Skip to content

divbase-cli files

Download/upload/list files to/from the project's store on DivBase.

Usage:

$ divbase-cli files [OPTIONS] COMMAND [ARGS]...

Options:

  • --install-completion: Install completion for the current shell.
  • --show-completion: Show completion for the current shell, to copy it or customize the installation.
  • --help: Show this message and exit.

Commands:

  • ls: List files and folders in the project's...
  • tree: Display the project's file store in a tree...
  • info: Get detailed information about a specific...
  • download: Download files from the project's store on...
  • download-all: Download all files in the project's store...
  • stream: Stream a file's content to standard output.
  • upload: Upload files to your project's store on...
  • mkdir: Create one or more directories in your...
  • rmdir: Remove one or more empty directories from...
  • rm: Soft delete files from the project's store...
  • restore: Restore soft deleted files from the...

divbase-cli files ls

List files and folders in the project's DivBase store.

Folders are shown highlighted before files. Use --detailed / -l to also show file sizes and upload dates.

Examples:

  • List all files and folders in the project: divbase-cli files ls

  • List files and folders in the 'vcfs/' folder (trailing '/' shows folder contents): divbase-cli files ls vcfs/

  • List all files and folders whose name starts with 'sample': divbase-cli files ls sample

  • Detailed listing with sizes and dates: divbase-cli files ls --detailed

  • List all files including DivBase query results files (hidden by default): divbase-cli files ls --include-results-files

Usage:

$ divbase-cli files ls [OPTIONS] [PREFIX]

Arguments:

  • [PREFIX]: Optional prefix to filter on. Use a '/' to specify a folder. e.g. 'vcfs/'

Options:

  • -l, --detailed: Show a detailed view including the file size and upload date.
  • -t, --tsv: If set, will print the output in .TSV format for easier programmatic parsing.
  • -r, --include-results-files: If set, will also show DivBase query results files which are hidden by default.
  • -s, --show-deleted-files: Show the files and folders in the project that are currently soft deleted. These can be recovered within a certain time frame after deletion.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files tree

Display the project's file store in a tree like manner.

Examples:

  • Show the full tree: divbase-cli files tree

  • Show the tree starting from the 'vcfs/' folder: divbase-cli files tree vcfs/

Usage:

$ divbase-cli files tree [OPTIONS] [PREFIX]

Arguments:

  • [PREFIX]: Optional prefix to filter on. Use a '/' to specify a folder. e.g. 'vcfs/'

Options:

  • -r, --include-results-files: If set, will also show DivBase query results files which are hidden by default.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files info

Get detailed information about a specific file in the project's DivBase store.

This includes all versions of the file and whether the file is currently marked as soft deleted.

Examples:

  • Get information about a specific file in the project: divbase-cli files info my_file.vcf.gz

Usage:

$ divbase-cli files info [OPTIONS] FILE_NAME

Arguments:

  • FILE_NAME: Name of the file to get information about. Must be a file, not a folder (no trailing '/'). [required]

Options:

  • -t, --tsv: If set, will print the output in .TSV format for easier programmatic parsing.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files download

Download files from the project's store on DivBase.

Files can be specified as a space-separated list or via --file-list. Use a trailing '/' to download everything inside a folder. By default, any folder structure from the project store is recreated locally; use --flatten to download all files into a single directory instead.

Examples:

  • Download specific files: divbase-cli files download file1.vcf.gz file2.tsv

  • Download all files inside the 'vcfs/' folder (this includes subfolders and their contents): divbase-cli files download vcfs/

  • Download and flatten all files into one directory: divbase-cli files download vcfs/ --flatten --download-dir /path/to/save

  • Download a specific older version of a file (obtain version ID from 'divbase-cli files info'): divbase-cli files download "my_file.vcf.gz:VERSION_ID"

  • Mix latest and versioned files in one command: divbase-cli files download file1.vcf.gz "file2.tsv:VERSION_ID"

Usage:

$ divbase-cli files download [OPTIONS] [FILES]...

Arguments:

  • [FILES]...: Space separated list of files to download. Use a trailing '/' to download all files inside a folder, e.g. 'vcfs/'.

Options:

  • --file-list PATH: Text file with list of files to download.
  • -d, --download-dir TEXT: Directory to download the files to. If not provided, defaults to what you specified in your user config. If also not specified in your user config, downloads to the current directory. You can also specify "." to download to the current directory.
  • -f, --flatten: Download all files directly into the download directory, ignoring any folder paths. By default the folder structure in the project store is preserved when downloading.
  • -n, --dry-run: If set, will not actually download the files, just print what would be downloaded.
  • -nc, --disable-verify-checksums: Turn off checksum verification which is on by default. Checksum verification means all downloaded files are verified against their MD5 checksums. It is recommended to leave checksum verification enabled unless you have a specific reason to disable it.
  • -pv, --project-version TEXT: User defined version of the project's at which to download the files. If not provided, downloads the latest version of all selected files.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files download-all

Download all files in the project's store on DivBase.

DivBase query results files are excluded. Before the download starts you will be prompted to confirm. By default, the folder structure from the project store is recreated locally; use --flatten to download all files into a single directory instead.

Examples:

  • Download all files, preserving the folder structure from the project store: divbase-cli files download-all --download-dir /path/to/save

  • Resume an interrupted download (use the same --download-dir as the original run): divbase-cli files download-all --resume --download-dir /path/to/save

  • Download all files into a single flat directory: divbase-cli files download-all --flatten --download-dir /path/to/save

  • Preview what would be downloaded without downloading: divbase-cli files download-all --dry-run

Usage:

$ divbase-cli files download-all [OPTIONS]

Options:

  • -d, --download-dir TEXT: Directory to download the files to. If not provided, defaults to what you specified in your user config. If also not specified in your user config, downloads to the current directory. You can also specify "." to download to the current directory.
  • -f, --flatten: Download all files directly into the download directory, ignoring any folder paths. By default the folder structure in the project store is preserved when downloading.
  • -r, --resume: If set, will attempt to resume an interrupted download. Will check which files have already been fully downloaded (by checking if a file with the same name and checksum already exists in the download directory) and skip downloading those files again.
  • -n, --dry-run: If set, will not actually download the files, just print what would be downloaded.
  • -nc, --disable-verify-checksums: Turn off checksum verification which is on by default. Checksum verification means all downloaded files are verified against their MD5 checksums. It is recommended to leave checksum verification enabled unless you have a specific reason to disable it.
  • -pv, --project-version TEXT: User defined version of the project's at which to download the files. If not provided, downloads the latest version of all selected files.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files stream

Stream a file's content to standard output.

This allows you to pipe the output to other tools like 'less', 'head', 'zcat' and 'bcftools'.

Examples:

  • View a file: divbase-cli files stream my_file.tsv | less

  • View a gzipped file: divbase-cli files stream my_file.vcf.gz | zcat | less

  • Run a bcftools command (the "-" tells bcftools to read from standard input): divbase-cli files stream my_file.vcf.gz | bcftools view -h -

Usage:

$ divbase-cli files stream [OPTIONS] FILE_NAME

Arguments:

  • FILE_NAME: Name of the file you want to stream. [required]

Options:

  • --version-id TEXT: Specify this if you want to look at an older/specific version of the file. If not provided, the latest version of the file is used. To get a file's version ids, use the 'divbase-cli files info [FILE_NAME]' command.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files upload

Upload files to your project's store on DivBase.

By default only the file name is used as the destination key (directory paths are stripped). Use '--to' to place files inside a remote folder. Use '--recursive' with a '**' glob to preserve subdirectory structure relative to the glob root.

Examples:

  • Upload multiple files: divbase-cli files upload file1.vcf.gz file2.tsv

  • Upload all .vcf.gz files in the current directory into a remote folder: divbase-cli files upload "*.vcf.gz" --to experiment1/

  • Upload all files in a directory to the root of the project store: divbase-cli files upload "/path/to/data/*"

  • Upload recursively, preserving subdirectory structure (wrap glob in quotes to prevent shell expansion): divbase-cli files upload --recursive "/path/to/data/**"

  • Upload recursively into a remote folder: divbase-cli files upload --recursive "/path/to/data/**" --to experiment1/

  • Upload from a text file list (one file path per line): divbase-cli files upload --file-list files_to_upload.txt --to experiment1/

Usage:

$ divbase-cli files upload [OPTIONS] [FILES]...

Arguments:

  • [FILES]...: Space separated list of files or glob patterns to upload.

Options:

  • -l, --file-list PATH: Text file with list of files to upload.
  • -t, --to TEXT: Remote folder to upload into, e.g. 'vcfs/batch1/'. Folder does not need to exist before uploading.
  • -s, --skip-existing: If set, will skip already uploaded files, from the files you provided in the command. Already uploaded files are determined by checking if a file with the same name and MD5 checksum already exists in the project's store on DivBase.
  • -r, -R, --recursive: If set, recursively include subdirectories contents when uploading (i.e. '' is expanded). Without this flag, patterns only match files in the specified directory. Put your argument in quotes to prevent shell expansion of the glob before it gets to the CLI, e.g. files upload 'data//*.vcf.gz'
  • -n, --dry-run: If set, will show what files would be uploaded, but not actually upload them.
  • --disable-safe-mode: Turn off safe mode which is on by default. Safe mode adds 2 extra bits of security by first calculating the MD5 checksum of each file that you're about to upload:(1) Checks if any of the files you're about to upload already exist (by comparing name and checksum) and if so stops the upload process.(2) Sends the file's checksum when the file is uploaded so the server can verify the upload was successful (by calculating and comparing the checksums).It is recommended to leave safe mode enabled unless you have a specific reason to disable it.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files mkdir

Create one or more directories in your project store.

Note: directories are created automatically when uploading files with '--to', so you only need this command if you want to create the folder structure ahead of time.

Examples:

  • Create a single directory: divbase-cli files mkdir vcfs

  • Create a nested directory structure: divbase-cli files mkdir vcfs/batch1/metadata

  • Create multiple directories at once: divbase-cli files mkdir vcfs metadata results

Usage:

$ divbase-cli files mkdir [OPTIONS] DIRECTORIES...

Arguments:

  • DIRECTORIES...: Space separated list of directories to create. A trailing '/' is added automatically if omitted. [required]

Options:

  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files rmdir

Remove one or more empty directories from your project store.

All files inside the directory must be deleted first using 'divbase-cli files rm'. If a directory does not exist, it is treated as a successful deletion.

Examples:

  • Remove a single directory: divbase-cli files rmdir vcfs/

  • Remove multiple directories at once: divbase-cli files rmdir vcfs/ metadata/

Usage:

$ divbase-cli files rmdir [OPTIONS] DIRECTORIES...

Arguments:

  • DIRECTORIES...: Space separated list of directories to remove. The directory must be empty (remove files first with 'divbase-cli files rm'). [required]

Options:

  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files rm

Soft delete files from the project's store on DivBase

Note that deleting a non existent file will be treated as a successful deletion.

Examples: - Delete specific files: divbase-cli files rm file1.vcf.gz file2.tsv

  • Delete files from a text file list (one file name per line): divbase-cli files rm --file-list files_to_delete.txt

  • Run a dry run to see what files would be deleted without actually deleting them: divbase-cli files rm file1.vcf.gz file2.tsv --dry-run

Usage:

$ divbase-cli files rm [OPTIONS] [FILES]...

Arguments:

  • [FILES]...: Space separated list of files/objects in the project's store on DivBase to delete.

Options:

  • --file-list PATH: Text file with list of files to delete.
  • --dry-run: If set, will not actually delete the files, just print what would be deleted.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.

divbase-cli files restore

Restore soft deleted files from the project's store on DivBase

Attempts to restore a file that is not soft deleted will be considered successful and the file will remain live.

Examples: - Restore specific files: divbase-cli files restore file1.vcf.gz file2.tsv

  • Restore files from a text file list (one file name per line): divbase-cli files restore --file-list files_to_restore.txt

Usage:

$ divbase-cli files restore [OPTIONS] [FILES]...

Arguments:

  • [FILES]...: Space separated list of files/objects in the project's store on DivBase to restore.

Options:

  • --file-list PATH: Text file with list of files to restore.
  • -p, --project TEXT: Name of the DivBase project, if not provided uses the default in your DivBase config file
  • --help: Show this message and exit.