DOCSEARCHbyAlgolia
Submit Docs

qshell Documentation

repository·master·Indexed 21 days ago

https://github.com/qiniu/qshell

qshell is a command-line tool for developers to interact with Qiniu's API services, including storage and CDN. It simplifies common tasks such as file uploads, downloads, and Base64 decoding. The tool includes specialized commands for bucket management (mkbucket, listbucket), asynchronous fetching (abfetch, acheck), and migrating data from AWS S3 (awsfetch, awslist) or Aliyun OSS (alilistbucket).

Tokens
86.4K
Snippets
393
Records
492
Agent score
75%

What's inside qshell

  1. Manage sandbox instances with the sandbox command

    master

    The sandbox command (alias sbx) is used to manage sandbox instances, templates, and injection rules. You can create, connect to, execute commands in, pause, resume, and terminate sandboxes, as well as view logs and resource metrics.

    Command Format:

    qshell sandbox <subcommand>
qshell sbx <subcommand>

    Available Subcommands:

    • list (ls): List sandboxes
    • create (cr): Create a sandbox and connect to the terminal
    • connect (cn): Connect to an existing sandbox terminal
    • kill (kl): Terminate a sandbox
    • pause (ps): Pause a sandbox
    • resume (rs): Resume a paused sandbox
    • exec (ex): Execute a command in the sandbox
    • logs (lg): View sandbox logs
    • metrics (mt): View sandbox resource metrics
    • template (tpl): Manage sandbox templates
    • injection-rule (ir): Manage sandbox injection rules
    qshell sandbox <subcommand>
qshell sbx <subcommand>

  2. Manage local accounts with the `user` command

    master
    The qshell user command is used to manage account information stored in the local database. You can use it to add new accounts, switch between active accounts, list all saved accounts, and remove specific accounts from your local machine.

  3. Configure the batchcopy input file format

    master

    The -i or --input-file option accepts a file where each line contains the mapping between source and destination keys.

    Formats

    1. Keep original name: Provide only the source key. <SrcKey>
    2. Rename file: Provide the source key and the destination key separated by a delimiter (default is a tab \t). <SrcKey><Sep><DestKey>

    Custom Delimiters

    If your filenames contain spaces or tabs, use the -F or --sep option to specify a custom delimiter (e.g., \t).

    // Example tocopy.txt content:
data/2015/02/01/bg.png\tbackground.png
data/2015/02/01/pig.jpg

  4. Configure abfetch input file formats

    master

    The -i or --input-file option specifies the list of resources to fetch. Each line represents one resource. If no file is provided, it reads from standard input.

    Supported line formats (fields separated by \t (tab)):

    1. [FileUrl]
    2. [FileUrl]\t[FileSize]
    3. [FileUrl]\t[FileSize]\t[Key]

    Notes:

    • FileSize is in bytes (B).
    • If no Key is provided, the path from the URL is used as the key (e.g., https://qiniu.com/a/b/c.png results in key a/b/c.png).
    • Providing FileSize allows the tool to estimate the required waiting time before checking for success, improving efficiency.
    https://qiniu.com/a.png\t1024\tb.png

  5. How file naming and directory structures work in qupload

    master

    By default, qupload preserves the relative path from the src_dir as a prefix for the file names in the bucket.

    Example: If src_dir is /Users/jemy/Temp/uptest and it contains a subdirectory 2017/01/03/cut1.png, the file will be uploaded to the bucket as 2017/01/03/cut1.png.

    If you want to ignore the directory structure and upload all files directly to the bucket root, use the ignore_dir: true parameter.

    {
  "src_dir" : "xxx",
  "bucket" : "xxx",
  "ignore_dir" : true
}

  6. Use `--enable-record` to resume interrupted batch rename tasks

    master

    The --enable-record option allows qshell to track the execution state of a task. If the command is interrupted or needs to be re-run, qshell will detect the state and skip tasks that have already been successfully completed.

    • --enable-record: Enables task state recording.
    • --record-redo-while-error: (Requires --enable-record) When re-running the command, this flag tells qshell to retry tasks that previously failed. If this is false (default), failed tasks are skipped during re-execution.

  7. Understand sandbox metrics output formats

    master

    The command supports two output modes:

    1. Pretty Mode (Default): Uses a single-line inline format with color-coded labels (cyan) for CPU:, Memory:, and Disk:. Example:

    [2024-01-01T00:00:00Z] CPU: 12.5% / 2 Cores | Memory: 256 MiB / 512 MiB | Disk: 100 MiB / 1024 MiB

    1. JSON Mode: Provides a structured JSON object suitable for programmatic parsing.

  8. Configure batchrestorear input files and separators

    master

    You can provide the list of files to restore via an input file or standard input.

    Input Formats:

    • Simple Key List: Each line contains a single <Key> (filename).
    • Key Mapping (Rename): Each line contains <Key><Separator><DestKey>, where <DestKey> is the target filename.

    Options:

    • -i / --input-file: Specifies the file containing the keys. If not provided, the command reads from stdin.
    • -F / --sep: Specifies a custom separator for the input file (e.g., a comma or pipe). The default is a tab character.
    qshell batchrestorear if-pbl 5 -i restorear.txt

  9. Check if files already exist in the bucket

    master

    To avoid redundant uploads, you can instruct qupload to check the bucket for existing files before uploading. This requires setting check_exists: true.

    When check_exists is enabled, the tool bypasses the local leveldb check and queries the Qiniu bucket directly. You can further refine this check:

    1. check_hash: true: Compares the MD5 hash of the local file with the hash of the file in the bucket. If they match, the upload is skipped. (Note: If both check_hash and check_size are true, check_hash takes precedence).
    2. check_size: true: If check_hash is not used, the tool compares the file size. If they match, the upload is skipped.

    Recommended Scenarios:

    • Multiple sync sources: Use check_exists: true to handle potential filename collisions from different sources.
    • Large files (e.g., Video): Use check_size: true instead of check_hash to avoid the overhead of reading large files into memory for hash calculation.

  10. Use the `func` command for Go template processing

    master

    The func command wraps Go language template functionality, allowing you to perform template-based string manipulations and data processing. It is particularly useful for validating templates used in qshell command callbacks (such as custom download paths in batch download operations) before deploying them.

    qshell passes internal data as a <ParamsJson> string, and you provide a <FuncTemplate> which acts like a callback function. The output of the template is treated as the return value for qshell.

    qshell func <ParamsJson> <FuncTemplate> [flags]

  11. How qupload handles interruptions and incremental uploads

    master

    To ensure efficiency and allow resuming interrupted tasks, qupload uses a local leveldb database to track successfully uploaded files.

    Resuming Uploads

    When a task is restarted, the tool checks the local database. If a file's record exists and its local last modification timestamp hasn't changed, the upload is skipped.

    Incremental Uploads

    To detect new files or modified files in the local directory, set rescan_local: true. By default (false), the tool uses the file list from the previous run to save time on large datasets. Setting it to true forces a fresh scan of the local directory.

    Overwriting Modified Files

    If a local file's timestamp has changed, qupload treats it as modified. To force the tool to overwrite the existing file in the bucket with the new version, set overwrite: true. Without this, the tool will log a warning and skip the file to prevent accidental data loss.

    {
  "overwrite" : true
}

  12. Use inline injection for API credentials and routing

    master

    Inline injection allows you to attach API keys or override base URLs for specific services within the sandbox. This is useful for providing credentials to LLMs or intercepting HTTP requests.

    Supported Types: openai, anthropic, gemini, qiniu, github, http.

    Parameters:

    • api-key: Used for openai, anthropic, gemini, qiniu, and github (as a token). The token is visible to the platform but not in plaintext inside the sandbox.
    • base-url: Overrides the default target address. Required for type=http. For type=qiniu, the default is api.qnaigc.com. For type=github, the host must be github.com or api.github.com.
    • headers: (Only for type=http) Semicolon-separated headers, e.g., Authorization=Bearer token;X-Env=prod.
    • if-headers: Condition for matching request headers (semicolon-separated).
    • if-queries: Condition for matching query parameters (semicolon-separated).
    # Inject OpenAI API key
qshell sbx cr my-template --inline-injection 'type=openai,api-key=sk-xxx'

# Inject HTTP interception rules
qshell sbx cr my-template --inline-injection 'type=http,base-url=https://api.example.com,headers=Authorization=Bearer token;X-Env=prod,if-headers=X-Scope=demo,if-queries=inject=true'

# Inject GitHub credentials
qshell sbx cr my-template --inline-injection 'type=github,api-key=ghp-xxx,base-url=https://api.github.com/repos/qiniu/*'