DOCSEARCHbyAlgolia
Submit Docs

MongoDB Tools Documentation

repository·master·Indexed 21 days ago

https://github.com/mongodb/mongo-tools

A collection of command-line utilities for MongoDB database administration, data movement, and real-time monitoring. Included tools are bsondump, mongoimport, mongoexport, mongodump, mongorestore, mongostat, mongofiles, and mongotop.

Tokens
2.4K
Snippets
5
Records
12
Agent score
77%

What's inside MongoDB Tools

  1. Overview of MongoDB Tools

    master

    MongoDB Tools is a suite of utilities for managing, monitoring, and interacting with MongoDB databases. The available tools include:

    • bsondump: Displays BSON files in a human-readable format.
    • mongoimport: Converts data from JSON, TSV, or CSV and inserts it into a collection.
    • mongoexport: Writes an existing collection to CSV or JSON format.
    • mongodump/mongorestore: Dumps MongoDB backups to disk in .BSON format or restores them to a live database.
    • mongostat: Monitors live MongoDB servers, replica sets, or sharded clusters.
    • mongofiles: Reads, writes, deletes, or updates files in GridFS.
    • mongotop: Monitors read/write activity on a MongoDB server.

  2. Security levels for TOOLS Jira tickets

    master

    Jira tickets under the TOOLS project have two primary security levels that determine visibility:

    1. Public (Default): Tickets are viewable by public users. These should be included in the CHANGELOG.md during a release.
    2. Mongo Internal: Tickets are restricted to internal users and cannot be viewed by external users. These must be excluded from the CHANGELOG.md when included in a release.

  3. Use mongodump_passthrough for resmoke testing

    master

    The mongodump_passthrough directory contains evergreen .yml files designed to support resmoke testing specifically for the mongodump and mongorestore toolchain. These files facilitate testing the passthrough capabilities of these tools.

    To use these files in a testing context, they should be included from mongo-tools/common.yml. When integrating, ensure that task or function names do not clash with those defined in common.yml.

  4. When to mark TOOLS tickets as `Mongo Internal`

    master

    To maintain security and privacy, certain types of tickets must be marked with the Mongo Internal security level:

    • HELP Ticket Investigations: Any TOOLS investigation ticket created from a HELP ticket must always be marked as Mongo Internal. This is because they often contain sensitive customer logs or internal links copied from the original HELP ticket.
    • Internal Process Changes: Tickets regarding changes to internal workflows (e.g., changes to the release process involving internal tools) should be marked as Mongo Internal.
    • Vulnerabilities (Pre-release): Vulnerabilities are created as Mongo Internal by default to prevent public disclosure before a fix is available.

  5. Handling Vulnerability ticket visibility

    master

    Vulnerabilities follow a specific lifecycle regarding their security level:

    1. Creation: Set to Mongo Internal by default.
    2. Resolution: Once the vulnerability is resolved (fixed or marked as a false positive), it remains internal until the fix is released.
    3. Public Disclosure: After the fix is included in a release, the release manager must:
      • Include the vulnerability in the CHANGELOG.md.
      • Update the ticket's Security Level to make it publicly viewable.

  6. Prepare mongo-tools for release

    master

    Once all tests for the new server version are passing, you must update the release configuration to unblock the server release process.

    Update the linuxRepoVersionsStable variable in release/release.go. Including the new version here triggers Barque (the DevProd service) to package the tools for various Linux repositories (such as Debian and RHEL).

    Note: This step is mandatory and must be completed before releasing the tools, as the release of any major/minor server version is blocked until a compatible version of the tools is released.

  7. Generate an SSDLC Compliance Report for a release

    master

    To create a compliance report for a specific release, follow these steps:

    1. Copy the template file and rename it to ssdlc-compliance-report.{{$tag}}.md.
    2. Replace all instances of the placeholder {{$tag}} with the actual release tag value.
    3. Delete the instructional text from the top of the document.
    4. Populate the sections with the relevant data from the following sources:
      • Release Creator: Identify the person who triggered the release by checking the CI run in the Evergreen branch project at the $release_project_url.
      • Third-Party Dependency Information: Reference the SBOM located at https://github.com/10gen/mongosync/blob/{{$tag}}/ssdlc/SBOM.{{$tag}}.bom.json.
      • Static Analysis Findings: Retrieve the SARIF report from the Evergreen logs under the gosec task output.
      • Signature Information: Refer to the data in the Papertrail service.
      • Security Testing/Assessment Reports: Request these from the Tools and Replicator team.
      • Known Vulnerabilities: Check the SBOM, the SARIF report, or individual Jira tickets in the release changelog.
    ssdlc-compliance-report.{{$tag}}.md

  8. Add support for a new MongoDB server version

    master

    To add support for a new MongoDB server version in mongo-tools, you must update the test matrix, adjust legacy JS shims if necessary, and prepare the release configuration. This process ensures that integration tests run against the new version and that the release pipeline unblocks the corresponding server release.

    1. Update the Test Matrix

    Update common.yml by adding a new task block for the target version. You should copy an existing version block and update the following fields:

    • The task name (e.g., integration-8.3)
    • The task tags (e.g., ["8.3"])
    • The mongo_version variable (use pre-release versions like 8.3.0-rc0 if the server is not yet released)
    • The load_libs_version variable (if present)

    Additionally, in the evergreen configuration:

    • Update the max_server_version variable in the global variables section.
    • Add the new version tag to each build variant. Note that for version 8.3, tags must be preceded by a dot (e.g., name: ".8.3").

    2. Update Server Download Logic

    Update the maxServerVersion variable in release/download/download_server.go. This allows the latest tests to correctly download and use the newest supported version.

    3. Adjust Load Libs (for legacy JS tests)

    - name: integration-8.3
  tags: ["8.3"]
  commands:
    - func: "fetch source"
    - command: expansions.update
    - func: "download mongod and shell"
      vars:
        mongo_version: "8.3.0-rc4"
    - func: "start mongod"
      vars:
        USE_TLS: "true"
    - func: "wait for mongod to be ready"
      vars:
        USE_TLS: "true"
    - func: "run make target"
      vars:
        target: build
    - func: "run make target"
      vars:
        target: test:integration -ssl=true ${testArgs}

  9. Shim legacy JS tests for new server versions

    master

    For legacy JavaScript tests running on server versions 8.1 and newer, you may need to provide shims for code removed from the MongoDB server repository.

    1. Locate the shim directory: test/shell_common/libs/.
    2. Create a new shim file named load_libs-<VERSION>.js (e.g., load_libs-8.3.js).
    3. You can base the new shim on the previous version's file. If tests fail due to missing server-side code, you may need to pull relevant code from the previous server version's source and include it in your shim.

  10. Build MongoDB Tools from source

    master

    To build the tools, you must clone the repository first; using go get directly will not work. The tools are built using Go version 1.15 (other versions are untested).

    Prerequisites:

    • Clone the repository.
    • Set the GOROOT environment variable to your Go root directory.

    Steps:

    1. Clone and enter the directory:
      git clone https://github.com/mongodb/mongo-tools
cd mongo-tools
    2. Set GOROOT (example path):
      export GOROOT=/usr/local/go
    3. Build all tools:
      ./make build
      The binaries will be placed in the bin directory.

    Build a subset of tools: Use the -pkgs option to build specific tools only. For example, to build only mongodump and mongorestore:

    ./make build -pkgs=mongodump,mongorestore
    git clone https://github.com/mongodb/mongo-tools
cd mongo-tools
export GOROOT=/usr/local/go
./make build

  11. Manage dependencies using go mod

    master

    Starting with version 100.3.1, MongoDB Tools uses go mod for dependency management. Dependencies are listed in go.mod and vendored in the vendor directory.

    To add or update a dependency:

    go mod edit -require=<package>@<version>

    To remove a dependency:

    go mod edit -droprequire=<package>

    To apply changes: After modifying go.mod, you must reconstruct the vendor directory:

    go mod vendor -v

    To sync go.mod with source code: Run go mod tidy -v to ensure the go.mod file matches the actual source code usage.

    go mod edit -require=<package>@<version>
go mod vendor -v
go mod tidy -v

  12. Fix 'killed' error when running built binaries on macOS

    master

    If you are running built binaries on macOS and the process is killed immediately (e.g., zsh: killed ./bin/mongodump --help), you must sign the binary to run it.

    Option 1: Sign the binary Use codesign to force a signature on the binary:

    codesign --force --sign - bin/mongodump

    Option 2: Update Security Policy (macOS Sonoma) Instead of signing, you can allow developer tools to run without signing by navigating to: System Settings > Privacy & Security > Developer Tools