DOCSEARCHbyAlgolia
Submit Docs

git-xargs

repository·master·Indexed 22 days ago

https://github.com/gruntwork-io/git-xargs

A high-performance CLI tool that automates bulk updates across multiple GitHub repositories. It runs commands or scripts in parallel using Go routines and automatically manages the workflow of cloning, committing changes, and opening pull requests.

Tokens
4.7K
Snippets
17
Records
24
Agent score
75%

What's inside git-xargs

  1. Overview of git-xargs

    master

    git-xargs is a CLI tool designed to perform updates across multiple GitHub repositories simultaneously. It automates the workflow of cloning repositories, executing a specified command or script, committing changes, and opening pull requests for each repository. It uses Go routines to process repositories in parallel for high performance.

    Common use cases include:

    • Adding a new file to all repositories in an organization.
    • Performing search-and-replace operations across many repos.
    • Upgrading syntax (e.g., Terraform modules) across a fleet of repositories.
    • Modifying CI/CD configuration files in place.

  2. How git-xargs works

    master

    The git-xargs tool automates repository maintenance by performing the following lifecycle for each targeted repository:

    1. Cloning: Clones the repository to your local /tmp/ directory. The local directory name is a combination of the repo name and a random number (e.g., terraform-aws-module-security3978298) to facilitate local debugging.
    2. Branching: Checks out a local branch specified by the --branch-name flag.
    3. Execution: Runs all selected scripts against the repository.
    4. Committing: Commits any changes made by the scripts using a commit message optionally specified via --commit-message.
    5. Pushing: Pushes the local branch to the repository's remote.
    6. Pull Request: Calls the GitHub API to open a pull request. You can customize the title and description using --pull-request-title and --pull-request-description. Use --skip-pull-requests to prevent this step.
    7. Reporting: Prints a detailed summary to STDOUT, including links to successful pull requests and detailed error reports for any failures (cloning errors, script errors, etc.).

  3. Understand branch and file staging behavior

    master

    Branching

    • The --branch-name (-b) flag is required.
    • If the branch exists on the remote, git-xargs pulls the latest changes before running your command.
    • If the branch is new, it is created locally and pushed after your changes are committed.
    • Pull requests are opened against the repository's default branch unless you specify --base-branch-name (which applies to all targeted repositories).

    File Staging

    git-xargs automatically finds and stages all new files and modified existing files within the repository before committing. Any changes made by your script will be included in the commit.

  4. Install git-xargs via Go

    master

    You can install git-xargs using the Go toolchain. Ensure Go is installed on your system first.

    # Install the latest release
go install github.com/gruntwork-io/git-xargs@latest

# Install a specific version
go install github.com/gruntwork-io/git-xargs@v0.0.5

# For Go 1.16 or earlier
go get github.com/gruntwork-io/git-xargs

  6. Target repositories with git-xargs

    master

    git-xargs supports four methods for selecting target repositories. It processes them in the order listed below; the first method found is used, and subsequent methods are ignored.

    1. GitHub Organization Lookup: Use --github-org <org-name> to target every repository in a specific GitHub organization.
    2. Flat File: Use --repos <path-to-file> to provide a file containing one repository per line in the format <github-organization>/<repo-name>. (Commas and extra spaces are ignored).
    3. Command Line Arguments: Use one or more --repo <org/repo> flags to specify individual repositories.
    4. Stdin: Pipe repository names into the command via stdin, separated by whitespace or newlines.
    # Option 1: GitHub Org
git-xargs --github-org my-org "$(pwd)/script.sh"

# Option 2: Flat File
git-xargs --repos data/repos.txt "$(pwd)/script.sh"

# Option 3: CLI Args
git-xargs --repo org/repo1 --repo org/repo2 "$(pwd)/script.sh"

# Option 4: Stdin
echo "org/repo1 org/repo2" | git-xargs --commit-message "msg" "$(pwd)/script.sh"

  7. Configure authentication for git-xargs

    master

    To interact with GitHub, you must provide an authentication token.

    GitHub.com

    Export a GitHub Personal Access Token to the GITHUB_OAUTH_TOKEN environment variable:

    export GITHUB_OAUTH_TOKEN=<your-secret-github-oauth-token>

    GitHub Enterprise

    If you are using a GitHub Enterprise server, set the GITHUB_HOSTNAME environment variable to your enterprise host:

    export GITHUB_HOSTNAME=<your-ghe-hostname.your-domain.com>
    export GITHUB_OAUTH_TOKEN=<your-secret-github-oauth-token>
export GITHUB_HOSTNAME=<your-ghe-hostname.your-domain.com>

  8. Install git-xargs via published binaries

    master
    1. Download the binary: Visit the releases page and download the version appropriate for your platform.
    2. Move to PATH: Save the binary to a directory in your PATH, such as /usr/local/bin/git-xargs.
    3. Set permissions: Ensure the binary is executable (e.g., chmod u+x /usr/local/bin/git-xargs).
    4. Verify: Run git-xargs --version to confirm the installation.
    chmod u+x /usr/local/bin/git-xargs
git-xargs --version

  11. Debug runtime errors with --loglevel DEBUG

    master

    By default, git-xargs uses an INFO log level and conceals runtime errors. To see detailed error output from your scripts or commands, pass the --loglevel DEBUG flag.

    git-xargs --loglevel DEBUG \
	--repo zack-test-org/terraform-aws-eks \
	--branch-name master \
	--commit-message "add blank file" \
	--skip-pull-requests touch foo.txt

  12. Best practices for writing scripts for git-xargs

    master

    When preparing scripts to be executed by git-xargs, follow these patterns:

    Single-repo focus

    Write your script as if it is operating on a single repository. git-xargs handles the orchestration of running that script once per targeted repository.

    Prerequisite management

    git-xargs does not automatically check for required binaries. It is the responsibility of the script author to ensure prerequisites are met. It is recommended to use patterns like the Gruntwork bash-commons assert_is_installed to verify required tools are present.

    Batching for safety

    To minimize risk when running ambitious scripts, break your target repositories into smaller batches (e.g., batch1.txt, batch2.txt). Start with a single repository or a very small batch to verify the script and the resulting pull request before expanding to larger batches.