DOCSEARCHbyAlgolia
Submit Docs

ghorg

repository·master·Indexed 24 days ago

https://github.com/gabrie30/ghorg

A CLI tool for bulk-cloning repositories from SCM providers including GitHub, GitLab, Bitbucket, Gitea, Codeberg, and Sourcehut. Optimized for local codebase searching, backups, and auditing, ghorg allows users to clone all repositories for an organization or specific user into a single directory. It supports advanced filtering via regex, target repository lists, and allowlist/denylist files (ghorgonly and ghorgignore), as well as mirror backups and wiki cloning.

Tokens
29.6K
Snippets
73
Records
187
Agent score
84%

What's inside ghorg

  1. What is ghorg and when to use it

    master

    ghorg (pronounced [gore-guh]) is a CLI tool used to quickly clone all repositories belonging to an organization or a specific user into a single directory.

    Common use cases include:

    • Searching an organization's or user's codebase using tools like ack, silver searcher, or grep.
    • Bash scripting.
    • Creating backups of repositories.
    • Onboarding new team members by cloning all relevant team repositories.
    • Performing security or code audits.

    Important Behavior Note: By default, if a repository already exists in the target directory, ghorg will perform a git pull and a git clean. This means any local changes in that directory will be overwritten by the remote state. To prevent losing local changes, either rename the directory or use the --no-clean flag in future runs.

  2. Clone all repositories from an organization or user

    master
    ghorg is a tool designed to quickly clone all repositories belonging to an entire organization or a specific user into a single directory. It supports multiple providers including GitHub, GitLab, Bitbucket, and others.

  3. Supported SCM Providers in ghorg

    master

    ghorg supports the following providers:

    • GitHub (Self Hosted & Cloud)
    • GitLab (Self Hosted & Cloud)
    • Bitbucket (Cloud & Self-hosted Server)
    • Gitea (Self Hosted Only)
    • Codeberg (Cloud & Self Hosted Forgejo)
    • Sourcehut (Limited Features)

  4. Preserve SCM hostname in directory structure

    master

    By default, ghorg clones repositories into $HOME/ghorg/<org>. If you use the --preserve-scm-hostname flag, ghorg will create an additional top-level folder named after the SCM hostname (e.g., github.com/ for GitHub Cloud, or your Enterprise URL).

    This is useful for organizing clones from multiple different providers or instances under a single root.

    Example (GitHub Cloud):

  5. Use `ghorgignore` for pattern-based denylisting

    master

    The ghorgignore mechanism automatically skips any repository whose clone URL contains a substring listed in a denylist file.

    • Default Location: If a file exists at $HOME/.config/ghorg/ghorgignore, it is used automatically.
    • Custom Location: Use --ghorgignore-path or set GHORG_IGNORE_PATH to specify a different file.
    • Matching: It performs a plain substring check against the full clone URL.
    • Best Practice: Make entries as specific as possible (e.g., use a full URL like https://github.com/user/repo.git instead of a short fragment) to avoid unintentional matches.

  6. How GitLab directory structures are managed in ghorg

    master

    GitLab's organization of projects into groups and subgroups can be handled in several ways using ghorg:

    1. --preserve-dir: When this flag is used, ghorg mirrors the nested directory structure of groups/subgroups/projects locally. This prevents name collisions. If omitted, all projects are cloned into a single flat directory. If collisions occur without this flag, the group/subgroup name is prepended to the project name.
    2. --output-dir: Overrides the default folder name. By default, ghorg uses the instance hostname (for all-groups/all-users) or the group name. When cloning a subgroup with --preserve-dir, it preserves the parent groups.
    3. --preserve-scm-hostname: Always creates a top-level folder named after the instance hostname (e.g., gitlab.com/ or the value of GHORG_SCM_BASE_URL) inside your target directory.
    4. Group Paths: Always use the URL-friendly path (e.g., my-group) rather than the display name (e.g., my group) when specifying targets.

  7. Important considerations for Codeberg users

    master

    When using ghorg with Codeberg, keep the following in mind:

    • Token Scopes: Ensure your token has read:organization and read:repository permissions.
    • Token Input: The --token flag accepts either a raw string or a path to a file (e.g., --token=~/.config/ghorg/codeberg-token.txt).
    • Overwrite Behavior: Running the same clone command a second time performs a git pull and git clean, which will overwrite local changes. To prevent this, use --no-clean or --protect-local.
    • Hostname Preservation: Use the --preserve-scm-hostname flag to create a top-level directory named after the instance hostname (e.g., codeberg.org) inside your clone path. This is useful when cloning from multiple providers.
    • Etiquette/Rate Limiting: Codeberg is a non-profit. To be respectful of their resources, consider lowering the --concurrency (default is 25) or adding --clone-delay-seconds=1 when cloning large organizations.

  8. Important considerations for Gitea cloning

    master

    When using ghorg with Gitea, keep the following in mind:

    • Token Input: The --token flag accepts either a raw token string or a path to a file containing the token (e.g., --token=~/.config/ghorg/gitea-token.txt).
    • Idempotency and Local Changes: Running the same clone command a second time will perform a git pull and git clean on every repository, which will overwrite any local changes. To prevent this, use --no-clean or --protect-local.
    • Self-Hosting: Gitea is self-hosted only; the --base-url pointing to your instance is always required.

  9. Use `--target-repos-path` for an explicit list of repositories

    master

    Use --target-repos-path (or the GHORG_TARGET_REPOS_PATH env var) to clone only a specific, known set of repositories. You must provide a path to a plain-text file containing one repository name per line.

    Key behaviors:

    • Matching is done against the repo name only (basename of the clone URL with .git stripped).
    • Matching is case-insensitive and exact; partial names will not match.
    • If a name in the file does not exist on the organization/user, it is reported in the clone summary.
    • When --clone-wiki is enabled, it also matches the corresponding .wiki repository.
    • GitLab snippets are matched against their parent repository name.

  10. Use `ghorgonly` for pattern-based allowlisting

    master

    The ghorgonly mechanism allows you to automatically clone repositories whose URLs contain specific substrings. This is useful for dynamic, pattern-based subsets (e.g., everything in an infra- namespace).

    • Default Location: If a file exists at $HOME/.config/ghorg/ghorgonly, it is used automatically.
    • Custom Location: Use --ghorgonly-path or set GHORG_ONLY_PATH to specify a different file.
    • Matching: It performs a plain substring check against the full clone URL.
    • Ordering: It is applied after --target-repos-path and before ghorgignore.

  12. Important Gitea usage considerations

    master

    When using ghorg with Gitea, keep the following in mind:

    • Base URL: Because Gitea is self-hosted, the --base-url flag is always required.
    • Token Input: The --token flag accepts either the raw token string or a path to a file containing the token (e.g., --token=~/.config/ghorg/gitea-token.txt).
    • Local Changes: Running the same clone command a second time will perform a git pull and git clean on every repository, which overwrites local changes. To prevent this, use --no-clean or --protect-local.
    • Configuration: All command line arguments can be permanently set in your $HOME/.config/ghorg/conf.yaml.