GitHub Models CLI Extension

The generate command is used for test generation and is based on the PromptPex framework. It allows you to automate the generation of prompts and test cases for evaluating model performance.

For detailed technical specifications and the full reference of how test generation works, refer to the PromptPex documentation.

Before submitting a pull request, run the following commands to ensure code quality:

• make check: Runs lint tests, unit tests, and all other Go-related checks.
• make test: Runs tests.
• make fmt: Automatically formats the code.
• make vet: Finds suspicious constructs.
• make tidy: Keeps dependencies up-to-date.

Integration tests for the gh-models CLI extension use the actual compiled binary rather than mocked clients. They verify the basic functionality of commands like list, run, view, and eval.

Tests are designed to be environment-aware: they will skip tests requiring live endpoints if GitHub authentication is not detected, allowing the suite to run partially even without credentials.

# From the integration directory
go test -v

To develop with gh-models, ensure you have the following prerequisites installed:

1. gh CLI: Must be installed and available in your PATH. You must also be authenticated via gh auth.
2. Go: A minimum version of 1.22 is required.

go version
go version go1.22.x <arch>

Before running integration tests, you must build the gh-models binary. For full test coverage (including live endpoint testing), you should also be authenticated via the GitHub CLI.

1. Build the binary: Run the build script from the repository root.
2. Authenticate (Optional): Use gh auth login to enable tests that require live GitHub endpoints.

# 1. Build the binary
cd ..
script/build

# 2. (Optional) Authenticate for full testing
gh auth login

To use the GitHub Models service from your terminal, you must first have the gh CLI installed and authenticated via gh auth. You can install the extension using the following command:

gh extension install https://github.com/github/gh-models

You can build the project using make build or the script/build script. Once built, you can execute the binary locally to test commands like list.

make build
./gh-models list

Integration tests are located in the integration/ directory and test the compiled binary against live endpoints. Note that tests requiring authentication will be automatically skipped if no GitHub token is available.

To run them:

1. Build the binary first.
2. Navigate to the integration directory.
3. Run go test -v.

make build
cd integration
go test -v

Because gh extension install and gh extension upgrade pull the latest release rather than the latest commit, you must create a new Git tag to publish changes. Tagging triggers the release action, which performs the production build.

git tag v0.0.x main
git push origin tag v0.0.x

When adding new commands or features to gh-models, follow these guidelines for creating integration tests:

1. Pattern: Follow the existing pattern of checking for binary existence and checking for authentication status.
2. Authentication Awareness: Ensure tests skip live endpoint checks if authentication is unavailable so the suite doesn't fail in unauthenticated environments.
3. Assertions: Keep assertions minimal but meaningful. Focus on:
   • Successful command execution without errors.
   • Presence and correct formatting of help text.
   • Basic output format compliance.
   • Respecting authentication requirements.

If you have previously installed the gh models extension and want to update to the latest version, use the upgrade command:

gh extension upgrade github/gh-models