pkgsite

repository·master·Indexed 23 days ago

https://github.com/golang/pkgsite

The source code for pkg.go.dev, featuring a documentation server for generating and viewing Go package documentation locally, a REST API (v1beta), and the pkgsite-cli tool for querying package discovery, metadata, version listings, and vulnerability reports.

Tokens
26.2K
Snippets
57
Records
142
Agent score
74%

What's inside pkgsite

  1. Use pkgsite-cli for package discovery and metadata

    master

    The pkgsite-cli is a lightweight client for the pkg.go.dev API. While go doc is used for reading documentation of local code, pkgsite-cli should be used for package discovery and looking up metadata that local tools cannot access, such as:

    • Version listings
    • Vulnerability reports
    • Reverse dependencies
    • Licenses
    • Search results for packages not yet downloaded
  2. Understand Cloud Build deployment workflows

    master

    Deployments are orchestrated via Cloud Build triggers. There are two primary deployment configurations:

    • deploy-env.yaml: Deploys the config, worker, and frontend components for a single environment. The specific environment is configurable via the Cloud Build trigger execution page.
    • deploy.yaml: Executes a full pipeline that deploys to staging, runs end-to-end (e2e) tests, and—if tests pass—deploys to production.
  3. How pkg.go.dev is architected

    master

    The pkg.go.dev system is composed of three primary high-level components that work together to serve Go package documentation:

    1. Frontend: An HTTP server that serves user-facing web pages. It retrieves module and package information from the database and uses Postgres text-search features to handle search queries.
    2. Worker: A stateless HTTP server responsible for data ingestion. It downloads new modules (via the Go Module Proxy), processes them (extracting READMEs, licenses, and documentation), and populates the database. It also computes importer counts and populates the search_documents table.
    3. Database: A Postgres database (managed via Google Cloud SQL) that stores all information served by the frontend.

    To manage processing loads, the worker uses a Google Cloud Tasks queue to process modules at a controlled rate, and Google Cloud Scheduler to trigger periodic activities like polling the index or updating importer counts.

  4. Structure the documentation outline tree

    master

    The documentation outline is implemented as a nested HTML list structure using the go-Tree and js-tree classes. To create a hierarchical navigation tree, use a <ul> element with role="tree" containing <li> elements. Each <li> can contain an <a> link for the current level and a nested <ul> to represent sub-levels.

    <ul class="go-Tree js-tree" role="tree">
      <li>
        <a href="#one">Level 1</a>
        <ul>
          <li>
            <a href="#two-one">Level 2-1</a>
          </li>
        </ul>
      </li>
    </ul>
  5. Resolve ambiguous package paths in pkgsite-cli

    master
    Unlike go mod tidy or the pkg.go.dev web interface, pkgsite-cli does not automatically use the "longest module path" rule to resolve ambiguous paths. If a package path exists in multiple modules, the API will return an error and a list of candidates. To resolve this, use the -module flag to explicitly specify the correct module path.
  6. Run frontend screentests

    master
    In addition to Go-based integration tests, the frontend uses image snapshot tests to ensure visual consistency. These tests are located in the tests/screentest/ directory. If a test fails, it will generate diffs for manual inspection.
  7. Run database tests with GO_DISCOVERY_TESTDB

    master
    By default, running go test ./... will skip database tests if a PostgreSQL instance is not running. To force the execution of these tests, set the GO_DISCOVERY_TESTDB environment variable to true.
  8. Test and lint frontend assets

    master

    Frontend development uses npm for testing and linting. These tools are used for type checking, unit testing, and style enforcement.

    • Testing: Runs the TypeScript type checker and unit tests.
    • Linting: Runs stylelint and eslint on CSS and TS files in content/static. You can use the --fix flag to automatically correct some errors.

    If you do not have npm installed locally, you can run these commands via Docker by prefixing them with ./all.bash.

  9. Install and run the pkgsite documentation server

    master

    The pkgsite program extracts and generates documentation for Go projects locally. You can install it using go install and run it against a local project directory. Using the -open flag will automatically open the documentation in your web browser.

    $ go install golang.org/x/pkgsite/cmd/pkgsite@latest
    $ cd myproject
    $ pkgsite -open .