The pkg.go.dev system is composed of three primary high-level components that work together to serve Go package documentation:
- Frontend: An HTTP server that serves user-facing web pages. It retrieves module and package information from the database and uses Postgres text-search features to handle search queries.
- Worker: A stateless HTTP server responsible for data ingestion. It downloads new modules (via the Go Module Proxy), processes them (extracting READMEs, licenses, and documentation), and populates the database. It also computes importer counts and populates the
search_documents table. - Database: A Postgres database (managed via Google Cloud SQL) that stores all information served by the frontend.
To manage processing loads, the worker uses a Google Cloud Tasks queue to process modules at a controlled rate, and Google Cloud Scheduler to trigger periodic activities like polling the index or updating importer counts.