OpenCNPJ Documentation

The ETL includes an IDataIntegration interface for sub-modules. The Portal da Transparência integration provides several datasets:

Current Photographs: favorecidos_pj, ceis, cepim, cnep, acordos_leniencia, convenios, and emendas_parlamentares.

Segmented Historical Datasets: licitacoes, contratos, renuncias_fiscais, notas_fiscais, and emendas_documentos.

Data Availability (Post-2013):

• licitacoes and contratos: Since 2013.
• renuncias_fiscais: Since 2015.
• notas_fiscais: Since November 2019.
• emendas_documentos: Since 2014.

Segmentation Logic: Historical datasets use segments. Closed years are consolidated into a YYYY segment, while the current year uses YYYY-MM segments. The API uses a binary routing index to find which segments contain a specific CNPJ and aggregates them in the response.

The Worker uses two layers of caching: Cloudflare Workers Cache (first level) and the Cache API (second level). To maximize cache hits and avoid unnecessary Worker invocations, follow these best practices:

1. Use unmasked CNPJs: Prefer numeric formats (e.g., 12345678000195) over masked formats (e.g., 12.345.678/0001-95).
2. Canonical Dataset Order: Always provide the datasets query parameter in a consistent, canonical order: receita,cno,rntrc.

The full URL, including the query string, composes the cache key. The Worker also maintains a hot in-memory cache within the isolate for recently read shard indices (limited to 32 MiB or a specific count).

The query interface is a React + TypeScript SPA located in src/Page. It is entirely static.

Commands:

# Install dependencies
npm install

# Start local development server
npm run dev

# Build the production static version
npm run build

Note: When deploying, use the generated files in src/Page/dist. Do not use the source src/Page/index.html used by Vite for production.

cd src/Page
npm install
npm run build

Navigate to src/ETL/Processor to execute the pipeline using the .NET CLI.

Commands:

# Run the pipeline for the most recent month available on WebDAV
dotnet run pipeline

# Run the pipeline for a specific month (YYYY-MM)
dotnet run pipeline -m 2023-01

# Force a specific remote release ID
dotnet run pipeline --release-id abc123...

# Run and clean up local temporary artifacts (downloads, extracted CSVs) upon success
dotnet run pipeline --cleanup-on-success

cd src/ETL/Processor
dotnet run pipeline

To run the OpenCNPJ pipeline and services, ensure the following are installed:

• .NET SDK 10.0+ (for the ETL Processor)
• rclone: Must be installed and authenticated to your target storage (e.g., Backblaze, R2, S3, Azure Storage).
• bq CLI: Required only if BigQuery.Enabled=true in your configuration.
• Disk Space & Connection: High disk availability and a stable connection are required, as the initial execution can take several days.

The src/script/deploy.sh script orchestrates the entire release process, including running the ETL, validating BigQuery (if enabled), copying assets to the Worker, and deploying the Cloudflare Worker.

Deployment Workflow:

1. Runs the ETL with a versioned release.
2. Validates BigQuery configuration and access via bq show.
3. Copies info.json and *.index.bin to src/Worker/assets.
4. Runs npm test on the Worker.
5. Executes npx wrangler deploy.
6. Validates the live /info endpoint and CNPJ lookups.
7. Removes the old release from the bucket only after successful validation.

To run the test suite for the Worker, navigate to the src/Worker directory and execute npm test.

cd src/Worker && npm test

To deploy the Worker, follow these steps:

1. Ensure you have adjusted the bucket_name in wrangler.toml to match your actual R2 bucket name.
2. Run the deployment script from the repository root:

./src/scripts/deploy.sh

Optional: You can pass a --base-url flag to the script if you want to validate and clean up an old release using a specific domain.

bash
./src/scripts/deploy.sh
# Or with an optional base-url
./src/scripts/deploy.sh --base-url https://example.com

The ETL behavior is controlled via src/ETL/Processor/config.json. You can adjust:

• Local folder paths
• Storage destinations
• Memory limits
• Parallelism settings
• rclone configurations for your specific storage provider.

Note: The downloader for Receita data currently uses WebDAV via the SERPRO+/Nextcloud public share.

The opencnpj-worker package provides an in-memory caching mechanism for frequently accessed data to improve performance. The cache handles three main types of data: BinaryShardIndex, SegmentRoutingIndex, and string-based chunks.

Caches are managed using a Least Recently Used (LRU) eviction policy based on both the number of entries and the total byte size. Each entry is also subject to a Time-To-Live (TTL) expiration.

Cache Types

• Hot Index Cache: Stores BinaryShardIndex objects.
• Hot Routing Index Cache: Stores SegmentRoutingIndex objects.
• Hot Chunk Cache: Stores string values (chunks).
• Runtime Info Cache: Stores RuntimeInfo metadata.