Cortex Documentation

Cloudflare's Email Protection feature hides email addresses on web pages to prevent malicious bots from harvesting them. When a user visits a protected page, they see an interstitial requiring JavaScript to be enabled. Once JavaScript is enabled, the obfuscated email address is decoded client-side. This protection is part of Cloudflare's Scrape Shield tools.

By maintaining one queue per tenant, a low demand tenant has the same opportunity to have a query serviced as a high demand tenant. This fairness only applies when queries are actually queued in the query frontend, which becomes more common as query sharding becomes more aggressive.

The optional query frontend provides querier API endpoints and accelerates reads by queuing queries, splitting large queries, and caching results. Queriers pull jobs from the frontend's queue, execute them, and return results for aggregation. Configure queriers with -querier.frontend-address to connect. Query frontends are stateless; two replicas suffice for most deployments. Can also proxy to other Prometheus-compatible services using -frontend.downstream-url.

The alertmanager is an optional service accepting alert notifications from the ruler, deduplicating and grouping them, and routing to notification channels (email, PagerDuty, OpsGenie). Built on Prometheus Alertmanager with multi-tenancy support. Requires a database for per-tenant configuration. Alertmanager is semi-stateful, persisting silences and active alerts to disk; simultaneous failure of all nodes results in data loss.

The Cortex community has published numerous blog posts covering architecture, operations, and scaling. Key articles include: 'How AWS and Grafana Labs are scaling Cortex for the cloud' (Dec 2020), 'How to switch Cortex from chunks to blocks storage' (Oct 2020), 'Cortex blocks storage for running Prometheus at scale' (Oct 2020 GA release), 'Scaling Prometheus: pushing Cortex blocks storage to its limit' (Aug 2020), 'How blocks storage reduces operational complexity' (Jul 2020), 'Cortex zone-aware replication' (Mar 2020), 'Using gossip to improve Cortex and Loki availability' (Mar 2020), 'The Future of Cortex: Into the Next Decade' (Jan 2020), and earlier articles on Prometheus scalability (Feb 2019), running Cortex in production (Feb 2019), and the original 'What is Cortex?' introduction (Sep 2018).

When running blocks storage, two additional Cortex services are required beyond the general architecture: store-gateway queries blocks and is used by the querier at query time. Compactor merges and deduplicates smaller blocks into larger ones to reduce stored blocks and improve query efficiency; it also keeps the bucket index updated. The alertmanager and ruler components can use object storage for configurations and rules, but require a separate bucket from blocks storage to avoid compactor issues.

The Cortex Alertmanager proposal aims to scale from ~2000 tenants to 10x that capacity by making the service horizontally scalable. Currently limited to 3 replicas for HA using gossip-based eventual consistency. The design preserves: no external downtime on replica crash, eventually consistent alert view favoring availability over consistency, and zero-downtime scaling/rollouts. The proposal divides the solution into four areas: Routing & Sharding, Persistence & State, Replication & Consistency, and Architecture.

The compactor is bounded by a 64GB index size limit. With unique timeseries, compaction doesn't reduce index size, and parallelism by time interval alone is insufficient for tenants with hundreds of millions of timeseries. The compactor can compact up to 400M timeseries within 12 hours before hitting the 64GB limit. The solution must handle the 64GB index limit and reduce overall compaction time by downloading data in smaller batches and reducing compaction duration. Currently, compaction time includes downloading blocks, merging timeseries, writing to disk, and uploading to object storage.

Query priority enables classifying queries by attributes (regex patterns, time ranges, API types, user agents, dashboard UIDs) and reserving dedicated querier capacity for each priority level. This prevents resource starvation where expensive analytical queries starve critical dashboard queries. Organizations can offer SLA differentiation: Tier 1 for real-time dashboards/alerts, Tier 2 for business intelligence, Tier 3 for ad-hoc exploration.

Cortex documentation is organized into: Getting Started (Single Binary and Microservices modes), Architecture, Blocks Storage (with components: Querier, Store-gateway, Compactor, and production tips), Configuration (file format and arguments), Guides (Kubernetes, gossip ring, alertmanager, auth, capacity planning, ruler sharding, HA pairs, TLS, security, shuffle sharding, tracing, OpenTelemetry, zone replication, and more), HTTP API, Operations (Query Auditor, Query Tee, request mirroring), Case Studies, and Proposals (including auth gateway, blocks storage, cross-tenant federation, parquet storage, and tenant retention).

Cortex exposes an HTTP API for pushing and querying time series data, and operating the cluster itself. API endpoints are grouped by service and behave differently depending on deployment mode: In microservices mode, each service exposes its own endpoints. In single-binary mode, the Cortex process exposes all API endpoints for the services running internally.