DOCSEARCHbyAlgolia
Submit Docs

Cortex Documentation

repository·master·Indexed 26 days ago

https://github.com/cortexproject/cortex

Cortex is a scalable, multi-tenant, long-term storage backend for Prometheus and OpenTelemetry metrics. It is designed to handle massive amounts of time-series data using cloud storage such as S3, GCS, Swift, and Microsoft Azure. The project features a modular architecture, a service lifecycle model for managing components, and a CLI for configuration and observability.

Tokens
130.5K
Snippets
160
Records
724
Agent score
90%

What's inside Cortex

  1. Overview of the Crossplane Provider for Cortex

    master

    The Crossplane provider for Cortex enables declarative management of Cortex tenant configurations using Kubernetes Custom Resources (CRDs). It is designed to replace manual API calls and CLI-based configurations with a GitOps-friendly approach, addressing issues like configuration drift, security, and tenant isolation.

    Key capabilities include:

    • Declarative Management: Manage recording rules, alerting rules, and Alertmanager configurations via Kubernetes.
    • Secure Credential Management: Uses Kubernetes secrets for authentication.
    • Multi-tenant Support: Provides isolation and governance controls for multiple tenants.
    • GitOps Integration: Compatible with standard Kubernetes workflows for configuration deployment.

  2. Overview of Cortex

    master
    Cortex is a horizontally scalable, highly available, and multi-tenant long-term storage solution designed for Prometheus and OpenTelemetry Metrics. It allows for high-throughput metric ingestion and storage across a cluster of machines and supports multiple independent tenants within a single cluster. For long-term storage, it integrates with cloud providers including S3, GCS, Swift, and Microsoft Azure.

  3. Overview of the Per-Tenant Cardinality API

    master

    The Per-Tenant Cardinality API is a diagnostic tool designed to help operators identify high-cardinality series within a Cortex tenant. It provides visibility into which metrics, labels, and label-value pairs contribute most to the series count, helping to debug resource usage and series limit issues.

    Key capabilities include:

    • Dual Data Source Support: Accesses both in-memory TSDB head data (from ingesters) and compacted blocks (from long-term object storage via store gateways).
    • Distributed Aggregation: Aggregates statistics across all ingesters or store gateways relevant to the requesting tenant.
    • Replication Awareness: Correctly accounts for the replication factor when summing series counts from ingesters to avoid over-counting.
    • Multi-tenant Security: Ensures tenants can only access cardinality statistics for their own data.

  4. Overview of Cortex HTTP API Endpoints

    master

    Cortex exposes an HTTP API for pushing and querying time series data, and operating the cluster. Endpoints are available in both microservices mode (each service exposes its own endpoints) and single-binary mode (the process exposes all endpoints for all internal services).

    Key service groups include:

    • Distributor: Remote write, OTLP receiver, tenant stats.
    • Ingester: Flush blocks, shutdown, ring status, tenant stats.
    • Querier/Query-frontend: Prometheus-compatible query endpoints (instant, range, exemplar, etc.).
    • Ruler: Rule management and alerting.
    • Alertmanager: Alert configuration and status.
    • Purger: Tenant deletion requests.
    • Overrides: User-specific configuration overrides.

  5. Overview of Cortex Blocks Storage

    master

    Cortex Blocks Storage is a storage engine based on Prometheus TSDB. It stores each tenant's time series into their own TSDB, writing series to on-disk blocks (defaulting to 2-hour periods). Each block consists of chunk files (timestamp-value pairs) and an index (mapping metric names and labels to series in chunks).

    Supported Backends:

    • Amazon S3
    • Google Cloud Storage
    • Microsoft Azure Storage
    • OpenStack Swift
    • Local Filesystem (single node only)

  6. Understand Parallel Compaction by Time Interval

    master

    Cortex's parallel compaction design aims to increase the active timeseries limit per tenant by allowing multiple non-overlapping time intervals to be compacted concurrently.

    In the standard architecture, a single tenant is sharded to a single compactor, which processes compaction groups (groups of Prometheus TSDB blocks) iteratively. This means blocks from different time intervals cannot be compacted at the same time for the same tenant.

    The proposed parallel approach uses shuffle-sharding, where a single tenant can belong to multiple compactors. Compactors determine responsibility for specific blocks by using a hash of the tenant id and the time interval, placing this information on the sharding ring. This allows for horizontal scaling of compaction throughput per tenant.

  7. Understand Partition Ring with Multi-AZ Replication

    master

    Cortex is proposing a transition from a token-based ring to a Partition Ring architecture to improve availability in multi-AZ deployments.

    In the current token-based ring, independent hashing of series can cause a single request containing multiple series to fail if any of its required ingesters are offline. This 'availability amplification' becomes more critical as batch sizes increase.

    The Partition Ring solves this by:

    1. Grouping Ingesters into Partitions: Each partition contains a set of tokens and a group of ingesters (one per availability zone).
    2. Deterministic Replication: Data sent to a partition is replicated across all ingesters within that partition, making replication patterns predictable.
    3. Reduced Failure Probability: It is significantly less likely that multiple ingesters in the same partition will fail simultaneously compared to random ingester failures across the fleet.

  8. Understand the Blocks Storage Bucket Index mechanism

    master

    Cortex uses a per-tenant bucket-index.json file to optimize how Queriers discover blocks and deletion marks. Instead of scanning the entire object store bucket (which causes high latency and API costs), Queriers download this single JSON file to get a complete view of a tenant's blocks.

    Key Benefits

    • Zero Startup Time: Queriers don't need to scan the bucket at startup.
    • Reduced API Costs: Replaces thousands of list objects and get object calls with a single get object call per tenant.
    • Efficient Memory Usage: The index is small (~150 bytes per block) and can be lazily loaded and periodically updated in memory.

    Index Structure

    The bucket-index.json file (stored at /<tenant-id>/bucket-index.json) contains:

    • Version
    • List of blocks: ID, MinTime, MaxTime, UploadedAt, SegmentsFormat (e.g., 1-based-6-digits), and SegmentsNum.
    • List of block deletion marks: ID and DeletionTime.
    • UpdatedAt timestamp.

    Staleness and Consistency

    Queriers track the UpdatedAt timestamp. If the index becomes too stale, queries may fail to ensure consistency. The maximum allowed staleness is determined by the minimum of:

    • New blocks from ingesters: min(-querier.query-ingesters-within, -blocks-storage.tsdb.retention-period) (default: 6h).
    • New blocks/deletions from compactor: -compactor.deletion-delay (default: 12h).

  9. Understand Cortex Service Architecture

    master

    Cortex uses a service-based architecture where components run separately and in parallel to perform specific tasks. Alternatively, it can run in a single process mode, which is useful for local testing and development.

    Core services include:

    • Distributor: Handles incoming Prometheus samples.
    • Ingester: Manages in-memory samples and flushes them to long-term storage.
    • Querier: Executes PromQL queries.
    • Compactor: Optimizes storage blocks.
    • Store gateway: Queries series from blocks.

    Optional services include Alertmanager, Configs API, Overrides exporter, Query frontend, Query scheduler, and Ruler.

  10. Understand the Store-gateway service

    master

    The store-gateway is a semi-stateful Cortex service responsible for querying series from blocks. It is a required component when running Cortex blocks storage. It maintains a view of the storage bucket to discover blocks belonging to its assigned shard.

    To maintain this view, it uses one of two methods:

    1. Periodic bucket scanning (default): The service iterates over the storage bucket to discover blocks and deletion marks.
    2. Bucket index: The service fetches a bucket index for each tenant in its shard.

    Note on Readiness: During the initial bucket synchronization phase at startup, the /ready readiness probe will fail. It is highly recommended to run the store-gateway with a persistent disk (e.g., using a StatefulSet with a PersistentVolumeClaim in Kubernetes) to store the index-header locally and avoid re-downloading it on restarts.

  11. Security and Multi-tenancy in Crossplane Provider for Cortex

    master

    The provider implements several security patterns for managing Cortex configurations via Kubernetes:

    Authentication and Authorization

    • Credential Storage: All credentials are stored in Kubernetes secrets.
    • Least Privilege: The provider service account is designed with minimal required permissions.
    • Bearer Tokens: Continuous authentication is maintained using bearer tokens retrieved from secrets.
    • Audit Logging: All configuration changes are logged for security auditing.

    Network Security

    • TLS Support: All HTTP communications can use TLS with certificate validation.
    • Custom CA: Supports private CAs and certificate pinning.
    • mTLS: Supports client certificate authentication.
    • Network Policies: Compatible with Kubernetes network policies for traffic isolation.

    Multi-tenancy

    • Tenant Isolation: TenantConfig manages separate tenant credentials and configurations.
    • Namespace Isolation: Resources can be deployed in separate namespaces to ensure isolation.
    • RBAC: Leverages standard Kubernetes RBAC for access control.
    • Validation: Configuration validation prevents cross-tenant configuration leakage.

  12. Understand the Ring Multikey Proposal

    master

    The Ring Multikey proposal aims to reduce concurrency issues and latency in Cortex services (like ingester or distributor) when using KV stores such as etcd or consul.

    Currently, services use a single key to store a large Desc struct containing all pod information. This causes high latency and write failures (5xx errors) as the number of pods scales, because all pods compete to update the same key.

    The Solution: Instead of one large key (e.g., ingester/ring/ingester), the data is split into multiple individual keys (e.g., ingester/ring/ingester-0, ingester/ring/ingester-1). This allows pods to update only their specific information without competing for a global lock or version tag on a single shared key.