Quickwit Search Engine

The OpenTelemetry Collector Proto package defines the protocol used for communication within the OpenTelemetry collector ecosystem. It is organized into several specialized packages:

• common: Contains shared messages used across different services.
• trace: Defines the Trace Service protocol messages.
• metrics: Defines the Metrics Service protocol messages.
• logs: Defines the Logs Service protocol messages.

Quickwit is a search engine designed to execute complex search and analytics queries directly on cloud storage (such as Amazon S3 or Azure Blob Storage) with sub-second latency. It is built in Rust and utilizes a decoupled compute and storage architecture, making it highly resource-efficient and capable of scaling to petabytes of data.

Core Architecture Benefits

• Decoupled Compute and Storage: Indexing and search workloads can be scaled independently.
• Cost-Effective Storage: Data is stored once on cheap, unlimited object storage.
• Rapid Scaling: Clusters can be scaled out in seconds without moving data.
• Multi-tenancy: Supports tenant isolation and usage-based charging.
• Schemaless Indexing: Allows indexing JSON documents with arbitrary fields without significant performance impact.

Quickwit is a search engine designed to execute complex search and analytics queries directly on cloud storage with sub-second latency. It uses a decoupled compute and storage architecture powered by Rust, making it suitable for scaling to petabytes of data while remaining resource-efficient.

Primary Use Cases

• Log Management: Efficiently searching and analyzing large volumes of logs.
• Distributed Tracing: Managing and querying trace data.
• Immutable Data: Handling conversational data (emails, texts, messaging) and event-based analytics.

Quickwit is a cloud-native search engine optimized for observability (logs, traces, and metrics). Key features include:

• Search Capabilities: Full-text search and aggregation queries.
• Cloud-Native Architecture: Decoupled compute and storage with stateless indexers and searchers. It provides sub-second search directly on cloud storage (Amazon S3, Azure Blob Storage, Google Cloud Storage, etc.).
• Compatibility: Elasticsearch/OpenSearch compatible API and Jaeger-native/OTEL-native support for logs and traces.
• Schema Flexibility: Supports both schemaless and strict schema indexing.
• Integrations: Grafana data source and Kubernetes readiness via Helm charts.
• Enterprise Features: Native support for Kafka, Kinesis, and Pulsar; multi-tenancy; retention policies; and GDPR-compliant delete tasks.

A Quickwit distributed cluster consists of five primary services:

All Quickwit REST API endpoints are prefixed with api/v1/.

• OpenAPI Specification: Available at /openapi.json.
• API Playground: A Swagger UI version is available at /ui/api-playground.
• URL Encoding: Parameters passed in the URL must be properly URL-encoded using UTF-8 for non-ASCII characters.
• Error Handling:
  • Successful requests return a 2xx HTTP status code.
  • Failed requests return a 4xx HTTP status code. The response body contains a JSON object with a message field describing the error.

{
 "message": "Failed to parse query"
}

Quickwit provides an Elasticsearch/OpenSearch compatible API to facilitate migrations and integrations with existing tools. Note that this API is currently incomplete; only specific endpoints are supported.

Supported endpoints include:

• _bulk: Batch ingestion
• _search: Index search
• _msearch: Multi search
• _search/scroll: Scroll API
• _cat: Cat API
• _field_caps: Field capabilities API

Quickwit is designed for efficient indexing of unstructured data and searching directly on cloud storage. It is optimized for log management use cases through native support for several ingestion protocols:

• OpenTelemetry (OTEL): Supports both gRPC and HTTP (protobuf only) protocols.
• REST API: Provides a REST API capable of ingesting any JSON-formatted logs.

Because it handles unstructured data efficiently and integrates with standard observability protocols, it serves as a high-performance alternative for log storage and search.

Quickwit is officially supported on Linux. While macOS and FreeBSD are not officially supported, they are expected to work.

Architecture Support:

• x86-64: Supported (SSE3 is recommended).
• aarch64 (ARM64): Supported, but currently considered experimental.

The Control Plane is responsible for ensuring that the state of the indexers matches the desired state defined by the Metastore. It operates based on three triggers:

1. Metastore Events: The scheduler listens for events such as source create, delete, toggle, or index delete. Upon receiving these, it generates a desired plan and sends tasks to indexers.
2. Heartbeats (Every 3 seconds): The scheduler checks if the current tasks running on indexers are in sync with the desired plan. If they are out of sync, it reapplies the plan.
3. Periodic Rebuild (Every minute): The scheduler rebuilds a plan using the latest Metastore state. This acts as a safety mechanism to recover from missed events due to network issues.