Chronicle Queue Documentation

Chronicle Queue is a broker-less, off-heap Java library designed for ultra-low-latency, persisted messaging. It allows for millions of events per second with microsecond latencies. Key features include:

• Off-heap storage: Minimizes Garbage Collection (GC) impact.
• Persistence: Uses memory-mapped files to store data on disk, allowing for unbounded queues limited only by disk space.
• Concurrency: Supports multiple writers via locking and multiple lock-less concurrent readers.
• Messaging Patterns: Supports appending messages, tailing (reading) messages, and random-access seeking.
• Low Latency: Achieves sub-microsecond latencies for same-machine IPC and under 10 microseconds for replicated machine-to-machine communication (Enterprise Edition).

Chronicle Queue documentation is divided into a User Guide and a collection of Runnable Demos. The User Guide covers the following core topics:

• Getting started: Introduction and hands-on instructions.
• Configuration: Fine-tuning options for various use cases.
• Appending: How to write messages to a queue.
• Tailing: How to read messages from a queue.
• Performance tuning: Key factors for optimizing performance in different environments.
• Enterprise Features (★): Includes Replication (automatic replication across machines/cores) and Encryption (protecting queue contents).

Chronicle Async Mode provides a high-performance, multi-writer, multi-reader, buffered queue implementation. By using an asynchronous buffer, you can mitigate OS-induced pauses that occur when using standard ChronicleQueue under heavy load. Configuring readBufferMode and writeBufferMode can reduce latency outlier events by up to 98% on loaded systems.

Key characteristics:

• Multi-writer/Multi-reader: Supports concurrent access.
• Zero-garbage collection: Designed for high-performance environments.
• Flexible Backing: Can be backed by an in-memory BytesStore (for single-process use) or a memory-mapped BytesStore (for multi-process use).

Chronicle Queue is an append-only data structure designed for high-performance sequential writes and reads. While primarily append-only, it supports random access and in-place updates (though entry sizes cannot be changed, they can be padded).

Key characteristics:

• Efficiency: Optimized for CPU L2 cache coherence and sequential disk access (HDD/SSD).
• Memory Mapping: Built on MappedBytes, it uses memory-mapped files to treat files as unbounded byte arrays. This allows the queue to exceed JVM heap limits, constrained only by available disk space.
• Thread Safety: Supports atomic operations like compareAndSet and atomic add on off-heap memory, enabling fast sharing between threads or processes (latency as low as 25 nanoseconds).

Chronicle Queue Zero (Queue Zero) is a high-performance variant of Chronicle Queue designed for applications that require significantly higher performance and better scaling at high percentiles, at the cost of some standard Chronicle Queue functionality.

Performance characteristics:

• Latency: Queue Zero typically provides approximately 10x lower latency compared to standard Chronicle Queue.
• Outliers: Queue Zero demonstrates a significant reduction in outliers at higher percentiles.

For implementation details or access to Queue Zero, contact sales@chronicle.software.

To use Chronicle Queue, you must understand its primary components:

• Excerpt: The fundamental data container. A message is written by starting, writing to, and finishing an excerpt.
• Appender: The source of messages. It appends data to the end of the queue. It does not support insertion or deletion.
• Tailer: An excerpt reader optimized for sequential reads. It can perform sequential and random reads (forwards and backwards). Unlike traditional queues, reading a message does not destroy it; every reader sees every message.
• Topics: A directory of queue files (e.g., mytopic/20160710.cq4).

Chronicle Queue uses specific terminology for its core operations:

• Appending: The act of writing data to the queue. This is performed using an appender.
• Tailing: The act of reading data from the queue. This is performed using a tailer.

Data is stored as a series of bytes, regardless of whether the original type was text, numbers, or serialized blobs.

Chronicle Queue manages data storage through a process called Rolling.

• Rolling: The act of writing to a new .cq4 file.
• Roll cycle: Defines the specific timestamp range of data that is stored within each individual Chronicle Queue file.

Chronicle Queue Enterprise is a commercially supported version of the Open Source Software (OSS) version. While the documentation covers OSS functionality, Enterprise-specific features are marked with a star symbol (★) in the guide.

Enterprise features include:

• Encryption: Protecting the contents of message queues and individual messages.
• TCP/IP Replication: Automatically replicating queues between hosts for real-time data backup.
• Timezone adapted rollover schedule: Managing queue file rollovers based on specific timezones.
• Async Mode: Improved performance for high throughput on slower filesystems.

The Queue Operations guide provides documentation for the core functional areas of Chronicle Queue. You can find detailed guides on the following operations:

• Appending: How to write data to the queue.
• Tailing: How to read data from the queue (typically following the tail).
• Indexing: How to index data for efficient retrieval.
• Read/Write Proxies: How to use proxies to manage read and write access.