DOCSEARCHbyAlgolia
Submit Docs

Apache Ignite 3

repository·main·Indexed 43 days ago

https://github.com/apache/ignite-3

A high-performance distributed database designed for ACID-compliant transactions, distributed SQL, and scalable compute tasks. It features a schema-driven architecture, pluggable storage engines like RocksDB, and a dedicated CLI for cluster management and SQL REPL.

Tokens
338.8K
Snippets
904
Records
1.4K
Agent score
50%

What's inside Apache Ignite 3

  1. What is Apache Ignite 3?

    main
    Apache Ignite is a distributed database designed for high-performance transactional and analytical workloads. It combines in-memory speed with disk persistence, providing both SQL and key-value access to the same data with full ACID transaction support. It is suitable for high-throughput OLTP, real-time analytics, caching layers, microservices shared state, and event processing.

  2. What is the Apache Ignite Bytecode module?

    main
    The Apache Ignite Bytecode module is a thin wrapper around the ASM library. It is designed to provide a convenient way to generate Java classes at runtime. This module is a fork of the PrestoDB Bytecode module (version 0.243), with the Guava dependency removed and tests migrated to JUnit 5.

  3. Overview of the Ignite RAFT module

    main

    The Ignite RAFT module provides an implementation of the RAFT (Reliable, Replicated, Redundant, And Fault-Tolerant) protocol. It is used to manage replication groups within Ignite.

    Key components:

    • JRaftServerImpl: The main entry point used to start a server node for handling RAFT messages.
    • NodeImpl: Responsible for managing one or many RAFT replication groups.
    • RAFT Node: Represents a single group node, which can be either a voting node or a non-voting node (also known as a learner).

  4. Overview of Apache Ignite 3 Native Clients

    main

    Apache Ignite 3 provides full-featured native clients that support the complete set of Ignite 3 APIs. These clients are designed for high-performance data access and full feature parity with the cluster.

    Key Features

    • Full Table API: Access via RecordView and KeyValueView.
    • SQL API: Support for prepared statements.
    • Transactions: Full transactional support.
    • Data Streaming: Capabilities for high-throughput data ingestion.
    • Compute: Execution of compute jobs on the cluster.
    • Cluster Topology: Access to information about the cluster state.
    • Async Operations: Support for asynchronous API calls.

    When to use Native Clients

    You should use a native client when your application requires:

    • Full access to all Ignite features (not just SQL).
    • Key-value operations.
    • Compute job deployment.
    • Event handling.
    • High-performance data access.

    Note: If you only require SQL connectivity, consider using a JDBC connection instead.

  5. Overview of Apache Ignite 3 Disaster Recovery

    main

    Disaster recovery in Apache Ignite 3 is used to restore a cluster when data operations become unfeasible due to consistency issues, typically caused by node failures or unreachability. Recovery is categorized into two main areas:

    1. Data Partition Recovery: Used when partition replicas in distribution zones become unavailable or inconsistent. The procedure depends on whether a minority or majority of replicas are affected.
    2. System Group Recovery: Used to restore critical cluster metadata and coordination components, specifically the Cluster Management Group (CMG) (topology and membership) and the Metastorage Group (metadata and configuration).

    For specific procedures, refer to the detailed guides for Data Partitions Recovery and System Groups Recovery.

  6. Overview of Apache Ignite 3 Java Examples

    main

    The examples/java module provides a comprehensive suite of examples demonstrating various Apache Ignite 3 APIs and storage engines:

    Data Access APIs

    • RecordViewExample: Usage of org.apache.ignite.table.RecordView.
    • KeyValueViewExample: Usage of org.apache.ignite.table.KeyValueView.
    • SqlJdbcExample: Usage of the Apache Ignite JDBC driver.
    • SqlApiExample: Usage of the Java API for SQL.

    Storage Engines

    • VolatilePageMemoryStorageExample: PageMemory storage with an in-memory data region.
    • PersistentPageMemoryStorageExample: PageMemory storage with a persistent data region.
    • RocksDbStorageExample: Usage of the RocksDB storage engine.

    Streaming and Data Ingestion

    Demonstrates DataStreamerTarget#streamData using Publisher and DataStreamerOptions with various targets:

    • KeyValueViewDataStreamerExample / KeyValueViewPojoDataStreamerExample (with POJOs)
    • RecordViewDataStreamerExample / RecordViewPojoDataStreamerExample (with POJOs)
    • ReceiverStreamProcessingExample: Stream processing of trades data.
    • ReceiverStreamProcessingWithResultSubscriberExample: Stream processing with result reception.
    • ReceiverStreamProcessingWithTableUpdateExample: Stream processing with table updates.

    Compute API

    Demonstrates IgniteCompute methods:

    • ComputeAsyncExample: executeAsync(JobTarget, JobDescriptor, Object).
    • ComputeBroadcastExample: execute(BroadcastJobTarget, JobDescriptor, Object).
    • ComputeCancellationExample: executeAsync with CancellationToken.
    • ComputeColocatedExample: execute with colocated JobTarget.
    • ComputeExample: Basic execute(JobTarget, JobDescriptor, Object).
    • ComputeJobPriorityExample: execute with different job priorities.
    • ComputeMapReduceExample: executeMapReduce(TaskDescriptor, Object).
    • ComputeWithCustomResultMarshallerExample: execute with a custom result marshaller.
    • ComputeWithResultExample: execute with a result return.

  7. Overview of the System Disaster Recovery API

    main
    The System Disaster Recovery API module provides specialized interfaces to recover from disasters affecting the core Apache Ignite 3 system Raft groups. Specifically, it targets recovery operations for the Cluster Management Group (CMG) and the Metastorage group, which are critical for maintaining system state and cluster coordination.

  8. Overview of the Apache Ignite 3 C++ API

    main

    The Apache Ignite 3 C++ API is a high-performance client designed with modern C++17 features and supports zero-copy operations. It provides thin client access to Ignite 3 clusters using an asynchronous execution model based on callbacks and supports configurable timeouts.

    Key features include:

    • Memory Management: Uses shared_ptr for automatic resource cleanup.
    • Data Access Models:
      • Typed Views: Uses template specialization for strongly-typed data access.
      • Binary Views: Uses ignite_tuple for schema-free operations.
    • Execution Model: Asynchronous operations via callbacks.

  9. Overview of Ignite 3 Metrics

    main
    The Metrics module is responsible for collecting and managing metrics across the Apache Ignite cluster. It provides visibility into the internal state and performance of the cluster components. For detailed design specifications and the underlying architecture of the metrics system, refer to IEP-116.

  10. Overview of data manipulation in Apache Ignite 3

    main

    Apache Ignite 3 provides several core capabilities for managing, processing, and interacting with data across a cluster. Key functional areas include:

    • Table API: Direct data access via key-value and record-based operations.
    • Transactions: ACID-compliant operations using Multi-Version Concurrency Control (MVCC) for cluster-wide consistency.
    • Data Streaming: High-throughput ingestion for bulk data loading.
    • Compute: Distributed computing that leverages colocated processing to perform operations locally where the data resides.
    • Serialization: Custom object serialization to optimize data transfer and storage efficiency.
    • Code Deployment: The ability to deploy custom code units to cluster nodes for distributed execution.
    • Events: An API to react to cluster events and data changes.
    • Java Integration: Programmatic table management from Java and logging configuration for the Java thin client.

  11. Overview of Apache Ignite 3 modules

    main
    Apache Ignite 3 is designed with a modular architecture to simplify testing and maintain clear boundaries. Each module exposes a specific public API intended for use by external modules. Developers should only interact with these exposed APIs and avoid using non-exposed internal APIs. The project structure prohibits cyclic dependencies between modules to ensure a clean dependency graph and facilitate future migrations to the Java JIGSAW module system.