Elassandra Documentation

The ingest-attachment plugin allows Elasticsearch to extract text and metadata from file attachments in common formats (such as PPT, XLS, and PDF) using the Apache Tika library. It serves as a replacement for the older mapper attachment plugin.

Data Format Requirements:

• The source field must be a base64 encoded binary string.
• Optimization Tip: To avoid the overhead of base64 encoding/decoding, you can use the CBOR format instead of JSON and specify the field as a bytes array. The processor will skip decoding in this case.

The ICU Analysis plugin integrates the Lucene ICU module into Elassandra, providing extended Unicode support via the ICU libraries. This enables advanced capabilities such as:

• Improved analysis of Asian languages (Thai, Lao, Chinese, Japanese, Korean, Myanmar, and Khmer).
• Unicode normalization and Unicode-aware case folding.
• Collation support (language-specific sorting).
• Transliteration (e.g., converting scripts to Latin).

Important: Backwards Compatibility ICU library updates (adding new characters, emojis, or improving collation orders) may affect search and sort results. While upgrades are restricted to major versions, an index created in a previous major version may require reindexing to ensure correct ordering and to support new characters.

The analysis-nori plugin integrates the Lucene nori analysis module into Elasticsearch. It uses the mecab-ko-dic dictionary to perform morphological analysis of Korean texts.

The standard nori analyzer is composed of the following components:

• nori_tokenizer (tokenizer)
• nori_part_of_speech (token filter)
• nori_readingform (token filter)
• lowercase (token filter)

Elassandra provides several command line tools for managing security configurations, performing system tasks, and managing users. The available tools are:

• certgen: Certificate generation.
• certutil: Certificate utility.
• migrate-tool: Migration tool.
• saml-metadata: SAML metadata generation.
• setup-passwords: Password setup utility.
• shard-tool: Shard management tool.
• syskeygen: System key generation.
• users-command: User management command.

Elassandra is an Apache Cassandra distribution that includes an embedded Elasticsearch search engine. It functions as a multi-master, multi-cloud database and search engine capable of replicating across multiple datacenters in active/active mode.

Key Architecture Concepts:

• Embedded Search: Elasticsearch code is embedded directly within Cassandra nodes, providing advanced search features on Cassandra tables.
• Unified Storage: Cassandra serves as the data and configuration store for Elasticsearch.
• Scalability: It supports Cassandra vnodes, allowing horizontal scaling by adding nodes without requiring manual index resharding.
• High Availability: Unlike standard Elasticsearch which is master-slave, Elassandra is masterless. It uses Cassandra lightweight transactions to manage cluster state, eliminating single points of write failure.

The repository-gcs plugin allows using Google Cloud Storage (GCS) as a repository for Snapshot/Restore operations.

Prerequisites:

1. A Google Cloud project with the Cloud Storage Service enabled.
2. A GCS bucket created via the Google Cloud Console (the plugin does not create buckets automatically).
3. A service account with 'Writer' access to the bucket.

Important Authentication Note: Standard 'application default credentials' (e.g., via GOOGLE_APPLICATION_CREDENTIALS environment variable) are not supported due to the Elasticsearch security manager. You must manually provide service account credentials via the Elasticsearch keystore.

The elasticsearch-certgen command is used to create Certificate Authorities (CA), Certificate Signing Requests (CSR), and signed certificates for the Elastic Stack.

Note: This command is deprecated as of version 6.1 and has been replaced by elasticsearch-certutil. However, existing certificates and CAs created with elasticsearch-certgen do not need to be replaced.

By default, the command runs in an interactive mode where you are prompted for instance details. An 'instance' refers to any component (Elasticsearch, Logstash, Kibana, or Beats) requiring TLS/SSL. The output is a ZIP file containing the signed certificates and private keys. If generating a CA, the CA certificate and private key are also included.

bin/elasticsearch-certgen

The Store SMB plugin is an experimental workaround for a performance issue occurring when using Java on Windows with shared file systems based on the SMB protocol (such as Azure File Service).

In these environments, Lucene's standard method of opening index segment files with a write-only flag disables the Windows SMB cache manager, resulting in slow writes (as described in LUCENE-6176). The Store SMB plugin provides alternative storage types that open files with a read flag to re-enable caching.

Warning: This plugin provides experimental support and should be used with caution. The long-term fix is expected to come from OpenJDK or Windows updates.

The significant_terms aggregation identifies interesting or unusual occurrences of terms by comparing a foreground set (the search results matched by a query) against a background set (typically the entire index).

Instead of just finding the most popular terms, it finds terms that have undergone a significant change in popularity. For example, if a term appears in 4% of your search results but only 0.01% of the total index, it is flagged as significant.

Common Use Cases:

• Suggesting keywords for refining searches.
• Identifying anomalies (e.g., a specific merchant appearing disproportionately in fraud reports).
• Spotting mis-categorized content by comparing structured fields to free-text descriptions.

The term suggester provides an API to access word alternatives on a per-token basis within a certain string distance. It allows accessing individual tokens while leaving the selection of the final phrase to the consumer.

The phrase suggester builds on top of the term suggester by adding logic to select entire corrected phrases instead of individual tokens. It uses ngram-language models to make better decisions about which tokens to pick based on co-occurrence and frequencies, resulting in more natural-sounding corrections.