Elasticsearch Documentation

Content connectors synchronize third-party data into Elasticsearch by extracting files, records, or objects and transforming them into searchable documents.

Key features supported by various connectors include:

• Advanced sync rules: Source-specific DSLs for remote filtering.
• Local binary extraction service: For extracting content from binary files.
• Incremental syncs: To sync only changed data.
• Document level security (DLS): To restrict access to specific documents.

Commonly available connectors include:

• Cloud Storage: Azure Blob, Google Cloud Storage, Amazon S3.
• SaaS/Collaboration: Box, Confluence (Cloud/Data Center/Server), Dropbox, GitHub, Google Drive, Notion, OneDrive, Sharepoint (Online/Server), Slack, Teams, Zoom.
• Databases: Microsoft SQL Server, MongoDB, MySQL, Oracle, PostgreSQL, Redis.
• Communication/Work: Gmail, Outlook, Jira (Cloud/Data Center/Server), ServiceNow.
• Other: GraphQL, Network drive, Salesforce.

Note: Always check the specific connector reference for prerequisites and configuration details.

Painless is the default scripting language for Elasticsearch, designed for security, performance, and flexibility. It is built on the Java Virtual Machine (JVM) and provides a Java-like syntax that compiles directly to JVM bytecode for native execution speed.

Key Features

• Security: Uses a sandbox environment and fine-grained allowlists to prevent access to restricted Java APIs.
• Performance: Direct compilation to bytecode eliminates interpretation overhead and leverages JVM optimizations.
• Flexibility: Supports context-aware scripting across the Elasticsearch stack, including search scoring, data processing, and ingest pipelines.

Execution Contexts

Scripts execute within specific contexts that determine available variables, allowed operations, and specific APIs. The available functionality changes depending on whether you are using the script for search, ingest, or other operational tasks.

An ingest pipeline is a sequence of processors applied to documents during ingestion. The order of processors is critical because each successive processor operates on the output of the previous one. Processors are categorized by their primary function:

Data Enrichment

Used to add information to documents.

• append: Appends a value to a field.
• enrich: Enriches documents using data from another index.
• inference: Uses Machine Learning to classify/tag text.
• geoip: Adds geographical info from IP addresses.
• attachment: Parses binary data (PDF, Word).
• fingerprint: Computes a hash of document content.

Data Transformation

Used to modify the structure or content of fields.

• convert: Changes field types (e.g., string to integer).
• dissect / grok: Extracts structured fields from text.
• gsub: Applies regex replacement to strings.
• rename: Renames an existing field.
• set: Sets a value on a field.
• split: Splits a field into an array.
• json: Parses a JSON string into a structured object.

Data Filtering

Used to include or exclude data.

• drop: Drops the document entirely.
• remove: Removes specific fields from the document.

Pipeline Handling

Used to control the flow of the ingest process.

• pipeline: Executes a different ingest pipeline.
• reroute: Sends documents to a different index or data stream.
• terminate: Stops the current pipeline immediately.
• fail: Raises an exception.

Array/JSON Handling

• for_each: Runs a processor on every element in an array/object.
• sort: Sorts array elements.
• script: Runs a Painless script in the ingest context.

libs/simdvec provides optimized vector distance and scoring kernels used by Elasticsearch's vector search features, including kNN, BBQ, and scalar quantization. It utilizes a hybrid approach:

• Java-side: Panama SIMD code.
• Native C++ library (libvec): Hand-tuned SIMD kernels loaded at runtime via FFI.

Native Code Tiers

The library selects kernels at runtime based on CPU and OS support probed by caps.cpp:

• Tier 1 (e.g., vec_1.cpp): Baseline support (AVX2 on x64, NEON + dotprod on ARM).
• Tier 2 (e.g., vec_2.cpp): Extended support (AVX-512 on x64, SVE on ARM).
• Tier 3 (e.g., vec_bf16_3.cpp): Specialized support (e.g., vdpbf16ps for native BF16 dot product on Intel Cooperlake).

Supported Quantization Formats

The kernels support single-pair and bulk scoring for:

• int7 (unsigned): dot-product, squared-euclidean, cosine (AVX2, AVX-512).
• int8 (signed): dot-product, squared-euclidean, cosine (AVX2, AVX-512, ARM/NEON).
• int4 (packed nibble): dot-product (AVX2, AVX-512, ARM/NEON).
• BBQ (binary quantized): 1-bit and 4-bit-to-1-bit dot products (AVX2, AVX-512, ARM/NEON, ARM/SVE).
• BFloat16: dot-product (AVX-512, AVX-512-BF16, ARM/NEON).
• float32: dot-product, squared-euclidean (AVX2, AVX-512, ARM/NEON, ARM/SVE).

The SharePoint Server connector synchronizes several object types from your SharePoint environment into Elasticsearch:

• Sites and Subsites
• Lists
• List Items and their attachment content
• Document Libraries and their attachment content (including Web Pages)

Key Limitations & Notes:

• File Size Limit: Content from files larger than 8 MiB will not be extracted by default. To handle larger files, you must use a self-managed local extraction service.
• Permissions: Permissions are not synced by default. Without Document Level Security (DLS) enabled, all documents indexed will be visible to any user with access to the Elasticsearch index.
• Sync Types: Full syncs are supported by default. Incremental syncs are supported but are disabled by default.