nrtSearch

nrtSearch is a high-performance gRPC server built on Apache Lucene 8.x that exposes Lucene's core functionality via a gRPC API. It optionally provides REST APIs on top of the gRPC layer.

Key features include:

• Near-Real-Time (NRT) Segment Replication: Uses a primary/writer node for indexing and segment merges, while replica nodes are dedicated entirely to search queries. This differs from document-based replication (like Elasticsearch) where every node is both a writer and a reader.
• Concurrent Query Execution: Supports running multiple queries simultaneously.
• Stateless Microservice Deployment: Indexes can be backed up to S3, allowing for easy scaling in environments like Kubernetes or Mesos. Nodes can bootstrap from previous backups upon restart.
• Streaming APIs: Provides gRPC streaming for both indexing and searching.

Every index in nrtsearch must contain an _ID field, and every document must provide a value for it. This field acts as the primary term for the document and is used to atomically replace existing documents when performing updates.

To configure an _ID field, you must set its type to _ID (or BOOLEAN as per the protobuf definition context) and ensure that either store or storeDocValues is set to true. The _ID field is always searchable by default.

{
    "name": "id_field",
    "type": "_ID",
    "storeDocValues": true
}

Normalizers are used for ATOM (keyword) fields to ensure consistent matching for term, prefix, and range queries. Unlike analyzers, normalizers treat the entire field value as a single token by using an internal keyword tokenizer. They only allow the application of character filters and token filters.

Predefined Normalizers

• lowercase: Lowercases the entire field value.

Custom Normalizers

Custom normalizers are defined via the CustomNormalizer message, specifying charFilters and tokenFilters. The available filters are the same as those used in custom analyzers.

message Normalizer {
    oneof NormalizerType {
        string predefined = 1;       // e.g., "lowercase"
        CustomNormalizer custom = 2;
    }
}

message CustomNormalizer {
    repeated NameAndParams charFilters = 1;
    repeated NameAndParams tokenFilters = 2;
}

A DisjunctionMaxQuery generates a union of documents produced by its subqueries. It scores each document using the maximum score produced by any single subquery, plus a tie-breaking increment for any additional matching subqueries.

This is particularly useful when searching for a term across multiple fields that have different boost factors, where a simple union would not allow for equivalent field combination. This query maps directly to Lucene's DisjunctionMaxQuery.

message DisjunctionMaxQuery {
    repeated Query disjuncts = 1; // A list of all the disjuncts to add
    float tieBreakerMultiplier = 2; // The score of each non-maximum disjunct for a document is multiplied by this weight and added into the final score.
}

To use innerHit, two conditions must be met:

1. A parent searchRequest must be present.
2. The index being searched must have the child object field registered as a nested field.

Example registration configuration:

{
  "name": "field_name",
  "nestedDoc": true,
  "multiValued": true,
  "type": "OBJECT",
  "childFields": [
    ...
  ]
}

When using the built-in "js" language (Lucene expressions), you can access specific data points within your script:

• _score: The document's score from the prior retrieval pass (Lucene relevance score for standard queries, or the blended score for multi-retriever requests).
• Document Fields: Numeric doc-values fields can be referenced directly by their name.
• Script Parameters: Named parameters passed via script.params can be used in the expression.

Note: This is a subset of JavaScript. It supports arithmetic (+, -, *, /) and math functions (min, max, abs, log, sqrt, pow, ln, exp, etc.), but it cannot access text fields directly.

// Example: multiply prior score by a popularity field
"_score * log(1 + popularity)"

// Example: using a named parameter
"_score * weight"

In nrtsearch, nested objects are stored as separate documents from their parent documents. While a NestedQuery allows you to filter parent documents based on whether they have matching child documents, the standard HitResponse only returns the parent documents.

To retrieve the actual matched child documents alongside their parents, you must use an innerHit. You can think of an innerHit as a second layer of search performed for each parent hit. Providing an empty innerHit query will return all children for each parent hit.

By default, most field types support the childFields property. This allows you to index the same input data using different configurations (such as different analyzers) without duplicating the data in your indexing request.

To access a child field, use dot notation: parent_field_name.child_field_name.

Note on Object Fields: The childFields behavior described above does not apply to OBJECT field types. For object fields, childFields is used to define the internal schema of the object's properties. See the field_types/object documentation for details.

{
    "name": "text_field",
    "type": "TEXT",
    "search": true,
    "childFields": [
        {
            "name": "keyword",
            "type": "ATOM",
            "search": true,
            "storeDocValues": true
        }
    ]
}