luqum

Luqum tracks the characters appearing before (head) and after (tail) each element in an expression tree. This is useful for preserving meaningful formatting (like spaces) when manipulating or reconstructing queries.

These properties are automatically computed during parsing. However, if you are building trees manually or modifying them computationally, you must set these properties yourself to avoid losing delimiters.

To automatically add minimal head and tail values to an existing tree, use the auto_head_tail utility.

from luqum.tree import AndOperation, Word, Not
from luqum.auto_head_tail import auto_head_tail

# Manual construction requires setting head/tail to preserve spaces
my_tree = AndOperation(Word('foo', tail=" "), Word('bar', head=" "))
print(my_tree)  # Output: foo AND bar

# Using the utility for automatic minimal spacing
my_tree = AndOperation(Word('foo'), Not(Word('bar')))
my_tree = auto_head_tail(my_tree)
print(my_tree)  # Output: foo AND NOT bar

Luqum supports 'Named Queries', which allow you to associate unique identifiers (names) with specific nodes in the query tree. This is primarily used to highlight which parts of a query matched a specific document in Elasticsearch.

Workflow:

1. Assign Names: Use auto_name(tree) to generate a dictionary mapping names to tree paths.
2. Navigate Tree: Use element_from_name(tree, name, names) or element_from_path(tree, path) to retrieve specific nodes.
3. Elasticsearch Integration: When building Elasticsearch queries (e.g., via an es_builder), the names are included in the query DSL under the _name key. Elasticsearch will return these names in its response for matched clauses.
4. Propagate Matches: Use MatchingPropagator to identify which parts of the tree correspond to the names returned by Elasticsearch.
5. Display Results: Use HTMLMarker to generate HTML strings that wrap matched and non-matched parts in CSS classes (e.g., <span class="ok">).

from luqum.naming import auto_name, element_from_name, element_from_path, MatchingPropagator, matching_from_names, HTMLMarker

# 1. Generate names
names = auto_name(tree)

# 2. Retrieve elements
elem = element_from_name(tree, "a", names)

# 3. Propagate matches from Elasticsearch response
# matched_queries is a list of names returned by Elasticsearch (e.g., ['b', 'c'])
propagate_matching = MatchingPropagator()
paths_ok, paths_ko = propagate_matching(tree, *matching_from_names(matched_queries, names))

# 4. Generate HTML for display
mark_html = HTMLMarker()
html_output = mark_html(tree, paths_ok, paths_ko)

While you can manually traverse and modify tree nodes by accessing children and value attributes, the recommended way to manipulate queries is by using the visitor pattern via TreeTransformer from luqum.visitor.

To use it, subclass TreeTransformer and implement visit_<node_type> methods (e.g., visit_search_field). Within these methods, you can yield a new node (created via node.clone_item()) to replace the existing one, or use yield from self.generic_visit(node, context) to continue traversing the tree.

from luqum.visitor import TreeTransformer

class MyTransformer(TreeTransformer):
    def visit_search_field(self, node, context):
        if node.expr.value == '"lazy dog"':
            # Replace the node with a cloned version containing a new value
            new_node = node.clone_item()
            new_node.expr = node.expr.clone_item(value = '"back to foo bar"')
            yield new_node
        else:
            yield from self.generic_visit(node, context)

transformer = MyTransformer()
new_tree = transformer.visit(tree)

To modify how the Elasticsearch DSL is generated (e.g., changing a term query to a match query), inherit from ElasticsearchQueryBuilder and override one of its element handling classes.

Available element classes to override include:

• E_MUST (EMust)
• E_MUST_NOT (EMustNot)
• E_SHOULD (EShould)
• E_WORD (EWord)
• E_PHRASE (EPhrase)
• E_RANGE (ERange)
• E_NESTED (ENested)
• E_BOOL_OPERATION (EBoolOperation)

from luqum.elasticsearch.visitor import EWord, ElasticsearchQueryBuilder

class EWordMatch(EWord):
    @property
    def json(self):
         if self.q == '*':
             return super().json
         return {"match": {self.field: self.q}}

class MyElasticsearchQueryBuilder(ElasticsearchQueryBuilder):
    E_WORD = EWordMatch

transformer = MyElasticsearchQueryBuilder()
# Now transformer(tree) will use 'match' for words instead of 'term'

You can convert Lucene query trees into Elasticsearch Query DSL using ElasticsearchQueryBuilder from luqum.elasticsearch.

Manual Configuration

When initializing ElasticsearchQueryBuilder, you can specify:

• not_analyzed_fields: A list of fields to be treated as terms rather than full-text searches.
• nested_fields: A dictionary mapping field names to sets of sub-fields that should be treated as nested queries.
• object_fields: A list of field paths to be treated as object fields.

Automatic Configuration via SchemaAnalyzer

If you have an Elasticsearch index schema (mappings and settings), you can use SchemaAnalyzer to automatically generate the correct configuration for the ElasticsearchQueryBuilder.

from luqum.elasticsearch import ElasticsearchQueryBuilder, SchemaAnalyzer
from luqum.parser import parser

# Option 1: Manual
es_builder = ElasticsearchQueryBuilder(not_analyzed_fields=["published", "tag"])
tree = parser.parse('tag:fable')
query = es_builder(tree)

# Option 2: Automatic via Schema
MESSAGES_SCHEMA = { ... }
schema_analyzer = SchemaAnalyzer(MESSAGES_SCHEMA)
message_es_builder = ElasticsearchQueryBuilder(**schema_analyzer.query_builder_options())

tree = parser.parse('message:"exciting news"')
query = message_es_builder(tree)