Luwak Documentation

repository·master·Indexed 18 days ago

A high-performance stored query engine based on Apache Lucene designed for reverse search and document routing. Luwak allows users to define complex queries upfront and monitor a stream of incoming documents to identify matches using components like the Monitor, Presearchers for optimization, and various CandidateMatcher implementations for reporting results.

Tokens: 1.6K
Snippets: 3
Records: 8
Agent score: 14%

What's inside Luwak

How Presearchers optimize query matching
master
A Presearcher reduces the number of queries the Monitor must run against an InputDocument by filtering them out early. Luwak provides three built-in implementations:
- MatchAllPresearcher: Performs no filtering; all registered queries are run against every document.
- TermFilteredPresearcher: Extracts terms from registered queries and indexes them in an internal index. At match-time, the InputDocument is tokenized, and only queries matching the document's terms are executed.
- MultipassTermFilteredPresearcher: An extension of TermFilteredPresearcher that improves filtering for phrase queries by indexing various combinations of terms.
https://github.com/flaxsearch/luwak/blob/master/README.md

Basic usage of the Luwak Monitor

master

The Monitor is the central component of Luwak. It manages a set of queries and matches them against incoming documents. You initialize it with a MonitorQueryParser and a Presearcher to optimize performance. You can then add queries using MonitorQuery and match documents either individually or in batches.

Monitor monitor = new Monitor(new LuceneQueryParser("field"), new TermFilteredPresearcher());

// Add a query
MonitorQuery mq = new MonitorQuery("query1", "field:text");
List<QueryError> errors = monitor.update(mq);

// Match one document at a time
InputDocument doc = InputDocument.builder("doc1")
                        .addField(textfield, document, new StandardAnalyzer())
                        .build();
Matches<QueryMatch> matches = monitor.match(doc, SimpleMatcher.FACTORY);

// Or match a batch of documents
Matches<QueryMatch> matches = monitor.match(DocumentBatch.of(listOfDocuments), SimpleMatcher.FACTORY);

Install Luwak via Maven
master
To use Luwak in your Java project, add the following dependency to your Maven pom.xml file:
```
<dependency>
  <groupId>com.github.flaxsearch</groupId>
  <artifactId>luwak</artifactId>
  <version>1.4.0</version>
</dependency>
```
https://github.com/flaxsearch/luwak/blob/master/README.md
Customize TermFilteredPresearcher weighting
master
To improve performance, you can control which terms are indexed by the TermFilteredPresearcher using a WeightPolicy. This policy uses WeightNorm implementations to assign weights to terms and a CombinePolicy to aggregate them. Fewer indexed terms result in a faster presearcher.
Available WeightNorms:
- FieldWeightNorm: Weights all terms in a specific field.
- FieldSpecificTermWeightNorm: Weights specific terms in specific fields.
- TermTypeNorm: Weights terms by type (e.g., EXACT, ANY, CUSTOM).
- TermWeightNorm: Weights a specific set of terms with a given value.
- TokenLengthNorm: Weights terms by their length.
- TermFrequencyWeightNorm: Weights terms by their frequency.
Available CombinePolicy:
- MinWeightCombiner: Sets a parent node's weight to the minimum weight of its children.
Example Configuration:
```
WeightPolicy weightPolicy = WeightPolicy.Default(new FieldWeightNorm("category", 0.01f));
CombinePolicy combinePolicy = new MinWeightCombiner();

TreeWeightor weightor = new TreeWeightor(weightPolicy, combinePolicy);
Presearcher presearcher = new TermFilteredPresearcher(weightor);
```
https://github.com/flaxsearch/luwak/blob/master/README.md
Add and update queries in the Monitor
master
The monitor is updated using MonitorQuery objects, which consist of an ID, a query string, and an optional metadata map.
Error Handling:
- In Luwak 1.5.0+: Adding queries that fail parsing throws an UpdateException detailing the failures.
- In Luwak 1.4 and below: monitor.update(mq) returns a List<QueryError>. An empty list indicates success; a non-empty list indicates parsing errors.
https://github.com/flaxsearch/luwak/blob/master/README.md
Implement a custom Presearcher
master
To create an entirely new query filtering mechanism, subclass Presearcher. You must implement two methods:
1. buildQuery(InputDocument, QueryTermFilter): Converts incoming documents into queries to be run against the Monitor's query index.
2. indexQuery(Query, Map<String,String>): Converts registered queries into a form that can be indexed.
Note: indexQuery should avoid using reserved field names _id or _query for the Monitor's internal index.
https://github.com/flaxsearch/luwak/blob/master/README.md

Implement a custom QueryTreeBuilder for new query types

master

If you are using custom Lucene query types, the TermFilteredPresearcher may not know how to extract terms from them. To support custom queries, subclass QueryTreeBuilder to define how to build a tree representation of your query, then pass it to the presearcher via a PresearcherComponent.

public class CustomQueryTreeBuilder extends QueryTreeBuilder<CustomQuery> {

    public CustomQueryTreeBuilder() {
        super(CustomQuery.class);
    }

    @Override
    public QueryTree buildTree(QueryAnalyzer builder, CustomQuery query) {
        return new TermNode(getYourTermFromCustomQuery(query));
    }

}

// Usage
Presearcher presearcher = new TermFilteredPresearcher(new PresearcherComponent(new CustomQueryTreeBuilder()));

Choose a CandidateMatcher for document matching
master
When calling monitor.match(), you must provide a CandidateMatcher implementation to determine how results are reported.
Standard Matchers:
- SimpleMatcher: Reports which queries matched the InputDocument.
- ScoringMatcher: Reports which queries matched, including their scores.
- ExplainingMatcher: Reports matches with an explanation for their scores.
- HighlightingMatcher: Reports matches with individual match highlights for each query.
Multithreaded Matchers:
- ParallelMatcher: Runs queries in multiple threads as they are collected from the Monitor.
- PartioningMatcher: Collects queries, partitions them into groups, and runs each group in its own thread.
https://github.com/flaxsearch/luwak/blob/master/README.md