Apache Lucene Documentation

The dev-tools directory contains various scripts and resources intended to assist developers of Lucene and Solr. Note that these tools are not maintained with the same rigor as the core Lucene codebase and may vary in utility.

Available toolsets include:

• aws-jmh: Scripts for executing microbenchmarks across various AWS EC2 instance types.
• doap: Lucene project descriptors provided in DOAP RDF format.
• scripts: Miscellaneous scripts used for tasks such as building releases.
• test-patch: Scripts designed for the automatic validation of patches.

Apache Lucene is a Java-based full-text search engine. It is provided as a code library and API rather than a standalone application, allowing developers to integrate search capabilities into their own Java applications.

Key resources:

• Website: https://lucene.apache.org/
• Mailing List: Join the java-user-subscribe@lucene.apache.org list for user discussions and support.
• Documentation: The primary documentation is located in the docs/index.html file within the distribution.

Lucene maintains compatibility with older indices by versioning codecs and file formats based on the minor version in which they were created. For example, Lucene87Codec is used for indices created in Lucene 8.7 and potentially later versions. Each segment explicitly records the codec name used to write it.

To support older versions, Lucene maintains legacy codec classes and file formats within the backwards-codecs package. Compatibility is verified through:

• Unit tests: Classes like TestLucene80NormsFormat verify that data can be written and then read using specific old formats.
• Integration tests: TestBackwardsCompatibility loads indices created in previous versions to ensure they remain searchable.

Starting with Lucene 10.5, [Byte|Float]VectorSimilarityQuery uses adaptive graph traversal instead of an explicit traversalSimilarity parameter. This improves the recall vs. latency tradeoff.

Users can tune the quality and performance using a decay factor, which should be a value in the range [0, 1]. Higher values produce better recall by encouraging more graph exploration.

The test-model-data directory contains small training datasets used to generate minimal OpenNLP models specifically for unit testing purposes. This data is derived from the Reuters corpus and tagged using CCG Urbana-Champaign online demos.

Note: The automated generation of models from this training data via Gradle is currently pending a specific feature implementation (referenced in issue #13002).

When working with multiple major versions of Lucene (e.g., switching between 8.x and 9.x), different branches may use different build systems (like ant vs gradle). To avoid orphaned build files and tool conflicts (IntelliJ, precommit), use git worktree. This allows you to maintain separate directories for different branches while sharing the same git metadata.

Follow these steps to set up a root directory and separate worktrees for main and a specific major branch:

mkdir lucene
cd lucene

# Clone the main repository into a directory named 'main'
git clone git@github.com:apache/lucene.git main
cd main

# Add a worktree for a specific branch (e.g., branch_10x) in a separate directory
git worktree add ../10x branch_10x

In Lucene 11, query caching is disabled by default. To enable it, you must manually configure it, typically in a static initialization block using IndexSearcher.setDefaultQueryCache.

int maxCachedQueries = 1_000;
long maxRamBytesUsed = 50 * 1024 * 1024; // 50MB
IndexSearcher.setDefaultQueryCache(new LRUQueryCache(maxCachedQueries, maxRamBytesUsed));

Lucene 11 introduces support for JUnit Jupiter. To migrate your test suite:

1. Extend LuceneTestCaseJupiter instead of LuceneTestCase.
2. Annotate test methods with @Test. Simply prefixing a method with test* is no longer sufficient and will cause validation errors.
3. Use JUnit 5 lifecycle annotations like @BeforeEach and @AfterEach instead of setUp and tearDown.
4. Do not call the static random() method on the parent class. Instead, add a Random parameter to your test methods or callbacks; the framework will automatically inject it.
5. Reference static utility methods via the parent class type or without an explicit type, as they have been moved to LuceneTestCaseParent.