DOCSEARCHbyAlgolia
Submit Docs

Apache OpenNLP Documentation

repository·main·Indexed 23 days ago

https://github.com/apache/opennlp

A machine learning-based Java toolkit for natural language processing tasks, including tokenization, sentence segmentation, part-of-speech tagging, and named entity extraction. Features include ONNX model support via opennlp-dl for GPU acceleration, SymSpell-based spell correction, and a modular architecture for JDK 21+. Provides both a Java API and a command-line interface for training, evaluating, and running models.

Tokens
8K
Snippets
17
Records
41
Agent score
82%

What's inside Apache OpenNLP

  1. Overview of Apache OpenNLP modules

    main

    Apache OpenNLP is modularized to allow developers to import only the components they need, reducing the dependency footprint. Key modules include:

    • opennlp-api: Public API defining core interfaces and abstractions.
    • opennlp-runtime: Core classes shared across components.
    • opennlp-ml-maxent, opennlp-ml-bayes, opennlp-ml-perceptron, opennlp-ml-libsvm: Specific machine learning implementations.
    • opennlp-dl / opennlp-dl-gpu: Adapters for ONNX models (with GPU support option).
    • opennlp-model-resolver: For discovering and loading models from the classpath.
    • opennlp-formats: Support for reading/writing NLP training and data formats.
    • opennlp-cli: Command-line tools for training, evaluating, and running models.
    • opennlp-tools: The full end-user toolkit containing all core components.
    • opennlp-morfologik: Morfologik-based dictionary and stemming support.
    • opennlp-spellcheck: SymSpell-based spell correction.
    • opennlp-uima: Apache UIMA annotators.

  2. Thread safety in Apache OpenNLP 3.x

    main

    Starting with version 3.0.0, the core *ME classes are thread-safe. This means a single instance of the following classes can be shared across multiple threads without the need for pooling or recreating instances per thread:

    • POSTaggerME
    • TokenizerME
    • SentenceDetectorME
    • ChunkerME
    • LemmatizerME
    • NameFinderME

    Legacy ThreadSafe*ME wrappers from 2.x are still supported but are now deprecated.

  3. Run ThreadSafetyBenchmarkIT correctness test

    main

    To verify that shared Maximum Entropy (ME) instances produce identical results to a single-threaded baseline under concurrent access, run the ThreadSafetyBenchmarkIT integration test. Note that you must use mvn verify instead of mvn test, as integration tests (*IT.java) are excluded from the standard test phase.

    mvn verify -pl opennlp-core/opennlp-runtime -am \
    -Dforbiddenapis.skip=true \
    -Dit.test=ThreadSafetyBenchmarkIT

  4. Post-generation: Reformat and License generated stemmer files

    main

    After generating the Java files, you must perform manual cleanup to ensure they are compatible with the OpenNLP codebase:

    1. Reformat Code: Open the generated files in an IDE and adjust indentation, spacing, and alignment to match the OpenNLP code style. You may also need to rename variables or methods to follow OpenNLP conventions.
    2. Add License Information: Ensure each generated file contains the appropriate license headers for both the Snowball project and the Apache Software Foundation (OpenNLP).

  5. Migrating from OpenNLP 2.x to 3.x

    main

    The 3.x release modularizes the project but introduces no known breaking changes to the core API.

    Key changes:

    • Modular Structure: Instead of using the monolithic opennlp-tools artifact, it is recommended to import only the specific modules you need (e.g., opennlp-runtime and opennlp-model-resolver).
    • Java Version: The minimum required Java version has been raised to JDK 21+ for the 3.0.0 release.
    • Package Namespace: Note that a future release (potentially 4.x) will change the package namespace from opennlp to org.apache.opennlp to align with Java conventions.

  6. Export Hugging Face models to ONNX

    main

    To use Hugging Face models with OpenNLP DL, you must first export them to the ONNX format using the transformers.onnx module.

    For NER (Named Entity Recognition):

    python -m transformers.onnx --model=dslim/bert-base-NER --feature token-classification exported

    For Sequence Classification (e.g., Sentiment):

    python -m transformers.onnx --model=nlptown/bert-base-multilingual-uncased-sentiment --feature sequence-classification exported

  8. Build and run JMH thread safety benchmarks

    main

    To measure the performance of different instance allocation strategies (like newInstancePerCall, instancePerThread, or sharedInstance) under concurrent load, you must build the project with the jmh profile and manually materialize the classpath to avoid ClassNotFoundException: ForkedMain errors in JMH's forked JVMs.

    Follow these steps to build and execute the benchmarks:

  9. Build and Integrate Snowball Stemmer for OpenNLP

    main

    This guide describes how to build the Snowball compiler, generate Java stemmer classes for various languages, and integrate them into the OpenNLP project. This is necessary if you need to add or update stemmer support using the Snowball language definitions.

    # 1. Clone and build the Snowball compiler
git clone https://github.com/snowballstem/snowball.git
cd snowball
make

# 2. Run the generation script (see 'Run the Snowball Compiler' record for details)
./generate_stemmers.sh

  10. Install Apache OpenNLP via Maven or Gradle

    main

    You can integrate Apache OpenNLP into your Java project using Maven or Gradle. The core toolkit is provided by opennlp-runtime. If you need to discover and load models from the classpath, you should also include opennlp-model-resolver.

    Note that opennlp-runtime includes the Maximum Entropy (MaxEnt) implementation by default. If your project requires other machine learning implementations (like Naive Bayes or Perceptron), you must add those specific dependencies explicitly.

    #### Maven

```xml
<dependency>
    <groupId>org.apache.opennlp</groupId>
    <artifactId>opennlp-runtime</artifactId>
    <version>${opennlp.version}</version>
</dependency>
<!-- if model support is needed -->
<dependency>
    <groupId>org.apache.opennlp</groupId>
    <artifactId>opennlp-model-resolver</artifactId>
    <version>${opennlp.version}</version>
</dependency>

    Gradle

    compile group: "org.apache.opennlp", name: "opennlp-runtime", version: "${opennlp.version}"
compile group: "org.apache.opennlp", name: "opennlp-model-resolver", version: "${opennlp.version}"

  11. Convert Sentence Transformers to ONNX

    main

    To use sentence vectors in OpenNLP, you can convert a sentence-transformers model to ONNX using the optimum library.

    1. Install dependencies:
    python3 -m pip install optimum onnx onnxruntime
    1. Run conversion script: Use ORTModelForFeatureExtraction to load the model and save both the ONNX checkpoint and the tokenizer to a local directory.
    from optimum.onnxruntime import ORTModelForFeatureExtraction
from transformers import AutoTokenizer
from pathlib import Path

model_id="sentence-transformers/all-MiniLM-L6-v2"
onnx_path = Path("onnx")

# load vanilla transformers and convert to onnx
model = ORTModelForFeatureExtraction.from_pretrained(model_id, from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# save onnx checkpoint and tokenizer
model.save_pretrained(onnx_path)
tokenizer.save_pretrained(onnx_path)

  12. Configure InferenceOptions for NameFinderDL and DocumentCategorizerDL

    main

    You can control text normalization behavior for both NameFinderDL and DocumentCategorizerDL using InferenceOptions. These options are useful for handling diverse Unicode inputs from web or PDF sources.

    • setNormalizeWhitespace(true): Converts each Unicode whitespace character into an ASCII space. This is length-preserving and does not move offsets.
    • setNormalizeDashes(true): Converts Unicode dashes to hyphen-minus. Note that while this can shrink non-BMP dashes by one UTF-16 unit, NameFinderDL.findInOriginal uses an Alignment to ensure reported spans remain correct relative to the original input.
    • setLowerCase(true): Enables lowercasing (required for uncased models).
    InferenceOptions options = new InferenceOptions();
options.setNormalizeWhitespace(true);
options.setNormalizeDashes(true);
NameFinderDL finder = new NameFinderDL(model, vocab, ids2Labels, options, sentenceDetector);