DOCSEARCHbyAlgolia
Submit Docs

Zemberek-NLP

repository·master·Indexed 23 days ago

https://github.com/ahmetaa/zemberek-nlp

A specialized Natural Language Processing library for the Turkish language. It provides tools for morphology, tokenization, Named Entity Recognition (NER), and text classification using a Java port of fastText. The library includes a gRPC server for remote access, a language identification system supporting 62 languages, and SmoothLm for compressed n-gram language model implementation.

Tokens
12.5K
Snippets
29
Records
67
Agent score
77%

What's inside Zemberek-NLP

  1. Overview of Zemberek Core Library

    master
    The Zemberek Core Library is a foundational component of the Zemberek NLP ecosystem. It provides specialized data structures and helper classes designed to support the more advanced NLP tasks performed by other Zemberek modules.

  2. Overview of Zemberek-NLP modules

    master

    Zemberek-NLP provides Natural Language Processing tools specifically for Turkish. The project is organized into several specialized modules that can be used individually via Maven:

    • Core (zemberek-core): Special Collections, Hash functions, and helpers.
    • Morphology (zemberek-morphology): Turkish morphological analysis, disambiguation, and word generation.
    • Tokenization (zemberek-tokenization): Turkish Tokenization and sentence boundary detection.
    • Normalization (zemberek-normalization): Basic spell checker, word suggestion, and noisy text normalization.
    • NER (zemberek-ner): Turkish Named Entity Recognition (Note: does not provide a model yet).
    • Classification (zemberek-classification): Text classification based on a Java port of the fastText project.
    • Language Identification (zemberek-lang-id): Fast identification of text language.
    • Language Modeling (zemberek-lm): Language model compression algorithm.
    • Applications (zemberek-apps): Console applications.
    • gRPC Server (zemberek-grpc): gRPC server for access from other programming languages.
    • Examples (zemberek-examples): Usage examples.

  3. What is SmoothLm language model compression?

    master

    SmoothLm is a compressed, optionally quantized, randomized back-off n-gram language model implementation. It uses Minimal Perfect Hash functions for compression, meaning actual n-gram values are not stored in the model.

    Key Characteristics:

    • Lossy Model: It may return an existing n-gram probability for a non-existing n-gram (false positive). The probability of a false positive depends on the fingerprint hash length (e.g., 8, 16, or 24 bits).
    • Quantization: Probability and back-off values can be quantized to 8, 16, or 24 bits for increased compactness.
    • Memory Usage: SmoothLm loads all model data into memory; it does not work directly from disk.
    • Capacity: It can only compress models where the n-gram amount for a given order is less than 2,147,483,648 ($2^{31}-1$).

  4. Initialize the LanguageIdentifier

    master

    To use the language identification library, you must first initialize a LanguageIdentifier instance. You can choose between loading all available models or a specific subset to optimize memory usage.

    • Load all languages: Use LanguageIdentifier.fromInternalModels() to load all 62 supported language models into memory.
    • Load a specific group: If you only need to identify a subset of languages, use LanguageIdentifier.fromInternalModelGroup(String groupName). For example, "tr_group" contains approximately 8 languages plus an *uknown* language ID.

  5. Normalize noisy Turkish text

    master

    The TurkishSentenceNormalizer is designed to correct informal speech or incorrectly written words commonly found in social media, chat, and messaging applications.

    Setup Requirements:

    1. Data Files: You must download the required lookup files and language models (approx. 100 MB). These include a compressed bi-gram language model (lm.2gram.slm) and normalization lookup tables.
    2. Preprocessing: Text should be divided into sentences (using the tokenization module) before being passed to the normalizer.

    Initialization and Usage: Initialize the TurkishSentenceNormalizer with TurkishMorphology, the path to the lookup directory, and the path to the language model file. Use the normalize(String sentence) method to process text.

    Note: The output is typically all lowercase and may occasionally change correct words or formatting.

  6. Create a TurkishMorphology object

    master

    The TurkishMorphology class is the primary entry point for morphological analysis and generation. Because instantiation involves generating a suffix graph and loading dictionaries, it is memory-intensive and time-consuming. You should use a single instance throughout the lifetime of your application.

    Default Initialization

    TurkishMorphology morphology = TurkishMorphology.createWithDefaults();

    Custom Lexicon and Builder

    You can use the RootLexicon.builder() to add default lexicons and custom text dictionaries. Use the TurkishMorphology.builder() to assemble the analyzer.

    RootLexicon lexicon = RootLexicon.builder()
    .addDefaultLexicon()
    .addTextDictionaries(Paths.get("my-dictionary.txt"))
    .build();

TurkishMorphology analyzer = TurkishMorphology.builder()
    .setLexicon(lexicon)
    .build();

    Disabling Cache

    If you use the builder mechanism, you can disable the built-in cache:

    TurkishMorphology analyzer = TurkishMorphology.builder()
    .setLexicon(RootLexicon.getDefault())
    .disableCache()
    .build();

  7. Prepare training data for Turkish NER

    master

    To train a Named Entity Recognition (NER) model, you must prepare a training set file where each sentence is on a new line. Sentences must be tokenized before annotation. Zemberek supports three annotation styles:

    1. Bracket Style: [TYPE text] Example: [ORG Enerji Verimliliği Merkezi] kurucu başkanı [PER Bülent Yeşilata]

    2. OpenNLP Style: <START:TYPE> text <END> Example: <START:ORG> Enerji Verimliliği Merkezi <END>

    3. Enamex Style: <b_enamex TYPE="TYPE">text<e_enamex> Example: <b_enamex TYPE="ORG">Enerji Verimliliği Merkezi<e_enamex>

    Common entity types include PER (Person), ORG (Organization), and LOC (Location), but these are arbitrary and user-defined. It is recommended to reserve approximately 10% of your data as a test set for evaluation.

  8. Generate inflections using Word Generation

    master

    Zemberek provides a word generation mechanism to create surface forms from a root form (or DictionaryItem) and a set of morphemes.

    Key Concepts:

    • Input Requirements: You need a root form or a DictionaryItem and the desired morphemes.
    • Morpheme Handling: The generator automatically handles empty morphemes in the search graph. For example, you do not need to explicitly provide morphemes like A3sg if their surface form is empty.
    • Output: The generate method returns a List of Result objects (an inner static class). Each Result object contains the generated surface form and the corresponding analysis results.
        String[] number = {"A3sg", "A3pl"};
    String[] possessives = {"P1sg", "P2sg", "P3sg"};
    String[] cases = {"Dat", "Loc", "Abl"};

    TurkishMorphology morphology =
        TurkishMorphology.builder().addDictionaryLines("armut").disableCache().build();

    DictionaryItem item = morphology.getLexicon().getMatchingItems("armut").get(0);
    for (String numberM : number) {
      for (String possessiveM : possessives) {
        for (String caseM : cases) {
          List<Result> results =
              morphology.getWordGenerator().generate(item, numberM, possessiveM, caseM);
          results.forEach(s->System.out.println(s.surface));
        }
      }
    }

  9. Train a text classification model with TrainClassifier

    master

    The TrainClassifier application generates a text classification model based on a Java port of the fastText library. It is optimized for sentence and short paragraph level texts.

    Training Set Requirements

    • Each line in the training set should contain a single document.
    • Document class labels must have the __label__ prefix.
      • Example: __label__sports Match ended in a draw.
    • A single document may contain more than one label.
    • It is recommended to apply tokenization, lower-casing, and other text operations to the training set before training.

    Optimization and Parameters

    • Large Label Sets: If you have many labels, use LossType set to HIERARCHICAL_SOFTMAX to increase training and runtime speed (with a small accuracy loss).
    • Compact Models: Use the -applyQuantization and -cutOff [dictionary-cut-off] parameters to generate smaller models.

  10. Resolve morphological ambiguity in sentences

    master

    Because Turkish is highly ambiguous, a single word can have many valid analyses. Zemberek uses an Averaged Perceptron mechanism to resolve this ambiguity at the sentence level.

    Workflow:

    1. Analyze the sentence: Use analyzeSentence(String sentence) to get a List<WordAnalysis>.
    2. Disambiguate: Pass the original sentence and the list of analyses to disambiguate(String sentence, List<WordAnalysis> analysis). This returns a SentenceAnalysis object.
    3. Get best results: Use .bestAnalysis() on the SentenceAnalysis to retrieve the most likely morphological sequence.
    TurkishMorphology morphology = TurkishMorphology.createWithDefaults();
String sentence = "Yarın kar yağacak.";
List<WordAnalysis> analysis = morphology.analyzeSentence(sentence);

SentenceAnalysis after = morphology.disambiguate(sentence, analysis);
after.bestAnalysis().forEach(s -> System.out.println(s.formatLong()));

  12. Enable informal Turkish word analysis

    master

    Zemberek supports analyzing informal Turkish words (e.g., okuycam). To enable this, you must initialize TurkishMorphology with .useInformalAnalysis().

    Informal morpheme names are identified by the _Informal suffix (e.g., Fut_Informal).

    Informal to Formal Conversion

    You can use InformalAnalysisConverter to generate the formal surface form of an informal word analysis. This requires the WordGenerator from your TurkishMorphology instance.

    TurkishMorphology morphology = TurkishMorphology.builder()        
    .setLexicon(RootLexicon.DEFAULT)
    .useInformalAnalysis()
    .build();

// Example conversion
InformalAnalysisConverter converter = new InformalAnalysisConverter(morphology.getWordGenerator());
String formalForm = converter.convert(analysis.surfaceForm(), analysis);