Catalyst NLP Library

The ImmutableDocument class provides an immutable, memory-efficient representation of a document. Use this when you need to ensure document data remains unchanged after processing. You can convert between mutable and immutable states using ToImmutable() and ToMutable().

// Convert to ImmutableDocument
ImmutableDocument immutableDoc = doc.ToImmutable();

// Convert back to mutable Document
Document mutableDoc = immutableDoc.ToMutable();

A Neuralyzer is a special component that can be added to a pipeline via nlp.UseNeuralyzer(neuralyzer). It allows you to define patterns to correct errors made by other models, such as adding or forgetting entities based on specific token patterns.

var neuralyzer = new Neuralyzer(Language.English, 0, "fixes");

// Teach the neuralyzer to add an 'Organization' entity when it sees 'Amazon'
neuralyzer.TeachAddPattern("Organization", "Amazon", mp => mp.Add(new PatternUnit(P.Single().WithToken("Amazon"))));

nlp.UseNeuralyzer(neuralyzer);

To use Catalyst, install the Catalyst NuGet package and the specific language package (e.g., Catalyst.Models.English). You must register the language before creating a pipeline. Catalyst uses a Storage mechanism to lazy-load models from disk or an online repository.

1. Register the language.
2. Configure Storage.Current (e.g., using DiskStorage).
3. Create a pipeline using Pipeline.ForAsync(Language).
4. Create a Document and process it using nlp.ProcessSingle(doc) or nlp.Process(docs) for batches.

Catalyst.Models.English.Register(); // Register the language

Storage.Current = new DiskStorage("catalyst-models");
var nlp = await Pipeline.ForAsync(Language.English);
var doc = new Document("The quick brown fox jumps over the lazy dog", Language.English);

nlp.ProcessSingle(doc);
Console.WriteLine(doc.ToJson());

The Pipeline class is the central orchestrator in Catalyst. To create a pipeline, you must first install the language-specific NuGet package and register its models.

Use Pipeline.ForAsync(Language) to create a default pipeline, which typically includes a tokenizer, a sentence detector, and a POS tagger.

# Example: Adding English language support
dotnet add package Catalyst.Models.English

// Register the language models
Catalyst.Models.English.Register();

// Create the default pipeline
var nlp = await Pipeline.ForAsync(Language.English);

The Document class is the primary data structure in Catalyst. It represents the text being processed and serves as a container for all linguistic annotations generated by the NLP pipeline. You can instantiate a new document by providing the raw text and its corresponding Language.

using Catalyst;
using Mosaik.Core;

var doc = new Document("The quick brown fox jumps over the lazy dog", Language.English);

After processing a document with an NER model, you can access entities by iterating through the document spans or using LINQ to flatten the entity collection.

// Using foreach
foreach (var span in doc)
{
    foreach (var entity in span.GetEntities())
    {
        Console.WriteLine($"Entity: {entity.Value} [{entity.EntityType.Type}]");
    }
}

// Using LINQ
var entities = doc.SelectMany(span => span.GetEntities());
foreach(var entity in entities)
{
    Console.WriteLine($"Entity: {entity.Value} [{entity.EntityType.Type}]");
}

Pipelines are flexible and can be modified by adding or removing components that implement IProcess.

• Adding Processes: Use nlp.Add(process) to add a model. Catalyst automatically maintains a logical execution order: Normalizers $\rightarrow$ Tokenizers $\rightarrow$ Sentence Detectors $\rightarrow$ Taggers $\rightarrow$ Others (e.g., Entity Recognizers).
• Removing Processes: Use nlp.RemoveAll(predicate) to remove specific models based on a condition.

// Adding an Entity Recognizer
var nlp = await Pipeline.ForAsync(Language.English);
nlp.Add(await AveragePerceptronEntityRecognizer.FromStoreAsync(Language.English, Version.Latest, "WikiNER"));

// Removing all taggers from the pipeline
nlp.RemoveAll(p => p is ITagger);

If you only require tokenization (and optionally sentence detection), use Pipeline.TokenizerForAsync. This is faster than the default pipeline because it skips the part-of-speech (POS) tagging step. You can also use Pipeline.ForAsync and explicitly set tagger: false to achieve a similar result.

// Option 1: Using the specialized tokenizer method
var nlp = await Pipeline.TokenizerForAsync(Language.English, sentenceDetector: true);

// Option 2: Disabling the tagger in the default pipeline
var nlp = await Pipeline.ForAsync(Language.English, tagger: false);

You can persist a configured pipeline and all its associated models into a single binary file using PackTo and reload it using Pipeline.LoadFromPackedAsync.

// Store the pipeline to a file
using(var f = File.OpenWrite("my-pipeline.bin"))
{
    nlp.PackTo(f);
}

// Load the pipeline from a file
using(var f = File.OpenRead("my-pipeline.bin"))
{
    var nlp2 = await Pipeline.LoadFromPackedAsync(f);
}

To use Catalyst, install the core Catalyst NuGet package. Because Catalyst is modular, language-specific data and models are provided in separate NuGet packages. You must install the package for the language you intend to use and register it in your code before processing.

# Install the core package (assumed)
dotnet add package Catalyst

# Install English language support
dotnet add package Catalyst.Models.English