PHP NlpTools

The LancasterStemmer constructor accepts an optional $ruleSet array. If an empty array is provided, the class defaults to its internal English ruleset.

Each rule in the array is an associative array containing the following keys:

• lookup_char: The character used to index the rule for faster lookup.
• ending_string: The suffix string to match.
• intact_flag: A flag (e.g., *) indicating if the rule should only apply if the word remains 'intact'.
• remove_total: The number of characters to remove from the end of the word.
• append_string: The string to append after removal.
• continue_flag: Controls the loop behavior (e.g., . to stop or > to continue).

use NlpTools\Stemmers\LancasterStemmer;

// Example of a custom rule set
$customRules = [
    [
        'lookup_char' => 'x',
        'ending_string' => 'xyz',
        'intact_flag' => '*',
        'remove_total' => '3',
        'append_string' => 'abc',
        'continue_flag' => '.'
    ]
];

$stemmer = new LancasterStemmer($customRules);

PHP NlpTools provides several categories of tools for NLP workflows:

Classification Models

• Multinomial Naive Bayes
• Maximum Entropy (Conditional Exponential model)

Topic Modeling

• Latent Dirichlet Allocation (Lda): Note that Lda is currently experimental and can be slow.

Clustering

• K-Means
• Hierarchical Agglomerative Clustering (supports SingleLink, CompleteLink, and GroupAverage)

Tokenizers

• WhitespaceTokenizer
• WhitespaceAndPunctuationTokenizer
• PennTreebankTokenizer
• RegexTokenizer
• ClassifierBasedTokenizer: Used for building complex, custom tokenizers.

Documents

• TokensDocument: Represents a bag of words model.
• WordDocument: Represents a single word within the context of a larger document.
• TrainingDocument: Represents a document with a known class.
• TrainingSet: A collection of TrainingDocument objects.

Feature Factories

• FunctionFeatures: Creates a feature factory from multiple callables.
• DataAsFeatures: Returns raw data as features.

Similarity Measures

• JaccardIndex
• CosineSimilarity
• Simhash
• Euclidean
• HammingDistance

Stemmers

• PorterStemmer
• RegexStemmer
• LancasterStemmer
• GreekStemmer

Optimizers (MaxEnt only)

• Gradient Descent: A simple PHP implementation for educational purposes.
• External Maxent Optimizer: A high-performance, parallel gradient descent optimizer written in Go, used via the ExternalMaxentOptimizer class.

The RegexTokenizer class implements TokenizerInterface and allows you to segment text using a sequence of regular expressions. It processes patterns iteratively: the output of one pattern becomes the input for the next.

There are three modes of operation based on the structure of the pattern provided in the constructor array:

1. Split Mode: If a pattern is provided as a single element (e.g., ['/pattern/']), it uses preg_split to break strings into smaller tokens. Empty results are discarded.
2. Match Mode: If a pattern is provided with an integer (e.g., ['/pattern/', 1]), it uses preg_match_all to extract specific capture groups. The integer specifies which index of the match array to keep.
3. Transform Mode: If a pattern is provided with a string (e.g., ['/pattern/', '/replacement/']), it uses preg_replace to transform existing tokens.

use NlpTools\Tokenizers\RegexTokenizer;

// Example: Splitting by whitespace, then matching words, then transforming
$patterns = [
    '/\s+/',               // Split mode: split by whitespace
    ['/(\w+)/', 0],        // Match mode: keep the whole match (index 0)
    ['/\w/', 'X']          // Transform mode: replace characters with 'X'
];

$tokenizer = new RegexTokenizer($patterns);
$tokens = $tokenizer->tokenize("Hello world");

The FunctionFeatures class implements FeatureFactoryInterface and allows you to generate features by executing a collection of callables (closures, function names, or array-based callables like [$object, 'method']).

Each callable is passed two arguments: the $class (a string representing the category/class) and the $d (a DocumentInterface instance).

Feature Extraction Logic:

• If a callable returns a string, that string is added as a feature.
• If a callable returns an array, every element in that array is added as an individual feature.
• Return values that evaluate to false are ignored.

Modeling Frequency vs. Presence:

• By default, the factory models presence (returning a list of unique feature strings).
• You can switch to modeling frequency (returning an associative array where keys are feature strings and values are their counts) using modelFrequency().

use NlpTools\FeatureFactories\FunctionFeatures;

// Initialize with an array of callables
$factory = new FunctionFeatures([
    function($class, $document) { return 'some_feature'; },
    [$myObject, 'extractFeatures']
]);

// To return counts instead of just unique keys:
$factory->modelFrequency();

// To return only unique keys (presence):
$factory->modelPresence();

// Get the features for a specific class and document
$features = $factory->getFeatureArray('my_class_name', $document);