quantulum3

To add support for a new language in quantulum3, you must provide a set of files and functions that follow the project's structure. This involves defining how numbers are pluralized, how they are converted to words, how they are matched via regex, how they are parsed/classified, and providing training/test data.

Required Components

1. load.py

Implement the following functions:

• pluralize(singular, count): Converts a word to its plural form based on the count. If count is None, return the plural form.
• number_to_words(number): Converts a float into a pronounceable string.

2. regex.py

Define regex patterns for number recognition:

• TEXT_PATTERN: A regex pattern for numbers spelled out in text. It must include the formatting groups used in the project's template. Note that braces intended to be part of the final regex must be escaped (e.g., {{3}} becomes {3}).
• NUM_PATTERN: A regex pattern describing the general form of a number. It must contain specific formatting groups (marked by {}) for number, decimals, scale, and fraction to ensure compatibility with the global parser.
• numberwords_regex: A concatenation of all possible number words for the language, joined by |.

3. parser.py, classifier.py, and speak.py

Implement the logic for parsing, unit classification, and spoken output. Refer to the en_US directory for reference implementations and required function signatures.

4. Training Data (train/)

Provide .json files containing lists of dictionaries for the classifier. Each dictionary must have:

• unit: The unit associated with the text.
• text: The text associated with the unit (used for tokenization and stemming).

5. Test Data (tests/)

Provide .json files to validate the language implementation. Tests are categorized into:

• expand: Validates parsing, text replacement, and pluralization.
• quantities: Validates unit parsing without the classifier.
• quantities.ambiguity: Validates the classifier's ability to disambiguate units.

6. Units and Entities (units.json, entities.json)

Create these files to define language-specific units and entities. These will override or supplement the global definitions. You must override the surfaces and URI fields for every language.

Use quantulum3.parser.parse() to extract quantities, measurements, and their units from unstructured text. The function returns a list of Quantity objects.

Key attributes of a Quantity object:

• value: The parsed numeric value (float).
• unit.name: The name of the parsed unit.
• unit.entity: The associated entity (e.g., volume).
• unit.uri: The Wikipedia URI for the unit.
• surface: The original text string extracted.
• span: The (start, end) character positions in the original text.
• uncertainty: The parsed uncertainty value (if present).

You can also use parser.inline_parse() to get a string where the extracted quantities are embedded as Quantity objects, which is useful for debugging.

from quantulum3 import parser

# Basic parsing
quants = parser.parse('I want 2 liters of wine')
# Output: [Quantity(2, 'litre')]

# Accessing attributes
print(quants[0].value)       # 2.0
print(quants[0].unit.name)   # 'litre'
print(quants[0].surface)     # '2 liters'
print(quants[0].span)        # (7, 15)

# Debugging with inline parsing
print(parser.inline_parse('I want 2 liters of wine'))
# Output: I want 2 liters {Quantity(2, "litre")} of wine

# Parsing dimensionless numbers
print(parser.parse('I want two'))
# Output: [Quantity(2, 'dimensionless')]

You can add new units to the units.json file to expand the library's extraction capabilities.

Each entry in units.json must follow this schema:

• Key: The unique name of the unit.
• URI: The Wikipedia page name.
• surfaces: A list of strings that refer to the unit (the library handles plurals automatically).
• entity: The name of the corresponding entity defined in entities.json.
• dimensions: A list of dictionaries defining dimensionality, where base refers to the name of another unit.
• symbols: A list of possible symbols or abbreviations (e.g., ["mps"]).
• prefixes (optional): A list of Metric or Binary prefixes (e.g., ["k", "M"]). Adding prefixes automatically generates related units. To add specific surfaces for a prefixed unit, you must create a separate entry for that specific version.

"metre per second": {
    "surfaces": ["metre per second", "meter per second"],
    "entity": "speed",
    "URI": "Metre_per_second",
    "dimensions": [{"base": "metre", "power": 1}, {"base": "second", "power": -1}],
    "symbols": ["mps"]
},
"year": {
    "surfaces": [ "year", "annum" ],
    "entity": "time",
    "URI": "Year",
    "dimensions": [],
    "symbols": [ "a", "y", "yr" ],
    "prefixes": [ "k", "M", "G", "T", "P", "E" ]
}

To replace the default unit and entity sets with your own, use load_custom_units and load_custom_entities. The recommended approach is to use a context manager to ensure default units are reloaded after use.

Using a context manager (Recommended):

from quantulum3 import load, parser

with load.CustomQuantities(["path/to/units.json"], ["path/to/entities.json"]):
    # Only your custom units/entities are available here
    results = parser.parse("...")

# Default units/entities are automatically restored here

Manual loading/unloading:

from quantulum3 import load, parser

# Load custom sets
load.load_custom_units(["path/to/units.json"])
load.load_custom_entities(["path/to/entities.json"])

# Reset to defaults
load.reset_quantities()

from quantulum3 import load, parser

# Using context manager for custom sets
with load.CustomQuantities(["path/to/units.json"], ["path/to/entities.json"]):
    results = parser.parse("This extremely sharp tool is precise up to 0.5 slp")

If the parser finds multiple matching units, it uses a classifier to disambiguate based on Wikipedia pages and GloVe vector representations. You can train your own classifier using the quantulum3-training CLI tool or the train_classifier method.

CLI Usage:

quantulum3-training --lang <language> --data <path/to/training/file.json> --output <path/to/output/file.joblib

Using a custom model in code: After training, pass the path to the .joblib file to the parser.

Note: Training requires the [classifier] installation extra.

from quantulum3 import parser

# Use a custom trained model
# Note: parser is typically used via the module-level parser or a Parser instance
# The documentation suggests: parser = Parser.parse(<text>, classifier_path="...")
# Assuming Parser class is available:
from quantulum3 import Parser

custom_parser = Parser(classifier_path="path/to/model.joblib")
# Then use custom_parser.parse(text)