CAMeL Tools Documentation

The camel_tools.disambig module provides tools for morphological disambiguation in Arabic. It contains several sub-modules implementing different disambiguation strategies:

• camel_tools.disambig.mle: Maximum Likelihood Estimation (MLE) based disambiguation.
• camel_tools.disambig.bert: Disambiguation using BERT-based models.
• camel_tools.disambig.common: Common utilities and base classes used across disambiguation modules.

To use these tools, you should select the module that best fits your performance and accuracy requirements (e.g., MLE for speed/efficiency or BERT for higher accuracy via transformer models).

When performing word disambiguation with camel_tools, the results are encapsulated in two primary data structures:

1. ScoredAnalysis: Represents a single morphological analysis associated with a confidence score.
2. DisambiguatedWord: Represents the final result for a specific word, typically containing the chosen analysis and potentially other metadata.

These classes are part of the camel_tools.disambig.common module and are used to handle the output of disambiguation processes.

The camel_tools.data module uses several specialized classes to represent different types of data entities:

• PackageType: Defines the category of the package being handled.
• PackageEntry: Represents a specific package within the system.
• DatasetEntry: Represents a specific dataset.
• ComponentEntry: Represents a specific component of the CAMeL Tools suite.
• FileEntry: Represents an individual file within a package or dataset.

The MorphologyDB class is responsible for parsing morphology database files and generating the indexes required by the morphological components of CAMeL Tools.

Note: You should not interact with MorphologyDB instances directly for processing. Instead, you create a MorphologyDB instance and pass it as an argument when initializing the Analyzer, Generator, or Reinflector components.

Markers are used to protect specific tokens from being diacritized (e.g., foreign words or names).

• Definition: A marker must be a single token without padding spaces (no whitespace at the beginning, middle, or end).
• Default: The default marker is @@IGNORE@@.
• Best Practice: Choose a marker that is unlikely to appear naturally in your text. For example, some NLP tools use @@LAT@@ to denote Latin/foreign text.
• Behavior: By default, tokens prefixed with the marker are skipped. Use the -I (--ignore-markers) flag to force diacritization of marked words, or the -S (--strip-markers) flag to remove the markers from the final output.

In camel_dediac, a marker is a string (without whitespace) used to identify tokens that should be treated specially.

By default, camel_dediac uses the marker @@IGNORE@@ to identify tokens that should not be de-diacritized.

Best Practices:

• Choose a marker that is unlikely to appear naturally in your text.
• Common alternatives include @@LAT@@ (often used by other Arabic NLP tools to denote Latin/foreign text).

Behavioral Flags:

• --marker <MARKER>: Sets the custom marker.
• --ignore-markers: Forces the tool to de-diacritize even the words prefixed with the marker.
• --strip-markers: Removes the marker from the output text (use this in conjunction with --ignore-markers).

CAMeL Tools uses pytest for testing, tox for automating tests across different Python environments, and pytest-cov for coverage reporting.

To run the full test suite across all configured Python environments, install tox and execute the tox command.

# Install tox
pip install tox

# Run the tests
tox

The camel_arclean utility is used to clean Arabic text by performing the following transformations:

• Deleting characters that are not in Arabic, ASCII, or Latin-1.
• Converting all spacing characters to an ASCII space character.
• Converting Indic digits into Arabic digits.
• Converting extended Arabic letters into basic Arabic letters.
• Converting 1-char presentation forms into simple basic forms.

Note

CAMeL Tools has been tested on Windows 10. The Dialect Identification component is currently unavailable on Windows.

Installation via pip

Use the following command to ensure compatibility with PyTorch dependencies:

pip install camel-tools -f https://download.pytorch.org/whl/torch_stable.html

Installation from source

# Clone the repo
git clone https://github.com/CAMeL-Lab/camel_tools.git
cd camel_tools

# Install from source
pip install -f https://download.pytorch.org/whl/torch_stable.html .

If you want to build a local HTML copy of the documentation, follow these steps:

1. Install documentation dependencies:

pip install sphinx myst-parser sphinx-rtd-theme

2. Navigate to the docs directory and build:

cd docs
make html

The compiled HTML files will be located in docs/build/html.

In Analyze Mode, camel_morphology identifies all possible out-of-context combinations of lemmas, morphological features, and diacritizations for undiacritized surface words.

Input Requirements

• The input must be a space-separated list of words.
• Punctuation must also be space-separated; otherwise, Arabic words adjacent to punctuation will be treated as foreign words.

Output Format

For every space-separated word in the input, the tool outputs:

1. A line starting with #WORD: followed by the word.
2. One or more lines containing feature-value pairs for each generated analysis.
3. If no analyses are possible, it outputs NO_ANALYSIS.
4. An empty line follows the analyses for a word.
5. An additional empty line is printed to indicate the end of the input stream.

$ camel_morphology analyze
مشيت في الشارع

To install from the repository source, clone the repository and use pip to install the local directory.

# Clone the repo
git clone https://github.com/CAMeL-Lab/camel_tools.git
cd camel_tools

# Install from source
pip install .