Morfessor 2.0

Morfessor 2.0 provides a Python library interface for integration into other Python applications. The interface is organized into three main functional areas:

• IO (morfessor.io): Handles reading and writing data, including corpus files, annotation files (gold standards), and binary model files.
• Model classes (morfessor.baseline): Contains the core model implementations (e.g., BaselineModel) used for training and segmentation.
• Evaluation classes (morfessor.evaluation): Provides tools for measuring model performance and conducting statistical significance tests (e.g., MorfessorEvaluation, WilcoxonSignedRank).

Morfessor 2.0 is designed for general string segmentation tasks. To use the tool correctly, understand its core abstractions which are independent of the specific task (e.g., morphology or chunking):

• Compounds: The largest sequences that a single construction can hold (e.g., word forms in morphology, or sentences in chunking).
• Constructions: The set of patterns found by the algorithm that describe the training corpus efficiently and accurately (e.g., morphs in morphology, or phrases in chunking).
• Atoms: The smallest constituent pieces of both constructions and compounds (e.g., characters in morphology, or words in chunking).

Morfessor supports different training paradigms depending on your data and goals:

• Batch Training: Each epoch is an iteration over the full training data. Epochs repeat until model cost converges. All data must be loaded before training starts.
• Online Training: The model is updated while data is being read. This is useful for rapid prototyping. Data is processed only once, so it is recommended to run batch training after online training.
• Recursive Training: For each compound, the current split is removed, and all possible splits are tried recursively. The best split is then selected.
• Local Viterbi Training: Compounds are processed sequentially. Each compound is removed from the corpus, segmented using Viterbi segmentation, and the result is put back into the model. Requires a non-zero --viterbi-smoothing parameter to allow new constructions.

The corpusweight ($\alpha$) balances the cost of the lexicon against the cost of the corpus. Morfessor provides several methods to tune this parameter during training:

• Fixed weight: Set a constant weight at the start of training using --corpusweight.
• Development set: Use a development set via --develset to balance the weight so that precision and recall on the development set are equal.
• Morph length: Tune the weight via --morph-length so the average length of morphs in the lexicon matches a target value.
• Num morph types: Tune the weight via --num-morph-types to target a specific approximate number of morph types in the lexicon.

The easiest way to install Morfessor 2.0 and its associated tools is via the Python Package Index (PyPI) using pip or easy_install. This method automatically handles downloading and installing the latest version.

It is recommended to run Morfessor within a virtual environment.

pip install morfessor
# or
easy_install morfessor

If you have downloaded a Morfessor 2.0 tarball or zip file from the Morpho project homepage or the Morfessor GitHub releases page, you can install it using one of the following two methods:

1. Using setup.py: Unpack the file and run the installation script directly.
2. Using pip: Point pip directly at the compressed archive.

Morfessor 2.0 uses standard Python packaging and is compatible with virtual environments.

# Method 1: Unpack and run setup.py
python setup.py install

# Method 2: Install directly from the archive
pip install morfessor-VERSION.tar.gz

To build the Morfessor 2.0 documentation from source, you need the sphinx and sphinxcontrib-napoleon packages. You can install these along with the project's documentation dependencies using pip with the [docs] extra:

pip install -e .[docs]

Once Sphinx is installed, navigate to the docs directory to generate documentation in various formats using the provided Makefile (Linux/macOS) or make.bat (Windows).

Common commands include:

• Generate a PDF: make latexpdf
• Generate a single HTML file: make singlehtml
• List all available formats: make help

# Example: Generate a single HTML file
make singlehtml

You can access the Morfessor 2.0 documentation in several ways:

• Online: The hosted documentation is available at http://morfessor.readthedocs.org/.
• Local Source: User instructions are located in the docs/ directory as Sphinx source files.
• Building Docs: To build the documentation locally, refer to the instructions in docs/README.