DOCSEARCHbyAlgolia
Submit Docs

SimAlign Documentation

repository·master·Indexed 19 days ago

https://github.com/cisnlp/simalign

A word alignment tool utilizing static and contextualized embeddings that does not require parallel training data. Features the SentenceAligner API for performing alignments and the calc_align_score.py script for evaluating alignment quality against gold standards.

Tokens
771
Snippets
2
Records
3
Agent score
14%

What's inside SimAlign

  1. Install SimAlign via pip

    master

    You can install SimAlign using PyPi or directly from the GitHub repository.

    Prerequisites:

    • Python 3.7
    • Transformers 3.1.0
    • Torch 1.5.0
    • networkx (optional, only required for the Match algorithm)

    Installation commands:

    # Install via PyPi
pip install simalign

# Or install directly from GitHub
pip install --upgrade git+https://github.com/cisnlp/simalign.git#egg=simalign
    pip install simalign

  2. Use the SentenceAligner API

    master

    The SentenceAligner class is the primary interface for performing word alignments.

    Initialization: You can specify the embedding model and alignment settings in the constructor. Common parameters include model, token_type, and matching_methods.

    Alignment Process:

    1. Initialize SentenceAligner.
    2. Provide source and target sentences as lists of tokens (they must be pre-tokenized to words).
    3. Call get_word_aligns(src_sentence, trg_sentence).

    Output Format: The method returns a dictionary where keys are the matching methods used and values are lists of tuples. Each tuple contains a pair of zero-indexed integers representing the aligned word indices (source_index, target_index).

    from simalign import SentenceAligner

# Initialize the aligner
myaligner = SentenceAligner(model="bert", token_type="bpe", matching_methods="mai")

# Sentences must be tokenized to words
src_sentence = ["This", "is", "a", "test", "."]
trg_sentence = ["Das", "ist", "ein", "Test", "."]

# Get alignments
alignments = myaligner.get_word_aligns(src_sentence, trg_sentence)

# Example iteration over results
for matching_method in alignments:
    print(matching_method, ":", alignments[matching_method])

# Expected output format:
# mwmf (Match): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]
# inter (ArgMax): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]
# itermax (IterMax): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]

  3. Evaluate alignments with calc_align_score.py

    master

    To evaluate the quality of your generated alignments against a gold standard, use the scripts/calc_align_score.py script.

    Requirements:

    • The gold alignment file must follow the same format as SimAlign outputs.
    • In the gold standard format:
      • Sure alignment edges use a hyphen (-) between source and target indices (e.g., 0-0).
      • Possible edges use a p between indices (e.g., 0p1).

    For sample parallel sentences and gold alignments, refer to the samples directory in the repository.