sense2vec

repository·master·Indexed 23 days ago

https://github.com/explosion/sense2vec

A library for contextually-keyed word vectors that incorporates part-of-speech tags and entity labels to provide detailed semantic representations. It can be used as a standalone class for vector management and similarity tasks or integrated as a spaCy v3 pipeline component. The library includes tools for training custom vectors using GloVe or fastText, a Streamlit interactive demo, and several Prodigy recipes for bootstrapping terminology lists and evaluating models.

Tokens
6.4K
Snippets
20
Records
24
Agent score
32%

What's inside sense2vec

  1. Understand pretrained Reddit vector senses

    master
    The pretrained Reddit vectors in sense2vec use 'senses' to provide contextually-keyed word vectors. These senses are represented by either Part-of-Speech (POS) tags or Named Entity Recognition (NER) entity labels. When using pretrained vectors, the similarity scores will be calculated based on these specific tags/labels attached to the words.
  2. Customize Sense2Vec using the registry

    master

    The registry (powered by catalogue) allows you to swap out the logic used to generate keys, split keys, or extract phrases. You can define custom functions, register them, and then pass them to Sense2Vec or Sense2VecComponent via the overrides dictionary.

    Available Registry Names:

    • registry.make_key: Generates a string key from a word and sense (e.g., word|sense).
    • registry.split_key: Splits a string key back into a (word, sense) tuple.
    • registry.make_spacy_key: Generates a (word, sense) tuple from a spaCy Token or Span.
    • registry.get_phrases: Returns a list of Span objects (e.g., noun phrases) for sense2vec.
    • registry.merge_phrases: Merges sense2vec phrases into single tokens in a Doc.

    Example Customization:

    from sense2vec import registry
    
    @registry.make_key.register("custom")
    def custom_make_key(word, sense):
        return f"{word}###{sense}"
    
    @registry.split_key.register("custom")
    def custom_split_key(key):
        word, sense = key.split("###")
        return word, sense
    
    # Use the custom functions via overrides
    overrides = {"make_key": "custom", "split_key": "custom"}
    s2v = Sense2Vec(overrides=overrides)
    from sense2vec import registry
    
    @registry.make_key.register("custom")
    def custom_make_key(word, sense):
        return f"{word}###{sense}"
    
    @registry.split_key.register("custom")
    def custom_split_key(key):
        word, sense = key.split("###")
        return word, sense
    
    overrides = {"make_key": "custom", "split_key": "custom"}
    s2v = Sense2Vec(overrides=overrides)
  3. Perform A/B evaluation with `sense2vec.eval-ab`

    master

    Compare two pretrained sense2vec vector models by comparing the most similar entries they return for a random phrase. The UI shows two randomized options (one from each model) and highlights the phrases that differ. At the end of the session, overall stats and the preferred model are displayed.

    prodigy sense2vec.eval-ab vectors_eval_sim /path/to/s2v_reddit_2015_md /path/to/s2v_reddit_2019_md --senses NOUN,ORG,PRODUCT
  4. Evaluate similarity with `sense2vec.eval-most-similar`

    master

    Evaluate a vectors model by looking at the most similar entries it returns for a random phrase and unselecting the mistakes.

    prodigy sense2vec.eval-most-similar vectors_eval_sim /path/to/s2v_reddit_2015_md --senses NOUN,ORG,PRODUCT
  5. Bootstrap a terminology list with `sense2vec.teach`

    master

    Use the sense2vec.teach Prodigy recipe to bootstrap a terminology list. Prodigy suggests similar terms based on the most similar phrases from your sense2vec vectors. As you annotate and accept phrases, the suggestions are adjusted. For each seed term, the best matching sense according to the vectors is used.

    To use this recipe, sense2vec must be installed in the same environment as Prodigy.

    prodigy sense2vec.teach tech_phrases /path/to/s2v_reddit_2015_md --seeds "natural language processing, machine learning, artificial intelligence"
  6. Train your own sense2vec vectors

    master

    To train custom sense2vec vectors, you must follow a multi-step pipeline using scripts located in the /scripts directory. The process is designed to be parallelizable by operating on single files and allows for resuming at different stages.

    Requirements

    • Large Text Corpus: Ideally at least 1 billion words (sense2vec vocabulary is sparser than word2vec).
    • Pretrained spaCy Model: Must provide part-of-speech tags, dependencies, named entities, and doc.noun_chunks. If your language lacks a syntax iterator for noun phrases, you must implement one, as doc.noun_chunks and doc.ents are used to identify phrases.
    • GloVe or fastText: Must be installed and built (via make or cmake). For fastText on Windows, an unofficial binary build may be required.

    Training Pipeline Steps

    StepScriptDescription
    101_parse.pyUses spaCy to parse raw text into binary DocBin collections.
    202_preprocess.pyConverts Doc objects into sense2vec format (one sentence per line with merged phrases and senses).
    303_glove_build_counts.pyBuilds vocabulary and counts using GloVe. (Skip if using fastText).
    404_glove_train_vectors.py or 04_fasttext_train_vectors.pyTrains the vectors using GloVe or fastText.
    505_export.pyExports vectors and frequencies into a sense2vec component compatible with Sense2Vec.from_disk.
    606_precompute_cache.pyOptional: Precomputes nearest-neighbor queries to accelerate Sense2Vec.most_similar calls.

    For detailed usage, run the scripts with the --help flag, e.g., python scripts/01_parse.py --help.

  7. Use sense2vec as a spaCy v3 pipeline component

    master

    The easiest way to integrate sense2vec is as a spaCy pipeline component. This adds Sense2VecComponent to your pipeline, which provides extension attributes and methods to Token and Span objects.

    It is recommended to add the component to the end of the pipeline so it has access to the dependency parse and named entities.

    import spacy
    from sense2vec import Sense2VecComponent
    
    nlp = spacy.load("en_core_web_sm")
    s2v = nlp.add_pipe("sense2vec")
    s2v.from_disk("/path/to/s2v_reddit_2015_md")
    
    doc = nlp("A sentence about natural language processing.")
    # Accessing attributes via the ._ property
    freq = doc[3:6]._.s2v_freq
    vector = doc[3:6]._.s2v_vec
    most_similar = doc[3:6]._.s2v_most_similar(3)
  8. Convert phrases to spaCy patterns with `sense2vec.to-patterns`

    master

    Convert a dataset of phrases (collected via sense2vec.teach) into token-based match patterns. These patterns can be used with spaCy's EntityRuler or recipes like ner.match. Multi-token terms are represented as tokenized lists (e.g., [{"LOWER": "new"}, {"LOWER": "balance"}]). If no --output-file is specified, patterns are written to stdout.

    prodigy sense2vec.to-patterns tech_phrases en_core_web_sm TECHNOLOGY --output-file /path/to/patterns.jsonl
  9. Evaluate a sense2vec model with `sense2vec.eval`

    master

    Evaluate a sense2vec model by presenting phrase triples: is word A more similar to word B, or to word C? If the human annotator agrees with the model, the model is considered good. The recipe only asks about vectors with the same sense and supports various selection strategies.

    Strategies:

    • most_similar (default): Picks a random word from a random sense and gets its most similar entries of the same sense. Asks about similarity to the last and middle entry.
    • most_least_similar: Picks a random word from a random sense and gets the least similar entry from its most similar entries, then the last most similar entry of that.
    • random: Picks a random sample of 3 words from the same random sense.
    prodigy sense2vec.eval vectors_eval /path/to/s2v_reddit_2015_md --senses NOUN,ORG,PRODUCT --threshold 0.5
  10. Configure sense2vec in a spaCy training config

    master

    To include a sense2vec component in a packaged spaCy pipeline, use the [initialize] block in your training configuration to specify the data path.

    [initialize.components]
    
    [initialize.components.sense2vec]
    data_path = "/path/to/s2v_reddit_2015_md"
  11. Use sense2vec in standalone mode

    master

    You can use the Sense2Vec class directly without spaCy. Load pretrained vectors using the .from_disk() method.

    Important: Keys must follow the format phrase_text|SENSE (using underscores instead of spaces and a pipe before the tag/label, e.g., machine_learning|NOUN). The vector table is case-sensitive.

    from sense2vec import Sense2Vec
    
    s2v = Sense2Vec().from_disk("/path/to/s2v_reddit_2015_md")
    query = "natural_language_processing|NOUN"
    assert query in s2v
    vector = s2v[query]
    freq = s2v.get_freq(query)
    most_similar = s2v.most_similar(query, n=3)