DOCSEARCHbyAlgolia
Submit Docs

spacymoji

repository·master·Indexed 41 days ago

https://github.com/explosion/spacymoji

A spaCy extension for detecting emojis in text and adding metadata to spaCy Doc, Token, and Span objects. It provides human-readable descriptions and the ability to merge multi-character emojis into single tokens. Compatible with spaCy v3.0.0+ (or v2.0.0 for spaCy v2.x).

Tokens
1.2K
Snippets
5
Records
5
Agent score
2%

What's inside spacymoji

  2. Add the emoji component to a spaCy pipeline

    master

    To use spacymoji, import the component and add it to your spaCy pipeline using the string name `

    import spacy

nlp = spacy.load("en_core_web_sm")
nlp.add_pipe("emoji", first=True)
doc = nlp("This is a test 😻 👍🏿")

# Accessing attributes
assert doc._.has_emoji is True
assert doc[4]._.is_emoji is True
assert doc[5]._.emoji_desc == "thumbs up dark skin tone"

  3. Configure the emoji component settings

    master

    You can customize the behavior of the emoji component by passing a config dictionary to nlp.add_pipe().

    Available configuration keys:

    • attrs (Tuple[str, str, str, str]): Custom names for the attributes set on the ._ property. Defaults to ('has_emoji', 'is_emoji', 'emoji_desc', 'emoji').
    • pattern_id (str): The ID of the match pattern. Defaults to 'EMOJI'.
    • merge_spans (bool): Whether to merge spans containing multi-character emoji into one token. Defaults to True.
    • lookup (Dict[str, str]): An optional dictionary mapping emoji strings to custom descriptions (e.g., for translations).
    import spacy

# Example of custom attribute names and a custom lookup table
emoji_config = {
    "attrs": ("has_e", "is_e", "e_desc", "e"), 
    "lookup": {"👨‍🎤": "David Bowie"}
}

nlp = spacy.blank("en")
nlp.add_pipe("emoji", first=True, config=emoji_config)

doc = nlp("We can be 👨‍🎤 heroes")
assert doc[3]._.is_e
assert doc[3]._.e_desc == "David Bowie"

  4. Configure emoji component in spaCy config.cfg

    master

    When training a spaCy pipeline, you can define the emoji component configuration directly in your config.cfg file under the [components.emoji] section.

    [nlp]
pipeline = ["emoji", "ner"]
# ...

[components.emoji]
factory = "emoji"
merge_spans = false

  5. Use spacymoji custom attributes

    master

    Once the emoji component is in your pipeline, it populates several custom attributes on Doc, Token, and Span objects via the ._. property. By default, these attributes are:

    | Attribute            | Type                       | Description                                                               |
| -------------------- | -------------------------- | -------------------------------------------------------------------------- |
| `Token._.is_emoji`   | bool                       | Whether the token is an emoji.                                             |
| `Token._.emoji_desc` | str                        | A human-readable description of the emoji.                                 |
| `Doc._.has_emoji`    | bool                       | Whether the document contains emoji.                                       |
| `Doc._.emoji`        | List[Tuple[str, int, str]] | `(emoji, index, description)` tuples of the document's emoji.              |
| `Span._.has_emoji`   | bool                       | Whether the span contains emoji.                                           |
| `Span._.emoji`       | List[Tuple[str, int, str]] | `(emoji, index, description)` tuples of the span's emoji.                  |