ekphrasis

ekphrasis is a collection of lightweight text tools specifically designed for social network text (e.g., Twitter, Facebook). Its core capabilities include:

1. Social Tokenizer: A tokenizer that handles complex emoticons, emojis, dates, times, and other unstructured expressions common in social media.
2. Word Segmentation: Splits long strings into constituent words, which is particularly useful for segmenting hashtags.
3. Spell Correction: Replaces misspelled words with the most probable candidates based on word statistics.
4. Customization:
   • Word Statistics: Segmentation and Spell Correction rely on word statistics. While Wikipedia and Twitter corpora are provided, you can generate statistics from your own domain-specific corpus (e.g., biomedical text) to prevent domain-specific terms from being flagged as misspellings.
   • Entity Identification: You can identify new entities by adding regular expressions to ekphrasis/regexes/expressions.txt.
5. Pre-Processing Pipeline: A way to combine tokenization, normalization, word annotation (labeling), and other steps into a single workflow for preparing datasets for machine learning or analysis.

You can build a sentiment analysis pipeline by combining TextPreProcessor for text cleaning and tokenization with the polarity utility to calculate sentiment scores.

1. Initialize TextPreProcessor with desired cleaning options (e.g., fix_text=True, unpack_contractions=True) and a tokenizer function.
2. Use pre_process_docs() to transform a list of raw strings into tokenized lists.
3. Pass the resulting tokens to polarity(sent) to obtain a sentiment polarity score and a dictionary of detailed scores.

from ekphrasis.classes.preprocessor import TextPreProcessor
from ekphrasis.classes.tokenizer import SocialTokenizer
from ekphrasis.utils.nlp import polarity

sentences = [
    "So there is no way for me to plug it in here in the US unless I go by a converter.",
    "Good case, Excellent value.",
    "Works great!",
    'The design is very odd, as the ear "clip" is not very comfortable at all.',
    "Needless to say, I wasted my money."
]

# define preprocessing pipeline
text_processor = TextPreProcessor(
    fix_text=True,
    unpack_contractions=True,
    tokenizer=SocialTokenizer(lowercase=True).tokenize,
)

# pass each sentence through the pipeline
tokenized_sentences = list(text_processor.pre_process_docs(sentences))
for sent in tokenized_sentences:
    _polarity, _scores = polarity(sent)
    print("{:.4f}\t".format(_polarity) + " ".join(sent))

Use the TextPreProcessor class to create a comprehensive pipeline for cleaning and normalizing social media text. The processor can handle normalization of specific terms (like URLs or emails), annotation of features (like hashtags or all-caps words), HTML fixing, word segmentation for hashtags, contraction unpacking, and spell correction.

Key configuration options:

• normalize: A list of term types to normalize (e.g., ['url', 'email', 'percent', 'money', 'phone', 'user', 'time', 'date', 'number']).
• annotate: A set of feature types to annotate (e.g., {'hashtag', 'allcaps', 'elongated', 'repeated', 'emphasis', 'censored'}).
• fix_html: Boolean to fix HTML tokens.
• segmenter: The corpus to use for word segmentation (e.g., 'twitter').
• corrector: The corpus to use for spell correction (e.g., 'twitter').
• unpack_hashtags: Boolean to perform word segmentation on hashtags.
• unpack_contractions: Boolean to expand contractions (e.g., can't -> can not).
• spell_correct_elong: Boolean to enable spell correction for elongated words.
• tokenizer: A callable that takes a string and returns a list of tokens (e.g., SocialTokenizer(lowercase=True).tokenize).
• dicts: A list of dictionaries used for replacing specific tokens with other expressions (e.g., emoticons).

from ekphrasis.classes.preprocessor import TextPreProcessor
from ekphrasis.classes.tokenizer import SocialTokenizer
from ekphrasis.dicts.emoticons import emoticons

text_processor = TextPreProcessor(
    normalize=['url', 'email', 'percent', 'money', 'phone', 'user', 'time', 'url', 'date', 'number'],
    annotate={'hashtag', 'allcaps', 'elongated', 'repeated', 'emphasis', 'censored'},
    fix_html=True,
    segmenter="twitter",
    corrector="twitter",
    unpack_hashtags=True,
    unpack_contractions=True,
    spell_correct_elong=False,
    tokenizer=SocialTokenizer(lowercase=True).tokenize,
    dicts=[emoticons]
)

sentences = ["CANT WAIT for the new season of #TwinPeaks ＼(^o^)／!!!"]
for s in sentences:
    print(" ".join(text_processor.pre_process_doc(s)))