DOCSEARCHbyAlgolia
Submit Docs

SnowNLP Documentation

repository·master·Indexed 27 days ago

https://github.com/isnowfy/snownlp

A Python library for simplified Chinese text processing. It provides features for word segmentation, part-of-speech tagging, sentiment analysis, Pinyin conversion, traditional to simplified Chinese conversion, keyword extraction, text summarization, sentence tokenization, TF/IDF calculation, text similarity, and text classification. The library allows for training custom models for segmentation, tagging, and sentiment analysis.

Tokens
475
Snippets
2
Records
3
Agent score
42%

What's inside SnowNLP

  1. Use SnowNLP for Chinese text processing

    master

    SnowNLP is a Python library for simplified Chinese text processing. It handles Unicode encoding, so ensure your input strings are decoded to Unicode.

    Key features include:

    • Word segmentation (.words)
    • Part-of-speech tagging (.tags)
    • Sentiment analysis (.sentiments)
    • Pinyin conversion (.pinyin)
    • Traditional to Simplified Chinese conversion (.han)
    • Keyword extraction (.keywords(n))
    • Text summarization (.summary(n))
    • Sentence tokenization (.sentences)
    • TF/IDF calculation
    • Text similarity (.sim(list))
    • Text classification

  2. Train SnowNLP models

    master

    SnowNLP allows you to train its own models for segmentation, part-of-speech tagging, and sentiment analysis.

    To train the segmentation model:

    1. Use seg.train('data.txt') with your training data.
    2. Save the model using seg.save('seg.marshal').
    3. To use the custom model, update the data_path in snownlp/seg/__init__.py to point to your saved .marshal file.

    Other available training modules:

    • Part-of-speech tagging: Use tag.train('file.txt') and tag.save('tag.marshal').
    • Sentiment analysis: Use sentiment.train('neg.txt', 'pos.txt') and sentiment.save('sentiment.marshal').
    from snownlp import seg
seg.train('data.txt')
seg.save('seg.marshal')

# For tagging:
# from snownlp import tag
# tag.train('199801.txt')
# tag.save('tag.marshal')

# For sentiment:
# from snownlp import sentiment
# sentiment.train('neg.txt', 'pos.txt')
# sentiment.save('sentiment.marshal')