HarvestText Documentation

Quickly search for documents containing specific entities (or their aliases) using an inverted index.

1. Build Index: Use ht.build_index(docs) to create the index.
2. Count Entities: Use ht.get_entity_counts(docs, inv_index) to get the frequency of all entities in the documents.
3. Search:
   • Single entity: ht.search_entity("EntityName", docs, inv_index)
   • Multiple entities (co-occurrence): ht.search_entity("EntityA EntityB", docs, inv_index)
   • Mixed entity and type: ht.search_entity("#Type# EntityName", docs, inv_index)
4. Filter by Type: Use ht.get_entity_counts(docs, inv_index, used_type=["Type"]) to restrict counts to specific entity types.

To use the library in your Python code, import the HarvestText class and instantiate it.

from harvesttext import HarvestText
ht = HarvestText()

You can install HarvestText using pip or by running the setup.py file from the repository directory.

Note that some features require additional libraries. If installation fails or features are missing, you may need to install them manually:

• For English language features: pip install pattern
• For Named Entity Recognition (NER) and Dependency Parsing: pip install pyhanlp (requires Python <= 3.8).

pip install --upgrade harvesttext

While primarily designed for Chinese data mining, HarvestText supports limited English features like sentiment analysis, sentence segmentation, and word segmentation. To use these, you must instantiate HarvestText with language="en".

ht_eng = HarvestText(language="en")

To use the entity discovery features (v0.7+), ensure you have the latest version of HarvestText installed via pip.

pip install --upgrade harvesttext

You can prevent the discovery of known words by providing an existing dictionary (e.g., from Jieba) to the word_discover method using the excluding_words parameter. Use exclude_number=True to also exclude numeric values from discovery.

from harvesttext.resources import get_jieba_dict
jieba_dict = get_jieba_dict(min_freq=100)
text = "1979-1998-2020的喜宝们 我现在记忆不太好..."
new_words_info = ht.word_discover(text, 
                                    excluding_words=set(jieba_dict),       # Exclude words already in dictionary
                                    exclude_number=True)                   # Exclude numbers (default True)     
new_words = new_words_info.index.tolist()
print(new_words)                                                         # ['喜宝']

The NaiveKGQA class provides a mechanism for answering questions based on provided SVOs (Subject-Verb-Object triples) and an entity type dictionary. It can answer questions about historical figures, relationships, and events.

QA = NaiveKGQA(SVOs, entity_type_dict=entity_type_dict)
questions = ["你好","孙中山干了什么事？","谁发动了什么？","清政府签订了哪些条约？",
		 "英国与鸦片战争的关系是什么？","谁复辟了帝制？"]
for question0 in questions:
	print("问："+question0)
	print("答："+QA.answer(question0))

Identify new words from large amounts of text using statistical indicators. You can provide seed words to guide the quality of discovery.

Use ht.word_discover(text, threshold_seeds=["SeedWord"]). The function returns a pandas.DataFrame containing quality information about the discovered words. You can extract the list of words using .index.tolist().

para = "上港的武磊和恒大的郜林..."
new_words_info = ht.word_discover(para, threshold_seeds=["武磊"])
new_words = new_words_info.index.tolist()

Create a graph structure (returning a networkx.Graph object) based on word co-occurrence to model relationships between entities, such as social networks.

• Add Entities: Use ht.add_new_entity(name, mention, type) to expand the entity library.
• Build Graph: Use ht.build_entity_graph(docs, used_types=["Type"]) to generate the graph. You can optionally restrict the graph to specific entity types.

ht.add_new_entity("颜骏凌", "颜骏凌", "球员")
docs = ["武磊和颜骏凌是队友", "武磊和郜林都是国内顶尖前锋"]
G = ht.build_entity_graph(docs, used_types=["球员"])

You can persist a HarvestText object to disk using saveHT and reload it later using loadHT. This allows you to preserve trained states or specific configurations.

from harvesttext import loadHT,saveHT
para = "上港的武磊和恒大的郜林，谁是中国最好的前锋？那当然是武磊武球王了，他是射手榜第一，原来是弱点的单刀也有了进步"
saveHT(ht,"ht_model1")
ht2 = loadHT("ht_model1")

# Clear records in the loaded model
ht2.clear()
print("cut with cleared model")
print(ht2.seg(para))

To ensure that discovered keywords are prioritized during subsequent segmentation, you can register them as new words or specific entities.

• Use add_new_words(list_of_words) to register words as general 'new words'.
• Use add_new_entity(word, mention0=..., type0=...) to register a word as a specific type of entity (e.g., a technical term).

def new_word_register():
    new_words = ["落叶球","666"]
    ht.add_new_words(new_words)   # Register as general "new words"
    ht.add_new_entity("落叶球", mention0="落叶球", type0="术语")  # Register as a specific type
    print(ht.seg("这个落叶球踢得真是666", return_sent=True))
    for word, flag in ht.posseg("这个落叶球踢得真是666"):
        print("%s:%s" % (word, flag), end=" ")