DOCSEARCHbyAlgolia
Submit Docs

HanLP Documentation

website·Indexed 33 days ago

https://hanlp.hankcs.com/

A natural language processing library featuring components for multi-task learning (MTL), including tasks for NER, POS tagging, constituency and dependency parsing, and SRL. The library provides BPE-based transformer tokenization, various dataset loaders for CoNLL and TSV formats, and utility modules for file handling and trie-based dictionaries.

Tokens
142.5K
Snippets
755
Records
997
Agent score
63%

What's inside HanLP

  1. Overview of HanLP NLP Datasets

    HanLP provides NLP datasets grouped by specific tasks. For each task, the library offers at least one torch.utils.data.Dataset compatible class and several open-source resources. Datasets are typically split into TRAIN, DEV, and TEST portions, which are stored as Python constants. These resources can be retrieved using the get_resource() utility function from hanlp.utils.io_util.

  2. Overview of Semantic Dependency Parsing (SDP) in HanLP

    Semantic Dependency Parsing (SDP) analyzes the semantic relationships between words in a sentence and represents them as a graph structure. Unlike standard dependency parsing, each node in an SDP graph can have any number of target nodes. HanLP supports several semantic frameworks: CSDP for Chinese, and DM, PAS, and PSD for English.

  3. Overview of Abstract Meaning Representation (AMR) in HanLP

    Abstract Meaning Representation (AMR) is a linguistic framework that represents the meaning of a sentence (who did what to whom, when, and where) as a single-source directed acyclic graph (DAG) where nodes are concepts. In HanLP, AMR can be used for downstream tasks such as machine translation, Question Answering (QA), and relation extraction. The resulting Meaning Representation (MR) format also includes coreference resolution results.

  4. Overview of the MultiTaskLearning framework

    The hanlp.components.mtl.multi_task_learning.MultiTaskLearning class provides a multi-task learning (MTL) framework that shares a single encoder across multiple decoders. It supports dependencies between decoders during the decoding process. To integrate a custom component into this framework, the component must implement the Task interface. The architecture is based on Clark et al. (2019) and He & Choi (2021), incorporating scalar mix tricks (Kondratyuk & Straka 2019) to allow tasks to attend to any subset of layers.

  5. Overview of HanLP Multilingual NLP Library

    HanLP is a multilingual Natural Language Processing (NLP) library designed for researchers and companies. It is built on PyTorch and TensorFlow 2.x and provides pretrained models for multiple languages, including English, Chinese, and Japanese. The library is designed to be efficient, user-friendly, and extendable for both academic and industrial deep learning applications.

  6. Overview of HanLP Multilingual NLP Library

    HanLP is a multilingual Natural Language Processing (NLP) library designed for researchers and companies. It is built on PyTorch and TensorFlow 2.x and provides pretrained models for various languages, including English, Chinese, and Japanese. The library is designed to be efficient, user-friendly, and extendable for both academic and industrial deep learning applications.

  7. Configure TransformerNamedEntityRecognizer training parameters

    When calling the fit method for TransformerNamedEntityRecognizer, the following parameters can be used to tune the model:

    • Model Architecture: crf (bool) enables the CRF layer; secondary_encoder provides additional representations.
    • Optimization: lr (decoder learning rate), transformer_lr (encoder learning rate), weight_decay, warmup_steps, and grad_norm (gradient clipping).
    • Regularization: word_dropout (randomly replaces subwords with MASK), hidden_dropout, and layer_dropout (zeros out transformer layer hidden states).
    • Sequence Handling: max_seq_len defines the limit before sliding window is used; sent_delimiter identifies where long sentences can be split; hard_constraint enforces length limits.
    • Data Processing: delimiter_in_entity joins tokens into entities during decoding; merge_types specifies consecutive entity types to merge; average_subwords averages subword representations.

  9. Use BiaffineDependencyParser for dependency parsing

    The BiaffineDependencyParser is a Torch-based component used for biaffine dependency parsing. It can predict dependency arcs and relations for given text data. It supports both flat input (single sentence) and batched input, and can optionally incorporate Part-of-Speech (POS) tags to improve parsing accuracy.
    from hanlp.components.parsers.biaffine.biaffine_dep import BiaffineDependencyParser

parser = BiaffineDependencyParser()
# Assuming the parser is loaded with a trained model
predictions = parser.predict(["This is a test sentence."])

  10. Understand the TransformerClassificationModel architecture

    The TransformerClassificationModel is the underlying PyTorch module for transformer-based classification. It consists of a transformer encoder, a dropout layer, and a linear classifier. If the input sequence length exceeds max_seq_length, it employs a sliding window approach via transformer_sliding_window to process the text.
    class TransformerClassificationModel(nn.Module):
    def __init__(self, transformer: PreTrainedModel, num_labels: int, max_seq_length=512) -> None:
        super().__init__()
        self.max_seq_length = max_seq_length
        self.transformer = transformer
        self.dropout = nn.Dropout(transformer.config.hidden_dropout_prob)
        self.classifier = nn.Linear(transformer.config.hidden_size, num_labels)

    def forward(self, input_ids, attention_mask, token_type_ids):
        # Handles long sequences via sliding window or standard forward pass
        # Returns logits for classification

  11. Use BiaffineNamedEntityRecognizer for Flat and Nested NER

    The BiaffineNamedEntityRecognizer is a PyTorch-based component that treats every possible span in a sentence as a candidate entity and predicts its label using a biaffine layer. Because it makes no assumptions about span boundaries, it naturally supports both flat NER and nested NER (where entities can be contained within other entities).
    from hanlp.components.ner.biaffine_ner.biaffine_ner import BiaffineNamedEntityRecognizer

# Initialize with a predefined config
recognizer = BiaffineNamedEntityRecognizer(**config)

  12. Initialize the Transformer model architecture

    The build_model method initializes the underlying PyTorch model. It uses AutoConfig and AutoModelForSequenceClassification from the transformers library, setting num_labels=1 for regression-based similarity scoring. If training is set to False, it initializes the model from a configuration rather than pre-trained weights.
    # Internal logic used by the component
model = self.build_model(transformer="bert-base-uncased", training=True)