DOCSEARCHbyAlgolia
Submit Docs

clulab/processors

repository·master·Indexed 19 days ago

https://github.com/clulab/processors

A collection of text processing and language understanding tools developed by the Computational Language Understanding (CLU) Lab at the University of Arizona. The library includes an embedding_generator script for analyzing text with Hugging Face models, a webapp for NER and extractor engine configuration, and a debugger for Odin rules. It supports CoNLL-U formatted output with additional columns for offsets, entities, and chunks.

Tokens
18.9K
Snippets
50
Records
84
Agent score
65%

What's inside clulab-processors

  1. Overview of the Odin Event Extraction framework

    master

    Odin (Open Domain INformer) is an event extraction (EE) framework designed to provide a simple, powerful, and robust rule-based language for information extraction. Unlike pure machine learning methods, Odin's rule-based approach offers interpretability and allows for rapid domain transfer.

    Key features include:

    • Simple Syntax: Uses a declarative syntax for $n$-ary events based on syntactic dependency representations. It captures event predicates (triggers) with lexical/morphological constraints and arguments via syntactic patterns and semantic constraints.
    • Complex Construct Support: Supports recursive events (events as arguments), extended regular expressions for directed graph traversal (using < and > operators), optional arguments, and multiple arguments with identical names.
    • Robustness: Allows mixing syntactic patterns with surface, token-based patterns (e.g., part of speech, lemma, named entity labels, or graph edges) to recover from syntactic errors.
    • High Performance: Uses trigger patterns as fast starting points; complex argument matching is only attempted after a trigger is detected. It can process over 100 sentences/second on standard laptop hardware.

  2. Overview of clulab/processors

    master
    The clulab/processors repository contains the main public code for the Computational Language Understanding (CLU) Lab at the University of Arizona. It is a collection of tools designed for text processing and language understanding tasks. For simple text parsing tasks, users are directed to the Basic Usage documentation.

  3. Understand Debugger Context

    master

    As the debugger runs, it maintains a context of the current state of the Odin algorithm. The context tracks instances of Sentence, Extractor, or Pattern.

    Currently, the context is organized into separate lists or stacks differentiated by the instance type. This allows the debugger to track what the system is working on at any given moment during the execution process.

  4. Understand the Sentiment Opinion Lexicons

    master

    The clulab/processors library includes sentiment lexicons for identifying Negative and Positive opinion words. These lexicons are based on research by Minqing Hu and Bing Liu.

    Usage Notes

    • Context Matters: The presence of an opinion word in a sentence does not guarantee that the sentence expresses a positive or negative opinion. Sentiment analysis requires considering context.
    • Misspellings: The lexicons contain many misspelled words. These are intentional and are included to match frequent patterns found in social media content.

    Citation

    If you use these lists in your work, please cite:

    Minqing Hu and Bing Liu. "Mining and Summarizing Customer Reviews." Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2004), Aug 22-25, 2004, Seattle, Washington, USA.

  6. Understand the embedding_generator output

    master

    When the script runs, it provides the following outputs:

    • Console Output:
      • Total number of tokens processed.
      • A 'Finished!' confirmation message.
      • Count of tokens appearing only once.
      • Performance tracking counts for every 1000 tokens from the vocabulary.
      • Count of unseen tokens (via two metrics) and a list of those tokens.
    • File Output:
      • A TSV file named <output>.tsv containing the results.

  7. Understand the difference between Static and Dynamic debugging views

    master

    The debugger collects two distinct types of information to help you troubleshoot Odin rules:

    1. Static Views: These deal with rules before they are applied to any text. They allow you to inspect the data structures created by rule parsers and compilers. Use these to verify that your YAML syntax, indentation, and identifiers are correctly understood by Odin.
    2. Dynamic Views: These deal with rules during execution against specific sentences. They show how extractors match against tokens and how they produce Mentions. Use these to understand why a rule is (or is not) matching specific text or why an action (like a filter or transformation) is behaving unexpectedly.

  8. Getting started with the Odin Rule Language and API

    master

    To learn how to use the Odin rule language and its API, refer to the following resources:

    • Comprehensive Documentation: Read the Odin Manual for a thorough description of the language and API.
    • Quick Introduction: Consult the LREC paper which summarizes the rule language.
    • Hands-on Learning: Use the interactive Odin tutorial to practice.
    • Example Domains:

  9. Use interactive mode and standard output

    master

    You can modify the command line behavior by omitting certain parameters:

    • Interactive Shell: If you omit the -input parameter (running just java -jar <JAR FILE NAME>), the software starts an interactive shell. You can type text directly, and the output will be displayed when you press Enter.
    • Standard Output: If you omit the -output parameter, the CoNLL-U formatted results will be printed directly to the standard output (stdout) instead of being saved to a file.

  10. Install and run processors via fat JAR

    master

    To use processors without installing from source, download the latest "fat" jar from here. You can then run the parser on a single text file using the java -jar command with -input and -output flags.

    java -jar <JAR FILE NAME> -input <INPUT TEXT FILE> -output <OUTPUT FILE>