DOCSEARCHbyAlgolia
Submit Docs

PORORO

repository·master·Indexed 23 days ago

https://github.com/kakaobrain/pororo

A comprehensive platform for Natural Language Processing (NLP) and Speech-related tasks. PORORO provides a unified interface via the `Pororo` class to solve various subtasks including translation, NER, summarization, speech recognition, sentiment analysis, and lemmatization by specifying a task name and language.

Tokens
9K
Snippets
22
Records
83
Agent score
77%

What's inside pororo

  1. Install dependencies for Optical Character Recognition (OCR)

    master

    To use the OCR module, you must install the system-level Mesa GL library and the following Python packages:

    1. Install libgl1-mesa-glx via apt-get.
    2. Install opencv-python and scikit-image via pip.
    apt-get install -y libgl1-mesa-glx
pip install opencv-python scikit-image

  2. Install Pororo common dependencies

    master

    When installing Pororo via pip, the following core libraries are installed automatically. You do not need to install them separately:

    • torch==1.6.0
    • torchvision==0.7.0
    • pillow>=4.1.1
    • fairseq>=0.10.2
    • transformers>=4.0.0
    • sentence_transformers>=0.4.1.2
    • nltk>=3.5
    • word2word
    • wget
    • joblib
    • lxml
    • g2p_en
    • whoosh
    • marisa-trie
    • kss
    • dataclasses (for Python < 3.7)

  3. Set up Automatic Speech Recognition (ASR)

    master

    To use the Automatic Speech Recognition module, you must install wav2letter.

    Requirements & Steps:

    1. CUDA: Must be pre-installed on your system.
    2. Installation: Run the provided installation script.

    Note: This task is supported on Linux.

    bash asr-install.sh

  4. Use the Pororo class for NLP and Speech tasks

    master

    The primary interface for PORORO is the Pororo class. You initialize it by specifying a task, an optional lang (language), and an optional model. Once initialized, the object can be called directly with an input string to perform the task.

    Workflow:

    1. Import Pororo from pororo.
    2. Initialize the object with task, lang, and/or model arguments.
    3. Call the object with your text input.

  5. Setup Automatic Speech Recognition (ASR)

    master

    To use the ASR module, you must install wav2letter.

    Prerequisites:

    • CUDA must be installed on your system before proceeding.

    Installation: Run the provided installation script in the repository root.

    Note: ASR, Speech Translation, OCR, and Image Captioning are supported on Linux.

    bash asr-install.sh

  7. Install additional dependencies for Korean NLP tasks

    master

    To perform specific Korean language tasks, you must install the following additional libraries:

    • Tokenization, PoS Tagging, and Dependency Parsing: Install python-mecab-ko==1.0.9.
    • Collocation: Install kollocate.
    • Morphological Inflection: Install koparadigm.
    • Grapheme-to-Phoneme: Install g2pk.

  8. Install additional libraries for Japanese NLP tasks

    master

    To perform Japanese NLP tasks, install the following:

    • Japanese RoBERTa and Japanese PoS Tagging: Install fugashi and ipadic.
    • Japanese Grapheme-to-Phoneme: Install romkan.
    pip install fugashi ipadic
pip install romkan

  9. Use Automated Essay Scoring (AES) in Pororo

    master

    The pororo.tasks.automated_essay_scoring module provides capabilities for Automated Essay Scoring (AES). This task is used to evaluate the quality of written essays automatically.

    Note: As this is a documentation stub generated via automodule, specific method signatures and class names should be discovered by inspecting the pororo.tasks.automated_essay_scoring module in the source code.

  11. Install additional libraries for Korean NLP tasks

    master

    Depending on the specific Korean NLP task you want to perform, you may need to install these additional libraries:

    • Korean Tokenization, PoS Tagging, Dependency Parsing, etc.: Install python-mecab-ko==1.0.9.
    • Korean Collocation: Install kollocate.
    • Korean Morphological Inflection: Install koparadigm.
    • Korean Grapheme-to-Phoneme: Install g2pk.
    pip install python-mecab-ko==1.0.9
pip install kollocate
pip install koparadigm
pip install g2pk

  12. Load a Transformer model and tokenizer

    master

    To use a Transformer model, follow these three steps:

    1. Download/Load the dictionary metadata: Use download_or_load with the model directory and language to retrieve the necessary paths.
    2. Initialize the model: Use TransformerModel.from_pretrained passing the paths retrieved from the metadata.
    3. Initialize the tokenizer: If the model requires a tokenizer, use CustomTokenizer.from_file pointing to the vocab.json and merges.txt files within the tokenizer directory.
    # 1. Pass the model name to download, and then get the path
load_dict = download_or_load(f"transformer/{self._n_model}", self._lang)

# 2. Use the path information to load the model
model = TransformerModel.from_pretrained(
  model_name_or_path=load_dict.path,
  checkpoint_file=f"{self._n_model}.pt",
  data_name_or_path=load_dict.dict_path,
  source_lang = load_dict.src_dict,
  target_lang = load_dict.tgt_dict,
)

# 3. Load the tokenizer, if necessary
tokenizer = CustomTokenizer.from_file(
    vocab_filename=f"{load_dict.src_tok}/vocab.json",
    merges_filename=f"{load_dict.src_tok}/merges.txt",
)