DOCSEARCHbyAlgolia
Submit Docs

Alan Nikos Blog

website·Indexed 30 days ago

https://alannikos.github.io

Personal blog focusing on Natural Language Processing (NLP), Large Language Models (LLMs), and high-performance computing, featuring technical notes on text generation tasks and performance evaluation.

Tokens
2.2K
Snippets
8
Records
17
Agent score
25%

What's inside Alan Nikos Blog

  1. Overview of Natural Language Generation (NLG)

    Natural Language Generation (NLG) is a subfield of artificial intelligence and computational linguistics focused on creating software systems that produce human-readable text. These systems accept various forms of input—including natural language text, images, or non-linguistic data (such as reports or visual information)—and automatically generate high-quality natural language outputs like summaries, stories, poems, or explanations.

  2. Understand Perplexity (PPL) in language models

    Perplexity is a common metric for evaluating language models and text generation. It measures how well a probability distribution or probability model predicts a sample. In simple terms, for a specific sequence, the higher the probability the model assigns to that sequence, the lower the perplexity. A lower perplexity indicates the model is less 'confused' by the data.

  3. Implement Open-Ended Text Generation (OETG)

    Open-Ended Text Generation (OETG) involves generating new text based on a provided context or prompt to ensure a coherent and fluent continuation. Common applications include:

    • Story Generation: Creating narratives based on a prompt.
    • Dialogue Generation: Generating conversational responses.

    Caveat: Models performing OETG may occasionally produce text that is semantically or logically inconsistent (hallucinations or lack of coherence).

  4. Implement Text Summarization tasks

    Text summarization involves condensing input text into a shorter version while preserving core meaning. It is divided into three main types:

    • Single-Document Summarization: Generating a core summary of a single input document.
    • Multi-Document Summarization: Extracting and synthesizing primary information from multiple documents sharing the same theme (e.g., multiple news reports on one event).
    • Dialogue Summarization: Generating summaries for conversations, such as meetings, call logs, online chats, and customer service interactions.

  6. Calculate METEOR score for text generation

    METEOR (Metric for Evaluation of Translation with Explicit ORdering) improves upon BLEU by incorporating word form matching (stemming), synonym matching (via WordNet), and a penalty for word order differences. It calculates a harmonic mean of precision and recall, then applies a penalty factor based on the alignment of chunks.
    from nltk.translate.meteor_score import meteor_score

def cal_meteor(reference, candidate):
    """
    :param candidate: Machine translation (list of tokens)
    :param reference: Reference translation (list of tokens)
    :return: meteor score
    """
    score = meteor_score(reference, candidate)
    return score

if __name__ == "__main__":
    reference = ['It', 'is', 'sunny', 'today', 'so', 'many', 'people', 'have', 'a', 'picnic']
    candidate = ['It', 'is', 'sunny', 'today', 'so', 'many', 'people', 'have', 'a']

    score = cal_meteor(reference, candidate)
    print(score)

  7. Calculate Perplexity (PPL) using PyTorch

    Perplexity (PPL) is used to evaluate the quality of a language model. It is mathematically related to cross-entropy; specifically, the perplexity is the exponentiation of the cross-entropy loss. A lower perplexity indicates a better-performing model.
    import torch
from torch import Tensor
import torch.nn.functional as F

def cal_perplexity(outputs: Tensor, targets: Tensor, config=None):
    """
    :param outputs: Model output (logits)
    :param targets: Reference output (ground truth labels)
    :return: Perplexity value
    """
    return float(torch.exp(F.cross_entropy(outputs, targets)))

if __name__ == "__main__":
    num_classes = 5
    batch_size  = 1
    torch.manual_seed(42)
    output = torch.rand(batch_size, num_classes)  # Model output
    target = torch.randint(num_classes, (batch_size,))  # Reference output

    print("PPL value: ", cal_perplexity(output, target))

  8. Calculate ROUGE scores for text summarization

    ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is primarily used for evaluating automatic summaries. Unlike BLEU, which focuses on precision, ROUGE emphasizes recall (how much of the reference text is captured by the generated text).

    • ROUGE-N: Measures the overlap of n-grams between the generated and reference summaries.
    • ROUGE-L: Based on the Longest Common Subsequence (LCS), capturing the longest sequence of words that appear in both texts in the same relative order.
    from rouge import Rouge
 
# Example texts
hypothesis = "the cat was found under the bed"
reference = "the cat was under the bed"
 
# Create Rouge object
rouge = Rouge()
 
# Calculate scores
scores = rouge.get_scores(hypothesis, reference)
 
# Extract ROUGE-L results (precision, recall, and F1 score)
print("ROUGE-L precision:", scores[0]["rouge-l"]["p"])
print("ROUGE-L recall:", scores[0]["rouge-l"]["r"])
print("ROUGE-L F1 score:", scores[0]["rouge-l"]["f"])

  9. Calculate BLEU score for text generation

    BLEU (Bilingual Evaluation Understudy) evaluates the quality of machine-translated text by comparing a candidate translation against one or more reference translations. It focuses on precision, measuring how many n-grams in the generated text appear in the reference text. It includes a brevity penalty (BP) to prevent overly short translations from achieving high scores.
    from nltk.translate.bleu_score import sentence_bleu

def cal_bleu(reference, candidate, weights):
    """
    :param candidate: Machine translation (list of tokens)
    :param reference: Reference translation (list of lists of tokens)
    :param weights: n-gram weights (e.g., (0.25, 0.25, 0.25, 0.25) for 4-grams)
    :return: bleu score
    """
    score = sentence_bleu(reference, candidate, weights=weights)
    return score

if __name__ == "__main__":
    # Reference translation must be a list of lists
    reference = [['It', 'is', 'sunny', 'today', 'so', 'many', 'people', 'have', 'a', 'picnic']]
    # Candidate translation is a list of tokens
    candidate = ['It', 'is', 'sunny', 'today', 'so', 'many', 'people', 'have', 'a']
    # Weights for 1, 2, 3, and 4-grams
    weights = (0.25, 0.25, 0.25, 0.25)
    
    score = cal_bleu(reference, candidate, weights)
    print(score)

  10. Choose a Machine Translation (MT) approach

    Machine Translation automatically converts a source language text into a target language text. There are three primary technical approaches:

    • Rule-Based Machine Translation (RBMT): Uses manually written linguistic rules covering grammar and vocabulary. It relies heavily on expert knowledge.
    • Statistical Machine Translation (SMT): A data-driven approach that learns translation patterns and rules from large bilingual text corpora.
    • Neural Machine Translation (NMT): Uses deep learning (RNNs, LSTMs, or Transformers) in an end-to-end fashion. NMT typically employs an Encoder-Decoder architecture, where the encoder converts the source sentence into a fixed-length tensor and the decoder generates the target text from that tensor.

  11. Understand CIDEr for Image Captioning Evaluation

    CIDEr (Consensus-based Image Description Evaluation) is a metric specifically designed for image captioning. It evaluates the similarity between a candidate sentence and a set of human-annotated reference sentences based on 'consensus'.

    Key characteristics:

    • It treats each sentence as a document.
    • It uses TF-IDF vectors to represent n-grams.
    • Similarity is measured using the cosine distance between the TF-IDF vectors of the candidate sentence and the reference sentences.
    • The final CIDEr score is the average of the scores calculated against all reference sentences.

  12. Understand Question-Answering (QA) system types

    Question-Answering systems provide natural language answers to user queries. Modern QA systems are generally categorized into three types:

    • Specialized QA Systems: Focused on a specific domain.
    • General QA Systems: Capable of answering a wide range of topics.
    • Multimodal QA Systems: Utilizing multimodal knowledge graphs and pre-trained models to handle diverse input types.