Hugging Face Course

The Hugging Face Hub is a central platform for discovering, using, and contributing to state-of-the-art models and datasets.

Key features include:

• Model Hosting: Supports various frameworks including 🤗 Transformers, Flair, AllenNLP, Asteroid, pyannote, and timm.
• Git-based Versioning: Each model is hosted as a Git repository, ensuring versioning and reproducibility.
• Inference API: Sharing a model on the Hub automatically deploys a hosted Inference API, allowing users to test models directly on their model pages using custom inputs and widgets.
• Accessibility: Public models are free to use and share. Private model sharing is available via paid plans.

Translation is a sequence-to-sequence (seq2seq) task where the goal is to transform one sequence of tokens into another. This approach can be adapted to other tasks such as:

• Style transfer: Translating text from one style to another (e.g., formal to casual).
• Generative question answering: Generating answers to questions based on a provided context.

For translation, you can either train a model from scratch using a large multilingual corpus or fine-tune an existing pretrained model. Common pretrained models for fine-tuning include multilingual models like mT5 or mBART, or specialized translation models like Marian.

Gradio provides two main APIs for building machine learning demos:

• Interface: A high-level API for quickly creating demos by providing a list of inputs and outputs.
• Blocks: A low-level API that offers full control over data flows, layout, and complex multi-step application logic.

Use Blocks when you need to group demos into tabs, customize layouts, create multi-step interfaces, or dynamically change component properties based on user input.

The 🤗 Transformers library provides a unified API for loading, training, and saving Transformer models. It is designed for ease of use, flexibility, and simplicity. Key characteristics include:

• Ease of use: Inference can be performed with minimal code (typically two lines).
• Flexibility: Models are implemented as standard PyTorch nn.Module classes, allowing them to be used within existing ML workflows.
• Simplicity: The library follows an "All in one file" concept where a model's forward pass is defined in a single file, making the code easy to understand and modify without relying on complex shared abstractions.

Extractive question answering involves posing questions about a document and identifying the answers as specific spans of text within that document.

Key Concepts:

• Extractive QA: Identifying text spans as answers. Typically performed using encoder-only models like BERT.
• Generative QA: Synthesizing answers to open-ended questions (e.g., "Why is the sky blue?"). This is typically performed using encoder-decoder models like T5 or BART.
• SQuAD Dataset: A common dataset used for training extractive QA models, consisting of questions posed on Wikipedia articles.

This course teaches how to use the Hugging Face ecosystem to work with Large Language Models (LLMs) and Natural Language Processing (NLP). You will learn to use core libraries including:

• 🤗 Transformers: For working with transformer-based models.
• 🤗 Datasets: For accessing and managing datasets.
• 🤗 Tokenizers: For text preprocessing.
• 🤗 Accelerate: For efficient model training and inference.
• Hugging Face Hub: For discovering and sharing models and datasets.

The course also covers external AI community tools and libraries.

Open R1 is a community project aimed at enabling Large Language Models (LLMs) to reason through complex problems using Reinforcement Learning (RL). The project focuses on training models to generate structured 'thoughts' alongside their final answers. This allows users to separate the reasoning process from the final output.

Models trained with this approach typically use a specific format to encapsulate reasoning:

<think>Reasoning process goes here</think>
Final answer goes here

Transformer models solve various tasks using three primary architectural variants: encoders, decoders, or encoder-decoder structures. The specific task is determined by how input data is prepared, which architecture variant is used, and how the output is processed.

Common task-to-model mappings include:

• Audio Classification & Automatic Speech Recognition (ASR): Wav2Vec2
• Image Classification: Vision Transformer (ViT) and ConvNeXT
• Object Detection: DETR
• Image Segmentation: Mask2Former
• Depth Estimation: GLPN
• NLP (Encoder-based): BERT for tasks like text classification, token classification, and question answering.
• NLP (Decoder-based): GPT2 for text generation.
• NLP (Encoder-Decoder): BART for summarization and translation.

Chapter 3 focuses on fine-tuning pretrained models for specific tasks using PyTorch. It covers the following core competencies:

• Dataset Preparation: Using the 🤗 Datasets library to prepare large datasets from the Hub.
• High-level Training: Using the Trainer API for fine-tuning with modern best practices.
• Custom Training: Implementing custom training loops with optimization techniques.
• Distributed Training: Leveraging the 🤗 Accelerate library to run distributed training on various hardware setups.
• Performance Optimization: Applying current fine-tuning best practices.

By the end of the chapter, you will have fine-tuned a BERT model for text classification.

Natural Language Processing (NLP) involves understanding human language by analyzing both individual words and their context. Common NLP tasks include:

• Classifying whole sentences: Sentiment analysis, spam detection, grammar checking, or logical relationship detection.
• Classifying each word in a sentence: Part-of-speech tagging (noun, verb, etc.) or Named Entity Recognition (NER) (identifying persons, locations, organizations).
• Generating text content: Prompt completion or masked word filling.
• Extracting an answer from a text: Question answering based on a provided context.
• Generating a new sentence from an input text: Machine translation or text summarization.
• Multimodal NLP: Speech recognition (audio transcripts) and image captioning (computer vision).