Hugging Face Computer Vision Course

Multimodal models integrate information from multiple sources (e.g., text and images) to perform various tasks. Key tasks include:

• Visual Question Answering (VQA) & Visual Reasoning: Answering questions about an image or inferring relationships/context within a scene.
• Document Visual Question Answering (DocVQA): Understanding both the text and layout of documents (like maps or contracts) to answer questions directly from the image.
• Image Captioning: Generating natural language descriptions of an image.
• Image-Text Retrieval: Matching images to their descriptions or vice versa (e.g., image search).
• Visual Grounding: Pinpointing specific objects or regions in an image based on natural language descriptions.
• Text-to-Image Generation: Creating unique images from written descriptions.

Computer vision (CV) is used across various industries to automate tasks that are often difficult or time-consuming for humans. Key application domains include:

• Autonomous Vehicles: Using cameras and sensors to identify objects (pedestrians, traffic signs, lane markings) for real-time decision making (steering, braking).
• Retail and E-commerce: Object recognition for product recommendations and computer vision-powered inventory tracking/customer behavior analysis in physical stores.
• Quality Control in Manufacturing: Real-time defect detection (scratches, dents, incorrect assembly) and automated inspection on assembly lines to ensure consistency.
• Medical Image Analysis: Interpreting X-rays, CT scans, MRIs, and ultrasounds for diagnostic assistance, organ segmentation, treatment planning, and drug research.

The NeRF field is evolving rapidly. Key advancements include:

• Faster Training/Inference:
  • Instant-ngp: Uses trainable hash-tables for encoding, allowing for much smaller MLPs and faster training.
  • Zip-NeRF: Combines hash-table encoding with scene contraction to handle real-world scenes in under an hour.
• Unbounded Scenes:
  • Mipnerf-360: Introduces scene contraction to represent unbounded environments.
• Other Research Areas:
  • Generative NeRFs
  • Pose Estimation
  • Deformable NeRFs
  • Compositionality
  • Real-time streaming (e.g., VR-NeRF, SMERF)

CapFilt is a mechanism introduced by BLIP to improve dataset quality. Because web-scraped alt-text is often noisy and inaccurate, CapFilt uses two models:

1. A filtering model that removes noisy image-text pairs.
2. A captioning model that generates new, accurate captions for images.

Both models are fine-tuned on human-annotated datasets. Using CapFilt to clean datasets results in superior performance compared to using raw web datasets.

Zero-shot learning (ZSL) is a machine learning setup where a model is presented with images belonging to classes it was not exposed to during training. In a classic ZSL setup, the training and testing sets are disjoint (the test set contains only unseen classes).

There is a more pragmatic variation called Generalized Zero-shot Learning (GZSL), which allows the test set to include both seen and unseen classes, making it more applicable to real-world scenarios.

BlenderProc is a modular pipeline built on top of Blender designed for generating realistic synthetic data for training convolutional neural networks (CNNs).

It is particularly useful for generating datasets that include:

• Segmentation maps
• Depth maps
• Surface normals
• Pose estimation

Key features include:

• Procedural Generation: Automated creation of complex 3D scenes with variations.
• Simulation: Integration of physics simulations to enhance realism.
• Large-Scale Generation: Designed for efficiency in high-volume scene generation.
• Automation: Support for Python scripting and parallel processing for scalability.

ViR is a general vision backbone that redesigns the retention mechanism to scale favorably to larger image resolutions in terms of throughput and memory consumption.

Key Architectural Differences from ViT:

1. Multi-Head Retention (MHR): Replaces Multi-Head Attention (MHA). The MHR mechanism is free of gating functions and can be switched between three modes: parallel, recurrent, or chunkwise (a hybrid mode).
2. Positional Embedding: In ViR, positional embeddings are added to the patch embedding before the [class] token is appended.

Neural Radiance Fields (NeRFs) are an implicit representation of a 3D scene. Instead of using explicit structures like voxel grids (which scale cubically in memory), NeRFs represent the scene using the weights of a Multi-Layer Perceptron (MLP).

This approach maps a 3D position $\mathbf{x}$ and a viewing direction $\boldsymbol{\theta}$ to a color $\mathbf{c}$ and volume density $\sigma$. NeRFs are primarily used for novel view synthesis, which is the task of generating images from camera perspectives that were not present in the original training set.