DOCSEARCHbyAlgolia
Submit Docs

KerasCV

repository·master·Indexed 22 days ago

https://github.com/keras-team/keras-cv

A library of modular computer vision components, including models, layers, metrics, and callbacks, built on Keras 3. It supports multi-framework training and inference natively with TensorFlow, JAX, or PyTorch. KerasCV provides tools for image preprocessing, data augmentation, and pretrained backbones, as well as specific implementations like Stable Diffusion v1-4.

Tokens
16.1K
Snippets
50
Records
68
Agent score
74%

What's inside KerasCV

  1. Overview of KerasCV Preprocessing and Augmentation Layers

    master

    KerasCV provides a variety of preprocessing and data augmentation layers designed for computer vision tasks including classification, object detection, and segmentation.

    A key feature of these layers is that when applied to training data, they automatically synchronize augmentations across multiple data types. If you augment an image, the corresponding class labels, bounding boxes (BBoxes), and segmentation masks are updated automatically to maintain spatial and semantic consistency.

  2. Stable Diffusion v1-4 Model Overview

    master

    Stable Diffusion is a latent text-to-image diffusion model that generates photo-realistic images from text prompts. The Stable-Diffusion-v1-4 checkpoint is a fine-tuned version of the v1-2 checkpoint, trained on 225k steps at 512x512 resolution using the 'laion-aesthetics v2 5+' dataset. It utilizes a fixed, pretrained CLIP ViT-L/14 text encoder.

    Key Details:

    • Model Type: Latent Diffusion Model (LDM).
    • Language: Primarily English.
    • License: CreativeML Open RAIL-M license.
    • Primary Use Case: Research, artistic generation, and creative tools.

  3. Limitations and Biases of Stable Diffusion v1-4

    master

    Users should be aware of the following technical and social limitations when using the Stable Diffusion v1-4 model:

    Technical Limitations

    • Photorealism: Does not achieve perfect photorealism.
    • Text Rendering: Cannot render legible text.
    • Compositionality: Struggles with complex spatial relationships (e.g., "A red cube on top of a blue sphere").
    • Human Generation: Faces and people may not be generated accurately.
    • Language: Performance is significantly lower for non-English prompts.
    • Lossy Encoding: The auto-encoding component is lossy.
    • Memorization: Due to lack of dataset deduplication, the model may exhibit memorization of images present in the training data.

    Bias and Safety

    • Cultural Bias: Trained on subsets of LAION-2B(en), which may result in Western/white cultures being treated as the default.
    • Dataset Content: The training dataset (LAION-5B) contains adult material and is not suitable for product use without additional safety mechanisms.
    • Social Bias: The model may reinforce or exacerbate social biases present in the training data.

  5. Understand KerasCV image rescaling and normalization

    master

    KerasCV is standardized to operate on images that have been rescaled using a simple 1/255 rescaling layer.

    While many historical models used manual normalization (e.g., subtracting ImageNet mean and dividing by standard deviation), KerasCV pipelines and code examples use the 1/255 rescaling scheme to ensure compatibility with modern deep learning architectures.

  6. Implement color-based preprocessing layers

    master

    Layers that perform color-based transformations (such as RandomBrightness, Equalize, or Solarization) must adhere to specific interface requirements to ensure consistency:

    1. value_range parameter: Must accept a value_range argument, which is a tuple of numbers.
    2. Default range: The value_range must default to (0, 255).
    3. Dtype flexibility: Layers must be able to accept input images of any dtype. Do not strictly assert a specific type (like int), as Keras layers may cast inputs automatically.

  7. Understand strength parameters and the factor convention

    master

    Many augmentation layers in KerasCV use a factor parameter to control the intensity of the transformation. To maintain consistency, follow these rules:

    • Range: The factor should ideally be in the range [0, 1], where 1 is the strongest transformation and 0 is a no-op.
    • Scaling: The strength of the augmentation should scale linearly with this factor.
    • Input Types: factor should support both float and tuple inputs.
    • Float Behavior: If a single float is passed (e.g., factor=0.5), the layer should default to applying transformations within the range [0, factor].

  8. Subclass BaseImageAugmentationLayer to create custom augmentations

    master

    When implementing custom preprocessing or augmentation layers, you should subclass keras_cv.layers.preprocessing.BaseImageAugmentationLayer. This base class provides a common call() method and handles auto-vectorization via tf.vectorized_map() for performance.

    When subclassing, you can override the following methods:

    • augment_image(): Required. Implements the core image-wise augmentation logic.
    • augment_label(): Optional. Allows you to update labels (e.g., segmentation maps) in response to image transformations.
    • augment_bounding_box(): Optional. Allows you to update bounding boxes in response to image transformations.
    # Example pattern for subclassing
class MyCustomAugmentation(keras_cv.layers.preprocessing.BaseImageAugmentationLayer):
    def augment_image(self, images, **kwargs):
        # Implement image-wise logic here
        return images

    def augment_bounding_box(self, bounding_boxes, **kwargs):
        # Implement bounding box updates here
        return bounding_boxes

  9. Quickstart: Build a complete training pipeline

    master

    This example demonstrates the standard KerasCV workflow: creating an augmentation pipeline, preparing a dataset, selecting a pretrained backbone, and training a model.

    import tensorflow as tf
import keras_cv
import tensorflow_datasets as tfds
import keras

# 1. Create a preprocessing pipeline with augmentations
BATCH_SIZE = 16
NUM_CLASSES = 3
augmenter = keras_cv.layers.Augmenter(
    [
        keras_cv.layers.RandomFlip(),
        keras_cv.layers.RandAugment(value_range=(0, 255)),
        keras_cv.layers.CutMix(),
    ],
)

def preprocess_data(images, labels, augment=False):
    labels = tf.one_hot(labels, NUM_CLASSES)
    inputs = {"images": images, "labels": labels}
    outputs = inputs
    if augment:
        outputs = augmenter(outputs)
    return outputs['images'], outputs['labels']

# 2. Load and prepare data
train_dataset, test_dataset = tfds.load(
    'rock_paper_scissors',
    as_supervised=True,
    split=['train', 'test'],
)
train_dataset = train_dataset.batch(BATCH_SIZE).map(
    lambda x, y: preprocess_data(x, y, augment=True),
        num_parallel_calls=tf.data.AUTOTUNE).prefetch(
            tf.data.AUTOTUNE)
test_dataset = test_dataset.batch(BATCH_SIZE).map(
    preprocess_data, num_parallel_calls=tf.data.AUTOTUNE).prefetch(
        tf.data.AUTOTUNE)

# 3. Create a model using a pretrained backbone
backbone = keras_cv.models.EfficientNetV2Backbone.from_preset(
    "efficientnetv2_b0_imagenet"
)
model = keras_cv.models.ImageClassifier(
    backbone=backbone,
    num_classes=NUM_CLASSES,
    activation="softmax",
)
model.compile(
    loss='categorical_crossentropy',
    optimizer=keras.optimizers.Adam(learning_rate=1e-5),
    metrics=['accuracy']
)

# 4. Train your model
model.fit(
    train_dataset,
    validation_data=test_dataset,
    epochs=8,
)
    import tensorflow as tf
import keras_cv
import tensorflow_datasets as tfds
import keras

# Create a preprocessing pipeline with augmentations
BATCH_SIZE = 16
NUM_CLASSES = 3
augmenter = keras_cv.layers.Augmenter(
    [
        keras_cv.layers.RandomFlip(),
        keras_cv.layers.RandAugment(value_range=(0, 255)),
        keras_cv.layers.CutMix(),
    ],
)

def preprocess_data(images, labels, augment=False):
    labels = tf.one_hot(labels, NUM_CLASSES)
    inputs = {"images": images, "labels": labels}
    outputs = inputs
    if augment:
        outputs = augmenter(outputs)
    return outputs['images'], outputs['labels']

train_dataset, test_dataset = tfds.load(
    'rock_paper_scissors',
    as_supervised=True,
    split=['train', 'test'],
)
train_dataset = train_dataset.batch(BATCH_SIZE).map(
    lambda x, y: preprocess_data(x, y, augment=True),
        num_parallel_calls=tf.data.AUTOTUNE).prefetch(
            tf.data.AUTOTUNE)
test_dataset = test_dataset.batch(BATCH_SIZE).map(
    preprocess_data, num_parallel_calls=tf.data.AUTOTUNE).prefetch(
        tf.data.AUTOTUNE)

# Create a model using a pretrained backbone
backbone = keras_cv.models.EfficientNetV2Backbone.from_preset(
    "efficientnetv2_b0_imagenet"
)
model = keras_cv.models.ImageClassifier(
    backbone=backbone,
    num_classes=NUM_CLASSES,
    activation="softmax",
)
model.compile(
    loss='categorical_crossentropy',
    optimizer=keras.optimizers.Adam(learning_rate=1e-5),
    metrics=['accuracy']
)

# Train your model
model.fit(
    train_dataset,
    validation_data=test_dataset,
    epochs=8,
)

  10. Configure the Keras backend (JAX, TensorFlow, or PyTorch)

    master

    If you are using Keras 3, you can switch between JAX, TensorFlow, and PyTorch backends by setting the KERAS_BACKEND environment variable.

    Crucial: You must set the KERAS_BACKEND environment variable before importing any Keras libraries.

    # In a shell
export KERAS_BACKEND=jax

# In a Python script or Colab
import os
os.environ["KERAS_BACKEND"] = "jax"
import keras_cv
import keras

  11. Install KerasCV

    master

    KerasCV supports both Keras 2 and Keras 3. Keras 3 is recommended as it allows using KerasCV with JAX, TensorFlow, or PyTorch.

    Keras 2 Installation

    To install KerasCV with Keras 2:

    pip install --upgrade keras-cv tensorflow

    Keras 3 Installation

    Option 1: Nightly (Latest changes)

    pip install --upgrade keras-cv-nightly tf-nightly

    Option 2: Stable (Recommended for most users) Install Keras 3 after installing KerasCV. This is necessary for environments where TensorFlow is still pinned to Keras 2 (relevant for TensorFlow versions before 2.16).

    pip install --upgrade keras-cv tensorflow
pip install --upgrade keras
    IMPORTANT

    Keras 3 will not function with TensorFlow 2.14 or earlier.

    pip install --upgrade keras-cv tensorflow
pip install --upgrade keras

  12. Prepare the ImageNet dataset for KerasCV

    master

    KerasCV does not provide the ImageNet dataset directly. To use it, you must first download the original ImageNet dataset from image-net.org and then convert the images into TFRecord format.

    To perform the parsing and conversion, use the official TensorFlow script. This script can parse images into TFRecords and upload them to Google Cloud Storage or local storage. Once the TFRecords are generated, you can use the KerasCV loader to load the data from your chosen storage location.