Feast (Feature Store) Documentation

Feast is an open-source feature store designed to manage and serve machine learning features for both batch and real-time applications. Key capabilities include:

• Point-in-time correctness: Prevents data leakage during training by using battle-tested logic for joins.
• Online feature availability: Manages deployment to online stores (e.g., DynamoDB, Redis, Google Cloud Datastore) to ensure features are available and fresh for real-time inference.
• Feature and model versioning: Enables discovery and collaboration through a centralized catalog and allows versioning of feature sets via feature services.
• Unified SDK and Service: Provides a Python SDK for data scientists and an optional hosted feature server for non-Python language integration.

The Feature Server is a core architectural component in Feast that provides low-latency feature retrieval and updates for machine learning applications. It is a stateless REST API server built with FastAPI that serves as an interface between your ML applications and the Feast Online Store and Registry.

Key capabilities:

• Serving Features: Real-time retrieval of feature values for specific entities.
• Data Integration: Pushing feature data into online or offline stores.
• Scalability: Supports horizontal scaling for high request volumes.
• Security: Supports TLS (SSL) for production environments.
• Deployment: Can be run locally, via Docker, or on Kubernetes using Helm charts.

Feast (Feature Store) is an open-source feature store designed to help teams manage, validate, and serve features for production AI/ML. It consists of two main components:

1. Offline Store: Used for historical feature extraction during model training.
2. Online Store: Used for low-latency feature serving in production applications.

Feast provides a Python SDK for defining features (entities, sources, transformations) and reading/writing data, an optional feature server for non-Python languages, a Web UI for exploration, and a CLI tool for management. It uses a push model for online serving, where the feature store pushes values to the online store to reduce retrieval latency.

Feast provides three primary deployment topologies for Kubernetes using the Feast Operator, categorized by organizational maturity and scale:

1. Minimal Production: Best for small teams, POCs, or low-traffic, non-critical workloads. It uses a single namespace with no high availability (HA) and a simple setup.
2. Standard Production (Recommended): Designed for most production workloads. It includes HA registry, autoscaling, TLS, and RBAC.
3. Enterprise Production: Designed for large-scale, multi-tenant organizations. It features namespace isolation, managed stores, and full observability.

Prerequisites:

• A running Kubernetes cluster.
• The Feast Operator installed.
• Familiarity with Feast concepts and components.

A Stream Processor in Feast consumes data from stream sources (e.g., Kafka, Kinesis) and loads it directly into the online store and, optionally, the offline store. It provides an abstraction over the specific technologies or frameworks used to materialize data.

Feast currently provides an experimental Spark Processor for Kafka. If the built-in processors do not meet your requirements, you can implement a custom processor.

The feast Python package is organized into several functional subpackages. Key subpackages include:

• feast.api: Core API definitions.
• feast.cli: Command-line interface tools.
• feast.aggregation: Logic for feature aggregations.
• feast.transformation: Transformation capabilities.
• feast.infra: Infrastructure-related components.
• feast.ui: Web UI components and servers.
• feast.dqm: Data Quality Management.
• feast.lineage & feast.openlineage: Data lineage tracking.
• feast.permissions: Access control and permissions.
• feast.protos: Protocol Buffer definitions.
• feast.dbt: Integration with dbt.
• feast.diff: Tools for comparing feature definitions.

The Feast Java ecosystem consists of two primary components designed for feature serving:

1. Feast Serving: A gRPC service responsible for serving the latest feature values to machine learning models.
2. Feast Serving Client: A client used by applications to retrieve features from the Feast Serving service.

Feast Serving depends on an online store (such as Redis) to retrieve feature values. The data ingestion process into the online store is decoupled from the feature retrieval process.

Feast supports building streaming feature pipelines through the following capabilities:

• Streaming Feature Views: Registering feature views that consume from streaming sources.
• Streaming Sources: Support for Kafka and Kinesis as streaming data sources.
• Stream Processor: An interface for stream processing. An example implementation using Kafka/Spark is available in the contrib folder of the repository.

For a complete end-to-end tutorial on building a versioned streaming pipeline (including registering transformations, features, and data sources), refer to the streaming-tutorial repository.

Feast uses a Role-Based Access Control (RBAC) model to secure and partition feature store resources. Permissions are defined as Feast objects and registered in the Feast registry using Python code.

A Permission object consists of three components:

1. Resource: The Feast object being secured (e.g., Project, Entity). It is identified by type, optional name patterns, and required tags.
2. Action: The operation performed on the resource. Actions include managing state (CREATE, DESCRIBE, UPDATE, DELETE) or accessing data (READ, WRITE—which are differentiated by ONLINE and OFFLINE store types).
3. Policy: The rule used to enforce authorization (the default implementation uses role-based policies).

Supported server types in an authorized environment:

• Online REST feature server
• Offline Arrow Flight feature server
• gRPC Registry server

A feature repository is the declarative source of truth for your feature store. It is a directory containing configuration and feature definitions that the Feast CLI uses to manage infrastructure (e.g., via feast apply).

A standard feature repository includes:

• feature_store.yaml: Located at the root, containing infrastructural configuration.
• Python files: Containing feature declarations (e.g., FeatureView, Entity).
• .feastignore (optional): A file specifying paths to ignore during feast apply.

Note: Feast reads all Python files recursively when feast apply is run, including those in subdirectories. It is recommended to use .feastignore to exclude imperative scripts or virtual environments.

$ tree -a
.
├── data
│   └── driver_stats.parquet
├── driver_features.py
├── feature_store.yaml
└── .feastignore

1 directory, 4 files