Apache Hamilton Documentation

Apache Hamilton is composed of five independently versioned packages. The core apache-hamilton package must be released before the others. The other four packages depend on the core library but do not depend on each other.

| Package | Key | Working Directory | Description |
|---|---|---|---|
| `apache-hamilton` | `hamilton` | `.` | Core library (must be released first) |
| `apache-hamilton-sdk` | `sdk` | `ui/sdk` | Tracking SDK |
| `apache-hamilton-contrib` | `contrib` | `contrib` | Community dataflows |
| `apache-hamilton-ui` | `ui` | `ui/backend` | Web UI server |
| `apache-hamilton-lsp` | `lsp` | `dev_tools/language_server` | Language server |

This application implements a Retrieval Augmented Generation (RAG) workflow using Apache Hamilton to manage dataflows.

Workflow Components

• Ingestion: Processes arXiv PDFs or arbitrary PDF files. Text is divided into chunks, embedded using OpenAI, and stored in a Weaviate vector database.
• Retrieval: When a user queries the system, the most relevant chunks are retrieved from Weaviate.
• Generation: A summary answer is generated using OpenAI based on the retrieved context.

Technology Stack

• Orchestration: Apache Hamilton (manages ingestion and retrieval dataflows).
• API Layer: FastAPI (exposes Hamilton dataflows as endpoints).
• Frontend: Streamlit (provides a web UI).
• Vector Database: Weaviate.
• LLM/Embeddings: OpenAI.
• Deployment: Docker Compose.

The Dagster example project follows this structure:

• tutorial/: The core Python module containing the Dagster project.
• pyproject.toml & setup.py: Configuration files used to install the tutorial/ module.
• tutorial/assets.py: Defines the data assets that are computed and materialized.
• tutorial/__init__.py: Registers the data assets, jobs, and resources for the Dagster orchestrator.
• tutorial/resources/: Contains configuration and logic to connect to external resources and APIs.

Beyond PySpark, Apache Hamilton integrates with several other distributed computing and scaling libraries depending on your use case:

• Dask: Supported via dedicated integration.
• Ray: Supported via dedicated integration.
• Modin: Supported (uses the pandas API with a different import).

Refer to the respective library directories (e.g., ../../dask/README.md) for specific setup and usage instructions.

This example demonstrates how to combine Ibis and Apache Hamilton for machine learning workflows:

• Ibis: A portable dataframe library used to write procedural data transformations in Python that can be executed on various SQL backends (e.g., DuckDB, Snowflake, Postgres, Flink).
• Apache Hamilton: Provides a declarative way to define modular, testable, and self-documenting dataflows that encode lineage and metadata.

In this specific integration, Hamilton manages the dataflow orchestration while Ibis handles the portable data transformations. The example covers creating feature transformations and training machine learning models.

The webscraper module is designed to collect specific HTML tags from a provided list of URLs while removing undesirable ones. It uses the tenacity package to handle HTTP request timeouts and retry logic.

Inputs

• urls (Required): A list of valid URLs to scrape.
• tags_to_extract: A list of HTML tags to extract from the pages.
• tags_to_remove: A list of HTML tags to remove from the pages.

The example is organized into several files to demonstrate how the same logic can be applied across different execution systems:

• doc_pipeline.py: The core Apache Hamilton code defining the document chunking pipeline.
• run.py: Entry point for local execution.
• run_dask.py: Entry point for Dask clusters/local Dask.
• run_ray.py: Entry point for Ray clusters/local Ray.
• spark/doc_pipeline.py: The pipeline logic adjusted specifically for PySpark.
• spark/spark_pipeline.py: Entry point for Spark clusters/local Spark.
• spark/README.md: Specific details regarding the Spark implementation differences.

This module implements forecasting using statistical methods via Nixtla's statsforecast. It allows you to fit and evaluate a list of models on a time series dataset, producing a cross-validation benchmark dataframe and prediction plots.

Dataset Requirements

Your input dataset must contain the following columns:

• unique_id: Identifies each individual time series.
• ds: Identifies the time step.
• y: Specifies the value of the series unique_id at time ds.

The People Data Labs example is organized into three main components to demonstrate how Apache Hamilton manages dataflows:

• notebook.ipynb: A step-by-step introduction to Apache Hamilton concepts.
• analysis.py: Contains the modular data transformations (the Hamilton functions) used in the analysis.
• run.py: The execution script that orchestrates the analysis using the transformations defined in analysis.py.