DOCSEARCHbyAlgolia
Submit Docs

Hamilton Dataflow Hub

website·Indexed 33 days ago

https://hub.dagworks.io

A curated collection of Hamilton dataflows, including officially maintained modules by DAGWorks and community-contributed examples. The hub provides instructions for implementing dataflows for RAG, fine-tuning, text summarization, and air quality analysis, with support for dynamic execution via ~/.hamilton/dataflows or the sf-hamilton-contrib library.

Tokens
34.3K
Snippets
151
Records
202
Agent score
63%

What's inside Hamilton Dataflow Hub

  1. Overview of the caption_images dataflow

    The caption_images module provides a Hamilton dataflow that generates captions for images using OpenAI. It supports two primary modes: a basic captioning mode and an extended mode that also generates embeddings for the resulting caption. This module requires the OPENAI_API_KEY environment variable to be set.

  2. Overview of the sphinx_doc_chunking module

    The sphinx_doc_chunking module is used to extract pages from Sphinx Furo themed documentation and chunk the text for further processing (e.g., creating embeddings). It is designed to be generic and extensible, running in any Python environment and supporting distributed execution via Ray, Dask, or PySpark.

  3. Overview of the convert_images_s3 dataflow

    The convert_images_s3 module allows for the batch conversion of images stored in S3, saving the converted versions in the same bucket next to the originals. It supports any image format compatible with the Pillow library. It assumes that files are correctly labeled by their extensions and that all files to be processed are in the same source format.

  4. Overview of nixtla_statsforecast Hamilton module

    The nixtla_statsforecast module implements time series forecasting using Nixtla's statsforecast library. It allows users to fit and evaluate multiple models on a dataset, generating cross-validation benchmark dataframes and prediction plots. The input dataset must contain the following columns: unique_id (series identifier), ds (timestamp), and y (target value). Note that this flow does not include dataset preprocessing steps.

  5. Overview of the customize_embeddings module

    The customize_embeddings module allows developers to customize text embeddings for specific tasks by generating a matrix that, when multiplied by original embeddings, produces 'custom embeddings'. This process can improve performance in tasks like binary classification by better distinguishing between similar and dissimilar text pairs. The module is based on an OpenAI cookbook example and utilizes libraries such as numpy, pandas, plotly, torch, sklearn, and openai.

  7. Overview of the text_summarization dataflow

    The text_summarization module implements a Hamilton dataflow that uses the OpenAI API to summarize text. It supports processing PDF files, plain text files, or raw text strings. Note that this module is limited to OpenAI and does not currently perform context length checks, which may lead to failures if the input text exceeds the model's context window.

  8. Overview of the llm_generate_code module

    The llm_generate_code module uses the OpenAI completion API to generate code in various languages. For Python code, the module can also execute the generated code in a subprocess and return the output and any errors encountered.

  9. Overview of nixtla_mlforecast dataflow

    The nixtla_mlforecast module implements machine learning forecasting using Nixtla's mlforecast. It allows users to fit and evaluate multiple models on a time series dataset to obtain cross-validation benchmarks and prediction plots. The input dataset must contain the following columns: unique_id (series identifier), ds (time step), and y (target value).

  10. Implement custom embeddings using the customize_embeddings module

    The customize_embeddings module provides a framework for generating and managing text embeddings using OpenAI. It includes utilities for downloading datasets (like SNLI), processing text pairs for similarity tasks, and caching embeddings to optimize performance. The module is designed to be used within a Hamilton dataflow, allowing for configuration overrides and data validation via Pandera schemas.

  11. Summarize text or PDFs using the text_summarization dataflow

    The text_summarization module provides a universal dataflow to summarize text or PDF files using an LLM of your choice (e.g., OpenAI, Cohere). By default, it uses OpenAI, but you can specify a different model by passing llm_name in the inputs dictionary during execution.

    Requirements:

    • You must set the appropriate LLM provider's API key in your environment variables (e.g., OPENAI_API_KEY or COHERE_API_KEY).

    Limitations:

    • The module does not check context length; it may fail if the input text exceeds the model's context window.

  12. Implement a RAG pipeline using the faiss_rag module

    The faiss_rag module provides a reference implementation of a Retrieval Augmented Generation (RAG) pipeline using Hamilton. It utilizes FAISS (via LangChain) for an in-memory vector store and OpenAI as the LLM provider. This module is designed to demonstrate how to structure RAG code within Hamilton.