DOCSEARCHbyAlgolia
Submit Docs

Semlib: Data Processing and Analysis Pipelines with LLMs

repository·master·Indexed 42 days ago

https://github.com/anishathalye/semlib

A Python library for building data processing and analysis pipelines using Large Language Models (LLMs). Semlib provides functional programming primitives such as map, reduce, sort, and filter that operate on semantic natural language descriptions rather than traditional code. It handles prompting, parsing, concurrency, and cost tracking, with support for structured outputs via Pydantic and Bare types. It uses LiteLLM as an abstraction layer to support various models including OpenAI, Anthropic, and Ollama.

Tokens
13.4K
Snippets
58
Records
68
Agent score
53%

What's inside Semlib

  1. Overview of Semlib: Semantic Data Processing

    master

    Semlib is a Python library designed for building data processing and analysis pipelines using Large Language Models (LLMs). Unlike traditional data processing that uses code-based logic, Semlib uses functional programming primitives—such as map, reduce, sort, and filter—that are programmed using natural language descriptions.

    Semlib abstracts away the complexities of LLM integration, including:

    • Prompting and parsing
    • Concurrency control
    • Caching
    • Cost tracking

    This approach allows you to handle arbitrary-sized datasets, improve result quality by breaking complex tasks into simpler steps, reduce latency through concurrency, and optimize costs by using smaller models for specific sub-tasks.

  2. Type checking and safety

    master

    Semlib makes extensive use of Python type annotations, generics, and overloads.

    Important: Semlib does not perform runtime checks for invalid arguments that would typically be caught by a type checker. To ensure code correctness and catch errors early, it is highly recommended to use a static type checker like mypy when developing with Semlib.

  3. Use semantic functional primitives in Semlib

    master

    Semlib provides semantic versions of standard functional programming operations. Instead of writing logic in Python, you provide natural language instructions to the LLM to perform the operation on your data.

    Common primitives include:

    • prompt: To generate data or answers from a query.
    • sort: To order data based on a semantic description (e.g., by="right-leaning").
    • find: To locate a specific item in a collection based on a description.
    • map: To transform each element in a collection using a natural language template.

    Note: These operations are asynchronous and typically require await.

    >>> presidents = await prompt(
...     "Who were the 39th through 42nd presidents of the United States?",
...     return_type=Bare(list[str])
... )

>>> await sort(presidents, by="right-leaning", reverse=True)  # highest first
['Ronald Reagan', 'George H. W. Bush', 'Bill Clinton', 'Jimmy Carter']

>>> await find(presidents, by="former actor")
'Ronald Reagan'

>>> await map(
...     presidents,
...     "How old was {} when he took office?",
...     return_type=Bare(int),
... )
[52, 69, 64, 46]

  4. Choose between asynchronous and synchronous interfaces

    master

    Semlib is optimized for asynchronous execution.

    • Asynchronous (Recommended): Use the standard API methods in async code.
    • Synchronous: Semlib exports a synchronous interface via functions with the _sync suffix. These functions wrap the asynchronous implementation using asyncio.run.

    Warning: If you must use the synchronous interface (_sync methods) inside an existing asynchronous loop, you may need to use nest_asyncio to avoid errors.

  5. Use structured output with return_type

    master

    Semlib simplifies working with structured outputs by allowing you to specify a return_type in many functions and methods (such as semlib.session.Session.map).

    • Default Behavior: If return_type is not provided, the output is assumed to be a str.
    • Pydantic Models: You can pass a Pydantic model as the return_type to have Semlib parse and validate the LLM's output into that model.
    • Bare Types: For standard Python types like int or list[float], use the semlib.bare.Bare class to wrap the type so Semlib knows to parse it.

    Note: To use this feature reliably, ensure the LLM model you select supports structured output.

    # Example using a Pydantic model
from pydantic import BaseModel
from semlib.session import Session

class User(BaseModel):
    name: str
    age: int

# session.map(..., return_type=User)

# Example using a bare type
from semlib.bare import Bare

# session.map(..., return_type=Bare[int])

  6. Run tests with Hatch

    master

    Execute the test suite using the hatch test command. You can customize the test execution with several options:

    • -c: Measure test coverage.
    • -a: Test against a matrix of Python versions.
    • Specific test targeting: Append the path to a specific test file or function (e.g., hatch test tests/path/to/test.py::test_name).
    hatch test

  8. Use Semlib for semantic data processing

    master

    Semlib allows you to build data processing pipelines using functional programming primitives (map, reduce, sort, filter) that are driven by natural language descriptions instead of traditional code. This enables you to perform complex data analysis tasks by leveraging LLMs for semantic understanding while Semlib handles prompting, parsing, concurrency, and cost tracking.

    Key operations include:

    • prompt: Execute a natural language query.
    • sort: Sort data based on a semantic description (e.g., by="right-leaning").
    • find: Locate a specific item based on a semantic description.
    • map: Transform data using a natural language template.

    Note: Most operations are asynchronous and should be awaited.

    >>> presidents = await prompt(
...     "Who were the 39th through 42nd presidents of the United States?",
...     return_type=Bare(list[str])
... )

>>> await sort(presidents, by="right-leaning", reverse=True)  # highest first
['Ronald Reagan', 'George H. W. Bush', 'Bill Clinton', 'Jimmy Carter']

>>> await find(presidents, by="former actor")
'Ronald Reagan'

>>> await map(
...     presidents,
...     "How old was {} when he took office?",
...     return_type=Bare(int),
... )
[52, 69, 64, 46]

  9. Preview and build documentation

    master

    Semlib uses [MkDocs] for its documentation. You can preview the documentation site locally with live-reloading or build the static site into the site/ directory.

    # Preview the documentation site locally at http://localhost:8000/
hatch run docs:serve

# Build the compiled site into the site/ directory
hatch run docs:build

  10. Configure LLM models via LiteLLM

    master

    Semlib uses LiteLLM as an abstraction layer, meaning you can use any model supported by LiteLLM by passing the appropriate string to the model argument.

    • Anthropic: Set ANTHROPIC_API_KEY and use model="anthropic/claude-sonnet-4-20250514".
    • Ollama: Set up Ollama and use model="ollama_chat/gpt-oss:20b".
    • Default: Semlib defaults to gpt-4o, which requires the OPENAI_API_KEY environment variable to be set.

    When using the return_type feature for structured output, ensure your chosen model supports it.

    # Example model selection
# model="anthropic/claude-sonnet-4-20250514"
# model="ollama_chat/gpt-oss:20b"

  11. Run static type checking with Mypy

    master

    Semlib uses mypy for static type checking. You can run the type checker on the source code or specifically on the documentation notebooks using the following Hatch commands.

    # Check types in the source code
hatch run types:check

# Check types in the documentation .ipynb notebooks
hatch run types:check-docs

  12. Set up the Semlib development environment with Hatch

    master

    Semlib uses the Hatch project manager to manage dependencies and run operations in isolated environments. To develop with Semlib, you should install Hatch following the official instructions.

    https://hatch.pypa.io/latest/install/