DOCSEARCHbyAlgolia
Submit Docs

smartpipe

website·Indexed 29 days ago

https://prabal-rje.github.io/smartpipe

A CLI tool for building AI pipelines using semantic verbs. smartpipe allows users to process data through a series of operations such as map, filter, embed, join, and top_k to perform tasks like log triage, document ranking, and knowledge graph creation. It supports structured JSON and plain text output, configuration management via CLI, and integration with AI agents.

Tokens
42K
Snippets
257
Records
328
Agent score
72%

What's inside smartpipe

  1. Overview of smartpipe semantic pipes

    smartpipe is a CLI tool that enables semantic processing of data (PDFs, images, audio, video, and text) using Unix-style pipes. It allows developers to compose operations using 'verbs' that understand their input. It supports cloud API keys (e.g., ChatGPT) or local models via Ollama, maintaining standard stdin/stdout composition. By default, each line of a text file is treated as one item, but the --as file flag can be used to treat a whole file as a single item.
    cat reviews.txt \
| smartpipe filter "the reviewer is sarcastic" \
| smartpipe map "Extract {product, complaint, anger number: 0 to 1}"

  2. Overview of smartpipe semantic operators

    smartpipe provides semantic operators for Unix pipes, allowing developers to treat LLM operations as distinct pipe stages rather than a single prompt command. Supported operators include map, filter, reduce, cluster, join, distinct, embed, and top_k.

  3. Overview of smartpipe CLI verbs

    The smartpipe CLI uses a series of 'verbs' to process streams of items. These verbs are categorized by whether they require model calls (LLM/Embeddings) or are 'free' deterministic operations.

  4. Understand the five laws of smartpipe data flow

    Everything in smartpipe is processed as an 'item' (a record). The system follows five fundamental laws:

    1. Everything is a record: Verbs interact with records and fields, not raw files or lines.
    2. Plain text is shorthand: A plain text line is treated as a text-only record (equivalent to {"text": "..."}).
    3. Files are crates: A file contains one or more items depending on the granularity setting.
    4. Verbs transform records: map returns a record, extend adds fields, and filter passes records through byte-identically.
    5. Edges determine format: Simple records are output as plain text (human-readable), while full records are output as JSONL (machine-readable).

  5. Understand Cloud vs Local model providers in smartpipe

    smartpipe supports both cloud-based models (OpenAI, Anthropic, Mistral, Gemini, OpenRouter) and local models via Ollama.

    • Cloud: Paid per use, data is sent to the provider, generally faster and more capable.
    • Local (Ollama): Free, data stays on your machine, performance depends on local hardware.

    If no model is configured, smartpipe attempts to use a local Ollama non-embedding model first. If neither a config nor a local model is found, it will trigger a setup screen.

  6. Perform basic Video Q&A with smartpipe

    Use the smartpipe map command to ask a natural language question about a video file. On Gemini, the model processes the entire file; for other models, smartpipe samples frames and the soundtrack to fit model constraints.
    smartpipe map "What is the speaker's main claim?" talk.mp4

  7. Understand smartpipe data flow and privacy

    smartpipe sends data only to the configured model endpoint. If using a local endpoint (e.g., Ollama), data stays on the local machine. If using a cloud provider, data is sent to that provider under their terms. By default, embeddings (using fastembed's nomic model) and local transcription (using whisper) are performed locally regardless of the chat model used.

  8. Understand smartpipe item granularity

    smartpipe treats data as a series of 'items'. The definition of an item depends on the input source:

    • Standard Input (stdin): Each line is treated as one item. This is used when piping data or using redirects.
    • Files: When using named files or the --from-files flag, each file is treated as one item by default (though this can be adjusted using the --as flag).
    # Example: Each line of the log is one item
cat server.log | smartpipe filter "database timeout"

# Example: Each PDF file is one item
smartpipe map "Summarize this document" 'reports/*.pdf'

  9. Determine item granularity in smartpipe

    Smartpipe determines the 'rung' of the granularity ladder (what constitutes one item) based on the following priority:

    1. Explicit Overrides: The --as flag overrides all other logic.
    2. Path Defaults: File extensions determine the default cut.
    3. Stdin Sniffing: If input comes from stdin, smartpipe analyzes each line to guess the type.

    Granularity Rungs:

    • file: A whole file (text + media). Used for document-level tasks like summarizing a report.
    • split: A piece of a file (page, N tokens, or N seconds). Used when documents are too large or need addressable pieces.
    • lines / rows: A single text line, JSONL record, or CSV row. Used for logs and datasets.

  10. Ingest data via stdin

    smartpipe can accept data via stdin, converting it into items based on the content type:

    • Text lines: Each line becomes one text item (as: lines).
    • JSONL rows: Each line becomes one record (as: jsonl).
    • Mixed streams: Both lines and records are captured, with a census note provided.
    • Single binary document (e.g., < report.pdf): Becomes one whole-document item.
    • Image/Audio/Video stream: Becomes one media item.

    To enforce strict row processing and error on mixed streams, use the --strict-rows flag or set the environment variable SMARTPIPE_STRICT_ROWS=1.

  11. Name the strongest links in a knowledge graph using LLMs

    To replace generic co-occurs relations with specific model-derived relations for the most important edges, use the --name-top N flag. This spends one model call per edge for the N heaviest links. Use --max-calls to set a total budget for the entire run (including embedding and naming calls). If the budget is exceeded, remaining edges stay as co-occurs.
    smartpipe graph "who works on which program" --name-top 5 'reports/*.pdf' --entities "aircraft, organization, person" --max-calls 12