DOCSEARCHbyAlgolia
Submit Docs

Pathway Python ETL Framework

repository·main·Indexed 13 days ago

https://github.com/pathwaycom/pathway

A Python ETL framework for stream processing, real-time analytics, and AI/RAG pipelines, powered by a high-performance Rust engine. Version 0.32.1 features a Live Data Framework capable of real-time document indexing via VectorStoreServer, integration with AG2 multi-agent systems, and support for streaming platforms like Kafka and Redpanda.

Tokens
271.8K
Snippets
832
Records
1.1K
Agent score
99%

What's inside Pathway

  1. Overview of Pathway Live Data Framework

    main

    Pathway is a Python ETL framework designed for stream processing, real-time analytics, LLM pipelines, and RAG (Retrieval-Augmented Generation).

    Key characteristics:

    • Unified Engine: Handles both batch and streaming data using the same code.
    • Scalable Rust Engine: While written in Python, the core execution is powered by a Rust engine based on Differential Dataflow, enabling multithreading, multiprocessing, and distributed computations.
    • Versatile Deployment: Suitable for local development, CI/CD, and production environments (Docker/Kubernetes).
    • Consistency: Manages time, late, and out-of-order data points automatically. The free version provides "at least once" consistency, while the enterprise version provides "exactly once" consistency.

  2. Overview of the Best-rated movies example project

    main

    This example project demonstrates an end-to-end application using the Pathway Live Data Framework to compute the K-best-rated items in a movie dataset. It specifically showcases the framework's ability to process data independently of the underlying streaming platform, allowing for a seamless switch between Kafka (with ZooKeeper) and Redpanda.

    The architecture consists of:

    1. A streaming platform (Kafka/ZooKeeper or Redpanda).
    2. A Python container that ingests a static CSV file and streams it to the platform.
    3. A Python container running the Pathway Live Data Framework to compute the K-best-rated items.

    The project uses a toy dataset formatted similarly to the MovieLens25M dataset.

  3. Overview of the Pathway LLM Extension Pack (XPack)

    main

    The LLM Extension Pack (XPack) provides specialized Pathway functions designed for building real-time data processing pipelines that incorporate Large Language Models (LLMs). It offers two main categories of functionality:

    1. Pathway-compatible wrappers (UDFs): User-Defined Functions for common AI and language modeling tasks, including:
      • Document parsing
      • Text embedding
      • LLM API calls
    2. Ready-made pipelines: Pre-built document processing workflows for common tasks like document indexing.

  4. Overview of Pathway Connectors

    main

    Connectors in the Pathway Live Data Framework are used to access external data sources. They are categorized into two types:

    1. Input Connectors: Used to ingest data into the Pathway framework.
    2. Output Connectors: Used to export processed data from the framework to external systems.

    Connectors behave differently depending on whether you are using Streaming mode or Static mode. It is recommended to understand the differences between these modes before selecting a connector.

  5. Overview of Pathway Key Features

    main

    Pathway is a Python-based framework powered by a scalable Rust engine. Key capabilities include:

    • Python Compatibility: Fully compatible with Python tools and ML libraries.
    • High Performance: A Rust engine with multithreading and multiprocessing (no JVM, no GIL).
    • Real-time Processing: Uses Differential Dataflow for incremental computations and handling out-of-order data.
    • Unified API: Use the same pipeline for both static (batch) and live (streaming) data.
    • Stateful Operations: Supports temporal operations like groupby and windows.
    • AI & LLM Tooling: Built-in support for online ML, RAG pipelines, and vector indexes.
    • Connectivity: 350+ connectors (including SharePoint) with support for custom implementations.
    • Reliability: Exactly-once consistency, in-memory processing, and persistence/backfilling to resume after failures.
    • Deployment: Easy deployment via Docker or Kubernetes, featuring an orchestrator and OpenTelemetry compatibility.

  6. Realtime log monitoring with Filebeat, Kafka, and Pathway

    main

    This project demonstrates an end-to-end real-time monitoring pipeline using the Pathway Live Data Framework. It monitors logs (e.g., nginx logs) by connecting Filebeat to Pathway via Kafka and sending alerts to a Slack channel.

    The architecture consists of four Docker containers:

    1. Filebeat: Monitors logs and sends updates to Kafka.
    2. Kafka and Zookeeper: Acts as a gateway between Filebeat and Pathway.
    3. Pathway Live Data Framework: Receives logs from Kafka, processes them, and sends alerts to Slack.

    Log Processing Logic (defined in ./pathway-src/alerts.py):

    • Extracts timestamps and messages from Filebeat JSON messages.
    • Converts ISO8601 timestamps to real timestamps.
    • Filters for messages within the last X seconds (default X=1s), where the current time is determined by the timestamp of the last log.
    • Triggers an alert (alert=True) if more than Y messages occur within that window (default Y=5).

  7. Benchmark components in the Kafka alternatives project

    main

    The benchmark is composed of several specialized scripts:

    • lib.py: Contains Python wrappers for Pathway Live Data Framework classes used to build the message queue on top of Delta Lake.
    • producer.py: Implements logic to generate messages and add them to the Delta Lake-based message queue at a specific streaming rate.
    • consumer.py: Implements logic to retrieve messages and track latency percentiles.
    • benchmark.py: The main entry point that coordinates the producer and consumer and manages result persistence.

  8. Use Pathway Application Templates for AI and ETL

    main

    Pathway provides ready-to-deploy ETL and RAG (Retrieval-Augmented Generation) pipeline templates. These templates allow you to quickly deploy production-grade AI applications that offer high-accuracy RAG at scale by processing real-time data from your sources.

    Templates are available in both YAML and Python formats, making them accessible for both developers and non-developers. They are designed to handle scalable, real-time data processing and AI-driven search capabilities.

  10. How Pathway Live Data Framework Connectors Work

    main

    Pathway's connector architecture is designed to ingest data from diverse sources (e.g., Kafka, NATS, MQTT, Kinesis, file systems, and databases) into a high-performance Rust runtime. This architecture provides two main benefits:

    1. Predictable Latency: The Rust core avoids Garbage Collection (GC) pauses, making it suitable for long-running streaming workloads.
    2. Parallelism: By moving the core to Rust, the framework bypasses the Python Global Interpreter Lock (GIL), enabling true parallel execution.

    The architecture separates the interaction with source systems from the logic of parsing data formats, allowing data formats (like JSON or JSON Lines) to be reused across different sources. It also integrates with Pathway's data persistence mechanism to ensure fault tolerance: pipelines can be stopped and restarted, resuming exactly where they left off without data loss or duplicate processing within the committed frontier.

  11. Core capabilities of Pathway

    main

    Pathway is a unified Live Data Framework designed for stream processing, data indexing, and real-time AI analytics. Key features include:

    • Unified Batch & Stream Processing: Use the same syntax and engine for both batch and streaming data without modifying your code.
    • Python Compatibility: Fully compatible with the Python ecosystem, allowing the use of favorite libraries, custom UDFs, and Machine Learning models.
    • High Performance: Powered by a Rust engine based on Differential Dataflow, providing multi-threading, multi-processing, and distributed computation capabilities without being limited by the Python GIL.
    • Real-time Data Indexing: Built-in support for vector search and full-text search, allowing you to synchronize indexes with data sources in real time without a dedicated vector store.
    • Advanced Temporal Operations: Built-in handling for complex streaming challenges like late or out-of-order data, including support for as-of-join and temporal windows.
    • Stateful Operations: Supports operations like groupby and windowing, with the ability to persist state for pipeline updates or recovery.

  12. Explore Pathway Live Data Framework examples

    main

    The examples/ directory contains various implementations of real-time and reactive data products using the Pathway Live Data Framework. These examples cover a range of use cases from simple Jupyter notebooks to complex production-ready pipelines.

    Key example categories include:

    • Interactive Exploration: Working with live data streams in Jupyter or Colab.
    • Monitoring & Observability: Real-time server log monitoring (e.g., Filebeat to Slack or Logstash to Elasticsearch).
    • AI & Vector Search: Always up-to-date data indexing pipelines for vector databases and building LLM applications.
    • Anomaly Detection: Detecting suspicious user activity using Tumbling Window group-by operations.
    • Stream Processing: Upsampling signal processing data by transforming multiple streams and computing graph algorithms like PageRank.
    • Social Media Analytics: Real-time Twitter analysis applications.