DOCSEARCHbyAlgolia
Submit Docs

Streamly

repository·master·Indexed 21 days ago

https://github.com/composewell/streamly

A high-performance Haskell streaming library providing declarative concurrency and a unified API for streaming, reactive programming, serialization, and I/O. It utilizes stream-fusion techniques to achieve C-like performance by eliminating intermediate allocations and function calls. The ecosystem includes specialized packages such as streamly-process, streamly-shell, and streamly-coreutils.

Tokens
51.2K
Snippets
108
Records
246
Agent score
73%

What's inside streamly

  1. Overview of Streamly

    master

    Streamly is a high-performance Haskell library designed for creating safe, scalable, and modular software. It combines Haskell's strong type safety with C-like performance by utilizing stream-fusion techniques.

    Key features include:

    • Blazing Fast Performance: Uses stream-fusion to eliminate intermediate allocations and function calls, often performing up to 100x faster than non-fused libraries.
    • Declarative Concurrency: Provides high-level abstractions for parallel execution (e.g., concurrent mapping, concurrent stream combination) without requiring manual management of locks or threads.
    • Unified API: A single interface covering streaming, concurrency, logic programming, reactive programming, arrays (pinned/unpinned), serialization, parsers, and I/O (file, network, filesystem events).
    • Highly Modular: Achieves high performance through GHC optimizations (like case-of-case and spec-constr) while maintaining modular code structures.

  2. Overview of the Streamly Ecosystem

    master

    Streamly provides a core streaming library supplemented by specialized ecosystem packages for common real-world tasks. These packages allow you to extend your streaming pipelines with process management, system utilities, statistics, and file system monitoring.

    Key ecosystem packages include:

    • streamly-process: Used to launch and compose executables, connect input/output streams, run shell commands, and integrate them into Haskell streaming pipelines.
    • streamly-coreutils: Provides safe, modular, high-performance Haskell equivalents for common shell or Python scripts.
    • streamly-statistics: Enables incremental statistical analysis on sliding windows of data within a pipeline.
    • streamly-fsevents: Generates event streams by watching files or directories for file system events (e.g., inotify).

  3. Explore related Haskell libraries for Streamly

    master

    Streamly operates in several domains including streaming, concurrency, and non-determinism. If you are looking for alternative libraries or complementary tools, the following categories represent the ecosystem Streamly interacts with:

    • Streaming: Alternatives like conduit, pipes, streaming, and io-streams.
    • Concurrency: Tools for parallel and asynchronous programming such as async, parallel, and monad-par.
    • Non-determinism: Libraries handling non-deterministic effects like logict and list-t.
    • FRP (Functional Reactive Programming): Frameworks like reflex, reactive-banana, and Yampa.
    • Folds: Specialized folding libraries like foldl and folds.
    • Distributed Computing: Libraries for distributed systems like cloud-haskell and distributed-process.
    • Resource Handling: Tools for safe resource management like resourcet and pipes-safe.
    • Web: Web-related frameworks such as reflex-dom and axiom.

  4. Understanding the Streamly module hierarchy

    master

    Streamly uses the Streamly prefix for all module names to prevent namespace conflicts on Hackage. The hierarchy is organized as follows:

    Data

    Contains basic data structures and processing primitives:

    • Streamly.Data.Stream: Effectful streams.
    • Streamly.Data.List: Pure streams (behave like lists).
    • Streamly.Data.Array: Unboxed immutable arrays.
    • Streamly.Data.Array.Generic: Boxed immutable arrays.
    • Streamly.Data.MutArray: Unboxed mutable arrays.
    • Streamly.Data.MutArray.Generic: Boxed mutable arrays.
    • Streamly.Data.Fold: Folding operations.
    • Streamly.Data.Unfold: Unfolding operations.
    • Streamly.Data.Parser: Parsing operations.

    Unicode

    For Unicode text processing:

    • Streamly.Unicode.Char: Operations on individual characters.
    • Streamly.Unicode.Stream: Operations on streams of Char.
    • Streamly.Unicode.Array.Char: Compact UTF-32 character arrays.
    • Streamly.Unicode.Array.Utf8: Compact UTF-8 encoded character arrays.

    Other Namespaces

    • Streamly.FileSystem: Data structures residing in files via a file system interface.
    • Streamly.Network: APIs for accessing data from remote computers over a network.

  5. Use streamly-coreutils for Unix-style operations

    master

    The streamly-coreutils package provides reimplementations of GNU coreutils as composable, concurrent Haskell functions. You can use these directly within streaming pipelines to perform file and system operations with the safety and performance of Haskell.

    Supported utilities include:

    • File manipulation: cp, mv, rm, mkdir, touch, ln
    • File information: ls, stat, test, readlink, pwd, cd
    • System/Other: which, sleep

  6. Accessing experimental APIs via Streamly.Internal

    master

    Streamly exposes most of its internal modules under the Streamly.Internal.* namespace. This allows users to experiment with experimental APIs and constructors that are not yet part of the stable public surface.

    Warning: APIs found in the Streamly.Internal namespace are subject to change without notice and may not be stable. Use them at your own risk for experimentation.

  7. Optimize performance via Chunked Processing

    master

    Processing data in larger chunks reduces loop overhead and improves cache locality, vectorization, and branching efficiency.

    • Inner Loop: Fusion is critical here. Use Unfold or fused Stream operations for chunk processing, and use fused Fold or Parser types on the consumer side.
    • Outer Loop: Fusion is less critical. You can use the fused Stream type or the more flexible CPS-based StreamK type.

  8. Manage stream state using adaptState to avoid concurrency issues

    master

    Streamly uses state-passing through its APIs, which requires careful handling of the state object to ensure correct behavior, especially when dealing with SVar sharing.

    When passing state to run a stream, follow these rules:

    1. Concurrent Streams: If you are building a concurrent stream that needs to share the same SVar, pass the incoming state as is.
    2. All other cases: You must not share the SVar. Every time you pass the state to run a stream, you must use adaptState to reset the SVar in the state.

    Best Practice: If you are unsure, always use adaptState on the state before passing it on. This ensures correct behavior, though it may result in a loss of concurrency at most.

    Note: There is currently no type-level enforcement for this pattern, so developers must manually ensure state is handled correctly. All transform operations should be verified against existing state-related tests.

  9. Use Streamly for concurrent and reactive programming

    master

    All Streamly abstractions are designed for native concurrent evaluation. You can compose serial processing pipelines with concurrent ones using the same patterns.

    Concurrency Patterns:

    • Map functions over a stream concurrently.
    • Generate multiple streams in parallel and merge them.
    • Split streams, scan branches concurrently, and combine results.
    • Distribute a stream to different concurrent folds.
    • Perform time-based operations: sampling, throttling, debouncing, or periodic actions.

    Reactive Programming: Streamly supports the functional reactive programming (FRP) model by treating events as streams. You can generate streams of events, merge them, and process them concurrently using time-specific and time-based sampling combinators found in Streamly.Data.Stream.Prelude.

  10. Core capabilities of the Streamly library

    master

    Streamly extends Haskell's base library with unified, streaming-capable APIs designed for high-performance and high-composability tasks. The library provides built-in concurrency across its APIs and includes support for:

    • Data Processing: Streams, folds, and parsers.
    • Storage: Arrays for random-access storage with streaming APIs.
    • Serialization: Binary serialization and deserialization.
    • I/O: Streaming file and directory I/O, and streaming Unicode text processing.
    • Networking: Streaming network APIs.
    • FRP: Time-based streaming APIs for Functional Reactive Programming.

  11. Understand traversal depth in directory IO

    master

    In Streamly's directory traversal, depth is defined as the number of recursive traversal steps from the starting traversal root.

    • Symlink Resolution: A symlink resolution counts as exactly one step, regardless of how many levels of indirection it involves.
    • Calculation: If descendant paths are constructed by appending child names returned by readdir (and readdir does not return . or ..), the traversal depth is identical to the lexical path depth relative to the root. This can be computed statelessly by stripping the traversal root prefix and counting the remaining path segments.

  12. Choose the right Stream type

    master

    Streamly provides different stream representations depending on whether you need static fusion or dynamic composition:

    • Stream m a: Statically fused, composable source streams. Best for high-performance pipelines where the structure is known at compile time. (Module: Streamly.Data.Stream)
    • Unfold m a b: Statically fused streams designed for nested fusion. (Module: Streamly.Data.Unfold)
    • StreamK m a: CPS-based source streams. Use these when you need dynamic composition (e.g., building streams at runtime based on logic). (Module: Streamly.Data.StreamK)