DOCSEARCHbyAlgolia
Submit Docs

Deephaven Community Core

repository·main·Indexed 43 days ago

https://github.com/deephaven/deephaven-core

A real-time, time-series, column-oriented analytics engine designed for high-performance data processing and visualization of historical and streaming data. Includes documentation on configuring authentication handlers such as Mutual TLS (mTLS), OpenID Connect (OIDC) via Keycloak, Pre-Shared Key (PSK), and PostgreSQL username/password authentication.

Tokens
602.5K
Snippets
1.4K
Records
2K
Agent score
50%

What's inside Deephaven

  1. What is deephaven.ui?

    main

    deephaven.ui is a React-like declarative framework for building interactive user interfaces entirely in Python. It allows you to create complex UIs without needing JavaScript or CSS knowledge.

    Key features include:

    • Components: Reusable building blocks defined in Python.
    • Live Dataframe Integration: Components are aware of Deephaven tables and update automatically when the underlying data changes (reactive UI).
    • Composability: Build complex interfaces by combining simple components.
    • React Spectrum Parity: It exposes Adobe's React Spectrum design system components. If you are familiar with React Spectrum, note that deephaven.ui uses snake_case for props instead of camelCase (e.g., on_press instead of onPress).

  3. Overview of the `deephaven` Python package submodules

    main

    The deephaven package is the primary interface for interacting with Deephaven tables and objects using Python. It is organized into several submodules, each providing specific functionality:

    Data Ingestion & Export

    • csv: Read/write CSV files.
    • parquet: Read/write Parquet files.
    • dbc: Connect to and execute queries on external databases (SQL).
    • stream: Integration with Apache Kafka and Table Publisher.
    • uri: Resolve URIs into objects.

    Table Manipulation & Creation

    • table: Core implementation of Deephaven tables and partitioned tables.
    • table_factory: Methods to create various table types (empty, input, function-generated, time, ring, etc.).
    • column: Implementation of columns, used primarily when creating new tables.
    • updateby: Tools for building cumulative and rolling aggregations for update_by operations.
    • agg: Combined aggregations usable in queries.
    • filters: Various filters for filtering operations.
    • table_listener: Implementation of Deephaven table listeners.

    Data Science & Interoperability

    • pandas: Convert between Deephaven tables and Pandas DataFrames.
    • pandasplugin: Display Pandas DataFrames as Deephaven tables in the UI.
    • numpy: Convert between Deephaven tables and NumPy arrays.
    • arrow: Convert between Deephaven tables and PyArrow tables.
    • learn: Framework for using machine learning libraries in Deephaven queries via efficient data transfer.

    System & Advanced Control

    • appmode: Support for writing Deephaven application mode Python scripts.
    • barrage: Access resources on remote Deephaven servers.
    • execution_context: Manage Deephaven query execution context on threads.
    • liveness_scope: Control cleanup of unreferenced nodes in the query update graph.
    • perfmon: Tools to analyze Deephaven system and query performance.
    • plugin: Framework for creating and using plugins.
    • server: Run Python operations from within a JVM.
    • update_graph: Access locks in the update graph for specific table operations.
    • jcompat: Java compatibility support for creating Java data structures from Python.
    • query_library: Import Java classes/packages into the query library for use in queries.

    Utilities

    • calendar: Working with business calendars.
    • constants: Global constants (e.g., NaN, null, infinity, max/min values).
    • dtypes: Supported Deephaven engine data types.
    • html: Export data in HTML format.
    • plot: Built-in plotting API.
    • replay: Support for replaying historical data.
    • time: Handling date-time data.
    • dherror: Custom exceptions for the package.
    • experimental: Experimental features (e.g., outer joins, AWS S3 support).

  4. Overview of Kafka integration in Deephaven

    main

    Deephaven provides a suite of tools designed to simplify Kafka integration. It allows developers to connect to Kafka streams, consume messages, and process data in real-time.

    Key benefits of using Deephaven with Kafka include:

    • Unified Abstractions: Deephaven uses common abstractions for both event streams and static tables, allowing live and historical data to share a common vocabulary.
    • Unified Operations: A single table operations library can be used for both streaming and historical data.
    • Fast R&D Cycles: The engine is designed to bridge the gap between data exploration, model fitting, backtesting, and deployment, which is critical for environments where trends change quickly (e.g., quantitative finance).
    • Language Interoperability: Code can run in-process within the data engine or externally via client APIs using a developer's language of choice.

  5. Overview of Deephaven Community Core

    main
    Deephaven Community Core is a real-time, time-series, column-oriented analytics engine with relational database features. It allows for seamless queries across both historical and real-time data, providing tools for data ingestion, computation, analysis, and visualization. It is the open version of Deephaven Enterprise.

  6. Overview of the Deephaven C++ Client

    main

    The Deephaven C++ client is a native library designed to interface with a Deephaven Core server. It enables C++ applications to:

    • Create and query tables on the server.
    • Pull table data into local memory as Apache Arrow tables.
    • Push local data up to the server.
    • Subscribe to ticking tables and receive real-time update streams.

    The client communicates using gRPC, Apache Arrow Flight, and the Barrage protocol for ticking data. It is composed of two primary libraries:

    • dhcore: Contains the client-side data model (tables, columns, chunks, row sets, types) and the ticking/Barrage state machine. It is independent of Arrow and gRPC.
    • dhclient: The primary user-facing API (including Client, TableHandleManager, and TableHandle), built on top of dhcore with Arrow Flight, gRPC, and protobuf support.

  7. Overview of the Deephaven Python Integration API

    main

    The Deephaven Python Integration Package, created by Deephaven Data Labs, enables Python developers and data scientists to interact with Deephaven data servers. It allows you to access data, run queries, and execute Python scripts directly inside the server for maximum performance.

    Key capabilities include:

    • Real-time streaming: Leveraging Deephaven's unique streaming table capabilities.
    • Data Ingestion: Connecting to various sources such as Kafka, Parquet, CSV, and SQL.
    • High-performance pipelines: Building real-time data processing pipelines that are easy to consume.

  8. Overview of deephaven.ui

    main

    deephaven.ui is a Python plugin for Deephaven designed to build real-time, data-focused browser dashboards. It allows developers to create interactive applications entirely in Python, eliminating the need for JavaScript or CSS.

    Key characteristics include:

    • Live Dataframe Awareness: Components can use Deephaven tables directly as data sources.
    • Continuous Streaming: Unlike frameworks that rely on periodic refreshes, deephaven.ui components are driven by continuous streaming data.
    • Reactive & Declarative: The UI automatically updates when underlying Python data changes, allowing you to describe the UI as a function of the data.

  9. Select and create columns in Deephaven

    main

    Deephaven provides five primary table operations to select, create, or modify columns. These operations work on real-time (updating), static, and snapshotted tables. The choice between them depends on whether you want to keep original columns and whether you want to compute values immediately (writing to memory) or on-demand (calculating when accessed).

    Available Operations

    • select: Returns only the columns specified in the formulas.
    • view: Returns only the columns specified in the formulas and calculates them on-demand.
    • update: Returns all original columns plus the new columns, writing results to memory immediately.
    • update_view: Returns all original columns plus the new columns, but calculates them on-demand.
    • lazy_update: Returns all original columns plus the new columns, calculating them on-demand with a memoization cache.
    from deephaven import new_table
from deephaven.column import int_col

source = new_table([int_col("Column_1", [1, 1, 2, 3, 4])]).update(
    formulas=["Column_2 = i*2"]
)

table_select = source.select(
    formulas=["Column_1", "Column_2", "New_Column_A = Column_1 + Column_2^2"]
)
table_view = source.view(
    formulas=["Column_1", "Column_2", "New_Column_A = Column_1 + Column_2^2"]
)

table_update = source.update(formulas=["New_Column_A = Column_1 + Column_2^2"])
table_update_view = source.update_view(["New_Column_A = Column_1 + Column_2^2"])

table_lazy_update = source.lazy_update(["New_Column_A = Column_1 + Column_2^2"])

  10. Connecting to a Deephaven Flight SQL server

    main

    Deephaven provides a built-in Flight SQL server that exposes Deephaven tables from the global query scope as named tables. This allows you to query tables using standard SQL via the Flight SQL protocol.

    Key Constraints:

    • Supported Operations: Only SELECT queries are supported. INSERT, UPDATE, and DELETE are not currently supported.
    • Data Model: Flight SQL APIs are not Live Dataframe APIs. If you require real-time updates, use Deephaven-native clients instead.
    • Authentication: When using clients that require cookies for authentication, you must set the x-deephaven-auth-cookie-request header to true.

  11. Navigate the Deephaven IDE interface

    main

    The Deephaven IDE consists of several key areas for real-time data analysis:

    • Console: Located on the left side, this is where you write and execute your code.
    • Workspace: Located on the right side, this area displays results such as tables and plots as interactive panels.
    • Tables Dropdown: Located at the top of the interface, this provides a list of all tables currently in your session.

  12. What the Deephaven R Client can do

    main

    The Deephaven Core R client is an R package that allows users to interface with a Deephaven server from RStudio or any other R interface.

    Key functionalities include:

    • Server Connection: Supports anonymous, basic (username/password), pre-shared key, and custom (key-value) authentication.
    • Script Execution: Run Python or Groovy scripts on the server if a console is equipped.
    • Table API Integration:
      • Create static and ticking tables on the server.
      • Construct and execute complex queries.
      • Retrieve table references.
      • Pull data into R formats: Arrow RecordBatchReader, Arrow Table, dplyr Tibble, or standard R Data Frame.
      • Create new server-side tables from R data structures (Arrow RecordBatchReader, Arrow Table, dplyr Tibble, or R Data Frame).
      • Call Deephaven table methods using familiar R functions.