Prophecy Documentation

Prophecy Professional Edition is a managed, AI-first analytics environment designed for data analyst teams to perform AI-driven analysis without managing data infrastructure. It uses a conversational workflow where an Agent generates pipelines and analyses based on user prompts. The environment runs on Prophecy-managed DuckDB, removing the need for users to configure fabrics, connections, or secrets.

Data lineage provides a visualization of the complete journey of data from source to destination, including all transformations. It is available for Enterprise Edition and supports Python/PySpark and Scala projects. Lineage is computed on-demand directly from Git code, allowing users to see how changes in branches affect lineage before committing.

The WindowFunction gem allows you to perform calculations for individual records across a defined window of rows. Unlike standard aggregations, window functions retain the original row structure while adding new columns based on neighboring rows. Common use cases include computing rolling averages, ranking rows within groups, comparing values between previous/next rows (lead/lag), and calculating running totals.

The SpatialMatch gem is used to find relationships between geometries (points, lines, and polygons) from two different datasets using spatial joins. It requires geometries to be in Well-Known Text (WKT) format. Common use cases include matching stores to delivery zones, identifying roads intersecting flood zones, or matching customers to service areas.

Prophecy Studio is a built-in IDE used for building data pipelines. It features a visual canvas for designing data flows using 'gems', a project browser for managing entities, and a code view for inspecting the compiled logic of visual pipelines.

Spark Copilot is an AI-powered assistant available in the Enterprise Edition of Prophecy. It provides suggestions within both the visual editor and code editor to help develop data pipelines and models. It fetches context exclusively from environment metadata and does not use knowledge graphs.

Pipelines are visual workflows used for data preparation and transforming warehouse data. They consist of a sequence of 'gems' that collect, transform, and move data from sources to storage or analytics systems. Pipelines compile to warehouse-native SQL, meaning all execution occurs within the user's warehouse using existing credentials and access controls. In the Prophecy model, Pipelines handle data preparation, while Analyses handle exploration and visualization.

A parameter set is a named collection of values for multiple parameters, allowing developers to group related settings that must change in sync (e.g., environment-specific settings like dev/prod, regional configurations, or tenant-specific values). Parameter sets can be created at both the project level and the pipeline level.

The FuzzyMatch gem is used to identify non-identical duplicates within a dataset. It compares records based on specified fields and a similarity threshold to find closely matching entries that are not exactly identical.

The Join gem allows you to join two or more DataFrames based on a given configuration. It supports various join types and Spark join hints to optimize performance.

Execution metrics allow developers to monitor and analyze Spark pipeline performance by tracking records processed, bytes processed, pipeline execution time, and data samples (interims) between components. These metrics are automatically collected during pipeline runs and stored in the user's data environment for later review via the Prophecy interface. This feature is available exclusively for Enterprise Edition and Spark pipelines.