repository·main·Indexed 43 days ago
https://github.com/snowflakedb/snowpark-pythonIntuitive Python APIs for querying and processing data directly within Snowflake. It features a native Snowpark Python API and a Snowpark pandas API (via Modin), enabling data pipeline development without moving data out of the Snowflake environment.
The Snowpark pandas API provides a pandas-compatible interface for performing distributed data processing on Snowflake. It allows users to use familiar pandas syntax (Series, DataFrames, indexing, etc.) while the underlying execution is offloaded to Snowflake's engine.
Key functional areas covered by the API include:
Series and DataFrame objects.groupby, resampling, window functions, and indexing.general_functions and numpy compatibility.hybrid_execution and performance optimizations.The snowflake.snowpark.secrets module provides functions and classes to retrieve sensitive information (secrets) stored in Snowflake. This allows you to securely access credentials like usernames, passwords, OAuth tokens, or cloud provider tokens within your Snowpark code without hardcoding them.
Available classes:
UsernamePassword: Represents a username and password credential pair.CloudProviderToken: Represents a token for a cloud provider.Available functions:
get_generic_secret_string: Retrieves a generic secret string.get_oauth_access_token: Retrieves an OAuth access token.get_secret_type: Determines the type of a secret.get_username_password: Retrieves username and password credentials.get_cloud_provider_token: Retrieves a cloud provider token.get_wif_token: Retrieves a Workload Identity Federation (WIF) token.The snowflake.snowpark.functions module provides a comprehensive suite of functions that can be used to manipulate Snowpark DataFrames. These functions are designed to mirror Snowflake SQL functions, allowing you to perform complex data transformations, aggregations, and analytical operations directly on your data within Snowflake.
Common categories of functions available include:
ai_complete, ai_summarize_agg, and ai_sentiment.array_construct, parse_json, and get.st_ prefixed functions like st_distance and st_intersects.abs, avg, stddev, and corr.date_trunc, datediff, and current_timestamp.rank, row_number, lead, and lag.concat, substring, and regexp_replace.Snowpark pandas (via Modin) supports a variety of general-purpose data manipulation and utility functions. These functions are categorized into data manipulation, missing data handling, numeric processing, datetime processing, and hybrid execution control.
### Data manipulations
- `melt`
- `crosstab`
- `pivot`
- `pivot_table`
- `cut`
- `qcut`
- `concat`
- `get_dummies`
- `merge`
- `merge_asof`
- `unique`
### Top-level missing data
- `isna`
- `isnull`
- `notna`
- `notnull`
### Top-level dealing with numeric data
- `to_numeric`
### Top-level dealing with datetimelike data
- `date_range`
- `bdate_range`
- `to_datetime`
### Hybrid Execution functions
- `explain_switch`The modin.pandas.groupby module provides a pandas-compatible API for grouping data. The supported operations are categorized into indexing/iteration, function application, and descriptive statistics for both DataFrameGroupBy and SeriesGroupBy objects.
For a complete list of specific method signatures and behaviors, refer to the individual API documentation for each method.
Snowpark pandas provides a subset of the pandas API implemented for Snowflake. Because new methods are merged into release branches continuously, the list of supported APIs is subject to change. To ensure you are using the correct documentation for your specific installation, always verify that you are viewing the documentation for the stable version that matches your installed library version.
The supported API surface is organized into several categories:
.dt and .str)snowflake.snowpark.catalog.Catalog module provides a set of methods to inspect, manage, and manipulate the Snowflake catalog. You can use it to check for the existence of objects (databases, schemas, tables, views, procedures, and UDFs), list metadata, and perform destructive operations like dropping objects. Most methods follow a snake_case naming convention (e.g., list_tables) but also support CamelCase aliases (e.g., listTables).When using Snowpark pandas, window function support is indicated by specific flags. Note that engine and engine_kwargs are always ignored in Snowpark pandas; the execution engine is always Snowflake.
Implementation Flags:
Y: Supported via distributed implementation.N: Not supported; the API will error out.P: Partial support; some parameters may not be supported.D: Defaults to single-node pandas execution via UDF/Sproc.The snowflake.snowpark.Table class represents a Snowflake table. It provides methods to perform DML (Data Manipulation Language) and DDL (Data Definition Language) operations directly on the table, such as deleting rows, updating rows, merging data, or dropping the table entirely.
from snowflake.snowpark import Table
# Assuming 'session' is an active Snowpark session
table = session.table("MY_TABLE_NAME")The CaseExpr class allows you to build CASE expressions in SQL using a fluent API. You can chain .when() calls to define multiple conditions and use .otherwise() to define the fallback value when no conditions are met.
# Example pattern for CaseExpr
from snowflake.snowpark.functions import col
# Logic: CASE WHEN condition1 THEN result1 WHEN condition2 THEN result2 ELSE result_else END
expr = col("MY_COL").when(col("MY_COL") > 10, "High").otherwise("Low")The Row class in snowflake.snowpark represents a single record from a Snowpark DataFrame. It allows you to access individual field values by name or index. Common operations include converting the row to a dictionary or retrieving its index.
from snowflake.snowpark import Row
# Example of creating a Row manually
row = Row({'col1': 1, 'col2': 'A'})
print(row['col1']) # Access by name
print(row[1]) # Access by index