DOCSEARCHbyAlgolia
Submit Docs

siuba

repository·main·Indexed 22 days ago

https://github.com/machow/siuba

A Python library that ports R's dplyr and tidyverse functionality to Python. It enables data analysis workflows across pandas DataFrames and SQL databases (postgres, redshift, sqlite) using a pipe-based syntax (>>), verbs like filter(), mutate(), and summarize(), and siu expressions (siuba._) for optimized execution.

Tokens
20.2K
Snippets
78
Records
105
Agent score
79%

What's inside siuba

  1. Overview of Siuba

    main
    Siuba is a Python library designed for quick, scrappy data analysis. It serves as a Python port of the R Tidyverse ecosystem, specifically implementing the APIs and philosophies of dplyr, tidyr, and other related R libraries. It allows users to perform data manipulation using a syntax familiar to R users while working within the Python ecosystem (typically powered by pandas).

  2. Use siu expressions instead of lambda functions

    main

    A siu expression (using siuba._) is a shorthand for a lambda function. It specifies what action to perform, allowing siuba to optimize the execution based on the data source (local DataFrame vs. remote SQL table).

    Instead of using a lambda:

    mtcars[lambda _: _.cyl == 4]

    You can use a siu expression:

    mtcars[_.cyl == 4]
    from siuba import _

# siu expression approach
mtcars[_.cyl == 4]

  3. Understand the Call abstraction in siuba.siu.calls

    main

    The siuba.siu.calls.Call class is the base class for all call expressions in the siu (siuba internal unit) system. It represents an operation or a function call that is captured as an expression rather than being executed immediately. This allows siuba to translate these expressions into other languages, such as SQL or specialized Python code, during evaluation.

    Key methods available on Call objects include:

    • copy(): Creates a copy of the call.
    • map_subcalls(): Allows for traversing and transforming sub-calls within the expression.
    • op_vars(): Retrieves the variables involved in the operation.

  4. Core concepts of siuba: Verbs, siu expressions, and pipes

    main

    Siuba's workflow is built on three fundamental concepts:

    1. Verbs: Functions that operate on a table (e.g., group_by(), summarize(), filter(), select(), mutate(), arrange()).
    2. Siu expressions: Expressions created using siuba._ that represent the actions you want to perform on columns. These allow siuba to decide whether to execute the action locally on a DataFrame or translate it into SQL for a remote database.
    3. Pipes: The >> operator used to chain verbs together, allowing for a readable, sequential data processing pipeline.

    Common verbs include:

    • select(): Keep certain columns.
    • filter(): Keep certain rows.
    • mutate(): Create or modify columns.
    • summarize(): Reduce columns to single values.
    • arrange(): Reorder rows.
    • group_by(): Group rows before applying verbs.
    • distinct(), count(), and joins: SQL-like operations.
    from siuba import group_by, summarize, _
from siuba.data import mtcars

(mtcars
  >> group_by(_.cyl)
  >> summarize(avg_hp = _.hp.mean())
  )

  5. Use Symbolic expressions in siuba

    main
    The siuba.siu.symbolic module provides the Symbolic class and the strip_symbolic function. These are used to handle symbolic representations of column names and expressions, allowing for the syntax used in siuba's DSL (Domain Specific Language) where column names can be treated as objects in expressions (e.g., _.column_name == value).

  6. Run analysis on SQL databases using siuba

    main

    Siuba allows you to run the same analysis code on a local pandas DataFrame or a remote SQL database (supporting postgres, redshift, or sqlite). To use a SQL table, wrap your SQLAlchemy engine and table name using the tbl() function.

    Example workflow:

    1. Create a SQLAlchemy engine.
    2. Use tbl(engine, "table_name") to create a siuba table object.
    3. Chain verbs using the >> operator.

    Note: When querying SQL, the output is a 'lazy query' that provides a preview of the results.

    from sqlalchemy import create_engine
from siuba import _, tbl, group_by, summarize
from siuba.data import mtcars

# 1. Setup engine and data
engine = create_engine("sqlite:///:memory:")
mtcars.to_sql("mtcars", engine, if_exists = "replace")

# 2. Connect with siuba
tbl_mtcars = tbl(engine, "mtcars")

# 3. Run analysis
(tbl_mtcars
  >> group_by(_.cyl)
  >> summarize(avg_hp = _.hp.mean())
  )

  8. Install siuba extra packages

    main

    siuba provides additional functionality through specialized submodules that extend its core capabilities. Depending on your needs, you can use functions from the following extra modules:

    • siuba.dply.vector: Provides missing vector functions (e.g., n()).
    • siuba.dply.forcats: Provides functions inspired by R's forcats library for handling categorical variables.
    • siuba.experimental.datetime: Provides functions for working with dates and times.

  9. How the SQL backend works

    main

    The SQL implementation is built on three main components:

    1. LazyTbl: A class that maintains a SQLAlchemy connection, the target table name, and a list of pending select statements.
    2. Verbs: Functions (like mutate) that dispatch on LazyTbl. Instead of executing immediately, they return a new LazyTbl containing the additional select statement.
    3. CallListeners: Components responsible for:
      • Translating siuba lazy expressions into SQL-specific functions.
      • Adding grouping information to SQL OVER clauses.

  10. Understand symbolic call rules for `_`

    main

    In siuba, the _ object is used to create symbolic expressions (siu expressions). Understanding how it behaves during calls is critical for writing correct expressions:

    1. Symbolic Call: When _() represents a call rather than executing one, it is a symbolic call. This happens by default.
    2. Automatic Non-Symbolic Behavior: _ performs a normal (non-symbolic) call immediately after:
      • A binary operation (e.g., _ + _)
      • A symbolic call (e.g., _())
      • An index operation (e.g., _['a'])
    3. Explicit Escaping: You can force _ to perform a normal call (instead of a symbolic one) using the ~~ operator.

    Common Patterns:

    • Binary Operation: _ + _ (No escaping needed)
    • Method Call: _.upper() (Symbolic call)
    • Index: _['a'] (No escaping needed)

  11. Chain operations using the pipe operator (>>)

    main

    Siuba supports a pipe operator >> to chain operations together, making code more readable and following the dplyr pattern. You can pipe a DataFrame into functions or use the Pipeable class to compose custom functions.

    # Chaining DataFrame operations
(df
  >> mutate(
       new_repo = _.repo + " waattt",
       case = case_when(_, {_.language == "python": "aw yeah", True: 'wat'})
     )
  >> filter(_.stars > 5000)
)

# Simple pipe
df >> group_by(_.language) >> summarize(wat = _.stars.mean())

  12. Regroup using a Grouper object

    main

    You can pass an existing Grouper object back into a groupby method. This is a key mechanism used by siuba to allow regrouping transformations and composing operations.

    # Assuming g_cyl is a GroupBy object
g_cyl2 = g_cyl.obj.groupby(g_cyl.grouper)

# The grouper remains the same
g_cyl.grouper is g_cyl2.grouper
    g_cyl2 = g_cyl.obj.groupby(g_cyl.grouper)

g_cyl.grouper is g_cyl2.grouper