DOCSEARCHbyAlgolia
Submit Docs

pyjanitor Documentation

repository·dev·Indexed 23 days ago

https://github.com/pyjanitor-devs/pyjanitor

A Python implementation of the R package 'janitor', providing a verb-based API to extend pandas with data-cleaning functions designed for method chaining. Features include tools for adding columns, converting currencies, filtering by date, truncating column names, and converting currency strings to numeric floats.

Tokens
11.8K
Snippets
40
Records
68
Agent score
80%

What's inside pyjanitor

  1. Explore Biology-related cleaning functions

    dev
    The janitor.biology module provides a collection of specialized cleaning and transformation functions designed for biological datasets. These functions are part of the pyjanitor ecosystem and are intended to streamline common data manipulation tasks encountered in bioinformatics and biological research.

  3. Use timeseries cleaning functions

    dev
    The janitor.timeseries module provides a suite of cleaning and transformation functions specifically designed for working with time-series data in pandas DataFrames. These functions allow you to manipulate temporal indices, handle frequency conversions, and clean time-related columns efficiently.

  4. Use pyjanitor with Polars

    dev
    pyjanitor provides a suite of cleaning and transformation functions specifically for Polars DataFrames. These functions are available under the janitor.polars namespace and allow you to perform common data cleaning tasks such as cleaning column names, handling missing values, and reshaping data.

  5. How pyjanitor's release system works

    dev

    pyjanitor uses a continuous delivery approach where release types are determined by how code is merged:

    • Patch releases: These are fully automated. Every successful merge to the dev branch triggers a patch release sequence.
    • Minor and Major releases: These are manual. Maintainers must explicitly trigger these when changes warrant a larger version bump.

    Automatic Release Sequence (Patch)

    When code is merged to dev, the following happens automatically if tests pass:

    1. Tests run: The pyjanitor tests workflow executes.
    2. Trigger: On success, the Auto-release workflow starts.
    3. Duplicate check: The system verifies if the commit already has a release tag to prevent infinite loops.
    4. Version bump: uv version --bump patch updates pyproject.toml.
    5. Release notes: llamabot CLI generates notes using an LLM.
    6. Publish: The package is built and published to PyPI via trusted publishing.
    7. GitHub Release: A GitHub release is created with the notes.

  6. How Pixi features and environments work for multi-Python testing

    dev

    The pyjanitor project uses Pixi to manage multiple Python versions for testing. This is achieved through two main abstractions: Features and Environments.

    1. Pixi Features

    Features are reusable sets of dependencies. In pyproject.toml, features are defined to pin specific Python versions using semantic versioning with wildcards (e.g., 3.11.*).

    2. Pixi Environments

    Environments are named configurations that combine multiple features. An environment typically combines:

    • Python version features (e.g., py311)
    • Testing features (e.g., tests containing pytest, hypothesis, etc.)
    • Setup features (e.g., setup for pre-commit hooks)

    When an environment is created, Pixi resolves all dependencies from the combined features to create an isolated environment with the exact Python version specified.

    [tool.pixi.feature.py311.dependencies]
python = "3.11.*"

[tool.pixi.environments]
py311 = { features = ["tests", "setup", "py311"] }

  7. How pyjanitor's API works

    dev

    The pyjanitor API is designed to enable clean, readable method chaining for data cleaning. There are three primary ways to use the library:

    1. Native Pandas Extension (Recommended): By importing janitor, its functions are automatically registered as methods on pandas.DataFrame objects. This allows you to call them directly in a chain.
    2. Functional API: Import specific functions from janitor and pass the DataFrame as the first argument.
    3. Pandas .pipe(): Use the standard pandas .pipe() method to pass the DataFrame through janitor functions.

    pyjanitor aims to provide 'explicitly named verbs' (e.g., remove_columns instead of drop) to make the data processing Directed Acyclic Graph (DAG) easier to read.

    import pandas as pd
import janitor

# 1. Native Extension (Recommended)
df = pd.DataFrame.from_dict(company_sales).clean_names().remove_empty()

# 2. Functional API
from janitor import clean_names, remove_empty
df = clean_names(pd.DataFrame.from_dict(company_sales))
df = remove_empty(df)

# 3. Using .pipe()
from janitor import clean_names, remove_empty
df = (
    pd.DataFrame.from_dict(company_sales)
    .pipe(clean_names)
    .pipe(remove_empty)
)

  8. Run tests and verify setup

    dev

    Use pixi run to execute the test suite. You can run the full suite or filter for faster tests to speed up your development cycle.

    • Run all tests:
      pixi run test
    • Run only fast tests (excludes slow "turtle" tests):
      pixi run pytest -m "not turtle"
    • Run tests for a specific module:
      pixi run pytest tests/functions/test_clean_names.py
    • Run a specific test function:
      pixi run pytest tests/functions/test_clean_names.py::test_clean_names_basic

    Note on Optional Dependencies: Tests for chemistry.py, biology.py, and spark.py are automatically skipped locally if required dependencies (like rdkit or pyspark) are not installed. These are always run in CI.

    pixi run test
pixi run pytest -m "not turtle"

  9. Check code style and formatting

    dev

    Before committing, you should manually run linting and formatting checks to ensure compliance with project standards. While pre-commit hooks run automatically during git commit, you can trigger them manually using pixi.

    • Run linting checks:
      pixi run lint
    • Run all checks (tests, docs, linting, and formatting):
      pixi run check
    pixi run lint
pixi run check

  10. Benchmark import performance for pyjanitor

    dev

    To measure how long import janitor takes and identify which sub-imports are causing delays, you can use Python's built-in -X importtime flag combined with the tuna visualization tool.

    1. Run the import command and redirect the timing output to a log file: python -X importtime -c "import janitor" 2> timing.log

    2. Visualize the log using tuna. If you don't have it, install it via pip install -U tuna. tuna timing.log

    This will open a web UI in your browser showing a visual breakdown of the import timings.

    python -X importtime -c "import janitor" 2> timing.log
tuna timing.log

  11. Trigger a manual minor or major release

    dev

    To release a version that is not a patch (e.g., adding new features or breaking changes), follow these steps in GitHub:

    1. Navigate to the Actions tab in the repository.
    2. Select Auto-release from the list of workflows on the left.
    3. Click the Run workflow dropdown.
    4. Select the desired version bump type: major, minor, or patch.
    5. Click the green Run workflow button.

    When to use each version type:

    • minor: Use for new features that are backward-compatible, significant enhancements, or new optional dependencies.
    • major: Use for breaking changes to the public API, removal of deprecated features, or fundamental architectural changes.

  12. Add a new Python version to the testing suite

    dev

    To extend testing to a new Python version (e.g., Python 3.14), follow these three steps:

    1. Define a new feature in pyproject.toml to pin the Python version:

      [tool.pixi.feature.py314.dependencies]
python = "3.14.*"

    2. Define a new environment in pyproject.toml that combines the new feature with existing tests and setup features:

      py314 = { features = ["tests", "setup", "py314"] }

    3. Update the GitHub Actions matrix in your workflow YAML to include the new environment name:

      matrix:
  environment: [py311, py312, py313, py314]
    [tool.pixi.feature.py314.dependencies]
python = "3.14.*"

[tool.pixi.environments]
py314 = { features = ["tests", "setup", "py314"] }
    matrix:
  environment: [py311, py312, py313, py314]