DOCSEARCHbyAlgolia
Submit Docs

benford_py

repository·master·Indexed 41 days ago

https://github.com/milcent/benford_py

A Python library for performing Benford's Law tests on numerical datasets to detect anomalies, errors, or fraud. It provides tools to analyze the distribution of leading digits using the Benford class and functions like first_digits(), second_digit(), and last_two_digits(). The library integrates with numpy, pandas, and matplotlib, supporting statistical evaluations including Z-score, Mean Absolute Deviation (MAD), Chi-Square, and Kolmogorov-Smirnov tests.

Tokens
1.9K
Snippets
7
Records
10
Agent score
99%

What's inside benford_py

  1. Overview of Benford's Law tests in Python

    master

    benford_py is a Python implementation of Benford's Law tests. It is designed to help users identify inconsistencies, errors, or potential fraud in numerical datasets by checking if the distribution of first digits follows the expected logarithmic pattern.

    Key features:

    • Uses numpy and pandas for data handling.
    • Uses matplotlib for visualization of results.
    • Supports automated scripting and interactive use.
    • Compatible with Python versions >= 3.6 on Linux, Windows, and Mac.

  2. Use the benford package modules

    master

    The benford package is organized into several specialized modules for performing Benford's Law analysis, calculating expected distributions, performing statistical tests, and visualizing results:

    • benford.benford: Core functionality for Benford's Law analysis.
    • benford.expected: Tools for generating or managing expected digit distributions.
    • benford.stats: Statistical testing modules to evaluate how well data conforms to Benford's Law.
    • benford.viz: Visualization tools for plotting digit distributions and comparisons.

  3. Install benford_py

    master

    You can install benford_py using pip from PyPI. The package is available under both benford_py and benford-py names.

    Alternatively, you can clone the repository directly into your Python environment's site-packages folder using git.

    pip install benford_py
# or
pip install benford-py

# Or via git
git clone https://github.com/milcent/benford_py

  4. Quick start with the Demo notebook

    master
    For a practical example of how to use the library, refer to the Demo.ipynb notebook in the repository. This notebook demonstrates how to run Benford's Law tests using SPY (S&P 500 ETF) daily returns.

  5. Use the `Benford` class for comprehensive analysis

    master

    The Benford class is a high-level wrapper that ingests data and automatically runs a suite of tests (First Digit, Second Digit, First Two, First Three, and Last Two Digits) along with their statistics.

    Initialization: You can initialize it with a pandas Series/numpy array, or a tuple containing a DataFrame and the column name: bf.Benford((df, 'column_name'), decimals=...).

    Key Attributes:

    • .tests: A list of all tests performed.
    • .data: The raw input data.
    • .base: The pre-processed data containing sequences (Seq), transformed values (ZN), and digit-specific records (F1D, F2D, F3D, SD, L2D).
    • .all_confidences: Returns the current confidence levels for all tests.

    Key Methods:

    • .report(): Called on a specific test attribute (e.g., benf.F1D.report()) to get detailed results.
    • .update_confidence(level, tests=None): Updates the confidence level for all tests (or a specific subset) without re-processing the data. This is useful for addressing the 'Power Problem' in large samples.
    • .summation(): Runs Summation Tests.
    • .sec_order(): Runs Second Order Tests.
    # Initialize with a DataFrame and column name
benf = bf.Benford((sp, 'l_r'), decimals=8, limit_N=1800)

# Get a report for the First Two Digits test
print(benf.F2D.report())

# Update confidence for specific tests
benf.update_confidence(99.999, tests=['F2D', 'L2D'])

  7. Run Second Digit and Last Two Digits Tests

    master

    For specific digit positions other than the leading digits, use the dedicated functions:

    • bf.second_digit(...): Analyzes the second digit of the numbers.
    • bf.last_two_digits(...): Analyzes the last two digits.

    These functions share the same parameter set as first_digits (including digs, decimals, confidence, and high_Z).

    # Second Digit Test
sd = bf.second_digit(data_series, decimals=8, confidence=85, high_Z='all')

# Last Two Digits Test
l2d = bf.last_two_digits(data_series, decimals=8, confidence=90, high_Z='neg')

  8. Run First Digits Tests with `bf.first_digits()`

    master

    The first_digits function analyzes a pandas Series or a numpy 1D array to check for Benford's Law compliance.

    Key Arguments:

    • digs: Specifies the test type. 1 for First Digit, 2 for First Two Digits, and 3 for First Three Digits.
    • decimals: The number of decimal places to consider during pre-processing. Defaults to 2. For very small numbers (like log returns), you should increase this (e.g., 8) to avoid division-by-zero errors. For integers, use 0. You can also use 'infer' to treat registries separately.
    • confidence: Sets the confidence level for Z-score statistical testing. Supported values: 80, 85, 90, 95, 99, 99.9, 99.99, 99.999, 99.9999, 99.99999.
    • sign: Filters data by sign. 'pos' (positive), 'neg' (negative), or 'all' (all non-zero entries). Defaults to 'all'.
    • verbose: If True, prints sample size and discarded records. Defaults to True.
    • high_Z: Controls which deviations are displayed when confidence is set. 'pos' (only positive deviations), 'neg' (only negative), 'all' (both), or an int (returns first $n$ entries).
    • show_plot: Whether to draw the test plot. Defaults to True.
    • MAD: If True, calculates the Mean Absolute Deviation.
    • chi_square: If True, computes the Chi-Square test.
    • KS: If True, computes the Kolmogorov-Smirnov test.
    import benford as bf
# Example: First digit test with 8 decimal places
f1d = bf.first_digits(data_series, digs=1, decimals=8)

  9. Calculate Mean Absolute Deviation (MAD)

    master

    The MAD measures the average absolute deviation between found proportions and Benford's expected proportions. A lower MAD indicates better conformity. You can calculate it using the bf.mad() function by specifying the test type:

    • 1 or 'F1D': First Digit
    • 2 or 'F2D': First Two Digits
    • 3 or 'F3D': First Three Digits
    • 22 or 'SD': Second Digit
    • -2 or 'L2D': Last Two Digits

    Alternatively, pass MAD=True into any of the digit test functions to include it in the results.

    # Using the standalone function
mad1 = bf.mad(data_series, test=1, decimals=8)

# Including MAD in a digit test
f2d = bf.first_digits(data_series, digs=2, decimals=8, MAD=True)

  10. Analyze Mantissas with `bf.mantissas()`

    master

    In a Benford-compliant dataset, the mantissas (decimal parts of logarithms) should be uniformly distributed. The bf.mantissas() function analyzes this property. It returns a Mantissa object where the results are stored in the .data attribute.

    # Returns a Mantissa object
mant = bf.mantissas(data_series)

# Access the data
print(mant.data.head())