DOCSEARCHbyAlgolia
Submit Docs

pyreadstat

repository·master·Indexed 19 days ago

https://github.com/roche/pyreadstat

A high-performance Python package for reading and writing SAS (.sas7bdat, .sas7bcat, .xport), SPSS (.sav, .zsav, .por), and Stata (.dta) data files. A Python wrapper around the Readstat C library, it provides faster I/O and better metadata handling (value labels and date types) than standard pandas methods. It supports outputting data to pandas or polars DataFrames and includes features for reading large files in chunks or via multiprocessing.

Tokens
6.3K
Snippets
19
Records
30
Agent score
63%

What's inside pyreadstat

  1. Overview of pyreadstat capabilities

    master

    pyreadstat is a Python wrapper around the Readstat C library. It allows you to read and write various statistical data files into pandas or polars dataframes.

    Supported Formats:

    • SAS: sas7bdat, sas7bcat, xport
    • SPSS: sav, zsav, por
    • STATA: dta

    Key Advantages over pandas.read_sas:

    • Performance: Significantly faster for large files (e.g., reading a 190MB SAS file in 7s vs 42s with pandas).
    • Value Labels: Can extract value labels from SPSS, STATA, and SAS catalog files.
    • Date/Datetime Accuracy: Correctly distinguishes between date and datetime types instead of converting all to datetime.
    • Encoding: Automatically handles character encodings and translates them to UTF-8, returning Python str instead of bytes.

  2. Understand the Metadata Object returned by parsing functions

    master

    Every parsing function in pyreadstat returns two objects: a data container (typically a pandas.DataFrame) and a metadata object. This metadata object provides structural and descriptive information about the dataset that is not contained within the raw data rows themselves.

    Key fields available in the metadata object include:

    • Structural Info: column_names (list), column_labels (list), column_names_to_labels (dict), number_columns (int), number_rows (int), and table_name (string).
    • File Info: file_encoding (string), creation_time, modification_time, and file_label (SAS specific).
    • Variable Properties:
      • variable_measure: The measurement level (nominal, ordinal, scale, or unknown).
      • variable_alignment: Display alignment (left, center, right, or unknown).
      • variable_storage_width and variable_display_width: Storage and display widths.
      • original_variable_types and readstat_variable_types: Debugging information regarding variable formats.
    • Labels and Values:
      • variable_value_labels: A nested dictionary mapping variable names to their value-to-label mappings.
      • value_labels: A dictionary of label names to their value-to-label mappings (used for SAS formats/catalogs).
      • variable_to_label: Maps variable names to their specific label name.
    • Missing Value Handling:
      • missing_ranges: For SPSS (.sav) files with user_missing=True, contains the lo and hi boundaries for user-defined missing values.
      • missing_user_values: For SAS and STATA, contains character representations of user-defined missing values.
    • Special Formats:
      • mr_sets: Definitions for multiple-response (MR) variables (currently only for SPSS .sav format), specifying type, is_dichotomy, counted_value, label, and variable_list.
    • Annotations:
      • notes: Text annotations attached to the file (supported for SPSS and STATA).

  3. Known limitations of pyreadstat

    master

    As pyreadstat is built on top of Readstat, it inherits several limitations:

    • SAS sas7bdat writing: While sas7bdat files can be written, they cannot be read in SAS. Consequently, writing them is not supported in pyreadstat.
    • Date/Time Ranges: Python's datetime object limitations can cause OverflowError for extreme dates (see workaround above).
    • Python 2.7: Support for Python 2.7 was dropped in version 1.2.3.
    • Non-ASCII paths: On Linux and Mac, reading may fail if the file path contains non-ASCII characters.

  5. Handle user-defined missing values

    master

    By default, pyreadstat translates system and user-defined missing values to NaN (pandas) or null (polars). To preserve the original user-defined missing values, use the user_missing=True argument.

    SPSS

    • Use user_missing=True in read_sav to see original values (e.g., 2, 3) instead of NaN.
    • Missing value definitions are stored in meta.missing_ranges as a list of {'hi': val, 'lo': val} dictionaries.
    • When writing, use missing_ranges in write_sav to define these values.

    SAS and STATA

    • Use user_missing=True in read_sas7bdat or read_dta to see values like A-Z or _ (SAS) and a-z (STATA).
    • Missing value definitions are stored in meta.missing_user_values.
    • When writing STATA files, use missing_user_values in write_dta (accepts single character strings).
    import pyreadstat

# Read SPSS file preserving user-defined missing values
df, meta = pyreadstat.read_sav("/path/to/file.sav", user_missing=True)

# Access missing range info for SPSS
print(meta.missing_ranges)

# Read SAS file preserving user-defined missing values
df, meta = pyreadstat.read_sas7bdat("/path/to/file.sas7bdat", user_missing=True)
print(meta.missing_user_values)

  7. Install pyreadstat from source

    master

    To install from the latest source code, clone the repository and run the setup script. You will need a working C compiler and cython >= 3.0.0.

    You can also install directly from the GitHub repository using pip.

    # From cloned repo
python3 setup.py install

# For users without admin rights
python3 setup.py install --user

# Directly from GitHub
pip install git+https://github.com/Roche/pyreadstat.git

  8. Compile pyreadstat on Windows using Microsoft Visual Studio (Recommended)

    master

    The recommended way to compile pyreadstat on Windows is using the Microsoft Visual Studio (MSVC) compiler. This approach allows for static compilation of external libraries, meaning pyreadstat will not depend on external DLL files.

    Prerequisites

    • MSVC must be installed and matching your Python version.
    • You must provide libiconv and zlib libraries. You can either compile them manually to ensure they match your MSVC version or install them via conda.

    Option 1: Manual Compilation of External Libraries (Static)

    To produce a standalone build (similar to PyPI wheels), compile libiconv and zlib using msbuild.exe and copy the resulting headers and static libraries into the pyreadstat/win_libs/64bit directory.

    Option 2: Installing via Conda

    If you use Conda, you can install the required libraries from conda-forge and then build pyreadstat normally.

    Building the extension

    Once the libraries are in place, run:

    python setup.py build_ext --inplace
    # Using Conda to provide dependencies
conda install -c conda-forge libiconv
conda install -c conda-forge zlib
python setup.py build_ext --inplace

  9. Install pyreadstat via pip

    master

    The easiest way to install pyreadstat is using pip. Pre-compiled wheels are available for Windows, Mac, and Linux. If no wheel is available for your Python version, pip will attempt to compile from source, which requires a C compiler.

    To install for the current user without admin rights, use the --user flag.

    pip install pyreadstat

# For users without admin rights
pip install pyreadstat --user

  10. Run pyreadstat tests

    master

    Depending on whether you have installed pyreadstat in your environment or built it in place, use the following commands to execute the test suite.

    If installed in your environment

    Run the specific test scripts directly using python3.

    If built in place

    Append the --inplace flag to the test commands to indicate the build is in place.

    # If installed in environment
python3 tests/test_basic.py
python3 tests/test_narwhalified.py --backend=pandas
python3 tests/test_narwhalified.py --backend=polars
python3 tests/test_http_integration.py

# If built in place
python3 tests/test_basic.py --inplace
python3 tests/test_narwhalified.py --inplace --backend=pandas
python3 tests/test_narwhalified.py --inplace --backend=polars
python3 tests/test_http_integration.py --inplace

  11. Read from file-like objects

    master

    Instead of a file path, you can pass a file-like object (e.g., io.BytesIO, zipfile.open, or a requests response stream) to any read_* function. This is useful for reading data directly from memory or remote URLs without saving to disk.

    import io
import requests
import pyreadstat

# Reading from a remote URL
response = requests.get('https://example.com/data.sav')
df, meta = pyreadstat.read_sav(io.BytesIO(response.content))

  12. Run all pyreadstat tests in place

    master

    To execute the entire test suite (basic, narwhalified with pandas/polars, HTTP integration, type hints, and runtime types) in a single command for an in-place build, use the following combined command:

    python tests/test_basic.py --inplace && python tests/test_narwhalified.py --inplace --backend=pandas && python tests/test_narwhalified.py --inplace --backend=polars && python tests/test_http_integration.py --inplace && pytest tests/test_typing.yml --mypy-ini-file=tests/test_mypy_setup.ini & python tests/test_runtime_types.py --inplace