DOCSEARCHbyAlgolia
Submit Docs

pandas: Python Library for Data Analysis

repository·main·Indexed 13 days ago

https://github.com/pandas-dev/pandas

A high-performance Python library for data analysis and manipulation. It provides core data structures like Series and DataFrames for working with labeled or relational data, featuring tools for missing data handling, data alignment, Group By operations, time series analysis, and hierarchical indexing.

Tokens
191.7K
Snippets
608
Records
934
Agent score
99%

What's inside pandas

  1. Overview of pandas I/O tools

    main

    The pandas I/O API consists of two main components:

    1. Readers: Top-level functions in the pandas namespace (e.g., pandas.read_csv) that load data from various formats into pandas objects like DataFrame or Series.
    2. Writers: Methods attached to pandas objects (e.g., DataFrame.to_csv) used to save data to specific file formats.

    Commonly supported formats include text-based formats (CSV, JSON, HTML, XML, Fixed-Width Text) and binary formats (Excel, HDF5, Feather, Parquet, Pickle, SQL).

  2. Performance improvements in pandas 1.4.0

    main

    Pandas 1.4.0 introduced significant performance optimizations across various core operations, including GroupBy methods, indexing, sparse data handling, and I/O functions. Key areas of improvement include:

    GroupBy Operations

    • Sampling: Improved performance for .DataFrameGroupBy.sample and .SeriesGroupBy.sample, particularly when using the weights argument.
    • Transform & Apply: Faster execution for .DataFrameGroupBy.transform and .SeriesGroupBy.transform when using user-defined functions, as well as certain .apply operations.
    • Reductions: Faster .DataFrameGroupBy.quantile, .count, .any, .all, .std, and .cumsum operations.
    • Shifting: Improved .DataFrameGroupBy.shift and .SeriesGroupBy.shift when a fill_value is provided.

    Indexing and Data Structures

    • Index Types: Faster indexing for UInt64Index, Float64Index, non-unique Index, and MultiIndex (both with listlike indexers and MultiIndex indexers).
    • DataFrame Construction: Optimized construction of DataFrame objects, including from array-like objects such as Pytorch tensors.
    • RangeIndex: Faster RangeIndex.sort_values and RangeIndex.insert.

    Sparse Data

    • Memory Efficiency: SparseArray.min and SparseArray.max no longer require conversion to a dense array.
    • Slicing: Indexing into a SparseArray with a slice where step=1 no longer requires conversion to a dense array.
    • Other: Improved SparseArray.take with allow_fill=False and SparseArray.__getitem__.

    I/O and Conversions

    • CSV: Improved pandas.read_csv performance with memory_map=True for UTF-8 encoded files, and when index_col is set to a numeric column. to_csv is faster when the index is a formatted datetime or when a MultiIndex has many unused levels.
    • Stata & SAS: Performance improvements in read_stata and read_sas.
    • Datetime: Faster to_datetime with uint dtypes or when infer_datetime_format=True.

    Other Core Methods

    • DataFrame: Faster DataFrame.corr (Pearson method on non-missing data), DataFrame.dropna, DataFrame.fillna, DataFrame.select_dtypes, DataFrame.insert, and DataFrame.values.
    • Series: Faster Series.unstack, Series.to_frame, Series.nsmallest, and Series.mad.
    • General: Improved merge, concat, and unstack operations.

  3. Performance improvements in v0.17.0

    main

    Version 0.17.0 introduced several significant performance optimizations across core pandas operations, including:

    • Indexing & Selection: 8x improvement in iloc using list-like input; significantly improved MultiIndex indexing with slicers; improved Series.isin for datetimelike/integer Series.
    • Data Manipulation: 20x improvement in concat of Categorical objects when categories are identical; performance improvements in DataFrame.drop_duplicates (with integer dtypes) and DataFrame.duplicated (with wide frames).
    • Time Series: 4x improvement in timedelta string parsing; 8x improvement in timedelta64 and datetime64 operations; improved to_datetime performance for ISO8601 format strings; enabled infer_datetime_format in to_datetime for non-zero-padded date components.
    • Aggregation: Performance improvements in Categorical.value_counts, SeriesGroupBy.nunique, SeriesGroupBy.value_counts, SeriesGroupBy.transform, and Series.value_counts (for float dtype).
    • Arithmetic: Improved addition/subtraction operations for DateOffset with Series or DatetimeIndex.

  4. Key features in pandas v0.21.0

    main

    Major highlights of the v0.21.0 release include:

    • Apache Parquet Integration: New support via read_parquet and DataFrame.to_parquet.
    • CategoricalDtype: Introduction of pandas.api.types.CategoricalDtype for specifying categoricals independently of the data.
    • Consistent Aggregations: sum and prod behavior on all-NaN Series/DataFrames is now consistent and independent of the bottleneck library. sum and prod on empty Series now return NaN instead of 0.
    • API Consistency: Improvements to drop, reindex, and rename APIs.
    • New Methods: Addition of DataFrame.infer_objects and GroupBy.pipe.
    • PyPy Compatibility: Various fixes to support the PyPy runtime.

  6. Core features of pandas

    main

    pandas is a high-level data analysis toolkit with several key capabilities:

    • Missing Data Handling: Easy management of NaN, NA, and NaT across various data types.
    • Data Structures: Supports Series and DataFrame objects with size mutability (inserting/deleting columns).
    • Data Alignment: Automatic and explicit alignment of data based on labels during computations.
    • Group By: Powerful split-apply-combine functionality for aggregating or transforming datasets.
    • Data Manipulation: Includes slicing, fancy indexing, subsetting, merging, joining, reshaping, and pivoting.
    • I/O Tools: Robust support for loading/saving data from CSV, Excel, SQL databases, and HDF5 formats.
    • Time Series: Specialized functionality for date range generation, frequency conversion, moving window statistics, and date shifting/lagging.
    • Hierarchical Indexing: Support for multi-level labeling of axes (MultiIndex).

  7. Performance improvements in pandas 2.1.0

    main

    Pandas 2.1.0 introduced significant performance optimizations across various operations, particularly for extension dtypes (like PyArrow-backed strings and nullable integers) and specific data structures. Key areas of improvement include:

    Data Loading & IO

    • read_orc: Faster reading of remote URI file paths.
    • read_parquet & DataFrame.to_parquet: Improved performance when reading remote files using engine="pyarrow".
    • read_parquet: Faster reading of string columns when use_nullable_dtypes=True is set.
    • read_csv: Improved performance with engine="c".

    String & PyArrow Operations

    • PyArrow-backed strings: Significant speedups for merge, Series.add (for string/binary dtypes), Series.str.get_dummies, Series.str.get, Series.str.split(expand=True), and Series.drop_duplicates.
    • Type Conversion: Faster parsing of strings to boolean[pyarrow] dtype and improved ArrowExtensionArray.astype when converting PyArrow timestamp/duration dtypes to NumPy.

    Indexing & Reshaping

    • MultiIndex: Faster set_levels and set_codes (with verify_integrity=True), sortlevel (when ascending is a list), and general indexing/setting operations.
    • DataFrame Indexing: Improved iloc (single integer access with extension dtypes), loc (row/column selection), and transpose (for single PyArrow or masked dtypes like Int64).
    • RangeIndex: Faster take operations and indexing via loc/iloc when using a RangeIndex.

    Aggregations & Reductions

    • GroupBy: Faster aggregations (e.g., .sum()) using engine="numba" and improved .groups access.
    • Reductions: Improved performance for axis=1 and axis=None reductions with extension dtypes, as well as general Series reductions.

    Other Key Optimizations

    • Period: The default formatter (period_format) is ~2x faster, accelerating str(), repr(), .strftime(), and to_csv operations.
    • Concatenation: concat is faster for homogeneous np.float64/np.float32 dtypes, when the axis is a MultiIndex, and when axis=1 with different indexes.
    • Filling/Missing Values: Faster ffill, bfill, fillna, and isna for PyArrow dtypes and ArrowExtensionArray.

  8. Performance Improvements in pandas 0.24.0

    main

    Version 0.24.0 introduced several performance optimizations across different data types and operations:

    Categorical Data

    • Slicing: Slicing Series and DataFrames with a monotonically increasing CategoricalIndex is now significantly faster (comparable to Int64Index) for both label-based (.loc) and position-based (.iloc) indexing.
    • Comparisons: Improved performance for CategoricalIndex.equals and membership checks (x in cat or CategoricalIndex.contains).
    • Construction: Improved performance of the Categorical constructor for Series objects.
    • Operations: Improved performance for ~DataFrame.where when using Categorical data.

    Datetime and Timezone Handling

    • Normalization: Improved performance for DatetimeIndex.normalize and Timestamp.normalize (specifically for timezone-naive or UTC datetimes).
    • Localization: Improved performance for DatetimeIndex.tz_localize and various DatetimeIndex attributes using the dateutil UTC timezone.
    • Binary Operations: Improved performance for tz-aware DatetimeArray binary operations.

    General Performance Gains

    • Indexing/Access: Improved performance for Series.at, Index.get_value (for Extension Arrays like Categorical), and IndexEngine.get_indexer_non_unique for sorted, non-unique indexes.
    • Aggregation/Stats: Improved performance for Series.describe (numeric dtypes) and .GroupBy.rank (handling tied rankings).
    • Data Structures: Improved performance for Period constructor (benefiting PeriodArray and PeriodIndex), PeriodIndex.unique, and concat for Series objects.
    • Iteration: DataFrame.itertuples now creates iterators without internally allocating lists of all elements, improving iteration speed over Series.
    • Other: Improved performance for HDFStore.groups and HDFStore.keys (membership checks), and pandas.get_dummies with sparse=True.

  9. Performance improvements in pandas 1.2.0

    main

    Version 1.2.0 introduced several performance optimizations, including:

    • String Handling: Faster creation of DataFrame or Series with str or StringDtype from large arrays.
    • GroupBy: Improvements in .agg and .transform when using the numba engine, and improvements in ExpandingGroupby.
    • Mapping: Faster Series.map when using very large dictionaries.
    • IO: read_json now avoids reading the entire file into memory when a chunksize is specified.
    • Memory: Reduced peak memory usage in DataFrame.to_pickle when using protocol=5 (Python 3.8+).
    • Indexing: Faster dir() calls for objects with many index labels and improved IntervalArray value setting.

  10. Key features in pandas 0.24.0

    main

    Pandas 0.24.0 introduced several significant enhancements:

    • Optional Integer NA Support: Support for missing values in integer-based data.
    • New Array Access APIs: New ways to access the underlying array backing a Series or Index.
    • Top-level Array Creation: A new top-level method for creating arrays.
    • Interval and Period Data Storage: Ability to store Interval and Period data within a Series or DataFrame.
    • MultiIndex Joins: Support for joining datasets on two MultiIndex objects.

  11. Enhance performance with Cython, Numba, and pandas.eval

    main

    For computationally heavy applications, you can speed up operations on a DataFrame by using Cython, Numba, or pandas.eval.

    • Cython and Numba typically offer the largest speedups but require more code implementation.
    • pandas.eval is a simpler way to evaluate expressions but generally offers less speedup than the compiled options.

    Recommendation: Before moving to Cython or Numba, always attempt to optimize in pure Python first by removing for-loops and utilizing NumPy vectorization. Additionally, ensure you have installed the recommended dependencies for pandas, as they provide significant speed improvements.

  12. Performance improvements in pandas 1.3.0

    main

    Pandas 1.3.0 introduced several performance optimizations across various data structures and operations. Key areas of improvement include:

    Nullable Data Types

    • Faster Series.mean and Series.isin for nullable data types.
    • Improved DataFrame.fillna performance when using method="pad" or method="backfill" with nullable floating and integer dtypes.
    • Faster DataFrameGroupBy.cummin, SeriesGroupBy.cummin, DataFrameGroupBy.cummax, and SeriesGroupBy.cummax for nullable data types.
    • Faster conversion of PyArrow Boolean arrays to pandas nullable Boolean arrays.

    GroupBy and Window Operations

    • Performance gains in .Rolling.corr and .Rolling.cov.
    • Performance gains in .RollingGroupby.corr, .ExpandingGroupby.corr, and .ExpandingGroupby.cov.
    • Improved performance for .ExpandingGroupby aggregation methods.
    • Faster .DataFrameGroupBy.apply and .SeriesGroupBy.apply when using the Python fallback implementation.

    Statistical and Correlation Methods

    • Faster DataFrame.corr for method="kendall" and method="spearman".
    • Improved Series.nunique performance when handling NaN values.

    Indexing and Transformation

    • Improved IntervalIndex.isin performance.
    • Faster unique function for object data types.
    • Improved DataFrame.transpose and Series.unstack performance when using DatetimeTZDtype.
    • Faster concatenation for data using CategoricalDtype.

    Styling and Plotting

    • Significant reduction (over 50%) in .Styler render times, now matching DataFrame.to_html performance.
    • .Styler.set_td_classes is now as performant as .Styler.apply and .Styler.applymap.
    • Faster Series.plot and DataFrame.plot via lazy loading of entry points.

    Other Utilities

    • Improved json_normalize performance for basic cases, including the use of separators.