soccerdata

SoccerData provides a unified interface for gathering soccer data from multiple sources including Club Elo, ESPN, FBref, FiveThirtyEight, Football-Data.co.uk, Sofascore, SoFIFA, Understat, and WhoScored.

Key Capabilities:

• Data Formats: All data is returned as Pandas DataFrames with consistent column names and identifiers, facilitating easy merging across different sources.
• Caching: Data is downloaded only when necessary and cached locally to optimize subsequent runs.
• Content Types: Access to fixtures, forecasts, match stats, and event stream data.
• Integration: Integrates with the socceraction package for advanced event stream analysis.

SoccerData supports multiple data sources including:

• Club Elo
• ESPN
• FBref
• Football-Data.co.uk
• Sofascore
• SoFIFA
• Understat
• WhoScored

Each scraper returns standardized Pandas DataFrames with matching column names and identifiers across different datasets.

The library uses two primary types of base readers depending on the technical requirements of the data source:

1. BaseRequestsReader: A wrapper around the requests library. Use this (or implement readers based on it) for scrapers that download static content and do not require JavaScript to be executed to render the data.
2. BaseSeleniumReader: A wrapper around the selenium library. Use this (or implement readers based on it) for scrapers that require a browser engine to execute JavaScript to load or display the data.

In soccerdata, the logic for downloading data from the web is encapsulated in base classes. These base classes are not intended for direct use by end-users. Instead, they serve as the foundation for specific reader classes (e.g., a Premier League reader) which implement the actual data parsing logic.

When choosing or extending a reader, the underlying mechanism is determined by whether the source requires JavaScript execution.

SoccerData provides a collection of scrapers to gather soccer data from various websites (e.g., FBref, ESPN, Sofascore) and returns the data as Pandas DataFrames. Data is downloaded on demand and cached locally.

To get started, import soccerdata, instantiate a scraper class for your desired league and season, and use the read_* methods to fetch data.

import soccerdata as sd

# Create a scraper class instance for the 2020/21 Premier League
fbref = sd.FBref('ENG-Premier League', '2021')

# Fetch data
games = fbref.read_schedule()
team_season_stats = fbref.read_team_season_stats(stat_type="passing")
player_season_stats = fbref.read_player_season_stats(stat_type="standard")

To avoid rate limits or access restricted data, you can route your scraper through a SOCKS5 proxy using Tor.

1. Install Tor following the official instructions.
2. Start the Tor service (e.g., by running tor in your terminal).
3. By default, Tor listens on localhost:9050. To use this default configuration, pass proxy='tor' when initializing your scraper class.

Note: This assumes Tor is running on the default port 9050.

ws = sd.WhoScored(proxy='tor')

To test code changes, you need a Python environment with all required dependencies. It is recommended to use uv for environment management.

Using uv (Recommended)

1. Install uv (macOS/Linux):

   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Create and activate a virtual environment with a supported Python version (e.g., 3.10) and sync dependencies:

   uv venv --python 3.10
   uv sync

Using pip

Alternatively, you can use standard pip within a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
python -m pip install -r requirements.txt

Each supported data source has a dedicated class with a uniform API. For example, use the soccerdata.FBref class to fetch data from fbref.com. Data is returned as a Pandas DataFrame.

To fetch aggregated shooting stats for all teams:

import soccerdata as sd

# Create scraper class instance
fbref = sd.FBref()

# Create dataframes
season_stats = fbref.read_team_season_stats(stat_type='shooting')

The project uses pytest for testing. You can run the full suite or target specific data sources.

• Run all tests: Use make test.
• Run tests for a specific class: Use make test-class <class_name>. For example, to run tests for ClubElo only:

  make test-class clubelo