DOCSEARCHbyAlgolia
Submit Docs

caRtola Data Science Repository

repository·master·Indexed 19 days ago

https://github.com/henriquepgomide/cartola

A data science project for Cartola FC, featuring data extraction from the Cartola API, exploratory analysis, and predictive models implemented in Python, R, and C++. The repository provides processed football data from 2014-22, including pre-trained neural network models (MLPRegressor), Markov Chain-based team scalers, and tutorials on player valuation and performance prediction.

Tokens
10K
Snippets
23
Records
32
Agent score
67%

What's inside caRtola

  1. Overview of Notebooks and Models

    master

    The notebooks/ directory contains the core data science workflows and pre-trained models for Cartola FC analysis:

    Notebooks and Scripts

    • Análise dos Dados.ipynb: Covers data import, cleaning, sample creation, and model training.
    • markov chain lpp: A team scaler based on predictions using Markov Chains and linear programming.

    Pre-trained Neural Network Models

    Located in the models/ folder:

    • nn.pkl: The current model, updated every round of the Brasileirão. Older versions can be retrieved via Git history.
    • nn_31.pkl: A specific model trained for round 31 of the 2017 Brasileirão, used for performance comparison against nn.pkl.

  2. Overview of R scripts in the Cartola repository

    master

    The src/R/ directory contains several R scripts designed for data collection, scraping, and processing related to Cartola FC and CBF data:

    • caRtola_fetch.R: Fetches data from the Cartola API.
    • team_data_scraper.R: Scrapes data from the CBF (Confederação Brasileira de Futebol) website.
    • data_wrangling: Aggregates Cartola scout data and creates variables for predictive models by combining multiple data sources.
    • rdata_2_sql.R: Transforms Cartola data objects into SQL files.

  3. Participate in the Desafio Valorização

    master

    The desafio_valorizacao folder contains a challenge to reverse-engineer the Cartola valuation algorithm from 2018.

    To contribute:

    1. Find the necessary data in ~/caRtola/data/desafio_valorizacao.
    2. Use the Python script located within the desafio_valorizacao folder to develop your solution.

  4. Explore Cartola FC data science tutorials

    master

    The repository provides several tutorials for performing data analysis on Cartola FC using different programming languages:

    Python

    • Reading all round files from the repository using BeautifulSoup and Pandas.
    • Automatic player selection using Markov Chains and linear programming.
    • Comparing global averages vs. home/away averages for player selection.
    • Analyzing the Cartola PFC player valuation algorithm.

    R

    • Analyzing forwards using Affinity Propagation.
    • Deciding between offensive or defensive midfielders.
    • Analyzing defensive players using Affinity Propagation.
    • Building defenses using Poisson regression.

    C++

    • Implementation of the Cartola FC valuation algorithm formula.

  5. Set up the Python environment using Miniconda

    master

    To run the notebooks and scripts in this repository, it is recommended to use Miniconda to manage dependencies. The project provides a cartola.yml file to automate the environment creation.

    Follow these steps:

    1. Create the environment from the configuration file.
    2. Activate the environment based on your operating system.
    3. Launch Jupyter Notebook to access the files.
    # Create the environment
conda env create -f cartola.yml

# Activate on Linux/Mac
source activate cartola

# Activate on Windows
activate cartola

# Launch Jupyter
jupyter notebook

  6. Analyze defender performance excluding clean sheets in R

    master

    To evaluate defenders (full-backs and center-backs) based on their performance metrics excluding the 'clean sheet' (saldo de gols) bonus, you can use the provided dataset. This approach helps isolate the core defensive and offensive contributions of players.

    Key data transformations used in this analysis include:

    • Filtering: Only defenders who played at least 10 matches in the season are included.
    • Scout Calculation: Scout averages are calculated up to the current round.
    • Score Adjustment: A score.no.cleansheets_mean metric is used, which represents the average points excluding clean sheet bonuses.
    • Fouls Scout: Created by summing yellow and red cards.
    • Shots Scout (shotsX): Created by summing shots off-target, defended shots, shots hitting the post, and goals.
    # Load the dataset
cartola <- read.csv("~/caRtola/src/R/tutoriais/tutorial-2-data.csv", stringsAsFactors = FALSE)

# Visualize the distribution of average scores excluding clean sheets by position
library(ggplot2)
library(ggthemes)

ggplot(data = cartola, aes(x = score.no.cleansheets_mean, fill = posicao)) +
  geom_density(alpha = .7) +
  labs(title = "CartolaFC - Comparação por Posição",
       subtitle = "Média sem saldo de gols entre Zagueiros e Laterais",
       caption = "Dados de defensores com no mínimo 10 partidas em 2018.") +
  theme_fivethirtyeight()

  9. Contribute to the caRtola repository

    master

    The repository is open to contributions. You can contribute by:

    • Sharing your own statistical analyses of Cartola FC.
    • Sharing predictive models for player performance.
    • Reporting errors in the existing data or analyses.

    Contributions can be made via Pull Requests or by opening an Issue.

  10. Prepare Cartola FC match data in R

    master

    To use the Dixon and Coles statistical model for predicting match outcomes, you must first prepare the match data. This involves loading the tidyverse and fbRanks packages, reading the match CSV files, and cleaning the data by converting date strings to Date objects and splitting the score column (e.g., "0 x 1") into separate home and away integer scores.

    # Carregar pacotes
library(tidyverse) # Manipulação dos dados
library(fbRanks)   # Uso do modelo de Dixon e Coles para estimativa da força de um time

# Abrir dados dos confrontos
matches <- read.csv("~/caRtola/data/2018/2018_partidas.csv", stringsAsFactors = FALSE)

# Limpeza de dados
# 1. Transformar coluna date em 'Date'
matches$date <- as.Date(matches$date, format = "%d/%m/%Y - %H:%M")

# 2. Separar dados da coluna "score"
matches <- separate(matches, score, c("home.score","vs","away.score"), convert = TRUE)
matches$home.score <- as.integer(matches$home.score)
matches$away.score <- as.integer(matches$away.score)

# 3. Retirar dados até a data desejada
matches <- filter(matches, date <= "2018-09-03")

# 4. Renomear colunas para R style
matches <- 
  matches %>%
  dplyr::select(game,round,date,
         home_team,home.score, away.score,
         away_team)

names(matches)[c(4,7)] <- c("home.team", "away.team")

  11. Configure local and base settings in Kedro

    master

    This project uses a split configuration structure to separate sensitive credentials from shared project settings.

    • local/ folder: Use this directory for user-specific settings (like IDE configurations) or protected information (like security keys and credentials). Never check the local folder into version control.
    • base/ folder: Use this directory for shared, non-sensitive, and project-related configuration that should be accessible to all team members. Do not put access credentials in the base folder.

  12. Convert categorical data to numerical for samples

    master

    Before training, categorical columns must be mapped to integers. The notebook uses the following mapping logic:

    • Positions (Posicao): {'gol': 1, 'zag': 2, 'lat': 3, 'mei': 4, 'ata': 5}
    • Teams (ClubeID): Mapped to an integer index based on unique values in the dataset.
    • Match Variable (variable): {'home.team': 1, 'away.team': 2}
    def dict_positions(to_int = True):
    dict_map = {'gol':1, 'zag':2, 'lat':3, 'mei':4, 'ata':5}
    return  dict_map if to_int else dict(zip(dict_map.values(), dict_map.keys()))

def dict_teams(to_int = True):
    teams_map = {team:(index+1) for index, team in enumerate(teams_full)}
    return teams_map if to_int else dict(zip(teams_map.values(), teams_map.keys()))