fftiers

The cron schedule is determined automatically by the dates set in src/config.R. The execution frequency changes based on the time of year:

This logic is handled by src/cron-run.sh, which reads the configuration to decide whether to execute the pipeline.

To manually run the pipeline to download data, generate charts, and upload them to S3, follow these steps:

1. Setup API Key: Create src/api_key.py with your FantasyPros API key (ensure this file is never committed to version control).
2. Install R Dependencies: Install the required R packages mclust and ggplot2.
3. Run Pipeline: You can either run the individual steps or the full master script.

Individual steps:

# Install R packages
Rscript -e 'install.packages(c("mclust","ggplot2"), repos="https://cloud.r-project.org")'

# Generate data
cd src
Rscript main.R TRUE

# Upload to S3
python3 push-to-s3.py

Full pipeline command:

python3 src/master.py

To automate the generation and upload process, set up the crontab using the provided example.

Setup steps:

1. Make the cron wrapper executable: chmod +x src/cron-run.sh.
2. Create a logs directory: mkdir -p logs.
3. Install the crontab: crontab src/crontab.example. Note: You must edit the path inside src/crontab.example if your repository is not located at /root/dev/fftiers.
4. Verify installation: crontab -l.

Running the cron wrapper manually:

• ./src/cron-run.sh: Runs the script but respects the schedule (skips if it is the wrong day).
• ./src/cron-run.sh --force: Forces the script to run immediately regardless of the schedule.

Important Notes:

• Logs are stored in logs/cron.log.
• Failures are emailed to boris.chen@gmail.com.
• Workbench Caveat: If the container sleeps while idle, missed 6:00 AM runs will not be automatically caught up.

The script retrieves the API key from a local file named api_key.py located in the same directory as fp_api.py.

To use the API functions, create a file at src/api_key.py containing your key as a single string on the first line.

Warning: Do not commit api_key.py to version control.

# src/api_key.py
YOUR_API_KEY_HERE

The convertTxtToCsv function parses raw text (containing HTML tables) and extracts data into a structured CSV file.

Parameters:

• infile: Path to the source file containing the HTML/text data.
• outfile: Path to the destination CSV file.
• ncol: The number of columns to include.

Behavioral Notes:

• If the string 'flex' is present in the outfile path, the function overrides the ncol parameter to 9 internally.
• It uses BeautifulSoup to find the first <table> in the file, extracts the headers from the first <tr>, and then iterates through subsequent rows to extract cell data.

# Example usage context
convertTxtToCsv(infile='raw_data.txt', outfile='data.csv', ncol=8)

The convertJsonToCsv function transforms a raw JSON file downloaded from the FantasyPros API into a structured CSV format.

Behavior by Week

The CSV header and columns change based on the week parameter:

• Week 0 (Preseason): Includes Rank,Player,Position,Best,Worst,Avg,Std Dev.
• Week 1+ (Regular Season): Includes Rank,Player,Opp,Best,Worst,Avg,Std Dev (adds the opponent column).

If a player record is malformed or missing required fields, it is skipped in the output.

convertJsonToCsv(json_path='path/to/data.json', out_csv='path/to/output.csv', week=1)

The perform_session_download function handles the authenticated session required to download protected FantasyPros data. It performs a login request using a CSRF token and then fetches the target URL.

Parameters:

• args: A dictionary containing username, password, and token (used for the CSRF middleware token).
• url: The string URL of the export XLS file.
• full_file_name: The full file path and name where the downloaded content will be saved.

Workflow:

1. Starts a requests.Session().
2. Fetches the login page to extract the csrfmiddlewaretoken.
3. Posts the credentials and token to the login URL.
4. Downloads the content from the provided url and writes it to full_file_name.

# Example usage context
userargs = {'username':'YOUR_USER', 'password':'YOUR_PASSWORD', 'token':'YOUR_TOKEN'}
perform_session_download(userargs, url, full_file_name)

You can download consensus rankings from the FantasyPros API and automatically convert them to a CSV file using the command line interface. This requires a valid FantasyPros API key to be stored in src/api_key.py.

CLI Arguments

• -j (required): The destination path for the raw JSON file.
• -c (required): The destination path for the converted CSV file.
• -y (required): The season year (e.g., 2022).
• -p (required): The player position (e.g., RB, WR).
• -w (required): The week number (use 0 for preseason/offseason).
• -s (required): The scoring format (e.g., STD).

python src/fp_api.py -j ./data/raw.json -c ./data/output.csv -y 2022 -p RB -w 1 -s STD

The fp_dl.py script provides a command-line interface to download FantasyPros export URLs (typically XLS files) and convert them into CSV format.

Note: The script contains hardcoded credentials in the userargs dictionary within the if __name__ == "__main__": block. Users should ensure these are configured correctly for their account before execution.

Required Arguments:

• -u: The FantasyPros export URL (e.g., an XLS export link).
• -d: The destination path for the raw downloaded file.
• -c: The destination path for the converted CSV file.
• -n: The number of columns to process.

python src/fp_dl.py -u https://www.fantasypros.com/nfl/rankings/half-point-ppr-flex.php?week=1\&export=xls -d ~/projects/fftiers/dat/2017/week-1-half-point-ppr-flex-raw.txt -c ~/projects/fftiers/dat/2017/week-1-half-point-ppr-flex-raw.csv -n 9

The push-to-s3.py script automates the uploading of generated tier assets (specifically from the out/current directory) to the S3 bucket prefix s3://fftiers/out/.

It looks for files in the following subdirectories within out/current/:

• png
• csv
• txt

AWS Authentication

The script determines which AWS profile to use in the following order:

1. The AWS_PROFILE environment variable.
2. A local AWS profile named personal (checked against ~/.aws/credentials and ~/.config).
3. If neither is found, it attempts to use default Workbench AWS credentials.

CLI Options

• --dry-run: Prints the local file paths and their intended S3 destinations without actually executing the aws s3 cp command.

# Perform a dry run to see what would be uploaded
python src/push-to-s3.py --dry-run

# Execute the actual upload
python src/push-to-s3.py