OddsHarvester

Localization breaks submarket selection because the full label (e.g., Over/Under +20.5 Games) is translated, but the specific line details are not.

The Fix: Match the untranslated tail of the label rather than the full string. While the prefix (e.g., Over/Under) changes per mirror, the numeric line and axis word (e.g., +20.5 Games or -2.5 Sets) remain byte-identical across all domains.

OddsPortalSelectors.submarket_match_text(specific_market, main_market) implements this by stripping the English main_market prefix and using a substring matcher on the remaining text.

For large-scale scraping, it is safer to use a two-pass approach: first collect all match links, then scrape the odds. This allows you to re-run only the failed links without re-scraping the entire dataset.

Workflow Example:

1. Pass 1: Collect links using --links-only.
2. Pass 2: Scrape odds from the collected file using --match-links-file and --append to fill in any recovered failures.

# Pass 1 - collect the season's match links
oddsharvester historic -s football -l england-premier-league --season 2022-2023 \
    --links-only -f csv -o links.csv

# Pass 2 - scrape odds from the collected links
tail -n +2 links.csv | cut -d, -f1 | sort -u > links.txt
oddsharvester upcoming -s football -m 1x2 -f csv -o odds.csv --append \
    --match-links-file links.txt

Replaying a HAR file containing a live match will not recreate the live view. Instead, it will render the pre-match variant of the event header.

• The Problem: The data-testid="live-info" element will not mount during replay, causing scrapers to incorrectly classify the match as not live.
• Consequence: You cannot write replay tests that assert the presence of the live header.
• Testing Strategy: When testing with HAR files, assert the shape of the live context (e.g., a period marker is present and the score matches \d+:\d+) and the fixed identity of the match (teams, league, date), but do not assert specific captured period or score values, as these are non-deterministic across replays.

When using --odds-history, the "Odds movement" tooltip header is localized (e.g., translated in Spanish). Matching by text like h3:text('Odds movement') will cause timeouts on regional mirrors.

The Fix: Match the header using its stable, language-independent CSS class: h3.font-semibold.uppercase.leading-6.

Note that the dates inside the tooltip are rendered via a hardcoded English JavaScript function (phpJsDate) and are not localized, so they can be parsed using standard English date formats regardless of the mirror domain.

A match being 'in play' (visible on /matches/<sport>/ with an elapsed time like 4') is not the same as a match having 'in-play odds'.

• The /inplay-odds/live-now/ listing only contains matches that have active bookmaker coverage for live betting.
• Many live matches will have no live odds available.

Best Practice: Always use the /inplay-odds/live-now/ listing as your entry point for live scraping. Do not fall back to the general /matches/ listing to 'fix' empty results, or you will collect matches that contain no odds data.

In multi-proxy setups, Odds format (e.g., Decimal Odds) and cookie consent are BrowserContext-scoped. Because each --proxy-url creates its own BrowserContext, you must ensure every context is "warmed up" before scraping to prevent parsing incorrect odds values.

The Risk: If a context is not warmed, it may default to a non-decimal format (like Fractional). The scraper will not throw an error; it will simply parse 5/2 as 2.5 (or vice versa), resulting in silently corrupted data.

Implementation Detail: The scraper uses base_scraper._warm_proxy_contexts() to navigate to the base URL, dismiss the cookie banner via CookieDismisser, and set the correct odds format exactly once per proxy context before any match scraping occurs.

By default, oddsharvester collects all matches and writes the batch at the end. To process data as it is scraped, use the --stream-ndjson flag. This emits each match as a single JSON line on stdout immediately upon completion.

Usage Pattern: Pipe the output into a tool like jq to process records in real-time.

oddsharvester upcoming -s football -d 20260918 -m 1x2 --stream-ndjson --headless \
    | while read -r line; do echo "$line" | jq -r '.home_team + " vs " + .away_team'; done

Note:

• Logs and errors are sent to stderr.
• If --output is provided, a batch file is still written, but the stream remains NDJSON.
• If --output is NOT provided, no result file is written; the stream is the only output.

To avoid inconsistent behavior across different sports or commands, never perform business logic normalization (like mapping --season current to a specific year) inside the CLI layer. The CLI should remain a thin pass-through that parses arguments and dispatches commands.

Correct Pattern

1. Move normalization to the lowest layer that owns the domain knowledge (e.g., URLBuilder for URL shapes, or SportMarketRegistry for market resolution).
2. Delete any hardcoded whitelists (e.g., if sport in {...}) inside CLI handlers.
3. Make the lower-layer behavior explicit in its docstrings.
4. Add tests at the lower layer covering all supported sports.

To avoid silent data errors on regional mirrors (where period labels like 1st Set are localized), select periods using the <scope> segment of the URL fragment (e.g., ...:over-under;12).

Implementation Details:

• Scope IDs are global: IDs like FullTime=2 are consistent across all sports and mirrors.
• Per-sport mapping: Some scopes are sport-specific (e.g., Baseball's FirstHalf uses scope 17, while Football's 1st Half uses scope 3).
• Verification: PeriodSelector.select_by_scope reads the current scope and only clicks tabs if necessary. It returns True only on an exact scope match, preventing the extractor from silently falling back to the wrong period (like Full Time) when a specific period is requested.