Data architecture and caching strategy
masterThe application uses a static site generation (SSG) approach with an ETL process for live data to handle high traffic efficiently.
Data Loading Pattern
To balance initial load time and interactivity, data is downloaded per province rather than per zone. This prevents downloading all data at once while avoiding frequent network requests when switching between zones in the same province.
Caching and Immutability
- Static Files: Served via Apache/Cloudflare with high cacheability.
- Live Data Reference (
/data/latest.json): A small file (<1kb) that points to specific data files. It has a short cache time (max-age=30) to allow updates. - Actual Data Files: These are immutable. Every update writes to a new file location. This eliminates the need for cache invalidation and allows for 'time travel' by loading specific historical data files.
Cache-Control Headers:
- Reference file:
public, max-age=30, stale-while-revalidate=30, stale-if-error=300, must-revalidate - Data files:
public, max-age=31536000, stale-while-revalidate=30, stale-if-error=300, immutable