Iowa Environmental Mesonet (IEM)

The AFOS archive undergoes specific processing and quality control to maintain data integrity:

Quality Control

• Format Validation: Ensures data is in ASCII format and free of unexpected control characters.
• Timestamp Validation: Checks that product timestamps are sane and close to real-time. For example, erroneous products from the NOAAPort SBN feed are filtered out.

Data Culling (Deletion)

To manage storage, certain high-frequency text products are culled.

• SHEF Products: These are significantly deleted to prevent overwhelming storage systems.
• RRM Products: These are generally SHEF products and were culled from the database. However, culling for RRM stopped on 31 March 2023 due to user requests, meaning the RRM archive only contains data up to that date.

For the exact list of AWIPS IDs used for daily database culling, refer to the clean_afos.py script.

Temperature data in the METAR archive is subject to precision challenges due to the conversion between Celsius and Fahrenheit.

Core Behavior

ASOS stations internally store temperatures in whole degree Fahrenheit. However, data transmission often occurs in whole degree Celsius, which can lead to rounding errors during conversion (e.g., 78°F $\rightarrow$ 26°C $\rightarrow$ 78.8°F $\rightarrow$ 79°F).

Source-Specific Temperature Reliability

• NOAA Satellite Feed (NOAAPort): Transmits in whole degree Celsius. The IEM prioritizes METARs that include the T-group to ensure reliable conversion back to Fahrenheit.
• MADIS 5-minute ASOS: Data transmitted from the FAA to the NWS is only in whole degree Celsius and cannot be reliably converted. The IEM database stores these values as missing and excludes them from standard data downloads. However, these values are preserved in the IEM-encoded raw METAR string available in the data downloads.
• NCEI ISD: Currently considered reliable for whole degree Fahrenheit conversion.

When working with Climodat data, be aware of the following temporal nuances:

Observation Timing

The date provided in the dataset typically represents the date on which the observation was taken, not necessarily a full local calendar day.

• Observations are typically taken around 7 AM.
• A single date entry usually represents the 24-hour period prior to 7 AM on that date.
• Some locations may have switched observation times over the years, or may observe 24-hour precipitation totals at different times than temperature values.

Solar Radiation Data

Since the NWS COOP Network does not provide daily solar radiation, the IEM provides point sampling from processed reanalysis datasets.

• Note: Because COOP observations are typically at 7 AM and gridded radiation data is extracted on a local calendar day basis, the 7 AM value is generally considered to be for the previous day.

When using the METAR dataset, be aware of the following regarding precipitation data:

• Non-US Locations: Precipitation data is often missing or reported as zero for locations outside the United States. This is due to data distribution agreements regarding high-value agricultural commodity data.
• Daily Totals Calculation: Because ASOS stations operate on local standard time, calculating true calendar day totals is complex. The IEM uses three methods to determine these totals:
  1. Manual Totals: A script manually sums hourly precipitation reports to compute a local calendar day total.
  2. Daily Summary Message (DSM): Real-time ingestion of daily totals from stations that issue DSM products.
  3. Climate Report (CLI): Real-time ingestion of daily totals from Climate Reports issued by local NWS Forecast Offices.

Note that not all stations provide DSM or CLI products, so the manual totaling serves as the baseline accounting.

The daily_analysis.py script is responsible for sourcing and gridding various variables. The logic depends on the specific variable and the time period being analyzed:

Temperature and Snow (12z variables)

Variables: high_tmpk_12z, low_tmpk_12z, snow_12z, snowd_12z

• Years 2009+: Uses archived COOP data from IEM and performs simple gridding.
• Pre-2009: Uses all existing observations available in the COOP database for gridding.

Precipitation (12z)

Variable: p01d_12z

• Sums up hourly IEMRE precipitation data.
• Pre-1997: Grids available Climatolog Station observations for the specific date.

Temperature and Wind (Daily)

Variables: high_tmpk, low_tmpk, avg_dwpk, wind_speed

• Post-1927: Grids observations found in the IEM Access database.

Precipitation (Daily)

Variable: p01d

• Constructs a midnight CST/CDT analysis by looking at the hourly grids.

Surface Downward Radiation

Variable: rsds

• Derived from IFS forecast processing, which writes into IEMRE hourly NetCDF files, which are then totaled by daily_analysis.py.

PRISM data grids can be inspected using gdalinfo or by examining the .hdr header file.

GDAL Grid Information

gdalinfo provides the bounding box coordinates (Upper Left, Lower Left, Upper Right, Lower Right) and the Center point in both decimal degrees and degrees/minutes/seconds.

Header (.hdr) File Parameters

The .hdr file contains spatial mapping parameters. A key parameter is ulxmap, which represents the x-axis map coordinate of the center of the upper-left pixel.

Example .hdr parameters:

ULXMAP         -125
ULYMAP         49.9166666666687
XDIM           0.04166666666667
YDIM           0.04166666666667

gdalinfo says the grid is:

Upper Left  (-125.0208333,  49.9375000) (125d 1'15.00"W, 49d56'15.00"N)
Lower Left  (-125.0208333,  24.0625000) (125d 1'15.00"W, 24d 3'45.00"N)
Upper Right ( -66.4791667,  49.9375000) ( 66d28'45.00"W, 49d56'15.00"N)
Lower Right ( -66.4791667,  24.0625000) ( 66d28'45.00"W, 24d 3'45.00"N)
Center      ( -95.7500000,  37.0000000) ( 95d45' 0.00"W, 37d 0' 0.00"N)

The IEM database stores the last known state of a NWS Watch, Warning, or Advisory (WaWA) event. Because NWS events undergo a lifecycle (updates to times, spatial extents, or upgrades like a Watch becoming a Warning), the database does not store the full history of every change, but rather a snapshot of the most recent metadata.

Key Lifecycle Behavior:

• Updates: When the NWS updates an event, fields like ISSUED and EXPIRED are updated to reflect the new times.
• Upgrades: If an event is upgraded (e.g., from a Watch to a Warning), the original event's EXPIRED time might be set to a time before its ISSUED time to denote its end.
• Recommendation for Time Handling: To avoid issues where EXPIRE < ISSUE due to upgrades, it is recommended to use the INIT_ISS (initial issuance) and INIT_EXP (initial expiration) columns to represent the intended start and end of the event, as these are not updated during the lifecycle.