openFDA Documentation

openFDA is a research project providing open APIs and raw data downloads for FDA public datasets. The repository contains the core components that power the api.fda.gov endpoints:

• Python Pipelines: Built with Luigi, these process FDA datasets (drugs, foods, medical devices, etc.) into JSON format for Elasticsearch.
• Elasticsearch Schemas: Definitions for the available datasets.
• Node.js API Server: An Express-based server using Elasticsearch.js and Elastic.js that provides the JSON interface for api.fda.gov by communicating with Elasticsearch.

The wide_sample.json file is a manually created sample used to generate the mapping schema for the deviceevent/maude Elasticsearch index.

Important Note on Date Fields: This file must include all expected date fields, even those that are null. This is because the join_maude.py process removes null date fields (as Elasticsearch cannot handle null dates), but the mapping file generation process requires these fields to be present in the sample to define the schema correctly.

The primarysource.reportercountry field uses different value formats depending on whether you are querying AERS or FAERS data:

• AERS: Uses full country names (e.g., United States, Japan).
• FAERS: Uses two-letter ISO country codes (e.g., US, JP).

The schemas/ directory contains two distinct types of schema files used for different purposes in the openFDA ecosystem:

1. Elasticsearch mapping files (_mapping.json): These define the structure and data types for the Elasticsearch instance. The data pipelines automatically use these files to create or update mappings if they do not already exist.
2. JSON Schema files (_schema.json): These define the structure and types of the API output. They are provided alongside data downloads to help developers understand the schema and data types of the API results they are consuming.

The following fields are treated as 'BAG OF WORDS' and may require specific extraction or tokenization logic:

• distribution_pattern: Currently free text; consider tokenizing on keywords (e.g., 'nation-wide') and commas to create an array of locations.
• code_info: Consider converting to an array of strings based on tokens.
• product_description: BAG OF WORDS.
• product_quantity: BAG OF WORDS.
• reason_for_recall: BAG OF WORDS.

Note on code_info: If code_info is implemented as an array of strings, a corresponding code_info_exact field must be added to maintain consistency with the openfda structure.

The reaction.reactionmeddrapt field has inconsistent casing between datasets: it is ALL CAPS in AERS and Not all caps in FAERS.

To ensure that count requests and searches work correctly despite this casing difference, use the reaction.reactionmeddrapt.exact field, which utilizes the upper_case_not_tokenized analyzer as a workaround.

When working with the data extraction and pipeline, adhere to the following naming and formatting rules:

• Naming Convention: All fields must use underscores (_) instead of hyphens (-). This follows the FAERS naming convention.
• Intermediate Formats: Ensure NDC/UPC data is included in the intermediate format, as the extraction process currently moves directly to the index format.
• Boolean Conversions: Consider converting voluntary_mandated values into separate boolean columns (e.g., Firm_Initiated: True).

Before running any scripts located in the scripts/ directory, you must ensure two conditions are met:

1. Bootstrap the environment: You must have already executed bootstrap.sh from the repository root.
2. Execution Context: All scripts assume they are being invoked from the top-level directory of the repository (e.g., by calling ./scripts/name_of_script.sh).

To execute the Python unit tests within the Docker environment, use the test profile with docker-compose:

docker-compose --profile test up test

You can run the entire stack using docker-compose up. This command orchestrates three main components:

1. Elasticsearch container: The data store.
2. API container: Exposes port 8000 for queries.
3. Python 3 container: Runs the following data pipelines to create indices in Elasticsearch:
   • NSDE
   • CAERS
   • Substance Data
   • Device Clearance
   • Device PMA
   • Device Event

Important: The API container starts immediately but will not serve data until the Python pipelines have finished processing. Use curl http://localhost:8000/status to monitor endpoint availability as pipelines progress.

The _mapping.json and _schema.json files serve as inputs to the scripts/generate_fields_yaml.py utility. This script generates a skeleton YAML file used to power the reference documentation pages on the open.fda.gov website. Note that the generated YAML file requires manual editing after generation.

python scripts/generate_fields_yaml.py

To improve the performance of bulk indexing operations in Elasticsearch, set the refresh_interval to 30s. This reduces the frequency of segment refreshes, allowing for higher ingestion throughput.

{
  "settings": {
    "index": {
      "refresh_interval": "30s"
    }
  }
}