comtradeapicall Python Package

repository·main·Indexed 40 days ago

https://github.com/uncomtrade/comtradeapicall

A Python wrapper for UN Comtrade APIs version 1.3.2 that simplifies the extraction of trade and tariff data. It provides functionality to retrieve data into pandas DataFrames, handle bulk downloads, manage asynchronous requests for datasets up to 2.5M records, and check data availability. Key components include Get/Preview for data extraction, BulkDownload for large files, and Metadata for publication notes and reference lists.

Tokens
17.3K
Snippets
52
Records
63
Agent score
50%

What's inside comtradeapicall

  1. Understand the downloaded file name convention

    main

    When downloading data via the UN Comtrade API, files follow a specific naming convention that encodes the data attributes. This allows you to identify the contents of a file without opening it.

    Pattern: COMTRADE-<DATA>-<TYPE><FREQ><COUNTRY CODE><YEAR[<MONTH>]><CLASSIFICATION CODE>[<RELEASE DATE>]

    Components:

    • <DATA>: The type of data (e.g., FINAL or TARIFFLINE).
    • <TYPE>: The data type (e.g., CM for merchandise trade).
    • <FREQ>: Frequency.
    • <COUNTRY CODE>: The numeric country code.
    • <YEAR[<MONTH>]>: The time period.
    • <CLASSIFICATION CODE>: The classification system used (e.g., H1).
    • <RELEASE DATE>: The date the data was released (optional, enclosed in brackets).

    Note: For asynchronous downloads, the original batch ID is retained.

    Examples:
    - Final merchandise trade data from Morocco (code 504) in March 2000 released on 3 Jan 2023 (H1 classification):
      COMTRADE-FINAL-CM504200003H1[2023-01-03]
    - Tariffline merchandise trade from Morocco (code 504) in March 2000 released on 3 Jan 2023 (H1 classification):
      COMTRADE-TARIFFLINE-CM504200003H1[2023-01-03]
  2. Configure the subscription key using environment variables

    main

    The API requires a subscription_key. It is best practice to store this in a .env file and load it using load_dotenv to avoid hardcoding credentials in your scripts.

    from dotenv import load_dotenv
    import os
    
    load_dotenv()
    subscription_key = os.getenv("SUBSCRIPTION_KEY")
  3. Install the comtradeapicall package

    main

    To use the UN Comtrade API in a Python environment (such as a Jupyter kernel), install the comtradeapicall package via pip. It is also recommended to install pandas, matplotlib, load_dotenv, and plotly for data manipulation and visualization.

    # Install the package and common dependencies
    python -m pip install --upgrade comtradeapicall
    python -m pip install pandas matplotlib load_dotenv plotly
  4. Handle Asynchronous Data Requests

    main

    For large or complex queries, you can submit requests asynchronously to avoid timeouts.

    1. Submit Request: Use submitAsyncFinalDataRequest or submitAsyncTarifflineDataRequest. This returns a JSON object containing a requestId.
    2. Check Status: Use checkAsyncDataRequest(subscription_key, batchId='...') to monitor the job progress.
    3. Download Results: Once complete, use downloadAsyncFinalDataRequest or downloadAsyncTarifflineDataRequest to retrieve the resulting file to a specified directory.
    # 1. Submit
    myJson = comtradeapicall.submitAsyncFinalDataRequest(subscription_key, typeCode='C', freqCode='M', clCode='HS', period='202205', reporterCode='36', cmdCode='91,90', flowCode='M', partnerCode=None, partner2Code=None, customsCode=None, motCode=None, aggregateBy=None, breakdownMode='classic')
    request_id = myJson['requestId']
    
    # 2. Check status
    mydf = comtradeapicall.checkAsyncDataRequest(subscription_key, batchId=request_id)
    
    # 3. Download
    comtradeapicall.downloadAsyncFinalDataRequest(subscription_key, directory, typeCode='C', freqCode='M', clCode='HS', period='202209', reporterCode=None, cmdCode='91,90', flowCode='M', partnerCode=None, partner2Code=None, customsCode=None, motCode=None)
  5. How to use grouping logic in API queries

    main

    The UN Comtrade API allows complex queries using specific syntax in the string parameters for reporterCode, partnerCode, cmdCode, and period:

    1. Combining Entities (+): Use the plus sign to treat multiple country codes as a single aggregate group. Example: reporterCode = '392+156' combines Japan and China.
    2. Multiple Queries (,): Use a comma to request data for multiple distinct entities in a single API call. Example: reporterCode = '276, 842' requests data for Germany and the USA separately.
    3. Period Ranges (+): Use a plus sign to combine years. Example: period = '2022+2023'.

    Refer to the official country groups JSON for valid codes.

  6. Optimize API calls by splitting periods

    main

    To avoid API timeouts when requesting monthly data, generate a list of periods and convert them into a single string delimited by commas. This allows the API to process the request in manageable chunks.

    from datetime import datetime
    import pandas as pd
    
    current_year = datetime.now().year
    period_start = f"{current_year - 1}-01-01"
    period_end = f"{current_year}-12-01"
    
    # Generate monthly periods
    periods = pd.date_range(period_start, period_end, freq='MS').strftime("%Y%m").tolist()
    
    # Convert to comma-delimited string
    delim = ","
    temp = list(map(str, periods))
    period_string = delim.join(temp)
  7. Visualize trade surpluses and deficits

    main

    After retrieving trade balance data, you can identify the top 5 surpluses and bottom 5 deficits by sorting the primaryValueBal column. The following pattern demonstrates how to combine these into a single horizontal bar chart using matplotlib.

    import pandas as pd
    import matplotlib.pyplot as plt
    
    # 1. Filter and sort for top/bottom balances (excluding partner 0)
    # Assuming df_TradeBalance is already fetched
    filtered_df = df_TradeBalance[df_TradeBalance['partnerCode'] != 0]
    
    bottom5_deficit = filtered_df.sort_values(by='primaryValueBal', ascending=True).head(5)
    top5_surplus = filtered_df.sort_values(by='primaryValueBal', ascending=False).head(5)
    
    # 2. Combine and prepare for plotting
    combined = pd.concat([bottom5_deficit, top5_surplus], ignore_index=True)
    colors = ['red'] * len(bottom5_deficit) + ['green'] * len(top5_surplus)
    
    # 3. Plot
    plt.figure(figsize=(12, 6))
    combined.plot(x='partnerDesc', y='primaryValueBal', kind='barh', color=colors, legend=False)
    plt.title('Trade Balance Surpluses and Deficits')
    plt.xlabel('Value')
    plt.ylabel('Partner Country')
    plt.tight_layout()
    plt.show()
  8. Async: Submit and download large asynchronous data jobs

    main

    For very large datasets (up to 2.5M records), use the Async component to submit jobs that run in the background and notify you via email upon completion.

    Async Workflow

    1. Submit: Use submitAsyncFinalDataRequest(subscription_key, SelectionCriteria, query_option) or submitAsyncTarifflineDataRequest(...) to start a job.
    2. Check Status: Use checkAsyncDataRequest(subscription_key, [batchId]) to monitor progress.
    3. Download: Use downloadAsyncFinalDataRequest(subscription_key, directory, SelectionCriteria, query_option) or downloadAsyncTarifflineDataRequest(...) to submit, wait, and download the resulting file.