DOCSEARCHbyAlgolia
Submit Docs

datamule-python

repository·main·Indexed 19 days ago

https://github.com/john-friedman/datamule-python

A Python package for working with SEC filings at scale. It provides tools to download and manage large datasets of SEC submissions via the SEC API or the datamule-tar hosted archive to bypass rate limits. Key features include the Portfolio class for ticker-based downloads, the Index class for searching filings with metadata filters, and the Book class for downloading pre-built datasets and performing S3 transfers of SEC documents.

Tokens
38.5K
Snippets
98
Records
131
Agent score
67%

What's inside datamule-python

  1. Overview of the datamule ecosystem

    main

    The datamule project is a suite of tools designed to simplify the manipulation and processing of Securities and Exchange Commission (SEC) data. While datamule-python is the primary Python package for programmatic access, the ecosystem includes specialized tools for data storage, parsing, indicator creation, and document conversion.

    Key components include:

    • datamule-python: The core Python package for interacting with SEC data.
    • datamule-data: A daily-updating data repository used to keep packages current.
    • datamule-indicators: Tools to derive financial indicators from SEC data.
    • secsgml: A parser for SEC Standardized Generalized Markup Language.
    • doc2dict: A utility to convert documents into dictionary formats.
    • txt2dataset: A tool to transform unstructured text into datasets.
    • secxbrl: A high-performance, lightweight parser for SEC Inline XBRL.
    • company-fundamentals: Standardizes SEC XBRL data into fundamental metrics like EBITDA.
    • secbrowser: A Flask-based web interface for interacting with SEC filings.

  2. Overview of datamule-python core classes

    main

    The datamule-python package provides programmatic access to SEC filings and datamule's proprietary data layers. The core functionality is organized into several specialized classes:

    • Portfolio: Used for interacting with SEC filings. It supports downloading data via the SEC directly or through the datamule archive. Note that using the datamule archive or the datamule websocket requires a datamule API key.
    • Sheet: Provides programmatic access to datamule's databases, such as ownership or institutional holdings. This requires an API_KEY.
    • Book: Provides access to datamule's S3 layer (e.g., extracting text from filings). This requires an API_KEY. It includes a utility function s3_transfer() for local machine execution to copy files from presigned URLs into your own S3 bucket.
    • Index: Provides programmatic access to search the SEC by keyword.
    • Cloud: Provides access to various miscellaneous datamule APIs. This requires an API_KEY.

  6. Filter submissions in a portfolio

    main

    You can apply filters to a portfolio to restrict which submissions are downloaded. Filters can be chained.

    Text Filtering: Use filter_text(text_query, ...) to filter submissions based on text content (e.g., searching for "climate change").

    XBRL Filtering: Use filter_xbrl(taxonomy, concept, unit, period, logic, value) to filter based on XBRL data.

    • taxonomy: e.g., dei, us-gaap.
    • concept: e.g., AccountsPayableCurrent.
    • unit: e.g., USD.
    • period: e.g., CY2019Q4I.
    • logic: Comparison operators: '>', '>=', '==', '!=', '<', '<='.
    # Chaining filters
portfolio.filter_text('"climate change"', filing_date=('2019-01-01', '2019-01-31'), submission_type='10-K')
portfolio.filter_text('drought', filing_date=('2019-01-01', '2019-01-31'), submission_type='10-K')

# Download only submissions that match all chained filters
portfolio.download_submissions(filing_date=('2019-01-01', '2019-01-31'), submission_type='10-K')

  7. Understand the Submission class

    main

    The Submission class represents an SEC filing (derived from the <SUBMISSION> tag in SEC SGML). It provides access to filing metadata, document paths, and parsed financial data.

    Key Attributes:

    • submission.path: The path to the submission.
    • submission.accession: The submission accession number.
    • submission.filing_date: The date the submission was filed.
    • submission.metadata: A dictionary containing additional metadata.
    • submission.xbrl: Parsed XBRL data (lazy-loaded).
    • submission.fundamentals: All fundamental financial data (lazy-loaded).

  8. Access financial statement categories dynamically

    main

    You can access specific financial statement categories directly as attributes on the submission.fundamentals object. The system dynamically attempts to match attribute names to the available categories in the fundamentals data.

    Common categories include:

    • balanceSheet
    • incomeStatement
    • CashFlowStatement
    # All of these work automatically
balance_sheet = submission.fundamentals.balanceSheet
income_stmt = submission.fundamentals.incomeStatement  
cash_flow = submission.fundamentals.CashFlowStatement

  9. Choose between SEC and datamule data providers

    main

    datamule supports two primary data providers depending on your speed and budget requirements:

    1. SEC: Direct access to SEC data. This provider is rate-limited to 5 requests per second, which may result in long download times (e.g., ~10 days for a full archive).
    2. datamule: Managed access via the datamule service. This provider has no rate limits and is significantly faster (e.g., ~1 hour for a full archive), but incurs a convenience fee.

  10. Parse result metadata from Index.search_submissions()

    main

    Each result returned by search_submissions() is a dictionary containing metadata and the document source.

    • _id: A string typically formatted as accession:filename. You can split this string at the first colon to separate the accession number from the filename.
    • _source: A dictionary containing the actual document fields. Common fields include file_date.
    # Example of parsing a result dictionary
doc_id = result['_id']
accession, filename = doc_id.split(':', 1)

filing_date = result['_source'].get('file_date', '')

  11. Use lazy loading for XBRL and fundamentals

    main

    The Submission class uses lazy loading for xbrl and fundamentals to optimize performance. You do not need to call parsing methods manually; the system triggers parsing automatically upon the first access to these attributes and caches the result for subsequent calls.

    • Accessing submission.xbrl automatically calls parse_xbrl().
    • Accessing submission.fundamentals automatically calls parse_fundamentals().