DOCSEARCHbyAlgolia
Submit Docs

edgartools

repository·main·Indexed 25 days ago

https://github.com/dgunning/edgartools

A Python library that transforms SEC EDGAR filings into structured Python objects and pandas DataFrames. It simplifies parsing XBRL data and SEC forms (10-K, 10-Q, Form 4, 13F, etc.) for financial analysis and AI pipelines. Features include an HTML parser with performance profiles, BDC reference data and DERA bulk dataset analysis, and AI-optimized integrations via Model Context Protocol (MCP) servers and specialized skills for Claude Desktop.

Tokens
319.3K
Snippets
763
Records
1.2K
Agent score
77%

What's inside edgartools

  1. Overview of the training package structure

    main

    The edgar/entity/training/ package contains the following core components:

    • run_learning.py: The canonical script for running the learning pipeline.
    • run_industry_learning.py: Used for industry-specific learning tasks.
    • deploy.py: Used to deploy learned data to the edgar/entity/data/ directory for library use.
    • view.py: Used to view the results of the learning process.
    • training/output/: The directory where generated JSON outputs are stored (gitignored).

  2. Overview of EdgarTools features and capabilities

    main

    EdgarTools is an AI-native platform for SEC filing analysis. It provides standardized, analysis-ready data from various filing types.

    Supported Filing Types

    • Annual Reports: 10-K, 20-F, 40-F
    • Quarterly Reports: 10-Q
    • Current Reports: 8-K, 6-K
    • Insider Transactions: Forms 3, 4, 5
    • Fund Holdings: 13F-HR
    • Proxy Statements: DEF 14A
    • Registration Statements: S-1, S-3, S-4, S-8
    • Other: 30+ other form types

    Key Capabilities

    • Standardized Financials: Access metrics like get_revenue() or get_net_income() that are automatically standardized across different companies (e.g., mapping various revenue labels to a single metric).
    • Data Extraction: Extract Balance Sheets, Income Statements, and Cash Flow statements; access XBRL facts with dimensional breakdowns; and perform clean text extraction from HTML sections (e.g., Risk Factors, MD&A).
    • AI/MCP Integration: Native support for AI agents with token-optimized outputs and specialized tools like edgar_company_research and edgar_analyze_financials.
    • Developer Experience: Built-in support for type hints, IntelliSense, Jupyter notebook rich displays, and automatic conversion to pandas DataFrames.

  3. Understanding the EdgarTools Verification Gap Analysis

    main

    The Verification Gap Analysis is a strategic document that identifies discrepancies between the current state of the edgartools testing infrastructure and the project's core principles. It categorizes gaps into several key areas: documentation verification, data correctness, user experience, SEC upstream changes, continuous integration efficiency, silence/warning handling, SEC breadth, API solvability, tiered testing spend, cassette (VCR) management, and the definition of done.

    Key areas of concern include:

    • Unverified Documentation: Code examples in documentation are not automatically verified against the actual API.
    • SEC Upstream Changes: There is currently no mechanism to distinguish between library bugs and changes in the SEC's data format.
    • Testing Scalability: The current testing model relies on linear network time, which becomes unsustainable as the codebase grows.
    • API Solvability: While an evaluation framework exists, it is not integrated into the CI pipeline to ensure users (human or agent) can successfully navigate the API from question to answer.

  4. EdgarTools feature overview

    main

    EdgarTools provides a comprehensive suite of features for SEC data analysis:

    FeatureDescription
    Company DataLook up any public company by ticker or CIK
    Filing AccessSearch and filter SEC filings by form type, date, company
    Financial StatementsParsed income statement, balance sheet, cash flow
    XBRL DataStructured access to all XBRL facts and dimensions
    Insider TradingParse Forms 3, 4, 5 for ownership transactions
    Fund Holdings13F institutional holdings reports
    Document ExtractionExtract text and exhibits from filings
    Local CachingIntelligent caching reduces API calls
    Rate LimitingAutomatic compliance with SEC limits

  5. Understand the XBRL Calculation Linkbase Parser design

    main

    The XBRL Calculation Linkbase Parser is designed to process mathematical relationships in XBRL calculation linkbases. These linkbases define how numeric values in financial statements sum up (e.g., how individual line items add up to a total).

    Core Workflow:

    1. Input: An XBRL calculation linkbase file (typically _cal.xml) and a reference to a taxonomy schema file to resolve element references.
    2. Processing: The parser extracts Extended Link Roles (ELRs), processes calculation arcs (relationships), and builds calculation trees.
    3. Output: A CalculationNetwork containing organized trees for each role, which can be used to validate the mathematical consistency of XBRL facts against instance data.

  6. Understand the edgar.ai package structure

    main

    The edgar.ai package is organized into several functional modules designed for AI-driven SEC filing analysis:

    • core.py: Contains the AI infrastructure, including TokenOptimizer for cost/context management, SemanticEnricher for business context, and the AIEnabled base class.
    • helpers.py: Provides high-level convenience wrappers for common analytical tasks like revenue trends and industry filtering.
    • mcp/: Implements the Model Context Protocol (MCP), allowing the library to function as a server with specialized tools for company, financial, and industry analysis.
    • skills/: The core logic for agentic behavior, containing the BaseSkill abstraction and the EdgarToolsSkill implementation.
    • exporters/: Utilities to export capabilities into formats compatible with AI interfaces, such as official Claude Skills or portable ZIP formats for Claude Desktop.
    • examples/: Contains reference implementations like basic_usage.py.

  7. Core components of the Investigation Toolkit

    main

    The toolkit consists of several specialized tools:

    • investigation_toolkit.py: The core analysis engine with pattern detection. Can be imported as a module.
    • quick_investigate.py: Used for rapid issue investigation and comparison.
    • create_reproduction.py: Generates reproduction scripts from existing templates.
    • visual_inspector.py: Provides rich visual inspection of statements, dataframes, and XBRL. Can be imported as a module.
    • quick_debug.py: An instant visual debugging tool for maintainers.

  8. Understand the EdgarTools Verification Gap Analysis

    main

    The Verification Gap Analysis is a principle-based assessment of the EdgarTools test suite against a 'Verification Constitution'. It identifies structural weaknesses in how the library ensures data correctness, documentation accuracy, and coverage across the diverse landscape of SEC filings.

    Key areas analyzed include:

    • Documentation vs. Specification: The decoupling of documented examples from actual test execution.
    • Data Correctness: The reliance on embedded ground-truth values in test code rather than externalized reference data.
    • User Experience: The tendency for tests to focus on internal implementation details (like DataFrame shapes) rather than user-facing behavior.
    • Upstream Reliability: The difficulty in distinguishing between library bugs and changes in SEC data/formats.
    • Continuous Verification: The challenges of maintaining a fast, scalable test suite as the project grows.
    • Failure Modes: The asymmetry in testing (verifying that data exists vs. verifying that errors/warnings surface correctly).
    • Coverage Breadth: The concentration of tests on specific large-cap tech companies (e.g., Apple) and standard forms, leaving international filers and edge-case forms unverified.

  9. Understand the XBRL Instance Document Parser design

    main

    The XBRL Instance Document Parser is designed to extract structured financial data from XBRL instance files (XML). It transforms raw XML input into a structured Parsed Data Model.

    Input:

    • An XBRL instance document (XML file, typically .xml).
    • Optional: Reference to associated taxonomy files for validation and context.

    Output:

    • A structured collection containing:
      • Entity information: Identity of the reporting company.
      • Reporting period information: Timeframes covered.
      • Contexts: The relationship between entities and periods.
      • Facts: The actual data points (values).
      • Units: Measurement scales (e.g., USD, shares).
      • Footnotes: Explanatory text linked to specific facts.

  10. Access SEC reference data with edgartools

    main

    EdgarTools provides access to a comprehensive set of SEC reference data, including ticker-to-CIK mappings, exchange listings, industry classifications (SIC), CUSIP lookups, place codes, and form descriptions.

    Most of this data is bundled with the package and works offline with zero configuration. For the full SEC ticker universe (including recent IPOs and mutual funds), you can perform a one-time local download to enable complete offline capabilities.

  11. Understand the EdgarTools test coverage and module status

    main

    The project maintains extensive test coverage across its core modules. Use the coverage map to understand which modules are well-tested and which may have lower reliability due to a lack of dedicated tests.

    Well-Covered Modules

    • Filings & Metadata: _filings.py (via test_filing.py, etc.)
    • Entities: entity/ and entity/entity_facts.py (using Network and VCR)
    • XBRL: xbrl/ (including statements, standardization, and stitching)
    • Documents: documents/ (HTML parsing and section detection)
    • Ownership: ownership/ and `beneficial_ownership/
    • Funds & Reference: funds/ and `reference/
    • Search & Proxy: search/ and `proxy/
    • AI/MCP: ai/mcp/ (using Async and mocks)

    Modules Without Dedicated Tests

    Use caution when working with these modules as they lack dedicated test files:

    • earnings.py
    • financials.py
    • ai/formats.py (Experimental)
    • ai/helpers.py (Experimental)
    • documents/migration.py (One-time utility)
    • standardization.py (Legacy; use xbrl/standardization/ instead)

    Regression Testing

    Regression tests are stored in tests/issues/regression/. Each file is tied to a specific GitHub issue and is marked with @pytest.mark.regression. Examples include test_cost_of_revenue_fix.py and test_bug_408_annual_periods.py.

  12. Use edgar.tools web UI for a no-code experience

    main

    If you prefer a visual interface over writing Python code, you can use edgar.tools. It provides the same data available in the Python library through a web UI, including:

    • Browsing company filings and financials.
    • Real-time filing updates.
    • Exploring insider trades with sentiment analysis.

    The service includes a free tier, a REST API, and a hosted MCP server for AI integrations.