DOCSEARCHbyAlgolia
Submit Docs

financial-datasets

repository·main·Indexed 19 days ago

https://github.com/virattt/financial-datasets

A Python library (v0.1.18) designed to automate the creation of high-quality question-and-answer datasets from financial documents, including 10-Ks, 10-Qs, and PDFs, using LLMs. It provides a DatasetGenerator for creating Q&A pairs and a FilingParser for extracting specific SEC filing items. The library uses Pydantic models (Dataset and DatasetItem) to structure data into question, answer, and context formats suitable for LLM evaluation or fine-tuning.

Tokens
4K
Snippets
20
Records
22
Agent score
56%

What's inside financial-datasets

  1. Understand the generated dataset format

    main

    The library returns a list of dictionaries. Each dictionary represents a single question-answer pair and includes the following keys:

    • question: The generated question.
    • answer: The answer to the question.
    • context: The specific snippet of text from the source used to derive the question and answer.
    [
  {
    "question": "What was Airbnb's revenue in 2023?",
    "answer": "$9.9 billion",
    "context": "In 2023, revenue increased by 18% to $9.9 billion..."
  }
]

  2. Install the financial-datasets library

    main

    You can install the library using pip or Poetry. To install directly from the source repository, clone the repo and use poetry install.

    # Using pip
pip install financial-datasets

# Using Poetry
poetry add financial-datasets

  3. Generate a dataset from a 10-K filing

    main

    Use the generate_from_10K method to automatically fetch and process a company's 10-K filing using a ticker and year. You can optionally specify item_names (e.g., ["Item 1", "Item 7"]) to narrow the scope of the generation.

    from financial_datasets.generator import DatasetGenerator

generator = DatasetGenerator(model="gpt-4-turbo", api_key="your-openai-key")

dataset = generator.generate_from_10K(
    ticker="AAPL",
    year=2023,
    max_questions=100,
    item_names=["Item 1", "Item 7"],  # optional
)

  4. Generate a dataset from a list of texts

    main

    Use the generate_from_texts method to create a dataset from a provided list of strings. This is the most flexible method for custom text sources.

    from financial_datasets.generator import DatasetGenerator

texts = ["your text 1", "your text 2"]
generator = DatasetGenerator(model="gpt-4-turbo", api_key="your-openai-key")

dataset = generator.generate_from_texts(
    texts=texts,
    max_questions=100,
)

  5. Generate a dataset from a PDF URL

    main

    Use the generate_from_pdf method to ingest a financial document directly from a web URL.

    from financial_datasets.generator import DatasetGenerator

generator = DatasetGenerator(model="gpt-4-turbo", api_key="your-openai-key")

dataset = generator.generate_from_pdf(
    url="https://www.berkshirehathaway.com/letters/2023ltr.pdf",
    max_questions=100,
)

  6. Initialize the DatasetGenerator

    main

    To begin generating datasets, instantiate the DatasetGenerator class from financial_datasets.generator. You must provide the LLM model name and your api_key.

    from financial_datasets.generator import DatasetGenerator

generator = DatasetGenerator(model="gpt-4-turbo", api_key="your-openai-key")

  7. Generate dataset from a PDF URL

    main

    Use generate_from_pdf to download a PDF from a URL, extract its text, and generate questions.

    Parameters:

    • url (str): The direct URL to the PDF file.
    • max_questions (int): Maximum questions to generate.
    • **kwargs:
      • chunk_size (int): Size of text chunks for splitting. Defaults to 1024.
      • chunk_overlap (int): Overlap between chunks. Defaults to 100.
      • system_prompt: Custom system prompt.

    Returns:

    • Dataset: A collection of generated DatasetItem objects.
    url = "https://example.com/report.pdf"
dataset = generator.generate_from_pdf(url=url, max_questions=15, chunk_size=512)

  8. Initialize DatasetGenerator

    main

    To use the DatasetGenerator, instantiate it with a supported OpenAI model (must start with gpt-) and your OpenAI API key. This class is the primary entry point for generating Q&A datasets from various text sources.

    from financial_datasets.generator import DatasetGenerator

generator = DatasetGenerator(model='gpt-4o', api_key='your-api-key')

  9. Extract items from 10-K filings using FilingParser.get_10K_items

    main

    Use FilingParser.get_10K_items to retrieve specific text sections (Items) from a company's annual 10-K filing.

    Parameters:

    • ticker (str): The stock ticker symbol (required).
    • year (int): The filing year (required).
    • item_names (List[str], optional): A list of specific FilingItem names to retrieve. If empty, all available items in the filing are returned.
    • sec_identity (str, optional): The identity string used for SEC API requests. Defaults to "gary gary@financialdatasets.org".

    Returns:

    • A List[str] containing the cleaned text of the requested items.

    Note on Cleaning: The returned text is automatically cleaned: newlines are replaced with spaces, outer whitespace is trimmed, and sequences of 3+ dashes/dots or 2+ plus signs are removed.

    from financial_datasets.parser import FilingParser

parser = FilingParser()
items = parser.get_10K_items(
    ticker="AAPL",
    year=2023,
    item_names=["Item 1", "Item 1A"]
)
print(items)

  10. Generate dataset from SEC 10-Q filings

    main

    Use generate_from_10Q to fetch and process a quarterly SEC filing (10-Q).

    Parameters:

    • ticker (str): The stock ticker symbol.
    • year (int): The filing year.
    • quarter (int): The quarter (1, 2, 3, or 4).
    • max_questions (int): Maximum questions to generate.
    • sec_identity (str): Identity for SEC API requests. Defaults to default_sec_identity.
    • **kwargs:
      • item_names (List[str]): Specific SEC item names to extract.
      • chunk_size (int): Text chunk size. Defaults to 1024.
      • chunk_overlap (int): Chunk overlap. Defaults to 100.
      • system_prompt: Custom system prompt.

    Returns:

    • Dataset: A collection of generated DatasetItem objects.
    dataset = generator.generate_from_10Q(
    ticker='TSLA', 
    year=2023, 
    quarter=2, 
    max_questions=10
)

  11. Define a dataset using Dataset and DatasetItem

    main

    The financial-datasets library uses Pydantic models to structure financial question-answering data.

    • DatasetItem: Represents a single data point containing a question, an answer, and the context used to derive them.
    • Dataset: A container that holds a list of DatasetItem objects.

    You can instantiate these models directly to create structured datasets for training or evaluation.

    from financial_datasets.dataset import Dataset, DatasetItem

item = DatasetItem(
    question="What was the revenue in 2023?",
    answer="$10 billion",
    context="In 2023, the company reported a total revenue of $10 billion."
)

dataset = Dataset(items=[item])

  12. Filter filings using filter_filings()

    main

    The filter_filings function searches through a list of EntityFilings objects to find the first filing that matches a specific form type and year.

    Parameters:

    • filings: A list of EntityFilings objects. Each object must have a form attribute (string) and a report_date attribute (string in %Y-%m-%d format).
    • form: The specific form type to match (e.g., '10-K').
    • year: The integer year to match.

    Returns:

    • The first EntityFiling object that matches both the form and the year.

    Errors:

    • Raises ValueError if no filing in the provided list matches the specified form and year.
    from financial_datasets.filings import filter_filings

# Assuming 'filings_list' is a list of EntityFiling objects
try:
    target_filing = filter_filings(filings_list, form="10-K", year=2023)
    print(f"Found filing: {target_filing.report_date}")
except ValueError as e:
    print(e)