ByteGenie Documentation

website·Indexed 31 days ago

https://handbook.byte-genie.com/README.html

A data processing and document management platform for extracting quantitative and qualitative data from PDFs. ByteGenie provides tools for KPI estimation, document sourcing, web search, structured data QA, and model training data generation. The API supports both sync and async modes for tasks such as PDF-to-markdown conversion, text classification, and layout parsing.

Tokens: 38.4K
Snippets: 186
Records: 217
Agent score: 50%

What's inside ByteGenie

Overview of ByteGenie AI Agent Architecture
ByteGenie allows developers to build specialized AI agents by composing three modular components:
1. Data Blocks: Sourced datasets (e.g., company reports, web search).
2. Model Layers: Foundation models combined with fine-tuned LoRA weights for specific tasks (e.g., data structuring, financial reasoning).
3. Tools: External API integrations (e.g., AWS Textract, Google Translate, code execution).
Agents are defined by configuration files that specify which data objects, model layers, and tools the agent can access. Agents can be toggled on or off to manage service costs.
https://handbook.byte-genie.com/README.html
Extract and structure quantitative metrics from document pages
The process of structuring quants from document pages involves a multi-step pipeline:
1. Extract and structure all quantitative metrics from filtered pages.
2. Score the extracted quants based on their relevance to specific KPIs (e.g., 'GHG Scope 1 emissions', 'gender pay gap').
3. Merge document metadata onto the estimated KPI values.
4. Standardize company names and dates.
5. Sort the final values by standardized company names, relevance score, and dates.
https://handbook.byte-genie.com/document_processing/extract_and_filter_quants_from_pages.html

Understand the synthesized quantitative data structure

The df_syn_quants data structure contains synthesized quantitative data extracted from documents. Each record includes metadata about the source document, the specific page, the extracted value, and the context in which it was found.

# Columns available in the synthesized quantitative data frame
columns = [
    'category',
    'company name',
    'company_verification_flag',
    'context',
    'date',
    'doc_name',
    'doc_org',
    'doc_type',
    'doc_year',
    'embedding',
    'fuzzy_verification_flag',
    'lm_verification_flag',
    'pagenum',
    'relevant quote',
    'relevant quote from text',
    'row_id',
    'unit',
    'value',
    'variable',
    'variable description'
]

Restructure extracted table data into a dataset

Use the create_dataset method to transform raw table data (such as a pandas DataFrame converted to records) into a structured dataset by specifying the desired attributes to extract. The result can be read back into a pandas DataFrame for further analysis.

# Convert pandas DataFrame to records and extract specific attributes
resp = bg_async.create_dataset(
    data=df_table.to_dict('records'),
    attrs=['product category', 'complete product description', 'key material used in the product',
           'per unit amount', 'phase 1 amount', 'phase 2 amount', 'currency']
)

# Check if the output file is ready and read the data
if resp.check_output_file_exists():
    df_dataset = resp.read_output_data()
    df_dataset = pd.DataFrame(df_dataset)
else:
    logger.info("create_dataset() output is not yet ready: wait some more")

Standardize KPI names using text classification

Because extracted quantity names can vary, use async_classify_texts to map raw quantity descriptions to a predefined list of KPI labels. This ensures consistency across different documents and companies.

# Create a unique description for classification
df_estimated_kpi['full_description'] = [
    f"{row.to_dict()}"
    for row_num, row in list(df_estimated_kpi[['quantity name', 'quantity description']].iterrows())
]

# Define and run classification task
tasks = [
    bg_async.async_classify_texts(
        texts=df_estimated_kpi['full_description'].unique().tolist(),
        labels=kpis, # List of target KPI labels
        multi_class=0
    )
]

# Process results
results = utils.async_utils.run_async_tasks(tasks)
df_name_std = pd.concat([pd.DataFrame(resp.get_output()) for resp in results])

# Keep only the label with the maximum score per sequence
df_name_std = df_name_std.groupby('sequence').apply(
    lambda x: x[x['scores'] == x['scores'].max()]
).reset_index(drop=True)

df_name_std = df_name_std.rename(
    columns={'sequence': 'full_description', 'labels': 'kpi', 'scores': 'score'}
)

# Merge back to main dataframe
df_estimated_kpi = pd.merge(
    left=df_estimated_kpi, 
    right=df_name_std, 
    on=['full_description'], 
    how='left'
)

Perform OCR to extract text and tables

Run OCR on documents to extract text blocks and table cells. Use extract_text to trigger the process. Extracted text is stored as CSVs under variable_desc=text-blocks/**.csv and table cells under variable_desc=table-cells/**.csv.

# Trigger OCR
for doc_name in doc_names:
    resp = bg_async.extract_text(doc_name=doc_name)

# List OCR text output files
tasks = [
    bg_sync.async_list_doc_files(
        doc_name=doc_name,
        file_pattern="variable_desc=text-blocks/**.csv"
    )
    for doc_name in doc_names
]
ocr_text_files = utils.async_utils.run_async_tasks(tasks)

# List OCR table cell output files
tasks = [
    bg_sync.async_list_doc_files(
        doc_name=doc_name,
        file_pattern="variable_desc=table-cells/**.csv"
    )
    for doc_name in doc_names
]
ocr_table_files = utils.async_utils.run_async_tasks(tasks)

List and read extracted text segment files

After documents are processed, you can retrieve the OCR and layout parsing results. Text segments are stored in CSV files, with one file generated per page. Use list_doc_files with a specific file_pattern to find these files, and read_file to load the data into a pandas DataFrame.

import pandas as pd

doc_names = ['document_id_1', 'document_id_2']
text_segment_files = []

for doc_name in doc_names:
    # List files matching the text-blocks pattern
    files = bg_sync.list_doc_files(
        doc_name=doc_name,
        file_pattern='variable_desc=text-blocks/**.csv',
    ).get_data()
    text_segment_files.extend(files)

# Read a specific text segment file into a DataFrame
df_text_segment = bg_sync.read_file(text_segment_files[0]).get_data()
df_text_segment = pd.DataFrame(df_text_segment)

Upload PDF files to ByteGenie

To upload PDF files, use the async_upload_data method. This method expects a list of file contents and a list of corresponding filenames. When processing multiple files, it is recommended to wrap these calls in async tasks and execute them using a utility like run_async_tasks.

import utils.common
import utils.async_utils

# 1. Get file contents from a folder
pdf_folder = "/tmp/PDF"
df_contents = utils.common.read_file_contents(directory=pdf_folder)

# 2. Create upload tasks
tasks = [
    bg_async.async_upload_data(
        contents=[df_contents['content'].tolist()[i]],
        filenames=[df_contents['filename'].tolist()[i]],
        username=bg_sync.read_username(),
    )
    for i in range(0, len(df_contents), 1)
]

# 3. Execute tasks
upload_responses = utils.async_utils.run_async_tasks(tasks)

Filter extracted document details for cement company analysis

After extracting document details into a DataFrame (df_doc_details), you can filter for high-quality, relevant reports by checking the document year, document type (e.g., annual or sustainability reports), and minimum page count.

import pandas as pd

# Read extracted data
df_doc_details = pd.read_csv("/tmp/doc-details_cement-companies.csv")

# Filter for reports after 2021, specific types, and at least 20 pages
df_doc_details = df_doc_details[
    (df_doc_details['doc_year_num'] > 2021) &
    (df_doc_details['doc_type'].str.contains('annual report|sustainability report')) &
    (df_doc_details['num_pages'] >= 20)
]

Rank document data by relevance using keyphrases in byte-genie

To filter the most relevant portions of extracted document data for downstream analysis, use the rank_data method. This process involves defining a set of quantitative and qualitative keyphrases and iterating through documents to rank data based on these attributes. The frac_rows_to_keep parameter controls the fraction of the most relevant rows to retain in the ranked output.

# Define keyphrases for ranking
keyphrases = {
    'quantitative': ['cement production', 'revenue', 'emissions by scope', 'emission intensity', 'energy consumption'],
    'qualitative': ['emission reduction measures', 'revenue growth projection', 'business risk', 
                    'cement industry trends', 'decarbonisation plans', 'climate risks', 'climate resilience']
}
frac_rows_to_keep = 0.1

# Iterate through documents and keyphrases to rank data
for doc_name in doc_names:
    for keyphase_type, phrases in keyphrases.items():
        for keyphrase in phrases:
            resp = bg_async.rank_data(
                doc_name=doc_name,
                attr=keyphrase,
                attr_type=keyphase_type,
                frac_rows_to_keep=frac_rows_to_keep,
            )
            
            # Verify and retrieve the output file
            if resp.check_output_file_exists():
                output_file = resp.get_output_file()

Read and process ranked text files

After ranking text data, use async_read_files to retrieve the content of the ranked files. The resulting data can be converted into a pandas DataFrame. To recover document metadata (like page numbers and document names) from the file paths, you can parse the file column using string splitting.

import pandas as pd

# Read the ranked files
tasks = [
    bg_sync.async_read_files(
        files=text_ranking_files,
        add_file=1,
    )
]
df_text_ranked_responses = utils.async_utils.run_async_tasks(tasks)

# Convert responses to a single pandas DataFrame
df_text_ranked = pd.concat([pd.DataFrame(resp.get_output()) for resp in df_text_ranked_responses])

# Extract page number from the file path (e.g., ..._pagenum-45_...)
df_text_ranked['pagenum'] = [
    int(file.split('_pagenum-')[-1].split('_')[0])
    for file in df_text_ranked['file']
]

# Extract document name from the file path (e.g., entity=userid_...)
df_text_ranked['doc_name'] = [
    file.split('entity=')[-1].split('/')[0]
    for file in df_text_ranked['file']
]

Retrieve and verify uploaded KPI data from the cloud

After uploading data via upload_data, you can retrieve the cloud storage path (href) from the response output. Use the read_file method of the byte-genie client to read the data back from the cloud and verify its integrity (e.g., by comparing row counts with the original local DataFrame).

import pandas as pd

# Get the cloud file path from the upload response
df_estimated_kpi_uploaded = pd.DataFrame(upload_resp.get_output())
estimated_kpi_file_path = df_estimated_kpi_uploaded['href'].tolist()[0]

# Read the file back from the cloud
saved_estimated_kpis = bg_sync.read_file(estimated_kpi_file_path).get_output()
saved_estimated_kpis = pd.DataFrame(saved_estimated_kpis)

# Verify row count matches original data
print(len(saved_estimated_kpis) == len(df_estimated_kpi))