iText2KG

iText2KG is a Python package for incrementally constructing consistent knowledge graphs (KGs) from unstructured text using LLMs. It uses a zero-shot approach to extract entities and relations, resolves ambiguities, and integrates the results into Neo4j for visualization.

The architecture consists of four core modules:

1. Document Distiller: Reformulates raw documents into semantic blocks based on a user-defined schema to improve the signal-to-noise ratio.
2. Incremental Entity Extractor: Extracts unique entities from semantic blocks and resolves ambiguities using cosine similarity to match local entities with global entities.
3. Incremental Relation Extractor: Identifies relationships between entities. It supports two modes: using global entities to enrich the graph or using local entities for higher precision.
4. Graph Integrator and Visualization: Integrates extracted data into a Neo4j database for interactive exploration.

ATOM (AdapTive and OptiMized Dynamic Temporal Knowledge Graph Construction Using LLMs) is a scalable approach for building and continuously updating Dynamic Temporal Knowledge Graphs (DTKGs) from unstructured text.

Unlike static KG construction, ATOM uses a three-module parallel pipeline to handle the dynamic and time-sensitive nature of data:

1. Module-1 (Atomic Fact Decomposition): Splits input documents into minimal, self-contained "atomic facts" (ideally <400 tokens) to ensure LLMs don't omit facts in long contexts.
2. Module-2 (Atomic TKGs Construction): Extracts 5-tuples (subject, predicate, object, t_start, t_end) in parallel from atomic facts. It uses dual-time modeling to distinguish between when a fact was observed (t_obs) and its validity period (t_start/t_end).
3. Module-3 (Parallel Atomic Merge): Uses a binary merge algorithm to iteratively merge atomic TKGs in parallel. It resolves entities (using cosine similarity with threshold $\theta_E = 0.8$), relations (threshold $\theta_R = 0.7$), and temporal validity periods.

The final result is a DTKG that evolves as new information is observed.

ATOM uses dual-time modeling to prevent temporal misattribution. It distinguishes between:

• t_obs: The date/time the information was actually observed (e.g., the publication date of a news article).
• t_start / t_end: The actual validity period of the fact being described.

Example Workflow:

1. Observation 1: "Steve Jobs was the CEO of Apple Inc. on January 9, 2007" $\rightarrow$ (Steve Jobs, is_ceo, Apple Inc., [09-01-2007], [.]).
2. Observation 2: "Steve Jobs is no longer the CEO of Apple Inc. on 05-10-2011".
3. Transformation: Module-2 transforms the "no longer" fact into an affirmative end-validity fact: (Steve Jobs, is_ceo, Apple Inc., [.], [05-10-2011]).
4. Resolution: Module-3 merges these into a single 5-tuple: (Steve Jobs, is_ceo, Apple Inc., [09-01-2007], [05-10-2011]).

This prevents the error of assuming an event happened exactly at the time the text was written, which is a common issue in other frameworks like Graphiti.

To build a Knowledge Graph that evolves over time, process time-series data (like news or social media posts) incrementally.

The Workflow:

1. Initialize DocumentDistiller and iText2KG_Star.
2. For the first document, extract facts and build the initial KnowledgeGraph using build_graph.
3. For every subsequent document:
   • Extract facts using DocumentDistiller.
   • Call iText2KG_Star.build_graph again, but this time pass the existing_knowledge_graph (using kg.model_copy()) and the new observation_date.
4. The resulting graph will contain relationships with observation_dates tracking when they were first seen.

# Incremental update loop snippet
for i in range(1, len(time_series_data)):
    facts = await document_distiller.distill(
        documents=[time_series_data[i]['content']], 
        IE_query=IE_query, 
        output_data_structure=Facts
    )
    
    # Update the existing KG with new information
    kg = await itext2kg_star.build_graph(
        sections=facts.facts,
        observation_date=time_series_data[i]['observation_date'],
        existing_knowledge_graph=kg.model_copy(),
        ent_threshold=0.8,
        rel_threshold=0.7
    )

This example demonstrates the full workflow: initializing LangChain models, preparing a dictionary of facts mapped to timestamps, building the TKG using build_graph_from_different_obs_times, and visualizing the result in Neo4j.

⚠️ Performance Note: Avoid running ATOM in Jupyter notebooks. Use dedicated Python scripts to prevent significant slowdowns caused by event loop conflicts and thread contention in notebook environments.

import pandas as pd
import asyncio
import ast
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from itext2kg.atom import Atom
from itext2kg import Neo4jStorage

# 1. Setup Models
openai_api_key = "#"
openai_llm_model = ChatOpenAI(
    api_key=openai_api_key,
    model="gpt-4.1-2025-04-14",
    temperature=0,
)

openai_embeddings_model = OpenAIEmbeddings(
    api_key=openai_api_key,
    model="text-embedding-3-large",
)

# 2. Prepare Data
news_covid = pd.read_pickle("../datasets/atom/nyt_news/2020_nyt_COVID_last_version_ready.pkl")

def to_dictionary(df:pd.DataFrame, max_elements: int | None = 20): 
    if isinstance(df['factoids_g_truth'][0], str):
        df["factoids_g_truth"] = df["factoids_g_truth"].apply(lambda x:ast.literal_eval(x))
    grouped_df = df.groupby("date")["factoids_g_truth"].sum().reset_index()[:max_elements]
    return {
        str(date): factoids for date, factoids in grouped_df.set_index("date")["factoids_g_truth"].to_dict().items()
        }

news_covid_dict = to_dictionary(news_covid)

# 3. Run ATOM
atom = Atom(llm_model=openai_llm_model, embeddings_model=openai_embeddings_model)

# Note: build_graph_from_different_obs_times is an async function
kg = await atom.build_graph_from_different_obs_times(
    atomic_facts_with_obs_timestamps=news_covid_dict,
)

# 4. Visualize
URI = "bolt://localhost:7687"
USERNAME = "neo4j"
PASSWORD = "##"
Neo4jStorage(uri=URI, username=USERNAME, password=PASSWORD).visualize_graph(knowledge_graph=kg)