repository·main·Indexed 25 days ago
https://github.com/yifanfeng97/hyper-extractAn LLM-powered knowledge extraction and evolution framework (version 0.3.0) that transforms unstructured text into structured, strongly-typed Knowledge Abstracts, including Graphs, Hypergraphs, and Spatio-Temporal Graphs. It features a CLI and Python API for automating extraction, semantic search capabilities via FAISS vector indexing, and support for multiple LLM providers including OpenAI, Bailian, and local vLLM instances. Includes tools for exporting knowledge to Obsidian vaults and an MCP server for agent integration.
The hyperextract-skills package provides specialized skills for designing knowledge templates.
| Skill | Purpose |
|---|---|
| Root | Entry point for template design |
| brainstorm | Requirements exploration and type discussion |
| record-designer | Design model/list/set structures |
| graph-designer | Design graph/hypergraph/etc structures |
| yaml-validator | Validate YAML configurations |
| multilingual | Convert to multi-language support |
| template-optimizer | Optimize templates and fix common issues |
BaseAutoType, which provides common methods like parse, feed_text, build_index, search, chat, dump, and load.Hyper-Extract offers two primary ways to interact with the system depending on your needs:
Hyper-Extract operates using a three-layer architecture that separates user interfaces from data structures and extraction algorithms:
| Layer | Purpose | Components |
|---|---|---|
| Auto-Types | Define data structures | 8 type classes |
| Methods | Extraction algorithms | RAG + Typical methods |
| Templates | Domain-specific configs | 80+ preset templates |
Auto-Types are intelligent, Pydantic-based data structures used to organize extracted knowledge. They act as containers that shape the output of document extraction.
Key characteristics include:
Hyper-Extract processes raw text through a four-stage pipeline to produce structured, type-safe data:
chunk_size. The default chunk size is 2048 characters with a 256-character overlap, using separators like paragraphs, sentences, and words.AutoTypeInstance for operations like search, chat, and visualization.Use the Template class to create an extraction pipeline, parse text, and visualize the resulting knowledge graph.
from hyperextract import Template
# Create template
ka = Template.create("general/biography_graph", language="en")
# Extract knowledge
with open("document.md") as f:
result = ka.parse(f.read())
# Access data
print(f"Nodes: {len(result.nodes)}")
print(f"Edges: {len(result.edges)}")
# Build index for search/chat capabilities
result.build_index()
# Visualize
result.show()from hyperextract import Template
# Create template
ka = Template.create("general/biography_graph", language="en")
# Extract knowledge
with open("document.md") as f:
result = ka.parse(f.read())
# Access data
print(f"Nodes: {len(result.nodes)}")
print(f"Edges: {len(result.edges)}")
# Build index for search/chat capabilities
result.build_index()
# Visualize
result.show()Hyper-Extract is a tool for designing YAML configuration templates used in structured knowledge extraction. The workflow follows a sequential process to move from initial requirements to a validated, multilingual YAML configuration:
record-designer (for models, lists, or sets) or graph-designer (for graphs, hypergraphs, temporal, or spatial structures).To create a temporal graph, add a time-related field to your relations and configure the identifiers block to recognize the time field. This allows the system to distinguish between multiple relations between the same entities occurring at different times.
Include a field (e.g., event_date) in the relations.fields list. It is recommended to set required: false and a default empty string to handle missing data.
In the identifiers block, set time_field to your chosen field name. Update relation_id to include the time field in its template (e.g., '{source}|{relation_type}|{target}|{event_date}') to ensure unique relation IDs across different timestamps.
Follow these guidelines for data consistency:
{observation_time}.2024-01-01).relations:
fields:
- name: source
type: str
- name: target
type: str
- name: relation_type
type: str
- name: event_date
type: str
description: 'When the relation occurred'
required: false
default: ''
identifiers:
entity_id: name
relation_id: '{source}|{relation_type}|{target}|{event_date}'
relation_members:
source: source
target: target
time_field: event_date
guideline:
rules_for_time:
- 'Observation time: {observation_time}'
- 'Absolute dates: Keep as-is (e.g., 2024-01-01)'
- 'Relative time: Convert to absolute'
- 'Fuzzy time: Leave empty, do not guess'Run a vLLM container with GPU support to host your local LLM service.
docker run --runtime nvidia --gpus all \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 \
vllm/vllm-openai:latest \
--model Qwen/Qwen3.5-9B \
--trust-remote-codeThe type field in your template determines which specialized sections are available in output, guideline, and identifiers.
type: list)Use items instead of entities or relations under output.
output:
items:
description:
zh: "列表项"
en: "List items"
fields:
- name: value
type: strtype: model)Use fields directly under output for flat extraction.
output:
fields:
- name: company_name
type: str
- name: revenue
type: floattype: temporal_graph)Includes specialized time extraction rules and identifier settings.
rules_for_time.time_field to specify which field contains the temporal data.guideline:
rules_for_time:
zh: ["规则1"]
en: ["Rule 1"]
identifiers:
time_field: timetype: spatial_graph)Includes specialized location extraction rules and field requirements.
location or coordinates fields in entities.rules_for_location.output:
entities:
fields:
- name: location
type: str
- name: coordinates
type: str
guideline:
rules_for_location:
zh: ["规则1"]
en: ["Rule 1"]You can use underlying extraction methods (like light_rag or graph_rag) instead of specific knowledge templates. Note that methods always use English prompts and do not require the --lang flag.
he parse document.md -m light_rag -o ./output/