DOCSEARCHbyAlgolia
Submit Docs

WeKnora Knowledge Framework

repository·main·Indexed 9 days ago

https://github.com/tencent/weknora

An LLM-powered knowledge framework for enterprise-grade document understanding and semantic retrieval. It utilizes RAG, autonomous ReAct agents, and an automated Wiki mode to turn documents into living knowledge. Features include a comprehensive CLI for managing knowledge bases, documents, and agents, support for the Model Context Protocol (MCP), and a Go client for programmatic integration.

Tokens
416.5K
Snippets
888
Records
1.5K
Agent score
94%

What's inside WeKnora

  1. Overview of WeKnora System Components

    main

    WeKnora is composed of several core and optional components that work together to provide RAG (Retrieval-Augmented Generation), Agent, and Auto-Wiki capabilities.

    Core Components

    ComponentTech StackDefault PortResponsibility
    app (Backend)Go / Gin8080REST API, Retrieval/QA, Agent engine, Asynchronous tasks (Asynq)
    frontendVue 3 + Nginx80Web console; Nginx proxies /api to the app
    docreaderPython / gRPC50051Document parsing, OCR, web scraping, image extraction
    postgresParadeDB (PostgreSQL 17)5432Primary database + default hybrid retrieval engine (via RETRIEVE_DRIVER=postgres)
    redisRedis 76379Stream management (SSE recovery) and Asynq task queues
    sandboxPython 3.11 + Node 20—One-time sandbox container for executing Agent Skills scripts

    Optional/Extensible Components

    WeKnora supports various pluggable backends via the RETRIEVE_DRIVER and STORAGE_TYPE environment variables:

    • Vector Databases: Qdrant, Milvus, Weaviate, Doris, Elasticsearch, or Tencent VectorDB.
    • Knowledge Graph: Neo4j (for GraphRAG).
    • Object Storage: MinIO (S3 compatible), COS, S3, OSS, OBS, or TOS (configured via STORAGE_TYPE).
    • Search: SearXNG (self-hosted web search).
    • Observability: Langfuse stack (Langfuse 3 + ClickHouse + MinIO).
    • MCP Server: mcp-server (Python) to expose WeKnora APIs as a Model Context Protocol server.
    • Hybrid Parsing: odl-hybrid (Docling) for OpenDataLoader PDF hybrid parsing.

  2. Overview of WeKnora API categories

    main

    The WeKnora API is organized into several functional categories. Use the following mapping to find specific documentation for your task:

    CategoryDescription
    Auth ManagementUser registration, login, token management, and OIDC flows
    Tenant ManagementCreating and managing space accounts
    Knowledge Base ManagementCreating, querying, and managing knowledge bases
    Knowledge ManagementUploading, retrieving, and managing knowledge content
    Model ManagementConfiguring and managing various AI models
    Chunk ManagementManaging knowledge chunks
    Tag ManagementManaging knowledge base tags
    FAQ ManagementManaging FAQ question-answer pairs
    Agent ManagementCreating and managing custom agents
    Session ManagementCreating and managing chat sessions
    Knowledge SearchSearching content within knowledge bases
    Chat FunctionalityQ&A based on knowledge bases and Agents
    Message ManagementRetrieving and managing chat messages
    EvaluationEvaluating model performance
    Initialization ManagementKnowledge base model configuration and Ollama management
    System ManagementSystem info, parsing engines, and storage engines
    MCP ServiceManaging MCP tool services
    Organization ManagementOrganizations, members, and sharing knowledge bases/agents
    SkillsPre-installed agent skills
    Web SearchWeb search service providers
    Vector StoreManaging vector database connections

  3. Overview of DocReader Service

    main
    DocReader is a gRPC service within the WeKnora project responsible for document parsing and processing. It supports multiple document formats, OCR recognition, and multimodal processing. It functions as a specialized worker that can be integrated with storage backends like MinIO, AWS S3, Tencent Cloud COS, or Alibaba Cloud OSS.

  4. Overview of WeKnora

    main

    WeKnora (维娜拉) is an enterprise-grade knowledge base and RAG (Retrieval-Augmented Generation) system open-sourced by Tencent. It is designed to turn documents into living knowledge using RAG, Agents, and Auto-Wiki capabilities.

    Core Architecture Components:

    • Backend: Go monolith.
    • Frontend: Vue 3.
    • Document Parsing: Python-based microservice (docreader).

    Key Capabilities:

    • Multi-tenancy and multiple knowledge bases.
    • Hybrid retrieval and Knowledge Graph support.
    • AI Agents and Wiki generation.
    • MCP (Model Context Protocol) integration.
    • Multi-platform IM integration and web embedding.
    • Chrome extension for web-based Q&A and clipping.

  5. Overview of WeKnora CLI commands

    main

    The weknora CLI provides a comprehensive interface for managing RAG (Retrieval-Augmented Generation) knowledge bases, documents, and agents.

    Key command groups include:

    • Knowledge Bases (kb): Manage and inspect knowledge bases.
    • Documents (doc): Upload, manage, and track document parsing status.
    • Authentication (auth, profile): Manage connection profiles and credentials.
    • RAG & Search (chat, search, chunk, session): Perform hybrid searches, ask streaming RAG questions, and manage chat sessions.
    • Agents (agent, model, skills): Manage custom agents, models, and install specialized Agent Skills.
    • AI Integration (mcp, api): Run as a Model Context Protocol (MCP) server or make raw API requests.
    $ weknora --help

  6. Overview of WeKnora LLM Knowledge Framework

    main

    WeKnora is an open-source LLM-based knowledge framework designed for enterprise-grade document understanding, semantic search, and autonomous reasoning. It transforms scattered documents into searchable, reason-capable, and evolving knowledge assets.

    Core Capabilities

    • RAG-based Quick Q&A: Optimized for everyday semantic search and retrieval-augmented generation.
    • ReAct Agent Reasoning: Orchestrates knowledge retrieval, MCP tools, and web searches to handle complex, multi-step tasks autonomously.
    • Wiki Mode: A novel feature where Agents autonomously generate and maintain an interconnected Markdown knowledge base and interactive knowledge graph from raw documents. It supports manual editing, version history, and one-click rollbacks.

    Key Features

    • Knowledge Management: Supports tree-structured folders (preserving upload directory structures), chunk editing with version history, and diff comparisons.
    • Data Integration: Automatically syncs with external platforms like Feishu Knowledge Base, Feishu Cloud Drive, Notion, Yuque, and RSS. Supports over 10 document formats including PDF, Word, Images, and Excel.
    • Deployment & Scalability: Modular architecture allows swapping LLMs, vector databases, and storage. Supports local and private cloud deployment for full data self-management. Includes a runtime task queue dashboard and worker pool management.
    • Enterprise Readiness: Features multi-tenant RBAC (4-layer role matrix + resource ownership + tenant audit logs), scoped API keys, and the Principal model for programmatic integration.
    • Observability: Seamlessly integrates with Langfuse to provide comprehensive observability for Agent reasoning, token consumption, and pipelines.

  7. Overview of WeKnora Observability and Auditing

    main

    WeKnora provides three primary mechanisms to monitor system health, performance, and security:

    1. Tracing (Langfuse): Use this to understand why a specific Q&A was slow or incorrect. It provides a full call chain including retrieval, reranking, and model calls (including token usage).
    2. Audit Logs: Use this to track who changed knowledge bases, members, or system settings. Access these via "Knowledge Base Settings -> Activity" or "Settings -> Audit Logs".
    3. Runtime Queue Dashboard: Use this to monitor background tasks (parsing, summarization, Wiki tasks) for accumulation or failures. Access via "Settings -> Runtime Queue".

    Quick Reference Table:

    GoalWhere to look
    Trace retrieval, model calls, and token usageLangfuse (after integration)
    Track changes to KB, members, or system settingsKnowledge Base "Activity" or "Settings -> Audit Logs"
    Monitor background task accumulation/failure"Settings -> Runtime Queue"
    Check service healthGET /health
    Correlate logs across servicesSearch by X-Request-ID header

  8. Overview of WeKnora features

    main

    WeKnora is a knowledge framework integrating RAG (Retrieval-Augmented Generation), Agent reasoning, and Auto-Wiki capabilities. Its features are categorized into three main areas:

    1. Intelligent Dialogue (Smart QA)

    • Reasoning: ReACT-based multi-step reasoning, orchestrating knowledge retrieval, MCP tools, and web search.
    • Wiki Mode: Agents automatically generate and maintain structured Markdown Wiki pages from raw documents, supporting manual editing, version history, and line-level diffs.
    • Tool Calling: Supports built-in tools, MCP tools (including OAuth2), and web search. Users can mention @Skill or @MCP to scope the Agent runtime.
    • RAG Capabilities: Fast Q&A based on knowledge bases, with support for citation overlays, RAG pipeline progress visualization, and multi-turn context awareness.

    2. Knowledge Management

    • Knowledge Base Types: FAQ, Documents, and Wiki. Supports folder/URL imports and multi-tag management.
    • Retrieval Strategies: BM25 (sparse), Dense retrieval, GraphRAG (graph-enhanced), Parent-Child chunking, and pgvector HNSW acceleration.
    • Chunk Editing: Direct editing of retrieval chunks with version snapshots, diffs, and one-click rollback. Editing automatically triggers index rebuilding.
    • Data Sources: Automatic synchronization with Feishu (Lark), Notion, Yuque, and RSS feeds.
    • Document Formats: PDF, Word, Txt, Markdown, HTML, EPUB, MHTML, Images, CSV, Excel, PPT, JSON.

    3. Integration & Extension

    • Model Support: Extensive list including OpenAI, Anthropic, DeepSeek, Qwen, Gemini, Ollama, and more.
    • Vector Databases: PostgreSQL (pgvector), Elasticsearch, Milvus, Qdrant, etc.
    • Object Storage: Local, AWS S3, Alibaba Cloud OSS, MinIO, etc. Supports multiple storage instances per space.
    • MCP Server: Official PyPI package tencent-weknora-mcp providing 29 tools via stdio, SSE, or HTTP.
    • Deployment: Local, Docker, or Kubernetes (Helm).

  9. Overview of WeKnora

    main

    WeKnora is an open-source, LLM-powered knowledge framework designed for enterprise-grade document understanding, semantic retrieval, and autonomous reasoning. It transforms scattered documents into a queryable, reasoning-capable, and evolving knowledge asset.

    Core Capabilities

    • RAG-based Quick Q&A: For everyday semantic lookups.
    • ReAct Agent: An autonomous agent that orchestrates retrieval, uses MCP tools, and performs web searches to handle complex multi-step tasks.
    • Wiki Mode: Agents distill raw documents into a self-maintaining, interlinked markdown knowledge base featuring an interactive knowledge graph, manual editing, revision history, and one-click rollback.

    Key Features

    • Multi-source Ingestion: Supports auto-syncing from Feishu wiki, Feishu Drive, Notion, Yuque, and RSS.
    • Document Support: Handles 10+ formats including PDF, Word, images, and Excel.
    • Knowledge Curation: Features a tree-structured folder view and chunk editing with revision history (diffing and reverting).
    • Deployment & Integration:
      • Website embed widgets for publishing agents.
      • Scoped API keys with a principal model for programmatic access.
      • Multi-instance storage backends per workspace.
      • 20+ LLM provider integrations (OpenAI, DeepSeek, Qwen, Gemini, Ollama, etc.).
      • Full Langfuse observability for agent reasoning and token usage.
      • Runtime task-queue dashboard with worker-pool governance.
    • Enterprise Readiness: Multi-workspace RBAC (4-tier role matrix), per-resource ownership, and per-workspace audit logs.
    • Connectivity: Serves Q&A via IM channels like WeCom, Feishu, Slack, and Telegram.

  10. What is WeKnora?

    main

    WeKnora is an open-source, LLM-based knowledge management framework designed for enterprise-grade document understanding, semantic retrieval, and intelligent reasoning. It integrates three core capabilities:

    1. RAG (Retrieval-Augmented Generation) Quick Q&A: For daily knowledge queries.
    2. ReAct Agent Intelligent Reasoning: Autonomous orchestration of knowledge retrieval, MCP tools, and web search to complete complex multi-step tasks.
    3. Wiki Mode: Agents autonomously generate interconnected Markdown knowledge bases and visual knowledge graphs from raw documents. This mode supports manual editing, version history, and one-click rollbacks.

    Key features include multi-source data ingestion (Feishu, Notion, Yuque, RSS, etc.), website embedding widgets, enterprise-grade RBAC, and modular architecture allowing for flexible replacement of LLMs, vector databases, and storage backends.

  12. Use the Data Processor skill for RAG analysis and data tasks

    main

    The Data Processor (数据处理器) is an enterprise-grade skill designed to process RAG (Retrieval-Augmented Generation) results and perform data analysis tasks. It is used when you need to perform statistical calculations, format conversions, information extraction, or report generation from retrieved knowledge base documents.

    Core Capabilities

    • Data Analysis: Statistical analysis of retrieved document data.
    • Format Conversion: Converting between JSON, CSV, and Markdown tables.
    • Information Extraction: Extracting structured data (numbers, dates, keywords) from unstructured text.
    • Report Generation: Creating summaries and data analysis reports.

    Common Use Cases

    • "Analyze this data", "Calculate the total/average", "Summarize the statistics".
    • "Convert this to JSON/CSV format".
    • "Extract key information", "Organize this into a table".
    • "Generate a report" or "Data summary".