DOCSEARCHbyAlgolia
Submit Docs

underthesea Vietnamese NLP Toolkit

repository·main·Indexed 23 days ago

https://github.com/undertheseanlp/underthesea

An open-source Agentic AI Toolkit and Vietnamese Natural Language Processing (NLP) toolkit. Version 9.5.0 features multi-provider AI Agent support, a RESTful API for conversation management, and specialized tools for tokenization, parsing, and sentiment analysis. It includes a variety of Vietnamese NLP datasets, such as the Vietnamese Legal Corpus (CP_Vietnamese-VLC), the Underthesea Vietnamese Dictionary (DI_Vietnamese-UVD), and the UTS2017_Bank banking domain dataset.

Tokens
75.5K
Snippets
239
Records
423
Agent score
77%

What's inside underthesea

  1. Overview of Open Vietnamese Dictionary

    main
    The Open Vietnamese Dictionary project aims to build a comprehensive Vietnamese dictionary from scratch. The initial phase focuses on establishing homogeneity for words and parts of speech within the corpus. The project leverages existing open resources, specifically the Hồ Ngọc Đức dictionary and Vietnamese Wiktionary, to populate its word definitions.

  3. Overview of the Vietnamese NLP Dataset for Continual Learning

    main

    The UD_Vietnamese-COL dataset is designed as a challenge for building continuous learning NLP systems specifically for the Vietnamese language. It focuses on tasks such as part-of-speech tagging and dependency parsing.

    The dataset is organized into three levels of sentence quality/annotation reliability:

    • Bronze: Sentences automatically tagged by the underthesea library.
    • Silver: Sentences that have been corrected by at least one human.
    • Gold: Sentences that have been verified by at least three humans.

  4. Overview of Vietnamese IPA datasets

    main

    The Vietnamese IPA extension provides access to several datasets containing Vietnamese syllables and their phonetic transcriptions. These datasets are useful for tasks involving Vietnamese pronunciation or phonetic analysis.

    Available datasets include:

    • viwik_syllables_v1_2022: 6,817 syllables extracted from the Vietnamese Wiki Dictionary.
    • sino_vietnamese_v1_2022: 1,942 syllables extracted from sino_reading_papers.
    • hieuthi_all_vietnamese_syllables: A comprehensive set containing 17,874 syllables representing all possible Vietnamese syllables.

  5. Overview of Underthesea Agent capabilities (v9.5.0)

    main

    As of version 9.5.0, the Underthesea Agent framework includes the following core features:

    • Agent Class: Supports tool loops using OpenAI function-calling.
    • Multi-provider LLM Support: Compatible with OpenAI, Azure, Anthropic, and Gemini.
    • Context Management: Includes SessionManager and ContextManager for context resets and handoffs (following the Anthropic pattern).
    • Memory: WikiAgent provides markdown-as-memory for personal knowledge bases.
    • Tracing: Supports LocalTracer and LangfuseTracer with auto-trace capabilities.
    • Agent Server: underthesea.agent.server provides an A2A-compatible server with bundled chat UI, per-session isolation, and tool streaming.
    • Lightweight Core: The core has zero external dependencies, relying only on urllib and json.

  6. Overview of Underthesea Core Functions

    main

    Underthesea provides a variety of NLP functions categorized by their underlying implementation.

    Core Functions (available in the base installation):

    • sent_tokenize: Sentence segmentation
    • text_normalize: Text normalization
    • word_tokenize: Word segmentation
    • pos_tag: Part-of-speech tagging
    • chunk: Phrase chunking
    • ner: Named entity recognition
    • classify: Text classification
    • sentiment: Sentiment analysis
    • convert_address: Address conversion (63→34 provinces)

    Deep Learning Functions (requires [deep] extra):

    • dependency_parse: Dependency parsing
    • translate: Vietnamese-English translation

    Additional Functions:

    • lang_detect: Language detection (requires [langdetect] extra)
    • tts: Text-to-speech (requires [voice] extra)
    • agent: Conversational AI agent (requires [agent] extra)

  7. Vietnamese Word Segmentation with underthesea

    main
    Vietnamese Word Segmentation (VWS) is the process of splitting a sequence of characters into meaningful chunks or 'words'. The underthesea toolkit provides VWS capabilities using a Conditional Random Fields (CRF) model. This model considers the entire sequence context to predict whether to split or not at each token, which is essential for handling the nuances of the Vietnamese language where multiple lexical units often form single written forms.

  8. Core Agent Features

    main

    The Agent implementation includes:

    • Multi-provider abstraction: A single interface for different LLMs.
    • Auto-detection: Automatically selects the provider from environment variables.
    • Chat capabilities: Supports both streaming and non-streaming modes.
    • Tool calling loop: Includes an iteration limit for tool execution.
    • Conversation management: Handles history and allows system prompt customization.
    • Parameter control: Supports temperature control and model overrides per call.

  9. Available Underthesea NLP Datasets

    main

    Underthesea provides several Vietnamese NLP datasets categorized by their research application. These datasets cover universal dependencies, parsing, tokenization, and domain-specific tasks.

    Universal Dependencies

    • UD_Vietnamese-COL: Treebank
    • UD_Vietnamese-UUD: Universal Dependencies

    Parsing & Dependencies

    • DI_Vietnamese-UVD: Dependency corpus
    • DI_Vietnamese-VFD: Feature detection
    • CP_Vietnamese-VLC: Constituency parsing
    • CP_Vietnamese-WIK: Wikipedia constituency

    Tokenization & Text

    • VLSP2013_WTK: VLSP 2013 tokenization
    • UTS_WTK: Underthesea tokenization
    • UTS_Text: Text processing
    • UTS_Dictionary: Dictionary resources

    Banking Domain

    • UTS2017_Bank: Vietnamese banking customer feedback dataset. Contains 2,471 examples (1,977 train, 494 test) for sentiment analysis, aspect classification, and aspect-based sentiment tasks across 14 banking categories. Licensed under Apache 2.0.

  10. Access Vietnamese stopwords

    main
    The underthesea repository provides a set of Vietnamese stopwords located in stopwords.txt. This dataset was developed as part of the August 2021 Continual Learning Challenge and is designed to be a continually updated set of stopwords for the Vietnamese language. The list is generated using methods such as Inverse Document Frequency (IDF) and Kullback-Leibler Divergence to identify words that carry low information content in the language.

  11. What is a Personal AI OS and how does it differ from standard agent frameworks?

    main

    A Personal AI OS (or Ambient/Always-on Agent) is a design paradigm for AI assistants that move beyond the request-response model used by enterprise frameworks like LangGraph or CrewAI.

    While standard frameworks typically require a developer to write Python code, deploy it to a server, and trigger it via HTTP or CLI (where the agent is stateless and shuts down after execution), a Personal AI OS is designed to be:

    • Always-on (Daemon-first): Runs as a long-lived process with open sockets to listen for events, rather than a CLI tool that runs and exits.
    • Multi-channel: Connects to various communication platforms (WhatsApp, iMessage, Slack, Telegram) simultaneously.
    • Multi-device: Operates across a mesh of nodes (laptop, phone, tablet) where each device exposes specific capabilities like camera, screen, mic, or location.
    • Memory-first: Uses persistent context (such as Markdown files or structured stores) to maintain state across sessions, days, or weeks.
    • Proactive (Ambient Triggers): Uses internal schedulers (Heartbeat), mailbox watchers, or file system listeners to take action based on events (e.g., "When X happens $\rightarrow$ agent does Y") rather than waiting for a user prompt.

  12. What is an Agent Harness vs. an Evaluation Harness?

    main

    It is important to distinguish between the infrastructure used to run an agent and the infrastructure used to test it:

    • Agent Harness (or Scaffold): The infrastructure layer that enables a model to act as an agent. It manages input processing, tool orchestration, memory management, and session continuity.
    • Evaluation Harness: The infrastructure used to run evaluations end-to-end. It handles task execution (often concurrently), manages isolated environments, and aggregates results.