DOCSEARCHbyAlgolia
Submit Docs

TypeAgent Documentation

repository·main·Indexed 20 days ago

https://github.com/microsoft/typeagent

TypeAgent is a collection of sample code exploring an architecture for building single personal agents with natural language interfaces, combining stochastic LLMs with structured software components. It includes Android samples for mobile and Wear OS clients, and AutoShell, a .NET 8 Windows shell automation console application for programmatic control of the Windows desktop environment via JSON-based commands.

Tokens
436.5K
Snippets
916
Records
2K
Agent score
69%

What's inside TypeAgent

  2. Overview of grammar-tools-core

    main

    The grammar-tools-core package (available as grammar-tools-core via pnpm --filter) is a framework-agnostic TypeScript package that wraps packages/actionGrammar. It provides a complete service surface for grammar-related tasks, including loading, diagnostics, symbol indexing, formatting, completion previews, match tracing, coverage reporting, and structural diffing.

    It is designed to serve as a stable API contract that can be used across various transports, such as VS Code webviews, web applications (via HTTP/WebSocket), shell IPC, or in-process calls.

  3. Overview of @typeagent/thoughts

    main

    The @typeagent/thoughts package is a TypeScript library that converts raw text or stream-of-consciousness input into well-formatted Markdown documents using Claude. It also supports transcribing WAV audio files into text using Azure Cognitive Services before processing them into Markdown.

    Key capabilities include:

    • CLI Utility: Convert text files, audio files, or stdin to Markdown via the command line.
    • Audio Transcription: Automatic WAV transcription via Azure Cognitive Services.
    • Customization: Use custom instructions to guide Markdown formatting, add keyword tags for organization, or use inline tags during input.
    • MCP Support: Can be integrated as a Model Context Protocol (MCP) server.

  4. Overview of @typeagent/copilot-plugin

    main

    The @typeagent/copilot-plugin integrates TypeAgent with the GitHub Copilot CLI. It acts as a bridge that intercepts user inputs to determine if they are Action Requests or General Questions.

    • Action Requests: Routed to TypeAgent. If recognized, TypeAgent processes the request directly, bypassing the Copilot LLM.
    • General Questions: Passed through to the Copilot LLM.

    The plugin operates in two modes:

    1. Direct Mode: Connects via WebSocket to TypeAgent. It is faster and saves LLM tokens but does not support streaming.
    2. MCP Mode: Instructs the LLM to call the typeagent-processCommand MCP tool. This supports streaming and LLM-formatted responses but is slower and consumes tokens.

  5. Overview of @typeagent/agent-flows functionalities

    main

    @typeagent/agent-flows provides shared infrastructure for TypeAgent flow agents, consolidating common functionalities to ensure consistent behavior across different agents (like browser-typeagent, powershell-typeagent, and taskflow-typeagent).

    Key capabilities include:

    • Script Validation: Prevents dangerous calls and blocked identifiers using createScriptValidator and transpileScript.
    • Script Execution: Runs scripts in controlled environments with overrides and timeouts via createScriptExecutor.
    • Grammar Generation: Constructs rules for parsing flow actions using generateGrammarRuleText, extractRuleNames, buildStartRule, and assembleDynamicGrammar.
    • Schema Building: Generates TypeScript types for type-safe flow action interactions via generateFlowActionTypes and buildUnionType.
    • Sandbox Declaration: Generates TypeScript declarations for sandbox environments using createSandboxDeclarationGenerator.

  6. Overview of @typeagent/action-schema-compiler

    main
    The @typeagent/action-schema-compiler is a TypeScript library used to preprocess action schemas authored in TypeScript. It compiles these TypeScript definitions into a ParsedActionSchemaGroup JSON format. This allows the TypeAgent dispatcher to load schema configurations from JSON at runtime, avoiding the need for runtime TypeScript compilation.

  7. Overview of @typeagent/common-utils utilities

    main

    The @typeagent/common-utils package provides shared utility functions and types used across the TypeAgent project. It is designed to work in both Node.js and browser environments, with platform-specific implementations for certain features like Base64 encoding/decoding.

    Key utility categories include:

    • Object Property Utilities: Safe retrieval and setting of properties using dot-separated strings (e.g., getObjectProperty, setObjectProperty).
    • Base64 Encoding/Decoding: Conversion between Uint8Array and base64 strings (uint8ArrayToBase64, base64ToUint8Array).
    • Promise Handling: Flexible promise management via createPromiseWithResolvers.
    • Rate Limiting: Controlling concurrent executions using createLimiter.
    • String Utilities: Regex generation for wildcard matching via simpleStarRegex.
    • Printing Utilities: Human-readable elapsed time formatting (getElapsedString, getColorElapsedString).
    • CLI Path Resolution: Resolving system paths for CLI executables (resolveCliOnPath, claudeExecutableOption).

  8. Overview of TextPro

    main

    TextPro is a sample code module designed for parsing, converting, and manipulating text data and documents. It uses markdown as its primary intermediate format.

    Key capabilities include:

    • Format Conversion: Converting formats like HTML into simplified markdown.
    • Document Analysis: Extracting document blocks/chunks (headings, lists, tables, links, images, etc.) from markdown.
    • Knowledge Inference: Extracting entities, topics, and structured tags from markdown information without requiring an LLM.

    TextPro is utilized by the Document Memory component to import both HTML and markdown documents.

  9. Overview of knowledge-processor capabilities

    main

    The knowledge-processor is a TypeScript library designed for Structured Retrieval-Augmented Generation (RAG). It is primarily used to implement Agent Memory by extracting, indexing, and utilizing knowledge from various sources.

    Core Functionalities

    • Knowledge Extraction: Uses actions like extractKnowledge to pull structured data from conversations, transcripts, images, and documents.
    • Knowledge Indexing: Organizes data via text indexing, key-value indexing, and temporal indexing for efficient retrieval.
    • Query Generation: Converts natural language into structured queries using actions like searchKnowledge.
    • Answer Generation: Generates natural language responses based on retrieved knowledge using actions like generateAnswer.
    • Conversation Management: Manages context, tracks topics, and handles messages using createMessage and generateAnswer.

  10. Overview of @typeagent/agent-rpc

    main

    The @typeagent/agent-rpc package is a TypeScript library providing Remote Procedure Call (RPC) capabilities for the TypeAgent SDK. It abstracts the transport layer using an RpcChannel interface, allowing distributed components of the TypeAgent system to communicate seamlessly.

    Key capabilities include:

    • Client-side RPC management: Managing operations, abort signals, and client-specific tasks.
    • Server-side RPC management: Creating and managing inbound RPC channels.
    • Rebindable RPC sessions: Creating durable sessions that can rebind to new transport channels upon reconnection, preserving state and object identity.

  11. Overview of grammar-tools-ui components

    main

    The grammar-tools-ui package provides Lit-based web components for visualizing, editing, and debugging grammar data. These components interact with a GrammarBackend interface.

    Key components include:

    • GtRuleList: Navigates and interacts with individual grammar rules.
    • GtSourceView: Explores the grammar's source code structure.
    • GtCompletionPanel: Previews completions based on partial input.
    • GtTraceTimeline: Visualizes the step-by-step trace of grammar matches.
    • GtCoverageHeatmap: Shows a heatmap of rule usage frequency.
    • GtDiffView: Compares grammar rule differences side-by-side.
    • GtDebugPanel: A composite component that integrates the above tools into a single interface.

  12. Overview of `grammar-tools-ui` shared widget bundle

    main

    The grammar-tools-ui package (located in packages/grammarTools/ui) provides a bundle of shared UI components designed to be hosted in various environments, including VS Code webviews, Vite SPAs, or shell BrowserWindow instances.

    Key characteristics:

    • Backend Agnostic: Components interact with grammar-tools-core through a GrammarBackend abstraction. This allows the UI to function whether the backend is running in-process or via RPC.
    • Technology Stack: Built using Lit, utilizing custom elements (@customElement()), reactive properties (@property()), and html tagged templates.
    • Distribution: Distributed as a single ESM bundle produced via Vite in library mode. Because components are registered as custom elements, they can be loaded by any host without specific entrypoints.