Agent KB Documentation

OpenHands Memory Monitor is a VSCode extension designed for real-time monitoring of system and process memory usage. It provides high-level visibility via the status bar and deep-dive analytics through a dedicated graphical interface.

Key capabilities include:

• Real-time Monitoring: Current system memory usage is visible in the status bar.
• Detailed Statistics: Access to total, used, and free memory metrics.
• Process Tracking: Identification of top processes by memory consumption.
• Historical Data: Interactive charts to track memory usage trends over time.
• Cross-Platform: Compatible with Windows, macOS, and Linux.

The Chat interface provides an AI-powered conversational experience designed for development tasks. Features include:

• Interactive AI Assistant: Engage in natural language conversations.
• Context-Aware Responses: The AI understands and responds to development-related queries.
• Action Suggestions: Receive actionable recommendations for specific tasks.
• Conversation Management: Create, delete, and manage different conversation threads.

The Workspace provides a comprehensive development environment for managing project files and directories. Key capabilities include:

• File Explorer: Browse, view, and manage project files.
• Project Management: Import, create, and navigate between different projects.
• File Operations: View contents, create new files/folders, and upload/download files.

The Terminal provides a command-line interface to interact directly with the system. You can:

• Execute Shell Commands: Run bash and system commands.
• Command History: Track and recall previously executed commands.
• Environment Interaction: Perform programming and system administration tasks directly via the command line.

The Browser feature enables web-based interaction and automation. It supports 15 different browser interaction functions, including:

• Web Page Navigation: Open and browse websites.
• Screenshot Capture: Automatically generate screenshots of web pages.
• Interaction Tools: Click elements, fill out forms, and scroll pages.

The SWE-bench evaluation results in this directory are generated using the official SWE-bench docker containerization. The report categorizes evaluation instances into several states to help developers debug agent performance:

• Resolved Instances: Instances where the agent successfully provided a patch that resolved the issue.
• Unresolved Instances: Instances where the agent attempted a solution but failed to resolve the issue.
• Error Instances: Instances that encountered runtime or execution errors during the evaluation process.
• Empty Patch Instances: Instances where the agent produced no code changes (an empty patch).

Detailed logs for each specific instance can be found in the ./eval_outputs/{instance_name}/run_instance.log file.

The Jupyter Notebook feature allows for interactive Python coding and data analysis within the application. It supports:

• Interactive Code Cells: Execute Python code in a cell-based interface.
• Input/Output Tracking: Maintains a history of code inputs and their corresponding outputs.
• Persistent Sessions: Preserves code execution context between cells for continuous data analysis tasks.

The Browsing Agent Framework implements a generic agent based on AgentLab that enables full-featured web browsing. The agent operates by receiving specific observations to navigate and understand web pages, including:

• Set-of-marks annotated web-page screenshots: Visual representations of the page with markers for interactive elements.
• Accessibility tree: A structured representation of the web page's accessibility layer.
• Historical context: All thoughts and actions from previous steps in the browsing session.

Note: This agent is distinct from the CodeAct agent. While this framework provides a specialized VisualBrowsingAgent, the CodeAct agent uses built-in browsing support via tools like browse_url and the browser tool.

The CodeAct Agent is OpenHands's primary agent implementation. It consolidates LLM actions into a unified code action space to improve simplicity and performance. The agent operates via a function calling interface and can perform two main types of actions:

1. Converse: Communicate with humans in natural language for clarification or confirmation.
2. CodeAct: Execute actions using a set of specialized tools for bash, Python, web browsing, and file editing.

The Memory Condenser is responsible for summarizing chunks of events to maintain an efficient context. It follows a specific priority order for summarization:

1. Agent Events: It first summarizes the earliest agent actions and observations occurring between two user messages.
2. Sequential Chunks: It then processes subsequent chunks of events between user messages.
3. User Messages: If no agent events remain, it summarizes user messages individually.
   • Conditions for User Message Summarization: User messages are only summarized if they are large enough and are not immediately following an AgentFinishAction event (as those are treated as important tasks).

Technical Implementation: Summaries are retrieved from the LLM as AgentSummarizeAction and are persisted in the State.