repository·main·Indexed 20 days ago
https://github.com/westlake-agi-lab/appagentxAn evolutionary framework for GUI agents designed for smartphone interaction. AppAgentX utilizes a memory mechanism to evolve repetitive action sequences into high-level shortcut actions. The system includes a Gradio-based demo and a containerized backend featuring an Image Feature Extraction Service (Port 8001) and a Screen Parsing Service based on OmniParser (Port 8000). It integrates with LangChain, LangGraph, Neo4j, Pinecone, and requires ADB for Android device connectivity.
AppAgentX relies on LangChain and LangGraph for its agent framework. To install the required Python dependencies, use pip to install from the provided requirements.txt file.
Note: You must also configure your LLM settings (such as OpenAI or DeepSeek) in the config.py file.
pip install -r requirements.txtAppAgentX requires a connection to an Android device or emulator via ADB (Android Debug Bridge).
If you do not have a physical device, use the Android Studio emulator:
Screen recognition and feature extraction services are containerized using Docker. This modular approach allows for easy replacement of parsing tools.
To deploy these services:
backend folder.backend/README.md file.The backend consists of two containerized FastAPI services: the Image Feature Extraction Service (Port 8001) and the Screen Parsing Service (Port 8000).
You must manually download model weights from OmniParser on Hugging Face and place them in the OmniParser/weights/ directory before starting, otherwise the API will fail to process images.
Required structure:
OmniParser/weights/icon_detect_v1_5/best.pt (YOLO detection model)OmniParser/weights/icon_caption_florence/ (Caption model files)To build and start the services in the foreground:
docker-compose up --buildTo run the services in the background:
docker-compose up -d --buildOnce your LLM, databases, backend services, and Android device/emulator are configured, you can launch the Gradio-based front-end interface using one of the following commands:
python demo.pyor
gradio demo.pyAll core service configurations are managed via the config.py file. You must provide credentials and connection details for the following services:
The Image Feature Extraction Service runs on http://localhost:8001. Use the following endpoints to manage models and extract image features:
| Method | Endpoint | Description |
|---|---|---|
GET | /available_models | Get list of available models |
POST | /set_model | Set the model to use |
POST | /extract_single/ | Extract features from a single image |
POST | /extract_batch/ | Batch extract features from multiple images |
GET | /model_info | Get current model information |
GET | /benchmark/ | Run performance tests |
The Image Parsing Service (based on OmniParser) runs on http://localhost:8000.
| Method | Endpoint | Description |
|---|---|---|
POST | /process_image/ | Process image and return parsing results |
Use the following Docker Compose commands to manage the lifecycle of the backend services:
Check service status:
docker-compose psView logs:
docker-compose logsomniparser): docker-compose logs -f omniparserStop and remove services:
docker-compose downdocker-compose stopdocker-compose down -vdocker-compose down --rmi all -vdocker-compose downThe following services are defined in the backend/docker-compose.yml file. Use these ports to communicate with the respective backend APIs.
services:
image-embedding:
ports:
- "8001:8001"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [ gpu ]
omni-parser:
ports:
- "8000:8000"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [ gpu ]