Xiaomi Miloco Documentation

The miloco-home-patrol skill acts as an automated home manager that runs as a cron task every 30 minutes. It performs two primary functions:

1. Device Automation: Adjusts the home environment based on preferences and rules found in the family profile.
2. Care Reminders: Sends notifications via miloco-notify when noteworthy events occur.

Important Constraints:

• Isolated Sessions: Each patrol run is an independent session with no cross-session memory. To prevent repetitive actions or notifications, the skill relies on a Patrol Log (巡检日志) to track what has already been handled.
• Identity Strictness: The skill is strictly prohibited from guessing identities. It only uses identities explicitly stated in perception memory. It does not perform identity inheritance (e.g., assuming a person in the same seat is the same person seen earlier) or continuous inference based on appearance.

Miloco is an open-source AI solution for whole-house intelligence designed to run on local hardware (e.g., NUC, Raspberry Pi, Mac Mini). It manages Mi家 (Mi Home) devices via the MiOT protocol, perceives the home environment, and drives automation through AI Agents.

Core Capabilities:

• Device Control: Control Mi Home devices (on/off, parameter tuning, status checks, scene triggering) via MiOT.
• Environmental Perception: A four-layer camera pipeline (MultimodalCollector → Gate → Identity → Omni) that converts video into structured events.
• Identity Recognition: Distinguishes between family members and strangers.
• Automation Rules: Uses natural language (e.g., "When X, do Y") where a VLM handles semantic judgment.
• Home Memory: Accumulates long-term knowledge from perception and dialogue to inject into Agent system prompts.
• Task Management: Creates persistent intents (rules, cron jobs, records) with behavior statistics.
• AI Agent Integration: Connects via OpenClaw plugins and miloco-* Skill sets.
• Real-time Camera Streaming: Browser-based live viewing sharing the same decoding pipeline as perception.
• Event Feedback: Allows users to report perception errors, packaging anonymized data for local model iteration.

The Event Feedback feature packages the omni_trace, original audio/video clips, and metadata into a local tar.gz file for debugging.

Key constraints and behaviors:

• Local Storage Only: Packages are saved to the local disk and are not automatically uploaded. Users must manually submit them via a Feishu questionnaire.
• Data Availability: The package contents depend on whether the omni_trace and clips were successfully saved at the time of the event. If components are missing, they will be marked as missing in the components field, but the packaging process will not fail.
• Trace Requirement: The feedback entry is only visible for events where has_trace is true (events with inference records).
• Privacy: Textual data is automatically desensitized for personal information. Person galleries (which may contain faces) are excluded by default unless the user explicitly selects them.
• File Management:
  • The reveal-dir command uses system file managers (macOS open or Linux xdg-open) to open the directory containing the package.
  • The system automatically cleans up old packages if the total size of the packaging directory exceeds a specific threshold.

The Miloco Knowledge Base is a unified resource for both product and R&D perspectives of the Xiaomi Miloco 2.0 platform. It is organized into seven functional directories to help developers and new members understand module responsibilities, architecture, and operational procedures without duplicating source code or low-level implementation details.

Directory Structure:

• 01-overview/: Global project map, system architecture, and end-to-end data flows.
• 02-strategy/: Product positioning, competitive analysis, and roadmap.
• 03-features/: Core module documentation (the main body), covering both product capabilities (L1) and architectural design (L2).
• 04-testing/: Evaluation methodologies and testing frameworks.
• 05-external-deps/: Integration constraints, responsibility boundaries, and version compatibility for external SDKs and services.
• 06-dev-guide/: Operational instructions including environment setup, installation, startup, and troubleshooting.
• 07-design/: Visual design language, component patterns, and layout specifications.

The miloco-create-task skill manages 'continuously running home tasks' by combining rules (automation), schedules (reminders), records (behavior statistics), and lifecycles (duration). It serves as the unified entry point for system messages from the [Perception Engine].

Supported operations based on user input:

Note: list, logs, disable/enable, and update are handled by references/crud-ops.md. delete is handled by miloco-terminate-task.

The Home Profile system manages long-term household memory through a two-stage process: a Candidate Area for accumulating new observations and a Formal Archive for verified knowledge.

Knowledge Writing Path

1. Perception Logs (Omni caption) $\rightarrow$ miloco-perception-digest (high frequency, minute-level) $\rightarrow$ Perception Memory Summary.
2. Daily Dreaming Process (runs nightly):
   • Observe (miloco-home-observe): Extracts knowledge from perception/interaction memory into the Candidate Area.
   • Promote (miloco-home-promote): Promotes qualified candidates to the Formal Archive.
   • Prune (miloco-home-prune): Unifies subjects (reassigns dispersed names) and calls HomeProfileService.commit() to handle expiration, weight recalculation, truncation, and archiving/activation.
   • Persistence: Results are written to profile.md.

Archive Consumption Path

• Main Agent: Does not receive the full profile in the system prompt. Instead, the agent retrieves information on-demand using the home-profile list skill.
• Omni Prompt: The home_profile_loader.py injects the profile into the Omni prompt dynamic layer for use during perception reasoning.

The perception pipeline processes multimodal data (audio/video) through several stages to perform scene reasoning using a Vision Language Model (VLM). The data flow is as follows:

1. Collection: CameraDeviceAdapter captures frames, and MultiTrackSyncBuffer aligns audio and video tracks.
2. Triggering: PerceptionRunner triggers processing when a window is ready or a timeout occurs.
3. Collection: MultimodalCollector gathers the synchronized data.
4. Processing: PipelineProcessor orchestrates the flow through:
   • Gate: Filters static windows using visual frame differencing and audio peak energy to save costs.
   • Identity: Maps tracks to people using DeepSortTrackingService and IdentityEngine.
   • Omni (VLM): Performs reasoning via PerceptionEngineProxy using a VLM (e.g., MiMo API).
5. Post-processing: Results are logged, rules are updated via RuleService.update_state, suggestions are sent to AgentDispatcher, and meaningful events are saved as artifacts.

Task Management in Miloco transforms one-time automations into trackable tasks with a lifecycle and cumulative statistics. It consists of two main components:

1. Task (task): The long-lived identity (defined by task_id) that can be enabled or paused. It acts as a container for rules, cron schedules, and records.
2. Task Record (task_record): The statistical carrier that tracks behavior (e.g., "how much", "how long", or "how many times").

Task Record Types (kind)

When creating a task, you must choose one of three mutually exclusive statistical forms. This kind cannot be changed after creation:

• Progress (progress): Accumulates towards a specific target value and unit (e.g., "Drink 8 cups of water daily").
• Duration (duration): Accumulates time using start/end segments (e.g., "Limit screen time to 2 hours").
• Event (event): Records occurrences and timestamps without a target or automatic reset (e.g., "Log every time a stranger enters").