SightFlow Desktop Agent

SightFlow abstracts the ability to 'analyze screenshots and generate replies' into independent Providers.

• Manifests: Providers declare their configuration structure via a manifest.json.
• Bundles: Providers implement logic via a provider.bundle.js entry point, which receives chat screenshots and returns events like reply_text, skip, or error.
• Discovery: The client pulls a list of available Providers from https://sightflow.dev/provider-hub.json.
• Caching: The list is cached locally. You can refresh it manually using the refresh button next to the Agent title.
• Fallback: The built-in 豆包 Seed provider is always available locally as a fallback.

SightFlow operates as a 'Work Memory Engine' that enables AI to interact with desktop software by following a four-stage lifecycle:

1. See (GUI Understanding): Uses visual models to understand any software interface and its current state.
2. Think (Planning): Combines context and history to decide on the next logical step.
3. Do (Execution): Performs actions like clicking, typing, switching windows, or sending messages, mimicking a human operator.
4. Learn (Trace Recording): Writes every execution into a structured work-trace, which serves as reusable work memory.

This approach differs from traditional RPA by recording not just the steps taken, but the rationale (the 'why') behind them, enabling replay, evaluation, and inheritance of expertise.

Every execution in SightFlow is recorded as a structured work-trace. This allows for replaying actions, evaluating model performance, and inheriting decision-making logic. A work-trace object contains:

• timestamp: The time of execution.
• ui_state: The state of the interface at that moment.
• rationale: The reasoning behind the action (why it was taken).
• action: The specific operation performed (e.g., click, input, switch, send).
• result: The outcome of the action.

work-trace = {
  timestamp,    # Timestamp
  ui_state,     # UI state
  rationale,    # Rationale (why this was done)
  action,       # Click / Input / Switch / Send
  result        # Result
}

The 'Learn' capability (Work Memory Engine) transforms raw agent execution into structured, actionable knowledge. It moves beyond simple logging to provide a three-layer product capability:

1. L1 Record Trace: Every execution is captured as a structured work trajectory (local, append-only, and privacy-preserving).
2. L2 Replay/Evaluate: A timeline-based replay of trajectories allows for quantitative comparison across different models or versions within the same scenario.
3. L3 Inherit Memory: Trajectories and human interventions are distilled into 'Experience Cards' (SOPs), which are then injected into the runtime to improve future performance.

This creates a closed loop of: Demonstration → SOP Induction → Execution → Refinement.

Unlike traditional RPA that only records steps, SightFlow is a working memory engine that records the reasoning behind actions. Every execution is captured as a structured work-trace.

A work-trace consists of:

• timestamp: When the action occurred.
• ui_state: A description/snapshot of what the screen looked like.
• rationale: The reasoning (WHY) behind the decision.
• action: The specific operation (e.g., click, type, switch, send).
• result: The outcome of the action.

This structure enables three core capabilities:

• Replay: Review every step and decision when something breaks.
• Eval: Compare different models or versions by evaluating their outcomes consistently.
• Inherit: Capture and reuse human-like judgment for future tasks.

In the context of specialized scenarios like Influencer Outreach (达人建联), the most valuable knowledge extracted is phrasing and judgment, rather than simple UI clicks. Knowledge is organized into "Strategy Cards" categorized by:

• Opening Scripts (开场话术): Segmented by influencer type, platform, or follower count.
• Objection Handling (异议处理): How to respond to high quotes, being ignored (read-only), or price negotiations.
• Rhythm Judgment (节奏判断): When to send case studies, when to provide quotes, or when to escalate to a phone call.
• Red Lines (红线): Prohibited actions (e.g., "never impersonate official staff", "never start with a price cut").

SightFlow operates through a continuous four-stage loop that transforms visual input into actionable intelligence:

1. See: A vision model understands the GUI and its current state.
2. Think: The agent plans its next move using the current context and historical work-traces.
3. Do: The agent performs human-like operations: clicking, typing, switching windows, or sending messages.
4. Learn: The execution is written as a structured work-trace, which compounds into the agent's long-term memory to improve future Think stages.

The SightFlow desktop client allows external integrators to add chat capabilities (analyzing screenshots and generating replies) by supplying a Provider. A Provider consists of a manifest.json file and a bundled entry file (e.g., provider.bundle.js). The app handles downloading, installation, configuration UI generation, and passing the necessary screenshot and context to your code.

Provider Package Structure:
```text
provider-root/
  manifest.json
  provider.bundle.js

SightFlow is a cross-platform desktop client built on Electron, React, and TypeScript. To set up a development environment, ensure you have Node.js (LTS) and npm installed.

Installation steps

1. Install dependencies: Run npm install to set up the project.
2. Run in development: Use npm run dev to start the development server.
3. Build a release: Use the platform-specific build commands to generate executable files:
   • Windows: npm run build:win
   • macOS: npm run build:mac
   • Linux: npm run build:linux

On the first launch, you must select your target application and complete the region selection. Then, navigate to Settings to configure your API keys.

npm install
npm run dev
# For builds:
npm run build:win
npm run build:mac
npm run build:linux

To install a new Provider:

1. Open the SightFlow app settings page.
2. Locate the "Chat service config manifest URL" field.
3. Enter the URL or file path to your manifest.json.
4. Click Install.

Supported Address Formats:

• HTTPS: https://example.com/provider/manifest.json
• Local File: file:///absolute/path/to/provider/manifest.json

Note: Provide the path to the manifest.json, not the bundle file. The app will resolve the bundle path using the entry field in the manifest.