DOCSEARCHbyAlgolia
Submit Docs

Remote WebView Client

repository·main·Indexed 40 days ago

https://github.com/strange-v/remotewebviewclient

An ESPHome-compatible component and browser-based test client that allows ESP32 devices with displays to render complex web pages by streaming image tiles from a headless browser server via WebSockets.

Tokens
6.2K
Snippets
17
Records
22
Agent score
1%

What's inside Remote WebView Client

  1. What is Remote WebView Client

    main
    Remote WebView Client is a client designed to connect to a Remote WebView Server. The server acts as a headless browser that renders web pages (such as Home Assistant dashboards) and streams them as image tiles over WebSocket to lightweight clients, typically ESP32-based displays. This allows low-power microcontrollers to display complex web interfaces.

  2. Features of the Browser Test Client

    main

    The Browser Test Client provides the following capabilities for testing remote WebView implementations:

    • WebSocket Connection: Includes built-in reconnect and keepalive logic.
    • Binary Protocol Support: Compatible with Frame, Touch, OpenURL, and Keepalive packets.
    • Rendering: Uses JPEG tile rendering to an HTML5 canvas.
    • Input Simulation: Supports pointer/touch forwarding (down/move/up) with move coalescing.

  4. Run the Browser Test Client locally

    main

    The Browser Test Client allows for faster development loops by simulating a remote WebView connection in a browser, avoiding the need to reflash ESP devices.

    To run the client locally:

    1. Install dependencies using npm install.
    2. Start the development server using npm run dev.
    3. Open the provided local Vite URL in your browser.
    4. Configure the server and display parameters.
    5. Click Connect to establish a connection.
    6. Use the Send open_url feature to perform navigation tests.
    npm install
npm run dev

  5. Performance recommendations for Remote WebView Client

    main

    To optimize the streaming experience, follow these guidelines:

    • Full-frame updates: Set full_frame_tile_count to 1 for the most efficient full-screen updates if your network and device memory allow it.
    • Frame rate: Keep every_nth_frame set to 1 to avoid missing changes. Increase min_frame_interval to be slightly larger than the render time reported by the self-test (use url: "self-test" to find this value).
    • Bandwidth and Speed:
      • Set max_bytes_per_msg larger than your maximum tile size (full-frame or partial).
      • Start jpeg_quality at 85. If you need more speed or lower bandwidth, drop it toward 70–75.
    • Color issues: If colors appear swapped or tinted, set big_endian: false.
    • Troubleshooting Red Tiles: A red tile or red screen indicates a tile payload exceeded max_bytes_per_msg. To fix this, increase max_bytes_per_msg or reduce tile_size/jpeg_quality.

  6. Install Remote WebView Client as an ESPHome component

    main

    The client is implemented as an ESPHome external component. To install it, add the component to your ESPHome YAML configuration using the external_components block pointing to the GitHub repository. You must also ensure your ESP32 configuration includes necessary components like esp_websocket_client and a JPEG decoder (e.g., bitbank2/jpegdec).

    external_components:
  - source: github://strange-v/RemoteWebViewClient@main
    refresh: 0s
    components: [ remote_webview ]

remote_webview:
  id: rwv
  server: 172.16.0.252:8081
  url: http://172.16.0.252:8123/dashboard-mobile/0

  7. Configure the remote_webview component

    main

    The remote_webview component is configured in ESPHome to define the connection to the server and the initial page to load. You can also tune performance parameters like tile counts, JPEG quality, and message size limits.

    remote_webview:
  id: rwv
  server: 172.16.0.252:8081
  url: http://172.16.0.252:8123/dashboard-mobile/0
  full_frame_tile_count: 1
  max_bytes_per_msg: 61440
  jpeg_quality: 85

  8. Configure RemoteWebViewBrowserClient connection settings

    main

    The client uses a UiSettings structure to manage connection parameters. These settings are persisted in the browser's localStorage under the key rwv-browser-client-settings.

    Settings Schema

    KeyTypeDescription
    serverstringThe server address (e.g., 172.16.0.252:8081)
    widthnumberViewport width
    heightnumberViewport height
    fullFrameTileCountnumberfftc - number of tiles for full frame
    scalePercentnumberDisplay scale (e.g., 100 for 100%)
    urlstringThe initial URL to open
    qualitynumberJPEG quality (1-100)
    maxBytesPerMsgnumbermbpm - maximum bytes per message

    Default Values

    • server: 127.0.0.1:8081
    • width: 480
    • height: 480
    • fullFrameTileCount: 1
    • scalePercent: 100
    • url: http://127.0.0.1:8123/
    • quality: 85
    • maxBytesPerMsg: 61440

  9. Reference: remote_webview configuration parameters

    main

    The following parameters are available for the remote_webview component in ESPHome:

    | YAML key                | Type      | Required | Example                          | Description |
|-------------------------|-----------|:--------:|----------------------------------|-------------|
| `display_id`            | id        | ❌       | `panel`                           | Display to draw on. Optional, if only one display is defined in the YAML.| |
| `touchscreen_id`        | id        | ❌       | `touch`                           | Touch input source. Optional, if only one touchscreen is defined in the YAML. |
| `server`                | string    | ✅       | `172.16.0.252:8081`              | WebSocket server address. Must be `hostname_or_IP:port`. |
| `url`                   | string    | ✅       | `http://…/dashboard`             | Page to open on connect. |
| `device_id`             | string    | ❌       | `"my-device"` or auto (`esp32-<mac>`) | Identifier used by the server. If not set, the component derives `esp32-<mac>` from the chip MAC and still sends it. |
| `tile_size`             | int       | ❌       | `32`                              | Tile edge size in pixels. Helps the server choose tile packing; it’s best to keep it a multiple of 16. |
| `full_frame_tile_count` | int       | ❌       | `4`                               | Number of tiles the server should use for full-frame updates. |
| `full_frame_area_threshold` | float | ❌       | `0.50`                            | Area delta (fraction of screen) above which the server should send a full frame. |
| `full_frame_every`      | int       | ❌       | `50`                              | Force a full-frame update every N frames (0 disables). |
| `every_nth_frame`       | int       | ❌       | `1`                               | Frame-rate divider. Server should send only every Nth frame. |
| `min_frame_interval`    | int (ms)  | ❌       | `80`                              | Minimum time between frames on the wire, in milliseconds. |
| `jpeg_quality`          | int       | ❌       | `85`                              | JPEG quality hint for the server’s encoder. |
| `max_bytes_per_msg`     | int (B)   | ❌       | `14336` or `61440`                | Upper bound for a single WS binary message. |
| `big_endian`            | bool      | ❌       | `true` or `false`                 | Use big-endian RGB565 pixel order for JPEG output (set false for little-endian panels). Default is `true`. |
| `rotation`              | int       | ❌       | 0, 90, 180, 270                   | Enables software rotation for both the display and touchscreen. |
| `on_frame_update`       | action    | ❌       | `- logger.log: "..."`            | Action that gets triggered each time the display updates (maximum of 1 trigger per second) |
| `current_url_sensor`    | text_sensor | ❌      | `name: "Current URL"`            | Exposes the URL currently loaded in the server's headless browser as an ESPHome Text Sensor. Supports on_value automations. |

  10. Build client-to-server command packets

    main

    The protocol provides helper functions to construct binary packets for sending commands from the client to the remote WebView server:

    • buildTouchPacket(type, pointerId, x, y): Creates a Touch message (MsgType.Touch) for touch input. type must be a TouchType (Down, Move, or Up).
    • buildOpenUrlPacket(url, flags): Creates an OpenURL message (MsgType.OpenURL) to instruct the remote browser to navigate to a specific URL.
    • buildKeepalivePacket(): Creates a Keepalive message (MsgType.Keepalive) to maintain the connection.
    import { buildTouchPacket, buildOpenUrlPacket, buildKeepalivePacket, TouchType } from './protocol';

// Send a touch down event
const touchDown = buildTouchPacket(TouchType.Down, 1, 100, 200);

// Request navigation
const nav = buildOpenUrlPacket('https://google.com');

// Send keepalive
const keepalive = buildKeepalivePacket();

  11. Navigate to a URL via sendOpenUrl()

    main

    Use sendOpenUrl to instruct the remote webview to navigate to a specific URL. This method returns a boolean indicating whether the command was successfully sent (returns false if the WebSocket is not in an OPEN state).

    const success = client.sendOpenUrl('https://example.com');
if (!success) {
  console.error('Failed to send URL command: WebSocket not connected');
}

  12. Build a WebSocket URI with `buildWsUri`

    main

    Use buildWsUri to construct the connection string for the WebSocket server. The function ensures the server address uses a ws:// or wss:// scheme and appends required and optional query parameters for session configuration.

    Required Parameters:

    • server: The base URL of the WebSocket server.
    • w: Width of the viewport.
    • h: Height of the viewport.

    Optional Parameters (QueryOptions):

    • id: A unique identifier for the session (defaults to browser-[8-char-uuid] if not provided).
    • r, ts, fftc, ffat, ffe, enf, mfi, q, mbpm: Various numeric configuration/tuning parameters.
    import { buildWsUri } from './protocol';

const uri = buildWsUri('ws://192.168.1.100:8080', {
  w: 480,
  h: 480,
  id: 'my-custom-client'
});
// Result: "ws://192.168.1.100:8080/?id=my-custom-client&w=480&h=480"