QtScrcpy

For users running QtScrcpy on macOS with Apple Silicon (M1/M2/M3), the recommended high-performance architecture (Scheme C) utilizes VideoToolbox for hardware-accelerated H.264 decoding and Metal for zero-copy GPU rendering. This approach significantly reduces CPU usage (from ~68% down to ~5-8%) by avoiding FFmpeg software decoding and bypassing the OpenGL-to-Metal translation layer.

Key Performance Benefits:

• Hardware Decoding: Uses the M1 Media Engine via VTDecompressionSession, resulting in ~0% CPU decoding load.
• Zero-Copy Data Flow: Leverages Unified Memory Architecture (UMA) by passing CVPixelBufferRef (IOSurface) directly from the decoder to Metal textures, eliminating CPU-to-GPU memory copies.
• Native Rendering: Uses Metal shaders to convert NV12 YUV data to BT.709 RGB, avoiding the slow paths associated with GL_LUMINANCE in OpenGL.

Architecture Overview

The data flow follows this pipeline:

1. Demuxer Thread: Receives AVPacket $\rightarrow$ VTDecoder::decode $\rightarrow$ creates CMSampleBuffer $\rightarrow$ hardware decodes via VTDecompressionSessionDecodeFrame $\rightarrow$ returns CVPixelBufferRef.
2. Main (GUI) Thread: Receives CVPixelBufferRef $\rightarrow$ CVMetalTextureCacheCreateTextureFromImage (creates zero-copy MTLTexture) $\rightarrow$ MTLCommandBuffer executes shaders $\rightarrow$ renders to a QWidget container.

The client's stream thread receives the H.264 video stream from the socket.

• Decoding: If a decoder is present (i.e., --no-display is not set), it uses libav (FFmpeg) to decode the stream and notifies the main thread when a new frame is ready.
• Double Buffering: To minimize latency, the client maintains two frames in memory: a decoding frame (written by the decoder thread) and a rendering frame (rendered as a texture in the main thread). The threads swap these frames using proper synchronization.
• Recording: If --record is enabled, the raw H.264 packets are muxed into an output video file by a recorder component.

                                   +----------+      +----------+ 
                              ---> | decoder  | ---> |  screen  | 
             +---------+     /     +----------+      +----------+ 
 socket ---> | stream  | ---- 
             +---------+     \ 
                              ---> | recorder | 

QtScrcpy operates using a client-server model consisting of two main components:

1. The Server (scrcpy-server): A Java application executed on the Android device as the shell user. It captures the screen, encodes it into an H.264 video stream, and receives input events (keyboard/mouse) from the client to inject them into the device.
2. The Client (scrcpy binary): Executed on the host computer. It is responsible for pushing the server to the device, starting its execution, decoding the video stream (using libav/FFmpeg), displaying it (using SDL), and capturing user input events to send back to the server.

Communication Flow:

• Video: The server sends a raw H.264 stream; the client decodes and displays it with minimal buffering to reduce latency.
• Input: The client captures keyboard/mouse events and transmits them to the server, which injects them into the Android system.
• Network Roles: At the application level, the server serves video and the client controls the device. However, at the network level, the client opens a server socket and listens, while the server connects to the client to avoid race conditions.

The key map JSON file consists of several top-level configuration sections:

• switchKey: The key used to toggle between the default mapping and your custom mapping.
• mouseMoveMap: Configuration for mouse movement mapping (useful for FPS games to control camera/vision).
• keyMapNodes: A JSON array containing all general key mappings (mapping keyboard keys to finger clicks/drags).

Mouse Movement Mapping (mouseMoveMap)

This maps mouse movement to a finger drag operation. When enabled, the mouse is hidden and its movement is restricted to a specific range.

For high-performance screen casting on Apple Silicon (M1/M2/M3), use the VideoToolbox + Metal pipeline. This approach utilizes hardware-accelerated decoding via VTDecompressionSession and zero-copy rendering via Metal and IOSurface.

Key architectural components:

• VTDecoder: Manages VTDecompressionSession and handles AVCDecoderConfigurationRecord parsing for SPS/PPS extraction.
• MetalVideoWindow: A QWindow subclass that uses CAMetalLayer and CVMetalTextureCache to render CVPixelBuffer frames with zero-copy.
• Zero-Copy Data Flow: AVPacket data is wrapped into a CMSampleBuffer using kCFAllocatorNull to avoid copying, and CVPixelBuffer is converted to MTLTexture via CVMetalTextureCache.

/* High-level flow for M1 acceleration */
// 1. Demuxer pushes AVPacket
// 2. VTDecoder parses SPS/PPS and creates VTDecompressionSession
// 3. VTDecoder wraps AVPacket data into CMSampleBuffer (zero-copy)
// 4. VTDecompressionSession decodes to CVPixelBuffer
// 5. MetalVideoWindow renders CVPixelBuffer via CVMetalTextureCache