A.D.A V2 uses hand tracking to enable gesture-based control of the UI. The system interprets specific hand landmarks to trigger actions:
- Cursor Movement: The cursor position is derived from hand landmarks and mapped to screen coordinates.
- Pinch Gesture (Clicking): Triggered when the distance between the index finger tip and the thumb tip falls below a specific threshold (
0.05). A pinch triggers a click() event on the element currently under the cursor. - Fist Gesture (Dragging): Detected when all finger tips are closer to the wrist than their respective MCP (knuckle) joints. When a fist is detected over a draggable element (like
cad, browser, kasa, or printer), the system enters a drag state. - Stable Dragging: To prevent jitter, dragging is controlled by the movement of the wrist rather than the index finger. The system calculates the delta between the current wrist position and the last known wrist position to update the element's coordinates.
- Snapping: The cursor automatically snaps to interactive elements (buttons, inputs, selects, or
.draggable elements) when within a certain threshold, providing visual feedback via a snap-highlight class and glow effects.
// Pinch Detection (Distance between Index and Thumb)
const distance = Math.sqrt(
Math.pow(indexTip.x - thumbTip.x, 2) + Math.pow(indexTip.y - thumbTip.y, 2)
);
const isPinchNow = distance < 0.05; // Threshold
if (isPinchNow && !isPinching) {
// Click Triggered
const el = document.elementFromPoint(finalX, finalY);
if (el) {
const clickable = el.closest('button, input, a, [role="button"]');
if (clickable && typeof clickable.click === 'function') {
clickable.click();
} else if (typeof el.click === 'function') {
el.click();
}
}
}
// Fist Detection for Gesture-Based Dragging
const isFist = isFingerFolded(8, 5) && isFingerFolded(12, 9) && isFingerFolded(16, 13) && isFingerFolded(20, 17);
if (isFist && activeDragElementRef.current) {
const dx = wristScreenX - lastWristPosRef.current.x;
const dy = wristScreenY - lastWristPosRef.current.y;
if (Math.abs(dx) > 0.5 || Math.abs(dy) > 0.5) {
updateElementPosition(activeDragElementRef.current, dx, dy);
}
lastWristPosRef.current = { x: wristScreenX, y: wristScreenY };
}
