DOCSEARCHbyAlgolia
Submit Docs

react-native-fast-opencv

repository·main·Indexed 42 days ago

https://github.com/lukaszkurantdev/react-native-fast-opencv

A high-performance OpenCV port for React Native that uses JSI to expose the OpenCV C++ API to JavaScript. Version 1.0.1 supports the New Architecture and React Native 0.85+. It provides efficient image processing via the Mat object and integrates with react-native-vision-camera and react-native-worklets-core.

Tokens
32.1K
Snippets
143
Records
164
Agent score
53%

What's inside react-native-fast-opencv

  1. Overview of react-native-fast-opencv

    main

    react-native-fast-opencv is a high-performance port of OpenCV for React Native designed to enable simple image processing operations directly from JavaScript code.

    Key Features:

    • Performance: Powered by JSI and uses the OpenCV C++ API for high-speed execution.
    • Integration: Designed for easy usage with react-native-vision-camera and react-native-worklets-core.
    • Standalone: Can be used independently without requiring integration with other libraries.

  2. How function dispatch works for custom C++ functions

    main

    The OpenCV object uses dynamic resolution for function calls. When you call a method on the OpenCV object in JavaScript that is not a built-in factory (like Mat or Scalar), the library automatically routes the call to FOCV_Function::invoke.

    Specifically, the OpenCVPlugin::get method in cpp/react-native-fast-opencv.cpp intercepts unknown property names and creates a host function. This host function forwards the call to FOCV_Function::invoke, passing the property name (the function name you called in JS) as the first argument in the arguments array.

  3. Manage memory for OpenCV objects

    main

    OpenCV objects are held in native C++ memory and exposed to JS as Host objects.

    Automatic Management

    Memory is managed by the JavaScript garbage collector (GC). When a Host object goes out of scope, the underlying native resources are automatically freed. The runtime reports the size of native buffers as external memory pressure to help the GC schedule collections.

    Manual Release

    While not strictly required, you should call .release() manually in high-performance scenarios to avoid memory spikes:

    • Inside Frame Processors: To prevent memory buildup at high frame rates.
    • In Tight Loops: When allocating many large objects (like Mat) rapidly.
    • Heavy Computations: To free large buffers immediately after use.
    const mat = Mat.createFromBase64(base64string);
// heavy calculations ...
mat.release(); // optional: free immediately instead of waiting for GC

  4. Manage Memory in V1

    main

    In V1, manual buffer cleanup via OpenCV.clearBuffers() is no longer required. Native memory is automatically reclaimed by the garbage collector when Host objects go out of scope.

    Optional Eager Release: If you are working with very heavy workloads (e.g., many large objects in a tight loop or high frame-rate frame processors), you can call .release() on an object to free its native memory immediately instead of waiting for the GC.

  5. Understand OpenCV versioning and platform differences

    main

    The library links prebuilt OpenCV binaries automatically during installation (pod install on iOS, Gradle on Android). You do not need to manually add OpenCV to your project.

    Important: Because the bundled OpenCV versions differ by platform, some functions may exhibit slightly different behaviors. Always test critical OpenCV operations on both iOS and Android.

    PlatformOpenCV versionSource
    iOS4.9.0FastOpenCV-iOS pod
    Android4.12.0org.opencv:opencv:4.12.0 (Maven Central)

  7. Implement real-time object detection with Vision Camera

    main

    To perform real-time detection, integrate react-native-fast-opencv with react-native-vision-camera's frame processor. The recommended workflow involves:

    1. Scaling: Use react-native-vision-camera-resizer to scale down the incoming frame for faster processing.
    2. Buffer Conversion: Convert the resized pixel buffer into a Uint8Array.
    3. Mat Creation: Create an OpenCV Mat object from the buffer using Mat.createFromBuffer.
    4. Processing: Apply OpenCV functions (like cvtColor, inRange, findContours) to identify objects.
    5. Rendering: Use react-native-vision-camera-skia within the render callback to draw detection results (e.g., rectangles) on top of the camera frame.

    Requirements:

    • react-native-fast-opencv
    • react-native-vision-camera + WorkletsCore
    • react-native-vision-camera-resizer
    • react-native-vision-camera-skia
    import { OpenCV, Mat, DataTypes, Scalar, MatVector } from 'react-native-fast-opencv';
import { SkiaCamera } from 'react-native-vision-camera-skia';
import { useResizer } from 'react-native-vision-camera-resizer';

// Inside your component/frame processor:
const { resizer } = useResizer({
  width: 320,
  height: 180,
  channelOrder: 'bgr',
  dataType: 'uint8',
  pixelLayout: 'interleaved',
  scaleMode: 'contain',
});

// In the onFrame callback:
// 1. Resize and get pixels
const resized = resizer!.resize(frame) as GPUFrame;
const pixels = new Uint8Array(resized.getPixelBuffer());

// 2. Create Mat from buffer
const src = Mat.createFromBuffer('uint8', height, width, 3, pixels);

// 3. Process (Example: Color thresholding)
const dst = Mat.create(0, 0, DataTypes.CV_8U);
const lowerBound = Scalar.create(30, 60, 60);
const upperBound = Scalar.create(50, 255, 255);
OpenCV.cvtColor(src, dst, ColorConversionCodes.COLOR_BGR2HSV);
OpenCV.inRange(dst, lowerBound, upperBound, dst);

// 4. Cleanup
resized.dispose();
frame.dispose();

  9. Replace `invoke` with Direct OpenCV Calls

    main

    In V1, OpenCV functions are called directly on the OpenCV object rather than using the invoke string-based method.

    // 0.4.X
OpenCV.invoke('cvtColor', src, dst, ColorConversionCodes.COLOR_BGR2HSV);
OpenCV.invoke('inRange', dst, lower, upper, dst);

// V1
OpenCV.cvtColor(src, dst, ColorConversionCodes.COLOR_BGR2HSV);
OpenCV.inRange(dst, lower, upper, dst);

  11. Migrate from 0.4.X to V1

    main

    Version V1 is a major rewrite. Objects are now exposed as JSI Host objects with their own classes, and OpenCV functions are called directly on the OpenCV object.

    Key architectural changes:

    • Direct API calls: Instead of using OpenCV.invoke('name', ...args), call OpenCV.name(...args) directly.
    • Static Factory Methods: Instead of OpenCV.createObject(ObjectType.X, ...), use X.create(...) (e.g., Mat.create(...)).
    • Automatic Memory Management: Native memory is managed by the garbage collector (GC). You no longer need to call OpenCV.clearBuffers(). For extremely heavy workloads, you can optionally call .release() to free memory eagerly.

    Important Compatibility Note: V1 supports only the new architecture and has been tested on React Native 0.85 and later. Ensure your app is running the new architecture before upgrading.

    // V1 Example: Vision Camera frame processor body
const src = Mat.createFromVisionCameraFrameBuffer(height, width, 3, resized);
const dst = Mat.create(0, 0, DataTypes.CV_8U);
const lower = Scalar.create(30, 60, 60);
const upper = Scalar.create(50, 255, 255);
OpenCV.cvtColor(src, dst, ColorConversionCodes.COLOR_BGR2HSV);
OpenCV.inRange(dst, lower, upper, dst);

const channels = MatVector.create();
OpenCV.split(dst, channels);
const gray = channels.get(0);

// no clearBuffers — the GC handles it

  12. Build and run the iOS app

    main

    To run the app on iOS, you must first install CocoaPods dependencies.

    1. If this is the first time running the project, install CocoaPods using Ruby bundler:
      bundle install
    2. Install or update native dependencies:
      bundle exec pod install
    3. Build and run the app:
      # Using npm
npm run ios

# OR using Yarn
yarn ios
    # Using npm
npm run ios

# OR using Yarn
yarn ios