repository·main·Indexed 42 days ago
https://github.com/capawesome-team/capacitor-mlkitA suite of plugins that brings Google's ML Kit machine learning capabilities to Capacitor applications on Android and iOS, including features for barcode scanning via custom UIs, ready-to-use native interfaces, and static image analysis.
Capacitor ML Kit is a collection of Capacitor plugins that provide access to the Google ML Kit SDKs within Capacitor applications.
Key features include:
The capacitor-mlkit repository provides a collection of individual plugins, each wrapping specific Google ML Kit capabilities for Capacitor applications. You should install only the specific plugin package required for your use case.
| Feature | Package Name |
|---|---|
| Barcode Scanning | @capacitor-mlkit/barcode-scanning |
| Digital Ink Recognition | @capacitor-mlkit/digital-ink-recognition |
| Document Scanner | @capacitor-mlkit/document-scanner |
| Entity Extraction | @capacitor-mlkit/entity-extraction |
| Face Detection | @capacitor-mlkit/face-detection |
| Face Mesh Detection | @capacitor-mlkit/face-mesh-detection |
| GenAI Image Description | @capacitor-mlkit/genai-image-description |
| GenAI Prompt | @capacitor-mlkit/genai-prompt |
| GenAI Proofreading | @capacitor-mlkit/genai-proofreading |
| GenAI Rewriting | @capacitor-mlkit/genai-rewriting |
| GenAI Speech Recognition | @capacitor-mlkit/genai-speech-recognition |
| GenAI Summarization | @capacitor-mlkit/genai-summarization |
| Image Labeling | @capacitor-mlkit/image-labeling |
| Language Identification | @capacitor-mlkit/language-identification |
| Object Detection | @capacitor-mlkit/object-detection |
| Pose Detection | @capacitor-mlkit/pose-detection |
| Selfie Segmentation | @capacitor-mlkit/selfie-segmentation |
| Smart Reply | @capacitor-mlkit/smart-reply |
| Subject Segmentation | @capacitor-mlkit/subject-segmentation |
| Text Recognition | @capacitor-mlkit/text-recognition |
| Translation | @capacitor-mlkit/translation |
The rewriting feature is performed on-device using Gemini Nano. To use this plugin, the device must meet the following requirements:
Use checkFeatureStatus(...) to verify if the device is compatible.
The ML Kit Document Scanner models, logic, and UI are dynamically downloaded by Google Play services. Because of this, the scanner might not be available on the first use.
To ensure a smooth user experience, you should:
isGoogleDocumentScannerModuleAvailable().installGoogleDocumentScannerModule().googleDocumentScannerModuleInstallProgress event to provide progress updates to the user.Note: This plugin is only available on Android via Google Play services. All methods will reject on other platforms.
// Example workflow for checking and installing the module
const available = await DocumentScanner.isGoogleDocumentScannerModuleAvailable();
if (!available) {
// Listen for progress updates
const listener = await DocumentScanner.addListener('googleDocumentScannerModuleInstallProgress', (info) => {
console.log('Installation progress:', info);
});
try {
await DocumentScanner.installGoogleDocumentScannerModule();
console.log('Installation complete');
} catch (err) {
console.error('Installation failed', err);
} finally {
listener.remove();
}
}The Text Recognition plugin supports five specific scripts. Each script uses a separate model that is bundled with your app to ensure they are available at runtime without additional downloads. You must use the script option in your request to specify which model to use.
Supported scripts:
The plugin supports two recognition modes on Android, each with different requirements:
Uses a traditional speech model.
en-US, fr-FR (beta), it-IT (beta), de-DE (beta), es-ES (beta), hi-IN (beta), ja-JP (beta), pt-BR (beta), tr-TR (beta), pl-PL (beta), cmn-Hans-CN (beta), ko-KR (beta), cmn-Hant-TW (beta), ru-RU (beta), vi-VN (beta).Uses a GenAI model.
en-US, ko-KR, es-ES, fr-FR, de-DE, it-IT, pt-PT, cmn-Hans-CN, cmn-Hant-TW, ja-JP, th-TH, ru-RU, nl-NL (beta), da-DK (beta), sv-SE (beta), pl-PL (beta), hi-IN (beta), vi-VN (beta), id-ID (beta), ar-SA (beta), tr-TR (beta).Speech recognition results are delivered via events. You must register listeners for these events before calling startRecognition(...) to ensure you don't miss any data.
partialResult: Emitted with intermediate text results while the user is actively speaking.finalResult: Emitted once the recognition session is complete with the final recognized text.// Example pattern for receiving results
SpeechRecognition.addListener('partialResult', (data) => {
console.log('Intermediate:', data.text);
});
SpeechRecognition.addListener('finalResult', (data) => {
console.log('Final:', data.text);
});
await SpeechRecognition.startRecognition();When using the Face Detection plugin, you can request different types of facial data:
landmarkMode option.contourMode option. Note that contours are only returned for up to 5 faces.Both are used to identify specific facial features but differ in the density of data provided.
The translation plugin performs all work on-device. To support offline translation, you must manage language models (approximately 30MB each) using the following methods:
downloadModel({ language }): Downloads the model for the specified language. It is best practice to do this over WiFi.getDownloadedModels(): Returns a list of languages that currently have models downloaded on the device.deleteDownloadedModel({ language }): Removes the model for the specified language to free up storage space.// Download a model
await Translation.downloadModel({ language: Language.French });
// Check what is already downloaded
const { languages } = await Translation.getDownloadedModels();
// Delete a model
await Translation.deleteDownloadedModel({ language: Language.French });The processImage(...) method is supported on the following platforms:
Web support is not available because the underlying Google ML Kit Pose Detection SDK does not support the Web platform.
A FaceMesh object represents a detected face. Depending on the useCase selected, it contains different levels of detail:
bounds: A Rect representing the axis-aligned bounding rectangle of the face.faceMeshPoints: A list of FaceMeshPoint objects representing the entire detected face (available in FaceMesh use case).contours: A Contours object containing specific facial feature landmarks (e.g., leftEye, noseBridge, upperLipTop).triangles: A list of Triangle objects representing logical surface areas of the face (available in FaceMesh use case).The plugin offers two modes of speech recognition, both performed entirely on-device:
Because availability varies, always use checkFeatureStatus(...) to determine which mode is supported on the user's device before attempting recognition.