repository·main·Indexed 41 days ago
https://github.com/microsoft/vscode-languagedetectionAn npm package (v1.0.23) that uses machine learning and the guesslang model to detect source code languages. It provides the ModelOperations class for running detection and returning ModelResult objects sorted by confidence, along with a diagnostic CLI for testing model results.
Install the package using npm or yarn to use machine learning-based language detection in your project.
npm install --save @vscode/vscode-languagedetection
# or using yarn
yarn add @vscode/vscode-languagedetectionFollow these steps to set up a local development environment for this repository:
npm install to install dependencies.npm run watch to start the development build process.Run npm test to execute the test suite.
To create a production-ready package:
npm run build.npm pack to create a tarball.To publish the package to a registry, run npm publish.
To report bugs, request new features, or ask questions about using vscode-languagedetection, use the GitHub Issues tracker.
Before filing a new issue, search the existing issues to ensure your topic has not already been addressed. If no relevant issue exists, create a new Issue for your bug report or feature request.
To detect the language of a code string, instantiate ModelOperations and call its runModel method. The method returns an array of objects containing languageId and confidence scores, sorted from highest to lowest confidence.
import { ModelOperations } from "@vscode/vscode-languagedetection";
const modulOperations = new ModelOperations();
const result = await modulOperations.runModel(`
function makeThing(): Thing {
let size = 0;
return {
// ... code snippet ...
};
}
`);The ModelOperations constructor accepts three optional parameters to customize how the model is loaded and how content is processed. This is useful for non-Node.js environments or fine-tuning detection sensitivity.
If you are not in a standard Node.js environment, you can provide custom functions to load the model files:
modelJSONFunc?: () => Promise<any>: A function that must return a JSON.parse() object of the model configuration.weightsFunc?: () => Promise<ArrayBuffer>: A function that returns the model weights as an ArrayBuffer.The third parameter is an options object with the following keys:
minContentSize?: number: The minimum number of characters required in a string for detection to occur. Defaults to 20.maxContentSize?: number: The maximum number of characters from the string that will be used for detection. Defaults to 100000.When instantiating ModelOperations, you can provide a ModelOperationsOptions object to customize how the model and its data are loaded and how text is processed.
| Option | Type | Default | Description |
|---|---|---|---|
modelJsonLoaderFunc | () => Promise<{ [key:string]: any }> | Node.js file system loader | A function that returns the model's JSON topology. |
weightsLoaderFunc | () => Promise<ArrayBuffer> | Node.js file system loader | A function that returns the model's weight binary data. |
minContentSize | number | 20 | The minimum number of characters required in the input string to attempt detection. If content is shorter, an empty array is returned. |
maxContentSize | number | 100000 | The maximum number of characters to process. Content exceeding this is truncated to prevent RangeError: Maximum call stack size exceeded in TensorFlow.js. |
normalizeNewline | boolean | true | If true, converts \r\n newlines to \n before processing. |
const options: ModelOperationsOptions = {
minContentSize: 50,
maxContentSize: 5000,
normalizeNewline: true,
// Custom loaders for non-Node environments or specific paths
modelJsonLoaderFunc: async () => { /* fetch or load JSON */ },
weightsLoaderFunc: async () => { /* fetch or load ArrayBuffer */ }
};
const modelOps = new ModelOperations(options);The ModelOperations class is the primary interface for running the language detection model. It takes text content and returns an array of ModelResult objects, sorted by confidence in descending order.
To use it, instantiate ModelOperations (optionally providing configuration) and call runModel(content). Note that the model is loaded lazily upon the first call to runModel.
import { ModelOperations } from '@vscode/vscode-languagedetection';
const modelOps = new ModelOperations();
const results = await modelOps.runModel('some text content');
// results is an array of { languageId: string, confidence: number }
for (const result of results) {
console.log(`Language: ${result.languageId}, Confidence: ${result.confidence}`);
}
// Clean up resources when finished
modelOps.dispose();Because ModelOperations manages a TensorFlow.js GraphModel, you should call dispose() to release the underlying model memory when it is no longer needed.
const modelOps = new ModelOperations();
// ... use modelOps ...
modelOps.dispose();The vscode-languagedetection CLI is a diagnostic tool intended for testing and inspecting the results of the @vscode/vscode-languagedetection model.
Warning: This CLI is for diagnostic purposes only and should not be used as a dependency in production systems.
To use the CLI, pass the text content you wish to analyze as a command-line argument. The CLI requires the input content to be longer than 20 characters to perform detection.
# Example usage (passing content as an argument)
node cli/index.js "This is a sample string of text to detect the language for."
# The output will be the JSON result from the model
# {"language": "english", ...}The ModelResult interface represents the output of a single language prediction from the model.
export interface ModelResult {
languageId: string;
confidence: number;
}