TinyLD Documentation

Most language detection libraries use statistical analysis, meaning accuracy improves as the input text length increases. For reliable results, it is recommended to pass larger documents.

To achieve a ~95% accuracy threshold:

• tinyld reaches this threshold at approximately 24 characters.
• langdetect and cld reach this threshold at approximately 48 characters.

Most libraries become highly accurate once the input exceeds 512 characters.

The following commands are available for managing the TinyLD development environment using yarn:

• Install dependencies: yarn
• Build the project: yarn build
• Run tests: yarn test
• Lint and auto-fix code style: yarn lint

# Install
yarn

# Build
yarn build

# Test
yarn test

# Lint / Auto-fix code style problems
yarn lint

To use TinyLD in a Node.js project, install it using your preferred package manager.

# for npm users
npm install --save tinyld

# for yarn users
yarn add tinyld

Language detection accuracy in Tinyld is highly dependent on text length because the underlying n-gram algorithm requires a sufficient sample of character patterns to be effective.

Accuracy Benchmarks:

• ~24 characters: Reaches the ~95% accuracy threshold.
• ~12 characters: Accuracy falls to ~80% (barely usable).
• < 10 characters: Detection is essentially random.

Tips for better results:

• Avoid detecting single words; aim for full sentences.
• Use the TinyLD Playground to test how length affects your specific use case.
• Be aware that similar languages (e.g., Spanish and Catalan) or generic brand names can increase error rates.

Select a library based on your runtime environment and the nature of your input text.

By Platform

• NodeJS: Use TinyLD, langdetect, or node-cld for high speed and accuracy.
• Browser: Use TinyLD Light or franc-min for small bundle size. franc-min supports more languages but has lower accuracy.

By Usage

• Short text (e.g., chatbots, keywords, database entries): Use TinyLD or langdetect.
• Long text (e.g., documents, full webpages): Use node-cld or TinyLD.

Avoid

• franc-all: Low accuracy due to attempting to detect 400+ languages using only 3-grams.
• languagedetect: Not accurate enough for reliable production use.

Generating profiles is an optional, resource-intensive process that analyzes large amounts of text to build statistics (words and n-grams) for language identification. The resulting profiles are stored in git.

Prerequisites

You must have the datasets downloaded and placed in the correct local directories before running the command:

1. Tatoeba sentence export: Download from Tatoeba and extract to data/tatoeba.csv.
2. UDHR: Download from Unicode and extract to data/udhr/.

Steps

1. Prepare the datasets as described above.
2. Run yarn train to build statistics for each language.
3. Once profiles are generated, run yarn build to create a build containing the new data.

To use Tinyld in applications like chat where individual messages are often very short, you should implement a context buffer to maintain stability and accuracy.

Instead of detecting the language of only the most recent message, maintain a buffer of the user's recent history (e.g., the last 256 characters). Since users rarely switch languages abruptly mid-conversation, checking the accumulated buffer provides a much more reliable language signature than checking a single short message.

When using the --verbose flag, TinyLD outputs the internal detection process. This includes:

• Grams detection: Shows the unique 1-grams and 2-grams identified in the input text.
• Gram scoring: Shows how specific grams (like 'a t' or ' te') contribute to the scores of various languages (e.g., ind, tgl, epo, spa, etc.).
• Final Result: Displays the ranked list of detected languages with their lang code, accuracy, and score.

yarn tinyld --verbose "this is a text"

Tinyld provides different versions (flavors) optimized for different environments and trade-offs between size, speed, and accuracy. You can select a flavor by changing your import path:

• tinyld: The general-purpose version (~500KB) supporting 64 languages.
• tinyld/light: Optimized for browser usage (~70KB) supporting 24 languages.
• tinyld/heavy: (Coming soon) Optimized for backend usage (several MB) focusing exclusively on maximum accuracy.

import { detect } from 'tinyld'      // General purpose (~500KB, 64 languages)
import { detect } from 'tinyld/light' // Browser optimized (~70KB, 24 languages)
import { detect } from 'tinyld/heavy' // Backend optimized (High accuracy)

You can use TinyLD directly in the browser by importing the module from a CDN like jsDelivr.

<script type="module">
  import { detect } from 'https://cdn.jsdelivr.net/npm/tinyld@1.3.0/dist/tinyld.normal.browser.js'
  // ...
</script>