lingua-go

lingua-go is a Go library designed for language detection. It identifies the language of a given text fragment, making it suitable as a preprocessing step for NLP tasks like text classification, spell checking, or routing communications (e.g., emails) to the correct departments.

Key features include:

• High accuracy on short text: Unlike many other libraries, it performs well on single words, phrases, and short snippets (like social media posts).
• Scalable accuracy: Detection quality remains high even as the number of supported languages increases.
• Offline capability: It uses a combination of rule-based and statistical methods without relying on external APIs or word dictionaries.
• Zero configuration: It is designed to work out of the box with minimal setup.

Lingua's performance is evaluated using bundled test data for each supported language. The accuracy is measured across three distinct types of input data:

1. Single word detection: Words with a minimum length of 5 characters.
2. Word pair detection: Pairs of words with a minimum length of 10 characters.
3. Sentence detection: Complete grammatical sentences of various lengths.

The test data is derived from the Wortschatz corpora provided by Leipzig University. For each test, a random subset of 1,000 single words, 1,000 word pairs, and 1,000 sentences is extracted from corpora consisting of 10,000 sentences from various websites.

To build the project from the source repository, ensure you have at least Go version 1.18 installed. Follow these steps:

1. Clone the repository.
2. Navigate to the directory.
3. Run the build command.

git clone https://github.com/pemistahl/lingua-go.git
cd lingua-go
go build

You can reproduce the accuracy results by generating test reports for both classifiers and all languages.

Steps:

1. Navigate to the cmd directory.
2. Run the accuracy_reporter.go script.

cd cmd
go run accuracy_reporter.go

Note for gocld3: To run the reporter successfully with gocld3, you must install the exact version 3.17.3 of Google's protocol buffers.

Generated reports are written to the /accuracy-reports directory.

The source code includes an extensive unit test suite. You can execute the tests using the standard Go test command:

go test

To use lingua-go in your project, add it as a dependency using go get:

go get github.com/pemistahl/lingua-go

To perform basic language detection, use lingua.NewLanguageDetectorBuilder() to specify the set of candidate languages via FromLanguages(...), then call Build() to create a LanguageDetector. Use DetectLanguageOf(text) to identify the most likely language. It returns the detected lingua.Language and a boolean exists indicating if a reliable detection was made.

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    if language, exists := detector.DetectLanguageOf("languages are awesome"); exists {
        fmt.Println(language)
    }

    // Output: English
}

By default, Lingua uses lazy-loading, loading language models only when they are deemed relevant by the rule-based engine. This saves memory but can introduce latency on the first request.

For web services where low latency is critical, you can enable eager-loading (preloading all models into memory) using WithPreloadedLanguageModels() in the builder. Multiple LanguageDetector instances share the same language models in memory.

lingua.NewLanguageDetectorBuilder().
    FromAllLanguages().
    WithPreloadedLanguageModels().
    Build()

Lingua offers two modes to balance accuracy and resource usage:

1. High Accuracy (Default): High detection accuracy but high memory consumption (~1,800 MB if all models are loaded) and slower performance.
2. Low Accuracy Mode: Significantly lower memory consumption (~110 MB) and faster performance. It achieves this by loading only a small subset of language models.

Trade-off: Accuracy for short texts (less than 120 characters) drops significantly in low accuracy mode, but accuracy for longer texts remains mostly unaffected. Use WithLowAccuracyMode() to enable this.

lingua.NewLanguageDetectorBuilder().
    FromAllLanguages().
    WithLowAccuracyMode().
    Build()

The accuracy_reporter is a CLI tool used to benchmark the language detection accuracy of lingua-go against other libraries like cld3 and whatlanggo. It evaluates detection performance across three different input granularities:

1. Single words: Detection accuracy for individual words.
2. Word pairs: Detection accuracy for pairs of words.
3. Sentences: Detection accuracy for full sentences.

The tool iterates through all supported languages, reads test data from a language-testdata directory, and generates detailed reports in Markdown format and an aggregated CSV file containing comparative accuracy metrics.