DOCSEARCHbyAlgolia
Submit Docs

jschardet

repository·main·Indexed 20 days ago

https://github.com/aadsm/jschardet

A high-performance character encoding detector for JavaScript and Node.js with zero runtime dependencies. Version 4.0.0-rc.1 is a TypeScript port of the Python chardet library, providing improved accuracy and throughput. It features a two-layer API consisting of a user-facing wrapper and a strict internal chardet API, supporting both single-match detection via detect(), multiple candidates via detectAll(), and incremental stream detection using UniversalDetector.

Tokens
7.5K
Snippets
31
Records
38
Agent score
72%

What's inside jschardet

  1. Understand jschardet 4.0.0 performance characteristics

    main

    jschardet 4.0.0 is a TypeScript port of chardet 7 and offers significant improvements over version 3.1.4 in accuracy, throughput, and memory efficiency, though it has a higher initial 'cold start' cost.

    Accuracy

    • Overall Accuracy: ~99.2% (compared to 42.0% in v3).
    • Language Detection: ~97.4% accuracy. Note that v3 does not support the language field.

    Throughput and Latency

    • Steady State: Much faster than v3, processing approximately 6× more files per second.
    • Tail Latency: Improved p95 latency compared to v3.

    Cold Start (First Call Latency)

    • Trade-off: The first call to detect() is slower in v4 (~45ms) than in v3 (~0.5ms). This is because v4 uses zlib-compressed bigram models that are lazily decompressed during the first detect() call. Subsequent calls benefit from the faster steady-state throughput.

    Memory Usage

    • Efficiency: v4 is significantly more memory-efficient. It uses a dense bigram model format (one 64 KiB lookup table per language) that is loaded once and shared. This avoids the high per-call sparse-map allocations found in v3, resulting in a much lower peak Resident Set Size (RSS).

  2. How bitwise flags work with EncodingEra and LanguageFilter

    main

    The library uses EncodingEra and LanguageFilter to allow users to restrict or prioritize certain types of encodings. While these are IntFlag enums in Python, in TypeScript they are implemented as as const objects containing power-of-two numbers.

    Because the members are plain numbers, you can use standard JavaScript bitwise operators (| for combining and & for checking) to work with them exactly as you would in Python.

  3. How model data is loaded and managed

    main

    The detection engine relies on pre-trained binary payloads for accuracy:

    • models.bin: Bigram frequency tables per language/encoding.
    • idf.bin: IDF weights for scoring input bigram profiles.
    • confusion.bin: Distinguishing-byte maps for resolving ties between similar single-byte encodings.

    In TypeScript, these are shipped as zlib-compressed base64 JS modules. To optimize startup time, they are lazy-loaded and decompressed only when the first detect() call is made. This ensures the library remains lightweight for users who do not perform character encoding detection.

  4. Understand jschardet's TypeScript declaration strategy

    main

    The project uses a flattened declaration strategy to support both ESM and CommonJS without requiring a types field in package.json. TypeScript resolves declarations by looking for files adjacent to the JavaScript entry points.

    • ESM: build/index.js is paired with build/index.d.ts.
    • CommonJS: build/index.cjs is paired with build/index.d.cts.

    Both declaration files are flattened versions of the internal types to prevent complex relative import issues (like TS1479) and to keep the public surface area clean.

  5. Understand how model data is loaded and decompressed

    main

    The jschardet library uses compressed binary models (models.bin, idf.bin, and confusion.bin) to perform character detection. Because browsers cannot load binary files from disk at runtime like Python, these models are embedded into JS modules as base64-encoded, zlib-compressed strings.

    Runtime Loading Behavior

    Each model module exports a readBytes() function. This function implements a lazy loading pattern:

    1. First call: Decompresses the base64 payload and caches the resulting Uint8Array.
    2. Subsequent calls: Returns the cached Uint8Array immediately without re-decompressing.

    Decompression Implementations

    The library uses different decompression paths depending on the environment:

    • Node.js: Uses the standard node:zlib library (zlib.inflateSync()).
    • Browser: Uses a specialized, lightweight DEFLATE decoder designed to handle the specific compression strategy used by the project.

  6. Understand the jschardet Two-Layer API

    main

    The project provides two distinct API layers depending on your needs:

    1. Public jschardet API (src/index.ts): This is the recommended user-facing wrapper. It is designed for ease of use and compatibility with previous versions of jschardet. It accepts both string and Uint8Array as input and returns an IDetectedMap shape.

    2. Internal chardet API (src/chardet.ts, src/detector.ts): This is a faithful TypeScript port of the original Python chardet public surface. It is more strict, accepting only Uint8Array. It includes features like detect(), detect_all(), and the UniversalDetector class (which supports streaming via feed() and close()). It also supports advanced options like encodingEra, preferSuperset, and compatNames.

  7. Benchmark decompression performance

    main

    If you are modifying the encoder strategy or the browser decoder implementation, you should run the built-in benchmark script to measure the first-call decompression cost of each payload and detect performance regressions.

    Run the following command from the repository root:

    npm run scripts/decompress-benchmark.js

  8. Choose the correct module format for your environment

    main

    The jschardet package provides different distribution bundles depending on whether you are running in Node.js or a browser. Choosing the correct entry point ensures optimal performance (e.g., using Node's native zlib instead of a slower JS-based inflate) and compatibility with your module system.

    ConsumerResolves toNotes
    import 'jschardet'build/index.jsESM, Node-native. Uses node:zlib for speed.
    require('jschardet')build/index.cjsCommonJS, Node-native. Uses node:zlib for speed.
    <script src="jschardet.min.js">dist/jschardet.min.jsIIFE, attaches jschardet to window.
    AMD loader (define)dist/jschardet.min.jsVia the define() call in the bundle footer.
    Browser bundlersdist/jschardet.esm.min.jsESM, uses a bundled JS decoder (no zlib).
    // Node.js ESM
import jschardet from 'jschardet';

// Node.js CommonJS
const jschardet = require('jschardet');

// Browser (Global)
// <script src="jschardet.min.js"></script>
console.log(window.jschardet);

  9. Verify parity with Python chardet

    main

    To ensure the TypeScript port matches the Python chardet implementation, use the parity check script. This runs detect() on both implementations against the tests/data/ corpus and diffs the results.

    Requirements:

    • python3 (with the chardet submodule checked out)
    • npx tsx
    • A populated tests/data/ corpus (run npm run test:accuracy if missing)

    Usage: Run the script and redirect stdout to capture the Markdown report:

    tests/compare-detect/run.sh > /tmp/parity.md
    tests/compare-detect/run.sh > /tmp/parity.md

  10. How to use byte literals in ported tests

    main

    When porting tests from Python to TypeScript, use specific patterns to ensure the byte sequences remain grep-able against the original Python chardet source:

    • Array of integers: For bytes([0x48, 0x65, ...]), use new Uint8Array([0x48, 0x65, ...]).
    • Mixed hex + ASCII: For b'\xef\xbb\xbfHello', use a bytes(s) helper function:
      function bytes(s: string): Uint8Array {
  return Uint8Array.from(s, c => c.charCodeAt(0));
}
    • Pure ASCII: For b'Hello world', use new TextEncoder().encode('Hello world').
    • Mixed sequences: Use a concat(...arrays) helper to join multiple Uint8Arrays.
    • Specific encodings: If a Python test uses a specific encoding (e.g., "text".encode("iso-8859-7")), inline the sequence as a new Uint8Array([...]) literal and add a comment naming the source encoding.
    function bytes(s: string): Uint8Array {
  return Uint8Array.from(s, c => c.charCodeAt(0));
}

  11. Reproduce jschardet performance benchmarks

    main

    You can run the performance benchmarks locally to measure accuracy, throughput, cold start, and memory usage. These commands run the benchmarking suite against the chardet test corpus.

    npm run benchmark:accuracy
npm run benchmark:throughput
npm run benchmark:coldstart
npm run benchmark:memory

  12. Run the jschardet test suites

    main

    The project uses Vitest and provides several configurations via npm scripts to run different types of tests:

    • Full suite (Node.js): npm test runs the complete test suite under Node.js.
    • Browser suite: npm run test:browser runs the suite in a real Chromium browser via Playwright.
    • Accuracy gate: npm run test:accuracy runs a corpus accuracy gate. Note: This will clone approximately 100 MB of data into tests/data/ on its first run.
    • Bundle tests: npm run test:bundles runs tests against the generated bundles in dist/ using headless Chromium. Requirement: You must run npm run build:bundles before executing this.
    npm test
npm run test:browser
npm run test:accuracy
npm run test:bundles