UTF Unknown

repository·master·Indexed 18 days ago

https://github.com/charsetdetector/utf-unknown

A .NET library for detecting the character encoding of files, streams, and byte arrays. A redesigned, object-oriented port of Ude and uchardet (Mozilla Universal Charset Detector), it supports .NET 8 and .NET Standard 2.0. The CharsetDetector class provides synchronous and asynchronous methods to return DetectionResult objects containing encoding names, confidence levels, and System.Text.Encoding objects.

Tokens
2K
Snippets
2
Records
3
Agent score
14%

What's inside utf-unknown

  1. Detect character sets using CharsetDetector

    master

    The CharsetDetector class provides static methods to detect the character encoding of files, streams, or byte arrays. Detection results are returned as a DetectionResult object, which contains a list of possible matches (Details) and the most likely match (Detected).

    Synchronous Detection

    • CharsetDetector.DetectFromFile(string path) or (FileInfo fileInfo): Detects encoding from a file.
    • CharsetDetector.DetectFromStream(Stream stream): Detects encoding from a stream.
    • CharsetDetector.DetectFromBytes(byte[] byteArray): Detects encoding from a byte array.

    Asynchronous Detection

    • await CharsetDetector.DetectFromFileAsync(string path, CancellationToken cancellationToken)
    • await CharsetDetector.DetectFromStreamAsync(Stream stream, CancellationToken cancellationToken)

    Accessing Results

    Once you have a DetectionResult, you can access the DetectionDetail for the best match via the .Detected property. From a DetectionDetail, you can retrieve:

    • EncodingName: The string alias of the encoding.
    • Encoding: The System.Text.Encoding object (may be null if the encoding is not available in the current environment).
    • Confidence: A float between 0 and 1 representing the detection certainty.
    • Details: An IList<DetectionDetail> containing all potential matches.
    // Detect from File 
    DetectionResult result = CharsetDetector.DetectFromFile("path/to/file.txt");
    
    // Get the best Detection
    DetectionDetail resultDetected = result.Detected;
    
    // Get the alias of the found encoding
    string encodingName = resultDetected.EncodingName;
    
    // Get the System.Text.Encoding of the found encoding (can be null if not available)
    Encoding encoding = resultDetected.Encoding;
    
    // Get the confidence of the found encoding (between 0 and 1)
    float confidence = resultDetected.Confidence;
    
    // Get all the details of the result
    IList<DetectionDetail> allDetails = result.Details;
  2. Supported Platforms and Compatibility

    master

    UTF Unknown supports the following platforms:

    • .NET 8
    • .NET Standard 2.0

    Note: Support for .NET 6 was removed in version 2.7.0. You can still register your own EncodingProvider so that Encoding.GetEncoding(...) attempts to find the detected encoding in your provider first.

  3. Reference of supported character sets

    master

    The library supports various encodings, including those with Byte Order Marks (BOM) and those without. If an encoding alias is not directly available in System.Text.Encoding, DetectionDetail.Encoding provides a suitable replacement.

    Encoding Replacements:

    • cp949 $\rightarrow$ use ks_c_5601-1987
    • iso-2022-cn $\rightarrow$ use x-cp50227
    ### Encodings with BOM:
    `utf-7`, `utf-8`, `utf-16be`/`utf-16le`, `utf-32be`/`utf-32le`, `X-ISO-10646-UCS-4-34121`/`X-ISO-10646-UCS-4-21431`, `gb18030`.
    
    ### Encodings without BOM (by language):
    | Language | Encodings |
    | :--- | :--- |
    | International (Unicode) | `utf-8` |
    | Arabic | `iso-8859-6`, `windows-1256` |
    | Bulgarian | `iso-8859-5`, `windows-1251` |
    | Chinese | `iso-2022-cn`, `big5`, `euc-tw`, `gb18030`, `hz-gb-2312` |
    | Croatian | `iso-8859-2`, `iso-8859-13`, `iso-8859-16`, `windows-1250`, `ibm852`, `x-mac-ce` |
    | Czech | `windows-1250`, `iso-8859-2`, `ibm852`, `x-mac-ce` |
    | Danish | `iso-8859-1`, `iso-8859-15`, `windows-1252` |
    | English | `ascii` |
    | Esperanto | `iso-8859-3` |
    | Estonian | `iso-8859-4`, `iso-8859-13`, `iso-8859-13`, `windows-1252`, `windows-1257` |
    | Finnish | `iso-8859-1`, `iso-8859-4`, `iso-8859-9`, `iso-8859-13`, `iso-8859-15`, `windows-1252` |
    | French | `iso-8859-1`, `iso-8859-15`, `windows-1252` |
    | German | `iso-8859-1`, `windows-1252`, `CP 850`/`IBM 00850` |
    | Greek | `iso-8859-7`, `windows-1253` |
    | Hebrew | `iso-8859-8`, `windows-1255` |
    | Hungarian | `iso-8859-2`, `windows-1250` |
    | Irish Gaelic | `iso-8859-1`, `iso-8859-9`, `iso-8859-15`, `windows-1252` |
    | Italian | `iso-8859-1`, `iso-8859-3`, `iso-8859-9`, `iso-8859-15`, `windows-1252` |
    | Japanese | `iso-2022-jp`, `shift-jis`, `euc-jp` |
    | Korean | `iso-2022-kr`, `euc-kr`/`uhc`, `cp949` |
    | Lithuanian | `iso-8859-4`, `iso-8859-10`, `iso-8859-13` |
    | Latvian | `iso-8859-4`, `iso-8859-10`, `iso-8859-13` |
    | Maltese | `iso-8859-3` |
    | Polish | `iso-8859-2`, `iso-8859-13`, `iso-8859-16`, `windows-1250`, `ibm852`, `x-mac-ce` |
    | Portuguese | `iso-8859-1`, `iso-8859-9`, `iso-8859-15`, `windows-1252` |
    | Romanian | `iso-8859-2`, `iso-8859-16`, `windows-1250`, `ibm852` |
    | Russian | `iso-8859-5`, `koi8-r`, `windows-1251`, `x-mac-cyrillic`, `ibm855`, `ibm866` |
    | Slovak | `windows-1250`, `iso-8859-2`, `ibm852`, `x-mac-ce` |
    | Slovene | `iso-8859-2`, `iso-8859-16`, `windows-1250`, `ibm852`, `x-mac-ce` |
    | Spanish | `iso-8859-1`, `iso-8859-15`, `windows-1252` |
    | Swedish | `iso-8859-1`, `iso-8859-4`, `iso-8859-9`, `iso-8859-15`, `windows-1252` |
    | Thai | `tis-620`, `iso-8859-11` |
    | Turkish | `iso-8859-3`, `iso-8859-9` |
    | Vietnamese | `viscii`, `windows-1258` |
    | Others | `windows-1252` |