DOCSEARCHbyAlgolia
Submit Docs

charset-normalizer Documentation

repository·master·Indexed 20 days ago

https://github.com/jawah/charset_normalizer

A universal charset encoding detector and MIT-licensed alternative to chardet. It focuses on readability and coherence to convert raw binary content into Unicode strings, supporting all IANA character set names provided by the Python core library. The package includes a CLI tool named `normalizer` and a Python API featuring functions like `detect`, `from_bytes`, `from_fp`, and `from_path` for encoding and language detection.

Tokens
15.9K
Snippets
42
Records
59
Agent score
72%

What's inside charset-normalizer

  1. Overview of Charset Normalizer

    master
    Charset Normalizer is a library designed to help you read text from unknown charset encodings. Unlike libraries that attempt to uncover the exact originating encoding used to create a file, Charset Normalizer focuses on finding the best encoding to transpose content into Unicode. It supports all IANA character set names for which the Python core library provides codecs. It is useful for scenarios where multiple encodings might yield the same valid Unicode result.

  2. Use Wassima for OS-native Root CA verification

    master

    Wassima allows you to use your operating system's trusted root CAs (Certificate Authorities) to verify peer certificates, rather than relying on a bundled certificate store. This can be used as a standalone solution to replace the certifi package. It is also enabled by default when using Niquests.

    https://github.com/jawah/wassima

  3. Key features of Charset Normalizer

    master

    Charset Normalizer provides the following capabilities:

    • Encoding detection: Works on file pointers (fp), bytes, or PathLike objects.
    • Unicode Transposition: Transposes encoded content to Unicode as accurately as possible.
    • Language Detection: Detects the spoken language within the text.
    • Binary Detection: Identifies if the content is binary.
    • CLI: Includes a comprehensive Command Line Interface.

  4. Explore Niquests (Requests fork)

    master

    Niquests is a fork of the requests library maintained by the same authors as charset-normalizer. It is designed to provide modern HTTP capabilities, such as HTTP/2 and HTTP/3 support, which are missing in standard requests. Because it is a fork of requests, it aims for effortless and safe migration with no breaking changes expected.

    https://github.com/jawah/niquests

  5. Known limitations of charset detection

    master

    When using charset-normalizer, be aware of the following constraints:

    • Language Ambiguity: Language detection may be unreliable if the text contains multiple languages that share the same character sets (e.g., English HTML tags mixed with Turkish content using Latin characters).
    • Content Size: Detection accuracy depends heavily on the amount of content provided. Avoid running detection on very small snippets of text, as there may not be enough data for a reliable match.

  6. Supported languages for detection

    master

    Charset Normalizer can detect the following languages within your content:

    English, German, French, Dutch, Italian, Polish, Spanish, Russian, Japanese, Portuguese, Swedish, Chinese, Ukrainian, Norwegian, Finnish, Vietnamese, Czech, Hungarian, Korean, Indonesian, Turkish, Romanian, Farsi, Arabic, Danish, Serbian, Lithuanian, Slovene, Slovak, Malay, Hebrew, Bulgarian, Croatian, Hindi, Estonian, Thai, Greek, and Tamil.

  7. Work with CharsetMatches results

    master

    The detection methods return a CharsetMatches object. This object behaves similarly to a list and is sorted by probability by default.

    List-like behavior

    You can use standard Python list operations on the CharsetMatches object:

    • Check for matches: Use if not results: to check if any matches were found.
    • Iteration: Use a for loop to iterate over CharsetMatch objects.
    • Indexing: Access specific matches using results[index].
    • Length: Use len(results) to get the number of matches.

    Extracting the best match

    Since the results are sorted, you can quickly retrieve the most probable match using .best() or .first(). These methods return a CharsetMatch object or None if no results exist.

    Class Aliases

    CharsetMatches is also accessible via the following aliases:

    • CharsetDetector
    • CharsetDoctor
    • CharsetNormalizerMatches
    # Iterate over results like a list
for match in results:
    print(match.encoding, 'can decode properly your sequence using', match.alphabets, 'and language', match.language)

# Get the most probable result
result = results.best()
# OR
result = results.first()

# Access by index
if results:
    print(str(results[0]))

  8. Understand the core philosophy of charset-normalizer

    master

    Unlike traditional detectors that focus on identifying the specific originating charset, charset-normalizer focuses on readability. Its goal is to convert raw binary content into a readable Unicode string by 'brute forcing' decoding. It prioritizes finding the encoding that produces the least amount of 'noise' (garbled text) and the highest 'coherence' (matching language-specific character frequencies).

    Note: Do not confuse this with ftfy. While ftfy repairs existing Unicode strings, charset-normalizer is designed to convert raw files of unknown encoding into Unicode.

  9. Understand the Charset Normalizer approach to encoding

    master

    Charset Normalizer does not aim to identify the precise originating encoding. Instead, it aims to find an encoding that allows the content to be correctly decoded into Unicode. Because multiple encodings can often decode the same byte sequence into the same Unicode string, the library treats these as equally valid solutions rather than searching for a single 'correct' source encoding.

    For example, if a byte string is encoded with cp1252, it might be equally valid to decode it using cp1256, cp1258, or iso8859_14 if they all result in the same text.

    # Example of multiple encodings yielding the same result
my_byte_str = 'Bonjour, je suis à la recherche d\'une aide sur les étoiles'.encode('cp1252')

# These decodings are equivalent
print(my_byte_str.decode('cp1252') == my_byte_str.decode('cp1256'))
print(my_byte_str.decode('cp1252') == my_byte_str.decode('iso8859_14'))

  10. Migrate from chardet to charset_normalizer using detect()

    master

    For backward compatibility with the chardet library, charset_normalizer provides a detect() function. This function is designed to be a drop-in replacement for chardet.detect(), returning a dictionary containing the detected encoding.

    To migrate, simply change your import from from chardet import detect to from charset_normalizer import detect.

    from charset_normalizer import detect

# Behaves exactly like chardet
result = detect(my_byte_str)

if result['encoding'] is not None:
    print('got', result['encoding'], 'as detected encoding')

  11. Compile charset-normalizer speedups using Mypyc

    master

    If your platform or architecture is not supported by the pre-built platform-specific wheels, you can manually compile the md.py module using Mypyc to achieve up to four times faster performance. This requires having the necessary toolchain installed on your system.

    To compile the extension, set the CHARSET_NORMALIZER_USE_MYPYC environment variable to 1 and install the package from source using --no-binary :all:.

    export CHARSET_NORMALIZER_USE_MYPYC=1
pip install mypy build wheel
pip install charset-normalizer --no-binary :all:

  12. Python version compatibility and support

    master

    Charset Normalizer has specific compatibility requirements based on your Python version:

    • Python >=2.7, <3.5: Unsupported.
    • Python 3.5: Requires charset-normalizer < 2.1.
    • Python 3.6: Requires charset-normalizer < 3.1.

    If you are using older Python versions, it is recommended to upgrade your Python interpreter as soon as possible.