ftfy (fixes text for you)

To prevent mojibake (encoding errors), assume text is in UTF-8 unless you have a specific reason to believe otherwise. In Python 3, use the Unicode string type (str) for all operations.

When opening text files, explicitly specify the encoding as utf-8 and use errors='replace' to handle potential issues gracefully:

openfile = open(filename, encoding='utf-8', errors='replace')

When converting bytes to text, decode them as UTF-8:

text = bytebuffer.decode('utf-8', 'replace')

The term "extended ASCII" is ambiguous because it refers to many incompatible 256-character encodings (e.g., IBM codepage 437, Windows-1252, Windows-1251, or Latin-1) depending on the user's region and OS. Using this term often leads to encoding errors.

Best Practices:

• Instead of saying "extended ASCII", use the specific name of the encoding (e.g., "Latin-1", "Windows-1252", "codepage 437").
• Transition your workflows to use UTF-8, which is a superset of ASCII and can represent all Unicode characters.

Standard Excel CSV exports use a 256-character encoding based on the operating system and default language, which often leads to mangled Unicode text and lack of interoperability.

To avoid this:

1. Use Google Sheets to create CSVs.
2. Keep Excel files in .xlsx format to preserve Unicode.
3. If you must use Excel for CSV-like files, select the option to export as "Unicode Text". This creates a tab-separated UTF-16 file which avoids mojibake.

Depending on your requirements, you may want to disable certain default fixes by passing them as keyword arguments set to False:

• HTML output: Set unescape_html=False if the output is intended to be interpreted as HTML.
• CJK text spacing: Set fix_character_width=False to preserve the spacing of CJK (Chinese, Japanese, Korean) text.
• Typography preservation: Set uncurl_quotes=False to preserve curly quotation marks.
• Strict mojibake repair: Set decode_inconsistent_utf8=False to only fix mojibake when it can be resolved with a consistent sequence of encoding and decoding steps.

ftfy is not an encoding detector; it is a mojibake detector and fixer. It is designed to take Unicode text that has already been decoded (even if it was decoded incorrectly, resulting in mojibake like réflexion) and fix it.

Important: ftfy does not accept bytes as input for its primary fixing operations. You should first attempt to decode your bytes into Unicode text before passing them to ftfy.

If you use ftfy as a data processing step in NLP research, please cite it. The project has a citable record on Zenodo.

Standard Citation: Robyn Speer. (2019). ftfy (Version 5.5). Zenodo. http://doi.org/10.5281/zenodo.2591652

BibTeX Format:

@misc{speer-2019-ftfy,
  author       = {Robyn Speer},
  title        = {ftfy},
  note         = {Version 5.5},
  year         = 2019,
  howpublished = {Zenodo},
  doi          = {10.5281/zenodo.2591652},
  url          = {https://doi.org/10.5281/zenodo.2591652}
}

You can install the ftfy library using pip to fix mojibake (encoding errors) in your text.

pip install ftfy

Do not use chardet to detect encodings on raw bytes. chardet relies on heuristics designed before the era of multilingual social media and emojis, making it unreliable for modern text. It often misidentifies UTF-8 (especially emojis) as other encodings like Windows-1254 or iso-8859-2.

Instead of attempting to detect the encoding, treat raw bytes as UTF-8 by default. Text is UTF-8 until proven otherwise.

Use ftfy.fix_and_explain() to fix broken text (mojibake) and receive a list of the transformations applied (e.g., unescaping HTML, encoding/decoding steps). This is useful for understanding how text was corrupted.

Note: Unlike fix_text, fix_and_explain treats the entire input as a single unit to find a unified explanation, rather than fixing it line-by-line.

from ftfy import fix_and_explain

shipping_label = "LóPEZ"
fixed, explanation = fix_and_explain(shipping_label)

# fixed: 'LóPEZ'
# explanation: [('apply', 'unescape_html'), ('apply', 'unescape_html'), ('apply', 'unescape_html'), ('encode', 'latin-1'), ('decode', 'utf-8')]