forceutf8

repository·master·Indexed 23 days ago

https://github.com/neitanod/forceutf8

A PHP utility class for handling encoding issues, specifically designed to fix mixed-encoding strings and repair garbled, double-encoded UTF-8 text. It provides methods such as Encoding::toUTF8() for converting Latin1 or Windows-1252 to UTF-8, Encoding::fixUTF8() for repairing multi-encoded strings, and Encoding::toLatin1() for converting strings back to legacy encodings. It also includes utilities to remove UTF-8 Byte Order Marks (BOM) and restore Windows-1252 characters.

Tokens
1.2K
Snippets
5
Records
11
Agent score
32%

What's inside forceutf8

  1. Fix garbled UTF-8 strings using Encoding::fixUTF8()

    master

    If you have a double-encoded or multiple-encoded UTF-8 string that appears garbled, use Encoding::fixUTF8() to repair it. It can handle various levels of corruption (e.g., Fédération becoming Fédération).

    use \ForceUTF8\Encoding;
    
    $utf8_string = Encoding::fixUTF8($garbled_utf8_string);
  2. Convert mixed or unknown encodings to UTF-8 using Encoding::toUTF8()

    master

    Use Encoding::toUTF8() to convert strings that may be Latin1 (ISO 8859-1), Windows-1252, or a mix of UTF-8 and other encodings into a clean UTF-8 string. This is useful when dealing with unreliable services that mix encodings in a single string.

    use \ForceUTF8\Encoding;
    
    $utf8_string = Encoding::toUTF8($utf8_or_latin1_or_mixed_string);
  3. Convert strings to Latin1 using Encoding::toLatin1()

    master

    Use Encoding::toLatin1() to convert a string (regardless of whether it is currently UTF-8, Latin1, or mixed) into Latin1 encoding.

    use \ForceUTF8\Encoding;
    
    $latin1_string = Encoding::toLatin1($utf8_or_latin1_or_mixed_string);
  4. Configure iconv flags for Encoding::fixUTF8()

    master

    By default, Encoding::fixUTF8() uses the Encoding::WITHOUT_ICONV flag, which means it does not use the PHP iconv extension to fix strings. If you want to utilize iconv processing to handle invalid or non-translatable characters, you can pass specific flags as the second argument to fixUTF8().

    Available flags:

    • Encoding::ICONV_IGNORE: Prevents the function from breaking on invalid characters by ignoring them (or keeping those present in Win1252).
    • Encoding::ICONV_TRANSLIT: Prevents the function from breaking by transliterating invalid characters into their closest approximations.
    use \ForceUTF8\Encoding;
    
    // Example: Preserving characters using IGNORE or TRANSLIT
    echo Encoding::fixUTF8($str, Encoding::ICONV_IGNORE);
    echo Encoding::fixUTF8($str, Encoding::ICONV_TRANSLIT);
  5. Convert strings to Latin-1, ISO-8859-1, or Windows-1252

    master

    The following methods convert UTF-8 strings to legacy encodings. They accept both single strings and arrays of strings. They also support an $option parameter to control iconv behavior if the iconv extension is available.

    • toWin1252($text, $option): Converts to Windows-1252.
    • toISO8859($text, $option): Alias for toWin1252.
    • toLatin1($text, $option): Alias for toWin1252.

    Options:

    • Encoding::WITHOUT_ICONV: (Default) Uses internal mapping instead of the iconv extension.
    • Encoding::ICONV_TRANSLIT: Uses iconv with //TRANSLIT.
    • Encoding::ICONV_IGNORE: Uses iconv with //IGNORE.
  6. Fix garbled multi-encoded UTF-8 strings with fixUTF8()

    master

    The fixUTF8() method is used to repair strings that have been incorrectly encoded multiple times (e.g., through repeated conversion cycles). It iteratively applies decoding and re-encoding to stabilize the string.

    It accepts both single strings and arrays of strings.

  7. Convert strings to UTF-8 with toUTF8()

    master

    The toUTF8() method converts non-UTF-8 strings to UTF-8. It assumes the original encoding is either Windows-1252 or ISO-8859-1. It leaves existing UTF-8 characters alone while converting other characters. This method accepts both single strings and arrays of strings.

    Limitations: Conversion may fail if certain characters are followed by specific character groups (e.g., certain accented characters followed by specific symbols) that could be interpreted as valid multi-byte Unicode characters.

  8. Encode text using a specific encoding label with encode()

    master

    The encode($encodingLabel, $text) method provides a high-level way to convert text based on a label. It normalizes the label first.

    • If the label is recognized as ISO-8859-1 (or aliases like LATIN1, WIN1252, etc.), it calls toLatin1().
    • Otherwise, it defaults to toUTF8().

    Supported Label Equivalences:

    • ISO88591, ISO8859, ISO, LATIN1, LATIN, WIN1252, WINDOWS1252 $\rightarrow$ ISO-8859-1
    • UTF8, UTF $\rightarrow$ UTF-8
  9. Fix Windows-1252 characters in UTF-8 strings with UTF8FixWin1252Chars()

    master
    If you have a UTF-8 string that was incorrectly converted from Windows-1252 as if it were ISO-8859-1 (effectively ignoring the Windows-1252 specific characters in the 0x80 to 0x9F range), use UTF8FixWin1252Chars() to restore them.