International Components for Unicode (ICU)

repository·main·Indexed 25 days ago

https://github.com/unicode-org/icu

A library providing robust support for Unicode, internationalization, and localization. Includes implementations for C/C++ (ICU4C) and Java (ICU4J), as well as tools for converting CLDR XML data to ICU resource files, managing break iterator rule source data, and building unsafe-backward collator data.

Tokens
190.5K
Snippets
317
Records
1K
Agent score
85%

What's inside ICU

  1. Overview of the ICU Data Build Tool

    main
    The ICU Data Build Tool allows you to customize and reduce the size of your ICU locale data file. By configuring the data with finer granularity, you can exclude specific locales or features that are not required for your application, making the resulting .dat file suitable for resource-constrained environments like smartphone apps or embedded devices.
  2. Overview of ICU Transforms

    main

    ICU Transforms provide a mechanism for processing Unicode text through various specialized operations. Key capabilities include:

    • Case Mappings: Handling upper-, lower-, and titlecase mappings across different languages and writing systems.
    • Normalization: Converting text into unique, equivalent forms (e.g., converting composite character sequences into precomposed characters). This also supports mapping legacy character encodings to and from Unicode.
    • General Transforms: A flexible package for tasks such as:
      • Uppercase, Lowercase, Titlecase, and Full/Halfwidth conversions.
      • Hex and Character Name conversions.
      • Script-to-Script conversion.
    • Bidirectional Algorithm: Specifying text directionality within a text flow.
  3. Overview of ICU versions (ICU4C and ICU4J)

    main

    ICU provides internationalization utilities for C, C++, and Java. While they share the same development goals and track the latest Unicode standard, they are distributed as distinct versions:

    • ICU4C: The C and C++ versions of the international functionality. It includes specific APIs like the character converter API.
    • ICU4J: The Java version, which includes enhanced versions of classes found in the Java Development Kit (JDK) and additional complementary classes.

    Both versions maintain a single, portable source code base.

  4. Introduction to ICU4J

    main

    ICU4J (International Components for Unicode for Java) is an add-on to the standard Java Runtime Environment (JRE) that provides advanced Unicode and internationalization services. It is designed to stay current with the evolving Unicode standard and provides richer, more performant APIs than the standard J2SE implementation.

    Key features include:

    • Collation: Rule-based, up-to-date Unicode Collation Algorithm (UCA) sorting.
    • Charset Detection: Recognition of various single and multibyte charsets.
    • UnicodeSet: Optimized set operations for Unicode characters.
    • Transforms: Flexible text conversions (Normalization, Case conversions, Transliteration, etc.).
    • Unicode Normalization: Support for NFC, NFD, NFKD, and NFKC.
    • International Calendars: Support for Arabic, Buddhist, Chinese, Hebrew, Japanese, Ethiopic, Islamic, Coptic, and more.
    • Date/Number Formatting: Enhanced patterns, relative date formatting, scientific notation, and spelled-out formats.
    • Boundary Analysis: Enhanced word-break detection for languages like Thai, Khmer, and Chinese.
    • Text Compression: Standard compression for Unicode text.
    • Charset Conversion: Conversion via the Java CharsetProvider SPI.
  5. Overview of ICU Punctuation Ignoring Options

    main

    By default, ICU treats spaces and punctuation as "variable" characters. These characters add primary (base character) differences and typically sort less-than digits and letters.

    To change this behavior, ICU provides "Variable Weighting" or "Alternate Handling" options. These options allow you to control how variable characters affect sorting at different collation levels (Primary, Secondary, Tertiary, and Quaternary).

    OptionBehavior Summary
    Non-ignorable (Default)Maps variable characters to their normal primary collation elements.
    BlankedIgnores variable characters entirely (effectively removes them before comparison).
    ShiftedSorts variable characters at the quaternary level, typically before regular characters.
    Shift-TrimmedNot implemented in ICU. Sorts variable characters such that inserting one makes a string sort greater-than the original.
    Variable-AfterNot implemented in ICU. Sorts variable characters after regular characters.
  6. Overview of ICU Calendar classes

    main

    ICU provides two primary calendar classes for parsing and formatting calendar information:

    1. Calendar: An abstract base class defining the calendar API. It supports converting UDate to specific fields and performing field arithmetic.
    2. GregorianCalendar: A concrete subclass of Calendar implementing the standard international Gregorian calendar.

    ICU also provides specialized subclasses for non-Gregorian systems, including Buddhist, Chinese, Coptic, Ethiopic, Hebrew, Indian, Islamic, Japanese, and Persian calendars.

  7. Overview of ICU Character Conversion

    main

    ICU provides a converter API designed to convert text between Unicode and other character encodings. The API is optimized for high performance, streaming (buffered) data, and consistent behavior across different computer platforms.

    Key features include:

    • Support for all major encodings and Unicode surrogate support.
    • Fallback mechanisms to the codepage or reverse fallbacks to Unicode.
    • Callbacks for handling and substituting invalid or unmapped byte sequences.
    • Ability to add support for unsupported encodings.
    • Support for streaming text data through the API.
  8. Overview of ICU 76 Release

    main

    ICU 76 is a major release that updates to Unicode 16 and CLDR 46.

    Key updates include:

    • Unicode 16: New characters, scripts (Garay, Gurung Khema, Kirat Rai, Ol Onal, Sunuwar), emoji, and improved line breaking (UAX #14).
    • CLDR 46: Significant locale data updates, new measurement units (night, light-speed), and improved Han Radical-Stroke order.
    • C/C++ Improvements: Introduction of C++ header-only APIs that wrap stable C APIs.
    • Java Improvements: Direct formatting of java.time (Temporal) types using existing ICU4J date/time formatting classes.
    • MessageFormat 2.0: Updated technology preview implementations for both Java and C++.

    Initial release version: 76.1 (Released 2024-10-24).

  9. Overview of ICU4C capabilities

    main

    ICU4C (International Components for Unicode) provides robust Unicode services for global software support. Key capabilities include:

    • Unicode Standard Support: Implementation of the latest Unicode versions.
    • Character Set Conversions: Support for over 220 codepages.
    • Locale Data: Support for more than 300 locales.
    • Text Processing: Language-sensitive collation (sorting), searching (Unicode Collation Algorithm/ISO 14651), and regular expression matching.
    • Transformations: Normalization, upper/lowercase, and script transliterations (50+ pairs).
    • Resource Management: Resource bundles for localized information.
    • Formatting: Date, Number, and Message formatting/parsing for culture-specific I/O.
    • Calendar & Boundaries: Calendar-specific date/time manipulation and text boundary analysis (characters, words, and sentences).
  10. Overview of UText abstraction

    main

    UText is a text abstraction facility in ICU designed to allow ICU services (like Regular Expressions and Break Iteration) to work with various text formats.

    Supported formats:

    • UTF-8 (char*)
    • UTF-16 (UChar* or UnicodeString)
    • Replaceable objects

    Extensibility: Developers can extend UText to support custom formats (e.g., UTF-32, discontiguous memory chunks, or non-Unicode code pages) by implementing a set of Text Provider Functions.

  11. Overview of ICU Calendar and TimeZone classes

    main

    ICU provides four primary classes for handling date and time logic:

    • Calendar: An abstract base class that maps UDate values to and from individual calendar fields (year, month, day, etc.) and performs field computations.
    • GregorianCalendar: A concrete subclass of Calendar implementing Julian and Gregorian calendar rules.
    • TimeZone: An abstract base class that maps between Universal Coordinated Time (UTC) and local time.
    • SimpleTimeZone: A concrete subclass of TimeZone that implements standard time and daylight savings time (DST) according to real-world rules.
  12. Overview of ICU Python utility libraries

    main

    The tools/py/libs directory contains a set of reusable Python helper modules designed to make ICU integration scripts succinct and reliable. These modules are intended to facilitate the migration of shell scripts to Python by providing standardized behaviors for logging, filesystem management, and process execution.

    Key modules include:

    • iculog: Handles progress reporting. It outputs colorized status messages to standard output while logging detailed information to log files for debugging.
    • icufs: Provides filesystem operations such as creating/removing directories and changing the current working directory.
    • icuproc: Manages external process execution with integrated logging and robust error detection.