UTF8-CPP Documentation

The library provides different API patterns depending on the C++ standard used by your compiler:

C++11 and later

The library exposes modern APIs that support standard Unicode strings (u16string) and move semantics.

Pre-C++11

You must use iterator-based functions and provide a container (like std::vector<unsigned short>) to receive the converted data.

Manual Standard Selection

If you want to override the automatic detection of the C++ standard (e.g., if your compiler's __cplusplus macro is unreliable, like in some versions of MSVC), define the UTF_CPP_CPLUSPLUS macro before including utf8.h. Assign it a value corresponding to the desired standard (matching the __cplusplus macro values).

The utf8::unchecked namespace provides faster versions of the standard utf8 functions.

WARNING: These functions are less safe. They do not check for the validity of the supplied UTF-8 sequences or code points and do not perform boundary checking. Using them on invalid data may produce invalid UTF-8 sequences or cause undefined behavior.

The utf8::iterator adapter allows you to iterate over UTF-8 encoded strings as if they were sequences of code points. This enables the use of standard STL algorithms with UTF-8 data.

Important Safety Note: utf8::iterator is a checked iterator. It operates within the specific range provided during construction. Any attempt to move the iterator outside this range, or to compare two iterators constructed from different ranges, will result in an exception.

Commonly, you should initialize it using the begin() and end() methods of a sequence container like std::string.

std::string s = "example";
utf8::iterator i (s.begin(), s.begin(), s.end());

UTF8-CPP is a header-only library. To install it, follow these steps:

1. Download a release from the GitHub releases page into a temporary directory.
2. Unzip the release.
3. Copy the contents of the utfcpp/source directory into your project's include directory.

Note: Using the CMakeLists.txt file to install the library is not a supported deployment method.

The utf8::distance function returns the number of UTF-8 encoded code points between two iterators. This is a linear $O(N)$ operation used to find the length of a UTF-8 string in terms of code points.

• first: Iterator to the beginning of the sequence.
• last: Iterator to the "post-end" of the last code point (can be the start of a new code point or not).

Exceptions:

• utf8::invalid_utf8: Thrown if an invalid UTF-8 sequence is encountered.
• utf8::not_enough_room: Thrown if last does not point to the past-of-end of a UTF-8 sequence.

char* twochars = "\xe6\x97\xa5\xd1\x88";
size_t dist = utf8::distance(twochars, twochars + 5);
assert (dist == 2);

Converts a UTF-32 encoded string to UTF-8 using iterators.

• Signature: octet_iterator utf32to8 (u32bit_iterator start, u32bit_iterator end, octet_iterator result)
• Parameters:
  • start: Iterator to the beginning of the UTF-32 string.
  • end: Iterator to the end of the UTF-32 string.
  • result: Output iterator where UTF-8 bytes are appended.
• Return value: An iterator pointing to the position after the appended UTF-8 string.

int utf32string[] = {0x448, 0x65E5, 0x10346, 0};
vector<unsigned char> utf8result;
utf32to8(utf32string, utf32string + 3, back_inserter(utf8result));
assert (utf8result.size() == 9);

Returns the 32-bit code point for the following sequence without advancing the iterator. This is useful for inspecting the next character without consuming it.

Throws utf8::not_enough_room if the iterator reaches end during extraction, and utf8::invalid_utf8 if the sequence is invalid.

Encodes a 32-bit code point as a UTF-16 sequence and appends it to a UTF-16 string or an output iterator.

Using an output iterator: Does not allocate memory. The caller must ensure sufficient space. It can add one or two words. Using std::back_inserter is recommended.

Using std::u16string: Directly appends the encoded code point to a std::u16string object (requires C++11).

Throws utf8::invalid_code_point if the code point is invalid.

Encodes a 32-bit code point as a UTF-8 sequence and appends it to a UTF-8 string or an output iterator.

Using an output iterator: append does not allocate memory; the caller must ensure sufficient space. It can add between 1 and 4 octets. Using std::back_inserter is recommended to handle memory allocation automatically.

Using std::string: Directly appends the encoded code point to a std::string object.

Throws utf8::invalid_code_point if the code point is invalid.