How well is Unicode supported in C++11?

How well does the C++ standard library support unicode?

Terribly.

A quick scan through the library facilities that might provide Unicode support gives me this list:

  • Strings library
  • Localization library
  • Input/output library
  • Regular expressions library

I think all but the first one provide terrible support. I’ll get back to it in more detail after a quick detour through your other questions.

Does std::string do what it should?

Yes. According to the C++ standard, this is what std::string and its siblings should do:

The class template basic_string describes objects that can store a sequence consisting of a varying number of arbitrary char-like objects with the first element of the sequence at position zero.

Well, std::string does that just fine. Does that provide any Unicode-specific functionality? No.

Should it? Probably not. std::string is fine as a sequence of char objects. That’s useful; the only annoyance is that it is a very low-level view of text and standard C++ doesn’t provide a higher-level one.

How do I use it?

Use it as a sequence of char objects; pretending it is something else is bound to end in pain.

Where are potential problems?

All over the place? Let’s see…

Strings library

The strings library provides us basic_string, which is merely a sequence of what the standard calls “char-like objects”. I call them code units. If you want a high-level view of text, this is not what you are looking for. This is a view of text suitable for serialization/deserialization/storage.

It also provides some tools from the C library that can be used to bridge the gap between the narrow world and the Unicode world: c16rtomb/mbrtoc16 and c32rtomb/mbrtoc32.

Localization library

The localization library still believes that one of those “char-like objects” equals one “character”. This is of course silly, and makes it impossible to get lots of things working properly beyond some small subset of Unicode like ASCII.

Consider, for example, what the standard calls “convenience interfaces” in the <locale> header:

template <class charT> bool isspace (charT c, const locale& loc);
template <class charT> bool isprint (charT c, const locale& loc);
template <class charT> bool iscntrl (charT c, const locale& loc);
// ...
template <class charT> charT toupper(charT c, const locale& loc);
template <class charT> charT tolower(charT c, const locale& loc);
// ...

How do you expect any of these functions to properly categorize, say, U+1F34C ʙᴀɴᴀɴᴀ, as in u8"🍌" or u8"\U0001F34C"? There’s no way it will ever work, because those functions take only one code unit as input.

This could work with an appropriate locale if you used char32_t only: U'\U0001F34C' is a single code unit in UTF-32.

However, that still means you only get the simple casing transformations with toupper and tolower, which, for example, are not good enough for some German locales: “ß” uppercases to “SS”☦ but toupper can only return one character code unit.

Next up, wstring_convert/wbuffer_convert and the standard code conversion facets.

wstring_convert is used to convert between strings in one given encoding into strings in another given encoding. There are two string types involved in this transformation, which the standard calls a byte string and a wide string. Since these terms are really misleading, I prefer to use “serialized” and “deserialized”, respectively, instead†.

The encodings to convert between are decided by a codecvt (a code conversion facet) passed as a template type argument to wstring_convert.

wbuffer_convert performs a similar function but as a wide deserialized stream buffer that wraps a byte serialized stream buffer. Any I/O is performed through the underlying byte serialized stream buffer with conversions to and from the encodings given by the codecvt argument. Writing serializes into that buffer, and then writes from it, and reading reads into the buffer and then deserializes from it.

The standard provides some codecvt class templates for use with these facilities: codecvt_utf8, codecvt_utf16, codecvt_utf8_utf16, and some codecvt specializations. Together these standard facets provide all the following conversions. (Note: in the following list, the encoding on the left is always the serialized string/streambuf, and the encoding on the right is always the deserialized string/streambuf; the standard allows conversions in both directions).

  • UTF-8 ↔ UCS-2 with codecvt_utf8<char16_t>, and codecvt_utf8<wchar_t> where sizeof(wchar_t) == 2;
  • UTF-8 ↔ UTF-32 with codecvt_utf8<char32_t>, codecvt<char32_t, char, mbstate_t>, and codecvt_utf8<wchar_t> where sizeof(wchar_t) == 4;
  • UTF-16 ↔ UCS-2 with codecvt_utf16<char16_t>, and codecvt_utf16<wchar_t> where sizeof(wchar_t) == 2;
  • UTF-16 ↔ UTF-32 with codecvt_utf16<char32_t>, and codecvt_utf16<wchar_t> where sizeof(wchar_t) == 4;
  • UTF-8 ↔ UTF-16 with codecvt_utf8_utf16<char16_t>, codecvt<char16_t, char, mbstate_t>, and codecvt_utf8_utf16<wchar_t> where sizeof(wchar_t) == 2;
  • narrow ↔ wide with codecvt<wchar_t, char_t, mbstate_t>
  • no-op with codecvt<char, char, mbstate_t>.

Several of these are useful, but there is a lot of awkward stuff here.

First off—holy high surrogate! that naming scheme is messy.

Then, there’s a lot of UCS-2 support. UCS-2 is an encoding from Unicode 1.0 that was superseded in 1996 because it only supports the basic multilingual plane. Why the committee thought desirable to focus on an encoding that was superseded over 20 years ago, I don’t know‡. It’s not like support for more encodings is bad or anything, but UCS-2 shows up too often here.

I would say that char16_t is obviously meant for storing UTF-16 code units. However, this is one part of the standard that thinks otherwise. codecvt_utf8<char16_t> has nothing to do with UTF-16. For example, wstring_convert<codecvt_utf8<char16_t>>().to_bytes(u"\U0001F34C") will compile fine, but will fail unconditionally: the input will be treated as the UCS-2 string u"\xD83C\xDF4C", which cannot be converted to UTF-8 because UTF-8 cannot encode any value in the range 0xD800-0xDFFF.

Still on the UCS-2 front, there is no way to read from an UTF-16 byte stream into an UTF-16 string with these facets. If you have a sequence of UTF-16 bytes you can’t deserialize it into a string of char16_t. This is surprising, because it is more or less an identity conversion. Even more suprising, though, is the fact that there is support for deserializing from an UTF-16 stream into an UCS-2 string with codecvt_utf16<char16_t>, which is actually a lossy conversion.

The UTF-16-as-bytes support is quite good, though: it supports detecting endianess from a BOM, or selecting it explicitly in code. It also supports producing output with and without a BOM.

There are some more interesting conversion possibilities absent. There is no way to deserialize from an UTF-16 byte stream or string into a UTF-8 string, since UTF-8 is never supported as the deserialized form.

And here the narrow/wide world is completely separate from the UTF/UCS world. There are no conversions between the old-style narrow/wide encodings and any Unicode encodings.

Input/output library

The I/O library can be used to read and write text in Unicode encodings using the wstring_convert and wbuffer_convert facilities described above. I don’t think there’s much else that would need to be supported by this part of the standard library.

Regular expressions library

I have expounded upon problems with C++ regexes and Unicode on Stack Overflow before. I will not repeat all those points here, but merely state that C++ regexes don’t have level 1 Unicode support, which is the bare minimum to make them usable without resorting to using UTF-32 everywhere.

That’s it?

Yes, that’s it. That’s the existing functionality. There’s lots of Unicode functionality that is nowhere to be seen like normalization or text segmentation algorithms.

U+1F4A9. Is there any way to get some better Unicode support in C++?

The usual suspects: ICU and Boost.Locale.

† A byte string is, unsurprisingly, a string of bytes, i.e., char objects. However, unlike a wide string literal, which is always an array of wchar_t objects, a “wide string” in this context is not necessarily a string of wchar_t objects. In fact, the standard never explicitly defines what a “wide string” means, so we’re left to guess the meaning from usage. Since the standard terminology is sloppy and confusing, I use my own, in the name of clarity.

Encodings like UTF-16 can be stored as sequences of char16_t, which then have no endianness; or they can be stored as sequences of bytes, which have endianness (each consecutive pair of bytes can represent a different char16_t value depending on endianness). The standard supports both of these forms. A sequence of char16_t is more useful for internal manipulation in the program. A sequence of bytes is the way to exchange such strings with the external world. The terms I’ll use instead of “byte” and “wide” are thus “serialized” and “deserialized”.

‡ If you are about to say “but Windows!” hold your 🐎🐎. All versions of Windows since Windows 2000 use UTF-16.

☦ Yes, I know about the großes Eszett (ẞ), but even if you were to change all German locales overnight to have ß uppercase to ẞ, there’s still plenty of other cases where this would fail. Try uppercasing U+FB00 ʟᴀᴛɪɴ sᴍᴀʟʟ ʟɪɢᴀᴛᴜʀᴇ ғғ. There is no ʟᴀᴛɪɴ ᴄᴀᴘɪᴛᴀʟ ʟɪɢᴀᴛᴜʀᴇ ғғ; it just uppercases to two Fs. Or U+01F0 ʟᴀᴛɪɴ sᴍᴀʟʟ ʟᴇᴛᴛᴇʀ ᴊ ᴡɪᴛʜ ᴄᴀʀᴏɴ; there’s no precomposed capital; it just uppercases to a capital J and a combining caron.

Related Posts:

  1. What is a lambda expression in C++11?
  2. ld: symbol(s) not found for architecture x86_64 error
  3. What is the ‘override’ keyword in C++ used for? [duplicate]
  4. What is a smart pointer and when should I use one?
  5. Meaning of = delete after function declaration
  6. C++ std::priority_queue uses the lambda expression
  7. Usage and Syntax of std::function
  8. What exactly is nullptr?
  9. Difference between `constexpr` and `const`
  10. Split a string using C++11
  11. How to create timer events using C++ 11?
  12. What does T&& (double ampersand) mean in C++11?
  13. Segmentation fault error 11 C++
  14. Why doesn’t C++ have a garbage collector?
  15. Compiling C++11 with g++
  16. terminate called after throwing an instance of ‘std::invalid_argument’ what(): stoi
  17. What exactly is std::atomic?
  18. Issue with std::stol – ‘std::invalid_argument’ what(): stol
  19. terminate called after throwing an instance of ‘std::out_of_range’
  20. Cleanest way to copy a constant size array in c++11
  21. push_back vs emplace_back
  22. undefined reference to ‘std::cout’
  23. What is move semantics?
  24. expression preceding parentheses of apparent call must have (pointer-to-) function type
  25. C++ error: Undefined symbols for architecture x86_64
  26. C++ terminate called without an active exception
  27. Thread pooling in C++11
  28. How does std::forward work? [duplicate]
  29. Difference in make_shared and normal shared_ptr in C++
  30. What is std::move(), and when should it be used?
  31. C++: std does not have member “string”
  32. push_back vs emplace_back
  33. error: use of deleted function
  34. Call to implicitly deleted copy constructor in LLVM
  35. cc1plus: error: unrecognized command line option “-std=c++11” with g++
  36. std::wstring VS std::string
  37. error C2679: binary ‘<<' : no operator found which takes a right-hand operand of type 'std::string' (or there is no acceptable conversion)
  38. no match for ‘operator<<’ in ‘std::operator
  39. Warning: non-static data member initializers only available with -std=c++11 or -std=gnu++11?
  40. Use the auto keyword in C++ STL
  41. Call to non-static member function without an object argument compiler error
  42. Does C++11 have C#-style properties?
  43. Iterator Loop vs index loop
  44. initialize a vector to zeros C++/C++11
  45. When is it safe to call this-> in constructor and destructor
  46. How to memset char array with null terminating character?
  47. How to automatically convert strongly typed enum into int?
  48. Why should I use a pointer rather than the object itself?
  49. too many initializers for ‘int [0]’ c++
  50. overloaded function with no contextual type information
  51. Creating folders in C++
  52. convert string to size_t
  53. Compiling C++11 with g++
  54. ERROR C2039: ‘vector’: is not a member of ‘std’
  55. Does static constexpr variable inside a function make sense?
  56. c++ vector bubble sort
  57. What does _T stands for in a CString
  58. How to use bitmask?
  59. Differences between unique_ptr and shared_ptr
  60. How do I print out the contents of a vector?
  61. ‘was not declared in this scope’ error
  62. What does `*&` in a function declaration mean?
  63. (->) arrow operator and (.) dot operator , class pointer
  64. How to reverse an std::string? [duplicate]
  65. What is the C version of RMI
  66. Difference between ‘strcpy’ and ‘strcpy_s’?
  67. What is uintptr_t data type
  68. What is the difference between a static and const variable?
  69. unsigned int vs. size_t
  70. error: expected primary-expression before ‘)’ token (C)
  71. What does -> mean in C++?
  72. Fill array with random numbers within a specified range (C++)
  73. Remove an array element and shift the remaining ones
  74. Right way to split an std::string into a vector
  75. string subscript out of range error
  76. No operator << matches these operands
  77. Returning an empty string : efficient way in c++
  78. How to find the size of an int[]?
  79. Append an int to a std::string
  80. Decimal to hex conversion c++ built-in function
  81. C++ IDE with repl?
  82. how to print a string to console in c++
  83. How to call a parent class function from derived class function?
  84. Error: No instance of constructor matches the argument list
  85. How can I get the list of files in a directory using C or C++?
  86. RPN Calculator for C++
  87. What does `Fatal Python error: PyThreadState_Get: no current thread` mean?
  88. c++ –
  89. C++ – statement cannot resolve address for overloaded function
  90. What’s the difference between * and & in C?
  91. is there is a compiler which convert c++ code into mips assembly language
  92. Stray ‘\342’ in C++ program
  93. gcc -g :what will happen
  94. QByteArray to QString
  95. Bitwise Less than or Equal to
  96. C++ Simple Dice roll – how to return multiple different random numbers [duplicate]
  97. How can I get a file’s size in C++?
  98. Why can’t I make a vector of references?
  99. What does “-Wall” in “g++ -Wall test.cpp -o test” do?
  100. C++ equivalent of StringBuffer/StringBuilder?

Leave a Comment