I'm using FreeType2 in one of my projects. In order to render a letter, I need to provide a Unicode two-byte character code. The char codes a program reads are in ASCII one-byte format though. It poses no problem for char codes below 128 (the character codes are the same), but the other 128 do not match. For instance:
'a' in ASCII is 0x61, 'a' in Unicode is 0x0061 – that's fine
'ą' in ASCII is 0xB9, 'ą' in Unicode is 0x0105 – completely different
I was trying to use WinAPI functions there, but I must be doing something wrong. Here's a sample:
unsigned char szTest1[] = "ąółź"; //ASCII format
wchar_t* wszTest2;
int size = MultiByteToWideChar(CP_UTF8, 0, (char*)szTest1, 4, NULL, 0);
printf("size = %d\n", size);
wszTest2 = new wchar_t[size];
MultiByteToWideChar(CP_UTF8, 0, (char*)szTest1, 4, wszTest2, size);
printf("HEX: %x\n", wszTest2[0]);
delete[] wszTest2;
I'm expecting a new wide string to be created, with no NULL at the end. However, the size variable always equals 0. Any idea what I'm doing wrong? Or maybe there's an easier way to solve the problem?
Best Answer
The "pure" ASCII set of characters is restricted in range 0-127 (7 bits). The 8-bit characters with most significant bit set (i.e. those in range 128-255) are not uniquely defined: their definition depends on the code page. So, your character
ą
(LATIN SMALL LETTER A WITH OGONEK) is represented by the value0xB9
in a particular code page, which should be Windows-1250. In other code pages, the value0xB9
is associated to a different character (for example, in Windows 1252 code page,0xB9
is associated to character¹
, i.e. a superscript digit 1).To convert characters from a particular code-page to Unicode UTF-16 using Windows Win32 APIs, you can use
MultiByteToWideChar
, specifying the correct code page (which is notCP_UTF8
as written in the code in your question; in fact,CP_UTF8
identifies Unicode UTF-8). You may want to try specifying1250
(ANSI Central European; Central European (Windows)) as proper code page identifier.If you can have access to ATL in your code, you can use the convenience of ATL string conversion helper classes like
CA2W
, which wraps theMultiByteToWideChar(
) call and memory allocation in a RAII class; e.g.:Now you should be able to use
test
string in your Unicode API's.If you don't have access to ATL or want a C++ STL-based solution, you may want to consider some code like this: