Fix accented chars when .po files are UTF8 encoded

[psa-jforestier]

Fix accented char display problem, where a blank space appears instead of the glyph, and fix accented glyph width calculation. Related to #377.

[psa-jforestier]

I review my own PR ...

I think the code could be optimized. Actually, the UTF8 parsing is done each time a string must be displayed : for each char, we look if < 0x80 and eventually convert the char to ASCII.
It could be better to make this UTF8 to ASCII translation when loading the PO file in src/stratagus/translate.cpp. So we will have the transcoded ASCII string in memory and we can display it as-it, without "live" transcodification and string parsing.

Will try to work on it in the coming days, but I would appreciate if someone (@timfel ? @Pointifurry ?) can validate this current PR to check if it works with German (ß working ?) and Spanish (ñ). On my Wargus installation, the Russian seems not working at all, because I do not have a RU version of War2. I dont know how to verify this language.

[timfel]

I just tried it with the German version, and the ß is mapping to the fallback ?, but the umlauts (ä, ö, ü, Ä, Ö, Ü) are working. I suspect the code to convert the warcraft strings to UTF-8 in wartool is just broken: https://github.com/Wargus/wargus/blob/master/wartool.cpp#L2069

[timfel]

Ah, yeah, the conversion is wrong. I guess the texts for European languages with Latin scripts are probably cp1252 in the original data. I'll fix the conversion in wartool.

[psa-jforestier]

On the GOG French version, the UI labels are in War2.Dat.mpq:/rez/MainMenu.bin, objectives are in /rez/objctivs.tbl The à is encoded 0xe0 , the é is 0xe9, it indicates it use the ISO-8859-1 or CP1252 encoding.

[timfel]

On the GOG French version, the UI labels are in War2.Dat.mpq:/rez/MainMenu.bin, objectives are in /rez/objctivs.tbl The à is encoded 0xe0 , the é is 0xe9, it indicates it use the ISO-8859-1 or CP1252 encoding.

Yes, I found those files. While we could extract these labels, our UI is completely different and whatever we do we always need translation additional files. Having translation files and just coding the UI in English by default makes it easier also to provide consistent different languages, even at the cost of hardcoding some strings that could be extracted.

Btw, to fix the ß in the German version, I need this patch on top of yours:

modified   src/video/font.cpp
@@ -87,7 +87,7 @@ static const int extended_ascii[] = {
 	0x00, 0xa0, 0x00, 0x00, 0x8e, 0x8f, 0x92, 0x80, /* 0xc0-0xc7 */
 	0x00, 0x90, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xc8-0xcf */
 	0x00, 0xa5, 0x00, 0x95, 0x00, 0x00, 0x99, 0x00, /* 0xd0-0xd7 */
-	0x9d, 0x97, 0x00, 0x00, 0x9a, 0x00, 0x00, 0x00, /* 0xd8-0xdf */
+	0x9d, 0x97, 0x00, 0x00, 0x9a, 0x00, 0x00, 0xe1, /* 0xd8-0xdf */
 	0x85, 0xa0, 0x83, 0x00, 0x84, 0x86, 0x91, 0x87, /* 0xe0-0xe7 */
 	0x8a, 0x82, 0x88, 0x89, 0x00, 0xa1, 0x8c, 0x8b, /* 0xe8-0xef */
 	0x00, 0xa4, 0x95, 0xa2, 0x93, 0x00, 0x94, 0x00, /* 0xf0-0xf7 */

[timfel]

For wartool, I added this which works with your PR here.

[timfel]

FYI, I'll be travelling for the next two weeks, so I won't have a chance to merge this until I get back

[timfel]

In case you're interested in helping with the French translation, I've setup Poeditor pages for the engine and game strings: https://github.com/Wargus/wargus/blob/master/README.md

[andreas-volz]

I could verify a problem with german data CD. Please see a gdb backtrace here: https://paste.ofcode.org/32uedC3XLN7zpuCTaw8UkVs
after reverting this pullrequest is works great.

[notpointify]

I apologize for the lateness, better late than never though I suppose 😅

I can confirm that this issue seems to be fixed for Spanish as of release 3.2.0.

BEFORE (3.1.2)

AFTER (3.2.0)

Kudos! 👍