Wrong characters codification in Xaos translation - broken UTF support

P.S.: I found out this problem because I'm translating XaoS for Hardy in Launchpad.
It wasn't translated in Italian before.

This is a known issue with XaoS on modern Linux systems. See the following bug in our tracker: http://sourceforge.net/tracker/index.php?func=detail&aid=1188322&group_id=5771&atid=105771

The problem is that XaoS's built-in text system can't handle UTF-8. It expects the text to be returned in Latin 1 or Latin 2, depending on the language, but Ubuntu and most other modern systems don't have these locales installed anymore. There is a workaround described in section 6 of the i18n README file: http://xaos.cvs.sourceforge.net/xaos/XaoS/src/i18n/README?revision=1.7&view=markup

Is adding support for the needed locales this something that could be addressed by the packager? I'm the Mac OS X maintainer, so I don't know a whole lot about debian packages.

A possible fix might be to put some function call to convert the text from UTF-8 returned by gettext to Latin 1 or 2 as appropriate before passing it into the text handling routine, but I have not looked into this.

Another possible fix (which would require a lot more work) is to replace XaoS's antiquated text system with the text drawing in a modern UI toolkit like gtk or qt. However, a prerequisite for this would be to write a new user interface for GTK or QT (easier than it sounds because of XaoS's portability framework, but still quite a bit of work). This is the approach I am taking for the Cocoa UI that I have written for Mac OS X. The menus and dialogs already handle UTF-8 correctly since they are native Mac OS X menus. However, the messages displayed on screen still go through XaoS's built-in text handling system and therefore don't show up correctly. The solution will be to replace calls to the built-in text handler with calls to the UI's text drawing functions. I'm not sure how much work that will be... possibly quite a bit.

This issue has been resolved in the upstream CVS and will be included in the upcoming 3.4 release. The problem is that XaoS's built-in text handling (the xprint function in grlib.c) only supports Latin-1 and Latin-2 encodings. On modern Linux systems (and Mac OS X), gettext returns strings in UTF-8. The solution is to convert the strings from UTF-8 back to Latin-1 or Latin-2 as appropriate before displaying them using XaoS's xprint function.

The meat of the changes are in grlib.c, but also, every call to xprint had to be updated to pass in the proper encoding, which is stored in the uih->encoding field (an integer 1 or 2 indicating Latin-1 or Latin-2 or 0 indicating that the encoding should not be changed). For the help and
tutorials, we pass in a 0 to prevent the text from being re-encoded. Tutorials pull strings directly from the appropriate catalog file instead of using gettext, so the text is already in the proper encoding. Help is not localized at all, so it also does not need to be re-encoded.

You may be able to use this fix against the 3.2 version that is included in ubuntu. You can probably use grlib.c directly as I don't think it contains any other changes between 3.2 until now. However, you will need to hand patch the files that call the xprint function, since these files may contain other changes between 3.2 and now. I would encourage Ubuntu to upgrade the package to the latest version. In addition to this fix, 3.3 (and the upcoming 3.4) will contain numerous other fixes, as well as an extremely important new feature, the ability to render user-defined formulas. Debian has already moved to 3.3 in testing and unstable. I will let you know when 3.4 has been released (it may be a few more weeks yet).

P.S., please submit your Italian translation for inclusion in the upstream package. We would love to have it. Version 3.4 will require a bit more development before we are ready to impose a string freeze. There will be quite a few string changes between 3.2 and 3.4, so there will probably be additional strings that need to be translated for 3.4.

1) It would be really nice having this new version ready for Hardy but we
have a very little time.
Do you think you could provide a patched version in time for it?
It would be great.

2) I can give you .po file. Just wait one day I need to check it. Can I put
it here as attachment?

2008/4/7, J.B. Langston III <email address hidden>:
>
> P.S., please submit your Italian translation for inclusion in the
> upstream package. We would love to have it. Version 3.4 will require a
> bit more development before we are ready to impose a string freeze.
> There will be quite a few string changes between 3.2 and 3.4, so there
> will probably be additional strings that need to be translated for 3.4.
>
>
> --
> Wrong characters codification in Xaos translation.
> https://bugs.launchpad.net/bugs/211710
> You received this bug notification because you are a direct subscriber
> of the bug.
>

P.S. Hardy final freeze is April 10th (this Thursday). very very little
time.

2008/4/7, Sergio Zanchetta <email address hidden>:
>
> 1) It would be really nice having this new version ready for Hardy but we
> have a very little time.
> Do you think you could provide a patched version in time for it?
> It would be great.
>
> 2) I can give you .po file. Just wait one day I need to check it. Can I
> put it here as attachment?
>
> 2008/4/7, J.B. Langston III <email address hidden>:
> >
> > P.S., please submit your Italian translation for inclusion in the
> > upstream package. We would love to have it. Version 3.4 will require a
> > bit more development before we are ready to impose a string freeze.
> > There will be quite a few string changes between 3.2 and 3.4, so there
> > will probably be additional strings that need to be translated for 3.4.
> >
> >
> > --
> > Wrong characters codification in Xaos translation.
> > https://bugs.launchpad.net/bugs/211710
> > You received this bug notification because you are a direct subscriber
> > of the bug.
> >
>
>

So you need a patch for this fix against the 3.2 sources with the ubuntu patches already applied? I don't know if i can get that to you by Thursday but I will try. I do not have a Linux environment set up right now since I use Mac OS X and my Ubuntu 7.10 VMWare image just got hosed. So even if I can produce the patch by Thursday, I won't be able to readily test it on Ubuntu.

Regarding the po file, please submit it as an attachment to our patch tracker: http://sourceforge.net/tracker/?group_id=5771&atid=305771. Also, to fully localize XaoS, you will also need to create a .cat file (in the catalogs directory). This is used to localize the tutorials. Please submit that along with the po file.

Here is the patch to fix the garbled text on systems using UTF-8. It should be applied against the 3.2 sources to which xaos_3.2-7ubuntu1.diff has already been applied. I have not had a chance to test it yet.

Here is the patch to fix the garbled text on systems using UTF-8. It should be applied against the 3.2 sources to which xaos_3.2-7ubuntu1.diff has already been applied. I have not had a chance to test it yet.

Maybe we could ask Tormod testing it, what do you think?

P.S. How can I create a .cat file?

Yes, I will test it, and suggest it for inclusion if it works. Thanks for the patch.

In the catalogs directory, copy english.cat to e.g., italiano.cat. Read the instructions in the comments at the beginning of the file, then translate each entry from English to Italian. This file is used to translate the tutorials. Without it, all the tutorials will still be in English, even if your locale is set to Italian.

@J.B. Langston III
Ah, ok. Now I understand why tutorials wasn't present in Launchpad for translations. I didn't know where to find them for translation.
I can provide you po file for now.
When I translate cat file I'll upload it as well. (Very very soon, but I don't think in time for Hardy by now, unless it is joined with patch you provided)
Thank you very much.

@Tormod
Thank you very much.

Your patch failed to build because HAVE_GETTEXT was probed before sourcing fconfig.h in grlib.c, and a missing comma in filesel.c:212. Having fixed that, I try to test it with: LC_ALL=fr xaos
but I just get:
An error occured in your setlocale/gettext installation.
I18n menus will not be available.

Am I testing it the right way?

You might have to do make install before it will work. XaoS expects its locale files to be in /usr/share/locale. I will try it to test it myself if I get a chance tonight.

@Tormod

I got my Ubuntu 7.10 VMWare image working again and I just tested the patch that you uploaded. It worked, but first I had to install the language support for all the languages I wanted to use via System>Administration>Language Support. After doing this and rebooting, I was able to set LANG to any of the languages supported by XaoS and the i18n text displayed correctly. Please give it a shot and let me know if it works for you. FYI, Hungarian or Czech are good languages to test with because they have lots of diacritical marks on the letters. French, German, and Spanish have them too, but you have to search around in the menus before you find a word that has them.

Sorry about the original patch not compiling correctly. I had to recreate the changes on 3.2 and I didn't have a Linux box available to test it at the time. Thank you for fixing it and helping test.

If you want some words to test in Italian language check this (all with
à->Ã)

Frattale -> Modalità Mandelbrot (this is what should be)
Frattale -> Tavolozza -> Velocità di ciclo dei colori (this is what should
be)
Calcolo -> Controllo di periodicità (this is what should be)
Calcolo -> Modalità julia veloce (this is what should be)
Calcolo -> Rotazione -> Velocità di rotazione (this is what should be)
IU -> Velocità di ingrandimento (this is what should be)
Aiuto -> Guide -> Novità (this is what should be)

You have to test with Hardy because it's not translated for Gutsy in
Launchpad (but now I upload .po file to sourceforge so you can use it
anyway)

2008/4/9, J.B. Langston III <email address hidden>:
>
> @Tormod
>
> I got my Ubuntu 7.10 VMWare image working again and I just tested the
> patch that you uploaded. It worked, but first I had to install the
> language support for all the languages I wanted to use via
> System>Administration>Language Support. After doing this and rebooting,
> I was able to set LANG to any of the languages supported by XaoS and the
> i18n text displayed correctly. Please give it a shot and let me know if
> it works for you. FYI, Hungarian or Czech are good languages to test
> with because they have lots of diacritical marks on the letters.
> French, German, and Spanish have them too, but you have to search around
> in the menus before you find a word that has them.
>
> Sorry about the original patch not compiling correctly. I had to
> recreate the changes on 3.2 and I didn't have a Linux box available to
> test it at the time. Thank you for fixing it and helping test.
>
>
> --
> Wrong characters codification in Xaos translation.
> https://bugs.launchpad.net/bugs/211710
> You received this bug notification because you are a direct subscriber
> of the bug.
>

I just uploaded po file to sourceforge.
Enjoy it. :-)

2008/4/9, Sergio Zanchetta <email address hidden>:
>
> If you want some words to test in Italian language check this (all with
> à->Ã)
>
> Frattale -> Modalità Mandelbrot (this is what should be)
> Frattale -> Tavolozza -> Velocità di ciclo dei colori (this is what should
> be)
> Calcolo -> Controllo di periodicità (this is what should be)
> Calcolo -> Modalità julia veloce (this is what should be)
> Calcolo -> Rotazione -> Velocità di rotazione (this is what should be)
> IU -> Velocità di ingrandimento (this is what should be)
> Aiuto -> Guide -> Novità (this is what should be)
>
> You have to test with Hardy because it's not translated for Gutsy in
> Launchpad (but now I upload .po file to sourceforge so you can use it
> anyway)
>
>
>
> 2008/4/9, J.B. Langston III <email address hidden>:
> >
> > @Tormod
> >
> > I got my Ubuntu 7.10 VMWare image working again and I just tested the
> > patch that you uploaded. It worked, but first I had to install the
> > language support for all the languages I wanted to use via
> > System>Administration>Language Support. After doing this and rebooting,
> > I was able to set LANG to any of the languages supported by XaoS and the
> > i18n text displayed correctly. Please give it a shot and let me know if
> > it works for you. FYI, Hungarian or Czech are good languages to test
> > with because they have lots of diacritical marks on the letters.
> > French, German, and Spanish have them too, but you have to search around
> > in the menus before you find a word that has them.
> >
> > Sorry about the original patch not compiling correctly. I had to
> > recreate the changes on 3.2 and I didn't have a Linux box available to
> > test it at the time. Thank you for fixing it and helping test.
> >
> >
> > --
> > Wrong characters codification in Xaos translation.
> > https://bugs.launchpad.net/bugs/211710
> > You received this bug notification because you are a direct subscriber
> > of the bug.
> >
>
>

Well, now I installed the French language packages and tried again. I get the same warning message and the menu entries are in English. The tutorial is in French and with characters as they should be.

However, I get the same results with the old package as with my new patched one!

If you are getting messages in English, it sounds like you are having a locale issue unrelated to the patch.

Tutorials were always displayed correctly even without the patch, because they don't use gettext but instead have their own message catalog format. This catalog is read directly from the file so it is already in the correct encoding (Latin 1 or 2) before it is displayed on screen. Menus, dialogs, and other messages are localized using gettext, which always returns strings encoded in whatever encoding is specified by your locale, regardless of the original encoding. On modern systems, this is usually UTF-8. Since the xaos text engine doesn't handle UTF-8, it tries to interpret the strings it receives from gettext as Latin 1 or Latin 2, depending on the language, and that is where the garbled characters come from. The patch converts the text received from gettext back to Latin 1 or Latin 2 before passing them to the text engine.

What is your LANG variable set to? The setting for LANG that worked for me is:

export LANG=fr_FR.UTF-8

Please make sure that you use this setting exactly, since I don't know if LANG=fr will work or not.

Also, after I installed support for the additional languages, I rebooted my system. I don't know if this is strictly necessary but you might want to give it a try.

Thanks, I was able to reproduce now. Using "LC_ALL=fr_FR.UTF-8 xaos" after installing French language packs showed garbled menu text with the current package, and the patched package worked fine.

Good news.
I completed italian translation of cat file.

Here you find both (po and cat)

http://sourceforge.net/tracker/?atid=305771&group_id=5771&func=browse

Please could you insert italian cat file together with the patch for this bug?
It would be really nice.

No matter about po file since it is automatically picked up from Launchpad Translations.

@Sergio

Thank you for all your work on this! I have committed both the cat and the po file to the upstream CVS. They will be included in version 3.4 of XaoS along with the bug fix for the text. I expect to make a pre-release within the next week and the final version within the next month or so.

I did find some small problems with your cat file. Several places, you had forgotten to include the ending double quote ("), and I also had to convert the file from UTF-8 to Latin 1 encoding. Even though the bug fix automatically converts the strings returned by gettext, it still expects the catalogs to be in Latin 1 or 2 since that is what all the other catalogs are in.

You can find the updated file at http://xaos.cvs.sourceforge.net/*checkout*/xaos/XaoS/catalogs/italiano.cat?revision=1.1. I confirmed that the tutorials successfully load and plays in Italian.

@Tormod

Great! Thanks for your effort testing this. Did the patch make it in time to be included in Hardy?

@J.B. Langston

Sorry about that.
It was very late when I finished last night, and I didn't check it all.

By the way, this morning I have deeply checked cat file again discovering untranslated parts and some mess.
Now it is ok.

I uploaded an updated version of cat file (called version 1.2, now in Latin 1 ;-) ) and po file (called "updated version", just a small change in here due to consistency with cat file).
Check it out at sourceforge.

@Tormod

Please pick up updated italian cat file and merge it with patch.
I corrected po file in Launchpad, so it will be updated automatically for Hardy release.

Thank you two for this great work.

Unfortunately, Hardy is in freeze now (as a penguin paradise as they said) so it can be difficult to get this accepted, unless i18n issues are given priority.

However, as soon as 3.4 gets into Debian, we'll sync/merge it into Intrepid Ibex (Hardy+1), and then we can request a backport to Hardy (the hardy-backports repository).

Could it be evaluated as a Freeze exception?
https://wiki.ubuntu.com/FreezeExceptionProcess

I think we can give it a try because this patch has a very little impact on
distribution (if not at all)
I think there is high probability of being accepted.

2008/4/11, Tormod Volden <email address hidden>:
>
> Unfortunately, Hardy is in freeze now (as a penguin paradise as they
> said) so it can be difficult to get this accepted, unless i18n issues
> are given priority.
>
> However, as soon as 3.4 gets into Debian, we'll sync/merge it into
> Intrepid Ibex (Hardy+1), and then we can request a backport to Hardy
> (the hardy-backports repository).
>
>
> --
> Wrong characters codification in Xaos translation - broken UTF support
> https://bugs.launchpad.net/bugs/211710
> You received this bug notification because you are a direct subscriber
> of the bug.
>

Oliver: can you please take a look at it?

FYI, this is fixed in upstream 3.4, but this is not in Debian yet and it can take some time since the package is orphaned.

xaos 3.4-1 was synced from debian, so this bug should be fixed

@Oliver

It was synced for Intrepid, but it has not be backported in Hardy yet.

can you provide a patch to the version in hardy ?

oh, sorry, silly me, tormod provided one ... setting to confirmed again

ubuntu-sru subscribed for a SRU request for Hardy.

Uploaded.

Accepted into hardy-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Oops, I forgot to put a valid address in debian/changelog. Don't know how important that is.

Any testers?

It works nice.

This bug was fixed in the package xaos - 3.2-7ubuntu2

---------------
xaos (3.2-7ubuntu2) hardy-proposed; urgency=low

  * fix translated texts to display correctly in UTF
    (inline) patch from J.B. Langston (upstream)
    (LP: #211710)

 -- Tormod Volden <tormod@viki> Tue, 08 Apr 2008 22:54:53 +0200