[Freeciv-Dev] (PR#1824) ruleset data is in incompatible charsets

["Jason Short via RT" <rt@xxxxxxxxxxxxxx>]

After discussing with Vasco and Per, I think we should do the following...

There are three encodings freeciv uses: local_encoding,
freeciv_encoding, and display_encoding.  The local encoding is
determined by $LANG, and is the encoding gettext translates into. 
printf's should use this encoding.  The freeciv_encoding is used for
storage (including in savegames, rulesets, in memory, and across the
network) of post-translation or untranslated strings (that is, strings
that have already been translated or those that don't need to be); this
should be UTF-8.  display_encoding is that used by the frontend - this
will be UTF-8 for the GTK2 client, UTF-16 for the SDL client, and
(AFAIK) local_encoding for all other clients and the server.

We provide conversion function for switching between encodings.  Three
forms of the functions are possible:

  char *xxx_to_xxx_string(char *buf, size_t bufsz, const char *text);
  char *xxx_to_xxx_string_malloc(const char *text);
  char *xxx_to_xxx_string_static(const char *text);

The first one converts the string into the provided buffer (as much of
it as is possible).  The second calls fc_malloc to allocate an
appropriately-sized buffer for the string.  The third one may not be
desirable; this is just a shortcut to place the text into a static
buffer (it saves a few lines at the cost of 512 bytes or so).

As many conversion functions as are needed will be provided, but I
suspect we will only need to convert local->freeciv->display.  This
means three types of conversions, times 2-3 conversion forms, is 6-9
functions.  Of course most of these are just wrappers for the base
conversion functions.

Note that there is really a fourth type of encoding, which I will call
ascii_encoding, used for not-yet-translated strings (i.e., those that
need to be translated).  We don't need to do any explicit conversion on
this since gettext will take care of it.  It is likely (but not certain)
that this encoding needs to be ascii so that these strings will work for
people without NLS.

Then, we simply convert between forms as needed.  When a translated
string is created and used directly in the GUI, we call
local_to_display_string() on it.  When a string is loaded from a
ruleset, translated, and stored, we call gettext and then
local_to_freeciv_string() on it.  When we get a city name from the
network and want to display it, we call freeciv_to_display_string() on
it.  As long as we follow the basic tenet that any string stored beyond
local scope is in freeciv_encoding, it should be easy for anybody who
needs to use this string to convert it.

It may even be desirable to provide super-short wrapper macros for these
operations - particularly the static one - so that we could do something
like

   gtk_label_set_text(FtoD(pcity->name));


All of this sounds crazy and complicated.  But it is the only way we
have hit upon to simplify the handling of the different encodings that
may be involved.  It makes the current complicated global problem a much
simpler local one.

jason