[Freeciv-Dev] Re: (PR#1824) ruleset data is in incompatible charsets

["Jason Short" <jdorje@xxxxxxxxxxxxxxxxxxxxx>]

<URL: http://rt.freeciv.org/Ticket/Display.html?id=1824 >

> Before we start with this can you outline:
>  - your solution
>  - a graphic with the data stores and flows and its encodings
>  - a list of changes to the code

Here's a flowchart.

c0803274538d174ec7777146d2a5ddea  iconv.png


Gettext takes only ascii text, and outputs whatever we tell it to (but 
it must be a single encoding).

fc_fprintf converts from a consistent encoding into the local encoding.

The server and the client each use a certain internal encoding.  This is 
the encoding that is output by gettext and accepted by fc_fprintf. 
There is also a universal encoding used by the data files and sent over 
the network.  To make things easier the server just uses this universal 
encoding as its internal encoding.  Meanwhile the client uses the GUI 
encoding as its internal encoding.

The universal encoding is most likely UTF-8.  However this means 
rulesets must be converted from latin1.  So my most recent patch 
#defined the universal encoding as latin1 (which will work fine for 
western European language users).  Later this should be changed (along 
with the rulesets) to UTF-8.

The GUI encoding depends on what GUI is in use.  For the GTK2 client it 
is always UTF-8.  For the XAW and GTK clients it is the same as the 
local encoding (this may be configurable).  For the SDL client it is 
always UTF-16.

The bits in purple are where we do character set conversions (gettext 
also changes the character set, but we don't have to worry about that). 
  These purple parts involve a call to a function in fciconv.c.  Since 
there are four purple bits there may be up to four iconv conversion 
functions.  I think the current fciconv isn't quite set up for this, 
though; it's got some extra functions as well (though it will be easy to 
change once we agree on the final design).

Of course the server and client must also call some fciconv function(s) 
to tell what the local and GUI encodings are.

jason