[Freeciv-Dev] Re: (PR#1824) ruleset data is in incompatible charsets

["Jason Short" <jdorje@xxxxxxxxxxxxxxxxxxxxx>]

<URL: http://rt.freeciv.org/Ticket/Display.html?id=1824 >

Raimar Falke wrote:

>>>The error is that GUI encoding will work with the functions in
>>>common/. We know from the past that this isn't the case. And it will
>>>also not be the case if the GUI encoding is latin1 (xaw) and the
>>>encoding used by common/ is utf8. The client and the server core needs
>>>the same encoding.
>>
>>Why won't it work?  What happened in the past?
> 
> For utf8 number of chars isn't number of bytes. There were problems
> with some buffers. Also is_sane_name uses is_iso_latin1.

The byte error has nothing to do with iconv.  It has nothing to do with 
changing the charset encoding (since we don't do this anyway).  The 
strings just need to be made smaller or the buffer bugger.

As for is_sane_name, it can convert from the internal encoding to the 
unicode encoding before proceeding.  So can any other functions that 
need a particular encoding (which is probably none).  This isn't an 
error; it's much simpler than changing all the GUI code.

>>>The problem here is that the common client code can't convert into the
>>>GUI encoding since the function signatures have to change for
>>>different encodings. For ASCII and utf8 it is "char *" for utf16 and
>>>ucs2 it is however "short int *".
>>
>>This isn't a problem so long as we assume we're always using 8-bit 
>>encodings.
> 
> gui-sdl isn't using an 8-bit encoding.

See my previous e-mail.  Using a 16-bit encoding means lots of problems. 
  There's really no way we can support both 16-bit and 8-bit encodings, 
even in the iconv code.  But gui-sdl already has the infrastructure to 
do the charset encoding itself so it is no problem.

>>The method you outline is possible.  However it will require many
>>more changes, since all GUIs will have to be changed to convert
>>to/from the internal encoding.  And I mean MANY changes.  And it
>>will make the GUI code harder to maintain because of all the places
>>where the conversions must be done.
> 
> You agree that it is the cleaner approach?!

No, it is absolutely not cleaner.  It gives a prettier flowchart but I 
believe the code will be unusable.

>>So we have to find the happy compromise between having the prettiest
>>flowchart and having the prettiest code.
> 
> _If_ we choose utf8 for the blue encoding the gtk2 client doesn't have
> much problems. The conversion from blue to red is a nop. xaw and gtk1
> would have to be changed.

And mui and win32???

jason