[Freeciv-Dev] Re: (PR#1824) ruleset data is in incompatible charsets

["Raimar Falke" <i-freeciv-lists@xxxxxxxxxxxxx>]

<URL: http://rt.freeciv.org/Ticket/Display.html?id=1824 >

On Sat, May 08, 2004 at 02:06:53AM -0700, Jason Short wrote:
> 
> <URL: http://rt.freeciv.org/Ticket/Display.html?id=1824 >
> 
> Raimar Falke wrote:
> > <URL: http://rt.freeciv.org/Ticket/Display.html?id=1824 >
> > 
> > On Fri, May 07, 2004 at 11:32:30AM -0700, Jason Short wrote:
> > 
> >>So with these changes the flow chart becomes much simpler.  See
> >>http://freeciv.org/~jdorje/iconv.png.
> >>
> >>Strings are stored in the server in unicode.  In the client they are
> >>stored in the GUI encoding.
> > 
> > There is a big error and a problem in this.
> > 
> > The error is that GUI encoding will work with the functions in
> > common/. We know from the past that this isn't the case. And it will
> > also not be the case if the GUI encoding is latin1 (xaw) and the
> > encoding used by common/ is utf8. The client and the server core needs
> > the same encoding.
> 
> Why won't it work?  What happened in the past?

For utf8 number of chars isn't number of bytes. There were problems
with some buffers. Also is_sane_name uses is_iso_latin1.

> > The problem here is that the common client code can't convert into the
> > GUI encoding since the function signatures have to change for
> > different encodings. For ASCII and utf8 it is "char *" for utf16 and
> > ucs2 it is however "short int *".
> 
> This isn't a problem so long as we assume we're always using 8-bit 
> encodings.

gui-sdl isn't using an 8-bit encoding.

> The method you outline is possible.  However it will require many
> more changes, since all GUIs will have to be changed to convert
> to/from the internal encoding.  And I mean MANY changes.  And it
> will make the GUI code harder to maintain because of all the places
> where the conversions must be done.

You agree that it is the cleaner approach?!

For gui-sdl and gui-fs these changes aren't this much of a problem
since all string data already goes through only a few functions.

> If you plan to change all internal freeciv strings to be utf-16/ucs2
> then you're talking about many more changes again.  And here I mean
> MANY MANY changes.  UCS2 gives us no advantage over UTF-8, since the
> constant width doesn't do anything useful for us.  It also adds a
> hard dependency of iconv (as opposed to a soft dependency).

This decision is independent of the above. And I agree with you that
utf-16 would require a lot of changes.

> So we have to find the happy compromise between having the prettiest
> flowchart and having the prettiest code.

_If_ we choose utf8 for the blue encoding the gtk2 client doesn't have
much problems. The conversion from blue to red is a nop. xaw and gtk1
would have to be changed.

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
 "> WHY?! Isn't it better to put $(shell cat cscope.files) on the list of
  I only have a yellow belt in makefile kungfu.  These fancy gnu make things
  are relatively new to some of us..."
    -- Mark Frazer to Vassilii Khachaturov in linux-kernel