[Freeciv-Dev] Re: (PR#2743) Blank messages and bad packet strings

["Vasco Alexandre Da Silva Costa via RT" <rt@xxxxxxxxxxxxxx>]

On Wed, 8 Jan 2003, Raimar Falke via RT wrote:

> On Wed, Jan 08, 2003 at 02:21:02AM -0800, kenn@xxxxxxxxxxxxxx via RT wrote:
> > On Tue, 2003-01-07 at 00:28, Vasco Alexandre Da Silva Costa via RT
> > wrote:
> > >
> > > On Mon, 6 Jan 2003, Guest via RT wrote:
> > >
> > > > Occasionally I get blank lines in my message window.
> > >
> > > Are you a RedHat 8.0 user?
> > Yes
> >
> > > Is your current charset UTF-8? If you are unsure check it by running the
> > > command 'locale'.
> > Yes.. they are all set to =en_CA.UTF-8
> >
> > > If you are one of these unfortunate users the problem is that the freeciv
> > > server is not charset clean. You must start the freeciv server in a
> > > ISO-8859-1 locale as well as the client.
> > >
> > > How to do it:
> > >
> > > export LANG=en_US.ISO-8859-1
> > > ./civ &
> > > ./ser
> > >
> > > Did this solve your problem?
> > Yes.. thanks!
>
> Vasco: is this documented somewhere?

No. Yes it should be.

RedHat 8.0 users have all sorts of problems with several apps. Most Motif
and GTK+ 1.2 apps just don't work well or even bomb on startup.

The bundled X-Chat for e.g. transmits multibyte characters when you
type non-ASCII text and can't even display the text you type in it.
Acrobat Reader 5.0.5 bombs.

Most apps do not work in UTF-8 mode because they are not locale clean.
They are dependent on a specific text encoding, like some programs have
problems with integer byte order on different platforms.

Civserver is very, very broken regarding I18N. We store the nation strings
in ISO-8859-1 format when it is not assured that the user actually uses
ISO-8859-1. For universal client<->server communication a common text
encoding format must be established. Be it ASCII, UTF-8 or otherwise.

I still think the best choice is to make everything sent and stored in
ASCII except the player chat messages.
gettext will take care of translating to the native language locally.

This means city names/leaders names etc must be ASCII. I think this is a
reasonable approach because it makes little sense to send key strings like
these, used to index important data, composed of characters some clients
cannot display (e.g. Japanese, Russian).

ASCII is so widespread we can rely on it for being displayable. UTF-8 for
player<->player messages is just icing on the cake. Why should we
constrain users so that they cannot chat in their native language?

---
Vasco Alexandre da Silva Costa @ Instituto Superior Tecnico, Lisboa