[Freeciv-Dev] (PR#2559) Font problem under BeOS

["Jason Short via RT" <rt@xxxxxxxxxxxxxx>]

So, to start the explanation from scratch...

The problem is that UTF-16 can be in either little-endian or big-endian
form.  The first two-byte character of the string is a byte-order-mark
(BOM) that shows what the endian-ness of the string is.  See
http://www.unicode.org/unicode/faq/utf_bom.html#22.

So, when using iconv to translate from the local encoding into UTF-16,
iconv sticks the BOM onto the front of the string.

gui-sdl just skips past this character.  Rafal apparently though it was
a bug.  SDL_ttf cannot handle the BOM, and in fact it appears SDL_ttf
needs the unicode to be little-endian (at least on little-endian
machines, see below).  The way to work around this is to use UTF-16LE,
which is always little-endian (and doesn't need the BOM).  See the
attached patch.

But, this could fail on big-endian machines.  gui-sdl breaks on Davide's
alpha and Bernd's beos machine.  Both are little-endian (I'm told), but
have big-endian UTF-16 strings for some reason.  But on a big-endian
machine it's possible (even likely) that we will need to use big-endian
unicode.  So in this case UTF-16LE wouldn't work - we'd probably have to
check the endian-ness of the machine (at compile time, for instance) and
use UTF-16BE for big-endian machines.

All of this seems like a gross hack.  SDL_ttf should be able to handle
UTF-16 with a BOM.

P.S.  The gui-sdl conversion using iconv is unnecessarily long.  Since
we know the target string's length (or think we do), we can do the
entire conversion with one call to iconv.  The loop is unnecessary, and
most of the extra variables are too.  But this function should go away
when this feature is provided by the core code, so we can wait until
then to change this.

jason

Index: client/gui-sdl/gui_iconv.c
===================================================================
RCS file: /home/freeciv/CVS/freeciv/client/gui-sdl/gui_iconv.c,v
retrieving revision 1.3
diff -u -r1.3 gui_iconv.c
--- client/gui-sdl/gui_iconv.c  2003/01/24 21:21:42     1.3
+++ client/gui-sdl/gui_iconv.c  2003/01/28 20:00:44
@@ -44,14 +44,13 @@
 Uint16 *convert_to_utf16(const char *pString)
 {
   /* Start Parametrs */
-  const char *pTocode = "UTF-16";
-  const char *pFromcode = LOCAL_CODE;
+  const char *pTocode = "UTF-16LE";
+  const char *pFromcode = "";
   const char *pStart = pString;
   const char *pEnd = pString + strlen(pString);
   /* ===== */
 
   char *pResult = NULL;
-  int iIter;
 
   /* From 8 bit code to UTF-16 (16 bit code)
      size_t length = 2*(strlen(pString)+1); */
@@ -72,7 +71,7 @@
   /* Do the conversion for real. */
   {
     const char *pInptr = pStart;
-    size_t Insize = pEnd - pStart;
+    size_t Insize = pEnd - pStart + 1;
 
     char *pOutptr = pResult;
     size_t Outsize = length;
@@ -109,13 +108,6 @@
     }
   }
 
-  /* Bug Corection */
-  for (iIter = 0; iIter < length - 2; iIter++) {
-    pResult[iIter] = pResult[iIter + 2];
-  }
-  pResult[length - 2] = 0;
-  pResult[length - 1] = 0;
-
   iconv_close(cd);
 
   return (Uint16 *) pResult;
@@ -127,14 +119,13 @@
 Uint16 *convertcopy_to_utf16(Uint16 * pToString, const char *pFromString)
 {
   /* Start Parametrs */
-  const char *pTocode = "UTF-16";
-  const char *pFromcode = LOCAL_CODE;
+  const char *pTocode = "UTF-16LE";
+  const char *pFromcode = "";
   const char *pStart = pFromString;
   const char *pEnd = pFromString + strlen(pFromString);
   /* ===== */
 
   char *pResult = (char *) pToString;
-  int iIter;
 
   /* From 8 bit code to UTF-16 (16 bit code)
      size_t length = 2*(strlen(pString)+1); */
@@ -157,7 +148,7 @@
   /* Do the conversion for real. */
   {
     const char *pInptr = pStart;
-    size_t Insize = pEnd - pStart;
+    size_t Insize = pEnd - pStart + 1;
 
     char *pOutptr = pResult;
     size_t Outsize = length;
@@ -191,13 +182,6 @@
       abort();
     }
   }
-
-  /* Bug Corection */
-  for (iIter = 0; iIter < length - 2; iIter++) {
-    pResult[iIter] = pResult[iIter + 2];
-  }
-  pResult[length - 2] = 0;
-  pResult[length - 1] = 0;
 
   iconv_close(cd);