[freeciv-i18n] how to make www.freeciv.org translatable (was: French pages of Freeciv.org with Php errors)

[Reinier Post <rp@xxxxxxxxxx>]

This is a request for discussion on the translation of www.freeciv.org.

On Mon, Sep 30, 2002 at 09:26:58AM +0200, Xavier Belanger wrote:
> Hello,
> 
> Thanks for bug report but... another error now :)
> 
> Fatal error: Call to undefined function: php_code_find()
> in /home/freeciv/html/website_l10n/fr/index.phtml on line 11

There have been several changes to the use of scripts on the site,
most of which were originally installed by Jeff Mallatt.

I have been trying to make the website more portable, so I can
use a "preview copy" on my own website to test changes.  We have also
had to make some changes due to hardware and software upgrades.

What follows is a - hopefully complete- overview of the existing support
for translation of www.freeciv.org.


 1)  URI mapping:

The URI

  http://www.freeciv.org/path/to/document.ext

is the English-language document corresponding to the $LANG-language
document with the URI

  http://www.freeciv.org/$LANG/path/to/document.ext

Here, $LANG is one of the following: de, fr, ro, jp.
In general, it can be any valid value for $LANG.

The website is at the webserver's document root, i.e. it cannot
be installed at http://my.test.host/freeciv/;
this assumption is hardcoded in many places.  But it is fairly
easy to install it at, e.g.,  http://my.test.host:8080/.


 2)  Filesystem mapping:

The URI

  http://www.freeciv.org/path/to/document.ext

if it is a plain file, is the file

  path/to/document.ext

but due to some moving around in the past, the URI

  http://www.freeciv.org/$LANG/path/to/document.ext

is actually the file

  website_l10n/$LANG/path/to/document.ext

This is also its location in the CVS repository.
The file can also be found as

  $LANG/path/to/document.ext

because of symlinks

  $LANG -> website_l10n

but CVS cannot store symlinks, so these have to be
maintained by hand on every checked oput copy of the website.

As a side effect, the translated documents can also be reached under

  http://www.freeciv.org/website_l10n/$LANG/path/to/document.ext

but this is not supported, the website partly breaks when
such a URI is used.

The use of website_l10n also makes it harder for translators
to check out the website for the first time.  It's an ugly sequence
of CVS commands that a first-time translator will not understand.

Perhaps we should remove the website_l10n directory?


  3) Incompleteness of URI mapping:

The mapping

  http://www.freeciv.org/path/to/document.ext
    <->
  http://www.freeciv.org/$LANG/path/to/document.ext

is not one to one.  I don't even know if translated documents
exist for which there is no English equivalent.  The webmasters
of the English site don't check this.  I do know that many
English language documents have no translated documents.

Often, this is because a translation has not yet been provided.

Therefore, mechanisms are in place to make sure that, whenever a
translated document is requested and does not exist, the English
language document is served instead.

There are several mechanisms to do this because there are several
categories of files to consider.

  3.1) fallback mechanism for plain files

For files served from the filesystem, the fallback is implemented
with URI remapping by the webserver itself.  (This is now defined in
the central webserver configuration file.)  If the files are served
as they are, this is all that is needed.

  3.2) fallback mechanism for PHP include files

For files processed with PHP (usually *.phtml), there is a library
of common utility functions: the set of files php_code/*.php.
Some of these files contain translatable text, and for that reason,
there must be a fallback mechanism for them as well.  This is
implemented with the php_code_find() function in php_code/php_code_find.php.
Note that the previous mechanism cannot be used, because PHP include()s
do not work via the webserver, but access the filesystem directly.

I think this is wrong; I think these included functions should *never*
generate any translatable text by design, and all translated websites should
use the same set of included functions.  This fallback mechanism can
then be removed.

  3.3) other fallback mechanisms for files on the filesystem used in PHP

The previous mechanism only works for include()d pieces of code, but
PHP scripts also read and write files directly from/to the filesystem.
The function fallback_find_file() in php_code/fallback_find_file.php
provides a fallback mechanism that can be used for these files.

Such files are used in many different ways, e.g.

 + one PHP script takes a screenshots.xml file with an ad-hoc format
   and a set of images, and produces an image viewer with thumbnails
 + two PHP scripts take a file in an ad-hoc *.qa format and translate
   it to HTML
 + several PHP scripts use the news items database, which is a set of
   plain text files in an ad-hoc format, and present them as HTML
 + one PHP script takes the HTML files generated by the metaserver
   and transforms them into RSS descriptions

The general pattern is that a PHP script combines one or multiple
data files and generates output (usually HTML).  In principle, all
the data files should be subject to the fallback mechanism, and the
fallback_find_file() function is available for this purpose.  But
it is not used in many places.

In de/php_code/locale_link.php there is another function, which
compares translated file and English language file.  It would
probably be a good idea to combine this into a set of functions
that all sites can use.

However, the PHP scripts also produce hardcoded translatable text
themselves.  For this reason they need to be translated and be subject
to the fallback mechanism themselves.  I think this is wrong again,
it means that translators have to update these files not only when the
textual content changes (which is OK) but also when the HTML layout or
the internal processing changes (but these are language independent).

For this reason, the screenshot generation script (the first example)
no longer contains any hardcoded translatable text.  To translate a
set of screenshots it suffices to translate the screenshots.xml file,
which contains all the translatable text.  The other scripts should be
rewritten to follow this example.

But there is a second problem: the data files often contain structured
information, e.g. lists of items.  If only one item changes, the whole
file must be retranslated.  This is not very convenient for the translators.
Various solutions can be adopted here:

  - split the files such that every unit of translation becomes a separate file
  - adopt gettext: maintain a list of translatable strings that occur within
    the files on the site, and ask translators to translate this list

I have no opinion on what should be done here.  We have to do something:
at the moment, many of these partly-generated webpages are not translated
at all, because there is no clear method for doing so.


  3.3) no fallback mechanism for other types of data

The site also uses MySQL database tables with textual information:
the projects & ports page, the poll page, the locales page, etc.
There is no arrangement at all for how these pages are be translated.

The LXR and CVSweb services provide code views.  There should be
translated versions of these services too.  I have no idea how hard
it would be to support this.  Similar for Jitterbug.

Some data are generated by offline scripts:

  - the metaserver
  - the latest.html page
  - webserver usage overview
  - some things in the manual

These should be translated as well, but again, it is not clear how to
do this.

As a co-webmaster of the English site I think we have a lot of work
to do before every part of the website can be translated to another
language in a systematic way.  We cannot do everything at once, but I
think it would help to standardise on the mechanisms as much as possible.
I think we as the English maintainers must change all the "list" pages
(screenshots, the FAQ, the list of news items, the list of downloads, the
list of projects and ports, etc.) to store all the items in the exact same
way, and we must choose a way that makes translation as easy as possible.
But this is just my idea.

So I ask the translators: what are the most urgent problems you have
in translating the site?  And what solution would you propose?

-- 
Reinier Post