[Freeciv-Dev] a civserver patch testing framework (was: [RFC][Patch] Inline)

[Reinier Post <rp@xxxxxxxxxx>]

On Thu, Mar 21, 2002 at 10:04:39AM +0100, Raimar Falke wrote:

> <dream mode on>
> There is an automatic patch tester. You give this bot a tree and a
> list of patches. It will apply the patch, configure and build the code
> (using options you specify).

I've thought out how to implement this in a Makefile
and wrote an auxiliary utility, but not the Makefile itself.

It would be used from a web interface; its purpose would be
to allow different versions to be run on civserver.freeciv.org.

> It will then run N autogames based on a
> rc-template. It will measure the performance. This will be done for
> every patch and the stock tree. Extra points for comparing the
> savegames to the stock version. Extra points for measuring average and
> variance. More extra points for feeding the results through
> gnuplot. Extra points for fetching the stock tree from the CVS
> repository.

We already have the framework running on civserver.freeciv.org: it
starts games, reading the rc files to use from a configuration file,
collecting all output files in a directory, and starting
a reporting script on the results, which among other things traverses
through all the savefiles and calls gnuplot; this would just be a
variation.  We also have a Makefile that CVS ups the source, applies
all patches found in patches/, ./configure && makes, mails any errors
to the maintainers, and optionally starts the resulting server.
What needs to be added is

 + support to run in autogame mode
   (trivial)
 + support to apply patches separately
   (easy, I'm working on this)
 + support to select the options to run with from a web interface
   (should be easy)
 + support to measure performance
   (probably easy)
 + support to publish the results (your 'extra points' points)
   (I'd prefer if you do this yourself)

> So I would just put some patched into the patches/ directory before I
> go to sleep and have a nice analysis the next morning.

This already happens on civserver.freeciv.org.  If tests
can be postponed to off-peak times and wrapped with nice and ulimit
it shouldn't hurt any active games.

> This will also
> allow an analysis which functions are worth inlining (DIR_CW for
> example isn't called in an autogame. Such automatic measurement will
> also give us a feeling what gains can be expected by inlining certain
> functions. I have currently no feeling for this.
> <dream mode off>

I don't understand your focus on inlining.  Unnecessary iterations
and inefficient lookups run a much higer risk of wasting CPU time.
CPU time is only important in AI games anyway; bandwidth consumption
is more important in interractive games.  Memory consumption may
be worth reporting.

> Results:
>  stock: 62.15s
>  with this patch and no inlining: 80.97s (difference because of
>  converting map_inx, dirstep,.. to functions)
>  with this patch and inlining: 40.46s
>  with this patch and inlining and NDEBUG: 36.83s
>  with this patch and inlining and -O3: 39.48s

You'd have to study the behaviour when AI players grow big, because
that's when CPU time becomes a factor.  It may be a good idea to
test on a given set of .savs.

-- 
Reinier