[Freeciv-Dev] Re: a civserver patch testing framework (was: [RFC][Patch] Inline)

[Raimar Falke <hawk@xxxxxxxxxxxxxxxxxxxxxxx>]

On Thu, Mar 21, 2002 at 02:28:43PM +0100, Reinier Post wrote:
> On Thu, Mar 21, 2002 at 10:04:39AM +0100, Raimar Falke wrote:
> 
> > <dream mode on>
> > There is an automatic patch tester. You give this bot a tree and a
> > list of patches. It will apply the patch, configure and build the code
> > (using options you specify).
> 
> I've thought out how to implement this in a Makefile
> and wrote an auxiliary utility, but not the Makefile itself.
> 
> It would be used from a web interface; its purpose would be
> to allow different versions to be run on civserver.freeciv.org.
> 
> > It will then run N autogames based on a
> > rc-template. It will measure the performance. This will be done for
> > every patch and the stock tree. Extra points for comparing the
> > savegames to the stock version. Extra points for measuring average and
> > variance. More extra points for feeding the results through
> > gnuplot. Extra points for fetching the stock tree from the CVS
> > repository.
> 
> We already have the framework running on civserver.freeciv.org: it
> starts games, reading the rc files to use from a configuration file,
> collecting all output files in a directory, and starting
> a reporting script on the results, which among other things traverses
> through all the savefiles and calls gnuplot; this would just be a
> variation.  We also have a Makefile that CVS ups the source, applies
> all patches found in patches/, ./configure && makes, mails any errors
> to the maintainers, and optionally starts the resulting server.
> What needs to be added is
> 
>  + support to run in autogame mode
>    (trivial)
>  + support to apply patches separately
>    (easy, I'm working on this)
>  + support to select the options to run with from a web interface
>    (should be easy)
>  + support to measure performance
>    (probably easy)
>  + support to publish the results (your 'extra points' points)
>    (I'd prefer if you do this yourself)

This tester should be easy to install and use. Reading "web interface"
means the opposite to me.

> > So I would just put some patched into the patches/ directory before I
> > go to sleep and have a nice analysis the next morning.
> 
> This already happens on civserver.freeciv.org.  If tests
> can be postponed to off-peak times and wrapped with nice and ulimit
> it shouldn't hurt any active games.

As I said this blurs the times. Probably due the context switches and
cache misses.

> > This will also
> > allow an analysis which functions are worth inlining (DIR_CW for
> > example isn't called in an autogame. Such automatic measurement will
> > also give us a feeling what gains can be expected by inlining certain
> > functions. I have currently no feeling for this.
> > <dream mode off>
> 
> I don't understand your focus on inlining.  

> Unnecessary iterations and inefficient lookups run a much higer risk
> of wasting CPU time.

Full ack. See the simple_ai_unit_type_iterate2.diff for this (which
btw isn't applied yet).

> CPU time is only important in AI games anyway; 

> bandwidth consumption is more important in interractive games

The people want this doesn't scream loud enough. And I have a 10mbit
connection.

> Memory consumption may be worth reporting.

> > Results:
> >  stock: 62.15s
> >  with this patch and no inlining: 80.97s (difference because of
> >  converting map_inx, dirstep,.. to functions)
> >  with this patch and inlining: 40.46s
> >  with this patch and inlining and NDEBUG: 36.83s
> >  with this patch and inlining and -O3: 39.48s
> 
> You'd have to study the behaviour when AI players grow big, because
> that's when CPU time becomes a factor.  It may be a good idea to
> test on a given set of .savs.

Ack. But I don't think that the mix of functions differ much in later
games. So the improvements you can achive on optimizing are the same.

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
  What's nice about GUI is that you see what you manipulate.
  What's bad about GUI is that you can only manipulate what you see.