[Freeciv-Dev] Re: a civserver patch testing framework (was: [RFC][Patch]
[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
On Thu, Mar 21, 2002 at 02:28:43PM +0100, Reinier Post wrote:
> On Thu, Mar 21, 2002 at 10:04:39AM +0100, Raimar Falke wrote:
> > <dream mode on>
> > There is an automatic patch tester. You give this bot a tree and a
> > list of patches. It will apply the patch, configure and build the code
> > (using options you specify).
> I've thought out how to implement this in a Makefile
> and wrote an auxiliary utility, but not the Makefile itself.
> It would be used from a web interface; its purpose would be
> to allow different versions to be run on civserver.freeciv.org.
> > It will then run N autogames based on a
> > rc-template. It will measure the performance. This will be done for
> > every patch and the stock tree. Extra points for comparing the
> > savegames to the stock version. Extra points for measuring average and
> > variance. More extra points for feeding the results through
> > gnuplot. Extra points for fetching the stock tree from the CVS
> > repository.
> We already have the framework running on civserver.freeciv.org: it
> starts games, reading the rc files to use from a configuration file,
> collecting all output files in a directory, and starting
> a reporting script on the results, which among other things traverses
> through all the savefiles and calls gnuplot; this would just be a
> variation. We also have a Makefile that CVS ups the source, applies
> all patches found in patches/, ./configure && makes, mails any errors
> to the maintainers, and optionally starts the resulting server.
> What needs to be added is
> + support to run in autogame mode
> + support to apply patches separately
> (easy, I'm working on this)
> + support to select the options to run with from a web interface
> (should be easy)
> + support to measure performance
> (probably easy)
> + support to publish the results (your 'extra points' points)
> (I'd prefer if you do this yourself)
This tester should be easy to install and use. Reading "web interface"
means the opposite to me.
> > So I would just put some patched into the patches/ directory before I
> > go to sleep and have a nice analysis the next morning.
> This already happens on civserver.freeciv.org. If tests
> can be postponed to off-peak times and wrapped with nice and ulimit
> it shouldn't hurt any active games.
As I said this blurs the times. Probably due the context switches and
> > This will also
> > allow an analysis which functions are worth inlining (DIR_CW for
> > example isn't called in an autogame. Such automatic measurement will
> > also give us a feeling what gains can be expected by inlining certain
> > functions. I have currently no feeling for this.
> > <dream mode off>
> I don't understand your focus on inlining.
> Unnecessary iterations and inefficient lookups run a much higer risk
> of wasting CPU time.
Full ack. See the simple_ai_unit_type_iterate2.diff for this (which
btw isn't applied yet).
> CPU time is only important in AI games anyway;
> bandwidth consumption is more important in interractive games
The people want this doesn't scream loud enough. And I have a 10mbit
> Memory consumption may be worth reporting.
> > Results:
> > stock: 62.15s
> > with this patch and no inlining: 80.97s (difference because of
> > converting map_inx, dirstep,.. to functions)
> > with this patch and inlining: 40.46s
> > with this patch and inlining and NDEBUG: 36.83s
> > with this patch and inlining and -O3: 39.48s
> You'd have to study the behaviour when AI players grow big, because
> that's when CPU time becomes a factor. It may be a good idea to
> test on a given set of .savs.
Ack. But I don't think that the mix of functions differ much in later
games. So the improvements you can achive on optimizing are the same.
What's nice about GUI is that you see what you manipulate.
What's bad about GUI is that you can only manipulate what you see.
[Freeciv-Dev] Re: [RFC][Patch] Inline, Jason Short, 2002/03/21
[Freeciv-Dev] a civserver patch testing framework (was: [RFC][Patch] Inline), Reinier Post, 2002/03/21
[Freeciv-Dev] Re: [RFC][Patch] Inline, Raimar Falke, 2002/03/21