[Freeciv-Dev] a civserver patch testing framework (was: [RFC][Patch] Inl
[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index] [Thread Index]
freeciv development list <freeciv-dev@xxxxxxxxxxx>
[Freeciv-Dev] a civserver patch testing framework (was: [RFC][Patch] Inline)
Reinier Post <rp@xxxxxxxxxx>
Thu, 21 Mar 2002 14:28:43 +0100
On Thu, Mar 21, 2002 at 10:04:39AM +0100, Raimar Falke wrote:
> <dream mode on>
> There is an automatic patch tester. You give this bot a tree and a
> list of patches. It will apply the patch, configure and build the code
> (using options you specify).
I've thought out how to implement this in a Makefile
and wrote an auxiliary utility, but not the Makefile itself.
It would be used from a web interface; its purpose would be
to allow different versions to be run on civserver.freeciv.org.
> It will then run N autogames based on a
> rc-template. It will measure the performance. This will be done for
> every patch and the stock tree. Extra points for comparing the
> savegames to the stock version. Extra points for measuring average and
> variance. More extra points for feeding the results through
> gnuplot. Extra points for fetching the stock tree from the CVS
We already have the framework running on civserver.freeciv.org: it
starts games, reading the rc files to use from a configuration file,
collecting all output files in a directory, and starting
a reporting script on the results, which among other things traverses
through all the savefiles and calls gnuplot; this would just be a
variation. We also have a Makefile that CVS ups the source, applies
all patches found in patches/, ./configure && makes, mails any errors
to the maintainers, and optionally starts the resulting server.
What needs to be added is
+ support to run in autogame mode
+ support to apply patches separately
(easy, I'm working on this)
+ support to select the options to run with from a web interface
(should be easy)
+ support to measure performance
+ support to publish the results (your 'extra points' points)
(I'd prefer if you do this yourself)
> So I would just put some patched into the patches/ directory before I
> go to sleep and have a nice analysis the next morning.
This already happens on civserver.freeciv.org. If tests
can be postponed to off-peak times and wrapped with nice and ulimit
it shouldn't hurt any active games.
> This will also
> allow an analysis which functions are worth inlining (DIR_CW for
> example isn't called in an autogame. Such automatic measurement will
> also give us a feeling what gains can be expected by inlining certain
> functions. I have currently no feeling for this.
> <dream mode off>
I don't understand your focus on inlining. Unnecessary iterations
and inefficient lookups run a much higer risk of wasting CPU time.
CPU time is only important in AI games anyway; bandwidth consumption
is more important in interractive games. Memory consumption may
be worth reporting.
> stock: 62.15s
> with this patch and no inlining: 80.97s (difference because of
> converting map_inx, dirstep,.. to functions)
> with this patch and inlining: 40.46s
> with this patch and inlining and NDEBUG: 36.83s
> with this patch and inlining and -O3: 39.48s
You'd have to study the behaviour when AI players grow big, because
that's when CPU time becomes a factor. It may be a good idea to
test on a given set of .savs.
[Freeciv-Dev] Re: [RFC][Patch] Inline, Jason Short, 2002/03/21
[Freeciv-Dev] a civserver patch testing framework (was: [RFC][Patch] Inline),
Reinier Post <=
[Freeciv-Dev] Re: [RFC][Patch] Inline, Raimar Falke, 2002/03/21