[Freeciv-Dev] Timing tool

[Raimar Falke <hawk@xxxxxxxxxxxxxxxxxxxxxxx>]

Attached is a first version of a script which should allow us to
easily measure performance. I wrote it in python but sh or perl should
also be possible. The basic program is test_one.py. You can see
possible uses in test.sh. It does:
 - copy the base tree
 - apply any patches
 - call the configure command
 - call make
 - do <runs> autogames
 - check if the savegames are equal
 - create some plots
 - it also handles profiling (just add -p and it will add -pg and run
 gprof for each run)

All the results are put into a new directory <name>. There is a
sub-dir for every run. A sample result dir may look like this:

.
|-- configure_command.out
|-- report
|-- run_00
|   |-- civgame+0001.sav.gz
|   |-- civgame+0020.sav.gz
|   |-- civgame-0250.sav.gz
|   |-- civgame-0500.sav.gz
|   |-- civgame-0750.sav.gz
|   |-- civgame-1000.sav.gz
|   |-- civgame-1500.sav.gz
|   |-- civgame-2000.sav.gz
|   |-- civgame-2500.sav.gz
|   |-- civgame-3000.sav.gz
|   |-- civgame-3500.sav.gz
|   |-- gamelog
|   `-- log
|-- run_01
|   |-- civgame+0001.sav.gz
|   |-- civgame+0020.sav.gz
|   |-- civgame-0250.sav.gz
|   |-- civgame-0500.sav.gz
|   |-- civgame-0750.sav.gz
|   |-- civgame-1000.sav.gz
|   |-- civgame-1500.sav.gz
|   |-- civgame-2000.sav.gz
|   |-- civgame-2500.sav.gz
|   |-- civgame-3000.sav.gz
|   |-- civgame-3500.sav.gz
|   |-- gamelog
|   `-- log
|-- run_02
|   |-- civgame+0001.sav.gz
|   |-- civgame+0020.sav.gz
|   |-- civgame-0250.sav.gz
|   |-- civgame-0500.sav.gz
|   |-- civgame-0750.sav.gz
|   |-- civgame-1000.sav.gz
|   |-- civgame-1500.sav.gz
|   |-- civgame-2000.sav.gz
|   |-- civgame-2500.sav.gz
|   |-- civgame-3000.sav.gz
|   |-- civgame-3500.sav.gz
|   |-- gamelog
|   `-- log
|-- startup.final
|-- system.png
|-- system.times
|-- user.png
|-- user.times
|-- wall.png
`-- wall.times

The file "report" contains all relevant data to repeat the experiment
and also the timing results.

Sample report:

<begin quote>
Name:              O2_no_CHECK_MAP_POS
Runs:              3
Base-tree:         /mnt/hdb7/hawk/freeciv/new
RC-template:       /mnt/hdb7/hawk/freeciv/test/rc.templ
Configure command: /mnt/hdb7/hawk/freeciv/test/conf
Profile:           0
CFLAGS:            -g -O2 -Wall
CPPFLAGS:          

Patches:
  /mnt/hdb7/hawk/freeciv/disable_CHECK_MAP_POS.diff applied with -p0

Configure command:
---START /mnt/hdb7/hawk/freeciv/test/conf---
#!/bin/sh

./configure --enable-client=no
---END---
  w=21.98s, s=3.86s, u=16.15s

Make:
  w=152.05s, s=7.1s, u=138.87s

Startup script:
---START /mnt/hdb7/hawk/freeciv/test/rc.templ---
set randseed 4321
set seed 1234
set aifill 7
hard
set endyear 1
#set saveturns 1
---END---
---START /mnt/hdb7/hawk/freeciv/test/O2_no_CHECK_MAP_POS/startup.final---
set randseed 4321
set seed 1234
set aifill 7
hard
set endyear 1
#set saveturns 1

set timeout -1
set savename civgame
set gamelog 40
create Caesar
start
---END---

Runs:
  0: w=80.27s, s=0.23s, u=76.34s
  1: w=78.95s, s=0.16s, u=75.99s
  2: w=79.07s, s=0.13s, u=76.25s

Average wall time: 79.430000
Average system time: 0.173333
Average user time: 76.193333
Savegames are equal

<end quote>

TODO:
 - add scripts which do the plots and savegame comparison for multiple
 settings
 - remove bugs

I haven't stressed it a lot since the runs are pretty time
intensive. But it should be usable.

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
 "> WHY?! Isn't it better to put $(shell cat cscope.files) on the list of
  I only have a yellow belt in makefile kungfu.  These fancy gnu make things
  are relatively new to some of us..."
    -- Mark Frazer to Vassilii Khachaturov in linux-kernel

test.sh
Description: Bourne shell script

conf
Description: Text document

rc.templ
Description: Text document

disable_CHECK_MAP_POS.diff
Description: Text document

test_one.py
Description: Text document