[Freeciv-Dev] Re: RFC: Testing framework

[Raimar Falke <rf13@xxxxxxxxxxxxxxxxx>]

On Wed, Jul 10, 2002 at 11:04:25PM +0200, Per I Mathisen wrote:
> This is a proof-of-concept testing framework for Freeciv. It loads a
> savegame created with civworld and carries out a series of tests on it.
> The tests are currently quite few and very simple, so as it is presented
> here it isn't yet useful.
> 
> However, when we get ten times as many tests (and more), I think it should
> start working nicely as a contribution towards making sure changes don't
> break stuff.
> 
> It currently loads ruleset info from the data/default directory, which is
> wrong. It should have its own rulesets, since we will want to test a lot
> of strange and unusual units, techs, buildings, nations, etc..
> 
> Currently, when you do "(cd tests; make check)" it outputs this:
> 
> == Test run "COMMON" ==
> Diplomat bribe unit ... ok
> !Diplomat bribe unitless tile ... ok
> Spy bribe unit ... ok
> Spy embassy city ... ok
> Spy steal tech from city ... ok
> Spy investigate city ... ok
> Spy is diplomat (sic) ... ok
> Fighter is air unit ... ok
> !Fighter is heli unit ... ok
> !Fighter is ground unit ... ok
> !Helicopter is air unit ... ok
> Helicopter is heli unit ... ok
> !Helicopter is ground unit ... ok
> 
> == Test run "SERVER" ==
> Counting diplomats ... ok
> City has barracks ... ok
> Find closest city ... ok
> City can work tile ... ok
> City can place worker at tile ... ok
> PASS: savegame
> ==================
> All 1 tests passed
> ==================
> 
> To begin with, I will use this unit testing framework to prove that my
> unit flags patch (which adds 6 new unit flags) does what it is supposed to
> do, and ensure that other patches don't break this patch later on. I also
> hope that the gen. impr. patch, for instance, can be proven to work this
> way.
> 
> The advantages of such unit testing are
>  - patch writer gets to test assumptions about the patch
>  - maintainers get a way to see that the patch does what it is supposed to
>    do and if patch writer has thought about all possibilities for failure
>  - future patch writers get a way to see if their new patch breaks
>    some rarely used features, breakage that won't show up in manual
>    testing using default rulesets
>  - less bugs, which means quicker releases :)
> 
> Comments?

If you look at the past bugs I think that you won't catch them this
way. You would have to do multiple actions to trigger them. Sometime
they are also timing issues. However if you do so you will need a big
number of tests (multiple thousands or larger).

While I would also like tests I think that this way will only catch
the easiest of the bugs (the one I hope we get by proof reading before
appling the patch). The only solution I have to hold the system from
breaking apart are constrains. They are easy to test with asserts at
runtime. I would like to codify more assumptions we made. An example
is sizeof(int)==sizeof(enum). If this one isn't true at least the
network layer will break badly.

So if you have found a real bug with you method I may change my
mind. An easier test would be to also test previous versions. For
example the first betas of the previous releases (these should be more
buggy then the others).

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
 "Make it idiot-proof and someone will make a better idiot."