[Freeciv-Dev] Re: RFC: Testing framework

["Per I. Mathisen" <per@xxxxxxxxxxx>]

On Thu, 11 Jul 2002, Raimar Falke wrote:
> If you look at the past bugs I think that you won't catch them this
> way.

Almost all of the big changes the last year have been in the client, not
in the server. This will change with the multiple generalisations being
worked on at the moment.

> You would have to do multiple actions to trigger them. Sometime
> they are also timing issues.

It will not catch them all.

> However if you do so you will need a big
> number of tests (multiple thousands or larger).

Yes, eventually.

> the easiest of the bugs (the one I hope we get by proof reading before
> appling the patch).

Not even the smartest maintainer can hold all the assumptions found in the
freeciv codebase in mind while proofreading patches.

> The only solution I have to hold the system from
> breaking apart are constrains. They are easy to test with asserts at
> runtime. I would like to codify more assumptions we made.

That is exactly what unit testing is all about, except on a higher level.
It is quite conceivable that bugs are detected during unit testing by
being caught by asserts. It is not just about testing function calls but
also "stimulating" various code paths rarely travelled to trigger asserts.

> So if you have found a real bug with you method I may change my
> mind. An easier test would be to also test previous versions. For
> example the first betas of the previous releases (these should be more
> buggy then the others).

That could be an interesting test.

Yours
Per