[Freeciv-Dev] Re: scorelog/savegame inconsistencies

[Reinier Post <rp@xxxxxxxxxx>]

On Sat, Jul 28, 2001 at 08:18:30PM -0400, Justin Moore wrote:
> 
>    If anyone's curious or would like to help me test the patch, you can
> grab it from
> 
> http://harmony.cs.virginia.edu/~jdm2d/public/scorelog-1.11.6-patch
> 
>    It applied cleanly to today's CVS, and was stable in the tests I ran.
> Any feedback, either on the patch itself or the idea, would be great.

(this is the idea:)

> >    As a possible (post 1.12.0) solution, what about merging the
> > civscore.log file with the save game file?  That way no matter when you
> > save the game, if you transfer it from one computer to another, or start
> > an older game, you have all the logging information with it.

I think the idea is conceptually wrong, but practically feasible.  The
reason it's conceptually wrong is that civscore.log consists of derived
information (the Reports that can be viewed during the game, plus a few
more).  civscore.log can be constructed from the savegames.  Actually,
I have a utility to do that, and I've used it in the past to construct
civscore.log files for games in which the civscore.log was missing or
corrupted.

There are some problems with the implementation:  civscore.log doesn't
record the names of players added while the game is running, which
Freeciv has been doing since January, 1999, and the code is hard to
maintain, being a single 400-line function with several state variables
and a goto.  Your patch appears to solve these problems.

This comes at the expense of grow memory use, by keeping the while
civscore.log info in memory, and disk use, by growing the savegames
with it.  The increase in size can be estimated by comparing the
relative size of compressed savegames and scorelogs.

> >  I'd like to get a sense of how quickly this will make the
> > saved files grow as time progresses.  Some early numbers have it as
> > growing 1K per player per 10 turns, but I'd like more accurate data.

You can get those, with some effort, from civserver.freeciv.org, e.g.,

sh> gameids=`GET 
'http://civserver.freeciv.org/query.phtml?minplayers=14&minyear=1800' | perl 
-ne '/game=(\d+)/ && print "$1 "'`

will get you the ids of the last 10 games played with at least 14 players
that didn't end before 1800 AD.

The scorelog and last savegame for each game are available
on the server: one way to list their lengths is this:

sh> for g in $gameids; do HEAD $repository/$g/civscore.log.gz; HEAD 
$repository/$g/civgame.sav.gz; done | grep -i '^content-[tl]'
Content-Length: 230195
Content-Type: application/x-freeciv-scorelog
Content-Length: 69285
Content-Type: application/x-freeciv-savedgame
Content-Length: 523662
Content-Type: application/x-freeciv-scorelog
Content-Length: 82024
Content-Type: application/x-freeciv-savedgame
Content-Length: 203360
Content-Type: application/x-freeciv-scorelog
Content-Length: 71082
Content-Type: application/x-freeciv-savedgame
Content-Length: 410191
Content-Type: application/x-freeciv-scorelog
Content-Length: 185667
Content-Type: application/x-freeciv-savedgame
Content-Length: 352250
Content-Type: application/x-freeciv-scorelog
Content-Length: 182749
Content-Type: application/x-freeciv-savedgame
Content-Length: 734905
Content-Type: application/x-freeciv-scorelog
Content-Length: 190108
Content-Type: application/x-freeciv-savedgame

So for this selection, with many players playing for a long time,
the scorelog is bigger than the savegame, but only by a few times.

For an idea of the average case, at

  http://www.freeciv.org/~hop/reports/year-savsize-scoresize.gz

you can find a table containing, for each of the 5183 saved games
on civserver.freeciv.org, a line with 3 numbers: the last year,
the size of civgame.sav.gz, and the size of civscore.log.gz.

It turns out that the summed sizes of savegames and scorelogs are
roughly equal. Limiting the comparison to use only recent games,
or only games that lasted until 1800 AD, or only games with a > 100K
savegame, doesn't have much effect:  the sum of the sizes still
compares as 1:1.5 or 1:2.

So I'd say that adding the civscore.log to the savegame will double
or triple it in size on average, and this is a fairly stable estimate.

> > Thanks.
> > -jdm

  Welcome,

-- 
Reinier Post