[Freeciv-Dev] Re: [Proposal] Changing the scorelog format

[Jeff Mallatt <jjm@xxxxxxxxxxxx>]

At 2000/09/24 23:14 , Wong TM (Huang Deming) wrote:
>Problem: I see that the log file can get very big(at least for an all AIs
game, 3MB
>for 20 players).
>
>Goal: Reduce log size.
>
>With feedback from CivLog members, I thinking of:
>
>- stop outputing dead player log and maybe add a 'dead' tag to signify
>end of dead player log.
>
>D  P YEAR DATA
>-  - ---- ----
>13 1 1100 100
>13 2 1100 120
>13 3 1100 dead <- player dead (not that much needed)
>13 4 1100 144
>[...]
>13 1 1120 110
>13 2 1120 111 <-\
>13 4 1120 111 <- dead player skipped
>
>*D - Data types (Pop, Mfg, City etc.)
>*P - Player

Dead players increase file size, but I believe there is another problem
which makes the current format very inadequate.  If a civil war happens, a
*new* player appears -- and I believe the log has no way to accommodate this.

>- Do not place the year in each row but make sections like this:
>
>Turn: 1100
>13 1 100
>13 2 120
>...
>Turn: 1120
>...

I think sectioning the file like this would work well.

>- It is also possible to omit lines if the value didn't change.
>
>Nevertheless all these changes complicate thing during the parsing.

This change of omitting lines has less affect on programs which process the
file than it does on the server which generates it.  I think omitting
unchanged lines is also acceptable.

>I will continue if that's not objections. Comments?

Here's a proposed complete spec.  It's just some thoughts I've had in the
past, merged with some ideas of yours, and is completely open to suggestion.


Lines:
------

There are two kinds of lines in the file: section lines and data lines.
Section lines introduce a section, and always begin with a "#" character.
Data lines contain data (surprise!), and may _not_ begin with a "#"
(actually, in the spec below, data lines always begin with a digit).

Header section (must be first line):

    #Freeciv scorelog X.Y.Z

where X.Y.Z is the version number.


Index sections:
---------------

Each "index" section sets the numeric indices for lines in the following
"turn" sections.  There are two types of index sections: "Figures of Merit"
("FOM") and "Players".

The FOM section looks something like:

    #FOM
    0 population
    ...
    9 land area
    ...
    13 spaceship

(note: no longer is there an "end" tag -- it's not needed because each
section begins with a line that begins with a "#" character).

The Players section looks something like:

    #Players
    0 Lincoln
    ...
    3 Bismarck
    ...
    7 Napoleon

(note: again, no "end" tag).

The indices in index sections must be sequential, starting at zero, and
without any gaps.

These sections may appear anywhere, any number of times, and affect all
following "turn" sections (up to another "index" section, which replaces
the previous one(s) of similar kind).


Turn sections:
--------------

Each "turn" section contains the log info for a single turn of the game.
They are formatted much as you described above.  However, there is no
"dead" tag -- that would be handled by emitting a new "players" section
that simply does not contain the dead player.  The turn section would look
something like:

    #Turn -1250
    0 0 82376476
    1 0 2364876
    ...
    13 0 98723466
    0 1 8723544
    1 1 463773
    ...
    13 7 12476734

where the number on the "#Turn" line is the year; the first number on each
data line is the FOM index; the second number is the Player index; the
third number is the value of the figure of merit for the player.

Any FOM/Player/Value triplet that hasn't changed since the previous turn
may be elided.  However, the turn section immediately following an index
section must emit *all* lines (the indices may have changed, so the
"carryovers" need to be reset).


Save sections:
--------------

One big problem with the current code is trying to synchronize loading a
save-game and continue appending to a civscore log file.

    #Save 1492 198763

where the first number on the "#Save" line is the year, and must match the
year in the corresponding save-file; the second number is the byte offset
into the log file of the previous Save section (or 0 if none).

Also, the turn section immediately following a save section must emit *all*
lines (this makes re-starting simpler; see next section).


Exit sections:
--------------

Restarting a game, and trying to continue appending to a civscore log file
would require re-reading the entire file if we didn't provide some
assistance.  The "exit" sections do this.  When a game is terminated, an
exit section is the last thing written to the log file.  It contains
information which makes it easy to start appending to the log when the game
is re-started.  It looks something like:

    #Exit 5423 9877 287645 2876556

where the first number is the byte offset into the log file of the latest
FOM index section; the second number is the byte offset into the log file
of the latest Players index section; the third number is the byte offset
into the log file of the latest Save section; the fourth number is the byte
offset into the log file of the latest Turns section which contains *all*
data lines.  In the case where any referenced section does not exist, a 0
is written (0 is an otherwise illegal value, because it would have to refer
to the Header section).

This provides all the information we need to quickly re-start.  An Exit
section is not required, because we can regenerate the information by
parsing the entire file -- it will just take longer.


Comments:
---------

Note that the Header, Save and Exit sections are all single-line sections
which may be simply ignored by processing programs.  Save and Exit are used
by the server to re-start a save game with a log file correctly, easily and
quickly.

jjm