[Freeciv-Dev] Re: [Proposal] Changing the scorelog format

[Jeff Mallatt <jjm@xxxxxxxxxxxx>]

At 2000/12/02 14:49 , Raimar Falke wrote:
>On Sat, Dec 02, 2000 at 11:29:47AM -0500, Jeff Mallatt wrote:
>> At 2000/09/24 23:14 , Wong TM (Huang Deming) wrote:
>
>I looks like you have a huge backlog of emails. *grin*

Groan.  And, this is far from the oldest...  ;-)

>> >Problem: I see that the log file can get very big(at least for an all AIs
>> game, 3MB
>> >for 20 players).
>> >
>> >Goal: Reduce log size.
>> >
>> >With feedback from CivLog members, I thinking of:
>> >
>> >- stop outputing dead player log and maybe add a 'dead' tag to signify
>> >end of dead player log.
>> >
>> >D  P YEAR DATA
>> >-  - ---- ----
>> >13 1 1100 100
>> >13 2 1100 120
>> >13 3 1100 dead <- player dead (not that much needed)
>> >13 4 1100 144
>> >[...]
>> >13 1 1120 110
>> >13 2 1120 111 <-\
>> >13 4 1120 111 <- dead player skipped
>> >
>> >*D - Data types (Pop, Mfg, City etc.)
>> >*P - Player
>> 
>> Dead players increase file size, but I believe there is another problem
>> which makes the current format very inadequate.  If a civil war happens, a
>> *new* player appears -- and I believe the log has no way to accommodate
this.
>
>I agree that file size isn't a problem. You are right there is no way
>to get the name of the new player.

Actually, though I didn't express myself well, I agree that file size is a
problem.  If file size were not a problem, *most* of what follows would be
unnecessary.

>> Lines:
>> ------
>> 
>> There are two kinds of lines in the file: section lines and data lines.
>> Section lines introduce a section, and always begin with a "#" character.
>> Data lines contain data (surprise!), and may _not_ begin with a "#"
>> (actually, in the spec below, data lines always begin with a digit).
>>
>> Header section (must be first line):
>> 
>>     #Freeciv scorelog X.Y.Z
>
>I like it.
>
>> where X.Y.Z is the version number.
>> 
>> 
>> Index sections:
>> ---------------
>> 
>> Each "index" section sets the numeric indices for lines in the following
>> "turn" sections.  There are two types of index sections: "Figures of Merit"
>> ("FOM") and "Players".
>> 
>> The FOM section looks something like:
>> 
>>     #FOM
>>     0 population
>>     ...
>>     9 land area
>>     ...
>>     13 spaceship
>> 
>> (note: no longer is there an "end" tag -- it's not needed because each
>> section begins with a line that begins with a "#" character).
>
>I dislike it. A line like "#End" makes the parsing easier and it is no
>overhead for the server.

Okay.  Explicit #End lines don't bother me.

>> The Players section looks something like:
>> 
>>     #Players
>>     0 Lincoln
>>     ...
>>     3 Bismarck
>>     ...
>>     7 Napoleon
>> 
>> (note: again, no "end" tag).
>> 
>> The indices in index sections must be sequential, starting at zero, and
>> without any gaps.
>> 
>> These sections may appear anywhere, any number of times, and affect all
>> following "turn" sections (up to another "index" section, which replaces
>> the previous one(s) of similar kind).
>
>Another way is to make changes more explicit. Instead of:
> player section
> turn sections
> player section
> turn sections
>
>it would possible to change the second player section to a "Player
>change section". This section contains lines like
>   #New player
>   8 Foobar
>and
>   #Remove player
>   7 Napoleon
>So the scorelog reader doesn't have to do this stuff be itself.

Hum?  This looks to me to be harder for both the server and the reader.

>> Turn sections:
>> --------------
>> 
>> Each "turn" section contains the log info for a single turn of the game.
>> They are formatted much as you described above.  However, there is no
>> "dead" tag -- that would be handled by emitting a new "players" section
>> that simply does not contain the dead player.  The turn section would look
>> something like:
>> 
>>     #Turn -1250
>>     0 0 82376476
>>     1 0 2364876
>>     ...
>>     13 0 98723466
>>     0 1 8723544
>>     1 1 463773
>>     ...
>>     13 7 12476734
>> 
>> where the number on the "#Turn" line is the year; the first number on each
>> data line is the FOM index; the second number is the Player index; the
>> third number is the value of the figure of merit for the player.
>
>I would like also a turn number in the section headline. Like:
>   #Turn -4000 0

Sure.

>> Any FOM/Player/Value triplet that hasn't changed since the previous turn
>> may be elided.  However, the turn section immediately following an index
>> section must emit *all* lines (the indices may have changed, so the
>> "carryovers" need to be reset).
>
>I think the omitting (sp?) is not a good idea since the size isn't a
>problem (if it is: add compression). It complicates parsing.

Again, in addition to fixing the bug when players are added or removed, the
other goal was to reduce the size of the file.

Using zlib to compress the output is certainly a possibility, but wouldn't
that make it yet even more difficult for readers?

Parsing is pretty trivial, anyway.  You have a 2-D array of FOMs.  Each
line specifies the index, player and value, and you just do:

  fom_player[index][player] = value;

and anything not written over on this turn remains what it was from the
previous turn.

(BTW "omitting" is correct.)

Also, if we are adding #End lines, shouldn't each #Turn section end with one?

>> Save sections:
>> --------------
>> 
>> One big problem with the current code is trying to synchronize loading a
>> save-game and continue appending to a civscore log file.
>> 
>>     #Save 1492 198763
>> 
>> where the first number on the "#Save" line is the year, and must match the
>> year in the corresponding save-file; the second number is the byte offset
>> into the log file of the previous Save section (or 0 if none).
>> 
>> Also, the turn section immediately following a save section must emit *all*
>> lines (this makes re-starting simpler; see next section).
>
>Ok.
>
>> Exit sections:
>> --------------
>> 
>> Restarting a game, and trying to continue appending to a civscore log file
>> would require re-reading the entire file if we didn't provide some
>> assistance.  The "exit" sections do this.  When a game is terminated, an
>> exit section is the last thing written to the log file.  It contains
>> information which makes it easy to start appending to the log when the game
>> is re-started.  It looks something like:
>> 
>>     #Exit 5423 9877 287645 2876556
>> 
>> where the first number is the byte offset into the log file of the latest
>> FOM index section; the second number is the byte offset into the log file
>> of the latest Players index section; the third number is the byte offset
>> into the log file of the latest Save section; the fourth number is the byte
>> offset into the log file of the latest Turns section which contains *all*
>> data lines.  In the case where any referenced section does not exist, a 0
>> is written (0 is an otherwise illegal value, because it would have to refer
>> to the Header section).
>> 
>> This provides all the information we need to quickly re-start.  An Exit
>> section is not required, because we can regenerate the information by
>> parsing the entire file -- it will just take longer.
>
>If we have the code to recover those information in the server omit it
>in the first place. So there is no double checking. The server is
>written in C and should be able to reread the scorelog in short time.

If we just accept that re-starting will require re-reading the entire
scorelog file, then we don't need the #Exit section *and* we don't need the
second parameter on the #Save section.

We can try it first without the extra information, and see if it's fast
enough.

jjm