[Freeciv-Dev] Re: [Proposal] Changing the scorelog format

[Raimar Falke <hawk@xxxxxxxxxxxxxxxxxxxxxxx>]

On Sat, Dec 02, 2000 at 11:29:47AM -0500, Jeff Mallatt wrote:
> At 2000/09/24 23:14 , Wong TM (Huang Deming) wrote:

I looks like you have a huge backlog of emails. *grin*

> >Problem: I see that the log file can get very big(at least for an all AIs
> game, 3MB
> >for 20 players).
> >
> >Goal: Reduce log size.
> >
> >With feedback from CivLog members, I thinking of:
> >
> >- stop outputing dead player log and maybe add a 'dead' tag to signify
> >end of dead player log.
> >
> >D  P YEAR DATA
> >-  - ---- ----
> >13 1 1100 100
> >13 2 1100 120
> >13 3 1100 dead <- player dead (not that much needed)
> >13 4 1100 144
> >[...]
> >13 1 1120 110
> >13 2 1120 111 <-\
> >13 4 1120 111 <- dead player skipped
> >
> >*D - Data types (Pop, Mfg, City etc.)
> >*P - Player
> 
> Dead players increase file size, but I believe there is another problem
> which makes the current format very inadequate.  If a civil war happens, a
> *new* player appears -- and I believe the log has no way to accommodate this.

I agree that file size isn't a problem. You are right there is no way
to get the name of the new player.

> >- Do not place the year in each row but make sections like this:
> >
> >Turn: 1100
> >13 1 100
> >13 2 120
> >...
> >Turn: 1120
> >...
> 
> I think sectioning the file like this would work well.
>
> >- It is also possible to omit lines if the value didn't change.
> >
> >Nevertheless all these changes complicate thing during the parsing.
> 
> This change of omitting lines has less affect on programs which process the
> file than it does on the server which generates it.  I think omitting
> unchanged lines is also acceptable.
> 
> >I will continue if that's not objections. Comments?
> 
> Here's a proposed complete spec.  It's just some thoughts I've had in the
> past, merged with some ideas of yours, and is completely open to suggestion.
> 
> 
> Lines:
> ------
> 
> There are two kinds of lines in the file: section lines and data lines.
> Section lines introduce a section, and always begin with a "#" character.
> Data lines contain data (surprise!), and may _not_ begin with a "#"
> (actually, in the spec below, data lines always begin with a digit).
>
> Header section (must be first line):
> 
>     #Freeciv scorelog X.Y.Z

I like it.

> where X.Y.Z is the version number.
> 
> 
> Index sections:
> ---------------
> 
> Each "index" section sets the numeric indices for lines in the following
> "turn" sections.  There are two types of index sections: "Figures of Merit"
> ("FOM") and "Players".
> 
> The FOM section looks something like:
> 
>     #FOM
>     0 population
>     ...
>     9 land area
>     ...
>     13 spaceship
> 
> (note: no longer is there an "end" tag -- it's not needed because each
> section begins with a line that begins with a "#" character).

I dislike it. A line like "#End" makes the parsing easier and it is no
overhead for the server.

> The Players section looks something like:
> 
>     #Players
>     0 Lincoln
>     ...
>     3 Bismarck
>     ...
>     7 Napoleon
> 
> (note: again, no "end" tag).
> 
> The indices in index sections must be sequential, starting at zero, and
> without any gaps.
> 
> These sections may appear anywhere, any number of times, and affect all
> following "turn" sections (up to another "index" section, which replaces
> the previous one(s) of similar kind).

Another way is to make changes more explicit. Instead of:
 player section
 turn sections
 player section
 turn sections

it would possible to change the second player section to a "Player
change section". This section contains lines like
   #New player
   8 Foobar
and
   #Remove player
   7 Napoleon
So the scorelog reader doesn't have to do this stuff be itself.

> Turn sections:
> --------------
> 
> Each "turn" section contains the log info for a single turn of the game.
> They are formatted much as you described above.  However, there is no
> "dead" tag -- that would be handled by emitting a new "players" section
> that simply does not contain the dead player.  The turn section would look
> something like:
> 
>     #Turn -1250
>     0 0 82376476
>     1 0 2364876
>     ...
>     13 0 98723466
>     0 1 8723544
>     1 1 463773
>     ...
>     13 7 12476734
> 
> where the number on the "#Turn" line is the year; the first number on each
> data line is the FOM index; the second number is the Player index; the
> third number is the value of the figure of merit for the player.

I would like also a turn number in the section headline. Like:
   #Turn -4000 0

> Any FOM/Player/Value triplet that hasn't changed since the previous turn
> may be elided.  However, the turn section immediately following an index
> section must emit *all* lines (the indices may have changed, so the
> "carryovers" need to be reset).

I think the omitting (sp?) is not a good idea since the size isn't a
problem (if it is: add compression). It complicates parsing.

> Save sections:
> --------------
> 
> One big problem with the current code is trying to synchronize loading a
> save-game and continue appending to a civscore log file.
> 
>     #Save 1492 198763
> 
> where the first number on the "#Save" line is the year, and must match the
> year in the corresponding save-file; the second number is the byte offset
> into the log file of the previous Save section (or 0 if none).
> 
> Also, the turn section immediately following a save section must emit *all*
> lines (this makes re-starting simpler; see next section).

Ok.

> Exit sections:
> --------------
> 
> Restarting a game, and trying to continue appending to a civscore log file
> would require re-reading the entire file if we didn't provide some
> assistance.  The "exit" sections do this.  When a game is terminated, an
> exit section is the last thing written to the log file.  It contains
> information which makes it easy to start appending to the log when the game
> is re-started.  It looks something like:
> 
>     #Exit 5423 9877 287645 2876556
> 
> where the first number is the byte offset into the log file of the latest
> FOM index section; the second number is the byte offset into the log file
> of the latest Players index section; the third number is the byte offset
> into the log file of the latest Save section; the fourth number is the byte
> offset into the log file of the latest Turns section which contains *all*
> data lines.  In the case where any referenced section does not exist, a 0
> is written (0 is an otherwise illegal value, because it would have to refer
> to the Header section).
> 
> This provides all the information we need to quickly re-start.  An Exit
> section is not required, because we can regenerate the information by
> parsing the entire file -- it will just take longer.

If we have the code to recover those information in the server omit it
in the first place. So there is no double checking. The server is
written in C and should be able to reread the scorelog in short time.

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
 Make a software that is foolproof, and only fools will want to use it.