[Freeciv-Dev] Re: [Proposal] Changing the scorelog format

[Raimar Falke <hawk@xxxxxxxxxxxxxxxxxxxxxxx>]

On Sun, Dec 03, 2000 at 10:48:24AM -0500, Jeff Mallatt wrote:
> At 2000/12/02 14:49 , Raimar Falke wrote:
> >On Sat, Dec 02, 2000 at 11:29:47AM -0500, Jeff Mallatt wrote:
> >> At 2000/09/24 23:14 , Wong TM (Huang Deming) wrote:
> >
> >I looks like you have a huge backlog of emails. *grin*
> 
> Groan.  And, this is far from the oldest...  ;-)
> 
> >> >Problem: I see that the log file can get very big(at least for an all AIs
> >> game, 3MB
> >> >for 20 players).
> >> >
> >> >Goal: Reduce log size.
> >> >
> >> >With feedback from CivLog members, I thinking of:
> >> >
> >> >- stop outputing dead player log and maybe add a 'dead' tag to signify
> >> >end of dead player log.
> >> >
> >> >D  P YEAR DATA
> >> >-  - ---- ----
> >> >13 1 1100 100
> >> >13 2 1100 120
> >> >13 3 1100 dead <- player dead (not that much needed)
> >> >13 4 1100 144
> >> >[...]
> >> >13 1 1120 110
> >> >13 2 1120 111 <-\
> >> >13 4 1120 111 <- dead player skipped
> >> >
> >> >*D - Data types (Pop, Mfg, City etc.)
> >> >*P - Player
> >> 
> >> Dead players increase file size, but I believe there is another problem
> >> which makes the current format very inadequate.  If a civil war happens, a
> >> *new* player appears -- and I believe the log has no way to accommodate
> this.
> >
> >I agree that file size isn't a problem. You are right there is no way
> >to get the name of the new player.
> 
> Actually, though I didn't express myself well, I agree that file size is a
> problem.  If file size were not a problem, *most* of what follows would be
> unnecessary.

Why is size a problem? Normally scorelogs aren't transfered over the
network. Also local storage isn't a problem. Our (CivLog) problem was
parsing speed. Our initial method was slow. The current method reads a
1.9M scorelog file (153k lines, 22 players, 298 turns = from -4000 to
1739) in 7 seconds. This is Java on a P2-400 with 128MB RAM.

I think we don't need to reduce the size through omitting. IMHO only
small changes are needed to provide all information for a program
which reads the scorelog.

> >> The Players section looks something like:
> >> 
> >>     #Players
> >>     0 Lincoln
> >>     ...
> >>     3 Bismarck
> >>     ...
> >>     7 Napoleon
> >> 
> >> (note: again, no "end" tag).
> >> 
> >> The indices in index sections must be sequential, starting at zero, and
> >> without any gaps.
> >> 
> >> These sections may appear anywhere, any number of times, and affect all
> >> following "turn" sections (up to another "index" section, which replaces
> >> the previous one(s) of similar kind).
> >
> >Another way is to make changes more explicit. Instead of:
> > player section
> > turn sections
> > player section
> > turn sections
> >
> >it would possible to change the second player section to a "Player
> >change section". This section contains lines like
> >   #New player
> >   8 Foobar
> >and
> >   #Remove player
> >   7 Napoleon
> >So the scorelog reader doesn't have to do this stuff be itself.
> 
> Hum?  This looks to me to be harder for both the server and the reader.

But AFAIK these are the informations/changes a program wants to
process/to show the user.

> Also, if we are adding #End lines, shouldn't each #Turn section end with one?

No problem.

> >> Exit sections:
> >> --------------
> >> 
> >> Restarting a game, and trying to continue appending to a civscore log file
> >> would require re-reading the entire file if we didn't provide some
> >> assistance.  The "exit" sections do this.  When a game is terminated, an
> >> exit section is the last thing written to the log file.  It contains
> >> information which makes it easy to start appending to the log when the game
> >> is re-started.  It looks something like:
> >> 
> >>     #Exit 5423 9877 287645 2876556
> >> 
> >> where the first number is the byte offset into the log file of the latest
> >> FOM index section; the second number is the byte offset into the log file
> >> of the latest Players index section; the third number is the byte offset
> >> into the log file of the latest Save section; the fourth number is the byte
> >> offset into the log file of the latest Turns section which contains *all*
> >> data lines.  In the case where any referenced section does not exist, a 0
> >> is written (0 is an otherwise illegal value, because it would have to refer
> >> to the Header section).
> >> 
> >> This provides all the information we need to quickly re-start.  An Exit
> >> section is not required, because we can regenerate the information by
> >> parsing the entire file -- it will just take longer.
> >
> >If we have the code to recover those information in the server omit it
> >in the first place. So there is no double checking. The server is
> >written in C and should be able to reread the scorelog in short time.
> 
> If we just accept that re-starting will require re-reading the entire
> scorelog file, then we don't need the #Exit section *and* we don't need the
> second parameter on the #Save section.
> 
> We can try it first without the extra information, and see if it's fast
> enough.

I like this approach. If Java takes 7 seconds C will take almost no
time. Especially since the server doesn't have build own internal data
structures like CivLog.

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
  The trick is to keep breathing.