[Freeciv-Dev] Re: (PR#11028) error on loading savegame

["Jason Short" <jdorje@xxxxxxxxxxxxxxxxxxxxx>]

<URL: http://rt.freeciv.org/Ticket/Display.html?id=11028 >

Jason Short wrote:

> The problem is the savegame is (unzipped) 10 megabytes in size.  This is
>  66k lines comprising 400k "entries" (I'm not entirely sure what an
> entry is).  Each entry requires a bucket in the hash table and for
> efficiency the hash table is sparse.  So there are 1.5M hash table
> buckets, each of which is 16 bytes.  This makes for a 25 megabyte hash
> table.
> 
> The next question, of course, is what we can do about it.

The way the current loading method works is this: the section file is 
scanned, one entry at a time, and loaded into a list (well a group of 
lists, one for each section).  Then each entry from all these lists is 
put into the hash.  Then the savegame code runs and accesses the value 
in the hash.

So in addition to the file on disc, there are three copies of the data 
in the code at the same time: the entry lists, the sectionfile hash, and 
the game data itself.  This makes for a very large amount of memory usage.

It might be possible to cut out the entry lists entirely.  Assuming the 
hash can resize itself dynamically (which should take time O(1) 
ammortized) we can load the entries directly into the hash.  However 
that doesn't save us much because we'll still have the 25 megabyte (or 
50 megabyte, or 500 megabyte) memory chunk, plus a much larger (2x to 
10x) amount of memory in small chunks for the entries themselves.

----------

So, I believe there are only two solutions to this.

We can rewrite the saving and loading code to do streaming saves and 
loads.  This wouldn't be impossible - we'd probably want to use XML with 
libexpat or something, and the data would just be streamed in and out. 
The problem is that savegame loading is very order-dependent - for many 
things there is a specific order in which they must be loaded - which 
does not lend itself well to a streaming approach.

Or, we can drop the sanity check on memory allocations, and simply allow 
a ballooning amount of memory for saving and loading.

jason