[Freeciv-Dev] Re: [RFC][Patch] Reduce bandwith by using deltas

[Raimar Falke <rf13@xxxxxxxxxxxxxxxxx>]

On Fri, Nov 01, 2002 at 12:28:15PM +0100, Raimar Falke wrote:

Changes for version2:
 - works for all packets and not just infos
 - change syntax of the definition file. It is now a lot like C but
 also a but different.
 - added a lot of options, rewrite of the generator

Overview: to not miss anything (and to have you read this ;) ) I will
just insert the header of the packets_gen.in file and some packets:

/*

Syntax:

 Comments:
 =========

  Supported are: c-style (/ * * /), c++-style (//) and python style
  (#). Empty lines will be ignored.

 Typedef lines:
 ==============
 
  Start with "type" and have the format "type <alias> = <src>". You
  can now use <alias> at every place a type is expected. Nested type
  defs are possible.

 Packets:
 ========
  
  They just start and end with a line only containing "end". Header
  line is: "<struct-name>;<packet-type>;<flags>". If packet-type isn't
  given the packet type is given as parameter. Flags is a comma
  seperated list of:

   Packet flags:
   -------------

     is-info: a second packet with the same content can be discarded

     pre, post: call pre-send and post-receive hooks

     no-delta: don't use the delta protocol. This is useful for
     packets which contain always different data
     (packet_generic_integer) or are only sent once
     (packet_req_join_game). Sadly this also disables the use of 0 as
     the default value.

     no-packet: don't generate a packet argument for the send
     function. Currently only used by packet_generic_empty.

  Each other packet line has the format "<type> <fields>;<flags>".

    Type:
    ----

      <type> is an alias or a basic type. A basic type has the format
      "<dataio-type>(<public-type>)". Exception here is the float
      type. You can specify with the dataio-type "float<number>" the
      transmission of a float in a uint32 multiplied by this factor.

    Fields:
    -------

      Comma seperated list of names. Each name can have zero, one or
      two array declarations. So "x", "x[10]" and "x[20][10]" is
      possible. The array size in the "[]" can be specified plain as a
      term. In this case all elements will be transmitted. If this is
      not-desired you can specify the amount of elements to be
      transfered by given the number. So the extenmded format is
      "[<full-array-size>:<elements-to-transfer>]". elements-to-transfer
      is relative to the packet.

   Field flags:
   ------------

      key: create multiple entries in the cache indexed by the key set
      (set of all fields which have the key attribute). This allow a
      better delta compression.

*/

type UINT8              = uint8(int)
type UINT16             = uint16(int)
type UINT32             = uint32(int)
type SINT8              = sint8(int)
type SINT16             = sint16(int)
type SINT32             = sint32(int)
type BOOL               = bool8(bool)
type FLOAT              = float10000(float)

type MEMORY             = memory(unsigned char)
type STRING             = string(char)
type BIT_STRING         = bit_string(char)
type CITY_MAP           = city_map(char)
type WORKLIST           = worklist(struct worklist)
type TECH_LIST          = tech_list(int)

type COORD              = UINT8
type PLAYER             = UINT8
type CITY               = UINT16
type YEAR               = SINT16
type UNIT               = UINT16
type HP                 = UINT8
type PERCENT            = UINT8
type TECH               = UINT8

packet_tile_info; PACKET_TILE_INFO; is-info
  COORD x,y; key

  UINT8 type, known;
  UINT16 special;
end

packet_unit_info; PACKET_UNIT_INFO; is-info
  UNIT id; key
  PLAYER owner;
  COORD x,y;
  CITY homecity;

  BOOL veteran, ai, paradropped, connecting, carried, select_it;

  UINT8 type, movesleft, hp, fuel, activity, activity_count;
  UINT8 unhappiness, upkeep, upkeep_food, upkeep_gold;
  COORD goto_dest_x,goto_dest_y;
  uint16(enum tile_special_type) activity_target;
  UINT8 packet_use;
  CITY info_city_id;
  UINT16 serial_num;
end

packet_city_request;; pre
  CITY city_id;                         # all
  UINT8 build_id;                       # change, sell
  BOOL is_build_id_unit_id;             # change
  COORD worker_x, worker_y;             # make_worker, make_specialist
  UINT8 specialist_from, specialist_to; # change_specialist
  STRING name[MAX_LEN_NAME];            # rename
  worklist(struct worklist) worklist;   # worklist
end

packet_attribute_chunk;PACKET_ATTRIBUTE_CHUNK; pre,post
  UINT32 offset, total_length, chunk_length;
  MEMORY data[ATTRIBUTE_CHUNK_SIZE:chunk_length];
end

----------------------- end ----------------------------

Note the chunk_length in the data field in the packet_attribute_chunk
packet.

The patch does bool-header-folding i.e. it will store the bool value
in the header instead of the has-the-value-changed bit.

The generator also generates the lsend_ functions.

All packets which are required for initial rejectiion are
non-delta. So you get a clean reject because of capabilities mismatch.

I didn't convert two packets. packet_ruleset_building because it
doesn't contain all data and uses *vec* calls. I want the removed. But
this is a larger task. Second problem is packet_goto_route which is
just dead ugly and I didn't take the time to decrypt it.

On top of the common/generate_packets.py script are

  fold_bool_into_header=1
  generate_stats=1
  generate_freelogs=1
  freelog_log_level="LOG_DEBUG"

which should be obvious. With stats and freelogs the sizes of the
generated files are:
    751    1494   22781 generate_packets.py
    958    2631   29569 packets.c
    227     491    6490 packets.h
  10829   28108  322850 packets_gen.c
    731    2095   23882 packets_gen.h
    702    1919   18648 packets_gen.in
without the stats the the freelogs the sizes are:
   8667   19989  215222 packets_gen.c
    731    2095   23882 packets_gen.h
_a_ bit smaller.

I have attached a diff which doesn't contain the generated
files. (40kb limit of the mailing list). The generated files in the
next mail.

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
 "We just typed make..."
    -- Stephen Lambrigh, Director of Server Product Marketing at Informix,
                         about porting their Database to Linux

delta2.diff.bz2
Description: Binary data