[Freeciv-Dev] Re: [RFC][Patch] Reduce bandwith by using deltas

[Raimar Falke <rf13@xxxxxxxxxxxxxxxxx>]

On Mon, Nov 04, 2002 at 12:26:24PM +0000, Per I. Mathisen wrote:
> On Sat, 2 Nov 2002, Raimar Falke wrote:
> > > Surely there must be a way to do this cleaner through some macro
> > > magic?
> >
> > I thought about it and came to the conclusion: no. The basic problem
> > is that we need the list of fields multiple times: in the struct
> > definition, the send function and the receive function. With normal
> > macros we can only generate one of these. We have to have some kind of
> > memory for the preprocessor. While this may be possible I don't know
> > how to do this.
> 
> Ah, I knew my experience with XSLT would come handy one day. You can solve
> any kind of text parsing quite nicely without memory if you can use
> recursion, as long as you keep the logical structure intact.
> 
> You can define packets in, say, packets.h.in, like this
> 
>       PACKET_START(packet_player_info)
>       DATA_STRING(name, MAX_LEN_NAME)
>       DATA_BOOL(is_name)
>       CAPABILITY("team", DATA_UNIT8(team), team = TEAM_NONE)
>       DATA_UNIT8(government)
>       ...
>       PACKET_END
> 
> Then packets.h like this
> 
>       #define PACKET_START(x) struct x {
>       #define DATA_STRING(x, len) char x[len];
>       ...
>       #define PACKET_END };
>       #include "packets.h.in"
> 
> Then packets.c like this
> 
>       #include "packets.h"
> 
>       #undef PACKET_START
>       #define PACKET_START(x) \
>               int send_x(struct connection *pc, \
>                            const struct x *packet) \ { ...
>       #undef DATA_STRING
>       #define DATA_STRING(x, len) \
>               dio_put_string(&dout, packet->x); /* len */
>       ...
>       #include "packets.h.in"
> 
>       #undef PACKET_START
>       #define PACKET_START(x) \
>               struct x *receive_x(struct connection *pc) \ { ...
>       #undef DATA_STRING
>       #define DATA_STRING(x, len) \
>               dio_get_string(&din, packet->name, sizeof(packet->name)); /* 
> len */
>       ...
>       #include "packets.h.in"
> 
> This will generate vanilla packets.c and packets.h without compile
> dependendency on perl. Also packets*.c|h will be much smaller (from 150k
> to maybe 2k), and you only need to update one file to add a new packet.
> You can even autogenerate the packet_type enum this way.
> 
> The delta code is a little bit more complicated, so you will need to
> restructure the logic of it, because the macros cannot change the logical
> layout of the transformed text. One way is to change this
> 
> /* ... lots of BV_SET() ... */
> +  if (!BV_ISSET_ANY(fields) && already_known) {
> +    freelog(LOG_DEBUG, "  no change -> discard");
> +    stats_packet_game_info_discarded++;
> +    return 0;
> +  }
> /* ... lots of BV_ISSET() ... */
> 
> to doing all the work on one field at once, and instead discard the packet
> after it has been fully assembled. This is cleanest, but will waste some
> CPU compared to the above solution.
> 
> A way to keep the CPU savings is to make the generated function
> recursive... in PACKET_START(x) you declare "static bool second_run =
> FALSE; second_run = !second_run;" and in PACKET_END(x) you do "if
> (second_run && BV_ISSET_ANY(fields)) second_run send_packet_x(packet);".
> Then you make data defines like
> 
>       #undef DATA_STRING
>       #define DATA_STRING(x, len) \
>               if (!second_run) { ... BV_SET( code ) ... } \
>               if (second_run) { ... BV_ISSET( code ) ... }
> 
> Now _please_ shoot this into flames if you think it is stupid before I
> start any coding :)

I also came to this solution (including something multiple times and
change the semantic of certain macros). I don't think it will be more
pretty. The disadvantage I see is that you can't look at the
output. And also can't save it (longer compile time for everyone).

I'm working on version 2 of the delta protocol/generating. It will
support all packets. Since this will break the complete protocol
(capabilities won't help you here) I'm thinking about what else can be
added at this time. One thing is sub-byte values. So instead of

packet_tile_info; PACKET_TILE_INFO
        COORD x,y; key

        UINT8 type, known;
        UINT16 special;
end
  you would write
packet_tile_info; PACKET_TILE_INFO
        COORD x,y; key

        UINT4 type;
        UINT2 known; # because enum known_type only needs 2 bits
        # one byte saved
        UINT16 special;
end

Is this a good idea?

Also delta2 will not waste a byte for a bool but instead will use the
bit in the fields to store the real value instead of the has-changed
status.

        Raimar

-- 
 email: rf13@xxxxxxxxxxxxxxxxx
  The trick is to keep breathing.