[Freeciv-Dev] Re: (PR#9947) insane inefficiency in packet sending

["Jason Short" <jdorje@xxxxxxxxxxxxxxxxxxxxx>]

<URL: http://rt.freeciv.org/Ticket/Display.html?id=9947 >

Jason Short wrote:
> <URL: http://rt.freeciv.org/Ticket/Display.html?id=9947 >
> 
> packets.c:148
> 
>        pc->compression.queued_packets = 
> fc_realloc(pc->compression.queued_packets, new_size);
> 
> because a realloc is done every time a packet is sent, this makes 
> sending packets O(n^2).
> 
> For instance if you have Apollo's, at the end of the turn you'll receive 
> 4000 packets of (maybe) 5 bytes on average.  This means 4000 reallocs of 
> average size 10,000.  That's 40 megabytes of memcpying.
> 
> Hopefully you're not playing on a 200x100 map.  Because that means 5x as 
> many packets and 25x as much memcpying.

You would be forgiven for thinking I'm wrong.  So try playing this patch 
and building apollo.  After one turn (size 29, topology 5):

2: 130750 sent, 1175687346 copied.

130 Kb are sent.  1.2 Gb are copied.  A 9000:1 ratio.  With the default 
size 4 it's a bit better:

2: 36896 sent, 109421796 copied.

Only 100 Mb sent here.  Without Apollo it's a bit better:

2: 27845 sent, 8542883 copied.

but still far from the 1:1 or 2:1 ratio that we could easily achieve.

Of course all these are on an empty map.  If the map were covered with 
cities, or if you were playing an empire with lots of cities using CMA, 
these numbers would go _way_ up (CMA requires lots of freezing).

jason

Index: common/packets.c
===================================================================
RCS file: /home/freeciv/CVS/freeciv/common/packets.c,v
retrieving revision 1.269
diff -u -r1.269 packets.c
--- common/packets.c    5 Sep 2004 05:10:50 -0000       1.269
+++ common/packets.c    5 Sep 2004 06:09:47 -0000
@@ -100,6 +100,7 @@
 {
   /* default for the server */
   int result = 0;
+  static size_t memcpying = 0;
 
   freelog(BASIC_PACKET_LOG_LEVEL, "sending packet type=%s(%d) len=%d",
          get_packet_name(data[2]), data[2], len);
@@ -143,6 +144,8 @@
 
     if (pc->compression.frozen_level > 0) {
       size_t new_size = pc->compression.queue_size + size;
+
+      memcpying += new_size;
       pc->compression.queued_packets = 
fc_realloc(pc->compression.queued_packets, new_size);
       memcpy(ADD_TO_POINTER(pc->compression.queued_packets, 
pc->compression.queue_size), data, len);
       pc->compression.queue_size = new_size;
@@ -217,6 +220,9 @@
   send_connection_data(pc, data, len);
 #endif
 
+  freelog(LOG_NORMAL, "%ld sent, %ld copied.",
+         (long)pc->statistics.bytes_send, (long)memcpying);
+
 #if PACKET_SIZE_STATISTICS
   {
     static struct {
Index: data/default/buildings.ruleset
===================================================================
RCS file: /home/freeciv/CVS/freeciv/data/default/buildings.ruleset,v
retrieving revision 1.51
diff -u -r1.51 buildings.ruleset
--- data/default/buildings.ruleset      4 Sep 2004 20:19:51 -0000       1.51
+++ data/default/buildings.ruleset      5 Sep 2004 06:09:48 -0000
@@ -1290,7 +1290,7 @@
 
 [building_apollo_program]
 name           = _("Apollo Program")
-tech_req       = "Space Flight"
+tech_req       = "None"
 bldg_req       = "None"
 graphic        = "b.apollo_program"
 graphic_alt    = "-"
@@ -1301,7 +1301,7 @@
 ;equiv_repl    =
 obsolete_by    = "None"
 is_wonder      = 1
-build_cost     = 600
+build_cost     = 1
 upkeep         = 0
 sabotage       = 0
 effect         =