Re: [PATCH][RFC] Re: high latency with TCP connections

[Rick Jones <rick.jones2@hp.com>]

Alexey Kuznetsov wrote:
> Hello!
> 
> 
>>transactions to data segments is fubar.  That issue is also why I wonder 
>>about the setting of tcp_abc.
> 
> 
> Yes, switching ABC on/off has visible impact on amount of segments.
> When ABC is off, amount of segments is almost the same as number of
> transactions. When it is on, ~1.5% are merged. But this is invisible
> in numbers of throughput/cpu usage.

Hmm, that would seem to suggest that for "new" the netperf/netserver 
were being fast enough that the code didn't perceive the receipt of 
back-to-back sub-MSS segments? (Is that even possible once -b is fairly 
large?)  Otherwise, with new I would have expected the segment count to 
be meaningfully > than the transaction count?

> 
> That' numbers:
> 
> 1Gig link. The first column is "b". - separates runs of netperf
> in backward direction.
> 
> Run #1. One host is slower.
> 
>         old,abc=0
> 	 new,abc=0
> 	  new,abc=1
> 	   old,abc=1
> 
> 2	23652.00  6.31   21.11  10.665  8.924
> 	 23622.16  6.47   21.01  10.951  8.893
> 	  23625.05  6.21   21.01  10.512  8.891
> 	   23725.12  6.46   20.31  10.898  8.559
> 	-
> 	23594.87  21.90  6.44   9.283   10.912
> 	 23631.52  20.30  6.36   8.592   10.766
> 	  23609.55  21.00  6.26   8.896   10.599
> 	   23633.75  21.10  5.44   8.929   9.206
> 
> 4	36349.11  8.71   31.21  9.584   8.585
> 	 36461.37  8.65   30.81  9.492   8.449
> 	  36723.72  8.22   31.31  8.949   8.526
> 	   35801.24  8.58   30.51  9.589   8.521
> 	-
> 	35127.34  33.80  8.43   9.621   9.605
> 	 36165.50  30.90  8.48   8.545   9.381
> 	  36201.45  31.10  8.31   8.592   9.185
> 	   35269.76  30.00  8.58   8.507   9.732
> 
> 8	41148.23  10.39  42.30  10.101  10.281
> 	 41270.06  11.04  31.31  10.698  7.585
> 	  41181.56  5.66   48.61  5.496   11.803
> 	   40372.37  9.68   56.50  9.591   13.996
> 	-
> 	40392.14  47.00  11.89  11.637  11.775
> 	 40613.80  36.90  9.16   9.086   9.019
> 	  40504.66  53.60  7.73   13.234  7.639
> 	   40388.99  48.70  11.93  12.058  11.814
> 
> 16	67952.27  16.27  43.70  9.576   6.432
> 	 68031.40  10.56  53.70  6.206   7.894
> 	  67777.95  12.81  46.90  7.559   6.920
> 	   67814.41  16.13  46.50  9.517   6.857
> 	-
> 	68031.46  51.30  11.53  7.541   6.781
> 	 68044.57  40.70  8.48   5.982   4.986
> 	  67808.13  39.60  15.86  5.840   9.355
> 	   67818.32  52.90  11.51  7.801   6.791
> 
> 32	90445.09  15.41  99.90  6.817   11.045
> 	 90210.34  16.11  100.00 7.143   11.085
> 	  90221.84  17.31  98.90  7.676   10.962
> 	   90712.78  18.41  99.40  8.120   10.958
> 	-
> 	89155.51  99.90  12.89  11.205  5.782
> 	 90058.54  99.90  16.16  11.093  7.179
> 	  90092.31  98.60  15.41  10.944  6.840
> 	   88688.96  99.00  17.59  11.163  7.933
> 
> 64	89983.76  13.66  100.00 6.071   11.113
> 	 90504.24  17.54  100.00 7.750   11.049
> 	  92043.36  17.44  99.70  7.580   10.832
> 	   90979.29  16.01  99.90  7.038   10.981
> 	-
> 	88615.27  99.90  14.91  11.273  6.729
> 	 89316.13  99.90  17.28  11.185  7.740
> 	  90622.85  99.90  16.81  11.024  7.420
> 	   89084.85  99.90  17.51  11.214  7.861
> 
> Run #2. Slower host is replaced with better one. ABC=0.
> No runs in backward directions.
> 
> 	new
> 	 old
> 
> 2	24009.73  8.80   6.49   3.667   10.806
> 	 24008.43  8.00   6.32   3.334   10.524
> 4	40012.53  18.30  8.79   4.574   8.783
> 	 39999.84  19.40  8.86   4.851   8.857
> 8	60500.29  26.30  12.78  4.348   8.452
> 	 60397.79  26.30  11.73  4.355   7.769
> 16	69619.95  39.80  14.03  5.717   8.063
> 	 70528.72  24.90  14.43  3.531   8.184
> 32	132522.01  53.20  21.28  4.015   6.424
> 	 132602.93  57.70  22.59  4.351   6.813
> 64	145738.83  60.30  25.01  4.138   6.865
> 	 143129.55  73.20  24.19  5.114   6.759
> 128	148184.21  69.70  24.96  4.704   6.739
> 	 148143.47  71.00  25.01  4.793   6.753
> 256	144798.91  69.40  25.01  4.793   6.908
> 	 144086.01  73.00  24.61  5.067   6.832
> 
> Frankly, I do not see any statistically valid correlations.

Does look like it jumps-around quite a bit - for example the run#2 with 
-b 16 had the CPU util all over the map on the netperf side.  That 
wasn't by any chance an SMP system?

>>that "linux" didn't seem to be doing the same thing. Hence my tweaking 
>>when seeing this patch come along...]
> 
> 
> netperf does not catch this. :-)

Nope :(  One of these days.... I need to teach netperf how to extract 
TCP statistics from as many platforms as possible.  Meantime it relies 
as always on the kindness of benchmarkers :) (My appologies to Tennesee 
Williams :)

> Even with this patch linux does not ack each second segment dumbly,
> it waits for some conditions, mostly read() emptying receive queue.

Good.  HP-UX is indeed dumb about this, but I'm assured it will be 
changing.  I forget what Solaris does in this situation - I thought I 
looked a while ago but cannot recall the result.

> To model this it is necessary to insert some gaps between
> bursted segments or to use slow network.
> 
> I have no doubts it is easy to model a situation when we send
> lots of useless ACKs. F.e. inserting 20ms gaps between requests.
> To see effect on thoughput/cpu, we could start enough of connections,
> doing the same thing.

Adding --enable-intervals might work there.  I don't recall how well it 
gets along with --enable-burst though, and you have already made lots of 
runs as it is.

rick