Linux 2.6.27.5 / SFQ/HTB scheduling problems

Linux 2.6.27.5 / SFQ/HTB scheduling problems

[Sami Farin]

idle:
--- 84.250.192.1 ping statistics ---
50 packets transmitted, 50 received, 0% packet loss, time 1218ms
rtt min/avg/max/mdev = 22.293/24.351/41.672/3.776 ms, pipe 2, ipg/ewma 24.876/24.431 ms

(also similar results with Linux 2.6.25.17 when uploading.)
Bother kernels have e1000-8.0.6.

But with Linux 2.6.27.5, only uploading a 3 MiB file using TCP/IP+HTTP,
I get over one second latencies when pinging etc, whereas with
2.6.25.17 I got about 25 ms.

However, when upstream bandwidth is taken by only DNS packets
(maybe 70-100 bytes each), latencies are just fine?!

This is the simplified init script:

--- 8< ---
SENDRATE="460Kbit"
PATH=/sbin:/usr/bin:/usr/sbin
DEV="eth0"
ATM="overhead 12 linklayer atm"
QDISC="sfq"

tc qdisc del dev $DEV root
tc qdisc del dev $DEV ingress
tc qdisc add dev $DEV root handle 1: htb default 19 r2q 1

# :12 is TCP ACKs, :15 is ICMP
tc class add dev $DEV parent 1: classid 1:1 htb rate $SENDRATE ceil $SENDRATE $ATM
tc class add dev $DEV parent 1:1  classid 1:12 htb rate  60Kbit prio 0 ceil 150Kbit $ATM
tc class add dev $DEV parent 1:1  classid 1:15 htb rate  30Kbit prio 0 ceil 150Kbit $ATM
tc class add dev $DEV parent 1:1  classid 1:19 htb rate 370Kbit prio 3 ceil $SENDRATE $ATM
tc qdisc add dev $DEV parent 1:12 handle 12: $QDISC
tc qdisc add dev $DEV parent 1:15 handle 15: $QDISC
tc qdisc add dev $DEV parent 1:19 handle 19: $QDISC
--- 8< ---

idle:
(why overhead is 0b?  linklayer not shown?  iproute2 is the latest git.)

class htb 1:19 parent 1:1 leaf 19: prio 3 quantum 46250 rate 370000bit ceil 460000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 0 
 Sent 108 bytes 2 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 2 borrowed: 0 giants: 0
 tokens: 31546 ctokens: 25374

class htb 1:1 root rate 460000bit ceil 460000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 7 
 Sent 108 bytes 2 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 25374 ctokens: 25374

class htb 1:12 parent 1:1 leaf 12: prio 0 quantum 7500 rate 60000bit ceil 150000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 0 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 208333 ctokens: 83333

class htb 1:15 parent 1:1 leaf 15: prio 0 quantum 3750 rate 30000bit ceil 150000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 0 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 416666 ctokens: 83333


qdisc htb 1: root r2q 1 default 19 direct_packets_stat 0 ver 3.17
 Sent 465 bytes 7 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc sfq 12: parent 1:12 limit 127p quantum 1486b flows 127/1024 
 Sent 66 bytes 1 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc sfq 15: parent 1:15 limit 127p quantum 1486b flows 127/1024 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc sfq 19: parent 1:19 limit 127p quantum 1486b flows 127/1024 
 Sent 399 bytes 6 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 


--------
uploading:

where did those "class sfq 19:XXX" come from?

--- 84.250.192.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 13656ms
rtt min/avg/max/mdev = 24.705/627.912/1156.690/366.291 ms, pipe 2, ipg/ewma 718.754/739.444 ms


class htb 1:19 parent 1:1 leaf 19: prio 3 quantum 46250 rate 370000bit ceil 460000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 0 
 Sent 4094513 bytes 2993 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 408744bit 37pps backlog 0b 41p requeues 0 
 lended: 214 borrowed: 99 giants: 0
 tokens: -1504233 ctokens: -1235857

class htb 1:1 root rate 460000bit ceil 460000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 7 
 Sent 3991086 bytes 4220 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 429992bit 39pps backlog 0b 0p requeues 0 
 lended: 99 borrowed: 0 giants: 0
 tokens: -1049505 ctokens: -1049505

class htb 1:12 parent 1:1 leaf 12: prio 0 quantum 7500 rate 60000bit ceil 150000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 0 
 Sent 92980 bytes 1356 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 120bit 0pps backlog 0b 0p requeues 0 
 lended: 1356 borrowed: 0 giants: 0
 tokens: 194532 ctokens: 77813

class htb 1:15 parent 1:1 leaf 15: prio 0 quantum 3750 rate 30000bit ceil 150000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 0 
 Sent 4834 bytes 47 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 88bit 0pps backlog 0b 0p requeues 0 
 lended: 47 borrowed: 0 giants: 0
 tokens: 336369 ctokens: 69286

class sfq 19:f parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:11 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 0 

class sfq 19:12 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:c2 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:c9 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot 0 

class sfq 19:ec parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:106 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:19a parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:19d parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:1d3 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:249 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 0 

class sfq 19:281 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:2a1 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:2ce parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot 1486 

class sfq 19:2f6 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 0 

class sfq 19:31a parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot 0 

class sfq 19:324 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 4p requeues 0 
 allot 0 

class sfq 19:34a parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:36a parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:372 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:3ae parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 7p requeues 0 
 allot 4828 

class sfq 19:3ed parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 

class sfq 19:3f5 parent 19: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 0 


qdisc htb 1: root r2q 1 default 19 direct_packets_stat 0 ver 3.17
 Sent 4456655 bytes 4605 pkt (dropped 0, overlimits 1029 requeues 0) 
 rate 0bit 0pps backlog 0b 60p requeues 0 
qdisc sfq 12: parent 1:12 limit 127p quantum 1486b flows 127/1024 
 Sent 92980 bytes 1356 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc sfq 15: parent 1:15 limit 127p quantum 1486b flows 127/1024 
 Sent 4834 bytes 47 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
qdisc sfq 19: parent 1:19 limit 127p quantum 1486b flows 127/1024 
 Sent 4358841 bytes 383 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 137239b 60p requeues 0 

(testmy.net speed test reports 28 KiB/s)

----------

STILL, if I limit bandwidth to 256Kbit (half of the link's capacity):

--- 84.250.192.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 9896ms
rtt min/avg/max/mdev = 22.259/396.306/1095.908/407.372 ms, pipe 3, ipg/ewma 520.890/432.970 ms

(testmy.net reports 16 KiB/s)

-- 
"None are so old as those who have outlived enthusiasm."
 - Henry David Thoreau

Re: Linux 2.6.27.5 / SFQ/HTB scheduling problems

[Sami Farin]

On Tue, Nov 11, 2008 at 17:17:22 +0200, Sami Farin wrote:
> (testmy.net speed test reports 28 KiB/s)

Calculating with tcpdump+tcptrace I get 48.7 KiB/s, that
site calculated speed from 2900 KiB when I uploaded 5282 KiB.

But that's irrelevant, I am more interested in finding
why I'm getting lagged to death.

Hmmh I am not getting 1s delays anymore, but still quite bad...

I noticed TSC sync is not working in 2.6.27.5, worked ok in 2.6.25.
I have Intel DQ965GF mobo.

[    0.000999] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
[    0.093769] CPU1:               Intel(R) Pentium(R) D CPU 2.80GHz stepping 07
[    0.094107] checking TSC synchronization [CPU#0 -> CPU#1]:
[    0.094999] Measured 3983 cycles TSC warp between CPUs, turning off TSC clock.
[    0.094999] Marking TSC unstable due to check_tsc_sync_source failed

486Kbit limit

--- 84.250.192.1 ping statistics ---
50 packets transmitted, 50 received, 0% packet loss, time 3517ms
rtt min/avg/max/mdev = 26.922/82.323/169.725/36.659 ms, pipe 5, ipg/ewma 71.791/80.044 ms

256Kbit limit

--- 84.250.192.1 ping statistics ---
50 packets transmitted, 50 received, 0% packet loss, time 3237ms
rtt min/avg/max/mdev = 22.174/53.266/190.722/37.690 ms, pipe 3, ipg/ewma 66.071/40.314 ms



Then I ruled out my ISP by rebooting to 2.6.25.19:

486Kbit limit

--- 84.250.192.1 ping statistics ---
50 packets transmitted, 50 received, 0% packet loss, time 1912ms
rtt min/avg/max/mdev = 22.270/38.025/64.843/10.604 ms, pipe 2, ipg/ewma 39.025/35.386 ms

State       Recv-Q Send-Q               Local Address:Port                 Peer Address:Port
ESTAB      0      150520        80.223.84.180:65184        74.54.226.166:80     timer:(on,1.564ms,0) uid:518 ino:17634 sk:7dd88d00ffff8100
         ts sackcubic wscale:8,3 rto:1621 rtt:895.125/65 cwnd:49 ssthresh:44 send 621.9Kbps rcv_space:5728



When 2.6.27.5 was sending:

2008-11-11T21:16:13.488287Z IP (tos 0x0, ttl 255, id 10323, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0x96a4), 4123538094:4123545194(7100) ack 1017886112 win 710 <nop,nop,timestamp 418352 372760162>
2008-11-11T21:16:13.620018Z IP (tos 0x0, ttl 255, id 10328, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xf1d0), 4123545194:4123552294(7100) ack 1017886112 win 710 <nop,nop,timestamp 418458 372760270>
2008-11-11T21:16:13.751751Z IP (tos 0x0, ttl 255, id 10333, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xc685), 4123552294:4123559394(7100) ack 1017886112 win 710 <nop,nop,timestamp 418562 372760374>

I noticed it's 7152 bytes a packet!
Is this a new trick in 2.6.27 or 2.6.26?
But:
<3>[ 1583.914947] 0000:00:19.0: eth0: Jumbo Frames not supported.

-- 
"Courage is grace under pressure." - Ernest Hemingway

Re: Linux 2.6.27.5 / SFQ/HTB scheduling problems

[Jarek Poplawski]

On 11-11-2008 22:47, Sami Farin wrote:
...
> When 2.6.27.5 was sending:
> 
> 2008-11-11T21:16:13.488287Z IP (tos 0x0, ttl 255, id 10323, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0x96a4), 4123538094:4123545194(7100) ack 1017886112 win 710 <nop,nop,timestamp 418352 372760162>
> 2008-11-11T21:16:13.620018Z IP (tos 0x0, ttl 255, id 10328, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xf1d0), 4123545194:4123552294(7100) ack 1017886112 win 710 <nop,nop,timestamp 418458 372760270>
> 2008-11-11T21:16:13.751751Z IP (tos 0x0, ttl 255, id 10333, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xc685), 4123552294:4123559394(7100) ack 1017886112 win 710 <nop,nop,timestamp 418562 372760374>
> 
> I noticed it's 7152 bytes a packet!
> Is this a new trick in 2.6.27 or 2.6.26?
> But:
> <3>[ 1583.914947] 0000:00:19.0: eth0: Jumbo Frames not supported.
> 

Could you check for TSO (and maybe turn this off) with ethtool?

Thanks,
Jarek P.

Re: Linux 2.6.27.5 / SFQ/HTB scheduling problems

[Sami Farin]

On Thu, Nov 13, 2008 at 07:15:43 +0000, Jarek Poplawski wrote:
> On 11-11-2008 22:47, Sami Farin wrote:
> ...
> > When 2.6.27.5 was sending:
> > 
> > 2008-11-11T21:16:13.488287Z IP (tos 0x0, ttl 255, id 10323, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0x96a4), 4123538094:4123545194(7100) ack 1017886112 win 710 <nop,nop,timestamp 418352 372760162>
> > 2008-11-11T21:16:13.620018Z IP (tos 0x0, ttl 255, id 10328, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xf1d0), 4123545194:4123552294(7100) ack 1017886112 win 710 <nop,nop,timestamp 418458 372760270>
> > 2008-11-11T21:16:13.751751Z IP (tos 0x0, ttl 255, id 10333, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xc685), 4123552294:4123559394(7100) ack 1017886112 win 710 <nop,nop,timestamp 418562 372760374>
> > 
> > I noticed it's 7152 bytes a packet!
> > Is this a new trick in 2.6.27 or 2.6.26?
> > But:
> > <3>[ 1583.914947] 0000:00:19.0: eth0: Jumbo Frames not supported.
> > 
> 
> Could you check for TSO (and maybe turn this off) with ethtool?

It was off:

# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off

Sometimes I am also sending over 15000 byte packets according to
tcpdump.  What I'd like to know is:
1) How can I get tcpdump to show the actual packets I am sending,
   without needing to buy another computer for sniffing the packets?
2) What does it really mean when 15000 byte packet is sent?
   Can other packets be queued normally "in between" or am I doomed
   to 250 ms latency, if I happen to e.g. ping one microsecond after
   the first of these packets is being sent?
   Or are many packets of the same stream just coalesced into one
   bigger, if there are no other packets from other streams in
   between, so that is looks neater when tcpdump is used?

P.S. google does not add References header field so scoring does
not work very well :(

-- 
Do what you love because life is too short for anything else.

Re: Linux 2.6.27.5 / SFQ/HTB scheduling problems

[Jarek Poplawski]

On 13-11-2008 12:48, Sami Farin wrote:
> On Thu, Nov 13, 2008 at 07:15:43 +0000, Jarek Poplawski wrote:
>> On 11-11-2008 22:47, Sami Farin wrote:
>> ...
>>> When 2.6.27.5 was sending:
>>>
>>> 2008-11-11T21:16:13.488287Z IP (tos 0x0, ttl 255, id 10323, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0x96a4), 4123538094:4123545194(7100) ack 1017886112 win 710 <nop,nop,timestamp 418352 372760162>
>>> 2008-11-11T21:16:13.620018Z IP (tos 0x0, ttl 255, id 10328, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xf1d0), 4123545194:4123552294(7100) ack 1017886112 win 710 <nop,nop,timestamp 418458 372760270>
>>> 2008-11-11T21:16:13.751751Z IP (tos 0x0, ttl 255, id 10333, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xc685), 4123552294:4123559394(7100) ack 1017886112 win 710 <nop,nop,timestamp 418562 372760374>
>>>
>>> I noticed it's 7152 bytes a packet!
>>> Is this a new trick in 2.6.27 or 2.6.26?
>>> But:
>>> <3>[ 1583.914947] 0000:00:19.0: eth0: Jumbo Frames not supported.
>>>
>> Could you check for TSO (and maybe turn this off) with ethtool?
> 
> It was off:
> 
> # ethtool -k eth0
> Offload parameters for eth0:
> rx-checksumming: off
> tx-checksumming: off
> scatter-gather: off
> tcp segmentation offload: off

I guess the later options are off as well? What is mtu btw?

> 
> Sometimes I am also sending over 15000 byte packets according to
> tcpdump.  What I'd like to know is:
> 1) How can I get tcpdump to show the actual packets I am sending,
>    without needing to buy another computer for sniffing the packets?

Since a driver and a hardware are later in the queue I doubt you can
do this with tcpdump easily without some loop or something... Maybe
you should try with driver's debug option? (But I'm not an expert.)

> 2) What does it really mean when 15000 byte packet is sent?
>    Can other packets be queued normally "in between" or am I doomed
>    to 250 ms latency, if I happen to e.g. ping one microsecond after
>    the first of these packets is being sent?
>    Or are many packets of the same stream just coalesced into one
>    bigger, if there are no other packets from other streams in
>    between, so that is looks neater when tcpdump is used?

Packets could be coalesced e.g. by TCP if TSO/GSO is on, but an actual
packet sent to the wire should be not more than mtu (15000 if "jumbo"
frames etc. are supported). Since you have warnings about this it looks
like some of your configs could be different between kernel versions.

Your stats from the first message show some differences: class htb 1:19
and qdisc sfq 19 show very different values. HTB can count gso segments,
then it seems some application could try this TSO or GSO. So to compare
these two kernels you should first find the reason why packets are
prepared differenly, and I doubt it's because of some HTB or SFQ change.

Anyway these TSO/GSO/jumbo_frames are usually bad idea with packet
schedulers (or need more tweaking - e.g. sfq quantum).

Jarek P.

Re: Linux 2.6.27.5 / SFQ/HTB scheduling problems

[Sami Farin]

On Thu, Nov 13, 2008 at 13:59:04 +0000, Jarek Poplawski wrote:
> On 13-11-2008 12:48, Sami Farin wrote:
> > On Thu, Nov 13, 2008 at 07:15:43 +0000, Jarek Poplawski wrote:
> >> On 11-11-2008 22:47, Sami Farin wrote:
> >> ...
> >>> When 2.6.27.5 was sending:
> >>>
> >>> 2008-11-11T21:16:13.488287Z IP (tos 0x0, ttl 255, id 10323, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0x96a4), 4123538094:4123545194(7100) ack 1017886112 win 710 <nop,nop,timestamp 418352 372760162>
> >>> 2008-11-11T21:16:13.620018Z IP (tos 0x0, ttl 255, id 10328, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xf1d0), 4123545194:4123552294(7100) ack 1017886112 win 710 <nop,nop,timestamp 418458 372760270>
> >>> 2008-11-11T21:16:13.751751Z IP (tos 0x0, ttl 255, id 10333, offset 0, flags [none], proto TCP (6), length 7152) 80.223.84.180.40449 > 74.54.226.166.80: ., cksum 0xee52 (incorrect (-> 0xc685), 4123552294:4123559394(7100) ack 1017886112 win 710 <nop,nop,timestamp 418562 372760374>
> >>>
> >>> I noticed it's 7152 bytes a packet!
> >>> Is this a new trick in 2.6.27 or 2.6.26?
> >>> But:
> >>> <3>[ 1583.914947] 0000:00:19.0: eth0: Jumbo Frames not supported.
> >>>
> >> Could you check for TSO (and maybe turn this off) with ethtool?
> > 
> > It was off:
> > 
> > # ethtool -k eth0
> > Offload parameters for eth0:
> > rx-checksumming: off
> > tx-checksumming: off
> > scatter-gather: off
> > tcp segmentation offload: off
> 
> I guess the later options are off as well? What is mtu btw?

What later options?  MTU is 1472.

Oh, I had old ethtool..
These with v 6:

# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: on

Wow.  I turned gso off and now it works just like before.
No packets over size of mtu anymore, either.

State       Recv-Q Send-Q               Local Address:Port                 Peer Address:Port
ESTAB      0      122334        80.223.84.180:57694        74.54.226.166:80     timer:(on,4.475ms,0) uid:518 ino:4546485 sk:2ea3ac80ffff8800
         ts sackcubic wscale:8,3 rto:4499 rtt:2914.12/378.5 cwnd:134 ssthresh:30 send 522.4Kbps rcv_space:5728

--- 84.250.192.1 ping statistics ---
50 packets transmitted, 50 received, 0% packet loss, time 1877ms
rtt min/avg/max/mdev = 22.463/38.354/99.126/13.738 ms, pipe 3, ipg/ewma 38.309/34.963 ms

... 
> Anyway these TSO/GSO/jumbo_frames are usually bad idea with packet
> schedulers (or need more tweaking - e.g. sfq quantum).

Well this GSO on by default was an unexpected surprise for me, for sure.

Maybe a warning on htb (or something) module load
"GSO enabled, everything might not work as expected, try 
'ethtool -K eth0 gso off' to turn GSO off"..? ;)

-- 
"That man is the richest whose pleasures are the cheapest."
 - Henry David Thoreau

Re: Linux 2.6.27.5 / SFQ/HTB scheduling problems

[Jarek Poplawski]

On 13-11-2008 15:29, Sami Farin wrote:
...
> Oh, I had old ethtool..
> These with v 6:
> 
> # ethtool -k eth0
> Offload parameters for eth0:
> rx-checksumming: off
> tx-checksumming: off
> scatter-gather: off
> tcp segmentation offload: off
> udp fragmentation offload: off
> generic segmentation offload: on
> 
> Wow.  I turned gso off and now it works just like before.
> No packets over size of mtu anymore, either.
> 
> State       Recv-Q Send-Q               Local Address:Port                 Peer Address:Port
> ESTAB      0      122334        80.223.84.180:57694        74.54.226.166:80     timer:(on,4.475ms,0) uid:518 ino:4546485 sk:2ea3ac80ffff8800
>          ts sackcubic wscale:8,3 rto:4499 rtt:2914.12/378.5 cwnd:134 ssthresh:30 send 522.4Kbps rcv_space:5728
> 
> --- 84.250.192.1 ping statistics ---
> 50 packets transmitted, 50 received, 0% packet loss, time 1877ms
> rtt min/avg/max/mdev = 22.463/38.354/99.126/13.738 ms, pipe 3, ipg/ewma 38.309/34.963 ms
> 
> ... 
>> Anyway these TSO/GSO/jumbo_frames are usually bad idea with packet
>> schedulers (or need more tweaking - e.g. sfq quantum).
> 
> Well this GSO on by default was an unexpected surprise for me, for sure.
> 
> Maybe a warning on htb (or something) module load
> "GSO enabled, everything might not work as expected, try 
> 'ethtool -K eth0 gso off' to turn GSO off"..? ;)

I agree, such a warning could be useful.

BTW, r2q == 1 in your config could harm latency too; if it's because
of quantum warnings, usually (if there are big rate differences
between classes) it's better to use htb class's "quantum" parameter
(tc class add htb help).

Jarek P.

PS: it seems your mailer setting prevents adding your email as "To:"
while doing "Reply All". Sometimes it's intended, but not the most
popular on this list (e.g. I'd prefer to be "To/Cc-ed", because I'm
not a subscriber at the moment).