Re: SFQ on HFSC leaf does not seem to work

[Eric Dumazet <eric.dumazet@gmail.com>]

Le vendredi 23 décembre 2011 à 09:38 -0500, John A. Sullivan III a
écrit :

> Thanks very much, Eric.  gso and gso only was enabled but disabling it
> does not seem to have solved the problem when I activate netem:
> 

And your kernel version is ?

> root@testswitch01:~# ./tcplay
> root@testswitch01:~# man ethtool
> root@testswitch01:~# ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: off
> large-receive-offload: off
> ntuple-filters: off
> receive-hashing: off
> root@testswitch01:~# ethtool -K eth1 gso off
> root@testswitch01:~# ethtool -k eth1
> Offload parameters for eth1:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: off
> large-receive-offload: off
> ntuple-filters: off
> receive-hashing: off
> ip ro flush cache
> 
> 64 bytes from 192.168.223.84: icmp_req=16 ttl=64 time=42.6 ms
> 64 bytes from 192.168.223.84: icmp_req=17 ttl=64 time=39.1 ms
> 64 bytes from 192.168.223.84: icmp_req=18 ttl=64 time=45.5 ms
> 64 bytes from 192.168.223.84: icmp_req=19 ttl=64 time=406 ms
> 64 bytes from 192.168.223.84: icmp_req=20 ttl=64 time=919 ms
> 64 bytes from 192.168.223.84: icmp_req=21 ttl=64 time=920 ms
> 64 bytes from 192.168.223.84: icmp_req=22 ttl=64 time=1013 ms
> 64 bytes from 192.168.223.84: icmp_req=23 ttl=64 time=1158 ms
> 64 bytes from 192.168.223.84: icmp_req=24 ttl=64 time=1521 ms
> 64 bytes from 192.168.223.84: icmp_req=25 ttl=64 time=1915 ms
> 64 bytes from 192.168.223.84: icmp_req=26 ttl=64 time=2371 ms
> 64 bytes from 192.168.223.84: icmp_req=27 ttl=64 time=2797 ms
> 64 bytes from 192.168.223.84: icmp_req=28 ttl=64 time=3161 ms
> 64 bytes from 192.168.223.84: icmp_req=29 ttl=64 time=3162 ms
> 64 bytes from 192.168.223.84: icmp_req=30 ttl=64 time=3163 ms
> 
> Just in case something is amiss in my methodology, I have four ssh
> sessions open to the test firewall; ssh is in a separate prioritized
> queue.  In one session I run:
> 	ping 192.168.223.84
> Then, in another, I do:
> 	nc 192.168.223.100 443 >/dev/null - this should go into a non-default,

So you _receive_ trafic ?

Are you aware you dont have SFQ in your ingress setup, only egress  ?

> prioritized queue.
> Pings are OK at this point.
> Then, in a third, I do:
> 	nc 192.168.223.100 80 >/dev/null - this goes into the default queue,

same here ?

> the same as ping, and is when the trouble starts.
> 
> I did alter the queue lengths in a recommendation from Dave Taht.  Here
> is my current script with netem:
> 
> tc qdisc add dev eth1 root handle 1: hfsc default 20
> tc class add dev eth1 parent 1: classid 1:1 hfsc sc rate 1490kbit ul
> rate 1490kbit
> tc class add dev eth1 parent 1:1 classid 1:20 hfsc rt rate 400kbit ls
> rate 200kbit
> tc qdisc add dev eth1 parent 1:20 handle 1201 sfq perturb 60 limit 30
> tc class add dev eth1 parent 1:1 classid 1:10 hfsc rt umax 16kbit dmax
> 50ms rate 200kbit ls rate 1000kbit
> tc qdisc add dev eth1 parent 1:10 handle 1101 sfq perturb 60 limit 30
> tc class add dev eth1 parent 1:1 classid 1:30 hfsc rt umax 1514b dmax
> 20ms rate 20kbit
> tc qdisc add dev eth1 parent 1:30 handle 1301 sfq perturb 60 limit 30
> iptables -t mangle -A POSTROUTING -p 6 --syn --dport 443 -j CONNMARK
> --set-mark 0x10
> iptables -t mangle -A PREROUTING -p 6 --syn --dport 822 -j CONNMARK
> --set-mark 0x11
> iptables -t mangle -A POSTROUTING -o eth1 -p 6 -j CONNMARK
> --restore-mark
> modprobe ifb
> ifconfig ifb0 up
> ifconfig ifb1 up
> tc qdisc add dev ifb0 root handle 1: hfsc default 20
> tc class add dev ifb0 parent 1: classid 1:1 hfsc sc rate 1490kbit ul
> rate 1490kbit
> tc class add dev ifb0 parent 1:1 classid 1:20 hfsc rt rate 400kbit ls
> rate 200kbit
> tc qdisc add dev ifb0 parent 1:20 handle 1201 netem delay 25ms 5ms
> distribution normal loss 0.1% 30%
> tc class add dev ifb0 parent 1:1 classid 1:10 hfsc rt umax 16kbit dmax
> 50ms rate 200kbit ls rate 1000kbit
> tc qdisc add dev ifb0 parent 1:10 handle 1101 netem delay 25ms 5ms
> distribution normal loss 0.1% 30%
> tc class add dev ifb0 parent 1:1 classid 1:30 hfsc rt umax 1514b dmax
> 20ms rate 20kbit
> tc qdisc add dev ifb0 parent 1:30 handle 1301 netem delay 25ms 5ms
> distribution normal loss 0.1% 30%
> tc filter add dev ifb0 parent 1:0 protocol ip prio 1 handle 6: u32
> divisor 1
> tc filter add dev ifb0 parent 1:0 protocol ip prio 1 u32 match ip
> protocol 6 0xff link 6: offset at 0 mask 0x0f00 shift 6 plus 0 eat


> tc filter add dev ifb0 parent 1:0 protocol ip prio 1 u32 ht 6:0 match
> tcp src 443 0x00ff flowid 1:10

why "src 443 0x00ff" ? It should be "src 443 0xffff"

> tc filter add dev ifb0 parent 1:0 protocol ip prio 1 u32 ht 6:0 match
> tcp dst 822 0xff00 flowid 1:30

same here : "dst 822 0xffff"

> tc qdisc add dev ifb1 root handle 2 netem delay 25ms 5ms distribution
> normal loss 0.1% 30%
> tc qdisc add dev eth1 ingress
> tc filter add dev eth1 parent ffff: protocol ip prio 50 u32 match u32 0
> 0 action mirred egress redirect dev ifb0
> tc filter add dev eth1 parent 1:1 protocol ip prio 1 handle 0x11 fw
> flowid 1:30
> tc filter add dev eth1 parent 1:1 protocol ip prio 1 handle 0x10 fw
> flowid 1:10
> tc filter add dev eth1 parent 1:1 protocol ip prio 2 u32 match u32 0 0
> flowid 1:20
> tc filter add dev eth1 parent 1:0 protocol ip prio 1 u32 match u32 0 0
> flowid 1:1 action mirred egress redirect dev ifb1
> ip link set eth1 txqueuelen 100
> ip link set ifb1 txqueuelen 100
> ip link set ifb0 txqueuelen 100
> 
> I'd love to solve this.  Just when I thought I was all finished having
> cracked the multiple filter problem to add netem to hfsc, I hit this.
> Thanks again - John
> 

Add some SFQ to your ingress too...