Re: HFSC not working as expected

[Alan Goodman <notifications@yescomputersolutions.com>]

On 07/07/14 10:54, Michal Soltys wrote:
> tc qdisc add dev ppp0 root handle 1:0 hfsc stab overhead 40 linklayer atm default 14
> tc class add dev ppp0 parent 1:0 classid 1:1 hfsc ls m2 1100000 ul m2 1100000
> tc class add dev ppp0 parent 1:1 classid 1:10 hfsc sc m2 100kbit #syn ack rst
> tc class add dev ppp0 parent 1:1 classid 1:11 hfsc sc m1 500kbit d 20ms m2 300kbit # Time critical
> tc class add dev ppp0 parent 1:1 classid 1:12 hfsc sc m2 200kbit #Interactive
> tc class add dev ppp0 parent 1:1 classid 1:13 hfsc sc m2 100kbit #bulk
> tc class add dev ppp0 parent 1:1 classid 1:14 hfsc sc m1 100kbit d 20ms m2 300kbit # torrent and not-classified junk

Hi,

Please note that the below relates to the updated traffic classes 
detailed within this email...

I have been spending further time testing stuff this morning/afternoon.  
I noticed that when download was busy and I started an upload the 
download speed was fluctuating up and down in a rhythmic pattern.  I 
theorise that this is likely caused by the downloads ACK packets (which 
are small and categorised the same as the download traffic) not getting 
sent.  During a download at line rate with no downstream traffic shaping 
enabled I see around 350kbit/sec flowing 'into' eth1 on the server.  
This will be traffic that is destined for the internet.

I am categorising the traffic as follows:
If a rule is matched we stop processing further rules.

Destination port 80 Packet length 0->512 bytes class 1:12
Destination port 80 Packet length 512+ AND bytes out LESS than 5MB class 
1:12
Destination port 80 Packet length 512+ AND bytes out MORE than 5MB class 
1:13

The bytes out rules look in conntrack to see how much data has flowed in 
that connection since it started.

I have confirmed that packets flowing up relating to this download are 
being categorised by the first rule, they are showing as 52 bytes long 
and have the ACK flag set.

With downstream traffic management disabled and idle upstream I am 
seeing steady line rate transfer on my download... (Output is wget 
--progress=dot:mega on a massive file hosted on a fast server.)

  24576K ........ ........ ........ ........ ........ ........  0% 2.03M 
53m9s
  27648K ........ ........ ........ ........ ........ ........  0% 2.03M 
51m16s
  30720K ........ ........ ........ ........ ........ ........  0% 2.00M 
49m46s
  33792K ........ ........ ........ ........ ........ ........  0% 2.03M 
48m28s
  36864K ........ ........ ........ ........ ........ ........  0% 2.03M 
47m23s

However once I start an sftp upload (which is categorised 1:13) my 
download speed becomes eratic with rhythmic slowdowns:

  86016K ........ ........ ........ ........ ........ ........  2% 2.00M 
40m20s
  89088K ........ ........ ........ ........ ........ ........  2% 2.05M 
40m5s
  92160K ........ ........ ........ ........ ........ ........  2% 2.03M 
39m52s
  95232K ........ ........ ........ ........ ........ ........  2% 2.00M 
39m41s
  98304K ........ ........ ........ ........ ........ ........  2% 1.21M 
40m11s
101376K ........ ........ ........ ........ ........ ........  2% 1.20M 
40m41s
104448K ........ ........ ........ ........ ........ ........  2% 1.39M 
40m55s
107520K ........ ........ ........ ........ ........ ........  2% 1.66M 
40m55s
110592K ........ ........ ........ ........ ........ ........  2% 1.66M 
40m54s
113664K ........ ........ ........ ........ ........ ........  2% 1.58M 
40m57s
116736K ........ ........ ........ ........ ........ ........  2% 1.75M 
40m53s
119808K ........ ........ ........ ........ ........ ........  2% 1.73M 
40m50s
122880K ........ ........ ........ ........ ........ ........  2% 1.86M 
40m42s
125952K ........ ........ ........ ........ ........ ........  2% 1.94M 
40m33s
129024K ........ ........ ........ ........ ........ ........  3% 1.89M 
40m26s
132096K ........ ........ ........ ........ ........ ........  3% 1.71M 
40m24s
135168K ........ ........ ........ ........ ........ ........  3% 1.70M 
40m22s
138240K ........ ........ ........ ........ ........ ........  3% 1.54M 
40m26s
141312K ........ ........ ........ ........ ........ ........  3% 1.17M 
40m47s
144384K ........ ........ ........ ........ ........ ........  3% 1.21M 
41m5s
147456K ........ ........ ........ ........ ........ ........  3% 1.54M 
41m8s
150528K ........ ........ ........ ........ ........ ........  3% 1.80M 
41m2s
153600K ........ ........ ........ ........ ........ ........  3% 1.77M 
40m58s
156672K ........ ........ ........ ........ ........ ........  3% 1.89M 
40m51s
159744K ........ ........ ........ ........ ........ ........  3% 1.80M 
40m46s
162816K ........ ........ ........ ........ ........ ........  3% 1.62M 
40m45s
165888K ........ ........ ........ ........ ........ ........  3% 1.84M 
40m40s
168960K ........ ........ ........ ........ ........ ........  3% 1.92M 
40m32s
172032K ........ ........ ........ ........ ........ ........  4% 1.71M 
40m30s
175104K ........ ........ ........ ........ ........ ........  4% 1.56M 
40m32s
178176K ........ ........ ........ ........ ........ ........  4% 1.70M 
40m29s
181248K ........ ........ ........ ........ ........ ........  4% 1.31M 
40m39s
184320K ........ ........ ........ ........ ........ ........  4% 1.21M 
40m53s
187392K ........ ........ ........ ........ ........ ........  4% 1.26M 
41m4s
190464K ........ ........ ........ ........ ........ ........  4% 1.74M 
41m0s
193536K ........ ........ ........ ........ ........ ........  4% 1.75M 
40m56s
196608K ........ ........ ........ ........ ........ ........  4% 1.87M 
40m50s
199680K ........ ........ ........ ........ ........ ........  4% 1.91M 
40m43s
202752K ........ ........ ........ ........ ........ ........  4% 1.89M 
40m37s
205824K ........ ........ ........ ........ ........ ........  4% 1.92M 
40m30s
208896K ........ ........ ........ ........ ........ ........  4% 1.96M 
40m23s

Here is the my tc structure

#QoS for Upload

tc qdisc del dev ppp0 root

tc qdisc add dev ppp0 stab overhead 42 linklayer atm mtu 1458 root 
handle 1:0 hfsc default 14

tc class add dev ppp0 parent 1:0 classid 1:1 hfsc sc rate 98mbit
tc class add dev ppp0 parent 1:1 classid 1:2 hfsc ls m2 1100kbit ul m2 
1100kbit

tc class add dev ppp0 parent 1:2 classid 1:11 hfsc sc m1 600kbit d 20ms 
m2 400kbit # Time critical
tc class add dev ppp0 parent 1:2 classid 1:12 hfsc sc m2 300kbit 
#Interactive
tc class add dev ppp0 parent 1:2 classid 1:13 hfsc sc m2 75kbit #bulk

tc class add dev ppp0 parent 1:1 classid 1:14 hfsc sc rate 98mbit

tc qdisc add dev ppp0 parent 1:11 handle 11: sfq (have also tried pfifo)
tc qdisc add dev ppp0 parent 1:12 handle 12: sfq limit 2 (have also 
tried "pfifo limit 2" "bfifo limit 2916b")
tc qdisc add dev ppp0 parent 1:13 handle 13: sfq limit 2 (have also 
tried "pfifo limit 2" "bfifo limit 2916b")
tc qdisc add dev ppp0 parent 1:14 handle 14: pfifo

tc filter add dev ppp0 parent 1:0 protocol ip prio 2 handle 11 fw flowid 
1:11
tc filter add dev ppp0 parent 1:0 protocol ip prio 3 handle 12 fw flowid 
1:12
tc filter add dev ppp0 parent 1:0 protocol ip prio 4 handle 13 fw flowid 
1:13

My method for devising the above was:

Total rt traffic: 975kbit
Time critical (voip, ping etc) guaranteed 400kbit
Left over bandwidth shared by ratio 3:1.  EG voip is using 400kbit this 
means there is 700kbit left.  If all classes were saturated this would 
mean interactive got 466kbit and bulk got 233kbit.

My testing has around 64bytes/sec hitting the time critical queue (just 
a ping packet every seconnd). This should mean that an http download 
sending 350kbit of ACKs will be able to send all of its packets because 
it should be allocated 3/4 of the remaining upload (777kbit) with the 
remaining 259kbit used by the bulk upload...

However I see more traffic dropping in my queue 1:12:

class hfsc 1:11 parent 1:2 leaf 11: sc m1 600000bit d 20.0ms m2 400000bit
  Sent 703310 bytes 4124 pkt (dropped 56, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0
  period 3987 work 703310 bytes rtwork 618669 bytes level 0

class hfsc 1:12 parent 1:2 leaf 13: sc m1 0bit d 0us m2 75000bit
  Sent 5553128 bytes 5024 pkt (dropped 452, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0
  period 4954 work 5553128 bytes rtwork 915469 bytes level 0

class hfsc 1:13 parent 1:2 leaf 12: sc m1 0bit d 0us m2 300000bit
  Sent 6719923 bytes 60826 pkt (dropped 1176, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0
  period 33210 work 6719923 bytes rtwork 3253617 bytes level 0

qdisc pfifo 11: parent 1:11 limit 3p
  Sent 736170 bytes 4324 pkt (dropped 56, overlimits 0 requeues 0)

qdisc bfifo 12: parent 1:12 limit 2916b
  Sent 6733385 bytes 60906 pkt (dropped 1176, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0

qdisc bfifo 13: parent 1:13 limit 2916b
  Sent 5554877 bytes 5030 pkt (dropped 452, overlimits 0 requeues 0)
  rate 0bit 0pps backlog 0b 0p requeues 0

I cant understand why queue 12 is dropping packets and not getting 3/4 
of the bandwidth share?  Maybe my STAB is still incorrect and all the 
small packets are throwing out the calculations?

Other information you might need:

I chose limit 2 based upon my desire to have no more than 60ms jitter.  
(1458/110000)*4=0.05301818 or 53ms.

Queue 1:11 is for time critical traffic.  I am categorising small tcp 
connection related packets (see below), VoIP, online games, ping and DNS 
here.
Queue 1:12 is for interactive traffic.  I categorise short ssh packets, 
short dport 80 short dport 443, RDP, email, and web flows less than 5MB 
but large packets here.
Queue 1:13 is for bulk traffic.  I categorise web flows over 5MB and 
everything else else into this queue.

TCP related packets hitting queue 1:11 are categorised by the following 
iptables rule:
iptables -t mangle -A ULQOS -p tcp --syn -m length --length 40:68 -j 
MARK --set-mark 11
iptables -t mangle -A ULQOS -p tcp --tcp-flags ALL SYN,ACK -m length 
--length 40:68 -j MARK --set-mark 11
iptables -t mangle -A ULQOS -p tcp --tcp-flags ALL RST -j MARK --set-mark 11
iptables -t mangle -A ULQOS -p tcp --tcp-flags ALL ACK,RST -j MARK 
--set-mark 11
iptables -t mangle -A ULQOS -p tcp --tcp-flags ALL ACK,FIN -j MARK 
--set-mark 11

Thanks again for your patience and support. Sorry for continuing to be a 
bit of a dunce!

Alan