[LARTC] tbf and prio blocking some flows entirely

[LARTC] tbf and prio blocking some flows entirely

[Brian J. Murrell]

[-- Attachment #1.1: Type: text/plain, Size: 5896 bytes --]

I thought I had this all worked out, but it seems not.  The following tc
configuration:

tc qdisc del dev ppp0 root 2> /dev/null > /dev/null
tc qdisc add dev ppp0 root handle 1: tbf rate 120kbit burst 1200 limit 1
tc qdisc add dev ppp0 parent 1:1 handle 2: prio bands 3
tc qdisc add dev ppp0 parent 2:1 handle 10: sfq perturb 20
tc qdisc add dev ppp0 parent 2:2 handle 20: sfq perturb 20
tc qdisc add dev ppp0 parent 2:3 handle 30: sfq perturb 20
# ICMP so we can see PRIO working
tc filter add dev ppp0 parent 2:0 protocol ip prio 10 u32 match ip protocol 1 0xff flowid 2:1

Is producing the following tc rules:

qdisc tbf 1: rate 120000bit burst 1199b lat 4294.9s
 Sent 2024323 bytes 38831 pkt (dropped 54, overlimits 280 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc prio 2: parent 1:1 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 2024323 bytes 38831 pkt (dropped 0, overlimits 0 requeues 280)
 rate 0bit 0pps backlog 0b 0p requeues 280
qdisc sfq 10: parent 2:1 limit 128p quantum 1452b perturb 20sec
 Sent 123362 bytes 1983 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 20: parent 2:2 limit 128p quantum 1452b perturb 20sec
 Sent 122669 bytes 1007 pkt (dropped 0, overlimits 0 requeues 127)
 rate 0bit 0pps backlog 0b 0p requeues 127
qdisc sfq 30: parent 2:3 limit 128p quantum 1452b perturb 20sec
 Sent 1778292 bytes 35841 pkt (dropped 0, overlimits 0 requeues 153)
 rate 0bit 0pps backlog 0b 0p requeues 153

And I have the following iptables rules:

Chain PREROUTING (policy ACCEPT 1564K packets, 875M bytes)
 pkts bytes target     prot opt in     out     source               destination
1560K  872M tcpre      all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain tcpre (1 references)
 pkts bytes target     prot opt in     out     source               destination
1504K  864M CONNMARK   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore mask 0xff
1280K  726M RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match !0x0/0xff
   86  8988 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           ipp2p v0.7.2 --bit MARK set 0x1
   86  8988 CONNMARK   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match 0x1/0xff CONNMARK save mask 0xff

Chain POSTROUTING (policy ACCEPT 1533K packets, 866M bytes)
 pkts bytes target     prot opt in     out     source               destination
 599K   29M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0            0.0.0.0/0           tcp flags:0x16/0x10 length 0:128 CLASSIFY set 2:1
1529K  864M tcpost     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain tcpost (1 references)
 pkts bytes target     prot opt in     out     source               destination
 212K  111M CLASSIFY   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CLASSIFY set 2:2
 549K   28M CLASSIFY   all  --  *      ppp0    0.0.0.0/0            0.0.0.0/0           MARK match 0x1/0xff CLASSIFY set 2:3
 713K  689M CLASSIFY   all  --  *      eth0    0.0.0.0/0            0.0.0.0/0           MARK match 0x1/0xff CLASSIFY set 2:3
30518 2862K CLASSIFY   all  --  *      *       10.75.22.1           66.1.2.3            CLASSIFY set 2:1
    0     0 CLASSIFY   all  --  *      *       66.1.2.3             66.9.8.7            CLASSIFY set 2:1
    0     0 CLASSIFY   udp  --  *      *       0.0.0.0/0            66.9.8.7            udp dpt:29378 CLASSIFY set 2:1
    0     0 CLASSIFY   udp  --  *      *       10.75.22.1           0.0.0.0/0           udp spt:29378 CLASSIFY set 2:1

But it seems that some outbound flows are being blocked entirely.  I
don't think they are being starved though.  Even if they end up in 2:3,
they should at least be treated fairly.  But I am producing a flow to
66.1.2.3 which does increment the counters in 2:1 but after a few
packets the flow stalls:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0   8240 10.75.22.1:55318            66.1.2.3:922                ESTABLISHED 21028/ssh

And that flow stays stuck with 8240 bytes in the Send-Q.

What have I done wrong?  I am so close to my desired behaviour with the
exception being of course that some flows seem to get stuck.  I should
add that some initial packets in that stuck flow above (port 922) are
processed.  It seems that at some point they just stop getting
processed.  The last packets that went out on that flow were:

11:26:47.691755 IP 66.1.2.3.922 > 66.9.8.7.55318: P 1872:1920(48) ack 2261 win 9968 <nop,nop,timestamp 8735287 43843955>
11:26:47.731489 IP 66.9.8.7.55318 > 66.1.2.3.922: . ack 1920 win 2548 <nop,nop,timestamp 43844116 8735287>
11:26:47.855781 IP 66.1.2.3.922 > 66.9.8.7.55318: P 1920:1968(48) ack 2261 win 9968 <nop,nop,timestamp 8735300 43844116>
11:26:47.856142 IP 66.9.8.7.55318 > 66.1.2.3.922: . ack 1968 win 2548 <nop,nop,timestamp 43844240 8735300>
11:26:47.857866 IP 66.9.8.7.55318 > 66.1.2.3.922: P 2261:2373(112) ack 1968 win 2548 <nop,nop,timestamp 43844242 8735300>
11:26:48.072776 IP 66.1.2.3.922 > 66.9.8.7.55318: P 1968:2016(48) ack 2373 win 9968 <nop,nop,timestamp 8735316 43844242>
11:26:48.321959 IP 66.1.2.3.922 > 66.9.8.7.55318: P 1968:2016(48) ack 2373 win 9968 <nop,nop,timestamp 8735352 43844242>
11:26:48.322384 IP 66.9.8.7.55318 > 66.1.2.3.922: . ack 2016 win 2548 <nop,nop,timestamp 43844707 8735352,nop,nop,sack 1 {1968:2016}>

So maybe for some reason that last ack is not being dequeued?  But
surely it should as classes 2:1 and 2:2 are virtually quiet and class
2:3 while having steady traffic, should at least treat it all fairly,
no?

b.

-- 
My other computer is your Microsoft Windows server.

Brian J. Murrell

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] tbf and prio blocking some flows entirely

[Andreas Klauer]

On Sunday 04 December 2005 17:36, Brian J. Murrell wrote:
> Even if they end up in 2:3, they should at least be treated fairly.

2:3 will not be treated at all as long as 2:1 and 2:2 (which have higher 
priority) are occupied. If the queues in 2:1 and 2:2 resp. never empty, 
the packets in 2:3 will never be sent. There is no fair treatment in PRIO. 
That's the whole purpose of this scheduler, to give one band of packets 
absolute priority over the other.

> What have I done wrong?

This is just out of personal interest, but could you try using instead of 
your TBF qdisc, a very simple HTB Qdisc / class with the same bandwidth 
limitation? If that solves the problem, then you're suffering from a 
problem that I failed to solve when I last tried to use TBF; for some 
reason it got stuck on me too.

> So maybe for some reason that last ack is not being dequeued?

I don't understand what you mean by fair treatment, but do try putting all 
ACKs into high priority band, then it will have to be dequeued.

Regards,
Andreas Klauer
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] tbf and prio blocking some flows entirely

[Brian J. Murrell]

[-- Attachment #1.1: Type: text/plain, Size: 3111 bytes --]

On Sun, 2005-12-04 at 17:54 +0100, Andreas Klauer wrote:
> On Sunday 04 December 2005 17:36, Brian J. Murrell wrote:
> > Even if they end up in 2:3, they should at least be treated fairly.
> 
> 2:3 will not be treated at all as long as 2:1 and 2:2 (which have higher 
> priority) are occupied.

Right.  I think I said in my last message that both 2:1 and 2:2 were
very quiet (which is what I would expect) and that there was a lot of
traffic going through 2:3.  My point there was that even if 2:3 was
processing a lot of packets, since it is an SFQ, everybody should
eventually get use of it.

>  If the queues in 2:1 and 2:2 resp. never empty, 
> the packets in 2:3 will never be sent.

Understood.  But they were empty and lots of packets from 2:3 were being
sent.

> There is no fair treatment in PRIO. 

No, it's priority based.  Got that.  Exactly what I am looking for in
fact.

> That's the whole purpose of this scheduler, to give one band of packets 
> absolute priority over the other.

Yup.  My "interactive"/latency sensitive traffic should always get best
service.

> This is just out of personal interest, but could you try using instead of 
> your TBF qdisc, a very simple HTB Qdisc / class with the same bandwidth 
> limitation?

Hrm.  OK.  But I'm not sure how I would use HTB to replace a classless
TBF though.  for TBF I use:

tc qdisc add dev ppp0 root handle 1: tbf rate 120kbit burst 1200 limit 1

So to replace that with HTB I tried:

tc qdisc add dev ppp0 root handle 1: htb default 10
tc class add dev ppp0 parent 1 classid 1:1 htb rate 120kbit

But nothing seems to be getting put into any of the child classes which
are configured as:

tc qdisc add dev ppp0 parent 1:1 handle 2: prio bands 3
tc qdisc add dev ppp0 parent 2:1 handle 10: sfq perturb 20
tc qdisc add dev ppp0 parent 2:2 handle 20: sfq perturb 20
tc qdisc add dev ppp0 parent 2:3 handle 30: sfq perturb 20

I'm probably misunderstanding all of the class naming and handles and
such.

> If that solves the problem, then you're suffering from a 
> problem that I failed to solve when I last tried to use TBF; for some 
> reason it got stuck on me too.

Well, TBF does not seem to be getting stuck.  There is still lots of
traffic moving when these other flows seem to just stop, so TBF can't be
the problem can it?  It has to be PRIO, not dequeuing anything for these
particular stalled flows to TBF right?

> I don't understand what you mean by fair treatment, but do try putting all 
> ACKs into high priority band, then it will have to be dequeued.

I think I am doing that.  I thought that is what:

 859K   42M CLASSIFY   tcp  --  *      ppp0    0.0.0.0/0            0.0.0.0/0           tcp flags:0x16/0x10 length 0:128 CLASSIFY set 2:1

in the POSTROUTING chain should be doing.  It's the result of the
iptables rule:

iptables -t mangle -I POSTROUTING -o ppp0 -p tcp -m tcp --tcp-flags SYN,RST,ACK ACK -m length --length :128 -j CLASSIFY --set-class 2:1

b.

-- 
My other computer is your Microsoft Windows server.

Brian J. Murrell

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] tbf and prio blocking some flows entirely

[Patrick McHardy]

Brian J. Murrell wrote:
> I thought I had this all worked out, but it seems not.  The following tc
> configuration:
> 
> tc qdisc del dev ppp0 root 2> /dev/null > /dev/null
> tc qdisc add dev ppp0 root handle 1: tbf rate 120kbit burst 1200 limit 1

> But it seems that some outbound flows are being blocked entirely.  I
> don't think they are being starved though.  Even if they end up in 2:3,
> they should at least be treated fairly.  But I am producing a flow to
> 66.1.2.3 which does increment the counters in 2:1 but after a few
> packets the flow stalls:

Your burst is too small. It needs to be at least one MTU.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] tbf and prio blocking some flows entirely

[Andreas Klauer]

On Sunday 04 December 2005 19:17, Brian J. Murrell wrote:
> > There is no fair treatment in PRIO.
>
> No, it's priority based.  Got that.  Exactly what I am looking for in
> fact.

Sorry, seems that I misunderstood you in your message in the point that you 
meant SFQ and not PRIO when you were talking about fair treatment.

> So to replace that with HTB I tried:
>
> tc qdisc add dev ppp0 root handle 1: htb default 10
> tc class add dev ppp0 parent 1 classid 1:1 htb rate 120kbit

Without additional filter rules, it should be 'default 1', and not 10. 
Otherwise HTB will try to put the classes in 1:10, which does not exist, 
and instead send them out directly without any shaping at all.

> Well, TBF does not seem to be getting stuck.  There is still lots of
> traffic moving when these other flows seem to just stop, so TBF can't be
> the problem can it?  It has to be PRIO, not dequeuing anything for these
> particular stalled flows to TBF right?

Hmmm. Well, I was just guessing, because I had this 'stuck' problem with 
TBF before. As I said, I never really solved that problem, so I can't say 
much about cause and such.

The way I understand it, the root qdisc will get the request to dequeue a 
packet, and it will forward this request to the underlying qdiscs. So when 
TBF is asked for a packet, it will decide wether to send one or not 
(depending on wether there is any bandwidth left). If it wants to dequeue 
one, it will forward this request to the underlying queue(s), PRIO in your 
case. PRIO will look at it's bands and dequeue a packet from the first 
band that has packets queued; in your case, it will have to ask the 
underlying SFQ queue to select a packet. SFQ will look at the packets it 
has queued and select one based on it's "stochastical fairness" algorithm.
SFQ then returns this packet to PRIO which in turn will return this packet 
to TBF which in turn will send this packet on it's way.

Please note that this is all guesswork. I haven't actually looked at the 
kernel code for all of that. :-) It might actually work in a totally 
different way.

Anyway, if my understanding model is correct, and some packets in your prio 
bands get sent and others don't, it should be the fault of SFQ, and not 
PRIO, for selecting the wrong packets. My guess before was that TBF was at 
fault, allowing too little bandwidth, which would lead to a general 
stalling, which would be most noticeable on connections that are bandwidth 
intensive.

A completely different reason may be that you've got a bad mixing of flows; 
for example, "important" traffic like web browsing etc., and P2P don't go 
well together in the same queue. This is simply because P2P has the habit 
of opening hundreds of connection, whereas WWW is just one or at least 
very few connections.

So if you've got maybe 5 WWW connections and 200 P2P connections flowing 
through the same SFQ queue, every connection will be able to send about 
the same number of packets, resulting in a lot of P2P packets and very few 
WWW packets, just because there are so many more P2P connections there.

For that very reason, in my P2P setups I'm actually using 4 prio bands, 
putting P2P alone into the 4th band, so that it may starve when there is 
any traffic other than P2P.

> I think I am doing that.  I thought that is what:
>
> iptables -t mangle -I POSTROUTING -o ppp0 -p tcp -m tcp --tcp-flags
> SYN,RST,ACK ACK -m length --length :128 -j CLASSIFY --set-class 2:1

I apologize; I'm guilty of not reading your messages carefully enough.

Regards,
Andreas Klauer
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] tbf and prio blocking some flows entirely

[Andy Furniss]

Patrick McHardy wrote:
> Brian J. Murrell wrote:
> 
>> I thought I had this all worked out, but it seems not.  The following tc
>> configuration:
>>
>> tc qdisc del dev ppp0 root 2> /dev/null > /dev/null
>> tc qdisc add dev ppp0 root handle 1: tbf rate 120kbit burst 1200 limit 1
> 
> 
>> But it seems that some outbound flows are being blocked entirely.  I
>> don't think they are being starved though.  Even if they end up in 2:3,
>> they should at least be treated fairly.  But I am producing a flow to
>> 66.1.2.3 which does increment the counters in 2:1 but after a few
>> packets the flow stalls:
> 
> 
> Your burst is too small. It needs to be at least one MTU.

Patrick do you know why prio is requeueing here?

I see the same testing with (a fixed) Brian's setup. If it's to get 
length then it's going to be a bit out sometimes with sfq attached - it 
will also messup sfq fairness whatever the reason.

Brian - limit is in bytes though you get away with it here by adding 
prio/sfq, limit 1 would stop traffic if tbf was alone. Also if you ever 
try tbf on ethX then burst/mtu/limit need to be your mtu + 14.

SFQ causes packet reordering when it perturbs and is best for bulk 
traffic. I would put a shortish bfifo on interactive class (though in 
practice you've kindof lost the battle if your interactive has queued, 
so its sfq should be empty when a packet arrives anyway).

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] tbf and prio blocking some flows entirely

[Brian J. Murrell]

[-- Attachment #1.1: Type: text/plain, Size: 364 bytes --]

On Sun, 2005-12-04 at 19:46 +0100, Patrick McHardy wrote:
> 
> Your burst is too small. It needs to be at least one MTU.

Bingo!  I guess it's obvious that I don't really understand how burst
works.  :-)  But setting it to 1600 seems to have solved the problem.

Thanx!

b.

-- 
My other computer is your Microsoft Windows server.

Brian J. Murrell

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc