Re: Auditing a broken and basic traffic shaping setup - PRIO

[OmegaPhil <OmegaPhil@startmail.com>]

[-- Attachment #1: Type: text/plain, Size: 7318 bytes --]

On 06/12/14 19:32, OmegaPhil wrote:
> Disclaimer: I don't do this very often so there is probably a retard
> error in here somewhere. I'm not expecting people to do my work for me,
> I'm just after a better understanding of the problem so I can get more
> control of the situation.
> 
> tldr: Custom priomap + iptables TOS isn't sorting packets correctly,
> Wireshark won't even filter on TOS...
> 
> ----
> 
> I'm currently attempting to implement a 4 band prio shaper with fq_codel
> queues on a 100Mbit connection (Debian Testing server):
> 
> ======================================================================
> 
> tc qdisc add dev eth0 root handle 1: htb default 1
> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 12800kibps ceil
> 12800kibps
> tc qdisc add dev eth0 parent 1:1 handle 100: prio bands 4 priomap  1 3 1
> 3 2 3 2 3 0 3 0 3 1 3 1 3
> tc qdisc add dev eth0 parent 100:1 handle 1001: fq_codel
> tc qdisc add dev eth0 parent 100:2 handle 1002: fq_codel
> tc qdisc add dev eth0 parent 100:3 handle 1003: fq_codel
> tc qdisc add dev eth0 parent 100:4 handle 1004: fq_codel
> 
> ======================================================================
> 
> Packets are tagged for the various prio queues via iptables:
> 
> ======================================================================
> 
> # ICMP
> $IPTABLES -t mangle -A POSTROUTING -o eth0 -p icmp -j TOS --set-tos
> Minimize-Delay
> 
> # TCP control packets
> $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags
> FIN,SYN,RST,ACK FIN,ACK -j TOS --set-tos Minimize-Delay
> $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags
> FIN,SYN,RST,ACK SYN,ACK -j TOS --set-tos Minimize-Delay
> $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags
> FIN,SYN,RST,ACK RST,ACK -j TOS --set-tos Minimize-Delay
> $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags
> FIN,SYN,RST,ACK RST -j TOS --set-tos Minimize-Delay
> $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --syn -j TOS --set-tos
> Minimize-Delay
> 
> # TCP ACK packets with no or very little data payload (p2p traffic sets
> all packets to ACK packets otherwise, source of size: http://phix.me/dm/)
> $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags
> FIN,SYN,RST,ACK ACK -m length --length 40:89 -j TOS --set-tos Minimize-Delay
> 
> # Band 2 prioritisation
> # Torrenting
> $IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge
> -j TOS --set-tos Maximize-Throughput
> 
> # Band 3 prioritisation
> #$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user1
> -j TOS --set-tos Minimize-Cost
> #$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user2
> -j TOS --set-tos Minimize-Cost
> 
> ======================================================================
> 
> This is based on an otherwise-successful configuration on a local Ubuntu
> server that admittedly doesn't originate traffic itself, without a
> custom priomap.
> 
> The general idea is:
> 
> Band 0: High priority TCP packets, Minimize Delay,
> Band 1: Normal (nothing targetted here)
> Band 2: Torrenting, Maximize Throughput
> Band 3: Special programs, Minimize Monetary Cost
> 
> When I let the above run, virtually all packets get dumped into band 1,
> whereas by far the bulk of the traffic is torrenting - the shaping code
> is behaving like iptables isn't tagging the packets properly, however
> 'iptables -v -L -t mangle' is showing a lot of packets going through the
> TOS rules.
> 
> I next captured packets and opened up with Wireshark to see what was
> going on (it would be nice if I could just capture from the queues
> directly but I've found no evidence this is possible), however the
> following expressions fail to return anything:
> 
> ip.tos
> ip.tos==8
> ip.tos==0x8
> 
> etc with other values - I then moved to ip.dsfield.dscp, this failed in
> a different way - ip.dsfield.dscp==2 returned packets with
> 'Differentiated Services Field: 0x08', ip.dsfield.dscp==2 returned 0x10
> - why?
> 
> At this point I stopped as I clearly didn't know what I was doing. Any
> pointers?
> 
> Thanks for any help.


This answering my own question for others that want a simple strict
priority hierarchy with a customisable band count:

I've finally managed to get a custom number of bands PRIO queue on my
server working now (no need to maintain a custom kernel, tc etc) - the
key was to drop the broken TOS classification and just the iptables
CLASSIFY target directly (no need to get involved in complicated tc
filter stuff either):

Aim:

Band 0: SSH traffic
Band 1: 'Normal' traffic, anything unclassified including iroffer
Band 2: Torrent traffic
Band 3: Darknet traffic

Setup 4 band PRIO qdisc:

=======================================================================

tc qdisc add dev eth0 parent root handle 1: prio bands 4 priomap 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1

=======================================================================

Handle must be 1+, it doesn't like 0, you end up with a 8000+ number
that naturally breaks any later references in iptables. Note that the
band number in priomap counts from 0, so the bands are 0, 1, 2 and 3 -
the actual qdisc IDs start from 1 (...). Dumping in band 1 (band 2 qdisc
ID) across the board acts as the default classification.

Setup usual fq_codel qdiscs:

=======================================================================

tc qdisc add dev eth0 parent 1:1 handle 101: fq_codel
tc qdisc add dev eth0 parent 1:2 handle 102: fq_codel
tc qdisc add dev eth0 parent 1:3 handle 103: fq_codel
tc qdisc add dev eth0 parent 1:4 handle 104: fq_codel

=======================================================================

The child PRIO qdiscs associated with your bands have been created for
you already and their ID starts from 1.

Now get iptables to do the classification:

SSH (port 22222 here):

=======================================================================

iptables -t mangle -A POSTROUTING -o eth0 -p tcp -s "$PUBLIC_IP" --sport
22222 -j CLASSIFY --set-class 1:1

=======================================================================

Torrenting:

=======================================================================

iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge -j
CLASSIFY --set-class 1:3

=======================================================================

Darknets:

=======================================================================

iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner
debian-tor -j CLASSIFY --set-class 1:4
iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner i2p -j
CLASSIFY --set-class 1:4

=======================================================================

Everything else ends up in 1:2 as mentioned previously due to the
initial priomap.

For a nice realtime view of how packets are flowing through the qdiscs
to prove things are actually doing what you told them to do, use bmon
(https://github.com/tgraf/bmon) - literally the 'bmon' command, then
move the left white cursor thing up and down to select the interface or
qdisc/class you are interested in.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]