From mboxrd@z Thu Jan 1 00:00:00 1970 From: OmegaPhil Date: Sun, 23 Aug 2015 19:45:02 +0000 Subject: Re: Auditing a broken and basic traffic shaping setup - PRIO Message-Id: <55DA22BE.90602@startmail.com> MIME-Version: 1 Content-Type: multipart/mixed; boundary="kgAw9jWAqBp26oce4I7m7QcOOHxdkp7PX" List-Id: References: <548359D6.7030505@startmail.com> In-Reply-To: <548359D6.7030505@startmail.com> To: lartc@vger.kernel.org This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --kgAw9jWAqBp26oce4I7m7QcOOHxdkp7PX Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 06/12/14 19:32, OmegaPhil wrote: > Disclaimer: I don't do this very often so there is probably a retard > error in here somewhere. I'm not expecting people to do my work for me,= > I'm just after a better understanding of the problem so I can get more > control of the situation. >=20 > tldr: Custom priomap + iptables TOS isn't sorting packets correctly, > Wireshark won't even filter on TOS... >=20 > ---- >=20 > I'm currently attempting to implement a 4 band prio shaper with fq_code= l > queues on a 100Mbit connection (Debian Testing server): >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > tc qdisc add dev eth0 root handle 1: htb default 1 > tc class add dev eth0 parent 1:0 classid 1:1 htb rate 12800kibps ceil > 12800kibps > tc qdisc add dev eth0 parent 1:1 handle 100: prio bands 4 priomap 1 3 = 1 > 3 2 3 2 3 0 3 0 3 1 3 1 3 > tc qdisc add dev eth0 parent 100:1 handle 1001: fq_codel > tc qdisc add dev eth0 parent 100:2 handle 1002: fq_codel > tc qdisc add dev eth0 parent 100:3 handle 1003: fq_codel > tc qdisc add dev eth0 parent 100:4 handle 1004: fq_codel >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > Packets are tagged for the various prio queues via iptables: >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > # ICMP > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p icmp -j TOS --set-tos > Minimize-Delay >=20 > # TCP control packets > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK FIN,ACK -j TOS --set-tos Minimize-Delay > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK SYN,ACK -j TOS --set-tos Minimize-Delay > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK RST,ACK -j TOS --set-tos Minimize-Delay > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK RST -j TOS --set-tos Minimize-Delay > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --syn -j TOS --set-to= s > Minimize-Delay >=20 > # TCP ACK packets with no or very little data payload (p2p traffic sets= > all packets to ACK packets otherwise, source of size: http://phix.me/dm= /) > $IPTABLES -t mangle -A POSTROUTING -o eth0 -p tcp --tcp-flags > FIN,SYN,RST,ACK ACK -m length --length 40:89 -j TOS --set-tos Minimize-= Delay >=20 > # Band 2 prioritisation > # Torrenting > $IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge > -j TOS --set-tos Maximize-Throughput >=20 > # Band 3 prioritisation > #$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user1 > -j TOS --set-tos Minimize-Cost > #$IPTABLES -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner user2 > -j TOS --set-tos Minimize-Cost >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > This is based on an otherwise-successful configuration on a local Ubunt= u > server that admittedly doesn't originate traffic itself, without a > custom priomap. >=20 > The general idea is: >=20 > Band 0: High priority TCP packets, Minimize Delay, > Band 1: Normal (nothing targetted here) > Band 2: Torrenting, Maximize Throughput > Band 3: Special programs, Minimize Monetary Cost >=20 > When I let the above run, virtually all packets get dumped into band 1,= > whereas by far the bulk of the traffic is torrenting - the shaping code= > is behaving like iptables isn't tagging the packets properly, however > 'iptables -v -L -t mangle' is showing a lot of packets going through th= e > TOS rules. >=20 > I next captured packets and opened up with Wireshark to see what was > going on (it would be nice if I could just capture from the queues > directly but I've found no evidence this is possible), however the > following expressions fail to return anything: >=20 > ip.tos > ip.tos=3D=3D8 > ip.tos=3D=3D0x8 >=20 > etc with other values - I then moved to ip.dsfield.dscp, this failed in= > a different way - ip.dsfield.dscp=3D=3D2 returned packets with > 'Differentiated Services Field: 0x08', ip.dsfield.dscp=3D=3D2 returned = 0x10 > - why? >=20 > At this point I stopped as I clearly didn't know what I was doing. Any > pointers? >=20 > Thanks for any help. This answering my own question for others that want a simple strict priority hierarchy with a customisable band count: I've finally managed to get a custom number of bands PRIO queue on my server working now (no need to maintain a custom kernel, tc etc) - the key was to drop the broken TOS classification and just the iptables CLASSIFY target directly (no need to get involved in complicated tc filter stuff either): Aim: Band 0: SSH traffic Band 1: 'Normal' traffic, anything unclassified including iroffer Band 2: Torrent traffic Band 3: Darknet traffic Setup 4 band PRIO qdisc: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D tc qdisc add dev eth0 parent root handle 1: prio bands 4 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Handle must be 1+, it doesn't like 0, you end up with a 8000+ number that naturally breaks any later references in iptables. Note that the band number in priomap counts from 0, so the bands are 0, 1, 2 and 3 - the actual qdisc IDs start from 1 (...). Dumping in band 1 (band 2 qdisc ID) across the board acts as the default classification. Setup usual fq_codel qdiscs: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D tc qdisc add dev eth0 parent 1:1 handle 101: fq_codel tc qdisc add dev eth0 parent 1:2 handle 102: fq_codel tc qdisc add dev eth0 parent 1:3 handle 103: fq_codel tc qdisc add dev eth0 parent 1:4 handle 104: fq_codel =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The child PRIO qdiscs associated with your bands have been created for you already and their ID starts from 1. Now get iptables to do the classification: SSH (port 22222 here): =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D iptables -t mangle -A POSTROUTING -o eth0 -p tcp -s "$PUBLIC_IP" --sport 22222 -j CLASSIFY --set-class 1:1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Torrenting: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner deluge -j CLASSIFY --set-class 1:3 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Darknets: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner debian-tor -j CLASSIFY --set-class 1:4 iptables -t mangle -A POSTROUTING -o eth0 -m owner --uid-owner i2p -j CLASSIFY --set-class 1:4 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Everything else ends up in 1:2 as mentioned previously due to the initial priomap. For a nice realtime view of how packets are flowing through the qdiscs to prove things are actually doing what you told them to do, use bmon (https://github.com/tgraf/bmon) - literally the 'bmon' command, then move the left white cursor thing up and down to select the interface or qdisc/class you are interested in. --kgAw9jWAqBp26oce4I7m7QcOOHxdkp7PX Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJV2iLCAAoJEBfSPH39wvOPLG4P/i1kkGS+wd/PxCUjZCmfZhIM 3/Rhb5N8RqCpGQi6uJRVGCunS4uLL4/GDpRFtMD4aUqT4g7E7lb2hHwQzMlRbXMA IQrt2KjUvKQLxPN6Ni8owjmFqq4VNFQrAurJ1ftZwE/SpoxozVo0YIqZ+pCwBDVo Y75UQIDCr1+q1eRkC8lwuTZffcT1UxPY/zpZnRhbruu9WJLdMoKK2qL9acX13aCD /Zzx1exbHBeKD0meiTEybQu+W8CGECR4cxgThvy7tLIHnzvu0uxRmx/BA7TPRc0A 3jEvTz9bAFVo7rP0kLJ4cMIlExm9ldOq/uT9OHpq8rxgSDCHB6KAxGsUwFcKn8Xr Sy+QzlQ+BhqM2rsMw8vzd6G7lIqItByIgFRiP7aZR/ijr5eMJqt4M1x82cqBNhCW 4zoooXzCmOArL+SmPE6kKi+gLTrVP38AV/19YWNYub9WenVsTAq97LNsLRCt6qhY UO6QJYps7GoTPdNFpsij0zQr3Zxfpa34IJ5uXRKU+/BQWYguhL8iJk58Kt7zsGly pqK1mGSgrGupl67OJKKkOBQ5uKktf71Rscu24gArs6htWhu/pdoffqbEGu9NBBsv 37o1LhE4/kaxaxPEHkKX2U5Z2GsK/njKh9+YmjijgfwgLBzGI/vf/mCCgDHJd85M r/V9lhe7YD/Aj0ppj3Sc =MGob -----END PGP SIGNATURE----- --kgAw9jWAqBp26oce4I7m7QcOOHxdkp7PX--