[LARTC] Profiling hotspots in my tc filter ruleset

[LARTC] Profiling hotspots in my tc filter ruleset

[Carl-Daniel Hailfinger]

Hi,

after I saw that my machine was having problems to forward
more than 200 Mbit/s, I decided to profile the kernel and
find out the hotspots. This is what I found:

[...]
1028     bridge.ko                __br_forward
1033     bridge.ko                br_nf_forward_finish
1074     bridge.ko                ip_sabotage_in
1119     ebtable_filter.ko        ebt_hook
1177     sky2.ko                  sky2_phy_stats
1304     bridge.ko                setup_pre_routing
1417     bridge.ko                br_fdb_cleanup
1425     sky2.ko                  sky2_put_idx
1766     ipt_physdev.ko           .text
2113     bridge.ko                br_nf_pre_routing_finish
2133     bridge.ko                br_nf_forward_ip
2314     bridge.ko                br_handle_frame_finish
2454     bridge.ko                br_dev_queue_push_xmit
2515     bridge.ko                ip_sabotage_out
2718     sky2.ko                  sky2_rx_add
2858     sch_htb.ko               htb_enqueue
3983     bridge.ko                br_handle_frame
4796     bridge.ko                br_nf_pre_routing
7047     bridge.ko                br_nf_post_routing
8158     sky2.ko                  sky2_xmit_frame
9519     sch_htb.ko               htb_classify
9910     sch_htb.ko               htb_dequeue
9916     ip_tables.ko             ipt_do_table
9944     bridge.ko                br_fdb_update
10094    bridge.ko                __br_fdb_get
14446    sky2.ko                  sky2_intr
15323    sky2.ko                  sky2_tx_complete
17745    ebt_ip.ko                ebt_filter_ip
55535    sky2.ko                  sky2_poll
82377    ebtables.ko              ebt_do_table
84971    cls_u32.ko               u32_classify
125089   af_packet.ko             packet_rcv

I admit that my rulesets are not really optimized and I could
probably fix that by building better rulesets. However, I was
interested in general where to start.

sky2: tuning with ethtool helped quite a bit
ebtables: will get mostly rid of that after reconfiguration
af_packet: that's probably the fault of a few tcpdump instances

This leaves cls_u32 to attack. Nearly all rules I have are added
with the following command:
# tc filter add dev eth0 parent 1:0 protocol ip prio $prio u32 match ip src $ip flowid 1:$flowid

Now my problem is that I have about 30-60 filter rules with
the above characteristics. Unfortunately the IPs I'm filtering
do have nothing in common and they generate only 3% of the
accumulated traffic. That means over 97% of the traffic is
checked against a linear list of filters without ever matching
anything.

I looked at hashed filters, but they seem not to be the right
thing because a hashing filter for a /16 network where only
30 IPs should be matched will probably make performance worse.

Only 30 IPs of the /16 subnet are filtered. Each of them gets
a different filter, but their collective traffic is only 3% of
the total traffic. So if I manage to catch them all with one
filter rule and attach my usual rules only to packets from that
filter, 97% of my traffic will hit only 1 filter instead of 30.

Is there anything like ipset available for tc?


Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc