Re: Conntrack Events Performance - Multipart Messages?

[Patrick McHardy <kaber@trash.net>]

[-- Attachment #1: Type: text/plain, Size: 1884 bytes --]

Fabian Hugelshofer wrote:
> Fabian Hugelshofer wrote:
>> Patrick McHardy wrote:
>>> Callgraph information would be useful since its unclear whether
>>> this is the memcpy triggered by netlink message trimming in
>>> af_netlink.c or something different. Unfortunately according
>>> to the documentation this is only supported on x86. I think
>>> selecting the netfilter options as modules should provide
>>> slightly more detail though.
> [...]
>>
>> memcpy is mostly invoked by skb_copy and netlink_broadcast 
>> (af_netlink). netlink_broadcast is expensive on its own and calls 
>> pskb_expand_head which is expensive as well. Using multipart messages 
>> would reduce the need to call netlink_broadcast.
> 
> I profiled again with nfnetlink and nf_conntrack compiled as modules:
>    103599 61.1842 vmlinux
>     24481 14.4582 ath_pci
>     19232 11.3582 nf_conntrack
>     10435  6.1628 wlan
>      3588  2.1190 nf_conntrack_netlink
>      2869  1.6944 oprofiled
>      1886  1.1138 nf_conntrack_ipv4
>      1447  0.8546 ath_rate_minstrel
>       627  0.3703 nfnetlink
>       237  0.1400 ld-uClibc-0.9.29.so
>       233  0.1376 libuClibc-0.9.29.so
>       183  0.1081 iptable_raw
>       174  0.1028 ctevtest
>       147  0.0868 busybox
>        85  0.0502 libnfnetlink.so.0.2.0
>        60  0.0354 libnetfilter_conntrack.so.1.2.0
>        38  0.0224 arp_tables
>         2  0.0012 arptable_filter
> 
> Again most of the time is spent in the kernel. Memory and skb operations 
> are accounted there. I suspect that they cause the most overhead.
> 
> Do you plan to dig deeper into optimising the non-optimal parts? I 
> consider myself not to have enough understanding to do it myself.

The first thing to try would be to use sane allocation sizes
for the event messages. This patch doesn't implement it properly
(uses probing), but should be enough to test whether it helps.





[-- Attachment #2: x --]
[-- Type: text/plain, Size: 991 bytes --]

diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c
index 105a616..0aa1b30 100644
--- a/net/netfilter/nf_conntrack_netlink.c
+++ b/net/netfilter/nf_conntrack_netlink.c
@@ -425,6 +425,7 @@ static int ctnetlink_conntrack_event(struct notifier_block *this,
 	unsigned int type;
 	sk_buff_data_t b;
 	unsigned int flags = 0, group;
+	static unsigned int size = 128;
 
 	/* ignore our fake conntrack entry */
 	if (ct == &nf_conntrack_untracked)
@@ -446,7 +447,8 @@ static int ctnetlink_conntrack_event(struct notifier_block *this,
 	if (!nfnetlink_has_listeners(group))
 		return NOTIFY_DONE;
 
-	skb = alloc_skb(NLMSG_GOODSIZE, GFP_ATOMIC);
+retry:
+	skb = alloc_skb(size, GFP_ATOMIC);
 	if (!skb)
 		return NOTIFY_DONE;
 
@@ -525,7 +527,8 @@ static int ctnetlink_conntrack_event(struct notifier_block *this,
 nlmsg_failure:
 nla_put_failure:
 	kfree_skb(skb);
-	return NOTIFY_DONE;
+	size <<= 1;
+	goto retry;
 }
 #endif /* CONFIG_NF_CONNTRACK_EVENTS */