ulogd2/NFLOG: nfnetlink_log: error creating log nlmsg

ulogd2/NFLOG: nfnetlink_log: error creating log nlmsg

[Thomas Jacob]

Hello List,

I am trying to get ulogd2 to log NFLOG-collected data
correctly, and am getting lots of "nfnetlink_log: error creating log
nlmsg" and only very few actual log entries.

As far as I can tell this happens when there is not enough skb_tailroom
for the log-message in __build_packet_message, but in
nfulnl_log_packet there also a check of skb_tailroom against a different
size value which is supposed to result in flushing messages if there
is no space.

Bug? Feature? If it's the latter, how can I prevent this from
happening?

   Thanks,
      Thomas

Installation details:

conntrack-tools-0.9.12
iptables-1.4.3.2
libnetfilter_conntrack-0.0.99
libnetfilter_log-0.0.16
libnfnetlink-0.0.41
linux-2.6.27.21 (Greg Kroah-Hartman's "stable" branch)
ulogd2 (fairly recent git pull, <1.5 month)

I am logging using "-m limit 1/s --burst 60 -j NFLOG --threshold 10
--prefix SOMETHING" with maybe 3-4 different rules like this in the main
data path. Log volume therefore shouldn't be that high.

Ulogd2 uses the example NFLOG stack [log1:NFLOG] (i.e.
LOGEMU/PRINTPKT...) and sync=0, the rest are default settings.

Re: ulogd2/NFLOG: nfnetlink_log: error creating log nlmsg

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

Thomas Jacob wrote:
> Hello List,
> 
> I am trying to get ulogd2 to log NFLOG-collected data
> correctly, and am getting lots of "nfnetlink_log: error creating log
> nlmsg" and only very few actual log entries.
> 
> As far as I can tell this happens when there is not enough skb_tailroom
> for the log-message in __build_packet_message, but in
> nfulnl_log_packet there also a check of skb_tailroom against a different
> size value which is supposed to result in flushing messages if there
> is no space.
> 
> Bug? Feature? If it's the latter, how can I prevent this from
> happening?

Does this patch helps? It seems that we're missing to add the size of
the hardware address since 2.6.27.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers

[-- Attachment #2: x --]
[-- Type: text/plain, Size: 632 bytes --]

diff --git a/net/netfilter/nfnetlink_log.c b/net/netfilter/nfnetlink_log.c
index fd326ac..66a6dd5 100644
--- a/net/netfilter/nfnetlink_log.c
+++ b/net/netfilter/nfnetlink_log.c
@@ -581,6 +581,12 @@ nfulnl_log_packet(u_int8_t pf,
 		+ nla_total_size(sizeof(struct nfulnl_msg_packet_hw))
 		+ nla_total_size(sizeof(struct nfulnl_msg_packet_timestamp));
 
+	if (in && skb_mac_header_was_set(skb)) {
+		size +=   nla_total_size(skb->dev->hard_header_len)
+			+ nla_total_size(sizeof(u_int16_t))	/* hwtype */
+			+ nla_total_size(sizeof(u_int16_t));	/* hwlen */
+	}
+
 	spin_lock_bh(&inst->lock);
 
 	if (inst->flags & NFULNL_CFG_F_SEQ)

Re: ulogd2/NFLOG: nfnetlink_log: error creating log nlmsg

[Thomas Jacob]

On Thu, May 21, 2009 at 07:24:58PM +0200, Pablo Neira Ayuso wrote:
> Thomas Jacob wrote:
> > Bug? Feature? If it's the latter, how can I prevent this from
> > happening?
> 
> Does this patch helps? It seems that we're missing to add the size of
> the hardware address since 2.6.27.

Thanks for the quick reply and the patch. Unfortunately I haven't found a way
to reproduce this problem in a lab environment yet, so I can't confirm
whether or not it does anything. Originally the issue
occurred when hot-replacing an older system with one that
runs NFLOG/ulogd2 and the versions listed, thereby generating lots
of invalid packets probably (which should have been logged) as well
as some intentional drops (Packet rate at the time maybe was 30kpps).

Any hints as to what kind of traffic I should be generating to create
a situation that your patch might be a remedy for ?

   Thanks, 
       Thomas

> -- 
> "Los honestos son inadaptados sociales" -- Les Luthiers

> diff --git a/net/netfilter/nfnetlink_log.c b/net/netfilter/nfnetlink_log.c
> index fd326ac..66a6dd5 100644
> --- a/net/netfilter/nfnetlink_log.c
> +++ b/net/netfilter/nfnetlink_log.c
> @@ -581,6 +581,12 @@ nfulnl_log_packet(u_int8_t pf,
>  		+ nla_total_size(sizeof(struct nfulnl_msg_packet_hw))
>  		+ nla_total_size(sizeof(struct nfulnl_msg_packet_timestamp));
>  
> +	if (in && skb_mac_header_was_set(skb)) {
> +		size +=   nla_total_size(skb->dev->hard_header_len)
> +			+ nla_total_size(sizeof(u_int16_t))	/* hwtype */
> +			+ nla_total_size(sizeof(u_int16_t));	/* hwlen */
> +	}
> +
>  	spin_lock_bh(&inst->lock);
>  
>  	if (inst->flags & NFULNL_CFG_F_SEQ)

Re: ulogd2/NFLOG: nfnetlink_log: error creating log nlmsg

[Thomas Jacob]

On Fri, 2009-05-22 at 14:12 +0200, Thomas Jacob wrote:
> On Thu, May 21, 2009 at 07:24:58PM +0200, Pablo Neira Ayuso wrote:
> > Thomas Jacob wrote:
> > > Bug? Feature? If it's the latter, how can I prevent this from
> > > happening?
> > 
> > Does this patch helps? It seems that we're missing to add the size of
> > the hardware address since 2.6.27.
> 
> Thanks for the quick reply and the patch. Unfortunately I haven't found a way
> to reproduce this problem in a lab environment yet, so I can't confirm
> whether or not it does anything. Originally the issue
> occurred when hot-replacing an older system with one that
> runs NFLOG/ulogd2 and the versions listed, thereby generating lots
> of invalid packets probably (which should have been logged) as well
> as some intentional drops (Packet rate at the time maybe was 30kpps).

Just to let you know, I still haven't been able to reproduce the
problem in the lab, but the errors went away in the production
environment after applying your patch, so presumably it did the trick.
Also, the lines in question are clearly missing anyway.

   Thanks again,
      Thomas
           

Re: ulogd2/NFLOG: nfnetlink_log: error creating log nlmsg

[Pablo Neira Ayuso]

Thomas Jacob wrote:
> On Fri, 2009-05-22 at 14:12 +0200, Thomas Jacob wrote:
>> On Thu, May 21, 2009 at 07:24:58PM +0200, Pablo Neira Ayuso wrote:
>>> Thomas Jacob wrote:
>>>> Bug? Feature? If it's the latter, how can I prevent this from
>>>> happening?
>>> Does this patch helps? It seems that we're missing to add the size of
>>> the hardware address since 2.6.27.
>> Thanks for the quick reply and the patch. Unfortunately I haven't found a way
>> to reproduce this problem in a lab environment yet, so I can't confirm
>> whether or not it does anything. Originally the issue
>> occurred when hot-replacing an older system with one that
>> runs NFLOG/ulogd2 and the versions listed, thereby generating lots
>> of invalid packets probably (which should have been logged) as well
>> as some intentional drops (Packet rate at the time maybe was 30kpps).
> 
> Just to let you know, I still haven't been able to reproduce the
> problem in the lab, but the errors went away in the production
> environment after applying your patch, so presumably it did the trick.
> Also, the lines in question are clearly missing anyway.

Thanks for the info. I think that it should be related with the patch.
I'll try make some tests by hot-replacing the daemon in my testbed to
see if I catch anything.

-- 
"Los honestos son inadaptados sociales" -- Les Luthiers