Re: ctnetlink kernel dump while running multiple libnfct clients

Re: ctnetlink kernel dump while running multiple libnfct clients

[Pablo Neira Ayuso]

On 25/03/11 01:21, Sam Roberts wrote:
> Screenshot attached.
>
> At the time I had 3 connections to nfnetlink open
> - a userspace connection tracker

what protocol are you tracking from user-space?

> - a utility of mine that prints conntrack and expect events
> (https://github.com/sam-github/libnet/blob/master/lua/nfct-events)
> - watch -n0.8 -d sudo conntrack -L e
>
> It was after 5 or so loops of the watch that the kernel BUGged out.
>
> kernel is 2.6.38-020638-generic (ubuntu's v2.6.28-natty).
>
> For what it's worth, ctnetlink_exp_dump_expect() seems to assume that
> the expectation being printed has a helper with a name that it can
> call strlen() on:
>
>                 helper = rcu_dereference(help->helper);
>                  if (helper)
>                          NLA_PUT_STRING(skb, CTA_EXPECT_HELP_NAME, helper->name);
>          }

AFAICS, the only way to hit this problem is to have some connection 
tracking helper in the kernel which overlaps your user-space helper, ie. 
someone is attaching a kernel helper to your conntrack.

> The expectation being printed is one I created from userspace, so I
> don't know what helper was found for it, or what it's name would be
> (if any).

Need more info to know what's going on.

Re: ctnetlink kernel dump while running multiple libnfct clients

[Sam Roberts]

On Mon, Mar 28, 2011 at 5:47 AM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> On 25/03/11 01:21, Sam Roberts wrote:
>>
>> Screenshot attached.
>>
>> At the time I had 3 connections to nfnetlink open
>> - a userspace connection tracker
>
> what protocol are you tracking from user-space?

A dummy protocol for purposes of developing this prototype, I call it
"echo port broker".

It listens on port 9999 for control connections. An echo port is
requested by the client, and server opens an ephemeral listen port and
returns the number. The client then reconnects to that ephemeral port,
which acts as an echo server.

> AFAICS, the only way to hit this problem is to have some connection tracking
> helper in the kernel which overlaps your user-space helper, ie. someone is
> attaching a kernel helper to your conntrack.

That's quite surprising, I've no firewall rules attaching anything
else to port 9999. See a dump of my rule setup at end of mail. Note it
assumes localhost client connects to localhost server.

> Need more info to know what's going on.

What info would you like me to provide?

Thanks,
Sam


cmd=<iptables -L -n>
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
cmd=<iptables -t filter -P INPUT DROP>
cmd=<iptables -t filter -A OUTPUT -p tcp --sport 9999 -j QUEUE>
cmd=<iptables -t filter -A INPUT -p tcp -m state --state
RELATED,ESTABLISHED -j ACCEPT>
cmd=<iptables -t filter -A OUTPUT -p tcp -m state --state
NEW,RELATED,ESTABLISHED -j ACCEPT>
cmd=<iptables -t filter -A INPUT -p tcp --dport 9999 -m state --state
NEW -j ACCEPT>
cmd=<iptables -L -n>
Chain INPUT (policy DROP)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state
RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp
dpt:9999 state NEW

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
QUEUE      tcp  --  0.0.0.0/0            0.0.0.0/0           tcp spt:9999
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state
NEW,RELATED,ESTABLISHED

Re: ctnetlink kernel dump while running multiple libnfct clients

[Pablo Neira Ayuso]

On 28/03/11 18:01, Sam Roberts wrote:
> On Mon, Mar 28, 2011 at 5:47 AM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>> On 25/03/11 01:21, Sam Roberts wrote:
>>>
>>> Screenshot attached.
>>>
>>> At the time I had 3 connections to nfnetlink open
>>> - a userspace connection tracker
>>
>> what protocol are you tracking from user-space?
> 
> A dummy protocol for purposes of developing this prototype, I call it
> "echo port broker".
> 
> It listens on port 9999 for control connections. An echo port is
> requested by the client, and server opens an ephemeral listen port and
> returns the number. The client then reconnects to that ephemeral port,
> which acts as an echo server.
> 
>> AFAICS, the only way to hit this problem is to have some connection tracking
>> helper in the kernel which overlaps your user-space helper, ie. someone is
>> attaching a kernel helper to your conntrack.
> 
> That's quite surprising, I've no firewall rules attaching anything
> else to port 9999. See a dump of my rule setup at end of mail. Note it
> assumes localhost client connects to localhost server.

Please, send me the code so I can reproduce the problem here.

Re: ctnetlink kernel dump while running multiple libnfct clients

[Sam Roberts]

On Tue, Mar 29, 2011 at 2:58 PM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> On 29/03/11 20:52, Sam Roberts wrote:
>> Even if that happened, why would that helper have a NULL name?
>
> My guess is that this is not related to the backtrace that you sent, the
> problem must be elsewhere.

> I never used the user-space expectations with permanency.

The version running when the kernel died didn't set permanent: the output shows
no sign of a helper associating with my userspace expectations.

>>> Please, send me the code so I can reproduce the problem here.

For what its worth, my prototype is here:

https://github.com/sam-github/libnet/blob/master/lua/echoconntracker
https://github.com/sam-github/libnet/blob/master/lua/echoserver
https://github.com/sam-github/libnet/blob/master/lua/echoclient

You have to build the whole libnet (and yes, this code has nothing to
do with libnet, and I'll rip it out somewhere else RSN).

>> Its not reproduceable, it seems more like a race condition leaving
>> pointers in an undefined state. Maybe printing of expectations while
>> expectations are being created/destroyed?
>
> Let me check this.

When this happened, I was running the prototype conntracker, and
several instances of conntrack -E (for each table). I typed a
conntrack -L command, and the kernel died after I hit ENTER. Could be
unrelated to that, but seems suspiciously coincidental that the
backtrace shown reflects something I did (it looks related to printing
conntracks). Still, you would know better than me what the kernel does
when things go wrong.

Thanks,
Sam

Re: ctnetlink kernel dump while running multiple libnfct clients

[Pablo Neira Ayuso]

On 30/03/11 00:44, Sam Roberts wrote:
> When this happened, I was running the prototype conntracker, and
> several instances of conntrack -E (for each table). I typed a
> conntrack -L command, and the kernel died after I hit ENTER. Could be
> unrelated to that, but seems suspiciously coincidental that the
> backtrace shown reflects something I did (it looks related to printing
> conntracks). Still, you would know better than me what the kernel does
> when things go wrong.

I think I've got the problematic scenario: the master ct is released
while there are still user-space expectations. In that case, the
expectations still point to the master, once they've been released. Once
the expectations expire, we dereference to the master ct which is not
valid anymore.

I'll send a patch to fix this.

Re: fix for userspace expectations

[Sam Roberts]

On Tue, Apr 12, 2011 at 1:27 AM, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> The main change is the following: You have to use the iptables CT target
> with the option --userspace-helper in the raw table of prerouting.

Patches to the iptables man page would be helpful.

Your example passes all traffic through the CT target.

Is that necessary? Can I match just tcp packets on the ports used by
the protocol who's connections are being tracked? Can the CT target be
used in the filter table?

>> Can you suggest a reliable way to reproduce the crash? I only saw it

Can you suggest a reliable way to reproduce the crash?

>> once, and have no way to reproduce reliably, so no way to assure you
>> the patch fixes the problem, I can only tell you if my code continues
>> to work correctly.
>
> I sent an email to describe the problem to the ML. Every expectation

Perhaps this was private email with some other netfilter devs? I can't
find it in the archives:

http://marc.info/?l=netfilter-devel&w=2&r=1&s=pablo&q=b

Cheers,
Sam