netlink recvmsg() and MSG_TRUNC

netlink recvmsg() and MSG_TRUNC

[David Miller]

So if you don't give a large enough buffer to
recvmsg() for the netlink response a few things
happen:

1) MSG_TRUNC is set
2) The length returned and the amount of data copied is the
   size given in the recvmsg() call
3) If enough other packets remain in the receive buffer,
   nlk->cb is left at non-NULL for a partial dump.  This
   means that you can't just immediately resubmit the
   original request else you'll get NLMSG_ERROR with error
   set to -EBUSY.  This is what netlink_dump_start() does
   when it sees nlk->cb non-NULL.

Now, the user is basically stuck and there is no real
way to recover from this besides doing something like
openning up a new netlink socket and then doing the recvmsg()
with a larger buffer, wash rinse repeat.

I looked at how some of our standard userspace code handles
this and it's not pretty:

1) iproute2 basically just uses a 16K buffer, signals an error
   when it sees MSG_TRUNC, and that's it, whoopee

2) Thomas's libnl believes that recvmsg() will return the
   true length necessary to receive the whole message, he
   signals on this to double the buffer size and try the
   recvmsg() again.  As mentioned recvmsg() never returns
   a length larger than the given buffer size, so this code
   never triggers, and if it did it would lose entries because
   netlink_recvmsg() drops the SKB even when it signals
   MSG_TRUNC.

The behavior of dropping the SKB matches what UDP does in
the case of MSG_TRUNC.

I guess one thing the user could do when it sees MSG_TRUNC
is keep calling recvmsg() until the receive queue is emptied
of packets, in order to get that pesky nlk->cb cleared to
NULL, then resubmit.

But that's rediculous and complicated.

Any ideas?

Re: netlink recvmsg() and MSG_TRUNC

[James Morris]

On Tue, 6 Mar 2007, David Miller wrote:

> I guess one thing the user could do when it sees MSG_TRUNC
> is keep calling recvmsg() until the receive queue is emptied
> of packets, in order to get that pesky nlk->cb cleared to
> NULL, then resubmit.
> 
> But that's rediculous and complicated.
> 
> Any ideas?

Only slightly less complicated: user calls recvmsg() once with a new flag 
MSG_FLUSH, which causes the queue to be flushed, then resubmits ?


- James
-- 
James Morris
<jmorris@namei.org>

Re: netlink recvmsg() and MSG_TRUNC

[Herbert Xu]

David Miller <davem@davemloft.net> wrote:
> 
> I guess one thing the user could do when it sees MSG_TRUNC
> is keep calling recvmsg() until the receive queue is emptied
> of packets, in order to get that pesky nlk->cb cleared to
> NULL, then resubmit.
> 
> But that's rediculous and complicated.
> 
> Any ideas?

Which netlink family generates (or needs to generate) unbounded
messages to user-space? Or indeed which ones generate messages
greater than 64K (or 4K for that matter)?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: netlink recvmsg() and MSG_TRUNC

[Stephen Hemminger]

On Wed, 07 Mar 2007 10:49:07 +1100
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> David Miller <davem@davemloft.net> wrote:
> > 
> > I guess one thing the user could do when it sees MSG_TRUNC
> > is keep calling recvmsg() until the receive queue is emptied
> > of packets, in order to get that pesky nlk->cb cleared to
> > NULL, then resubmit.
> > 
> > But that's rediculous and complicated.
> > 
> > Any ideas?
> 
> Which netlink family generates (or needs to generate) unbounded
> messages to user-space? Or indeed which ones generate messages
> greater than 64K (or 4K for that matter)?
> 
> Cheers,

I know some commands send big blocks down of configuration information.
One example is netem statistical data, but there are others.

-- 
Stephen Hemminger <shemminger@linux-foundation.org>

Re: netlink recvmsg() and MSG_TRUNC

[David Miller]

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Wed, 07 Mar 2007 10:49:07 +1100

> Which netlink family generates (or needs to generate) unbounded
> messages to user-space? Or indeed which ones generate messages
> greater than 64K (or 4K for that matter)?

Create a lot of intefaces, try to dump them :-)

GLIBC can even hit this via it's ifaddrs.c code.

Re: netlink recvmsg() and MSG_TRUNC

[Herbert Xu]

On Tue, Mar 06, 2007 at 04:02:02PM -0800, David Miller wrote:
>
> Create a lot of intefaces, try to dump them :-)

Dumps should be done using 4K (NLMSG_GOODSIZE) skb's, where is the problem?

> GLIBC can even hit this via it's ifaddrs.c code.

Do you have a simple test case that I can run?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: netlink recvmsg() and MSG_TRUNC

[Herbert Xu]

On Tue, Mar 06, 2007 at 03:57:50PM -0800, Stephen Hemminger wrote:
> 
> I know some commands send big blocks down of configuration information.
> One example is netem statistical data, but there are others.

You mean dumps? Unless someone is coalescing them I don't see a problem
there.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: netlink recvmsg() and MSG_TRUNC

[David Miller]

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Wed, 7 Mar 2007 11:04:19 +1100

> On Tue, Mar 06, 2007 at 04:02:02PM -0800, David Miller wrote:
> >
> > Create a lot of intefaces, try to dump them :-)
> 
> Dumps should be done using 4K (NLMSG_GOODSIZE) skb's, where is the problem?

Actually, more accurately it's using PAGE_SIZE. :)

> > GLIBC can even hit this via it's ifaddrs.c code.
> 
> Do you have a simple test case that I can run?

I see, so the better fix would be to make glibc's
netlink_request() function start with a getpagesize()'d
buffer.

Re: netlink recvmsg() and MSG_TRUNC

[Herbert Xu]

On Tue, Mar 06, 2007 at 04:05:57PM -0800, David Miller wrote:
>
> Actually, more accurately it's using PAGE_SIZE. :)

Aha it's you non-i386 people :)

> I see, so the better fix would be to make glibc's
> netlink_request() function start with a getpagesize()'d
> buffer.

Yes that's a good idea.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt