Regression: "rtnetlink: Compute and store minimum ifinfo dump size" breaks glibc's getifaddrs()

Regression: "rtnetlink: Compute and store minimum ifinfo dump size" breaks glibc's getifaddrs()

[David Vrabel]

Changeset c7ac8679bec9397afe8918f788cbcef88c38da54 (rtnetlink: Compute
and store minimum ifinfo dump size) applied to 3.1 increased the maximum
size of the RTM_GETLINK message response.

glibc's getifaddrs() function uses a page sized (4 KiB) buffer for the
RTM_GETLINK response and returns a failure if the message is truncated.
This buffer is not large enough if there is a network card with many
virtual functions.

What do you recommend to resolve this regression?

David

Re: Regression: "rtnetlink: Compute and store minimum ifinfo dump size" breaks glibc's getifaddrs()

[David Miller]

From: David Vrabel <david.vrabel@citrix.com>
Date: Fri, 27 Jan 2012 12:36:47 +0000

> Changeset c7ac8679bec9397afe8918f788cbcef88c38da54 (rtnetlink: Compute
> and store minimum ifinfo dump size) applied to 3.1 increased the maximum
> size of the RTM_GETLINK message response.
> 
> glibc's getifaddrs() function uses a page sized (4 KiB) buffer for the
> RTM_GETLINK response and returns a failure if the message is truncated.
> This buffer is not large enough if there is a network card with many
> virtual functions.
> 
> What do you recommend to resolve this regression?

Actually, glibc technically uses the CPP define value "PAGE_SIZE" if
available, which is potentially different from the system page size.

Using a statically defined PAGE_SIZE is wrong if the program is
subsequently executed on a system with a different page size.

On sparc, and powerpc I believe, this happens commonly.  A 32-bit
executable will see a PAGE_SIZE value of 4K, but when executed on a
64-bit system the page size is actually 8K or larger.  That's why
__getpagesize() should always be used.

Anyways, if the page sizes are correct we're in a bit of a pickle.

Do you have any idea what the computed value of min_ifinfo_dump_size
is at the time of the failure?

Greg, I think we're kind of screwed.  The defined minimum appropriate
buffer size for a recvmsg() call on a netlink socket is defined as:

	getpagesize() < 8192 ? getpagesize() : 8192

and glibc essentially abides by this by unconditionally using page
size, and therefore if an interface with many virtual interfaces takes
us over this limit, we break basically every properly written piece of
netlink code out there.

iproute2 is funny, it uses a static 16K buffer size for netlink
message reception, which tends to paper over this problem :-/

Stefan, this plays into your work too, we're starting to create
situations which are going to cause very serious problems.  Imagine
an interface with lots of these new macvlan source rules, it would
potentially exceed the limit in the above equation as well.

Thanks.

Re: Regression: "rtnetlink: Compute and store minimum ifinfo dump size" breaks glibc's getifaddrs()

[Štefan Gula]

2012/1/27 David Miller <davem@davemloft.net>:
> From: David Vrabel <david.vrabel@citrix.com>
> Date: Fri, 27 Jan 2012 12:36:47 +0000
>
>> Changeset c7ac8679bec9397afe8918f788cbcef88c38da54 (rtnetlink: Compute
>> and store minimum ifinfo dump size) applied to 3.1 increased the maximum
>> size of the RTM_GETLINK message response.
>>
>> glibc's getifaddrs() function uses a page sized (4 KiB) buffer for the
>> RTM_GETLINK response and returns a failure if the message is truncated.
>> This buffer is not large enough if there is a network card with many
>> virtual functions.
>>
>> What do you recommend to resolve this regression?
>
> Actually, glibc technically uses the CPP define value "PAGE_SIZE" if
> available, which is potentially different from the system page size.
>
> Using a statically defined PAGE_SIZE is wrong if the program is
> subsequently executed on a system with a different page size.
>
> On sparc, and powerpc I believe, this happens commonly.  A 32-bit
> executable will see a PAGE_SIZE value of 4K, but when executed on a
> 64-bit system the page size is actually 8K or larger.  That's why
> __getpagesize() should always be used.
>
> Anyways, if the page sizes are correct we're in a bit of a pickle.
>
> Do you have any idea what the computed value of min_ifinfo_dump_size
> is at the time of the failure?
>
> Greg, I think we're kind of screwed.  The defined minimum appropriate
> buffer size for a recvmsg() call on a netlink socket is defined as:
>
>        getpagesize() < 8192 ? getpagesize() : 8192
>
> and glibc essentially abides by this by unconditionally using page
> size, and therefore if an interface with many virtual interfaces takes
> us over this limit, we break basically every properly written piece of
> netlink code out there.
>
> iproute2 is funny, it uses a static 16K buffer size for netlink
> message reception, which tends to paper over this problem :-/
>
> Stefan, this plays into your work too, we're starting to create
> situations which are going to cause very serious problems.  Imagine
> an interface with lots of these new macvlan source rules, it would
> potentially exceed the limit in the above equation as well.
>
> Thanks.
I can theoretically avoid this situation by using multiple small
rtnetlink messages instead of one huge. I believe that ipset code is
using that approach - just have to figure out proper changes to my
macvlan code. But this doesn't solves the issue you are mentioning, it
only prohibits to happen for macvlan code as one page size will be
more than enough to allocate buffer....

RE: Regression: "rtnetlink: Compute and store minimum ifinfo dump size" breaks glibc's getifaddrs()

[Rose, Gregory V]

> -----Original Message-----
> From: David Miller [mailto:davem@davemloft.net]
> Sent: Friday, January 27, 2012 1:54 PM
> To: david.vrabel@citrix.com
> Cc: netdev@vger.kernel.org; Rose, Gregory V; steweg@gmail.com
> Subject: Re: Regression: "rtnetlink: Compute and store minimum ifinfo dump
> size" breaks glibc's getifaddrs()
> 
> From: David Vrabel <david.vrabel@citrix.com>
> Date: Fri, 27 Jan 2012 12:36:47 +0000
> 
> > Changeset c7ac8679bec9397afe8918f788cbcef88c38da54 (rtnetlink: Compute
> > and store minimum ifinfo dump size) applied to 3.1 increased the maximum
> > size of the RTM_GETLINK message response.
> >
> > glibc's getifaddrs() function uses a page sized (4 KiB) buffer for the
> > RTM_GETLINK response and returns a failure if the message is truncated.
> > This buffer is not large enough if there is a network card with many
> > virtual functions.
> >
> > What do you recommend to resolve this regression?
> 
> Actually, glibc technically uses the CPP define value "PAGE_SIZE" if
> available, which is potentially different from the system page size.
> 
> Using a statically defined PAGE_SIZE is wrong if the program is
> subsequently executed on a system with a different page size.
> 
> On sparc, and powerpc I believe, this happens commonly.  A 32-bit
> executable will see a PAGE_SIZE value of 4K, but when executed on a
> 64-bit system the page size is actually 8K or larger.  That's why
> __getpagesize() should always be used.
> 
> Anyways, if the page sizes are correct we're in a bit of a pickle.
> 
> Do you have any idea what the computed value of min_ifinfo_dump_size
> is at the time of the failure?
> 
> Greg, I think we're kind of screwed.  The defined minimum appropriate
> buffer size for a recvmsg() call on a netlink socket is defined as:
> 
> 	getpagesize() < 8192 ? getpagesize() : 8192
> 
> and glibc essentially abides by this by unconditionally using page
> size, and therefore if an interface with many virtual interfaces takes
> us over this limit, we break basically every properly written piece of
> netlink code out there.

Yep, we're screwed.  And as more features and capabilities get added to virtual functions that we want to control through netlink it gets even worse.

Maybe we should have 'ip link show' just display the number of VFs and then have a new 'ip' tool syntax along the lines of 'ip link show eth(x) vf (n)' where eth(x) is the PF and n is the number of the VF.  Then it could show all relevant information for just that VF.  Scripts could parse the number of VFs from the first call to 'ip link show' and then loop to show the details of each VF.

Just an idea... maybe there are other ones out there but it's just getting ridiculous how much data has to be transferred back and forth during the basic 'ip link show' command when the interface as subordinate VFs now that we're getting to devices with up to 256 VFs.

- Greg

Re: Regression: "rtnetlink: Compute and store minimum ifinfo dump size" breaks glibc's getifaddrs()

[David Miller]

From: "Rose, Gregory V" <gregory.v.rose@intel.com>
Date: Mon, 30 Jan 2012 16:50:58 +0000

> Maybe we should have 'ip link show' just display the number of VFs
> and then have a new 'ip' tool syntax along the lines of 'ip link
> show eth(x) vf (n)' where eth(x) is the PF and n is the number of
> the VF.  Then it could show all relevant information for just that
> VF.  Scripts could parse the number of VFs from the first call to
> 'ip link show' and then loop to show the details of each VF.
> 
> Just an idea... maybe there are other ones out there but it's just
> getting ridiculous how much data has to be transferred back and
> forth during the basic 'ip link show' command when the interface as
> subordinate VFs now that we're getting to devices with up to 256
> VFs.

The fix is almost certainly that we need to require that the application
turn on the publishing of features it is interested in.

The inet_diag sockets do something like this, wherein you set bits in
some flags of the request saying what attributes you want to see.

It seems clear that we must, at this point, turn off VF attributes by
default.

Could you do some work on this?  If you don't have the time I'll look
into it, because we'll need to backport this to -stable too.

Thanks.

RE: Regression: "rtnetlink: Compute and store minimum ifinfo dump size" breaks glibc's getifaddrs()

[Rose, Gregory V]

> -----Original Message-----
> From: David Miller [mailto:davem@davemloft.net]
> Sent: Monday, January 30, 2012 8:58 AM
> To: Rose, Gregory V
> Cc: david.vrabel@citrix.com; netdev@vger.kernel.org; steweg@gmail.com
> Subject: Re: Regression: "rtnetlink: Compute and store minimum ifinfo dump
> size" breaks glibc's getifaddrs()
> 
> From: "Rose, Gregory V" <gregory.v.rose@intel.com>
> Date: Mon, 30 Jan 2012 16:50:58 +0000
> 
> > Maybe we should have 'ip link show' just display the number of VFs
> > and then have a new 'ip' tool syntax along the lines of 'ip link
> > show eth(x) vf (n)' where eth(x) is the PF and n is the number of
> > the VF.  Then it could show all relevant information for just that
> > VF.  Scripts could parse the number of VFs from the first call to
> > 'ip link show' and then loop to show the details of each VF.
> >
> > Just an idea... maybe there are other ones out there but it's just
> > getting ridiculous how much data has to be transferred back and
> > forth during the basic 'ip link show' command when the interface as
> > subordinate VFs now that we're getting to devices with up to 256
> > VFs.
> 
> The fix is almost certainly that we need to require that the application
> turn on the publishing of features it is interested in.
> 
> The inet_diag sockets do something like this, wherein you set bits in
> some flags of the request saying what attributes you want to see.
> 
> It seems clear that we must, at this point, turn off VF attributes by
> default.
> 
> Could you do some work on this?  If you don't have the time I'll look
> into it, because we'll need to backport this to -stable too.

Yeah, I can push out some other things I'm working on to get on this.

- Greg

> 
> Thanks.