ipset restore behavior with newer glibc

ipset restore behavior with newer glibc

[Josh Hunt]

When doing an ipset restore with newer versions of glibc I'm seeing some 
extra syscall overhead that I was not seeing with glibc 2.4. I was 
wondering if anyone has seen such behavior and could help me understand 
what is going on?

Here is a snippet of strace during the restore with glibc 2.4: 
http://pastebin.com/qxkPF7FB and one with glibc 2.7: 
http://pastebin.com/wga9SN0E
I've also seen similar behavior with glibc 2.11.

You'll notice that with the newer version a second netlink socket is 
created and it appears some data is sent and info received back from the 
kernel and that socket is closed. This creating and closing of sockets 
continues until the restore operation completes. This causes the restore 
to grow from taking ~1 second to ~5 seconds. This is not a huge deal, 
but I'm more curious as to why this behavior has changed.

These runs were done with ipset v6.11 and kernel 3.2. It appears to me 
this is a glibc change b/c if I swap out only that component I see this 
change in behavior.

Thanks
Josh

Re: ipset restore behavior with newer glibc

[Jan Engelhardt]

On Wednesday 2012-10-31 17:27, Josh Hunt wrote:

> When doing an ipset restore with newer versions of glibc I'm seeing some extra
> syscall overhead that I was not seeing with glibc 2.4. I was wondering if
> anyone has seen such behavior and could help me understand what is going on?
>
> Here is a snippet of strace during the restore with glibc 2.4:
> http://pastebin.com/qxkPF7FB and one with glibc 2.7:
> http://pastebin.com/wga9SN0E
> I've also seen similar behavior with glibc 2.11.
>
> You'll notice that with the newer version a second netlink socket is created
> and it appears some data is sent and info received back from the kernel

I have observed such as well in other programs as well. Without
looking into this too deeply, I suspect that a program, or a library
on its behalf, is using the interface name<->index resolution
functions if_nametoindex(3) et al, for which netlink is used in
sufficiently new glibc where socket ioctls were (probably) used
previously. Could this be it?

Re: ipset restore behavior with newer glibc

[Jozsef Kadlecsik]

On Wed, 31 Oct 2012, Jan Engelhardt wrote:

> On Wednesday 2012-10-31 17:27, Josh Hunt wrote:
> 
> > When doing an ipset restore with newer versions of glibc I'm seeing 
> > some extra syscall overhead that I was not seeing with glibc 2.4. I 
> > was wondering if anyone has seen such behavior and could help me 
> > understand what is going on?
> >
> > Here is a snippet of strace during the restore with glibc 2.4:
> > http://pastebin.com/qxkPF7FB and one with glibc 2.7:
> > http://pastebin.com/wga9SN0E
> > I've also seen similar behavior with glibc 2.11.
> >
> > You'll notice that with the newer version a second netlink socket is created
> > and it appears some data is sent and info received back from the kernel
> 
> I have observed such as well in other programs as well. Without
> looking into this too deeply, I suspect that a program, or a library
> on its behalf, is using the interface name<->index resolution
> functions if_nametoindex(3) et al, for which netlink is used in
> sufficiently new glibc where socket ioctls were (probably) used
> previously. Could this be it?

ipset does not check interface names (except the length of the string), so 
does not call if_nametoindex.

The extra syscalls come from "getaddrinfo", which is used by ipset to 
parse every IP address. In eglibc 2.11 the implementation of "getaddrinfo" 
contains the comment and the uncoditional call:

  /* We might need information about what interfaces are available.
     Also determine whether we have IPv4 or IPv6 interfaces or both.  We
     cannot cache the results since new interfaces could be added at
     any time.  */
  __check_pf (&seen_ipv4, &seen_ipv6, &in6ai, &in6ailen);

And indeed, __check_pf opens up a netlink socket, makes a request then 
closes it.

I haven't checked the source code of glibc itself but I suppose it works 
the same way.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset restore behavior with newer glibc

[Jozsef Kadlecsik]

On Wed, 31 Oct 2012, Jozsef Kadlecsik wrote:

> On Wed, 31 Oct 2012, Jan Engelhardt wrote:
> 
> > On Wednesday 2012-10-31 17:27, Josh Hunt wrote:
> > 
> > > When doing an ipset restore with newer versions of glibc I'm seeing 
> > > some extra syscall overhead that I was not seeing with glibc 2.4. I 
> > > was wondering if anyone has seen such behavior and could help me 
> > > understand what is going on?
> > >
> > > Here is a snippet of strace during the restore with glibc 2.4:
> > > http://pastebin.com/qxkPF7FB and one with glibc 2.7:
> > > http://pastebin.com/wga9SN0E
> > > I've also seen similar behavior with glibc 2.11.
> > >
> > > You'll notice that with the newer version a second netlink socket is created
> > > and it appears some data is sent and info received back from the kernel
> > 
> > I have observed such as well in other programs as well. Without
> > looking into this too deeply, I suspect that a program, or a library
> > on its behalf, is using the interface name<->index resolution
> > functions if_nametoindex(3) et al, for which netlink is used in
> > sufficiently new glibc where socket ioctls were (probably) used
> > previously. Could this be it?
> 
> ipset does not check interface names (except the length of the string), so 
> does not call if_nametoindex.
> 
> The extra syscalls come from "getaddrinfo", which is used by ipset to 
> parse every IP address. In eglibc 2.11 the implementation of "getaddrinfo" 
> contains the comment and the uncoditional call:
> 
>   /* We might need information about what interfaces are available.
>      Also determine whether we have IPv4 or IPv6 interfaces or both.  We
>      cannot cache the results since new interfaces could be added at
>      any time.  */
>   __check_pf (&seen_ipv4, &seen_ipv6, &in6ai, &in6ailen);
> 
> And indeed, __check_pf opens up a netlink socket, makes a request then 
> closes it.
> 
> I haven't checked the source code of glibc itself but I suppose it works 
> the same way.

It seems to me, the best is to add a stripped down version of 
"getaddrinfo" to ipset itself. That'll be the first library workaround...

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset restore behavior with newer glibc

[Josh Hunt]

On 10/31/2012 02:07 PM, Jozsef Kadlecsik wrote:
>
> It seems to me, the best is to add a stripped down version of
> "getaddrinfo" to ipset itself. That'll be the first library workaround...
>

Is this something that would be an acceptable solution? If so I can try 
my hand at it unless someone else is eager to do it.

Josh

Re: ipset restore behavior with newer glibc

[Jozsef Kadlecsik]

On Tue, 27 Nov 2012, Josh Hunt wrote:

> On 10/31/2012 02:07 PM, Jozsef Kadlecsik wrote:
> > 
> > It seems to me, the best is to add a stripped down version of
> > "getaddrinfo" to ipset itself. That'll be the first library workaround...
> 
> Is this something that would be an acceptable solution? If so I can try my
> hand at it unless someone else is eager to do it.

In ipset 6.15 getaddrinfo was replaced by gethostbyname2 (where it's 
available in libc).

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset restore behavior with newer glibc

[Josh Hunt]

On 11/27/2012 01:51 PM, Jozsef Kadlecsik wrote:
> On Tue, 27 Nov 2012, Josh Hunt wrote:
>
>> On 10/31/2012 02:07 PM, Jozsef Kadlecsik wrote:
>>>
>>> It seems to me, the best is to add a stripped down version of
>>> "getaddrinfo" to ipset itself. That'll be the first library workaround...
>>
>> Is this something that would be an acceptable solution? If so I can try my
>> hand at it unless someone else is eager to do it.
>
> In ipset 6.15 getaddrinfo was replaced by gethostbyname2 (where it's
> available in libc).

Sweet! :D

Re: ipset restore behavior with newer glibc

[Josh Hunt]

On 10/31/2012 01:49 PM, Jozsef Kadlecsik wrote:
> On Wed, 31 Oct 2012, Jan Engelhardt wrote:
>
>> On Wednesday 2012-10-31 17:27, Josh Hunt wrote:
>>
>>> When doing an ipset restore with newer versions of glibc I'm seeing
>>> some extra syscall overhead that I was not seeing with glibc 2.4. I
>>> was wondering if anyone has seen such behavior and could help me
>>> understand what is going on?
>>>
>>> Here is a snippet of strace during the restore with glibc 2.4:
>>> http://pastebin.com/qxkPF7FB and one with glibc 2.7:
>>> http://pastebin.com/wga9SN0E
>>> I've also seen similar behavior with glibc 2.11.
>>>
>>> You'll notice that with the newer version a second netlink socket is created
>>> and it appears some data is sent and info received back from the kernel
>>
>> I have observed such as well in other programs as well. Without
>> looking into this too deeply, I suspect that a program, or a library
>> on its behalf, is using the interface name<->index resolution
>> functions if_nametoindex(3) et al, for which netlink is used in
>> sufficiently new glibc where socket ioctls were (probably) used
>> previously. Could this be it?
>
> ipset does not check interface names (except the length of the string), so
> does not call if_nametoindex.
>
> The extra syscalls come from "getaddrinfo", which is used by ipset to
> parse every IP address. In eglibc 2.11 the implementation of "getaddrinfo"
> contains the comment and the uncoditional call:
>
>    /* We might need information about what interfaces are available.
>       Also determine whether we have IPv4 or IPv6 interfaces or both.  We
>       cannot cache the results since new interfaces could be added at
>       any time.  */
>    __check_pf (&seen_ipv4, &seen_ipv6, &in6ai, &in6ailen);
>
> And indeed, __check_pf opens up a netlink socket, makes a request then
> closes it.
>
> I haven't checked the source code of glibc itself but I suppose it works
> the same way.
>
> Best regards,
> Jozsef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>            H-1525 Budapest 114, POB. 49, Hungary
>
Jozsef

It looks like you're right. glibc doesn't do the socket in __check_pf, 
but does add a socket call inside of getaddrinfo.

Josh