Re: 2.6.0: something is leaking memory

Re: 2.6.0: something is leaking memory

[Linus Torvalds]

On Sun, 4 Jan 2004, Erik Hensema wrote:
> Linus Torvalds (torvalds@osdl.org) wrote:
> >
> > Can you do /proc/slabinfo too?
> 
> Sure, this is of course my currently running system, 4 days, 9:53
> uptime.
> 
> slabinfo - version: 2.0
> # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
> tcp6_sock          19729  19732   1024    4    1 : tunables   54   27    0 : slabdata   4933   4933      0

You've got 19 _megabytes_ allocated to "tcp6_sock", and they are all 
marked as "active". That's almost certainly the leaking bug.

Everything else looks reasonably normal.

> I do use IPv6. I've got three active tunnels and native IPv6 over
> ethernet.

Yeah, but there is no way you have 19 MB worth of sockets active for three 
tunnels.

David?

		Linus

Re: 2.6.0: something is leaking memory

[David S. Miller]

On Sun, 4 Jan 2004 18:30:21 -0800 (PST)
Linus Torvalds <torvalds@osdl.org> wrote:

> You've got 19 _megabytes_ allocated to "tcp6_sock", and they are all 
> marked as "active". That's almost certainly the leaking bug.
> 
> Everything else looks reasonably normal.
 ...
> David?

Fixed by changeset 1.1496.16.1 which is in 2.6.1-rc1

Re: (usagi-core 16947) Re: 2.6.0: something is leaking memory

[YOSHIFUJI Hideaki / 吉藤英明]

In article <slrnbvh1hd.jl6.erik@dexter.hensema.net> (at Sun, 4 Jan 2004 21:31:26 +0000 (UTC)), Erik Hensema <erik@hensema.net> says:

> > Can you do /proc/slabinfo too?
> 
> Sure, this is of course my currently running system, 4 days, 9:53
> uptime.
> 
> slabinfo - version: 2.0
> # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
:
> tcp6_sock          19729  19732   1024    4    1 : tunables   54   27    0 : slabdata   4933   4933      0
:
> > Clearly the memory leak isn't in the page cache, so the most likely source 
> > is network buffers, and most likely in iptables connection tracking or 
> > similar. If you actually _use_ IPv6, then that is also more likely to have 
> > leaks just due to less testing.
> 
> I do use IPv6. I've got three active tunnels and native IPv6 over
> ethernet.
> 
> I've always had problems with nscd leaking filedescriptors, all
> IPv6 connections to my LDAP server. This started after upgrading
> suse 8.0 to 8.2 (I think the problem is in nss_ldap).
> I'm restarting nscd using a cronjob every night now. Output of
> netstat --inet6 -avpn is below. All sockets in CLOSE_WAIT are
> leaked and will go away after a nscd restart.

How about /proc/slabinfo just after restarting nss_ldap?

> The server isn't very critical, but I do need it. I'm willing to
> try some patches (or do an upgrade to -mm), but nothing to wild.
> 
> netstat --inet6 -avpn
> 
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
> tcp        0      0 :::22                   :::*                    LISTEN      1208/sshd           
> tcp        0      0 :::119                  :::*                    LISTEN      1364/innd           
> tcp        0      0 :::25                   :::*                    LISTEN      1433/sendmail: acce 
> tcp        0      0 :::953                  :::*                    LISTEN      1175/named          
> tcp        0      0 ::1:6010                :::*                    LISTEN      19900/sshd          
> tcp        0      0 ::1:6011                :::*                    LISTEN      20150/sshd          
> tcp        1      0 ::1:50565               2001:888:10a1::1:389    CLOSE_WAIT  26536/nscd          
> tcp        1      0 ::1:50224               2001:888:10a1::1:389    CLOSE_WAIT  26536/nscd          
> tcp        0      0 2001:888:10a1::1:389    ::1:55936               ESTABLISHED 1145/slapd          
> tcp        1      0 ::1:50343               2001:888:10a1::1:389    CLOSE_WAIT  26536/nscd          
> tcp        1      0 ::1:50988               2001:888:10a1::1:389    CLOSE_WAIT  26536/nscd          
:

There're too many sockets in CLOSE_WAIT, but the number is very different from
"tcp6_sock."


And, what is happened when you use ipv4 in your nscd?

-- 
Hideaki YOSHIFUJI @ USAGI Project <yoshfuji@linux-ipv6.org>
GPG FP: 9022 65EB 1ECF 3AD1 0BDF  80D8 4807 F894 E062 0EEA

Re: 2.6.0: something is leaking memory

[Erik Hensema]

On Sun, Jan 04, 2004 at 08:48:34PM -0800, David S. Miller wrote:
> On Sun, 4 Jan 2004 18:30:21 -0800 (PST)
> Linus Torvalds <torvalds@osdl.org> wrote:
> 
> > You've got 19 _megabytes_ allocated to "tcp6_sock", and they are all 
> > marked as "active". That's almost certainly the leaking bug.
> > 
> > Everything else looks reasonably normal.
>  ...
> > David?
> 
> Fixed by changeset 1.1496.16.1 which is in 2.6.1-rc1

Are you sure?

tcp6_sock           1110   1136   1024    4    1 : tunables   54   27    0
: slabdata    284    284      0

dexter:~ # netstat --inet6 | wc -l
     21
dexter:~ # uptime
 19:13:22 up  4:36,  1 user,  load average: 0.02, 0.08, 0.09

This is 2.6.1-rc1.

I can't say anything definitive until after a few days of uptime, but it
sure seems the leak is still there.
-- 
Erik Hensema (erik@hensema.net)

Re: 2.6.0: something is leaking memory

[Erik Hensema]

On Sun, Jan 04, 2004 at 08:48:34PM -0800, David S. Miller wrote:
> On Sun, 4 Jan 2004 18:30:21 -0800 (PST)
> Linus Torvalds <torvalds@osdl.org> wrote:
> 
> > You've got 19 _megabytes_ allocated to "tcp6_sock", and they are all 
> > marked as "active". That's almost certainly the leaking bug.
> > 
> > Everything else looks reasonably normal.
>  ...
> > David?
> 
> Fixed by changeset 1.1496.16.1 which is in 2.6.1-rc1

The leak seems to be in 2.6.1-rc1 too and I can't find any IPv6 related
fixes wrt memory in the long format changelog of 2.6.1-rc1.

David: are you sure it was fixed in rc1?

It doesn't seem to be in -rc2 either.

This is after 26 hours uptime:

tcp6_sock           6246   6248   1024    4    1 : tunables   54   27    0
: slabdata   1562   1562      0


-- 
Erik Hensema (erik@hensema.net)

Re: 2.6.0: something is leaking memory

[David S. Miller]

On Tue, 6 Jan 2004 16:59:34 +0100
Erik Hensema <erik@hensema.net> wrote:

> David: are you sure it was fixed in rc1?
> 
> It doesn't seem to be in -rc2 either.
> 
> This is after 26 hours uptime:
> 
> tcp6_sock           6246   6248   1024    4    1 : tunables   54   27    0
> : slabdata   1562   1562      0

Someone mentioned about a bug in the userland program you're
using that is openning these sockets?  Something about leaving sockets
not closed.

(Arnaldo, we aparently still have a TCP ipv6 socket leak...)

Re: 2.6.0: something is leaking memory

[Arnaldo Carvalho de Melo]

Em Tue, Jan 06, 2004 at 09:59:09AM -0800, David S. Miller escreveu:
> On Tue, 6 Jan 2004 16:59:34 +0100
> Erik Hensema <erik@hensema.net> wrote:
> 
> > David: are you sure it was fixed in rc1?
> > 
> > It doesn't seem to be in -rc2 either.
> > 
> > This is after 26 hours uptime:
> > 
> > tcp6_sock           6246   6248   1024    4    1 : tunables   54   27    0
> > : slabdata   1562   1562      0
> 
> Someone mentioned about a bug in the userland program you're
> using that is openning these sockets?  Something about leaving sockets
> not closed.
> 
> (Arnaldo, we aparently still have a TCP ipv6 socket leak...)

I'll take a look at this.

Re: 2.6.0: something is leaking memory

[Erik Hensema]

On Tue, Jan 06, 2004 at 09:59:09AM -0800, David S. Miller wrote:
> On Tue, 6 Jan 2004 16:59:34 +0100
> Erik Hensema <erik@hensema.net> wrote:
> 
> > David: are you sure it was fixed in rc1?
> > 
> > It doesn't seem to be in -rc2 either.
> > 
> > This is after 26 hours uptime:
> > 
> > tcp6_sock           6246   6248   1024    4    1 : tunables   54   27    0
> > : slabdata   1562   1562      0
> 
> Someone mentioned about a bug in the userland program you're
> using that is openning these sockets?  Something about leaving sockets
> not closed.

That's correct. I suspect that this exposes the leak in the kernel more
promimently than on other systems.

The leak is in nscd, it doesn't properly close sockets to my LDAP server.
This is not a kernel problem I think, because it also leaks in 2.4.x. The
kernelspace leak however is not in 2.4, only in 2.6.

I'm restarting nscd in cron every night, which makes the CLOSE_WAIT sockets
go away. However the kernel resources are not freed it seems.
-- 
Erik Hensema (erik@hensema.net)

Re: (usagi-core 16947) Re: 2.6.0: something is leaking memory

[YOSHIFUJI Hideaki / 吉藤英明]

In article <slrnbvh1hd.jl6.erik@dexter.hensema.net> (at Sun, 4 Jan 2004 21:31:26 +0000 (UTC)), Erik Hensema <erik@hensema.net> says:

> > Can you do /proc/slabinfo too?
> 
> Sure, this is of course my currently running system, 4 days, 9:53
> uptime.
> 
> slabinfo - version: 2.0
> # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <batchcount> <limit> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
:
> tcp6_sock          19729  19732   1024    4    1 : tunables   54   27    0 : slabdata   4933   4933      0
:
> > Clearly the memory leak isn't in the page cache, so the most likely source 
> > is network buffers, and most likely in iptables connection tracking or 
> > similar. If you actually _use_ IPv6, then that is also more likely to have 
> > leaks just due to less testing.
> 
> I do use IPv6. I've got three active tunnels and native IPv6 over
> ethernet.
> 
> I've always had problems with nscd leaking filedescriptors, all
> IPv6 connections to my LDAP server. This started after upgrading
> suse 8.0 to 8.2 (I think the problem is in nss_ldap).
> I'm restarting nscd using a cronjob every night now. Output of
> netstat --inet6 -avpn is below. All sockets in CLOSE_WAIT are
> leaked and will go away after a nscd restart.

How about /proc/slabinfo just after restarting nss_ldap?

> The server isn't very critical, but I do need it. I'm willing to
> try some patches (or do an upgrade to -mm), but nothing to wild.
> 
> netstat --inet6 -avpn
> 
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
> tcp        0      0 :::22                   :::*                    LISTEN      1208/sshd           
> tcp        0      0 :::119                  :::*                    LISTEN      1364/innd           
> tcp        0      0 :::25                   :::*                    LISTEN      1433/sendmail: acce 
> tcp        0      0 :::953                  :::*                    LISTEN      1175/named          
> tcp        0      0 ::1:6010                :::*                    LISTEN      19900/sshd          
> tcp        0      0 ::1:6011                :::*                    LISTEN      20150/sshd          
> tcp        1      0 ::1:50565               2001:888:10a1::1:389    CLOSE_WAIT  26536/nscd          
> tcp        1      0 ::1:50224               2001:888:10a1::1:389    CLOSE_WAIT  26536/nscd          
> tcp        0      0 2001:888:10a1::1:389    ::1:55936               ESTABLISHED 1145/slapd          
> tcp        1      0 ::1:50343               2001:888:10a1::1:389    CLOSE_WAIT  26536/nscd          
> tcp        1      0 ::1:50988               2001:888:10a1::1:389    CLOSE_WAIT  26536/nscd          
:

There're too many sockets in CLOSE_WAIT, but the number is very different from
"tcp6_sock."


And, what is happened when you use ipv4 in your nscd?

-- 
Hideaki YOSHIFUJI @ USAGI Project <yoshfuji@linux-ipv6.org>
GPG FP: 9022 65EB 1ECF 3AD1 0BDF  80D8 4807 F894 E062 0EEA