[BUG] Linux 2.6.25.4 task_struct leak

[BUG] Linux 2.6.25.4 task_struct leak

[Thorsten Knabe]

[-- Attachment #1: Type: text/plain, Size: 6223 bytes --]

[1.] One line summary of the problem:
Linux 2.6.25.4 x86_64 task_struct leak on host when running UML
2.6.23.16 compiled for i386

[2.] Full description of the problem/report:
I'm seeing a massive task_struct leak on vanilla Linux 2.6.25.4 x86_64
(64-bit) running User Mode Linux 2.6.23.16 kernels compiled for i386
(32-bit). It seems a task_struct leaks on the HOST for every process
that has been created and destroyed by an UML guest. The task_structs
remain allocated on the host even after the UML guests have been shut
down completely.
Other 32-bit applications do NOT leak task_structs.
Also there is no task_struct leak when running the same 32-bit UML
guests on a Linux 2.6.25.4 i386 (32-bit) host kernel.

[3.] Keywords (i.e., modules, networking, kernel):
task_struct leak, UML

[4.] Kernel information
[4.1.] Kernel version (from /proc/version):
Linux version 2.6.25.4 (tek@tek01) (gcc version 3.3.6 (Debian
1:3.3.6-15)) #2 SMP PREEMPT Sat May 24 21:46:37 CEST 2008

[4.2.] Kernel .config file:
Config attached!

[5.] Most recent kernel version which did not have the bug:
2.6.23.17 does not leak task_structs.
2.6.24 - 2.6.25.3 has not been tested
Running 2.6.23.16 i386 UML on a 2.6.25.4 i386 host does NOT leak
task_structs.
Running 64-bit UML guests has not been tested.

[6.] Output of Oops.. message (if applicable) with symbolic information
     resolved (see Documentation/oops-tracing.txt)
No Ooops!

[7.] A small shell script or example program which triggers the
     problem (if possible)
Start 32-bit UML on 64-bit host, then shut it down again. Compare
/proc/slabinfo with number of running tasks/threads.
Before running UML guests, number of active task_structs in
/proc/slabinfo matches number of running tasks/threads on host.
After running and shutting down UML guest differ. I had >500000 active
task_structs in /proc/slabinfo with less than 500 tasks/threads running
on the host.

[8.] Environment
[8.1.] Software (add the output of the ver_linux script here)
Linux tek01 2.6.25.4 #2 SMP PREEMPT Sat May 24 21:46:37 CEST 2008 x86_64
GNU/Linux

Gnu C                  4.1.2
Gnu make               3.81
binutils               2.17
util-linux             2.12r
mount                  2.12r
module-init-tools      3.3-pre2
e2fsprogs              1.40-WIP
reiserfsprogs          3.6.19
xfsprogs               2.8.11
Linux C Library        2.3.6
Dynamic linker (ldd)   2.3.6
Procps                 3.2.7
Net-tools              1.60
Console-tools          0.2.3
oprofile               0.9.2
Sh-utils               5.97
udev                   105
Modules Loaded         nvidia rfcomm l2cap bluetooth nfs lockd nfs_acl
sunrpc af_packet ppdev lp nf_conntrack_ipv6 ip6t_REJECT ip6t_LOG
ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_state ipt_REJECT
ipt_LOG iptable_raw iptable_mangle iptable_nat iptable_filter dm_crypt
crypto_blkcipher dm_snapshot dm_mirror dm_mod deadline_iosched fuse
cryptoloop loop tun zaphfc zaptel crc_ccitt powernow_k8 freq_table
nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack_ipv4 nf_conntrack
ip_tables x_tables w83627hf hwmon_vid eeprom i2c_viapro sg sr_mod sbp2
cx88_blackbird cx2341x usbhid snd_via82xx wm8775 gameport
snd_mpu401_uart tuner tea5767 tda8290 tda18271 tda827x tuner_xc2028
xc5000 snd_via82xx_modem snd_ac97_codec ac97_bus firmware_class
snd_seq_oss tda9887 tuner_simple mt20xx tea5761 snd_seq_midi cx88_alsa
snd_pcm_oss snd_mixer_oss snd_rawmidi snd_seq_midi_event snd_pcm snd_seq
firewire_ohci firewire_core cx8802 cx8800 cx88xx ir_common
compat_ioctl32 videodev v4l1_compat hisax snd_timer k8temp
snd_seq_device i2c_algo_bit tveeprom crc_itu_t isdn snd_page_alloc
v4l2_common i2c_core psmouse rtc snd hwmon ehci_hcd parport_pc parport
serio_raw pcspkr uhci_hcd soundcore btcx_risc videobuf_dma_sg
videobuf_core usbcore slhc skge ohci1394 ieee1394 thermal button
processor evdev

[8.2.] Processor information (from /proc/cpuinfo):
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 35
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+
stepping        : 2
cpu MHz         : 2200.000
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
fxsr_opt lm 3dnowext 3dnow rep_good pni lahf_lm cmp_legacy
bogomips        : 4409.04
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 35
model name      : AMD Athlon(tm) 64 X2 Dual Core Processor 4400+
stepping        : 2
cpu MHz         : 2200.000
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
fxsr_opt lm 3dnowext 3dnow rep_good pni lahf_lm cmp_legacy
bogomips        : 4405.71
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp


[8.3.] Module information (from /proc/modules):
Not relevant!

[8.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
Not relevant!

[8.5.] PCI information ('lspci -vvv' as root)
Not relevant!

[8.6.] SCSI information (from /proc/scsi/scsi)
Not relevant!

[8.7.] Other information that might be relevant to the problem
       (please look in /proc and include all information that you
       think to be relevant):
[X.] Other notes, patches, fixes, workarounds:

Probably some kind of 64-bit kernel running 32-bit user-space
applications, that is only triggered by running UML.

Regards,
Thorsten

-- 
___
 |        | /                 E-Mail: linux@thorsten-knabe.de
 |horsten |/\nabe                WWW: http://linux.thorsten-knabe.de

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 17044 bytes --]

Re: [BUG] Linux 2.6.25.4 task_struct leak

[Chris Wright]

(Cc: Jeff)

* Thorsten Knabe (linux@thorsten-knabe.de) wrote:
> [5.] Most recent kernel version which did not have the bug:
> 2.6.23.17 does not leak task_structs.
> 2.6.24 - 2.6.25.3 has not been tested

Would you be able bisect this (or even just narrow it down a bit)?

> Running 2.6.23.16 i386 UML on a 2.6.25.4 i386 host does NOT leak
> task_structs.
> Running 64-bit UML guests has not been tested.

Re: [BUG] Linux 2.6.25.4 task_struct leak

[Jeff Dike]

On Sun, Jun 01, 2008 at 02:31:39PM -0700, Chris Wright wrote:
> * Thorsten Knabe (linux@thorsten-knabe.de) wrote:
> > [5.] Most recent kernel version which did not have the bug:
> > 2.6.23.17 does not leak task_structs.
> > 2.6.24 - 2.6.25.3 has not been tested

Has this been seen on non-UML?

    	      	   Jeff

-- 
Work email - jdike at linux dot intel dot com

Re: [BUG] Linux 2.6.25.4 task_struct leak

[Thorsten Knabe]

Jeff Dike wrote:
> On Sun, Jun 01, 2008 at 02:31:39PM -0700, Chris Wright wrote:
>> * Thorsten Knabe (linux@thorsten-knabe.de) wrote:
>>> [5.] Most recent kernel version which did not have the bug:
>>> 2.6.23.17 does not leak task_structs.
>>> 2.6.24 - 2.6.25.3 has not been tested
> 
> Has this been seen on non-UML?

Hello Jeff.

I'm not sure, if I understood your question correctly. The task_struct
leak is on the host running a vanilla Linux 2.6.25.4 x86_64 on real
hardware and not within the UML instances. I'll try to describe the
situation more precisely:

The host system is a vanilla Linux 2.6.25.4 x86_64 running on real
hardware with a Debian Etch x86_64 user space. However I've created a
Debian Etch i386 (32bit) userspace chroot environment on the system,
which is used, among other things, to compile the Linux 2.6.23.17 UML
kernels and start various UML instances.

I can start other 32-bit applications, for example compile an UML
kernel, within the chroot without leaking task_structs, but as soon as I
start an UML instance, I see leaked task_structs. Starting and
immediately shutting down an UML instacne leaks approximately 2000
task_structs. The number of leaked task_structs on the host seems to be
equal to the number of processes that have been created (and destroyed)
within the UML instances.

The host system was upgraded from 2.6.23.17 x86_64 which did not leak
task_structs to 2.6.25.4 x86_64. Also running the same UML kernels and
filesystems on my laptop with a vanilla Linux 2.6.25.4 i386 kernel does
not leak task_structs. The version of the UML kernel (tested 2.6.23.17
and 2.6.25.4) does not seem to matter.

As far as I understand the UML code in the kernel, an UML kernel uses
some unusual clone() flags when creating new processes, which are seldom
used by other applications and could be related to the bug.

If there is no obvious bug in the code, I'll try to bisect that beast
next weekend as suggested by Chris.

Regards
Thorsten

-- 
___
 |        | /                 E-Mail: linux@thorsten-knabe.de
 |horsten |/\nabe                WWW: http://linux.thorsten-knabe.de

Re: [BUG] Linux 2.6.25.4 task_struct leak

[Jeff Dike]

On Thu, Jun 05, 2008 at 12:40:07AM +0200, Thorsten Knabe wrote:
> I can start other 32-bit applications, for example compile an UML
> kernel, within the chroot without leaking task_structs, but as soon as I
> start an UML instance, I see leaked task_structs. Starting and
> immediately shutting down an UML instacne leaks approximately 2000
> task_structs. The number of leaked task_structs on the host seems to be
> equal to the number of processes that have been created (and destroyed)
> within the UML instances.

I misunderstood - I thought you were seeing a task_struct leak within
UML rather than a leak on the host elicited by UML.

> As far as I understand the UML code in the kernel, an UML kernel uses
> some unusual clone() flags when creating new processes, which are seldom
> used by other applications and could be related to the bug.

Yes, it does.  I don't see the flags causing a leak, though.  What
might be more likely (although I really have no idea) is ptrace.
Possibly a reference is held when it should have been dropped.  This
might also show up with strace or gdb.

      	   	   Jeff

-- 
Work email - jdike at linux dot intel dot com

Re: [BUG] Linux 2.6.25.4 task_struct leak

[Chris Wright]

* Jeff Dike (jdike@addtoit.com) wrote:
> Yes, it does.  I don't see the flags causing a leak, though.  What
> might be more likely (although I really have no idea) is ptrace.

Yeah, ptrace was my guess as well, and I was hoping you'd have an idea ;-)

> Possibly a reference is held when it should have been dropped.  This
> might also show up with strace or gdb.

Re: [BUG] Linux 2.6.25.4 task_struct leak

[Thorsten Knabe]

Jeff Dike wrote:
> I misunderstood - I thought you were seeing a task_struct leak within
> UML rather than a leak on the host elicited by UML.
> 
>> As far as I understand the UML code in the kernel, an UML kernel uses
>> some unusual clone() flags when creating new processes, which are seldom
>> used by other applications and could be related to the bug.
> 
> Yes, it does.  I don't see the flags causing a leak, though.  What
> might be more likely (although I really have no idea) is ptrace.
> Possibly a reference is held when it should have been dropped.  This
> might also show up with strace or gdb.

Hello Jeff.

Your assumption about ptrace causing the task_struct leak seems to be
right. I bisected the problem down to a few commits using the repository
at git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git.

Commit b7b71725fb9584454bfe5f231223bd63421798fb is the last known commit
that does not leak task_structs, whereas commit
a97f52e67890fda6b373c1c1895ff1c1c69b36c8 is leaking task_structs.
Revisions in between do not even compile.
Also I had to apply the changes from commit
f9cb02b0be4de3c51edfdd701754e13d9a2d20d6 to most of the kernels I have
tested, otherwise the UML process would crash on startup.

HTH
Thorsten

-- 
___
 |        | /                 E-Mail: linux@thorsten-knabe.de
 |horsten |/\nabe                WWW: http://linux.thorsten-knabe.de

Re: [BUG] Linux 2.6.25.4 task_struct leak

[WANG Cong]

On Sun, Jun 08, 2008 at 01:39:11PM +0200, Thorsten Knabe wrote:
>
>Hello Jeff.
>
>Your assumption about ptrace causing the task_struct leak seems to be
>right. I bisected the problem down to a few commits using the repository
>at git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git.
>
>Commit b7b71725fb9584454bfe5f231223bd63421798fb is the last known commit
>that does not leak task_structs, whereas commit
>a97f52e67890fda6b373c1c1895ff1c1c69b36c8 is leaking task_structs.
>Revisions in between do not even compile.
>Also I had to apply the changes from commit
>f9cb02b0be4de3c51edfdd701754e13d9a2d20d6 to most of the kernels I have
>tested, otherwise the UML process would crash on startup.

Add Cc to Roland McGrath <roland@redhat.com>.


-- 
Hi, I'm a .signature virus, please copy/paste me to help me spread
all over the world.

Re: [BUG] Linux 2.6.25.4 task_struct leak

[Roland McGrath]

Oops, mea culpa.  I added the bug in commit
5a4646a4efed8c835f76c3b88f3155f6ab5b8d9b.
It's already gone for 2.6.26 with the compat_arch_ptrace cleanup.

-stable patch to follow.


Thanks,
Roland

[PATCH stable-2.6.25] x86_64 ptrace: fix sys32_ptrace task_struct leak

[Roland McGrath]

Commit 5a4646a4efed8c835f76c3b88f3155f6ab5b8d9b introduced a leak of
task_struct refs into sys32_ptrace.  This bug has already gone away in
for 2.6.26 in commit 562b80bafffaf42a6d916b0a2ee3d684220a1c10.

Signed-off-by: Roland McGrath <roland@redhat.com>
---
 arch/x86/kernel/ptrace.c |   45 ++++++++++++++++++++++++++-------------------
 1 files changed, 26 insertions(+), 19 deletions(-)

diff --git a/arch/x86/kernel/ptrace.c b/arch/x86/kernel/ptrace.c
index 9003e0b..a10ba65 100644
--- a/arch/x86/kernel/ptrace.c
+++ b/arch/x86/kernel/ptrace.c
@@ -1309,42 +1309,49 @@ asmlinkage long sys32_ptrace(long request, u32 pid, u32 addr, u32 data)
 		break;
 
 	case PTRACE_GETREGS:	/* Get all gp regs from the child. */
-		return copy_regset_to_user(child, &user_x86_32_view,
-					   REGSET_GENERAL,
-					   0, sizeof(struct user_regs_struct32),
-					   datap);
+		ret = copy_regset_to_user(child, &user_x86_32_view,
+					  REGSET_GENERAL,
+					  0, sizeof(struct user_regs_struct32),
+					  datap);
+		break;
 
 	case PTRACE_SETREGS:	/* Set all gp regs in the child. */
-		return copy_regset_from_user(child, &user_x86_32_view,
-					     REGSET_GENERAL, 0,
-					     sizeof(struct user_regs_struct32),
-					     datap);
+		ret = copy_regset_from_user(child, &user_x86_32_view,
+					    REGSET_GENERAL, 0,
+					    sizeof(struct user_regs_struct32),
+					    datap);
+		break;
 
 	case PTRACE_GETFPREGS:	/* Get the child FPU state. */
-		return copy_regset_to_user(child, &user_x86_32_view,
-					   REGSET_FP, 0,
-					   sizeof(struct user_i387_ia32_struct),
-					   datap);
+		ret = copy_regset_to_user(child, &user_x86_32_view,
+					  REGSET_FP, 0,
+					  sizeof(struct user_i387_ia32_struct),
+					  datap);
+		break;
 
 	case PTRACE_SETFPREGS:	/* Set the child FPU state. */
-		return copy_regset_from_user(
+		ret = copy_regset_from_user(
 			child, &user_x86_32_view, REGSET_FP,
 			0, sizeof(struct user_i387_ia32_struct), datap);
+		break;
 
 	case PTRACE_GETFPXREGS:	/* Get the child extended FPU state. */
-		return copy_regset_to_user(child, &user_x86_32_view,
-					   REGSET_XFP, 0,
-					   sizeof(struct user32_fxsr_struct),
-					   datap);
+		ret = copy_regset_to_user(child, &user_x86_32_view,
+					  REGSET_XFP, 0,
+					  sizeof(struct user32_fxsr_struct),
+					  datap);
+		break;
 
 	case PTRACE_SETFPXREGS:	/* Set the child extended FPU state. */
-		return copy_regset_from_user(child, &user_x86_32_view,
+		ret = copy_regset_from_user(child, &user_x86_32_view,
 					     REGSET_XFP, 0,
 					     sizeof(struct user32_fxsr_struct),
 					     datap);
+		break;
 
 	default:
-		return compat_ptrace_request(child, request, addr, data);
+		ret = compat_ptrace_request(child, request, addr, data);
+		break;
 	}
 
  out:

Re: [PATCH stable-2.6.25] x86_64 ptrace: fix sys32_ptrace task_struct leak

[Ingo Molnar]

* Roland McGrath <roland@redhat.com> wrote:

> Commit 5a4646a4efed8c835f76c3b88f3155f6ab5b8d9b introduced a leak of 
> task_struct refs into sys32_ptrace.  This bug has already gone away in 
> for 2.6.26 in commit 562b80bafffaf42a6d916b0a2ee3d684220a1c10.
> 
> Signed-off-by: Roland McGrath <roland@redhat.com>

Acked-by: Ingo Molnar <mingo@elte.hu>

	Ingo