Size-128 slab leak

Size-128 slab leak

[Kevin O'Connor]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

Hi,

I have an annoying slab leak on my kernel.  Every day, I lose about
50Megs of memory to the leak.  It seems to be related to disk
accesses, because the count only goes up noticeable around 4:00am when
the system locate utility runs.

I can tell there is a leak because /proc/slabinfo shows "size-128"
growing continuously.  For example, it currently reads:

size-128          4086041 4106550    128   30    1 : tunables  120   60    8 : slabdata 136885 136885      0

The machine is a vanilla lkml kernel:

Linux double 2.6.15 #1 SMP Wed Jan 4 23:13:51 EST 2006 x86_64 x86_64 x86_64 GNU/Linux

I've noticed this bug on a 2.6.14 kernel also.  This machine is using
libata (sata_uli) along with reiserfs, ext3, and lvm.  I'm interested
in finding ways of diagnosing this problem.  I can provide more
information on demand.  Please CC me on any replies.

Thanks,
-Kevin

[-- Attachment #2: lspci-20060130 --]
[-- Type: text/plain, Size: 1672 bytes --]

00:00.0 Host bridge: ALi Corporation M1695 K8 Northbridge [PCI Express and HyperTransport]
00:01.0 PCI bridge: ALi Corporation: Unknown device 524b
00:02.0 PCI bridge: ALi Corporation: Unknown device 524c
00:04.0 Host bridge: ALi Corporation M1689 K8 Northbridge [Super K8 Single Chip]
00:05.0 PCI bridge: ALi Corporation AGP8X Controller
00:06.0 PCI bridge: ALi Corporation M5249 HTT to PCI Bridge
00:07.0 ISA bridge: ALi Corporation M1563 HyperTransport South Bridge (rev 70)
00:07.1 Bridge: ALi Corporation M7101 Power Management Controller [PMU]
00:08.0 Multimedia audio controller: ALi Corporation M5455 PCI AC-Link Controller Audio Device (rev 20)
00:11.0 Ethernet controller: ALi Corporation M5263 Ethernet Controller (rev 40)
00:12.0 IDE interface: ALi Corporation M5229 IDE (rev c7)
00:12.1 IDE interface: ALi Corporation ULi 5289 SATA (rev 10)
00:13.0 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.1 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.2 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
00:13.3 USB Controller: ALi Corporation USB 2.0 Controller (rev 01)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
03:00.0 VGA compatible controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (rev 01)
03:00.1 Display controller: ATI Technologies Inc: Unknown device 5940 (rev 01)

[-- Attachment #3: cpuinfo-20060130 --]
[-- Type: text/plain, Size: 1276 bytes --]

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 43
model name	: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
stepping	: 1
cpu MHz		: 1000.051
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips	: 2002.42
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 15
model		: 43
model name	: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
stepping	: 1
cpu MHz		: 1000.051
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 1
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips	: 2002.42
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: ts fid vid ttp


[-- Attachment #4: lsmod-20060130 --]
[-- Type: text/plain, Size: 2883 bytes --]

Module                  Size  Used by
cramfs                 46144  1 
loop                   19472  2 
ppp_deflate             8192  0 
zlib_deflate           26016  1 ppp_deflate
ppp_async              15488  0 
crc_ccitt               3200  1 ppp_async
ppp_generic            36384  2 ppp_deflate,ppp_async
slhc                    8960  1 ppp_generic
vfat                   17152  0 
fat                    60976  1 vfat
usb_storage            86592  0 
snd_rtctimer            4888  0 
udf                    92064  1 
nls_utf8                3328  0 
ipaq                   40880  0 
usbserial              38612  1 ipaq
radeon                119584  1 
drm                   100776  2 radeon
ipv6                  311680  14 
parport_pc             33900  0 
lp                     16960  0 
parport                46604  2 parport_pc,lp
autofs4                25608  1 
w83627hf               35344  0 
hwmon_vid               3712  1 w83627hf
hwmon                   4616  1 w83627hf
eeprom                  9744  0 
i2c_isa                 7552  1 w83627hf
i2c_dev                14208  0 
i2c_core               27904  4 w83627hf,eeprom,i2c_isa,i2c_dev
sunrpc                184504  1 
pcmcia                 48816  0 
yenta_socket           31628  0 
rsrc_nonstatic         15872  1 yenta_socket
pcmcia_core            50228  3 pcmcia,yenta_socket,rsrc_nonstatic
reiserfs              285688  2 
video                  20488  0 
button                  8992  0 
battery                12168  0 
ac                      6792  0 
ohci_hcd               24708  0 
ehci_hcd               38920  0 
shpchp                 53888  0 
snd_intel8x0           39592  0 
snd_ac97_codec        117180  1 snd_intel8x0
snd_ac97_bus            3840  1 snd_ac97_codec
snd_seq_dummy           5380  0 
snd_seq_oss            41700  0 
snd_seq_midi_event     10368  1 snd_seq_oss
snd_seq                70616  5 snd_seq_dummy,snd_seq_oss,snd_seq_midi_event
snd_seq_device         12048  3 snd_seq_dummy,snd_seq_oss,snd_seq
snd_pcm_oss            63264  0 
snd_mixer_oss          21632  1 snd_pcm_oss
snd_pcm               111624  3 snd_intel8x0,snd_ac97_codec,snd_pcm_oss
snd_timer              30600  3 snd_rtctimer,snd_seq,snd_pcm
snd                    73696  9 snd_intel8x0,snd_ac97_codec,snd_seq_oss,snd_seq,snd_seq_device,snd_pcm_oss,snd_mixer_oss,snd_pcm,snd_timer
soundcore              13088  1 snd
snd_page_alloc         13840  2 snd_intel8x0,snd_pcm
uli526x                21268  0 
dm_snapshot            18768  0 
dm_zero                 2816  0 
dm_mirror              25320  0 
ext3                  152848  2 
jbd                    68904  1 ext3
dm_mod                 69064  8 dm_snapshot,dm_zero,dm_mirror
sata_uli                8964  1 
libata                 67224  1 sata_uli
sd_mod                 21632  1 
scsi_mod              166712  3 usb_storage,libata,sd_mod

[-- Attachment #5: slabinfo-20060130.gz --]
[-- Type: application/x-gzip, Size: 2415 bytes --]

Re: Size-128 slab leak

[Andrew Morton]

"Kevin O'Connor" <kevin@koconnor.net> wrote:
>
> I have an annoying slab leak on my kernel.  Every day, I lose about
>  50Megs of memory to the leak.  It seems to be related to disk
>  accesses, because the count only goes up noticeable around 4:00am when
>  the system locate utility runs.
> 
>  I can tell there is a leak because /proc/slabinfo shows "size-128"
>  growing continuously.  For example, it currently reads:
> 
>  size-128          4086041 4106550    128   30    1 : tunables  120   60    8 : slabdata 136885 136885      0
> 
>  The machine is a vanilla lkml kernel:
> 
>  Linux double 2.6.15 #1 SMP Wed Jan 4 23:13:51 EST 2006 x86_64 x86_64 x86_64 GNU/Linux
> 
>  I've noticed this bug on a 2.6.14 kernel also.  This machine is using
>  libata (sata_uli) along with reiserfs, ext3, and lvm.  I'm interested
>  in finding ways of diagnosing this problem.

-mm kernels have a patch (slab-leak-detector.patch) which will help. 
Here's a version for 2.6.16-rc1.  It requires CONFIG_DEBUG_SLAB.  Thanks.


From: Manfred Spraul <manfred@colorfullife.com>

Maintenance work from Alexander Nyberg <alexn@telia.com>

With the patch applied,

	echo "size-4096 0 0 0" > /proc/slabinfo

walks the objects in the size-4096 slab, printing out the calling address
of whoever allocated that object.

It is for leak detection.

Signed-off-by: Andrew Morton <akpm@osdl.org>
---

 mm/slab.c |   46 +++++++++++++++++++++++++++++++++++++++++++---
 1 files changed, 43 insertions(+), 3 deletions(-)

diff -puN mm/slab.c~slab-leak-detector mm/slab.c
--- devel/mm/slab.c~slab-leak-detector	2006-02-01 23:05:08.000000000 -0800
+++ devel-akpm/mm/slab.c	2006-02-01 23:07:43.000000000 -0800
@@ -198,7 +198,7 @@
  * is less than 512 (PAGE_SIZE<<3), but greater than 256.
  */
 
-typedef unsigned int kmem_bufctl_t;
+typedef unsigned long kmem_bufctl_t;
 #define BUFCTL_END	(((kmem_bufctl_t)(~0U))-0)
 #define BUFCTL_FREE	(((kmem_bufctl_t)(~0U))-1)
 #define	SLAB_LIMIT	(((kmem_bufctl_t)(~0U))-2)
@@ -2393,7 +2393,7 @@ static void check_slabp(kmem_cache_t *ca
 		     i < sizeof(slabp) + cachep->num * sizeof(kmem_bufctl_t);
 		     i++) {
 			if ((i % 16) == 0)
-				printk("\n%03x:", i);
+				printk("\n%04lx:", i);
 			printk(" %02x", ((unsigned char *)slabp)[i]);
 		}
 		printk("\n");
@@ -2550,6 +2550,15 @@ static void *cache_alloc_debugcheck_afte
 		*dbg_redzone1(cachep, objp) = RED_ACTIVE;
 		*dbg_redzone2(cachep, objp) = RED_ACTIVE;
 	}
+	{
+		int objnr;
+		struct slab *slabp;
+
+		slabp = page_get_slab(virt_to_page(objp));
+
+		objnr = (objp - slabp->s_mem) / cachep->objsize;
+		slab_bufctl(slabp)[objnr] = (unsigned long)caller;
+	}
 	objp += obj_dbghead(cachep);
 	if (cachep->ctor && cachep->flags & SLAB_POISON) {
 		unsigned long ctor_flags = SLAB_CTOR_CONSTRUCTOR;
@@ -2691,7 +2700,7 @@ static void free_block(kmem_cache_t *cac
 		check_spinlock_acquired_node(cachep, node);
 		check_slabp(cachep, slabp);
 
-#if DEBUG
+#if 0
 		/* Verify that the slab belongs to the intended node */
 		WARN_ON(slabp->nodeid != node);
 
@@ -3573,6 +3582,36 @@ struct seq_operations slabinfo_op = {
 	.show = s_show,
 };
 
+static void do_dump_slabp(kmem_cache_t *cachep)
+{
+#if DEBUG
+	struct list_head *q;
+	int node;
+
+	check_irq_on();
+	spin_lock_irq(&cachep->spinlock);
+	for_each_online_node(node) {
+		struct kmem_list3 *rl3 = cachep->nodelists[node];
+		spin_lock(&rl3->list_lock);
+
+		list_for_each(q, &rl3->slabs_full) {
+			int i;
+			struct slab *slabp = list_entry(q, struct slab, list);
+
+			for (i = 0; i < cachep->num; i++) {
+				unsigned long sym = slab_bufctl(slabp)[i];
+
+				printk("obj %p/%d: %p", slabp, i, (void *)sym);
+				print_symbol(" <%s>", sym);
+				printk("\n");
+			}
+		}
+		spin_unlock(&rl3->list_lock);
+	}
+	spin_unlock_irq(&cachep->spinlock);
+#endif
+}
+
 #define MAX_SLABINFO_WRITE 128
 /**
  * slabinfo_write - Tuning for the slab allocator
@@ -3612,6 +3651,7 @@ ssize_t slabinfo_write(struct file *file
 			if (limit < 1 ||
 			    batchcount < 1 ||
 			    batchcount > limit || shared < 0) {
+				do_dump_slabp(cachep);
 				res = 0;
 			} else {
 				res = do_tune_cpucache(cachep, limit,
_

Re: Size-128 slab leak

[Pekka Enberg]

Hi,

On 2/2/06, Andrew Morton <akpm@osdl.org> wrote:
> @@ -2550,6 +2550,15 @@ static void *cache_alloc_debugcheck_afte
>                 *dbg_redzone1(cachep, objp) = RED_ACTIVE;
>                 *dbg_redzone2(cachep, objp) = RED_ACTIVE;
>         }
> +       {
> +               int objnr;
> +               struct slab *slabp;
> +
> +               slabp = page_get_slab(virt_to_page(objp));
> +
> +               objnr = (objp - slabp->s_mem) / cachep->objsize;
> +               slab_bufctl(slabp)[objnr] = (unsigned long)caller;
> +       }

We already have last caller in dbg_userword. Manfred, is there a
reason we're not using it?

                               Pekka

Re: Size-128 slab leak

[Kevin O'Connor]

On Wed, Feb 01, 2006 at 11:10:01PM -0800, Andrew Morton wrote:
> "Kevin O'Connor" <kevin@koconnor.net> wrote:
> >
> > I have an annoying slab leak on my kernel.  Every day, I lose about
> >  50Megs of memory to the leak.  It seems to be related to disk
> >  accesses, because the count only goes up noticeable around 4:00am when
> >  the system locate utility runs.
> > 
> >  I can tell there is a leak because /proc/slabinfo shows "size-128"
> >  growing continuously.  For example, it currently reads:
>
> -mm kernels have a patch (slab-leak-detector.patch) which will help. 
> Here's a version for 2.6.16-rc1.  It requires CONFIG_DEBUG_SLAB.  Thanks.

Thanks Andrew.

I've applied the patch and found the leak.  It's in kzalloc.  :-)

With kzalloc inlined, however, it appears that selinux is the likely
culprit.  I would not have expected that.

After running updatedb I got 23530 occurrences of:

kernel: obj ffff81003f04f000/12: ffffffff801ed7b7 <selinux_inode_alloc_security+0x37/0x100>

I'm not sure how to debug selinux issues, but at least I can disable
it.

-Kevin

Re: Size-128 slab leak

[Manfred Spraul]

Pekka Enberg wrote:

>We already have last caller in dbg_userword. Manfred, is there a
>reason we're not using it?
>
>  
>
IIRC only due to historical reasons: dbg_userword is newer than this patch.

--
    Manfred

Re: Size-128 slab leak

[Stephen Smalley]

On Thu, 2006-02-02 at 23:00 -0500, Kevin O'Connor wrote:
> Thanks Andrew.
> 
> I've applied the patch and found the leak.  It's in kzalloc.  :-)
> 
> With kzalloc inlined, however, it appears that selinux is the likely
> culprit.  I would not have expected that.
> 
> After running updatedb I got 23530 occurrences of:
> 
> kernel: obj ffff81003f04f000/12: ffffffff801ed7b7 <selinux_inode_alloc_security+0x37/0x100>
> 
> I'm not sure how to debug selinux issues, but at least I can disable
> it.

Hmm...that allocation call occurs upon alloc_inode() via
security_inode_alloc, and the associated free call occurs upon
destroy_inode() via security_inode_free.  However, when Jeff Mahoney
introduced the support for "private inodes" (S_PRIVATE flag) to support
reiserfs xattrs-as-files, he added the IS_PRIVATE guards to both
security_inode_alloc and security_inode_free.  I think that this ends up
causing SELinux to allocate a security structure for every reiserfs
inode including private inodes since they are not marked until later by
reiserfs, while preventing SELinux from ever freeing the security
structure for the private inodes.  Note that
selinux_inode_free_security() should be safe even for the private
inodes, as it doesn't assume any other initialization beyond the
allocation-time initialization.  Patch below.  

BTW, Jeff - I assume you know about the unrelated breakage in
SELinux/reiserfs support that was introduced by the changes for atomic
security labeling of inodes in 2.6.14.  If you care, you might want to
use the same workaround upstreamed for xfs for 2.6.16.  But I understand
that SELinux is not a priority in SuSE presently.

BTW, Chris - are you still serving as LSM maintainer?  MAINTAINERS still
lists your osdl address.

---

Remove private inode tests from security_inode_alloc and security_inode_free,
as we otherwise end up leaking inode security structures for private inodes.

Signed-off-by:  Stephen Smalley <sds@tycho.nsa.gov>

---

 include/linux/security.h |    4 ----
 1 file changed, 4 deletions(-)

Index: linux-2.6/include/linux/security.h
===================================================================
RCS file: /nfshome/pal/CVS/linux-2.6/include/linux/security.h,v
retrieving revision 1.50
diff -u -p -r1.50 security.h
--- linux-2.6/include/linux/security.h	3 Jan 2006 16:36:48 -0000	1.50
+++ linux-2.6/include/linux/security.h	3 Feb 2006 12:50:57 -0000
@@ -1437,15 +1437,11 @@ static inline void security_sb_post_pivo
 
 static inline int security_inode_alloc (struct inode *inode)
 {
-	if (unlikely (IS_PRIVATE (inode)))
-		return 0;
 	return security_ops->inode_alloc_security (inode);
 }
 
 static inline void security_inode_free (struct inode *inode)
 {
-	if (unlikely (IS_PRIVATE (inode)))
-		return;
 	security_ops->inode_free_security (inode);
 }
 


-- 
Stephen Smalley
National Security Agency

Re: Size-128 slab leak

[James Morris]

On Fri, 3 Feb 2006, Stephen Smalley wrote:

> Signed-off-by:  Stephen Smalley <sds@tycho.nsa.gov>

Acked-by: James Morris <jmorris@namei.org>


-- 
James Morris
<jmorris@namei.org>

Re: Size-128 slab leak

[Kevin O'Connor]

On Fri, Feb 03, 2006 at 08:21:12AM -0500, Stephen Smalley wrote:
> On Thu, 2006-02-02 at 23:00 -0500, Kevin O'Connor wrote:
> > After running updatedb I got 23530 occurrences of:
> > 
> > kernel: obj ffff81003f04f000/12: ffffffff801ed7b7 <selinux_inode_alloc_security+0x37/0x100>
> > 
> Hmm...that allocation call occurs upon alloc_inode() via
> security_inode_alloc, and the associated free call occurs upon
> destroy_inode() via security_inode_free.  However, when Jeff Mahoney
> introduced the support for "private inodes" (S_PRIVATE flag) to support
> reiserfs xattrs-as-files, he added the IS_PRIVATE guards to both
> security_inode_alloc and security_inode_free.  I think that this ends up
> causing SELinux to allocate a security structure for every reiserfs
> inode including private inodes since they are not marked until later by
> reiserfs, while preventing SELinux from ever freeing the security
> structure for the private inodes.  Note that
> selinux_inode_free_security() should be safe even for the private
> inodes, as it doesn't assume any other initialization beyond the
> allocation-time initialization.  Patch below.  

Hi Stephen,

I've applied your patch.  It seems to be working.  (Multiple runs of
updatedb no longer grow the size-128 slab.)

Thanks,
-Kevin

Re: Size-128 slab leak

[Jeffrey Mahoney]

Stephen Smalley wrote:
> BTW, Jeff - I assume you know about the unrelated breakage in
> SELinux/reiserfs support that was introduced by the changes for atomic
> security labeling of inodes in 2.6.14.  If you care, you might want to
> use the same workaround upstreamed for xfs for 2.6.16.  But I understand
> that SELinux is not a priority in SuSE presently.

Hi Stephen -

Thanks for the reminder on this one. I have a series of patches that
rework the reiserfs xattr implementation. One of the patches adds
journalling for xattrs, including putting the inherited ACLs in the same
transaction as the object creation. Adding the SELinux attribute to the
same transaction wouldn't be too much of a problem.

I should have a patch tomorrow, though it will depend on the xattr patchset.

-Jeff

--
Jeff Mahoney
SUSE Labs