yet more slab corruption.

yet more slab corruption.

[Dave Jones]

[-- Attachment #1: Type: text/plain, Size: 301 bytes --]

Garg. Is there no end to these ?
That kernel is based off 2.6.16rc5-git8

This brings the current count up to 8 different patterns filed
against our 2.6.16rc tree in Fedora bugzilla.
(One of them doesn't count as it's against the out-of-tree bcm43xx driver).

		Dave

-- 
http://www.codemonkey.org.uk

[-- Attachment #2: Type: message/rfc822, Size: 10032 bytes --]

From: bugzilla@redhat.com
To: kernel-maint@redhat.com
Subject: [Bug 184310] New: Slab corruption in 2.6.15-1.2025_FC5smp
Date: Tue, 7 Mar 2006 17:37:25 -0500
Message-ID: <bug-184310-176318@bugzilla.redhat.com>

Please do not reply directly to this email. All additional
comments should be made in the comments box of this bug report.




https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=184310

           Summary: Slab corruption in 2.6.15-1.2025_FC5smp
           Product: Fedora Core
           Version: devel
          Platform: i686
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: normal
         Component: kernel
        AssignedTo: kernel-maint@redhat.com
        ReportedBy: hongjiu.lu@intel.com
         QAContact: bbrock@redhat.com
                CC: wtogami@redhat.com


I am running 2.6.15-1.2025_FC5smp. When I was building gcc with the gcc source
tree on a NFS server running RHEL 4 U2, I got

Slab corruption: (Not tainted) start=c574aeb8, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02182d0>](release_mem+0xe6/0x1de)
 [<c01602a5>] check_poison_obj+0x6a/0x15d     [<c01603ba>]
cache_alloc_debugcheck_after+0x22/0xf9
 [<c02c26d5>] tcp_sendmsg+0x153/0x983     [<c0160bfa>]
__kmalloc_track_caller+0xb5/0xbf
 [<c02c26d5>] tcp_sendmsg+0x153/0x983     [<c029c5db>] __alloc_skb+0x4d/0xf2
 [<c02c26d5>] tcp_sendmsg+0x153/0x983     [<c02d980d>] inet_sendmsg+0x35/0x3f
 [<c02972da>] sock_sendmsg+0xd2/0xec     [<c013465b>]
autoremove_wake_function+0x0/0x2d
 [<c0298acb>] kernel_sendmsg+0x26/0x2c     [<f8c4fd26>]
xs_tcp_send_request+0x107/0x302 [sunrpc]
 [<f8c4f050>] xprt_transmit+0xd7/0x1c8 [sunrpc]     [<f8d386dc>]
nfs3_xdr_fhandle+0x0/0x21 [nfs]
 [<f8c4e1ba>] call_transmit+0x198/0x1cf [sunrpc]     [<f8c51dd7>]
__rpc_execute+0x7a/0x193 [sunrpc]
 [<f8c4d2e4>] rpc_call_sync+0x6b/0x91 [sunrpc]     [<f8d35f3d>]
nfs3_rpc_wrapper+0x1f/0x5b [nfs]
 [<f8d36939>] nfs3_proc_getattr+0x74/0x96 [nfs]     [<f8d2f1f7>]
__nfs_revalidate_inode+0x113/0x24e [nfs]
 [<c01780e5>] __d_lookup+0xbc/0xee     [<c02f5136>] _read_unlock_irq+0x5/0x7
 [<c014d5b6>] __do_page_cache_readahead+0x10f/0x212     [<f8d2b251>]
nfs_lookup_revalidate+0x18a/0x318 [nfs]
 [<c017837a>] dput+0x31/0x164     [<f8d2b2d3>] nfs_lookup_revalidate+0x20c/0x318
[nfs]
 [<c0160420>] cache_alloc_debugcheck_after+0x88/0xf9     [<c01d765c>]
vsnprintf+0x422/0x461
 [<f8c52a58>] rpcauth_lookup_credcache+0x153/0x1ef [sunrpc]     [<c01780e5>]
__d_lookup+0xbc/0xee
 [<c016fbd7>] do_lookup+0x11d/0x14d     [<c01716da>] __link_path_walk+0x834/0xc7d
 [<c017bfe2>] mntput_no_expire+0x11/0x6e     [<c0171afd>]
__link_path_walk+0xc57/0xc7d
 [<c0171b6c>] link_path_walk+0x49/0xbd     [<c0171f07>] do_path_lookup+0x1e3/0x248
 [<c01727bf>] __path_lookup_intent_open+0x42/0x72     [<c017283e>]
path_lookup_open+0xf/0x13
 [<c0172938>] open_namei+0x73/0x54c     [<c0162c1f>] do_filp_open+0x1c/0x31
 [<c0163ab8>] do_sys_open+0x3e/0xaa     [<c0103d81>] syscall_call+0x7/0xb
0b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff
0c0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Prev obj: start=c574a6ac, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c029c35c>](kfree_skbmem+0x8/0x63)
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Next obj: start=c574b6c4, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02182d0>](release_mem+0xe6/0x1de)
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Slab corruption: (Not tainted) start=c574b6c4, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02182d0>](release_mem+0xe6/0x1de)
 [<c01602a5>] check_poison_obj+0x6a/0x15d     [<c0170dc4>] lookup_one_len+0x4b/0x59
 [<c01603ba>] cache_alloc_debugcheck_after+0x22/0xf9     [<c02181d5>]
alloc_tty_struct+0x10/0x25
 [<c0160510>] kmem_cache_alloc+0x7f/0x89     [<c02181d5>] alloc_tty_struct+0x10/0x25
 [<c02181d5>] alloc_tty_struct+0x10/0x25     [<c0218446>] init_dev+0x7e/0x47a
 [<c021ac1b>] ptmx_open+0xf8/0x1b6     [<c02f525b>] lock_kernel+0x25/0x34
 [<c016ba4e>] chrdev_open+0x104/0x148     [<c016b94a>] chrdev_open+0x0/0x148
 [<c0162aac>] __dentry_open+0xc7/0x1ab     [<c0162bf4>] nameidata_to_filp+0x19/0x28
 [<c0162c2e>] do_filp_open+0x2b/0x31     [<c0163ab8>] do_sys_open+0x3e/0xaa
 [<c0103d81>] syscall_call+0x7/0xb    <3>0b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
6b ff ff ff ff
0c0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Prev obj: start=c574aeb8, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c029c35c>](kfree_skbmem+0x8/0x63)
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Slab corruption: (Not tainted) start=c574a6ac, len=2048
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c02182d0>](release_mem+0xe6/0x1de)
 [<c01602a5>] check_poison_obj+0x6a/0x15d
 [<c01603ba>] cache_alloc_debugcheck_after+0x22/0xf9     [<c02c26d5>]
tcp_sendmsg+0x153/0x983
 [<c0160bfa>] __kmalloc_track_caller+0xb5/0xbf     [<c02c26d5>]
tcp_sendmsg+0x153/0x983
 [<c029c5db>] __alloc_skb+0x4d/0xf2     [<c02c26d5>] tcp_sendmsg+0x153/0x983
 [<c02d980d>] inet_sendmsg+0x35/0x3f     [<c02972da>] sock_sendmsg+0xd2/0xec
 [<c013465b>] autoremove_wake_function+0x0/0x2d     [<c0298acb>]
kernel_sendmsg+0x26/0x2c
 [<f8c4fd26>] xs_tcp_send_request+0x107/0x302 [sunrpc]     [<f8c4f050>]
xprt_transmit+0xd7/0x1c8 [sunrpc]
 [<f8d38c69>] nfs3_xdr_diropargs+0x0/0x2e [nfs]     [<f8c4e1ba>]
call_transmit+0x198/0x1cf [sunrpc]
 [<f8c51dd7>] __rpc_execute+0x7a/0x193 [sunrpc]     [<f8c4d2e4>]
rpc_call_sync+0x6b/0x91 [sunrpc]
 [<f8d35f3d>] nfs3_rpc_wrapper+0x1f/0x5b [nfs]     [<f8d36a59>]
nfs3_proc_lookup+0xfe/0x1a6 [nfs]
 [<c01780e5>] __d_lookup+0xbc/0xee     [<c02f5136>] _read_unlock_irq+0x5/0x7
 [<c014d5b6>] __do_page_cache_readahead+0x10f/0x212     [<c017837a>] dput+0x31/0x164
 [<f8d2b822>] nfs_lookup+0x96/0x104 [nfs]     [<c0160420>]
cache_alloc_debugcheck_after+0x88/0xf9
 [<c01d765c>] vsnprintf+0x422/0x461     [<c0160459>]
cache_alloc_debugcheck_after+0xc1/0xf9
 [<c017816f>] d_alloc+0x1d/0x1c1     [<c0160510>] kmem_cache_alloc+0x7f/0x89
 [<c017816f>] d_alloc+0x1d/0x1c1     [<c0178307>] d_alloc+0x1b5/0x1c1
 [<c016fb6c>] do_lookup+0xb2/0x14d     [<c01716da>] __link_path_walk+0x834/0xc7d
[<c017837a>] dput+0x31/0x164     [<c0171b6c>] link_path_walk+0x49/0xbd
 [<c0171f07>] do_path_lookup+0x1e3/0x248     [<c0172661>] __user_walk_fd+0x29/0x3a
 [<c016c470>] vfs_stat_fd+0x15/0x3c     [<c016c524>] sys_stat64+0xf/0x23
 [<c02f5e9f>] do_page_fault+0x0/0x5e2     [<c0103d81>] syscall_call+0x7/0xb
0b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff
0c0: 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Next obj: start=c574aeb8, len=2048
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<c02181d5>](alloc_tty_struct+0x10/0x25)
000: 01 54 00 00 d4 a4 d5 f7 02 00 00 00 03 54 00 00
010: 7c 85 32 c0 00 00 00 00 01 00 00 00 4c b8 21 c0

-- 
Configure bugmail: https://bugzilla.redhat.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Re: yet more slab corruption.

[Andrew Morton]

Dave Jones <davej@redhat.com> wrote:
>
> Garg. Is there no end to these ?
> That kernel is based off 2.6.16rc5-git8
> 
> This brings the current count up to 8 different patterns filed
> against our 2.6.16rc tree in Fedora bugzilla.
> (One of them doesn't count as it's against the out-of-tree bcm43xx driver).
> 

A use-after-free on size-2048.  We wrote -1L and 0L apparently 0x6b8 bytes
into the object.  That's an awfully large offset for tty_struct - off the
end.  Sometimes the buffer was used as skb data too.

Unless it was a DMA scribble, CONFIG_DEBUG_PAGEALLOC should catch this.

Re: yet more slab corruption.

[Andreas Gruenbacher]

On Wednesday, 08 March 2006 08:14, Andrew Morton wrote:
> Dave Jones <davej@redhat.com> wrote:
> > Garg. Is there no end to these ?
> > That kernel is based off 2.6.16rc5-git8
> >
> > This brings the current count up to 8 different patterns filed
> > against our 2.6.16rc tree in Fedora bugzilla.
> > (One of them doesn't count as it's against the out-of-tree bcm43xx
> > driver).

We have two similar bug reports to 
https://bugzilla.redhat.com/bugzilla/184310, slab corruption in an object 
freed by release_mem:

	https://bugzilla.novell.com/151111 (i386)
	https://bugzilla.novell.com/154601 (x86_64)

So this bug seems to trigger on different architectures, and with different 
hardware.

> A use-after-free on size-2048.  We wrote -1L and 0L apparently 0x6b8 bytes
> into the object.  That's an awfully large offset for tty_struct - off the
> end.  Sometimes the buffer was used as skb data too.

I don't know about offset 0x6b8: 0x6b is POISON_FREE in mm/slab.c, so this is 
probably a misread. I think the correct offset is 0xbc. Bug 151111 has the 
corruption at the same offset. On x86_64, the offset is 0x124. This seems to 
be tty_struct->winsize in all cases, even though I can't tell for sure for 
most of the reports anymore.

So this could be a tty_struct locking bug -- it's surprising that we are also 
seeing filesystem corruption, but only in some cases. There was a recent 
locking fix in this area by Paul Fulghum <paulkf@microgate.com> from Feb 15 
which might be related, but the fix looks good to me: 
http://www.kernel.org/hg/linux-2.6/?cs=623e3c38a511.

> Unless it was a DMA scribble, CONFIG_DEBUG_PAGEALLOC should catch this.

This didn't trigger anything in an overnight run. Any further ideas?

Thanks,
Andreas

-- 
Andreas Gruenbacher <agruen@suse.de>
Novell / SUSE Labs

Re: yet more slab corruption.

[Jan Niehusmann]

On Tue, Apr 04, 2006 at 03:44:01PM +0200, Andreas Gruenbacher wrote:
> We have two similar bug reports to 
> https://bugzilla.redhat.com/bugzilla/184310, slab corruption in an object 
> freed by release_mem:
> 
> 	https://bugzilla.novell.com/151111 (i386)
> 	https://bugzilla.novell.com/154601 (x86_64)
> 
> So this bug seems to trigger on different architectures, and with different 
> hardware.

I also got a few of these, just reported in a different thread, 
see http://lkml.org/lkml/2006/4/4/86 for details.

Jan