* 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor
@ 2004-12-20 22:28 Manfred Schwarb
2004-12-21 16:46 ` Marcelo Tosatti
0 siblings, 1 reply; 7+ messages in thread
From: Manfred Schwarb @ 2004-12-20 22:28 UTC (permalink / raw)
To: linux-kernel
Hi,
yesterday, I got at the start of a rsync backup over nfs
an Oops in find_inode/reiserfs_find_actor.
Some googling revealed that this seems to be an
inode list corruption, but how can this happen?
There seem to be quite some reports about similar problems,
so doing rsync over nfs-mounted partitions might be a bad idea?
Is there anything I can do? Any help is appreciated.
This machine is an athlon i386 with vanilla 2.4.29-pre2.
Below you may find the ksymoopsified Oops:
Dec 19 21:00:01 kernel: Unable to handle kernel NULL pointer dereference at
virtual address 00000028
Dec 19 21:00:01 kernel: c0153b09
Dec 19 21:00:01 kernel: *pde = 1d678067
Dec 19 21:00:01 kernel: Oops: 0000
Dec 19 21:00:01 kernel: CPU: 0
Dec 19 21:00:01 kernel: EIP: 0010:[find_inode+25/112] Not tainted
Dec 19 21:00:01 kernel: EIP: 0010:[<c0153b09>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Dec 19 21:00:01 kernel: EFLAGS: 00213213
Dec 19 21:00:01 kernel: eax: e0b3c9e0 ebx: 00000000 ecx: 0000000f
edx: dff80000
Dec 19 21:00:01 kernel: esi: 00000000 edi: dffa2a58 ebp: 000000fa
esp: d7f05d60
Dec 19 21:00:01 kernel: ds: 0018 es: 0018 ss: 0018
Dec 19 21:00:01 kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
Dec 19 21:00:01 kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4 dffa2a58
000000fa da2f6000 c0153f4e
Dec 19 21:00:01 kernel: da2f6000 000000fa dffa2a58 e0b3c9e0 d7f05db8
d7f05de4 d7f05e48 d7f05db8
Dec 19 21:00:01 kernel: da2f6000 e0b3ca60 da2f6000 000000fa e0b3c9e0
d7f05db8 00000011 d7f05de4
Dec 19 21:00:01 kernel: Call Trace: [iget4_locked+94/272]
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96]
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]
Dec 19 21:00:01 kernel: Call Trace: [<c0153f4e>] [<e0b3c9e0>]
[<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
Dec 19 21:00:01 kernel: [<c0151d1c>] [<c01491cf>] [<c0149279>]
[<e118fd71>] [<e1196d39>] [<e119de8c>]
Dec 19 21:00:01 kernel: [<e118c68d>] [<c027ab3e>] [<e119de8c>]
[<e119d758>] [<e119d778>] [<e118c3cb>]
Dec 19 21:00:01 kernel: [<c010729b>] [<e118c210>]
Dec 19 21:00:01 kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0 00
00 00 75 e5 8b
>>EIP; c0153b09 <find_inode+19/70> <=====
>>eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
>>edx; dff80000 <_end+1fbfd7f4/20792854>
>>edi; dffa2a58 <_end+1fc2024c/20792854>
>>esp; d7f05d60 <_end+17b83554/20792854>
Trace; c0153f4e <iget4_locked+5e/110>
Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0>
Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120>
Trace; c0151d1c <d_alloc+1c/1d0>
Trace; c01491cf <lookup_hash+9f/d0>
Trace; c0149279 <lookup_one_len+79/90>
Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490>
Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140>
Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220>
Trace; c027ab3e <svc_process+3de/590>
Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
Trace; e119d758 <[nfsd]nfsd_version3+0/10>
Trace; e119d778 <[nfsd]nfsd_program+0/28>
Trace; e118c3cb <[nfsd]nfsd+1bb/330>
Trace; c010729b <arch_kernel_thread+2b/40>
Trace; e118c210 <[nfsd]nfsd+0/330>
Code; c0153b09 <find_inode+19/70>
00000000 <_EIP>:
Code; c0153b09 <find_inode+19/70> <=====
0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
Code; c0153b0c <find_inode+1c/70>
3: 89 de mov %ebx,%esi
Code; c0153b0e <find_inode+1e/70>
5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b10 <find_inode+20/70>
7: 8b 44 24 20 mov 0x20(%esp,1),%eax
Code; c0153b14 <find_inode+24/70>
b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
Code; c0153b1a <find_inode+2a/70>
11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b1c <find_inode+2c/70>
13: 8b 00 mov (%eax),%eax
--
Psssst! Mit GMX Handyrechnung senken: http://www.gmx.net/de/go/mail
100 FreeSMS/Monat (GMX TopMail), 50 (GMX ProMail), 10 (GMX FreeMail)
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor
2004-12-20 22:28 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor Manfred Schwarb
@ 2004-12-21 16:46 ` Marcelo Tosatti
2004-12-21 19:34 ` Trond Myklebust
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Marcelo Tosatti @ 2004-12-21 16:46 UTC (permalink / raw)
To: Manfred Schwarb; +Cc: linux-kernel, reiserfs-dev, Trond Myklebust
CCing the ReiserFS devel list because they might have some clue.
On Mon, Dec 20, 2004 at 11:28:52PM +0100, Manfred Schwarb wrote:
> Hi,
> yesterday, I got at the start of a rsync backup over nfs
> an Oops in find_inode/reiserfs_find_actor.
> Some googling revealed that this seems to be an
> inode list corruption, but how can this happen?
> There seem to be quite some reports about similar problems,
> so doing rsync over nfs-mounted partitions might be a bad idea?
No, it should work.
> Is there anything I can do? Any help is appreciated.
>
> This machine is an athlon i386 with vanilla 2.4.29-pre2.
> Below you may find the ksymoopsified Oops:
>
>
> Dec 19 21:00:01 kernel: Unable to handle kernel NULL pointer dereference at
> virtual address 00000028
> Dec 19 21:00:01 kernel: c0153b09
> Dec 19 21:00:01 kernel: *pde = 1d678067
> Dec 19 21:00:01 kernel: Oops: 0000
> Dec 19 21:00:01 kernel: CPU: 0
> Dec 19 21:00:01 kernel: EIP: 0010:[find_inode+25/112] Not tainted
> Dec 19 21:00:01 kernel: EIP: 0010:[<c0153b09>] Not tainted
> Using defaults from ksymoops -t elf32-i386 -a i386
> Dec 19 21:00:01 kernel: EFLAGS: 00213213
> Dec 19 21:00:01 kernel: eax: e0b3c9e0 ebx: 00000000 ecx: 0000000f
> edx: dff80000
> Dec 19 21:00:01 kernel: esi: 00000000 edi: dffa2a58 ebp: 000000fa
> esp: d7f05d60
> Dec 19 21:00:01 kernel: ds: 0018 es: 0018 ss: 0018
> Dec 19 21:00:01 kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
> Dec 19 21:00:01 kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4 dffa2a58
> 000000fa da2f6000 c0153f4e
> Dec 19 21:00:01 kernel: da2f6000 000000fa dffa2a58 e0b3c9e0 d7f05db8
> d7f05de4 d7f05e48 d7f05db8
> Dec 19 21:00:01 kernel: da2f6000 e0b3ca60 da2f6000 000000fa e0b3c9e0
> d7f05db8 00000011 d7f05de4
> Dec 19 21:00:01 kernel: Call Trace: [iget4_locked+94/272]
> [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]
> [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96]
> [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]
> [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]
> Dec 19 21:00:01 kernel: Call Trace: [<c0153f4e>] [<e0b3c9e0>]
> [<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
> Dec 19 21:00:01 kernel: [<c0151d1c>] [<c01491cf>] [<c0149279>]
> [<e118fd71>] [<e1196d39>] [<e119de8c>]
> Dec 19 21:00:01 kernel: [<e118c68d>] [<c027ab3e>] [<e119de8c>]
> [<e119d758>] [<e119d778>] [<e118c3cb>]
> Dec 19 21:00:01 kernel: [<c010729b>] [<e118c210>]
> Dec 19 21:00:01 kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0 00
> 00 00 75 e5 8b
>
>
> >>EIP; c0153b09 <find_inode+19/70> <=====
>
> >>eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> >>edx; dff80000 <_end+1fbfd7f4/20792854>
> >>edi; dffa2a58 <_end+1fc2024c/20792854>
> >>esp; d7f05d60 <_end+17b83554/20792854>
>
> Trace; c0153f4e <iget4_locked+5e/110>
> Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0>
> Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120>
> Trace; c0151d1c <d_alloc+1c/1d0>
> Trace; c01491cf <lookup_hash+9f/d0>
> Trace; c0149279 <lookup_one_len+79/90>
> Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490>
> Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140>
> Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
> Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220>
> Trace; c027ab3e <svc_process+3de/590>
> Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
> Trace; e119d758 <[nfsd]nfsd_version3+0/10>
> Trace; e119d778 <[nfsd]nfsd_program+0/28>
> Trace; e118c3cb <[nfsd]nfsd+1bb/330>
> Trace; c010729b <arch_kernel_thread+2b/40>
> Trace; e118c210 <[nfsd]nfsd+0/330>
>
> Code; c0153b09 <find_inode+19/70>
> 00000000 <_EIP>:
> Code; c0153b09 <find_inode+19/70> <=====
> 0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
> Code; c0153b0c <find_inode+1c/70>
> 3: 89 de mov %ebx,%esi
> Code; c0153b0e <find_inode+1e/70>
> 5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
> Code; c0153b10 <find_inode+20/70>
> 7: 8b 44 24 20 mov 0x20(%esp,1),%eax
> Code; c0153b14 <find_inode+24/70>
> b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
> Code; c0153b1a <find_inode+2a/70>
> 11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
> Code; c0153b1c <find_inode+2c/70>
> 13: 8b 00 mov (%eax),%eax
This is indeed corruption - an inode in this hash bucket has "->next" as
NULL, so find_inode goes boom.
Something is leaving this hash bucket list corrupt.
Have you ever seen this crash before ? Can you reproduce it?
The only inode corruption case that was reliable was from Chris Caputo
and we end up agreeing that it was most likely a hardware issue, because it was
hard to reproduce and strange in several ways. Can you point me at
the "quite some similar reports" you have found, please ?
The inode number being looked up is in %ebp, which is 250.
Trond, the following issue noted by you could not generate such
corruption could it?
Here's one question:
Given the fact that in iput(), the inode remains hashed and the
inode->i_state does not change until after we've dropped the inode_lock,
called write_inode_now(), and then retaken the inode_lock, exactly what
is preventing a third party task from grabbing that inode?
(Better still: write_inode_now() itself actually calls __iget(), which
could cause that inode to be plonked right back onto the "inode_in_use"
list if ever refile_inode() gets called.)
So does the following patch help?
Cheers,
Trond
--- linux-2.4.27-pre3/fs/inode.c.orig 2004-05-20 20:41:41.000000000 -0400
+++ linux-2.4.27-pre3/fs/inode.c 2004-06-19 23:22:29.000000000 -0400
@@ -1200,6 +1200,7 @@ void iput(struct inode *inode)
struct super_block *sb = inode->i_sb;
struct super_operations *op = NULL;
+again:
if (inode->i_state == I_CLEAR)
BUG();
@@ -1241,11 +1242,16 @@ void iput(struct inode *inode)
if (!(inode->i_state & (I_DIRTY|I_LOCK)))
__refile_inode(inode);
inodes_stat.nr_unused++;
- spin_unlock(&inode_lock);
- if (!sb || (sb->s_flags & MS_ACTIVE))
+ if (!sb || (sb->s_flags & MS_ACTIVE)) {
+ spin_unlock(&inode_lock);
return;
- write_inode_now(inode, 1);
- spin_lock(&inode_lock);
+ }
+ if (inode->i_state & I_DIRTY) {
+ __iget(inode);
+ spin_unlock(&inode_lock);
+ write_inode_now(inode, 1);
+ goto again;
+ }
inodes_stat.nr_unused--;
list_del_init(&inode->i_hash);
}
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor
2004-12-21 16:46 ` Marcelo Tosatti
@ 2004-12-21 19:34 ` Trond Myklebust
2004-12-21 23:56 ` Manfred Schwarb
2004-12-21 23:56 ` Manfred Schwarb
2 siblings, 0 replies; 7+ messages in thread
From: Trond Myklebust @ 2004-12-21 19:34 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Manfred Schwarb, linux-kernel, reiserfs-dev
ty den 21.12.2004 Klokka 14:46 (-0200) skreiv Marcelo Tosatti:
> Trond, the following issue noted by you could not generate such
> corruption could it?
>
I doubt it. That bug requires the MS_ACTIVE flag to have been cleared.
IOW, you would have to be in the middle of unmounting the filesystem,
and that doesn't really square with the fact that it is being exported
by knfsd.
Cheers,
Trond
--
Trond Myklebust <trond.myklebust@fys.uio.no>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor
2004-12-21 16:46 ` Marcelo Tosatti
2004-12-21 19:34 ` Trond Myklebust
@ 2004-12-21 23:56 ` Manfred Schwarb
2004-12-22 0:59 ` Willy Tarreau
2004-12-21 23:56 ` Manfred Schwarb
2 siblings, 1 reply; 7+ messages in thread
From: Manfred Schwarb @ 2004-12-21 23:56 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: linux-kernel, reiserfs-dev, trond.myklebust
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="us-ascii", Size: 6285 bytes --]
>
> CCing the ReiserFS devel list because they might have some clue.
>
> On Mon, Dec 20, 2004 at 11:28:52PM +0100, Manfred Schwarb wrote:
>
> > Hi,
> > yesterday, I got at the start of a rsync backup over nfs
> > an Oops in find_inode/reiserfs_find_actor.
> > Some googling revealed that this seems to be an
> > inode list corruption, but how can this happen?
> > There seem to be quite some reports about similar problems,
> > so doing rsync over nfs-mounted partitions might be a bad idea?
>
> No, it should work.
>
> > Is there anything I can do? Any help is appreciated.
> >
> > This machine is an athlon i386 with vanilla 2.4.29-pre2.
> > Below you may find the ksymoopsified Oops:
> >
> >
> > Dec 19 21:00:01 kernel: Unable to handle kernel NULL pointer
> dereference at
> > virtual address 00000028
> > Dec 19 21:00:01 kernel: c0153b09
> > Dec 19 21:00:01 kernel: *pde = 1d678067
> > Dec 19 21:00:01 kernel: Oops: 0000
> > Dec 19 21:00:01 kernel: CPU: 0
> > Dec 19 21:00:01 kernel: EIP: 0010:[find_inode+25/112] Not tainted
> > Dec 19 21:00:01 kernel: EIP: 0010:[<c0153b09>] Not tainted
> > Using defaults from ksymoops -t elf32-i386 -a i386
> > Dec 19 21:00:01 kernel: EFLAGS: 00213213
> > Dec 19 21:00:01 kernel: eax: e0b3c9e0 ebx: 00000000 ecx: 0000000f
> > edx: dff80000
> > Dec 19 21:00:01 kernel: esi: 00000000 edi: dffa2a58 ebp: 000000fa
> > esp: d7f05d60
> > Dec 19 21:00:01 kernel: ds: 0018 es: 0018 ss: 0018
> > Dec 19 21:00:01 kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
> > Dec 19 21:00:01 kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4
> dffa2a58
> > 000000fa da2f6000 c0153f4e
> > Dec 19 21:00:01 kernel: da2f6000 000000fa dffa2a58 e0b3c9e0
> d7f05db8
> > d7f05de4 d7f05e48 d7f05db8
> > Dec 19 21:00:01 kernel: da2f6000 e0b3ca60 da2f6000 000000fa
> e0b3c9e0
> > d7f05db8 00000011 d7f05de4
> > Dec 19 21:00:01 kernel: Call Trace: [iget4_locked+94/272]
> >
>
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]
> >
>
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96]
> >
>
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]
> >
>
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]
> > Dec 19 21:00:01 kernel: Call Trace: [<c0153f4e>] [<e0b3c9e0>]
> > [<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
> > Dec 19 21:00:01 kernel: [<c0151d1c>] [<c01491cf>] [<c0149279>]
> > [<e118fd71>] [<e1196d39>] [<e119de8c>]
> > Dec 19 21:00:01 kernel: [<e118c68d>] [<c027ab3e>] [<e119de8c>]
> > [<e119d758>] [<e119d778>] [<e118c3cb>]
> > Dec 19 21:00:01 kernel: [<c010729b>] [<e118c210>]
> > Dec 19 21:00:01 kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0
> 00
> > 00 00 75 e5 8b
> >
> >
> > >>EIP; c0153b09 <find_inode+19/70> <=====
> >
> > >>eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > >>edx; dff80000 <_end+1fbfd7f4/20792854>
> > >>edi; dffa2a58 <_end+1fc2024c/20792854>
> > >>esp; d7f05d60 <_end+17b83554/20792854>
> >
> > Trace; c0153f4e <iget4_locked+5e/110>
> > Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0>
> > Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120>
> > Trace; c0151d1c <d_alloc+1c/1d0>
> > Trace; c01491cf <lookup_hash+9f/d0>
> > Trace; c0149279 <lookup_one_len+79/90>
> > Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490>
> > Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140>
> > Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
> > Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220>
> > Trace; c027ab3e <svc_process+3de/590>
> > Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
> > Trace; e119d758 <[nfsd]nfsd_version3+0/10>
> > Trace; e119d778 <[nfsd]nfsd_program+0/28>
> > Trace; e118c3cb <[nfsd]nfsd+1bb/330>
> > Trace; c010729b <arch_kernel_thread+2b/40>
> > Trace; e118c210 <[nfsd]nfsd+0/330>
> >
> > Code; c0153b09 <find_inode+19/70>
> > 00000000 <_EIP>:
> > Code; c0153b09 <find_inode+19/70> <=====
> > 0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
> > Code; c0153b0c <find_inode+1c/70>
> > 3: 89 de mov %ebx,%esi
> > Code; c0153b0e <find_inode+1e/70>
> > 5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
> > Code; c0153b10 <find_inode+20/70>
> > 7: 8b 44 24 20 mov 0x20(%esp,1),%eax
> > Code; c0153b14 <find_inode+24/70>
> > b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
> > Code; c0153b1a <find_inode+2a/70>
> > 11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
> > Code; c0153b1c <find_inode+2c/70>
> > 13: 8b 00 mov (%eax),%eax
>
> This is indeed corruption - an inode in this hash bucket has "->next" as
> NULL, so find_inode goes boom.
>
> Something is leaving this hash bucket list corrupt.
>
> Have you ever seen this crash before ? Can you reproduce it?
>
No, not at all. This machine was running for 10 months with 2.4.xx
kernels without any problems. Since this oops, I tried to reproduce
the particular situation (rsync over nfs, and some additional load),
put I had no success in crashing the box:
I mirrored the box (ca. 1 million files) 5 times, no problem.
> The only inode corruption case that was reliable was from Chris Caputo
> and we end up agreeing that it was most likely a hardware issue, because
> it was
> hard to reproduce and strange in several ways. Can you point me at
> the "quite some similar reports" you have found, please ?
>
Just my first impression, a closer look showed that most of the
cases group around 2.4.1[789] because of lacking reiserfs_find_actor.
Sorry for the overstatement.
A recent oops report shows some similarity, using a 2.6.7 kernel:
http://marc.theaimsgroup.com/?l=linux-kernel&m=109278828905885&w=2
Hardware issue: you mean memory? Last winter I ran memtest86
during a weekend, everything was fine. At the moment I can't
take this box offline for a longer period to test again, so I
tend to belive memory is ok, and knock on wood...
--
+++ Sparen Sie mit GMX DSL +++ http://www.gmx.net/de/go/dsl
AKTION für Wechsler: DSL-Tarife ab 3,99 EUR/Monat + Startguthaben
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor
2004-12-21 16:46 ` Marcelo Tosatti
2004-12-21 19:34 ` Trond Myklebust
2004-12-21 23:56 ` Manfred Schwarb
@ 2004-12-21 23:56 ` Manfred Schwarb
2004-12-28 11:24 ` Marcelo Tosatti
2 siblings, 1 reply; 7+ messages in thread
From: Manfred Schwarb @ 2004-12-21 23:56 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: linux-kernel, reiserfs-dev, trond.myklebust
>
> CCing the ReiserFS devel list because they might have some clue.
>
> On Mon, Dec 20, 2004 at 11:28:52PM +0100, Manfred Schwarb wrote:
>
> > Hi,
> > yesterday, I got at the start of a rsync backup over nfs
> > an Oops in find_inode/reiserfs_find_actor.
> > Some googling revealed that this seems to be an
> > inode list corruption, but how can this happen?
> > There seem to be quite some reports about similar problems,
> > so doing rsync over nfs-mounted partitions might be a bad idea?
>
> No, it should work.
>
> > Is there anything I can do? Any help is appreciated.
> >
> > This machine is an athlon i386 with vanilla 2.4.29-pre2.
> > Below you may find the ksymoopsified Oops:
> >
> >
> > Dec 19 21:00:01 kernel: Unable to handle kernel NULL pointer
> dereference at
> > virtual address 00000028
> > Dec 19 21:00:01 kernel: c0153b09
> > Dec 19 21:00:01 kernel: *pde = 1d678067
> > Dec 19 21:00:01 kernel: Oops: 0000
> > Dec 19 21:00:01 kernel: CPU: 0
> > Dec 19 21:00:01 kernel: EIP: 0010:[find_inode+25/112] Not tainted
> > Dec 19 21:00:01 kernel: EIP: 0010:[<c0153b09>] Not tainted
> > Using defaults from ksymoops -t elf32-i386 -a i386
> > Dec 19 21:00:01 kernel: EFLAGS: 00213213
> > Dec 19 21:00:01 kernel: eax: e0b3c9e0 ebx: 00000000 ecx: 0000000f
> > edx: dff80000
> > Dec 19 21:00:01 kernel: esi: 00000000 edi: dffa2a58 ebp: 000000fa
> > esp: d7f05d60
> > Dec 19 21:00:01 kernel: ds: 0018 es: 0018 ss: 0018
> > Dec 19 21:00:01 kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
> > Dec 19 21:00:01 kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4
> dffa2a58
> > 000000fa da2f6000 c0153f4e
> > Dec 19 21:00:01 kernel: da2f6000 000000fa dffa2a58 e0b3c9e0
> d7f05db8
> > d7f05de4 d7f05e48 d7f05db8
> > Dec 19 21:00:01 kernel: da2f6000 e0b3ca60 da2f6000 000000fa
> e0b3c9e0
> > d7f05db8 00000011 d7f05de4
> > Dec 19 21:00:01 kernel: Call Trace: [iget4_locked+94/272]
> >
>
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]
> >
>
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96]
> >
>
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]
> >
>
[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]
> > Dec 19 21:00:01 kernel: Call Trace: [<c0153f4e>] [<e0b3c9e0>]
> > [<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
> > Dec 19 21:00:01 kernel: [<c0151d1c>] [<c01491cf>] [<c0149279>]
> > [<e118fd71>] [<e1196d39>] [<e119de8c>]
> > Dec 19 21:00:01 kernel: [<e118c68d>] [<c027ab3e>] [<e119de8c>]
> > [<e119d758>] [<e119d778>] [<e118c3cb>]
> > Dec 19 21:00:01 kernel: [<c010729b>] [<e118c210>]
> > Dec 19 21:00:01 kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0
> 00
> > 00 00 75 e5 8b
> >
> >
> > >>EIP; c0153b09 <find_inode+19/70> <=====
> >
> > >>eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > >>edx; dff80000 <_end+1fbfd7f4/20792854>
> > >>edi; dffa2a58 <_end+1fc2024c/20792854>
> > >>esp; d7f05d60 <_end+17b83554/20792854>
> >
> > Trace; c0153f4e <iget4_locked+5e/110>
> > Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0>
> > Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120>
> > Trace; c0151d1c <d_alloc+1c/1d0>
> > Trace; c01491cf <lookup_hash+9f/d0>
> > Trace; c0149279 <lookup_one_len+79/90>
> > Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490>
> > Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140>
> > Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
> > Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220>
> > Trace; c027ab3e <svc_process+3de/590>
> > Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
> > Trace; e119d758 <[nfsd]nfsd_version3+0/10>
> > Trace; e119d778 <[nfsd]nfsd_program+0/28>
> > Trace; e118c3cb <[nfsd]nfsd+1bb/330>
> > Trace; c010729b <arch_kernel_thread+2b/40>
> > Trace; e118c210 <[nfsd]nfsd+0/330>
> >
> > Code; c0153b09 <find_inode+19/70>
> > 00000000 <_EIP>:
> > Code; c0153b09 <find_inode+19/70> <=====
> > 0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
> > Code; c0153b0c <find_inode+1c/70>
> > 3: 89 de mov %ebx,%esi
> > Code; c0153b0e <find_inode+1e/70>
> > 5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
> > Code; c0153b10 <find_inode+20/70>
> > 7: 8b 44 24 20 mov 0x20(%esp,1),%eax
> > Code; c0153b14 <find_inode+24/70>
> > b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
> > Code; c0153b1a <find_inode+2a/70>
> > 11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
> > Code; c0153b1c <find_inode+2c/70>
> > 13: 8b 00 mov (%eax),%eax
>
> This is indeed corruption - an inode in this hash bucket has "->next" as
> NULL, so find_inode goes boom.
>
> Something is leaving this hash bucket list corrupt.
>
> Have you ever seen this crash before ? Can you reproduce it?
>
No, not at all. This machine was running for 10 months with 2.4.xx
kernels without any problems. Since this oops, I tried to reproduce
the particular situation (rsync over nfs, and some additional load),
put I had no success in crashing the box:
I mirrored the box (ca. 1 million files) 5 times, no problem.
> The only inode corruption case that was reliable was from Chris Caputo
> and we end up agreeing that it was most likely a hardware issue, because
> it was
> hard to reproduce and strange in several ways. Can you point me at
> the "quite some similar reports" you have found, please ?
>
Just my first impression, a closer look showed that most of the
cases group around 2.4.1[789] because of lacking reiserfs_find_actor.
Sorry for the overstatement.
A recent oops report shows some similarity, using a 2.6.7 kernel:
http://marc.theaimsgroup.com/?l=linux-kernel&m=109278828905885&w=2
Hardware issue: you mean memory? Last winter I ran memtest86
during a weekend, everything was fine. At the moment I can't
take this box offline for a longer period to test again, so I
tend to belive memory is ok, and knock on wood...
--
Psssst! Mit GMX Handyrechnung senken: http://www.gmx.net/de/go/mail
100 FreeSMS/Monat (GMX TopMail), 50 (GMX ProMail), 10 (GMX FreeMail)
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor
2004-12-21 23:56 ` Manfred Schwarb
@ 2004-12-22 0:59 ` Willy Tarreau
0 siblings, 0 replies; 7+ messages in thread
From: Willy Tarreau @ 2004-12-22 0:59 UTC (permalink / raw)
To: Manfred Schwarb
Cc: Marcelo Tosatti, linux-kernel, reiserfs-dev, trond.myklebust
Hi,
On Wed, Dec 22, 2004 at 12:56:05AM +0100, Manfred Schwarb wrote:
> No, not at all. This machine was running for 10 months with 2.4.xx
> kernels without any problems. Since this oops, I tried to reproduce
> the particular situation (rsync over nfs, and some additional load),
> put I had no success in crashing the box:
> I mirrored the box (ca. 1 million files) 5 times, no problem.
(...)
> Hardware issue: you mean memory? Last winter I ran memtest86
> during a weekend, everything was fine. At the moment I can't
> take this box offline for a longer period to test again, so I
> tend to belive memory is ok, and knock on wood...
Does this machine have ECC memory ? If so, have you tried to check the
error counters ? And if not, well, it might be some random bit flips
in memory.
Last year, I encountered a really strange situation. During about one week,
I had 3 or 4 machines that have panic'd several times each, and sometimes
only some multi-threaded apps would freeze (eg: pdnsd). These machines
were installed at different customers' in different locations, and at
different times. They had never crashed before, nor did they after. Most
other similar machines at other customer's, or some at the sames did not
experience this. This was during the first week of november, when there
were the very strong solar flares. I never told the customers about my
thoughts on the subject, they would have definitely made fun of me. So
we replaced the RAM sticks, to report something more believable...
Regards,
Willy
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor
2004-12-21 23:56 ` Manfred Schwarb
@ 2004-12-28 11:24 ` Marcelo Tosatti
0 siblings, 0 replies; 7+ messages in thread
From: Marcelo Tosatti @ 2004-12-28 11:24 UTC (permalink / raw)
To: Manfred Schwarb; +Cc: linux-kernel, reiserfs-dev, trond.myklebust, Hans Reiser
On Wed, Dec 22, 2004 at 12:56:06AM +0100, Manfred Schwarb wrote:
> > > >>EIP; c0153b09 <find_inode+19/70> <=====
> > >
> > > >>eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > > >>edx; dff80000 <_end+1fbfd7f4/20792854>
> > > >>edi; dffa2a58 <_end+1fc2024c/20792854>
> > > >>esp; d7f05d60 <_end+17b83554/20792854>
> > >
> > > Trace; c0153f4e <iget4_locked+5e/110>
> > > Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > > Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0>
> > > Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40>
> > > Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120>
> > > Trace; c0151d1c <d_alloc+1c/1d0>
> > > Trace; c01491cf <lookup_hash+9f/d0>
> > > Trace; c0149279 <lookup_one_len+79/90>
> > > Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490>
> > > Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140>
> > > Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
> > > Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220>
> > > Trace; c027ab3e <svc_process+3de/590>
> > > Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320>
> > > Trace; e119d758 <[nfsd]nfsd_version3+0/10>
> > > Trace; e119d778 <[nfsd]nfsd_program+0/28>
> > > Trace; e118c3cb <[nfsd]nfsd+1bb/330>
> > > Trace; c010729b <arch_kernel_thread+2b/40>
> > > Trace; e118c210 <[nfsd]nfsd+0/330>
> > >
> > > Code; c0153b09 <find_inode+19/70>
> > > 00000000 <_EIP>:
> > > Code; c0153b09 <find_inode+19/70> <=====
> > > 0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
> > > Code; c0153b0c <find_inode+1c/70>
> > > 3: 89 de mov %ebx,%esi
> > > Code; c0153b0e <find_inode+1e/70>
> > > 5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
> > > Code; c0153b10 <find_inode+20/70>
> > > 7: 8b 44 24 20 mov 0x20(%esp,1),%eax
> > > Code; c0153b14 <find_inode+24/70>
> > > b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
> > > Code; c0153b1a <find_inode+2a/70>
> > > 11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
> > > Code; c0153b1c <find_inode+2c/70>
> > > 13: 8b 00 mov (%eax),%eax
> >
> > This is indeed corruption - an inode in this hash bucket has "->next" as
> > NULL, so find_inode goes boom.
> >
> > Something is leaving this hash bucket list corrupt.
> >
> > Have you ever seen this crash before ? Can you reproduce it?
> >
>
> No, not at all. This machine was running for 10 months with 2.4.xx
> kernels without any problems. Since this oops, I tried to reproduce
> the particular situation (rsync over nfs, and some additional load),
> put I had no success in crashing the box:
> I mirrored the box (ca. 1 million files) 5 times, no problem.
>
>
> > The only inode corruption case that was reliable was from Chris Caputo
> > and we end up agreeing that it was most likely a hardware issue, because
> > it was
> > hard to reproduce and strange in several ways. Can you point me at
> > the "quite some similar reports" you have found, please ?
> >
>
> Just my first impression, a closer look showed that most of the
> cases group around 2.4.1[789] because of lacking reiserfs_find_actor.
> Sorry for the overstatement.
>
> A recent oops report shows some similarity, using a 2.6.7 kernel:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=109278828905885&w=2
>
>
> Hardware issue: you mean memory? Last winter I ran memtest86
> during a weekend, everything was fine. At the moment I can't
> take this box offline for a longer period to test again, so I
> tend to belive memory is ok, and knock on wood...
Yes, what I'm saying is that no reliable inode corruption case has been
reported recently, except when hardware was flaky (usually memory errors).
I'm not saying that this is your case - its just one explanation to the
problem. It might well be a software problem.
Hans, did any of your developers see similar inode cache hashtable corruption
in v2.4.x?
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2004-12-28 14:21 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-12-20 22:28 2.4.29-pre2 Oops at find_inode/reiserfs_find_actor Manfred Schwarb
2004-12-21 16:46 ` Marcelo Tosatti
2004-12-21 19:34 ` Trond Myklebust
2004-12-21 23:56 ` Manfred Schwarb
2004-12-22 0:59 ` Willy Tarreau
2004-12-21 23:56 ` Manfred Schwarb
2004-12-28 11:24 ` Marcelo Tosatti
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox