Re: Reiserfs 3.6 + quota enabled, crash on delete (or maybe truncate)

Re: Reiserfs 3.6 + quota enabled, crash on delete (or maybe truncate)

[Guillaume Pelat]

Hi,

 >> I'm having a crash with reiserfs 3.6 + user quota enabled, on
 >> 2.6.11.10 kernel (no smp), apparently when deleting files (or maybe
 >>  during a runcate operation). The problem seems to happen under high
 >>  load.
 >> When the error occurs, all the processes accessing the reiserfs
 >> partition seems to hang. This problem happened several times on
 >> different servers (having the same hardware configuration) during
 >> last weeks.

 > The attached patch should help to get rid of clm-2100 and to avoid
 > crash.
 > Also, I think you should reiserfsck sda3.

Thanks for your answer. I tried the patch attached to your mail, but it 
seems that it doesnt solve my problem :)
The error also occured after i did a reiserfsck on sda3, I also tested 
2.6.13-rc4 with your patch applied, without success (the error also 
occured on serveral servers having the same hardware configuration).

Here are the error logs:

------------[ cut here ]------------
kernel BUG at fs/reiserfs/prints.c:362!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c019ae2f>]    Not tainted VLI
EFLAGS: 00010296   (2.6.13-rc4-endy)
EIP is at reiserfs_panic+0x4f/0x80
eax: 00000053   ebx: c02b8fde   ecx: 00000000   edx: c02dfdac
esi: 00000000   edi: 00000140   ebp: e75b383c   esp: e75b3824
ds: 007b   es: 007b   ss: 0068
Process pure-ftpd (pid: 12771, threadinfo=e75b2000 task=f091d530)
Stack: c02bd610 c02b8fde c03acdc0 00000fa0 c0971154 00000002 e75b3864 
c01ac75f
        00000000 c02bf89c 00000fa0 00000002 00020000 00000000 c097101c 
00000000
        e75b38b8 c01939d3 c097101c 00000fd0 00000000 00000000 00000000 
00000000
Call Trace:
  [<c0102e5f>] show_stack+0x7f/0xa0
  [<c0103002>] show_registers+0x152/0x1c0
  [<c01031f8>] die+0xc8/0x140
  [<c0103325>] do_trap+0xb5/0xc0
  [<c010366c>] do_invalid_op+0xbc/0xd0
  [<c0102aa3>] error_code+0x4f/0x54
  [<c01ac75f>] direntry_check_left+0x8f/0x90
  [<c01939d3>] get_num_ver+0x303/0x350
  [<c01949ac>] ip_check_balance+0x3dc/0xbc0
  [<c0195948>] check_balance+0x58/0x70
  [<c019623b>] fix_nodes+0x15b/0x420
  [<c01a2daf>] reiserfs_cut_from_item+0x10f/0x570
  [<c01a359b>] reiserfs_do_truncate+0x2db/0x5e0
  [<c01a282f>] reiserfs_delete_object+0x3f/0x80
  [<c0189baf>] reiserfs_delete_inode+0xaf/0x150
  [<c0161835>] generic_delete_inode+0x95/0x130
  [<c0161a18>] generic_drop_inode+0x18/0x30
  [<c0161a86>] iput+0x56/0x80
  [<c018d07d>] reiserfs_new_inode+0x16d/0x7e0
  [<c0187d31>] reiserfs_create+0xc1/0x1f0
  [<c0156a4f>] vfs_create+0x9f/0x120
  [<c015732c>] open_namei+0x5cc/0x620
  [<c0146eac>] filp_open+0x3c/0x60
  [<c01471c5>] sys_open+0x55/0x90
  [<c0102889>] syscall_call+0x7/0xb
Code: 01 00 00 89 04 24 e8 31 fd ff ff c7 04 24 10 d6 2b c0 85 f6 89 d8 
0f 45 c7
ba c0 cd 3a c0 89
54 24 08 89 44 24 04 e8 21 80 f7 ff <0f> 0b 6a 01 2f 95 2b c0 c7 04 24 
34 d6 2b
c0 85 f6 be c0 cd 3
a
  Badness in do_exit at kernel/exit.c:787
  [<c0102e9e>] dump_stack+0x1e/0x30
  [<c0114fdc>] do_exit+0x2ec/0x300
  [<c010326f>] die+0x13f/0x140
  [<c0103325>] do_trap+0xb5/0xc0
  [<c010366c>] do_invalid_op+0xbc/0xd0
  [<c0102aa3>] error_code+0x4f/0x54
  [<c01ac75f>] direntry_check_left+0x8f/0x90
  [<c01939d3>] get_num_ver+0x303/0x350
  [<c01949ac>] ip_check_balance+0x3dc/0xbc0
  [<c0195948>] check_balance+0x58/0x70
  [<c019623b>] fix_nodes+0x15b/0x420
  [<c01a2daf>] reiserfs_cut_from_item+0x10f/0x570
  [<c01a359b>] reiserfs_do_truncate+0x2db/0x5e0
  [<c01a282f>] reiserfs_delete_object+0x3f/0x80
  [<c0189baf>] reiserfs_delete_inode+0xaf/0x150
  [<c0161835>] generic_delete_inode+0x95/0x130
  [<c0161a18>] generic_drop_inode+0x18/0x30
  [<c0161a86>] iput+0x56/0x80
  [<c018d07d>] reiserfs_new_inode+0x16d/0x7e0
  [<c0187d31>] reiserfs_create+0xc1/0x1f0
  [<c0156a4f>] vfs_create+0x9f/0x120
  [<c015732c>] open_namei+0x5cc/0x620
  [<c0146eac>] filp_open+0x3c/0x60
  [<c01471c5>] sys_open+0x55/0x90
  [<c0102889>] syscall_call+0x7/0xb

Best Regards,

Guillaume Pelat

Re: Reiserfs 3.6 + quota enabled, crash on delete (or maybe truncate)

[Vladimir V. Saveliev]

[-- Attachment #1: Type: text/plain, Size: 2488 bytes --]

Hello

Guillaume Pelat wrote:
> Hi,
> 
>  >> I'm having a crash with reiserfs 3.6 + user quota enabled, on
>  >> 2.6.11.10 kernel (no smp), apparently when deleting files (or maybe
>  >>  during a runcate operation). The problem seems to happen under high
>  >>  load.
>  >> When the error occurs, all the processes accessing the reiserfs
>  >> partition seems to hang. This problem happened several times on
>  >> different servers (having the same hardware configuration) during
>  >> last weeks.
> 
>  > The attached patch should help to get rid of clm-2100 and to avoid
>  > crash.
>  > Also, I think you should reiserfsck sda3.
> 
> Thanks for your answer. I tried the patch attached to your mail, but it 
> seems that it doesnt solve my problem :)
> The error also occured after i did a reiserfsck on sda3, I also tested 
> 2.6.13-rc4 with your patch applied, without success (the error also 
> occured on serveral servers having the same hardware configuration).
> 

Would you, please, try to reproduce the problem having reiserfs check mode on.
(it is File systems->Reiserfs support->Enable reiserfs debug mode in kernel configuration)
and with attached patch.

> Here are the error logs:
> 

Was there anything about reiserfs in the logs before this dump?

> ------------[ cut here ]------------
> kernel BUG at fs/reiserfs/prints.c:362!
> invalid operand: 0000 [#1]
> CPU:    0
> EIP:    0060:[<c019ae2f>]    Not tainted VLI
> EFLAGS: 00010296   (2.6.13-rc4-endy)
> EIP is at reiserfs_panic+0x4f/0x80
> eax: 00000053   ebx: c02b8fde   ecx: 00000000   edx: c02dfdac
> esi: 00000000   edi: 00000140   ebp: e75b383c   esp: e75b3824
> ds: 007b   es: 007b   ss: 0068
> Process pure-ftpd (pid: 12771, threadinfo=e75b2000 task=f091d530)
> Stack: c02bd610 c02b8fde c03acdc0 00000fa0 c0971154 00000002 e75b3864 
> c01ac75f
>        00000000 c02bf89c 00000fa0 00000002 00020000 00000000 c097101c 
> 00000000
>        e75b38b8 c01939d3 c097101c 00000fd0 00000000 00000000 00000000 
> 00000000
> Call Trace:
>  [<c0102e5f>] show_stack+0x7f/0xa0
>  [<c0103002>] show_registers+0x152/0x1c0
>  [<c01031f8>] die+0xc8/0x140
>  [<c0103325>] do_trap+0xb5/0xc0
>  [<c010366c>] do_invalid_op+0xbc/0xd0
>  [<c0102aa3>] error_code+0x4f/0x54
>  [<c01ac75f>] direntry_check_left+0x8f/0x90
>  [<c01939d3>] get_num_ver+0x303/0x350
>  [<c01949ac>] ip_check_balance+0x3dc/0xbc0
>  [<c0195948>] check_balance+0x58/0x70
>  [<c019623b>] fix_nodes+0x15b/0x420
>  [<c01a2daf>] reiserfs_cut_from_item+0x10f/0x570

[-- Attachment #2: reiserfs-panic-fix.patch --]
[-- Type: text/plain, Size: 694 bytes --]

 fs/reiserfs/prints.c |    3 ---
 1 files changed, 3 deletions(-)

diff -puN fs/reiserfs/prints.c~reiserfs-panic-fix fs/reiserfs/prints.c
--- linux-2.6.13-rc4-mm1/fs/reiserfs/prints.c~reiserfs-panic-fix	2005-08-05 13:36:09.947391212 +0400
+++ linux-2.6.13-rc4-mm1-vs/fs/reiserfs/prints.c	2005-08-05 13:36:23.346671407 +0400
@@ -359,9 +359,6 @@ void reiserfs_panic(struct super_block *
 	do_reiserfs_warning(fmt);
 	printk(KERN_EMERG "REISERFS: panic (device %s): %s\n",
 	       reiserfs_bdevname(sb), error_buf);
-	BUG();
-
-	/* this is not actually called, but makes reiserfs_panic() "noreturn" */
 	panic("REISERFS: panic (device %s): %s\n",
 	      reiserfs_bdevname(sb), error_buf);
 }

_

Re: Reiserfs 3.6 + quota enabled, crash on delete (or maybe truncate)

[Guillaume Pelat]

Hi,

> Would you, please, try to reproduce the problem having reiserfs check 
> mode on.
> (it is File systems->Reiserfs support->Enable reiserfs debug mode in 
> kernel configuration)
> and with attached patch.

Here is the error log with reiserfs check mode on + patch applied :

ReiserFS: sda3: found reiserfs format "3.6" with standard journal
ReiserFS: sda3: warning: CONFIG_REISERFS_CHECK is set ON
ReiserFS: sda3: warning: - it is slow mode for debugging.
ReiserFS: sda3: using ordered data mode
ReiserFS: sda3: journal params: device sda3, size 8192, journal first 
block 18, max trans len 1024, max batch 900, max commit age 30, max 
trans age 30
ReiserFS: sda3: checking transaction log (sda3)
ReiserFS: sda3: journal-1153: found in header: first_unflushed_offset 
4607, last_flushed_trans_id 401988
ReiserFS: sda3: journal-1206: Starting replay from offset 
1726529608356351, trans_id 0
ReiserFS: sda3: journal-1299: Setting newest_mount_id to 23
ReiserFS: sda3: Using r5 hash to sort names
ReiserFS: sda3: warning: vs-8301: reiserfs_kmalloc: allocated memory 202992
[..a few days later..]
REISERFS: panic (device Null superblock): vs-8025: set_entry_sizes: 
(mode==c, insert_size==-4958), invalid length of directory item
Kernel panic - not syncing: REISERFS: panic (device Null superblock): 
vs-8025: set_entry_sizes: (mode==c, insert_size==-4958
), invalid length of directory item

The partition had just been checked with reiserfsck (2 days before) and 
it was ok. I didnt reboot between the reiserfsck and the crash.

Here was the result of reiserfsck before the crash:
Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
finished
No corruptions found
There are on the filesystem:
         Leaves 423085
         Internal nodes 2932
         Directories 1046685
         Other files 8739829
         Data block pointers 75038187 (0 of them are zero)
         Safe links 0
###########

Btw, i forgot to mention the mount options:
noatime,notail,usrquota


I just applied the patch submitted by Jan Kara:
http://bugzilla.kernel.org/show_bug.cgi?id=4771#c3
I dont know yet if it solves the problem :)

Best Regards,

Guillaume Pelat

Re: Reiserfs 3.6 + quota enabled, crash on delete (or maybe truncate)

[Jan Kara]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

  Hi,

> I just applied the patch submitted by Jan Kara:
> http://bugzilla.kernel.org/show_bug.cgi?id=4771#c3
> I dont know yet if it solves the problem :)
  I actually should cure your problem but can have some unexpected
sideffects. Please try an attached patch (it's the new one I posted 
to bugzilla) - that should be a better fix.

								Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

[-- Attachment #2: reiser-2.6.13-rc6-create_fix.diff --]
[-- Type: text/plain, Size: 821 bytes --]

Initialize key object ID in inode so that we don't try to remove the inode
when we fail on some checks even before we manage to allocate something.

Signed-off-by: Jan Kara <jack@suse.cz>

diff -rupX /home/jack/.kerndiffexclude linux-2.6.13-rc6/fs/reiserfs/namei.c linux-2.6.13-rc6-reiser_create_fix/fs/reiserfs/namei.c
--- linux-2.6.13-rc6/fs/reiserfs/namei.c	2005-08-12 10:39:25.000000000 +0200
+++ linux-2.6.13-rc6-reiser_create_fix/fs/reiserfs/namei.c	2005-08-12 10:39:07.000000000 +0200
@@ -593,6 +593,9 @@ static int new_inode_init(struct inode *
 	 */
 	inode->i_uid = current->fsuid;
 	inode->i_mode = mode;
+	/* Make inode invalid - just in case we are going to drop it before
+	 * the initialization happens */
+	INODE_PKEY(inode)->k_objectid = 0;
 
 	if (dir->i_mode & S_ISGID) {
 		inode->i_gid = dir->i_gid;