XFS crash on linux raid

XFS crash on linux raid

[Emmanuel Florac]

Hello, 
Apparently quite a lot of people do encounter the same problem from
time to time, but I couldn't find any solution. 

When writing quite a lot to the filesystem (heavy load on the
fileserver), the filesystem crashes when filled at 2.5~3TB (varies from
time to time). The filesystems tested where always running on a software
raid 0, with disabled barriers. I tend to think that disabled write
barriers are causing the crash but I'll do some more tests to get sure.

I've met this problem for the first time on 12/23 (yup... merry
christmas :) when a 13 TB filesystem went belly up :

Dec 23 01:38:10 storiq1 -- MARK --
Dec 23 01:58:10 storiq1 -- MARK --
Dec 23 02:10:29 storiq1 kernel: xfs_iunlink_remove: xfs_itobp()
returned an error 990 on md0. Returning error.
Dec 23 02:10:29 storiq1 kernel: xfs_inactive:^Ixfs_ifree() returned an
error = 990 on md0
Dec 23 02:10:29 storiq1 kernel: xfs_force_shutdown(md0,0x1) called from
line 1763 of file fs/xfs/xfs_vnodeops.c. Return address = 0xc027f78b
Dec 23 02:38:11 storiq1 -- MARK --
Dec 23 02:58:11 storiq1 -- MARK -- 


When mounting, it did that :

Filesystem "md0": Disabling barriers, not supported by the underlying
device XFS mounting filesystem md0
Starting XFS recovery on filesystem: md0 (logdev: internal)
Filesystem "md0": xfs_inode_recover: Bad inode magic number, dino ptr =
0xf7196600, dino bp = 0xf718e980, ino = 119318 Filesystem "md0": XFS
internal error xlog_recover_do_inode_trans(1) at line 2352 of file
fs/xfs/xfs_log_recover.c. Caller 0xc025d180 <c025cb9d>
xlog_recover_do_inode_trans+0x93d/0xa00 <c025d180>
xlog_recover_do_trans+0x140/0x160 <c0277afb>
xfs_buf_delwri_queue+0x2b/0xb0 <c025d180>
xlog_recover_do_trans+0x140/0x160 <c027385f> kmem_zalloc+0x1f/0x50
<c025d27f> xlog_recover_commit_trans+0x3f/0x50 <c025d39a>
xlog_recover_process_data+0xea/0x240 <c025e48a>
xlog_do_recovery_pass+0x39a/0xb70 <c013d189>
hrtimer_run_queues+0x29/0x110 <c025ecf6>
xlog_do_log_recovery+0x96/0xd0 <c025ed6b> xlog_do_recover+0x3b/0x170
<c025ef7d> xlog_recover+0xdd/0xf0 <c0255da1> xfs_log_mount+0xa1/0x110
<c02607e5> xfs_mountfs+0x825/0xf30 <c02420c7> xfs_fs_cmn_err+0x27/0x30
<c0251137> xfs_ioinit+0x27/0x50 <c02693ef> xfs_mount+0x2ff/0x520
<c027f3f3> vfs_mount+0x43/0x50 <c027f20a> xfs_fs_fill_super+0x9a/0x200
<c013e42d> debug_mutex_add_waiter+0x3d/0xd0 <c029eb07>
snprintf+0x27/0x30 <c01a9204> disk_name+0xb4/0xc0 <c01727bf>
sb_set_blocksize+0x1f/0x50 <c01721a6> get_sb_bdev+0x106/0x150
<c027f3a0> xfs_fs_get_sb+0x30/0x40 <c027f170>
xfs_fs_fill_super+0x0/0x200 <c01723ff> do_kern_mount+0x5f/0xe0
<c0189e57> do_new_mount+0x77/0xc0 <c018a4ad> do_mount+0x18d/0x1f0
<c014007b> take_cpu_down+0xb/0x20 <c018a2c3>
copy_mount_options+0x63/0xc0 <c018a85f> sys_mount+0x9f/0xe0 <c01031d7>
syscall_call+0x7/0xb XFS: log mount/recovery failed: error 990 XFS: log
mount failed 

XFS_repair (too old a version...) hosed the filesystem and destroyed
most of the 2.6TB of data. Yes, there were no backup, I wrote a recovery
tool to restore the video data from the raw device but the is a
different story.

The system was running vanilla 2.6.17.9, and md0 was made of 3 striped
RAID-5 on 3 3Ware-9550 cards, each hardware RAID-5 made of 8 750 GB
drives.

On a similar hardware with 2 3Ware-9550 16x750GB striped together, but
running 2.6.17.13, I had a similar fs crash last week. Unfortunately I
don't have the logs at hand, but we where able to reproduce several
times the crash at home :

Filesystem "md0": XFS internal error xfs_btree_check_sblock at line 336
of file fs/xfs/xfs_btree.c.  Caller 0xc01fb282 <c0214568>
xfs_btree_check_sblock+0x58/0xe0  <c01fb282>
xfs_alloc_lookup+0x142/0x400 <c01fb282> xfs_alloc_lookup+0x142/0x400
<c0254e59> kmem_zone_alloc+0x59/0xd0 <c02149e3>
xfs_btree_init_cursor+0x23/0x190  <c01f7834>
xfs_alloc_ag_vextent_near+0x54/0x9e0 <c0204893>
xfs_bmap_add_extent+0x383/0x430  <c020a9e6>
xfs_bmap_search_multi_extents+0x76/0xf0 <c01f7619>
xfs_alloc_ag_vextent+0x119/0x120  <c01f9b1b>
xfs_alloc_vextent+0x3db/0x4f0 <c0208e4e> xfs_bmap_btalloc+0x3ee/0x890
<c020cf96> xfs_bmapi+0x1216/0x1690 <c021b2e6>
xfs_dir2_grow_inode+0xf6/0x400  <c015f636>
cache_alloc_refill+0xb6/0x1e0 <c023263b> xfs_idata_realloc+0x3b/0x130
<c021cebc> xfs_dir2_sf_to_block+0xac/0x5d0 <c021ad69>
xfs_dir2_lookup+0x129/0x130  <c0223147> xfs_dir2_sf_addname+0x97/0x110
<c021ac34> xfs_dir2_createname+0x144/0x150  <c0249f9b>
xfs_trans_ijoin+0x2b/0x80 <c0245b74> xfs_rename+0x354/0x9f0  <c024ec6f>
xfs_access+0x3f/0x50 <c025c278> xfs_vn_rename+0x48/0xa0  <c017146c>
__link_path_walk+0xc7c/0xc90 <c024dbcf> xfs_getattr+0x23f/0x2f0
<c017e72b> mntput_no_expire+0x1b/0x80 <c015f636>
cache_alloc_refill+0xb6/0x1e0  <c0173946> vfs_rename_other+0x96/0xd0
<c0173bd8> vfs_rename+0x258/0x2d0  <c0173dc1> do_rename+0x171/0x1a0
<c015f52b> cache_grow+0x10b/0x160  <c015f636>
cache_alloc_refill+0xb6/0x1e0 <c017000b> do_getname+0x4b/0x80
<c0173e37> sys_renameat+0x47/0x80 <c0173e98> sys_rename+0x28/0x30
<c0103037> syscall_call+0x7/0xb Filesystem "md0": XFS internal error
xfs_trans_cancel at line 1150 of file fs/xfs/xfs_trans.c.  Caller
0xc0245ec7 <c0248aa0> xfs_trans_cancel+0xd0/0x100  <c0245ec7>
xfs_rename+0x6a7/0x9f0 <c0245ec7> xfs_rename+0x6a7/0x9f0  <c024ec6f>
xfs_access+0x3f/0x50 <c025c278> xfs_vn_rename+0x48/0xa0  <c017146c>
__link_path_walk+0xc7c/0xc90 <c024dbcf> xfs_getattr+0x23f/0x2f0
<c017e72b> mntput_no_expire+0x1b/0x80 <c015f636>
cache_alloc_refill+0xb6/0x1e0  <c0173946> vfs_rename_other+0x96/0xd0
<c0173bd8> vfs_rename+0x258/0x2d0  <c0173dc1> do_rename+0x171/0x1a0
<c015f52b> cache_grow+0x10b/0x160  <c015f636>
cache_alloc_refill+0xb6/0x1e0 <c017000b> do_getname+0x4b/0x80
<c0173e37> sys_renameat+0x47/0x80 <c0173e98> sys_rename+0x28/0x30
<c0103037> syscall_call+0x7/0xb xfs_force_shutdown(md0,0x8) called from
line 1151 of file fs/xfs/xfs_trans.c.  Return address = 0xc025f7b9
Filesystem "md0": Corruption of in-memory data detected.  Shutting down
filesystem: md0 Please umount the filesystem, and rectify the
problem(s) xfs_force_shutdown(md0,0x1) called from line 338 of file
fs/xfs/xfs_rw.c.  Return address = 0xc025f7b9
xfs_force_shutdown(md0,0x1) called from line 338 of file
fs/xfs/xfs_rw.c.  Return address = 0xc025f7b9

After xfs_repair, the fs is fine. However, it crashes again when
writing again a couple of GBs of data. It crashes again under 2.6.17.13,
2.6.17.13 SMP, 2.6.18.8, 2.6.16.36... 

Out of curiosity, I've tried to use reiserfs (just to see how it
compares regarding this). Reiserfs crashed before even writing 100MB!

So I tend to believe this is a "write barrier" problem and it looks
really nasty!!! 

To sort this out I've started a test on a single 3Ware raid, without
software raid. 

Any idea on how to circumvent the problem to make software RAID/LVM
usable?

-- 
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------

Re: XFS crash on linux raid

[Chris Wedgwood]

On Thu, May 03, 2007 at 04:45:21PM +0200, Emmanuel Florac wrote:

> After xfs_repair, the fs is fine. However, it crashes again when
> writing again a couple of GBs of data. It crashes again under
> 2.6.17.13, 2.6.17.13 SMP, 2.6.18.8, 2.6.16.36...

4K stacks?

> So I tend to believe this is a "write barrier" problem and it looks
> really nasty!!!

You could try "mount -o nobarrier ...."

Re: XFS crash on linux raid

[David Chinner]

On Thu, May 03, 2007 at 04:45:21PM +0200, Emmanuel Florac wrote:
> 
> Hello, 
> Apparently quite a lot of people do encounter the same problem from
> time to time, but I couldn't find any solution. 
> 
> When writing quite a lot to the filesystem (heavy load on the
> fileserver), the filesystem crashes when filled at 2.5~3TB (varies from
> time to time). The filesystems tested where always running on a software
> raid 0, with disabled barriers. I tend to think that disabled write
> barriers are causing the crash but I'll do some more tests to get sure.
> 
> I've met this problem for the first time on 12/23 (yup... merry
> christmas :) when a 13 TB filesystem went belly up :
> 
> Dec 23 01:38:10 storiq1 -- MARK --
> Dec 23 01:58:10 storiq1 -- MARK --
> Dec 23 02:10:29 storiq1 kernel: xfs_iunlink_remove: xfs_itobp()
> returned an error 990 on md0. Returning error.
> Dec 23 02:10:29 storiq1 kernel: xfs_inactive:^Ixfs_ifree() returned an
> error = 990 on md0
> Dec 23 02:10:29 storiq1 kernel: xfs_force_shutdown(md0,0x1) called from
> line 1763 of file fs/xfs/xfs_vnodeops.c. Return address = 0xc027f78b
> Dec 23 02:38:11 storiq1 -- MARK --
> Dec 23 02:58:11 storiq1 -- MARK -- 

So, trying to remove an inode there was a corruption found on disk
and it shut the filesystem down.

Where there any I/o errors reported before the shutdown?

> When mounting, it did that :
> 
> Filesystem "md0": Disabling barriers, not supported by the underlying
> device XFS mounting filesystem md0
> Starting XFS recovery on filesystem: md0 (logdev: internal)
> Filesystem "md0": xfs_inode_recover: Bad inode magic number, dino ptr =
> 0xf7196600, dino bp = 0xf718e980, ino = 119318 Filesystem "md0": XFS

Which was found again during log recovery.

> The system was running vanilla 2.6.17.9, and md0 was made of 3 striped
> RAID-5 on 3 3Ware-9550 cards, each hardware RAID-5 made of 8 750 GB
> drives.
> 
> On a similar hardware with 2 3Ware-9550 16x750GB striped together, but
> running 2.6.17.13, I had a similar fs crash last week. Unfortunately I
> don't have the logs at hand, but we where able to reproduce several
> times the crash at home :

Hmm - 750GB drives are brand new. i wouldn't rule out media issues
at this point...

> Filesystem "md0": XFS internal error xfs_btree_check_sblock at line 336
> of file fs/xfs/xfs_btree.c.  Caller 0xc01fb282 <c0214568>

Memory corruption?

> line 1151 of file fs/xfs/xfs_trans.c.  Return address = 0xc025f7b9
> Filesystem "md0": Corruption of in-memory data detected.  Shutting down
> filesystem: md0 Please umount the filesystem, and rectify the
> problem(s) xfs_force_shutdown(md0,0x1) called from line 338 of file
> fs/xfs/xfs_rw.c.  Return address = 0xc025f7b9
> xfs_force_shutdown(md0,0x1) called from line 338 of file
> fs/xfs/xfs_rw.c.  Return address = 0xc025f7b9
> 
> After xfs_repair, the fs is fine. However, it crashes again when
> writing again a couple of GBs of data. It crashes again under 2.6.17.13,
> 2.6.17.13 SMP, 2.6.18.8, 2.6.16.36... 
> 
> Out of curiosity, I've tried to use reiserfs (just to see how it
> compares regarding this). Reiserfs crashed before even writing 100MB!

That indicates there's something wrong other than the filesystem.
I'd suggest making sure your raid arrays, memory, etc are all
functioning correctly first.

What platform are you running on? Are you running ia32 with 4k stacks?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Fri, 4 May 2007 10:59:22 +1000 vous écriviez:

> Where there any I/o errors reported before the shutdown?
> 

Nope. To make it clear : the problem can be reproduce on several
different systems, different motherboards, different drives, different
RAID controllers... This isn't a hardware problem.

> > On a similar hardware with 2 3Ware-9550 16x750GB striped together,
> > but running 2.6.17.13, I had a similar fs crash last week.
> > Unfortunately I don't have the logs at hand, but we where able to
> > reproduce several times the crash at home :  
> 
> Hmm - 750GB drives are brand new. i wouldn't rule out media issues
> at this point...

The problem is quite easily reproduced with 500GB drives too.
 
> > Filesystem "md0": XFS internal error xfs_btree_check_sblock at line
> > 336 of file fs/xfs/xfs_btree.c.  Caller 0xc01fb282 <c0214568>  
> 
> Memory corruption?

Tried with different RAMs, and the problem occurs on ECC RAM too.

> > 
> > Out of curiosity, I've tried to use reiserfs (just to see how it
> > compares regarding this). Reiserfs crashed before even writing
> > 100MB!  
> 
> That indicates there's something wrong other than the filesystem.
> I'd suggest making sure your raid arrays, memory, etc are all
> functioning correctly first.

They are. I've tested 5 different machines so far (Supermicro or Tyan
mobos, kingston RAM, Intel or AMD cpus, hitachi and seagate drives...)
 
> What platform are you running on? Are you running ia32 with 4k stacks?

Yes. I'll try this week 2.6.18.8 thoroughly and 2.6.20.11 too. Then
jfs, just to be sure.

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[David Chinner]

On Fri, May 04, 2007 at 09:06:13AM +0200, Emmanuel Florac wrote:
> Le Fri, 4 May 2007 10:59:22 +1000 vous écriviez:
> > What platform are you running on? Are you running ia32 with 4k stacks?
> 
> Yes. I'll try this week 2.6.18.8 thoroughly and 2.6.20.11 too. Then
> jfs, just to be sure.

Well, there's your problem. Stack overflows.  IMO, if you use a
filesystem, you shouldn't use 4k stacks. ;)

If you remake you kernel with 8k stacks then your problems will
most likely go away.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Fri, 4 May 2007 17:33:44 +1000
David Chinner <dgc@sgi.com> écrivait:

> Well, there's your problem. Stack overflows.  IMO, if you use a
> filesystem, you shouldn't use 4k stacks. ;)
> 
> If you remake you kernel with 8k stacks then your problems will
> most likely go away.

Well, I've double-checked the asm-i386/module.h, and it actually looks
like 4K stacks is NOT the default, so I must be using 8K, isn't it?

I've ran the same test on the same machine but WITHOUT software raid-0
(so write barriers are in use), and all went well, more than 3TB
written without a glitch. I still think there's something related to
the write barriers here. I'll try with another RAID controller, Adaptec
for instance, to get sure the 3ware driver isn't involved. I'll also try
again with an amd64 kernel.

I'd really like to sort this out...

-- 
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------

Re: XFS crash on linux raid

[Eric Sandeen]

Emmanuel Florac wrote:
> Le Fri, 4 May 2007 17:33:44 +1000
> David Chinner <dgc@sgi.com> écrivait:
> 
>> Well, there's your problem. Stack overflows.  IMO, if you use a
>> filesystem, you shouldn't use 4k stacks. ;)
>>
>> If you remake you kernel with 8k stacks then your problems will
>> most likely go away.
> 
> Well, I've double-checked the asm-i386/module.h, and it actually looks
> like 4K stacks is NOT the default, so I must be using 8K, isn't it?

Depends on how you config'd it, just look at the .config you built with, 
and search for CONFIG_4KSTACKS

On Fedora at least (and I can't remember - I don't think this is a 
fedora-ism...) you can do "modinfo" on some module, and see:

vermagic:       2.6.21 SMP mod_unload 686 4KSTACKS

-Eric

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Fri, 04 May 2007 09:55:30 -0500
Eric Sandeen <sandeen@sandeen.net> écrivait:

> Emmanuel Florac wrote:
> > Le Fri, 4 May 2007 17:33:44 +1000
> > David Chinner <dgc@sgi.com> écrivait:
> > 
> >> Well, there's your problem. Stack overflows.  IMO, if you use a
> >> filesystem, you shouldn't use 4k stacks. ;)
> >>
> >> If you remake you kernel with 8k stacks then your problems will
> >> most likely go away.
> > 
> > Well, I've double-checked the asm-i386/module.h, and it actually
> > looks like 4K stacks is NOT the default, so I must be using 8K,
> > isn't it?
> 
> Depends on how you config'd it, just look at the .config you built
> with, and search for CONFIG_4KSTACKS

config-2.6.17.13:

# CONFIG_4KSTACKS is not set

So the problem lies elsewhere...

-- 
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------

Re: XFS crash on linux raid

[Chris Wedgwood]

On Fri, May 04, 2007 at 05:30:49PM +0200, Emmanuel Florac wrote:

> # CONFIG_4KSTACKS is not set
>
> So the problem lies elsewhere...

CONFIG_4KSTACKS is badly named.

It means you have 4K process + 4K interrupt stacks.  Without this set
you have just a single 8K stack for processes and interrupts.

One argument for 4K+4K stacks is that 8K+0K isn't really safer in many
cases --- it just appears that way becasue the problems are harder to
hit.

Almost three years ago I posted patches to split the CONFIG_4KSTACKS
option into two options.  I quickly just ported that to 2.6.21 just
now (very quickly, I might have goofed fixing up the rejects).

You could if you have time try this and enable CONFIG_I386_IRQSTACKS
but don't enable CONFIG_I386_4KSTACKS and see if that helps...




diff --git a/arch/i386/Kconfig.debug b/arch/i386/Kconfig.debug
index 458bc16..f32fbec 100644
--- a/arch/i386/Kconfig.debug
+++ b/arch/i386/Kconfig.debug
@@ -56,15 +56,22 @@ config DEBUG_RODATA
 	  portion of the kernel code won't be covered by a 2MB TLB anymore.
 	  If in doubt, say "N".
 
-config 4KSTACKS
+config I386_4KSTACKS
 	bool "Use 4Kb for kernel stacks instead of 8Kb"
 	depends on DEBUG_KERNEL
 	help
 	  If you say Y here the kernel will use a 4Kb stacksize for the
 	  kernel stack attached to each process/thread. This facilitates
 	  running more threads on a system and also reduces the pressure
-	  on the VM subsystem for higher order allocations. This option
-	  will also use IRQ stacks to compensate for the reduced stackspace.
+	  on the VM subsystem for higher order allocations.
+
+config I386_IRQSTACKS
+	bool "Allocate separate IRQ stacks"
+	depends on DEBUG_KERNEL
+	default y
+	help
+	  If you say Y here the kernel will allocate and use separate
+	  stacks for interrupts.
 
 config X86_FIND_SMP_CONFIG
 	bool
diff --git a/arch/i386/defconfig b/arch/i386/defconfig
diff --git a/arch/i386/kernel/irq.c b/arch/i386/kernel/irq.c
index 8db8d51..f6224fd 100644
--- a/arch/i386/kernel/irq.c
+++ b/arch/i386/kernel/irq.c
@@ -47,7 +47,7 @@ void ack_bad_irq(unsigned int irq)
 #endif
 }
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 /*
  * per-CPU IRQ handling contexts (thread information and stack)
  */
@@ -58,7 +58,7 @@ union irq_ctx {
 
 static union irq_ctx *hardirq_ctx[NR_CPUS] __read_mostly;
 static union irq_ctx *softirq_ctx[NR_CPUS] __read_mostly;
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 /*
  * do_IRQ handles all normal device IRQ's (the special
@@ -71,7 +71,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	/* high bit used in ret_from_ code */
 	int irq = ~regs->orig_eax;
 	struct irq_desc *desc = irq_desc + irq;
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
 #endif
@@ -99,7 +99,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	}
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 
 	curctx = (union irq_ctx *) current_thread_info();
 	irqctx = hardirq_ctx[smp_processor_id()];
@@ -136,7 +136,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 			: "memory", "cc"
 		);
 	} else
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 		desc->handle_irq(irq, desc);
 
 	irq_exit();
@@ -144,7 +144,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	return 1;
 }
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 
 /*
  * These should really be __section__(".bss.page_aligned") as well, but
@@ -234,7 +234,7 @@ asmlinkage void do_softirq(void)
 }
 
 EXPORT_SYMBOL(do_softirq);
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 /*
  * Interrupt statistics:
diff --git a/include/asm-i386/irq.h b/include/asm-i386/irq.h
index 11761cd..7db95e1 100644
--- a/include/asm-i386/irq.h
+++ b/include/asm-i386/irq.h
@@ -24,14 +24,14 @@ static __inline__ int irq_canonicalize(int irq)
 # define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
   extern void irq_ctx_init(int cpu);
   extern void irq_ctx_exit(int cpu);
 # define __ARCH_HAS_DO_SOFTIRQ
-#else
+#else /* !CONFIG_I386_IRQSTACKS */
 # define irq_ctx_init(cpu) do { } while (0)
 # define irq_ctx_exit(cpu) do { } while (0)
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 #ifdef CONFIG_IRQBALANCE
 extern int irqbalance_disable(char *str);
diff --git a/include/asm-i386/module.h b/include/asm-i386/module.h
index 02f8f54..7d5d2df 100644
--- a/include/asm-i386/module.h
+++ b/include/asm-i386/module.h
@@ -62,11 +62,11 @@ struct mod_arch_specific
 #error unknown processor family
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_4KSTACKS
 #define MODULE_STACKSIZE "4KSTACKS "
-#else
+#else /* not using CONFIG_I386_4KSTACKS */
 #define MODULE_STACKSIZE ""
-#endif
+#endif /* CONFIG_I386_4KSTACKS */
 
 #define MODULE_ARCH_VERMAGIC MODULE_PROC_FAMILY MODULE_STACKSIZE
 
diff --git a/include/asm-i386/thread_info.h b/include/asm-i386/thread_info.h
index 4b187bb..f5268e0 100644
--- a/include/asm-i386/thread_info.h
+++ b/include/asm-i386/thread_info.h
@@ -53,7 +53,7 @@ struct thread_info {
 #endif
 
 #define PREEMPT_ACTIVE		0x10000000
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_4KSTACKS
 #define THREAD_SIZE            (4096)
 #else
 #define THREAD_SIZE		(8192)

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Fri, 4 May 2007 16:20:28 -0700 vous écriviez:

> You could if you have time try this and enable CONFIG_I386_IRQSTACKS
> but don't enable CONFIG_I386_4KSTACKS and see if that helps...

That sounds very interesting, I'll give it a try monday.

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Eric Sandeen]

Emmanuel Florac wrote:
> Le Fri, 4 May 2007 16:20:28 -0700 vous écriviez:
> 
>> You could if you have time try this and enable CONFIG_I386_IRQSTACKS
>> but don't enable CONFIG_I386_4KSTACKS and see if that helps...
> 
> That sounds very interesting, I'll give it a try monday.
> 

There are also stack debugging config options; one that will warn if you 
are about to overflow (CONFIG_DEBUG_STACKOVERFLOW) and one that will 
print max stack depth in sysrq-t output (CONFIG_DEBUG_STACK_USAGE).

-Eric

Re: XFS crash on linux raid

[Chris Wedgwood]

On Sat, May 05, 2007 at 11:50:12AM -0500, Eric Sandeen wrote:

> There are also stack debugging config options; one that will warn if
> you are about to overflow (CONFIG_DEBUG_STACKOVERFLOW) and one that
> will print max stack depth in sysrq-t output
> (CONFIG_DEBUG_STACK_USAGE).

I was in such a hurry I don't think I tweaked that sanely.  I'll go
over the patch checking that and test it later today.

Is there some preferred kernel version people would like?

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Sat, 5 May 2007 13:35:25 -0700 vous écriviez:

> Is there some preferred kernel version people would like?
> 

Well I prefer staying away from the very latest bleeding edge, so I
stick to 2.6.20.11 for now.

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Chris Wedgwood]

On Sat, May 05, 2007 at 10:58:19PM +0200, Emmanuel Florac wrote:

> Well I prefer staying away from the very latest bleeding edge, so I
> stick to 2.6.20.11 for now.

diff --git a/arch/i386/Kconfig.debug b/arch/i386/Kconfig.debug
index f68cc6f..908b755 100644
--- a/arch/i386/Kconfig.debug
+++ b/arch/i386/Kconfig.debug
@@ -56,15 +56,22 @@ config DEBUG_RODATA
 	  portion of the kernel code won't be covered by a 2MB TLB anymore.
 	  If in doubt, say "N".
 
-config 4KSTACKS
+config I386_4KSTACKS
 	bool "Use 4Kb for kernel stacks instead of 8Kb"
 	depends on DEBUG_KERNEL
 	help
 	  If you say Y here the kernel will use a 4Kb stacksize for the
 	  kernel stack attached to each process/thread. This facilitates
 	  running more threads on a system and also reduces the pressure
-	  on the VM subsystem for higher order allocations. This option
-	  will also use IRQ stacks to compensate for the reduced stackspace.
+	  on the VM subsystem for higher order allocations.
+
+config I386_IRQSTACKS
+	bool "Allocate separate IRQ stacks"
+	depends on DEBUG_KERNEL
+	default y
+	help
+	  If you say Y here the kernel will allocate and use separate
+	  stacks for interrupts.
 
 config X86_FIND_SMP_CONFIG
 	bool
diff --git a/arch/i386/kernel/irq.c b/arch/i386/kernel/irq.c
index 3201d42..0da8251 100644
--- a/arch/i386/kernel/irq.c
+++ b/arch/i386/kernel/irq.c
@@ -33,7 +33,7 @@ void ack_bad_irq(unsigned int irq)
 }
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 /*
  * per-CPU IRQ handling contexts (thread information and stack)
  */
@@ -44,7 +44,7 @@ union irq_ctx {
 
 static union irq_ctx *hardirq_ctx[NR_CPUS] __read_mostly;
 static union irq_ctx *softirq_ctx[NR_CPUS] __read_mostly;
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 /*
  * do_IRQ handles all normal device IRQ's (the special
@@ -57,7 +57,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	/* high bit used in ret_from_ code */
 	int irq = ~regs->orig_eax;
 	struct irq_desc *desc = irq_desc + irq;
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
 #endif
@@ -85,7 +85,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	}
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 
 	curctx = (union irq_ctx *) current_thread_info();
 	irqctx = hardirq_ctx[smp_processor_id()];
@@ -122,7 +122,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 			: "memory", "cc"
 		);
 	} else
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 		desc->handle_irq(irq, desc);
 
 	irq_exit();
@@ -130,7 +130,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	return 1;
 }
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 
 /*
  * These should really be __section__(".bss.page_aligned") as well, but
@@ -220,7 +220,7 @@ asmlinkage void do_softirq(void)
 }
 
 EXPORT_SYMBOL(do_softirq);
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 /*
  * Interrupt statistics:
diff --git a/include/asm-i386/irq.h b/include/asm-i386/irq.h
index 11761cd..7db95e1 100644
--- a/include/asm-i386/irq.h
+++ b/include/asm-i386/irq.h
@@ -24,14 +24,14 @@ static __inline__ int irq_canonicalize(int irq)
 # define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
   extern void irq_ctx_init(int cpu);
   extern void irq_ctx_exit(int cpu);
 # define __ARCH_HAS_DO_SOFTIRQ
-#else
+#else /* !CONFIG_I386_IRQSTACKS */
 # define irq_ctx_init(cpu) do { } while (0)
 # define irq_ctx_exit(cpu) do { } while (0)
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 #ifdef CONFIG_IRQBALANCE
 extern int irqbalance_disable(char *str);
diff --git a/include/asm-i386/module.h b/include/asm-i386/module.h
index 02f8f54..7d5d2df 100644
--- a/include/asm-i386/module.h
+++ b/include/asm-i386/module.h
@@ -62,11 +62,11 @@ struct mod_arch_specific
 #error unknown processor family
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_4KSTACKS
 #define MODULE_STACKSIZE "4KSTACKS "
-#else
+#else /* not using CONFIG_I386_4KSTACKS */
 #define MODULE_STACKSIZE ""
-#endif
+#endif /* CONFIG_I386_4KSTACKS */
 
 #define MODULE_ARCH_VERMAGIC MODULE_PROC_FAMILY MODULE_STACKSIZE
 
diff --git a/include/asm-i386/thread_info.h b/include/asm-i386/thread_info.h
index 4b187bb..f5268e0 100644
--- a/include/asm-i386/thread_info.h
+++ b/include/asm-i386/thread_info.h
@@ -53,7 +53,7 @@ struct thread_info {
 #endif
 
 #define PREEMPT_ACTIVE		0x10000000
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_4KSTACKS
 #define THREAD_SIZE            (4096)
 #else
 #define THREAD_SIZE		(8192)

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Sat, 5 May 2007 15:12:50 -0700 vous écriviez:

> diff --git a/arch/i386/Kconfig.debug b/arch/i386/Kconfig.debug

Thanks!

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Sat, 05 May 2007 11:50:12 -0500 vous écriviez:

> There are also stack debugging config options; one that will warn if
> you are about to overflow (CONFIG_DEBUG_STACKOVERFLOW) and one that
> will print max stack depth in sysrq-t output
> (CONFIG_DEBUG_STACK_USAGE).

Fine, I'll try that.

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Alexander Bergolth]

[-- Attachment #1: Type: text/plain, Size: 5257 bytes --]

Hi!

Several months ago, you posted a patch for 8k stacks together with
irqstacks:

On 05/05/2007 01:20 AM, Chris Wedgwood wrote:
> Almost three years ago I posted patches to split the CONFIG_4KSTACKS
> option into two options.  I quickly just ported that to 2.6.21 just
> now (very quickly, I might have goofed fixing up the rejects).

Do you have a working version of your patch for 2.6.23?
I've been using a similar patch (attached) for several years now but
since 2.6.23, it produces oopses like below. The patch applies cleanly
but there seems to be some major change between 2.6.22 and 2.6.23 that
isn't covered correctly.

Thanks,
--leo

P.S.: The attached patch is part of ATrpms' 8k kernel:
http://atrpms.net/dist/f8/kernel-tuxonice/
2.6.23.1-49_0.99.cubbi_tuxonice_8k and
2.6.23.1-42_0.99.cubbi_tuxonice_8k both show the problem, the
corresponding versions without _8k work without any problem.
2.6.22.1-41_0.99.cubbi_tuxonice_8k on Fedora 7 worked fine too.

-------------------- 8< --------------------
BUG: unable to handle kernel NULL pointer dereference at virtual address
00000050
printing eip: 00000050 *pde = 3ed4c067
Oops: 0000 [#1] SMP
Modules linked in: ipv6 ext2 mbcache loop ahci sr_mod cdrom ata_generic
firewire_ohci firewire_core pata_pdc2027x crc_itu_t sata_sil iTCO_wdt
i2c_i801 button parport_pc parport iTCO_vendor_support i2c_core ata_piix
pcspkr intel_agp sky2 floppy sg dm_snapshot dm_zero dm_mirror dm_mod
pata_it821x libata sd_mod scsi_mod raid456 async_xor async_memcpy
async_tx xor raid1 xfs uhci_hcd ohci_hcd ehci_hcd
CPU:    0
EIP:    0060:[<00000050>]    Not tainted VLI
EFLAGS: 00210082   (2.6.23.1-49_0.99.cubbi_tuxonice_8k.fc8 #1)
EIP is at 0x50
eax: 00000001   ebx: f8947796   ecx: 01073000   edx: c0799200
esi: f8897310   edi: f79d8000   ebp: f79d8a2c   esp: c07defa0
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process rklogd (pid: 1787, ti=c07de000 task=f74fb840 task.ti=f744e000)
Stack: 00200292 f894472d 00000000 000000dc f7f5a6a0 f7f8f2b4 00000002
00000001
       00200292 00000001 00000012 00000012 c041e022 c0465fcb c0741700
c0741700
       00000012 00000000 c04672e5 c07def7c 00000012 c0467249 c0741700
c04074cb
Call Trace:
 [<f894472d>] ata_interrupt+0x1ab/0x1be [libata]
 [<c041e022>] ack_ioapic_quirk_irq+0x34/0x86
 [<c0465fcb>] handle_IRQ_event+0x23/0x51
 [<c04672e5>] handle_fasteoi_irq+0x9c/0xa6
 [<c0467249>] handle_fasteoi_irq+0x0/0xa6
 [<c04074cb>] do_IRQ+0x8c/0xb9
 =======================
Code:  Bad EIP value.
EIP: [<00000050>] 0x50 SS:ESP 0068:c07defa0
BUG: unable to handle kernel paging request at virtual address 010b2fc8
printing eip: c07dee7c *pde = 00000000
Oops: 0002 [#2] SMP
Modules linked in: ipv6 ext2 mbcache loop ahci sr_mod cdrom ata_generic
firewire_ohci firewire_core pata_pdc2027x crc_itu_t sata_sil iTCO_wdt
i2c_i801 button parport_pc parport iTCO_vendor_support i2c_core ata_piix
pcspkr intel_agp sky2 floppy sg dm_snapshot dm_zero dm_mirror dm_mod
pata_it821x libata sd_mod scsi_mod raid456 async_xor async_memcpy
async_tx xor raid1 xfs uhci_hcd ohci_hcd ehci_hcd
CPU:    0
EIP:    0060:[<c07dee7c>]    Tainted: G      D VLI
EFLAGS: 00210086   (2.6.23.1-49_0.99.cubbi_tuxonice_8k.fc8 #1)
EIP is at hardirq_stack+0x1e7c/0x40000
eax: 00200046   ebx: f74fb88c   ecx: 00200286   edx: 01073000
esi: 00051b65   edi: 001e8480   ebp: 00200006   esp: c07dee58
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process rklogd (pid: 1787, ti=c07de000 task=f74fb840 task.ti=f744e000)
Stack: c0425763 153fdb8a 00000001 13125d1e 00000001 f7bb5840 f74fb840
f7bb5840
       c07dee90 c042ab0f 001e8480 00000000 00000001 00000000 00200082
c0427c1d
       ffffff10 c04300ad 00000000 0000000f f7bb5840 00000000 c180c200
00000000
Call Trace:
 [<c0425763>] __check_preempt_curr_fair+0x55/0x86
 [<c042ab0f>] check_preempt_curr_fair+0x6b/0x71
 [<c0427c1d>] try_to_wake_up+0x2ef/0x2f9
 [<c04300ad>] do_exit+0x11b/0x6fc
 [<c0434d58>] del_timer+0x48/0x4e
 [<f894564f>] ata_scsi_qc_complete+0x3bd/0x3cb [libata]
 [<c062cdeb>] do_page_fault+0x521/0x5ef
 [<f893feea>] __ata_qc_complete+0x8c/0x92 [libata]
 [<f8940dd9>] ata_hsm_move+0x6d1/0x70c [libata]
 [<f8897310>] it821x_passthru_bmdma_stop+0x17/0x36 [pata_it821x]
 [<c062c8ca>] do_page_fault+0x0/0x5ef
 [<c062b5b2>] error_code+0x72/0x78
 [<f8947796>] ata_altstatus+0x1c/0x20 [libata]
 [<f894796c>] ata_bmdma_stop+0x1a/0x23 [libata]
 [<f8947796>] ata_altstatus+0x1c/0x20 [libata]
 [<f8897310>] it821x_passthru_bmdma_stop+0x17/0x36 [pata_it821x]
 [<f894472d>] ata_interrupt+0x1ab/0x1be [libata]
 [<c041e022>] ack_ioapic_quirk_irq+0x34/0x86
 [<c0465fcb>] handle_IRQ_event+0x23/0x51
 [<c04672e5>] handle_fasteoi_irq+0x9c/0xa6
 [<c0467249>] handle_fasteoi_irq+0x0/0xa6
 [<c04074cb>] do_IRQ+0x8c/0xb9
 =======================
Code: 00 00 00 86 00 21 00 63 57 42 c0 8a db 3f 15 01 00 00 00 1e 5d 12
13 01 00 00 00 40 58 bb f7 40 b8 4f f7 40 58 bb f7 90 ee 7d c0 <0f> ab
42 c0 80 84 1e 00 00 00 00 00 01 00 00 00 00 00 00 00 82
EIP: [<c07dee7c>] hardirq_stack+0x1e7c/0x40000 SS:ESP 0068:c07dee58
Fixing recursive fault but reboot is needed!
-------------------- 8< --------------------

-- 
e-mail   ::: Alexander.Bergolth (at) wu-wien.ac.at
fax      ::: +43-1-31336-906050
location ::: Computer Center | Vienna University of Economics | Austria


[-- Attachment #2: linux-2.6.22-8k_stacks.patch --]
[-- Type: text/x-patch, Size: 1884 bytes --]

--- linux-2.6.22.i686/arch/i386/Kconfig.debug.orig	2007-07-09 01:32:17.000000000 +0200
+++ linux-2.6.22.i686/arch/i386/Kconfig.debug	2007-07-22 13:49:57.000000000 +0200
@@ -85,4 +85,9 @@
           option saves about 4k and might cause you much additional grey
           hair.
 
+ config IRQSTACKS
+ 	bool "use IRQ stacks"
+ 	depends on !4KSTACKS
+ 	default n
+ 
 endmenu
--- linux-2.6.22.i686/arch/i386/kernel/irq.c.orig	2007-07-09 01:32:17.000000000 +0200
+++ linux-2.6.22.i686/arch/i386/kernel/irq.c	2007-07-22 13:50:50.000000000 +0200
@@ -50,7 +50,7 @@
 #endif
 }
 
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
 /*
  * per-CPU IRQ handling contexts (thread information and stack)
  */
@@ -74,7 +74,7 @@
 	/* high bit used in ret_from_ code */
 	int irq = ~regs->orig_eax;
 	struct irq_desc *desc = irq_desc + irq;
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
 #endif
@@ -102,7 +102,7 @@
 	}
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
 
 	curctx = (union irq_ctx *) current_thread_info();
 	irqctx = hardirq_ctx[smp_processor_id()];
@@ -147,7 +147,7 @@
 	return 1;
 }
 
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
 
 static char softirq_stack[NR_CPUS * THREAD_SIZE]
 		__attribute__((__section__(".bss.page_aligned")));
--- linux-2.6.22.i686/include/asm-i386/irq.h.orig	2007-07-09 01:32:17.000000000 +0200
+++ linux-2.6.22.i686/include/asm-i386/irq.h	2007-07-22 13:51:34.000000000 +0200
@@ -24,7 +24,7 @@
 # define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
   extern void irq_ctx_init(int cpu);
   extern void irq_ctx_exit(int cpu);
 # define __ARCH_HAS_DO_SOFTIRQ

Re: XFS crash on linux raid

[Chris Wedgwood]

On Mon, Nov 19, 2007 at 07:10:32PM +0100, Alexander Bergolth wrote:

> Several months ago, you posted a patch for 8k stacks together with
> irqstacks:

I've posted it off and on for two or three years now --- quite a few
people are using it or the equivalent of it, but thus far people
resist merging this.

The other idea (also nixed) that I posed was to dosable XFS if people
are using 4KSTACKS

> Do you have a working version of your patch for 2.6.23?

No.  I can go digg out a 32-bit machine and retest though.

> I've been using a similar patch (attached) for several years now but
> since 2.6.23, it produces oopses like below. The patch applies
> cleanly but there seems to be some major change between 2.6.22 and
> 2.6.23 that isn't covered correctly.

I'll take a look.  Can you make sure it works with 8K stacks (more or
less works).

Re: XFS crash on linux raid

[Alexander 'Leo' Bergolth]

On 11/20/2007 12:44 AM, Chris Wedgwood wrote:
> On Mon, Nov 19, 2007 at 07:10:32PM +0100, Alexander Bergolth wrote:
>> Several months ago, you posted a patch for 8k stacks together with
>> irqstacks:
 >
> I'll take a look.  Can you make sure it works with 8K stacks (more or
> less works).

It works (more or less) fine without the patch that splits irqstacks 
from 4k stacks. (I tried it both with CONFIG_4KSTACKS set and unset.)

Thanks,
--leo
-- 
e-mail   ::: Alexander.Bergolth (at) wu-wien.ac.at
fax      ::: +43-1-31336-906050
location ::: Computer Center | Vienna University of Economics | Austria

Re: XFS crash on linux raid

[Martin Steigerwald]

Am Freitag 04 Mai 2007 schrieb Emmanuel Florac:

> I've ran the same test on the same machine but WITHOUT software raid-0
> (so write barriers are in use), and all went well, more than 3TB
> written without a glitch. I still think there's something related to
> the write barriers here. I'll try with another RAID controller, Adaptec
> for instance, to get sure the 3ware driver isn't involved. I'll also
> try again with an amd64 kernel.

Hello Emmanuel!

When you can't use write barriers as XFS tell you in the logs, you better 
switch of write caching for the harddisks / raid controller, unless you 
happen to have NVRAM or safe power supply.

But then using write cache without barrier should not make any difference 
unless you actually have a crash or power failure during write operation.

Did you test with ext3 as well? You wrote it crashes with ReiserFS 
(version 3) even faster. When it crashes with several filesystems its 
unlikely to be a filesystem issue.

Regards,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Fri, 4 May 2007 17:58:21 +0200 vous écriviez:

> Did you test with ext3 as well? You wrote it crashes with ReiserFS 
> (version 3) even faster. When it crashes with several filesystems its 
> unlikely to be a filesystem issue.

Unfortunately ext3 doesn't support volumes bigger than 8TB, so that's
useless to me. I plan to test jfs, however.
I think it's more a dm/md issue, but I'm not sure...

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Eric Sandeen]

Emmanuel Florac wrote:
> Le Fri, 4 May 2007 17:58:21 +0200 vous écriviez:
> 
>> Did you test with ext3 as well? You wrote it crashes with ReiserFS 
>> (version 3) even faster. When it crashes with several filesystems its 
>> unlikely to be a filesystem issue.
> 
> Unfortunately ext3 doesn't support volumes bigger than 8TB, so that's
> useless to me. I plan to test jfs, however.
> I think it's more a dm/md issue, but I'm not sure...
> 

Most recent kernels (2.6.19 or so IIRC) & cvs e2fsprogs (or that from 
rhel5/centos5) can do up to 16T ext3 filesystems, so you should be able 
to test that if you like.

-Eric

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Fri, 04 May 2007 23:49:28 -0500 vous écriviez:

> 
> Most recent kernels (2.6.19 or so IIRC) & cvs e2fsprogs (or that from 
> rhel5/centos5) can do up to 16T ext3 filesystems, so you should be
> able to test that if you like.

Thanks, I'll try that too. Though it won't cover all my needs (I plan
to set up 50 and 150TB systems really soon).

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Eric Sandeen]

Emmanuel Florac wrote:
> Le Fri, 04 May 2007 23:49:28 -0500 vous écriviez:
> 
>> Most recent kernels (2.6.19 or so IIRC) & cvs e2fsprogs (or that from 
>> rhel5/centos5) can do up to 16T ext3 filesystems, so you should be
>> able to test that if you like.
> 
> Thanks, I'll try that too. Though it won't cover all my needs (I plan
> to set up 50 and 150TB systems really soon).
> 

Sure, I understand - it may be helpful in figuring out what the problem 
is, though.  I'll be curious to see how it goes...

Oh, btw, you'll need the -F (force) flag for mkfs.ext3

-Eric

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Sat, 05 May 2007 11:47:58 -0500 vous écriviez:

> Sure, I understand - it may be helpful in figuring out what the
> problem is, though.  I'll be curious to see how it goes...

Sure, stay tuned!
 
> Oh, btw, you'll need the -F (force) flag for mkfs.ext3

Thanks!

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Sat, 5 May 2007 14:00:02 -0700 vous écriviez:

> A 50TB filesystem might suck horrible on a 32-bit platform.  I'm not
> sure there is *ANY* way you coiuld fsck that should you need in some
> cases.
> 
> Is that what you're planning to do?

Nope, I'll use an x86_64 system running an x86_64 kernel :)

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Chris Wedgwood]

On Sun, May 06, 2007 at 07:21:04PM +0200, Emmanuel Florac wrote:

> Nope, I'll use an x86_64 system running an x86_64 kernel :)

How much RAM?  I think you'll want 10s of GBs possibly (well, it
depends very much on what you're storing but you can fit a lot of
small files in 150TB...)

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Sun, 6 May 2007 10:26:06 -0700 vous écriviez:

> How much RAM?  I think you'll want 10s of GBs possibly (well, it
> depends very much on what you're storing but you can fit a lot of
> small files in 150TB...)

It will be video storage, big to huge file mainly. But I'll remember to
stick as much RAM as I can :)

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Chris Wedgwood]

On Fri, May 04, 2007 at 11:43:57PM +0200, Emmanuel Florac wrote:

> Unfortunately ext3 doesn't support volumes bigger than 8TB, so
> that's useless to me. I plan to test jfs, however.

Is jfs supported by anyone right now?

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Sat, 5 May 2007 13:56:17 -0700 vous écriviez:

> Is jfs supported by anyone right now?

Huh, IBM I hope :)

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[Martin Steigerwald]

Am Samstag 05 Mai 2007 schrieb Chris Wedgwood:
> On Fri, May 04, 2007 at 11:43:57PM +0200, Emmanuel Florac wrote:
> > Unfortunately ext3 doesn't support volumes bigger than 8TB, so
> > that's useless to me. I plan to test jfs, however.
>
> Is jfs supported by anyone right now?

David 'Dave' Kleikamp was still taking care of JFS as I asked him some 
questions about write barrier support back in July 2007. He concentrated 
on bug fixes tough, not on new features.

Regards,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: XFS crash on linux raid

[David Chinner]

On Fri, May 04, 2007 at 03:25:46PM +0200, Emmanuel Florac wrote:
> Le Fri, 4 May 2007 17:33:44 +1000
> David Chinner <dgc@sgi.com> écrivait:
> 
> > Well, there's your problem. Stack overflows.  IMO, if you use a
> > filesystem, you shouldn't use 4k stacks. ;)
> > 
> > If you remake you kernel with 8k stacks then your problems will
> > most likely go away.
> 
> Well, I've double-checked the asm-i386/module.h, and it actually looks
> like 4K stacks is NOT the default, so I must be using 8K, isn't it?

Yes.

> I've ran the same test on the same machine but WITHOUT software raid-0
> (so write barriers are in use), and all went well, more than 3TB
> written without a glitch. I still think there's something related to
> the write barriers here. I'll try with another RAID controller, Adaptec
> for instance, to get sure the 3ware driver isn't involved. I'll also try
> again with an amd64 kernel.

So you use software raid and you get corruptions, right? I doubt this has
anything to do with write barriers - if it does thats an indication
of broken drivers or hardware.....

Can you run with "-o nobarrier" and no software raid and see if you
still have a problem?

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS crash on linux raid

[Emmanuel Florac]

Le Mon, 7 May 2007 12:11:22 +1000 vous écriviez:

> So you use software raid and you get corruptions, right? I doubt this
> has anything to do with write barriers - if it does thats an
> indication of broken drivers or hardware.....
> 
> Can you run with "-o nobarrier" and no software raid and see if you
> still have a problem?

I tried on the same machine without software RAID and barriers, and i
worked OK. I'll try today with nobarrier. Stay tuned :)

-- 
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Re: XFS crash on linux raid

[richid]

Emmanuel Florac wrote:
> 
> Le Mon, 7 May 2007 12:11:22 +1000 vous écriviez:
> 
>> So you use software raid and you get corruptions, right? I doubt this
>> has anything to do with write barriers - if it does thats an
>> indication of broken drivers or hardware.....
>> 
>> Can you run with "-o nobarrier" and no software raid and see if you
>> still have a problem?
> 
> I tried on the same machine without software RAID and barriers, and i
> worked OK. I'll try today with nobarrier. Stay tuned :)
> 
> -- 
> --------------------------------------------------
> Emmanuel Florac               www.intellique.com   
> --------------------------------------------------
> 

Can you update us on the status?  Not to thread hijack, but I am having the
same problem on my fileserver.  Searched all over and can't seem to find a
definitive answer.

-- 
View this message in context: http://www.nabble.com/XFS-crash-on-linux-raid-tf3687076.html#a11857832
Sent from the Xfs - General mailing list archive at Nabble.com.