4K stack kernel get Oops in Filesystem stress test

4K stack kernel get Oops in Filesystem stress test

[Cahya Wirawan]

Hi,
I use vanila kernel 2.6.7 and 2.6.8-rc1 with 4K stack enabled,
but if I execute the filesystem stress test from ltp.sf.net (linux test
project) the machine always crash immediately. Or it will crash also after few
hours if I do kernel compile test repeatedly. But if I use 8K stack,
my server survive this filesystem stress test.
Also my notebook get oops if I used 4k stack in kernel , but it crashed 
after few minutes running the filesystem stress test (not immediately).
My configuration is
compaq proliant ML530/G2 , 2 processor intel 2.4Ghz, 1GB ram ,
LVM1 volume with XFS filesystem.
and here is the Oops message:

Unable to handle kernel paging request at virtual address 770000d5
 printing eip:
c01153ef
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP 
Modules linked in: eepro100
CPU:    10
EIP:    0060:[<c01153ef>]    Not tainted
EFLAGS: 00010893   (2.6.7) 
EIP is at do_page_fault+0x51/0x583
eax: cafc4000   ebx: cafda000   ecx: 0000007b   edx: 7700006d
esi: 00000000   edi: c011539e   ebp: cafc40f0   esp: cafc4048
ds: 007b   es: 007b   ss: 0068
Process  (pid: -1072123377, threadinfo=cafc3000 task=dd8b4000)
Stack: 65ff0066 6c69616d 6669672e 2e780500 00666967 770000d5 7700006d 672e6f01 
       00006669 00030001 2e746573 00666967 78656eff 69672e74 78040066 6669672e 
       7270fc00 672e7665 04006669 69672e78 69fe0066 672e746e 03006669 69672e78 
Call Trace:
 [<c0106d63>] show_stack+0x80/0x96
 [<c0106efa>] show_registers+0x15f/0x1ae
 [<c0107087>] die+0xa5/0x11e
 [<c011558f>] do_page_fault+0x1f1/0x583
 [<c0106a05>] error_code+0x2d/0x38
 [<c0106a05>] error_code+0x2d/0x38
 ... Another hundreds of error_code+0x2d/0x38
 [<c0106a05>] error_code+0x2d/0x38
 [<c01068e8>] common_interrupt+0x18/0x20
 [<c031d52b>] dm_request+0x28/0xcd
 [<c02ada03>] generic_make_request+0x151/0x1cf
 [<c02adaef>] submit_bio+0x6e/0x11a
 [<c02632ee>] _pagebuf_ioapply+0x1c3/0x2c2
 [<c0263468>] pagebuf_iorequest+0x7b/0x147
 [<c0262fb2>] pagebuf_iostart+0x72/0xa0
 [<c02626d2>] pagebuf_get+0x174/0x1d6
 [<c025448b>] xfs_trans_read_buf+0x17f/0x38b
 [<c021d0a3>] xfs_btree_read_bufs+0x71/0x87
 [<c02029e7>] xfs_alloc_lookup+0x109/0x3ea
 [<c01ff43c>] xfs_alloc_ag_vextent_size+0x74/0x4d7
 [<c01fe5b5>] xfs_alloc_ag_vextent+0x102/0x128
 [<c0200eb3>] xfs_alloc_vextent+0x3d8/0x4cc
 [<c0210311>] xfs_bmap_alloc+0x816/0x1746
 [<c021445c>] xfs_bmapi+0x500/0x1457
 [<c0240e20>] xfs_iomap_write_allocate+0x299/0x4d9
 [<c023fe46>] xfs_iomap+0x37d/0x4a7
 [<c0260102>] xfs_map_blocks+0x6f/0x14a
 [<c0261132>] xfs_page_state_convert+0x42b/0x542
 [<c026184e>] linvfs_writepage+0x63/0xec
 [<c017bd4d>] mpage_writepages+0x1a0/0x30b
 [<c013e55d>] do_writepages+0x3e/0x44
 [<c0138663>] __filemap_fdatawrite+0x7a/0x7c
 [<c013869b>] filemap_flush+0x19/0x1d
 [<c024000c>] xfs_flush_space+0x9c/0xc2
 [<c0240a03>] xfs_iomap_write_delay+0x573/0x6f7
 [<c023fd92>] xfs_iomap+0x2c9/0x4a7
 [<c02612d8>] linvfs_get_block_core+0x8f/0x28e
 [<c026151c>] linvfs_get_block+0x45/0x49
 [<c015b8e6>] __block_prepare_write+0x1d3/0x3bb
 [<c015c2fa>] block_prepare_write+0x32/0x42
 [<c013a97e>] generic_file_aio_write_nolock+0x3d2/0xc11
 [<c0268454>] xfs_write+0x243/0x7bf
 [<c026450e>] linvfs_writev+0xdb/0x15d
 [<c0158637>] do_readv_writev+0x1f0/0x23a
 [<c015872e>] vfs_writev+0x52/0x5b
 [<c01587d3>] sys_writev+0x3f/0x5d
 [<c0105f7b>] syscall_call+0x7/0xb
 =======================
 [<c0106d63>] show_stack+0x80/0x96
 [<c0106efa>] show_registers+0x15f/0x1ae
 [<c0107087>] die+0xa5/0x11e
 [<c011558f>] do_page_fault+0x1f1/0x583
 [<c0106a05>] error_code+0x2d/0x38
 [<c0106a05>] error_code+0x2d/0x38
 ... Another hundreds of error_code+0x2d/0x38
 [<c0106a05>] error_code+0x2d/0x38
 [<c01068e8>] common_interrupt+0x18/0x20
 [<c0106a05>] erro_code+0x2d/0x38
 [<c0106a05>] erro_code+0x2d/0x38
 [<c0106a05>] erro_code+0x2d/0x38
 [<c031d52b>] dm_request+0x28/0xcd
 [<c02ada03>] generic_make_request+0x151/0x1cf
 [<c02adaef>] submit_bio+0x6e/0x11a
 [<c02632ee>] _pagebuf_ioapply+0x1c3/0x2c2
 [<c0263468>] pagebuf_iorequest+0x7b/0x147
 [<c0262fb2>] pagebuf_iostart+0x72/0xa0
 [<c02626d2>] pagebuf_get+0x174/0x1d6
 [<c025448b>] xfs_trans_read_buf+0x17f/0x38b
 [<c021d0a3>] xfs_btree_read_bufs+0x71/0x87
 [<c02029e7>] xfs_alloc_lookup+0x109/0x3ea
 [<c01ff43c>] xfs_alloc_ag_vextent_size+0x74/0x4d7
 [<c01fe5b5>] xfs_alloc_ag_vextent+0x102/0x128
 [<c0200eb3>] xfs_alloc_vextent+0x3d8/0x4cc
 [<c0210311>] xfs_bmap_alloc+0x816/0x1746
 [<c021445c>] xfs_bmapi+0x500/0x1457
 [<c0240e20>] xfs_iomap_write_allocate+0x299/0x4d9
 [<c023fe46>] xfs_iomap+0x37d/0x4a7
 [<c0260102>] xfs_map_blocks+0x6f/0x14a
 [<c0261132>] xfs_page_state_convert+0x42b/0x542
 [<c026184e>] linvfs_writepage+0x63/0xec
 [<c017bd4d>] mpage_writepages+0x1a0/0x30b
 [<c013e55d>] do_writepages+0x3e/0x44
 [<c0138663>] __filemap_fdatawrite+0x7a/0x7c
 [<c013869b>] filemap_flush+0x19/0x1d
 [<c024000c>] xfs_flush_space+0x9c/0xc2
 [<c0240a03>] xfs_iomap_write_delay+0x573/0x6f7
 [<c023fd92>] xfs_iomap+0x2c9/0x4a7
 [<c02612d8>] linvfs_get_block_core+0x8f/0x28e
 [<c026151c>] linvfs_get_block+0x45/0x49
 [<c015b8e6>] __block_prepare_write+0x1d3/0x3bb
 [<c015c2fa>] block_prepare_write+0x32/0x42
 [<c013a97e>] generic_file_aio_write_nolock+0x3d2/0xc11
 [<c0268454>] xfs_write+0x243/0x7bf
 [<c026450e>] linvfs_writev+0xdb/0x15d
 [<c0158637>] do_readv_writev+0x1f0/0x23a
 [<c015872e>] vfs_writev+0x52/0x5b
 [<c01587d3>] sys_writev+0x3f/0x5d
 [<c0105f7b>] syscall_call+0x7/0xb
Unable to handle kernel paging request at virtual address 66690030
 printing eip:
c0106ca0
*pde = 00000000

Thanks,
Cahya.

Re: 4K stack kernel get Oops in Filesystem stress test

[Steve Lord]

Cahya Wirawan wrote:
> Hi,
> I use vanila kernel 2.6.7 and 2.6.8-rc1 with 4K stack enabled,
> but if I execute the filesystem stress test from ltp.sf.net (linux test
> project) the machine always crash immediately. Or it will crash also after few
> hours if I do kernel compile test repeatedly. But if I use 8K stack,
> my server survive this filesystem stress test.
> Also my notebook get oops if I used 4k stack in kernel , but it crashed 
> after few minutes running the filesystem stress test (not immediately).
> My configuration is
> compaq proliant ML530/G2 , 2 processor intel 2.4Ghz, 1GB ram ,
> LVM1 volume with XFS filesystem.
> and here is the Oops message:


Don't use 4K stacks and XFS. What you hit here is a path where the
filesystem is getting full and it needs to free some reserved space
by flushing cached data which is using reserved extents. Reserved
extents do not yet have an on disk address and they include a
reservation for the worst case metadata usage. Flushing them will
get you room back.

As you can see, it is a pretty deep call stack, most of XFS is going
to work just fine with a 4K stack, but there are end cases like
this one which will just not fit.

Steve

Re: 4K stack kernel get Oops in Filesystem stress test

[Jeffrey E. Hundstad]

Steve Lord wrote:

> Don't use 4K stacks and XFS. What you hit here is a path where the
> filesystem is getting full and it needs to free some reserved space
> by flushing cached data which is using reserved extents. Reserved
> extents do not yet have an on disk address and they include a
> reservation for the worst case metadata usage. Flushing them will
> get you room back.
>
> As you can see, it is a pretty deep call stack, most of XFS is going
> to work just fine with a 4K stack, but there are end cases like
> this one which will just not fit.


If this is a known truth with XFS maybe it would be a good idea to have 
4K stacks and XFS be an impossible combination using the config tool.

-- 
jeffrey hundstad

[2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Adrian Bunk]

On Tue, Jul 20, 2004 at 09:39:21AM -0500, Jeffrey E. Hundstad wrote:
> Steve Lord wrote:
> 
> >Don't use 4K stacks and XFS. What you hit here is a path where the
> >filesystem is getting full and it needs to free some reserved space
> >by flushing cached data which is using reserved extents. Reserved
> >extents do not yet have an on disk address and they include a
> >reservation for the worst case metadata usage. Flushing them will
> >get you room back.
> >
> >As you can see, it is a pretty deep call stack, most of XFS is going
> >to work just fine with a 4K stack, but there are end cases like
> >this one which will just not fit.
> 
> 
> If this is a known truth with XFS maybe it would be a good idea to have 
> 4K stacks and XFS be an impossible combination using the config tool.


The patch below does:

1. let 4KSTACKS depend on EXPERIMENTAL
Rationale:
4Kb stacks on i386 are the future. But currently this option might still 
cause problems in some areas of the kernel. OTOH, 4Kb stacks isn't a big 
gain for most people.
2.6 is a stable kernel series, and 4KSTACKS=n is the safe choice.
Once all issues with 4KSTACKS=y are resolved this can be reverted.

2. let XFS depend on (4KSTACKS=n || BROKEN)
Rationale:
Mark Loy said:
  Don't use 4K stacks and XFS.
Mark this combination as BROKEN until XFS is fixed.
This might result in XFS support disappearing for some people, but if 
they use EXPERIMENTAL=y they should know what they are doing.

The 4KSTACKS option has to be moved for that it's asked before XFS in 
"make config".

diffstat output:
 arch/i386/Kconfig |   19 ++++++++++---------
 fs/Kconfig        |    1 +
 2 files changed, 11 insertions(+), 9 deletions(-)


Signed-off-by: Adrian Bunk <bunk@fs.tum.de>

--- linux-2.6.8-rc2-full/arch/i386/Kconfig.old	2004-07-20 21:00:32.000000000 +0200
+++ linux-2.6.8-rc2-full/arch/i386/Kconfig	2004-07-20 21:03:30.000000000 +0200
@@ -865,6 +865,16 @@
 	generate incorrect output with certain kernel constructs when
 	-mregparm=3 is used.
 
+config 4KSTACKS
+	bool "Use 4Kb for kernel stacks instead of 8Kb"
+	depends on EXPERIMENTAL
+	help
+	  If you say Y here the kernel will use a 4Kb stacksize for the
+	  kernel stack attached to each process/thread. This facilitates
+	  running more threads on a system and also reduces the pressure
+	  on the VM subsystem for higher order allocations. This option
+	  will also use IRQ stacks to compensate for the reduced stackspace.
+
 endmenu
 
 
@@ -1289,15 +1299,6 @@
 	  If you don't debug the kernel, you can say N, but we may not be able
 	  to solve problems without frame pointers.
 
-config 4KSTACKS
-	bool "Use 4Kb for kernel stacks instead of 8Kb"
-	help
-	  If you say Y here the kernel will use a 4Kb stacksize for the
-	  kernel stack attached to each process/thread. This facilitates
-	  running more threads on a system and also reduces the pressure
-	  on the VM subsystem for higher order allocations. This option
-	  will also use IRQ stacks to compensate for the reduced stackspace.
-
 config X86_FIND_SMP_CONFIG
 	bool
 	depends on X86_LOCAL_APIC || X86_VOYAGER
--- linux-2.6.8-rc2-full/fs/Kconfig.old	2004-07-20 21:04:02.000000000 +0200
+++ linux-2.6.8-rc2-full/fs/Kconfig	2004-07-20 21:04:25.000000000 +0200
@@ -294,6 +294,7 @@
 
 config XFS_FS
 	tristate "XFS filesystem support"
+	depends on (4KSTACKS=n || BROKEN)
 	help
 	  XFS is a high performance journaling filesystem which originated
 	  on the SGI IRIX platform.  It is completely multi-threaded, can

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Chris Wedgwood]

On Tue, Jul 20, 2004 at 09:50:12PM +0200, Adrian Bunk wrote:

> 1. let 4KSTACKS depend on EXPERIMENTAL

i don't like this change, despite what i might have claimed earlier :)

the reason i say this is if XFS blows up with 4K stacks then it
probably can with 8K stacks but it will be much harder, so it's not
really fixing anything but just papering over the problem

the reason for this is 8K stacks means you don't have separate irq
stacks, so if and interrupt comes along at the right time and the
codes paths are just right, you can still overflow (arguably you have
less overall space than with 4K stacks and separate irq stacks)

that said, separate irq stacks *and* 8k thread stacks would be safe,
but i'd love to see ideas on how to get the stack utilization down
(it's actually really hard)


  --cw

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Adrian Bunk]

On Tue, Jul 20, 2004 at 01:42:38PM -0700, Chris Wedgwood wrote:
> On Tue, Jul 20, 2004 at 09:50:12PM +0200, Adrian Bunk wrote:
> 
> > 1. let 4KSTACKS depend on EXPERIMENTAL
> 
> i don't like this change, despite what i might have claimed earlier :)
> 
> the reason i say this is if XFS blows up with 4K stacks then it
> probably can with 8K stacks but it will be much harder, so it's not
> really fixing anything but just papering over the problem
>...

2.6 is a stable kernel series used in production environments.

The correct solution is to fix XFS (and other problems with 4kb stacks   
if they occur), and my patch is only a short-term workaround.

4KSTACKS=n is simply the better tested case, and 4KSTACKS=y uncovers 
some issues you might not want to see in production environments.

>   --cw

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Chris Wedgwood]

On Tue, Jul 20, 2004 at 10:50:31PM +0200, Adrian Bunk wrote:

> 2.6 is a stable kernel series used in production environments.

so is 2.4.x and problems i mentioned can occur there too but are
harder to hit

> The correct solution is to fix XFS (and other problems with 4kb
> stacks if they occur), and my patch is only a short-term workaround.

it's not really a workaround, it just makes the problems harder to hit

a real fix is going to be hard, it's partly the fact there are
insanely long complicated paths and partly the fact for ia32 gcc
spills register space badly and bloats functions (afaik amd64 uses
significantly less stack in some functions)

> 4KSTACKS=n is simply the better tested case, and 4KSTACKS=y uncovers
> some issues you might not want to see in production environments.

neither address the real problem though


  --cw

Re: 4K stack kernel get Oops in Filesystem stress test

[Amon Ott]

On Dienstag, 20. Juli 2004 16:39, Jeffrey E. Hundstad wrote:
> Steve Lord wrote:
> > Don't use 4K stacks and XFS. What you hit here is a path where the
> > filesystem is getting full and it needs to free some reserved space
> > by flushing cached data which is using reserved extents. Reserved
> > extents do not yet have an on disk address and they include a
> > reservation for the worst case metadata usage. Flushing them will
> > get you room back.
> >
> > As you can see, it is a pretty deep call stack, most of XFS is going
> > to work just fine with a 4K stack, but there are end cases like
> > this one which will just not fit.
> 
> If this is a known truth with XFS maybe it would be a good idea to have 
> 4K stacks and XFS be an impossible combination using the config tool.

It would be good if there was some warning in the 4K stack option help, 
there have been quite many cases already where the kernel broke with odd 
symptoms because of this switch.

E.g.
Warning: Use this option with care, as it might break your system under 
load. If you experience weird crashes or oopses, please retry with this 
option turned off.

Amon.
-- 
http://www.rsbac.org - GnuPG: 2048g/5DEAAA30 2002-10-22

Re: 4K stack kernel get Oops in Filesystem stress test

[Nathan Scott]

On Thu, Jul 22, 2004 at 09:27:44AM +0200, Amon Ott wrote:
> On Dienstag, 20. Juli 2004 16:39, Jeffrey E. Hundstad wrote:
> > Steve Lord wrote:
> > > Don't use 4K stacks and XFS. What you hit here is a path where the
> > > filesystem is getting full and it needs to free some reserved space
> > > by flushing cached data which is using reserved extents. Reserved
> > > extents do not yet have an on disk address and they include a
> > > reservation for the worst case metadata usage. Flushing them will
> > > get you room back.
> > >
> > > As you can see, it is a pretty deep call stack, most of XFS is going
> > > to work just fine with a 4K stack, but there are end cases like
> > > this one which will just not fit.

Actually, this area of the code has been a source of several
deadlocks, so we need to consider reworking how we do this
anyay (helper thread or something along those lines) - that
would also help to address the stack depth problem here, and
this spot is probably the worst-case situation for stack use
in XFS.

> > If this is a known truth with XFS maybe it would be a good idea to have 
> > 4K stacks and XFS be an impossible combination using the config tool.

I would prefer not to do that, we want to know where the problems
are - if we hide them like this it will just make them harder to
find and resolve.  Certainly XFS is not the only subsystem with
problems here (even saw an _ext2_ stack overflow go past recently,
and I believe other filesystems can be much worse) - & when stacked
volume managers and other drivers enter the picture...

> It would be good if there was some warning in the 4K stack option help, 
> there have been quite many cases already where the kernel broke with odd 
> symptoms because of this switch.
> 
> E.g.
> Warning: Use this option with care, as it might break your system under 
> load. If you experience weird crashes or oopses, please retry with this 
> option turned off.

If its not done already, perhaps 4KSTACKS could be reported in
the oops message (like PREEMPT and one or two other things are,
IIRC)?

cheers.

-- 
Nathan

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Nathan Scott]

On Tue, Jul 20, 2004 at 09:50:12PM +0200, Adrian Bunk wrote:
> 
> The patch below does:
> 
> 1. let 4KSTACKS depend on EXPERIMENTAL
> Rationale:
> 4Kb stacks on i386 are the future. But currently this option might still 
> cause problems in some areas of the kernel. OTOH, 4Kb stacks isn't a big 
> gain for most people.
> 2.6 is a stable kernel series, and 4KSTACKS=n is the safe choice.
> Once all issues with 4KSTACKS=y are resolved this can be reverted.

Seems fine.

> 2. let XFS depend on (4KSTACKS=n || BROKEN)
> Rationale:
> Mark Loy said:
>   Don't use 4K stacks and XFS.

Who is Mark Loy?  (and what does he know about XFS?)

> Mark this combination as BROKEN until XFS is fixed.

This part is not useful.  We want to hear about problems
that people hit with 4K stacks so we can try to address
them, and it mostly works as is.

cheers.

-- 
Nathan

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Adrian Bunk]

On Thu, Jul 29, 2004 at 04:09:00PM +1000, Nathan Scott wrote:
> On Tue, Jul 20, 2004 at 09:50:12PM +0200, Adrian Bunk wrote:
> > 
> > The patch below does:
> > 
> > 1. let 4KSTACKS depend on EXPERIMENTAL
> > Rationale:
> > 4Kb stacks on i386 are the future. But currently this option might still 
> > cause problems in some areas of the kernel. OTOH, 4Kb stacks isn't a big 
> > gain for most people.
> > 2.6 is a stable kernel series, and 4KSTACKS=n is the safe choice.
> > Once all issues with 4KSTACKS=y are resolved this can be reverted.
> 
> Seems fine.
> 
> > 2. let XFS depend on (4KSTACKS=n || BROKEN)
> > Rationale:
> > Mark Loy said:
> >   Don't use 4K stacks and XFS.
> 
> Who is Mark Loy?  (and what does he know about XFS?)

I don't know where I got this name from.

I wanted to write "Steve Lord".

(Sorry for the confusion.)

> > Mark this combination as BROKEN until XFS is fixed.
> 
> This part is not useful.  We want to hear about problems
> that people hit with 4K stacks so we can try to address
> them, and it mostly works as is.

2.6 is a stable kernel series used in production environments.

Regarding Linus' tree, it's IMHO the best solution to work around it 
this way until all issues are sorted out.

Feel free to revert it in -mm later, since there are many brave souls  
running -mm you'll still get to hear about problems.

> cheers.
> Nathan

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]


> > > Mark this combination as BROKEN until XFS is fixed.
> > 
> > This part is not useful.  We want to hear about problems
> > that people hit with 4K stacks so we can try to address
> > them, and it mostly works as is.
> 
> 2.6 is a stable kernel series used in production environments.
> 
> Regarding Linus' tree, it's IMHO the best solution to work around it 
> this way until all issues are sorted out.
> 
> Feel free to revert it in -mm later, since there are many brave souls  
> running -mm you'll still get to hear about problems.

can you then also mark XFS broken in 2.4 entirely?
2.4 has a nett stack of also 4Kb... 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Mika Bostrom]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

On Thu, Jul 29, 2004 at 04:09:00PM +1000, Nathan Scott wrote:
> On Tue, Jul 20, 2004 at 09:50:12PM +0200, Adrian Bunk wrote:
> > Mark this combination as BROKEN until XFS is fixed.
> This part is not useful.  We want to hear about problems
> that people hit with 4K stacks so we can try to address
> them, and it mostly works as is.

  I can mention one. This is not a bugreport (yet).

  I can reproduce a complete hard-lock on 2.6.7 vanilla with the
following setup:

  * VMWare 4.5.2 (taints kernel, I know)
    * NIC is bridged
  * XFS filesystem
  * 4k stacks

  Inside vmware, I am running an instance of modified OpenBSD. With 4k
stacks, at least once a day the system locks hard. The shortest time
between two forced boots was about 20 minutes.

  The hang is triggered everytime by a single instance: guest OS sends a
DHCP-request, and this causes the linux kernel to hang. This does not
happen every time, but perhaps every fourth or fifth time. On one
occasion, it was the first time immediately after boot.

  Compiling 2.6.7 with 8k stacks has so far solved this issue. No random
hangs.

  Now, the reason this can't be any kind of bugreport is clear:
  1) kernel is tainted
  2) VMWare's modules are not yet updated to cope with 2.6.7 kernel

  So until VMWare updates their product, I consider this a bug in their
modules. When they do, I intend to test 4k stacks again. If the hangs
continue, then I shall see with their support whether it can be tracked
to their code or not.

  But at least at the moment if you wish to use VMWare and XFS, using 4k
stacks is, in my experience, asking for trouble. 

-- 
 Mika Boström      +358-40-525-7347  \-/  "World peace will be achieved
 Bostik@iki.fi    www.iki.fi/bostik   X    when the last man has killed
 Security freak, and proud of it.    /-\   the second-to-last." -anon?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Zwane Mwaikambo]

On Thu, 29 Jul 2004, Mika Bostrom wrote:

>   Now, the reason this can't be any kind of bugreport is clear:
>   1) kernel is tainted
>   2) VMWare's modules are not yet updated to cope with 2.6.7 kernel
>
>   So until VMWare updates their product, I consider this a bug in their
> modules. When they do, I intend to test 4k stacks again. If the hangs
> continue, then I shall see with their support whether it can be tracked
> to their code or not.
>
>   But at least at the moment if you wish to use VMWare and XFS, using 4k
> stacks is, in my experience, asking for trouble.

Given that XFS and 4k stacks has known issues perhaps it isn't a fault of
VMWare. I've been using VMWare 4 with 4K stacks running linux, netbsd and
win2k on a system with a 30day uptime, i'm using ext3 on 2.6.7-rc3-mm2.

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS

[Mika Bostrom]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

On Thu, Jul 29, 2004 at 12:09:29PM -0400, Zwane Mwaikambo wrote:
> On Thu, 29 Jul 2004, Mika Bostrom wrote:
> 
> >   Now, the reason this can't be any kind of bugreport is clear:
> >   1) kernel is tainted
> >   2) VMWare's modules are not yet updated to cope with 2.6.7 kernel
> >
> >   So until VMWare updates their product, I consider this a bug in their
> > modules. When they do, I intend to test 4k stacks again. If the hangs
> > continue, then I shall see with their support whether it can be tracked
> > to their code or not.
> >
> >   But at least at the moment if you wish to use VMWare and XFS, using 4k
> > stacks is, in my experience, asking for trouble.
> 
> Given that XFS and 4k stacks has known issues perhaps it isn't a fault of
> VMWare. I've been using VMWare 4 with 4K stacks running linux, netbsd and
> win2k on a system with a 30day uptime, i'm using ext3 on 2.6.7-rc3-mm2.

  Quite true, and a good point. However, this is one case that can be
reproduced - not systematically, but with certainty - and which displays
a clear difference between 4k and 8k stacks. Enabling noisy debugs and
tagging on a serial console might catch something. 

  My co-worker uses a FC2 stock kernel and I remember someone from RH
(Ingo Molnar?) saying that those are currently built with 4k stacks.  He
has had no such troubles, but then, he isn't using XFS.

  I'm merely providing this as an additional info for those who assume
that VMWare knows what they are doing and wish to start debugging the
combination sooner rather than later. (Even if 'later' proves to
eliminate some dead-end test-case scenarios.) Nathan Scott explicitly
requested for information on troublesome setups, so one could say I'm
humouring him and granting a wish, of a sort :)

-- 
 Mika Boström      +358-40-525-7347  \-/  "World peace will be achieved
 Bostik@iki.fi    www.iki.fi/bostik   X    when the last man has killed
 Security freak, and proud of it.    /-\   the second-to-last." -anon?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Adrian Bunk]

On Thu, Jul 29, 2004 at 01:46:52PM +0200, Arjan van de Ven wrote:
> 
> > > > Mark this combination as BROKEN until XFS is fixed.
> > > 
> > > This part is not useful.  We want to hear about problems
> > > that people hit with 4K stacks so we can try to address
> > > them, and it mostly works as is.
> > 
> > 2.6 is a stable kernel series used in production environments.
> > 
> > Regarding Linus' tree, it's IMHO the best solution to work around it 
> > this way until all issues are sorted out.
> > 
> > Feel free to revert it in -mm later, since there are many brave souls  
> > running -mm you'll still get to hear about problems.
> 
> can you then also mark XFS broken in 2.4 entirely?
> 2.4 has a nett stack of also 4Kb... 

There are reports of breakages with 4kb stacks in 2.6, but AFAIK no 
similar reports for 2.4 .

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Chris Wedgwood]

On Thu, Jul 29, 2004 at 11:11:37PM +0200, Adrian Bunk wrote:

> There are reports of breakages with 4kb stacks in 2.6, but AFAIK no
> similar reports for 2.4 .

2.4.x uses the stack(s) differently than 2.6.x so it will usually be
harder (but not impossible) to break and less easy to detect.

I believe what Arjan is saying that that 2.4.x effectively really only
has 4K of safely usable stack anyhow (we have some on-stack allocated
data and interrupts use the same stack).

Also, FWIW I do think there were been reports of problems in 2.4.x
that looked like they might be stack-size related (things dying
horribly after an interrupt for example).


  --cw

Re: [xfs-masters] Re: [2.6 patch] let 4KSTACKS depend on EXPERIMENTAL and XFS on 4KSTACKS=n

[Nathan Scott]

Arjan wrote:
> can you then also mark XFS broken in 2.4 entirely?
> 2.4 has a nett stack of also 4Kb... 

The assumptions there are incorrect - 2.4 is now quite a
different kernel - we haven't seen problems like this on
2.4 at all, and I routinely test that failing code path
in our regression tests every other night on 2.4.  There
have certainly been stack consumers in the 2.6 VFS that
weren't there in 2.4 (like AIO and struct kiocb, etc) so
thats not an apples-to-apples comparison anymore.

Adrian wrote:
> 2.6 is a stable kernel series used in production environments.
> 
> Regarding Linus' tree, it's IMHO the best solution to work around it 
> this way until all issues are sorted out.

I'm not really convinced - the EXPERIMENTAL marking should
be plenty of a deterent to folks in production environments.
There are reports of stack overruns on other filesystems as
well with 4KSTACKS, so doesn't seem worthwhile to me to do
this just for XFS.

cheers.

-- 
Nathan

[2.6 patch] let 4KSTACKS depend on EXPERIMENTAL

[Adrian Bunk]

On Fri, Jul 30, 2004 at 08:30:40AM +1000, Nathan Scott wrote:
>...
> Adrian wrote:
> > 2.6 is a stable kernel series used in production environments.
> > 
> > Regarding Linus' tree, it's IMHO the best solution to work around it 
> > this way until all issues are sorted out.
> 
> I'm not really convinced - the EXPERIMENTAL marking should
> be plenty of a deterent to folks in production environments.
> There are reports of stack overruns on other filesystems as
> well with 4KSTACKS, so doesn't seem worthwhile to me to do
> this just for XFS.


OK, below is a patch that only adds a dependency of 4KSTACKS on 
EXPERIMENTAL.

Considering that not all issues with 4kb stacks are currently corrected, 
this patch should IMHO go in 2.6.8 .


Signed-off-by: Adrian Bunk <bunk@fs.tum.de>

--- linux-2.6.8-rc2-mm1-full/arch/i386/Kconfig.old	2004-08-01 20:59:02.000000000 +0200
+++ linux-2.6.8-rc2-mm1-full/arch/i386/Kconfig	2004-08-01 20:59:46.000000000 +0200
@@ -1474,7 +1474,8 @@
 	  to solve problems without frame pointers.
 
 config 4KSTACKS
-	bool "Use 4Kb for kernel stacks instead of 8Kb"
+	bool "Use 4Kb for kernel stacks instead of 8Kb (EXPERIMENTAL)"
+	depends on EXPERIMENTAL
 	help
 	  If you say Y here the kernel will use a 4Kb stacksize for the
 	  kernel stack attached to each process/thread. This facilitates