Re: PROBLEM + POSS FIX: kernel stack overflow, xfs, many disks, heavy write load, 8k stack, x86-64

[John Berthels <john@humyo.com>]

[-- Attachment #1: Type: text/plain, Size: 5328 bytes --]

Chris Mason wrote:
> shrink_zone on my box isn't 500 bytes, but lets try the easy stuff
> first.  This is against .34, if you have any trouble applying to .32,
> just add the word noinline after the word static on the function
> definitions.

Hi Chris,

Thanks for this, we've been soaking it for a while and get the stack 
trace below (which is still >8k), which still has shrink_zone at 528 bytes.

I find it odd that the shrink_zone stack usage is different on our 
systems. This is a stock kernel 2.6.33.2 kernel, x86_64 arch (plus your 
patch + Dave Chinner's patch) built using ubuntu make-kpkg, with gcc 
(Ubuntu 4.3.3-5ubuntu4) 4.3.3 (.vmscan.o.cmd with full build options is 
below, gzipped .config attached).

Can you see any difference between your system and ours which might 
explain the discrepancy? I note -g and -pg in there. (Does -pg have any 
stack overhead? It seems to be enabled in ubuntu release kernels).

regards,

jb



mm/.vmscan.o.cmd:

cmd_mm/vmscan.o := gcc -Wp,-MD,mm/.vmscan.o.d  -nostdinc -isystem 
/usr/lib/gcc/x86_64-linux-gnu/4.3.3/include 
-I/usr/local/src/kern/linux-2.6.33.2/arch/x86/include -Iinclude 
-include include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef 
-Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common 
-Werror-implicit-function-declaration -Wno-format-security 
-fno-delete-null-pointer-checks -O2 -m64 -mtune=generic -mno-red-zone 
-mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args 
-fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -pipe 
-Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx 
-mno-sse2 -mno-3dnow -fno-omit-frame-pointer -fno-optimize-sibling-calls 
-g -pg -Wdeclaration-after-statement -Wno-pointer-sign 
-fno-strict-overflow   -D"KBUILD_STR(s)=\#s" 
-D"KBUILD_BASENAME=KBUILD_STR(vmscan)" 
-D"KBUILD_MODNAME=KBUILD_STR(vmscan)"  -c -o mm/.tmp_vmscan.o mm/vmscan.c



Apr 12 22:06:35 nas17 kernel: [36346.599076] apache2 used greatest stack
depth: 7904 bytes left
         Depth    Size   Location    (56 entries)
         -----    ----   --------
   0)     7904      48   __call_rcu+0x67/0x190
   1)     7856      16   call_rcu_sched+0x15/0x20
   2)     7840      16   call_rcu+0xe/0x10
   3)     7824     272   radix_tree_delete+0x159/0x2e0
   4)     7552      32   __remove_from_page_cache+0x21/0x110
   5)     7520      64   __remove_mapping+0xe8/0x130
   6)     7456     384   shrink_page_list+0x400/0x860
   7)     7072     528   shrink_zone+0x636/0xdc0
   8)     6544     112   do_try_to_free_pages+0xc2/0x3c0
   9)     6432     112   try_to_free_pages+0x64/0x70
  10)     6320     256   __alloc_pages_nodemask+0x3d2/0x710
  11)     6064      48   alloc_pages_current+0x8c/0xe0
  12)     6016      32   __page_cache_alloc+0x67/0x70
  13)     5984      80   find_or_create_page+0x50/0xb0
  14)     5904     160   _xfs_buf_lookup_pages+0x145/0x350 [xfs]
  15)     5744      64   xfs_buf_get+0x74/0x1d0 [xfs]
  16)     5680      48   xfs_buf_read+0x2f/0x110 [xfs]
  17)     5632      80   xfs_trans_read_buf+0x2bf/0x430 [xfs]
  18)     5552      80   xfs_btree_read_buf_block+0x5d/0xb0 [xfs]
  19)     5472     176   xfs_btree_rshift+0xd7/0x530 [xfs]
  20)     5296      96   xfs_btree_make_block_unfull+0x5b/0x190 [xfs]
  21)     5200     224   xfs_btree_insrec+0x39c/0x5b0 [xfs]
  22)     4976     128   xfs_btree_insert+0x86/0x180 [xfs]
  23)     4848      96   xfs_alloc_fixup_trees+0x1fa/0x350 [xfs]
  24)     4752     144   xfs_alloc_ag_vextent_near+0x916/0xb30 [xfs]
  25)     4608      32   xfs_alloc_ag_vextent+0xe5/0x140 [xfs]
  26)     4576      96   xfs_alloc_vextent+0x49f/0x630 [xfs]
  27)     4480     160   xfs_bmbt_alloc_block+0xbe/0x1d0 [xfs]
  28)     4320     208   xfs_btree_split+0xb3/0x6a0 [xfs]
  29)     4112      96   xfs_btree_make_block_unfull+0x151/0x190 [xfs]
  30)     4016     224   xfs_btree_insrec+0x39c/0x5b0 [xfs]
  31)     3792     128   xfs_btree_insert+0x86/0x180 [xfs]
  32)     3664     352   xfs_bmap_add_extent_delay_real+0x41e/0x1670 [xfs]
  33)     3312     208   xfs_bmap_add_extent+0x41c/0x450 [xfs]
  34)     3104     448   xfs_bmapi+0x982/0x1200 [xfs]
  35)     2656     256   xfs_iomap_write_allocate+0x248/0x3c0 [xfs]
  36)     2400     208   xfs_iomap+0x3d8/0x410 [xfs]
  37)     2192      32   xfs_map_blocks+0x2c/0x30 [xfs]
  38)     2160     256   xfs_page_state_convert+0x443/0x730 [xfs]
  39)     1904      64   xfs_vm_writepage+0xab/0x160 [xfs]
  40)     1840      32   __writepage+0x1a/0x60
  41)     1808     288   write_cache_pages+0x1f7/0x400
  42)     1520      16   generic_writepages+0x27/0x30
  43)     1504      48   xfs_vm_writepages+0x5a/0x70 [xfs]
  44)     1456      16   do_writepages+0x24/0x40
  45)     1440      64   writeback_single_inode+0xf1/0x3e0
  46)     1376     128   writeback_inodes_wb+0x31e/0x510
  47)     1248      16   writeback_inodes_wbc+0x1e/0x20
  48)     1232     224   balance_dirty_pages_ratelimited_nr+0x277/0x410
  49)     1008     192   generic_file_buffered_write+0x19b/0x240
  50)      816     288   xfs_write+0x849/0x930 [xfs]
  51)      528      16   xfs_file_aio_write+0x5b/0x70 [xfs]
  52)      512     272   do_sync_write+0xd1/0x120
  53)      240      48   vfs_write+0xcb/0x1a0
  54)      192      64   sys_write+0x55/0x90
  55)      128     128   system_call_fastpath+0x16/0x1b


[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 28595 bytes --]