From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from cuda.sgi.com (cuda2.sgi.com [192.48.176.25]) by oss.sgi.com (8.14.3/8.14.3/SuSE Linux 0.8) with ESMTP id o4LLsNeC032930 for ; Fri, 21 May 2010 16:54:23 -0500 Received: from mail.sandeen.net (localhost [127.0.0.1]) by cuda.sgi.com (Spam Firewall) with ESMTP id 2B38F36540D for ; Fri, 21 May 2010 14:56:42 -0700 (PDT) Received: from mail.sandeen.net (64-131-60-146.usfamily.net [64.131.60.146]) by cuda.sgi.com with ESMTP id Be9cMHp3jpPC7nGu for ; Fri, 21 May 2010 14:56:42 -0700 (PDT) Message-ID: <4BF70199.1000607@sandeen.net> Date: Fri, 21 May 2010 16:56:41 -0500 From: Eric Sandeen MIME-Version: 1.0 Subject: Re: kernel crash: scheduling while atomic References: <201005212344.41367@zmi.at> In-Reply-To: <201005212344.41367@zmi.at> List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: xfs-bounces@oss.sgi.com Errors-To: xfs-bounces@oss.sgi.com To: Michael Monnerie Cc: xfs@oss.sgi.com Michael Monnerie wrote: > Ist this caused by xfs? > > I just rsynced from another server to this one, the filesystem is nearly full, > I was in the process of vgextend (done), lvextend (done) and just wanted > to call xfs_growfs but before I could press enter the system was gone. > It's a VM within XenServer, if that matters. > Kernel 2.6.31.12-0.2-xen by openSUSE 11.2 I'm guessing you blew the stack. I've seen stack overruns hit thread flags which then cause this to trip, and that backtrace looks loooooong. oh and indeed: > [21877.951770] Thread overran stack, or stack corrupted \o/ I put that test in years ago, this is about the first time I've seen it be useful ;) So, this is a stack overrun issue. Reclaim recursed back into the filesystem. This should be a _NOFS allocation I'd think, but I don't have a suse kernel handy... upstream I wonder if xfs should be setting AOP_FLAG_NOFS in xfs_vm_write_begin .... -Eric > [21877.948005] BUG: scheduling while atomic: rsync/2345/0xffff8800 > [21877.948005] Modules linked in: af_packet nfs lockd fscache nfs_acl auth_rpcgss sunrpc ipv6 ramzswap xvmalloc lzo_decompress lzo_compress loop dm_mod reiserfs xfs exportfs xennet xenblk cdrom > [21877.948005] Pid: 2345, comm: rsync Not tainted 2.6.31.12-0.2-xen #1 > [21877.948005] Call Trace: > [21877.949649] [] try_stack_unwind+0x189/0x1b0 > [21877.949659] [] dump_trace+0xa6/0x1e0 > [21877.949666] [] show_trace_log_lvl+0x64/0x90 > [21877.949676] [] show_trace+0x23/0x40 > [21877.949684] [] dump_stack+0x81/0x9e > [21877.949695] [] __schedule_bug+0x78/0x90 > [21877.949702] [] thread_return+0x1d7/0x3fb > [21877.949709] [] schedule_timeout+0x195/0x200 > [21877.949717] [] wait_for_common+0x10b/0x230 > [21877.949726] [] wait_for_completion+0x2b/0x50 > [21877.949768] [] xfs_buf_iowait+0x31/0x80 [xfs] > [21877.949894] [] _xfs_buf_read+0x70/0x80 [xfs] > [21877.949992] [] xfs_buf_read_flags+0x8b/0xd0 [xfs] > [21877.950089] [] xfs_trans_read_buf+0x1e9/0x320 [xfs] > [21877.950174] [] xfs_btree_read_buf_block+0x68/0xe0 [xfs] > [21877.950232] [] xfs_btree_lookup_get_block+0x8e/0x110 [xfs] > [21877.950281] [] xfs_btree_lookup+0xdf/0x4d0 [xfs] > [21877.950329] [] xfs_alloc_lookup_eq+0x27/0x50 [xfs] > [21877.950361] [] xfs_alloc_fixup_trees+0x249/0x370 [xfs] > [21877.950397] [] xfs_alloc_ag_vextent_near+0x4e0/0x9a0 [xfs] > [21877.950432] [] xfs_alloc_ag_vextent+0x105/0x160 [xfs] > [21877.950471] [] xfs_alloc_vextent+0x3b4/0x4b0 [xfs] > [21877.950504] [] xfs_bmbt_alloc_block+0xf8/0x210 [xfs] > [21877.950550] [] xfs_btree_split+0xc7/0x720 [xfs] > [21877.950597] [] xfs_btree_make_block_unfull+0x15c/0x1c0 [xfs] > [21877.950643] [] xfs_btree_insrec+0x40f/0x5c0 [xfs] > [21877.950689] [] xfs_btree_insert+0xa1/0x1b0 [xfs] > [21877.950748] [] xfs_bmap_add_extent_delay_real+0x82e/0x12a0 [xfs] > [21877.950787] [] xfs_bmap_add_extent+0x424/0x450 [xfs] > [21877.950833] [] xfs_bmapi+0xda3/0x1320 [xfs] > [21877.950879] [] xfs_iomap_write_allocate+0x1d8/0x3f0 [xfs] > [21877.950953] [] xfs_iomap+0x2c9/0x300 [xfs] > [21877.951021] [] xfs_map_blocks+0x38/0x60 [xfs] > [21877.951108] [] xfs_page_state_convert+0x3fa/0x720 [xfs] > [21877.951204] [] xfs_vm_writepage+0x84/0x160 [xfs] > [21877.951301] [] pageout+0x143/0x2b0 > [21877.951308] [] shrink_page_list+0x26e/0x650 > [21877.951314] [] shrink_inactive_list+0x2d3/0x7c0 > [21877.951320] [] shrink_list+0x5b/0x110 > [21877.951325] [] shrink_zone+0x171/0x250 > [21877.951330] [] shrink_zones+0x83/0x120 > [21877.951336] [] do_try_to_free_pages+0x9e/0x380 > [21877.951342] [] try_to_free_pages+0x77/0xa0 > [21877.951349] [] __alloc_pages_slowpath+0x2d3/0x5c0 > [21877.951355] [] __alloc_pages_nodemask+0x151/0x160 > [21877.951362] [] __page_cache_alloc+0x27/0x50 > [21877.951368] [] grab_cache_page_write_begin+0x9a/0xe0 > [21877.951376] [] block_write_begin+0xae/0x120 > [21877.951396] [] xfs_vm_write_begin+0x34/0x50 [xfs] > [21877.951482] [] generic_perform_write+0xc1/0x1f0 > [21877.951489] [] generic_file_buffered_write+0x90/0x160 > [21877.951512] [] xfs_write+0x521/0xb60 [xfs] > [21877.951624] [] xfs_file_aio_write+0x70/0xa0 [xfs] > [21877.951711] [] do_sync_write+0x102/0x160 > [21877.951718] [] vfs_write+0xd8/0x1c0 > [21877.951723] [] sys_write+0x5b/0xa0 > [21877.951729] [] system_call_fastpath+0x16/0x1b > [21877.951736] [<00007fc41b0fab10>] 0x7fc41b0fab10 > [21877.951750] BUG: unable to handle kernel paging request at 0000000108743280 > [21877.951755] IP: [] dequeue_task+0x72/0x110 > [21877.951766] PGD 31c6f067 PUD 0 > [21877.951770] Thread overran stack, or stack corrupted > [21877.951773] Oops: 0000 [#1] SMP > [21877.951777] last sysfs file: /sys/kernel/uevent_seqnum > [21877.951780] CPU 0 > [21877.951783] Modules linked in: af_packet nfs lockd fscache nfs_acl auth_rpcgss sunrpc ipv6 ramzswap xvmalloc lzo_decompress lzo_compress loop dm_mod reiserfs xfs exportfs xennet xenblk cdrom > [21877.951817] Pid: 2345, comm: rsync Not tainted 2.6.31.12-0.2-xen #1 > [21877.951821] RIP: e030:[] [] dequeue_task+0x72/0x110 > [21877.951826] RSP: e02b:ffff880030ff80e8 EFLAGS: 00010012 > [21877.951830] RAX: 000000000000a380 RBX: ffff880028d72700 RCX: 0000000030ff8028 > [21877.951833] RDX: 0000000000000001 RSI: ffff880028d72700 RDI: ffffc9000000a380 > [21877.951836] RBP: ffff880030ff8108 R08: ffff880030ff8000 R09: 0000000000000000 > [21877.951839] R10: 00000000000010f3 R11: 0000000000000000 R12: 0000000000000000 > [21877.951841] R13: ffff88000d864bf0 R14: 0000000000007d00 R15: 0000000000000000 > [21877.951848] FS: 00007fc41bbc86f0(0000) GS:ffffc90000000000(0000) knlGS:000000000000000 > [21877.951851] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > [21877.951854] CR2: 0000000108743280 CR3: 0000000031c75000 CR4: 0000000000000660 > [21877.951858] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [21877.951861] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [21877.951864] Process rsync (pid: 2345, threadinfo ffff880030ff8000, task ffff880028d72700) > [21877.951867] Stack: > [21877.951869] ffff880030ff8138 0000000077adb4da 0000000000000000 ffffc9000000a380 > [21877.951874] <0> ffff880030ff8138 ffffffff80034968 ffffc9000000a380 0000000077adb4da > [21877.951880] <0> ffff88000d864bf0 ffffc9000000a380 ffff880030ff8248 ffffffff8046c8fc > [21877.951892] Call Trace: > [21877.951898] [] deactivate_task+0x38/0x60 > [21877.951907] [] thread_return+0x157/0x3fb > [21877.951913] [] schedule_timeout+0x195/0x200 > [21877.951922] [] wait_for_common+0x10b/0x230 > [21877.951928] [] wait_for_completion+0x2b/0x50 > [21877.951953] [] xfs_buf_iowait+0x31/0x80 [xfs] > [21877.951991] [] _xfs_buf_read+0x70/0x80 [xfs] > [21877.951991] [] xfs_buf_read_flags+0x8b/0xd0 [xfs] > [21877.951991] [] xfs_trans_read_buf+0x1e9/0x320 [xfs] > [21877.951991] [] xfs_btree_read_buf_block+0x68/0xe0 [xfs] > [21877.951991] [] xfs_btree_lookup_get_block+0x8e/0x110 [xfs] > [21877.951991] [] xfs_btree_lookup+0xdf/0x4d0 [xfs] > [21877.951991] [] xfs_alloc_lookup_eq+0x27/0x50 [xfs] > [21877.951991] [] xfs_alloc_fixup_trees+0x249/0x370 [xfs] > [21877.951991] [] xfs_alloc_ag_vextent_near+0x4e0/0x9a0 [xfs] > [21877.951991] [] xfs_alloc_ag_vextent+0x105/0x160 [xfs] > [21877.951991] [] xfs_alloc_vextent+0x3b4/0x4b0 [xfs] > [21877.951991] [] xfs_bmbt_alloc_block+0xf8/0x210 [xfs] > [21877.951991] [] xfs_btree_split+0xc7/0x720 [xfs] > [21877.951991] [] xfs_btree_make_block_unfull+0x15c/0x1c0 [xfs] > [21877.951991] [] xfs_btree_insrec+0x40f/0x5c0 [xfs] > [21877.951991] [] xfs_btree_insert+0xa1/0x1b0 [xfs] > [21877.951991] [] xfs_bmap_add_extent_delay_real+0x82e/0x12a0 [xfs] > [21877.951991] [] xfs_bmap_add_extent+0x424/0x450 [xfs] > [21877.951991] [] xfs_bmapi+0xda3/0x1320 [xfs] > [21877.951991] [] xfs_iomap_write_allocate+0x1d8/0x3f0 [xfs] > [21877.951991] [] xfs_iomap+0x2c9/0x300 [xfs] > [21877.951991] [] xfs_map_blocks+0x38/0x60 [xfs] > [21877.951991] [] xfs_page_state_convert+0x3fa/0x720 [xfs] > [21877.951991] [] xfs_vm_writepage+0x84/0x160 [xfs] > [21877.951991] [] pageout+0x143/0x2b0 > [21877.951991] [] shrink_page_list+0x26e/0x650 > [21877.951991] [] shrink_inactive_list+0x2d3/0x7c0 > [21877.951991] [] shrink_list+0x5b/0x110 > [21877.951991] [] shrink_zone+0x171/0x250 > [21877.951991] [] shrink_zones+0x83/0x120 > [21877.951991] [] do_try_to_free_pages+0x9e/0x380 > [21877.951991] [] try_to_free_pages+0x77/0xa0 > [21877.951991] [] __alloc_pages_slowpath+0x2d3/0x5c0 > [21877.951991] [] __alloc_pages_nodemask+0x151/0x160 > [21877.951991] [] __page_cache_alloc+0x27/0x50 > [21877.951991] [] grab_cache_page_write_begin+0x9a/0xe0 > [21877.951991] [] block_write_begin+0xae/0x120 > [21877.951991] [] xfs_vm_write_begin+0x34/0x50 [xfs] > [21877.951991] [] generic_perform_write+0xc1/0x1f0 > [21877.951991] [] generic_file_buffered_write+0x90/0x160 > [21877.951991] [] xfs_write+0x521/0xb60 [xfs] > [21877.951991] [] xfs_file_aio_write+0x70/0xa0 [xfs] > [21877.951991] [] do_sync_write+0x102/0x160 > [21877.951991] [] vfs_write+0xd8/0x1c0 > [21877.951991] [] sys_write+0x5b/0xa0 > [21877.951991] [] system_call_fastpath+0x16/0x1b > [21877.951991] [<00007fc41b0fab10>] 0x7fc41b0fab10 > [21877.951991] Code: 00 48 29 c8 48 29 f0 48 c1 f8 03 48 01 f0 48 89 83 a0 00 00 00 4c 8b 43 08 4c 8b 8b 18 02 00 00 48 c7 c0 80 a3 00 00 41 8b 48 18 <48> 8b 0c cd 40 31 78 80 48 8b b4 08 40 08 00 00 31 c9 48 c7 83 > [21877.951991] RIP [] dequeue_task+0x72/0x110 > [21877.951991] RSP > [21877.951991] CR2: 0000000108743280 > [21877.951991] ---[ end trace 4e4e8115170a7eca ]--- > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > xfs mailing list > xfs@oss.sgi.com > http://oss.sgi.com/mailman/listinfo/xfs _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs