From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753078Ab2GTPVG (ORCPT ); Fri, 20 Jul 2012 11:21:06 -0400 Received: from mx1.redhat.com ([209.132.183.28]:35063 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752386Ab2GTPVB (ORCPT ); Fri, 20 Jul 2012 11:21:01 -0400 Message-ID: <50097758.3040803@redhat.com> Date: Fri, 20 Jul 2012 10:20:56 -0500 From: Eric Sandeen User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: Jeff Moyer CC: kernel list Subject: Re: Stack overrun in 3.5.0-rc7 w/ cfq References: <5008C388.2020803@redhat.com> In-Reply-To: X-Enigmail-Version: 1.4.3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 7/20/12 9:07 AM, Jeff Moyer wrote: > Eric Sandeen writes: > >> I got this oops & stack overrun warning while running mkfs.ext4 on a sparse 4T file hosted on xfs. >> >> Should CFQ be issuing IO here? > > Yes. The on-stack plugging gets flushed when a process is scheduled > out. Seriously, Eric, all of that xfs stuff in the stack trace, and you > want to pick on cfq!?! For shame! ;) I had to get your attention ;) And Dave put me up to it. > Am I reading that wrong, or is there no separate IRQ stack? Or did > taking the IRQ cause the stack overrun? Um, I'm not sure. I think Dave's point was whether it should be issuing IO from the schedule, related to this? /* * If 'from_schedule' is true, then postpone the dispatch of requests * until a safe kblockd context. We due this to avoid accidental big * additional stack usage in driver dispatch, in places where the originally * plugger did not intend it. */ static void queue_unplugged(struct request_queue *q, unsigned int depth, bool from_schedule) FWIW, here's a better trace of how deep XFS itself went on that callchain, about 5k: [root@inode ~]# cat /sys/kernel/debug/tracing/stack_trace Depth Size Location (56 entries) ----- ---- -------- 0) 6920 16 rcu_read_lock_held+0x9/0x50 1) 6904 64 __module_address+0x105/0x110 2) 6840 32 __module_text_address+0x16/0x70 3) 6808 32 is_module_text_address+0x27/0x40 4) 6776 32 __kernel_text_address+0x58/0x80 5) 6744 112 print_context_stack+0x81/0x140 6) 6632 144 dump_trace+0x17f/0x300 7) 6488 32 save_stack_trace+0x2f/0x50 8) 6456 64 set_track+0x67/0x100 9) 6392 48 alloc_debug_processing+0x170/0x180 10) 6344 240 __slab_alloc+0x153/0x6f0 11) 6104 80 kmem_cache_alloc+0x212/0x220 12) 6024 16 mempool_alloc_slab+0x15/0x20 13) 6008 144 mempool_alloc+0x60/0x180 14) 5864 128 get_request+0x251/0x660 15) 5736 192 get_request_wait+0x2c/0x330 16) 5544 96 blk_queue_bio+0x105/0x430 17) 5448 48 generic_make_request+0xca/0x100 18) 5400 112 submit_bio+0x85/0x110 19) 5288 112 _xfs_buf_ioapply+0x170/0x1e0 [xfs] 20) 5176 48 xfs_buf_iorequest+0x4d/0x110 [xfs] 21) 5128 32 _xfs_buf_read+0x31/0x50 [xfs] 22) 5096 48 xfs_buf_read+0x113/0x170 [xfs] 23) 5048 96 xfs_trans_read_buf+0x325/0x610 [xfs] 24) 4952 96 xfs_btree_read_buf_block+0x5e/0xd0 [xfs] 25) 4856 96 xfs_btree_lookup_get_block+0x81/0xf0 [xfs] 26) 4760 176 xfs_btree_lookup+0xbf/0x470 [xfs] 27) 4584 16 xfs_alloc_lookup_ge+0x1c/0x20 [xfs] 28) 4568 240 xfs_alloc_ag_vextent_near+0x98/0xe10 [xfs] 29) 4328 32 xfs_alloc_ag_vextent+0xd5/0x100 [xfs] 30) 4296 112 __xfs_alloc_vextent+0x57a/0x7b0 [xfs] 31) 4184 272 xfs_alloc_vextent+0x198/0x1a0 [xfs] 32) 3912 256 xfs_bmbt_alloc_block+0xdb/0x210 [xfs] 33) 3656 240 xfs_btree_split+0xbd/0x710 [xfs] 34) 3416 96 xfs_btree_make_block_unfull+0x12d/0x190 [xfs] 35) 3320 224 xfs_btree_insrec+0x3ef/0x5a0 [xfs] 36) 3096 144 xfs_btree_insert+0x6d/0x190 [xfs] 37) 2952 256 xfs_bmap_add_extent_delay_real+0xf56/0x1cf0 [xfs] 38) 2696 80 xfs_bmapi_allocate+0x238/0x2d0 [xfs] 39) 2616 336 xfs_bmapi_write+0x521/0x790 [xfs] 40) 2280 192 xfs_iomap_write_allocate+0x13d/0x370 [xfs] 41) 2088 112 xfs_map_blocks+0x299/0x310 [xfs] 42) 1976 208 xfs_vm_writepage+0x197/0x5c0 [xfs] 43) 1768 32 __writepage+0x1a/0x50 44) 1736 336 write_cache_pages+0x206/0x5f0 45) 1400 96 generic_writepages+0x54/0x80 46) 1304 48 xfs_vm_writepages+0x5c/0x80 [xfs] 47) 1256 16 do_writepages+0x23/0x40 48) 1240 80 __writeback_single_inode+0x46/0x1e0 49) 1160 192 writeback_sb_inodes+0x27d/0x500 50) 968 80 __writeback_inodes_wb+0x9e/0xd0 51) 888 192 wb_writeback+0x2db/0x5c0 52) 696 176 wb_do_writeback+0x130/0x300 53) 520 160 bdi_writeback_thread+0xc3/0x410 54) 360 176 kthread+0xc6/0xd0 55) 184 184 kernel_thread_helper+0x4/0x10 > Cheers, > Jeff > >> >> -Eric >> >> [10821.639839] BUG: unable to handle kernel paging request at fffffffb900148a0 >> [10821.640820] IP: [] cpuacct_charge+0xb4/0x210 >> [10821.640820] PGD 1c0d067 PUD 0 >> [10821.640820] Thread overran stack, or stack corrupted >> [10821.640820] Oops: 0000 [#1] SMP >> [10821.640820] CPU 1 >> [10821.640820] Modules linked in:[10821.640820] xfs sunrpc ip6table_filter ip6_tables binfmt_misc vhost_net macvtap macvlan tun iTCO_wdt iTCO_vendor_support dcdbas microcode i2c_i801 lpc_ich mfd_core tg3 shpchp i3000_edac edac_core ext3 jbd mbcache ata_generic pata_acpi pata_sil680 radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan] >> >> [10821.640820] Pid: 2914, comm: flush-8:16 Not tainted 3.5.0-rc7+ #65 Dell Computer Corporation PowerEdge 860/0RH817 >> [10821.640820] RIP: 0010:[] [] cpuacct_charge+0xb4/0x210 >> [10821.640820] RSP: 0018:ffff88007d003d58 EFLAGS: 00010082 >> [10821.640820] RAX: 00000000001d6fe8 RBX: 00000000000b67c5 RCX: 0000000000000003 >> [10821.640820] RDX: 0000000000000001 RSI: ffffffff81c2fae0 RDI: 0000000000000046 >> [10821.640820] RBP: ffff88007d003d88 R08: 0000000000000003 R09: 0000000000000001 >> [10821.640820] R10: 0000000000000001 R11: 0000000000000004 R12: ffff88007b8d0000 >> [10821.640820] R13: ffffffff81c60ee0 R14: ffffffff820f7d40 R15: 00000153b050614b >> [10821.640820] FS: 0000000000000000(0000) GS:ffff88007d000000(0000) knlGS:0000000000000000 >> [10821.640820] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >> [10821.640820] CR2: fffffffb900148a0 CR3: 000000007a3ab000 CR4: 00000000000007e0 >> [10821.640820] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> [10821.640820] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >> [10821.640820] Process flush-8:16 (pid: 2914, threadinfo ffff88007981a000, task ffff88007b8d0000) >> [10821.640820] Stack: >> [10821.640820] ffffffff810a0978 000000001dcd6500 ffff88007d1d42b8 00000000000b67c5 >> [10821.640820] ffff88007b8d0048 ffff88007b8d0000 ffff88007d003dc8 ffffffff810a82ef >> [10821.640820] ffffffff81810920 ffff88007d1d42b8 ffff88007b8d0048 0000000000000000 >> [10821.640820] Call Trace: >> [10821.640820] >> [10821.640820] [] ? cpuacct_charge+0x28/0x210 >> [10821.640820] [] update_curr+0x13f/0x220 >> [10821.640820] [] task_tick_fair+0xbd/0x140 >> [10821.640820] [] scheduler_tick+0xde/0x150 >> [10821.640820] [] update_process_times+0x6e/0x90 >> [10821.640820] [] tick_sched_timer+0x66/0xc0 >> [10821.640820] [] __run_hrtimer+0x83/0x320 >> [10821.640820] [] ? tick_nohz_handler+0x100/0x100 >> [10821.640820] [] hrtimer_interrupt+0x106/0x280 >> [10821.640820] [] ? sched_clock_local+0x25/0x90 >> [10821.640820] [] smp_apic_timer_interrupt+0x69/0x99 >> [10821.640820] [] apic_timer_interrupt+0x6f/0x80 >> [10821.640820] >> [10821.640820] [] ? _raw_spin_unlock_irq+0x34/0x50 >> [10821.640820] [] scsi_request_fn+0xc8/0x5b0 >> [10821.640820] [] __blk_run_queue+0x1e/0x20 >> [10821.640820] [] cfq_insert_request+0x398/0x720 >> [10821.640820] [] ? cfq_insert_request+0x4c/0x720 >> [10821.640820] [] ? attempt_merge+0x21/0x520 >> [10821.640820] [] ? ftrace_call+0x5/0x2b >> [10821.640820] [] __elv_add_request+0x220/0x2e0 >> [10821.640820] [] blk_flush_plug_list+0x1a4/0x260 >> [10821.640820] [] schedule+0x50/0x70 >> [10821.640820] [] schedule_timeout+0x315/0x410 >> [10821.640820] [] ? mark_held_locks+0x8d/0x140 >> [10821.640820] [] ? _raw_spin_unlock_irq+0x30/0x50 >> [10821.640820] [] ? trace_hardirqs_on_caller+0x105/0x190 >> [10821.640820] [] wait_for_common+0x12b/0x180 >> [10821.640820] [] ? try_to_wake_up+0x2e0/0x2e0 >> [10821.640820] [] ? ftrace_call+0x5/0x2b >> [10821.640820] [] ? _xfs_buf_read+0x41/0x50 [xfs] >> [10821.640820] [] ? xfs_trans_read_buf+0x325/0x610 [xfs] >> [10821.640820] [] wait_for_completion+0x1d/0x20 >> [10821.640820] [] xfs_buf_iowait+0xc5/0x1b0 [xfs] >> [10821.640820] [] _xfs_buf_read+0x41/0x50 [xfs] >> [10821.640820] [] xfs_buf_read+0x113/0x170 [xfs] >> [10821.640820] [] xfs_trans_read_buf+0x325/0x610 [xfs] >> [10821.640820] [] xfs_btree_read_buf_block+0x5e/0xd0 [xfs] >> [10821.640820] [] ? ftrace_call+0x5/0x2b >> [10821.640820] [] xfs_btree_lookup_get_block+0x81/0xf0 [xfs] >> [10821.640820] [] ? xfs_btree_ptr_offset+0x4c/0x90 [xfs] >> [10821.640820] [] xfs_btree_lookup+0xbf/0x470 [xfs] >> [10821.640820] [] ? ftrace_call+0x5/0x2b >> [10821.640820] [] xfs_alloc_lookup_eq+0x19/0x20 [xfs] >> [10821.640820] [] xfs_alloc_fixup_trees+0x269/0x340 [xfs] >> [10821.640820] [] xfs_alloc_ag_vextent_near+0x852/0xe10 [xfs] >> [10821.640820] [] xfs_alloc_ag_vextent+0xd5/0x100 [xfs] >> [10821.640820] [] __xfs_alloc_vextent+0x57a/0x7b0 [xfs] >> [10821.640820] [] ? lockdep_init_map+0x59/0x150 >> [10821.640820] [] xfs_alloc_vextent+0x198/0x1a0 [xfs] >> [10821.640820] [] xfs_bmbt_alloc_block+0xdb/0x210 [xfs] >> [10821.640820] [] xfs_btree_split+0xbd/0x710 [xfs] >> [10821.640820] [] ? ftrace_call+0x5/0x2b >> [10821.640820] [] xfs_btree_make_block_unfull+0x12d/0x190 [xfs] >> [10821.640820] [] xfs_btree_insrec+0x3ef/0x5a0 [xfs] >> [10821.640820] [] ? ftrace_call+0x5/0x2b >> [10821.640820] [] xfs_btree_insert+0x6d/0x190 [xfs] >> [10821.640820] [] xfs_bmap_add_extent_delay_real+0x72b/0x1cf0 [xfs] >> [10821.640820] [] ? kmem_cache_alloc+0x113/0x220 >> [10821.640820] [] xfs_bmapi_allocate+0x238/0x2d0 [xfs] >> [10821.640820] [] ? ftrace_call+0x5/0x2b >> [10821.640820] [] xfs_bmapi_write+0x521/0x790 [xfs] >> [10821.640820] [] xfs_iomap_write_allocate+0x13d/0x370 [xfs] >> [10821.640820] [] xfs_map_blocks+0x299/0x310 [xfs] >> [10821.640820] [] xfs_vm_writepage+0x197/0x5c0 [xfs] >> [10821.640820] [] __writepage+0x1a/0x50 >> [10821.640820] [] write_cache_pages+0x206/0x5f0 >> [10821.640820] [] ? bdi_position_ratio+0x130/0x130 >> [10821.640820] [] generic_writepages+0x54/0x80 >> [10821.640820] [] xfs_vm_writepages+0x5c/0x80 [xfs] >> [10821.640820] [] do_writepages+0x23/0x40 >> [10821.640820] [] __writeback_single_inode+0x46/0x1e0 >> [10821.640820] [] writeback_sb_inodes+0x27d/0x500 >> [10821.640820] [] ? _raw_spin_unlock+0x2b/0x50 >> [10821.640820] [] __writeback_inodes_wb+0x9e/0xd0 >> [10821.640820] [] wb_writeback+0x2db/0x5c0 >> [10821.640820] [] ? ftrace_call+0x5/0x2b >> [10821.640820] [] wb_do_writeback+0x130/0x300 >> [10821.640820] [] bdi_writeback_thread+0xc3/0x410 >> [10821.640820] [] ? wb_do_writeback+0x300/0x300 >> [10821.640820] [] ? wb_do_writeback+0x300/0x300 >> [10821.640820] [] kthread+0xc6/0xd0 >> [10821.640820] [] kernel_thread_helper+0x4/0x10 >> [10821.640820] [] ? retint_restore_args+0x13/0x13 >> [10821.640820] [] ? __init_kthread_worker+0x70/0x70 >> [10821.640820] [] ? gs_change+0x13/0x13 >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/