From: Minchan Kim <minchan@kernel.org>
To: Dave Chinner <david@fromorbit.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
linux-mm <linux-mm@kvack.org>, "H. Peter Anvin" <hpa@zytor.com>,
Ingo Molnar <mingo@kernel.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
Mel Gorman <mgorman@suse.de>, Rik van Riel <riel@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Hugh Dickins <hughd@google.com>,
Rusty Russell <rusty@rustcorp.com.au>,
"Michael S. Tsirkin" <mst@redhat.com>,
Dave Hansen <dave.hansen@intel.com>,
Steven Rostedt <rostedt@goodmis.org>
Subject: Re: [RFC 2/2] x86_64: expand kernel stack to 16K
Date: Fri, 30 May 2014 08:36:38 +0900 [thread overview]
Message-ID: <20140529233638.GJ10092@bbox> (raw)
In-Reply-To: <20140529015830.GG6677@dastard>
Hello Dave,
On Thu, May 29, 2014 at 11:58:30AM +1000, Dave Chinner wrote:
> On Thu, May 29, 2014 at 11:30:07AM +1000, Dave Chinner wrote:
> > On Wed, May 28, 2014 at 03:41:11PM -0700, Linus Torvalds wrote:
> > commit a237c1c5bc5dc5c76a21be922dca4826f3eca8ca
> > Author: Jens Axboe <jaxboe@fusionio.com>
> > Date: Sat Apr 16 13:27:55 2011 +0200
> >
> > block: let io_schedule() flush the plug inline
> >
> > Linus correctly observes that the most important dispatch cases
> > are now done from kblockd, this isn't ideal for latency reasons.
> > The original reason for switching dispatches out-of-line was to
> > avoid too deep a stack, so by _only_ letting the "accidental"
> > flush directly in schedule() be guarded by offload to kblockd,
> > we should be able to get the best of both worlds.
> >
> > So add a blk_schedule_flush_plug() that offloads to kblockd,
> > and only use that from the schedule() path.
> >
> > Signed-off-by: Jens Axboe <jaxboe@fusionio.com>
> >
> > And now we have too deep a stack due to unplugging from io_schedule()...
>
> So, if we make io_schedule() push the plug list off to the kblockd
> like is done for schedule()....
>
> > > IOW, swap-out directly caused that extra 3kB of stack use in what was
> > > a deep call chain (due to memory allocation). I really don't
> > > understand why you are arguing anything else on a pure technicality.
> > >
> > > I thought you had some other argument for why swap was different, and
> > > against removing that "page_is_file_cache()" special case in
> > > shrink_page_list().
> >
> > I've said in the past that swap is different to filesystem
> > ->writepage implementations because it doesn't require significant
> > stack to do block allocation and doesn't trigger IO deep in that
> > allocation stack. Hence it has much lower stack overhead than the
> > filesystem ->writepage implementations and so is much less likely to
> > have stack issues.
> >
> > This stack overflow shows us that just the memory reclaim + IO
> > layers are sufficient to cause a stack overflow,
>
> .... we solve this problem directly by being able to remove the IO
> stack usage from the direct reclaim swap path.
>
> IOWs, we don't need to turn swap off at all in direct reclaim
> because all the swap IO can be captured in a plug list and
> dispatched via kblockd. This could be done either by io_schedule()
> or a new blk_flush_plug_list() wrapper that pushes the work to
> kblockd...
I did below hacky test to apply your idea and the result is overflow again.
So, again it would second stack expansion. Otherwise, we should prevent
swapout in direct reclaim.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index f5c6635b806c..95f169e85dbe 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4241,10 +4241,13 @@ EXPORT_SYMBOL_GPL(yield_to);
void __sched io_schedule(void)
{
struct rq *rq = raw_rq();
+ struct blk_plug *plug = current->plug;
delayacct_blkio_start();
atomic_inc(&rq->nr_iowait);
- blk_flush_plug(current);
+ if (plug)
+ blk_flush_plug_list(plug, true);
+
current->in_iowait = 1;
schedule();
current->in_iowait = 0;
[ 1209.764725] kworker/u24:0 (23627) used greatest stack depth: 304 bytes left
[ 1510.835509] kworker/u24:1 (25817) used greatest stack depth: 144 bytes left
[ 3701.482790] PANIC: double fault, error_code: 0x0
[ 3701.483297] CPU: 8 PID: 6117 Comm: kworker/u24:1 Not tainted 3.14.0+ #201
[ 3701.483980] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 3701.484366] Workqueue: writeback bdi_writeback_workfn (flush-253:0)
[ 3701.484366] task: ffff8800353c41c0 ti: ffff880000106000 task.ti: ffff880000106000
[ 3701.484366] RIP: 0010:[<ffffffff810a5390>] [<ffffffff810a5390>] __lock_acquire+0x170/0x1ca0
[ 3701.484366] RSP: 0000:ffff880000105f58 EFLAGS: 00010046
[ 3701.484366] RAX: 0000000000000001 RBX: ffff8800353c41c0 RCX: 0000000000000002
[ 3701.484366] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff81c4a1e0
[ 3701.484366] RBP: ffff880000106048 R08: 0000000000000001 R09: 0000000000000001
[ 3701.484366] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 3701.484366] R13: 0000000000000000 R14: ffffffff81c4a1e0 R15: 0000000000000000
[ 3701.484366] FS: 0000000000000000(0000) GS:ffff880037d00000(0000) knlGS:0000000000000000
[ 3701.484366] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 3701.484366] CR2: ffff880000105f48 CR3: 0000000001c0b000 CR4: 00000000000006e0
[ 3701.484366] Stack:
[ 3701.484366] BUG: unable to handle kernel paging request at ffff880000105f58
[ 3701.484366] IP: [<ffffffff81004e14>] show_stack_log_lvl+0x134/0x1a0
[ 3701.484366] PGD 28c5067 PUD 28c6067 PMD 28c7067 PTE 8000000000105060
[ 3701.484366] Thread overran stack, or stack corrupted
[ 3701.484366] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 3701.484366] Dumping ftrace buffer:
[ 3701.484366] ---------------------------------
[ 3701.484366] <...>-6117 8d..4 3786719374us : stack_trace_call: Depth Size Location (46 entries)
[ 3701.484366] ----- ---- --------
[ 3701.484366] <...>-6117 8d..4 3786719395us : stack_trace_call: 0) 7200 8 _raw_spin_lock_irqsave+0x51/0x60
[ 3701.484366] <...>-6117 8d..4 3786719395us : stack_trace_call: 1) 7192 296 get_page_from_freelist+0x886/0x920
[ 3701.484366] <...>-6117 8d..4 3786719395us : stack_trace_call: 2) 6896 352 __alloc_pages_nodemask+0x5e1/0xb20
[ 3701.484366] <...>-6117 8d..4 3786719396us : stack_trace_call: 3) 6544 8 alloc_pages_current+0x10f/0x1f0
[ 3701.484366] <...>-6117 8d..4 3786719396us : stack_trace_call: 4) 6536 168 new_slab+0x2c5/0x370
[ 3701.484366] <...>-6117 8d..4 3786719396us : stack_trace_call: 5) 6368 8 __slab_alloc+0x3a9/0x501
[ 3701.484366] <...>-6117 8d..4 3786719396us : stack_trace_call: 6) 6360 80 __kmalloc+0x1cb/0x200
[ 3701.484366] <...>-6117 8d..4 3786719396us : stack_trace_call: 7) 6280 376 vring_add_indirect+0x36/0x200
[ 3701.484366] <...>-6117 8d..4 3786719397us : stack_trace_call: 8) 5904 144 virtqueue_add_sgs+0x2e2/0x320
[ 3701.484366] <...>-6117 8d..4 3786719397us : stack_trace_call: 9) 5760 288 __virtblk_add_req+0xda/0x1b0
[ 3701.484366] <...>-6117 8d..4 3786719397us : stack_trace_call: 10) 5472 96 virtio_queue_rq+0xd3/0x1d0
[ 3701.484366] <...>-6117 8d..4 3786719397us : stack_trace_call: 11) 5376 128 __blk_mq_run_hw_queue+0x1ef/0x440
[ 3701.484366] <...>-6117 8d..4 3786719397us : stack_trace_call: 12) 5248 16 blk_mq_run_hw_queue+0x35/0x40
[ 3701.484366] <...>-6117 8d..4 3786719397us : stack_trace_call: 13) 5232 96 blk_mq_insert_requests+0xdb/0x160
[ 3701.484366] <...>-6117 8d..4 3786719398us : stack_trace_call: 14) 5136 112 blk_mq_flush_plug_list+0x12b/0x140
[ 3701.484366] <...>-6117 8d..4 3786719398us : stack_trace_call: 15) 5024 112 blk_flush_plug_list+0xc7/0x220
[ 3701.484366] <...>-6117 8d..4 3786719398us : stack_trace_call: 16) 4912 128 blk_mq_make_request+0x42a/0x600
[ 3701.484366] <...>-6117 8d..4 3786719398us : stack_trace_call: 17) 4784 48 generic_make_request+0xc0/0x100
[ 3701.484366] <...>-6117 8d..4 3786719398us : stack_trace_call: 18) 4736 112 submit_bio+0x86/0x160
[ 3701.484366] <...>-6117 8d..4 3786719398us : stack_trace_call: 19) 4624 160 __swap_writepage+0x198/0x230
[ 3701.484366] <...>-6117 8d..4 3786719399us : stack_trace_call: 20) 4464 32 swap_writepage+0x42/0x90
[ 3701.484366] <...>-6117 8d..4 3786719399us : stack_trace_call: 21) 4432 320 shrink_page_list+0x676/0xa80
[ 3701.484366] <...>-6117 8d..4 3786719399us : stack_trace_call: 22) 4112 208 shrink_inactive_list+0x262/0x4e0
[ 3701.484366] <...>-6117 8d..4 3786719399us : stack_trace_call: 23) 3904 304 shrink_lruvec+0x3e1/0x6a0
[ 3701.484366] <...>-6117 8d..4 3786719399us : stack_trace_call: 24) 3600 80 shrink_zone+0x3f/0x110
[ 3701.484366] <...>-6117 8d..4 3786719400us : stack_trace_call: 25) 3520 128 do_try_to_free_pages+0x156/0x4c0
[ 3701.484366] <...>-6117 8d..4 3786719400us : stack_trace_call: 26) 3392 208 try_to_free_pages+0xf7/0x1e0
[ 3701.484366] <...>-6117 8d..4 3786719400us : stack_trace_call: 27) 3184 352 __alloc_pages_nodemask+0x783/0xb20
[ 3701.484366] <...>-6117 8d..4 3786719400us : stack_trace_call: 28) 2832 8 alloc_pages_current+0x10f/0x1f0
[ 3701.484366] <...>-6117 8d..4 3786719400us : stack_trace_call: 29) 2824 200 __page_cache_alloc+0x13f/0x160
[ 3701.484366] <...>-6117 8d..4 3786719400us : stack_trace_call: 30) 2624 80 find_or_create_page+0x4c/0xb0
[ 3701.484366] <...>-6117 8d..4 3786719401us : stack_trace_call: 31) 2544 112 __getblk+0x109/0x2f0
[ 3701.484366] <...>-6117 8d..4 3786719401us : stack_trace_call: 32) 2432 224 ext4_ext_insert_extent+0x4d8/0x1270
[ 3701.484366] <...>-6117 8d..4 3786719401us : stack_trace_call: 33) 2208 256 ext4_ext_map_blocks+0x8d4/0x1010
[ 3701.484366] <...>-6117 8d..4 3786719401us : stack_trace_call: 34) 1952 160 ext4_map_blocks+0x325/0x530
[ 3701.484366] <...>-6117 8d..4 3786719401us : stack_trace_call: 35) 1792 384 ext4_writepages+0x6d1/0xce0
[ 3701.484366] <...>-6117 8d..4 3786719402us : stack_trace_call: 36) 1408 16 do_writepages+0x23/0x40
[ 3701.484366] <...>-6117 8d..4 3786719402us : stack_trace_call: 37) 1392 96 __writeback_single_inode+0x45/0x2e0
[ 3701.484366] <...>-6117 8d..4 3786719402us : stack_trace_call: 38) 1296 176 writeback_sb_inodes+0x2ad/0x500
[ 3701.484366] <...>-6117 8d..4 3786719402us : stack_trace_call: 39) 1120 80 __writeback_inodes_wb+0x9e/0xd0
[ 3701.484366] <...>-6117 8d..4 3786719402us : stack_trace_call: 40) 1040 160 wb_writeback+0x29b/0x350
[ 3701.484366] <...>-6117 8d..4 3786719402us : stack_trace_call: 41) 880 208 bdi_writeback_workfn+0x11c/0x480
[ 3701.484366] <...>-6117 8d..4 3786719403us : stack_trace_call: 42) 672 144 process_one_work+0x1d2/0x570
[ 3701.484366] <...>-6117 8d..4 3786719403us : stack_trace_call: 43) 528 112 worker_thread+0x116/0x370
[ 3701.484366] <...>-6117 8d..4 3786719403us : stack_trace_call: 44) 416 240 kthread+0xf3/0x110
[ 3701.484366] <...>-6117 8d..4 3786719403us : stack_trace_call: 45) 176 176 ret_from_fork+0x7c/0xb0
[ 3701.484366] ---------------------------------
[ 3701.484366] Modules linked in:
[ 3701.484366] CPU: 8 PID: 6117 Comm: kworker/u24:1 Not tainted 3.14.0+ #201
[ 3701.484366] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 3701.484366] Workqueue: writeback bdi_writeback_workfn (flush-253:0)
[ 3701.484366] task: ffff8800353c41c0 ti: ffff880000106000 task.ti: ffff880000106000
[ 3701.484366] RIP: 0010:[<ffffffff81004e14>] [<ffffffff81004e14>] show_stack_log_lvl+0x134/0x1a0
[ 3701.484366] RSP: 0000:ffff880037d06e58 EFLAGS: 00010046
[ 3701.484366] RAX: 000000000000000c RBX: 0000000000000000 RCX: 0000000000000000
[ 3701.484366] RDX: ffff880037cfffc0 RSI: ffff880037d06f58 RDI: 0000000000000000
[ 3701.484366] RBP: ffff880037d06ea8 R08: ffffffff81a0804c R09: 0000000000000000
[ 3701.484366] R10: 0000000000000002 R11: 0000000000000002 R12: ffff880037d06f58
[ 3701.484366] R13: 0000000000000000 R14: ffff880000105f58 R15: ffff880037d03fc0
[ 3701.484366] FS: 0000000000000000(0000) GS:ffff880037d00000(0000) knlGS:0000000000000000
[ 3701.484366] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 3701.484366] CR2: ffff880000105f58 CR3: 0000000001c0b000 CR4: 00000000000006e0
[ 3701.484366] Stack:
[ 3701.484366] 0000000000000000 ffff880000105f58 ffff880037d06f58 ffff880000105f58
[ 3701.484366] ffff880000106000 ffff880037d06f58 0000000000000040 ffff880037d06f58
[ 3701.484366] ffff880000105f58 0000000000000000 ffff880037d06ef8 ffffffff81004f1c
[ 3701.484366] Call Trace:
[ 3701.484366] <#DF>
[ 3701.484366] [<ffffffff81004f1c>] show_regs+0x9c/0x1f0
[ 3701.484366] [<ffffffff8103aa37>] df_debug+0x27/0x40
[ 3701.484366] [<ffffffff81003361>] do_double_fault+0x61/0x80
[ 3701.484366] [<ffffffff816f0907>] double_fault+0x27/0x30
[ 3701.484366] [<ffffffff810a5390>] ? __lock_acquire+0x170/0x1ca0
[ 3701.484366] <<EOE>>
--
Kind regards,
Minchan Kim
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2014-05-29 23:36 UTC|newest]
Thread overview: 97+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-28 6:53 [PATCH 1/2] ftrace: print stack usage right before Oops Minchan Kim
2014-05-28 6:53 ` [RFC 2/2] x86_64: expand kernel stack to 16K Minchan Kim
2014-05-28 8:37 ` Dave Chinner
2014-05-28 9:13 ` Dave Chinner
2014-05-28 16:06 ` Johannes Weiner
2014-05-28 21:55 ` Dave Chinner
2014-05-29 6:06 ` Minchan Kim
2014-05-28 9:04 ` Michael S. Tsirkin
2014-05-29 1:09 ` Minchan Kim
2014-05-29 2:44 ` Steven Rostedt
2014-05-29 4:11 ` Minchan Kim
2014-05-29 2:47 ` Rusty Russell
2014-05-28 9:27 ` Borislav Petkov
2014-05-29 13:23 ` One Thousand Gnomes
2014-05-28 14:14 ` Steven Rostedt
2014-05-28 14:23 ` H. Peter Anvin
2014-05-28 22:11 ` Dave Chinner
2014-05-28 22:42 ` H. Peter Anvin
2014-05-28 23:17 ` Dave Chinner
2014-05-28 23:21 ` H. Peter Anvin
2014-05-28 15:43 ` Richard Weinberger
2014-05-28 16:08 ` Steven Rostedt
2014-05-28 16:11 ` Richard Weinberger
2014-05-28 16:13 ` Linus Torvalds
2014-05-28 16:09 ` Linus Torvalds
2014-05-28 22:31 ` Dave Chinner
2014-05-28 22:41 ` Linus Torvalds
2014-05-29 1:30 ` Dave Chinner
2014-05-29 1:58 ` Dave Chinner
2014-05-29 2:51 ` Linus Torvalds
2014-05-29 23:36 ` Minchan Kim [this message]
2014-05-30 0:05 ` Linus Torvalds
2014-05-30 0:20 ` Minchan Kim
2014-05-30 0:31 ` Linus Torvalds
2014-05-30 0:50 ` Minchan Kim
2014-05-30 1:24 ` Linus Torvalds
2014-05-30 1:58 ` Dave Chinner
2014-05-30 2:13 ` Linus Torvalds
2014-05-30 6:21 ` Minchan Kim
2014-05-30 1:30 ` Linus Torvalds
2014-05-30 0:15 ` Dave Chinner
2014-05-30 2:12 ` Minchan Kim
2014-05-30 4:37 ` Linus Torvalds
2014-05-31 1:45 ` Linus Torvalds
2014-05-30 6:12 ` Minchan Kim
2014-06-03 13:28 ` Rasmus Villemoes
2014-06-03 19:04 ` Linus Torvalds
2014-05-29 2:42 ` Linus Torvalds
2014-05-29 5:14 ` H. Peter Anvin
2014-05-29 6:01 ` Rusty Russell
2014-05-29 7:26 ` virtio ring cleanups, which save stack on older gcc Rusty Russell
2014-05-29 7:26 ` [PATCH 1/4] Hack: measure stack taken by vring from virtio_blk Rusty Russell
2014-05-29 15:39 ` Linus Torvalds
2014-05-29 7:26 ` [PATCH 2/4] virtio_net: pass well-formed sg to virtqueue_add_inbuf() Rusty Russell
2014-05-29 10:07 ` Michael S. Tsirkin
2014-05-29 7:26 ` [PATCH 3/4] virtio_ring: assume sgs are always well-formed Rusty Russell
2014-05-29 11:18 ` Michael S. Tsirkin
2014-05-29 7:26 ` [PATCH 4/4] virtio_ring: unify direct/indirect code paths Rusty Russell
2014-05-29 7:52 ` Peter Zijlstra
2014-05-29 11:05 ` Rusty Russell
2014-05-29 11:33 ` Michael S. Tsirkin
2014-05-29 11:29 ` Michael S. Tsirkin
2014-05-30 2:37 ` Rusty Russell
2014-05-29 7:41 ` virtio ring cleanups, which save stack on older gcc Minchan Kim
2014-05-29 10:39 ` Dave Chinner
2014-05-29 11:08 ` Rusty Russell
2014-05-29 23:45 ` Minchan Kim
2014-05-30 1:06 ` Minchan Kim
2014-05-30 6:56 ` Rusty Russell
2014-05-29 7:26 ` [RFC 2/2] x86_64: expand kernel stack to 16K Dave Chinner
2014-05-29 15:24 ` Linus Torvalds
2014-05-29 23:40 ` Minchan Kim
2014-05-29 23:53 ` Dave Chinner
2014-05-30 0:06 ` Dave Jones
2014-05-30 0:21 ` Dave Chinner
2014-05-30 0:29 ` Dave Jones
2014-05-30 0:32 ` Minchan Kim
2014-05-30 1:34 ` Dave Chinner
2014-05-30 15:25 ` H. Peter Anvin
2014-05-30 15:41 ` Linus Torvalds
2014-05-30 15:52 ` H. Peter Anvin
2014-05-30 16:06 ` Linus Torvalds
2014-05-30 17:24 ` Dave Hansen
2014-05-30 18:12 ` H. Peter Anvin
2014-05-30 9:48 ` Richard Weinberger
2014-05-30 15:36 ` Linus Torvalds
2014-05-31 2:06 ` Jens Axboe
2014-06-02 22:59 ` Dave Chinner
2014-06-03 13:02 ` Konstantin Khlebnikov
2014-05-29 3:46 ` Minchan Kim
2014-05-29 4:13 ` Linus Torvalds
2014-05-29 5:10 ` Minchan Kim
2014-05-30 21:23 ` Andi Kleen
2014-05-28 16:18 ` [PATCH 1/2] ftrace: print stack usage right before Oops Steven Rostedt
2014-05-29 3:52 ` Minchan Kim
2014-05-29 3:01 ` Steven Rostedt
2014-05-29 3:49 ` Minchan Kim
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140529233638.GJ10092@bbox \
--to=minchan@kernel.org \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=dave.hansen@intel.com \
--cc=david@fromorbit.com \
--cc=hannes@cmpxchg.org \
--cc=hpa@zytor.com \
--cc=hughd@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=mingo@kernel.org \
--cc=mst@redhat.com \
--cc=riel@redhat.com \
--cc=rostedt@goodmis.org \
--cc=rusty@rustcorp.com.au \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).