From: "Michael S. Tsirkin" <mst@redhat.com>
To: Minchan Kim <minchan@kernel.org>
Cc: linux-kernel@vger.kernel.org,
Andrew Morton <akpm@linux-foundation.org>,
linux-mm@kvack.org, "H. Peter Anvin" <hpa@zytor.com>,
Ingo Molnar <mingo@kernel.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
Mel Gorman <mgorman@suse.de>, Rik van Riel <riel@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Hugh Dickins <hughd@google.com>,
rusty@rustcorp.com.au, Dave Hansen <dave.hansen@intel.com>,
Steven Rostedt <rostedt@goodmis.org>
Subject: Re: [RFC 2/2] x86_64: expand kernel stack to 16K
Date: Wed, 28 May 2014 12:04:09 +0300 [thread overview]
Message-ID: <20140528090409.GA16795@redhat.com> (raw)
In-Reply-To: <1401260039-18189-2-git-send-email-minchan@kernel.org>
On Wed, May 28, 2014 at 03:53:59PM +0900, Minchan Kim wrote:
> While I play inhouse patches with much memory pressure on qemu-kvm,
> 3.14 kernel was randomly crashed. The reason was kernel stack overflow.
>
> When I investigated the problem, the callstack was a little bit deeper
> by involve with reclaim functions but not direct reclaim path.
>
> I tried to diet stack size of some functions related with alloc/reclaim
> so did a hundred of byte but overflow was't disappeard so that I encounter
> overflow by another deeper callstack on reclaim/allocator path.
>
> Of course, we might sweep every sites we have found for reducing
> stack usage but I'm not sure how long it saves the world(surely,
> lots of developer start to add nice features which will use stack
> agains) and if we consider another more complex feature in I/O layer
> and/or reclaim path, it might be better to increase stack size(
> meanwhile, stack usage on 64bit machine was doubled compared to 32bit
> while it have sticked to 8K. Hmm, it's not a fair to me and arm64
> already expaned to 16K. )
>
> So, my stupid idea is just let's expand stack size and keep an eye
> toward stack consumption on each kernel functions via stacktrace of ftrace.
> For example, we can have a bar like that each funcion shouldn't exceed 200K
> and emit the warning when some function consumes more in runtime.
> Of course, it could make false positive but at least, it could make a
> chance to think over it.
>
> I guess this topic was discussed several time so there might be
> strong reason not to increase kernel stack size on x86_64, for me not
> knowing so Ccing x86_64 maintainers, other MM guys and virtio
> maintainers.
>
> [ 1065.604404] kworker/-5766 0d..2 1071625990us : stack_trace_call: Depth Size Location (51 entries)
> [ 1065.604404] ----- ---- --------
> [ 1065.604404] kworker/-5766 0d..2 1071625991us : stack_trace_call: 0) 7696 16 lookup_address+0x28/0x30
> [ 1065.604404] kworker/-5766 0d..2 1071625991us : stack_trace_call: 1) 7680 16 _lookup_address_cpa.isra.3+0x3b/0x40
> [ 1065.604404] kworker/-5766 0d..2 1071625991us : stack_trace_call: 2) 7664 24 __change_page_attr_set_clr+0xe0/0xb50
> [ 1065.604404] kworker/-5766 0d..2 1071625991us : stack_trace_call: 3) 7640 392 kernel_map_pages+0x6c/0x120
> [ 1065.604404] kworker/-5766 0d..2 1071625992us : stack_trace_call: 4) 7248 256 get_page_from_freelist+0x489/0x920
> [ 1065.604404] kworker/-5766 0d..2 1071625992us : stack_trace_call: 5) 6992 352 __alloc_pages_nodemask+0x5e1/0xb20
> [ 1065.604404] kworker/-5766 0d..2 1071625992us : stack_trace_call: 6) 6640 8 alloc_pages_current+0x10f/0x1f0
> [ 1065.604404] kworker/-5766 0d..2 1071625992us : stack_trace_call: 7) 6632 168 new_slab+0x2c5/0x370
> [ 1065.604404] kworker/-5766 0d..2 1071625992us : stack_trace_call: 8) 6464 8 __slab_alloc+0x3a9/0x501
> [ 1065.604404] kworker/-5766 0d..2 1071625993us : stack_trace_call: 9) 6456 80 __kmalloc+0x1cb/0x200
> [ 1065.604404] kworker/-5766 0d..2 1071625993us : stack_trace_call: 10) 6376 376 vring_add_indirect+0x36/0x200
> [ 1065.604404] kworker/-5766 0d..2 1071625993us : stack_trace_call: 11) 6000 144 virtqueue_add_sgs+0x2e2/0x320
> [ 1065.604404] kworker/-5766 0d..2 1071625993us : stack_trace_call: 12) 5856 288 __virtblk_add_req+0xda/0x1b0
> [ 1065.604404] kworker/-5766 0d..2 1071625993us : stack_trace_call: 13) 5568 96 virtio_queue_rq+0xd3/0x1d0
virtio stack usage seems very high.
Here is virtio_ring.su generated using -fstack-usage flag for gcc 4.8.2.
virtio_ring.c:107:35:sg_next_arr 16 static
<--- this is a surprise, I really expected it to be inlined
same for sg_next_chained.
<--- Rusty: should we force compiler to inline it?
virtio_ring.c:584:6:virtqueue_disable_cb 16 static
virtio_ring.c:604:10:virtqueue_enable_cb_prepare 16 static
virtio_ring.c:632:6:virtqueue_poll 16 static
virtio_ring.c:652:6:virtqueue_enable_cb 16 static
virtio_ring.c:845:14:virtqueue_get_vring_size 16 static
virtio_ring.c:854:6:virtqueue_is_broken 16 static
virtio_ring.c:101:35:sg_next_chained 16 static
virtio_ring.c:436:6:virtqueue_notify 24 static
virtio_ring.c:672:6:virtqueue_enable_cb_delayed 16 static
virtio_ring.c:820:6:vring_transport_features 16 static
virtio_ring.c:472:13:detach_buf 40 static
virtio_ring.c:518:7:virtqueue_get_buf 32 static
virtio_ring.c:812:6:vring_del_virtqueue 16 static
virtio_ring.c:394:6:virtqueue_kick_prepare 16 static
virtio_ring.c:464:6:virtqueue_kick 32 static
virtio_ring.c:186:19:4 16 static
virtio_ring.c:733:13:vring_interrupt 24 static
virtio_ring.c:707:7:virtqueue_detach_unused_buf 32 static
virtio_config.h:84:20:7 16 static
virtio_ring.c:753:19:vring_new_virtqueue 80 static
virtio_ring.c:374:5:virtqueue_add_inbuf 56 static
virtio_ring.c:352:5:virtqueue_add_outbuf 56 static
virtio_ring.c:314:5:virtqueue_add_sgs 112 static
as you see, vring_add_indirect was inlined within virtqueue_add_sgs by my gcc.
Taken together, they add up to only 112 bytes: not 1/2K as they do for you.
Which compiler version and flags did you use?
> [ 1065.604404] kworker/-5766 0d..2 1071625994us : stack_trace_call: 14) 5472 128 __blk_mq_run_hw_queue+0x1ef/0x440
> [ 1065.604404] kworker/-5766 0d..2 1071625994us : stack_trace_call: 15) 5344 16 blk_mq_run_hw_queue+0x35/0x40
> [ 1065.604404] kworker/-5766 0d..2 1071625994us : stack_trace_call: 16) 5328 96 blk_mq_insert_requests+0xdb/0x160
> [ 1065.604404] kworker/-5766 0d..2 1071625994us : stack_trace_call: 17) 5232 112 blk_mq_flush_plug_list+0x12b/0x140
> [ 1065.604404] kworker/-5766 0d..2 1071625994us : stack_trace_call: 18) 5120 112 blk_flush_plug_list+0xc7/0x220
> [ 1065.604404] kworker/-5766 0d..2 1071625995us : stack_trace_call: 19) 5008 64 io_schedule_timeout+0x88/0x100
> [ 1065.604404] kworker/-5766 0d..2 1071625995us : stack_trace_call: 20) 4944 128 mempool_alloc+0x145/0x170
> [ 1065.604404] kworker/-5766 0d..2 1071625995us : stack_trace_call: 21) 4816 96 bio_alloc_bioset+0x10b/0x1d0
> [ 1065.604404] kworker/-5766 0d..2 1071625995us : stack_trace_call: 22) 4720 48 get_swap_bio+0x30/0x90
> [ 1065.604404] kworker/-5766 0d..2 1071625995us : stack_trace_call: 23) 4672 160 __swap_writepage+0x150/0x230
> [ 1065.604404] kworker/-5766 0d..2 1071625996us : stack_trace_call: 24) 4512 32 swap_writepage+0x42/0x90
> [ 1065.604404] kworker/-5766 0d..2 1071625996us : stack_trace_call: 25) 4480 320 shrink_page_list+0x676/0xa80
> [ 1065.604404] kworker/-5766 0d..2 1071625996us : stack_trace_call: 26) 4160 208 shrink_inactive_list+0x262/0x4e0
> [ 1065.604404] kworker/-5766 0d..2 1071625996us : stack_trace_call: 27) 3952 304 shrink_lruvec+0x3e1/0x6a0
> [ 1065.604404] kworker/-5766 0d..2 1071625996us : stack_trace_call: 28) 3648 80 shrink_zone+0x3f/0x110
> [ 1065.604404] kworker/-5766 0d..2 1071625997us : stack_trace_call: 29) 3568 128 do_try_to_free_pages+0x156/0x4c0
> [ 1065.604404] kworker/-5766 0d..2 1071625997us : stack_trace_call: 30) 3440 208 try_to_free_pages+0xf7/0x1e0
> [ 1065.604404] kworker/-5766 0d..2 1071625997us : stack_trace_call: 31) 3232 352 __alloc_pages_nodemask+0x783/0xb20
> [ 1065.604404] kworker/-5766 0d..2 1071625997us : stack_trace_call: 32) 2880 8 alloc_pages_current+0x10f/0x1f0
> [ 1065.604404] kworker/-5766 0d..2 1071625997us : stack_trace_call: 33) 2872 200 __page_cache_alloc+0x13f/0x160
> [ 1065.604404] kworker/-5766 0d..2 1071625998us : stack_trace_call: 34) 2672 80 find_or_create_page+0x4c/0xb0
> [ 1065.604404] kworker/-5766 0d..2 1071625998us : stack_trace_call: 35) 2592 80 ext4_mb_load_buddy+0x1e9/0x370
> [ 1065.604404] kworker/-5766 0d..2 1071625998us : stack_trace_call: 36) 2512 176 ext4_mb_regular_allocator+0x1b7/0x460
> [ 1065.604404] kworker/-5766 0d..2 1071625998us : stack_trace_call: 37) 2336 128 ext4_mb_new_blocks+0x458/0x5f0
> [ 1065.604404] kworker/-5766 0d..2 1071625998us : stack_trace_call: 38) 2208 256 ext4_ext_map_blocks+0x70b/0x1010
> [ 1065.604404] kworker/-5766 0d..2 1071625999us : stack_trace_call: 39) 1952 160 ext4_map_blocks+0x325/0x530
> [ 1065.604404] kworker/-5766 0d..2 1071625999us : stack_trace_call: 40) 1792 384 ext4_writepages+0x6d1/0xce0
> [ 1065.604404] kworker/-5766 0d..2 1071625999us : stack_trace_call: 41) 1408 16 do_writepages+0x23/0x40
> [ 1065.604404] kworker/-5766 0d..2 1071625999us : stack_trace_call: 42) 1392 96 __writeback_single_inode+0x45/0x2e0
> [ 1065.604404] kworker/-5766 0d..2 1071625999us : stack_trace_call: 43) 1296 176 writeback_sb_inodes+0x2ad/0x500
> [ 1065.604404] kworker/-5766 0d..2 1071626000us : stack_trace_call: 44) 1120 80 __writeback_inodes_wb+0x9e/0xd0
> [ 1065.604404] kworker/-5766 0d..2 1071626000us : stack_trace_call: 45) 1040 160 wb_writeback+0x29b/0x350
> [ 1065.604404] kworker/-5766 0d..2 1071626000us : stack_trace_call: 46) 880 208 bdi_writeback_workfn+0x11c/0x480
> [ 1065.604404] kworker/-5766 0d..2 1071626000us : stack_trace_call: 47) 672 144 process_one_work+0x1d2/0x570
> [ 1065.604404] kworker/-5766 0d..2 1071626000us : stack_trace_call: 48) 528 112 worker_thread+0x116/0x370
> [ 1065.604404] kworker/-5766 0d..2 1071626001us : stack_trace_call: 49) 416 240 kthread+0xf3/0x110
> [ 1065.604404] kworker/-5766 0d..2 1071626001us : stack_trace_call: 50) 176 176 ret_from_fork+0x7c/0xb0
>
> Signed-off-by: Minchan Kim <minchan@kernel.org>
> ---
> arch/x86/include/asm/page_64_types.h | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/page_64_types.h b/arch/x86/include/asm/page_64_types.h
> index 8de6d9cf3b95..678205195ae1 100644
> --- a/arch/x86/include/asm/page_64_types.h
> +++ b/arch/x86/include/asm/page_64_types.h
> @@ -1,7 +1,7 @@
> #ifndef _ASM_X86_PAGE_64_DEFS_H
> #define _ASM_X86_PAGE_64_DEFS_H
>
> -#define THREAD_SIZE_ORDER 1
> +#define THREAD_SIZE_ORDER 2
> #define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)
> #define CURRENT_MASK (~(THREAD_SIZE - 1))
>
> --
> 1.9.2
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2014-05-28 9:05 UTC|newest]
Thread overview: 97+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-28 6:53 [PATCH 1/2] ftrace: print stack usage right before Oops Minchan Kim
2014-05-28 6:53 ` [RFC 2/2] x86_64: expand kernel stack to 16K Minchan Kim
2014-05-28 8:37 ` Dave Chinner
2014-05-28 9:13 ` Dave Chinner
2014-05-28 16:06 ` Johannes Weiner
2014-05-28 21:55 ` Dave Chinner
2014-05-29 6:06 ` Minchan Kim
2014-05-28 9:04 ` Michael S. Tsirkin [this message]
2014-05-29 1:09 ` Minchan Kim
2014-05-29 2:44 ` Steven Rostedt
2014-05-29 4:11 ` Minchan Kim
2014-05-29 2:47 ` Rusty Russell
2014-05-28 9:27 ` Borislav Petkov
2014-05-29 13:23 ` One Thousand Gnomes
2014-05-28 14:14 ` Steven Rostedt
2014-05-28 14:23 ` H. Peter Anvin
2014-05-28 22:11 ` Dave Chinner
2014-05-28 22:42 ` H. Peter Anvin
2014-05-28 23:17 ` Dave Chinner
2014-05-28 23:21 ` H. Peter Anvin
2014-05-28 15:43 ` Richard Weinberger
2014-05-28 16:08 ` Steven Rostedt
2014-05-28 16:11 ` Richard Weinberger
2014-05-28 16:13 ` Linus Torvalds
2014-05-28 16:09 ` Linus Torvalds
2014-05-28 22:31 ` Dave Chinner
2014-05-28 22:41 ` Linus Torvalds
2014-05-29 1:30 ` Dave Chinner
2014-05-29 1:58 ` Dave Chinner
2014-05-29 2:51 ` Linus Torvalds
2014-05-29 23:36 ` Minchan Kim
2014-05-30 0:05 ` Linus Torvalds
2014-05-30 0:20 ` Minchan Kim
2014-05-30 0:31 ` Linus Torvalds
2014-05-30 0:50 ` Minchan Kim
2014-05-30 1:24 ` Linus Torvalds
2014-05-30 1:58 ` Dave Chinner
2014-05-30 2:13 ` Linus Torvalds
2014-05-30 6:21 ` Minchan Kim
2014-05-30 1:30 ` Linus Torvalds
2014-05-30 0:15 ` Dave Chinner
2014-05-30 2:12 ` Minchan Kim
2014-05-30 4:37 ` Linus Torvalds
2014-05-31 1:45 ` Linus Torvalds
2014-05-30 6:12 ` Minchan Kim
2014-06-03 13:28 ` Rasmus Villemoes
2014-06-03 19:04 ` Linus Torvalds
2014-05-29 2:42 ` Linus Torvalds
2014-05-29 5:14 ` H. Peter Anvin
2014-05-29 6:01 ` Rusty Russell
2014-05-29 7:26 ` virtio ring cleanups, which save stack on older gcc Rusty Russell
2014-05-29 7:26 ` [PATCH 1/4] Hack: measure stack taken by vring from virtio_blk Rusty Russell
2014-05-29 15:39 ` Linus Torvalds
2014-05-29 7:26 ` [PATCH 2/4] virtio_net: pass well-formed sg to virtqueue_add_inbuf() Rusty Russell
2014-05-29 10:07 ` Michael S. Tsirkin
2014-05-29 7:26 ` [PATCH 3/4] virtio_ring: assume sgs are always well-formed Rusty Russell
2014-05-29 11:18 ` Michael S. Tsirkin
2014-05-29 7:26 ` [PATCH 4/4] virtio_ring: unify direct/indirect code paths Rusty Russell
2014-05-29 7:52 ` Peter Zijlstra
2014-05-29 11:05 ` Rusty Russell
2014-05-29 11:33 ` Michael S. Tsirkin
2014-05-29 11:29 ` Michael S. Tsirkin
2014-05-30 2:37 ` Rusty Russell
2014-05-29 7:41 ` virtio ring cleanups, which save stack on older gcc Minchan Kim
2014-05-29 10:39 ` Dave Chinner
2014-05-29 11:08 ` Rusty Russell
2014-05-29 23:45 ` Minchan Kim
2014-05-30 1:06 ` Minchan Kim
2014-05-30 6:56 ` Rusty Russell
2014-05-29 7:26 ` [RFC 2/2] x86_64: expand kernel stack to 16K Dave Chinner
2014-05-29 15:24 ` Linus Torvalds
2014-05-29 23:40 ` Minchan Kim
2014-05-29 23:53 ` Dave Chinner
2014-05-30 0:06 ` Dave Jones
2014-05-30 0:21 ` Dave Chinner
2014-05-30 0:29 ` Dave Jones
2014-05-30 0:32 ` Minchan Kim
2014-05-30 1:34 ` Dave Chinner
2014-05-30 15:25 ` H. Peter Anvin
2014-05-30 15:41 ` Linus Torvalds
2014-05-30 15:52 ` H. Peter Anvin
2014-05-30 16:06 ` Linus Torvalds
2014-05-30 17:24 ` Dave Hansen
2014-05-30 18:12 ` H. Peter Anvin
2014-05-30 9:48 ` Richard Weinberger
2014-05-30 15:36 ` Linus Torvalds
2014-05-31 2:06 ` Jens Axboe
2014-06-02 22:59 ` Dave Chinner
2014-06-03 13:02 ` Konstantin Khlebnikov
2014-05-29 3:46 ` Minchan Kim
2014-05-29 4:13 ` Linus Torvalds
2014-05-29 5:10 ` Minchan Kim
2014-05-30 21:23 ` Andi Kleen
2014-05-28 16:18 ` [PATCH 1/2] ftrace: print stack usage right before Oops Steven Rostedt
2014-05-29 3:52 ` Minchan Kim
2014-05-29 3:01 ` Steven Rostedt
2014-05-29 3:49 ` Minchan Kim
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140528090409.GA16795@redhat.com \
--to=mst@redhat.com \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=dave.hansen@intel.com \
--cc=hannes@cmpxchg.org \
--cc=hpa@zytor.com \
--cc=hughd@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=minchan@kernel.org \
--cc=mingo@kernel.org \
--cc=riel@redhat.com \
--cc=rostedt@goodmis.org \
--cc=rusty@rustcorp.com.au \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).