From: Max Kellermann <max@blarg.de>
To: Justin Piszcz <jpiszcz@lucidpixels.com>
Cc: "Kirill A. Shutemov" <kirill@shutemov.name>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: 5.1 kernel: khugepaged stuck at 100%
Date: Thu, 6 Jun 2019 19:24:40 +0200 [thread overview]
Message-ID: <20190606172440.GA24838@swift.blarg.de> (raw)
In-Reply-To: <CAO9zADww2v2ckHsNDwRgiyMr9b3JH1xOOSiRJ0Uh2XZT5c=MEQ@mail.gmail.com>
On 2019/05/16 16:14, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
> Kernel: 5.1.2
>
> $ sudo cat /proc/$(pidof khugepaged)/stack
> [<0>] 0xffffffffffffffff
>
> $ perf top
>
> PerfTop: 3716 irqs/sec kernel:92.9% exact: 99.1% lost: 68/68
> drop: 0/0 [4000Hz cycles], (all, 12 CPUs)
> -------------------------------------------------------------------------------
>
> 47.53% [kernel] [k] compaction_alloc
> 38.88% [kernel] [k] __pageblock_pfn_to_page
> 6.68% [kernel] [k] nmi
> 0.58% [kernel] [k] __list_del_entry_valid
> 0.48% [kernel] [k] format_decode
> 0.39% [kernel] [k] __rb_insert_augmented
> 0.25% libdbus-1.so.3.19.9 [.] _dbus_string_hex_decode
> 0.24% [kernel] [k] entry_SYSCALL_64_after_hwframe
> 0.20% perf [.] rb_next
> 0.19% perf [.] __symbols__insert
I have the same problem (kernel 5.1.7), but over here, it's a PHP
process, not khugepaged, which is looping inside compaction_alloc.
This is from "perf report":
100.00% 0.00% php-cgi7.0 php-cgi7.0 [.] 0x000055d0e88bc5ee
|
---0x55d0e88bc5ee
page_fault
__do_page_fault
handle_mm_fault
__handle_mm_fault
do_huge_pmd_anonymous_page
__alloc_pages_nodemask
__alloc_pages_slowpath
__alloc_pages_direct_compact
try_to_compact_pages
compact_zone_order
compact_zone
migrate_pages
compaction_alloc
|
|--24.43%--__pageblock_pfn_to_page
|
--2.22%--_cond_resched
|
--0.89%--rcu_all_qs
ftrace:
<...>-263514 [012] .... 109004.793009: rcu_all_qs <-_cond_resched
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793009: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793010: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: __pageblock_pfn_to_page <-compaction_alloc
<...>-263514 [012] .... 109004.793011: _cond_resched <-compaction_alloc
(Repeating this sequence over and over.)
Nothing useful in /proc/263514/{stack,wchan,syscall}.
What else can I do to collect more information to aid fix this?
Max
next prev parent reply other threads:[~2019-06-06 17:30 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-09 9:54 5.1 kernel: khugepaged stuck at 100% Justin Piszcz
2019-05-09 11:13 ` Kirill A. Shutemov
2019-05-15 7:25 ` Justin Piszcz
2019-05-16 14:14 ` Justin Piszcz
2019-05-16 14:17 ` Justin Piszcz
2019-05-16 14:27 ` Justin Piszcz
2019-05-24 11:43 ` Oleksandr Natalenko
2019-06-06 17:24 ` Max Kellermann [this message]
2019-06-07 7:40 ` Max Kellermann
2019-06-07 7:50 ` Oleksandr Natalenko
2019-06-07 7:53 ` Max Kellermann
2019-05-09 11:17 ` Justin Piszcz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190606172440.GA24838@swift.blarg.de \
--to=max@blarg.de \
--cc=jpiszcz@lucidpixels.com \
--cc=kirill@shutemov.name \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.