From: Juri Lelli <juri.lelli@redhat.com>
To: Huang Shijie <shijie@os.amperecomputing.com>
Cc: mingo@redhat.com, patches@amperecomputing.com,
peterz@infradead.org, vincent.guittot@linaro.org,
dietmar.eggemann@arm.com, rostedt@goodmis.org,
bsegall@google.com, mgorman@suse.de, vschneid@redhat.com,
vineeth@bitbyteword.org, bristot@kernel.org,
linux-kernel@vger.kernel.org, hongyan.xia2@arm.com
Subject: Re: [PATCH v3] sched/deadline: fix the hang in dl_task_of
Date: Thu, 29 Aug 2024 10:14:02 +0200 [thread overview]
Message-ID: <ZtAtymo-KB9c20Q8@jlelli-thinkpadt14gen4.remote.csb> (raw)
In-Reply-To: <20240829031111.12142-1-shijie@os.amperecomputing.com>
Hi,
On 29/08/24 11:11, Huang Shijie wrote:
> When we enable the schedstats, we will meet an OS hang like this:
> --------------------------------------------------------
> [ 134.104253] kernel BUG at kernel/sched/deadline.c:63!
> [ 134.132013] ------------[ cut here ]------------
> [ 134.133441] x27: 0000000000000001
> [ 134.138048] kernel BUG at kernel/sched/deadline.c:63!
> [ 134.146478] x26: 0000000000000001 x25: 0000000000000000 x24: 0000000000000001
> [ 134.153607] x23: 0000000000000001 x22: 0000000000000000 x21: 0000000000000001
> [ 134.160734] x20: ffff007dbf1b6d00 x19: ffff007dbf1b7610 x18: 0000000000000014
> [ 134.162027] ------------[ cut here ]------------
> [ 134.167861] x17: 000000009deab6cd x16: 00000000527c9a1c x15: 00000000000000dc
> [ 134.172473] kernel BUG at kernel/sched/deadline.c:63!
> [ 134.179595] x14: 0000000001200011 x13: 0000000040001000 x12: 0000ffffb6df05bc
> [ 134.191760] x11: ffff007dbf1b6d00 x10: ffff0001062dd2e8 x9 : ffff8000801215ac
> [ 134.192036] ------------[ cut here ]------------
> [ 134.198888] x8 : 0000000000000000 x7 : 0000000000000021 x6 : ffff0001764ed280
> [ 134.203498] kernel BUG at kernel/sched/deadline.c:63!
> [ 134.210622] x5 : 0000000000000000 x4 : 0000000000000001 x3 : ffff807d3dd24000
> [ 134.222787] x2 : 000000028b77a140 x1 : 0000003400000000 x0 : ffff007dbf1b6c80
> [ 134.229915] Call trace:
> [ 134.232353] dl_task_of.part.0+0x0/0x10
> [ 134.236182] dl_server_start+0x54/0x158
> [ 134.240013] enqueue_task_fair+0x138/0x420
> [ 134.244100] enqueue_task+0x44/0xb0
> [ 134.247584] wake_up_new_task+0x1c0/0x3a0
> [ 134.251584] kernel_clone+0xe8/0x3e8
> [ 134.252022] ------------[ cut here ]------------
> [ 134.255156] __do_sys_clone+0x70/0xa8
> [ 134.259764] kernel BUG at kernel/sched/deadline.c:63!
> [ 134.263412] __arm64_sys_clone+0x28/0x40
> [ 134.272360] invoke_syscall+0x50/0x120
> [ 134.276101] el0_svc_common+0x44/0xf8
> [ 134.279753] do_el0_svc+0x28/0x40
> [ 134.283058] el0_svc+0x40/0x150
> [ 134.286195] el0t_64_sync_handler+0x100/0x130
> [ 134.290546] el0t_64_sync+0x1a4/0x1a8
> [ 134.294200] Code: 35ffffa2 17ffffe3 d4210000 17ffffb4 (d4210000)
> [ 134.300283] ---[ end trace 0000000000000000 ]---
> [ 134.304890] Kernel panic - not syncing: Oops - BUG: Fatal exception
> [ 134.311147] SMP: stopping secondary CPUs
> [ 135.365096] SMP: failed to stop secondary CPUs 8-9,16,30,43,86,88,121,149
> [ 135.371884] Kernel Offset: disabled
> [ 135.375361] CPU features: 0x00,00100003,80153d29,d75ffea7
> [ 135.380749] Memory Limit: none
> [ 135.383793] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception ]
> --------------------------------------------------------
>
> In dl_server_start(), we set the dl_se->dl_server with 1. When schedstats
> is enabled, in the following:
> dl_server_start() --> enqueue_dl_entity() --> update_stats_enqueue_dl()
> __schedstats_from_dl_se() -->dl_task_of()
>
> we will meet the BUG_ON.
>
> Since the fair task has already had its own schedstats, there is no need
> to track anything for the associated dl_server.
>
> So add check in:
> update_stats_wait_start_dl()
> update_stats_wait_end_dl()
> update_stats_enqueue_dl()
> update_stats_dequeue_dl()
>
> return early for a dl_server dl_se.
>
> Tested this patch with memcached in Altra.
>
> Fixes: 5f6bd380c7bd ("sched/rt: Remove default bandwidth control")
> Signed-off-by: Huang Shijie <shijie@os.amperecomputing.com>
> ---
> v2 --> v3:
> Return early in:
> update_stats_wait_start_dl()
> update_stats_wait_end_dl()
> update_stats_enqueue_dl()
> update_stats_dequeue_dl()
This looks better, thanks.
Peter, what do you think?
Best,
Juri
next prev parent reply other threads:[~2024-08-29 8:14 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-08-29 3:11 [PATCH v3] sched/deadline: fix the hang in dl_task_of Huang Shijie
2024-08-29 8:14 ` Juri Lelli [this message]
2024-09-02 11:14 ` Peter Zijlstra
2024-09-02 12:49 ` Juri Lelli
2024-09-03 1:48 ` Shijie Huang
2024-09-03 13:38 ` [tip: sched/core] sched/deadline: Fix schedstats vs deadline servers tip-bot2 for Huang Shijie
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZtAtymo-KB9c20Q8@jlelli-thinkpadt14gen4.remote.csb \
--to=juri.lelli@redhat.com \
--cc=bristot@kernel.org \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=hongyan.xia2@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=patches@amperecomputing.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=shijie@os.amperecomputing.com \
--cc=vincent.guittot@linaro.org \
--cc=vineeth@bitbyteword.org \
--cc=vschneid@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox