From: "Jason A. Donenfeld" <Jason@zx2c4.com>
To: Guenter Roeck <linux@roeck-us.net>
Cc: david@redhat.com, linux-kernel@vger.kernel.org,
wsa+renesas@sang-engineering.com, nicholas@linux.ibm.com,
windhl@126.com, cuigaosheng1@huawei.com, mikey@neuling.org,
paul@paul-moore.com, haren@linux.ibm.com, joel@jms.id.au,
lukas.bulwahn@gmail.com, nathanl@linux.ibm.com,
ajd@linux.ibm.com, ye.xingchen@zte.com.cn, npiggin@gmail.com,
nathan@kernel.org, rmclure@linux.ibm.com, hbathini@linux.ibm.com,
atrajeev@linux.vnet.ibm.com, yuanjilin@cdjrlc.com,
pali@kernel.org, farosas@linux.ibm.com, geoff@infradead.org,
Linus Torvalds <torvalds@linux-foundation.org>,
gustavoars@kernel.org, lihuafei1@huawei.com,
aneesh.kumar@linux.ibm.com, zhengyongjun3@huawei.com,
linuxppc-dev@lists.ozlabs.org
Subject: Re: [GIT PULL] Please pull powerpc/linux.git powerpc-6.1-1 tag
Date: Wed, 12 Oct 2022 12:37:50 -0600 [thread overview]
Message-ID: <Y0cJfoPLGeuzm/gQ@zx2c4.com> (raw)
In-Reply-To: <20221012174826.GB2995920@roeck-us.net>
On Wed, Oct 12, 2022 at 10:48:26AM -0700, Guenter Roeck wrote:
> > I've also managed to not hit this bug a few times. When it triggers,
> > after "kprobes: kprobe jump-optimization is enabled. All kprobes are
> > optimized if possible.", there's a long hang - tens seconds before it
> > continues. When it doesn't trigger, there's no hang at that point in the
> > boot process.
> >
>
> That probably explains why my attempts to bisect the problem were
> unsuccessful.
So I just did this:
diff --git a/drivers/char/random.c b/drivers/char/random.c
index 2fe28eeb2f38..2d70bc09db7e 100644
--- a/drivers/char/random.c
+++ b/drivers/char/random.c
@@ -1212,6 +1212,7 @@ static void __cold try_to_generate_entropy(void)
struct entropy_timer_state stack;
unsigned int i, num_different = 0;
unsigned long last = random_get_entropy();
+ return;
for (i = 0; i < NUM_TRIAL_SAMPLES - 1; ++i) {
stack.entropy = random_get_entropy();
And then ran it, and now we get the lockup from the idle process:
udhcpc: started, v1.33.0
udhcpc: sending discover
watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.0.0-28380-gde492c83cae0-dirty #10
Hardware name: PowerMac3,1 PPC970FX 0x3c0301 PowerMac
NIP: c0000000000300f8 LR: c0000000000304e8 CTR: c00000000001a410
REGS: c0000000028c79a8 TRAP: 0900 Not tainted (6.0.0-28380-gde492c83cae0-dirty)
MSR: 800000000000b032 <SF,EE,FP,ME,IR,DR,RI> CR: 24088442 XER: 00000000
IRQMASK: 0
GPR00: c0000000000304e8 c0000000028c7b30 c000000001435500 c0000000028c79a8
GPR04: c0000000013366c0 0000000000000000 000000000010029c 0000000000000000
GPR08: c000000002d3bbb0 0000000000000000 c000000002883d00 c000000002915500
GPR12: 0000000044088442 c000000002e00000 0000000000000007 0000000002295698
GPR16: 00000000039400e8 0000000002295258 0000000002295660 00000000022953d0
GPR20: 0000000002295b10 00000000022b34d0 0000000002295b38 0000000003945500
GPR24: 0000000003945500 0000000000080000 c000000002883d80 c000000002883d00
GPR28: c00000000290d0c0 0000000000000001 c00000000290d018 c00000000290cc78
NIP [c0000000000300f8] .replay_soft_interrupts+0x28/0x2d0
LR [c0000000000304e8] .arch_local_irq_restore+0x148/0x1a0
Call Trace:
[c0000000028c7b30] [c0000000000304e8] .arch_local_irq_restore+0x148/0x1a0 (unreliable)
[c0000000028c7bb0] [c00000000001a388] .arch_cpu_idle+0xb8/0x140
[c0000000028c7c30] [c000000000fd4940] .default_idle_call+0x80/0xc8
[c0000000028c7ca0] [c000000000148480] .do_idle+0x150/0x1a0
[c0000000028c7d50] [c000000000148748] .cpu_startup_entry+0x38/0x40
[c0000000028c7dd0] [c0000000000113a8] .rest_init+0x168/0x170
[c0000000028c7e60] [c000000002004224] .arch_post_acpi_subsys_init+0x0/0x24
[c0000000028c7ed0] [c000000002004ba8] .start_kernel+0x8d0/0x924
[c0000000028c7f90] [c00000000000d4ac] start_here_common+0x1c/0x20
Instruction dump:
60000000 60000000 7c0802a6 f8010010 f821fe01 60000000 60000000 38610078
e92d0af8 f92101f8 39200000 4803a491 <60000000> 39200000 e9410180 f92101b0
Kernel panic - not syncing: softlockup: hung tasks
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G L 6.0.0-28380-gde492c83cae0-dirty #10
Hardware name: PowerMac3,1 PPC970FX 0x3c0301 PowerMac
Call Trace:
[c0000000028c74a0] [c000000000f93b90] .dump_stack_lvl+0x7c/0xc4 (unreliable)
[c0000000028c7530] [c0000000000d2a58] .panic+0x180/0x438
[c0000000028c75e0] [c000000000232424] .watchdog_timer_fn+0x3a4/0x410
[c0000000028c76a0] [c0000000001cb964] .__hrtimer_run_queues+0x1f4/0x590
[c0000000028c77a0] [c0000000001cc354] .hrtimer_interrupt+0x134/0x300
[c0000000028c7860] [c000000000021cd4] .timer_interrupt+0x1c4/0x5d0
[c0000000028c7930] [c0000000000302f8] .replay_soft_interrupts+0x228/0x2d0
[c0000000028c7b30] [c0000000000304e8] .arch_local_irq_restore+0x148/0x1a0
[c0000000028c7bb0] [c00000000001a388] .arch_cpu_idle+0xb8/0x140
[c0000000028c7c30] [c000000000fd4940] .default_idle_call+0x80/0xc8
[c0000000028c7ca0] [c000000000148480] .do_idle+0x150/0x1a0
[c0000000028c7d50] [c000000000148748] .cpu_startup_entry+0x38/0x40
[c0000000028c7dd0] [c0000000000113a8] .rest_init+0x168/0x170
[c0000000028c7e60] [c000000002004224] .arch_post_acpi_subsys_init+0x0/0x24
[c0000000028c7ed0] [c000000002004ba8] .start_kernel+0x8d0/0x924
[c0000000028c7f90] [c00000
WARNING: multiple messages have this Message-ID (diff)
From: "Jason A. Donenfeld" <Jason@zx2c4.com>
To: Guenter Roeck <linux@roeck-us.net>
Cc: Michael Ellerman <mpe@ellerman.id.au>,
Linus Torvalds <torvalds@linux-foundation.org>,
ajd@linux.ibm.com, aneesh.kumar@linux.ibm.com,
atrajeev@linux.vnet.ibm.com, christophe.leroy@csgroup.eu,
cuigaosheng1@huawei.com, david@redhat.com, farosas@linux.ibm.com,
geoff@infradead.org, gustavoars@kernel.org, haren@linux.ibm.com,
hbathini@linux.ibm.com, joel@jms.id.au, lihuafei1@huawei.com,
linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
lukas.bulwahn@gmail.com, mikey@neuling.org, nathan@kernel.org,
nathanl@linux.ibm.com, nicholas@linux.ibm.com, npiggin@gmail.com,
pali@kernel.org, paul@paul-moore.com, rmclure@linux.ibm.com,
ruscur@russell.cc, windhl@126.com,
wsa+renesas@sang-engineering.com, ye.xingchen@zte.com.cn,
yuanjilin@cdjrlc.com, zhengyongjun3@huawei.com
Subject: Re: [GIT PULL] Please pull powerpc/linux.git powerpc-6.1-1 tag
Date: Wed, 12 Oct 2022 12:37:50 -0600 [thread overview]
Message-ID: <Y0cJfoPLGeuzm/gQ@zx2c4.com> (raw)
In-Reply-To: <20221012174826.GB2995920@roeck-us.net>
On Wed, Oct 12, 2022 at 10:48:26AM -0700, Guenter Roeck wrote:
> > I've also managed to not hit this bug a few times. When it triggers,
> > after "kprobes: kprobe jump-optimization is enabled. All kprobes are
> > optimized if possible.", there's a long hang - tens seconds before it
> > continues. When it doesn't trigger, there's no hang at that point in the
> > boot process.
> >
>
> That probably explains why my attempts to bisect the problem were
> unsuccessful.
So I just did this:
diff --git a/drivers/char/random.c b/drivers/char/random.c
index 2fe28eeb2f38..2d70bc09db7e 100644
--- a/drivers/char/random.c
+++ b/drivers/char/random.c
@@ -1212,6 +1212,7 @@ static void __cold try_to_generate_entropy(void)
struct entropy_timer_state stack;
unsigned int i, num_different = 0;
unsigned long last = random_get_entropy();
+ return;
for (i = 0; i < NUM_TRIAL_SAMPLES - 1; ++i) {
stack.entropy = random_get_entropy();
And then ran it, and now we get the lockup from the idle process:
udhcpc: started, v1.33.0
udhcpc: sending discover
watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.0.0-28380-gde492c83cae0-dirty #10
Hardware name: PowerMac3,1 PPC970FX 0x3c0301 PowerMac
NIP: c0000000000300f8 LR: c0000000000304e8 CTR: c00000000001a410
REGS: c0000000028c79a8 TRAP: 0900 Not tainted (6.0.0-28380-gde492c83cae0-dirty)
MSR: 800000000000b032 <SF,EE,FP,ME,IR,DR,RI> CR: 24088442 XER: 00000000
IRQMASK: 0
GPR00: c0000000000304e8 c0000000028c7b30 c000000001435500 c0000000028c79a8
GPR04: c0000000013366c0 0000000000000000 000000000010029c 0000000000000000
GPR08: c000000002d3bbb0 0000000000000000 c000000002883d00 c000000002915500
GPR12: 0000000044088442 c000000002e00000 0000000000000007 0000000002295698
GPR16: 00000000039400e8 0000000002295258 0000000002295660 00000000022953d0
GPR20: 0000000002295b10 00000000022b34d0 0000000002295b38 0000000003945500
GPR24: 0000000003945500 0000000000080000 c000000002883d80 c000000002883d00
GPR28: c00000000290d0c0 0000000000000001 c00000000290d018 c00000000290cc78
NIP [c0000000000300f8] .replay_soft_interrupts+0x28/0x2d0
LR [c0000000000304e8] .arch_local_irq_restore+0x148/0x1a0
Call Trace:
[c0000000028c7b30] [c0000000000304e8] .arch_local_irq_restore+0x148/0x1a0 (unreliable)
[c0000000028c7bb0] [c00000000001a388] .arch_cpu_idle+0xb8/0x140
[c0000000028c7c30] [c000000000fd4940] .default_idle_call+0x80/0xc8
[c0000000028c7ca0] [c000000000148480] .do_idle+0x150/0x1a0
[c0000000028c7d50] [c000000000148748] .cpu_startup_entry+0x38/0x40
[c0000000028c7dd0] [c0000000000113a8] .rest_init+0x168/0x170
[c0000000028c7e60] [c000000002004224] .arch_post_acpi_subsys_init+0x0/0x24
[c0000000028c7ed0] [c000000002004ba8] .start_kernel+0x8d0/0x924
[c0000000028c7f90] [c00000000000d4ac] start_here_common+0x1c/0x20
Instruction dump:
60000000 60000000 7c0802a6 f8010010 f821fe01 60000000 60000000 38610078
e92d0af8 f92101f8 39200000 4803a491 <60000000> 39200000 e9410180 f92101b0
Kernel panic - not syncing: softlockup: hung tasks
CPU: 0 PID: 0 Comm: swapper/0 Tainted: G L 6.0.0-28380-gde492c83cae0-dirty #10
Hardware name: PowerMac3,1 PPC970FX 0x3c0301 PowerMac
Call Trace:
[c0000000028c74a0] [c000000000f93b90] .dump_stack_lvl+0x7c/0xc4 (unreliable)
[c0000000028c7530] [c0000000000d2a58] .panic+0x180/0x438
[c0000000028c75e0] [c000000000232424] .watchdog_timer_fn+0x3a4/0x410
[c0000000028c76a0] [c0000000001cb964] .__hrtimer_run_queues+0x1f4/0x590
[c0000000028c77a0] [c0000000001cc354] .hrtimer_interrupt+0x134/0x300
[c0000000028c7860] [c000000000021cd4] .timer_interrupt+0x1c4/0x5d0
[c0000000028c7930] [c0000000000302f8] .replay_soft_interrupts+0x228/0x2d0
[c0000000028c7b30] [c0000000000304e8] .arch_local_irq_restore+0x148/0x1a0
[c0000000028c7bb0] [c00000000001a388] .arch_cpu_idle+0xb8/0x140
[c0000000028c7c30] [c000000000fd4940] .default_idle_call+0x80/0xc8
[c0000000028c7ca0] [c000000000148480] .do_idle+0x150/0x1a0
[c0000000028c7d50] [c000000000148748] .cpu_startup_entry+0x38/0x40
[c0000000028c7dd0] [c0000000000113a8] .rest_init+0x168/0x170
[c0000000028c7e60] [c000000002004224] .arch_post_acpi_subsys_init+0x0/0x24
[c0000000028c7ed0] [c000000002004ba8] .start_kernel+0x8d0/0x924
[c0000000028c7f90] [c00000
next prev parent reply other threads:[~2022-10-12 18:39 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-09 11:01 [GIT PULL] Please pull powerpc/linux.git powerpc-6.1-1 tag Michael Ellerman
2022-10-09 11:01 ` Michael Ellerman
2022-10-09 21:17 ` pr-tracker-bot
2022-10-09 21:17 ` pr-tracker-bot
2022-10-10 19:25 ` Jason A. Donenfeld
2022-10-10 19:25 ` Jason A. Donenfeld
2022-10-10 20:03 ` Jason A. Donenfeld
2022-10-10 20:03 ` Jason A. Donenfeld
2022-10-10 22:26 ` Jason A. Donenfeld
2022-10-10 22:26 ` Jason A. Donenfeld
2022-10-11 0:00 ` Andrew Donnellan
2022-10-11 0:00 ` Andrew Donnellan
2022-10-11 0:13 ` Jason A. Donenfeld
2022-10-11 0:13 ` Jason A. Donenfeld
2022-10-11 1:44 ` Michael Ellerman
2022-10-11 1:44 ` Michael Ellerman
2022-10-11 2:57 ` Jason A. Donenfeld
2022-10-11 2:57 ` Jason A. Donenfeld
2022-10-11 9:34 ` Michael Ellerman
2022-10-11 9:34 ` Michael Ellerman
2022-10-11 1:53 ` Michael Ellerman
2022-10-11 1:53 ` Michael Ellerman
2022-10-11 2:57 ` Jason A. Donenfeld
2022-10-11 2:57 ` Jason A. Donenfeld
2022-10-11 9:35 ` Michael Ellerman
2022-10-11 9:35 ` Michael Ellerman
2022-10-11 11:10 ` Nicholas Piggin
2022-10-11 11:10 ` Nicholas Piggin
2022-10-12 14:18 ` Guenter Roeck
2022-10-12 14:18 ` Guenter Roeck
2022-10-12 15:49 ` Jason A. Donenfeld
2022-10-12 15:49 ` Jason A. Donenfeld
2022-10-12 16:44 ` Guenter Roeck
2022-10-12 16:44 ` Guenter Roeck
2022-10-12 17:20 ` Jason A. Donenfeld
2022-10-12 17:20 ` Jason A. Donenfeld
2022-10-12 17:48 ` Guenter Roeck
2022-10-12 17:48 ` Guenter Roeck
2022-10-12 18:37 ` Jason A. Donenfeld [this message]
2022-10-12 18:37 ` Jason A. Donenfeld
2022-10-13 5:17 ` Nicholas Piggin
2022-10-13 5:17 ` Nicholas Piggin
2022-10-12 22:16 ` Guenter Roeck
2022-10-12 22:16 ` Guenter Roeck
2022-10-13 0:03 ` Michael Ellerman
2022-10-13 0:03 ` Michael Ellerman
2022-10-13 0:21 ` Guenter Roeck
2022-10-13 0:21 ` Guenter Roeck
2022-10-13 5:03 ` Nicholas Piggin
2022-10-13 5:03 ` Nicholas Piggin
2022-10-13 5:19 ` Jason A. Donenfeld
2022-10-13 5:19 ` Jason A. Donenfeld
2022-10-13 5:20 ` Guenter Roeck
2022-10-13 5:20 ` Guenter Roeck
2022-10-13 5:22 ` Nicholas Piggin
2022-10-13 5:22 ` Nicholas Piggin
2022-10-13 4:43 ` Guenter Roeck
2022-10-13 4:43 ` Guenter Roeck
2022-10-13 5:14 ` Nicholas Piggin
2022-10-13 5:14 ` Nicholas Piggin
2022-10-13 18:55 ` Guenter Roeck
2022-10-13 18:55 ` Guenter Roeck
2022-10-12 16:45 ` Jason A. Donenfeld
2022-10-12 16:45 ` Jason A. Donenfeld
2022-10-12 16:49 ` Guenter Roeck
2022-10-12 16:49 ` Guenter Roeck
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y0cJfoPLGeuzm/gQ@zx2c4.com \
--to=jason@zx2c4.com \
--cc=ajd@linux.ibm.com \
--cc=aneesh.kumar@linux.ibm.com \
--cc=atrajeev@linux.vnet.ibm.com \
--cc=cuigaosheng1@huawei.com \
--cc=david@redhat.com \
--cc=farosas@linux.ibm.com \
--cc=geoff@infradead.org \
--cc=gustavoars@kernel.org \
--cc=haren@linux.ibm.com \
--cc=hbathini@linux.ibm.com \
--cc=joel@jms.id.au \
--cc=lihuafei1@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux@roeck-us.net \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=lukas.bulwahn@gmail.com \
--cc=mikey@neuling.org \
--cc=nathan@kernel.org \
--cc=nathanl@linux.ibm.com \
--cc=nicholas@linux.ibm.com \
--cc=npiggin@gmail.com \
--cc=pali@kernel.org \
--cc=paul@paul-moore.com \
--cc=rmclure@linux.ibm.com \
--cc=torvalds@linux-foundation.org \
--cc=windhl@126.com \
--cc=wsa+renesas@sang-engineering.com \
--cc=ye.xingchen@zte.com.cn \
--cc=yuanjilin@cdjrlc.com \
--cc=zhengyongjun3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.