From: Russell King - ARM Linux admin <linux@armlinux.org.uk>
To: Will Deacon <will@kernel.org>
Cc: linux-arch@vger.kernel.org, gregkh@linuxfoundation.org,
Xogium <contact@xogium.me>,
linux-kernel@vger.kernel.org, mingo@redhat.com, bp@alien8.de,
tglx@linutronix.de, linux-arm-kernel@lists.infradead.org
Subject: Re: [breakage] panic() does not halt arm64 systems under certain conditions
Date: Tue, 17 Sep 2019 11:51:36 +0100 [thread overview]
Message-ID: <20190917105136.GK25745@shell.armlinux.org.uk> (raw)
In-Reply-To: <20190917104518.ovg6ivadyst7h76o@willie-the-truck>
On Tue, Sep 17, 2019 at 11:45:19AM +0100, Will Deacon wrote:
> Hi,
>
> [Expanding CC list; original message is here:
> https://lore.kernel.org/linux-arm-kernel/BX1W47JXPMR8.58IYW53H6M5N@dragonstone/]
>
> On Mon, Sep 16, 2019 at 09:35:36PM -0400, Xogium wrote:
> > On arm64 in some situations userspace will continue running even after a
> > panic. This means any userspace watchdog daemon will continue pinging,
> > that service managers will keep running and displaying messages in certain
> > cases, and that it is possible to enter via ssh in the now unstable system
> > and to do almost anything except reboot/power off and etc. If
> > CONFIG_PREEMPT=n is set in the kernel's configuration, the issue is fixed.
> > I have reproduced the very same behavior with linux 4.19, 5.2 and 5.3. On
> > x86/x86_64 the issue does not seem to be present at all.
>
> I've managed to reproduce this under both 32-bit and 64-bit ARM kernels.
> The issue is that the infinite loop at the end of panic() can run with
> preemption enabled (particularly when invoking by echoing 'c' to
> /proc/sysrq-trigger), so we end up rescheduling user tasks. On x86, this
> doesn't happen because smp_send_stop() disables the local APIC in
> native_stop_other_cpus() and so interrupts are effectively masked while
> spinning.
>
> A straightforward fix is to disable preemption explicitly on the panic()
> path (diff below), but I've expanded the cc list to see both what others
> think,
Yep, and it looks like this bug goes back into the dim and distant past.
At least to the start of modern git history, 2.6.12-rc2.
> but also in case smp_send_stop() is supposed to have the side-effect
> of disabling interrupt delivery for the local CPU.
That can't fix it. Consider a preemptive non-SMP kernel.
smp_send_stop() becomes a no-op there.
I'd suggest that a preemptive UP kernel on x86 hardware will suffer
this same issue - it will be able to preempt out of this loop and
continue running userspace.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
WARNING: multiple messages have this Message-ID (diff)
From: Russell King - ARM Linux admin <linux@armlinux.org.uk>
To: Will Deacon <will@kernel.org>
Cc: Xogium <contact@xogium.me>,
linux-arm-kernel@lists.infradead.org, tglx@linutronix.de,
mingo@redhat.com, bp@alien8.de, gregkh@linuxfoundation.org,
linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [breakage] panic() does not halt arm64 systems under certain conditions
Date: Tue, 17 Sep 2019 11:51:36 +0100 [thread overview]
Message-ID: <20190917105136.GK25745@shell.armlinux.org.uk> (raw)
Message-ID: <20190917105136.vm_z7TPsI0iVuTTHudiyqmcSb7T2veB-Bnd-_C59UA4@z> (raw)
In-Reply-To: <20190917104518.ovg6ivadyst7h76o@willie-the-truck>
On Tue, Sep 17, 2019 at 11:45:19AM +0100, Will Deacon wrote:
> Hi,
>
> [Expanding CC list; original message is here:
> https://lore.kernel.org/linux-arm-kernel/BX1W47JXPMR8.58IYW53H6M5N@dragonstone/]
>
> On Mon, Sep 16, 2019 at 09:35:36PM -0400, Xogium wrote:
> > On arm64 in some situations userspace will continue running even after a
> > panic. This means any userspace watchdog daemon will continue pinging,
> > that service managers will keep running and displaying messages in certain
> > cases, and that it is possible to enter via ssh in the now unstable system
> > and to do almost anything except reboot/power off and etc. If
> > CONFIG_PREEMPT=n is set in the kernel's configuration, the issue is fixed.
> > I have reproduced the very same behavior with linux 4.19, 5.2 and 5.3. On
> > x86/x86_64 the issue does not seem to be present at all.
>
> I've managed to reproduce this under both 32-bit and 64-bit ARM kernels.
> The issue is that the infinite loop at the end of panic() can run with
> preemption enabled (particularly when invoking by echoing 'c' to
> /proc/sysrq-trigger), so we end up rescheduling user tasks. On x86, this
> doesn't happen because smp_send_stop() disables the local APIC in
> native_stop_other_cpus() and so interrupts are effectively masked while
> spinning.
>
> A straightforward fix is to disable preemption explicitly on the panic()
> path (diff below), but I've expanded the cc list to see both what others
> think,
Yep, and it looks like this bug goes back into the dim and distant past.
At least to the start of modern git history, 2.6.12-rc2.
> but also in case smp_send_stop() is supposed to have the side-effect
> of disabling interrupt delivery for the local CPU.
That can't fix it. Consider a preemptive non-SMP kernel.
smp_send_stop() becomes a no-op there.
I'd suggest that a preemptive UP kernel on x86 hardware will suffer
this same issue - it will be able to preempt out of this loop and
continue running userspace.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
WARNING: multiple messages have this Message-ID (diff)
From: Russell King - ARM Linux admin <linux@armlinux.org.uk>
To: Will Deacon <will@kernel.org>
Cc: linux-arch@vger.kernel.org, gregkh@linuxfoundation.org,
Xogium <contact@xogium.me>,
linux-kernel@vger.kernel.org, mingo@redhat.com, bp@alien8.de,
tglx@linutronix.de, linux-arm-kernel@lists.infradead.org
Subject: Re: [breakage] panic() does not halt arm64 systems under certain conditions
Date: Tue, 17 Sep 2019 11:51:36 +0100 [thread overview]
Message-ID: <20190917105136.GK25745@shell.armlinux.org.uk> (raw)
In-Reply-To: <20190917104518.ovg6ivadyst7h76o@willie-the-truck>
On Tue, Sep 17, 2019 at 11:45:19AM +0100, Will Deacon wrote:
> Hi,
>
> [Expanding CC list; original message is here:
> https://lore.kernel.org/linux-arm-kernel/BX1W47JXPMR8.58IYW53H6M5N@dragonstone/]
>
> On Mon, Sep 16, 2019 at 09:35:36PM -0400, Xogium wrote:
> > On arm64 in some situations userspace will continue running even after a
> > panic. This means any userspace watchdog daemon will continue pinging,
> > that service managers will keep running and displaying messages in certain
> > cases, and that it is possible to enter via ssh in the now unstable system
> > and to do almost anything except reboot/power off and etc. If
> > CONFIG_PREEMPT=n is set in the kernel's configuration, the issue is fixed.
> > I have reproduced the very same behavior with linux 4.19, 5.2 and 5.3. On
> > x86/x86_64 the issue does not seem to be present at all.
>
> I've managed to reproduce this under both 32-bit and 64-bit ARM kernels.
> The issue is that the infinite loop at the end of panic() can run with
> preemption enabled (particularly when invoking by echoing 'c' to
> /proc/sysrq-trigger), so we end up rescheduling user tasks. On x86, this
> doesn't happen because smp_send_stop() disables the local APIC in
> native_stop_other_cpus() and so interrupts are effectively masked while
> spinning.
>
> A straightforward fix is to disable preemption explicitly on the panic()
> path (diff below), but I've expanded the cc list to see both what others
> think,
Yep, and it looks like this bug goes back into the dim and distant past.
At least to the start of modern git history, 2.6.12-rc2.
> but also in case smp_send_stop() is supposed to have the side-effect
> of disabling interrupt delivery for the local CPU.
That can't fix it. Consider a preemptive non-SMP kernel.
smp_send_stop() becomes a no-op there.
I'd suggest that a preemptive UP kernel on x86 hardware will suffer
this same issue - it will be able to preempt out of this loop and
continue running userspace.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2019-09-17 10:51 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-17 1:35 [breakage] panic() does not halt arm64 systems under certain conditions Xogium
2019-09-17 10:45 ` Will Deacon
2019-09-17 10:45 ` Will Deacon
2019-09-17 10:51 ` Russell King - ARM Linux admin [this message]
2019-09-17 10:51 ` Russell King - ARM Linux admin
2019-09-17 10:51 ` Russell King - ARM Linux admin
2019-09-17 11:05 ` Will Deacon
2019-09-17 11:05 ` Will Deacon
2019-09-17 11:05 ` Will Deacon
2019-09-20 4:25 ` Jookia
2019-09-20 4:25 ` Jookia
2019-09-30 13:53 ` Will Deacon
2019-09-30 13:53 ` Will Deacon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190917105136.GK25745@shell.armlinux.org.uk \
--to=linux@armlinux.org.uk \
--cc=bp@alien8.de \
--cc=contact@xogium.me \
--cc=gregkh@linuxfoundation.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=tglx@linutronix.de \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.