From mboxrd@z Thu Jan 1 00:00:00 1970 From: Russell King - ARM Linux admin Subject: Re: [breakage] panic() does not halt arm64 systems under certain conditions Date: Tue, 17 Sep 2019 11:51:36 +0100 Message-ID: <20190917105136.GK25745@shell.armlinux.org.uk> References: <20190917104518.ovg6ivadyst7h76o@willie-the-truck> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <20190917104518.ovg6ivadyst7h76o@willie-the-truck> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=m.gmane.org@lists.infradead.org To: Will Deacon Cc: linux-arch@vger.kernel.org, gregkh@linuxfoundation.org, Xogium , linux-kernel@vger.kernel.org, mingo@redhat.com, bp@alien8.de, tglx@linutronix.de, linux-arm-kernel@lists.infradead.org List-Id: linux-arch.vger.kernel.org On Tue, Sep 17, 2019 at 11:45:19AM +0100, Will Deacon wrote: > Hi, > > [Expanding CC list; original message is here: > https://lore.kernel.org/linux-arm-kernel/BX1W47JXPMR8.58IYW53H6M5N@dragonstone/] > > On Mon, Sep 16, 2019 at 09:35:36PM -0400, Xogium wrote: > > On arm64 in some situations userspace will continue running even after a > > panic. This means any userspace watchdog daemon will continue pinging, > > that service managers will keep running and displaying messages in certain > > cases, and that it is possible to enter via ssh in the now unstable system > > and to do almost anything except reboot/power off and etc. If > > CONFIG_PREEMPT=n is set in the kernel's configuration, the issue is fixed. > > I have reproduced the very same behavior with linux 4.19, 5.2 and 5.3. On > > x86/x86_64 the issue does not seem to be present at all. > > I've managed to reproduce this under both 32-bit and 64-bit ARM kernels. > The issue is that the infinite loop at the end of panic() can run with > preemption enabled (particularly when invoking by echoing 'c' to > /proc/sysrq-trigger), so we end up rescheduling user tasks. On x86, this > doesn't happen because smp_send_stop() disables the local APIC in > native_stop_other_cpus() and so interrupts are effectively masked while > spinning. > > A straightforward fix is to disable preemption explicitly on the panic() > path (diff below), but I've expanded the cc list to see both what others > think, Yep, and it looks like this bug goes back into the dim and distant past. At least to the start of modern git history, 2.6.12-rc2. > but also in case smp_send_stop() is supposed to have the side-effect > of disabling interrupt delivery for the local CPU. That can't fix it. Consider a preemptive non-SMP kernel. smp_send_stop() becomes a no-op there. I'd suggest that a preemptive UP kernel on x86 hardware will suffer this same issue - it will be able to preempt out of this loop and continue running userspace. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from pandora.armlinux.org.uk ([78.32.30.218]:39420 "EHLO pandora.armlinux.org.uk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725270AbfIQKvy (ORCPT ); Tue, 17 Sep 2019 06:51:54 -0400 Date: Tue, 17 Sep 2019 11:51:36 +0100 From: Russell King - ARM Linux admin Subject: Re: [breakage] panic() does not halt arm64 systems under certain conditions Message-ID: <20190917105136.GK25745@shell.armlinux.org.uk> References: <20190917104518.ovg6ivadyst7h76o@willie-the-truck> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190917104518.ovg6ivadyst7h76o@willie-the-truck> Sender: linux-arch-owner@vger.kernel.org List-ID: To: Will Deacon Cc: Xogium , linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, gregkh@linuxfoundation.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Message-ID: <20190917105136.vm_z7TPsI0iVuTTHudiyqmcSb7T2veB-Bnd-_C59UA4@z> On Tue, Sep 17, 2019 at 11:45:19AM +0100, Will Deacon wrote: > Hi, > > [Expanding CC list; original message is here: > https://lore.kernel.org/linux-arm-kernel/BX1W47JXPMR8.58IYW53H6M5N@dragonstone/] > > On Mon, Sep 16, 2019 at 09:35:36PM -0400, Xogium wrote: > > On arm64 in some situations userspace will continue running even after a > > panic. This means any userspace watchdog daemon will continue pinging, > > that service managers will keep running and displaying messages in certain > > cases, and that it is possible to enter via ssh in the now unstable system > > and to do almost anything except reboot/power off and etc. If > > CONFIG_PREEMPT=n is set in the kernel's configuration, the issue is fixed. > > I have reproduced the very same behavior with linux 4.19, 5.2 and 5.3. On > > x86/x86_64 the issue does not seem to be present at all. > > I've managed to reproduce this under both 32-bit and 64-bit ARM kernels. > The issue is that the infinite loop at the end of panic() can run with > preemption enabled (particularly when invoking by echoing 'c' to > /proc/sysrq-trigger), so we end up rescheduling user tasks. On x86, this > doesn't happen because smp_send_stop() disables the local APIC in > native_stop_other_cpus() and so interrupts are effectively masked while > spinning. > > A straightforward fix is to disable preemption explicitly on the panic() > path (diff below), but I've expanded the cc list to see both what others > think, Yep, and it looks like this bug goes back into the dim and distant past. At least to the start of modern git history, 2.6.12-rc2. > but also in case smp_send_stop() is supposed to have the side-effect > of disabling interrupt delivery for the local CPU. That can't fix it. Consider a preemptive non-SMP kernel. smp_send_stop() becomes a no-op there. I'd suggest that a preemptive UP kernel on x86 hardware will suffer this same issue - it will be able to preempt out of this loop and continue running userspace. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up