Re: arm64: kdump broken on a large CPU system

["AKASHI, Takahiro" <takahiro.akashi@linaro.org>]

On Tue, Dec 11, 2018 at 11:34:22AM +0000, James Morse wrote:
> Hi Qian, Marc,
> 
> On 11/12/2018 10:09, Marc Zyngier wrote:
> > On 10/12/2018 22:30, Qian Cai wrote:
> >> On this HPE Apollo 70 arm64 server with 256 CPUs, triggering a crash dump just
> >> hung (4.20-rc6 as well as 4.18). It was confirmed that the executing went as far
> >> as entering __cpu_soft_restart(),
> > 
> > You can forget about 4.18 altogether, it will never correctly kexec.
> > I've used 4.20 + kexec on a TX2 system though, and although it takes
> > absolutely ages, it reliably works.
> 
> >> __crash_kexec
> >>   machine_kexec
> >>     cpu_soft_restart
> >>       restart
> >>         __cpu_soft_restart

@Qian, how did you confirm that you reached here?

> >>
> >> The earlycon was enabled but had no output from the 2nd kernel, so it was pretty
> >> much stuck in all those assembly code in arm64/kernel/head.S or the early part
> >> of start_kernel() before earlycon was initialized.
> > 
> > Could it instead be in the purgatory code provided by userspace?
> 
> Yes, this could be anything between entering __cpu_soft_restart(), purgatory and
> the earlycon driver in the new kernel.

To be in purgatory, or not to be, that is the question.
(I'm serious.)

> 
> >> It turned out this has something to do with nr_cpus in the 1st kernel, although
> >> the 2nd kernel always has nr_cpus=1 [1]. It was tested with both
> >> crashkernel=512M or 768M.
> > 
> > James was saying something about a timeout, which may or may not be long
> > enough.
> 
> This comes from arch/arm64/kernel/smp.c:crash_smp_send_stop()
> It sends IPIs to all other CPUs, then waits one second before timing-out.
> This may not be enough time for a system with hundreds of CPUs.
> 
> Increasing the timeout may help, but I don't understand why extra CPUs would
> matter if we're getting as far as __cpu_soft_restart().

Indeed.

> 
> >> nr_cpus <= 96  GOOD (2nd kernel was up in 2-3 mins.)
> >> nr_cpus=256    BAD  (2nd kernel was NOT up after 1 hour.)
> >> nr_cpus=127    BAD  (2nd kernel was NOT up after 10 mins.)
> >>
> >> I did also test with and without CONFIG_ARM64_VHE (i.e., el2_switch) made no
> >> difference.
> 
> >> [1] KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=1 swiotlb=noforce reset_devices"
> 
> >> I am still figuring out a way to debug those assembly code to where it actually
> >> hung, 

In my experiences, I have used the patch I mention below
as well as a hw debugger, DS-5 with FVP in my case, for examining
purgatory-related issues.


> There were some earlier patches to purgatory to let it write the console, but
> this didn't scale as purgatory isn't an operating-system. (Reducing purgatory to
> be as simple as possible is better, with kexec_file_load() we don't use it all.)
> 
> If kexec-tools still has a 'ARM64_DEBUG_PORT' you may be able to get it to write
> to your uart. (no idea which uarts it supports, or how it tells pl011 and 8250
> apart).

@James, are you sure? I don't see it.
@Qian, I can give you a small patch of enabling printf in purgatory,
although it's quite hacky, if you want.
(As thunder X2 has a pl011, the patch should work.)

> Some threads to pull on:
> https://patchwork.kernel.org/patch/6121951/
> https://patchwork.kernel.org/patch/9238475/
> (search for 'TX as the first port?' in the last one)
> 
> 
> >> but the server was hooked up with a conserver that was not able to
> >> generate any sysrq and I have no shell access to the conserver, so seems a bit
> >> difficult to use kgdb or kdb in this case.
> 
> More recent kexec tools has a 'lite' or 'no-checks' option that tells it not to
> bother checksumming the kdump kernel.

Are you sure? I remember that Geoff's original patch was rejected.

> This is what takes a long time as its done
> without the MMU+caches enabled.

Pratyush has a patch of enabling MMU in purgatory, but again
it was rejected.

Thanks,
-Takahiro Akashi

> It shouldn't be possible for the old-kernel to corrupt it, as its not mapped
> unless its being loaded (or save/restored by hibernate). I'm not sure how the
> crash-regs get written to the elfcore header though...
> 
> 
> Thanks,
> 
> James

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel