Re: arm64 board boot pauses on linux-next (resend)

From: Will Deacon <will.deacon@arm.com>
To: John Garry <john.garry@huawei.com>
Cc: ard.biesheuvel@linaro.org, Robin Murphy <robin.murphy@arm.com>,
	Linuxarm <linuxarm@huawei.com>,
	linux-arm-kernel@lists.infradead.org,
	Suzuki K Poulose <suzuki.poulose@arm.com>
Subject: Re: arm64 board boot pauses on linux-next (resend)
Date: Thu, 13 Dec 2018 13:02:10 +0000	[thread overview]
Message-ID: <20181213130210.GC31177@edgewater-inn.cambridge.arm.com> (raw)
In-Reply-To: <eac9866b-3cb4-f7af-dc21-7f685bd3aa96@huawei.com>

[+Ard]

On Thu, Dec 13, 2018 at 11:27:09AM +0000, John Garry wrote:
> On 12/12/2018 19:16, Suzuki K Poulose wrote:
> > On 12/12/2018 06:07 PM, John Garry wrote:
> > > On 12/12/2018 17:17, Robin Murphy wrote:
> > > > On 12/12/2018 16:48, John Garry wrote:
> > > > > I am finding our arm64 D05 board particularly slow to boot from
> > > > > linux-next, specifically a 30+ second pause when setting up CPU
> > > > > features.
> > > > 
> > > > That does look like it's almost certainly spending the mystery time in
> > > > setup_cpu_features() itself.
> > > 
> > > Seems to be "Kernel page table isolation (KPTI)" feature which we hang
> > > on, but I would not say that's conclusive.
> > > 
> > 
> > I think I have an idea what could be happening. The cpu_enable()
> > for KPTI, waits for all the secondary CPUs to enter a busy loop,
> > before installing the non-global mapping. So, with the changes in
> > -next, we batch the cpu_enabl() callbacks, which implies, the seconary
> > CPUs end up in the "cpu_enable()" for KPTI at different times.
> > 
> > Could you check if the following hack makes it any better ?
> > 
> 
> Unfortunately it does not help:
> [    5.502243] CPU63: Booted secondary processor 0x0000070303 [0x410fd082]
> [    5.502329] smp: Brought up 4 nodes, 64 CPUs
> [    7.442722] SMP: Total of 64 processors activated.
> [    7.447567] CPU features: detected: GIC system register CPU interface
> [    7.454098] CPU features: detected: 32-bit EL0 Support
> [    7.459291] CPU features: detected: CRC32 instructions
> [   40.236781] CPU: All CPU(s) started at EL2
> [   40.241062] alternatives: patching kernel code
> [   40.263213] devtmpfs: initialized
> [   40.267140] clocksource: jiffies: mask: 0xffffffff max_cycles:
> 0xffffffff, max_idle_ns: 7645041785100000 ns
> [   40.277147] futex hash table entries: 16384 (order: 8, 1048576 bytes)
> [   40.284424] pinctrl core: initialized pinctrl subsystem
> [   40.290081] SMBIOS 3.0.0 present.
> 
> BTW, If you guys know the reason for this delay and it is not going to be an
> issue, then that's fine. I just wanted to raise awareness.

I think I've managed to reproduce the issue locally and it appears to be
because of the default rodata=full changes. One impact of the change is
that the linear map is now mapped at page granularity, so the kpti work
to convert everything to non-global takes considerably longer.

Given that most machines are not affected by meltdown, I think we should
probably look at expanding the whitelist we have, rather than pile more
complexity into the early page table code.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel