Re: [PATCH v7 11/16] irqchip/gic-v3: Add support for ACPI's disabled but 'online capable' CPUs

[Jonathan Cameron <Jonathan.Cameron@Huawei.com>]

On Thu, 25 Apr 2024 16:00:17 +0100
Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:

> On Thu, 25 Apr 2024 13:31:50 +0100
> Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:
> 
> > On Wed, 24 Apr 2024 16:33:22 +0100
> > Marc Zyngier <maz@kernel.org> wrote:
> >   
> > > On Wed, 24 Apr 2024 13:54:38 +0100,
> > > Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:    
> > > > 
> > > > On Tue, 23 Apr 2024 13:01:21 +0100
> > > > Marc Zyngier <maz@kernel.org> wrote:
> > > >       
> > > > > On Mon, 22 Apr 2024 11:40:20 +0100,
> > > > > Jonathan Cameron <Jonathan.Cameron@Huawei.com> wrote:      
> > > > > > 
> > > > > > On Thu, 18 Apr 2024 14:54:07 +0100
> > > > > > Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:      
> > > 
> > > [...]
> > >     
> > > > > >         
> > > > > > > +	/*
> > > > > > > +	 * Capable but disabled CPUs can be brought online later. What about
> > > > > > > +	 * the redistributor? ACPI doesn't want to say!
> > > > > > > +	 * Virtual hotplug systems can use the MADT's "always-on" GICR entries.
> > > > > > > +	 * Otherwise, prevent such CPUs from being brought online.
> > > > > > > +	 */
> > > > > > > +	if (!(gicc->flags & ACPI_MADT_ENABLED)) {
> > > > > > > +		pr_warn_once("CPU %u's redistributor is inaccessible: this CPU can't be brought online\n", cpu);
> > > > > > > +		set_cpu_present(cpu, false);
> > > > > > > +		set_cpu_possible(cpu, false);
> > > > > > > +		return 0;
> > > > > > > +	}        
> > > > > 
> > > > > It seems dangerous to clear those this late in the game, given how
> > > > > disconnected from the architecture code this is. Are we sure that
> > > > > nothing has sampled these cpumasks beforehand?      
> > > > 
> > > > Hi Marc,
> > > > 
> > > > Any firmware that does this is being considered as buggy already
> > > > but given it is firmware and the spec doesn't say much about this,
> > > > there is always the possibility.      
> > > 
> > > There is no shortage of broken firmware out there, and I expect this
> > > trend to progress.
> > >     
> > > > Not much happens between the point where these are setup and
> > > > the point where the the gic inits and this code runs, but even if careful
> > > > review showed it was fine today, it will be fragile to future changes.
> > > > 
> > > > I'm not sure there is a huge disadvantage for such broken firmware in
> > > > clearing these masks from the point of view of what is used throughout
> > > > the rest of the kernel. Here I think we are just looking to prevent the CPU
> > > > being onlined later.      
> > > 
> > > I totally agree on the goal, I simply question the way you get to it.
> > >     
> > > > 
> > > > We could add a set_cpu_broken() with appropriate mask.
> > > > Given this is very arm64 specific I'm not sure Rafael will be keen on
> > > > us checking such a mask in the generic ACPI code, but we could check it in
> > > > arch_register_cpu() and just not register the cpu if it matches.
> > > > That will cover the vCPU hotplug case.
> > > >
> > > > Does that sounds sensible, or would you prefer something else?      
> > > 
> > > 
> > > Such a 'broken_rdists' mask is exactly what I have in mind, just
> > > keeping it private to the GIC driver, and not expose it anywhere else.
> > > You can then fail the hotplug event early, and avoid changing the
> > > global masks from within the GIC driver. At least, we don't mess with
> > > the internals of the kernel, and the CPU is properly marked as dead
> > > (that mechanism should already work).
> > > 
> > > I'd expect the handling side to look like this (will not compile, but
> > > you'll get the idea):    
> > Hi Marc,
> > 
> > In general this looks good - but...
> > 
> > I haven't gotten to the bottom of why yet (and it might be a side
> > effect of how I hacked the test by lying in minimal fashion and
> > just frigging the MADT read functions) but the hotplug flow is only getting
> > as far as calling __cpu_up() before it seems to enter an infinite loop.
> > That is it never gets far enough to fail this test.
> > 
> > Getting stuck in a psci cpu_on call.  I'm guessing something that
> > we didn't get to in the earlier gicv3 calls before bailing out is blocking that?
> > Looks like it gets to
> > SMCCC smc
> > and is never seen again.
> > 
> > Any ideas on where to look?  The one advantage so far of the higher level
> > approach is we never tried the hotplug callbacks at all so avoided hitting
> > that call.  One (little bit horrible) solution that might avoid this would 
> > be to add another cpuhp state very early on and fail at that stage.
> > I'm not keen on doing that without a better explanation than I have so far!  
> 
> Whilst it still doesn't work I suspect I'm loosing ability to print to the console
> between that point and somewhat later and real problem is elsewhere.

Hi again,

Found it I think.  cpuhp calls between cpu:bringup and ap:online 
arm made from notify_cpu_starting() are clearly marked as nofail with a comment.
STARTING must not fail!

https://elixir.bootlin.com/linux/latest/source/kernel/cpu.c#L1642

Whilst I have no immediate idea why that comment is there it is pretty strong
argument against trying to have the CPUHP_AP_IRQ_GIC_STARTING callback fail
and expecting it to carry on working :( 
There would have been a nice print message, but given I don't appear to have
a working console after that stage I never see it.

So the best I have yet come up with for this is the option of a new callback registered
in gic_smp_init()

cpuhp_setup_state_nocalls(CPUHP_BP_PREPARE_DYN,
			  "irqchip/arm/gicv3:checkrdist",
			  gic_broken_rdist, NULL);

with callback being simply 

static int gic_broken_rdist(unsigned int cpu)
{
	if (cpumask_test_cpu(cpu, &broken_rdists))
		return -EINVAL;

	return 0;
}

That gets called cpuhp_up_callbacks() and is allows to fail and roll back the steps.

Not particularly satisfying but keeps the logic confined to the gicv3 driver.

What do you think?

Jonathan

> 
> Jonathan
> 
> > 
> > Thanks,
> > 
> > J
> > 
> >    
> > > diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> > > index 6fb276504bcc..e8f02bfd0e21 100644
> > > --- a/drivers/irqchip/irq-gic-v3.c
> > > +++ b/drivers/irqchip/irq-gic-v3.c
> > > @@ -1009,6 +1009,9 @@ static int __gic_populate_rdist(struct redist_region *region, void __iomem *ptr)
> > >  	u64 typer;
> > >  	u32 aff;
> > >  
> > > +	if (cpumask_test_cpu(smp_processor_id(), &broken_rdists))
> > > +		return 1;
> > > +
> > >  	/*
> > >  	 * Convert affinity to a 32bit value that can be matched to
> > >  	 * GICR_TYPER bits [63:32].
> > > @@ -1260,14 +1263,15 @@ static int gic_dist_supports_lpis(void)
> > >  		!gicv3_nolpi);
> > >  }
> > >  
> > > -static void gic_cpu_init(void)
> > > +static int gic_cpu_init(void)
> > >  {
> > >  	void __iomem *rbase;
> > > -	int i;
> > > +	int ret, i;
> > >  
> > >  	/* Register ourselves with the rest of the world */
> > > -	if (gic_populate_rdist())
> > > -		return;
> > > +	ret = gic_populate_rdist();
> > > +	if (ret)
> > > +		return ret;
> > >  
> > >  	gic_enable_redist(true);
> > >  
> > > @@ -1286,6 +1290,8 @@ static void gic_cpu_init(void)
> > >  
> > >  	/* initialise system registers */
> > >  	gic_cpu_sys_reg_init();
> > > +
> > > +	return 0;
> > >  }
> > >  
> > >  #ifdef CONFIG_SMP
> > > @@ -1295,7 +1301,11 @@ static void gic_cpu_init(void)
> > >  
> > >  static int gic_starting_cpu(unsigned int cpu)
> > >  {
> > > -	gic_cpu_init();
> > > +	int ret;
> > > +
> > > +	ret = gic_cpu_init();
> > > +	if (ret)
> > > +		return ret;
> > >  
> > >  	if (gic_dist_supports_lpis())
> > >  		its_cpu_init();
> > > 
> > > But the question is: do you rely on these masks having been
> > > "corrected" anywhere else?
> > > 
> > > Thanks,
> > > 
> > > 	M.
> > >     
> > 
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel  
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel