Re: [PATCH] target/ppc: cpu_init: Clean up stop state on cpu reset

["Cédric Le Goater" <clg@kaod.org>]

On 6/15/22 09:17, Frederic Barrat wrote:
> 
> 
> On 15/06/2022 07:23, Cédric Le Goater wrote:
>> On 6/14/22 10:29, Frederic Barrat wrote:
>>> The 'resume_as_sreset' attribute of a cpu can be set when a thread is
>>> entering a stop state on ppc books. It causes the thread to be
>>> re-routed to vector 0x100 when woken up by an exception. So it must be
>>> cleaned on reset or a thread might be re-routed unexpectedly after a
>>> reset, when it was not in a stop state and/or when the appropriate
>>> exception handler isn't set up yet.
>>
>> What is the test scenario ? and what are the symptoms ?
> 
> 
> I was hitting it because of another bug in skiboot: if you have many chips, we spend way too much time in add_opal_interrupts(), especially on powernv10 (I'm working on a separate patch in skiboot to fix that). Sufficiently so that the watchdog timer resets the system. When it happens, all the secondary threads are in stopped state, only the main thread is working. That's how I was reproducing.
> 
> What happens after the reset can vary a bit due to timing, but the most likely scenario is that we go through another primary thread election in skiboot. If the primary thread is the same as before, then there's no problem. If it's a different primary, then it will enter main_cpu_entry() while the other threads wait as secondaries. At some point, the primary thread (which still carries the wrong resume_as_sreset value from before reset) will enable the decrementer interrupt. The vector for the decrementer exception 0x900 is defined, so that shouldn't be a problem. However, because of the wrong resume_as_sreset value, it is re-routed to vector 0x100, which is still defined as the default boot-time handler, which is the entry point for BML. So the thread restarts as new, but this time it will be elected secondary. And we end up with all threads waiting as secondaries and a system stuck. All that happen before we've init the uart, so there's not a single trace on the console. 
> Fun :-)

Great analysis !

I think this deserve a v2 just to put in the commit log what you
just wrote :)

Thanks,

C.