Re: Thoughts on kexec / SBI

[Nick Kossifidis <mick@ics.forth.gr>]

Hello Atish,

Στις 2019-03-24 00:31, Atish Patra έγραψε:
> Hi Nick,
> 
> On Mar 23, 2019, at 11:05 AM, Nick Kossifidis <mick@ics.forth.gr>
> wrote:
> 
>> Hello all,
>> 
>> I'm working on implementing kexec on RISC-V (just kexec for now, not
>> kdump yet) since we want to be able to test new kernel images easily
>> on our testbeds. It's part of a larger project where I'm trying to
>> have a unified way of testing various linux-capable RISC-V targets
>> (https://github.com/mickflemm/yarvt [1]) that we have on the lab.
> 
> Great. This repo will be very helpful to beginners. I had a quick look
> at the sifive-fu540/commands.sh.
> objcopy step is redundant for OpenSBI as it already generates the
> fw_payload.bin as well.
> 

I hope so, thanks for the update, I prefer it this way since it shows 
how the .bin is generated.

>> The issue is that we don't have a way of stopping the secondary
>> harts in a recoverable way, so at this point I have two options,
>> either I'm going to call smp_send_stop() on machine_shutdown() and
>> come back from kexec with a single hart running (which should be
>> fine for kdump btw), or I need to have a reserved memory space where
>> I'll have to keep some code to be executed by the secondary harts,
>> that will be patched by the boot hart once the new kernel is in
>> place, to let them execute the new kernel. The second option is too
>> complicated for no reason and it also reduces the flexibility of the
>> process since we can't use the whole RAM for the loaded kimage, plus
>> it will be obsolete when we have proper handling of CPU suspend /
>> per-hart reset, through SBI. So I'm going for the first option until
>> then.
>> 
>> I'd like to jump-start the discussion on how we can handle things
>> through SBI, my initial through was this:
>> 
>> a) Have a new IPI code IPI_SUSPEND, handled by the firmware (its
>> code/data is persistent across kexecs), that puts the hart on a wfi
>> loop, checking for a variable that contains the new virtual address
>> to jump to (where the new kernel is located) on this hart's scratch
>> buffer. Alternatively this call may do power management and
>> completely shut down the CPU, if the hardware supports this.
>> 
>> b) Have another IPI code IPI_RESUME, that wakes up the harts by
>> providing them with the virtual address to jump to. This can be a
>> simple write of the new address on the remote hart's scratch buffer
>> + interrupt, or it can use power management to power up the hart and
>> either provide it with a new reset vector, or do some handling on
>> the BootROM code for that (this btw is also an open discussion on
>> the TEE group where we are discussing Secure Boot). Also in case the
>> hart is already running (an IPI_SUSPEND hasn't been sent on it
>> before, e.g. during boot), the firmware will ignore the event,
>> alternatively we may call IPI_SUSPEND on the firmware during boot so
>> that the boot process happens with a single hart and other harts
>> wait for the OS to wake them up.
> 
> Looks good to me. I had similar IPI approach(IPI WAKEUP) for cpu
> hotplug implementation except the resume address requirement.
> https://lkml.org/lkml/2018/9/6/214 [2]
> 
> The only issue is that with all this IPIs will be redundant when hart
> power management SBI extension (a much cleaner interface) will be
> available. My hotplug patch is on hold because of that reason as well.
> 
> 
> Regards,
> Atish
> 

My proposal is about the new SBI API/extension, as I mention these IPI 
calls i'm talking about need to be implemented on the firmware side, 
they won't be handled by the OS like IPI_SOFT. So since you prefer to 
have separate calls for each of the hart states the idea is:

sbi_hart_suspend(cpumask) -> send an IPI_SUSPEND to cpus on cpumask
sbi_hart_resume(cpumask) -> send an IPI_RESUME to cpus on cpumask

Regards,
Nick

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv