Re: [RFC 0/3] Introduce a new Qemu machine for RISC-V

[Atish Patra <atishp@atishpatra.org>]

On Sun, May 22, 2022 at 10:59 PM Alistair Francis <alistair23@gmail.com> wrote:
>
> On Wed, May 18, 2022 at 4:38 PM Atish Patra <atishp@atishpatra.org> wrote:
> >
> > On Tue, May 17, 2022 at 1:54 PM Alistair Francis <alistair23@gmail.com> wrote:
> > >
> > > On Tue, May 17, 2022 at 6:52 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > >
> > > > On Tue, May 17, 2022 at 03:03:38PM +1000, Alistair Francis wrote:
> > > > > On Sat, May 7, 2022 at 6:30 AM Atish Kumar Patra <atishp@rivosinc.com> wrote:
> > > > > >
> > > > > > On Fri, May 6, 2022 at 4:00 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > > > > > >
> > > > > > > On Fri, 6 May 2022 at 09:18, Daniel P. Berrangé <berrange@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Fri, May 06, 2022 at 06:34:47AM +1000, Alistair Francis wrote:
> > > > > > > > > Even if we didn't worry about backwards compatibility the current virt
> > > > > > > > > machine would still be what most users want. It's just a small number
> > > > > > > > > of users who don't want MMIO devices and instead want to use PCIe for
> > > > > > > > > everything. Realistically it's only HPC users who would want this type
> > > > > > > > > of machine, at least that's my understanding.
> > > > > > > >
> > > > > > > > I'm not so sure about that. Every other architecture has ended up
> > > > > > > > standardizing on PCI for general purpose virtual machines. IIRC,
> > > > > > > > aarch64 started off with MMIO, but switched to PCI as it matured.
> > > > > > > >
> > > > > > > > In terms of having VM mgmt tools "just work" for risc-v, I think
> > > > > > > > it will be very compelling for the general 'virt' machine to be
> > > > > > > > PCI based, otherwise all the assumptions about PCI in mgmt apps
> > > > > > > > are going to break requiring never ending riscv fixes.
> > > > > > >
> > > > > > > Mmm, my experience with aarch64 virt is that PCI is much nicer
> > > > > > > as a general preference. aarch64 virt has some MMIO devices
> > > > > > > for historical reasons and some because you can't reasonably
> > > > > > > do the necessary things with PCI, but I'm actively trying to
> > > > > > > push people who submit new MMIO device features for virt to
> > > > > > > try to use a PCI-based solution instead if they possibly can.
> > > > >
> > > > > Interesting...
> > > > >
> > > > > Ok, maybe calling this "virt-pcie" might be a good start, with the
> > > > > expectation to eventually replace the current virt with the new
> > > > > virt-pcie at some point.
> > > >
> > > > Delaying the inevitable by leaving PCIE support in a separate
> > > > machine type initially is going to be more painful long term.
> > > >
> > > > > The other option would be to try and gradually change from the current
> > > > > virt machine to this new virt machine
> > > >
> > > > Yes, I really think the 'virt' machine type needs to aim for PCIE
> > > > support sooner rather than later, if RISC-V wants to get on part
> > > > with other architectures. The best time to have added PCIE support
> > > > to 'virt' was when it was first created, the next best time is now.
> > >
> > > So maybe instead we lock in the current virt machine as the 7.1 virt
> > > machine for QEMU 7.1, then work on migrating to a PCIe only machine
> > > with versions (similar to the other archs)
> > >
> >
> > I am not quite sure what exactly you mean here. Do you mean to modify
> > the current virt
> > machine to be PCIE only after QEMU 7.1 or the new PCIE only machine
> > (with the versioning)
> > which will be the default machine in the future
>
> I mean that we call the current virt machine the virt machine for QEMU
> 7.1. Then for future releases we can make breaking changes, where we
> have the old 7.1 virt machine for backwards compatibility.
>
> >
> > If you intend to say the former, few issues with that approach.
> >
> > 1. virt machine is not well documented and already bloated. There is
> > no specification for virt machine as such.
> > Putting restrictions after a certain release will lead to confusion.
> > 2. Do we support existing MMIO devices after that specific version or not ?
>
> Yeah, so I guess this doesn't achieve the same outcome you want. I
> would say we would still include some MMIO devices, like UART for
> example.
>

Why ? We can just rely on the pcie based uart (virtio-serial-pci or
serial-pci) should be enough.
The only MMIO devices that should be allowed are the ones that can't
be behind pcie.

> But we could simplify things a bit. So for example maybe we could use
> AIA by default and then remove the PLIC code. That would help cleanup
> the board file. Then we could add a `msi-only` option that would be
> similar to https://github.com/atishp04/qemu/commit/d7fc1c6aa7855b414b3484672a076140166a2dcd.
> But without the PLIC it should hopefully be cleaner
>
> We would need AIA ratified before we could remove the PLIC though.
>

And AIA patches available in the upstream Linux kernel.
Even after that, can we just remove the PLIC ?
That would mean everybody has to use the latest kernel with AIA
support after that.

> > 3. The user has to be aware of which version of virt machine it is
> > running in order to test specific features (i.e. flash, reset, wired
> > interrupts)
>
> That's true, but I think we are going to have this issue someday
> anyway. We don't want to use the SiFive CLINT and PLIC forever,
> eventually we will want to use the newer hardware.
>

Agreed. But It should be disabed by default at first. In the future it
can be removed.
Otherwise, we end up breaking a bunch of user configurations.

> > 4. Based on the version of the virt machine, the command line options
> > will change. This may also be confusing.
> >
> > On the other hand, the second approach will be much cleaner without
> > any baggage. The RISC-V eco-system is still maturing and this is the
> > right time
> > to start something fresh. Let's call the new machine virt-pcie for
> > now. Here are a few things that can be implemented for this machine.
> >
> > 1. It must support versioning and any changes to the machine code must
> > modify the version of the machine.
> > 2. It must only support MSI based PCIe devices. No support for wired interrupts.
> >     The only allowed MMIO devices are
> >            -- mtimer/mtimecmp (there is no other option provided in ISA)
> >            -- reset/rtc device (If there is a way we can emulate these
> > two over PCIe, that would be great)
> >            -- flash
> > Beyond this, adding any other MMIO device must be strongly discouraged.
> > 3. The virt-pcie machine must be well documented similar to a hardware
> > specification (including address range, restrictions and
> > implemented/allowed devices)
> > 4. No dependence on virtio framework but must work with virtio-pcie backend.
> > 5. Migration must be supported.
>
> I'm on board with these! They would all be great to have.
>
> I'm open to a virt-pcie, but as others have pointed out it might be
> easier to just make the switch now to the general board.
>
> > 6. No command line option is required.
>
> I don't see this being achievable unfortunately
>

I meant no command line configurability options for the machine
itself. We probably should
allow using different versions of the machine via command line.

> > 7. Any other ?
> >
> > Once we have these policies in place, this can be the preferred virt
> > machine and the current virt machine can be phased out or continue to
> > co-exist
> > as a more flexible experimental platform.
> >
> > Thoughts ?
> >
> > > Alistair
> > >
> > > >
> > > > With regards,
> > > > Daniel
> > > > --
> > > > |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> > > > |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> > > > |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> > > >
> >
> >
> >
> > --
> > Regards,
> > Atish



-- 
Regards,
Atish