QEMU features useful for Xen development?

QEMU features useful for Xen development?

[Alex Bennée]

Dear Xen community,

Linaro is significantly invested in QEMU development, with a special
focus on Arm-related aspects. We recognize the value of QEMU as a
readily available software reference platform for projects that need to
test their software well before the availability of real hardware.

The primary focus of our effort is on adding core architectural elements
to the CPU emulation. For an overview of the current feature set, please
see:

  https://qemu.readthedocs.io/en/master/system/arm/emulation.html

Besides the -cpu max, providing an approximation of a v9.0 baseline CPU,
we have also recently added several specific CPU types like the
Neoverse-N1 and V1 processor types as well as numerous Cortex CPU
models.

Our most utilized machine model is "virt", which is primarily designed
for guest operation and therefore has minimal resemblance to actual
hardware. "sbsa-ref" was implemented to more closely simulate a real
machine that aligns with Arm's SBSA specification.

In our work on VirtIO, we often use QEMU. Most of our rust-vmm
vhost-device backends, for instance, were initially tested on QEMU.

Now that everyone is up-to-date, I would welcome any feedback from the
Xen community on features that would increase QEMU's usefulness as a
development target.

Do you have interest in any upcoming Arm CPU features? For example, we
recently added FEAT_RME support for Arm's new confidential computing,
but currently do not implement FEAT_NV/NV2.

How about the HW emulation in QEMU? Is the PCI emulation reliable enough
to ensure confidence while testing changes to Xen's PCI management? What
about the few peripherals that the hypervisor accesses directly?

Are there other development features you consider essential? Have you
noticed any limitations with gdbstub? Does anyone use the record/replay
or reverse debug functions? Has anyone tried TCG plugins for analysing
the behavior of the hypervisor?

While I cannot promise to implement every wish-list item (performance
counter emulation, for example, as we are not a uArch simulator), I am
eager to gather feedback on how QEMU could be improved to help the Xen
community deliver it's roadmap faster.

Thank you for your time and I look forward to any feedback :-)

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: QEMU features useful for Xen development?

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3404 bytes --]

Hi Alex,

Thanks for reaching out. QEMU is an important development tool for the
Xen community and we are using QEMU as part of our upstream gitlab-ci
testing, see automation/scripts/qemu-*.

As Xen is gaining R52 and R82 support, it would be great to be able to
use QEMU for development and testing there as well, but I don't think
QEMU can emulate EL2 properly for the Cortex-R architecture. We would
need EL2 support in the GIC/timer for R52/R82 as well.

On Cortex-As, in addition to a PCI root complex and an arbitrary PCI
device, SMMUv3 emulation (both stages) and GICv3 ITS are needed to be
able to test PCI Passthrough. However, if I recall correctly SMMUv3
emulation in QEMU might not be complete enough to enable us to use it.

For Virtio, using QEMU on target (not develpment/testing, but
production), it would greatly help if we could improve the build system
to only build what is strictly necessary for the xenpvh machine to run.

Cheers,

Stefano


On Wed, 30 Aug 2023, Alex Bennée wrote:
> Dear Xen community,
> 
> Linaro is significantly invested in QEMU development, with a special
> focus on Arm-related aspects. We recognize the value of QEMU as a
> readily available software reference platform for projects that need to
> test their software well before the availability of real hardware.
> 
> The primary focus of our effort is on adding core architectural elements
> to the CPU emulation. For an overview of the current feature set, please
> see:
> 
>   https://qemu.readthedocs.io/en/master/system/arm/emulation.html
> 
> Besides the -cpu max, providing an approximation of a v9.0 baseline CPU,
> we have also recently added several specific CPU types like the
> Neoverse-N1 and V1 processor types as well as numerous Cortex CPU
> models.
> 
> Our most utilized machine model is "virt", which is primarily designed
> for guest operation and therefore has minimal resemblance to actual
> hardware. "sbsa-ref" was implemented to more closely simulate a real
> machine that aligns with Arm's SBSA specification.
> 
> In our work on VirtIO, we often use QEMU. Most of our rust-vmm
> vhost-device backends, for instance, were initially tested on QEMU.
> 
> Now that everyone is up-to-date, I would welcome any feedback from the
> Xen community on features that would increase QEMU's usefulness as a
> development target.
> 
> Do you have interest in any upcoming Arm CPU features? For example, we
> recently added FEAT_RME support for Arm's new confidential computing,
> but currently do not implement FEAT_NV/NV2.
> 
> How about the HW emulation in QEMU? Is the PCI emulation reliable enough
> to ensure confidence while testing changes to Xen's PCI management? What
> about the few peripherals that the hypervisor accesses directly?
> 
> Are there other development features you consider essential? Have you
> noticed any limitations with gdbstub? Does anyone use the record/replay
> or reverse debug functions? Has anyone tried TCG plugins for analysing
> the behavior of the hypervisor?
> 
> While I cannot promise to implement every wish-list item (performance
> counter emulation, for example, as we are not a uArch simulator), I am
> eager to gather feedback on how QEMU could be improved to help the Xen
> community deliver it's roadmap faster.
> 
> Thank you for your time and I look forward to any feedback :-)
> 
> -- 
> Alex Bennée
> Virtualisation Tech Lead @ Linaro
> 

Re: QEMU features useful for Xen development?

[Peter Maydell]

On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
> As Xen is gaining R52 and R82 support, it would be great to be able to
> use QEMU for development and testing there as well, but I don't think
> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
> need EL2 support in the GIC/timer for R52/R82 as well.

We do actually have a Cortex-R52 model which at least in theory
should include EL2 support, though as usual with newer QEMU
stuff it quite likely has lurking bugs; I'm not sure how much
testing it's had. Also there is currently no board model which
will work with the Cortex-R52 so it's a bit tricky to use in practice.
(What sort of board model would Xen want to use it with?)

The Cortex-R82 would be more work, because (unlike the R52) it's
AArch64, and we don't have Armv8-R AArch64 support yet, only the AArch32.

I haven't looked at whether GIC on R-profile requires any changes
from the A-profile GIC; on A-profile obviously we emulate the
virtualization support already.

> On Cortex-As, in addition to a PCI root complex and an arbitrary PCI
> device, SMMUv3 emulation (both stages) and GICv3 ITS are needed to be
> able to test PCI Passthrough. However, if I recall correctly SMMUv3
> emulation in QEMU might not be complete enough to enable us to use it.

Yeah, at the moment the SMMU emulation supports stage 1 and stage 2,
but not both at the same time. This is good enough for PCI passthrough
with a Linux guest using KVM to pass a device through to a nested
Linux guest.

thanks
-- PMM

Re: QEMU features useful for Xen development?

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> As Xen is gaining R52 and R82 support, it would be great to be able to
>> use QEMU for development and testing there as well, but I don't think
>> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
>> need EL2 support in the GIC/timer for R52/R82 as well.
>
> We do actually have a Cortex-R52 model which at least in theory
> should include EL2 support, though as usual with newer QEMU
> stuff it quite likely has lurking bugs; I'm not sure how much
> testing it's had. Also there is currently no board model which
> will work with the Cortex-R52 so it's a bit tricky to use in practice.
> (What sort of board model would Xen want to use it with?)

We already model a bunch of the mps2/mps3 images so I'm assuming adding
the mps3-an536 would be a fairly simple step to do (mps2tz.c is mostly
tweaking config values). The question is would it be a useful target for
Xen?

  https://developer.arm.com/documentation/dai0536/latest/

> The Cortex-R82 would be more work, because (unlike the R52) it's
> AArch64, and we don't have Armv8-R AArch64 support yet, only the AArch32.
>
> I haven't looked at whether GIC on R-profile requires any changes
> from the A-profile GIC; on A-profile obviously we emulate the
> virtualization support already.
>
>> On Cortex-As, in addition to a PCI root complex and an arbitrary PCI
>> device, SMMUv3 emulation (both stages) and GICv3 ITS are needed to be
>> able to test PCI Passthrough.

We have ITS emulation support and it was recently plumbed into the
"sbsa-ref" board as it is needed for higher level SBSA compliance.

>> However, if I recall correctly SMMUv3
>> emulation in QEMU might not be complete enough to enable us to use it.
>
> Yeah, at the moment the SMMU emulation supports stage 1 and stage 2,
> but not both at the same time. This is good enough for PCI passthrough
> with a Linux guest using KVM to pass a device through to a nested
> Linux guest.

Is this a missing feature for SMMUv3 or something introduced in the
later revisions?

We have sketched out the tasks for SMMUv3.2
(https://linaro.atlassian.net/browse/QEMU-558) with a view to whats
needed for RME guests to access hardware. However I think there is a lot
of other stuff needed specifically for RME including what we do about
modelling things like TDISP. Realistically it will be awhile before we
get to completing all of that.


>
> thanks
> -- PMM


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: QEMU features useful for Xen development?

[Peter Maydell]

On Thu, 31 Aug 2023 at 10:53, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
> > On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >> As Xen is gaining R52 and R82 support, it would be great to be able to
> >> use QEMU for development and testing there as well, but I don't think
> >> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
> >> need EL2 support in the GIC/timer for R52/R82 as well.
> >
> > We do actually have a Cortex-R52 model which at least in theory
> > should include EL2 support, though as usual with newer QEMU
> > stuff it quite likely has lurking bugs; I'm not sure how much
> > testing it's had. Also there is currently no board model which
> > will work with the Cortex-R52 so it's a bit tricky to use in practice.
> > (What sort of board model would Xen want to use it with?)
>
> We already model a bunch of the mps2/mps3 images so I'm assuming adding
> the mps3-an536 would be a fairly simple step to do (mps2tz.c is mostly
> tweaking config values). The question is would it be a useful target for
> Xen?

All our MPS2/MPS3 boards are M-profile. That means we have the
device models for all the interesting devices on the board, but
it would be simpler to write the an536 board model separately.
(In particular, the M-profile boards are wrappers around an
"ARMSSE" sort-of-like-an-SoC component; there's no equivalent
for the Cortex-R52.)

>   https://developer.arm.com/documentation/dai0536/latest/
>
> > The Cortex-R82 would be more work, because (unlike the R52) it's
> > AArch64, and we don't have Armv8-R AArch64 support yet, only the AArch32.
> >
> > I haven't looked at whether GIC on R-profile requires any changes
> > from the A-profile GIC; on A-profile obviously we emulate the
> > virtualization support already.
> >
> >> On Cortex-As, in addition to a PCI root complex and an arbitrary PCI
> >> device, SMMUv3 emulation (both stages) and GICv3 ITS are needed to be
> >> able to test PCI Passthrough.
>
> We have ITS emulation support and it was recently plumbed into the
> "sbsa-ref" board as it is needed for higher level SBSA compliance.
>
> >> However, if I recall correctly SMMUv3
> >> emulation in QEMU might not be complete enough to enable us to use it.
> >
> > Yeah, at the moment the SMMU emulation supports stage 1 and stage 2,
> > but not both at the same time. This is good enough for PCI passthrough
> > with a Linux guest using KVM to pass a device through to a nested
> > Linux guest.
>
> Is this a missing feature for SMMUv3 or something introduced in the
> later revisions?

It's a missing feature. The SMMUv3 spec allows an implementation
to implement stage 1, stage 2 or both. We started with just a
stage-1-only implementation because Linux doesn't need any more.
Stage-2-only just landed recently. Nobody's looked at both-stages yet.

thanks
-- PMM

Re: QEMU features useful for Xen development?

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Thu, 31 Aug 2023 at 10:53, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Peter Maydell <peter.maydell@linaro.org> writes:
>>
>> > On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
>> >> As Xen is gaining R52 and R82 support, it would be great to be able to
>> >> use QEMU for development and testing there as well, but I don't think
>> >> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
>> >> need EL2 support in the GIC/timer for R52/R82 as well.
>> >
>> > We do actually have a Cortex-R52 model which at least in theory
>> > should include EL2 support, though as usual with newer QEMU
>> > stuff it quite likely has lurking bugs; I'm not sure how much
>> > testing it's had. Also there is currently no board model which
>> > will work with the Cortex-R52 so it's a bit tricky to use in practice.
>> > (What sort of board model would Xen want to use it with?)
>>
>> We already model a bunch of the mps2/mps3 images so I'm assuming adding
>> the mps3-an536 would be a fairly simple step to do (mps2tz.c is mostly
>> tweaking config values). The question is would it be a useful target for
>> Xen?
>
> All our MPS2/MPS3 boards are M-profile. That means we have the
> device models for all the interesting devices on the board, but
> it would be simpler to write the an536 board model separately.
> (In particular, the M-profile boards are wrappers around an
> "ARMSSE" sort-of-like-an-SoC component; there's no equivalent
> for the Cortex-R52.)
>
>>   https://developer.arm.com/documentation/dai0536/latest/

It's not super clear from the design notes but it does mention the
SSE-200 sub-system as the basis for peripherals. Specifically the blocks
are:

  Arm Cortex-R52 Processor
  Arm CoreSight SoC-400 (n/a for QEMU)
  Cortex-M System Design Kit
  PL022 Serial Port
  NIC-400 Network interconnect

But if writing it from scratch is simpler so be it. The real question is
what new hardware would we need to model to be able to bring something
up that is useful to Xen?

>> > The Cortex-R82 would be more work, because (unlike the R52) it's
>> > AArch64, and we don't have Armv8-R AArch64 support yet, only the AArch32.
>> >
>> > I haven't looked at whether GIC on R-profile requires any changes
>> > from the A-profile GIC; on A-profile obviously we emulate the
>> > virtualization support already.
>> >

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: QEMU features useful for Xen development?

[Peter Maydell]

On Thu, 31 Aug 2023 at 11:49, Alex Bennée <alex.bennee@linaro.org> wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> > All our MPS2/MPS3 boards are M-profile. That means we have the
> > device models for all the interesting devices on the board, but
> > it would be simpler to write the an536 board model separately.
> > (In particular, the M-profile boards are wrappers around an
> > "ARMSSE" sort-of-like-an-SoC component; there's no equivalent
> > for the Cortex-R52.)
> >
> >>   https://developer.arm.com/documentation/dai0536/latest/
>
> It's not super clear from the design notes but it does mention the
> SSE-200 sub-system as the basis for peripherals. Specifically the blocks
> are:
>
>   Arm Cortex-R52 Processor
>   Arm CoreSight SoC-400 (n/a for QEMU)
>   Cortex-M System Design Kit
>   PL022 Serial Port
>   NIC-400 Network interconnect
>
> But if writing it from scratch is simpler so be it. The real question is
> what new hardware would we need to model to be able to bring something
> up that is useful to Xen?

Just the board, basically. The SSE-200 is specifically a
dual-Cortex-M33 block; all the references to it in the
AN536 appnote look like cases where text wasn't sufficiently
cleaned up when creating it based on the M-profile doc...

-- PMM

Re: QEMU features useful for Xen development?

[Ayan Kumar Halder]

Hi Peter/Alex,

Appreciate your help. :)

On 31/08/2023 11:03, Peter Maydell wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> On Thu, 31 Aug 2023 at 10:53, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Peter Maydell <peter.maydell@linaro.org> writes:
>>
>>> On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>> As Xen is gaining R52 and R82 support, it would be great to be able to
>>>> use QEMU for development and testing there as well, but I don't think
>>>> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
>>>> need EL2 support in the GIC/timer for R52/R82 as well.
>>> We do actually have a Cortex-R52 model which at least in theory
>>> should include EL2 support, though as usual with newer QEMU
>>> stuff it quite likely has lurking bugs; I'm not sure how much
>>> testing it's had. Also there is currently no board model which
>>> will work with the Cortex-R52 so it's a bit tricky to use in practice.
>>> (What sort of board model would Xen want to use it with?)
>> We already model a bunch of the mps2/mps3 images so I'm assuming adding
>> the mps3-an536 would be a fairly simple step to do (mps2tz.c is mostly
>> tweaking config values). The question is would it be a useful target for
>> Xen?
> All our MPS2/MPS3 boards are M-profile. That means we have the
> device models for all the interesting devices on the board, but
> it would be simpler to write the an536 board model separately.
> (In particular, the M-profile boards are wrappers around an
> "ARMSSE" sort-of-like-an-SoC component; there's no equivalent
> for the Cortex-R52.)
>
>>    https://developer.arm.com/documentation/dai0536/latest/

Yes, it will be helpful if Qemu can model this board. We have a 
downstream port of Xen on R52 (upstreaming is in progress).

So, we can test the Qemu model with Xen.

Also if all works fine, we might consider adding this to the upstream 
Xen CI docker.

Out of curiosity, are you planning to add Qemu R52 SoC support to Zephyr ?

- Ayan

>>
>>> The Cortex-R82 would be more work, because (unlike the R52) it's
>>> AArch64, and we don't have Armv8-R AArch64 support yet, only the AArch32.
>>>
>>> I haven't looked at whether GIC on R-profile requires any changes
>>> from the A-profile GIC; on A-profile obviously we emulate the
>>> virtualization support already.
>>>
>>>> On Cortex-As, in addition to a PCI root complex and an arbitrary PCI
>>>> device, SMMUv3 emulation (both stages) and GICv3 ITS are needed to be
>>>> able to test PCI Passthrough.
>> We have ITS emulation support and it was recently plumbed into the
>> "sbsa-ref" board as it is needed for higher level SBSA compliance.
>>
>>>> However, if I recall correctly SMMUv3
>>>> emulation in QEMU might not be complete enough to enable us to use it.
>>> Yeah, at the moment the SMMU emulation supports stage 1 and stage 2,
>>> but not both at the same time. This is good enough for PCI passthrough
>>> with a Linux guest using KVM to pass a device through to a nested
>>> Linux guest.
>> Is this a missing feature for SMMUv3 or something introduced in the
>> later revisions?
> It's a missing feature. The SMMUv3 spec allows an implementation
> to implement stage 1, stage 2 or both. We started with just a
> stage-1-only implementation because Linux doesn't need any more.
> Stage-2-only just landed recently. Nobody's looked at both-stages yet.
>
> thanks
> -- PMM
>

Re: QEMU features useful for Xen development?

[Alex Bennée]

Ayan Kumar Halder <ayankuma@amd.com> writes:

> Hi Peter/Alex,
>
> Appreciate your help. :)
>
> On 31/08/2023 11:03, Peter Maydell wrote:
>> On Thu, 31 Aug 2023 at 10:53, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>> Peter Maydell <peter.maydell@linaro.org> writes:
>>>
>>>> On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>>> As Xen is gaining R52 and R82 support, it would be great to be able to
>>>>> use QEMU for development and testing there as well, but I don't think
>>>>> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
>>>>> need EL2 support in the GIC/timer for R52/R82 as well.
>>>> We do actually have a Cortex-R52 model which at least in theory
>>>> should include EL2 support, though as usual with newer QEMU
>>>> stuff it quite likely has lurking bugs; I'm not sure how much
>>>> testing it's had. Also there is currently no board model which
>>>> will work with the Cortex-R52 so it's a bit tricky to use in practice.
>>>> (What sort of board model would Xen want to use it with?)
>>> We already model a bunch of the mps2/mps3 images so I'm assuming adding
>>> the mps3-an536 would be a fairly simple step to do (mps2tz.c is mostly
>>> tweaking config values). The question is would it be a useful target for
>>> Xen?
>> All our MPS2/MPS3 boards are M-profile. That means we have the
>> device models for all the interesting devices on the board, but
>> it would be simpler to write the an536 board model separately.
>> (In particular, the M-profile boards are wrappers around an
>> "ARMSSE" sort-of-like-an-SoC component; there's no equivalent
>> for the Cortex-R52.)
>>
>>>    https://developer.arm.com/documentation/dai0536/latest/
>
> Yes, it will be helpful if Qemu can model this board. We have a
> downstream port of Xen on R52 (upstreaming is in progress).
>
> So, we can test the Qemu model with Xen.
>
> Also if all works fine, we might consider adding this to the upstream
> Xen CI docker.
>
> Out of curiosity, are you planning to add Qemu R52 SoC support to Zephyr ?

Generally enabling other software platforms is out of scope for the QEMU
team as we have plenty enough to do in QEMU itself. However its
certainly useful to have images we can test with.

Eyeballing the Zephyr docs it looks like it already supports some
R-profile cores on various boards, including CPU_CORTEX_R52 for the
NXPS32Z/E board. The BSP sections mostly look like config fragments but
I'm not really familiar with how Zephyr goes together.

I can ask our micro-controller experts what might be missing and need
implementing but I can't commit them to work on it ;-)

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: QEMU features useful for Xen development?

[Alex Bennée]

Ayan Kumar Halder <ayankuma@amd.com> writes:

> Hi Peter/Alex,
>
> Appreciate your help. :)
>
> On 31/08/2023 11:03, Peter Maydell wrote:
>> CAUTION: This message has originated from an External Source. Please
>> use proper judgment and caution when opening attachments, clicking
>> links, or responding to this email.
>>
>>
>> On Thu, 31 Aug 2023 at 10:53, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>> Peter Maydell <peter.maydell@linaro.org> writes:
>>>
>>>> On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>>> As Xen is gaining R52 and R82 support, it would be great to be able to
>>>>> use QEMU for development and testing there as well, but I don't think
>>>>> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
>>>>> need EL2 support in the GIC/timer for R52/R82 as well.
>>>> We do actually have a Cortex-R52 model which at least in theory
>>>> should include EL2 support, though as usual with newer QEMU
>>>> stuff it quite likely has lurking bugs; I'm not sure how much
>>>> testing it's had. Also there is currently no board model which
>>>> will work with the Cortex-R52 so it's a bit tricky to use in practice.
>>>> (What sort of board model would Xen want to use it with?)
>>> We already model a bunch of the mps2/mps3 images so I'm assuming adding
>>> the mps3-an536 would be a fairly simple step to do (mps2tz.c is mostly
>>> tweaking config values). The question is would it be a useful target for
>>> Xen?
>> All our MPS2/MPS3 boards are M-profile. That means we have the
>> device models for all the interesting devices on the board, but
>> it would be simpler to write the an536 board model separately.
>> (In particular, the M-profile boards are wrappers around an
>> "ARMSSE" sort-of-like-an-SoC component; there's no equivalent
>> for the Cortex-R52.)
>>
>>>    https://developer.arm.com/documentation/dai0536/latest/
>
> Yes, it will be helpful if Qemu can model this board. We have a
> downstream port of Xen on R52 (upstreaming is in progress).
>
> So, we can test the Qemu model with Xen.
>
> Also if all works fine, we might consider adding this to the upstream
> Xen CI docker.

We wrote up whats required on our JIRA:

  https://linaro.atlassian.net/browse/QEMU-598

Given we have most of the bits already it shouldn't take long to build a
model for it. However I want to draw every ones attention to the notes:

  The real FPGA image does not have a global monitor that can support
  LDREX/STREX between the two CPUs for accesses in the DDR4 RAM or BRAM.
  This means that effectively guest software will only be able to use
  one CPU (the local monitor within the CPU works fine). This
  restriction won’t apply to the QEMU model, but is important to be
  aware of if you’re writing guest software on the QEMU model that you
  also want to be able to run on the FPGA board.

However if that meets the requirements for Xen testing and no one tries
to run SMP loads on real HW then I can try and find space on our roadmap
to do it (rough guess 9.1 or 9.2?).

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: QEMU features useful for Xen development?

[Ayan Kumar Halder]

Hi Alex,

On 20/10/2023 16:15, Alex Bennée wrote:
> Ayan Kumar Halder <ayankuma@amd.com> writes:
>
>> Hi Peter/Alex,
>>
>> Appreciate your help. :)
>>
>> On 31/08/2023 11:03, Peter Maydell wrote:
>>> CAUTION: This message has originated from an External Source. Please
>>> use proper judgment and caution when opening attachments, clicking
>>> links, or responding to this email.
>>>
>>>
>>> On Thu, 31 Aug 2023 at 10:53, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>> Peter Maydell <peter.maydell@linaro.org> writes:
>>>>
>>>>> On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
>>>>>> As Xen is gaining R52 and R82 support, it would be great to be able to
>>>>>> use QEMU for development and testing there as well, but I don't think
>>>>>> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
>>>>>> need EL2 support in the GIC/timer for R52/R82 as well.
>>>>> We do actually have a Cortex-R52 model which at least in theory
>>>>> should include EL2 support, though as usual with newer QEMU
>>>>> stuff it quite likely has lurking bugs; I'm not sure how much
>>>>> testing it's had. Also there is currently no board model which
>>>>> will work with the Cortex-R52 so it's a bit tricky to use in practice.
>>>>> (What sort of board model would Xen want to use it with?)
>>>> We already model a bunch of the mps2/mps3 images so I'm assuming adding
>>>> the mps3-an536 would be a fairly simple step to do (mps2tz.c is mostly
>>>> tweaking config values). The question is would it be a useful target for
>>>> Xen?
>>> All our MPS2/MPS3 boards are M-profile. That means we have the
>>> device models for all the interesting devices on the board, but
>>> it would be simpler to write the an536 board model separately.
>>> (In particular, the M-profile boards are wrappers around an
>>> "ARMSSE" sort-of-like-an-SoC component; there's no equivalent
>>> for the Cortex-R52.)
>>>
>>>>     https://developer.arm.com/documentation/dai0536/latest/
>> Yes, it will be helpful if Qemu can model this board. We have a
>> downstream port of Xen on R52 (upstreaming is in progress).
>>
>> So, we can test the Qemu model with Xen.
>>
>> Also if all works fine, we might consider adding this to the upstream
>> Xen CI docker.
> We wrote up whats required on our JIRA:
>
>    https://linaro.atlassian.net/browse/QEMU-598
>
> Given we have most of the bits already it shouldn't take long to build a
> model for it. However I want to draw every ones attention to the notes:
>
>    The real FPGA image does not have a global monitor that can support
>    LDREX/STREX between the two CPUs for accesses in the DDR4 RAM or BRAM.
>    This means that effectively guest software will only be able to use
>    one CPU (the local monitor within the CPU works fine). This
>    restriction won’t apply to the QEMU model, but is important to be
>    aware of if you’re writing guest software on the QEMU model that you
>    also want to be able to run on the FPGA board.
>
> However if that meets the requirements for Xen testing and no one tries
> to run SMP loads on real HW then I can try and find space on our roadmap
> to do it (rough guess 9.1 or 9.2?).

Yes, this meets the requirements for Xen testing.

It will be great to use Qemu for Xen testing.

Please keep me posted on this feature as I am plumbing the R52 bits in 
upstream Xen.

- Ayan

>

Re: QEMU features useful for Xen development?

[Peter Maydell]

On Thu, 31 Aug 2023 at 11:32, Ayan Kumar Halder <ayankuma@amd.com> wrote:
> On 31/08/2023 11:03, Peter Maydell wrote:
> > On Thu, 31 Aug 2023 at 10:53, Alex Bennée <alex.bennee@linaro.org> wrote:
> >> Peter Maydell <peter.maydell@linaro.org> writes:
> >>> On Thu, 31 Aug 2023 at 01:57, Stefano Stabellini <sstabellini@kernel.org> wrote:
> >>>> As Xen is gaining R52 and R82 support, it would be great to be able to
> >>>> use QEMU for development and testing there as well, but I don't think
> >>>> QEMU can emulate EL2 properly for the Cortex-R architecture. We would
> >>>> need EL2 support in the GIC/timer for R52/R82 as well.

> >>> (What sort of board model would Xen want to use it with?)

> >> We already model a bunch of the mps2/mps3 images so I'm assuming adding
> >> the mps3-an536 would be a fairly simple step to do (mps2tz.c is mostly
> >> tweaking config values). The question is would it be a useful target for
> >> Xen?

> Yes, it will be helpful if Qemu can model this board. We have a
> downstream port of Xen on R52 (upstreaming is in progress).
>
> So, we can test the Qemu model with Xen.

Hi, all. I just wanted to provide an update on this. We've now
completed the mps3-an536 board model, and you can find it if
you check out the head-of-git QEMU. The new board will be in the
9.0 QEMU release, but if you have a chance to give it a spin now
we'll be able to fix any bugs or problems with it before the release.

The documentation for the board is here:
https://www.qemu.org/docs/master/system/arm/mps2.html
and it lists the limitations/missing features. (If any of those
are important let me know and we can look at scheduling the work
to fill them in.)

I'd also like to draw your attention to the note about the UART
ordering on the AN536; unfortunately the hardware setup is a bit
awkward here, so if you have an "I don't see any output" problem
make sure your guest is sending to the same UART you're asking
QEMU to show you the output from :-)

thanks
-- PMM