Re: [PATCH v1 00/27] KVM: s390: Introduce arm64 KVM

[Steffen Eiden <seiden@linux.ibm.com>]

Hi Marc & Will,

On Mon, Apr 20, 2026 at 11:57:38AM +0100, Marc Zyngier wrote:

> Hi Steffen, s390 folks,
> 
> On Thu, 02 Apr 2026 05:20:56 +0100,
> Steffen Eiden <seiden@linux.ibm.com> wrote:
> > 
> > By introducing a novel virtualization acceleration for the ARM architecture on
> > s390 architecture, we aim to expand the platform's software ecosystem. This
> > initial patch series lays the groundwork by enabling KVM-accelerated ARM CPU
> > virtualization on s390. To achieve this, a common KVM layer between s390 and
> > arm64 is introduced (see below for more details). Design considerations of
> > arm64 on the s390 Architecture The s390 virtualization architecture is extended
> > with a set of new instructions dedicated to supporting ARM-based virtual
> > machines. The s390 KVM host acts as EL2 (hypervisor) for a EL1/EL0
> > (OS/application) arm64 guest. To achieve this, the new Start-Arm-Execution
> > (SAE) instruction enables accelerated execution of arm64 VMs.  Additional new
> > s390 instructions aLre introduced to query available arm64 features, used to
> > populate the arm64 ID register contents, as well as, new s390 instructions to
> > save/restore various arm64 registers in the VM context.
> 
> Apologises for the delay in responding to this, things got delayed a
> bit with the Easter break. Since then, Will and I have been discussing
> this series and what it means for the future of the arm64 port.
> 
> By way of opening the discussion, we want to be clear that we are
> supportive of the effort. Our comments here should be seen as areas of
> potential improvement and not as rejection of what you are trying to
> achieve.
>
Thank you for your answer. We are happy to hear that you support our
efforts.
 
> * Code movement:
> 
>   The patches you have posted demonstrate that it is possible to
>   expose a large amount of arm64-specific code and definition to s390,
>   and yet still manage to build both architectures without regression.
>   However, the result looks rather messy and may adversely affect
>   maintainability on the arm64 side.
> 
>   The moving of files into shared locations is particularly painful,
>   and gets in the way of overall maintainability. Not only does it
>   break our comfortable habits, it makes the backporting of fixes
>   harder.  Importantly, these changes come with no benefit on the
>   arm64 side.
> 
>   Would it be possible to try some other means of reaching the
>   arm64-specific files *in situ*, either by making use of relative
>   paths, or by using symbolic links? Even better, files that are
>   generated on arm64 (such as the sysreg definitions) should equally
>   be generated for s390, locally to the s390 part of the tree.
> 
Yes, we can do that. Our first iteration had an extensive use of symlinks for
headers. We feared that this approach would gain no big support as it was
quite messy and gave a lot of surface for future errors. So we moved to the
current implementation. For the non-KVM headers I could see moving back to the
symlink approach to reduce the backport & maintainability burden for you.
Preferably, those headers are kept clean of any arm implementation specific
things (e.g. sysreg vs sysreg-defs).

For shared kvm headers and code, we think moving them to the proposed location
helps reducing regression issues when someone changes a function as that
location makes it 100% clear that this is shared code. Tagging a file 
as __shared__ by other means (e.g. by a file name suffix) may 
be OK for us as well if that reduces your maintenance burden.

The generated (e.g. sysreg) definitions are already generated into the s390
tree:
{outdir}/arch/s390/include/generated/asm/sysreg-gen-defs.h
We just reuse the makefile definitions from arm.

As a side note: We tried to reuse as much arm code as possible - to not
reinvent the wheel - while keeping the arm churn minimal. While going through
the arm code, we tried to spot parts that could benefit from refactoring and
did that. By moving especially the kvm code to another location we wanted to
emphasize that this code is shared between arm and s390 and possibly other
architectures in the future.

We will prototype alternatives including using symlinks and post them soon here. 

>   But that doesn't mean that we consider that the arm64 tree is
>   immutable and that we are not open to change, quite the opposite.
>   Most of the KVM/arm64 include files are an unholy mix of arch
>   definitions, data structures that have some arch relevance, but also
>   code and data that is strictly implementation specific. Splitting
>   these (as you already have for some include files) could both help
>   with sharing what is actually needed, keep the arm64-specific stuff
>   at bay, *and* benefit arm64's overall maintainability. We would need
>   some tooling to enforce the split and avoid regressing it, something
>   that could happen quickly given the level of activity on arm64. Yet
>   another way to achieve this could be to mechanically process the
>   arm64 files as part of the s390 build to extract the relevant
>   information, and we could help with this.

That is good to hear. We of course also wanted arm to improve with our changes.

> 
>   Looking a bit more into the distance, it is likely that KVM/arm64
>   will grow feature support quicker than s390 can absorb them, and
>   that some feature won't ever make any sense of s390 (pKVM, for
>   example).  We need to establish how these features can be built
>   without arm64 being hindered by s390. This is also true when adding
>   architectural support for features that don't exist in the s390 view
>   of arm64.

Yes, of course s390 should not hinder arm64 to progress. 120% agree! We will
be available in case that happens. However, I do not think this is a big
problem. By defining some arm feature macros to false we already did turn off
few arm features for us at compile time. Compiler optimization is a very good
friend here.

> 
> * UAPI and guest API:
> 
>   Obviously, one of our biggest concerns is the userspace API. We
>   appreciate that you want to reuse it as it is, warts and all, and
>   directly incorporate additional feature support as it becomes

Yes, implementing the arm64 kvm-UAPI was one of our primary goals so that we
can reuse existing arm64 VMMs e.g. Qemu.

>   available. This means that, should any divergence in UAPI appear,
>   the source of truth must be on the arm64 side. This has the
>   following consequences:

Yes, arm64 is the source of truth for us. That is the exact reason we did not
copy the (UAPI) headers but moved & share them.

> 
>     - s390 cannot add extensions to the UAPI
> 
>     - s390 must be compatible with all future arm64 extensions
yes
> 
>   Similar concerns exist on the guest/hypervisor API, including:
> 
>   - errata mitigation: this is unsurprisingly a hot topic, which keeps
>     causing us some massive headaches. We are particularly concerned
>     about errata that need to be disclosed to the guest and acted upon
>     via a hypercall. Should there be a need for those, how will we
>     coordinate the deployment of such hypercall?
> 
>     The way it has been deployed so far is that PSCI has grown an
>     errata discovery mechanism. ARM assigns function numbers and
>     specifies what these hypercalls mitigate. KVM, in turn, takes part
>     in implementing the mitigation. We expect that s390 would follow
>     the same behaviour, including coordination with ARM for the
>     function numbering.

Yes. s390 will follow those things. We are planning to reuse the complete 
arm hypercall code including the current (and future) errata detection that
comes with it.
This change will come in one of the future series. 

> 
>   - device assignment: this is unknown territory for us, as we
>     commonly use vfio-pci (and more occasionally vfio-platform). How
>     would that look for an arm64 guest on s390?
> 
We plan to work with virtio-pci and vfio-pci. No plans to assign ccw devices to
 arm guests.

>   - s390-specific ISA extension: although we obviously cannot control
>     how you will decide to expose features to your arm64 guests,
>     KVM/arm64 makes a point of forbidding any use of implementation
>     specific instruction or system registers. We expect the s390
>     implementation to uphold this.
>

We have no plans of using private ISA extensions or deviations.

>   - s390-specific hypercalls: aside from the errata handling
>     mentioned above, we would very much like to avoid anything that is
>     implementation specific, and keep the hypercall space as small as
>     possible. In other words, an unenlightened arm64 guest must work
>     and continue to work.

Of course an unenlightened arm64 guest must keep working and it should
work with good performance and usability. This is another primary goal of
this project.
Given that we have some history of paravirtual optimizations on s390, we 
might propose some hypercalls in the future. But this will then very likely to
the benefit of all arm platforms and implemented on both host variants. 

> 
> * Overall maintenance
> 
>   Unsurprisingly, we are not totally familiar with s390. To say that
>   there is a learning gap would only be an understatement. So how do
>   we make sure we don't break things out of pure ignorance? Is there
>   any documentation we can refer to when hacking on code that will
>   eventually run on your side of the computing universe?

I am aware of the Kernel Documentation for s390.
Also, for the z/Architecture in general:
Principles of Operation might be a good start to learn about s390 architecture.

Of course we will be available for answering s390 architecture questions. 

> 
>   We need to be able to build and test what we produce. How do we go
>   about that? We appreciate that you may not be in a position to help
>   with this right now, but at least having a plan would be reassuring.
>   This should include things like automatic testing of our CI branches.
>   We are happy to test build s390 as part of our maintenance flow, if
>   pointed to existing binary toolchains compiled for arm64 and x86,
>   together with a typical configuration.
>

For your side:
Cross compiling the kernel is a good starting point. GCC/Clang cross compile
toolchains for s390 are available on all major distros. They are typically
postfixed with ‘-s390x-linux-gnu‘, note the x after s390. defconfig has KVMARM
in it - this should be suitable for testing you do not break s390 compilation.

You can get access to s390 resources for doing native builds in the LinuxONE
community cloud. Those are VMs itself -> run tests are not possible.
https://community.ibm.com/zsystems/l1cc
They also offer permanent access to OSS communities if necessary. We can help
to connect you with those teams.

Another option would be to spin-up a s390 qemu-tcg guest to build the kernel
'native'.

For us:
Yes we are planning to do regular tests to prevent breaking arm. Testing your
CI branches seems to be a good starting point for this. Do you have a few
pointers which are suited best?

>   What about debugging? We expect that you'd have to help, should an
>   arm64 change cause a regression on s390, as it is fairly unlikely
>   that we would be able to reproduce it.

Positive, we will do whatever we can to support you in any way.

> 
>   Finally, we feel it would be beneficial for both projects to swap
>   prisoners and have cross-reviewers in MAINTAINERS, so that there is
>   an s390 reviewer added to KVM/arm64, and an arm64 reviewer added to
>   KVM/s390.

Great Idea and I like the wording :)
We’ll start with the exchange. I (Steffen) would volunteer to be sent over to you.
I will add myself as kvm/arm64 reviewer in v2 of this series if that is OK for
you.

For the other way we appreciate any volunteers and also will ask around for suitable
people with arm and preferably also s390 knowledge. 

> 
> It probably would be beneficial to work through some of these things
> face-to-face. Maybe around LPC or KVM Forum if you manage to get
> there? Or some other place/time?

Totally agree, although I would prefer an earlier date (probably virtual) to 
get rid of any serious misunderstandings that may be there early. 
Surely, we can meet at LPC and/or KVM-forum as well to discuss even more.

Thank you very much for your openness and your constructive, honest
feedback.

	Steffen & the KVM/s390 team

> 
> Thanks,
> 
> 	Marc and Will
> 
> -- 
> Without deviation from the norm, progress is not possible.