Re: Discussion about virtual iommu support for Xen guest

[Paul Durrant <Paul.Durrant@citrix.com>]

> -----Original Message-----
> From: Xen-devel [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of
> Tian, Kevin
> Sent: 27 May 2016 09:35
> To: Andrew Cooper; Lan, Tianyu; jbeulich@suse.com; sstabellini@kernel.org;
> Ian Jackson; xen-devel@lists.xensource.com; Eddie Dong; Nakajima, Jun;
> yang.zhang.wz@gmail.com; Anthony Perard
> Subject: Re: [Xen-devel] Discussion about virtual iommu support for Xen
> guest
> 
> > From: Andrew Cooper [mailto:andrew.cooper3@citrix.com]
> > Sent: Thursday, May 26, 2016 7:36 PM
> >
> > On 26/05/16 09:29, Lan Tianyu wrote:
> > > Hi All:
> > > We try pushing virtual iommu support for Xen guest and there are some
> > > features blocked by it.
> > >
> > > Motivation:
> > > -----------------------
> > > 1) Add SVM(Shared Virtual Memory) support for Xen guest
> > > To support iGFX pass-through for SVM enabled devices, it requires
> > > virtual iommu support to emulate related registers and intercept/handle
> > > guest SVM configure in the VMM.
> > >
> > > 2) Increase max vcpu support for one VM.
> > >
> > > So far, max vcpu for Xen hvm guest is 128. For HPC(High Performance
> > > Computing) cloud computing, it requires more vcpus support in a single
> > > VM. The usage model is to create just one VM on a machine with the
> > > same number vcpus as logical cpus on the host and pin vcpu on each
> > > logical cpu in order to get good compute performance.
> > >
> > > Intel Xeon phi KNL(Knights Landing) is dedicated to HPC market and
> > > supports 288 logical cpus. So we hope VM can support 288 vcpu
> > > to meet HPC requirement.
> > >
> > > Current Linux kernel requires IR(interrupt remapping) when MAX APIC
> > > ID is > 255 because interrupt only can be delivered among 0~255 cpus
> > > without IR. IR in VM relies on the virtual iommu support.
> > >
> > > KVM Virtual iommu support status
> > > ------------------------
> > > Current, Qemu has a basic virtual iommu to do address translation for
> > > virtual device and it only works for the Q35 machine type. KVM reuses it
> > > and Redhat is adding IR to support more than 255 vcpus.
> > >
> > > How to add virtual iommu for Xen?
> > > -------------------------
> > > First idea came to my mind is to reuse Qemu virtual iommu but Xen didn't
> > > support Q35 so far. Enabling Q35 for Xen seems not a short term task.
> > > Anthony did some related jobs before.
> > >
> > > I'd like to see your comments about how to implement virtual iommu for
> Xen.
> > >
> > > 1) Reuse Qemu virtual iommu or write a separate one for Xen?
> > > 2) Enable Q35 for Xen to reuse Qemu virtual iommu?
> > >
> > > Your comments are very appreciated. Thanks a lot.
> >
> > To be viable going forwards, any solution must work with PVH/HVMLite as
> > much as HVM.  This alone negates qemu as a viable option.
> 
> KVM wants things done in Qemu as much as possible. Now Xen may
> have more things moved into hypervisor instead for HVMLite. The end
> result is that many new platform features from IHVs will require
> double effort in the future (nvdimm is another example) which means
> much longer enabling path to bring those new features to customers.
> 
> I can understand the importance of covering HVMLite in Xen community,
> but is it really the only factor to negate Qemu option?
> 
> >
> > From a design point of view, having Xen needing to delegate to qemu to
> > inject an interrupt into a guest seems backwards.
> >
> >
> > A whole lot of this would be easier to reason about if/when we get a
> > basic root port implementation in Xen, which is necessary for HVMLite,
> > and which will make the interaction with qemu rather more clean.  It is
> > probably worth coordinating work in this area.
> 
> Would it make Xen too complex? Qemu also has its own root port
> implementation, and then you need some tricks within Qemu to not
> use its own root port but instead registering to Xen root port. Why is
> such movement more clean?
> 

Upstream QEMU already registers PCI BDFs with Xen, and Xen already handles cf8 and cfc accesses (to turn them into single config space read/write ioreqs). So, it really isn't much of a leap to put the root port implementation in Xen.

  Paul

> >
> >
> > As for the individual issue of 288vcpu support, there are already issues
> > with 64vcpu guests at the moment.  While it is certainly fine to remove
> > the hard limit at 255 vcpus, there is a lot of other work required to
> > even get 128vcpu guests stable.
> >
> 
> Thanks
> Kevin
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel