Re: Enable more than 255 VCPU support without irq remapping function in the guest

[Yang Zhang <yang.zhang.wz@gmail.com>]

On 2016/5/3 12:55, Jan Kiszka wrote:
> On 2016-05-03 04:03, Nadav Amit wrote:
>> Yang Zhang <yang.zhang.wz@gmail.com> wrote:
>>
>>> I think it is not only interrupt. There must have the DMAR emulation and
>>> the cost for DMA is heavy in VM(DMA operations are very frequently). I
>>> cannot remember whether there are strong dependency in hardware between
>>> DMAR and IR(I know IR is relying on QI). Even hardware dependency is ok,
>>> is it ok for OS running in hardware with IR but without DMAR?
>>
>> Do you know a way for the IOMMU to report that DMAR is disabled, while IR
>> is enabled?
>
> The hardware cannot decide about disabling this, but the guest can, of
> course. In fact, you can even configure Linux to have DMAR off by
> default until you pass "intel_iommu=on" on the command line (I think
> distros still do this - at least they used to). No idea about other
> OSes, though.

If we can disable DMAR in guest, it should be enough.

>
>>
>> Anyhow, the VM can use IOMMU passthrough mode to avoid most IOMMU overhead.
>> Regardless, a recent patch-set should improve DMAR performance
>> considerably [1].
>
> The bottleneck with emulated DMAR is rather in QEMU. But DMAR can be
> almost as cheap as IR once we get it running for VFIO and vhost: both
> need proper caching because they do not work with QEMU in the loop for
> each and every DMA transfer. Still no need to deviate from physical
> hardware.

Sorry, i don't know detail about how VFIO and vhost work with IR. But it 
seems hard to do proper caching since DMA allocations are very 
frequently in Linux unless we move the whole iommu emulation to kernel. 
Another idea is using two iommus: one for Qemu and one for device in 
kernel like vfio and vhost. I did the similar thing in Xen in several 
years ago which uses two iommus solution and it works well in my 
experiment environment. Besides, this solution is easy for nested device 
pass-through. The Page 32 of [1] has more detail.

[1] http://docplayer.net/10559370-Nested-virtualization.html

-- 
best regards
yang