Re: [Qemu-devel] [PATCH] vfio: Align iova also to IOMMU page size

[Alex Williamson <alex.williamson@redhat.com>]

On Tue, 2015-11-24 at 18:24 +0300, Pavel Fedin wrote:
>  Hello!
> 
> > >  So, i've got this problem on ARM64. On ARM64 we actually can never have 1K pages. This page
> > > size was supported only by old 32-bit ARM CPUs, up to ARMv5 IIRC, then it was dropped. Linux
> > > OS never even used it.
> > >  But, since qemu can emulate those ancient CPUs, TARGET_PAGE_BITS is defined to 10 for ARM.
> > > And, ARM64 and ARM32 is actually the same target for qemu, so this is why we still get it.
> > >  Perhaps, TARGET_PAGE_BITS should be a variable for ARM, and we should set it according to
> > > the actual used CPU. Then this IOMMU alignment problem would disappear automatically. What do
> > > you think?
> > >  Cc'ed Peter since he is the main ARM guy here.
> > 
> > Do we only see these alignments when we're emulating those old 1k page
> > processors?
> 
>  No. We see them when we are emulating brand new ARM64. And we see them only because TARGET_PAGE_BITS is still hardcoded to 10.
> 
> > If not, should we really be telling a 4k page VM about 1k
> > aligned memory?
> 
>  Well, first of all, we are not telling about aligned memory at all. I've researched for the origin of this address, it appears when we are mapping MSI-X BAR of the VFIO device.
>  My device defines this BAR to be of 2M size. In this case qemu splits it up into three regions:
>  1) Region below the MSI-X table (it's called "mmap", for me it's empty because table offset is 0)
>  2) MSI-X table itself (20 vectors = 0x00000140 bytes for me).
>  3) Region above the MSi-X table (it's called "mmap msix-hi"). Size for my case is 2M - REAL_HOST_PAGE_ALIGN(0x140) = 0x1FF000.
> Regions (1) and (3) are passed through and directly mapped by VFIO, region (2) is emulated.
>  Now, we have PBA. The device says that PBA is placed in memory specified by the same BAR as table, and its offset is 0x000f0000. PBA is also emulated by qemu, and as a result it overlaps "mmap msix-hi" region, which breaks up into two fragments as a consequence:
>  3a) offset 0x00000 size 0x0EF000
>  3b) offset 0xEF008 size 0x10FFF8
>  PBA sits between these fragments, having only 8 bytes in size.
>  Attempt to map (3b) causes the problem. vfio_mmap_region() is expected to align this up, and it does, but it does it to a minimum possible granularity for ARM platform, which, as qemu thinks, is always 1K.

Yep, on x86 we'd have target page size == host page size == minimum
iommu page size and we'd align that up to something that can be mapped,
which means we wouldn't have an iommu mapping for peer-to-peer in this
space, but nobody is likely to notice.  There are devices that want to
start taking advantage of the fact that we map mmio through the iommu
and start enabling p2p though, so it's also not something I'm eager to
remove.

Really, the problem seems to be the flat _processor_ view of memory vs
what we'd like to map through the iommu, the flat _device_ view of
memory.  All the quirks and QEMU trapping of sub-regions get wiped away
from the device view.  Thanks,

Alex