Re: [Qemu-devel] [virtio-dev] Re: [virtio-dev] Re: [virtio-dev] Re: [PATCH v2 00/16] Vhost-pci for inter-VM communication

["Michael S. Tsirkin" <mst@redhat.com>]

On Sun, Jun 04, 2017 at 06:34:45PM +0800, Wei Wang wrote:
> On 05/26/2017 01:57 AM, Michael S. Tsirkin wrote:
> > 
> > I think that's a very valid point. Linux isn't currently optimized to
> > handle packets in device BAR.
> > There are several issues here and you do need to address them in the
> > kernel, no way around that:
> > 
> > 1. lots of drivers set protection to
> >          vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
> > 
> Sorry for my late reply.
> 
> In the implementation tests, I didn't find an issue when letting the
> guest directly access the bar MMIO returned by ioremap_cache().
> If that's conventionally improper, we can probably make a new
> function similar to ioremap_cache, as the 2nd comment suggests
> below.

Right. And just disable the driver on architectures that don't support it.

> So, in any case, the vhost-pci driver uses ioremap_cache() or a similar
> function, which sets the memory type to WB.
> 

And that's great. AFAIK VFIO doesn't though, you will need to
teach it to do that to use userspace drivers.

> 
> >     vfio certainly does, and so I think does pci sysfs.
> >     You won't get good performance with this, you want to use
> >     a cacheable mapping.
> >     This needs to be addressed for pmd to work well.
> 
> In case it's useful for the discussion here, introduce a little background
> about how the bar MMIO is used in vhost-pci:
> The device in QEMU sets up the MemoryRegion of the bar as  "ram" type,
> which will finally have translation mappings created in EPT. So, the memory
> setup of the bar is the same as adding a regular RAM. It's like we are
> passing through a bar memory to the guest which allows the guest to
> directly access to the bar memory.
> 
> Back to the comments, why it is not cacheable memory when the
> vhost-pci driver explicitly uses ioremap_cache()?

It is. But when you write a userspace driver, you will need
to teach vfio to allow cacheable access from userspace.

> > 
> > 2. linux mostly assumes PCI BAR isn't memory, ioremap_cache returns __iomem
> >     pointers which aren't supposed to be dereferenced directly.
> >     You want a new API that does direct remap or copy if not possible.
> >     Alternatively remap or fail, kind of like pci_remap_iospace.
> >     Maybe there's already something like that - I'm not sure.
> > 
> 
> For the vhost-pci case, the bar is known to be a portion physical memory.

Yes but AFAIK __iomem mappings still can't be portably dereferenced on all
architectures. ioremap_cache simply doesn't always give you
a dereferencable address.

> So, in this case, would it be an issue if the driver directly accesses to
> it?
> (as mentioned above, the implementation functions correctly)
> 
> Best,
> Wei

you mean like this:
        void __iomem *baseptr = ioremap_cache(....);

        unsigned long signature = *(unsigned int *)baseptr;


It works on intel. sparse will complain though. See
Documentation/bus-virt-phys-mapping.txt


-- 
MST