Re: qemu/hw/device-assignment: questions about msix_table_page

[Sheng Yang <sheng@linux.intel.com>]

On Monday 27 April 2009 22:15:04 Michael S. Tsirkin wrote:
> On Mon, Apr 27, 2009 at 10:03:59PM +0800, Sheng Yang wrote:
> > On Monday 27 April 2009 21:51:34 Michael S. Tsirkin wrote:
> > > On Mon, Apr 27, 2009 at 09:16:14PM +0800, Sheng Yang wrote:
> > > > On Monday 27 April 2009 18:41:17 Michael S. Tsirkin wrote:
> > > > > Sheng, Marcelo,
> > > > > I've been reading code in qemu/hw/device-assignment.c, and
> > > > > I have a couple of questions about msi-x implementation:
> > > >
> > > > Hi Michael
> > > >
> > > > > 1. What is the reason that msix_table_page is allocated
> > > > >    with mmap and not with e.g. malloc?
> > > >
> > > > msix_table_page is a page, and mmap allocate memory on page boundary.
> > > > So I use it.
> > >
> > > Just wondering, would e.g. posix_memalign work here as well?
> >
> > Um, I think it should work too.
> >
> > > > > 2. msix_table_page has the guest view of the msix table for the
> > > > > device. However, even this memory isn't mapped into guest directly,
> > > > > instead msix_mmio_read/msix_mmio_write perform the write in qemu.
> > > > > Won't it be possible to map this page directly into
> > > > >    guest memory, reducing the overhead for table writes?
> > > >
> > > > First, Linux configured the real MSI-X table in device, which is out
> > > > of our scope. KVM accepted the interrupt from Linux, then inject it
> > > > to the guest according to the MSI-X table setting of guest. So KVM
> > > > should know about the page modification. For example, MSI-X table got
> > > > mask bit which can be written by guest at any time(this bit haven't
> > > > been implement yet, but should be soon), then we should mask the
> > > > correlated vector of real MSI-X table; then guest may modified the
> > > > MSI address/data, that also should be intercepted by KVM and used to
> > > > update our knowledge of guest. So we can't passthrough the
> > > > modification.
> > >
> > > Right, I see that. However all msix_mmio_write does is a memcpy.
> > > So what I don't understand yet, what causes the real MSI-X table to be
> > > modified? Where's that code?
> >
> > Now it haven't been allowed to do dynamically change...:( For now, please
> > refer to assigned_device_update_msix_mmio in qemu/hw/device-assignment.c.
> > It would scan the MSI-X page and transfer to KVM through ioctl. And for
> > the kernel part, please refer to pci_enable_msix().
>
> Right, I noticed that. So msix_mmio_read/write are placeholders, right?
> For exiting functionality we thinkably could let the guest write into
> msix_table_page.

Yeah, in some meaning. But it's not suggested let guest write into the 
msix_table_page.

1. For some drivers don't set/unset mask bit, guest won't access the page 
frequently, so it shouldn't be a performance critical one. 
2. With mask bit, some drivers would access the page more frequently, but you 
have to intercept it...

My suggestion is get mask bit support first, then consider optimization. Mask 
bit would affect the performance on large scaled system, so it should be done. 
So current MSI-X status is indeed temporarily...

I guess what you see now maybe accessing is frequently due to mask bit, but we 
ignored it for now, so it may can be optimized. But I really think it's 
temporarily status, so I think it's better not to spend much effect on 
optimize before mask bit support done...

-- 
regards
Yang, Sheng

>
> > The userspace/kernel both still need change to support mask/unmask
> > feature. That's still in TODO list(and I hope it can catch up with 2.6.31
> > merge window).
> >
> > --
> > regards
> > Yang, Sheng
> >
> > > > If guest can write to the real device MSI-X table directly, it would
> > > > cause chaos on interrupt delivery, for what guest see is totally
> > > > different with what's host see...
> > >
> > > Obviously.
> > >
> > > Thanks,