Re: pv-ops domU not working with MSI interrupts on Nehalem

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Oct 13, 2010 at 02:36:47PM -0700, Bruce Edge wrote:
> On Mon, Oct 11, 2010 at 2:46 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Mon, Oct 11, 2010 at 02:12:22PM -0700, Bruce Edge wrote:
> >> On Fri, Oct 1, 2010 at 2:11 PM, Konrad Rzeszutek Wilk
> >> <konrad.wilk@oracle.com> wrote:
> >> > On Mon, Sep 27, 2010 at 08:52:39AM -0700, Bruce Edge wrote:
> >> >> One of our developers who is working on a tachyon driver is
> >> >> complaining that the pvops domU kernel is not working for these MSI
> >> >> interrupts.
> >> >> This is using the current head of xen/2.6.32.x on both a single
> >> >> Nahelam 920 and a dual E5540. This behavior is consistent with Xen
> >> >> 4.0.1, 4.0.2.rc1-pre and 4.1.
> >> >
> >> >
> >> > I just checked on my SuperMicro X8DTN, this combination
> >> >  - For Dom0, git commit fe999249 (2.6.32.18)
> >> >  - For DomU, devel/xen-pcifront-0.6 or devel/xen-pcifront-0.7
> >> >  - For Hypervisor I used cs 21976, but found that the latest (22155) works too
> >> >
> >> > with which where I passed in PCI devices with legacy IRQ, MSI, and MSI-X. I tried
> >> > a combination of doing this with IOMMU (VT-d) and without - both cases these devices:
> >> >
> >> > 00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
> >> > 00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
> >> > 00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
> >> > 00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
> >> > 03:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
> >> > 0a:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
> >> >
> >> > worked just fine (either defining pci=["..."] or just using pci-attach).
> >> >
> >> > But if I use the latest xen/next or xen/stable-2.6.32.x it does not look
> >> > that happy :-(
> >> >
> >> >
> >>
> >> Konrad,
> >> To try eliminate the remaining differences here, could you post your
> >> dom0/domU config files?
> >
> > Sure. See attached
> 
> Konrad,
> That made a big difference. Looks much better now. It's been kicked
> over to several developers who have each got our tachyon driver
> working a little bit better.

Hmm, I didn't really try to do anything fancy with the configs. Any
inklings of what config option might have caused all this headache?

> 
> Now the sticking point is an apparent limitation on the amount of
> memory one can request using pci_map_single. It appears that we can
> only ask for 256K or less.  We need a 2MB DMA buffer.

You can't use SG buffers? And chain them together and provide them
to the device? A lot of other drivers do this ...

> Is there some alternate mechanism for getting a larger physically
> contiguous buffer under pvops?

Why the contiguous requirement?
> 
> Thanks
> 
> -Bruce
> 
> >> I'd like to build the same kernels to get an apples->apples comparison.
> >>
> >> Also, could you include your grub info and domU cfg file?
> >>
> >> These may eliminate some of the remaining diffs in the configs and
> >> show why your's works while mine does not.
> >>
> >> Thanks
> >>
> >> -Bruce
> >