Re: FreeBSD guest with VTD NIC not passing traffic

[Alex Williamson <alex.williamson@redhat.com>]

On Fri, 2012-01-13 at 22:00 +0100, Jan Kiszka wrote:
> On 2012-01-04 04:21, Alex Williamson wrote:
> > On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote:
> >> Hi,
> >>        I am running Ubuntu 10.10 (amd64) on a 2 socket  nehalem based
> >> server with IOH 5520. 5520 supports VTD.
> >> I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter
> >> which I assigned through VT-D to FreeBSD 8.2 running
> >> as guest os. The ixgbe driver detects the device and the driver
> >> successfully configures the device. But the link
> >> never comes up. It looks like link up/down interrupts are not
> >> delivered. Then I checked kvm interrupt assignment and as expected
> >> kvm could not make MSI-X entries for the VT-d guest. So no output from
> >> "grep kvm /proc/interrupt". By enabling some debugs in the
> >> qemu-kvm I figured out that the MSI-x updates are not received
> >> properly. It does look like Linux updates MSI-X table in a batch
> >> fashion
> >> which qemu-kvm gets in  one shot and every thing works fine in case of
> >> linux. In case of FreeBSD PCIE updates come /MSI-X entry
> >> which qemu-kvm can't make use.
> > 
> > That's right, Linux and Windows both seem to setup the MSI-X table then
> > enable it in one shot, so we only trigger the interrupt programming when
> > the enable bit is set.  We don't trigger changes on writes to the MSI-X
> > table... not very accurate emulation of mask bits.
> 
> According to the PCI spec, updates that happen while a vector is
> unmasked, need not be considered by the hardware (thus the hypervisor
> here). Is that the scenario here?

I'm assuming the vector is masked in the MSI-X table.  So Linux/Windows
do:

a) program MSI-X table
b) enable MSI-X in capability register

Whereas FreeBSD does:

a) enable MSI-X in capability register (vectors masked in table)
b) program and unmask individual vectors

Thanks,

Alex