Re: FreeBSD guest with VTD NIC not passing traffic

[Alex Williamson <alex.williamson@redhat.com>]

On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote:
> Hi,
>        I am running Ubuntu 10.10 (amd64) on a 2 socket  nehalem based
> server with IOH 5520. 5520 supports VTD.
> I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter
> which I assigned through VT-D to FreeBSD 8.2 running
> as guest os. The ixgbe driver detects the device and the driver
> successfully configures the device. But the link
> never comes up. It looks like link up/down interrupts are not
> delivered. Then I checked kvm interrupt assignment and as expected
> kvm could not make MSI-X entries for the VT-d guest. So no output from
> "grep kvm /proc/interrupt". By enabling some debugs in the
> qemu-kvm I figured out that the MSI-x updates are not received
> properly. It does look like Linux updates MSI-X table in a batch
> fashion
> which qemu-kvm gets in  one shot and every thing works fine in case of
> linux. In case of FreeBSD PCIE updates come /MSI-X entry
> which qemu-kvm can't make use.

That's right, Linux and Windows both seem to setup the MSI-X table then
enable it in one shot, so we only trigger the interrupt programming when
the enable bit is set.  We don't trigger changes on writes to the MSI-X
table... not very accurate emulation of mask bits.

> To overcome this I compiled latest intel ixgbe driver with MSI-X
> diabled. This time MSI interrupt got allocated both in the guest
> and in the qemu-kvm (host kernel). Still I could not get link UP. I
> modified the FreeBSD driver to poll for Link in some local_timer
> task. Link comes up but no traffic flows. The MAC statistics show
> packets received but packets do not reach the guest.
> DMAing of packet may be failing. I could not find out the reason.
> The same happens with legacy interrupt allocated in the guest. I even
> tried qemu-kvm prefer_msi=off.
> 
> I think there are two problems1
> 1. Interrupt delivery either because of interrupt remapping failure or
> all out interrupt allocation failure in qemu-kvm
> 2. Packets not getting DMAed to the guest possibly because of some DMAR issue.
> 
> Before doing VT-d I made sure connections are fine and the adapter
> works fine in bare metal Linux.
> I also tried VT-d of 82599 with Linux as guest (both 32 bits(PAE and
> non-PAE) and amd64) and it just works magically.

Unknowns: does the intel ixgbe driver work in FreeBSD with MSI-X
disabled, does polling for link UP work.  The way you describe MSI-X
interrupt setup for FreeBSD is likely something we don't handle
correctly.  You might do a hack in device-assignment.c to call
assigned_dev_update_msix from msix_mmio_writel any time we write to the
vector control dword.  We really have no support for either function
mask or per vector mask right now.  I doubt you're having DMAR issues or
you'd likely be seeing errors in the host dmesg.

> Is this a known issue I am hitting with non-Linux guest oses and VT-d
> ? Appreciate any help in debugging this problem.
> 
> -Shashidhar
> 
> Linux kernel - 2.6.35
> kvm - 0.14.1

It'd be helpful to test on something newer too.  Thanks,

Alex