Re: FreeBSD guest with VTD NIC not passing traffic

[Jan Kiszka <jan.kiszka@web.de>]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

On 2012-01-13 22:05, Alex Williamson wrote:
> On Fri, 2012-01-13 at 22:00 +0100, Jan Kiszka wrote:
>> On 2012-01-04 04:21, Alex Williamson wrote:
>>> On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote:
>>>> Hi,
>>>>        I am running Ubuntu 10.10 (amd64) on a 2 socket  nehalem based
>>>> server with IOH 5520. 5520 supports VTD.
>>>> I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter
>>>> which I assigned through VT-D to FreeBSD 8.2 running
>>>> as guest os. The ixgbe driver detects the device and the driver
>>>> successfully configures the device. But the link
>>>> never comes up. It looks like link up/down interrupts are not
>>>> delivered. Then I checked kvm interrupt assignment and as expected
>>>> kvm could not make MSI-X entries for the VT-d guest. So no output from
>>>> "grep kvm /proc/interrupt". By enabling some debugs in the
>>>> qemu-kvm I figured out that the MSI-x updates are not received
>>>> properly. It does look like Linux updates MSI-X table in a batch
>>>> fashion
>>>> which qemu-kvm gets in  one shot and every thing works fine in case of
>>>> linux. In case of FreeBSD PCIE updates come /MSI-X entry
>>>> which qemu-kvm can't make use.
>>>
>>> That's right, Linux and Windows both seem to setup the MSI-X table then
>>> enable it in one shot, so we only trigger the interrupt programming when
>>> the enable bit is set.  We don't trigger changes on writes to the MSI-X
>>> table... not very accurate emulation of mask bits.
>>
>> According to the PCI spec, updates that happen while a vector is
>> unmasked, need not be considered by the hardware (thus the hypervisor
>> here). Is that the scenario here?
> 
> I'm assuming the vector is masked in the MSI-X table.  So Linux/Windows
> do:
> 
> a) program MSI-X table
> b) enable MSI-X in capability register
> 
> Whereas FreeBSD does:
> 
> a) enable MSI-X in capability register (vectors masked in table)
> b) program and unmask individual vectors

That should work with the current code. It checks the number of vectors
on each config write, iterates the whole table, and then updates the
kernel configuration accordingly. It even requires the enable bit in the
cap register to be set before doing this.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]