Re: FreeBSD guest with VTD NIC not passing traffic

[Jan Kiszka <jan.kiszka@web.de>]

[-- Attachment #1: Type: text/plain, Size: 3204 bytes --]

On 2012-01-13 22:56, Alex Williamson wrote:
> On Fri, 2012-01-13 at 22:33 +0100, Jan Kiszka wrote:
>> On 2012-01-13 22:05, Alex Williamson wrote:
>>> On Fri, 2012-01-13 at 22:00 +0100, Jan Kiszka wrote:
>>>> On 2012-01-04 04:21, Alex Williamson wrote:
>>>>> On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote:
>>>>>> Hi,
>>>>>>        I am running Ubuntu 10.10 (amd64) on a 2 socket  nehalem based
>>>>>> server with IOH 5520. 5520 supports VTD.
>>>>>> I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter
>>>>>> which I assigned through VT-D to FreeBSD 8.2 running
>>>>>> as guest os. The ixgbe driver detects the device and the driver
>>>>>> successfully configures the device. But the link
>>>>>> never comes up. It looks like link up/down interrupts are not
>>>>>> delivered. Then I checked kvm interrupt assignment and as expected
>>>>>> kvm could not make MSI-X entries for the VT-d guest. So no output from
>>>>>> "grep kvm /proc/interrupt". By enabling some debugs in the
>>>>>> qemu-kvm I figured out that the MSI-x updates are not received
>>>>>> properly. It does look like Linux updates MSI-X table in a batch
>>>>>> fashion
>>>>>> which qemu-kvm gets in  one shot and every thing works fine in case of
>>>>>> linux. In case of FreeBSD PCIE updates come /MSI-X entry
>>>>>> which qemu-kvm can't make use.
>>>>>
>>>>> That's right, Linux and Windows both seem to setup the MSI-X table then
>>>>> enable it in one shot, so we only trigger the interrupt programming when
>>>>> the enable bit is set.  We don't trigger changes on writes to the MSI-X
>>>>> table... not very accurate emulation of mask bits.
>>>>
>>>> According to the PCI spec, updates that happen while a vector is
>>>> unmasked, need not be considered by the hardware (thus the hypervisor
>>>> here). Is that the scenario here?
>>>
>>> I'm assuming the vector is masked in the MSI-X table.  So Linux/Windows
>>> do:
>>>
>>> a) program MSI-X table
>>> b) enable MSI-X in capability register
>>>
>>> Whereas FreeBSD does:
>>>
>>> a) enable MSI-X in capability register (vectors masked in table)
>>> b) program and unmask individual vectors
>>
>> That should work with the current code. It checks the number of vectors
>> on each config write, iterates the whole table, and then updates the
>   ^^^^^^^^^^^^^^^^^^^^
>> kernel configuration accordingly. It even requires the enable bit in the
>> cap register to be set before doing this.
> 
> That's the problem, we only do it on config writes overlapping the MSI-X
> flags.  We don't do anything for writes to the MSI-X table.  It might be
> as simple as calling assigned_dev_update_msix() from msix_mmio_writel()
> when the mask bit is toggled.  I'm not sure what might fall out of that
> though.

Ah indeed. Now I recall to have fixed this in my MSI-X refactoring
series. I introduced config notifiers that are triggered by the MSI-X
layer on every relevant modification, and the device assignment code
hook the update function into this. I really need to dig into that
series soon again and refresh it.

In the meantime, we could try what you suggest (if the cap enable bit is
set).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]