From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kiszka Subject: Re: FreeBSD guest with VTD NIC not passing traffic Date: Fri, 13 Jan 2012 23:15:19 +0100 Message-ID: <4F10ACF7.2000900@web.de> References: <1325647262.4305.47.camel@bling.home> <4F109B65.8020702@web.de> <1326488745.5945.42.camel@bling.home> <4F10A320.1090202@web.de> <1326491780.5945.57.camel@bling.home> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigBB8928EC09384DA3BBFF84CC" Cc: Shashidhar Patil , kvm@vger.kernel.org To: Alex Williamson Return-path: Received: from fmmailgate02.web.de ([217.72.192.227]:38303 "EHLO fmmailgate02.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756018Ab2AMWPV (ORCPT ); Fri, 13 Jan 2012 17:15:21 -0500 Received: from moweb002.kundenserver.de (moweb002.kundenserver.de [172.19.20.108]) by fmmailgate02.web.de (Postfix) with ESMTP id 584FC1BFA176F for ; Fri, 13 Jan 2012 23:15:20 +0100 (CET) In-Reply-To: <1326491780.5945.57.camel@bling.home> Sender: kvm-owner@vger.kernel.org List-ID: This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBB8928EC09384DA3BBFF84CC Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 2012-01-13 22:56, Alex Williamson wrote: > On Fri, 2012-01-13 at 22:33 +0100, Jan Kiszka wrote: >> On 2012-01-13 22:05, Alex Williamson wrote: >>> On Fri, 2012-01-13 at 22:00 +0100, Jan Kiszka wrote: >>>> On 2012-01-04 04:21, Alex Williamson wrote: >>>>> On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote: >>>>>> Hi, >>>>>> I am running Ubuntu 10.10 (amd64) on a 2 socket nehalem ba= sed >>>>>> server with IOH 5520. 5520 supports VTD. >>>>>> I enabled DMAR with intel_iommu=3Don. The box has intel 82599 adap= ter >>>>>> which I assigned through VT-D to FreeBSD 8.2 running >>>>>> as guest os. The ixgbe driver detects the device and the driver >>>>>> successfully configures the device. But the link >>>>>> never comes up. It looks like link up/down interrupts are not >>>>>> delivered. Then I checked kvm interrupt assignment and as expected= >>>>>> kvm could not make MSI-X entries for the VT-d guest. So no output = from >>>>>> "grep kvm /proc/interrupt". By enabling some debugs in the >>>>>> qemu-kvm I figured out that the MSI-x updates are not received >>>>>> properly. It does look like Linux updates MSI-X table in a batch >>>>>> fashion >>>>>> which qemu-kvm gets in one shot and every thing works fine in cas= e of >>>>>> linux. In case of FreeBSD PCIE updates come /MSI-X entry >>>>>> which qemu-kvm can't make use. >>>>> >>>>> That's right, Linux and Windows both seem to setup the MSI-X table = then >>>>> enable it in one shot, so we only trigger the interrupt programming= when >>>>> the enable bit is set. We don't trigger changes on writes to the M= SI-X >>>>> table... not very accurate emulation of mask bits. >>>> >>>> According to the PCI spec, updates that happen while a vector is >>>> unmasked, need not be considered by the hardware (thus the hyperviso= r >>>> here). Is that the scenario here? >>> >>> I'm assuming the vector is masked in the MSI-X table. So Linux/Windo= ws >>> do: >>> >>> a) program MSI-X table >>> b) enable MSI-X in capability register >>> >>> Whereas FreeBSD does: >>> >>> a) enable MSI-X in capability register (vectors masked in table) >>> b) program and unmask individual vectors >> >> That should work with the current code. It checks the number of vector= s >> on each config write, iterates the whole table, and then updates the > ^^^^^^^^^^^^^^^^^^^^ >> kernel configuration accordingly. It even requires the enable bit in t= he >> cap register to be set before doing this. >=20 > That's the problem, we only do it on config writes overlapping the MSI-= X > flags. We don't do anything for writes to the MSI-X table. It might b= e > as simple as calling assigned_dev_update_msix() from msix_mmio_writel()= > when the mask bit is toggled. I'm not sure what might fall out of that= > though. Ah indeed. Now I recall to have fixed this in my MSI-X refactoring series. I introduced config notifiers that are triggered by the MSI-X layer on every relevant modification, and the device assignment code hook the update function into this. I really need to dig into that series soon again and refresh it. In the meantime, we could try what you suggest (if the cap enable bit is set). Jan --------------enigBB8928EC09384DA3BBFF84CC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8QrPcACgkQitSsb3rl5xRjpwCglUSZmYc0Kq14c3SR6AIa40MK eK0An2x7Hjudwe8nEyrszN92Amqp8hTQ =6V6F -----END PGP SIGNATURE----- --------------enigBB8928EC09384DA3BBFF84CC--