From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kiszka Subject: Re: FreeBSD guest with VTD NIC not passing traffic Date: Fri, 13 Jan 2012 22:33:20 +0100 Message-ID: <4F10A320.1090202@web.de> References: <1325647262.4305.47.camel@bling.home> <4F109B65.8020702@web.de> <1326488745.5945.42.camel@bling.home> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig9B440CBD378FE4620D8BE20D" Cc: Shashidhar Patil , kvm@vger.kernel.org To: Alex Williamson Return-path: Received: from fmmailgate02.web.de ([217.72.192.227]:52342 "EHLO fmmailgate02.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753875Ab2AMVdq (ORCPT ); Fri, 13 Jan 2012 16:33:46 -0500 Received: from moweb001.kundenserver.de (moweb001.kundenserver.de [172.19.20.114]) by fmmailgate02.web.de (Postfix) with ESMTP id 4F39B1BFA0A1E for ; Fri, 13 Jan 2012 22:33:21 +0100 (CET) In-Reply-To: <1326488745.5945.42.camel@bling.home> Sender: kvm-owner@vger.kernel.org List-ID: This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig9B440CBD378FE4620D8BE20D Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 2012-01-13 22:05, Alex Williamson wrote: > On Fri, 2012-01-13 at 22:00 +0100, Jan Kiszka wrote: >> On 2012-01-04 04:21, Alex Williamson wrote: >>> On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote: >>>> Hi, >>>> I am running Ubuntu 10.10 (amd64) on a 2 socket nehalem base= d >>>> server with IOH 5520. 5520 supports VTD. >>>> I enabled DMAR with intel_iommu=3Don. The box has intel 82599 adapte= r >>>> which I assigned through VT-D to FreeBSD 8.2 running >>>> as guest os. The ixgbe driver detects the device and the driver >>>> successfully configures the device. But the link >>>> never comes up. It looks like link up/down interrupts are not >>>> delivered. Then I checked kvm interrupt assignment and as expected >>>> kvm could not make MSI-X entries for the VT-d guest. So no output fr= om >>>> "grep kvm /proc/interrupt". By enabling some debugs in the >>>> qemu-kvm I figured out that the MSI-x updates are not received >>>> properly. It does look like Linux updates MSI-X table in a batch >>>> fashion >>>> which qemu-kvm gets in one shot and every thing works fine in case = of >>>> linux. In case of FreeBSD PCIE updates come /MSI-X entry >>>> which qemu-kvm can't make use. >>> >>> That's right, Linux and Windows both seem to setup the MSI-X table th= en >>> enable it in one shot, so we only trigger the interrupt programming w= hen >>> the enable bit is set. We don't trigger changes on writes to the MSI= -X >>> table... not very accurate emulation of mask bits. >> >> According to the PCI spec, updates that happen while a vector is >> unmasked, need not be considered by the hardware (thus the hypervisor >> here). Is that the scenario here? >=20 > I'm assuming the vector is masked in the MSI-X table. So Linux/Windows= > do: >=20 > a) program MSI-X table > b) enable MSI-X in capability register >=20 > Whereas FreeBSD does: >=20 > a) enable MSI-X in capability register (vectors masked in table) > b) program and unmask individual vectors That should work with the current code. It checks the number of vectors on each config write, iterates the whole table, and then updates the kernel configuration accordingly. It even requires the enable bit in the cap register to be set before doing this. Jan --------------enig9B440CBD378FE4620D8BE20D Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8QoyAACgkQitSsb3rl5xTEQwCg694GXEM/sitqBAj+gwVoqaLu jUwAnj6PBJXRmb30pIvFC1aH8bBUHbG7 =gu46 -----END PGP SIGNATURE----- --------------enig9B440CBD378FE4620D8BE20D--