From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1131C33C8C for ; Mon, 6 Jan 2020 14:04:57 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 75926207FD for ; Mon, 6 Jan 2020 14:04:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="eNJ5nW6k" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 75926207FD Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ioSzi-0000mF-13; Mon, 06 Jan 2020 14:04:34 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ioSzg-0000mA-TT for xen-devel@lists.xenproject.org; Mon, 06 Jan 2020 14:04:33 +0000 X-Inumbo-ID: 71320300-308d-11ea-88e7-bc764e2007e4 Received: from wout2-smtp.messagingengine.com (unknown [64.147.123.25]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 71320300-308d-11ea-88e7-bc764e2007e4; Mon, 06 Jan 2020 14:04:24 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id F1ADC74E; Mon, 6 Jan 2020 09:04:22 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Mon, 06 Jan 2020 09:04:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=snV/PI Er4cnpaqIEpiZU2KzSbWgTDlfLuyvlbX4dIjQ=; b=eNJ5nW6kRktYepwYYtmrHU Xbq8iZJgNJs1EGZd86WItwmpNSVyWIU+wIWmOu3cn6vXwx+IK9+ml2Of9+i9mYUL 2RIQGgwoCfYhFiQVCuUvr0f0CP8YCStUQ8swKASXRCcoTTVuXuJRtLQNAqJeE4Wc VEZMCzr1WL8E50+w0Pau3MXHWEQ21CJXXFMuG0JCtr5BRMYvnjyZabYpEJcneM9j uoNsXAS79XGUl82PElYWG4sfaUg3m/HWxyKHpKMukqEPvoZ+R7zw91D6+fo+/Lbl 9anyMGYK+C9XHTdgYE2mHxEjGtgYAL4EYJGA7fwtwM210LNQlMmMyd9fvS84Dt0w == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrvdehtddgieduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfhfgggtuggjsehgtderredttdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucfkphepledurdeihedrfeegrdef feenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhvihhsihgslh gvthhhihhnghhslhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from mail-itl (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id BE5AF80059; Mon, 6 Jan 2020 09:04:21 -0500 (EST) Date: Mon, 6 Jan 2020 15:04:18 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Jan Beulich Message-ID: <20200106140418.GH1314@mail-itl> References: <20200104010759.GA2507@mail-itl> MIME-Version: 1.0 In-Reply-To: Subject: Re: [Xen-devel] Broken PCI device passthrough, after XSA-302 fix? X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: xen-devel Content-Type: multipart/mixed; boundary="===============7619410958256783281==" Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" --===============7619410958256783281== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="CGDBiGfvSTbxKZlW" Content-Disposition: inline --CGDBiGfvSTbxKZlW Content-Type: text/plain; protected-headers=v1; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Subject: Re: Broken PCI device passthrough, after XSA-302 fix? On Mon, Jan 06, 2020 at 12:18:31PM +0100, Jan Beulich wrote: > On 04.01.2020 02:07, Marek Marczykowski-G=C3=B3recki wrote: > > I have a multi-function PCI device, behind a PCI bridge, that normally > > I assign to a single domain. But now it fails with: > >=20 > > (XEN) [VT-D]d14: 0000:04:00.0 owned by d0!<0>assign 0000:05:00.0 to = dom14 failed (-22) >=20 > Is this on the 1st attempt, or after the device had already been > assigned to some (same or other) guest? After quite a bit of > staring at the code I can't seem to be able to spot a difference > in behavior for the 1st attempt, but you not saying explicitly > that it would only happen on subsequent ones makes me assume you > run into the issue right away. Yes, it was the first try. > > This is Xen 4.8.5 + XSA patches. It started happening after some update > > during last few months, not really sure which one. >=20 > Having a smaller window would of course help, as would ... The working version was just before XSAs of 2019-10-31 (which include XSA-302). But at this point, I'm not sure if no other configuration changes were made (see below). > > I guess it is because quarantine feature, so initial ownership of > > 0000:05:00.0 is different than the bridge it is connected to. > > I'm not sure if relevant for this case, but I also set > > pcidev->rdm_policy =3D LIBXL_RDM_RESERVE_POLICY_RELAXED. > >=20 > > Booting with iommu=3Dno-quarantine helps. Note I do not use `xl > > pci-assignable-add` command, only bind the device to the pciback driver > > in dom0. >=20 > ... knowing whether behavior differs when using this preparatory > step. xl pci-assignable-add doesn't make a difference with XSA-306 applied. But I've tried xl pci-assignable-remove with interesting result: It succeeded for 0000:05:00.0 and 0000:05:00.2, but failed for 0000:05:00.1 with this message: (XEN) [VT-D]d0: 0000:05:00.1 owned by d32753!<0>deassign 0000:05:00.1 =66rom dom32753 failed (-22) Anyway, I think my previous testing was inaccurate: Looks like the issue is caused by me failing to set rdm_policy, contrary to the above message. I get the above error only without LIBXL_RDM_RESERVE_POLICY_RELAXED set. When I set it properly, domain starts even without iommu=3Dno-quarantine. I still have some issues with the device within the domain, but not sure if relevant to this or something else. Does it make sense now? Is the patch from your other message still relevant? --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? --CGDBiGfvSTbxKZlW Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl4TPmIACgkQ24/THMrX 1yyCqgf/SbsRmDqrE7IdVlXvU3xudxiWn1oigXa1EYb9GaIczz5yq84Nd/iU53wK jxE0hTcpw0AhNFZivge0U0GeZ6IY18teQRYsunlbby4ZStDh6eRFkh1MIyhSLQTI jboBF18Jkp3zBG1PGFNcEnnpENVfMRfw56LreBUKvOU7txziWt6Q+79QJI66f+Vi Pls1vg4CUYKN03ytKyQw3oAen6E1UCZ7zy6EEBdsqVS4JHqfSD9j3UHOGfDJH+iY DQZf8D4/j0baexP2n1VaafLSd0wo1YoyuFhMSmiBst+Gsq7FR7YNduZm+YjNzfma 7BTjkgJHBrwnup9ivVdDptCRNcC2+g== =1rVE -----END PGP SIGNATURE----- --CGDBiGfvSTbxKZlW-- --===============7619410958256783281== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVucHJvamVjdC5vcmcKaHR0cHM6Ly9saXN0 cy54ZW5wcm9qZWN0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3hlbi1kZXZlbA== --===============7619410958256783281==--