From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48095) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1elxYz-0006Fh-Ou for qemu-devel@nongnu.org; Wed, 14 Feb 2018 08:57:35 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1elxYw-0002hS-NF for qemu-devel@nongnu.org; Wed, 14 Feb 2018 08:57:33 -0500 Received: from outprodmail02.cc.columbia.edu ([128.59.72.51]:47099) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1elxYw-0002hH-Ii for qemu-devel@nongnu.org; Wed, 14 Feb 2018 08:57:30 -0500 Received: from hazelnut (hazelnut.cc.columbia.edu [128.59.213.250]) by outprodmail02.cc.columbia.edu (8.14.4/8.14.4) with ESMTP id w1EDtbJn041214 for ; Wed, 14 Feb 2018 08:57:29 -0500 Received: from hazelnut (localhost.localdomain [127.0.0.1]) by hazelnut (Postfix) with ESMTP id AA6486D for ; Wed, 14 Feb 2018 08:57:30 -0500 (EST) Received: from sendprodmail04.cc.columbia.edu (sendprodmail04.cc.columbia.edu [128.59.72.16]) by hazelnut (Postfix) with ESMTP id 955017E for ; Wed, 14 Feb 2018 08:57:30 -0500 (EST) Received: from mail-wr0-f200.google.com (mail-wr0-f200.google.com [209.85.128.200]) by sendprodmail04.cc.columbia.edu (8.14.4/8.14.4) with ESMTP id w1EDvSGw047566 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 14 Feb 2018 08:57:29 -0500 Received: by mail-wr0-f200.google.com with SMTP id d17so23063wrc.19 for ; Wed, 14 Feb 2018 05:57:29 -0800 (PST) Received: from mail-wr0-f172.google.com (mail-wr0-f172.google.com. [209.85.128.172]) by smtp.gmail.com with ESMTPSA id y3sm8983139edb.1.2018.02.14.05.57.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Feb 2018 05:57:26 -0800 (PST) Received: by mail-wr0-f172.google.com with SMTP id b52so12660wrd.10 for ; Wed, 14 Feb 2018 05:57:26 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20180214053626.GJ4472@xz-mi> References: <20180214053626.GJ4472@xz-mi> From: Jintack Lim Date: Wed, 14 Feb 2018 08:57:25 -0500 Message-ID: Content-Type: text/plain; charset="UTF-8" Subject: Re: [Qemu-devel] Assigning network devices to nested VMs results in driver errors in nested VMs List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Xu Cc: QEMU Devel Mailing List , vfio-users@redhat.com On Wed, Feb 14, 2018 at 12:36 AM, Peter Xu wrote: > On Tue, Feb 13, 2018 at 11:44:09PM -0500, Jintack Lim wrote: >> Hi, >> >> I'm trying to assign network devices to nested VMs on x86 using KVM, >> but I got network device driver errors in the nested VMs. (I've tried >> this about an year ago when vIOMMU patches were not upstreamed, and I >> got similar errors at that time.) >> >> This could be network driver issues, but I'd like to get some help if >> somebody encountered similar issues. >> >> I'm using v4.15.0 kernel and v2.11.0 QEMU, and I followed this [1] >> guide. I had no problem with assigning devices to the first level VMs >> (L1 VMs). And I also checked that the devices were assigned to nested >> VMs with the lspci command in the nested VMs. But network device >> drivers failed to initialize the device. I tried two network cards - >> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection and >> Mellanox Technologies MT27500 Family. >> >> Intel driver error in the nested VM looks like this. >> [ 1.939552] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - >> version 5.1.0-k >> [ 1.949796] ixgbe: Copyright (c) 1999-2016 Intel Corporation. >> [ 2.210024] ixgbe 0000:00:04.0: HW Init failed: -12 >> [ 2.218144] ixgbe: probe of 0000:00:04.0 failed with error -12 >> >> and I saw lots of these messages in the host (L0) kernel log when >> booting the nested VM. >> >> [ 1557.404173] DMAR: DRHD: handling fault status reg 102 >> [ 1557.409813] DMAR: [DMA Read] Request device [06:00.0] fault addr >> 90000 [fault reason 06] PTE Read access is not set >> [ 1561.383957] DMAR: DRHD: handling fault status reg 202 >> [ 1561.389598] DMAR: [DMA Read] Request device [06:00.0] fault addr >> 90000 [fault reason 06] PTE Read access is not set >> >> This is Mellanox driver error in another nested VM. >> [ 2.481694] mlx4_core: Initializing 0000:00:04.0 >> [ 3.519422] mlx4_core 0000:00:04.0: Installed FW has unsupported >> command interface revision 0 >> [ 3.537769] mlx4_core 0000:00:04.0: (Installed FW version is 0.0.000) >> [ 3.551733] mlx4_core 0000:00:04.0: This driver version supports >> only revisions 2 to 3 >> [ 3.568758] mlx4_core 0000:00:04.0: QUERY_FW command failed, aborting >> [ 3.582789] mlx4_core 0000:00:04.0: Failed to init fw, aborting. >> >> The host showed similar messages as above. >> >> I wonder what could be the cause of these errors. Please let me know >> if further information is needed. >> >> [1] https://wiki.qemu.org/Features/VT-d > > Hi, Jintack, Hi Peter, > > Thanks for reporting the problem. > > I haven't been playing with nested assignment much recently (and even > before), but I think I encountered similar problem too in the past. Oh, that's good to hear that :) > > Will let you know if I had any progress, but it's possibly not gonna > happen in a few days since there'll be a whole week holiday starting > from tomorrow (which is Chinese Spring Festival). Thanks a lot. Enjoy Lunar New Year! > > -- > Peter Xu >