From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:42509) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XDFrx-0007xI-G9 for qemu-devel@nongnu.org; Fri, 01 Aug 2014 12:39:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XDFrp-0001ST-Vl for qemu-devel@nongnu.org; Fri, 01 Aug 2014 12:39:49 -0400 Received: from mout.web.de ([212.227.15.14]:61662) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XDFrp-0001S4-Lm for qemu-devel@nongnu.org; Fri, 01 Aug 2014 12:39:41 -0400 Message-ID: <53DBC2C4.3030801@web.de> Date: Fri, 01 Aug 2014 18:39:32 +0200 From: Jan Kiszka MIME-Version: 1.0 References: <20140730131605.GA25923@redhat.com> <1406731566.2963.53.camel@ul30vt.home> <1406906753.2963.188.camel@ul30vt.home> <1406910924.2963.227.camel@ul30vt.home> In-Reply-To: <1406910924.2963.227.camel@ul30vt.home> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="27DugNG7jvC2VIj1bETl2QxLMx7RLhfLC" Subject: Re: [Qemu-devel] vfio in the guest: no available reset mechanism List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alex Williamson , Le Tan Cc: qemu-devel , "Michael S. Tsirkin" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --27DugNG7jvC2VIj1bETl2QxLMx7RLhfLC Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 2014-08-01 18:35, Alex Williamson wrote: > On Fri, 2014-08-01 at 09:25 -0600, Alex Williamson wrote: >> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote: >>> Hi Alex, >>> >>> 2014-07-30 22:46 GMT+08:00 Alex Williamson : >>>> On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote: >>>>> Hi Michael, >>>>> >>>>> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin : >>>>>> On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote: >>>>>>> Hi, >>>>>>> I am testing vfio in L1 with my VT-d emulation project. I assigne= d one >>>>>>> of the two AHCI controllers in L1 to L2 via vfio. After I ran the= QEMU >>>>>>> in L1, it complains that: >>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no >>>>>>> available reset mechanism. >>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no >>>>>>> available reset mechanism. >>>>>>> >>>>>>> Then L2 paused when the SeaBIOS executed in ahci_controller_setup= (). I >>>>>>> look into this and found that: >>>>>>> val =3D ahci_ctrl_readl(ctrl, HOST_CTL); >>>>>>> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN); >>>>>>> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, = which >>>>>>> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit= >>>>>>> should be cleared by default. So maybe L1 didn't reset the device= >>>>>>> before assigning it to L2? >>>>>>> Then the BIOS tried to write back to HOST_CTL and it was stuck he= re. :( >>>>>>> >>>>>>> So can anyone give me some advice? About the state of PCI device = or >>>>>>> bus-level reset? >>>>>>> >>>>>>> Here is the detail of the environment and the way I did the vfio.= >>>>>>> 1. lspci in L1 said: >>>>>>> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH >>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev= 02) >>>>>>> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC >>>>>>> Interface Controller [8086:2918] (rev 02) >>>>>>> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH >>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev= 02) >>>>>>> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBu= s >>>>>>> Controller [8086:2930] (rev 02) >>>>>>> 2. Unbind 00:03.0 and do vfio: >>>>>>> modprobe -r vfio_iommu_type1 >>>>>>> modprobe vfio_iommu_type1 allow_unsafe_interrupts=3D1 >>>>>>> modprobe vfio-pci >>>>>>> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/un= bind >>>>>>> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id >>>>>>> 3. run L2 with "-device vfio-pci,host=3D00:03.0" >>>>>>> >>>>>>> Any help is appreciated! Thanks very much! >>>>>>> >>>>>>> Regards, >>>>>>> Le >>>>>> >>>>>> Clearly, bus level reset can't work for the root bus :) >>>>> >>>>> Thanks very much! >>>>> I test the vfio with a second-bus ahci controller and it didn't >>>>> complain about the lack of reset mechanism. :) And the return val o= f >>>>> HOST_CTL is normal now (the same as emulated ahci controller). >>>>> However, it still paused when the BIOS tried to write to the HOST_C= TL. >>>>> Do you have any idea? >>>>> And we should just test vfio and legacy pci-assignment with second = bus >>>>> devices, not considering the root-bus devices? >>>> >>>> AHCI seems like a poor choice of device for this work, they typicall= y >>>> don't support any kind of reset and they can be troublesome even for= the >>>> L1 assignment. You really want something with FLR support so that b= oth >>>> the host and L1 guest can potentially reset the device. That said, = you >>>> may still run into bugs with a L1 guest directed FLR. Thanks, >>>> >>>> Alex >>>> >>> >>> So what device do you think is suitable for the pci-assign test? e100= 0? >>> I just tested it with sound card ac97 and USB controller. But I don't= >>> know how to attach them to a pcie-to-pci bridge, so maybe they weren'= t >>> reset before being assigned to L2. But it seems that they can work. >>> 1. With the sound card, I assigned it to L2 via both vfio and legacy >>> pci-assign and I could hear the music played in L2 from host's >>> speakers. Of course, the vfio also complained about the lack of reset= >>> mechanism. >> >> First off, maybe I'm a little confused, are you trying to assigned >> emulated devices for the L1 guest to the L2 guest or are these assigne= d >> devices to the L1 guest that you want to re-assign to L2? AFAIK, we >> don't have any emulated devices that support reset, but it wouldn't be= >> that hard to add a PM capability to one that advertises a soft reset o= n >> D3hot->D0 transition and calls the QEMU driver reset function when tha= t >> occurs. This would provide the most flexibility. >=20 > Or even better might be to add an Advanced Features capability to each > device so it can do an explicit FLR. We don't tend to trust PM reset a= s > being reliably on all devices. But FLR is PCIe-only, isn't it? Also, it may let some of our device models deviate from their real versions (I suppose, e.g., none of the e1000 devices we currently emulate exposed FLR). In any case, I suggested Le to focus on his GSoC task again (VT-d emulation) and postpone this topic to later point (likely outside GSoC). Only few weeks left... Jan --27DugNG7jvC2VIj1bETl2QxLMx7RLhfLC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlPbwsgACgkQitSsb3rl5xT6awCgqMQdW4YulQML5kMI68jFugal g7wAn3hDBjS7YBLe0Kb+8oAcBVc3ToJ7 =/Wht -----END PGP SIGNATURE----- --27DugNG7jvC2VIj1bETl2QxLMx7RLhfLC--