Re: [Qemu-devel] vfio in the guest: no available reset mechanism

[Jan Kiszka <jan.kiszka@web.de>]

[-- Attachment #1: Type: text/plain, Size: 5295 bytes --]

On 2014-08-01 18:35, Alex Williamson wrote:
> On Fri, 2014-08-01 at 09:25 -0600, Alex Williamson wrote:
>> On Fri, 2014-08-01 at 09:35 +0800, Le Tan wrote:
>>> Hi Alex,
>>>
>>> 2014-07-30 22:46 GMT+08:00 Alex Williamson <alex.williamson@redhat.com>:
>>>> On Wed, 2014-07-30 at 22:16 +0800, Le Tan wrote:
>>>>> Hi Michael,
>>>>>
>>>>> 2014-07-30 21:16 GMT+08:00 Michael S. Tsirkin <mst@redhat.com>:
>>>>>> On Wed, Jul 30, 2014 at 08:24:04PM +0800, Le Tan wrote:
>>>>>>> Hi,
>>>>>>> I am testing vfio in L1 with my VT-d emulation project. I assigned one
>>>>>>> of the two AHCI controllers in L1 to L2 via vfio. After I ran the QEMU
>>>>>>> in L1, it complains that:
>>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>>>>>>> available reset mechanism.
>>>>>>> qemu-system-x86_64: vfio: Cannot reset device 0000:00:03.0, no
>>>>>>> available reset mechanism.
>>>>>>>
>>>>>>> Then L2 paused when the SeaBIOS executed in ahci_controller_setup(). I
>>>>>>> look into this and found that:
>>>>>>> val = ahci_ctrl_readl(ctrl, HOST_CTL);
>>>>>>> ahci_ctrl_writel(ctrl, HOST_CTL, val | HOST_CTL_AHCI_EN);
>>>>>>> When the BIOS tried to read the HOST_CTL, it returns 0x80000002, which
>>>>>>> bit 2 (Interrupt Enable) is 1. The AHCI manual says that this bit
>>>>>>> should be cleared by default. So maybe L1 didn't reset the device
>>>>>>> before assigning it to L2?
>>>>>>> Then the BIOS tried to write back to HOST_CTL and it was stuck here. :(
>>>>>>>
>>>>>>> So can anyone give me some advice? About the state of PCI device or
>>>>>>> bus-level reset?
>>>>>>>
>>>>>>> Here is the detail of the environment and the way I did the vfio.
>>>>>>> 1. lspci in L1 said:
>>>>>>> 00:03.0 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>>>>>>> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC
>>>>>>> Interface Controller [8086:2918] (rev 02)
>>>>>>> 00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH
>>>>>>> (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
>>>>>>> 00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus
>>>>>>> Controller [8086:2930] (rev 02)
>>>>>>> 2. Unbind 00:03.0 and do vfio:
>>>>>>> modprobe -r vfio_iommu_type1
>>>>>>> modprobe vfio_iommu_type1 allow_unsafe_interrupts=1
>>>>>>> modprobe vfio-pci
>>>>>>> echo 0000:00:03.0 > /sys/bus/pci/devices/0000\:00\:03.0/driver/unbind
>>>>>>> echo "8086 2922" > /sys/bus/pci/drivers/vfio-pci/new_id
>>>>>>> 3. run L2 with "-device vfio-pci,host=00:03.0"
>>>>>>>
>>>>>>> Any help is appreciated! Thanks very much!
>>>>>>>
>>>>>>> Regards,
>>>>>>> Le
>>>>>>
>>>>>> Clearly, bus level reset can't work for the root bus :)
>>>>>
>>>>> Thanks very much!
>>>>> I test the vfio with a second-bus ahci controller and it didn't
>>>>> complain about the lack of reset mechanism. :) And the return val of
>>>>> HOST_CTL is normal now (the same as emulated ahci controller).
>>>>> However, it still paused when the BIOS tried to write to the HOST_CTL.
>>>>> Do you have any idea?
>>>>> And we should just test vfio and legacy pci-assignment with second bus
>>>>> devices, not considering the root-bus devices?
>>>>
>>>> AHCI seems like a poor choice of device for this work, they typically
>>>> don't support any kind of reset and they can be troublesome even for the
>>>> L1 assignment.  You really want something with FLR support so that both
>>>> the host and L1 guest can potentially reset the device.  That said, you
>>>> may still run into bugs with a L1 guest directed FLR.  Thanks,
>>>>
>>>> Alex
>>>>
>>>
>>> So what device do you think is suitable for the pci-assign test? e1000?
>>> I just tested it with sound card ac97 and USB controller. But I don't
>>> know how to attach them to a pcie-to-pci bridge, so maybe they weren't
>>> reset before being assigned to L2. But it seems that they can work.
>>> 1. With the sound card, I assigned it to L2 via both vfio and legacy
>>> pci-assign and I could hear the music played in L2 from host's
>>> speakers. Of course, the vfio also complained about the lack of reset
>>> mechanism.
>>
>> First off, maybe I'm a little confused, are you trying to assigned
>> emulated devices for the L1 guest to the L2 guest or are these assigned
>> devices to the L1 guest that you want to re-assign to L2?  AFAIK, we
>> don't have any emulated devices that support reset, but it wouldn't be
>> that hard to add a PM capability to one that advertises a soft reset on
>> D3hot->D0 transition and calls the QEMU driver reset function when that
>> occurs.  This would provide the most flexibility.
> 
> Or even better might be to add an Advanced Features capability to each
> device so it can do an explicit FLR.  We don't tend to trust PM reset as
> being reliably on all devices.

But FLR is PCIe-only, isn't it? Also, it may let some of our device
models deviate from their real versions (I suppose, e.g., none of the
e1000 devices we currently emulate exposed FLR).

In any case, I suggested Le to focus on his GSoC task again (VT-d
emulation) and postpone this topic to later point (likely outside GSoC).
Only few weeks left...

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]