Re: USB Passthrough 1.1 performance problem...

[Erik Brakkee <erik@brakkee.org>]

Erik Brakkee wrote:
> Erik Brakkee wrote:
>> <quote who="Kenni Lund">
>>> 2010/12/14 Erik Brakkee<erik@brakkee.org>:
>>>> Daniel P. Berrange wrote:
>>>>> On Tue, Dec 14, 2010 at 12:55:04PM +0100, Kenni Lund wrote:
>>>>>
>>>>>> 2010/12/14 Erik Brakkee<erik@brakkee.org>:
>>>>>>
>>>>>>>> From: Kenni Lund<kenni@kelu.dk>
>>>>>>>> 2010/12/14 Erik Brakkee<erik@brakkee.org>:
>>>>>>>>
>>>>>>>>>> From: Kenni Lund<kenni@kelu.dk>
>>>>>>>>>>
>>>>>>>>>>> Does this mean I have a chance now that PCI passthrough of my
>>>>>>>>>>> WinTV
>>>>>>>>>>> PVR-500
>>>>>>>>>>> might work now?
>>>>>>>>>>>
>>>>>>>>>> Passthrough of a PVR-500 has been working for a long time. I've
>>>>>>>>>> been
>>>>>>>>>> running with passthrough of a PVR-500 in my HTPC, since
>>>>>>>>>> November/December 2009...so it should work with any recent 
>>>>>>>>>> kernel
>>>>>>>>>> and
>>>>>>>>>> any recent version of qemu-kvm you can find today - No patching
>>>>>>>>>> needed. The only issue I had with the PVR-500 card, was when *I*
>>>>>>>>>> didn't free up the shared interrupts...once I fixed that, it 
>>>>>>>>>> "just
>>>>>>>>>> worked".
>>>>>>>>>>
>>>>>>>>> How did you free up those shared interrupts then? I tried 
>>>>>>>>> different
>>>>>>>>> slots
>>>>>>>>> but always get conflicts with the USB irqs.
>>>>>>>>>
>>>>>>>> I did an unbind of the conflicting device (eg. disabled it). I 
>>>>>>>> moved
>>>>>>>> the PVR-500 card around in the different slots and once I got a
>>>>>>>> conflict with the integrated sound card, I left the PVR-500 
>>>>>>>> card in
>>>>>>>> that slot (it's a headless machine, so no need for sound) and
>>>>>>>> configured unbind of the sound card at boot time. On my old 
>>>>>>>> system I
>>>>>>>> think it was conflicting with one of the USB controllers as well,
>>>>>>>> but
>>>>>>>> it didn't really matter, as I only lost a few of the ports on the
>>>>>>>> back
>>>>>>>> of the computer for that particular USB controller - I still had
>>>>>>>> plenty of USB ports left and if I really needed more ports, I 
>>>>>>>> could
>>>>>>>> just plug in an extra USB PCI card.
>>>>>>>>
>>>>>>>> My /etc/rc.local boot script looks like the following today:
>>>>>>>> -- 
>>>>>>>> #Remove HDA conflicting with ivtv1
>>>>>>>> echo "0000:00:1b.0">   /sys/bus/pci/drivers/HDA\ Intel/unbind
>>>>>>>>
>>>>>>>> # ivtv0
>>>>>>>> echo "4444 0016">   /sys/bus/pci/drivers/pci-stub/new_id
>>>>>>>> echo "0000:04:08.0">   /sys/bus/pci/drivers/ivtv/unbind
>>>>>>>> echo "0000:04:08.0">   /sys/bus/pci/drivers/pci-stub/bind
>>>>>>>> echo "4444 0016">   /sys/bus/pci/drivers/pci-stub/remove_id
>>>>>>>>
>>>>>>>> # ivtv1
>>>>>>>> echo "4444 0016">   /sys/bus/pci/drivers/pci-stub/new_id
>>>>>>>> echo "0000:04:09.0">   /sys/bus/pci/drivers/ivtv/unbind
>>>>>>>> echo "0000:04:09.0">   /sys/bus/pci/drivers/pci-stub/bind
>>>>>>>> echo "4444 0016">   /sys/bus/pci/drivers/pci-stub/remove_id
>>>>>>>>
>>>>>>> I did not try unbinding the usb device so I can also try that.
>>>>>>>
>>>>>>> I don'.t understand what is happening with the 4444 0016. I
>>>>>>> configured
>>>>>>> the
>>>>>>> pci card in kvm and I believe kvm does the binding to pci-stub in
>>>>>>> recent
>>>>>>> versions. Where is the 4444 0016%oming from?
>>>>>>>
>>>>>> Okay, qemu-kvm might do it today, I don't know - I haven't changed
>>>>>> that script for the past year. But are you sure that it's not
>>>>>> libvirt/virsh/virt-manager which does that for you?
>>>>>>
>>>>> If you use the managed="yes" attribute on the<hostdev>   in libvirt
>>>>> XML, then libvirt will automatically do the pcistub bind/unbind,
>>>>> followed by a device reset at guest startup&   the reverse at 
>>>>> shutdown.
>>>>> If you have conflicting devices on the bus though, libvirt won't
>>>>> attempt to unbind them, unless you had also explicitly assigned all
>>>>> those conflicting devices to the same guest.
>>>>>
>>>>> Daniel
>>>>>
>>>> I definitely have to try again (right now having some stability 
>>>> problems
>>>> on
>>>> the server that I am debugging).
>>>>
>>>> The shared IRQs are as follows:
>>>>
>>>>   16:          0          0          0          0          
>>>> 0          0
>>>>     0          0   IO-APIC-fasteoi   uhci_hcd:usb3
>>>>   18:     252995          0          0          0          
>>>> 0          0
>>>>     0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb8, 
>>>> ivtv0
>>>>   19:      58281          0          0          0          
>>>> 0          0
>>>>     0          0   IO-APIC-fasteoi   ata_piix, ata_piix, 
>>>> uhci_hcd:usb5,
>>>> uhci_hcd:usb7, ivtv1
>>>>   21:          0          0          0          0          
>>>> 0          0
>>>>     0          0   IO-APIC-fasteoi   uhci_hcd:usb4
>>>>   23:        713       6906          0      76919          
>>>> 0          0
>>>>     0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb6
>>>>
>>>> So I have IRQ sharing with usb1, usb8, usb5, usb7.
>>> Uff....and your ata HDD controller. I guess i was much luckier than
>>> you are, my ivtv0 didn't conflict at all and ivtv1 only conflicted
>>> with USB.
>>>
>>>> I have also read that
>>>> ehci refers to USB 2.0 and uhci to USB 1.1 is that correct? Anyway, 
>>>> how
>>>> would I now identify the USB PCI devices that I would need to 
>>>> unbind to
>>>> get
>>>> rid of the sharing with the USB ports?
>>> Play around with:
>>> lspci -v
>>> lspci -n
>>> lsusb -v
>>> lsusb -t
>>>
>>> You can also just start by unbinding the first one and take note when
>>> you hit the right ones...once you unbind one, it will disappear from
>>> cat /proc/interrupts. When you're down to having only ivtv0 on one
>>> interrupt and only ivtv1 on another interrupt, then you're ready to
>>> bind with pci-stub and boot your guest.
>>>
>>>> It also doesn't really matter in
>>>> which slot I put the PVR-500 card because both cards share IRQs 
>>>> with USB
>>>> in
>>>> all cases.
>>> You also have your conflicting ata controller to take into
>>> consideration. I don't remember if "ata_piix" and "ata_piix" is IDE
>>> only, if it is, you might not even use it. Otherwise it might be
>>> easier for you to run qemu-kvm git with the new patches for shared
>>> interrupts...it really depends on your setup.
>>>
>>>> I have also used an add on USB PCI card but still got these 
>>>> conflicts. I
>>>> was
>>>> considering to get a PCIe USB card instead to try out in the hope that
>>>> that
>>>> would use different IRQs. Is that a realistic expectation? That way, I
>>>> could
>>>> disable all on-board USB (in the BIOS even) and use the add-on USB 
>>>> only.
>>> Most likely, the PCIe USB 3.0 card I bought supported MSI-X, so it got
>>> its own unique IRQs which wasn't shared with anything.
>>>
>> I think I am a bit out of luck here. The two IRQs that are 
>> interesting are
>> 18 and 19. One of them has the SMBUS and the other one has SATA 
>> drives. I
>> think the best for me is to wait until shared IRQ support in KVM is
>> available. My only option is to try it out every once in a while.
>>
>>
>>
> Meanwhile, I haven't given up and experimented with determining the 
> effect of unbinding the various USB PCI devices. I haven't tried 
> virtualization yet though. The problem is I managed to free up one 
> tuner (ivtv0) from shared IRQs, all USB ports are still operational, 
> but then how would I determine which USB ports still support USB 2.0?
>
> I have now at least found a way to make ivtv0 available without any 
> shared IRQ by unbinding two USB PCI devices: one UHCI and one EHCI 
> device.
>
> The list of USB PCI devices is:
>
> 00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
> UHCI Controller #4
> 00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
> UHCI Controller #5
> 00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
> UHCI Controller #6
> 00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 
> EHCI Controller #2
> 00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
> UHCI Controller #1
> 00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
> UHCI Controller #2
> 00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
> UHCI Controller #3
> 00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 
> EHCI Controller #1
>
> I unbound  00:1a.7 and 00:1d.2. To my great surprise, all USB ports on 
> the server are still operational. I tried this by inserting a USB 
> memory stick and trying to mount it. The motherboard has a total of 8 
> USB connections with 6 USB in use, I don't really understand this. Was 
> I simply in luck?
>
> The output of lsucb after doing the unbinding is:
>
> Bus 005 Device 003: ID 046b:ff10 American Megatrends, Inc.
> Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
> Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
> Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
> Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
> Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
> Bus 006 Device 002: ID 045e:001e Microsoft Corp. IntelliMouse Explorer
> Bus 006 Device 003: ID 051d:0002 American Power Conversion 
> Uninterruptible Power Supply
> Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
>
> Alternatively, I am also considering to by a new PCIe card. In 
> particular, the ASUS U3S6 looks interesting because it has 2 PCIe 
> ports and 2 USB 3.0 ports. This means that I could disable 2 more USB 
> devices (00:1a.2 and 00:1d.1) and get some extra USB devices. The 
> ata_piix drivers are being used for an internal disk and CDROM. The 
> RAID array is managed through arcmsr (Areca) so I don't have a problem 
> there. Of course, if I managed to get completely free of shared IRQs 
> with this add-on card, then the issue of determining which of the 
> remaining ports is still USB 2.0 is not that important anymore.
>
> So, enough ideas to try out.
>
In the mean while I figured it out completely. I disabled the 4 USB PCI 
devices as mentioned above which leads to one USB port becoming unusable 
(acceptable). Then I could configure the SATA 'handling' in the BIOS. 
The default was plain SATA, but I could also select Intel RAID, Adaptec 
RAID, and AHCI. In any case Intel RAID and AHCI solved the problem with 
the ata_piix shared IRQ (did not check), so I used the most modern one 
(Intel RAID). (motherboard Supermicro X8DTi-F).

Meanwhile I did a successful PCI passthrough test recording two channels 
simultanesously and it worked!

PS. I used to do backups to a backup SATA disk over USB but since I 
turned on the Intel RAID, the SATA ports on the motherboard became hot 
swappable so I can now backup over (e)SATA which gives me a 5 times 
speedup in bandwidth and frees up one USB port, compensating for the one 
I lost.

So all is well and I can continue my virtualized setup according to my 
original plans.

Cheers
   Erik