Re: USB Passthrough 1.1 performance problem...

[Erik Brakkee <erik@brakkee.org>]

Erik Brakkee wrote:
> <quote who="Kenni Lund">
>    
>> 2010/12/14 Erik Brakkee<erik@brakkee.org>:
>>      
>>> Daniel P. Berrange wrote:
>>>        
>>>> On Tue, Dec 14, 2010 at 12:55:04PM +0100, Kenni Lund wrote:
>>>>
>>>>          
>>>>> 2010/12/14 Erik Brakkee<erik@brakkee.org>:
>>>>>
>>>>>            
>>>>>>> From: Kenni Lund<kenni@kelu.dk>
>>>>>>> 2010/12/14 Erik Brakkee<erik@brakkee.org>:
>>>>>>>
>>>>>>>                
>>>>>>>>> From: Kenni Lund<kenni@kelu.dk>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>>>> Does this mean I have a chance now that PCI passthrough of my
>>>>>>>>>> WinTV
>>>>>>>>>> PVR-500
>>>>>>>>>> might work now?
>>>>>>>>>>
>>>>>>>>>>                      
>>>>>>>>> Passthrough of a PVR-500 has been working for a long time. I've
>>>>>>>>> been
>>>>>>>>> running with passthrough of a PVR-500 in my HTPC, since
>>>>>>>>> November/December 2009...so it should work with any recent kernel
>>>>>>>>> and
>>>>>>>>> any recent version of qemu-kvm you can find today - No patching
>>>>>>>>> needed. The only issue I had with the PVR-500 card, was when *I*
>>>>>>>>> didn't free up the shared interrupts...once I fixed that, it "just
>>>>>>>>> worked".
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>> How did you free up those shared interrupts then? I tried different
>>>>>>>> slots
>>>>>>>> but always get conflicts with the USB irqs.
>>>>>>>>
>>>>>>>>                  
>>>>>>> I did an unbind of the conflicting device (eg. disabled it). I moved
>>>>>>> the PVR-500 card around in the different slots and once I got a
>>>>>>> conflict with the integrated sound card, I left the PVR-500 card in
>>>>>>> that slot (it's a headless machine, so no need for sound) and
>>>>>>> configured unbind of the sound card at boot time. On my old system I
>>>>>>> think it was conflicting with one of the USB controllers as well,
>>>>>>> but
>>>>>>> it didn't really matter, as I only lost a few of the ports on the
>>>>>>> back
>>>>>>> of the computer for that particular USB controller - I still had
>>>>>>> plenty of USB ports left and if I really needed more ports, I could
>>>>>>> just plug in an extra USB PCI card.
>>>>>>>
>>>>>>> My /etc/rc.local boot script looks like the following today:
>>>>>>> --
>>>>>>> #Remove HDA conflicting with ivtv1
>>>>>>> echo "0000:00:1b.0">   /sys/bus/pci/drivers/HDA\ Intel/unbind
>>>>>>>
>>>>>>> # ivtv0
>>>>>>> echo "4444 0016">   /sys/bus/pci/drivers/pci-stub/new_id
>>>>>>> echo "0000:04:08.0">   /sys/bus/pci/drivers/ivtv/unbind
>>>>>>> echo "0000:04:08.0">   /sys/bus/pci/drivers/pci-stub/bind
>>>>>>> echo "4444 0016">   /sys/bus/pci/drivers/pci-stub/remove_id
>>>>>>>
>>>>>>> # ivtv1
>>>>>>> echo "4444 0016">   /sys/bus/pci/drivers/pci-stub/new_id
>>>>>>> echo "0000:04:09.0">   /sys/bus/pci/drivers/ivtv/unbind
>>>>>>> echo "0000:04:09.0">   /sys/bus/pci/drivers/pci-stub/bind
>>>>>>> echo "4444 0016">   /sys/bus/pci/drivers/pci-stub/remove_id
>>>>>>>
>>>>>>>                
>>>>>> I did not try unbinding the usb device so I can also try that.
>>>>>>
>>>>>> I don'.t understand what is happening with the 4444 0016. I
>>>>>> configured
>>>>>> the
>>>>>> pci card in kvm and I believe kvm does the binding to pci-stub in
>>>>>> recent
>>>>>> versions. Where is the 4444 0016%oming from?
>>>>>>
>>>>>>              
>>>>> Okay, qemu-kvm might do it today, I don't know - I haven't changed
>>>>> that script for the past year. But are you sure that it's not
>>>>> libvirt/virsh/virt-manager which does that for you?
>>>>>
>>>>>            
>>>> If you use the managed="yes" attribute on the<hostdev>   in libvirt
>>>> XML, then libvirt will automatically do the pcistub bind/unbind,
>>>> followed by a device reset at guest startup&   the reverse at shutdown.
>>>> If you have conflicting devices on the bus though, libvirt won't
>>>> attempt to unbind them, unless you had also explicitly assigned all
>>>> those conflicting devices to the same guest.
>>>>
>>>> Daniel
>>>>
>>>>          
>>> I definitely have to try again (right now having some stability problems
>>> on
>>> the server that I am debugging).
>>>
>>> The shared IRQs are as follows:
>>>
>>>   16:          0          0          0          0          0          0
>>>     0          0   IO-APIC-fasteoi   uhci_hcd:usb3
>>>   18:     252995          0          0          0          0          0
>>>     0          0   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb8, ivtv0
>>>   19:      58281          0          0          0          0          0
>>>     0          0   IO-APIC-fasteoi   ata_piix, ata_piix, uhci_hcd:usb5,
>>> uhci_hcd:usb7, ivtv1
>>>   21:          0          0          0          0          0          0
>>>     0          0   IO-APIC-fasteoi   uhci_hcd:usb4
>>>   23:        713       6906          0      76919          0          0
>>>     0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb6
>>>
>>> So I have IRQ sharing with usb1, usb8, usb5, usb7.
>>>        
>> Uff....and your ata HDD controller. I guess i was much luckier than
>> you are, my ivtv0 didn't conflict at all and ivtv1 only conflicted
>> with USB.
>>
>>      
>>> I have also read that
>>> ehci refers to USB 2.0 and uhci to USB 1.1 is that correct? Anyway, how
>>> would I now identify the USB PCI devices that I would need to unbind to
>>> get
>>> rid of the sharing with the USB ports?
>>>        
>> Play around with:
>> lspci -v
>> lspci -n
>> lsusb -v
>> lsusb -t
>>
>> You can also just start by unbinding the first one and take note when
>> you hit the right ones...once you unbind one, it will disappear from
>> cat /proc/interrupts. When you're down to having only ivtv0 on one
>> interrupt and only ivtv1 on another interrupt, then you're ready to
>> bind with pci-stub and boot your guest.
>>
>>      
>>> It also doesn't really matter in
>>> which slot I put the PVR-500 card because both cards share IRQs with USB
>>> in
>>> all cases.
>>>        
>> You also have your conflicting ata controller to take into
>> consideration. I don't remember if "ata_piix" and "ata_piix" is IDE
>> only, if it is, you might not even use it. Otherwise it might be
>> easier for you to run qemu-kvm git with the new patches for shared
>> interrupts...it really depends on your setup.
>>
>>      
>>> I have also used an add on USB PCI card but still got these conflicts. I
>>> was
>>> considering to get a PCIe USB card instead to try out in the hope that
>>> that
>>> would use different IRQs. Is that a realistic expectation? That way, I
>>> could
>>> disable all on-board USB (in the BIOS even) and use the add-on USB only.
>>>        
>> Most likely, the PCIe USB 3.0 card I bought supported MSI-X, so it got
>> its own unique IRQs which wasn't shared with anything.
>>
>>      
> I think I am a bit out of luck here. The two IRQs that are interesting are
> 18 and 19. One of them has the SMBUS and the other one has SATA drives. I
> think the best for me is to wait until shared IRQ support in KVM is
> available. My only option is to try it out every once in a while.
>
>
>
>    
Meanwhile, I haven't given up and experimented with determining the 
effect of unbinding the various USB PCI devices. I haven't tried 
virtualization yet though. The problem is I managed to free up one tuner 
(ivtv0) from shared IRQs, all USB ports are still operational, but then 
how would I determine which USB ports still support USB 2.0?

I have now at least found a way to make ivtv0 available without any 
shared IRQ by unbinding two USB PCI devices: one UHCI and one EHCI device.

The list of USB PCI devices is:

00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 
EHCI Controller #2
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB 
UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 
EHCI Controller #1

I unbound  00:1a.7 and 00:1d.2. To my great surprise, all USB ports on 
the server are still operational. I tried this by inserting a USB memory 
stick and trying to mount it. The motherboard has a total of 8 USB 
connections with 6 USB in use, I don't really understand this. Was I 
simply in luck?

The output of lsucb after doing the unbinding is:

Bus 005 Device 003: ID 046b:ff10 American Megatrends, Inc.
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 002: ID 045e:001e Microsoft Corp. IntelliMouse Explorer
Bus 006 Device 003: ID 051d:0002 American Power Conversion 
Uninterruptible Power Supply
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

Alternatively, I am also considering to by a new PCIe card. In 
particular, the ASUS U3S6 looks interesting because it has 2 PCIe ports 
and 2 USB 3.0 ports. This means that I could disable 2 more USB devices 
(00:1a.2 and 00:1d.1) and get some extra USB devices. The ata_piix 
drivers are being used for an internal disk and CDROM. The RAID array is 
managed through arcmsr (Areca) so I don't have a problem there. Of 
course, if I managed to get completely free of shared IRQs with this 
add-on card, then the issue of determining which of the remaining ports 
is still USB 2.0 is not that important anymore.

So, enough ideas to try out.