From mboxrd@z Thu Jan 1 00:00:00 1970 From: Erik Brakkee Subject: Re: USB Passthrough 1.1 performance problem... Date: Fri, 24 Dec 2010 19:19:42 +0100 Message-ID: <4D14E43E.5020404@brakkee.org> References: <4D04B645.3010100@brakkee.org> <4D0537A8.8000607@brakkee.org> <4D0549AA.2020007@web.de> <4D054D37.7040107@brakkee.org> <3047113345.976756218@brakkee.org> <5085473063.976781602@brakkee.org> <20101214120519.GH32153@redhat.com> <4D07E5F7.1000400@brakkee.org> <65300.194.237.142.7.1292516619.squirrel@brakkee.org> <4D0A88BE.8000707@brakkee.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Kenni Lund , kvm To: Erik Brakkee Return-path: Received: from cpsmtp-fia02.kpnxchange.com ([195.121.247.5]:2195 "EHLO cpsmtp-fia02.kpnxchange.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752584Ab0LXSTm (ORCPT ); Fri, 24 Dec 2010 13:19:42 -0500 In-Reply-To: <4D0A88BE.8000707@brakkee.org> Sender: kvm-owner@vger.kernel.org List-ID: Erik Brakkee wrote: > Erik Brakkee wrote: >> >>> 2010/12/14 Erik Brakkee: >>>> Daniel P. Berrange wrote: >>>>> On Tue, Dec 14, 2010 at 12:55:04PM +0100, Kenni Lund wrote: >>>>> >>>>>> 2010/12/14 Erik Brakkee: >>>>>> >>>>>>>> From: Kenni Lund >>>>>>>> 2010/12/14 Erik Brakkee: >>>>>>>> >>>>>>>>>> From: Kenni Lund >>>>>>>>>> >>>>>>>>>>> Does this mean I have a chance now that PCI passthrough of my >>>>>>>>>>> WinTV >>>>>>>>>>> PVR-500 >>>>>>>>>>> might work now? >>>>>>>>>>> >>>>>>>>>> Passthrough of a PVR-500 has been working for a long time. I've >>>>>>>>>> been >>>>>>>>>> running with passthrough of a PVR-500 in my HTPC, since >>>>>>>>>> November/December 2009...so it should work with any recent >>>>>>>>>> kernel >>>>>>>>>> and >>>>>>>>>> any recent version of qemu-kvm you can find today - No patching >>>>>>>>>> needed. The only issue I had with the PVR-500 card, was when *I* >>>>>>>>>> didn't free up the shared interrupts...once I fixed that, it >>>>>>>>>> "just >>>>>>>>>> worked". >>>>>>>>>> >>>>>>>>> How did you free up those shared interrupts then? I tried >>>>>>>>> different >>>>>>>>> slots >>>>>>>>> but always get conflicts with the USB irqs. >>>>>>>>> >>>>>>>> I did an unbind of the conflicting device (eg. disabled it). I >>>>>>>> moved >>>>>>>> the PVR-500 card around in the different slots and once I got a >>>>>>>> conflict with the integrated sound card, I left the PVR-500 >>>>>>>> card in >>>>>>>> that slot (it's a headless machine, so no need for sound) and >>>>>>>> configured unbind of the sound card at boot time. On my old >>>>>>>> system I >>>>>>>> think it was conflicting with one of the USB controllers as well, >>>>>>>> but >>>>>>>> it didn't really matter, as I only lost a few of the ports on the >>>>>>>> back >>>>>>>> of the computer for that particular USB controller - I still had >>>>>>>> plenty of USB ports left and if I really needed more ports, I >>>>>>>> could >>>>>>>> just plug in an extra USB PCI card. >>>>>>>> >>>>>>>> My /etc/rc.local boot script looks like the following today: >>>>>>>> -- >>>>>>>> #Remove HDA conflicting with ivtv1 >>>>>>>> echo "0000:00:1b.0"> /sys/bus/pci/drivers/HDA\ Intel/unbind >>>>>>>> >>>>>>>> # ivtv0 >>>>>>>> echo "4444 0016"> /sys/bus/pci/drivers/pci-stub/new_id >>>>>>>> echo "0000:04:08.0"> /sys/bus/pci/drivers/ivtv/unbind >>>>>>>> echo "0000:04:08.0"> /sys/bus/pci/drivers/pci-stub/bind >>>>>>>> echo "4444 0016"> /sys/bus/pci/drivers/pci-stub/remove_id >>>>>>>> >>>>>>>> # ivtv1 >>>>>>>> echo "4444 0016"> /sys/bus/pci/drivers/pci-stub/new_id >>>>>>>> echo "0000:04:09.0"> /sys/bus/pci/drivers/ivtv/unbind >>>>>>>> echo "0000:04:09.0"> /sys/bus/pci/drivers/pci-stub/bind >>>>>>>> echo "4444 0016"> /sys/bus/pci/drivers/pci-stub/remove_id >>>>>>>> >>>>>>> I did not try unbinding the usb device so I can also try that. >>>>>>> >>>>>>> I don'.t understand what is happening with the 4444 0016. I >>>>>>> configured >>>>>>> the >>>>>>> pci card in kvm and I believe kvm does the binding to pci-stub in >>>>>>> recent >>>>>>> versions. Where is the 4444 0016%oming from? >>>>>>> >>>>>> Okay, qemu-kvm might do it today, I don't know - I haven't changed >>>>>> that script for the past year. But are you sure that it's not >>>>>> libvirt/virsh/virt-manager which does that for you? >>>>>> >>>>> If you use the managed="yes" attribute on the in libvirt >>>>> XML, then libvirt will automatically do the pcistub bind/unbind, >>>>> followed by a device reset at guest startup& the reverse at >>>>> shutdown. >>>>> If you have conflicting devices on the bus though, libvirt won't >>>>> attempt to unbind them, unless you had also explicitly assigned all >>>>> those conflicting devices to the same guest. >>>>> >>>>> Daniel >>>>> >>>> I definitely have to try again (right now having some stability >>>> problems >>>> on >>>> the server that I am debugging). >>>> >>>> The shared IRQs are as follows: >>>> >>>> 16: 0 0 0 0 >>>> 0 0 >>>> 0 0 IO-APIC-fasteoi uhci_hcd:usb3 >>>> 18: 252995 0 0 0 >>>> 0 0 >>>> 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb8, >>>> ivtv0 >>>> 19: 58281 0 0 0 >>>> 0 0 >>>> 0 0 IO-APIC-fasteoi ata_piix, ata_piix, >>>> uhci_hcd:usb5, >>>> uhci_hcd:usb7, ivtv1 >>>> 21: 0 0 0 0 >>>> 0 0 >>>> 0 0 IO-APIC-fasteoi uhci_hcd:usb4 >>>> 23: 713 6906 0 76919 >>>> 0 0 >>>> 0 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb6 >>>> >>>> So I have IRQ sharing with usb1, usb8, usb5, usb7. >>> Uff....and your ata HDD controller. I guess i was much luckier than >>> you are, my ivtv0 didn't conflict at all and ivtv1 only conflicted >>> with USB. >>> >>>> I have also read that >>>> ehci refers to USB 2.0 and uhci to USB 1.1 is that correct? Anyway, >>>> how >>>> would I now identify the USB PCI devices that I would need to >>>> unbind to >>>> get >>>> rid of the sharing with the USB ports? >>> Play around with: >>> lspci -v >>> lspci -n >>> lsusb -v >>> lsusb -t >>> >>> You can also just start by unbinding the first one and take note when >>> you hit the right ones...once you unbind one, it will disappear from >>> cat /proc/interrupts. When you're down to having only ivtv0 on one >>> interrupt and only ivtv1 on another interrupt, then you're ready to >>> bind with pci-stub and boot your guest. >>> >>>> It also doesn't really matter in >>>> which slot I put the PVR-500 card because both cards share IRQs >>>> with USB >>>> in >>>> all cases. >>> You also have your conflicting ata controller to take into >>> consideration. I don't remember if "ata_piix" and "ata_piix" is IDE >>> only, if it is, you might not even use it. Otherwise it might be >>> easier for you to run qemu-kvm git with the new patches for shared >>> interrupts...it really depends on your setup. >>> >>>> I have also used an add on USB PCI card but still got these >>>> conflicts. I >>>> was >>>> considering to get a PCIe USB card instead to try out in the hope that >>>> that >>>> would use different IRQs. Is that a realistic expectation? That way, I >>>> could >>>> disable all on-board USB (in the BIOS even) and use the add-on USB >>>> only. >>> Most likely, the PCIe USB 3.0 card I bought supported MSI-X, so it got >>> its own unique IRQs which wasn't shared with anything. >>> >> I think I am a bit out of luck here. The two IRQs that are >> interesting are >> 18 and 19. One of them has the SMBUS and the other one has SATA >> drives. I >> think the best for me is to wait until shared IRQ support in KVM is >> available. My only option is to try it out every once in a while. >> >> >> > Meanwhile, I haven't given up and experimented with determining the > effect of unbinding the various USB PCI devices. I haven't tried > virtualization yet though. The problem is I managed to free up one > tuner (ivtv0) from shared IRQs, all USB ports are still operational, > but then how would I determine which USB ports still support USB 2.0? > > I have now at least found a way to make ivtv0 available without any > shared IRQ by unbinding two USB PCI devices: one UHCI and one EHCI > device. > > The list of USB PCI devices is: > > 00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #4 > 00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #5 > 00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #6 > 00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 > EHCI Controller #2 > 00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #1 > 00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #2 > 00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #3 > 00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 > EHCI Controller #1 > > I unbound 00:1a.7 and 00:1d.2. To my great surprise, all USB ports on > the server are still operational. I tried this by inserting a USB > memory stick and trying to mount it. The motherboard has a total of 8 > USB connections with 6 USB in use, I don't really understand this. Was > I simply in luck? > > The output of lsucb after doing the unbinding is: > > Bus 005 Device 003: ID 046b:ff10 American Megatrends, Inc. > Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub > Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub > Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub > Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub > Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub > Bus 006 Device 002: ID 045e:001e Microsoft Corp. IntelliMouse Explorer > Bus 006 Device 003: ID 051d:0002 American Power Conversion > Uninterruptible Power Supply > Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub > > Alternatively, I am also considering to by a new PCIe card. In > particular, the ASUS U3S6 looks interesting because it has 2 PCIe > ports and 2 USB 3.0 ports. This means that I could disable 2 more USB > devices (00:1a.2 and 00:1d.1) and get some extra USB devices. The > ata_piix drivers are being used for an internal disk and CDROM. The > RAID array is managed through arcmsr (Areca) so I don't have a problem > there. Of course, if I managed to get completely free of shared IRQs > with this add-on card, then the issue of determining which of the > remaining ports is still USB 2.0 is not that important anymore. > > So, enough ideas to try out. > In the mean while I figured it out completely. I disabled the 4 USB PCI devices as mentioned above which leads to one USB port becoming unusable (acceptable). Then I could configure the SATA 'handling' in the BIOS. The default was plain SATA, but I could also select Intel RAID, Adaptec RAID, and AHCI. In any case Intel RAID and AHCI solved the problem with the ata_piix shared IRQ (did not check), so I used the most modern one (Intel RAID). (motherboard Supermicro X8DTi-F). Meanwhile I did a successful PCI passthrough test recording two channels simultanesously and it worked! PS. I used to do backups to a backup SATA disk over USB but since I turned on the Intel RAID, the SATA ports on the motherboard became hot swappable so I can now backup over (e)SATA which gives me a 5 times speedup in bandwidth and frees up one USB port, compensating for the one I lost. So all is well and I can continue my virtualized setup according to my original plans. Cheers Erik