From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexey Kardashevskiy Subject: Re: [RFC PATCH] PCI: Introduce INTx check & mask API Date: Fri, 25 May 2012 21:26:13 +1000 Message-ID: <4FBF6C55.1070603@ozlabs.ru> References: <4FBDE6D6.80700@ozlabs.ru> <4FBE2349.6040800@siemens.com> <4FBEDDF3.20108@ozlabs.ru> <4FBEEEA4.2060504@web.de> <4FBEF2C7.4000708@ozlabs.ru> <4FBF6238.7030407@siemens.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: kvm@vger.kernel.org, qemu-devel@nongnu.org, Alex Graf , Alex Williamson , David Gibson To: Jan Kiszka Return-path: Received: from mail-pb0-f46.google.com ([209.85.160.46]:61251 "EHLO mail-pb0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751923Ab2EYL0P (ORCPT ); Fri, 25 May 2012 07:26:15 -0400 Received: by pbbrp8 with SMTP id rp8so1644059pbb.19 for ; Fri, 25 May 2012 04:26:15 -0700 (PDT) In-Reply-To: <4FBF6238.7030407@siemens.com> Sender: kvm-owner@vger.kernel.org List-ID: 25.05.2012 20:43, Jan Kiszka =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB= : > On 2012-05-24 23:47, Alexey Kardashevskiy wrote: >> On 25/05/12 12:29, Jan Kiszka wrote: >>> On 2012-05-24 22:18, Alexey Kardashevskiy wrote: >>>> On 24/05/12 22:02, Jan Kiszka wrote: >>>>> On 2012-05-24 04:44, Alexey Kardashevskiy wrote: >>>>>> [Found while debugging VFIO on POWER but it is platform independ= ent] >>>>>> >>>>>> There is a feature in PCI (>=3D2.3?) to mask/unmask INTx via PCI= _COMMAND and >>>>>> PCI_STATUS registers. >>>>> >>>>> Yes, 2.3 introduced this. Masking is done via command register, c= hecking >>>>> if the source was the PCI in question via the status register. Th= e >>>>> latter is important for supporting IRQ sharing - and that's why w= e >>>>> introduced this masking API to the PCI layer. >>>> >>>> >>>> Is not it just a quite small optimization to not to disable interr= upts on all devices which share >>>> the same IRQ but just on those who fired an interrupt? If so, do P= CI devices really often share >>>> IRQs? Does not supporting this mean real slowdown on such devices? >>>> >>>> As far as I understand, everyone who cares about performance uses = MSI/MSIX, no? >>> >>> Not everyone is blessed with MSI-only PCI devices. From my notebook= : >>> >>> # cat /proc/interrupts >>> [...] >>> 22: [...] IO-APIC-fasteoi ehci_hcd:usb1, ehci_hcd:usb2 >>> >>> So, if I want to assign one EHCI controller to a guest, I have to >>> disable the other as well. The same can happen quickly if you attac= h a >>> few legacy PCI adapters to a system and want to pass them through. >> >> Why? vfio-pci receives interrupt, disables it, handles it, enables i= nterrupt back. Yes, handling is >> a bit longer and includes passing interrupt to QEMU and then to the = guest (can be optimized to avoid >> QEMU) and waiting for EOI notification but this is all the differenc= e. >=20 > You can disable the complete IRQ line as you cannot predict when the > untrusted device driver that VFIO, KVM, or UIO serves will finally > decide to silence the IRQ reason in hardware. If you did this, you ri= sk > a DoS attack on those other devices. Untrusted device still can pull down (or up? do not remember :) ) hardware INT# line, stop other devices on this line and you cannot do anything about it. How does INTx help if the device is that broken? >> Does the current kernel use INTx bit for your USB controllers now, w= ithout any KVM, etc? >=20 > No, it is only used for KVM device assignment when it grabs a device = and > uio_pci_generic. If a host driver uses the device, and can silence > interrupts in a device-specific way. >=20 >> >> So, is it just an optimization or it is something bigger that I miss= ed? >=20 > It is not an optimization but an essential feature to support INTx > sharing between an untrusted device driver and some other driver. So you propose to trust every hardware adapter and only some drivers, i= s it what you are saying? And I thought it is all for the kernel to understand what device called interrupt and disable it without calling every device which uses the same line, and that's it. Am I wrong? >>>>>> And there is some API to support that (commit a2e27787f893621c5a= 6b865acf6b7766f8671328). >>>>>> >>>>>> I have a network adapter: >>>>>> 0001:00:01.0 Ethernet controller: Chelsio Communications Inc T31= 0 10GbE Single Port Adapter >>>>>> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- Par= Err+ Stepping- SERR+ FastB2B- DisINTx- >>>>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=3Dfast >TAbort= - SERR- >>>>> >>>>>> pci_intx_mask_supported() reports that the feature is supported = for this adapter >>>>>> BUT the adapter does not set PCI_STATUS_INTERRUPT so pci_check_a= nd_set_intx_mask() >>>>>> never changes PCI_COMMAND and INTx does not work on it when we u= se it as VFIO-PCI device. >>>>>> >>>>>> If I remove the check of this bit, it works fine as it is called= from an interrupt handler and >>>>>> Status bit check is redundant. >>>>>> >>>>>> Opened a spec: >>>>>> PCI LOCAL BUS SPECIFICATION, REV. 3.0, Table 6-2: Status Registe= r Bits >>>>>> =3D=3D=3D >>>>>> 3 This read-only bit reflects the state of the interrupt in the >>>>>> device/function. Only when the Interrupt Disable bit in the comm= and >>>>>> register is a 0 and this Interrupt Status bit is a 1, will the >>>>>> device=E2=80=99s/function=E2=80=99s INTx# signal be asserted. Se= tting the Interrupt >>>>>> Disable bit to a 1 has no effect on the state of this bit. >>>>>> =3D=3D=3D >>>>>> With this adapter, INTx# is asserted but Status bit is still 0. >>>>>> >>>>>> Is it mandatory for a device to set Status bit if it supports IN= Tx masking? >>>>>> >>>>>> 2 Alex: if it is mandatory, then we need to be able to disable p= ci_2_3 in VFIO-PCI >>>>>> somehow. >>>>> >>>>> Since PCI 2.3, this bit is mandatory, and it should be independen= t of >>>>> the masking bit. The question is, if your device is supposed to s= upport >>>>> 2.3, thus is just buggy, or if our detection algorithm is unrelia= ble. It >>>>> basically builds on the assumption that, if we can flip the mask = bit, >>>>> the feature should be present. I guess that is the best we can do= =2E Maybe >>>>> we can augment this with a blacklist of devices that "support" fl= ipping >>>>> without actually providing the feature. >>>> >>>> It is a good moment to start :) >>>> Not sure where - in VFIO or along with that PCI INTx API. >>> >>> At PCI level as the API is VFIO agnostic (it was introduced for >>> "classic" KVM device assignment, in fact). >>>> Here is that broken device: >>>> aik@vpl2:~$ lspci -s 1:1:0.0 >>>> 0001:01:00.0 Ethernet controller: Chelsio Communications Inc T310 = 10GbE Single Port Adapter >>>> aik@vpl2:~$ lspci -ns 1:1:0.0 >>>> 0001:01:00.0 0200: 1425:0030 >>> >>> A patch to add the infrastructure as well would be even more welcom= e. :) >>> You could have a look at drivers/pci/quirks.c for patterns how to d= o this. >> >> I am not sure yet that we need this feature at all ;) I would rather= prefer to have some way to >> disable it in VFIO rather than to add yet another quirk for the feat= ure which nobody uses at the moment. >> Really, this device supports MSI/MSIX and in real life nobody is goi= ng to use INTx on it. The only >> need for it is testing. >=20 > These are wrong assumptions, both that it has to be addressed at VFIO > level and that it has no serious use case. We will need this feature = for > quite a while until legacy PCI finally died. Bets are taken when this > will happen, but I would be careful with any date in this decade. ;) Heh. I bet that legacy PCI will never be a serious target for legacy PC= I ;) I really do not understand. You want to do PCI pass through for old devices which share INT# line and can screw the devices they share interrupt with. And you trust drivers of devices which support INTx. I believe that this is not enough for trust, you need hardware isolatio= n. At least, you should put all the devices which share the same IRQ to on= e IOMMU group. --=20 With best regards Alexey Kardashevskiy -- icq: 52150396