From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH 0/5] Add MSI support to XEN Date: Thu, 27 Mar 2008 07:56:08 +0000 Message-ID: References: <823A93EED437D048963A3697DB0E35DE0139CE15@pdsmsx414.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2064661193==" Return-path: In-Reply-To: <823A93EED437D048963A3697DB0E35DE0139CE15@pdsmsx414.ccr.corp.intel.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Shan, Haitao" , xen-devel Cc: "Tian, Kevin" , "Jiang, Yunhong" , "Li, Xin B" List-Id: xen-devel@lists.xenproject.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============2064661193== Content-type: multipart/alternative; boundary="B_3289449374_11578502" > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3289449374_11578502 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Thanks, I=B9ll have to look at the patches regarding the per-domain pirq changes. Tha= t sounds like it probably makes sense, but I seem to remember there were big changes to the irq architecture and irq naming in the hypervisor in previou= s iterations of these patches, which I didn=B9t understand. This IRQ storm issue still needs properly resolving. Noone has yet explaine= d how a message-based interrupt source can cause an irq storm. Storms are inherently a property of level-triggered sources, where ACK/EOI immediately causes re-sampling of the interrupt line and re-assertion of the interrupt at the CPU. How can anything similar happen with MSI? You (Intel) are probably uniquely placed to answer this question, since you manufacture the chipset and NIC which exhibit this problem. -- Keir On 27/3/08 06:55, "Shan, Haitao" wrote: > The basic idea including: > 1) Keep vector global resource owned by xen, while split pirq into per-do= main > information. > 2) Domain0 kernel will operate msi resource for domain0/domU, while QEMU = will > operate MSI resource for HVM domain. > 3) Xen will do EOI for MSI interrupt. >=20 > Signed-off-by: Yunhong Jiang > > =20 > There are no much changes made compared with the original patches. Bu= t > there do have some issues that we need your kind comments. > 1> ACK-NEW method is necessary to avoid IRQ storm. But it causes the > deadlock.=20 > During my tests, I do find there can be deadlock with patches > applied. When assigned a NIC device to HVM domain, the scenario is: Dom0 = is > waiting to IDE interrupt (vector 0x21); HVM domain is waiting for qemu=B9s = IDE > emulation and thus blocked; NIC interrupt (MSI vector 0x31) is waiting fo= r > injection to HVM domain since it is blocked now; IDE interrupt is waiting= for > NIC interrupt since NIC interrupt is of high priority but not ACKed by XE= N > now. When IDE interrupt and NIC interrupt are delivered to the same CPU, = and > when guest OS is Vista, the phenomenon is easy to be observed. > 2> Without ACK-NEW, some naughty NIC devices as we observed will brin= g IRQ > storms. For this phenomenon, I think Yunhong can comment more. Basically, > writing EOI without mask the source of MSI will bring IRQ storm. Although= the > reason is under investigation, XEN should anyhow handle such bogous devic= e, > right? > 3> Using ACK-OLD and masking the MSI when writing EOI can be solution= . > However, XEN does not own PCI configuration spaces. > =20 > We also tried some work arounds. > One work around might be using a timer to force a EOI within some tim= e > interval. This method is already implemented in VT-D=B9s code. However, wit= h > this approach, if the timer is fired and EOI is written, this is essentia= lly > the same apporach as option 2. > Another approach is to never deliver these two IRQs to the same CPU. = But > this is really ugly and can not be applied to UP. > We have also considered using VT-D 2 interrupt remapping feature. > According to the spec, there is no bit in the remapping table to mask the > interrupt. Therefore, this can not be combined with option 2 to solve the > issue. Masking the interrupt still needs accessing PCI configuration spac= es. > =20 > We think the most clean method may be to move ownership from dom0 to = VMM. > However, this is a great change. This should be well discussed in communi= ty > and need your comments. > =20 > These patch series sent out can be served as a discussion materials. = What > is your comments on the patches and the issues, Keir? > =20 > Thanks! --B_3289449374_11578502 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Re: [Xen-devel] [PATCH 0/5] Add MSI support to XEN Thank= s,

I’ll have to look at the patches regarding the per-domain pirq change= s. That sounds like it probably makes sense, but I seem to remember there we= re big changes to the irq architecture and irq naming in the hypervisor in p= revious iterations of these patches, which I didn’t understand.

This IRQ storm issue still needs properly resolving. Noone has yet explaine= d how a message-based interrupt source can cause an irq storm. Storms are in= herently a property of level-triggered sources, where ACK/EOI immediately= causes re-sampling of the interrupt line and re-assertion of the interr= upt at the CPU. How can anything similar happen with MSI? You (Intel) are pr= obably uniquely placed to answer this question, since you manufacture the ch= ipset and NIC which exhibit this problem.

 -- Keir

On 27/3/08 06:55, "Shan, Haitao" <haitao.shan@intel.com> wr= ote:

The basic idea including:
1) Keep vector global resource owned by xen, while split pirq into per-doma= in information.
2) Domain0 kernel will operate msi resource for domain0/domU, while QEMU wi= ll operate MSI resource for HVM domain.
3) Xen will do EOI for MSI interrupt.

Signed-off-by: Yunhong Jiang <yunhong.jiang@int= el.com <mailto:yunhong.jiang@intel.com> >
 
    There are no much changes made compared with the or= iginal patches. But there do have some issues that we need your kind comment= s.
    1> ACK-NEW method is necessary to avoid IRQ stor= m. But it causes the deadlock.
         During my tests, I do= find there can be deadlock with patches applied. When assigned a NIC device= to HVM domain, the scenario is: Dom0 is waiting to IDE interrupt (vector 0x= 21); HVM domain is waiting for qemu’s IDE emulation and thus blocked; = NIC interrupt (MSI vector 0x31) is waiting for injection to HVM domain since= it is blocked now; IDE interrupt is waiting for NIC interrupt since NIC int= errupt is of high priority but not ACKed by XEN now. When IDE interrupt and = NIC interrupt are delivered to the same CPU, and when guest OS is Vista, the= phenomenon is easy to be observed.
    2> Without ACK-NEW, some naughty NIC devices as = we observed will bring IRQ storms. For this phenomenon, I think Yunhong can = comment more. Basically, writing EOI without mask the source of MSI will bri= ng IRQ storm. Although the reason is under investigation, XEN should anyhow = handle such bogous device, right?
    3> Using ACK-OLD and masking the MSI when writin= g EOI can be solution. However, XEN does not own PCI configuration spaces.  
    We also tried some work arounds.
    One work around might be using a timer to force a E= OI within some time interval. This method is already implemented in VT-DR= 17;s code. However, with this approach, if the timer is fired and EOI is wri= tten, this is essentially the same apporach as option 2.
    Another approach is to never deliver these two IRQs= to the same CPU. But this is really ugly and can not be applied to UP.
    We have also considered using VT-D 2 interrupt rema= pping feature. According to the spec, there is no bit in the remapping table= to mask the interrupt. Therefore, this can not be combined with option 2 to= solve the issue. Masking the interrupt still needs accessing PCI configurat= ion spaces.
 
    We think the most clean method may be to move owner= ship from dom0 to VMM. However, this is a great change. This should be well = discussed in community and need your comments.
 
    These patch series sent out can be served as a disc= ussion materials. What is your comments on the patches and the issues, Keir?=
    
Thanks!

 --B_3289449374_11578502-- --===============2064661193== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============2064661193==--