From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhao, Yu" Subject: Re: XEN Panic with VT-d support enabled on Xeon E55xxplatform Date: Wed, 08 Apr 2009 13:54:48 +0800 Message-ID: <49DC3C28.9070503@intel.com> References: <49DABAFD.4070301@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Isabelle, Francois" Cc: "xen-devel@lists.xensource.com" , "Cui, Dexuan" List-Id: xen-devel@lists.xenproject.org Francois, Thank you for the log. (XEN) [VT-D]iommu.c:1237:d32767 domain_context_mapping:PCI: bdf =3D 0:1d.= 0 Did you see 1 second delay after above line was printed? Or the system=20 got panic immediately following it? (XEN) DMAR_IQA_REG =3D 7f79d000 (XEN) DMAR_IQH_REG =3D 0 (XEN) DMAR_IQT_REG =3D 20 (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) queue invalidate wait descriptor was not executed (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... Isabelle, Francois wrote: >>> When you only use iommu=3Dno-intremap and see the panic, can you atta= ched >>the xen log? >>> And can you also try iommu=3Dpassthrough,no-intremap and attach the l= og? >=20 > See attached files. >=20 >>> VT-d Context Entry is used, the DMA requests related to the Context E= ntry can bypass DMA remapping. This can be faster than DMA remapping. > The BIOS does not have an option for this on my board. But I did mistoo= k it for DMA remapping. >=20 >=20 > And thank you for answering my questions. >=20 > Fran=E7ois Isabelle >=20 >=20 >=20 >=20 > -----Message d'origine----- > De : Cui, Dexuan [mailto:dexuan.cui@intel.com] > Envoy=E9 : 7 avril 2009 10:22 > =C0 : Isabelle, Francois; Zhao, Yu > Cc : xen-devel@lists.xensource.com > Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55= xxplatform >=20 >>>> iommu=3Dxxx >> Dexuan, 'iommu=3Dno-intremap' is not enough to make it run, ' iommu=3D= no-qinval,no-intremap' allows booting. > When you only use iommu=3Dno-intremap and see the panic, can you attach= ed the xen log? > And can you also try iommu=3Dpassthrough,no-intremap and attach the log= ? >=20 >>> (XEN) [VT-D]dmar.c:485: Host address width 39 >> Shouldn't this be 38 ? > The Host Address Width (HAW) of the platform is computed as (N+1), wher= e N is the value reported in the Host Address Width field of DMA Remappin= g Reporting Structure. > Please see acpi_parse_dmar() or VT-d spec for details. >=20 >>> (XEN) Intel VT-d Snoop Control supported. >>> (XEN) Intel VT-d DMA Passthrough not supported. >> Shouldn't that be enabled to support DMA from domU? > If the "DMA Passthrough" (please don't mistake it for DMA remapping:-) = of VT-d Context Entry is used, the DMA requests related to the Context En= try can bypass DMA remapping. This can be faster than DMA remapping. If y= ou host supports DMA passthrough (you could check your BIOS setup menu), = in Xen we can enable DMA passthrough by iommu=3Dpassthrough (Note, even w= ith this parameter, we only enable passthrough for Dom0 considering secri= ty) >=20 >=20 >>> (XEN) HPET broadcast init failed, turn to PIT broadcast. >> Does that mean the HPET is not functional? >>>From hpet_fsb_cap_lookup(), looks your HPET timers may work, but they d= on't support FSB routing, so won't be used in power management code. >=20 >> Can we expect a good level of performance with interrupt remapping dis= abled? Will a domU be able to load a 82576 driver and use the NIC at all? > I think we should have almost the same performance with Interrupt Remap= ping enabled or disabled. > 82576 should work in DomU. >=20 > Thanks, > -- Dexuan >=20 >=20 >=20 > -----Original Message----- > From: Isabelle, Francois [mailto:Francois.Isabelle@ca.kontron.com] > Sent: Tuesday, April 07, 2009 9:12 PM > To: Cui, Dexuan; Zhao, Yu > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E5= 5xxplatform >=20 > Hi. >=20 > Here is what you both asked for: >=20 >>> Do you have this problem all the times? Can you please grab some logs >>> and send them to me? >=20 > Yes Yu, all the times. I will package the logs as attachment to this po= st. >=20 >>> iommu=3Dxxx > Dexuan, 'iommu=3Dno-intremap' is not enough to make it run, ' iommu=3Dn= o-qinval,no-intremap' allows booting. >=20 > Here are some parts of the log I don't understand... >=20 >> (XEN) [VT-D]dmar.c:485: Host address width 39 > Shouldn't this be 38 ? >=20 >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. > Shouldn't that be enabled to support DMA from domU? >=20 >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping supported. >=20 >> (XEN) HPET broadcast init failed, turn to PIT broadcast. > Does that mean the HPET is not functional? >=20 > Can we expect a good level of performance with interrupt remapping disa= bled? Will a domU be able to load a 82576 driver and use the NIC at all? >=20 > Thank you both for your replies. >=20 > Fran=E7ois Isabelle >=20 > -----Message d'origine----- > De : Cui, Dexuan [mailto:dexuan.cui@intel.com] > Envoy=E9 : 7 avril 2009 03:05 > =C0 : Zhao, Yu; Isabelle, Francois > Cc : xen-devel@lists.xensource.com > Objet : RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E55= xxplatform >=20 >>> (XEN) DMAR_IQA_REG =3D 7f79d000 >>> (XEN) DMAR_IQH_REG =3D 0 >>> (XEN) DMAR_IQT_REG =3D 20 >> Here the IQT is such a big number 0x20... this seems pretty strange. > Oh, actually this 0x20 is normal -- actually QT =3D IQT_REG >> 4 accori= ding the format of IQT_REG in VT-d spec. So QT =3D 0x2 -- this is ok. > So looks the panic somehow happens on the first invalidation: enable_in= tremap() -> iommu_flush_iec_global(). I expect your host can boot fine wi= th iommu=3Dno-intremap -- if this still doesn't work, could you please tr= y iommu=3Dno-qinval,no-intremap ? >=20 > Thanks, > -- Dexuan >=20 >=20 >=20 > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@l= ists.xensource.com] On Behalf Of Cui, Dexuan > Sent: Tuesday, April 07, 2009 12:40 PM > To: Zhao, Yu; Isabelle, Francois > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E5= 5xx platform >=20 >> (XEN) DMAR_IQA_REG =3D 7f79d000 >> (XEN) DMAR_IQH_REG =3D 0 >> (XEN) DMAR_IQT_REG =3D 20 > Here the IQT is such a big number 0x20... this seems pretty strange. >=20 > Hi Fran=E7ois, > And, could you please try the xen parameter "iommu=3Dno-intremap" ? -- = If you still see the panic, please also add the parameter "nosmp" to see= if there would any change. > We don't have the same host at hand. :-( So we need try these on your = host. > BTW, can you boot the same host using the latest native linux-2.6.29-rc= 8 when you iommu_intel=3Don and interrupt remapping is enabled? >=20 > PS, what's your BIOS version and date -- could you try to find the late= st BIOS for the host? I personally tend to think this is a BIOS issue. >=20 > Thanks, > -- Dexuan >=20 >=20 >=20 > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@l= ists.xensource.com] On Behalf Of Zhao, Yu > Sent: Tuesday, April 07, 2009 10:31 AM > To: Isabelle, Francois > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] XEN Panic with VT-d support enabled on Xeon E5= 5xx platform >=20 > Hi Francois, >=20 > Do you have this problem all the times? Can you please grab some logs > and send them to me? The dmesg from a native kernel if you can't boot u= p > any version of Xen, and the whole Xen hypervisor log would help us to > figure out the problem. >=20 > Thanks, > Yu >=20 >=20 > Isabelle, Francois wrote: >> Hi all, >> >> I'm working with a board on which XEN panics with the following messag= e (see below). From what I can see on the lists, most of the problems on = these platforms seem to be related to broken DMAR tables and RMRR, but th= is platform passes the VT-d firmware toolkit (http://edc.intel.com/Platfo= rms/Xeon-5500/) successfully so I'm tented to blame XEN for it... should = I file a bug entry? >> >> >From what I can see in the 'queue invalidation' code, a timeout is re= ached. As anyone seen this on a similar platform? >> >> >> (XEN) Xen version 3.4-unstable (root@localdomain) (gcc version 4.1.2 2= 0080704 (Red Hat 4.1.2-44)) Fri Apr 3 14:20:00 EDT 2009 >> (XEN) Latest ChangeSet: Mon Mar 30 16:48:26 2009 +0100 19433:d5ddc782b= c49 >> (XEN) Command line: console=3Dcom1 com1=3D115200,8n1 iommu=3D1 loglvl= =3Dall loglvl_guest=3Dall ro >> >> .. >> (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs >> (XEN) ACPI: HPET id: 0xffffffff base: 0xfed00000 >> (XEN) [VT-D]dmar.c:485: Host address width 39 >> (XEN) [VT-D]dmar.c:494: found ACPI_DMAR_DRHD >> (XEN) [VT-D]dmar.c:349: dmaru->address =3D fbffe000 >> (XEN) [VT-D]dmar.c:306: found IOAPIC: bdf =3D f0:1f.7 >> (XEN) [VT-D]dmar.c:358: found INCLUDE_ALL >> (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf =3D 0:1d.0 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf =3D 0:1d.1 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf =3D 0:1d.2 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf =3D 0:1d.7 >> (XEN) [VT-D]dmar.c:498: found ACPI_DMAR_RMRR >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf =3D 0:1d.0 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf =3D 0:1d.1 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf =3D 0:1d.2 >> (XEN) [VT-D]dmar.c:300: found endpoint: bdf =3D 0:1d.7 >> (XEN) [VT-D]dmar.c:502: found ACPI_DMAR_ATSR >> (XEN) [VT-D]dmar.c:287: found bridge: bdf =3D 0:3.0 sec =3D 5 sub =3D= 5 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf =3D 0:4.0 sec =3D 6 sub =3D= e >> (XEN) [VT-D]dmar.c:287: found bridge: bdf =3D 0:5.0 sec =3D 3 sub =3D= 3 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf =3D 0:7.0 sec =3D 18 sub =3D= 18 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf =3D 0:8.0 sec =3D f sub =3D= 17 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf =3D 0:9.0 sec =3D 4 sub =3D= 4 >> (XEN) [VT-D]dmar.c:287: found bridge: bdf =3D 0:a.0 sec =3D 2 sub =3D= 2 >> .. >> (XEN) Intel VT-d Snoop Control supported. >> (XEN) Intel VT-d DMA Passthrough not supported. >> (XEN) Intel VT-d Queued Invalidation supported. >> (XEN) Intel VT-d Interrupt Remapping supported. >> (XEN) DMAR_IQA_REG =3D 7f79d000 >> (XEN) DMAR_IQH_REG =3D 0 >> (XEN) DMAR_IQT_REG =3D 20 >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 0: >> (XEN) queue invalidate wait descriptor was not executed >> (XEN) **************************************** >> (XEN) >> (XEN) Reboot in five seconds... >> >> Fran=E7ois Isabelle >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >=20