From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: Skylake: VT-d and other error messages Date: Sat, 28 Nov 2015 22:14:52 +0000 Message-ID: <565A275C.108@citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0873049562345046086==" Return-path: Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1a2nlm-0005Qq-K4 for xen-devel@lists.xenproject.org; Sat, 28 Nov 2015 22:15:02 +0000 In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Eric Shelton , xen-devel , Yang Zhang , Kevin Tian , Keir Fraser , Jan Beulich List-Id: xen-devel@lists.xenproject.org --===============0873049562345046086== Content-Type: multipart/alternative; boundary="------------030009040905080605030009" --------------030009040905080605030009 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit On 28/11/15 20:46, Eric Shelton wrote: > Looking through the output of 'xl dmesg' on a Skylake system > (i5-6600K), I found a number of error messages that I do not encounter > on a Haswell-based system. I have tried two motherboards from > different manufacturers, with pretty much the same results. Below are > some of the unexpected messages: > > Not enabling x2APIC (upon firmware request) > ... > mwait-idle: does not run on family 6 model 94 > ... > [VT-D] iommu.c:875: iommu_fault_status: Primary Pending Fault > [VT-D] INTR-REMAP: Request device [0000:f0:1f.0] fault index 0, iommu > reg = ffff82c000201000 > (on motherboard 1) [VT-D] INTR-REMAP: reason 22 - Present field in the > IRTE entry is clear > (on motherboard 2) [VT-D] INTR-REMAP: reason 25 - Blocked a > compatibility format interrupt request > > This leads to a few questions: > 1) Is there some reason x2APIC should not be enabled on Skylake? What > consequence, if any, is there not having x2APIC enabled? In this case, the firmware has set the x2apic opt-out bit in the DMAR table, indicating that Xen should not use x2apic. You might find an option in your BIOS to undo this; there have been enough errata in the past in this area that I would expect it to be a tweakable. x2apic is the extension to xapic, which permits more than 255 cpus. So long as you don't have that many, there isn't a specific problem with missing x2apic mode. > 2) Should mwait-idle be available on Skylake? We probably need to resync the mwait driver with Linux. It is whitelisted on known cpu model numbers. > 3) What about the IOMMU errors on Skylake - are they a concern? Yes. In both cases, PCI device f0:1f.0 is misconfigured or misbehaving. On motherboard 1, it is delivering an interrupt for which no remapping entry has been set up. On motherboard 2, it is delivering an compatibility-format interrupt, as opposed to a remapable-format interrupt. For motherboard 2, Xen should disallow such a configuration. Either interrupt remapping is enabled and all devices should be configured to issue remmapable interrupts, or interrupt remapping is disabled and everything should be configured to issue compatibility-format interrupts. Either way, diagnosing the problem here starts with identifying what f0:1f.0 is. ~Andrew --------------030009040905080605030009 Content-Type: text/html; charset="utf-8" Content-Length: 3862 Content-Transfer-Encoding: quoted-printable
On 28/11/15 20:46, Eric Shelton wrote:
Looking through the output of 'xl dmesg' on a Skylake system (i5-6600K), I found a number of error messages that I do not encounter on a Haswell-based system.=C2=A0 I have tried two motherboards from different manufacturers, with pretty much the same results.=C2=A0 Below are some of the unexpected messages:

Not enabling x2APIC (upon firmware request)
...
mwait-idle: does not run on family 6 model 94
...
[VT-D] iommu.c:875: iommu_fault_status: Primary Pending Fault
[VT-D] INTR-REMAP: Request device [0000:f0:1f.0] fault index 0, iommu reg =3D ffff82c000201000
(on motherboard 1) [VT-D] INTR-REMAP: reason 22 - Present field in the IRTE entry is clear
(on motherboard 2) [VT-D] INTR-REMAP: reason 25 - Blocked a compatibility format interrupt request

This leads to a few questions:
1) Is there some reason x2APIC should not be enabled on Skylake=3F=C2=A0 What consequence, if any, is there not having x2APIC enabled=3F

In this case, the firmware has set the x2apic opt-out bit in the DMAR table, indicating that Xen should not use x2apic.

You might find an option in your BIOS to undo this; there have been enough errata in the past in this area that I would expect it to be a tweakable.

x2apic is the extension to xapic, which permits more than 255 cpus.=C2=A0 So long as you don't have that many, there isn't a specific problem with missing x2apic mode.

2) Should mwait-idle be available on Skylake=3F

We probably need to resync the mwait driver with Linux.=C2=A0 It is whitelisted on known cpu model numbers.

3) What about the IOMMU errors on Skylake - are they a concern=3F

Yes.

In both cases, PCI device f0:1f.0 is misconfigured or misbehaving.

On motherboard 1, it is delivering an interrupt for which no remapping entry has been set up.

On motherboard 2, it is delivering an compatibility-format interrupt, as opposed to a remapable-format interrupt.

For motherboard 2, Xen should disallow such a configuration.=C2=A0 Either interrupt remapping is enabled and all devices should be configured to issue remmapable interrupts, or interrupt remapping is disabled and everything should be configured to issue compatibility-format interrupts.

Either way, diagnosing the problem here starts with identifying what f0:1f.0 is.

~Andrew
--------------030009040905080605030009-- --===============0873049562345046086== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============0873049562345046086==--