Re: VT-D RMRR is incorrect

Re: VT-D RMRR is incorrect

[Christian Tramnitz]

Dustin,

I couldn't really get what you were trying to say (I never mentioned 
AMD...) but anyway this isn't really Linux specific it's a BIOS bug. 
Unfortunately the only way to get in contact with that bug is to either 
use Xen or enable DMRR in the vanilla Linux kernel. If this cannot be 
fixed in software I'll probably return the board to the store and get an 
Intel DX58SO instead...


Yoshiharu, have you tested Xen-unstable yet? When commenting out the 
EFAULT you mentioned I get an immediate panic related to the rmrr 
address conflict on the latest 3.4-unstable tree.


@Xen devs: Is this something that could be worked around in Xen or do we 
really have to rely on exact RMRR values for vtd to work?
See http://permalink.gmane.org/gmane.comp.emulators.xen.user/43937 for 
more details.


Thanks,
    Christian

Dustin Henning schrieb:
> 	I would recommend opening a new ticket with ASUS and telling them
> you were running Windows if you can get the same output Yoshiharu-san did
> and it fails to meet AMD specifications the way his fails to meet Intel
> specifications, after all, that would mean their board was out of spec with
> AMD, not Linux.  However, in my experience, ASUS can't fix anything (I tried
> to be an early adopter of the SiI port multiplier via their P5WD2-Premium
> onboard SiI SATA and a port multiplier from someone else, after all, their
> documentation clearly stated that they supported port multipliers, and they
> didn't make their own, however, it wouldn't work with port multipliers, they
> "escalated it to their engineering," and I never heard back, so the issue
> was never resolved).  I'd recommend going with Gigabyte or SuperMicro,
> because I've had equally bad support from MSI (I have several computers
> whose systems monitoring software goes off continuously until reboot after
> using a floppy drive, that would have to be a firmware issue, and they can't
> fix it).  All the other manufacturers are either awfully new to the market
> or awfully unreliable (Biostar, for instance, isn't likely to have a board
> last much longer than their warranty in my experience, regardless of
> platform), SuperMicro would probably be the better bet compatibility wise,
> but they have a short warranty and less features (read: less additional
> features that probably don't work with Linux anyway).  Good luck finding
> something that works, be sure to let everyone know what it is.
> 	Dustin

Re: Re: VT-D RMRR is incorrect

[Yoshiharu Mori]

[-- Attachment #1.1: Type: text/plain, Size: 959 bytes --]

Hi Christian


> Yoshiharu, have you tested Xen-unstable yet? When commenting out the EFAULT
> you mentioned I get an immediate panic related to the rmrr address conflict
> on the latest 3.4-unstable tree.


I also tried Xen-unstable, but could not boot xen when I commented out the
EFAULT.
(The boot message was too fast and my machine was rebooted soon, I counldn't
understand details.)

BTW, I got new BIOS from SUPERMICRO. But I cannot test till next week end.
Another trouble was happend on my board, I sent it to distributor and change
it.


@Xen devs: Is this something that could be worked around in Xen or do we
> really have to rely on exact RMRR values for vtd to work?
> See http://permalink.gmane.org/gmane.comp.emulators.xen.user/43937 for
> more details.


The RMRR regions are expencted to be used only for USB and UMA Graphics
legacy usages for reserved memory, I think it may be possible to use vtd
with limitation.


Thanks,
Yoshiharu Mori

[-- Attachment #1.2: Type: text/html, Size: 1471 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

RE: Re: [Xen-users] Re: VT-D RMRR is incorrect

[Ross Philipson]

[-- Attachment #1.1: Type: text/plain, Size: 2196 bytes --]

Hi folks,

This sounds a lot like an issue I am working on right now. A recent change to the vtd code changed the way the 1 to 1 page mapping for dom0 is done (look at intel_iommu_domain_init() for dom0). The change was to query the e820 map for memory regions that were usable RAM and only map those into the iommus for dom0. Thus reserved regions were no longer mapped in. When the RMRR values do not report the correct values for required 1 to 1 reserved memory regions, the needed regions do not get mapped in. Thus device DMA to these regions causes vtd faults and in out case a hang. It is indeed a BIOS bug; perhaps this is the same issue you are seeing?

Anyway, I am working on a workaround that I will be pushing upstream this week. A parameter will allow you to include reserved memory regions in the mappings. It is not ideal (less secure) but better than hanging or crashing...

Thanks
Ross

From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Yoshiharu Mori
Sent: Saturday, February 21, 2009 11:58 AM
To: Christian Tramnitz
Cc: xen-devel@lists.xensource.com; xen-users@lists.xensource.com
Subject: [Xen-devel] Re: [Xen-users] Re: VT-D RMRR is incorrect

Hi Christian

Yoshiharu, have you tested Xen-unstable yet? When commenting out the EFAULT you mentioned I get an immediate panic related to the rmrr address conflict on the latest 3.4-unstable tree.

I also tried Xen-unstable, but could not boot xen when I commented out the EFAULT.
(The boot message was too fast and my machine was rebooted soon, I counldn't understand details.)

BTW, I got new BIOS from SUPERMICRO. But I cannot test till next week end.
Another trouble was happend on my board, I sent it to distributor and change it.

@Xen devs: Is this something that could be worked around in Xen or do we really have to rely on exact RMRR values for vtd to work?
See http://permalink.gmane.org/gmane.comp.emulators.xen.user/43937 for more details.

The RMRR regions are expencted to be used only for USB and UMA Graphics legacy usages for reserved memory, I think it may be possible to use vtd with limitation.


Thanks,
Yoshiharu Mori

[-- Attachment #1.2: Type: text/html, Size: 6170 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Re: VT-D RMRR is incorrect

[Dustin Henning]

Christian,
	I am not sure what lead me to believe you had an AMD system.  To
summarize what I was trying to say, ASUS is claiming that this issue is
unsupported because you use Linux, but at the end of the day, the issue has
nothing to do with Linux directly (any virtualization software that used
VT-D could have the same problem, including those run on Windows.
Therefore, if you tell ASUS that you are running Windows and then tell them
that your issue is that their BIOS doesn't meet Intel's specifications, then
you might be able to get the case escalated (not that anyone at ASUS could
ever get it fixed), and if it were resolved, this would be good for the Xen
community.  Barring that, especially if you are in a hurry to get this
working, going with an Intel board should be a safe bet.  I mentioned my
thoughts on some other motherboard manufacturers as well, and I must redact
one of those, based on a thread I read this morning, Gigabyte motherboards
don't even support VT-D (or at least Gigabyte had no intention of supporting
it not too long ago), see this thread:
http://lists.xensource.com/archives/html/xen-users/2009-02/msg00630.html.
Good luck,
	Dustin

-----Original Message-----
From: xen-users-bounces@lists.xensource.com
[mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Christian
Tramnitz
Sent: Saturday, February 21, 2009 10:31
To: xen-users@lists.xensource.com
Cc: xen-devel@lists.xensource.com
Subject: [Xen-users] Re: VT-D RMRR is incorrect

Dustin,

I couldn't really get what you were trying to say (I never mentioned 
AMD...) but anyway this isn't really Linux specific it's a BIOS bug. 
Unfortunately the only way to get in contact with that bug is to either 
use Xen or enable DMRR in the vanilla Linux kernel. If this cannot be 
fixed in software I'll probably return the board to the store and get an 
Intel DX58SO instead...


Yoshiharu, have you tested Xen-unstable yet? When commenting out the 
EFAULT you mentioned I get an immediate panic related to the rmrr 
address conflict on the latest 3.4-unstable tree.


@Xen devs: Is this something that could be worked around in Xen or do we 
really have to rely on exact RMRR values for vtd to work?
See http://permalink.gmane.org/gmane.comp.emulators.xen.user/43937 for 
more details.


Thanks,
    Christian

Dustin Henning schrieb:
> 	I would recommend opening a new ticket with ASUS and telling them
> you were running Windows if you can get the same output Yoshiharu-san did
> and it fails to meet AMD specifications the way his fails to meet Intel
> specifications, after all, that would mean their board was out of spec
with
> AMD, not Linux.  However, in my experience, ASUS can't fix anything (I
tried
> to be an early adopter of the SiI port multiplier via their P5WD2-Premium
> onboard SiI SATA and a port multiplier from someone else, after all, their
> documentation clearly stated that they supported port multipliers, and
they
> didn't make their own, however, it wouldn't work with port multipliers,
they
> "escalated it to their engineering," and I never heard back, so the issue
> was never resolved).  I'd recommend going with Gigabyte or SuperMicro,
> because I've had equally bad support from MSI (I have several computers
> whose systems monitoring software goes off continuously until reboot after
> using a floppy drive, that would have to be a firmware issue, and they
can't
> fix it).  All the other manufacturers are either awfully new to the market
> or awfully unreliable (Biostar, for instance, isn't likely to have a board
> last much longer than their warranty in my experience, regardless of
> platform), SuperMicro would probably be the better bet compatibility wise,
> but they have a short warranty and less features (read: less additional
> features that probably don't work with Linux anyway).  Good luck finding
> something that works, be sure to let everyone know what it is.
> 	Dustin


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: Re: VT-D RMRR is incorrect

[Steve Reaver]

On Tue, Feb 24, 2009 at 12:36 AM, Dustin Henning
<Dustin.Henning@prd-inc.com> wrote:
> Christian,
>        I am not sure what lead me to believe you had an AMD system.  To
> summarize what I was trying to say, ASUS is claiming that this issue is
> unsupported because you use Linux, but at the end of the day, the issue has
> nothing to do with Linux directly (any virtualization software that used
> VT-D could have the same problem, including those run on Windows.

Well this should be easy to resolve... can anyone tell me what Windows
virtualisation software supports VT-D ??

 it would be no problem for me to put another drive in my P6T with
windows on it and test it out..... that way I can log a case about it
not working.

Re: [Xen-users] Re: VT-D RMRR is incorrect

[Christian Tramnitz]

Ross Philipson schrieb:
> Hi folks,
> 
> This sounds a lot like an issue I am working on right now. A recent 
> change to the vtd code changed the way the 1 to 1 page mapping for dom0 
> is done (look at intel_iommu_domain_init() for dom0). The change was to 
> query the e820 map for memory regions that were usable RAM and only map 
> those into the iommus for dom0. Thus reserved regions were no longer 
> mapped in. When the RMRR values do not report the correct values for 
> required 1 to 1 reserved memory regions, the needed regions do not get 
> mapped in. Thus device DMA to these regions causes vtd faults and in out 
> case a hang. It is indeed a BIOS bug; perhaps this is the same issue you 
> are seeing?
> 
> Anyway, I am working on a workaround that I will be pushing upstream 
> this week. A parameter will allow you to include reserved memory regions 
> in the mappings. It is not ideal (less secure) but better than hanging 
> or crashing…


Hi Ross,

I tried your patch from you other post, but even with the new parameter 
enabled I still get the DMAR error messages and vt-d being disabled:

(XEN) Command line: dom0_mem=1024M iommu=1 iommu_include_reserved=1
...
(XEN) [VT-D]dmar.c:473: Host address width 39
(XEN) [VT-D]dmar.c:482: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:337: dmaru->address = fbfff000
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1b.0
(XEN) [VT-D]dmar.c:482: found ACPI_DMAR_DRHD
(XEN) [VT-D]dmar.c:337: dmaru->address = fbffe000
(XEN) [VT-D]dmar.c:294: found IOAPIC: bdf = f0:1f.7
(XEN) [VT-D]dmar.c:294: found IOAPIC: bdf = 0:13.0
(XEN) [VT-D]dmar.c:346: found INCLUDE_ALL
(XEN) [VT-D]dmar.c:486: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1d.0
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1d.1
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1d.2
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1d.7
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1a.0
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1a.1
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1a.2
(XEN) [VT-D]dmar.c:288: found endpoint: bdf = 0:1a.7
(XEN) [VT-D]dmar.c:486: found ACPI_DMAR_RMRR
(XEN) [VT-D]dmar.c:376: RMRR error: base_addr bf7dc000 end_address bf7dbfff
(XEN) Failed to parse ACPI DMAR.  Disabling VT-d.


Best regards,
    Christian