From: Fengzhe Zhang <fengzhe.zhang@intel.com>
To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: xen-devel <xen-devel@lists.xensource.com>,
"Dong, Eddie" <eddie.dong@intel.com>,
"Li, Xin" <xin.li@intel.com>
Subject: Re: [PATCH] iomem: Prevent Dom0 pci bus from allocating RAM as I/O space
Date: Sun, 20 Feb 2011 22:14:14 +0800 [thread overview]
Message-ID: <4D6121B6.2060109@intel.com> (raw)
In-Reply-To: <20110218220558.GA18213@dumpdata.com>
On 2011/2/19 6:05, Konrad Rzeszutek Wilk wrote:
> On Wed, Feb 16, 2011 at 10:26:20PM +0800, Zhang, Fengzhe wrote:
>> iomem: Prevent Dom0 pci bus from allocating RAM as I/O space
>>
>> In Dom0, pci bus dynamically allocates I/O address resources from memory hole within 4GB physical space, which can be RAM space not allocated to Dom0. This patch set physical RAM space to be unusable in Dom0 E820 map if they are not owned by Dom0 to prevent them from being misused as I/O address space. Dom0 is assumed to look for MMIO space only below 4GB. If this assumption is broken, additional fixes are required.
>
> So I am coming back to your patch trying to understand how it makes
> the intel-agp crash go away. What this patch in effect does is
> inhibit ioremap from inadvertly mapping System RAM, which it could do
> before this - b/c it considered the "zapped" System RAM (so e820->size = 0)
> as gap.
>
> But looking at the intel-agp.c code works, it seems like it wouldn't
> really care about it initially. It figures out where to physical GTT is by
> poking the "intel_private.registers+I810_PGETBL_CTL". Earlier we also make a
> call in "agp_intel_probe" to do "pci_assign_resource(pdev, 0)" which
> irregardless of this patch or not, would work (since the BARs don't change
> with this patch? or do they?).
>
> The ioremap which is done in "intel_i9xx_setup_flush" would now potentially
> _not_ work since the region the GTT might be in the RAM region, and ioremap
> would consult the e820 table and find it is "UNUSABLE".. We then would
> continue on without the flush page with "can't ioremap flush page - no chipset flushing"
intel_i9xx_setup_flush checks IFP(Intel Flush Page) BAR is 0 on machine
bootup and calls pci_bus_alloc_resource to get an iommu page.
pci_bus_alloc_resource allocates io address resource from the largest
hole it finds in e820 map. Before patching, the hole would be from the
top of dom0 RAM to ACPI space, which overlaps with real RAM. After
patching, the hole would no longer overlap with any real RAM.
The following is the E820 map on the test machine with 4GB memory:
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 0000000000097c00 (usable)
(XEN) 0000000000097c00 - 00000000000a0000 (reserved)
(XEN) 00000000000e8000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000defafe00 (usable)
(XEN) 00000000defafe00 - 00000000defb1ea0 (ACPI NVS)
(XEN) 00000000defb1ea0 - 00000000e0000000 (reserved)
(XEN) 00000000f4000000 - 00000000f8000000 (reserved)
(XEN) 00000000fec00000 - 00000000fed40000 (reserved)
(XEN) 00000000fed45000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 000000011c000000 (usable)
The following is the E820 map seen in Dom0 before patching:
(Dom0 assigned with 1GB mem)
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: 0000000000000000 - 0000000000097c00 (usable)
[ 0.000000] Xen: 0000000000097c00 - 0000000000100000 (reserved)
[ 0.000000] Xen: 0000000000100000 - 0000000040000000 (usable)
[ 0.000000] Xen: 00000000defafe00 - 00000000defb1ea0 (ACPI NVS)
[ 0.000000] Xen: 00000000defb1ea0 - 00000000e0000000 (reserved)
[ 0.000000] Xen: 00000000f4000000 - 00000000f8000000 (reserved)
[ 0.000000] Xen: 00000000fec00000 - 00000000fed40000 (reserved)
[ 0.000000] Xen: 00000000fed45000 - 0000000100000000 (reserved)
[ 0.000000] Xen: 0000000100000000 - 00000001bafaf000 (usable)
The following is the E820 map seen in Dom0 after patching:
(Dom0 assigned with 1GB mem)
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: 0000000000000000 - 0000000000097c00 (usable)
[ 0.000000] Xen: 0000000000097c00 - 0000000000100000 (reserved)
[ 0.000000] Xen: 0000000000100000 - 0000000040000000 (usable)
[ 0.000000] Xen: 0000000040000000 - 00000000defafe00 (unusable)
[ 0.000000] Xen: 00000000defafe00 - 00000000defb1ea0 (ACPI NVS)
[ 0.000000] Xen: 00000000defb1ea0 - 00000000e0000000 (reserved)
[ 0.000000] Xen: 00000000f4000000 - 00000000f8000000 (reserved)
[ 0.000000] Xen: 00000000fec00000 - 00000000fed40000 (reserved)
[ 0.000000] Xen: 00000000fed45000 - 0000000100000000 (reserved)
[ 0.000000] Xen: 0000000100000000 - 00000001bafaf000 (usable)
The following is the /proc/iomem layout before patching:
(Dom0 assigned with 1GB mem)
00000000-00000fff : System RAM
00001000-00005fff : reserved
00006000-00097bff : System RAM
00097c00-000fffff : reserved
00100000-7fffffff : System RAM
01000000-01513391 : Kernel code
01513392-018713d7 : Kernel data
018fa000-019c9483 : Kernel bss
40000000-401fffff : PCI Bus 0000:01
40200000-403fffff : PCI Bus 0000:01
40400000-405fffff : PCI Bus 0000:02
40600000-407fffff : PCI Bus 0000:02
40800000-408fffff : PCI Bus 0000:03
40800000-408fffff : 0000:03:04.0
40900000-40900fff : Intel Flush Page //notice this line
defafe00-defb1e9f : ACPI Non-volatile Storage
defb1ea0-dfffffff : reserved
e0000000-efffffff : 0000:00:02.0
f0000000-f01fffff : PCI Bus 0000:03
f0000000-f00fffff : 0000:03:04.0
f0000000-f00fffff : e100
f0100000-f0100fff : 0000:03:04.0
f0100000-f0100fff : e100
f0200000-f02fffff : 0000:00:02.0
f0300000-f037ffff : 0000:00:02.0
f0380000-f03fffff : 0000:00:02.1
f0400000-f041ffff : 0000:00:19.0
f0400000-f041ffff : e1000e
f0420000-f0423fff : 0000:00:1b.0
f0420000-f0423fff : ICH HD audio
f0424000-f0424fff : 0000:00:03.3
f0425000-f0425fff : 0000:00:19.0
f0425000-f0425fff : e1000e
f0426000-f04267ff : 0000:00:1f.2
f0426000-f04267ff : ahci
f0426800-f0426bff : 0000:00:1a.7
f0426800-f0426bff : ehci_hcd
f0426c00-f0426fff : 0000:00:1d.7
f0426c00-f0426fff : ehci_hcd
f0427100-f042710f : 0000:00:03.0
f4000000-f7ffffff : PCI MMCONFIG 0 [00-3f]
f4000000-f7ffffff : reserved
f4000000-f7ffffff : pnp 00:10
fec00000-fed3ffff : reserved
fec00000-fec00fff : IOAPIC 0
fec01000-fecfffff : pnp 00:10
fed00000-fed003ff : HPET 0
fed00400-fed3ffff : pnp 00:10
fed45000-ffffffff : reserved
fed45000-ffffffff : pnp 00:10
100000000-17afaefff : System RAM
17afaf000-17bffffff : RAM buffer
The following is the /proc/iomem layout after patching:
(Dom0 assigned with 1GB mem)
00000000-00000fff : System RAM
00001000-00005fff : reserved
00006000-00097bff : System RAM
00097c00-000fffff : reserved
00100000-3fffffff : System RAM
01000000-015134f1 : Kernel code
015134f2-018713d7 : Kernel data
018fa000-019c9483 : Kernel bss
40000000-defafdff : Unusable memory
defafe00-defb1e9f : ACPI Non-volatile Storage
defb1ea0-dfffffff : reserved
e0000000-efffffff : 0000:00:02.0
f0000000-f01fffff : PCI Bus 0000:03
f0000000-f00fffff : 0000:03:04.0
f0000000-f00fffff : e100
f0100000-f0100fff : 0000:03:04.0
f0100000-f0100fff : e100
f0200000-f02fffff : 0000:00:02.0
f0300000-f037ffff : 0000:00:02.0
f0380000-f03fffff : 0000:00:02.1
f0400000-f041ffff : 0000:00:19.0
f0400000-f041ffff : e1000e
f0420000-f0423fff : 0000:00:1b.0
f0420000-f0423fff : ICH HD audio
f0424000-f0424fff : 0000:00:03.3
f0425000-f0425fff : 0000:00:19.0
f0425000-f0425fff : e1000e
f0426000-f04267ff : 0000:00:1f.2
f0426000-f04267ff : ahci
f0426800-f0426bff : 0000:00:1a.7
f0426800-f0426bff : ehci_hcd
f0426c00-f0426fff : 0000:00:1d.7
f0426c00-f0426fff : ehci_hcd
f0427100-f042710f : 0000:00:03.0
f0428000-f0428fff : Intel Flush Page //notice this line
f0500000-f06fffff : PCI Bus 0000:01
f0700000-f08fffff : PCI Bus 0000:01
f0900000-f0afffff : PCI Bus 0000:02
f0b00000-f0cfffff : PCI Bus 0000:02
f0d00000-f0dfffff : PCI Bus 0000:03
f0d00000-f0dfffff : 0000:03:04.0
f4000000-f7ffffff : PCI MMCONFIG 0 [00-3f]
f4000000-f7ffffff : reserved
f4000000-f7ffffff : pnp 00:10
fec00000-fed3ffff : reserved
fec00000-fec00fff : IOAPIC 0
fec01000-fecfffff : pnp 00:10
fed00000-fed003ff : HPET 0
fed00400-fed3ffff : pnp 00:10
fed45000-ffffffff : reserved
fed45000-ffffffff : pnp 00:10
100000000-1bafaefff : System RAM
1bafaf000-1bbffffff : RAM buffer
-Fengzhe
>
> Is that really what this patch achieves? How is this related
> to the igb driver?
>
> Can you provide a serial output of before this patch, and after this
> patch? I am really curious to see how the intel agp functions.
>
>
>>
>> Signed-off-by: Fengzhe Zhang<fengzhe.zhang@intel.com>
>>
>> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
>> index 1a1934a..f1a3896 100644
>> --- a/arch/x86/xen/setup.c
>> +++ b/arch/x86/xen/setup.c
>> @@ -189,6 +189,16 @@ char * __init xen_memory_setup(void)
>> end -= delta;
>>
>> extra_pages += PFN_DOWN(delta);
>> +
>> + /*
>> + * Set RAM below 4GB that are not owned by Dom0 to be unusable.
>> + * This prevents RAM-backed address space from being used as
>> + * I/O address in Dom0. Dom0 is assumed to look for MMIO
>> + * space only below 4GB. If this assumption is broken, additional
>> + * fixes are required.
>> + */
>> + if (delta&& end< 0x100000000UL)
>> + e820_add_region(end, delta, E820_UNUSABLE);
>> }
>>
>> if (map[i].size> 0&& end> xen_extra_mem_start)
>
>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
next prev parent reply other threads:[~2011-02-20 14:14 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-02-16 14:26 [PATCH] iomem: Prevent Dom0 pci bus from allocating RAM as I/O space Zhang, Fengzhe
2011-02-16 15:06 ` Konrad Rzeszutek Wilk
2011-02-16 15:20 ` Konrad Rzeszutek Wilk
2011-02-16 15:47 ` Konrad Rzeszutek Wilk
2011-02-17 5:32 ` Fengzhe Zhang
2011-02-17 14:23 ` [PATCH] iomem: Prevent Dom0 pci bus from allocating RAM as I/O space in 2.6.32.27 tree Konrad Rzeszutek Wilk
2011-02-17 15:07 ` Li, Xin
2011-02-17 15:35 ` Konrad Rzeszutek Wilk
2011-02-18 15:16 ` [PATCH] iomem: Prevent Dom0 pci bus from allocating RAM as I/O space Konrad Rzeszutek Wilk
2011-02-20 13:58 ` Fengzhe Zhang
2011-02-17 4:11 ` Fengzhe Zhang
2011-02-18 22:05 ` Konrad Rzeszutek Wilk
2011-02-20 14:14 ` Fengzhe Zhang [this message]
2011-02-21 10:35 ` Li, Xin
2011-02-21 19:20 ` Konrad Rzeszutek Wilk
2011-02-22 4:55 ` Li, Xin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4D6121B6.2060109@intel.com \
--to=fengzhe.zhang@intel.com \
--cc=eddie.dong@intel.com \
--cc=konrad.wilk@oracle.com \
--cc=xen-devel@lists.xensource.com \
--cc=xin.li@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.