[PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[Konrad Rzeszutek Wilk]

In a couple of days I thinking to ask Linus to pull this branch:

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git stable/bug-fixes-for-rc5

Which as two fixes that fix a bootup (Linux can't boot under Xen
at all) regression introduced by "x86-64, mm: Put early page table high"
(git commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e). Stefano
and Yinghai have been working on patches fixing this regression
when the patch was still in x86/mm-core before the 2.6.39 merge window openned.

But we haven't come up with an acceptable general solution yet, so this
patchset provides a workaround for the problem. Peter, Yinghai - what would be
the best forum/email/conference to hammer out a general solution for this?

Currently, there are couple of ways of fixing this:
 - use pvops hooks: http://marc.info/?i=1302607192-21355-2-git-send-email-stefano.stabellini@eu.citrix.com
 - have a workaround in Xen MMU's early bootup code (which is what these
   two patches to this email have).
 - or remove the patch introducing the regression altogether.

Foremost important is to fix the regression, and attached patches 
achieve that. I want to remove this workaround patch when we
hammer out more appropriate semantics for the page table creation - but
that will take some time and the runway to do that in 2.6.39 is gone.

Konrad Rzeszutek Wilk (1):
      xen/mmu: Add workaround "x86-64, mm: Put early page table high"

Stefano Stabellini (1):
      xen: mask_rw_pte mark RO all pagetable pages up to pgt_buf_top

 arch/x86/xen/mmu.c |  125 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 124 insertions(+), 1 deletions(-)

[PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Konrad Rzeszutek Wilk]

As a consequence of the commit:

commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
Author: Yinghai Lu <yinghai@kernel.org>
Date:   Fri Dec 17 16:58:28 2010 -0800

    x86-64, mm: Put early page table high

it causes the Linux kernel to crash under Xen:

mapping kernel into physical memory
Xen: setup ISA identity maps
about to get started...
(XEN) mm.c:2466:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn b1d89 (pfn bacf7)
(XEN) mm.c:3027:d0 Error while pinning mfn b1d89
(XEN) traps.c:481:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 0 (vcpu#0) crashed on cpu#0:
...

The reason is that at some point init_memory_mapping is going to reach
the pagetable pages area and map those pages too (mapping them as normal
memory that falls in the range of addresses passed to init_memory_mapping
as argument). Some of those pages are already pagetable pages (they are
in the range pgt_buf_start-pgt_buf_end) therefore they are going to be
mapped RO and everything is fine.
Some of these pages are not pagetable pages yet (they fall in the range
pgt_buf_end-pgt_buf_top; for example the page at pgt_buf_end) so they
are going to be mapped RW.  When these pages become pagetable pages and
are hooked into the pagetable, xen will find that the guest has already
a RW mapping of them somewhere and fail the operation.
The reason Xen requires pagetables to be RO is that the hypervisor needs
to verify that the pagetables are valid before using them. The validation
operations are called "pinning" (more details in arch/x86/xen/mmu.c).

In order to fix the issue we mark all the pages in the entire range
pgt_buf_start-pgt_buf_top as RO, however when the pagetable allocation
is completed only the range pgt_buf_start-pgt_buf_end is reserved by
init_memory_mapping. Hence the kernel is going to crash as soon as one
of the pages in the range pgt_buf_end-pgt_buf_top is reused (b/c those
ranges are RO).

For this reason, this function is introduced which is called _after_
the init_memory_mapping has completed (in a perfect world we would
call this function from init_memory_mapping, but lets ignore that).

Because we are called _after_ init_memory_mapping the pgt_buf_[start,
end,top] have all changed to new values (b/c another init_memory_mapping
is called). Hence, the first time we enter this function, we save
away the pgt_buf_start value and update the pgt_buf_[end,top].

When we detect that the "old" pgt_buf_start through pgt_buf_end
PFNs have been reserved (so memblock_x86_reserve_range has been called),
we immediately set out to RW the "old" pgt_buf_end through pgt_buf_top.

And then we update those "old" pgt_buf_[end|top] with the new ones
so that we can redo this on the next pagetable.

Reviewed-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
[v1: Updated with Jeremy's comments]
[v2: Added the crash output]
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/xen/mmu.c |  123 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 123 insertions(+), 0 deletions(-)

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index aef7af9..1bca25f 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1463,6 +1463,119 @@ static int xen_pgd_alloc(struct mm_struct *mm)
 	return ret;
 }
 
+#ifdef CONFIG_X86_64
+static __initdata u64 __last_pgt_set_rw = 0;
+static __initdata u64 __pgt_buf_start = 0;
+static __initdata u64 __pgt_buf_end = 0;
+static __initdata u64 __pgt_buf_top = 0;
+/*
+ * As a consequence of the commit:
+ * 
+ * commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
+ * Author: Yinghai Lu <yinghai@kernel.org>
+ * Date:   Fri Dec 17 16:58:28 2010 -0800
+ * 
+ *     x86-64, mm: Put early page table high
+ * 
+ * at some point init_memory_mapping is going to reach the pagetable pages
+ * area and map those pages too (mapping them as normal memory that falls
+ * in the range of addresses passed to init_memory_mapping as argument).
+ * Some of those pages are already pagetable pages (they are in the range
+ * pgt_buf_start-pgt_buf_end) therefore they are going to be mapped RO and
+ * everything is fine.
+ * Some of these pages are not pagetable pages yet (they fall in the range
+ * pgt_buf_end-pgt_buf_top; for example the page at pgt_buf_end) so they
+ * are going to be mapped RW.  When these pages become pagetable pages and
+ * are hooked into the pagetable, xen will find that the guest has already
+ * a RW mapping of them somewhere and fail the operation.
+ * The reason Xen requires pagetables to be RO is that the hypervisor needs
+ * to verify that the pagetables are valid before using them. The validation
+ * operations are called "pinning".
+ * 
+ * In order to fix the issue we mark all the pages in the entire range
+ * pgt_buf_start-pgt_buf_top as RO, however when the pagetable allocation
+ * is completed only the range pgt_buf_start-pgt_buf_end is reserved by
+ * init_memory_mapping. Hence the kernel is going to crash as soon as one
+ * of the pages in the range pgt_buf_end-pgt_buf_top is reused (b/c those
+ * ranges are RO).
+ * 
+ * For this reason, 'mark_rw_past_pgt' is introduced which is called _after_
+ * the init_memory_mapping has completed (in a perfect world we would
+ * call this function from init_memory_mapping, but lets ignore that).
+ * 
+ * Because we are called _after_ init_memory_mapping the pgt_buf_[start,
+ * end,top] have all changed to new values (b/c init_memory_mapping
+ * is called and setting up another new page-table). Hence, the first time
+ * we enter this function, we save away the pgt_buf_start value and update
+ * the pgt_buf_[end,top].
+ * 
+ * When we detect that the "old" pgt_buf_start through pgt_buf_end
+ * PFNs have been reserved (so memblock_x86_reserve_range has been called),
+ * we immediately set out to RW the "old" pgt_buf_end through pgt_buf_top.
+ * 
+ * And then we update those "old" pgt_buf_[end|top] with the new ones
+ * so that we can redo this on the next pagetable.
+ */
+static __init void mark_rw_past_pgt(void) {
+
+	if (pgt_buf_end > pgt_buf_start) {
+		u64 addr, size;
+
+		/* Save it away. */
+		if (!__pgt_buf_start) {
+			__pgt_buf_start = pgt_buf_start;
+			__pgt_buf_end = pgt_buf_end;
+			__pgt_buf_top = pgt_buf_top;
+			return;
+		}
+		/* If we get the range that starts at __pgt_buf_end that means
+		 * the range is reserved, and that in 'init_memory_mapping'
+		 * the 'memblock_x86_reserve_range' has been called with the
+		 * outdated __pgt_buf_start, __pgt_buf_end (the "new"
+		 * pgt_buf_[start|end|top] refer now to a new pagetable.
+		 * Note: we are called _after_ the pgt_buf_[..] have been
+		 * updated.*/
+
+		addr = memblock_x86_find_in_range_size(PFN_PHYS(__pgt_buf_start),
+						       &size, PAGE_SIZE);
+
+		/* Still not reserved, meaning 'memblock_x86_reserve_range'
+		 * hasn't been called yet. Update the _end and _top.*/
+		if (addr == PFN_PHYS(__pgt_buf_start)) {
+			__pgt_buf_end = pgt_buf_end;
+			__pgt_buf_top = pgt_buf_top;
+			return;
+		}
+
+		/* OK, the area is reserved, meaning it is time for us to
+		 * set RW for the old end->top PFNs. */
+
+		/* ..unless we had already done this. */
+		if (__pgt_buf_end == __last_pgt_set_rw)
+			return;
+
+		addr = PFN_PHYS(__pgt_buf_end);
+		
+		/* set as RW the rest */
+		printk(KERN_DEBUG "xen: setting RW the range %llx - %llx\n",
+			PFN_PHYS(__pgt_buf_end), PFN_PHYS(__pgt_buf_top));
+		
+		while (addr < PFN_PHYS(__pgt_buf_top)) {
+			make_lowmem_page_readwrite(__va(addr));
+			addr += PAGE_SIZE;
+		}
+		/* And update everything so that we are ready for the next
+		 * pagetable (the one created for regions past 4GB) */
+		__last_pgt_set_rw = __pgt_buf_end;
+		__pgt_buf_start = pgt_buf_start;
+		__pgt_buf_end = pgt_buf_end;
+		__pgt_buf_top = pgt_buf_top;
+	}
+	return;
+}
+#else
+static __init void mark_rw_past_pgt(void) { }
+#endif
 static void xen_pgd_free(struct mm_struct *mm, pgd_t *pgd)
 {
 #ifdef CONFIG_X86_64
@@ -1489,6 +1602,14 @@ static __init pte_t mask_rw_pte(pte_t *ptep, pte_t pte)
 	unsigned long pfn = pte_pfn(pte);
 
 	/*
+	 * A bit of optimization. We do not need to call the workaround
+	 * when xen_set_pte_init is called with a PTE with 0 as PFN.
+	 * That is b/c the pagetable at that point are just being populated
+	 * with empty values and we can save some cycles by not calling
+	 * the 'memblock' code.*/
+	if (pfn)
+		mark_rw_past_pgt();
+	/*
 	 * If the new pfn is within the range of the newly allocated
 	 * kernel pagetable, and it isn't being mapped into an
 	 * early_ioremap fixmap slot as a freshly allocated page, make sure
@@ -1997,6 +2118,8 @@ __init void xen_ident_map_ISA(void)
 
 static __init void xen_post_allocator_init(void)
 {
+	mark_rw_past_pgt();
+
 #ifdef CONFIG_XEN_DEBUG
 	pv_mmu_ops.make_pte = PV_CALLEE_SAVE(xen_make_pte_debug);
 #endif
-- 
1.7.1

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Daniel Kiper]

On Mon, May 02, 2011 at 01:22:21PM -0400, Konrad Rzeszutek Wilk wrote:
> As a consequence of the commit:
>
> commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> Author: Yinghai Lu <yinghai@kernel.org>
> Date:   Fri Dec 17 16:58:28 2010 -0800
>
>     x86-64, mm: Put early page table high
>
> it causes the Linux kernel to crash under Xen:
>
> mapping kernel into physical memory
> Xen: setup ISA identity maps
> about to get started...
> (XEN) mm.c:2466:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn b1d89 (pfn bacf7)
> (XEN) mm.c:3027:d0 Error while pinning mfn b1d89
> (XEN) traps.c:481:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
> (XEN) domain_crash_sync called from entry.S
> (XEN) Domain 0 (vcpu#0) crashed on cpu#0:
> ...

I was hit by this bug when I was working on memory hotplug.
After some investigation I found myself above mentioned patch
as a guilty and later I discovered that you are working on that
issue. I have tested your patch and discoverd some issues with it.
First of all it has compilation issues on gcc version 4.1.2 20061115
(prerelease) (Debian 4.1.1-21). Details below.

Additionlly, I think that your patch does not work as you expected.
I found that git commit 24bdb0b62cc82120924762ae6bc85afc8c3f2b26
(xen: do not create the extra e820 region at an addr lower than 4G)
do this work (to some extent). When this patch is removed domU
is crashing with following error:

(early) Linux version 2.6.39-rc5-x86_64.xenU.all.r0+ (root@dev-00) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #5 SMP Tue May 3 01:43:26 CEST 2011
(early) Command line: root=/dev/xvda debug earlyprintk=xen noapic nolapic console=hvc0
(early) ACPI in unprivileged domain disabled
(early) released 0 pages of unused memory
(early) Set 0 page(s) to 1-1 mapping.
(early) BIOS-provided physical RAM map:
(early)  Xen: 0000000000000000 - 00000000000a0000 (usable)
(early)  Xen: 00000000000a0000 - 0000000000100000 (reserved)
(early)  Xen: 0000000000100000 - 0000000026000000 (usable)
(early) bootconsole [xenboot0] enabled
(early) NX (Execute Disable) protection: active
(early) DMI not present or invalid.
(early) e820 update range: 0000000000000000 - 0000000000010000 (early) (usable)(early)  ==> (early) (reserved)(early)
(early) e820 remove range: 00000000000a0000 - 0000000000100000 (early) (usable)(early)
(early) No AGP bridge found
(early) last_pfn = 0x26000 max_arch_pfn = 0x400000000
(early) initial memory mapped : 0 - 01693000
(early) Base memory trampoline at [ffff88000009e000] 9e000 size 8192
(early) init_memory_mapping: 0000000000000000-0000000026000000
(early)  0000000000 - 0026000000 page 4k
(early) kernel direct mapping tables up to 26000000 @ 256ce000-25800000
(early) BUG: unable to handle kernel (early) NULL pointer dereference(early)  at           (null)
(early) IP:(early)  [<ffffffff814a33f2>] find_range_array+0x4e/0x57
(early) PGD 0 (early)
(early) Oops: 0003 [#1] (early) SMP (early)
(early) last sysfs file:
(early) CPU 0 (early)
(early) Modules linked in:(early)
(early)
(early) Pid: 0, comm: swapper Not tainted 2.6.39-rc5-x86_64.xenU.all.r0+ #5(early)   (early)
(early) RIP: e030:[<ffffffff814a33f2>] (early)  [<ffffffff814a33f2>] find_range_array+0x4e/0x57
(early) RSP: e02b:ffffffff81427e58  EFLAGS: 00010046
(early) RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 00000000000000e0
(early) RDX: ffff8800257fff20 RSI: 00000000257fff20 RDI: ffff8800257fff20
(early) RBP: ffffffff81427e68 R08: 0000000000000005 R09: 0000000000000050
(early) R10: 0000000000000005 R11: 0000000025800000 R12: ffffffff814bd000
(early) R13: 0000000000000000 R14: 000000000000000e R15: 0000000000001000
(early) FS:  0000000000000000(0000) GS:ffffffff8147f000(0000) knlGS:0000000000000000
(early) CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
(early) CR2: 0000000000000000 CR3: 0000000001441000 CR4: 0000000000002660
(early) DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
(early) DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
(early) Process swapper (pid: 0, threadinfo ffffffff81426000, task ffffffff81449020)
(early) Stack:
(early)  ffffffff81427e88(early)  0000000000000000(early)  ffffffff81427ea8(early)  ffffffff814a343d(early)
(early)  0000000000000100(early)  0000000026000000(early)  ffffffff814bd000(early)  ffffffffffffffff(early)
(early)  0000000000000000(early)  0000000000000000(early)  ffffffff81427eb8(early)  ffffffff814a3577(early)
(early) Call Trace:
(early)  [<ffffffff814a343d>] __memblock_x86_memory_in_range+0x42/0x171
(early)  [<ffffffff814a3577>] memblock_x86_memory_in_range+0xb/0xd
(early)  [<ffffffff81497348>] memblock_find_dma_reserve+0x15/0x3b
(early)  [<ffffffff81496b50>] setup_arch+0x721/0x7e5
(early)  [<ffffffff810065fd>] ? __raw_callee_save_xen_irq_disable+0x11/0x1e
(early)  [<ffffffff81493955>] start_kernel+0x8a/0x2db
(early)  [<ffffffff81493299>] x86_64_start_reservations+0x84/0x88
(early)  [<ffffffff814947d9>] xen_start_kernel+0x3e1/0x3e8
(early) Code: (early) 66 (early) 00 (early) 00 (early) 48 (early) 85 (early) c0 (early) 75 (early) 0c (early) 48 (early) c7 (early) c7 (early) 86 (early) 80 (early) 3a (early) 81 (early) e8 (early) 55 (early) 41 (early) b9 (early) ff (early) 48 (early) bf (early) 00 (early) 00 (early) 00 (early) 00 (early) 00 (early) 88 (early) ff (early) ff (early) 48 (early) 89 (early) d9 (early) 48 (early) 8d (early) 14 (early) 38 (early) 31 (early) c0 (early) fc (early) 48 (early) 89 (early) d7 (early) <f3> (early) aa (early) 48 (early) 89 (early) d0 (early) 5f (early) 5b (early) c9 (early) c3 (early) 55 (early) 48 (early) 89 (early) e5 (early) 41 (early) 57 (early) 49 (early) 89 (early) f7 (early) 49 (early) c1 (early) ef (early)
(early) RIP (early)  [<ffffffff814a33f2>] find_range_array+0x4e/0x57
(early)  RSP <ffffffff81427e58>
(early) CR2: 0000000000000000
(early) ---[ end trace 4eaa2a86a8e2da22 ]---
(early) Kernel panic - not syncing: Attempted to kill the idle task!
(early) Pid: 0, comm: swapper Tainted: G      D     2.6.39-rc5-x86_64.xenU.all.r0+ #5
(early) Call Trace:
(early)  [<ffffffff810375ed>] panic+0xbd/0x1c7
(early)  [<ffffffff810383c7>] ? printk+0x67/0x69
(early)  [<ffffffff811fd7b0>] ? account+0xe1/0xf0
(early)  [<ffffffff8103a641>] do_exit+0xb4/0x676
(early)  [<ffffffff81037b53>] ? spin_unlock_irqrestore+0x9/0xb
(early)  [<ffffffff81038c24>] ? kmsg_dump+0x4a/0xd9
(early)  [<ffffffff8100d11d>] oops_end+0xc1/0xc9
(early)  [<ffffffff810252b1>] no_context+0x1f5/0x204
(early)  [<ffffffff81025448>] __bad_area_nosemaphore+0x188/0x1ab
(early)  [<ffffffff810065df>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
(early)  [<ffffffff810254e1>] bad_area_nosemaphore+0xe/0x10
(early)  [<ffffffff81025991>] do_page_fault+0x18c/0x337
(early)  [<ffffffff810065df>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
(early)  [<ffffffff810065df>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
(early)  [<ffffffff810065df>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
(early)  [<ffffffff812eebd5>] page_fault+0x25/0x30
(early)  [<ffffffff814a33f2>] ? find_range_array+0x4e/0x57
(early)  [<ffffffff814a33ca>] ? find_range_array+0x26/0x57
(early)  [<ffffffff814a343d>] __memblock_x86_memory_in_range+0x42/0x171
(early)  [<ffffffff814a3577>] memblock_x86_memory_in_range+0xb/0xd
(early)  [<ffffffff81497348>] memblock_find_dma_reserve+0x15/0x3b
(early)  [<ffffffff81496b50>] setup_arch+0x721/0x7e5
(early)  [<ffffffff810065fd>] ? __raw_callee_save_xen_irq_disable+0x11/0x1e
(early)  [<ffffffff81493955>] start_kernel+0x8a/0x2db
(early)  [<ffffffff81493299>] x86_64_start_reservations+0x84/0x88
(early)  [<ffffffff814947d9>] xen_start_kernel+0x3e1/0x3e8

I think that (Stefano please confirm or not) this patch was prepared
as workaround for similar issues. However, I do not like this patch
because on systems with small amount of memory it leaves huge (to some
extent) hole between max_low_pfn and 4G. Additionally, it affects
memory hotplug a bit because it allocates memory starting from current
max_mfn. It also breaks memory hotplug on i386 (maybe also others
thinks, however, I could not confirm that). If it stay for some
reason it should be amended in follwing way:

#ifdef CONFIG_X86_32
        xen_extra_mem_start = mem_end;
#else
        xen_extra_mem_start = max((1ULL << 32), mem_end);
#endif

Regarding comment for this patch it should be mentioned that without this
patch e820_end_of_low_ram_pfn() is not broken. It is not called simply.

Last but least. I found that memory sizes below and including exactly 1 GiB and
exactly 2 GiB, 3 GiB (maybe higher, i.e. 4 GiB, 5 GiB, ...; I was not able to test
them because I do not have sufficient memory) are magic. It means that if memory
is set with those sizes everything is working good (without 4b239f458c229de044d6905c2b0f9fe16ed9e01e
and 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 applied). It means that domU
should be tested with sizes which are not power of two nor multiple of that.

> The reason is that at some point init_memory_mapping is going to reach
> the pagetable pages area and map those pages too (mapping them as normal
> memory that falls in the range of addresses passed to init_memory_mapping
> as argument). Some of those pages are already pagetable pages (they are
> in the range pgt_buf_start-pgt_buf_end) therefore they are going to be
> mapped RO and everything is fine.
> Some of these pages are not pagetable pages yet (they fall in the range
> pgt_buf_end-pgt_buf_top; for example the page at pgt_buf_end) so they
> are going to be mapped RW.  When these pages become pagetable pages and
> are hooked into the pagetable, xen will find that the guest has already
> a RW mapping of them somewhere and fail the operation.
> The reason Xen requires pagetables to be RO is that the hypervisor needs
> to verify that the pagetables are valid before using them. The validation
> operations are called "pinning" (more details in arch/x86/xen/mmu.c).
> 
> In order to fix the issue we mark all the pages in the entire range
> pgt_buf_start-pgt_buf_top as RO, however when the pagetable allocation
> is completed only the range pgt_buf_start-pgt_buf_end is reserved by
> init_memory_mapping. Hence the kernel is going to crash as soon as one
> of the pages in the range pgt_buf_end-pgt_buf_top is reused (b/c those
> ranges are RO).
> 
> For this reason, this function is introduced which is called _after_
> the init_memory_mapping has completed (in a perfect world we would
> call this function from init_memory_mapping, but lets ignore that).
> 
> Because we are called _after_ init_memory_mapping the pgt_buf_[start,
> end,top] have all changed to new values (b/c another init_memory_mapping
> is called). Hence, the first time we enter this function, we save
> away the pgt_buf_start value and update the pgt_buf_[end,top].
> 
> When we detect that the "old" pgt_buf_start through pgt_buf_end
> PFNs have been reserved (so memblock_x86_reserve_range has been called),
> we immediately set out to RW the "old" pgt_buf_end through pgt_buf_top.
> 
> And then we update those "old" pgt_buf_[end|top] with the new ones
> so that we can redo this on the next pagetable.
> 
> Reviewed-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> [v1: Updated with Jeremy's comments]
> [v2: Added the crash output]
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  arch/x86/xen/mmu.c |  123 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 123 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> index aef7af9..1bca25f 100644
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -1463,6 +1463,119 @@ static int xen_pgd_alloc(struct mm_struct *mm)
>  	return ret;
>  }
>  
> +#ifdef CONFIG_X86_64
> +static __initdata u64 __last_pgt_set_rw = 0;
> +static __initdata u64 __pgt_buf_start = 0;
> +static __initdata u64 __pgt_buf_end = 0;
> +static __initdata u64 __pgt_buf_top = 0;

Please look into include/linux/init.h for proper
usage of __init macros. It should be changed to

static u64 __last_pgt_set_rw __initdata = 0;
...
...

Additionally,

static const struct pv_mmu_ops xen_mmu_ops __initdata = {

should be changed to:

static const struct pv_mmu_ops xen_mmu_ops __initconst = {

It is not in your patch, however, it conflicts
with your definitions.

> +/*
> + * As a consequence of the commit:
> + * 
> + * commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> + * Author: Yinghai Lu <yinghai@kernel.org>
> + * Date:   Fri Dec 17 16:58:28 2010 -0800
> + * 
> + *     x86-64, mm: Put early page table high
> + * 
> + * at some point init_memory_mapping is going to reach the pagetable pages
> + * area and map those pages too (mapping them as normal memory that falls
> + * in the range of addresses passed to init_memory_mapping as argument).
> + * Some of those pages are already pagetable pages (they are in the range
> + * pgt_buf_start-pgt_buf_end) therefore they are going to be mapped RO and
> + * everything is fine.
> + * Some of these pages are not pagetable pages yet (they fall in the range
> + * pgt_buf_end-pgt_buf_top; for example the page at pgt_buf_end) so they
> + * are going to be mapped RW.  When these pages become pagetable pages and
> + * are hooked into the pagetable, xen will find that the guest has already
> + * a RW mapping of them somewhere and fail the operation.
> + * The reason Xen requires pagetables to be RO is that the hypervisor needs
> + * to verify that the pagetables are valid before using them. The validation
> + * operations are called "pinning".
> + * 
> + * In order to fix the issue we mark all the pages in the entire range
> + * pgt_buf_start-pgt_buf_top as RO, however when the pagetable allocation
> + * is completed only the range pgt_buf_start-pgt_buf_end is reserved by
> + * init_memory_mapping. Hence the kernel is going to crash as soon as one
> + * of the pages in the range pgt_buf_end-pgt_buf_top is reused (b/c those
> + * ranges are RO).
> + * 
> + * For this reason, 'mark_rw_past_pgt' is introduced which is called _after_
> + * the init_memory_mapping has completed (in a perfect world we would
> + * call this function from init_memory_mapping, but lets ignore that).
> + * 
> + * Because we are called _after_ init_memory_mapping the pgt_buf_[start,
> + * end,top] have all changed to new values (b/c init_memory_mapping
> + * is called and setting up another new page-table). Hence, the first time
> + * we enter this function, we save away the pgt_buf_start value and update
> + * the pgt_buf_[end,top].
> + * 
> + * When we detect that the "old" pgt_buf_start through pgt_buf_end
> + * PFNs have been reserved (so memblock_x86_reserve_range has been called),
> + * we immediately set out to RW the "old" pgt_buf_end through pgt_buf_top.
> + * 
> + * And then we update those "old" pgt_buf_[end|top] with the new ones
> + * so that we can redo this on the next pagetable.
> + */
> +static __init void mark_rw_past_pgt(void) {

Please look into include/linux/init.h. I found much more similar
mistakes in current Xen code. I will prepare relevant patch
shortly.

> +	if (pgt_buf_end > pgt_buf_start) {
> +		u64 addr, size;
> +
> +		/* Save it away. */
> +		if (!__pgt_buf_start) {
> +			__pgt_buf_start = pgt_buf_start;
> +			__pgt_buf_end = pgt_buf_end;
> +			__pgt_buf_top = pgt_buf_top;
> +			return;
> +		}
> +		/* If we get the range that starts at __pgt_buf_end that means
> +		 * the range is reserved, and that in 'init_memory_mapping'
> +		 * the 'memblock_x86_reserve_range' has been called with the
> +		 * outdated __pgt_buf_start, __pgt_buf_end (the "new"
> +		 * pgt_buf_[start|end|top] refer now to a new pagetable.
> +		 * Note: we are called _after_ the pgt_buf_[..] have been
> +		 * updated.*/
> +
> +		addr = memblock_x86_find_in_range_size(PFN_PHYS(__pgt_buf_start),
> +						       &size, PAGE_SIZE);
> +
> +		/* Still not reserved, meaning 'memblock_x86_reserve_range'
> +		 * hasn't been called yet. Update the _end and _top.*/
> +		if (addr == PFN_PHYS(__pgt_buf_start)) {
> +			__pgt_buf_end = pgt_buf_end;
> +			__pgt_buf_top = pgt_buf_top;
> +			return;
> +		}
> +
> +		/* OK, the area is reserved, meaning it is time for us to
> +		 * set RW for the old end->top PFNs. */
> +
> +		/* ..unless we had already done this. */
> +		if (__pgt_buf_end == __last_pgt_set_rw)
> +			return;
> +
> +		addr = PFN_PHYS(__pgt_buf_end);
> +		
> +		/* set as RW the rest */
> +		printk(KERN_DEBUG "xen: setting RW the range %llx - %llx\n",
> +			PFN_PHYS(__pgt_buf_end), PFN_PHYS(__pgt_buf_top));
> +		
> +		while (addr < PFN_PHYS(__pgt_buf_top)) {
> +			make_lowmem_page_readwrite(__va(addr));
> +			addr += PAGE_SIZE;
> +		}
> +		/* And update everything so that we are ready for the next
> +		 * pagetable (the one created for regions past 4GB) */
> +		__last_pgt_set_rw = __pgt_buf_end;
> +		__pgt_buf_start = pgt_buf_start;
> +		__pgt_buf_end = pgt_buf_end;
> +		__pgt_buf_top = pgt_buf_top;
> +	}
> +	return;

I think that this return is superfluous.

> +}
> +#else
> +static __init void mark_rw_past_pgt(void) { }

Dito.

> +#endif
>  static void xen_pgd_free(struct mm_struct *mm, pgd_t *pgd)
>  {
>  #ifdef CONFIG_X86_64
> @@ -1489,6 +1602,14 @@ static __init pte_t mask_rw_pte(pte_t *ptep, pte_t pte)
>  	unsigned long pfn = pte_pfn(pte);
>  
>  	/*
> +	 * A bit of optimization. We do not need to call the workaround
> +	 * when xen_set_pte_init is called with a PTE with 0 as PFN.
> +	 * That is b/c the pagetable at that point are just being populated
> +	 * with empty values and we can save some cycles by not calling
> +	 * the 'memblock' code.*/
> +	if (pfn)
> +		mark_rw_past_pgt();
> +	/*
>  	 * If the new pfn is within the range of the newly allocated
>  	 * kernel pagetable, and it isn't being mapped into an
>  	 * early_ioremap fixmap slot as a freshly allocated page, make sure
> @@ -1997,6 +2118,8 @@ __init void xen_ident_map_ISA(void)
>  
>  static __init void xen_post_allocator_init(void)

Dito.

Daniel

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Konrad Rzeszutek Wilk]

On Tue, May 03, 2011 at 02:55:27AM +0200, Daniel Kiper wrote:
> On Mon, May 02, 2011 at 01:22:21PM -0400, Konrad Rzeszutek Wilk wrote:
> > As a consequence of the commit:
> >
> > commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > Author: Yinghai Lu <yinghai@kernel.org>
> > Date:   Fri Dec 17 16:58:28 2010 -0800
> >
> >     x86-64, mm: Put early page table high
> >
> > it causes the Linux kernel to crash under Xen:
> >
> > mapping kernel into physical memory
> > Xen: setup ISA identity maps
> > about to get started...
> > (XEN) mm.c:2466:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn b1d89 (pfn bacf7)
> > (XEN) mm.c:3027:d0 Error while pinning mfn b1d89
> > (XEN) traps.c:481:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
> > (XEN) domain_crash_sync called from entry.S
> > (XEN) Domain 0 (vcpu#0) crashed on cpu#0:
> > ...
> 
> I was hit by this bug when I was working on memory hotplug.
> After some investigation I found myself above mentioned patch
> as a guilty and later I discovered that you are working on that
> issue. I have tested your patch and discoverd some issues with it.
> First of all it has compilation issues on gcc version 4.1.2 20061115
> (prerelease) (Debian 4.1.1-21). Details below.
> 
> Additionlly, I think that your patch does not work as you expected.
> I found that git commit 24bdb0b62cc82120924762ae6bc85afc8c3f2b26
> (xen: do not create the extra e820 region at an addr lower than 4G)
> do this work (to some extent). When this patch is removed domU
> is crashing with following error:

Which is "this patch" ? The 24bdb0b62cc82120924762ae6bc85afc8c3f2b26?

> 
> (early) Linux version 2.6.39-rc5-x86_64.xenU.all.r0+ (root@dev-00) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #5 SMP Tue May 3 01:43:26 CEST 2011
> (early) Command line: root=/dev/xvda debug earlyprintk=xen noapic nolapic console=hvc0
> (early) ACPI in unprivileged domain disabled
> (early) released 0 pages of unused memory
> (early) Set 0 page(s) to 1-1 mapping.
> (early) BIOS-provided physical RAM map:
> (early)  Xen: 0000000000000000 - 00000000000a0000 (usable)
> (early)  Xen: 00000000000a0000 - 0000000000100000 (reserved)
> (early)  Xen: 0000000000100000 - 0000000026000000 (usable)
> (early) bootconsole [xenboot0] enabled
> (early) NX (Execute Disable) protection: active
> (early) DMI not present or invalid.
> (early) e820 update range: 0000000000000000 - 0000000000010000 (early) (usable)(early)  ==> (early) (reserved)(early)
> (early) e820 remove range: 00000000000a0000 - 0000000000100000 (early) (usable)(early)
> (early) No AGP bridge found
> (early) last_pfn = 0x26000 max_arch_pfn = 0x400000000
> (early) initial memory mapped : 0 - 01693000
> (early) Base memory trampoline at [ffff88000009e000] 9e000 size 8192
> (early) init_memory_mapping: 0000000000000000-0000000026000000
> (early)  0000000000 - 0026000000 page 4k
> (early) kernel direct mapping tables up to 26000000 @ 256ce000-25800000
> (early) BUG: unable to handle kernel (early) NULL pointer dereference(early)  at           (null)
> (early) IP:(early)  [<ffffffff814a33f2>] find_range_array+0x4e/0x57
> (early) PGD 0 (early)
> (early) Oops: 0003 [#1] (early) SMP (early)
> (early) last sysfs file:
> (early) CPU 0 (early)
> (early) Modules linked in:(early)
> (early)
> (early) Pid: 0, comm: swapper Not tainted 2.6.39-rc5-x86_64.xenU.all.r0+ #5(early)   (early)
> (early) RIP: e030:[<ffffffff814a33f2>] (early)  [<ffffffff814a33f2>] find_range_array+0x4e/0x57
> (early) RSP: e02b:ffffffff81427e58  EFLAGS: 00010046
> (early) RAX: 0000000000000000 RBX: 00000000000000e0 RCX: 00000000000000e0
> (early) RDX: ffff8800257fff20 RSI: 00000000257fff20 RDI: ffff8800257fff20
> (early) RBP: ffffffff81427e68 R08: 0000000000000005 R09: 0000000000000050
> (early) R10: 0000000000000005 R11: 0000000025800000 R12: ffffffff814bd000
> (early) R13: 0000000000000000 R14: 000000000000000e R15: 0000000000001000
> (early) FS:  0000000000000000(0000) GS:ffffffff8147f000(0000) knlGS:0000000000000000
> (early) CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> (early) CR2: 0000000000000000 CR3: 0000000001441000 CR4: 0000000000002660
> (early) DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> (early) DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> (early) Process swapper (pid: 0, threadinfo ffffffff81426000, task ffffffff81449020)
> (early) Stack:
> (early)  ffffffff81427e88(early)  0000000000000000(early)  ffffffff81427ea8(early)  ffffffff814a343d(early)
> (early)  0000000000000100(early)  0000000026000000(early)  ffffffff814bd000(early)  ffffffffffffffff(early)
> (early)  0000000000000000(early)  0000000000000000(early)  ffffffff81427eb8(early)  ffffffff814a3577(early)
> (early) Call Trace:
> (early)  [<ffffffff814a343d>] __memblock_x86_memory_in_range+0x42/0x171
> (early)  [<ffffffff814a3577>] memblock_x86_memory_in_range+0xb/0xd
> (early)  [<ffffffff81497348>] memblock_find_dma_reserve+0x15/0x3b
> (early)  [<ffffffff81496b50>] setup_arch+0x721/0x7e5
> (early)  [<ffffffff810065fd>] ? __raw_callee_save_xen_irq_disable+0x11/0x1e
> (early)  [<ffffffff81493955>] start_kernel+0x8a/0x2db
> (early)  [<ffffffff81493299>] x86_64_start_reservations+0x84/0x88
> (early)  [<ffffffff814947d9>] xen_start_kernel+0x3e1/0x3e8
> (early) Code: (early) 66 (early) 00 (early) 00 (early) 48 (early) 85 (early) c0 (early) 75 (early) 0c (early) 48 (early) c7 (early) c7 (early) 86 (early) 80 (early) 3a (early) 81 (early) e8 (early) 55 (early) 41 (early) b9 (early) ff (early) 48 (early) bf (early) 00 (early) 00 (early) 00 (early) 00 (early) 00 (early) 88 (early) ff (early) ff (early) 48 (early) 89 (early) d9 (early) 48 (early) 8d (early) 14 (early) 38 (early) 31 (early) c0 (early) fc (early) 48 (early) 89 (early) d7 (early) <f3> (early) aa (early) 48 (early) 89 (early) d0 (early) 5f (early) 5b (early) c9 (early) c3 (early) 55 (early) 48 (early) 89 (early) e5 (early) 41 (early) 57 (early) 49 (early) 89 (early) f7 (early) 49 (early) c1 (early) ef (early)
> (early) RIP (early)  [<ffffffff814a33f2>] find_range_array+0x4e/0x57
> (early)  RSP <ffffffff81427e58>
> (early) CR2: 0000000000000000
> (early) ---[ end trace 4eaa2a86a8e2da22 ]---
> (early) Kernel panic - not syncing: Attempted to kill the idle task!
> (early) Pid: 0, comm: swapper Tainted: G      D     2.6.39-rc5-x86_64.xenU.all.r0+ #5
> (early) Call Trace:
> (early)  [<ffffffff810375ed>] panic+0xbd/0x1c7
> (early)  [<ffffffff810383c7>] ? printk+0x67/0x69
> (early)  [<ffffffff811fd7b0>] ? account+0xe1/0xf0
> (early)  [<ffffffff8103a641>] do_exit+0xb4/0x676
> (early)  [<ffffffff81037b53>] ? spin_unlock_irqrestore+0x9/0xb
> (early)  [<ffffffff81038c24>] ? kmsg_dump+0x4a/0xd9
> (early)  [<ffffffff8100d11d>] oops_end+0xc1/0xc9
> (early)  [<ffffffff810252b1>] no_context+0x1f5/0x204
> (early)  [<ffffffff81025448>] __bad_area_nosemaphore+0x188/0x1ab
> (early)  [<ffffffff810065df>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
> (early)  [<ffffffff810254e1>] bad_area_nosemaphore+0xe/0x10
> (early)  [<ffffffff81025991>] do_page_fault+0x18c/0x337
> (early)  [<ffffffff810065df>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
> (early)  [<ffffffff810065df>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
> (early)  [<ffffffff810065df>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
> (early)  [<ffffffff812eebd5>] page_fault+0x25/0x30
> (early)  [<ffffffff814a33f2>] ? find_range_array+0x4e/0x57
> (early)  [<ffffffff814a33ca>] ? find_range_array+0x26/0x57
> (early)  [<ffffffff814a343d>] __memblock_x86_memory_in_range+0x42/0x171
> (early)  [<ffffffff814a3577>] memblock_x86_memory_in_range+0xb/0xd
> (early)  [<ffffffff81497348>] memblock_find_dma_reserve+0x15/0x3b
> (early)  [<ffffffff81496b50>] setup_arch+0x721/0x7e5
> (early)  [<ffffffff810065fd>] ? __raw_callee_save_xen_irq_disable+0x11/0x1e
> (early)  [<ffffffff81493955>] start_kernel+0x8a/0x2db
> (early)  [<ffffffff81493299>] x86_64_start_reservations+0x84/0x88
> (early)  [<ffffffff814947d9>] xen_start_kernel+0x3e1/0x3e8
> 
> I think that (Stefano please confirm or not) this patch was prepared
> as workaround for similar issues. However, I do not like this patch
> because on systems with small amount of memory it leaves huge (to some
> extent) hole between max_low_pfn and 4G. Additionally, it affects
> memory hotplug a bit because it allocates memory starting from current
> max_mfn. It also breaks memory hotplug on i386 (maybe also others
> thinks, however, I could not confirm that). If it stay for some
> reason it should be amended in follwing way:
> 
> #ifdef CONFIG_X86_32
>         xen_extra_mem_start = mem_end;
> #else
>         xen_extra_mem_start = max((1ULL << 32), mem_end);
> #endif
> 
> Regarding comment for this patch it should be mentioned that without this
> patch e820_end_of_low_ram_pfn() is not broken. It is not called simply.
> 
> Last but least. I found that memory sizes below and including exactly 1 GiB and
> exactly 2 GiB, 3 GiB (maybe higher, i.e. 4 GiB, 5 GiB, ...; I was not able to test
> them because I do not have sufficient memory) are magic. It means that if memory
> is set with those sizes everything is working good (without 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> and 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 applied). It means that domU
> should be tested with sizes which are not power of two nor multiple of that.

Hmm, I thought I did test 1500M.

> > +#ifdef CONFIG_X86_64
> > +static __initdata u64 __last_pgt_set_rw = 0;
> > +static __initdata u64 __pgt_buf_start = 0;
> > +static __initdata u64 __pgt_buf_end = 0;
> > +static __initdata u64 __pgt_buf_top = 0;
> 
> Please look into include/linux/init.h for proper
> usage of __init macros. It should be changed to
> 
> static u64 __last_pgt_set_rw __initdata = 0;
> ...
> ...
> 
> Additionally,
> 
> static const struct pv_mmu_ops xen_mmu_ops __initdata = {
> 
> should be changed to:
> 
> static const struct pv_mmu_ops xen_mmu_ops __initconst = {
> 
> It is not in your patch, however, it conflicts
> with your definitions.

Ok. I am not that worried about the changes this patch brings. I hope
to have it removed in 2.6.40-ish time -frame.
> 
> > +/*
> > + * As a consequence of the commit:
> > + * 
> > + * commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > + * Author: Yinghai Lu <yinghai@kernel.org>
> > + * Date:   Fri Dec 17 16:58:28 2010 -0800
> > + * 
> > + *     x86-64, mm: Put early page table high
> > + * 
> > + * at some point init_memory_mapping is going to reach the pagetable pages
> > + * area and map those pages too (mapping them as normal memory that falls
> > + * in the range of addresses passed to init_memory_mapping as argument).
> > + * Some of those pages are already pagetable pages (they are in the range
> > + * pgt_buf_start-pgt_buf_end) therefore they are going to be mapped RO and
> > + * everything is fine.
> > + * Some of these pages are not pagetable pages yet (they fall in the range
> > + * pgt_buf_end-pgt_buf_top; for example the page at pgt_buf_end) so they
> > + * are going to be mapped RW.  When these pages become pagetable pages and
> > + * are hooked into the pagetable, xen will find that the guest has already
> > + * a RW mapping of them somewhere and fail the operation.
> > + * The reason Xen requires pagetables to be RO is that the hypervisor needs
> > + * to verify that the pagetables are valid before using them. The validation
> > + * operations are called "pinning".
> > + * 
> > + * In order to fix the issue we mark all the pages in the entire range
> > + * pgt_buf_start-pgt_buf_top as RO, however when the pagetable allocation
> > + * is completed only the range pgt_buf_start-pgt_buf_end is reserved by
> > + * init_memory_mapping. Hence the kernel is going to crash as soon as one
> > + * of the pages in the range pgt_buf_end-pgt_buf_top is reused (b/c those
> > + * ranges are RO).
> > + * 
> > + * For this reason, 'mark_rw_past_pgt' is introduced which is called _after_
> > + * the init_memory_mapping has completed (in a perfect world we would
> > + * call this function from init_memory_mapping, but lets ignore that).
> > + * 
> > + * Because we are called _after_ init_memory_mapping the pgt_buf_[start,
> > + * end,top] have all changed to new values (b/c init_memory_mapping
> > + * is called and setting up another new page-table). Hence, the first time
> > + * we enter this function, we save away the pgt_buf_start value and update
> > + * the pgt_buf_[end,top].
> > + * 
> > + * When we detect that the "old" pgt_buf_start through pgt_buf_end
> > + * PFNs have been reserved (so memblock_x86_reserve_range has been called),
> > + * we immediately set out to RW the "old" pgt_buf_end through pgt_buf_top.
> > + * 
> > + * And then we update those "old" pgt_buf_[end|top] with the new ones
> > + * so that we can redo this on the next pagetable.
> > + */
> > +static __init void mark_rw_past_pgt(void) {
> 
> Please look into include/linux/init.h. I found much more similar
> mistakes in current Xen code. I will prepare relevant patch
> shortly.

Excellent. Looking forward to them.
..
> > +	return;
> 
> I think that this return is superfluous.

>nods>
> 
> > +}
> > +#else
> > +static __init void mark_rw_past_pgt(void) { }
> 
> Dito.

Ok. Not that much worried - I think we will get rid of this in 2.6.40 anyhow
(or I hope so).
> 
> > +#endif
> >  static void xen_pgd_free(struct mm_struct *mm, pgd_t *pgd)
> >  {
> >  #ifdef CONFIG_X86_64
> > @@ -1489,6 +1602,14 @@ static __init pte_t mask_rw_pte(pte_t *ptep, pte_t pte)
> >  	unsigned long pfn = pte_pfn(pte);
> >  
> >  	/*
> > +	 * A bit of optimization. We do not need to call the workaround
> > +	 * when xen_set_pte_init is called with a PTE with 0 as PFN.
> > +	 * That is b/c the pagetable at that point are just being populated
> > +	 * with empty values and we can save some cycles by not calling
> > +	 * the 'memblock' code.*/
> > +	if (pfn)
> > +		mark_rw_past_pgt();
> > +	/*
> >  	 * If the new pfn is within the range of the newly allocated
> >  	 * kernel pagetable, and it isn't being mapped into an
> >  	 * early_ioremap fixmap slot as a freshly allocated page, make sure
> > @@ -1997,6 +2118,8 @@ __init void xen_ident_map_ISA(void)
> >  
> >  static __init void xen_post_allocator_init(void)
> 
> Dito.

<nods> That looks like a candidate for another patch.
> 
> Daniel

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Daniel Kiper]

On Tue, May 03, 2011 at 11:12:06AM -0400, Konrad Rzeszutek Wilk wrote:
> On Tue, May 03, 2011 at 02:55:27AM +0200, Daniel Kiper wrote:
> > On Mon, May 02, 2011 at 01:22:21PM -0400, Konrad Rzeszutek Wilk wrote:
> > > As a consequence of the commit:
> > >
> > > commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > > Author: Yinghai Lu <yinghai@kernel.org>
> > > Date:   Fri Dec 17 16:58:28 2010 -0800
> > >
> > >     x86-64, mm: Put early page table high
> > >
> > > it causes the Linux kernel to crash under Xen:
> > >
> > > mapping kernel into physical memory
> > > Xen: setup ISA identity maps
> > > about to get started...
> > > (XEN) mm.c:2466:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn b1d89 (pfn bacf7)
> > > (XEN) mm.c:3027:d0 Error while pinning mfn b1d89
> > > (XEN) traps.c:481:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
> > > (XEN) domain_crash_sync called from entry.S
> > > (XEN) Domain 0 (vcpu#0) crashed on cpu#0:
> > > ...
> > 
> > I was hit by this bug when I was working on memory hotplug.
> > After some investigation I found myself above mentioned patch
> > as a guilty and later I discovered that you are working on that
> > issue. I have tested your patch and discoverd some issues with it.
> > First of all it has compilation issues on gcc version 4.1.2 20061115
> > (prerelease) (Debian 4.1.1-21). Details below.
> > 
> > Additionlly, I think that your patch does not work as you expected.
> > I found that git commit 24bdb0b62cc82120924762ae6bc85afc8c3f2b26
> > (xen: do not create the extra e820 region at an addr lower than 4G)
> > do this work (to some extent). When this patch is removed domU
> > is crashing with following error:
> 
> Which is "this patch" ? The 24bdb0b62cc82120924762ae6bc85afc8c3f2b26?

Yep.

[...]

> > I think that (Stefano please confirm or not) this patch was prepared
> > as workaround for similar issues. However, I do not like this patch
> > because on systems with small amount of memory it leaves huge (to some
> > extent) hole between max_low_pfn and 4G. Additionally, it affects
> > memory hotplug a bit because it allocates memory starting from current
> > max_mfn. It also breaks memory hotplug on i386 (maybe also others
> > thinks, however, I could not confirm that). If it stay for some
> > reason it should be amended in follwing way:
> > 
> > #ifdef CONFIG_X86_32
> >         xen_extra_mem_start = mem_end;
> > #else
> >         xen_extra_mem_start = max((1ULL << 32), mem_end);
> > #endif
> > 
> > Regarding comment for this patch it should be mentioned that without this
> > patch e820_end_of_low_ram_pfn() is not broken. It is not called simply.
> > 
> > Last but least. I found that memory sizes below and including exactly 1 GiB and
> > exactly 2 GiB, 3 GiB (maybe higher, i.e. 4 GiB, 5 GiB, ...; I was not able to test
> > them because I do not have sufficient memory) are magic. It means that if memory
> > is set with those sizes everything is working good (without 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > and 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 applied). It means that domU
> > should be tested with sizes which are not power of two nor multiple of that.
> 
> Hmm, I thought I did test 1500M.

It does not work on my machine (24bdb0b62cc82120924762ae6bc85afc8c3f2b26
removed and 4b239f458c229de044d6905c2b0f9fe16ed9e01e applied).

Daniel

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Daniel Kiper]

On Tue, May 03, 2011 at 09:51:41PM +0200, Daniel Kiper wrote:
> On Tue, May 03, 2011 at 11:12:06AM -0400, Konrad Rzeszutek Wilk wrote:
> > On Tue, May 03, 2011 at 02:55:27AM +0200, Daniel Kiper wrote:
> > > On Mon, May 02, 2011 at 01:22:21PM -0400, Konrad Rzeszutek Wilk wrote:

[...]

> > > I think that (Stefano please confirm or not) this patch was prepared
> > > as workaround for similar issues. However, I do not like this patch
> > > because on systems with small amount of memory it leaves huge (to some
> > > extent) hole between max_low_pfn and 4G. Additionally, it affects
> > > memory hotplug a bit because it allocates memory starting from current
> > > max_mfn. It also breaks memory hotplug on i386 (maybe also others
> > > thinks, however, I could not confirm that). If it stay for some
> > > reason it should be amended in follwing way:
> > >
> > > #ifdef CONFIG_X86_32
> > >         xen_extra_mem_start = mem_end;
> > > #else
> > >         xen_extra_mem_start = max((1ULL << 32), mem_end);
> > > #endif
> > >
> > > Regarding comment for this patch it should be mentioned that without this
> > > patch e820_end_of_low_ram_pfn() is not broken. It is not called simply.
> > >
> > > Last but least. I found that memory sizes below and including exactly 1 GiB and
> > > exactly 2 GiB, 3 GiB (maybe higher, i.e. 4 GiB, 5 GiB, ...; I was not able to test
> > > them because I do not have sufficient memory) are magic. It means that if memory
> > > is set with those sizes everything is working good (without 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > > and 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 applied). It means that domU
> > > should be tested with sizes which are not power of two nor multiple of that.
> >
> > Hmm, I thought I did test 1500M.
>
> It does not work on my machine (24bdb0b62cc82120924762ae6bc85afc8c3f2b26
> removed and 4b239f458c229de044d6905c2b0f9fe16ed9e01e applied).

It does not work on my machine (x86_64) with Linux Kernel Ver. 2.6.39-rc6 without
git commit 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
e820 region at an addr lower than 4G). As I said ealier bug introduced by git
commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e (x86-64, mm: Put early page table
high) is probably hidden (repaird/workarounded ???) by git commit
24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
e820 region at an addr lower than 4G).

Konrad, Stefano could you confirm that ??? If it is true
how could I help you in removing this bug ???

Daniel

Re: [Xen-devel] Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Konrad Rzeszutek Wilk]

On Wed, May 04, 2011 at 08:59:03PM +0200, Daniel Kiper wrote:
> On Tue, May 03, 2011 at 09:51:41PM +0200, Daniel Kiper wrote:
> > On Tue, May 03, 2011 at 11:12:06AM -0400, Konrad Rzeszutek Wilk wrote:
> > > On Tue, May 03, 2011 at 02:55:27AM +0200, Daniel Kiper wrote:
> > > > On Mon, May 02, 2011 at 01:22:21PM -0400, Konrad Rzeszutek Wilk wrote:
> 
> [...]
> 
> > > > I think that (Stefano please confirm or not) this patch was prepared
> > > > as workaround for similar issues. However, I do not like this patch

It was actually to fix SandyBridge boxes. Their last E820 reserved
region was around fed40000 and then the RAM region started at
100000000. Which meant that we misinterpreted the gap (starting at fed40 mfn)
as the start of RAM.

> > > > because on systems with small amount of memory it leaves huge (to some
> > > > extent) hole between max_low_pfn and 4G. Additionally, it affects
> > > > memory hotplug a bit because it allocates memory starting from current
> > > > max_mfn. It also breaks memory hotplug on i386 (maybe also others
> > > > thinks, however, I could not confirm that). If it stay for some
> > > > reason it should be amended in follwing way:
> > > >
> > > > #ifdef CONFIG_X86_32
> > > >         xen_extra_mem_start = mem_end;
> > > > #else
> > > >         xen_extra_mem_start = max((1ULL << 32), mem_end);
> > > > #endif
> > > >
> > > > Regarding comment for this patch it should be mentioned that without this
> > > > patch e820_end_of_low_ram_pfn() is not broken. It is not called simply.

Hmm. What is max_pfn set to?
Can you send the full dmesg of your guest?

> > > >
> > > > Last but least. I found that memory sizes below and including exactly 1 GiB and
> > > > exactly 2 GiB, 3 GiB (maybe higher, i.e. 4 GiB, 5 GiB, ...; I was not able to test
> > > > them because I do not have sufficient memory) are magic. It means that if memory
> > > > is set with those sizes everything is working good (without 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > > > and 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 applied). It means that domU
> > > > should be tested with sizes which are not power of two nor multiple of that.
> > >
> > > Hmm, I thought I did test 1500M.
> >
> > It does not work on my machine (24bdb0b62cc82120924762ae6bc85afc8c3f2b26
> > removed and 4b239f458c229de044d6905c2b0f9fe16ed9e01e applied).
> 
> It does not work on my machine (x86_64) with Linux Kernel Ver. 2.6.39-rc6 without
> git commit 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
> e820 region at an addr lower than 4G). As I said ealier bug introduced by git
> commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e (x86-64, mm: Put early page table
> high) is probably hidden (repaird/workarounded ???) by git commit
> 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
> e820 region at an addr lower than 4G).

There are a couple of things that have been going to fix "x86-64, mm: Put
early page table high" and also .. "cleanup highmem" (something) - which
has been plaguing us since 2.6.32 (and was the one you hit long time ago).

Anyhow, regarding the setting xen_extra_mem_start to 4GB or higher should
be reworked. Not sure yet how. 
> 
> Konrad, Stefano could you confirm that ??? If it is true
> how could I help you in removing this bug ???
> 
> Daniel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Daniel Kiper]

[-- Attachment #1: Type: text/plain, Size: 3638 bytes --]

On Wed, May 04, 2011 at 03:33:53PM -0400, Konrad Rzeszutek Wilk wrote:
> On Wed, May 04, 2011 at 08:59:03PM +0200, Daniel Kiper wrote:
> > On Tue, May 03, 2011 at 09:51:41PM +0200, Daniel Kiper wrote:
> > > On Tue, May 03, 2011 at 11:12:06AM -0400, Konrad Rzeszutek Wilk wrote:
> > > > On Tue, May 03, 2011 at 02:55:27AM +0200, Daniel Kiper wrote:
> > > > > On Mon, May 02, 2011 at 01:22:21PM -0400, Konrad Rzeszutek Wilk wrote:
> > 
> > [...]
> > 
> > > > > I think that (Stefano please confirm or not) this patch was prepared
> > > > > as workaround for similar issues. However, I do not like this patch
> 
> It was actually to fix SandyBridge boxes. Their last E820 reserved
> region was around fed40000 and then the RAM region started at
> 100000000. Which meant that we misinterpreted the gap (starting at fed40 mfn)
> as the start of RAM.

Thanks.

> > > > > because on systems with small amount of memory it leaves huge (to some
> > > > > extent) hole between max_low_pfn and 4G. Additionally, it affects
> > > > > memory hotplug a bit because it allocates memory starting from current
> > > > > max_mfn. It also breaks memory hotplug on i386 (maybe also others
> > > > > thinks, however, I could not confirm that). If it stay for some
> > > > > reason it should be amended in follwing way:
> > > > >
> > > > > #ifdef CONFIG_X86_32
> > > > >         xen_extra_mem_start = mem_end;
> > > > > #else
> > > > >         xen_extra_mem_start = max((1ULL << 32), mem_end);
> > > > > #endif
> > > > >
> > > > > Regarding comment for this patch it should be mentioned that without this
> > > > > patch e820_end_of_low_ram_pfn() is not broken. It is not called simply.
> 
> Hmm. What is max_pfn set to?
> Can you send the full dmesg of your guest?

Look into attachments. Both dmesgs are from plain 2.6.39-rc6.
Guests had allocated 2 GiB of memory.

> > > > > Last but least. I found that memory sizes below and including exactly 1 GiB and
> > > > > exactly 2 GiB, 3 GiB (maybe higher, i.e. 4 GiB, 5 GiB, ...; I was not able to test
> > > > > them because I do not have sufficient memory) are magic. It means that if memory
> > > > > is set with those sizes everything is working good (without 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > > > > and 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 applied). It means that domU
> > > > > should be tested with sizes which are not power of two nor multiple of that.
> > > >
> > > > Hmm, I thought I did test 1500M.
> > >
> > > It does not work on my machine (24bdb0b62cc82120924762ae6bc85afc8c3f2b26
> > > removed and 4b239f458c229de044d6905c2b0f9fe16ed9e01e applied).
> > 
> > It does not work on my machine (x86_64) with Linux Kernel Ver. 2.6.39-rc6 without
> > git commit 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
> > e820 region at an addr lower than 4G). As I said ealier bug introduced by git
> > commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e (x86-64, mm: Put early page table
> > high) is probably hidden (repaird/workarounded ???) by git commit
> > 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
> > e820 region at an addr lower than 4G).
> 
> There are a couple of things that have been going to fix "x86-64, mm: Put
> early page table high" and also .. "cleanup highmem" (something) - which
> has been plaguing us since 2.6.32 (and was the one you hit long time ago).
> 
> Anyhow, regarding the setting xen_extra_mem_start to 4GB or higher should
> be reworked. Not sure yet how. 

OK. As I can see it is __VERY__ difficult problem. I will wait for
proper solution. However, if I could help you in any way
please drop me a line.

Daniel

[-- Attachment #2: dmesg.i386 --]
[-- Type: text/plain, Size: 8003 bytes --]

Reserving virtual address space above 0xf5800000
Linux version 2.6.39-rc6-i386.xenU.all.r0+ (root@dte40r0-um-i386) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Wed May 4 19:50:41 CEST 2011
ACPI in unprivileged domain disabled
released 0 pages of unused memory
Set 0 page(s) to 1-1 mapping.
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000080000000 (usable)
 Xen: 0000000100000000 - 0000000100800000 (usable)
NX (Execute Disable) protection: active
DMI not present or invalid.
e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0x100800 max_arch_pfn = 0x1000000
initial memory mapped : 0 - 016ba000
Base memory trampoline at [c009f000] 9f000 size 4096
init_memory_mapping: 0000000000000000-000000002d3fe000
 0000000000 - 002d3fe000 page 4k
kernel direct mapping tables up to 2d3fe000 @ 154d000-16ba000
3380MB HIGHMEM available.
723MB LOWMEM available.
  mapped low ram: 0 - 2d3fe000
  low ram: 0 - 2d3fe000
Zone PFN ranges:
  DMA      0x00000010 -> 0x00001000
  Normal   0x00001000 -> 0x0002d3fe
  HighMem  0x0002d3fe -> 0x00100800
Movable zone start PFN for each node
early_node_map[3] active PFN ranges
    0: 0x00000010 -> 0x000000a0
    0: 0x00000100 -> 0x00080000
    0: 0x00100000 -> 0x00100800
On node 0 totalpages: 526224
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 3952 pages, LIFO batch:0
  Normal zone: 1416 pages used for memmap
  Normal zone: 179830 pages, LIFO batch:31
  HighMem zone: 6761 pages used for memmap
  HighMem zone: 334233 pages, LIFO batch:31
Using APIC driver default
SMP: Allowing 4 CPUs, 0 hotplug CPUs
APIC: disable apic facility
APIC: switched to apic NOOP
nr_irqs_gsi: 16
Allocating PCI resources starting at 80000000 (gap: 80000000:80000000)
Booting paravirtualized kernel on Xen
Xen version: 4.1.1-rc1-pre (preserve-AD)
setup_percpu: NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 12 pages/cpu @ebfa6000 s28160 r0 d20992 u49152
pcpu-alloc: s28160 r0 d20992 u49152 alloc=12*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 518015
Kernel command line: root=/dev/xvda noapic nolapic softdog.nowayout=1 softdog.soft_margin=180 tmem console=hvc0
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Initializing CPU#0
Initializing HighMem for node 0 (0002d3fe:00100800)
Memory: 2067168k/4202496k available (2573k kernel code, 29536k reserved, 1179k data, 356k init, 1355784k highmem)
virtual kernel memory layout:
    fixmap  : 0xf5766000 - 0xf57ff000   ( 612 kB)
    pkmap   : 0xf5400000 - 0xf5600000   (2048 kB)
    vmalloc : 0xedbfe000 - 0xf53fe000   ( 120 MB)
    lowmem  : 0xc0000000 - 0xed3fe000   ( 723 MB)
      .init : 0xc13ab000 - 0xc1404000   ( 356 kB)
      .data : 0xc12837d4 - 0xc13aa640   (1179 kB)
      .text : 0xc1000000 - 0xc12837d4   (2573 kB)
SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Hierarchical RCU implementation.
	RCU-based detection of stalled CPUs is disabled.
NR_IRQS:384
CPU 0 irqstacks, hard=eb80c000 soft=eb80e000
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
Detected 2666.856 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5333.71 BogoMIPS (lpj=26668560)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
SMP alternatives: switching to UP code
cpu 0 spinlock event irq 17
Performance Events: unsupported p6 CPU model 15 no PMU driver, software events only.
CPU 1 irqstacks, hard=eb870000 soft=eb872000
installing Xen timer for CPU 1
cpu 1 spinlock event irq 23
SMP alternatives: switching to SMP code
Initializing CPU#1
CPU 2 irqstacks, hard=eb87a000 soft=eb87c000
installing Xen timer for CPU 2
cpu 2 spinlock event irq 29
Initializing CPU#2
CPU 3 irqstacks, hard=eb884000 soft=eb886000
installing Xen timer for CPU 3
cpu 3 spinlock event irq 35
Initializing CPU#3
Brought up 4 CPUs
Grant table initialized
NET: Registered protocol family 16
PCI: setting up Xen PCI frontend stub
PCI: pci_cache_line_size set to 64 bytes
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen/balloon: Initialising balloon driver.
last_pfn = 0x100800 max_arch_pfn = 0x1000000
xen-balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
libata version 3.00 loaded.
PCI: System does not support PCI
PCI: System does not support PCI
Switching to clocksource xen
pnp: PnP ACPI: disabled
Switched to NOHz mode on CPU #1
Switched to NOHz mode on CPU #0
Switched to NOHz mode on CPU #2
Switched to NOHz mode on CPU #3
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
UDP hash table entries: 512 (order: 2, 16384 bytes)
UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
NET: Registered protocol family 1
PCI: CLS 0 bytes, default 64
platform rtc_cmos: registered platform RTC device (no PNP device found)
highmem bounce pool size: 64 pages
HugeTLB registered 2 MB page size, pre-allocated 0 pages
NTFS driver 2.1.30 [Flags: R/W].
msgmni has been set to 1405
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered (default)
io scheduler cfq registered
Event-channel device installed.
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
loop: module loaded
e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
e1000: Copyright (c) 1999-2006 Intel Corporation.
Initialising Xen virtual ethernet driver.
blkfront: xvda: barriers enabled
 xvda:
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
i8042: PNP: No PS/2 controller found. Probing ports directly.
i8042: No controller found
blkfront: xvdb: barriers enabled
mousedev: PS/2 mouse device common for all mice
 xvdb: unknown partition table
Setting capacity to 2097152
xvdb: detected capacity change from 0 to 1073741824
rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
rtc_cmos: probe of rtc_cmos failed with error -38
Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=180 sec soft_panic=0 (nowayout= 1)
cpuidle: using governor ladder
cpuidle: using governor menu
Netfilter messages via NETLINK v0.30.
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ctnetlink v0.93: registering with nfnetlink.
xt_time: kernel timezone is -0000
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
NET: Registered protocol family 17
Using IPI No-Shortcut mode
XENBUS: Device with no driver: device/console/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
REISERFS (device xvda): found reiserfs format "3.6" with standard journal
REISERFS (device xvda): using ordered data mode
REISERFS (device xvda): journal params: device xvda, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
REISERFS (device xvda): checking transaction log (xvda)
REISERFS (device xvda): Using r5 hash to sort names
VFS: Mounted root (reiserfs filesystem) on device 202:0.
Freeing unused kernel memory: 356k freed
udevd (502): /proc/502/oom_adj is deprecated, please use /proc/502/oom_score_adj instead.
Adding 1048572k swap on /dev/xvdb.  Priority:-1 extents:1 across:1048572k SS
blkfront: xvda: empty write barrier op failed
blkfront: xvda: barriers disabled

[-- Attachment #3: dmesg.x86_64 --]
[-- Type: text/plain, Size: 8093 bytes --]

Linux version 2.6.39-rc6-x86_64.xenU.all.r0+ (root@dte40r0-um-i386) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #1 SMP Wed May 4 19:32:40 CEST 2011
Command line: root=/dev/xvda noapic nolapic softdog.nowayout=1 softdog.soft_margin=180 tmem console=hvc0
ACPI in unprivileged domain disabled
released 0 pages of unused memory
Set 0 page(s) to 1-1 mapping.
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000080000000 (usable)
 Xen: 0000000100000000 - 0000000100800000 (usable)
NX (Execute Disable) protection: active
DMI not present or invalid.
e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
No AGP bridge found
last_pfn = 0x100800 max_arch_pfn = 0x400000000
last_pfn = 0x80000 max_arch_pfn = 0x400000000
initial memory mapped : 0 - 01693000
Base memory trampoline at [ffff88000009e000] 9e000 size 8192
init_memory_mapping: 0000000000000000-0000000080000000
 0000000000 - 0080000000 page 4k
kernel direct mapping tables up to 80000000 @ 7fbfd000-80000000
init_memory_mapping: 0000000100000000-0000000100800000
 0100000000 - 0100800000 page 4k
kernel direct mapping tables up to 100800000 @ 7f3f3000-7fbfd000
xen: setting RW the range 7ffec000 - 80000000
Zone PFN ranges:
  DMA      0x00000010 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100800
Movable zone start PFN for each node
early_node_map[3] active PFN ranges
    0: 0x00000010 -> 0x000000a0
    0: 0x00000100 -> 0x00080000
    0: 0x00100000 -> 0x00100800
On node 0 totalpages: 526224
  DMA zone: 56 pages used for memmap
  DMA zone: 2 pages reserved
  DMA zone: 3926 pages, LIFO batch:0
  DMA32 zone: 14280 pages used for memmap
  DMA32 zone: 505912 pages, LIFO batch:31
  Normal zone: 28 pages used for memmap
  Normal zone: 2020 pages, LIFO batch:0
xen: setting RW the range 7f3f8000 - 7fbfd000
SMP: Allowing 4 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
APIC: switched to apic NOOP
nr_irqs_gsi: 16
Allocating PCI resources starting at 80000000 (gap: 80000000:80000000)
Booting paravirtualized kernel on Xen
Xen version: 4.1.1-rc1-pre (preserve-AD)
setup_percpu: NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 27 pages/cpu @ffff88007fb45000 s78592 r8192 d23808 u110592
pcpu-alloc: s78592 r8192 d23808 u110592 alloc=27*4096
pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 511858
Kernel command line: root=/dev/xvda noapic nolapic softdog.nowayout=1 softdog.soft_margin=180 tmem console=hvc0
PID hash table entries: 4096 (order: 3, 32768 bytes)
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
Checking aperture...
No AGP bridge found
Memory: 1982568k/4202496k available (3016k kernel code, 2097600k absent, 122328k reserved, 1580k data, 476k init)
SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
Hierarchical RCU implementation.
	CONFIG_RCU_FANOUT set to non-default value of 32
	RCU-based detection of stalled CPUs is disabled.
NR_IRQS:384
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
Xen: using vcpuop timer interface
installing Xen timer for CPU 0
Detected 2666.856 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5333.71 BogoMIPS (lpj=26668560)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 256
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
SMP alternatives: switching to UP code
cpu 0 spinlock event irq 17
Performance Events: unsupported p6 CPU model 15 no PMU driver, software events only.
installing Xen timer for CPU 1
cpu 1 spinlock event irq 23
SMP alternatives: switching to SMP code
installing Xen timer for CPU 2
cpu 2 spinlock event irq 29
installing Xen timer for CPU 3
cpu 3 spinlock event irq 35
Brought up 4 CPUs
Grant table initialized
NET: Registered protocol family 16
PCI: setting up Xen PCI frontend stub
PCI: pci_cache_line_size set to 64 bytes
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen/balloon: Initialising balloon driver.
last_pfn = 0x100800 max_arch_pfn = 0x400000000
xen-balloon: Initialising balloon driver.
vgaarb: loaded
SCSI subsystem initialized
libata version 3.00 loaded.
PCI: System does not support PCI
PCI: System does not support PCI
Switching to clocksource xen
pnp: PnP ACPI: disabled
Switched to NOHz mode on CPU #0
Switched to NOHz mode on CPU #1
Switched to NOHz mode on CPU #2
Switched to NOHz mode on CPU #3
NET: Registered protocol family 2
IP route cache hash table entries: 65536 (order: 7, 524288 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
UDP hash table entries: 1024 (order: 3, 32768 bytes)
UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
NET: Registered protocol family 1
PCI: CLS 0 bytes, default 64
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Placing 64MB software IO TLB between ffff880079000000 - ffff88007d000000
software IO TLB at phys 0x79000000 - 0x7d000000
platform rtc_cmos: registered platform RTC device (no PNP device found)
HugeTLB registered 2 MB page size, pre-allocated 0 pages
NTFS driver 2.1.30 [Flags: R/W].
msgmni has been set to 3872
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered (default)
io scheduler cfq registered
Event-channel device installed.
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
loop: module loaded
e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
e1000: Copyright (c) 1999-2006 Intel Corporation.
Initialising Xen virtual ethernet driver.
blkfront: xvda: barriers enabled
 xvda:
Setting capacity to 209715200
xvda: detected capacity change from 0 to 107374182400
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
blkfront: xvdb: barriers enabled
i8042: PNP: No PS/2 controller found. Probing ports directly.
i8042: No controller found
mousedev: PS/2 mouse device common for all mice
 xvdb: unknown partition table
Setting capacity to 2097152
xvdb: detected capacity change from 0 to 1073741824
rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
rtc_cmos: probe of rtc_cmos failed with error -38
Software Watchdog Timer: 0.07 initialized. soft_noboot=0 soft_margin=180 sec soft_panic=0 (nowayout= 1)
cpuidle: using governor ladder
cpuidle: using governor menu
Netfilter messages via NETLINK v0.30.
nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
ctnetlink v0.93: registering with nfnetlink.
xt_time: kernel timezone is -0000
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
NET: Registered protocol family 17
XENBUS: Device with no driver: device/console/0
drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
REISERFS (device xvda): found reiserfs format "3.6" with standard journal
REISERFS (device xvda): using ordered data mode
REISERFS (device xvda): journal params: device xvda, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
REISERFS (device xvda): checking transaction log (xvda)
REISERFS (device xvda): Using r5 hash to sort names
VFS: Mounted root (reiserfs filesystem) on device 202:0.
Freeing unused kernel memory: 476k freed
udevd (520): /proc/520/oom_adj is deprecated, please use /proc/520/oom_score_adj instead.
ioctl32(fgconsole:774): Unknown cmd fd(3) cmd(00005603){t:'V';sz:0} arg(ffbbd53e) on /dev/console
Adding 1048572k swap on /dev/xvdb.  Priority:-1 extents:1 across:1048572k SS
blkfront: xvda: empty write barrier op failed
blkfront: xvda: barriers disabled
ioctl32(fgconsole:1046): Unknown cmd fd(3) cmd(00005603){t:'V';sz:0} arg(ffe8c38e) on /dev/console

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Stefano Stabellini]

On Wed, 4 May 2011, Daniel Kiper wrote:
> On Tue, May 03, 2011 at 09:51:41PM +0200, Daniel Kiper wrote:
> > On Tue, May 03, 2011 at 11:12:06AM -0400, Konrad Rzeszutek Wilk wrote:
> > > On Tue, May 03, 2011 at 02:55:27AM +0200, Daniel Kiper wrote:
> > > > On Mon, May 02, 2011 at 01:22:21PM -0400, Konrad Rzeszutek Wilk wrote:
> 
> [...]
> 
> > > > I think that (Stefano please confirm or not) this patch was prepared
> > > > as workaround for similar issues. However, I do not like this patch
> > > > because on systems with small amount of memory it leaves huge (to some
> > > > extent) hole between max_low_pfn and 4G. Additionally, it affects
> > > > memory hotplug a bit because it allocates memory starting from current
> > > > max_mfn. It also breaks memory hotplug on i386 (maybe also others
> > > > thinks, however, I could not confirm that). If it stay for some
> > > > reason it should be amended in follwing way:
> > > >
> > > > #ifdef CONFIG_X86_32
> > > >         xen_extra_mem_start = mem_end;
> > > > #else
> > > >         xen_extra_mem_start = max((1ULL << 32), mem_end);
> > > > #endif
> > > >
> > > > Regarding comment for this patch it should be mentioned that without this
> > > > patch e820_end_of_low_ram_pfn() is not broken. It is not called simply.
> > > >
> > > > Last but least. I found that memory sizes below and including exactly 1 GiB and
> > > > exactly 2 GiB, 3 GiB (maybe higher, i.e. 4 GiB, 5 GiB, ...; I was not able to test
> > > > them because I do not have sufficient memory) are magic. It means that if memory
> > > > is set with those sizes everything is working good (without 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > > > and 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 applied). It means that domU
> > > > should be tested with sizes which are not power of two nor multiple of that.
> > >
> > > Hmm, I thought I did test 1500M.
> >
> > It does not work on my machine (24bdb0b62cc82120924762ae6bc85afc8c3f2b26
> > removed and 4b239f458c229de044d6905c2b0f9fe16ed9e01e applied).
> 
> It does not work on my machine (x86_64) with Linux Kernel Ver. 2.6.39-rc6 without
> git commit 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
> e820 region at an addr lower than 4G). As I said ealier bug introduced by git
> commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e (x86-64, mm: Put early page table
> high) is probably hidden (repaird/workarounded ???) by git commit
> 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
> e820 region at an addr lower than 4G).
> 
> Konrad, Stefano could you confirm that ??? If it is true
> how could I help you in removing this bug ???

The reason why "xen: do not create the extra e820 region at an addr
lower than 4G" is needed is the following:

starting the extra memory region at an address lower than 4G is
dangerous because if the extra memory region spans the 4G boundary then
e820_end_of_ram_pfn will get confused and return 0x100000000, therefore
init_memory_mapping will setup 1:1 mapping for 0-0x100000000, including
the whole 3G-4G range.
Of course this is wrong because among other things will cause the kernel
to remap the IOAPIC MMIO registers that should go through the fixmap
instead.
I don't think "x86-64, mm: Put early page table high" is related to the
bug that this commit is trying to solve.

Could you please explain in details what problems do this patch create?

In any case you are right about the fact that the change is not needed
on X86_32 so if it has any bad side effects on X86_32 we can always do
the following, like you suggested:

diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index 90bac0a..721f576 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -227,7 +227,11 @@ char * __init xen_memory_setup(void)
 
 	memcpy(map_raw, map, sizeof(map));
 	e820.nr_map = 0;
+#ifdef CONFIG_X86_32
+	xen_extra_mem_start = mem_end;
+#else
 	xen_extra_mem_start = max((1ULL << 32), mem_end);
+#endif
 	for (i = 0; i < memmap.nr_entries; i++) {
 		unsigned long long end;
 

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Konrad Rzeszutek Wilk]

On Thu, May 05, 2011 at 01:55:22PM +0100, Stefano Stabellini wrote:
> On Wed, 4 May 2011, Daniel Kiper wrote:
> > On Tue, May 03, 2011 at 09:51:41PM +0200, Daniel Kiper wrote:
> > > On Tue, May 03, 2011 at 11:12:06AM -0400, Konrad Rzeszutek Wilk wrote:
> > > > On Tue, May 03, 2011 at 02:55:27AM +0200, Daniel Kiper wrote:
> > > > > On Mon, May 02, 2011 at 01:22:21PM -0400, Konrad Rzeszutek Wilk wrote:
> > 
> > [...]
> > 
> > > > > I think that (Stefano please confirm or not) this patch was prepared
> > > > > as workaround for similar issues. However, I do not like this patch
> > > > > because on systems with small amount of memory it leaves huge (to some
> > > > > extent) hole between max_low_pfn and 4G. Additionally, it affects
> > > > > memory hotplug a bit because it allocates memory starting from current
> > > > > max_mfn. It also breaks memory hotplug on i386 (maybe also others
> > > > > thinks, however, I could not confirm that). If it stay for some
> > > > > reason it should be amended in follwing way:
> > > > >
> > > > > #ifdef CONFIG_X86_32
> > > > >         xen_extra_mem_start = mem_end;
> > > > > #else
> > > > >         xen_extra_mem_start = max((1ULL << 32), mem_end);
> > > > > #endif
> > > > >
> > > > > Regarding comment for this patch it should be mentioned that without this
> > > > > patch e820_end_of_low_ram_pfn() is not broken. It is not called simply.
> > > > >
> > > > > Last but least. I found that memory sizes below and including exactly 1 GiB and
> > > > > exactly 2 GiB, 3 GiB (maybe higher, i.e. 4 GiB, 5 GiB, ...; I was not able to test
> > > > > them because I do not have sufficient memory) are magic. It means that if memory
> > > > > is set with those sizes everything is working good (without 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > > > > and 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 applied). It means that domU
> > > > > should be tested with sizes which are not power of two nor multiple of that.
> > > >
> > > > Hmm, I thought I did test 1500M.
> > >
> > > It does not work on my machine (24bdb0b62cc82120924762ae6bc85afc8c3f2b26
> > > removed and 4b239f458c229de044d6905c2b0f9fe16ed9e01e applied).
> > 
> > It does not work on my machine (x86_64) with Linux Kernel Ver. 2.6.39-rc6 without
> > git commit 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
> > e820 region at an addr lower than 4G). As I said ealier bug introduced by git
> > commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e (x86-64, mm: Put early page table
> > high) is probably hidden (repaird/workarounded ???) by git commit
> > 24bdb0b62cc82120924762ae6bc85afc8c3f2b26 (xen: do not create the extra
> > e820 region at an addr lower than 4G).
> > 
> > Konrad, Stefano could you confirm that ??? If it is true
> > how could I help you in removing this bug ???
> 
> The reason why "xen: do not create the extra e820 region at an addr
> lower than 4G" is needed is the following:
> 
> starting the extra memory region at an address lower than 4G is
> dangerous because if the extra memory region spans the 4G boundary then
> e820_end_of_ram_pfn will get confused and return 0x100000000, therefore
> init_memory_mapping will setup 1:1 mapping for 0-0x100000000, including
> the whole 3G-4G range.
> Of course this is wrong because among other things will cause the kernel
> to remap the IOAPIC MMIO registers that should go through the fixmap
> instead.
> I don't think "x86-64, mm: Put early page table high" is related to the
> bug that this commit is trying to solve.
> 
> Could you please explain in details what problems do this patch create?
> 
> In any case you are right about the fact that the change is not needed
> on X86_32 so if it has any bad side effects on X86_32 we can always do

Why not? Can't you boot a 32-bit Dom0 on those machines?

> the following, like you suggested:
> 
> diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
> index 90bac0a..721f576 100644
> --- a/arch/x86/xen/setup.c
> +++ b/arch/x86/xen/setup.c
> @@ -227,7 +227,11 @@ char * __init xen_memory_setup(void)
>  
>  	memcpy(map_raw, map, sizeof(map));
>  	e820.nr_map = 0;
> +#ifdef CONFIG_X86_32
> +	xen_extra_mem_start = mem_end;
> +#else
>  	xen_extra_mem_start = max((1ULL << 32), mem_end);
> +#endif
>  	for (i = 0; i < memmap.nr_entries; i++) {
>  		unsigned long long end;
>  
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Stefano Stabellini]

On Thu, 5 May 2011, Konrad Rzeszutek Wilk wrote:
> > In any case you are right about the fact that the change is not needed
> > on X86_32 so if it has any bad side effects on X86_32 we can always do
> 
> Why not? Can't you boot a 32-bit Dom0 on those machines?
> 

Because e820_end_of_low_ram_pfn() is not called on 32-bit,
find_low_pfn_range() is called instead and works differently.
AFAICT find_low_pfn_range() is not affected by the issue described in
the previous email.

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Daniel Kiper]

On Thu, May 05, 2011 at 03:06:18PM +0100, Stefano Stabellini wrote:
> On Thu, 5 May 2011, Konrad Rzeszutek Wilk wrote:
> > > In any case you are right about the fact that the change is not needed
> > > on X86_32 so if it has any bad side effects on X86_32 we can always do
> >
> > Why not? Can't you boot a 32-bit Dom0 on those machines?
>
> Because e820_end_of_low_ram_pfn() is not called on 32-bit,
> find_low_pfn_range() is called instead and works differently.
> AFAICT find_low_pfn_range() is not affected by the issue described in
> the previous email.

Yes, you are right. Additionally, I tested your patch with my amendments ealier
and it works on i386. I am going to prepare relevant patch today or tomorrow.

Daniel

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Stefano Stabellini]

On Mon, 2 May 2011, Konrad Rzeszutek Wilk wrote:
> As a consequence of the commit:
> 
> commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> Author: Yinghai Lu <yinghai@kernel.org>
> Date:   Fri Dec 17 16:58:28 2010 -0800
> 
>     x86-64, mm: Put early page table high
> 
> it causes the Linux kernel to crash under Xen:
> 
> mapping kernel into physical memory
> Xen: setup ISA identity maps
> about to get started...
> (XEN) mm.c:2466:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn b1d89 (pfn bacf7)
> (XEN) mm.c:3027:d0 Error while pinning mfn b1d89
> (XEN) traps.c:481:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
> (XEN) domain_crash_sync called from entry.S
> (XEN) Domain 0 (vcpu#0) crashed on cpu#0:
> ...
> 
> The reason is that at some point init_memory_mapping is going to reach
> the pagetable pages area and map those pages too (mapping them as normal
> memory that falls in the range of addresses passed to init_memory_mapping
> as argument). Some of those pages are already pagetable pages (they are
> in the range pgt_buf_start-pgt_buf_end) therefore they are going to be
> mapped RO and everything is fine.
> Some of these pages are not pagetable pages yet (they fall in the range
> pgt_buf_end-pgt_buf_top; for example the page at pgt_buf_end) so they
> are going to be mapped RW.  When these pages become pagetable pages and
> are hooked into the pagetable, xen will find that the guest has already
> a RW mapping of them somewhere and fail the operation.
> The reason Xen requires pagetables to be RO is that the hypervisor needs
> to verify that the pagetables are valid before using them. The validation
> operations are called "pinning" (more details in arch/x86/xen/mmu.c).
> 
> In order to fix the issue we mark all the pages in the entire range
> pgt_buf_start-pgt_buf_top as RO, however when the pagetable allocation
> is completed only the range pgt_buf_start-pgt_buf_end is reserved by
> init_memory_mapping. Hence the kernel is going to crash as soon as one
> of the pages in the range pgt_buf_end-pgt_buf_top is reused (b/c those
> ranges are RO).
> 
> For this reason, this function is introduced which is called _after_
> the init_memory_mapping has completed (in a perfect world we would
> call this function from init_memory_mapping, but lets ignore that).
> 
> Because we are called _after_ init_memory_mapping the pgt_buf_[start,
> end,top] have all changed to new values (b/c another init_memory_mapping
> is called). Hence, the first time we enter this function, we save
> away the pgt_buf_start value and update the pgt_buf_[end,top].
> 
> When we detect that the "old" pgt_buf_start through pgt_buf_end
> PFNs have been reserved (so memblock_x86_reserve_range has been called),
> we immediately set out to RW the "old" pgt_buf_end through pgt_buf_top.
> 
> And then we update those "old" pgt_buf_[end|top] with the new ones
> so that we can redo this on the next pagetable.
> 
> Reviewed-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> [v1: Updated with Jeremy's comments]
> [v2: Added the crash output]
> Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
>  arch/x86/xen/mmu.c |  123 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 123 insertions(+), 0 deletions(-)
> 
> diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
> index aef7af9..1bca25f 100644
> --- a/arch/x86/xen/mmu.c
> +++ b/arch/x86/xen/mmu.c
> @@ -1463,6 +1463,119 @@ static int xen_pgd_alloc(struct mm_struct *mm)
>  	return ret;
>  }
>  
> +#ifdef CONFIG_X86_64
> +static __initdata u64 __last_pgt_set_rw = 0;
> +static __initdata u64 __pgt_buf_start = 0;
> +static __initdata u64 __pgt_buf_end = 0;
> +static __initdata u64 __pgt_buf_top = 0;
> +/*
> + * As a consequence of the commit:
> + * 
> + * commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> + * Author: Yinghai Lu <yinghai@kernel.org>
> + * Date:   Fri Dec 17 16:58:28 2010 -0800
> + * 
> + *     x86-64, mm: Put early page table high
> + * 
> + * at some point init_memory_mapping is going to reach the pagetable pages
> + * area and map those pages too (mapping them as normal memory that falls
> + * in the range of addresses passed to init_memory_mapping as argument).
> + * Some of those pages are already pagetable pages (they are in the range
> + * pgt_buf_start-pgt_buf_end) therefore they are going to be mapped RO and
> + * everything is fine.
> + * Some of these pages are not pagetable pages yet (they fall in the range
> + * pgt_buf_end-pgt_buf_top; for example the page at pgt_buf_end) so they
> + * are going to be mapped RW.  When these pages become pagetable pages and
> + * are hooked into the pagetable, xen will find that the guest has already
> + * a RW mapping of them somewhere and fail the operation.
> + * The reason Xen requires pagetables to be RO is that the hypervisor needs
> + * to verify that the pagetables are valid before using them. The validation
> + * operations are called "pinning".
> + * 
> + * In order to fix the issue we mark all the pages in the entire range
> + * pgt_buf_start-pgt_buf_top as RO, however when the pagetable allocation
> + * is completed only the range pgt_buf_start-pgt_buf_end is reserved by
> + * init_memory_mapping. Hence the kernel is going to crash as soon as one
> + * of the pages in the range pgt_buf_end-pgt_buf_top is reused (b/c those
> + * ranges are RO).
> + * 
> + * For this reason, 'mark_rw_past_pgt' is introduced which is called _after_
> + * the init_memory_mapping has completed (in a perfect world we would
> + * call this function from init_memory_mapping, but lets ignore that).
> + * 
> + * Because we are called _after_ init_memory_mapping the pgt_buf_[start,
> + * end,top] have all changed to new values (b/c init_memory_mapping
> + * is called and setting up another new page-table). Hence, the first time
> + * we enter this function, we save away the pgt_buf_start value and update
> + * the pgt_buf_[end,top].
> + * 
> + * When we detect that the "old" pgt_buf_start through pgt_buf_end
> + * PFNs have been reserved (so memblock_x86_reserve_range has been called),
> + * we immediately set out to RW the "old" pgt_buf_end through pgt_buf_top.
> + * 
> + * And then we update those "old" pgt_buf_[end|top] with the new ones
> + * so that we can redo this on the next pagetable.
> + */
> +static __init void mark_rw_past_pgt(void) {
> +
> +	if (pgt_buf_end > pgt_buf_start) {
> +		u64 addr, size;
> +
> +		/* Save it away. */
> +		if (!__pgt_buf_start) {
> +			__pgt_buf_start = pgt_buf_start;
> +			__pgt_buf_end = pgt_buf_end;
> +			__pgt_buf_top = pgt_buf_top;
> +			return;
> +		}
> +		/* If we get the range that starts at __pgt_buf_end that means
> +		 * the range is reserved, and that in 'init_memory_mapping'
> +		 * the 'memblock_x86_reserve_range' has been called with the
> +		 * outdated __pgt_buf_start, __pgt_buf_end (the "new"
> +		 * pgt_buf_[start|end|top] refer now to a new pagetable.
> +		 * Note: we are called _after_ the pgt_buf_[..] have been
> +		 * updated.*/
> +
> +		addr = memblock_x86_find_in_range_size(PFN_PHYS(__pgt_buf_start),
> +						       &size, PAGE_SIZE);
> +
> +		/* Still not reserved, meaning 'memblock_x86_reserve_range'
> +		 * hasn't been called yet. Update the _end and _top.*/
> +		if (addr == PFN_PHYS(__pgt_buf_start)) {
> +			__pgt_buf_end = pgt_buf_end;
> +			__pgt_buf_top = pgt_buf_top;
> +			return;
> +		}
> +
> +		/* OK, the area is reserved, meaning it is time for us to
> +		 * set RW for the old end->top PFNs. */
> +
> +		/* ..unless we had already done this. */
> +		if (__pgt_buf_end == __last_pgt_set_rw)
> +			return;
> +
> +		addr = PFN_PHYS(__pgt_buf_end);
> +		
> +		/* set as RW the rest */
> +		printk(KERN_DEBUG "xen: setting RW the range %llx - %llx\n",
> +			PFN_PHYS(__pgt_buf_end), PFN_PHYS(__pgt_buf_top));
> +		
> +		while (addr < PFN_PHYS(__pgt_buf_top)) {
> +			make_lowmem_page_readwrite(__va(addr));
> +			addr += PAGE_SIZE;
> +		}
> +		/* And update everything so that we are ready for the next
> +		 * pagetable (the one created for regions past 4GB) */
> +		__last_pgt_set_rw = __pgt_buf_end;
> +		__pgt_buf_start = pgt_buf_start;
> +		__pgt_buf_end = pgt_buf_end;
> +		__pgt_buf_top = pgt_buf_top;
> +	}
> +	return;
> +}
> +#else
> +static __init void mark_rw_past_pgt(void) { }
> +#endif
>  static void xen_pgd_free(struct mm_struct *mm, pgd_t *pgd)
>  {
>  #ifdef CONFIG_X86_64
> @@ -1489,6 +1602,14 @@ static __init pte_t mask_rw_pte(pte_t *ptep, pte_t pte)
>  	unsigned long pfn = pte_pfn(pte);
>  
>  	/*
> +	 * A bit of optimization. We do not need to call the workaround
> +	 * when xen_set_pte_init is called with a PTE with 0 as PFN.
> +	 * That is b/c the pagetable at that point are just being populated
> +	 * with empty values and we can save some cycles by not calling
> +	 * the 'memblock' code.*/
> +	if (pfn)
> +		mark_rw_past_pgt();
> +	/*
>  	 * If the new pfn is within the range of the newly allocated
>  	 * kernel pagetable, and it isn't being mapped into an
>  	 * early_ioremap fixmap slot as a freshly allocated page, make sure
> @@ -1997,6 +2118,8 @@ __init void xen_ident_map_ISA(void)
>  
>  static __init void xen_post_allocator_init(void)
>  {
> +	mark_rw_past_pgt();
> +
>  #ifdef CONFIG_XEN_DEBUG
>  	pv_mmu_ops.make_pte = PV_CALLEE_SAVE(xen_make_pte_debug);
>  #endif



Unless I am missing something there is no guarantee that somebody else
won't use memory in the pgt_buf_end-pgt_buf_top range when the range is
still RO before mark_rw_past_pgt() is called again.  If so this code
works by coincidence, that is the reason why I didn't try to reuse the
pagetable_setup_done or the pagetable_setup_start hooks.

In any case this code looks very ugly and fragile, do we really want to
add a workaround as bad as this one rather than reverting the original
commit? I think it creates a bad precedent.

Re: [PATCH 1/2] xen/mmu: Add workaround "x86-64, mm: Put early page table high"

[Konrad Rzeszutek Wilk]

On Tue, May 03, 2011 at 02:20:25PM +0100, Stefano Stabellini wrote:
> On Mon, 2 May 2011, Konrad Rzeszutek Wilk wrote:
> > As a consequence of the commit:
> > 
> > commit 4b239f458c229de044d6905c2b0f9fe16ed9e01e
> > Author: Yinghai Lu <yinghai@kernel.org>
> > Date:   Fri Dec 17 16:58:28 2010 -0800
> > 
> >     x86-64, mm: Put early page table high
> > 
> > it causes the Linux kernel to crash under Xen:
> > 
> > mapping kernel into physical memory
> > Xen: setup ISA identity maps
> > about to get started...
> > (XEN) mm.c:2466:d0 Bad type (saw 7400000000000001 != exp 1000000000000000) for mfn b1d89 (pfn bacf7)
> > (XEN) mm.c:3027:d0 Error while pinning mfn b1d89
> > (XEN) traps.c:481:d0 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
> > (XEN) domain_crash_sync called from entry.S
> > (XEN) Domain 0 (vcpu#0) crashed on cpu#0:

.. snip..
> 
> 
> Unless I am missing something there is no guarantee that somebody else
> won't use memory in the pgt_buf_end-pgt_buf_top range when the range is
> still RO before mark_rw_past_pgt() is called again.  If so this code
> works by coincidence, that is the reason why I didn't try to reuse the
> pagetable_setup_done or the pagetable_setup_start hooks.

It looks that during sequence of events after the initial pagetable is created
and when we get to the post-allocator nobody is touching those pages.
(also one of them - the 0-4GB pagetable has been .. made RW). But if you do find it
dying/crashing, please tell so that we can revert it and use the generic work-around
or revert Yinghai's patch.
 
> In any case this code looks very ugly and fragile, do we really want to
> add a workaround as bad as this one rather than reverting the original
> commit? I think it creates a bad precedent.

This is the second one. We had the swapper_pg_dir/initial_page_table dance in x86_32
where we mark it RO, then RW (after a cr3 load) then RO and then back to RW.
(Details escape me, but it was some form of that)

But I wonder how many workarounds the generic code has because of us?

I think we need to setup some form of meeting with the x86 maintainers
to figure out some better way of handling this.

[PATCH 2/2] xen: mask_rw_pte mark RO all pagetable pages up to pgt_buf_top

[Konrad Rzeszutek Wilk]

From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

mask_rw_pte is currently checking if a pfn is a pagetable page if it
falls in the range pgt_buf_start - pgt_buf_end but that is incorrect
because pgt_buf_end is a moving target: pgt_buf_top is the real
boundary.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
 arch/x86/xen/mmu.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 1bca25f..55c965b 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1616,7 +1616,7 @@ static __init pte_t mask_rw_pte(pte_t *ptep, pte_t pte)
 	 * it is RO.
 	 */
 	if (((!is_early_ioremap_ptep(ptep) &&
-			pfn >= pgt_buf_start && pfn < pgt_buf_end)) ||
+			pfn >= pgt_buf_start && pfn < pgt_buf_top)) ||
 			(is_early_ioremap_ptep(ptep) && pfn != (pgt_buf_end - 1)))
 		pte = pte_wrprotect(pte);
 
-- 
1.7.1

Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[H. Peter Anvin]

On 05/02/2011 10:22 AM, Konrad Rzeszutek Wilk wrote:
> 
> But we haven't come up with an acceptable general solution yet, so this
> patchset provides a workaround for the problem. Peter, Yinghai - what would be
> the best forum/email/conference to hammer out a general solution for this?
> 
> Currently, there are couple of ways of fixing this:
>  - use pvops hooks: http://marc.info/?i=1302607192-21355-2-git-send-email-stefano.stabellini@eu.citrix.com
>  - have a workaround in Xen MMU's early bootup code (which is what these
>    two patches to this email have).
>  - or remove the patch introducing the regression altogether.
> 
> Foremost important is to fix the regression, and attached patches 
> achieve that. I want to remove this workaround patch when we
> hammer out more appropriate semantics for the page table creation - but
> that will take some time and the runway to do that in 2.6.39 is gone.
> 

My inclination would be to apply your workaround -- are there any
adverse effects to doing that?

	-hpa

[Sorry if I have missed any emails recently... apparently my email was
significantly on the fritz over the last few days.]

Re: [Xen-devel] Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[Konrad Rzeszutek Wilk]

On Mon, May 02, 2011 at 10:31:21AM -0700, H. Peter Anvin wrote:
> On 05/02/2011 10:22 AM, Konrad Rzeszutek Wilk wrote:
> > 
> > But we haven't come up with an acceptable general solution yet, so this
> > patchset provides a workaround for the problem. Peter, Yinghai - what would be
> > the best forum/email/conference to hammer out a general solution for this?
> > 
> > Currently, there are couple of ways of fixing this:
> >  - use pvops hooks: http://marc.info/?i=1302607192-21355-2-git-send-email-stefano.stabellini@eu.citrix.com
> >  - have a workaround in Xen MMU's early bootup code (which is what these
> >    two patches to this email have).
> >  - or remove the patch introducing the regression altogether.
> > 
> > Foremost important is to fix the regression, and attached patches 
> > achieve that. I want to remove this workaround patch when we
> > hammer out more appropriate semantics for the page table creation - but
> > that will take some time and the runway to do that in 2.6.39 is gone.
> > 
> 
> My inclination would be to apply your workaround -- are there any
> adverse effects to doing that?

There is a bootup slowdown (not noticeable). That is because we call the
'memblock_find_range' function on every PTE table creation (only during
bootup of course). 

Testing wise, on the machines on which the regression occurred, with these two
patches the regression disappears - so that is a good sign. I am testing
it on some more today to assure myself I am not missing anything.

> 
> 	-hpa
> 
> [Sorry if I have missed any emails recently... apparently my email was
> significantly on the fritz over the last few days.]

Yikes - I hate when that happens. I was wondering why you went so silent
on some of the emails.

Re: Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[H. Peter Anvin]

On 05/02/2011 11:08 AM, Konrad Rzeszutek Wilk wrote:
>>
>> My inclination would be to apply your workaround -- are there any
>> adverse effects to doing that?
> 
> There is a bootup slowdown (not noticeable). That is because we call the
> 'memblock_find_range' function on every PTE table creation (only during
> bootup of course). 
> 
> Testing wise, on the machines on which the regression occurred, with these two
> patches the regression disappears - so that is a good sign. I am testing
> it on some more today to assure myself I am not missing anything.
> 

OK, sounds like a plan then.  I like it because it doesn't affect the
native kernel.

	-hpa

Re: Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[Konrad Rzeszutek Wilk]

On Mon, May 02, 2011 at 11:33:16AM -0700, H. Peter Anvin wrote:
> On 05/02/2011 11:08 AM, Konrad Rzeszutek Wilk wrote:
> >>
> >> My inclination would be to apply your workaround -- are there any
> >> adverse effects to doing that?
> > 
> > There is a bootup slowdown (not noticeable). That is because we call the
> > 'memblock_find_range' function on every PTE table creation (only during
> > bootup of course). 
> > 
> > Testing wise, on the machines on which the regression occurred, with these two
> > patches the regression disappears - so that is a good sign. I am testing
> > it on some more today to assure myself I am not missing anything.
> > 
> 
> OK, sounds like a plan then.  I like it because it doesn't affect the
> native kernel.

<laughs> I figured :-)

Moving forward I really want to get rid of this wart.

Not sure if it is possible, but I was thinking it would be nice to get
you, Yinghai, Jeremy, Stefano all in one place (phone or web-conference thing)
to sketch out some ideas and hammer something out.

What days would work best?

Re: [Xen-devel] Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[H. Peter Anvin]

On 05/02/2011 12:34 PM, Konrad Rzeszutek Wilk wrote:
> On Mon, May 02, 2011 at 11:33:16AM -0700, H. Peter Anvin wrote:
>> On 05/02/2011 11:08 AM, Konrad Rzeszutek Wilk wrote:
>>>>
>>>> My inclination would be to apply your workaround -- are there any
>>>> adverse effects to doing that?
>>>
>>> There is a bootup slowdown (not noticeable). That is because we call the
>>> 'memblock_find_range' function on every PTE table creation (only during
>>> bootup of course). 
>>>
>>> Testing wise, on the machines on which the regression occurred, with these two
>>> patches the regression disappears - so that is a good sign. I am testing
>>> it on some more today to assure myself I am not missing anything.
>>>
>>
>> OK, sounds like a plan then.  I like it because it doesn't affect the
>> native kernel.
> 
> <laughs> I figured :-)
> 
> Moving forward I really want to get rid of this wart.
> 
> Not sure if it is possible, but I was thinking it would be nice to get
> you, Yinghai, Jeremy, Stefano all in one place (phone or web-conference thing)
> to sketch out some ideas and hammer something out.
> 
> What days would work best?

First things first... are you pushing the workaround (you can add my
Acked-by:) or should I?

Second... I don't know what physical locations and/or time zones every
one is in, which is probably the first thing.

Third, I will be travelling a lot over the next two weeks, plus we are
getting close to merge window time, so it might be hard to schedule
before the merge window.  Are you looking at trying to push something
for .40?  If so, you probably should have an implementation in mind
already.  If not, I suggest we aim for after the merge window is over or
at least quieted down.
	
	-hpa

Re: [Xen-devel] Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[Konrad Rzeszutek Wilk]

On Mon, May 02, 2011 at 12:59:08PM -0700, H. Peter Anvin wrote:
> On 05/02/2011 12:34 PM, Konrad Rzeszutek Wilk wrote:
> > On Mon, May 02, 2011 at 11:33:16AM -0700, H. Peter Anvin wrote:
> >> On 05/02/2011 11:08 AM, Konrad Rzeszutek Wilk wrote:
> >>>>
> >>>> My inclination would be to apply your workaround -- are there any
> >>>> adverse effects to doing that?
> >>>
> >>> There is a bootup slowdown (not noticeable). That is because we call the
> >>> 'memblock_find_range' function on every PTE table creation (only during
> >>> bootup of course). 
> >>>
> >>> Testing wise, on the machines on which the regression occurred, with these two
> >>> patches the regression disappears - so that is a good sign. I am testing
> >>> it on some more today to assure myself I am not missing anything.
> >>>
> >>
> >> OK, sounds like a plan then.  I like it because it doesn't affect the
> >> native kernel.
> > 
> > <laughs> I figured :-)
> > 
> > Moving forward I really want to get rid of this wart.
> > 
> > Not sure if it is possible, but I was thinking it would be nice to get
> > you, Yinghai, Jeremy, Stefano all in one place (phone or web-conference thing)
> > to sketch out some ideas and hammer something out.
> > 
> > What days would work best?
> 
> First things first... are you pushing the workaround (you can add my
> Acked-by:) or should I?

I will do it tomorrow and stick your Acked-by on both patches.
> 
> Second... I don't know what physical locations and/or time zones every
> one is in, which is probably the first thing.

EST for me, Jeremy is PST (California), Stefano is in UK, so that is UTC+0, and
no idea where Yinghai is.

But people are on vacation time right now so or until next week, plus
> Third, I will be travelling a lot over the next two weeks, plus we are

.. you are traveling.
> getting close to merge window time, so it might be hard to schedule
> before the merge window.  Are you looking at trying to push something
> for .40?  If so, you probably should have an implementation in mind
> already.  If not, I suggest we aim for after the merge window is over or
> at least quieted down.

.. and I don't have any implementation in mind yet. Was hoping we could
brainstorm something together.

OK, let me bug you guys when the merge window is over.

> 	
> 	-hpa

Re: Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high'

[Yinghai Lu]

On Mon, May 2, 2011 at 1:07 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Mon, May 02, 2011 at 12:59:08PM -0700, H. Peter Anvin wrote:
>> >> OK, sounds like a plan then.  I like it because it doesn't affect the
>> >> native kernel.

Xen should set RAM for page-table to RO after init_memory mapping.

>> >
>> > <laughs> I figured :-)
>> >

...

> EST for me, Jeremy is PST (California), Stefano is in UK, so that is UTC+0, and
> no idea where Yinghai is.

California

Thanks

Yinghai

Re: Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high' - does not work.

[Konrad Rzeszutek Wilk]

> >> OK, sounds like a plan then.  I like it because it doesn't affect the
> >> native kernel.
.. snip..

> First things first... are you pushing the workaround (you can add my
> Acked-by:) or should I?

Grrr.. While it works on my machines, it does not work on some of the AMD opteron
CPU machines. I asked Stefan Bader from Canonical to run a simple bootup test with
2.6.39-rc6 (Linus picked it already), and if it crashed, to use the 
one that Stefano came up with.

And sure enough - it crashed and Stefano's fix worked. I've Stefano's
patch in stable/bug-fixes-for-x86 and I am OK reverting the fix I came up.

Here is the email correspondence:

On 05/05/2011 04:56 PM, Konrad Rzeszutek Wilk wrote:
> On Thu, May 05, 2011 at 04:20:47PM +0200, Stefan Bader wrote:
>> On 05/04/2011 05:12 PM, Konrad Rzeszutek Wilk wrote:
>>> Hey Stefan,
>>>
>>> I have you as the only contact from Canonical who does Xen. Not sure
>>> if that is correct - if not, please forward me to the right person.
>>>
>>> When v2.6.39 merge window openned a patch from Yinghai ("Put early
>>> page table high") made x86_64 Linux kernels unbootable under Xen.
>>> A hack got added in v2.6.39-rc6 (git commit  a38647837a411f7df79623128421eef2118b5884)
>>> which takes care of it. But it is a hack, and the testing we did
>>> does include a lot of hardware and config options  - but not everything
>>> so we might have missed some edge cases where this particular fix
>>> won't work right.
>>>
>>> Hence I was wondering when you guys get to test this kernel (v2.6.39-rc6
>>> or later) under Xen if you could be mindful of this particular fix - and
>>> see if does the job or if it is insufficient. If it is insufficient
>>> there is a backup patch in 'stable/bug-fixes-for-x86'
>>> (git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git)
>>> that is more surgical in its fixing (but the x86 maintainers
>>> don't like it).
>>>
>>> Thanks,
>>>
>>> Konrad
>>
>> Hi Konrad,
>>
>> I just did the tests with the -rc6 kernel. While a smaller memory setup (615M,
>> one vcpu) boots successful, I see an eary crash with a bigger setup (7680M, 2
> 
> ok..
>> vcpus) which I do not get with a 2.6.38 based kernel (the crash log is attached
>> but I did not do any real analysis).
> 
> <nods>
>>
>> My test system is CentOS 5.6 based with Xen 3.4.3 (from gitco) on an AMD opteron
>> CPU. As said, I have not really looked into the dump as I wanted to let know
>> early. I would try the backup patch next and see how that goes.
> 
> Great!  Thank you. 
>>
>> -Stefan
> 
>> [    0.000000] 
>> <6>[    0.000000] Initializing cgroup subsys cpuset
>> <6>[    0.000000] Initializing cgroup subsys cpu
>> <5>[    0.000000] Linux version 2.6.39-0-virtual (root@tangerine) (gcc version 4.6.1 20110428 (prerelease) (Ubuntu 4.6.0-6ubuntu1) ) #6~smb1 SMP Thu May 5 09:28:20 UTC 2011 (Ubuntu 2.6.39-0.6~smb1-virtual 2.6.39-rc5)
>> <6>[    0.000000] Command line: root=LABEL=uec-rootfs ro console=hvc0 
>> <6>[    0.000000] KERNEL supported cpus:
>> <6>[    0.000000]   Intel GenuineIntel
>> <6>[    0.000000]   AMD AuthenticAMD
>> <6>[    0.000000]   Centaur CentaurHauls
>> <6>[    0.000000] ACPI in unprivileged domain disabled
>> <6>[    0.000000] released 0 pages of unused memory
>> <6>[    0.000000] Set 0 page(s) to 1-1 mapping.
>> <6>[    0.000000] BIOS-provided physical RAM map:
>> <6>[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
>> <6>[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
>> <6>[    0.000000]  Xen: 0000000000100000 - 00000001e0800000 (usable)
>> <6>[    0.000000] NX (Execute Disable) protection: active
>> <6>[    0.000000] DMI not present or invalid.
>> <7>[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
>> <7>[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
>> <6>[    0.000000] No AGP bridge found
>> <6>[    0.000000] last_pfn = 0x1e0800 max_arch_pfn = 0x400000000
>> <6>[    0.000000] last_pfn = 0x100000 max_arch_pfn = 0x400000000
>> <7>[    0.000000] initial memory mapped : 0 - 02c3a000
>> <7>[    0.000000] Base memory trampoline at [ffff88000009b000] 9b000 size 20480
>> <6>[    0.000000] init_memory_mapping: 0000000000000000-0000000100000000
>> <7>[    0.000000]  0000000000 - 0100000000 page 4k
>> <7>[    0.000000] kernel direct mapping tables up to 100000000 @ ff7fb000-100000000
>> <6>[    0.000000] init_memory_mapping: 0000000100000000-00000001e0800000
>> <7>[    0.000000]  0100000000 - 01e0800000 page 4k
>> <7>[    0.000000] kernel direct mapping tables up to 1e0800000 @ 1df0f3000-1e0000000
>> <7>[    0.000000] xen: setting RW the range fffdc000 - 100000000
>> <6>[    0.000000] RAMDISK: 0203b000 - 02c3a000
>> <6>[    0.000000] No NUMA configuration found
>> <6>[    0.000000] Faking a node at 0000000000000000-00000001e0800000
>> <7>[    0.000000] NUMA: Using 63 for the hash shift.
>> <6>[    0.000000] Initmem setup node 0 0000000000000000-00000001e0800000
>> <6>[    0.000000]   NODE_DATA [00000001dfffb000 - 00000001dfffffff]
>> <1>[    0.000000] BUG: unable to handle kernel NULL pointer dereference at           (null)
>> <1>[    0.000000] IP: [<ffffffff81cf6a75>] setup_node_bootmem+0x18a/0x1ea
>> <4>[    0.000000] PGD 0 
>> <0>[    0.000000] Oops: 0003 [#1] SMP 
>> <0>[    0.000000] last sysfs file: 
>> <4>[    0.000000] CPU 0 
>> <4>[    0.000000] Modules linked in:
>> <4>[    0.000000] 
>> <4>[    0.000000] Pid: 0, comm: swapper Not tainted 2.6.39-0-virtual #6~smb1  
>> <4>[    0.000000] RIP: e030:[<ffffffff81cf6a75>]  [<ffffffff81cf6a75>] setup_node_bootmem+0x18a/0x1ea
>> <4>[    0.000000] RSP: e02b:ffffffff81c01e38  EFLAGS: 00010046
>> <4>[    0.000000] RAX: 0000000000000000 RBX: 00000001e0800000 RCX: 0000000000001040
>> <4>[    0.000000] RDX: 0000000000004100 RSI: 0000000000000000 RDI: ffff8801dfffb000
>> <4>[    0.000000] RBP: ffffffff81c01e58 R08: 0000000000000020 R09: 0000000000000000
>> <4>[    0.000000] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
>> <4>[    0.000000] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000bfe400
>> <4>[    0.000000] FS:  0000000000000000(0000) GS:ffffffff81cca000(0000) knlGS:0000000000000000
>> <4>[    0.000000] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
>> <4>[    0.000000] CR2: 0000000000000000 CR3: 0000000001c03000 CR4: 0000000000000660
>> <4>[    0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> <4>[    0.000000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>> <4>[    0.000000] Process swapper (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c0b020)
>> <0>[    0.000000] Stack:
>> <4>[    0.000000]  0000000000000040 0000000000000001 0000000000000000 ffffffffffffffff
>> <4>[    0.000000]  ffffffff81c01e88 ffffffff81cf6c25 0000000000000000 0000000000000000
>> <4>[    0.000000]  ffffffff81cf687f 0000000000000000 ffffffff81c01ea8 ffffffff81cf6e45
>> <0>[    0.000000] Call Trace:
>> <4>[    0.000000]  [<ffffffff81cf6c25>] numa_register_memblks.constprop.3+0x150/0x181
>> <4>[    0.000000]  [<ffffffff81cf687f>] ? numa_add_memblk+0x7c/0x7c
>> <4>[    0.000000]  [<ffffffff81cf6e45>] numa_init.part.2+0x1c/0x7c
>> <4>[    0.000000]  [<ffffffff81cf687f>] ? numa_add_memblk+0x7c/0x7c
>> <4>[    0.000000]  [<ffffffff81cf6f67>] numa_init+0x6c/0x70
>> <4>[    0.000000]  [<ffffffff81cf7057>] initmem_init+0x39/0x3b
>> <4>[    0.000000]  [<ffffffff81ce5865>] setup_arch+0x64e/0x769
>> <4>[    0.000000]  [<ffffffff815e43c1>] ? printk+0x51/0x53
>> <4>[    0.000000]  [<ffffffff81cdf92b>] start_kernel+0xd4/0x3f3
>> <4>[    0.000000]  [<ffffffff81cdf388>] x86_64_start_reservations+0x132/0x136
>> <4>[    0.000000]  [<ffffffff81ce2ed4>] xen_start_kernel+0x588/0x58f
>> <0>[    0.000000] Code: 41 00 00 48 8b 3c c5 a0 24 cc 81 31 c0 40 f6 c7 01 74 05 aa 66 ba ff 40 40 f6 c7 02 74 05 66 ab 83 ea 02 89 d1 c1 e9 02 f6 c2 02 <f3> ab 74 02 66 ab 80 e2 01 74 01 aa 49 63 c4 48 c1 eb 0c 44 89 
>> <1>[    0.000000] RIP  [<ffffffff81cf6a75>] setup_node_bootmem+0x18a/0x1ea
>> <4>[    0.000000]  RSP <ffffffff81c01e38>
>> <0>[    0.000000] CR2: 0000000000000000
>> <4>[    0.000000] ---[ end trace a7919e7f17c0a725 ]---
>> <0>[    0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
>> <4>[    0.000000] Pid: 0, comm: swapper Tainted: G      D     2.6.39-0-virtual #6~smb1
>> <4>[    0.000000] Call Trace:
>> <4>[    0.000000]  [<ffffffff815e426d>] panic+0x91/0x194
>> <4>[    0.000000]  [<ffffffff81063ffb>] do_exit+0x40b/0x440
>> <4>[    0.000000]  [<ffffffff815fb990>] oops_end+0xb0/0xf0
>> <4>[    0.000000]  [<ffffffff815e332b>] no_context+0x145/0x152
>> <4>[    0.000000]  [<ffffffff815e34c6>] __bad_area_nosemaphore+0x18e/0x1b1
>> <4>[    0.000000]  [<ffffffff815e34fc>] bad_area_nosemaphore+0x13/0x15
>> <4>[    0.000000]  [<ffffffff815fe29d>] do_page_fault+0x43d/0x530
>> <4>[    0.000000]  [<ffffffff815fa70e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
>> <4>[    0.000000]  [<ffffffff8100721f>] ? __raw_callee_save_xen_restore_fl+0x11/0x1e
>> <4>[    0.000000]  [<ffffffff81060161>] ? vprintk+0x231/0x4b0
>> <4>[    0.000000]  [<ffffffff815facd5>] page_fault+0x25/0x30
>> <4>[    0.000000]  [<ffffffff81cf6a75>] ? setup_node_bootmem+0x18a/0x1ea
>> <4>[    0.000000]  [<ffffffff81cf6a16>] ? setup_node_bootmem+0x12b/0x1ea
>> <4>[    0.000000]  [<ffffffff81cf6c25>] numa_register_memblks.constprop.3+0x150/0x181
>> <4>[    0.000000]  [<ffffffff81cf687f>] ? numa_add_memblk+0x7c/0x7c
>> <4>[    0.000000]  [<ffffffff81cf6e45>] numa_init.part.2+0x1c/0x7c
>> <4>[    0.000000]  [<ffffffff81cf687f>] ? numa_add_memblk+0x7c/0x7c
>> <4>[    0.000000]  [<ffffffff81cf6f67>] numa_init+0x6c/0x70
>> <4>[    0.000000]  [<ffffffff81cf7057>] initmem_init+0x39/0x3b
>> <4>[    0.000000]  [<ffffffff81ce5865>] setup_arch+0x64e/0x769
>> <4>[    0.000000]  [<ffffffff815e43c1>] ? printk+0x51/0x53
>> <4>[    0.000000]  [<ffffffff81cdf92b>] start_kernel+0xd4/0x3f3
>> <4>[    0.000000]  [<ffffffff81cdf388>] x86_64_start_reservations+0x132/0x136
>> <4>[    0.000000]  [<ffffffff81ce2ed4>] xen_start_kernel+0x588/0x58f
> 

This seems to work better. I reverted the work-around patch and applied the two
patches at the tip of your branch. Both configurations boot now. To get a bit
more coverage, I would also put this second version into some ec2 instances
(they usually are Intel based). But at least locally the second approach seems
better (even with x86 maintainer disklike ;)).

-Stefan


--------------000200000708070501040204
Content-Type: text/x-log;
 name="dmesg.log"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="dmesg.log"

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.39-0-virtual (root@tangerine) (gcc version 4.6.1 20110428 (prerelease) (Ubuntu 4.6.0-6ubuntu1) ) #6~smb2 SMP Thu May 5 14:44:59 UTC 2011 (Ubuntu 2.6.39-0.6~smb2-virtual 2.6.39-rc5)
[    0.000000] Command line: root=LABEL=uec-rootfs ro console=hvc0 
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] Set 0 page(s) to 1-1 mapping.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 00000001e0800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) ==> (reserved)
[    0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x1e0800 max_arch_pfn = 0x400000000
[    0.000000] last_pfn = 0x100000 max_arch_pfn = 0x400000000
[    0.000000] initial memory mapped : 0 - 02c3a000
[    0.000000] Base memory trampoline at [ffff88000009b000] 9b000 size 20480
[    0.000000] init_memory_mapping: 0000000000000000-0000000100000000
[    0.000000]  0000000000 - 0100000000 page 4k
[    0.000000] kernel direct mapping tables up to 100000000 @ ff7fb000-100000000
[    0.000000] xen: setting RW the range fffdc000 - 100000000
[    0.000000] init_memory_mapping: 0000000100000000-00000001e0800000
[    0.000000]  0100000000 - 01e0800000 page 4k
[    0.000000] kernel direct mapping tables up to 1e0800000 @ 1df0f3000-1e0000000
[    0.000000] xen: setting RW the range 1df7fb000 - 1e0000000
[    0.000000] RAMDISK: 0203b000 - 02c3a000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at 0000000000000000-00000001e0800000
[    0.000000] NUMA: Using 63 for the hash shift.
[    0.000000] Initmem setup node 0 0000000000000000-00000001e0800000
[    0.000000]   NODE_DATA [00000001dfffb000 - 00000001dfffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x001e0800
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x001e0800
[    0.000000] On node 0 totalpages: 1968016
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 5 pages reserved
[    0.000000]   DMA zone: 3923 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 14280 pages used for memmap
[    0.000000]   DMA32 zone: 1030200 pages, LIFO batch:31
[    0.000000]   Normal zone: 12572 pages used for memmap
[    0.000000]   Normal zone: 906980 pages, LIFO batch:31
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] nr_irqs_gsi: 16
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] PCI: Warning: Cannot find a gap in the 32bit address range
[    0.000000] PCI: Unassigned devices with 32bit resource registers may break!
[    0.000000] Allocating PCI resources starting at 1e0900000 (gap: 1e0900000:400000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 3.4.3 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff8801dff74000 s84608 r8192 d21888 u114688
[    0.000000] pcpu-alloc: s84608 r8192 d21888 u114688 alloc=28*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 1941103
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: root=LABEL=uec-rootfs ro console=hvc0 
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 7629212k/7872512k available (6169k kernel code, 448k absent, 242852k reserved, 6921k data, 912k init)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:288 16
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] allocated 63963136 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2000.138 MHz processor.
[    0.010000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4000.27 BogoMIPS (lpj=20001380)
[    0.010000] pid_max: default: 32768 minimum: 301
[    0.010000] Security Framework initialized
[    0.010000] AppArmor: AppArmor initialized
[    0.010000] Yama: becoming mindful.
[    0.010000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.010000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.010000] Mount-cache hash table entries: 256
[    0.010000] Initializing cgroup subsys ns
[    0.010000] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.010000] Initializing cgroup subsys cpuacct
[    0.010000] Initializing cgroup subsys memory
[    0.010000] Initializing cgroup subsys devices
[    0.010000] Initializing cgroup subsys freezer
[    0.010000] Initializing cgroup subsys net_cls
[    0.010000] Initializing cgroup subsys blkio
[    0.010000] Initializing cgroup subsys perf_event
[    0.010000] tseg: 00dff00000
[    0.010000] CPU: Physical Processor ID: 0
[    0.010000] CPU: Processor Core ID: 0
[    0.010000] SMP alternatives: switching to UP code
[    0.010000] ftrace: allocating 25924 entries in 102 pages
[    0.010144] cpu 0 spinlock event irq 17
[    0.010188] Performance Events: 
[    0.010195] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.010204] no hardware sampling interrupt available.
[    0.010241] Broken PMU hardware detected, using software events only.
[    0.010952] installing Xen timer for CPU 1
[    0.010978] cpu 1 spinlock event irq 23
[    0.011022] SMP alternatives: switching to SMP code
[    0.020145] Brought up 2 CPUs
[    0.020248] devtmpfs: initialized
[    0.021630] Grant table initialized
[    0.021630] print_constraints: dummy: 
[    0.043023] Time: 165:165:165  Date: 165/165/65
[    0.043085] NET: Registered protocol family 16
[    0.043283] Trying to unpack rootfs image as initramfs...
[    0.050186] Extended Config Space enabled on 0 nodes
[    0.050794] PCI: setting up Xen PCI frontend stub
[    0.050794] PCI: pci_cache_line_size set to 64 bytes
[    0.052494] bio: create slab <bio-0> at 0
[    0.052633] ACPI: Interpreter disabled.
[    0.053004] xen/balloon: Initialising balloon driver.
[    0.053004] last_pfn = 0x1e0800 max_arch_pfn = 0x400000000
[    0.054255] xen-balloon: Initialising balloon driver.
[    0.054385] vgaarb: loaded
[    0.054385] SCSI subsystem initialized
[    0.054792] libata version 3.00 loaded.
[    0.054792] usbcore: registered new interface driver usbfs
[    0.054792] usbcore: registered new interface driver hub
[    0.054932] usbcore: registered new device driver usb
[    0.055054] PCI: System does not support PCI
[    0.055054] PCI: System does not support PCI
[    0.055144] NetLabel: Initializing
[    0.055144] NetLabel:  domain hash size = 128
[    0.055144] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.055144] NetLabel:  unlabeled traffic allowed by default
[    0.055144] Switching to clocksource xen
[    0.055446] Switched to NOHz mode on CPU #0
[    0.055443] Switched to NOHz mode on CPU #1
[    0.057737] AppArmor: AppArmor Filesystem Enabled
[    0.057771] pnp: PnP ACPI: disabled
[    0.059315] NET: Registered protocol family 2
[    0.059780] Freeing initrd memory: 12284k freed
[    0.059878] IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.062730] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    0.066505] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.066992] TCP: Hash tables configured (established 524288 bind 65536)
[    0.067003] TCP reno registered
[    0.067043] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[    0.067149] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[    0.067347] NET: Registered protocol family 1
[    0.070009] PCI: CLS 0 bytes, default 64
[    0.070009] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.070009] Placing 64MB software IO TLB between ffff8800fb7fb000 - ffff8800ff7fb000
[    0.070009] software IO TLB at phys 0xfb7fb000 - 0xff7fb000
[    0.072259] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.072597] audit: initializing netlink socket (disabled)
[    0.072627] type=2000 audit(1304608433.099:1): initialized
[    0.096031] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.098001] VFS: Disk quotas dquot_6.5.2
[    0.098076] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.098825] fuse init (API version 7.16)
[    0.098933] msgmni has been set to 14924
[    0.099356] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.099423] io scheduler noop registered
[    0.099431] io scheduler deadline registered (default)
[    0.099487] io scheduler cfq registered
[    0.099580] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.099699] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.100155] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    0.440229] Linux agpgart interface v0.103
[    0.441329] brd: module loaded
[    0.441840] loop: module loaded
[    0.444335] blkfront device/vbd/2049 num-ring-pages 1 nr_ents 32.
[    0.448802] blkfront device/vbd/2050 num-ring-pages 1 nr_ents 32.
[    0.449898] blkfront: xvde1: barriers disabled
[    0.450534] Fixed MDIO Bus: probed
[    0.450575] PPP generic driver version 2.4.2
[    0.450642] Initialising Xen virtual ethernet driver.
[    0.453787] blkfront: xvde2: barriers disabled
[    0.453890] tun: Universal TUN/TAP device driver, 1.6
[    0.453902] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    0.454012] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.454033] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.454047] uhci_hcd: USB Universal Host Controller Interface driver
[    0.454104] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    0.454935] i8042: No controller found
[    0.455000] mousedev: PS/2 mouse device common for all mice
[    0.455045] Setting capacity to 24576000
[    0.455063] xvde2: detected capacity change from 0 to 12582912000
[    0.495214] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.495280] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.495448] device-mapper: uevent: version 1.0.3
[    0.495539] device-mapper: ioctl: 4.20.0-ioctl (2011-02-02) initialised: dm-devel@redhat.com
[    0.495724] device-mapper: multipath: version 1.3.0 loaded
[    0.495734] device-mapper: multipath round-robin: version 1.0.0 loaded
[    0.495813] cpuidle: using governor ladder
[    0.495820] cpuidle: using governor menu
[    0.495825] EFI Variables Facility v0.08 2004-May-17
[    0.496143] TCP cubic registered
[    0.496282] NET: Registered protocol family 10
[    0.497112] NET: Registered protocol family 17
[    0.497137] Registering the dns_resolver key type
[    0.497188] powernow-k8: Found 1 AMD Opteron(tm) Processor 6128 (2 cpu cores) (version 2.20.00)
[    0.497216] [Firmware Bug]: powernow-k8: No compatible ACPI _PSS objects found.
[    0.497218] [Firmware Bug]: powernow-k8: Try again with latest BIOS.
[    0.497312] PM: Hibernation image not present or could not be loaded.
[    0.497327] registered taskstats version 1
[    0.497356] XENBUS: Device with no driver: device/console/0
[    0.497376]   Magic number: 1:252:3141
[    0.497426] /home/smb/oneiric-amd64/ubuntu-2.6/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    0.497437] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    0.497444] EDD information not available.
[    0.498007] Freeing unused kernel memory: 912k freed
[    0.498343] Write protecting the kernel read-only data: 12288k
[    0.506235] Freeing unused kernel memory: 2004k freed
[    0.507671] Freeing unused kernel memory: 1380k freed
[    0.560935] udev[59]: starting version 167
[    0.692128] EXT4-fs (xvde1): mounted filesystem with ordered data mode. Opts: (null)
[    0.908740] EXT4-fs (xvde1): re-mounted. Opts: (null)
[    0.938669] udev[206]: starting version 167
[    0.968617] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    1.186280] type=1400 audit(1304608434.208:2): apparmor="STATUS" operation="profile_load" name="/sbin/dhclient" pid=294 comm="apparmor_parser"
[    1.191109] type=1400 audit(1304608434.218:3): apparmor="STATUS" operation="profile_load" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=294 comm="apparmor_parser"
[    1.191399] type=1400 audit(1304608434.218:4): apparmor="STATUS" operation="profile_load" name="/usr/lib/connman/scripts/dhclient-script" pid=294 comm="apparmor_parser"
[    1.535007] type=1400 audit(1304608434.558:5): apparmor="STATUS" operation="profile_replace" name="/sbin/dhclient" pid=401 comm="apparmor_parser"
[    1.538107] type=1400 audit(1304608434.558:6): apparmor="STATUS" operation="profile_replace" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=401 comm="apparmor_parser"
[    1.538405] type=1400 audit(1304608434.558:7): apparmor="STATUS" operation="profile_replace" name="/usr/lib/connman/scripts/dhclient-script" pid=401 comm="apparmor_parser"
[    1.547190] type=1400 audit(1304608434.568:8): apparmor="STATUS" operation="profile_load" name="/usr/sbin/tcpdump" pid=402 comm="apparmor_parser"
[   12.210059] eth0: no IPv6 routers present

Re: [Xen-devel] Re: [PATCH] Two patches fixing regression introduced by 'x86-64, mm: Put early page table high' - does not work.

[Konrad Rzeszutek Wilk]

On Thu, May 05, 2011 at 12:28:49PM -0400, Konrad Rzeszutek Wilk wrote:
> > >> OK, sounds like a plan then.  I like it because it doesn't affect the
> > >> native kernel.
> .. snip..
> 
> > First things first... are you pushing the workaround (you can add my
> > Acked-by:) or should I?
> 
> Grrr.. While it works on my machines, it does not work on some of the AMD opteron
> CPU machines. I asked Stefan Bader from Canonical to run a simple bootup test with
> 2.6.39-rc6 (Linus picked it already), and if it crashed, to use the 
> one that Stefano came up with.
> 
> And sure enough - it crashed and Stefano's fix worked. I've Stefano's
> patch in stable/bug-fixes-for-x86 and I am OK reverting the fix I came up.

To make it easier, I made a branch called 

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git stable/bug-fixes-for-rc6

that has all the right magic sauce in it:

Konrad Rzeszutek Wilk (1):
      Revert "xen/mmu: Add workaround "x86-64, mm: Put early page table high""

Sedat Dilek (1):
      x86/mm: Fix section mismatch derived from native_pagetable_reserve()

Stefano Stabellini (1):
      x86,xen: introduce x86_init.mapping.pagetable_reserve