PAE xen + linux kernel boots ...

PAE xen + linux kernel boots ...

[Gerd Knorr]

... linux userspace doesn't come up yet though, thus it is still some
way to go.  I didn't even chech whenever non-pae still works with my
current code base.

If you want to have a look nevertheless, patches are here:

  Xen: http://dl.bytesex.org/patches/xen-4/pae-support
  Linux: http://dl.bytesex.org/patches/linux-xen-4/

The most intresting patch for Linux is "pae-support" as well (for
those who just want read the diffs).  To build a kernel you need
all of them, on top of a vanilla 2.6.11 kernel, applied in the
order specified in the "series" file.

Comments?  Questions?

  Gerd

==============================[ cut here ]==============================
 __  __            _____  ___         _                _ 
 \ \/ /___ _ __   |___ / / _ \     __| | _____   _____| |
  \  // _ \ '_ \    |_ \| | | |__ / _` |/ _ \ \ / / _ \ |
  /  \  __/ | | |  ___) | |_| |__| (_| |  __/\ V /  __/ |
 /_/\_\___|_| |_| |____(_)___/    \__,_|\___| \_/ \___|_|
                                                         
 http://www.cl.cam.ac.uk/netos/xen
 University of Cambridge Computer Laboratory

 Xen version 3.0-devel (kraxel@strusel007.de) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) Mon Apr 25 18:46:53 CEST 2005
 Latest ChangeSet: information unavailable

(XEN) Physical RAM map:
(XEN)  0000000000000000 - 000000000009f000 (usable)
(XEN)  000000000009f000 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 000000003fffc000 (usable)
(XEN)  000000003fffc000 - 000000003ffff000 (ACPI data)
(XEN)  000000003ffff000 - 0000000040000000 (ACPI NVS)
(XEN)  00000000ffff0000 - 0000000100000000 (reserved)
(XEN) System RAM: 1023MB (1048172kB)
(XEN) Xen heap: 10MB (10588kB)
(XEN) CPU0: Before vendor init, caps: 0383f9ff 00000000 00000000, vendor = 0
(XEN) CPU caps: 0383f9ff 00000000 00000000 00000000
(XEN) PAE enabled, limit: 16 GB
(XEN) ACPI: RSDP (v000 ASUS                                      ) @ 0x000f5a60
(XEN) ACPI: RSDT (v001 ASUS   P3B_F    0x30303031 MSFT 0x31313031) @ 0x3fffc000
(XEN) ACPI: FADT (v001 ASUS   P3B_F    0x30303031 MSFT 0x31313031) @ 0x3fffc080
(XEN) ACPI: BOOT (v001 ASUS   P3B_F    0x30303031 MSFT 0x31313031) @ 0x3fffc040
(XEN) ACPI: DSDT (v001   ASUS P3B_F    0x00001000 MSFT 0x0100000b) @ 0x00000000
(XEN) Local APIC disabled by BIOS -- reenabling.
(XEN) Found and enabled local APIC!
(XEN) Using scheduler: Borrowed Virtual Time (bvt)
(XEN) Initializing CPU#0
(XEN) write_ptbase: type 0 pfn 101
(XEN) Detected 451.030 MHz processor.
(XEN) CPU0 booted
(XEN) SMP motherboard not detected.
(XEN) enabled ExtINT on CPU#0
(XEN) Using local APIC timer interrupts.
(XEN) Calibrating APIC timer for CPU0...
(XEN) ..... CPU clock speed is 451.0244 MHz.
(XEN) ..... host bus clock speed is 100.2273 MHz.
(XEN) ..... bus_scale = 0x000066A2
(XEN) Time init:
(XEN) .... System Time: 12215956ns
(XEN) .... cpu_freq:    00000000:1AE22F74
(XEN) .... scale:       00000002:3796AEE9
(XEN) .... Wall Clock:  1114447958s 30000us
(XEN) PCI: PCI BIOS revision 2.10 entry at 0xf08c0, last bus=1
(XEN) PCI: Using configuration type 1
(XEN) PCI: Probing PCI hardware
(XEN) PCI: Probing PCI hardware (bus 00)
(XEN) PCI: Using IRQ router PIIX/ICH [8086/7110] at 00:04.0
(XEN) Limiting direct PCI/PCI transfers.
(XEN) mtrr: v2.0 (20020519)
(XEN) (file=grant_table.c, line=1229) Grant table init
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen-ELF header found: 'GUEST_OS=linux,GUEST_VER=2.6,XEN_VER=3.0,VIRT_BASE=0xC0000000,PAE=yes,LOADER=generic'
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   10000000->20000000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: c0100000->c0383234
(XEN)  Init. ramdisk: c0384000->c25e7800
(XEN)  Phys-Mach map: c25e8000->c2628000
(XEN)  Page tables:   c2628000->c2641000
(XEN)  Start info:    c2641000->c2642000
(XEN)  Boot stack:    c2642000->c2643000
(XEN)  TOTAL:         c0000000->c2800000
(XEN)  ENTRY ADDRESS: c0100000
(XEN) write_ptbase: type 78000002 pfn 12628
(XEN) Initrd len 0x2263800, start at 0xc0384000
(XEN) write_ptbase: type 0 pfn 101
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen).
(XEN) write_ptbase: type 78000002 pfn 12628
(XEN) write_ptbase: type 78000002 pfn 10102
Linux version 2.6.11-xen-unstable (kraxel@eskarina) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) #125 Mon Apr 25 18:44:11 CEST 2005
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000010000000 (usable)
0MB HIGHMEM available.
256MB LOWMEM available.
DMI 2.3 present.
IRQ lockup detection disabled
Allocating PCI resources starting at 10000000 (gap: 10000000:f0000000)
Built 1 zonelists
Kernel command line: 
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 32768 bytes)
Xen reported: 451.030 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 220928k/262144k available (1410k kernel code, 41084k reserved, 848k data, 116k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
(XEN) mod_l3_entry: ptr fec5b000 entry 0 | type 78000003 pfn 10102
(XEN) mod_l3_entry: ptr fec5e008 entry 0 | type 78000003 pfn 10102
(XEN) mod_l3_entry: ptr fec61010 entry 0 | type 78000003 pfn 10102
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: L2 cache: 512K
CPU: Intel Pentium III (Katmai) stepping 03
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
checking if image is initramfs... it is
Freeing initrd memory: 35214k freed
NET: Registered protocol family 16
PCI: Using configuration type Xen
xen_mem: Initialising balloon driver.
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Probing PCI hardware (bus 01)
PCI: Probing PCI hardware
Grant table initialized
Initializing Cryptographic API
Limiting direct PCI/PCI transfers.
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
io scheduler noop registered
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
Xen virtual console successfully installed as ttyS0
Event-channel device installed.
Initialising Xen netif backend
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard on isa0060/serio0
NET: Registered protocol family 2
IP: routing cache hash table of 2048 buckets, 16Kbytes
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 16384 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
NET: Registered protocol family 1
Freeing unused kernel memory: 116k freed
(XEN) debugtrace_dump() starting
(XEN) debugtrace_dump() finished
(XEN) BUG at mm.c:706
(XEN) CPU:    0
(XEN) EIP:    0808:[<ff137d8f>]      
(XEN) EFLAGS: 00010292   CONTEXT: hypervisor
(XEN) eax: ff177418   ebx: 00000000   ecx: 00000000   edx: 00000000
(XEN) esi: 00007ff0   edi: c0351018   ebp: ff107e00   esp: ff107dc8
(XEN) ds: 0810   es: 0810   fs: 0810   gs: 0810   ss: 0810   cs: 0808
(XEN) Stack trace from ESP=ff107dc8:
(XEN)    ff16fead ff16ffd7 000002c2 [ff156b8e] ff179730 10351000 00000001 00000063 
(XEN)    cccccccc cccccccc ff1ab000 00000000 00007ff0 c0351018 ff107e40 [ff130bcc] 
(XEN)    fec71000 cccccccc 00010351 ffbfac00 c0000000 00000131 00000004 00000132 
(XEN)    00000131 00000004 fec71000 c0000000 00010351 ffbfac00 ff107e60 [ff131f63] 
(XEN)    f6984f98 00000131 c28e1dd8 00000069 60000000 10102000 ff107eb0 [ff13260a] 
(XEN)    f6984f98 60000000 ffbfac00 ffbfac00 ffbfac00 80000002 80000003 00000004 
(XEN)    00000000 f0000000 f0000000 00000004 60000001 f0000000 f6984fac f0000000 
(XEN)    f0000000 60000001 ff107ee0 [ff12fea7] f6984f98 60000000 00000000 00000000 
(XEN)    00000000 00000001 ffbfac00 00000000 00007ff0 f6984f98 ff107fb0 [ff132e0f] 
(XEN)    00010351 60000000 ffbfac00 ff107fb8 c262ea88 ff178ee0 00000004 00000000 
(XEN)    002736ca 00000000 8288f4e0 00000000 82b02baa 00000000 00000001 ffbf8080 
(XEN)    ffbf8080 ff107f4c [ff1571da] c01e3d3b 00000000 ffbfac00 00000000 ffbfac00 
(XEN)    00000601 00000000 00000000 ff107f98 c28e1ddc 0000000c 0001262e [ff1132de] 
(XEN)    ffbf8080 ff107fac [ff147606] ffbfac00 ffbf8080 f6984f98 00000000 60000000 
(XEN)    00000000 00000001 00000000 00000000 00000002 00010351 c0351120 c262ea88 
(XEN)    ffbf8080 00007ff0 c0351000 [ff1563b3] c28e1ddc 00000001 00000000 00007ff0 
(XEN)    c0351018 c0351000 0000001a 000e0003 c011550f 00000061 00000246 c28e1dcc 
(XEN)    00000069 0000007b 0000007b 00000000 00000000 ffbf8080 
(XEN) Call Trace from ESP=ff107dc8:
(XEN)    [<ff156b8e>] [<ff130bcc>] [<ff131f63>] [<ff13260a>] [<ff12fea7>] [<ff132e0f>] 
(XEN)    [<ff1571da>] [<ff1132de>] [<ff147606>] [<ff1563b3>] 
(XEN) debugtrace_dump() starting
(XEN) debugtrace_dump() finished

****************************************
CPU0 FATAL TRAP: vector = 6 (invalid operand)
[error_code=0000]
Aieee! CPU0 is toast...
****************************************

Reboot in five seconds...

Re: PAE xen + linux kernel boots ...

[Gerd Knorr]

Gerd Knorr <kraxel@bytesex.org> writes:

> ... linux userspace doesn't come up yet though, thus it is still some
> way to go.

Well, now it does, boots up to a login prompt ;)

>   Xen: http://dl.bytesex.org/patches/xen-4/pae-support
>   Linux: http://dl.bytesex.org/patches/linux-xen-4/

New versions of the patches are in xen-5 + linux-xen-5, they are
against changeset 1.1385 (xen) 2.6.11 vanilla (linux-xen).

Currently open issues are:

  * general: more testing, cleanups, drop debug code, review
    all the FIXME's in the code ...

  * interfaces: do_update_va_mapping() and do_mmu_update() use 32bit
    values for page table entry updates, which will stop working as
    soon as we'll attempt to use memory above 4GB (which isn't the
    case yet).

  * xen: quite a few places are not fixed yet to deal with physical
    addresses above 4GB, i.e. are still 32 bit only.

  * xen: writable pagetables need fixing, they use emulation at the
    moment.  I was very surprised that it did "just work", but that
    was just the bug fixed by Keir recently causing the code do
    emulation all the time ;-)

  * xenlinux: Not sure how to deal best with the vsyscall page.
    xenlinux places it just before the hypervisor area, which is a
    different address in pae/non-pae mode.  And there seems to be no
    obvious way to make that depend on CONFIG_X86_PAE, the linker
    script where the address is in is not processed by cpp ...

I think that's it for the moment.  Comments are welcome.

Enjoy,

  Gerd

-- 
#define printk(args...) fprintf(stderr, ## args)

Re: PAE xen + linux kernel boots ...

[Chris Wright]

* Gerd Knorr (kraxel@bytesex.org) wrote:
> 
>   * xenlinux: Not sure how to deal best with the vsyscall page.
>     xenlinux places it just before the hypervisor area, which is a
>     different address in pae/non-pae mode.  And there seems to be no
>     obvious way to make that depend on CONFIG_X86_PAE, the linker
>     script where the address is in is not processed by cpp ...

I have some local changes for xenlinux to use real vsyscall.ld.S and
preprocess, they aren't yet stable, but should fix this.

thanks,
-chris

Re: PAE xen + linux kernel boots ...

[Gerd Knorr]

On Thu, Apr 28, 2005 at 11:41:58AM -0700, Chris Wright wrote:
> * Gerd Knorr (kraxel@bytesex.org) wrote:
> 
> I have some local changes for xenlinux to use real vsyscall.ld.S and
> preprocess, they aren't yet stable, but should fix this.

Ah, cool, so I don't have to worry about that one ;)

  Gerd

-- 
#define printk(args...) fprintf(stderr, ## args)

Re: PAE xen + linux kernel boots ...

[Scott Parish]

[-- Attachment #1: Type: text/plain, Size: 200 bytes --]

On Wed, Apr 27, 2005 at 02:03:17PM +0200, Gerd Knorr wrote:

>   * general: more testing, cleanups, drop debug code, review
>     all the FIXME's in the code ...

Here's one...

sRp

-- 
Scott Parish

[-- Attachment #2: show-page-walk.diff --]
[-- Type: text/plain, Size: 1560 bytes --]

diff -rN -u -p old-xen-unstable/xen/arch/x86/x86_32/traps.c new-xen-unstable/xen/arch/x86/x86_32/traps.c
--- old-xen-unstable/xen/arch/x86/x86_32/traps.c	2005-04-27 20:19:53.000000000 +0000
+++ new-xen-unstable/xen/arch/x86/x86_32/traps.c	2005-04-30 08:45:13.000000000 +0000
@@ -159,17 +159,32 @@ void show_registers(struct xen_regs *reg
 
 void show_page_walk(unsigned long addr)
 {
-#if defined(__i386__) && defined(CONFIG_X86_PAE)
-    printk("FIXME: PAE code needed here: %s:%d (%s)\n",
-	   __FILE__, __LINE__, __FUNCTION__);
-#else
-    unsigned long page;
-
     if ( addr < PAGE_OFFSET )
         return;
 
     printk("Pagetable walk from %08lx:\n", addr);
     
+#if defined(__i386__) && defined(CONFIG_X86_PAE)
+    l3_pgentry_t *pl3e;
+    l2_pgentry_t *pl2e;
+    l1_pgentry_t *pl1e;
+        
+    pl3e = &idle_pg_table[l3_table_offset(addr)];
+    printk(" L3 = 0x%016llx\n", l3e_get_value(*pl3e));
+
+    pl2e = l3e_to_l2e(*pl3e) + l2_table_offset(addr);
+
+    printk(" L2 = 0x%016llx %s\n", l2e_get_value(*pl2e),
+           (l2e_get_flags(*pl2e) & _PAGE_PSE) ? "(2MB)" : "");
+    if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) ||
+         (l2e_get_flags(*pl2e) & _PAGE_PSE) )
+        return;
+
+    pl1e = l2e_to_l1e(*pl2e) + l1_table_offset(addr);
+    printk(" L1 = 0x%016llx\n", l1e_get_value(*pl1e));
+#else
+    unsigned long page;
+    
     page = l2e_get_value(idle_pg_table[l2_table_offset(addr)]);
     printk(" L2 = %08lx %s\n", page, (page & _PAGE_PSE) ? "(4MB)" : "");
     if ( !(page & _PAGE_PRESENT) || (page & _PAGE_PSE) )

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PAE xen + linux kernel boots ...

[Scott Parish]

[-- Attachment #1: Type: text/plain, Size: 691 bytes --]

On Wed, Apr 27, 2005 at 02:03:17PM +0200, Gerd Knorr wrote:

> Well, now it does, boots up to a login prompt ;)

I'm still not quite there, but the attached patch gets me a lot
closer.

The hypervisor was taking a pagefault in ptwr_emulated_update() when
pl1e (a map_domain_mem() mapped page) was dereferenced to be
copied. pl1e is a 64 bit type with pae, but only the first 4 bytes
were getting mapped, and there was a case where pl1e would straddle
a page boundary, so it was all over when the high bits were read.

There's probably a better solution, but this at least got me past
linux's kernel_physical_mapping_init(), which is where i was
consistently crashing.

sRp

-- 
Scott Parish

[-- Attachment #2: map-pairs.diff --]
[-- Type: text/plain, Size: 2364 bytes --]

diff -rN -u -p old-xen-unstable/xen/arch/x86/x86_32/domain_page.c new-xen-unstable/xen/arch/x86/x86_32/domain_page.c
--- old-xen-unstable/xen/arch/x86/x86_32/domain_page.c	2005-04-27 20:19:53.000000000 +0000
+++ new-xen-unstable/xen/arch/x86/x86_32/domain_page.c	2005-04-30 08:42:30.000000000 +0000
@@ -20,7 +20,7 @@
 #include <asm/hardirq.h>
 
 l1_pgentry_t *mapcache;
-static unsigned int map_idx, epoch, shadow_epoch[NR_CPUS];
+static unsigned int map_idx = 0, epoch, shadow_epoch[NR_CPUS];
 static spinlock_t map_lock = SPIN_LOCK_UNLOCKED;
 
 /* Use a spare PTE bit to mark entries ready for recycling. */
@@ -39,7 +39,7 @@ static void flush_all_ready_maps(void)
 }
 
 
-void *map_domain_mem(unsigned long pa)
+void *map_domain_mem(unsigned long long pa)
 {
     unsigned long va;
     unsigned int idx, cpu = smp_processor_id();
@@ -62,7 +62,7 @@ void *map_domain_mem(unsigned long pa)
     }
 
     do {
-        idx = map_idx = (map_idx + 1) & (MAPCACHE_ENTRIES - 1);
+        idx = map_idx = (map_idx + 2) & (MAPCACHE_ENTRIES - 1);
         if ( unlikely(idx == 0) )
         {
             ASSERT(flush_count++ == 0);
@@ -75,6 +75,7 @@ void *map_domain_mem(unsigned long pa)
     while ( l1e_get_flags(cache[idx]) & _PAGE_PRESENT );
 
     cache[idx] = l1e_create_phys(pa, __PAGE_HYPERVISOR);
+    cache[idx + 1] = l1e_create_phys(pa + sizeof(u32), __PAGE_HYPERVISOR);
 
     spin_unlock(&map_lock);
 
@@ -89,4 +90,5 @@ void unmap_domain_mem(void *va)
     ASSERT(va < (void *)MAPCACHE_VIRT_END);
     idx = ((unsigned long)va - MAPCACHE_VIRT_START) >> PAGE_SHIFT;
     l1e_add_flags(&mapcache[idx], READY_FOR_TLB_FLUSH);
+    l1e_add_flags(&mapcache[idx + 1], READY_FOR_TLB_FLUSH);
 }
diff -rN -u -p old-xen-unstable/xen/include/asm-x86/x86_32/domain_page.h new-xen-unstable/xen/include/asm-x86/x86_32/domain_page.h
--- old-xen-unstable/xen/include/asm-x86/x86_32/domain_page.h	2005-04-27 20:19:53.000000000 +0000
+++ new-xen-unstable/xen/include/asm-x86/x86_32/domain_page.h	2005-04-30 08:05:25.000000000 +0000
@@ -19,7 +19,7 @@ extern l1_pgentry_t *mapcache;
  * The entire page containing that VA is now accessible until a 
  * corresponding call to unmap_domain_mem().
  */
-extern void *map_domain_mem(unsigned long pa);
+extern void *map_domain_mem(unsigned long long pa);
 
 /*
  * Pass a VA within a page previously mapped with map_domain_mem().

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PAE xen + linux kernel boots ...

[Scott Parish]

On Sat, Apr 30, 2005 at 09:01:17AM +0000, Scott Parish wrote:

> On Wed, Apr 27, 2005 at 02:03:17PM +0200, Gerd Knorr wrote:
> 
> > Well, now it does, boots up to a login prompt ;)
> 
> pl1e would straddle a page boundary

I swear there is a muse associated with the send button on email
clients.

In this case the epiphany was the obvious--the problem was that we're
missing alignment. But why?

On the linux side of things we have the following in pgtable-3level.h:

   #if 1 /* writable pagetables */
   static inline void set_pte(pte_t *ptep, pte_t pte)
   {
	    ptep->pte_high = pte.pte_high;
	    smp_wmb();
	    ptep->pte_low = pte.pte_low;
   }
   ...

Here's what (i'm thinking) is going on. We go to set the high bits
(first for atomicy: we don't set the active bit till last), but take
a page fault, on the high bits--a 4 byte offset.

Switch to xen, which is going to emulate some instructions and fake
the writing. We eventually end up in ptwr_emulated_update(), who among
other things, tries to copy the full l1_pgentry_t (64bits), but from
the 4 byte offset, that is the 4 high bytes and then 4 bytes of
undefined memory that may even be in another page.

sRp

-- 
Scott Parish

Re: PAE xen + linux kernel boots ...

[Keir Fraser]

On 30 Apr 2005, at 10:51, Scott Parish wrote:

> Switch to xen, which is going to emulate some instructions and fake
> the writing. We eventually end up in ptwr_emulated_update(), who among
> other things, tries to copy the full l1_pgentry_t (64bits), but from
> the 4 byte offset, that is the 4 high bytes and then 4 bytes of
> undefined memory that may even be in another page.

There's code in the 32-bit ptwr_emulated_update() to turn a sub-pte 
access into a full-pte access. Either this is missing/broken in the 
64-bit version, or the emulator is broken and passing the wrong operand 
size.

  -- Keir

Re: PAE xen + linux kernel boots ...

[Gerd Knorr]

> +#if defined(__i386__) && defined(CONFIG_X86_PAE)
> +    l3_pgentry_t *pl3e;
> +    l2_pgentry_t *pl2e;
> +    l1_pgentry_t *pl1e;
> +        
> +    pl3e = &idle_pg_table[l3_table_offset(addr)];
> +    printk(" L3 = 0x%016llx\n", l3e_get_value(*pl3e));

Well, that isn't needed.

>      page = l2e_get_value(idle_pg_table[l2_table_offset(addr)]);

Just make that "idle_pg_table_l2[l2_linear_offset(addr)]" should work
ok.  The idle_pg tables are contignous in physical (and virtual) memory,
so you basically don't have to care about the idle_pg_table_l3 at all
and can simply use idle_pg_table_l2 directly.

  Gerd

-- 
#define printk(args...) fprintf(stderr, ## args)

Re: PAE xen + linux kernel boots ...

[Scott Parish]

On Sun, May 01, 2005 at 01:01:54AM +0200, Gerd Knorr wrote:

> > The hypervisor was taking a pagefault in ptwr_emulated_update() when
> > pl1e (a map_domain_mem() mapped page) was dereferenced to be
> > copied. pl1e is a 64 bit type with pae, but only the first 4 bytes
> > were getting mapped, and there was a case where pl1e would straddle
> > a page boundary,
> 
> Huh?  page table entries must be 8-byte aligned, so they never ever
> can cross a page border.  Must be something else.

Right, the bug is in the partial write support in ptwr_emulated_update().
Working on a patch now.

sRp

-- 
Scott Parish

Re: PAE xen + linux kernel boots ...

[Gerd Knorr]

> The hypervisor was taking a pagefault in ptwr_emulated_update() when
> pl1e (a map_domain_mem() mapped page) was dereferenced to be
> copied. pl1e is a 64 bit type with pae, but only the first 4 bytes
> were getting mapped, and there was a case where pl1e would straddle
> a page boundary,

Huh?  page table entries must be 8-byte aligned, so they never ever
can cross a page border.  Must be something else.

> -void *map_domain_mem(unsigned long pa)
> +void *map_domain_mem(unsigned long long pa)

Hmm, I guess the most sane approach is to add a new type for 
physical addresses, simply using "unsigned long long" isn't
a good idea ...

> -        idx = map_idx = (map_idx + 1) & (MAPCACHE_ENTRIES - 1);
> +        idx = map_idx = (map_idx + 2) & (MAPCACHE_ENTRIES - 1);

>      cache[idx] = l1e_create_phys(pa, __PAGE_HYPERVISOR);
> +    cache[idx + 1] = l1e_create_phys(pa + sizeof(u32), __PAGE_HYPERVISOR);

That looks a bit fishy, like hiding a bug somewhere else.
And most likely will break for non-pae ...

  Gerd

-- 
#define printk(args...) fprintf(stderr, ## args)

Re: PAE xen + linux kernel boots ...

[Gerd Knorr]

> On the linux side of things we have the following in pgtable-3level.h:
> 
>    #if 1 /* writable pagetables */
>    static inline void set_pte(pte_t *ptep, pte_t pte)
>    {
> 	    ptep->pte_high = pte.pte_high;
> 	    smp_wmb();
> 	    ptep->pte_low = pte.pte_low;
>    }
>    ...

Note there is also set_pte_atomic ...

> Switch to xen, which is going to emulate some instructions and fake
> the writing. We eventually end up in ptwr_emulated_update(), who among
> other things, tries to copy the full l1_pgentry_t (64bits), but from
> the 4 byte offset, that is the 4 high bytes and then 4 bytes of
> undefined memory that may even be in another page.

Having a close look at the emulation is on my todo list.

Note that ptwr_emulated_update takes "unsigned long", i.e. 32-bit
values (on x86_32) as parameters, so chances are pretty good that
there are issues with 64bit updates.  It works fine for me nevertheless,
for some reason, maybe just pure luck ;)

Turning off emulation works fine for me as well btw. (just delete
the tree lines which force the emulation path for PAE), so I obviously
got the PGT_va backref stuff right ;)

  Gerd

PS: there is revision #6 of the patches on the usual location,
    I hadn't announced those yet.

-- 
#define printk(args...) fprintf(stderr, ## args)

Re: PAE xen + linux kernel boots ...

[Scott Parish]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

On Sat, Apr 30, 2005 at 11:54:06AM +0100, Keir Fraser wrote:

> 
> On 30 Apr 2005, at 10:51, Scott Parish wrote:
> 
> >Switch to xen, which is going to emulate some instructions and fake
> >the writing. We eventually end up in ptwr_emulated_update(), who among
> >other things, tries to copy the full l1_pgentry_t (64bits), but from
> >the 4 byte offset, that is the 4 high bytes and then 4 bytes of
> >undefined memory that may even be in another page.
>
>  There's code in the 32-bit ptwr_emulated_update() to turn a sub-pte
> access into a full-pte access. Either this is missing/broken in the
> 64-bit version, or the emulator is broken and passing the wrong
> operand size.

The bitshifting stuff in ptwr_emulated_update() had some problems, 
although its possible that it somehow worked for whatever cases it
was needed for before.

Adding a physaddr_t, and fixing the bitshifting took care of the
problem.

While i was at it, i wired up support for cmpxchg8b emulation under
PAE, and then tried to use set_pte_atomic(). That didn't quite do
the trick, but killing the machine_to_physical() conversion in
pte_val() got things working.

I was getting to dom0 prompt under the pae 5th patch release, testing
my patches under the 6th, the kernel fell apart in very late
boot. Haven't looked into this just yet.

sRp

-- 
Scott Parish

[-- Attachment #2: ptwr-em-up.diff --]
[-- Type: text/plain, Size: 3372 bytes --]

diff -rN -u -p old-xen-unstable-1/xen/arch/x86/mm.c new-xen-unstable-1/xen/arch/x86/mm.c
--- old-xen-unstable-1/xen/arch/x86/mm.c	2005-04-30 08:42:30.000000000 +0000
+++ new-xen-unstable-1/xen/arch/x86/mm.c	2005-05-01 07:24:40.000000000 +0000
@@ -2768,8 +2768,8 @@ void ptwr_flush(struct domain *d, const 
 
 static int ptwr_emulated_update(
     unsigned long addr,
-    unsigned long old,
-    unsigned long val,
+    physaddr_t old,
+    physaddr_t val,
     unsigned int bytes,
     unsigned int do_cmpxchg)
 {
@@ -2787,21 +2787,22 @@ static int ptwr_emulated_update(
     }
 
     /* Turn a sub-word access into a full-word access. */
-    /* FIXME: needs tweaks for PAE */
-    if ( (addr & ((BITS_PER_LONG/8)-1)) != 0 )
+    if (bytes != sizeof(physaddr_t))
     {
         int           rc;
-        unsigned long full;
-        unsigned int  mask = addr & ((BITS_PER_LONG/8)-1);
+        physaddr_t    full;
+        unsigned int  offset = addr & (sizeof(physaddr_t)-1);
+
         /* Align address; read full word. */
-        addr &= ~((BITS_PER_LONG/8)-1);
-        if ( (rc = x86_emulate_read_std(addr, &full, BITS_PER_LONG/8)) )
-            return rc;
+        addr &= ~(sizeof(physaddr_t)-1);
+        if ( (rc = x86_emulate_read_std(addr, (unsigned long *)&full,
+					sizeof(physaddr_t))) )
+            return rc; 
         /* Mask out bits provided by caller. */
-        full &= ~((1UL << (bytes*8)) - 1UL) << (mask*8);
+        full &= ~((((physaddr_t)1 << (bytes*8)) - 1) << (offset*8));
         /* Shift the caller value and OR in the missing bits. */
-        val  &= (1UL << (bytes*8)) - 1UL;
-        val <<= mask*8;
+        val  &= (((physaddr_t)1 << (bytes*8)) - 1);
+        val <<= (offset)*8;
         val  |= full;
     }
 
@@ -2889,11 +2890,29 @@ static int ptwr_emulated_cmpxchg(
     return ptwr_emulated_update(addr, old, new, bytes, 1);
 }
 
+#if defined(CONFIG_X86_PAE)
+static int ptwr_emulated_cmpxchg8b(
+    unsigned long addr,
+    unsigned long old,
+    unsigned long old_hi,
+    unsigned long new,
+    unsigned long new_hi)
+{
+    return ptwr_emulated_update(addr,
+				(u64)old_hi << sizeof(old_hi)*8 | old,
+				(u64)new_hi << sizeof(new_hi)*8 | new,
+				sizeof(u64), 1);
+}
+#endif
+
 static struct x86_mem_emulator ptwr_mem_emulator = {
     .read_std         = x86_emulate_read_std,
     .write_std        = x86_emulate_write_std,
     .read_emulated    = x86_emulate_read_std,
     .write_emulated   = ptwr_emulated_write,
+#if defined(CONFIG_X86_PAE)
+    .cmpxchg8b_emulated = ptwr_emulated_cmpxchg8b,
+#endif
     .cmpxchg_emulated = ptwr_emulated_cmpxchg
 };
 
diff -rN -u -p old-xen-unstable-1/xen/include/asm-x86/types.h new-xen-unstable-1/xen/include/asm-x86/types.h
--- old-xen-unstable-1/xen/include/asm-x86/types.h	2005-04-04 22:03:05.000000000 +0000
+++ new-xen-unstable-1/xen/include/asm-x86/types.h	2005-05-01 01:25:20.000000000 +0000
@@ -44,11 +44,17 @@ typedef signed long long s64;
 typedef unsigned long long u64;
 #define BITS_PER_LONG 32
 typedef unsigned int size_t;
+#if defined(CONFIG_X86_PAE)
+typedef u64 physaddr_t;
+#else
+typedef u32 physaddr_t;
+#endif
 #elif defined(__x86_64__)
 typedef signed long s64;
 typedef unsigned long u64;
 #define BITS_PER_LONG 64
 typedef unsigned long size_t;
+typedef u64 physaddr_t;
 #endif
 
 /* DMA addresses come in generic and 64-bit flavours.  */

[-- Attachment #3: bez-m2p.diff --]
[-- Type: text/plain, Size: 809 bytes --]

--- include/asm-xen/asm-i386/page.h.orig	2005-05-01 07:08:44.000000000 +0000
+++ include/asm-xen/asm-i386/page.h	2005-05-01 07:21:09.000000000 +0000
@@ -92,18 +92,7 @@ typedef struct { unsigned long long pgpr
     (((_x)&1) ? ((pgd_t) {phys_to_machine(_x)}) : ((pgd_t) {(_x)})); })
 #define __pmd(x) ({ unsigned long long _x = (x); \
     (((_x)&1) ? ((pmd_t) {phys_to_machine(_x)}) : ((pmd_t) {(_x)})); })
-static inline unsigned long long pte_val(pte_t x)
-{
-	unsigned long long ret;
-
-	if (x.pte_low) {
-		ret = x.pte_low | (unsigned long long)x.pte_high << 32;
-		ret = machine_to_phys(ret) | 1;
-	} else {
-		ret = 0;
-	}
-	return ret;
-}
+#define pte_val(x) ((x).pte_low | (unsigned long long)(x).pte_high << 32)
 static inline unsigned long long pmd_val(pmd_t x)
 {
 	unsigned long long ret = x.pmd;

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PAE xen + linux kernel boots ...

[Gerd Knorr]

> The bitshifting stuff in ptwr_emulated_update() had some problems, 
> although its possible that it somehow worked for whatever cases it
> was needed for before.
> 
> Adding a physaddr_t, and fixing the bitshifting took care of the
> problem.

Looks fine, added it into my patches.

> While i was at it, i wired up support for cmpxchg8b emulation under
> PAE, and then tried to use set_pte_atomic().

Hmm, that's dead code right now, isn't it?

> That didn't quite do the trick, but killing the machine_to_physical()
> conversion in pte_val() got things working.

That looks like hiding a bug somewhere else.  All the p??_val()
functions have a separate _ma() version which doesn't do the machine
to physical translation, so I'd suspect that happening due to some
place in the code using the wrong version of the macro, not due to
the macro itself being broken ...

> I was getting to dom0 prompt under the pae 5th patch release, testing
> my patches under the 6th, the kernel fell apart in very late
> boot. Haven't looked into this just yet.

There used to be some issues independent of the PAE stuff, at least with
the 1.1385 bk version (IIRC) I've used to create patchset #5 .  I've
created patchset #6 with 1.1399, worked fine.  Updated to 1.1413 today,
also no problems so far.

  Gerd

Re: PAE xen + linux kernel boots ...

[Scott Parish]

On Mon, May 02, 2005 at 04:03:05PM +0200, Gerd Knorr wrote:

> > While i was at it, i wired up support for cmpxchg8b emulation under
> > PAE, and then tried to use set_pte_atomic().
> 
> Hmm, that's dead code right now, isn't it?

Was dead code, but worked when i got through with it. I'll take another
look at the macros for the ma thing.

sRp

-- 
Scott Parish

Re: PAE xen + linux kernel boots ...

[Scott Parish]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

On Mon, May 02, 2005 at 04:03:05PM +0200, Gerd Knorr wrote:

> > That didn't quite do the trick, but killing the machine_to_physical()
> > conversion in pte_val() got things working.
> 
> That looks like hiding a bug somewhere else.  All the p??_val()
> functions have a separate _ma() version which doesn't do the machine
> to physical translation, so I'd suspect that happening due to some
> place in the code using the wrong version of the macro, not due to
> the macro itself being broken ...

I'm pretty sure that set_pte_atomic() should be calling the _ma version
of pte_val; compare and note that set_pte() is doing no machine_to_phys().

sRp

Signed-off-by: Scott Parish <srparish@us.ibm.com>

-- 
Scott Parish

[-- Attachment #2: set-pte-atomic.diff --]
[-- Type: text/plain, Size: 1046 bytes --]

* looking for pae@localhost/linux--pae--1.0--patch-2 to compare with
* comparing to pae@localhost/linux--pae--1.0--patch-2
M  include/asm-xen/asm-i386/page.h
M  include/asm-xen/asm-i386/pgtable-3level.h

* modified files

--- orig/include/asm-xen/asm-i386/page.h
+++ mod/include/asm-xen/asm-i386/page.h
@@ -116,7 +116,10 @@
 	if (ret) ret = machine_to_phys(ret) | 1;
 	return ret;
 }
-#define pte_val_ma(v) ((v).pte_low) /* FIXME */
+static inline unsigned long long pte_val_ma(pte_t x)
+{
+	return (unsigned long long)x.pte_high << 32 | x.pte_low;
+}
 #define HPAGE_SHIFT	21
 #else
 typedef struct { unsigned long pte_low; } pte_t;


--- orig/include/asm-xen/asm-i386/pgtable-3level.h
+++ mod/include/asm-xen/asm-i386/pgtable-3level.h
@@ -60,7 +60,7 @@
 	ptep->pte_low = pte.pte_low;
 }
 # define set_pte_atomic(pteptr,pteval) \
-		set_64bit((unsigned long long *)(pteptr),pte_val(pteval))
+		set_64bit((unsigned long long *)(pteptr),pte_val_ma(pteval))
 #else
 # define set_pte(pteptr,pteval)				\
 		xen_l1_entry_update((pteptr), (pteval))




[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PAE xen + linux kernel boots ...

[Scott Parish]

On Mon, May 02, 2005 at 04:03:05PM +0200, Gerd Knorr wrote:

> All the p??_val() functions have a separate _ma() version which
> doesn't do the machine to physical translation

What does "ma" stand for anyway?

sRp

-- 
Scott Parish

Re: PAE xen + linux kernel boots ...

[Kip Macy]

ma == machine address 
physical address is the logical offset in the allocated address space - machine 
address is the actual page frame in RAM.


			-Kip


					


On Wed, 4 May 2005, Scott Parish wrote:

> On Mon, May 02, 2005 at 04:03:05PM +0200, Gerd Knorr wrote:
> 
> > All the p??_val() functions have a separate _ma() version which
> > doesn't do the machine to physical translation
> 
> What does "ma" stand for anyway?
> 
> sRp
> 
> 

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: PAE xen + linux kernel boots ...

[Scott Parish]

That's what i thought. So, for example, why does pfn_pte() call
pfn_to_mfn() where pfn_pte_ma() does not. I bet i'm missing some
trivial thing, but this just seems backwards.

sRp

On Tue, May 03, 2005 at 08:22:53PM -0700, Kip Macy wrote:

> ma == machine address 
> physical address is the logical offset in the allocated address space - machine 
> address is the actual page frame in RAM.
> 
> 
> 			-Kip
> 
> 
> 					
> 
> 
> On Wed, 4 May 2005, Scott Parish wrote:
> 
> > On Mon, May 02, 2005 at 04:03:05PM +0200, Gerd Knorr wrote:
> > 
> > > All the p??_val() functions have a separate _ma() version which
> > > doesn't do the machine to physical translation
> > 
> > What does "ma" stand for anyway?
> > 
> > sRp
> > 
> > 
> 
> -- 
> "Debugging is twice as hard as writing the code in the first place.
> Therefore, if you write the code as cleverly as possible, you are,
> by definition, not smart enough to debug it." - Brian W. Kernighan
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

-- 
Scott Parish

Re: PAE xen + linux kernel boots ...

[Gerd Knorr]

On Wed, May 04, 2005 at 03:23:31AM +0000, Scott Parish wrote:
> That's what i thought. So, for example, why does pfn_pte() call
> pfn_to_mfn() where pfn_pte_ma() does not. I bet i'm missing some
> trivial thing, but this just seems backwards.

Well, the names are a bit confusing.

 * "machine address" is the address of the real machine.
 * "physical address" is the address within the virtual machine.

So what you have in the page table entries is the "machine
address", but the virtualized linux usually has to translate 
that into the "physical address", when looking up the page
information in the virtual machines frame table (mem_map[]
in linux IIRC) for example.

  Gerd

Re: PAE xen + linux kernel boots ...

[Gerd Knorr]

> I'm pretty sure that set_pte_atomic() should be calling the _ma version
> of pte_val; compare and note that set_pte() is doing no machine_to_phys().

Yes, you are right.  The __pte() macro used to create page table
entries does the translation, set_pte() just writes the values
and shouldn't translate anything.

Do you run a SMP kernel btw.?  That would explain why I didn't
notice that bug, I have a UP machine ...

  Gerd

-- 
#define printk(args...) fprintf(stderr, ## args)

Re: PAE xen + linux kernel boots ...

[Scott Parish]

On Wed, May 04, 2005 at 10:03:59AM +0200, Gerd Knorr wrote:

> > I'm pretty sure that set_pte_atomic() should be calling the _ma version
> > of pte_val; compare and note that set_pte() is doing no machine_to_phys().
> 
> Yes, you are right.  The __pte() macro used to create page table
> entries does the translation, set_pte() just writes the values
> and shouldn't translate anything.
> 
> Do you run a SMP kernel btw.?  That would explain why I didn't
> notice that bug, I have a UP machine ...

I'm running on a dual opteron box with 4g ram; from grub i'm specifying
nosmp and dom0_mem=512M. 

sRp

-- 
Scott Parish