Re: Failed to enable MMU when booting EFI on Seattle

[Suravee Suthikulanit <suravee.suthikulpanit@amd.com>]

On 10/7/2014 4:21 AM, Ian Campbell wrote:
> On Mon, 2014-10-06 at 22:29 -0500, Suravee Suthikulpanit wrote:
>> Ian/Roy,
>>
>> So, I have found that in arch/arm/arm64/head.S, after we enable MMU and
>> execute "br x1" (where x1 = 0x2005F8, the VA of "paging:"), it went to
>> address 0x2005F8, which has all zeros value. This is why we getting the
>> "Synchronous Exception"
>>
>> ......
>> 1:
>>           PRINT("- Turning on paging -\r\n")
>>
>>           ldr   x1, =paging            /* Explicit vaddr, not RIP-relative */
>>           mrs   x0, SCTLR_EL2
>>           orr   x0, x0, #SCTLR_M       /* Enable MMU */
>>           orr   x0, x0, #SCTLR_C       /* Enable D-cache */
>>           dsb   sy                     /* Flush PTE writes and finish
>> reads */
>> 	b .
>>           msr   SCTLR_EL2, x0          /* now paging is enabled */
>>           isb                          /* Now, flush the icache */
>>           br    x1                     /* Get a proper vaddr into PC */
>> paging:
>> .....
>>
>> Notes:
>> * This is not the case when booting non-EFI.
>>
>> * If I do "add x1, x1, x20", which puts the PA of paging into x1, it
>> seems to branch to "paging:" fine. Although, I don't this this is the
>> right thing to do since it should already be using VA).
>>
>> So, I inspected the page tables and compare the ones from booting with
>> and without EFI. At this point, it seems that the contents of the page
>> tables are set up properly. The only thing different is the phys-offset
>> and the location of the page table:
>>
>> BOOTING WITH EFI:
>>
>> 	x20 (phys-offset) = 0x83fc2d0000
>> 	x19 (start paddr) = 0x83fc4d0000
>> 	(.text starts at 2MB = 0x200000 offset)
>>
>> 	boot_pgtable (pa) = x20 + 0x2f2000 = 0x83fc5c2000
>> 	boot_first (pa)   = x20 + 0x2f3000 = 0x83fc5c3000
>> 	boot_second(pa)   = x20 + 0x2f5000 = 0x83fc5c5000
>> 	boot_third (pa)   = x20 + 0x2f6000 = 0x83fc5c6000
>>
>> BOOTING WITHOUT EFI:
>>
>> 	x20 (phys-offset) = 0x8008500000
>> 	x19 (start paddr) = 0x8008700000
>> 	(.text starts at 2MB = 0x200000 offset)
>>
>> 	boot_pgtable (pa) = x20 + 0x2f2000 = 0x80087f2000
>> 	boot_first (pa)   = x20 + 0x2f3000 = 0x80087f3000
>> 	boot_second(pa)   = x20 + 0x2f5000 = 0x80087f5000
>> 	boot_third (pa)   = x20 + 0x2f6000 = 0x80087f6000
>>
>> Here is the memory map available from UEFI:
>> 	
>> Shell> memmap
>> Type      Start            End              #pages             Attributes
>> RT_Data   0000008000000000-0000008000FFFFFF 0000000000001000
>> 800000000000000F
>> Available 0000008001000000-0000008007FFDFFF 0000000000006FFE
>> 000000000000000F
>> BS_Code   0000008007FFE000-0000008007FFFFFF 0000000000000002
>> 000000000000000F
>> Available 0000008008000000-000000801FFFDFFF 0000000000017FFE
>> 000000000000000F
>> BS_Data   000000801FFFE000-000000801FFFFFFF 0000000000000002
>> 000000000000000F
>> Available 0000008020000000-000000802FFFFFFF 0000000000010000
>> 000000000000000F
>> RT_Code   0000008030000000-0000008030000FFF 0000000000000001
>> 800000000000000F
>> Available 0000008030001000-00000083F0FFFFFF 00000000003C0FFF
>> 000000000000000F
>> BS_Data   00000083F1000000-00000083F101FFFF 0000000000000020
>> 000000000000000F
>> Available 00000083F1020000-00000083FC5D2FFF 000000000000B5B3
>> 000000000000000F
>> LoaderCode 00000083FC5D3000-00000083FC6B1FFF 00000000000000DF
>> 000000000000000F
>>
>> However, it seems that the location falls in the "Available", which
>> should be ok to use.
>
> Right, I don't see anything wrong with the location of the page tables.
>
> What do the contents of these PTs look like?
>
> In particular for Xen's load address of 0x200000 I think entries at
> boot_pgtable[0], boot_first[0], boot_second[0] and boot_third[0x20] are
> of interest. Do they correctly map their next level and/or the physical
> address with the actual Xen bits in?
>
> For completeness it would also be worth knowing the entries
> corresponding to the 1:1 pa mapping.
>
> For phys-offset 0x8008500000 those would be offsets [0x1, 0x0, 0x42,
> 0x100] (I think, but please do check my maths!). I think this will
> result in boot_pgtable[1] mapping boot_first_id which will contain a
> gigabyte page super mapping in slot [0].
>
> Phys-offset 0x83fc2d0000 ought to be offsets [0x1, 0xf, 0x1e1, 0xd].
> Which I would again expect to result in boot_pgtable[1] mapping
> boot_first_id with a gigabyte mapping in slot 0xf this time.
>
> I expect the PA mapping to be correct, since as you say we are able to
> run off it, it's worth checking though in case we are somehow creating
> VA and PA-1:1 mappings which conflict (e.g. writing boot_first instead
> of boot_first_id when creating the PA mappings -- that sort of thing).
>
> Ian.

So, I have experimented with:

         /*
          * Preserve x0 (fdt pointer) across call to __flush_dcache_area,
          * restore for entry into Xen.
          */
         mov   x20, x0

         /*
          * Flush dcache covering current runtime addresses
          * of xen text/data. Then flush all of icache.
          */
         adrp  x1, _start
         add   x1, x1, #:lo12:_start
         mov   x0, x1	<--- Fixed X0
         adrp  x2, _end
         add   x2, x2, #:lo12:_end
         sub   x1, x2, x1

         bl    __flush_dcache_area
         ic    ialluis
         tlbi  alle2     <---- Added this

Now, that I added the "tlbi alle2", it works after we enable MMU :) I 
guess even though we set up the new page table, but it was still using 
the stale info in the TLB.

On to the next hurdle:

Shell> FS0:xen -cfg=xen-seattle.cfg
Xen 4.5-unstable (c/s Mon Oct 6 10:16:52 2014 -0500 git:f0ffd29-dirty) 
EFI loader
Image: 0x00000083fbbca000-0x00000083fc4ca890
- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Setup boot first -
- Setup boot second -
- Setup boot third -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000008001000000 - 0000008007ffdfff
(XEN) RAM: 0000008007ffe000 - 0000008007ffffff
(XEN) RAM: 0000008008000000 - 000000801fffdfff
(XEN) RAM: 000000801fffe000 - 000000801fffffff
(XEN) RAM: 0000008020000000 - 000000802fffffff
(XEN) RAM: 0000008030001000 - 00000083f0ffffff
(XEN) RAM: 00000083f1000000 - 00000083f101ffff
(XEN) RAM: 00000083f1020000 - 00000083fbbc7fff
(XEN) RAM: 00000083fc4ce000 - 00000083fc4cefff
(XEN) RAM: 00000083fc6b2000 - 00000083fec25fff
(XEN) RAM: 00000083fec26000 - 00000083fee8bfff
(XEN) RAM: 00000083fee8c000 - 00000083ff225fff
(XEN) RAM: 00000083ff226000 - 00000083ff263fff
(XEN) RAM: 00000083ff265000 - 00000083ff2c4fff
(XEN) RAM: 00000083ffe70000 - 00000083ffffffff
(XEN)
(XEN) MODULE[0]: 00000083fc4cb000 - 00000083fc4ce000 Device Tree
(XEN) MODULE[1]: 00000083fbbca000 - 00000083fc4ca890 Kernel 
console=hvc0 console=ttyAMA0,115200 earlycon=pl011,0xe1010000 
show_styx_info root=/dev/sda2 rootwait maxcpus=1
(XEN) MODULE[2]: 0000008020000000 - 00000080209e6950 Kernel
(XEN)
(XEN) Command line: FS0:xen no-bootscrub console=dtuart conswitch=x 
dtuart=serial0 noreboot sync_console dom0_mem=256M dom0_max_vcpus=2
(XEN) Placing Xen at 0x000000802fe00000-0x0000008030000000
(XEN) Update BOOTMOD_XEN from 00000083fc4d0000-00000083fc5d2d81 => 
000000802fe00000-000000802ff02d81
(XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44
(XEN) CPU0: Unexpected Trap: Hypervisor
(XEN) ----[ Xen-4.5-unstable  arm64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) PC:     00000000002794b4 bootmem_region_add+0x194/0x1c4
(XEN) LR:     00000000002794ac
(XEN) SP:     00000000002afd50
(XEN) CPSR:   800003c9 MODE:64-bit EL2h (Hypervisor, handler)
(XEN)      X0: 0000800000000000  X1: 0000800000000000  X2: 0000000000000000
(XEN)      X3: 0000800000000000  X4: ffffffffffffffff  X5: 0000000000000000
(XEN)      X6: 0000800000000010  X7: 000000000000000a  X8: 0000000000000000
(XEN)      X9: 0000000000000010 X10: 00000000002afbb8 X11: 0000000000000038
(XEN)     X12: 000000000000000a X13: 000000000025d380 X14: 0000000000000030
(XEN)     X15: 0000000000400000 X16: 0000000000000000 X17: 0000000000287fdc
(XEN)     X18: 000000802feea000 X19: 0000000008001001 X20: 0000000000290048
(XEN)     X21: 0000000008007ffe X22: 0000000000000000 X23: 0000008007ffe000
(XEN)     X24: 00000000002794e4 X25: 00000000002794e4 X26: ffffffffffffffff
(XEN)     X27: 0000000000298038 X28: 0000008007ffe000  FP: 00000000002afd50
(XEN)
(XEN)   VTCR_EL2: 80000000
(XEN)  VTTBR_EL2: 0000000000000000
(XEN)
(XEN)  SCTLR_EL2: 30cd183d
(XEN)    HCR_EL2: 000000000038643f
(XEN)  TTBR0_EL2: 000000802feec000
(XEN)
(XEN)    ESR_EL2: 96000044
(XEN)  HPFAR_EL2: 0000000000000000
(XEN)    FAR_EL2: 0000800000000000
(XEN)
(XEN) Xen stack trace from sp=00000000002afd50:
(XEN)    00000000002afd80 0000000000279530 0000000000290048 0000000000000000
(XEN)    0000008001000000 0000008001000000 00000000002afdd0 000000000024b154
(XEN)    0000000000000000 0000000000000000 0000008001000000 0000008001000000
(XEN)    0000008007ffe000 00000000002794e4 00000000002afe20 00000000002834ac
(XEN)    00000000002afe20 0000000000283d50 0000000000298030 0000000000000418
(XEN)    0000008001000000 0000008007ffe000 0000008007ffe000 0000000000000000
(XEN)    0000000000298038 000000000fffffff 00000083fff702b0 0000000000200690
(XEN)    00000083fc4d0000 00000083fc2d0000 00000083fc4cb000 0000000000000000
(XEN)    0000000000400000 0000000000000000 0000000000000001 00000083fc544be0
(XEN)    00000083fc5800f0 00000083fc5800e0 0000000000000000 0000000000003000
(XEN)    00000083fc4cb000 0000000006ffe000 0000000000000000 0000008001000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<00000000002794b4>] bootmem_region_add+0x194/0x1c4 (PC)
(XEN)    [<00000000002794ac>] bootmem_region_add+0x18c/0x1c4 (LR)
(XEN)    [<0000000000279530>] init_boot_pages+0x4c/0x174
(XEN)    [<000000000024b154>] dt_unreserved_regions+0xbc/0xd0
(XEN)    [<0000000000283d50>] start_xen+0xc38/0xc58
(XEN)    [<0000000000200690>] paging+0x88/0xc0
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) CPU0: Unexpected Trap: Hypervisor
(XEN)
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)

Any thoughts??

Thanks,

Suravee