* Failed to enable MMU when booting EFI on Seattle
@ 2014-10-07 3:29 Suravee Suthikulpanit
2014-10-07 9:21 ` Ian Campbell
0 siblings, 1 reply; 4+ messages in thread
From: Suravee Suthikulpanit @ 2014-10-07 3:29 UTC (permalink / raw)
To: Ian Campbell
Cc: Roy.Franz@linaro.org, Julien Grall,
stefano.stabellini@eu.citrix.com, xen-devel
Ian/Roy,
So, I have found that in arch/arm/arm64/head.S, after we enable MMU and
execute "br x1" (where x1 = 0x2005F8, the VA of "paging:"), it went to
address 0x2005F8, which has all zeros value. This is why we getting the
"Synchronous Exception"
......
1:
PRINT("- Turning on paging -\r\n")
ldr x1, =paging /* Explicit vaddr, not RIP-relative */
mrs x0, SCTLR_EL2
orr x0, x0, #SCTLR_M /* Enable MMU */
orr x0, x0, #SCTLR_C /* Enable D-cache */
dsb sy /* Flush PTE writes and finish
reads */
b .
msr SCTLR_EL2, x0 /* now paging is enabled */
isb /* Now, flush the icache */
br x1 /* Get a proper vaddr into PC */
paging:
.....
Notes:
* This is not the case when booting non-EFI.
* If I do "add x1, x1, x20", which puts the PA of paging into x1, it
seems to branch to "paging:" fine. Although, I don't this this is the
right thing to do since it should already be using VA).
So, I inspected the page tables and compare the ones from booting with
and without EFI. At this point, it seems that the contents of the page
tables are set up properly. The only thing different is the phys-offset
and the location of the page table:
BOOTING WITH EFI:
x20 (phys-offset) = 0x83fc2d0000
x19 (start paddr) = 0x83fc4d0000
(.text starts at 2MB = 0x200000 offset)
boot_pgtable (pa) = x20 + 0x2f2000 = 0x83fc5c2000
boot_first (pa) = x20 + 0x2f3000 = 0x83fc5c3000
boot_second(pa) = x20 + 0x2f5000 = 0x83fc5c5000
boot_third (pa) = x20 + 0x2f6000 = 0x83fc5c6000
BOOTING WITHOUT EFI:
x20 (phys-offset) = 0x8008500000
x19 (start paddr) = 0x8008700000
(.text starts at 2MB = 0x200000 offset)
boot_pgtable (pa) = x20 + 0x2f2000 = 0x80087f2000
boot_first (pa) = x20 + 0x2f3000 = 0x80087f3000
boot_second(pa) = x20 + 0x2f5000 = 0x80087f5000
boot_third (pa) = x20 + 0x2f6000 = 0x80087f6000
Here is the memory map available from UEFI:
Shell> memmap
Type Start End #pages Attributes
RT_Data 0000008000000000-0000008000FFFFFF 0000000000001000
800000000000000F
Available 0000008001000000-0000008007FFDFFF 0000000000006FFE
000000000000000F
BS_Code 0000008007FFE000-0000008007FFFFFF 0000000000000002
000000000000000F
Available 0000008008000000-000000801FFFDFFF 0000000000017FFE
000000000000000F
BS_Data 000000801FFFE000-000000801FFFFFFF 0000000000000002
000000000000000F
Available 0000008020000000-000000802FFFFFFF 0000000000010000
000000000000000F
RT_Code 0000008030000000-0000008030000FFF 0000000000000001
800000000000000F
Available 0000008030001000-00000083F0FFFFFF 00000000003C0FFF
000000000000000F
BS_Data 00000083F1000000-00000083F101FFFF 0000000000000020
000000000000000F
Available 00000083F1020000-00000083FC5D2FFF 000000000000B5B3
000000000000000F
LoaderCode 00000083FC5D3000-00000083FC6B1FFF 00000000000000DF
000000000000000F
However, it seems that the location falls in the "Available", which
should be ok to use.
Not sure at this point what I might be missing :(
Suravee
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Failed to enable MMU when booting EFI on Seattle
2014-10-07 3:29 Failed to enable MMU when booting EFI on Seattle Suravee Suthikulpanit
@ 2014-10-07 9:21 ` Ian Campbell
2014-10-08 1:45 ` Suravee Suthikulanit
0 siblings, 1 reply; 4+ messages in thread
From: Ian Campbell @ 2014-10-07 9:21 UTC (permalink / raw)
To: Suravee Suthikulpanit
Cc: Roy.Franz@linaro.org, Julien Grall,
stefano.stabellini@eu.citrix.com, xen-devel
On Mon, 2014-10-06 at 22:29 -0500, Suravee Suthikulpanit wrote:
> Ian/Roy,
>
> So, I have found that in arch/arm/arm64/head.S, after we enable MMU and
> execute "br x1" (where x1 = 0x2005F8, the VA of "paging:"), it went to
> address 0x2005F8, which has all zeros value. This is why we getting the
> "Synchronous Exception"
>
> ......
> 1:
> PRINT("- Turning on paging -\r\n")
>
> ldr x1, =paging /* Explicit vaddr, not RIP-relative */
> mrs x0, SCTLR_EL2
> orr x0, x0, #SCTLR_M /* Enable MMU */
> orr x0, x0, #SCTLR_C /* Enable D-cache */
> dsb sy /* Flush PTE writes and finish
> reads */
> b .
> msr SCTLR_EL2, x0 /* now paging is enabled */
> isb /* Now, flush the icache */
> br x1 /* Get a proper vaddr into PC */
> paging:
> .....
>
> Notes:
> * This is not the case when booting non-EFI.
>
> * If I do "add x1, x1, x20", which puts the PA of paging into x1, it
> seems to branch to "paging:" fine. Although, I don't this this is the
> right thing to do since it should already be using VA).
>
> So, I inspected the page tables and compare the ones from booting with
> and without EFI. At this point, it seems that the contents of the page
> tables are set up properly. The only thing different is the phys-offset
> and the location of the page table:
>
> BOOTING WITH EFI:
>
> x20 (phys-offset) = 0x83fc2d0000
> x19 (start paddr) = 0x83fc4d0000
> (.text starts at 2MB = 0x200000 offset)
>
> boot_pgtable (pa) = x20 + 0x2f2000 = 0x83fc5c2000
> boot_first (pa) = x20 + 0x2f3000 = 0x83fc5c3000
> boot_second(pa) = x20 + 0x2f5000 = 0x83fc5c5000
> boot_third (pa) = x20 + 0x2f6000 = 0x83fc5c6000
>
> BOOTING WITHOUT EFI:
>
> x20 (phys-offset) = 0x8008500000
> x19 (start paddr) = 0x8008700000
> (.text starts at 2MB = 0x200000 offset)
>
> boot_pgtable (pa) = x20 + 0x2f2000 = 0x80087f2000
> boot_first (pa) = x20 + 0x2f3000 = 0x80087f3000
> boot_second(pa) = x20 + 0x2f5000 = 0x80087f5000
> boot_third (pa) = x20 + 0x2f6000 = 0x80087f6000
>
> Here is the memory map available from UEFI:
>
> Shell> memmap
> Type Start End #pages Attributes
> RT_Data 0000008000000000-0000008000FFFFFF 0000000000001000
> 800000000000000F
> Available 0000008001000000-0000008007FFDFFF 0000000000006FFE
> 000000000000000F
> BS_Code 0000008007FFE000-0000008007FFFFFF 0000000000000002
> 000000000000000F
> Available 0000008008000000-000000801FFFDFFF 0000000000017FFE
> 000000000000000F
> BS_Data 000000801FFFE000-000000801FFFFFFF 0000000000000002
> 000000000000000F
> Available 0000008020000000-000000802FFFFFFF 0000000000010000
> 000000000000000F
> RT_Code 0000008030000000-0000008030000FFF 0000000000000001
> 800000000000000F
> Available 0000008030001000-00000083F0FFFFFF 00000000003C0FFF
> 000000000000000F
> BS_Data 00000083F1000000-00000083F101FFFF 0000000000000020
> 000000000000000F
> Available 00000083F1020000-00000083FC5D2FFF 000000000000B5B3
> 000000000000000F
> LoaderCode 00000083FC5D3000-00000083FC6B1FFF 00000000000000DF
> 000000000000000F
>
> However, it seems that the location falls in the "Available", which
> should be ok to use.
Right, I don't see anything wrong with the location of the page tables.
What do the contents of these PTs look like?
In particular for Xen's load address of 0x200000 I think entries at
boot_pgtable[0], boot_first[0], boot_second[0] and boot_third[0x20] are
of interest. Do they correctly map their next level and/or the physical
address with the actual Xen bits in?
For completeness it would also be worth knowing the entries
corresponding to the 1:1 pa mapping.
For phys-offset 0x8008500000 those would be offsets [0x1, 0x0, 0x42,
0x100] (I think, but please do check my maths!). I think this will
result in boot_pgtable[1] mapping boot_first_id which will contain a
gigabyte page super mapping in slot [0].
Phys-offset 0x83fc2d0000 ought to be offsets [0x1, 0xf, 0x1e1, 0xd].
Which I would again expect to result in boot_pgtable[1] mapping
boot_first_id with a gigabyte mapping in slot 0xf this time.
I expect the PA mapping to be correct, since as you say we are able to
run off it, it's worth checking though in case we are somehow creating
VA and PA-1:1 mappings which conflict (e.g. writing boot_first instead
of boot_first_id when creating the PA mappings -- that sort of thing).
Ian.
>
> Not sure at this point what I might be missing :(
>
> Suravee
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Failed to enable MMU when booting EFI on Seattle
2014-10-07 9:21 ` Ian Campbell
@ 2014-10-08 1:45 ` Suravee Suthikulanit
2014-10-08 4:02 ` Roy Franz
0 siblings, 1 reply; 4+ messages in thread
From: Suravee Suthikulanit @ 2014-10-08 1:45 UTC (permalink / raw)
To: Ian Campbell
Cc: Roy.Franz@linaro.org, Julien Grall,
stefano.stabellini@eu.citrix.com, xen-devel
On 10/7/2014 4:21 AM, Ian Campbell wrote:
> On Mon, 2014-10-06 at 22:29 -0500, Suravee Suthikulpanit wrote:
>> Ian/Roy,
>>
>> So, I have found that in arch/arm/arm64/head.S, after we enable MMU and
>> execute "br x1" (where x1 = 0x2005F8, the VA of "paging:"), it went to
>> address 0x2005F8, which has all zeros value. This is why we getting the
>> "Synchronous Exception"
>>
>> ......
>> 1:
>> PRINT("- Turning on paging -\r\n")
>>
>> ldr x1, =paging /* Explicit vaddr, not RIP-relative */
>> mrs x0, SCTLR_EL2
>> orr x0, x0, #SCTLR_M /* Enable MMU */
>> orr x0, x0, #SCTLR_C /* Enable D-cache */
>> dsb sy /* Flush PTE writes and finish
>> reads */
>> b .
>> msr SCTLR_EL2, x0 /* now paging is enabled */
>> isb /* Now, flush the icache */
>> br x1 /* Get a proper vaddr into PC */
>> paging:
>> .....
>>
>> Notes:
>> * This is not the case when booting non-EFI.
>>
>> * If I do "add x1, x1, x20", which puts the PA of paging into x1, it
>> seems to branch to "paging:" fine. Although, I don't this this is the
>> right thing to do since it should already be using VA).
>>
>> So, I inspected the page tables and compare the ones from booting with
>> and without EFI. At this point, it seems that the contents of the page
>> tables are set up properly. The only thing different is the phys-offset
>> and the location of the page table:
>>
>> BOOTING WITH EFI:
>>
>> x20 (phys-offset) = 0x83fc2d0000
>> x19 (start paddr) = 0x83fc4d0000
>> (.text starts at 2MB = 0x200000 offset)
>>
>> boot_pgtable (pa) = x20 + 0x2f2000 = 0x83fc5c2000
>> boot_first (pa) = x20 + 0x2f3000 = 0x83fc5c3000
>> boot_second(pa) = x20 + 0x2f5000 = 0x83fc5c5000
>> boot_third (pa) = x20 + 0x2f6000 = 0x83fc5c6000
>>
>> BOOTING WITHOUT EFI:
>>
>> x20 (phys-offset) = 0x8008500000
>> x19 (start paddr) = 0x8008700000
>> (.text starts at 2MB = 0x200000 offset)
>>
>> boot_pgtable (pa) = x20 + 0x2f2000 = 0x80087f2000
>> boot_first (pa) = x20 + 0x2f3000 = 0x80087f3000
>> boot_second(pa) = x20 + 0x2f5000 = 0x80087f5000
>> boot_third (pa) = x20 + 0x2f6000 = 0x80087f6000
>>
>> Here is the memory map available from UEFI:
>>
>> Shell> memmap
>> Type Start End #pages Attributes
>> RT_Data 0000008000000000-0000008000FFFFFF 0000000000001000
>> 800000000000000F
>> Available 0000008001000000-0000008007FFDFFF 0000000000006FFE
>> 000000000000000F
>> BS_Code 0000008007FFE000-0000008007FFFFFF 0000000000000002
>> 000000000000000F
>> Available 0000008008000000-000000801FFFDFFF 0000000000017FFE
>> 000000000000000F
>> BS_Data 000000801FFFE000-000000801FFFFFFF 0000000000000002
>> 000000000000000F
>> Available 0000008020000000-000000802FFFFFFF 0000000000010000
>> 000000000000000F
>> RT_Code 0000008030000000-0000008030000FFF 0000000000000001
>> 800000000000000F
>> Available 0000008030001000-00000083F0FFFFFF 00000000003C0FFF
>> 000000000000000F
>> BS_Data 00000083F1000000-00000083F101FFFF 0000000000000020
>> 000000000000000F
>> Available 00000083F1020000-00000083FC5D2FFF 000000000000B5B3
>> 000000000000000F
>> LoaderCode 00000083FC5D3000-00000083FC6B1FFF 00000000000000DF
>> 000000000000000F
>>
>> However, it seems that the location falls in the "Available", which
>> should be ok to use.
>
> Right, I don't see anything wrong with the location of the page tables.
>
> What do the contents of these PTs look like?
>
> In particular for Xen's load address of 0x200000 I think entries at
> boot_pgtable[0], boot_first[0], boot_second[0] and boot_third[0x20] are
> of interest. Do they correctly map their next level and/or the physical
> address with the actual Xen bits in?
>
> For completeness it would also be worth knowing the entries
> corresponding to the 1:1 pa mapping.
>
> For phys-offset 0x8008500000 those would be offsets [0x1, 0x0, 0x42,
> 0x100] (I think, but please do check my maths!). I think this will
> result in boot_pgtable[1] mapping boot_first_id which will contain a
> gigabyte page super mapping in slot [0].
>
> Phys-offset 0x83fc2d0000 ought to be offsets [0x1, 0xf, 0x1e1, 0xd].
> Which I would again expect to result in boot_pgtable[1] mapping
> boot_first_id with a gigabyte mapping in slot 0xf this time.
>
> I expect the PA mapping to be correct, since as you say we are able to
> run off it, it's worth checking though in case we are somehow creating
> VA and PA-1:1 mappings which conflict (e.g. writing boot_first instead
> of boot_first_id when creating the PA mappings -- that sort of thing).
>
> Ian.
So, I have experimented with:
/*
* Preserve x0 (fdt pointer) across call to __flush_dcache_area,
* restore for entry into Xen.
*/
mov x20, x0
/*
* Flush dcache covering current runtime addresses
* of xen text/data. Then flush all of icache.
*/
adrp x1, _start
add x1, x1, #:lo12:_start
mov x0, x1 <--- Fixed X0
adrp x2, _end
add x2, x2, #:lo12:_end
sub x1, x2, x1
bl __flush_dcache_area
ic ialluis
tlbi alle2 <---- Added this
Now, that I added the "tlbi alle2", it works after we enable MMU :) I
guess even though we set up the new page table, but it was still using
the stale info in the TLB.
On to the next hurdle:
Shell> FS0:xen -cfg=xen-seattle.cfg
Xen 4.5-unstable (c/s Mon Oct 6 10:16:52 2014 -0500 git:f0ffd29-dirty)
EFI loader
Image: 0x00000083fbbca000-0x00000083fc4ca890
- UART enabled -
- CPU 00000000 booting -
- Current EL 00000008 -
- Xen starting at EL2 -
- Zero BSS -
- Setting up control registers -
- Setup boot first -
- Setup boot second -
- Setup boot third -
- Turning on paging -
- Ready -
(XEN) Checking for initrd in /chosen
(XEN) RAM: 0000008001000000 - 0000008007ffdfff
(XEN) RAM: 0000008007ffe000 - 0000008007ffffff
(XEN) RAM: 0000008008000000 - 000000801fffdfff
(XEN) RAM: 000000801fffe000 - 000000801fffffff
(XEN) RAM: 0000008020000000 - 000000802fffffff
(XEN) RAM: 0000008030001000 - 00000083f0ffffff
(XEN) RAM: 00000083f1000000 - 00000083f101ffff
(XEN) RAM: 00000083f1020000 - 00000083fbbc7fff
(XEN) RAM: 00000083fc4ce000 - 00000083fc4cefff
(XEN) RAM: 00000083fc6b2000 - 00000083fec25fff
(XEN) RAM: 00000083fec26000 - 00000083fee8bfff
(XEN) RAM: 00000083fee8c000 - 00000083ff225fff
(XEN) RAM: 00000083ff226000 - 00000083ff263fff
(XEN) RAM: 00000083ff265000 - 00000083ff2c4fff
(XEN) RAM: 00000083ffe70000 - 00000083ffffffff
(XEN)
(XEN) MODULE[0]: 00000083fc4cb000 - 00000083fc4ce000 Device Tree
(XEN) MODULE[1]: 00000083fbbca000 - 00000083fc4ca890 Kernel
console=hvc0 console=ttyAMA0,115200 earlycon=pl011,0xe1010000
show_styx_info root=/dev/sda2 rootwait maxcpus=1
(XEN) MODULE[2]: 0000008020000000 - 00000080209e6950 Kernel
(XEN)
(XEN) Command line: FS0:xen no-bootscrub console=dtuart conswitch=x
dtuart=serial0 noreboot sync_console dom0_mem=256M dom0_max_vcpus=2
(XEN) Placing Xen at 0x000000802fe00000-0x0000008030000000
(XEN) Update BOOTMOD_XEN from 00000083fc4d0000-00000083fc5d2d81 =>
000000802fe00000-000000802ff02d81
(XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44
(XEN) CPU0: Unexpected Trap: Hypervisor
(XEN) ----[ Xen-4.5-unstable arm64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) PC: 00000000002794b4 bootmem_region_add+0x194/0x1c4
(XEN) LR: 00000000002794ac
(XEN) SP: 00000000002afd50
(XEN) CPSR: 800003c9 MODE:64-bit EL2h (Hypervisor, handler)
(XEN) X0: 0000800000000000 X1: 0000800000000000 X2: 0000000000000000
(XEN) X3: 0000800000000000 X4: ffffffffffffffff X5: 0000000000000000
(XEN) X6: 0000800000000010 X7: 000000000000000a X8: 0000000000000000
(XEN) X9: 0000000000000010 X10: 00000000002afbb8 X11: 0000000000000038
(XEN) X12: 000000000000000a X13: 000000000025d380 X14: 0000000000000030
(XEN) X15: 0000000000400000 X16: 0000000000000000 X17: 0000000000287fdc
(XEN) X18: 000000802feea000 X19: 0000000008001001 X20: 0000000000290048
(XEN) X21: 0000000008007ffe X22: 0000000000000000 X23: 0000008007ffe000
(XEN) X24: 00000000002794e4 X25: 00000000002794e4 X26: ffffffffffffffff
(XEN) X27: 0000000000298038 X28: 0000008007ffe000 FP: 00000000002afd50
(XEN)
(XEN) VTCR_EL2: 80000000
(XEN) VTTBR_EL2: 0000000000000000
(XEN)
(XEN) SCTLR_EL2: 30cd183d
(XEN) HCR_EL2: 000000000038643f
(XEN) TTBR0_EL2: 000000802feec000
(XEN)
(XEN) ESR_EL2: 96000044
(XEN) HPFAR_EL2: 0000000000000000
(XEN) FAR_EL2: 0000800000000000
(XEN)
(XEN) Xen stack trace from sp=00000000002afd50:
(XEN) 00000000002afd80 0000000000279530 0000000000290048 0000000000000000
(XEN) 0000008001000000 0000008001000000 00000000002afdd0 000000000024b154
(XEN) 0000000000000000 0000000000000000 0000008001000000 0000008001000000
(XEN) 0000008007ffe000 00000000002794e4 00000000002afe20 00000000002834ac
(XEN) 00000000002afe20 0000000000283d50 0000000000298030 0000000000000418
(XEN) 0000008001000000 0000008007ffe000 0000008007ffe000 0000000000000000
(XEN) 0000000000298038 000000000fffffff 00000083fff702b0 0000000000200690
(XEN) 00000083fc4d0000 00000083fc2d0000 00000083fc4cb000 0000000000000000
(XEN) 0000000000400000 0000000000000000 0000000000000001 00000083fc544be0
(XEN) 00000083fc5800f0 00000083fc5800e0 0000000000000000 0000000000003000
(XEN) 00000083fc4cb000 0000000006ffe000 0000000000000000 0000008001000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN) [<00000000002794b4>] bootmem_region_add+0x194/0x1c4 (PC)
(XEN) [<00000000002794ac>] bootmem_region_add+0x18c/0x1c4 (LR)
(XEN) [<0000000000279530>] init_boot_pages+0x4c/0x174
(XEN) [<000000000024b154>] dt_unreserved_regions+0xbc/0xd0
(XEN) [<0000000000283d50>] start_xen+0xc38/0xc58
(XEN) [<0000000000200690>] paging+0x88/0xc0
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) CPU0: Unexpected Trap: Hypervisor
(XEN)
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)
Any thoughts??
Thanks,
Suravee
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Failed to enable MMU when booting EFI on Seattle
2014-10-08 1:45 ` Suravee Suthikulanit
@ 2014-10-08 4:02 ` Roy Franz
0 siblings, 0 replies; 4+ messages in thread
From: Roy Franz @ 2014-10-08 4:02 UTC (permalink / raw)
To: Suravee Suthikulanit
Cc: Julien Grall, Ian Campbell, stefano.stabellini@eu.citrix.com,
xen-devel
On Tue, Oct 7, 2014 at 6:45 PM, Suravee Suthikulanit
<suravee.suthikulpanit@amd.com> wrote:
> On 10/7/2014 4:21 AM, Ian Campbell wrote:
>>
>> On Mon, 2014-10-06 at 22:29 -0500, Suravee Suthikulpanit wrote:
>>>
>>> Ian/Roy,
>>>
>>> So, I have found that in arch/arm/arm64/head.S, after we enable MMU and
>>> execute "br x1" (where x1 = 0x2005F8, the VA of "paging:"), it went to
>>> address 0x2005F8, which has all zeros value. This is why we getting the
>>> "Synchronous Exception"
>>>
>>> ......
>>> 1:
>>> PRINT("- Turning on paging -\r\n")
>>>
>>> ldr x1, =paging /* Explicit vaddr, not
>>> RIP-relative */
>>> mrs x0, SCTLR_EL2
>>> orr x0, x0, #SCTLR_M /* Enable MMU */
>>> orr x0, x0, #SCTLR_C /* Enable D-cache */
>>> dsb sy /* Flush PTE writes and finish
>>> reads */
>>> b .
>>> msr SCTLR_EL2, x0 /* now paging is enabled */
>>> isb /* Now, flush the icache */
>>> br x1 /* Get a proper vaddr into PC */
>>> paging:
>>> .....
>>>
>>> Notes:
>>> * This is not the case when booting non-EFI.
>>>
>>> * If I do "add x1, x1, x20", which puts the PA of paging into x1, it
>>> seems to branch to "paging:" fine. Although, I don't this this is the
>>> right thing to do since it should already be using VA).
>>>
>>> So, I inspected the page tables and compare the ones from booting with
>>> and without EFI. At this point, it seems that the contents of the page
>>> tables are set up properly. The only thing different is the phys-offset
>>> and the location of the page table:
>>>
>>> BOOTING WITH EFI:
>>>
>>> x20 (phys-offset) = 0x83fc2d0000
>>> x19 (start paddr) = 0x83fc4d0000
>>> (.text starts at 2MB = 0x200000 offset)
>>>
>>> boot_pgtable (pa) = x20 + 0x2f2000 = 0x83fc5c2000
>>> boot_first (pa) = x20 + 0x2f3000 = 0x83fc5c3000
>>> boot_second(pa) = x20 + 0x2f5000 = 0x83fc5c5000
>>> boot_third (pa) = x20 + 0x2f6000 = 0x83fc5c6000
>>>
>>> BOOTING WITHOUT EFI:
>>>
>>> x20 (phys-offset) = 0x8008500000
>>> x19 (start paddr) = 0x8008700000
>>> (.text starts at 2MB = 0x200000 offset)
>>>
>>> boot_pgtable (pa) = x20 + 0x2f2000 = 0x80087f2000
>>> boot_first (pa) = x20 + 0x2f3000 = 0x80087f3000
>>> boot_second(pa) = x20 + 0x2f5000 = 0x80087f5000
>>> boot_third (pa) = x20 + 0x2f6000 = 0x80087f6000
>>>
>>> Here is the memory map available from UEFI:
>>>
>>> Shell> memmap
>>> Type Start End #pages Attributes
>>> RT_Data 0000008000000000-0000008000FFFFFF 0000000000001000
>>> 800000000000000F
>>> Available 0000008001000000-0000008007FFDFFF 0000000000006FFE
>>> 000000000000000F
>>> BS_Code 0000008007FFE000-0000008007FFFFFF 0000000000000002
>>> 000000000000000F
>>> Available 0000008008000000-000000801FFFDFFF 0000000000017FFE
>>> 000000000000000F
>>> BS_Data 000000801FFFE000-000000801FFFFFFF 0000000000000002
>>> 000000000000000F
>>> Available 0000008020000000-000000802FFFFFFF 0000000000010000
>>> 000000000000000F
>>> RT_Code 0000008030000000-0000008030000FFF 0000000000000001
>>> 800000000000000F
>>> Available 0000008030001000-00000083F0FFFFFF 00000000003C0FFF
>>> 000000000000000F
>>> BS_Data 00000083F1000000-00000083F101FFFF 0000000000000020
>>> 000000000000000F
>>> Available 00000083F1020000-00000083FC5D2FFF 000000000000B5B3
>>> 000000000000000F
>>> LoaderCode 00000083FC5D3000-00000083FC6B1FFF 00000000000000DF
>>> 000000000000000F
>>>
>>> However, it seems that the location falls in the "Available", which
>>> should be ok to use.
>>
>>
>> Right, I don't see anything wrong with the location of the page tables.
>>
>> What do the contents of these PTs look like?
>>
>> In particular for Xen's load address of 0x200000 I think entries at
>> boot_pgtable[0], boot_first[0], boot_second[0] and boot_third[0x20] are
>> of interest. Do they correctly map their next level and/or the physical
>> address with the actual Xen bits in?
>>
>> For completeness it would also be worth knowing the entries
>> corresponding to the 1:1 pa mapping.
>>
>> For phys-offset 0x8008500000 those would be offsets [0x1, 0x0, 0x42,
>> 0x100] (I think, but please do check my maths!). I think this will
>> result in boot_pgtable[1] mapping boot_first_id which will contain a
>> gigabyte page super mapping in slot [0].
>>
>> Phys-offset 0x83fc2d0000 ought to be offsets [0x1, 0xf, 0x1e1, 0xd].
>> Which I would again expect to result in boot_pgtable[1] mapping
>> boot_first_id with a gigabyte mapping in slot 0xf this time.
>>
>> I expect the PA mapping to be correct, since as you say we are able to
>> run off it, it's worth checking though in case we are somehow creating
>> VA and PA-1:1 mappings which conflict (e.g. writing boot_first instead
>> of boot_first_id when creating the PA mappings -- that sort of thing).
>>
>> Ian.
>
>
> So, I have experimented with:
>
> /*
> * Preserve x0 (fdt pointer) across call to __flush_dcache_area,
> * restore for entry into Xen.
> */
> mov x20, x0
>
> /*
> * Flush dcache covering current runtime addresses
> * of xen text/data. Then flush all of icache.
> */
> adrp x1, _start
> add x1, x1, #:lo12:_start
> mov x0, x1 <--- Fixed X0
> adrp x2, _end
> add x2, x2, #:lo12:_end
> sub x1, x2, x1
>
> bl __flush_dcache_area
> ic ialluis
> tlbi alle2 <---- Added this
>
> Now, that I added the "tlbi alle2", it works after we enable MMU :) I guess
> even though we set up the new page table, but it was still using the stale
> info in the TLB.
>
> On to the next hurdle:
>
> Shell> FS0:xen -cfg=xen-seattle.cfg
> Xen 4.5-unstable (c/s Mon Oct 6 10:16:52 2014 -0500 git:f0ffd29-dirty) EFI
> loader
> Image: 0x00000083fbbca000-0x00000083fc4ca890
> - UART enabled -
> - CPU 00000000 booting -
> - Current EL 00000008 -
> - Xen starting at EL2 -
> - Zero BSS -
> - Setting up control registers -
> - Setup boot first -
> - Setup boot second -
> - Setup boot third -
> - Turning on paging -
> - Ready -
> (XEN) Checking for initrd in /chosen
> (XEN) RAM: 0000008001000000 - 0000008007ffdfff
> (XEN) RAM: 0000008007ffe000 - 0000008007ffffff
> (XEN) RAM: 0000008008000000 - 000000801fffdfff
> (XEN) RAM: 000000801fffe000 - 000000801fffffff
> (XEN) RAM: 0000008020000000 - 000000802fffffff
> (XEN) RAM: 0000008030001000 - 00000083f0ffffff
> (XEN) RAM: 00000083f1000000 - 00000083f101ffff
> (XEN) RAM: 00000083f1020000 - 00000083fbbc7fff
> (XEN) RAM: 00000083fc4ce000 - 00000083fc4cefff
> (XEN) RAM: 00000083fc6b2000 - 00000083fec25fff
> (XEN) RAM: 00000083fec26000 - 00000083fee8bfff
> (XEN) RAM: 00000083fee8c000 - 00000083ff225fff
> (XEN) RAM: 00000083ff226000 - 00000083ff263fff
> (XEN) RAM: 00000083ff265000 - 00000083ff2c4fff
> (XEN) RAM: 00000083ffe70000 - 00000083ffffffff
> (XEN)
> (XEN) MODULE[0]: 00000083fc4cb000 - 00000083fc4ce000 Device Tree
> (XEN) MODULE[1]: 00000083fbbca000 - 00000083fc4ca890 Kernel console=hvc0
> console=ttyAMA0,115200 earlycon=pl011,0xe1010000 show_styx_info
> root=/dev/sda2 rootwait maxcpus=1
> (XEN) MODULE[2]: 0000008020000000 - 00000080209e6950 Kernel
> (XEN)
> (XEN) Command line: FS0:xen no-bootscrub console=dtuart conswitch=x
> dtuart=serial0 noreboot sync_console dom0_mem=256M dom0_max_vcpus=2
> (XEN) Placing Xen at 0x000000802fe00000-0x0000008030000000
> (XEN) Update BOOTMOD_XEN from 00000083fc4d0000-00000083fc5d2d81 =>
> 000000802fe00000-000000802ff02d81
> (XEN) Hypervisor Trap. HSR=0x96000044 EC=0x25 IL=1 Syndrome=0x44
> (XEN) CPU0: Unexpected Trap: Hypervisor
> (XEN) ----[ Xen-4.5-unstable arm64 debug=y Not tainted ]----
> (XEN) CPU: 0
> (XEN) PC: 00000000002794b4 bootmem_region_add+0x194/0x1c4
> (XEN) LR: 00000000002794ac
> (XEN) SP: 00000000002afd50
> (XEN) CPSR: 800003c9 MODE:64-bit EL2h (Hypervisor, handler)
> (XEN) X0: 0000800000000000 X1: 0000800000000000 X2: 0000000000000000
> (XEN) X3: 0000800000000000 X4: ffffffffffffffff X5: 0000000000000000
> (XEN) X6: 0000800000000010 X7: 000000000000000a X8: 0000000000000000
> (XEN) X9: 0000000000000010 X10: 00000000002afbb8 X11: 0000000000000038
> (XEN) X12: 000000000000000a X13: 000000000025d380 X14: 0000000000000030
> (XEN) X15: 0000000000400000 X16: 0000000000000000 X17: 0000000000287fdc
> (XEN) X18: 000000802feea000 X19: 0000000008001001 X20: 0000000000290048
> (XEN) X21: 0000000008007ffe X22: 0000000000000000 X23: 0000008007ffe000
> (XEN) X24: 00000000002794e4 X25: 00000000002794e4 X26: ffffffffffffffff
> (XEN) X27: 0000000000298038 X28: 0000008007ffe000 FP: 00000000002afd50
> (XEN)
> (XEN) VTCR_EL2: 80000000
> (XEN) VTTBR_EL2: 0000000000000000
> (XEN)
> (XEN) SCTLR_EL2: 30cd183d
> (XEN) HCR_EL2: 000000000038643f
> (XEN) TTBR0_EL2: 000000802feec000
> (XEN)
> (XEN) ESR_EL2: 96000044
> (XEN) HPFAR_EL2: 0000000000000000
> (XEN) FAR_EL2: 0000800000000000
> (XEN)
> (XEN) Xen stack trace from sp=00000000002afd50:
> (XEN) 00000000002afd80 0000000000279530 0000000000290048 0000000000000000
> (XEN) 0000008001000000 0000008001000000 00000000002afdd0 000000000024b154
> (XEN) 0000000000000000 0000000000000000 0000008001000000 0000008001000000
> (XEN) 0000008007ffe000 00000000002794e4 00000000002afe20 00000000002834ac
> (XEN) 00000000002afe20 0000000000283d50 0000000000298030 0000000000000418
> (XEN) 0000008001000000 0000008007ffe000 0000008007ffe000 0000000000000000
> (XEN) 0000000000298038 000000000fffffff 00000083fff702b0 0000000000200690
> (XEN) 00000083fc4d0000 00000083fc2d0000 00000083fc4cb000 0000000000000000
> (XEN) 0000000000400000 0000000000000000 0000000000000001 00000083fc544be0
> (XEN) 00000083fc5800f0 00000083fc5800e0 0000000000000000 0000000000003000
> (XEN) 00000083fc4cb000 0000000006ffe000 0000000000000000 0000008001000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN) 0000000000000000 0000000000000000
> (XEN) Xen call trace:
> (XEN) [<00000000002794b4>] bootmem_region_add+0x194/0x1c4 (PC)
> (XEN) [<00000000002794ac>] bootmem_region_add+0x18c/0x1c4 (LR)
> (XEN) [<0000000000279530>] init_boot_pages+0x4c/0x174
> (XEN) [<000000000024b154>] dt_unreserved_regions+0xbc/0xd0
> (XEN) [<0000000000283d50>] start_xen+0xc38/0xc58
> (XEN) [<0000000000200690>] paging+0x88/0xc0
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) CPU0: Unexpected Trap: Hypervisor
> (XEN)
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
>
> Any thoughts??
>
> Thanks,
>
> Suravee
>
>
>
>
After running a bit with your patch (minus the TLB flushing), I found
I was seeing some crashes with a corrupt FDT. From looking at the
code,
I didn't think we would need to flush the FDT, as I don't think it is
accessed before mmu/caching is turned back on. This was an
intermittent crash
on the FVP model with cache state modelling enabled.
The following hacky patch seems (in limited testing) to fix this:
diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S
index 2b8998b..1c2346b 100644
--- a/xen/arch/arm/arm64/head.S
+++ b/xen/arch/arm/arm64/head.S
@@ -744,6 +744,8 @@ ENTRY(efi_xen_start)
* restore for entry into Xen.
*/
mov x20, x0
+ mov x1, 0x100000
+ bl __flush_dcache_area
/*
* Flush dcache covering current runtime addresses
1 MByte should cover the FDT - if we really do need to flush it I will
do a proper flush of the correct size.
The code where you are crashing it is examining the FDT to see if
there are any mem-reserve regions, so if the FDT
is corrupt it could be causing your crashes as well. (There won't be
any mem-reserve regions, as the EFI boot code
removes them along with the memory nodes from the FDT.)
I'll be interested to hear if flushing the FDT fixes your crashes, and
I need to further investigate why this flushing seems
to be required.
Roy
^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-10-08 4:02 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-10-07 3:29 Failed to enable MMU when booting EFI on Seattle Suravee Suthikulpanit
2014-10-07 9:21 ` Ian Campbell
2014-10-08 1:45 ` Suravee Suthikulanit
2014-10-08 4:02 ` Roy Franz
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.