* Bug report: kernel paniced while booting
@ 2023-06-05 10:52 Song Shuai
2023-06-05 10:58 ` Conor Dooley
2023-06-05 14:25 ` Alexandre Ghiti
0 siblings, 2 replies; 16+ messages in thread
From: Song Shuai @ 2023-06-05 10:52 UTC (permalink / raw)
To: alexghiti, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan,
mason.huo, paul.walmsley, conor.dooley, guoren
Cc: linux-riscv, linux-kernel
Description of problem:
Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic.
The entire log has been posted at this link : https://termbin.com/nga4.
You can reproduce it with the following step :
1. prepare the environment with
- Qemu-virt: v8.0.0 (with OpenSbi v1.2)
- edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests")
- Linux : v6.4-rc1 and later version
2. start the Qemu virt board
```sh
$ cat ~/8_riscv/start_latest.sh
#!/bin/bash
/home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \
-s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \
-machine virt,acpi=off -smp 2 -m 2G \
-kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \
-initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \
-append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest
```
3. Then you will encounter the kernel panic logged in the above link
Other Information:
1. -------
This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them
are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping").
With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000)
from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]).
In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000,
so there is no "access fault" but "page fault".
And reverting commit 3335068f8721 indeed fixed this panic.
2. -------
As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store.
Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong?
```
(gdb) p/x $satp
$1 = 0xa000000000081708
(gdb) pt -satp 0xa000000000081708
Address : Length Permissions
0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1
0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1
0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1
0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1
0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1
0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000
0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1
0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1
0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1
0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1
```
3. ------
You can also reproduce similar panic by appending "memtest" in kernel cmdline.
I have posted the memtest boot log at this link: https://termbin.com/1twl.
Please correct me if I'm wrong.
[1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/
[2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/
[3]: https://github.com/martinradev/gdb-pt-dump
_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: Bug report: kernel paniced while booting 2023-06-05 10:52 Bug report: kernel paniced while booting Song Shuai @ 2023-06-05 10:58 ` Conor Dooley 2023-06-05 14:25 ` Alexandre Ghiti 1 sibling, 0 replies; 16+ messages in thread From: Conor Dooley @ 2023-06-05 10:58 UTC (permalink / raw) To: Song Shuai Cc: alexghiti, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel [-- Attachment #1.1: Type: text/plain, Size: 1040 bytes --] On Mon, Jun 05, 2023 at 10:52:14AM +0000, Song Shuai wrote: > This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > > With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). Just to be clear, you enabled NONPORTABLE & hibernation, and then hibernating panicked for you on a system that uses standard OpenSBI? This is the existing, known, issue and not something you are trying to report as a new problem here, right? (Sorry, I am a bit slow today) Cheers, Conor. > In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > so there is no "access fault" but "page fault". > > And reverting commit 3335068f8721 indeed fixed this panic. [-- Attachment #1.2: signature.asc --] [-- Type: application/pgp-signature, Size: 228 bytes --] [-- Attachment #2: Type: text/plain, Size: 161 bytes --] _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-05 10:52 Bug report: kernel paniced while booting Song Shuai 2023-06-05 10:58 ` Conor Dooley @ 2023-06-05 14:25 ` Alexandre Ghiti 2023-06-05 15:12 ` Sunil V L 2023-06-06 10:04 ` Song Shuai 1 sibling, 2 replies; 16+ messages in thread From: Alexandre Ghiti @ 2023-06-05 14:25 UTC (permalink / raw) To: Song Shuai Cc: robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel Hi Song, On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > > Description of problem: > > Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. > The entire log has been posted at this link : https://termbin.com/nga4. > > You can reproduce it with the following step : > > 1. prepare the environment with > - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") > - Linux : v6.4-rc1 and later version > > 2. start the Qemu virt board > > ```sh > $ cat ~/8_riscv/start_latest.sh > #!/bin/bash > /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > -machine virt,acpi=off -smp 2 -m 2G \ > -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest > ``` > 3. Then you will encounter the kernel panic logged in the above link > > Other Information: > > 1. ------- > > This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > > With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). > > In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > so there is no "access fault" but "page fault". > > And reverting commit 3335068f8721 indeed fixed this panic. > > 2. ------- > > As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. > Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? > > ``` > (gdb) p/x $satp > $1 = 0xa000000000081708 > (gdb) pt -satp 0xa000000000081708 > Address : Length Permissions > 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 > 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > > ``` > > 3. ------ > > You can also reproduce similar panic by appending "memtest" in kernel cmdline. > I have posted the memtest boot log at this link: https://termbin.com/1twl. > > Please correct me if I'm wrong. > > [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > [3]: https://github.com/martinradev/gdb-pt-dump Thanks for the thorough report, really appreciated. So there are multiple issues here: - the first one is that the memory region for opensbi is marked as not cacheable in the efi memory map, and then this region is not mapped in the linear mapping: [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | | | | | | | | | | | | |UC] - the second one (that I feel a bit ashamed of...) is that I did not check the alignment of the virtual address when choosing the map size in best_map_size() and then we end up trying to map a physical region aligned on 2MB that is actually not aligned on 2MB virtually because the opensbi region is not mapped at all. - the possible third one is that we should not map the linear mapping using 4K pages, this would be slow in my opinion, and I think we should waste a bit of memory to align va and pa on a 2MB boundary. So I'll fix the second issue, and possibly the third one, and if no one looks into why the opensbi region is mapped in UC, I'll take a look at edk2. Sorry for that, Alex _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-05 14:25 ` Alexandre Ghiti @ 2023-06-05 15:12 ` Sunil V L 2023-06-05 20:55 ` Atish Patra 2023-06-05 21:42 ` Jessica Clarke 2023-06-06 10:04 ` Song Shuai 1 sibling, 2 replies; 16+ messages in thread From: Sunil V L @ 2023-06-05 15:12 UTC (permalink / raw) To: Alexandre Ghiti Cc: Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: > Hi Song, > > On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > > > > Description of problem: > > > > Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. > > The entire log has been posted at this link : https://termbin.com/nga4. > > > > You can reproduce it with the following step : > > > > 1. prepare the environment with > > - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > > - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") > > - Linux : v6.4-rc1 and later version > > > > 2. start the Qemu virt board > > > > ```sh > > $ cat ~/8_riscv/start_latest.sh > > #!/bin/bash > > /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > > -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > > -machine virt,acpi=off -smp 2 -m 2G \ > > -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > > -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > > -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest > > ``` > > 3. Then you will encounter the kernel panic logged in the above link > > > > Other Information: > > > > 1. ------- > > > > This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > > are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > > > > With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > > from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). > > > > In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > > so there is no "access fault" but "page fault". > > > > And reverting commit 3335068f8721 indeed fixed this panic. > > > > 2. ------- > > > > As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. > > Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? > > > > ``` > > (gdb) p/x $satp > > $1 = 0xa000000000081708 > > (gdb) pt -satp 0xa000000000081708 > > Address : Length Permissions > > 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > > 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > > 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > > 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > > 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > > 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 > > 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > > 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > > 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > > 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > > > > ``` > > > > 3. ------ > > > > You can also reproduce similar panic by appending "memtest" in kernel cmdline. > > I have posted the memtest boot log at this link: https://termbin.com/1twl. > > > > Please correct me if I'm wrong. > > > > [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > > [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > > [3]: https://github.com/martinradev/gdb-pt-dump > > Thanks for the thorough report, really appreciated. > > So there are multiple issues here: > > - the first one is that the memory region for opensbi is marked as not > cacheable in the efi memory map, and then this region is not mapped in > the linear mapping: > [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | > | | | | | | | | | | | |UC] > > - the second one (that I feel a bit ashamed of...) is that I did not > check the alignment of the virtual address when choosing the map size > in best_map_size() and then we end up trying to map a physical region > aligned on 2MB that is actually not aligned on 2MB virtually because > the opensbi region is not mapped at all. > > - the possible third one is that we should not map the linear mapping > using 4K pages, this would be slow in my opinion, and I think we > should waste a bit of memory to align va and pa on a 2MB boundary. > > So I'll fix the second issue, and possibly the third one, and if no > one looks into why the opensbi region is mapped in UC, I'll take a > look at edk2. > Hi Alex, EDK2 marks opensbi range as reserved memory in EFI map. According to DT spec, if the no-map is not set, we need to mark it as EfiBootServicesData but EfiBootServicesData is actually considered as free memory in kernel, as per UEFI spec. To avoid kernel using this memory, we deviated from the DT spec for opensbi ranges. Let me know your thoughts how we can handle this better in EDK2 considering it has to support ACPI also. Thanks, Sunil _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-05 15:12 ` Sunil V L @ 2023-06-05 20:55 ` Atish Patra 2023-06-06 0:48 ` Icenowy Zheng 2023-06-06 7:20 ` Alexandre Ghiti 2023-06-05 21:42 ` Jessica Clarke 1 sibling, 2 replies; 16+ messages in thread From: Atish Patra @ 2023-06-05 20:55 UTC (permalink / raw) To: Sunil V L Cc: Alexandre Ghiti, Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On Mon, Jun 5, 2023 at 8:13 AM Sunil V L <sunilvl@ventanamicro.com> wrote: > > On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: > > Hi Song, > > > > On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > > > > > > Description of problem: > > > > > > Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. > > > The entire log has been posted at this link : https://termbin.com/nga4. > > > > > > You can reproduce it with the following step : > > > > > > 1. prepare the environment with > > > - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > > > - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") > > > - Linux : v6.4-rc1 and later version > > > > > > 2. start the Qemu virt board > > > > > > ```sh > > > $ cat ~/8_riscv/start_latest.sh > > > #!/bin/bash > > > /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > > > -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > > > -machine virt,acpi=off -smp 2 -m 2G \ > > > -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > > > -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > > > -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest > > > ``` > > > 3. Then you will encounter the kernel panic logged in the above link > > > > > > Other Information: > > > > > > 1. ------- > > > > > > This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > > > are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > > > > > > With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > > > from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). > > > > > > In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > > > so there is no "access fault" but "page fault". > > > > > > And reverting commit 3335068f8721 indeed fixed this panic. > > > > > > 2. ------- > > > > > > As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. > > > Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? > > > > > > ``` > > > (gdb) p/x $satp > > > $1 = 0xa000000000081708 > > > (gdb) pt -satp 0xa000000000081708 > > > Address : Length Permissions > > > 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > > > 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > > > 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > > > 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > > > 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > > > 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 > > > 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > > > 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > > > 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > > > 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > > > > > > ``` > > > > > > 3. ------ > > > > > > You can also reproduce similar panic by appending "memtest" in kernel cmdline. > > > I have posted the memtest boot log at this link: https://termbin.com/1twl. > > > > > > Please correct me if I'm wrong. > > > > > > [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > > > [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > > > [3]: https://github.com/martinradev/gdb-pt-dump > > > > Thanks for the thorough report, really appreciated. > > > > So there are multiple issues here: > > > > - the first one is that the memory region for opensbi is marked as not > > cacheable in the efi memory map, and then this region is not mapped in > > the linear mapping: > > [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | > > | | | | | | | | | | | |UC] > > @Alex: The OpenSBI region is marked reserved because EDK2 chooses to do that explicitly as explained by Sunil. I don't think UC has to do anything with it. All the EFI memory regions are UC. > > - the second one (that I feel a bit ashamed of...) is that I did not > > check the alignment of the virtual address when choosing the map size > > in best_map_size() and then we end up trying to map a physical region > > aligned on 2MB that is actually not aligned on 2MB virtually because > > the opensbi region is not mapped at all. > > > > - the possible third one is that we should not map the linear mapping > > using 4K pages, this would be slow in my opinion, and I think we > > should waste a bit of memory to align va and pa on a 2MB boundary. > > > > So I'll fix the second issue, and possibly the third one, and if no > > one looks into why the opensbi region is mapped in UC, I'll take a > > look at edk2. > > > Hi Alex, > > EDK2 marks opensbi range as reserved memory in EFI map. According to DT > spec, if the no-map is not set, we need to mark it as > EfiBootServicesData but EfiBootServicesData is actually considered as > free memory in kernel, as per UEFI spec. To avoid kernel using this > memory, we deviated from the DT spec for opensbi ranges. > IMO, that should be the correct way unless we can change it to EfiRunServicesData/Code. Looking at U-Boot code, it sets the no-map region to EfiBootServicesData which may cause issues in RISC-V as well if we linear mapping sets up the initial 2MB. > Let me know your thoughts how we can handle this better in EDK2 > considering it has to support ACPI also. > > Thanks, > Sunil > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv -- Regards, Atish _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-05 20:55 ` Atish Patra @ 2023-06-06 0:48 ` Icenowy Zheng 2023-06-06 2:29 ` Jessica Clarke 2023-06-06 7:20 ` Alexandre Ghiti 1 sibling, 1 reply; 16+ messages in thread From: Icenowy Zheng @ 2023-06-06 0:48 UTC (permalink / raw) To: Atish Patra, Sunil V L Cc: Alexandre Ghiti, Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel 在 2023-06-05星期一的 13:55 -0700，Atish Patra写道： > On Mon, Jun 5, 2023 at 8:13 AM Sunil V L <sunilvl@ventanamicro.com> > wrote: > > > > On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: > > > Hi Song, > > > > > > On Mon, Jun 5, 2023 at 12:52 PM Song Shuai > > > <songshuaishuai@tinylab.org> wrote: > > > > > > > > Description of problem: > > > > > > > > Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO > > > > page fault was trapped to trigger a kernel panic. > > > > The entire log has been posted at this link : > > > > https://termbin.com/nga4. > > > > > > > > You can reproduce it with the following step : > > > > > > > > 1. prepare the environment with > > > > - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > > > > - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: > > > > Reduce the number of random tests") > > > > - Linux : v6.4-rc1 and later version > > > > > > > > 2. start the Qemu virt board > > > > > > > > ```sh > > > > $ cat ~/8_riscv/start_latest.sh > > > > #!/bin/bash > > > > /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system- > > > > riscv64 \ > > > > -s -nographic -drive > > > > file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE > > > > _GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > > > > -machine virt,acpi=off -smp 2 -m 2G \ > > > > -kernel > > > > /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > > > > -initrd > > > > /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > > > > -append "root=/dev/ram ro console=ttyS0 > > > > earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 > > > > memblock=debug" ## also panic by memtest > > > > ``` > > > > 3. Then you will encounter the kernel panic logged in the above > > > > link > > > > > > > > Other Information: > > > > > > > > 1. ------- > > > > > > > > This report is not identical to my prior report -- "kernel > > > > paniced when system hibernates" [1], but both of them > > > > are closely related with the commit (3335068f8721 "riscv: Use > > > > PUD/P4D/PGD pages for the linear mapping"). > > > > > > > > With this commit, hibernation is trapped with "access fault" > > > > while accessing the PMP-protected regions > > > > (mmode_resv0@80000000) > > > > from OpenSbi (BTW, hibernation is marked as nonportable by > > > > Conor[2]). > > > > > > > > In this report, efi_init handoffs the memory mapping from Boot > > > > Services to memblock where reserves mmode_resv0@80000000, > > > > so there is no "access fault" but "page fault". > > > > > > > > And reverting commit 3335068f8721 indeed fixed this panic. > > > > > > > > 2. ------- > > > > > > > > As the gdb-pt-dump [3] tool shows, the PTE which covered the > > > > fault virtual address had the appropriate permission to store. > > > > Is there another way to trigger the "Store/AMO page fault"? Or > > > > the creation of linear mapping in commit 3335068f8721 did > > > > something wrong? > > > > > > > > ``` > > > > (gdb) p/x $satp > > > > $1 = 0xa000000000081708 > > > > (gdb) pt -satp 0xa000000000081708 > > > > Address : Length Permissions > > > > 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > > > > 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > > > > 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > > > > 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // > > > > badaddr: ff6000007fdb1000 > > > > 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > > > > 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > > > > 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > > > > > > > > ``` > > > > > > > > 3. ------ > > > > > > > > You can also reproduce similar panic by appending "memtest" in > > > > kernel cmdline. > > > > I have posted the memtest boot log at this link: > > > > https://termbin.com/1twl. > > > > > > > > Please correct me if I'm wrong. > > > > > > > > [1]: > > > > https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > > > > [2]: > > > > https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > > > > [3]: https://github.com/martinradev/gdb-pt-dump > > > > > > Thanks for the thorough report, really appreciated. > > > > > > So there are multiple issues here: > > > > > > - the first one is that the memory region for opensbi is marked > > > as not > > > cacheable in the efi memory map, and then this region is not > > > mapped in > > > the linear mapping: > > > [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved > > > | | > > > | | | | | | | | | | | |UC] > > > > > @Alex: The OpenSBI region is marked reserved because EDK2 chooses to > do that explicitly as explained by Sunil. > I don't think UC has to do anything with it. All the EFI memory > regions are UC. > > > > - the second one (that I feel a bit ashamed of...) is that I did > > > not > > > check the alignment of the virtual address when choosing the map > > > size > > > in best_map_size() and then we end up trying to map a physical > > > region > > > aligned on 2MB that is actually not aligned on 2MB virtually > > > because > > > the opensbi region is not mapped at all. > > > > > > - the possible third one is that we should not map the linear > > > mapping > > > using 4K pages, this would be slow in my opinion, and I think we > > > should waste a bit of memory to align va and pa on a 2MB > > > boundary. > > > > > > So I'll fix the second issue, and possibly the third one, and if > > > no > > > one looks into why the opensbi region is mapped in UC, I'll take > > > a > > > look at edk2. > > > > > Hi Alex, > > > > EDK2 marks opensbi range as reserved memory in EFI map. According > > to DT > > spec, if the no-map is not set, we need to mark it as > > EfiBootServicesData but EfiBootServicesData is actually considered > > as > > free memory in kernel, as per UEFI spec. To avoid kernel using this > > memory, we deviated from the DT spec for opensbi ranges. > > > > IMO, that should be the correct way unless we can change it to > EfiRunServicesData/Code. > Looking at U-Boot code, it sets the no-map region to > EfiBootServicesData which may cause > issues in RISC-V as well if we linear mapping sets up the initial > 2MB. Semantically I think no-map means the kernel should not be utilizing it, so it should be EfiRunServicesData instead. > > > > Let me know your thoughts how we can handle this better in EDK2 > > considering it has to support ACPI also. > > > > Thanks, > > Sunil > > > > _______________________________________________ > > linux-riscv mailing list > > linux-riscv@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-riscv > > > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-06 0:48 ` Icenowy Zheng @ 2023-06-06 2:29 ` Jessica Clarke 0 siblings, 0 replies; 16+ messages in thread From: Jessica Clarke @ 2023-06-06 2:29 UTC (permalink / raw) To: Icenowy Zheng Cc: robh, mason.huo, Alexandre Ghiti, jeeheng.sia, anup, paul.walmsley, conor.dooley, palmer, linux-kernel, leyfoon.tan, Song Shuai, guoren, Atish Patra, linux-riscv, ajones On 6 Jun 2023, at 01:48, Icenowy Zheng <uwu@icenowy.me> wrote: > > 在 2023-06-05星期一的 13:55 -0700，Atish Patra写道： >> On Mon, Jun 5, 2023 at 8:13 AM Sunil V L <sunilvl@ventanamicro.com> >> wrote: >>> >>> On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: >>>> Hi Song, >>>> >>>> On Mon, Jun 5, 2023 at 12:52 PM Song Shuai >>>> <songshuaishuai@tinylab.org> wrote: >>>>> >>>>> Description of problem: >>>>> >>>>> Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO >>>>> page fault was trapped to trigger a kernel panic. >>>>> The entire log has been posted at this link : >>>>> https://termbin.com/nga4. >>>>> >>>>> You can reproduce it with the following step : >>>>> >>>>> 1. prepare the environment with >>>>> - Qemu-virt: v8.0.0 (with OpenSbi v1.2) >>>>> - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: >>>>> Reduce the number of random tests") >>>>> - Linux : v6.4-rc1 and later version >>>>> >>>>> 2. start the Qemu virt board >>>>> >>>>> ```sh >>>>> $ cat ~/8_riscv/start_latest.sh >>>>> #!/bin/bash >>>>> /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system- >>>>> riscv64 \ >>>>> -s -nographic -drive >>>>> file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE >>>>> _GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ >>>>> -machine virt,acpi=off -smp 2 -m 2G \ >>>>> -kernel >>>>> /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ >>>>> -initrd >>>>> /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ >>>>> -append "root=/dev/ram ro console=ttyS0 >>>>> earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 >>>>> memblock=debug" ## also panic by memtest >>>>> ``` >>>>> 3. Then you will encounter the kernel panic logged in the above >>>>> link >>>>> >>>>> Other Information: >>>>> >>>>> 1. ------- >>>>> >>>>> This report is not identical to my prior report -- "kernel >>>>> paniced when system hibernates" [1], but both of them >>>>> are closely related with the commit (3335068f8721 "riscv: Use >>>>> PUD/P4D/PGD pages for the linear mapping"). >>>>> >>>>> With this commit, hibernation is trapped with "access fault" >>>>> while accessing the PMP-protected regions >>>>> (mmode_resv0@80000000) >>>>> from OpenSbi (BTW, hibernation is marked as nonportable by >>>>> Conor[2]). >>>>> >>>>> In this report, efi_init handoffs the memory mapping from Boot >>>>> Services to memblock where reserves mmode_resv0@80000000, >>>>> so there is no "access fault" but "page fault". >>>>> >>>>> And reverting commit 3335068f8721 indeed fixed this panic. >>>>> >>>>> 2. ------- >>>>> >>>>> As the gdb-pt-dump [3] tool shows, the PTE which covered the >>>>> fault virtual address had the appropriate permission to store. >>>>> Is there another way to trigger the "Store/AMO page fault"? Or >>>>> the creation of linear mapping in commit 3335068f8721 did >>>>> something wrong? >>>>> >>>>> ``` >>>>> (gdb) p/x $satp >>>>> $1 = 0xa000000000081708 >>>>> (gdb) pt -satp 0xa000000000081708 >>>>> Address : Length Permissions >>>>> 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 >>>>> 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 >>>>> 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 >>>>> 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 >>>>> 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 >>>>> 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // >>>>> badaddr: ff6000007fdb1000 >>>>> 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 >>>>> 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 >>>>> 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 >>>>> 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 >>>>> >>>>> ``` >>>>> >>>>> 3. ------ >>>>> >>>>> You can also reproduce similar panic by appending "memtest" in >>>>> kernel cmdline. >>>>> I have posted the memtest boot log at this link: >>>>> https://termbin.com/1twl. >>>>> >>>>> Please correct me if I'm wrong. >>>>> >>>>> [1]: >>>>> https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ >>>>> [2]: >>>>> https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ >>>>> [3]: https://github.com/martinradev/gdb-pt-dump >>>> >>>> Thanks for the thorough report, really appreciated. >>>> >>>> So there are multiple issues here: >>>> >>>> - the first one is that the memory region for opensbi is marked >>>> as not >>>> cacheable in the efi memory map, and then this region is not >>>> mapped in >>>> the linear mapping: >>>> [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved >>>> | | >>>> | | | | | | | | | | | |UC] >>>> >> >> @Alex: The OpenSBI region is marked reserved because EDK2 chooses to >> do that explicitly as explained by Sunil. >> I don't think UC has to do anything with it. All the EFI memory >> regions are UC. >> >>>> - the second one (that I feel a bit ashamed of...) is that I did >>>> not >>>> check the alignment of the virtual address when choosing the map >>>> size >>>> in best_map_size() and then we end up trying to map a physical >>>> region >>>> aligned on 2MB that is actually not aligned on 2MB virtually >>>> because >>>> the opensbi region is not mapped at all. >>>> >>>> - the possible third one is that we should not map the linear >>>> mapping >>>> using 4K pages, this would be slow in my opinion, and I think we >>>> should waste a bit of memory to align va and pa on a 2MB >>>> boundary. >>>> >>>> So I'll fix the second issue, and possibly the third one, and if >>>> no >>>> one looks into why the opensbi region is mapped in UC, I'll take >>>> a >>>> look at edk2. >>>> >>> Hi Alex, >>> >>> EDK2 marks opensbi range as reserved memory in EFI map. According >>> to DT >>> spec, if the no-map is not set, we need to mark it as >>> EfiBootServicesData but EfiBootServicesData is actually considered >>> as >>> free memory in kernel, as per UEFI spec. To avoid kernel using this >>> memory, we deviated from the DT spec for opensbi ranges. >>> >> >> IMO, that should be the correct way unless we can change it to >> EfiRunServicesData/Code. >> Looking at U-Boot code, it sets the no-map region to >> EfiBootServicesData which may cause >> issues in RISC-V as well if we linear mapping sets up the initial >> 2MB. > > Semantically I think no-map means the kernel should not be utilizing > it, so it should be EfiRunServicesData instead. no-map *is* EfiRuntimeServicesData in the DT spec; the problem is that OpenSBI isn’t marking it no-map, but has nothing in its place to mark it as “you can map it but you really cannot touch it”; normally reserved memory is “don’t treat it as normal memory but you can use it for specific purposes (e.g. DMA memory pool)”, so it gets treated as the latter (what that state has always meant) despite definitely not being that. Or, to put it another way, OpenSBI went from something that was correct-but-slow to something that is incorrect-but-fast and the consumers of the FDT are now paying the price for its brokenness. So we need some other approach that isn’t incorrect by definition. Jess >> >> >>> Let me know your thoughts how we can handle this better in EDK2 >>> considering it has to support ACPI also. >>> >>> Thanks, >>> Sunil >>> >>> _______________________________________________ >>> linux-riscv mailing list >>> linux-riscv@lists.infradead.org >>> http://lists.infradead.org/mailman/listinfo/linux-riscv >> >> >> > > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-05 20:55 ` Atish Patra 2023-06-06 0:48 ` Icenowy Zheng @ 2023-06-06 7:20 ` Alexandre Ghiti 2023-06-06 9:01 ` Sunil V L 1 sibling, 1 reply; 16+ messages in thread From: Alexandre Ghiti @ 2023-06-06 7:20 UTC (permalink / raw) To: Atish Patra, Sunil V L Cc: Alexandre Ghiti, Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On 05/06/2023 22:55, Atish Patra wrote: > On Mon, Jun 5, 2023 at 8:13 AM Sunil V L <sunilvl@ventanamicro.com> wrote: >> On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: >>> Hi Song, >>> >>> On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: >>>> Description of problem: >>>> >>>> Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. >>>> The entire log has been posted at this link : https://termbin.com/nga4. >>>> >>>> You can reproduce it with the following step : >>>> >>>> 1. prepare the environment with >>>> - Qemu-virt: v8.0.0 (with OpenSbi v1.2) >>>> - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") >>>> - Linux : v6.4-rc1 and later version >>>> >>>> 2. start the Qemu virt board >>>> >>>> ```sh >>>> $ cat ~/8_riscv/start_latest.sh >>>> #!/bin/bash >>>> /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ >>>> -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ >>>> -machine virt,acpi=off -smp 2 -m 2G \ >>>> -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ >>>> -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ >>>> -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest >>>> ``` >>>> 3. Then you will encounter the kernel panic logged in the above link >>>> >>>> Other Information: >>>> >>>> 1. ------- >>>> >>>> This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them >>>> are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). >>>> >>>> With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) >>>> from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). >>>> >>>> In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, >>>> so there is no "access fault" but "page fault". >>>> >>>> And reverting commit 3335068f8721 indeed fixed this panic. >>>> >>>> 2. ------- >>>> >>>> As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. >>>> Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? >>>> >>>> ``` >>>> (gdb) p/x $satp >>>> $1 = 0xa000000000081708 >>>> (gdb) pt -satp 0xa000000000081708 >>>> Address : Length Permissions >>>> 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 >>>> 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 >>>> 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 >>>> 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 >>>> 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 >>>> 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 >>>> 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 >>>> 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 >>>> 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 >>>> 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 >>>> >>>> ``` >>>> >>>> 3. ------ >>>> >>>> You can also reproduce similar panic by appending "memtest" in kernel cmdline. >>>> I have posted the memtest boot log at this link: https://termbin.com/1twl. >>>> >>>> Please correct me if I'm wrong. >>>> >>>> [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ >>>> [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ >>>> [3]: https://github.com/martinradev/gdb-pt-dump >>> Thanks for the thorough report, really appreciated. >>> >>> So there are multiple issues here: >>> >>> - the first one is that the memory region for opensbi is marked as not >>> cacheable in the efi memory map, and then this region is not mapped in >>> the linear mapping: >>> [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | >>> | | | | | | | | | | | |UC] >>> > @Alex: The OpenSBI region is marked reserved because EDK2 chooses to > do that explicitly as explained by Sunil. > I don't think UC has to do anything with it. All the EFI memory regions are UC. Oops, thanks for correcting me, UC has nothing to do with that indeed, I misread is_usable_memory() (https://elixir.bootlin.com/linux/latest/source/drivers/firmware/efi/efi-init.c#L127): EFI_RESERVED_TYPE will return false. And then this region won't get mapped, so that's the equivalent of "no-map". > >>> - the second one (that I feel a bit ashamed of...) is that I did not >>> check the alignment of the virtual address when choosing the map size >>> in best_map_size() and then we end up trying to map a physical region >>> aligned on 2MB that is actually not aligned on 2MB virtually because >>> the opensbi region is not mapped at all. >>> >>> - the possible third one is that we should not map the linear mapping >>> using 4K pages, this would be slow in my opinion, and I think we >>> should waste a bit of memory to align va and pa on a 2MB boundary. >>> >>> So I'll fix the second issue, and possibly the third one, and if no >>> one looks into why the opensbi region is mapped in UC, I'll take a >>> look at edk2. >>> >> Hi Alex, >> >> EDK2 marks opensbi range as reserved memory in EFI map. According to DT >> spec, if the no-map is not set, we need to mark it as >> EfiBootServicesData but EfiBootServicesData is actually considered as >> free memory in kernel, as per UEFI spec. To avoid kernel using this >> memory, we deviated from the DT spec for opensbi ranges. >> > IMO, that should be the correct way unless we can change it to > EfiRunServicesData/Code. I'd say that everything depends on "reusable" no? If "no-map" then EfiReserved else if reusable then EfiBootServicesData else EfiRuntimeServicesData endif endif For now, I think only the following is done (based on "3.5.4 /reserved-memory and UEFI" of the DT spec https://github.com/devicetree-org/devicetree-specification/releases/download/v0.4-rc1/devicetree-specification-v0.4-rc1.pdf): If "no-map" then EfiReserved else EfiBootServicesData endif > Looking at U-Boot code, it sets the no-map region to > EfiBootServicesData which may cause > issues in RISC-V as well if we linear mapping sets up the initial 2MB. Hmm https://elixir.bootlin.com/u-boot/v2023.07-rc3/source/lib/efi_loader/efi_dt_fixup.c#L25 seems to follow what I quoted from the DT spec above: if (nomap) type = EFI_RESERVED_MEMORY_TYPE; else type = EFI_BOOT_SERVICES_DATA; And I don't find anything regarding "reusable" in u-boot code. > > >> Let me know your thoughts how we can handle this better in EDK2 >> considering it has to support ACPI also. >> >> Thanks, >> Sunil >> >> _______________________________________________ >> linux-riscv mailing list >> linux-riscv@lists.infradead.org >> http://lists.infradead.org/mailman/listinfo/linux-riscv > > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-06 7:20 ` Alexandre Ghiti @ 2023-06-06 9:01 ` Sunil V L 2023-06-06 11:50 ` Alexandre Ghiti 0 siblings, 1 reply; 16+ messages in thread From: Sunil V L @ 2023-06-06 9:01 UTC (permalink / raw) To: Alexandre Ghiti Cc: Atish Patra, Alexandre Ghiti, Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On Tue, Jun 06, 2023 at 09:20:34AM +0200, Alexandre Ghiti wrote: > > On 05/06/2023 22:55, Atish Patra wrote: > > On Mon, Jun 5, 2023 at 8:13 AM Sunil V L <sunilvl@ventanamicro.com> wrote: > > > On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: > > > > Hi Song, > > > > > > > > On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > > > > > Description of problem: > > > > > > > > > > Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. > > > > > The entire log has been posted at this link : https://termbin.com/nga4. > > > > > > > > > > You can reproduce it with the following step : > > > > > > > > > > 1. prepare the environment with > > > > > - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > > > > > - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") > > > > > - Linux : v6.4-rc1 and later version > > > > > > > > > > 2. start the Qemu virt board > > > > > > > > > > ```sh > > > > > $ cat ~/8_riscv/start_latest.sh > > > > > #!/bin/bash > > > > > /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > > > > > -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > > > > > -machine virt,acpi=off -smp 2 -m 2G \ > > > > > -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > > > > > -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > > > > > -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest > > > > > ``` > > > > > 3. Then you will encounter the kernel panic logged in the above link > > > > > > > > > > Other Information: > > > > > > > > > > 1. ------- > > > > > > > > > > This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > > > > > are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > > > > > > > > > > With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > > > > > from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). > > > > > > > > > > In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > > > > > so there is no "access fault" but "page fault". > > > > > > > > > > And reverting commit 3335068f8721 indeed fixed this panic. > > > > > > > > > > 2. ------- > > > > > > > > > > As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. > > > > > Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? > > > > > > > > > > ``` > > > > > (gdb) p/x $satp > > > > > $1 = 0xa000000000081708 > > > > > (gdb) pt -satp 0xa000000000081708 > > > > > Address : Length Permissions > > > > > 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > > 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > > 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > > > > > 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > > > > > 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > > > > > 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 > > > > > 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > > > > > 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > > 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > > > > > 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > > > > > > > > > > ``` > > > > > > > > > > 3. ------ > > > > > > > > > > You can also reproduce similar panic by appending "memtest" in kernel cmdline. > > > > > I have posted the memtest boot log at this link: https://termbin.com/1twl. > > > > > > > > > > Please correct me if I'm wrong. > > > > > > > > > > [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > > > > > [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > > > > > [3]: https://github.com/martinradev/gdb-pt-dump > > > > Thanks for the thorough report, really appreciated. > > > > > > > > So there are multiple issues here: > > > > > > > > - the first one is that the memory region for opensbi is marked as not > > > > cacheable in the efi memory map, and then this region is not mapped in > > > > the linear mapping: > > > > [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | > > > > | | | | | | | | | | | |UC] > > > > > > @Alex: The OpenSBI region is marked reserved because EDK2 chooses to > > do that explicitly as explained by Sunil. > > I don't think UC has to do anything with it. All the EFI memory regions are UC. > > > Oops, thanks for correcting me, UC has nothing to do with that indeed, I > misread is_usable_memory() (https://elixir.bootlin.com/linux/latest/source/drivers/firmware/efi/efi-init.c#L127): > EFI_RESERVED_TYPE will return false. > > And then this region won't get mapped, so that's the equivalent of "no-map". > > > > > > > > - the second one (that I feel a bit ashamed of...) is that I did not > > > > check the alignment of the virtual address when choosing the map size > > > > in best_map_size() and then we end up trying to map a physical region > > > > aligned on 2MB that is actually not aligned on 2MB virtually because > > > > the opensbi region is not mapped at all. > > > > > > > > - the possible third one is that we should not map the linear mapping > > > > using 4K pages, this would be slow in my opinion, and I think we > > > > should waste a bit of memory to align va and pa on a 2MB boundary. > > > > > > > > So I'll fix the second issue, and possibly the third one, and if no > > > > one looks into why the opensbi region is mapped in UC, I'll take a > > > > look at edk2. > > > > > > > Hi Alex, > > > > > > EDK2 marks opensbi range as reserved memory in EFI map. According to DT > > > spec, if the no-map is not set, we need to mark it as > > > EfiBootServicesData but EfiBootServicesData is actually considered as > > > free memory in kernel, as per UEFI spec. To avoid kernel using this > > > memory, we deviated from the DT spec for opensbi ranges. > > > > > IMO, that should be the correct way unless we can change it to > > EfiRunServicesData/Code. > > > I'd say that everything depends on "reusable" no? > > If "no-map" then > EfiReserved > else > if reusable then > EfiBootServicesData > else > EfiRuntimeServicesData > endif > endif > Hi Alex, Is this your proposal or documented somewhere? If it is proposal, this option was intentionally not allowed as I understand from Ard's response at [1]. [1] - https://github.com/ARM-software/ebbr/issues/52#issuecomment-690362294 Thanks, Sunil _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-06 9:01 ` Sunil V L @ 2023-06-06 11:50 ` Alexandre Ghiti 0 siblings, 0 replies; 16+ messages in thread From: Alexandre Ghiti @ 2023-06-06 11:50 UTC (permalink / raw) To: Sunil V L Cc: Alexandre Ghiti, Atish Patra, Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On Tue, Jun 6, 2023 at 11:01 AM Sunil V L <sunilvl@ventanamicro.com> wrote: > > On Tue, Jun 06, 2023 at 09:20:34AM +0200, Alexandre Ghiti wrote: > > > > On 05/06/2023 22:55, Atish Patra wrote: > > > On Mon, Jun 5, 2023 at 8:13 AM Sunil V L <sunilvl@ventanamicro.com> wrote: > > > > On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: > > > > > Hi Song, > > > > > > > > > > On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > > > > > > Description of problem: > > > > > > > > > > > > Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. > > > > > > The entire log has been posted at this link : https://termbin.com/nga4. > > > > > > > > > > > > You can reproduce it with the following step : > > > > > > > > > > > > 1. prepare the environment with > > > > > > - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > > > > > > - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") > > > > > > - Linux : v6.4-rc1 and later version > > > > > > > > > > > > 2. start the Qemu virt board > > > > > > > > > > > > ```sh > > > > > > $ cat ~/8_riscv/start_latest.sh > > > > > > #!/bin/bash > > > > > > /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > > > > > > -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > > > > > > -machine virt,acpi=off -smp 2 -m 2G \ > > > > > > -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > > > > > > -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > > > > > > -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest > > > > > > ``` > > > > > > 3. Then you will encounter the kernel panic logged in the above link > > > > > > > > > > > > Other Information: > > > > > > > > > > > > 1. ------- > > > > > > > > > > > > This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > > > > > > are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > > > > > > > > > > > > With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > > > > > > from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). > > > > > > > > > > > > In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > > > > > > so there is no "access fault" but "page fault". > > > > > > > > > > > > And reverting commit 3335068f8721 indeed fixed this panic. > > > > > > > > > > > > 2. ------- > > > > > > > > > > > > As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. > > > > > > Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? > > > > > > > > > > > > ``` > > > > > > (gdb) p/x $satp > > > > > > $1 = 0xa000000000081708 > > > > > > (gdb) pt -satp 0xa000000000081708 > > > > > > Address : Length Permissions > > > > > > 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > > > 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > > > 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > > > > > > 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > > > > > > 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > > > > > > 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 > > > > > > 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > > > > > > 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > > > > > > 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > > > > > > 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > > > > > > > > > > > > ``` > > > > > > > > > > > > 3. ------ > > > > > > > > > > > > You can also reproduce similar panic by appending "memtest" in kernel cmdline. > > > > > > I have posted the memtest boot log at this link: https://termbin.com/1twl. > > > > > > > > > > > > Please correct me if I'm wrong. > > > > > > > > > > > > [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > > > > > > [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > > > > > > [3]: https://github.com/martinradev/gdb-pt-dump > > > > > Thanks for the thorough report, really appreciated. > > > > > > > > > > So there are multiple issues here: > > > > > > > > > > - the first one is that the memory region for opensbi is marked as not > > > > > cacheable in the efi memory map, and then this region is not mapped in > > > > > the linear mapping: > > > > > [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | > > > > > | | | | | | | | | | | |UC] > > > > > > > > @Alex: The OpenSBI region is marked reserved because EDK2 chooses to > > > do that explicitly as explained by Sunil. > > > I don't think UC has to do anything with it. All the EFI memory regions are UC. > > > > > > Oops, thanks for correcting me, UC has nothing to do with that indeed, I > > misread is_usable_memory() (https://elixir.bootlin.com/linux/latest/source/drivers/firmware/efi/efi-init.c#L127): > > EFI_RESERVED_TYPE will return false. > > > > And then this region won't get mapped, so that's the equivalent of "no-map". > > > > > > > > > > > > - the second one (that I feel a bit ashamed of...) is that I did not > > > > > check the alignment of the virtual address when choosing the map size > > > > > in best_map_size() and then we end up trying to map a physical region > > > > > aligned on 2MB that is actually not aligned on 2MB virtually because > > > > > the opensbi region is not mapped at all. > > > > > > > > > > - the possible third one is that we should not map the linear mapping > > > > > using 4K pages, this would be slow in my opinion, and I think we > > > > > should waste a bit of memory to align va and pa on a 2MB boundary. > > > > > > > > > > So I'll fix the second issue, and possibly the third one, and if no > > > > > one looks into why the opensbi region is mapped in UC, I'll take a > > > > > look at edk2. > > > > > > > > > Hi Alex, > > > > > > > > EDK2 marks opensbi range as reserved memory in EFI map. According to DT > > > > spec, if the no-map is not set, we need to mark it as > > > > EfiBootServicesData but EfiBootServicesData is actually considered as > > > > free memory in kernel, as per UEFI spec. To avoid kernel using this > > > > memory, we deviated from the DT spec for opensbi ranges. > > > > > > > IMO, that should be the correct way unless we can change it to > > > EfiRunServicesData/Code. > > > > > > I'd say that everything depends on "reusable" no? > > > > If "no-map" then > > EfiReserved > > else > > if reusable then > > EfiBootServicesData > > else > > EfiRuntimeServicesData > > endif > > endif > > > Hi Alex, > > Is this your proposal or documented somewhere? If it is proposal, this > option was intentionally not allowed as I understand from Ard's response > at [1]. > > [1] - https://github.com/ARM-software/ebbr/issues/52#issuecomment-690362294 You mean because EfiRuntimeServicesData are actually not mapped in the linear mapping by the kernel right? In that case, yes you're right! > > Thanks, > Sunil _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-05 15:12 ` Sunil V L 2023-06-05 20:55 ` Atish Patra @ 2023-06-05 21:42 ` Jessica Clarke 2023-06-06 6:40 ` Sunil V L 1 sibling, 1 reply; 16+ messages in thread From: Jessica Clarke @ 2023-06-05 21:42 UTC (permalink / raw) To: Sunil V L Cc: Alexandre Ghiti, Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On 5 Jun 2023, at 16:12, Sunil V L <sunilvl@ventanamicro.com> wrote: > > On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: >> Hi Song, >> >> On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: >>> >>> Description of problem: >>> >>> Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. >>> The entire log has been posted at this link : https://termbin.com/nga4. >>> >>> You can reproduce it with the following step : >>> >>> 1. prepare the environment with >>> - Qemu-virt: v8.0.0 (with OpenSbi v1.2) >>> - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") >>> - Linux : v6.4-rc1 and later version >>> >>> 2. start the Qemu virt board >>> >>> ```sh >>> $ cat ~/8_riscv/start_latest.sh >>> #!/bin/bash >>> /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ >>> -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ >>> -machine virt,acpi=off -smp 2 -m 2G \ >>> -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ >>> -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ >>> -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest >>> ``` >>> 3. Then you will encounter the kernel panic logged in the above link >>> >>> Other Information: >>> >>> 1. ------- >>> >>> This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them >>> are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). >>> >>> With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) >>> from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). >>> >>> In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, >>> so there is no "access fault" but "page fault". >>> >>> And reverting commit 3335068f8721 indeed fixed this panic. >>> >>> 2. ------- >>> >>> As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. >>> Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? >>> >>> ``` >>> (gdb) p/x $satp >>> $1 = 0xa000000000081708 >>> (gdb) pt -satp 0xa000000000081708 >>> Address : Length Permissions >>> 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 >>> 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 >>> 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 >>> 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 >>> 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 >>> 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 >>> 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 >>> 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 >>> 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 >>> 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 >>> >>> ``` >>> >>> 3. ------ >>> >>> You can also reproduce similar panic by appending "memtest" in kernel cmdline. >>> I have posted the memtest boot log at this link: https://termbin.com/1twl. >>> >>> Please correct me if I'm wrong. >>> >>> [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ >>> [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ >>> [3]: https://github.com/martinradev/gdb-pt-dump >> >> Thanks for the thorough report, really appreciated. >> >> So there are multiple issues here: >> >> - the first one is that the memory region for opensbi is marked as not >> cacheable in the efi memory map, and then this region is not mapped in >> the linear mapping: >> [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | >> | | | | | | | | | | | |UC] >> >> - the second one (that I feel a bit ashamed of...) is that I did not >> check the alignment of the virtual address when choosing the map size >> in best_map_size() and then we end up trying to map a physical region >> aligned on 2MB that is actually not aligned on 2MB virtually because >> the opensbi region is not mapped at all. >> >> - the possible third one is that we should not map the linear mapping >> using 4K pages, this would be slow in my opinion, and I think we >> should waste a bit of memory to align va and pa on a 2MB boundary. >> >> So I'll fix the second issue, and possibly the third one, and if no >> one looks into why the opensbi region is mapped in UC, I'll take a >> look at edk2. >> > Hi Alex, > > EDK2 marks opensbi range as reserved memory in EFI map. According to DT > spec, if the no-map is not set, we need to mark it as > EfiBootServicesData but EfiBootServicesData is actually considered as > free memory in kernel, as per UEFI spec. To avoid kernel using this > memory, we deviated from the DT spec for opensbi ranges. Violating specs is never the answer. Do one of: 1. Use no-map and take the performance hit 2. Exclude the memory range from /memory itself 3. Come up with a new no-access property that’s a weaker no-map (i.e. that allows mapping and speculative access) and uses EfiRuntimeServicesData in EFI land 2 feels most normal to me, personally, but all are fine. Jess > Let me know your thoughts how we can handle this better in EDK2 > considering it has to support ACPI also. > > Thanks, > Sunil > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-05 21:42 ` Jessica Clarke @ 2023-06-06 6:40 ` Sunil V L 2023-06-06 7:25 ` Alexandre Ghiti 0 siblings, 1 reply; 16+ messages in thread From: Sunil V L @ 2023-06-06 6:40 UTC (permalink / raw) To: Jessica Clarke Cc: Alexandre Ghiti, Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On Mon, Jun 05, 2023 at 10:42:33PM +0100, Jessica Clarke wrote: > On 5 Jun 2023, at 16:12, Sunil V L <sunilvl@ventanamicro.com> wrote: > > > > On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: > >> Hi Song, > >> > >> On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > >>> > >>> Description of problem: > >>> > >>> Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. > >>> The entire log has been posted at this link : https://termbin.com/nga4. > >>> > >>> You can reproduce it with the following step : > >>> > >>> 1. prepare the environment with > >>> - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > >>> - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") > >>> - Linux : v6.4-rc1 and later version > >>> > >>> 2. start the Qemu virt board > >>> > >>> ```sh > >>> $ cat ~/8_riscv/start_latest.sh > >>> #!/bin/bash > >>> /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > >>> -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > >>> -machine virt,acpi=off -smp 2 -m 2G \ > >>> -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > >>> -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > >>> -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest > >>> ``` > >>> 3. Then you will encounter the kernel panic logged in the above link > >>> > >>> Other Information: > >>> > >>> 1. ------- > >>> > >>> This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > >>> are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > >>> > >>> With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > >>> from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). > >>> > >>> In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > >>> so there is no "access fault" but "page fault". > >>> > >>> And reverting commit 3335068f8721 indeed fixed this panic. > >>> > >>> 2. ------- > >>> > >>> As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. > >>> Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? > >>> > >>> ``` > >>> (gdb) p/x $satp > >>> $1 = 0xa000000000081708 > >>> (gdb) pt -satp 0xa000000000081708 > >>> Address : Length Permissions > >>> 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > >>> 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > >>> 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > >>> 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > >>> 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > >>> 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 > >>> 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > >>> 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > >>> 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > >>> 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > >>> > >>> ``` > >>> > >>> 3. ------ > >>> > >>> You can also reproduce similar panic by appending "memtest" in kernel cmdline. > >>> I have posted the memtest boot log at this link: https://termbin.com/1twl. > >>> > >>> Please correct me if I'm wrong. > >>> > >>> [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > >>> [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > >>> [3]: https://github.com/martinradev/gdb-pt-dump > >> > >> Thanks for the thorough report, really appreciated. > >> > >> So there are multiple issues here: > >> > >> - the first one is that the memory region for opensbi is marked as not > >> cacheable in the efi memory map, and then this region is not mapped in > >> the linear mapping: > >> [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | > >> | | | | | | | | | | | |UC] > >> > >> - the second one (that I feel a bit ashamed of...) is that I did not > >> check the alignment of the virtual address when choosing the map size > >> in best_map_size() and then we end up trying to map a physical region > >> aligned on 2MB that is actually not aligned on 2MB virtually because > >> the opensbi region is not mapped at all. > >> > >> - the possible third one is that we should not map the linear mapping > >> using 4K pages, this would be slow in my opinion, and I think we > >> should waste a bit of memory to align va and pa on a 2MB boundary. > >> > >> So I'll fix the second issue, and possibly the third one, and if no > >> one looks into why the opensbi region is mapped in UC, I'll take a > >> look at edk2. > >> > > Hi Alex, > > > > EDK2 marks opensbi range as reserved memory in EFI map. According to DT > > spec, if the no-map is not set, we need to mark it as > > EfiBootServicesData but EfiBootServicesData is actually considered as > > free memory in kernel, as per UEFI spec. To avoid kernel using this > > memory, we deviated from the DT spec for opensbi ranges. > > Violating specs is never the answer. Do one of: > > 1. Use no-map and take the performance hit > 2. Exclude the memory range from /memory itself > 3. Come up with a new no-access property that’s a weaker no-map > (i.e. that allows mapping and speculative access) and uses > EfiRuntimeServicesData in EFI land > > 2 feels most normal to me, personally, but all are fine. > Hi Jess, IMO, all the physical memory installed by the user should be visible. Some part of the memory may be reserved and not available for the user but excluding from /memory can cause issues. Whether we mark as EfiReservedMemory or EfiRuntimeServiceData, I think it will be marked as no-map in memblock and can not be used by the OS linear mapping. Alex can confirm this. So, my preference is option 1. Thanks, Sunil _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-06 6:40 ` Sunil V L @ 2023-06-06 7:25 ` Alexandre Ghiti 2023-06-06 18:21 ` Atish Patra 0 siblings, 1 reply; 16+ messages in thread From: Alexandre Ghiti @ 2023-06-06 7:25 UTC (permalink / raw) To: Sunil V L, Jessica Clarke Cc: Alexandre Ghiti, Song Shuai, robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On 06/06/2023 08:40, Sunil V L wrote: > On Mon, Jun 05, 2023 at 10:42:33PM +0100, Jessica Clarke wrote: >> On 5 Jun 2023, at 16:12, Sunil V L <sunilvl@ventanamicro.com> wrote: >>> On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: >>>> Hi Song, >>>> >>>> On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: >>>>> Description of problem: >>>>> >>>>> Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. >>>>> The entire log has been posted at this link : https://termbin.com/nga4. >>>>> >>>>> You can reproduce it with the following step : >>>>> >>>>> 1. prepare the environment with >>>>> - Qemu-virt: v8.0.0 (with OpenSbi v1.2) >>>>> - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") >>>>> - Linux : v6.4-rc1 and later version >>>>> >>>>> 2. start the Qemu virt board >>>>> >>>>> ```sh >>>>> $ cat ~/8_riscv/start_latest.sh >>>>> #!/bin/bash >>>>> /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ >>>>> -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ >>>>> -machine virt,acpi=off -smp 2 -m 2G \ >>>>> -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ >>>>> -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ >>>>> -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest >>>>> ``` >>>>> 3. Then you will encounter the kernel panic logged in the above link >>>>> >>>>> Other Information: >>>>> >>>>> 1. ------- >>>>> >>>>> This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them >>>>> are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). >>>>> >>>>> With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) >>>>> from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). >>>>> >>>>> In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, >>>>> so there is no "access fault" but "page fault". >>>>> >>>>> And reverting commit 3335068f8721 indeed fixed this panic. >>>>> >>>>> 2. ------- >>>>> >>>>> As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. >>>>> Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? >>>>> >>>>> ``` >>>>> (gdb) p/x $satp >>>>> $1 = 0xa000000000081708 >>>>> (gdb) pt -satp 0xa000000000081708 >>>>> Address : Length Permissions >>>>> 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 >>>>> 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 >>>>> 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 >>>>> 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 >>>>> 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 >>>>> 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 >>>>> 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 >>>>> 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 >>>>> 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 >>>>> 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 >>>>> >>>>> ``` >>>>> >>>>> 3. ------ >>>>> >>>>> You can also reproduce similar panic by appending "memtest" in kernel cmdline. >>>>> I have posted the memtest boot log at this link: https://termbin.com/1twl. >>>>> >>>>> Please correct me if I'm wrong. >>>>> >>>>> [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ >>>>> [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ >>>>> [3]: https://github.com/martinradev/gdb-pt-dump >>>> Thanks for the thorough report, really appreciated. >>>> >>>> So there are multiple issues here: >>>> >>>> - the first one is that the memory region for opensbi is marked as not >>>> cacheable in the efi memory map, and then this region is not mapped in >>>> the linear mapping: >>>> [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | >>>> | | | | | | | | | | | |UC] >>>> >>>> - the second one (that I feel a bit ashamed of...) is that I did not >>>> check the alignment of the virtual address when choosing the map size >>>> in best_map_size() and then we end up trying to map a physical region >>>> aligned on 2MB that is actually not aligned on 2MB virtually because >>>> the opensbi region is not mapped at all. >>>> >>>> - the possible third one is that we should not map the linear mapping >>>> using 4K pages, this would be slow in my opinion, and I think we >>>> should waste a bit of memory to align va and pa on a 2MB boundary. >>>> >>>> So I'll fix the second issue, and possibly the third one, and if no >>>> one looks into why the opensbi region is mapped in UC, I'll take a >>>> look at edk2. >>>> >>> Hi Alex, >>> >>> EDK2 marks opensbi range as reserved memory in EFI map. According to DT >>> spec, if the no-map is not set, we need to mark it as >>> EfiBootServicesData but EfiBootServicesData is actually considered as >>> free memory in kernel, as per UEFI spec. To avoid kernel using this >>> memory, we deviated from the DT spec for opensbi ranges. >> Violating specs is never the answer. Do one of: >> >> 1. Use no-map and take the performance hit >> 2. Exclude the memory range from /memory itself >> 3. Come up with a new no-access property that’s a weaker no-map >> (i.e. that allows mapping and speculative access) and uses >> EfiRuntimeServicesData in EFI land >> >> 2 feels most normal to me, personally, but all are fine. >> > Hi Jess, > > IMO, all the physical memory installed by the user should be visible. > Some part of the memory may be reserved and not available for the user > but excluding from /memory can cause issues. > > Whether we mark as EfiReservedMemory or EfiRuntimeServiceData, I think > it will be marked as no-map in memblock and can not be used by the OS > linear mapping. Alex can confirm this. Yes, I think you're right, EfiRuntimeServiceData will be marked as no-map anyway (see is_usable_memory()). > > So, my preference is option 1. Yes, again, I think you're right, this is feeling more and more like the most "natural" solution to me too, we are struggling for a performance benefit that was never proven... > > Thanks, > Sunil > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-06 7:25 ` Alexandre Ghiti @ 2023-06-06 18:21 ` Atish Patra 0 siblings, 0 replies; 16+ messages in thread From: Atish Patra @ 2023-06-06 18:21 UTC (permalink / raw) To: Alexandre Ghiti Cc: robh, mason.huo, Alexandre Ghiti, jeeheng.sia, anup, conor.dooley, palmer, linux-kernel, leyfoon.tan, Song Shuai, guoren, paul.walmsley, linux-riscv, Jessica Clarke, ajones On Tue, Jun 6, 2023 at 12:26 AM Alexandre Ghiti <alex@ghiti.fr> wrote: > > > On 06/06/2023 08:40, Sunil V L wrote: > > On Mon, Jun 05, 2023 at 10:42:33PM +0100, Jessica Clarke wrote: > >> On 5 Jun 2023, at 16:12, Sunil V L <sunilvl@ventanamicro.com> wrote: > >>> On Mon, Jun 05, 2023 at 04:25:06PM +0200, Alexandre Ghiti wrote: > >>>> Hi Song, > >>>> > >>>> On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > >>>>> Description of problem: > >>>>> > >>>>> Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. > >>>>> The entire log has been posted at this link : https://termbin.com/nga4. > >>>>> > >>>>> You can reproduce it with the following step : > >>>>> > >>>>> 1. prepare the environment with > >>>>> - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > >>>>> - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") > >>>>> - Linux : v6.4-rc1 and later version > >>>>> > >>>>> 2. start the Qemu virt board > >>>>> > >>>>> ```sh > >>>>> $ cat ~/8_riscv/start_latest.sh > >>>>> #!/bin/bash > >>>>> /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > >>>>> -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > >>>>> -machine virt,acpi=off -smp 2 -m 2G \ > >>>>> -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > >>>>> -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > >>>>> -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest > >>>>> ``` > >>>>> 3. Then you will encounter the kernel panic logged in the above link > >>>>> > >>>>> Other Information: > >>>>> > >>>>> 1. ------- > >>>>> > >>>>> This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > >>>>> are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > >>>>> > >>>>> With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > >>>>> from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). > >>>>> > >>>>> In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > >>>>> so there is no "access fault" but "page fault". > >>>>> > >>>>> And reverting commit 3335068f8721 indeed fixed this panic. > >>>>> > >>>>> 2. ------- > >>>>> > >>>>> As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. > >>>>> Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? > >>>>> > >>>>> ``` > >>>>> (gdb) p/x $satp > >>>>> $1 = 0xa000000000081708 > >>>>> (gdb) pt -satp 0xa000000000081708 > >>>>> Address : Length Permissions > >>>>> 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > >>>>> 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > >>>>> 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > >>>>> 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > >>>>> 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > >>>>> 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 > >>>>> 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > >>>>> 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > >>>>> 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > >>>>> 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > >>>>> > >>>>> ``` > >>>>> > >>>>> 3. ------ > >>>>> > >>>>> You can also reproduce similar panic by appending "memtest" in kernel cmdline. > >>>>> I have posted the memtest boot log at this link: https://termbin.com/1twl. > >>>>> > >>>>> Please correct me if I'm wrong. > >>>>> > >>>>> [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > >>>>> [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > >>>>> [3]: https://github.com/martinradev/gdb-pt-dump > >>>> Thanks for the thorough report, really appreciated. > >>>> > >>>> So there are multiple issues here: > >>>> > >>>> - the first one is that the memory region for opensbi is marked as not > >>>> cacheable in the efi memory map, and then this region is not mapped in > >>>> the linear mapping: > >>>> [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | > >>>> | | | | | | | | | | | |UC] > >>>> > >>>> - the second one (that I feel a bit ashamed of...) is that I did not > >>>> check the alignment of the virtual address when choosing the map size > >>>> in best_map_size() and then we end up trying to map a physical region > >>>> aligned on 2MB that is actually not aligned on 2MB virtually because > >>>> the opensbi region is not mapped at all. > >>>> > >>>> - the possible third one is that we should not map the linear mapping > >>>> using 4K pages, this would be slow in my opinion, and I think we > >>>> should waste a bit of memory to align va and pa on a 2MB boundary. > >>>> > >>>> So I'll fix the second issue, and possibly the third one, and if no > >>>> one looks into why the opensbi region is mapped in UC, I'll take a > >>>> look at edk2. > >>>> > >>> Hi Alex, > >>> > >>> EDK2 marks opensbi range as reserved memory in EFI map. According to DT > >>> spec, if the no-map is not set, we need to mark it as > >>> EfiBootServicesData but EfiBootServicesData is actually considered as > >>> free memory in kernel, as per UEFI spec. To avoid kernel using this > >>> memory, we deviated from the DT spec for opensbi ranges. > >> Violating specs is never the answer. Do one of: > >> > >> 1. Use no-map and take the performance hit > >> 2. Exclude the memory range from /memory itself > >> 3. Come up with a new no-access property that’s a weaker no-map > >> (i.e. that allows mapping and speculative access) and uses > >> EfiRuntimeServicesData in EFI land > >> > >> 2 feels most normal to me, personally, but all are fine. > >> > > Hi Jess, > > > > IMO, all the physical memory installed by the user should be visible. > > Some part of the memory may be reserved and not available for the user > > but excluding from /memory can cause issues. > > > > Whether we mark as EfiReservedMemory or EfiRuntimeServiceData, I think > > it will be marked as no-map in memblock and can not be used by the OS > > linear mapping. Alex can confirm this. > > > Yes, I think you're right, EfiRuntimeServiceData will be marked as > no-map anyway (see is_usable_memory()). > > > > > > So, my preference is option 1. > > > Yes, again, I think you're right, this is feeling more and more like the > most "natural" solution to me too, we are struggling for a performance > benefit that was never proven... > I am inclined towards this option as well. After going through the rationale for marking any /reserved-memory node without "no-map" as EfiBootServicesData, this will trip kernel in future if not happening already. Any region marked as EfiBootServicesData will be available to the kernel for use after ExitBootServices. Let's have a no-map set for the reserved memory set for the firmware. The fallout would be anybody with kernel > 6.4 has to upgrade the firmware version that sets the no-map correctly if they care about hibernation or EFI booting. OpenSBI v1.3 is planned this month anyway. We can communicate the same to the rust-sbi project as well. Any thoughts ? > > > > > Thanks, > > Sunil > > > > _______________________________________________ > > linux-riscv mailing list > > linux-riscv@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-riscv > > _______________________________________________ > linux-riscv mailing list > linux-riscv@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-riscv -- Regards, Atish _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-05 14:25 ` Alexandre Ghiti 2023-06-05 15:12 ` Sunil V L @ 2023-06-06 10:04 ` Song Shuai 2023-06-06 11:46 ` Alexandre Ghiti 1 sibling, 1 reply; 16+ messages in thread From: Song Shuai @ 2023-06-06 10:04 UTC (permalink / raw) To: Alexandre Ghiti Cc: robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel 在 2023/6/5 22:25, Alexandre Ghiti 写道: > Hi Song, > > On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: >> >> Description of problem: >> >> Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. >> The entire log has been posted at this link : https://termbin.com/nga4. >> >> You can reproduce it with the following step : >> >> 1. prepare the environment with >> - Qemu-virt: v8.0.0 (with OpenSbi v1.2) >> - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") >> - Linux : v6.4-rc1 and later version >> >> 2. start the Qemu virt board >> >> ```sh >> $ cat ~/8_riscv/start_latest.sh >> #!/bin/bash >> /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ >> -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ >> -machine virt,acpi=off -smp 2 -m 2G \ >> -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ >> -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ >> -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest >> ``` >> 3. Then you will encounter the kernel panic logged in the above link >> >> Other Information: >> >> 1. ------- >> >> This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them >> are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). >> >> With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) >> from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). >> >> In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, >> so there is no "access fault" but "page fault". >> >> And reverting commit 3335068f8721 indeed fixed this panic. >> >> 2. ------- >> >> As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. >> Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? >> >> ``` >> (gdb) p/x $satp >> $1 = 0xa000000000081708 >> (gdb) pt -satp 0xa000000000081708 >> Address : Length Permissions >> 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 >> 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 >> 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 >> 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 >> 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 >> 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 >> 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 >> 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 >> 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 >> 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 >> >> ``` >> >> 3. ------ >> >> You can also reproduce similar panic by appending "memtest" in kernel cmdline. >> I have posted the memtest boot log at this link: https://termbin.com/1twl. >> >> Please correct me if I'm wrong. >> >> [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ >> [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ >> [3]: https://github.com/martinradev/gdb-pt-dump > > Thanks for the thorough report, really appreciated. > > So there are multiple issues here: > > - the first one is that the memory region for opensbi is marked as not > cacheable in the efi memory map, and then this region is not mapped in > the linear mapping: > [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | > | | | | | | | | | | | |UC] > > - the second one (that I feel a bit ashamed of...) is that I did not > check the alignment of the virtual address when choosing the map size > in best_map_size() and then we end up trying to map a physical region > aligned on 2MB that is actually not aligned on 2MB virtually because > the opensbi region is not mapped at all. > The issue 2 should be the root cause of this panic. Here is my understanding of the necessity of the 2M-aligned VA for linear PMD mapping. Please correct me if I'm wrong. I logged the `create_linear_mapping_range()` function. ``` song # lowmem region: [0x0000000081800000 -- 0x00000000ffe3d000], va: 0xff6000007fbc0000, pa: 0x00000000ffc00000, map_size: 200000 ,pg: e7 song # lowmem region: [0x0000000081800000 -- 0x00000000ffe3d000], va: 0xff6000007fdc0000, pa: 0x00000000ffe00000, map_size: 1000 ,pg: e7 ``` The PA `0x00000000ffc00000` of this PMD mapping is aligned with PMD_SIZE but VA `0xff6000007fbc0000` is not. After the `pmd_index()`, this 2M PA region is actually mapping the effective VA region `[0xff6000007fa00000,0xff6000007fc00000)`, and any access of VA hole between the end of the effective VA region and the start VA of the next 4K mapping (`0xff6000007fdc0000`) will fault. In this report, the memtest fault VA (`0xff6000007fc00000`) and the booting fault VA (`ff6000007fdb1000`) lie right in the VA hole. When reverting the commit 3335068f8721, the kernel load address is always offseted by PMD_SIZE, kernel_map.va_pa_offset and MIN_MEMBLOCK_ADDR follow it. So the linear PMD mapping will always take 2M-aligned VA. That's why this reverting works. > - the possible third one is that we should not map the linear mapping > using 4K pages, this would be slow in my opinion, and I think we > should waste a bit of memory to align va and pa on a 2MB boundary. I also noticed this one. > > So I'll fix the second issue, and possibly the third one, and if no Thanks for your attention to this report, looking for your fixup. > one looks into why the opensbi region is mapped in UC, I'll take a > look at edk2. > > Sorry for that, > > Alex > -- Song Shuai Thanks _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: Bug report: kernel paniced while booting 2023-06-06 10:04 ` Song Shuai @ 2023-06-06 11:46 ` Alexandre Ghiti 0 siblings, 0 replies; 16+ messages in thread From: Alexandre Ghiti @ 2023-06-06 11:46 UTC (permalink / raw) To: Song Shuai Cc: robh, ajones, anup, palmer, jeeheng.sia, leyfoon.tan, mason.huo, paul.walmsley, conor.dooley, guoren, linux-riscv, linux-kernel On Tue, Jun 6, 2023 at 12:04 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > > > > 在 2023/6/5 22:25, Alexandre Ghiti 写道: > > Hi Song, > > > > On Mon, Jun 5, 2023 at 12:52 PM Song Shuai <songshuaishuai@tinylab.org> wrote: > >> > >> Description of problem: > >> > >> Booting Linux With RiscVVirtQemu edk2 firmware, a Store/AMO page fault was trapped to trigger a kernel panic. > >> The entire log has been posted at this link : https://termbin.com/nga4. > >> > >> You can reproduce it with the following step : > >> > >> 1. prepare the environment with > >> - Qemu-virt: v8.0.0 (with OpenSbi v1.2) > >> - edk2 : at commit (2bc8545883 "UefiCpuPkg/CpuPageTableLib: Reduce the number of random tests") > >> - Linux : v6.4-rc1 and later version > >> > >> 2. start the Qemu virt board > >> > >> ```sh > >> $ cat ~/8_riscv/start_latest.sh > >> #!/bin/bash > >> /home/song/8_riscv/3_acpi/qemu/ooo/usr/local/bin/qemu-system-riscv64 \ > >> -s -nographic -drive file=/home/song/8_riscv/3_acpi/Build_virt/RiscVVirtQemu/RELEASE_GCC5/FV/RISCV_VIRT.fd,if=pflash,format=raw,unit=1 \ > >> -machine virt,acpi=off -smp 2 -m 2G \ > >> -kernel /home/song/9_linux/linux/00_rv_def/arch/riscv/boot/Image \ > >> -initrd /home/song/8_riscv/3_acpi/buildroot/output/images/rootfs.ext2 \ > >> -append "root=/dev/ram ro console=ttyS0 earlycon=uart8250,mmio,0x10000000 efi=debug loglevel=8 memblock=debug" ## also panic by memtest > >> ``` > >> 3. Then you will encounter the kernel panic logged in the above link > >> > >> Other Information: > >> > >> 1. ------- > >> > >> This report is not identical to my prior report -- "kernel paniced when system hibernates" [1], but both of them > >> are closely related with the commit (3335068f8721 "riscv: Use PUD/P4D/PGD pages for the linear mapping"). > >> > >> With this commit, hibernation is trapped with "access fault" while accessing the PMP-protected regions (mmode_resv0@80000000) > >> from OpenSbi (BTW, hibernation is marked as nonportable by Conor[2]). > >> > >> In this report, efi_init handoffs the memory mapping from Boot Services to memblock where reserves mmode_resv0@80000000, > >> so there is no "access fault" but "page fault". > >> > >> And reverting commit 3335068f8721 indeed fixed this panic. > >> > >> 2. ------- > >> > >> As the gdb-pt-dump [3] tool shows, the PTE which covered the fault virtual address had the appropriate permission to store. > >> Is there another way to trigger the "Store/AMO page fault"? Or the creation of linear mapping in commit 3335068f8721 did something wrong? > >> > >> ``` > >> (gdb) p/x $satp > >> $1 = 0xa000000000081708 > >> (gdb) pt -satp 0xa000000000081708 > >> Address : Length Permissions > >> 0xff1bfffffea39000 : 0x1000 | W:1 X:0 R:1 S:1 > >> 0xff1bfffffebf9000 : 0x1000 | W:1 X:0 R:1 S:1 > >> 0xff1bfffffec00000 : 0x400000 | W:1 X:0 R:1 S:1 > >> 0xff60000000000000 : 0x1c0000 | W:1 X:0 R:1 S:1 > >> 0xff60000000200000 : 0xa00000 | W:0 X:0 R:1 S:1 > >> 0xff60000000c00000 : 0x7f000000 | W:1 X:0 R:1 S:1 // badaddr: ff6000007fdb1000 > >> 0xff6000007fdc0000 : 0x3d000 | W:1 X:0 R:1 S:1 > >> 0xff6000007ffbf000 : 0x1000 | W:1 X:0 R:1 S:1 > >> 0xffffffff80000000 : 0xc00000 | W:0 X:1 R:1 S:1 > >> 0xffffffff80c00000 : 0xa00000 | W:1 X:0 R:1 S:1 > >> > >> ``` > >> > >> 3. ------ > >> > >> You can also reproduce similar panic by appending "memtest" in kernel cmdline. > >> I have posted the memtest boot log at this link: https://termbin.com/1twl. > >> > >> Please correct me if I'm wrong. > >> > >> [1]: https://lore.kernel.org/linux-riscv/CAAYs2=gQvkhTeioMmqRDVGjdtNF_vhB+vm_1dHJxPNi75YDQ_Q@mail.gmail.com/ > >> [2]: https://lore.kernel.org/linux-riscv/20230526-astride-detonator-9ae120051159@wendy/ > >> [3]: https://github.com/martinradev/gdb-pt-dump > > > > Thanks for the thorough report, really appreciated. > > > > So there are multiple issues here: > > > > - the first one is that the memory region for opensbi is marked as not > > cacheable in the efi memory map, and then this region is not mapped in > > the linear mapping: > > [ 0.000000] efi: 0x000080000000-0x00008003ffff [Reserved | | > > | | | | | | | | | | | |UC] > > > > - the second one (that I feel a bit ashamed of...) is that I did not > > check the alignment of the virtual address when choosing the map size > > in best_map_size() and then we end up trying to map a physical region > > aligned on 2MB that is actually not aligned on 2MB virtually because > > the opensbi region is not mapped at all. > > > The issue 2 should be the root cause of this panic. > > Here is my understanding of the necessity of the 2M-aligned VA for > linear PMD mapping. Please correct me if I'm wrong. > > I logged the `create_linear_mapping_range()` function. > > ``` > song # lowmem region: [0x0000000081800000 -- 0x00000000ffe3d000], va: > 0xff6000007fbc0000, pa: 0x00000000ffc00000, map_size: 200000 ,pg: e7 > song # lowmem region: [0x0000000081800000 -- 0x00000000ffe3d000], va: > 0xff6000007fdc0000, pa: 0x00000000ffe00000, map_size: 1000 ,pg: e7 > ``` > > The PA `0x00000000ffc00000` of this PMD mapping is aligned with PMD_SIZE > but VA `0xff6000007fbc0000` is not. > After the `pmd_index()`, this 2M PA region is actually mapping the > effective VA region `[0xff6000007fa00000,0xff6000007fc00000)`, > and any access of VA hole between the end of the effective VA region and > the start VA of the next 4K mapping (`0xff6000007fdc0000`) will fault. > > In this report, the memtest fault VA (`0xff6000007fc00000`) and the > booting fault VA (`ff6000007fdb1000`) lie right in the VA hole. > > When reverting the commit 3335068f8721, the kernel load address is > always offseted by PMD_SIZE, kernel_map.va_pa_offset and > MIN_MEMBLOCK_ADDR follow it. So the linear PMD mapping will always take > 2M-aligned VA. That's why this reverting works. Yes, I agree with you, this is quite an oversight on my side! > > > - the possible third one is that we should not map the linear mapping > > using 4K pages, this would be slow in my opinion, and I think we > > should waste a bit of memory to align va and pa on a 2MB boundary. > I also noticed this one. > > > > So I'll fix the second issue, and possibly the third one, and if no > Thanks for your attention to this report, looking for your fixup. It's running our internal CI, I hope to send it this afternoon or tomorrow morning so that it can be discussed during the patchwork meeting. Thanks again, Alex > > one looks into why the opensbi region is mapped in UC, I'll take a > > look at edk2. > > > > Sorry for that, > > > > Alex > > > > -- > Song Shuai > Thanks > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2023-06-06 18:22 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-06-05 10:52 Bug report: kernel paniced while booting Song Shuai 2023-06-05 10:58 ` Conor Dooley 2023-06-05 14:25 ` Alexandre Ghiti 2023-06-05 15:12 ` Sunil V L 2023-06-05 20:55 ` Atish Patra 2023-06-06 0:48 ` Icenowy Zheng 2023-06-06 2:29 ` Jessica Clarke 2023-06-06 7:20 ` Alexandre Ghiti 2023-06-06 9:01 ` Sunil V L 2023-06-06 11:50 ` Alexandre Ghiti 2023-06-05 21:42 ` Jessica Clarke 2023-06-06 6:40 ` Sunil V L 2023-06-06 7:25 ` Alexandre Ghiti 2023-06-06 18:21 ` Atish Patra 2023-06-06 10:04 ` Song Shuai 2023-06-06 11:46 ` Alexandre Ghiti
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).