[bug report] xen/arm64: singlestep doesn't work on Dom0

[bug report] xen/arm64: singlestep doesn't work on Dom0

[Masami Hiramatsu]

Hello,

When I tested the kprobes on Dom0 kernel, it caused a kernel panic.

Here is an example, to clarify the bug is in the single-stepping
(software-step exception), I added a printk() where the kprobes setup
single-stepping.

root@develbox:~# cd /sys/kernel/debug/tracing/
root@develbox:/sys/kernel/debug/tracing# echo p vfs_read > kprobe_events
root@develbox:/sys/kernel/debug/tracing# echo 1 > events/kprobes/enable
root@develbox:/sys/kernel/debug/tracing# [  112.282480] kprobes:
singlestep ool insn at ffff800011785000    <--- This shows single
stepping buffer (trampoline)
[  112.288077] ------------[ cut here ]------------
[  112.292745] kernel BUG at arch/arm64/kernel/traps.c:406!
[  112.298129] Internal error: Oops - BUG: 0 [#1] SMP
[  112.302987] Modules linked in: fuse bridge stp llc binfmt_misc
nls_ascii nls_cp437 vfat fat ahci libahci libata hid_generic udlfb
scsi_mod aes_ce_blk crypto_simd evdev cryptd aes_ce_cipher usbhid
ghash_ce realtek gf128mul hid netsec sha2_ce mdio_devres i2c_algo_bit
of_mdio sha256_arm64 fb_ddc sha1_ce fixed_phy gpio_keys leds_gpio
libphy bpf_preload ip_tables x_tables autofs4 xhci_pci
xhci_pci_renesas xhci_hcd usbcore gpio_mb86s7x
[  112.341097] CPU: 13 PID: 1045 Comm: bash Not tainted 5.10.0-rc1+ #44
[  112.347515] Hardware name: Socionext Developer Box (DT)
[  112.352813] pstate: 00000085 (nzcv daIf -PAN -UAO -TCO BTYPE=--)
[  112.358897] pc : do_undefinstr+0x354/0x378
[  112.363053] lr : do_undefinstr+0x270/0x378
[  112.367218] sp : ffff8000122fbc50
[  112.370603] x29: ffff8000122fbc50 x28: ffff00084bc9e080
[  112.375985] x27: 0000000000000000 x26: 0000000000000000
[  112.381366] x25: 0000000000000000 x24: 0000000000000000
[  112.386748] x23: 0000000080000085 x22: ffff800011785004
[  112.392129] x21: ffff8000122fbe00 x20: ffff8000122fbcc0
[  112.397511] x19: ffff800011249988 x18: 0000000000000000
[  112.402892] x17: 0000000000000000 x16: 0000000000000000
[  112.408274] x15: 0000000000000000 x14: 0000000000000000
[  112.413655] x13: 0000000000000000 x12: 0000000000000000
[  112.419037] x11: 0000000000000000 x10: 0000000000000000
[  112.424426] x9 : ffff800010314614 x8 : 0000000000000000
[  112.429801] x7 : 0000000000000000 x6 : ffff8000122fbca8
[  112.435189] x5 : 0000000000000000 x4 : ffff800011400110
[  112.440564] x3 : 00000000d5300000 x2 : ffff800011255f78
[  112.445946] x1 : ffff800011400110 x0 : 0000000080000085
[  112.451328] Call trace:
[  112.453848]  do_undefinstr+0x354/0x378
[  112.457669]  el1_sync_handler+0xa8/0x138
[  112.461658]  el1_sync+0x7c/0x100
[  112.464958]  0xffff800011785004     /// <- Undefined instruction
error happens on the next instruction of single stepping buffer.
[  112.468172]  __arm64_sys_read+0x24/0x30
[  112.472078]  el0_svc_common.constprop.3+0x94/0x178
[  112.476936]  do_el0_svc+0x2c/0x98
[  112.480321]  el0_sync_handler+0x118/0x168
[  112.484407]  el0_sync+0x158/0x180
[  112.487789] Code: d2801400 17ffffbe a9025bf5 f9001bf7 (d4210000)
[  112.493951] ---[ end trace 3564a3bf75d1618c ]---

So, this seems that the Linux kernel couldn't catch the software-step exception.

I confirmed the same kernel doesn't cause this error without Xen. I
guess Xen is not correctly setting the debug registers when the cpu
goes EL1.
(Or, would we handle debug exceptions in Xen and transfer it to EL1
OS? I'm not sure how it was designed)

Thank you,

-- 
Masami Hiramatsu

Re: [bug report] xen/arm64: singlestep doesn't work on Dom0

[Alex Bennée]

Masami Hiramatsu <masami.hiramatsu@linaro.org> writes:

> Hello,
>
> When I tested the kprobes on Dom0 kernel, it caused a kernel panic.
>
> Here is an example, to clarify the bug is in the single-stepping
> (software-step exception), I added a printk() where the kprobes setup
> single-stepping.
>
> root@develbox:~# cd /sys/kernel/debug/tracing/
> root@develbox:/sys/kernel/debug/tracing# echo p vfs_read > kprobe_events
> root@develbox:/sys/kernel/debug/tracing# echo 1 > events/kprobes/enable
> root@develbox:/sys/kernel/debug/tracing# [  112.282480] kprobes:
> singlestep ool insn at ffff800011785000    <--- This shows single
> stepping buffer (trampoline)
> [  112.288077] ------------[ cut here ]------------
> [  112.292745] kernel BUG at arch/arm64/kernel/traps.c:406!
> [  112.298129] Internal error: Oops - BUG: 0 [#1] SMP
> [  112.302987] Modules linked in: fuse bridge stp llc binfmt_misc
> nls_ascii nls_cp437 vfat fat ahci libahci libata hid_generic udlfb
> scsi_mod aes_ce_blk crypto_simd evdev cryptd aes_ce_cipher usbhid
> ghash_ce realtek gf128mul hid netsec sha2_ce mdio_devres i2c_algo_bit
> of_mdio sha256_arm64 fb_ddc sha1_ce fixed_phy gpio_keys leds_gpio
> libphy bpf_preload ip_tables x_tables autofs4 xhci_pci
> xhci_pci_renesas xhci_hcd usbcore gpio_mb86s7x
> [  112.341097] CPU: 13 PID: 1045 Comm: bash Not tainted 5.10.0-rc1+ #44
> [  112.347515] Hardware name: Socionext Developer Box (DT)
> [  112.352813] pstate: 00000085 (nzcv daIf -PAN -UAO -TCO BTYPE=--)
> [  112.358897] pc : do_undefinstr+0x354/0x378
> [  112.363053] lr : do_undefinstr+0x270/0x378
> [  112.367218] sp : ffff8000122fbc50
> [  112.370603] x29: ffff8000122fbc50 x28: ffff00084bc9e080
> [  112.375985] x27: 0000000000000000 x26: 0000000000000000
> [  112.381366] x25: 0000000000000000 x24: 0000000000000000
> [  112.386748] x23: 0000000080000085 x22: ffff800011785004
> [  112.392129] x21: ffff8000122fbe00 x20: ffff8000122fbcc0
> [  112.397511] x19: ffff800011249988 x18: 0000000000000000
> [  112.402892] x17: 0000000000000000 x16: 0000000000000000
> [  112.408274] x15: 0000000000000000 x14: 0000000000000000
> [  112.413655] x13: 0000000000000000 x12: 0000000000000000
> [  112.419037] x11: 0000000000000000 x10: 0000000000000000
> [  112.424426] x9 : ffff800010314614 x8 : 0000000000000000
> [  112.429801] x7 : 0000000000000000 x6 : ffff8000122fbca8
> [  112.435189] x5 : 0000000000000000 x4 : ffff800011400110
> [  112.440564] x3 : 00000000d5300000 x2 : ffff800011255f78
> [  112.445946] x1 : ffff800011400110 x0 : 0000000080000085
> [  112.451328] Call trace:
> [  112.453848]  do_undefinstr+0x354/0x378
> [  112.457669]  el1_sync_handler+0xa8/0x138
> [  112.461658]  el1_sync+0x7c/0x100
> [  112.464958]  0xffff800011785004     /// <- Undefined instruction
> error happens on the next instruction of single stepping buffer.
> [  112.468172]  __arm64_sys_read+0x24/0x30
> [  112.472078]  el0_svc_common.constprop.3+0x94/0x178
> [  112.476936]  do_el0_svc+0x2c/0x98
> [  112.480321]  el0_sync_handler+0x118/0x168
> [  112.484407]  el0_sync+0x158/0x180
> [  112.487789] Code: d2801400 17ffffbe a9025bf5 f9001bf7 (d4210000)
> [  112.493951] ---[ end trace 3564a3bf75d1618c ]---
>
> So, this seems that the Linux kernel couldn't catch the software-step exception.
>
> I confirmed the same kernel doesn't cause this error without Xen. I
> guess Xen is not correctly setting the debug registers when the cpu
> goes EL1.

Having worked on the arm64 KVM debug logic I have a little familiarity
with how it works on KVM. 

> (Or, would we handle debug exceptions in Xen and transfer it to EL1
> OS? I'm not sure how it was designed)

Xen looks as though it should be trapping a chunk of debug accesses:

    /* Trap Debug and Performance Monitor accesses */
    WRITE_SYSREG(HDCR_TDRA|HDCR_TDOSA|HDCR_TDA|HDCR_TPM|HDCR_TPMCR,
                 MDCR_EL2);

but it doesn't set HDCR_TDE so it won't re-route debug exceptions to EL2
which should be the place that is dealing with them. Also I can't see
where the debug registers are saved/restored. In KVM we maintain a
shadow copy of the guest debug register state while guest debugging is
in effect as any breakpoints you want to trigger need to be copied
across.

I also can't see where CPSR_SS or DBG_MDSCR_SS is set in the mdscr.

FWIW most of the logic in KVM is contained in:

  arch/arm64/kvm/debug.c

with some smatterings of handling the traps and context swapping
elsewhere in the code.

>
> Thank you,


-- 
Alex Bennée

Re: [bug report] xen/arm64: singlestep doesn't work on Dom0

[Julien Grall]

On 28/10/2020 08:50, Masami Hiramatsu wrote:
> Hello,

Hi,

> When I tested the kprobes on Dom0 kernel, it caused a kernel panic.
> 
> Here is an example, to clarify the bug is in the single-stepping
> (software-step exception), I added a printk() where the kprobes setup
> single-stepping.
> 
> root@develbox:~# cd /sys/kernel/debug/tracing/
> root@develbox:/sys/kernel/debug/tracing# echo p vfs_read > kprobe_events
> root@develbox:/sys/kernel/debug/tracing# echo 1 > events/kprobes/enable
> root@develbox:/sys/kernel/debug/tracing# [  112.282480] kprobes:
> singlestep ool insn at ffff800011785000    <--- This shows single
> stepping buffer (trampoline)
> [  112.288077] ------------[ cut here ]------------
> [  112.292745] kernel BUG at arch/arm64/kernel/traps.c:406!
> [  112.298129] Internal error: Oops - BUG: 0 [#1] SMP
> [  112.302987] Modules linked in: fuse bridge stp llc binfmt_misc
> nls_ascii nls_cp437 vfat fat ahci libahci libata hid_generic udlfb
> scsi_mod aes_ce_blk crypto_simd evdev cryptd aes_ce_cipher usbhid
> ghash_ce realtek gf128mul hid netsec sha2_ce mdio_devres i2c_algo_bit
> of_mdio sha256_arm64 fb_ddc sha1_ce fixed_phy gpio_keys leds_gpio
> libphy bpf_preload ip_tables x_tables autofs4 xhci_pci
> xhci_pci_renesas xhci_hcd usbcore gpio_mb86s7x
> [  112.341097] CPU: 13 PID: 1045 Comm: bash Not tainted 5.10.0-rc1+ #44
> [  112.347515] Hardware name: Socionext Developer Box (DT)
> [  112.352813] pstate: 00000085 (nzcv daIf -PAN -UAO -TCO BTYPE=--)
> [  112.358897] pc : do_undefinstr+0x354/0x378
> [  112.363053] lr : do_undefinstr+0x270/0x378
> [  112.367218] sp : ffff8000122fbc50
> [  112.370603] x29: ffff8000122fbc50 x28: ffff00084bc9e080
> [  112.375985] x27: 0000000000000000 x26: 0000000000000000
> [  112.381366] x25: 0000000000000000 x24: 0000000000000000
> [  112.386748] x23: 0000000080000085 x22: ffff800011785004
> [  112.392129] x21: ffff8000122fbe00 x20: ffff8000122fbcc0
> [  112.397511] x19: ffff800011249988 x18: 0000000000000000
> [  112.402892] x17: 0000000000000000 x16: 0000000000000000
> [  112.408274] x15: 0000000000000000 x14: 0000000000000000
> [  112.413655] x13: 0000000000000000 x12: 0000000000000000
> [  112.419037] x11: 0000000000000000 x10: 0000000000000000
> [  112.424426] x9 : ffff800010314614 x8 : 0000000000000000
> [  112.429801] x7 : 0000000000000000 x6 : ffff8000122fbca8
> [  112.435189] x5 : 0000000000000000 x4 : ffff800011400110
> [  112.440564] x3 : 00000000d5300000 x2 : ffff800011255f78
> [  112.445946] x1 : ffff800011400110 x0 : 0000000080000085
> [  112.451328] Call trace:
> [  112.453848]  do_undefinstr+0x354/0x378
> [  112.457669]  el1_sync_handler+0xa8/0x138
> [  112.461658]  el1_sync+0x7c/0x100
> [  112.464958]  0xffff800011785004     /// <- Undefined instruction
> error happens on the next instruction of single stepping buffer.
> [  112.468172]  __arm64_sys_read+0x24/0x30
> [  112.472078]  el0_svc_common.constprop.3+0x94/0x178
> [  112.476936]  do_el0_svc+0x2c/0x98
> [  112.480321]  el0_sync_handler+0x118/0x168
> [  112.484407]  el0_sync+0x158/0x180
> [  112.487789] Code: d2801400 17ffffbe a9025bf5 f9001bf7 (d4210000)
> [  112.493951] ---[ end trace 3564a3bf75d1618c ]---
> 
> So, this seems that the Linux kernel couldn't catch the software-step exception.
> 
> I confirmed the same kernel doesn't cause this error without Xen. I
> guess Xen is not correctly setting the debug registers when the cpu
> goes EL1.
> (Or, would we handle debug exceptions in Xen and transfer it to EL1
> OS? I'm not sure how it was designed)
Debug registers (including single-step) is not supported by Xen today. I 
vaguely remember that they are quite tricky to implement it.

Although, it would be nice to have it supported!

Cheers,

-- 
Julien Grall

Re: [bug report] xen/arm64: singlestep doesn't work on Dom0

[Masami Hiramatsu]

Hi Alex,

2020年10月28日(水) 21:03 Alex Bennée <alex.bennee@linaro.org>:
>
>
> Masami Hiramatsu <masami.hiramatsu@linaro.org> writes:
>
> > Hello,
> >
> > When I tested the kprobes on Dom0 kernel, it caused a kernel panic.
> >
> > Here is an example, to clarify the bug is in the single-stepping
> > (software-step exception), I added a printk() where the kprobes setup
> > single-stepping.
> >
> > root@develbox:~# cd /sys/kernel/debug/tracing/
> > root@develbox:/sys/kernel/debug/tracing# echo p vfs_read > kprobe_events
> > root@develbox:/sys/kernel/debug/tracing# echo 1 > events/kprobes/enable
> > root@develbox:/sys/kernel/debug/tracing# [  112.282480] kprobes:
> > singlestep ool insn at ffff800011785000    <--- This shows single
> > stepping buffer (trampoline)
> > [  112.288077] ------------[ cut here ]------------
> > [  112.292745] kernel BUG at arch/arm64/kernel/traps.c:406!
> > [  112.298129] Internal error: Oops - BUG: 0 [#1] SMP
> > [  112.302987] Modules linked in: fuse bridge stp llc binfmt_misc
> > nls_ascii nls_cp437 vfat fat ahci libahci libata hid_generic udlfb
> > scsi_mod aes_ce_blk crypto_simd evdev cryptd aes_ce_cipher usbhid
> > ghash_ce realtek gf128mul hid netsec sha2_ce mdio_devres i2c_algo_bit
> > of_mdio sha256_arm64 fb_ddc sha1_ce fixed_phy gpio_keys leds_gpio
> > libphy bpf_preload ip_tables x_tables autofs4 xhci_pci
> > xhci_pci_renesas xhci_hcd usbcore gpio_mb86s7x
> > [  112.341097] CPU: 13 PID: 1045 Comm: bash Not tainted 5.10.0-rc1+ #44
> > [  112.347515] Hardware name: Socionext Developer Box (DT)
> > [  112.352813] pstate: 00000085 (nzcv daIf -PAN -UAO -TCO BTYPE=--)
> > [  112.358897] pc : do_undefinstr+0x354/0x378
> > [  112.363053] lr : do_undefinstr+0x270/0x378
> > [  112.367218] sp : ffff8000122fbc50
> > [  112.370603] x29: ffff8000122fbc50 x28: ffff00084bc9e080
> > [  112.375985] x27: 0000000000000000 x26: 0000000000000000
> > [  112.381366] x25: 0000000000000000 x24: 0000000000000000
> > [  112.386748] x23: 0000000080000085 x22: ffff800011785004
> > [  112.392129] x21: ffff8000122fbe00 x20: ffff8000122fbcc0
> > [  112.397511] x19: ffff800011249988 x18: 0000000000000000
> > [  112.402892] x17: 0000000000000000 x16: 0000000000000000
> > [  112.408274] x15: 0000000000000000 x14: 0000000000000000
> > [  112.413655] x13: 0000000000000000 x12: 0000000000000000
> > [  112.419037] x11: 0000000000000000 x10: 0000000000000000
> > [  112.424426] x9 : ffff800010314614 x8 : 0000000000000000
> > [  112.429801] x7 : 0000000000000000 x6 : ffff8000122fbca8
> > [  112.435189] x5 : 0000000000000000 x4 : ffff800011400110
> > [  112.440564] x3 : 00000000d5300000 x2 : ffff800011255f78
> > [  112.445946] x1 : ffff800011400110 x0 : 0000000080000085
> > [  112.451328] Call trace:
> > [  112.453848]  do_undefinstr+0x354/0x378
> > [  112.457669]  el1_sync_handler+0xa8/0x138
> > [  112.461658]  el1_sync+0x7c/0x100
> > [  112.464958]  0xffff800011785004     /// <- Undefined instruction
> > error happens on the next instruction of single stepping buffer.
> > [  112.468172]  __arm64_sys_read+0x24/0x30
> > [  112.472078]  el0_svc_common.constprop.3+0x94/0x178
> > [  112.476936]  do_el0_svc+0x2c/0x98
> > [  112.480321]  el0_sync_handler+0x118/0x168
> > [  112.484407]  el0_sync+0x158/0x180
> > [  112.487789] Code: d2801400 17ffffbe a9025bf5 f9001bf7 (d4210000)
> > [  112.493951] ---[ end trace 3564a3bf75d1618c ]---
> >
> > So, this seems that the Linux kernel couldn't catch the software-step exception.
> >
> > I confirmed the same kernel doesn't cause this error without Xen. I
> > guess Xen is not correctly setting the debug registers when the cpu
> > goes EL1.
>
> Having worked on the arm64 KVM debug logic I have a little familiarity
> with how it works on KVM.
>
> > (Or, would we handle debug exceptions in Xen and transfer it to EL1
> > OS? I'm not sure how it was designed)
>
> Xen looks as though it should be trapping a chunk of debug accesses:
>
>     /* Trap Debug and Performance Monitor accesses */
>     WRITE_SYSREG(HDCR_TDRA|HDCR_TDOSA|HDCR_TDA|HDCR_TPM|HDCR_TPMCR,
>                  MDCR_EL2);
>
> but it doesn't set HDCR_TDE so it won't re-route debug exceptions to EL2
> which should be the place that is dealing with them.

Would we need to re-route the debug exception to EL2? I thought it was
OK to send it to EL1 (Dom0 kernel).

> Also I can't see
> where the debug registers are saved/restored. In KVM we maintain a
> shadow copy of the guest debug register state while guest debugging is
> in effect as any breakpoints you want to trigger need to be copied
> across.

Hmm, so KVM will trap the access of the debug registers in EL1?


> I also can't see where CPSR_SS or DBG_MDSCR_SS is set in the mdscr.
>
> FWIW most of the logic in KVM is contained in:
>
>   arch/arm64/kvm/debug.c
>
> with some smatterings of handling the traps and context swapping
> elsewhere in the code.

Thank you,

-- 
Masami Hiramatsu