QEMU+KVM on RISC-V + Hypervisor Extension

QEMU+KVM on RISC-V + Hypervisor Extension

[Ralf Ramsauer]

Hi,

I'm trying to run Linux/QEMU+KVM inside an emulated
qemu-system-riscv64 VM (x86 host). On latest&greatest QEMU (1416688c53), 
I run Linux inside QEMU. On host side:

qemu-system-riscv64 -nographic \
         -machine virt \
         -cpu 'rv64,h=true' \
         -smp 8 -m 8G \
         -bios $latest_opensbi \
         -kernel $latest_upstream_linux \
	[…snip…]

Inside that machine, I boot Linux with KVM enabled:

[…]
[    0.228939] kvm [1]: hypervisor extension available
[    0.229000] kvm [1]: using Sv48x4 G-stage page table format
[    0.229033] kvm [1]: VMID 14 bits available
[…]

KVM seems to load correctly. Inside that machine, I compiled QEMU from 
the same sources with the KVM accelerator enabled. When I try to start 
QEMU with KVM enabled, I get

./qemu-system-riscv64 -nographic \
	-monitor none \
	-machine virt \
	-smp 1 -m 1G \
	-bios ./pc-bios/opensbi-riscv64-generic-fw_dynamic.bin \
	-serial stdio \
	-enable-kvm

[ 4860.559194] kvm [9942]: VCPU exit error -95
[ 4860.584262] kvm [9942]: SEPC=0x0 SSTATUS=0x200004120 HSTATUS=0x2002001c0
[ 4860.586839] kvm [9942]: SCAUSE=0x14 STVAL=0x0 HTVAL=0x0 HTINST=0x0

on the ringbuffer, together with a register dump of qemu [1] on the 
console. Needless to say, but without -enable-kvm, it works fine.

As far as I see that, SCAUSE=0x14 reports a 'Instruction Guest PF', 
which would be kind of correct, if the guest's PC really tries to 
execute at phys 0x0. DRAM of the 'virt' machine definition should start 
at 0x80000000, where OpenSBI resides. So I wonder if an erroneous reset 
PC might be the culprit…

Before digging deeper into that issue, I wanted to ask if Qemu/KVM 
inside an emulated riscv64+H-extension is actually supported, or if this 
is a known bug and has some ongoing work.

Thanks
   Ralf

[1]

  pc       0000000000000000
  mhartid  0000000000000000
  mstatus  0000000200000000
  mip      0000000000000000
  mie      0000000000000000
  mideleg  0000000000000000
  medeleg  0000000000000000
  mtvec    0000000000000000
  stvec    0000000000000000
  mepc     0000000000000000
  sepc     0000000000000000
  mcause   0000000000000000
  scause   0000000000000000
  mtval    0000000000000000
  stval    0000000000000000
  mscratch 0000000000000000
  sscratch 0000000000000000
  satp     0000000000000000
  x0/zero  0000000000000000 x1/ra    0000000000000000 x2/sp 
0000000000000000 x3/gp    0000000000000000
  x4/tp    0000000000000000 x5/t0    0000000000000000 x6/t1 
0000000000000000 x7/t2    0000000000000000
  x8/s0    0000000000000000 x9/s1    0000000000000000 x10/a0 
0000000000000000 x11/a1   00000000bf000000
  x12/a2   0000000000000000 x13/a3   0000000000000000 x14/a4 
0000000000000000 x15/a5   0000000000000000
  x16/a6   0000000000000000 x17/a7   0000000000000000 x18/s2 
0000000000000000 x19/s3   0000000000000000
  x20/s4   0000000000000000 x21/s5   0000000000000000 x22/s6 
0000000000000000 x23/s7   0000000000000000
  x24/s8   0000000000000000 x25/s9   0000000000000000 x26/s10 
0000000000000000 x27/s11  0000000000000000
  x28/t3   0000000000000000 x29/t4   0000000000000000 x30/t5 
0000000000000000 x31/t6   0000000000000000

Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Alistair Francis]

On Sun, Mar 13, 2022 at 12:12 PM Ralf Ramsauer
<ralf.ramsauer@oth-regensburg.de> wrote:
>
> Hi,
>
> I'm trying to run Linux/QEMU+KVM inside an emulated
> qemu-system-riscv64 VM (x86 host). On latest&greatest QEMU (1416688c53),
> I run Linux inside QEMU. On host side:
>
> qemu-system-riscv64 -nographic \
>          -machine virt \
>          -cpu 'rv64,h=true' \
>          -smp 8 -m 8G \
>          -bios $latest_opensbi \
>          -kernel $latest_upstream_linux \
>         […snip…]
>
> Inside that machine, I boot Linux with KVM enabled:
>
> […]
> [    0.228939] kvm [1]: hypervisor extension available
> [    0.229000] kvm [1]: using Sv48x4 G-stage page table format
> [    0.229033] kvm [1]: VMID 14 bits available
> […]
>
> KVM seems to load correctly. Inside that machine, I compiled QEMU from
> the same sources with the KVM accelerator enabled. When I try to start
> QEMU with KVM enabled, I get
>
> ./qemu-system-riscv64 -nographic \
>         -monitor none \
>         -machine virt \
>         -smp 1 -m 1G \
>         -bios ./pc-bios/opensbi-riscv64-generic-fw_dynamic.bin \
>         -serial stdio \
>         -enable-kvm
>
> [ 4860.559194] kvm [9942]: VCPU exit error -95
> [ 4860.584262] kvm [9942]: SEPC=0x0 SSTATUS=0x200004120 HSTATUS=0x2002001c0
> [ 4860.586839] kvm [9942]: SCAUSE=0x14 STVAL=0x0 HTVAL=0x0 HTINST=0x0
>
> on the ringbuffer, together with a register dump of qemu [1] on the
> console. Needless to say, but without -enable-kvm, it works fine.
>
> As far as I see that, SCAUSE=0x14 reports a 'Instruction Guest PF',
> which would be kind of correct, if the guest's PC really tries to
> execute at phys 0x0. DRAM of the 'virt' machine definition should start
> at 0x80000000, where OpenSBI resides. So I wonder if an erroneous reset
> PC might be the culprit…
>
> Before digging deeper into that issue, I wanted to ask if Qemu/KVM
> inside an emulated riscv64+H-extension is actually supported, or if this
> is a known bug and has some ongoing work.

RISC-V KVM should work. I haven't had a chance to try it myself though.

I have CCed two people who hopefully can help.

Alistair

>
> Thanks
>    Ralf
>
> [1]
>
>   pc       0000000000000000
>   mhartid  0000000000000000
>   mstatus  0000000200000000
>   mip      0000000000000000
>   mie      0000000000000000
>   mideleg  0000000000000000
>   medeleg  0000000000000000
>   mtvec    0000000000000000
>   stvec    0000000000000000
>   mepc     0000000000000000
>   sepc     0000000000000000
>   mcause   0000000000000000
>   scause   0000000000000000
>   mtval    0000000000000000
>   stval    0000000000000000
>   mscratch 0000000000000000
>   sscratch 0000000000000000
>   satp     0000000000000000
>   x0/zero  0000000000000000 x1/ra    0000000000000000 x2/sp
> 0000000000000000 x3/gp    0000000000000000
>   x4/tp    0000000000000000 x5/t0    0000000000000000 x6/t1
> 0000000000000000 x7/t2    0000000000000000
>   x8/s0    0000000000000000 x9/s1    0000000000000000 x10/a0
> 0000000000000000 x11/a1   00000000bf000000
>   x12/a2   0000000000000000 x13/a3   0000000000000000 x14/a4
> 0000000000000000 x15/a5   0000000000000000
>   x16/a6   0000000000000000 x17/a7   0000000000000000 x18/s2
> 0000000000000000 x19/s3   0000000000000000
>   x20/s4   0000000000000000 x21/s5   0000000000000000 x22/s6
> 0000000000000000 x23/s7   0000000000000000
>   x24/s8   0000000000000000 x25/s9   0000000000000000 x26/s10
> 0000000000000000 x27/s11  0000000000000000
>   x28/t3   0000000000000000 x29/t4   0000000000000000 x30/t5
> 0000000000000000 x31/t6   0000000000000000
>

Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Anup Patel]

On Tue, Mar 15, 2022 at 12:18 PM Alistair Francis <alistair23@gmail.com> wrote:
>
> On Sun, Mar 13, 2022 at 12:12 PM Ralf Ramsauer
> <ralf.ramsauer@oth-regensburg.de> wrote:
> >
> > Hi,
> >
> > I'm trying to run Linux/QEMU+KVM inside an emulated
> > qemu-system-riscv64 VM (x86 host). On latest&greatest QEMU (1416688c53),
> > I run Linux inside QEMU. On host side:
> >
> > qemu-system-riscv64 -nographic \
> >          -machine virt \
> >          -cpu 'rv64,h=true' \
> >          -smp 8 -m 8G \
> >          -bios $latest_opensbi \
> >          -kernel $latest_upstream_linux \
> >         […snip…]
> >
> > Inside that machine, I boot Linux with KVM enabled:
> >
> > […]
> > [    0.228939] kvm [1]: hypervisor extension available
> > [    0.229000] kvm [1]: using Sv48x4 G-stage page table format
> > [    0.229033] kvm [1]: VMID 14 bits available
> > […]
> >
> > KVM seems to load correctly. Inside that machine, I compiled QEMU from
> > the same sources with the KVM accelerator enabled. When I try to start
> > QEMU with KVM enabled, I get
> >
> > ./qemu-system-riscv64 -nographic \
> >         -monitor none \
> >         -machine virt \
> >         -smp 1 -m 1G \
> >         -bios ./pc-bios/opensbi-riscv64-generic-fw_dynamic.bin \
> >         -serial stdio \
> >         -enable-kvm

The QEMU RISC-V KVM support is quite new so I haven't got a chance
to play with it.

Nevertheless, we should not pass any firmware when using KVM
so can you try "-bios none" ?

Regards,
Anup

> >
> > [ 4860.559194] kvm [9942]: VCPU exit error -95
> > [ 4860.584262] kvm [9942]: SEPC=0x0 SSTATUS=0x200004120 HSTATUS=0x2002001c0
> > [ 4860.586839] kvm [9942]: SCAUSE=0x14 STVAL=0x0 HTVAL=0x0 HTINST=0x0
> >
> > on the ringbuffer, together with a register dump of qemu [1] on the
> > console. Needless to say, but without -enable-kvm, it works fine.
> >
> > As far as I see that, SCAUSE=0x14 reports a 'Instruction Guest PF',
> > which would be kind of correct, if the guest's PC really tries to
> > execute at phys 0x0. DRAM of the 'virt' machine definition should start
> > at 0x80000000, where OpenSBI resides. So I wonder if an erroneous reset
> > PC might be the culprit…
> >
> > Before digging deeper into that issue, I wanted to ask if Qemu/KVM
> > inside an emulated riscv64+H-extension is actually supported, or if this
> > is a known bug and has some ongoing work.
>
> RISC-V KVM should work. I haven't had a chance to try it myself though.
>
> I have CCed two people who hopefully can help.
>
> Alistair
>
> >
> > Thanks
> >    Ralf
> >
> > [1]
> >
> >   pc       0000000000000000
> >   mhartid  0000000000000000
> >   mstatus  0000000200000000
> >   mip      0000000000000000
> >   mie      0000000000000000
> >   mideleg  0000000000000000
> >   medeleg  0000000000000000
> >   mtvec    0000000000000000
> >   stvec    0000000000000000
> >   mepc     0000000000000000
> >   sepc     0000000000000000
> >   mcause   0000000000000000
> >   scause   0000000000000000
> >   mtval    0000000000000000
> >   stval    0000000000000000
> >   mscratch 0000000000000000
> >   sscratch 0000000000000000
> >   satp     0000000000000000
> >   x0/zero  0000000000000000 x1/ra    0000000000000000 x2/sp
> > 0000000000000000 x3/gp    0000000000000000
> >   x4/tp    0000000000000000 x5/t0    0000000000000000 x6/t1
> > 0000000000000000 x7/t2    0000000000000000
> >   x8/s0    0000000000000000 x9/s1    0000000000000000 x10/a0
> > 0000000000000000 x11/a1   00000000bf000000
> >   x12/a2   0000000000000000 x13/a3   0000000000000000 x14/a4
> > 0000000000000000 x15/a5   0000000000000000
> >   x16/a6   0000000000000000 x17/a7   0000000000000000 x18/s2
> > 0000000000000000 x19/s3   0000000000000000
> >   x20/s4   0000000000000000 x21/s5   0000000000000000 x22/s6
> > 0000000000000000 x23/s7   0000000000000000
> >   x24/s8   0000000000000000 x25/s9   0000000000000000 x26/s10
> > 0000000000000000 x27/s11  0000000000000000
> >   x28/t3   0000000000000000 x29/t4   0000000000000000 x30/t5
> > 0000000000000000 x31/t6   0000000000000000
> >

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Ralf Ramsauer]

On 15/03/2022 09:33, Anup Patel wrote:
> On Tue, Mar 15, 2022 at 12:18 PM Alistair Francis <alistair23@gmail.com> wrote:
>>
>> On Sun, Mar 13, 2022 at 12:12 PM Ralf Ramsauer
>> <ralf.ramsauer@oth-regensburg.de> wrote:
>>>
>>> Hi,
>>>
>>> I'm trying to run Linux/QEMU+KVM inside an emulated
>>> qemu-system-riscv64 VM (x86 host). On latest&greatest QEMU (1416688c53),
>>> I run Linux inside QEMU. On host side:
>>>
>>> qemu-system-riscv64 -nographic \
>>>           -machine virt \
>>>           -cpu 'rv64,h=true' \
>>>           -smp 8 -m 8G \
>>>           -bios $latest_opensbi \
>>>           -kernel $latest_upstream_linux \
>>>          […snip…]
>>>
>>> Inside that machine, I boot Linux with KVM enabled:
>>>
>>> […]
>>> [    0.228939] kvm [1]: hypervisor extension available
>>> [    0.229000] kvm [1]: using Sv48x4 G-stage page table format
>>> [    0.229033] kvm [1]: VMID 14 bits available
>>> […]
>>>
>>> KVM seems to load correctly. Inside that machine, I compiled QEMU from
>>> the same sources with the KVM accelerator enabled. When I try to start
>>> QEMU with KVM enabled, I get
>>>
>>> ./qemu-system-riscv64 -nographic \
>>>          -monitor none \
>>>          -machine virt \
>>>          -smp 1 -m 1G \
>>>          -bios ./pc-bios/opensbi-riscv64-generic-fw_dynamic.bin \
>>>          -serial stdio \
>>>          -enable-kvm
> 
> The QEMU RISC-V KVM support is quite new so I haven't got a chance
> to play with it.

Btw, the kernel's KVM selftests seem to run.

> 
> Nevertheless, we should not pass any firmware when using KVM
> so can you try "-bios none" ?

I have no kernel specified in the guest, so I'd then expect to boot into 
an completely empty machine (besides the tiny startup stub in the MROM). 
However, I still get the exact same error as mentioned in my initial mail.

Again, without "-enable-kvm", everything works as expected (i.e., I 
don't see any output, because there's basically no guest code to execute).

Just out of curiosity: if you don't pass a bios to the machine, what 
would then the boot sequence be?

Thanks
   Ralf

> 
> Regards,
> Anup
> 
>>>
>>> [ 4860.559194] kvm [9942]: VCPU exit error -95
>>> [ 4860.584262] kvm [9942]: SEPC=0x0 SSTATUS=0x200004120 HSTATUS=0x2002001c0
>>> [ 4860.586839] kvm [9942]: SCAUSE=0x14 STVAL=0x0 HTVAL=0x0 HTINST=0x0
>>>
>>> on the ringbuffer, together with a register dump of qemu [1] on the
>>> console. Needless to say, but without -enable-kvm, it works fine.
>>>
>>> As far as I see that, SCAUSE=0x14 reports a 'Instruction Guest PF',
>>> which would be kind of correct, if the guest's PC really tries to
>>> execute at phys 0x0. DRAM of the 'virt' machine definition should start
>>> at 0x80000000, where OpenSBI resides. So I wonder if an erroneous reset
>>> PC might be the culprit…
>>>
>>> Before digging deeper into that issue, I wanted to ask if Qemu/KVM
>>> inside an emulated riscv64+H-extension is actually supported, or if this
>>> is a known bug and has some ongoing work.
>>
>> RISC-V KVM should work. I haven't had a chance to try it myself though.
>>
>> I have CCed two people who hopefully can help.
>>
>> Alistair
>>
>>>
>>> Thanks
>>>     Ralf
>>>
>>> [1]
>>>
>>>    pc       0000000000000000
>>>    mhartid  0000000000000000
>>>    mstatus  0000000200000000
>>>    mip      0000000000000000
>>>    mie      0000000000000000
>>>    mideleg  0000000000000000
>>>    medeleg  0000000000000000
>>>    mtvec    0000000000000000
>>>    stvec    0000000000000000
>>>    mepc     0000000000000000
>>>    sepc     0000000000000000
>>>    mcause   0000000000000000
>>>    scause   0000000000000000
>>>    mtval    0000000000000000
>>>    stval    0000000000000000
>>>    mscratch 0000000000000000
>>>    sscratch 0000000000000000
>>>    satp     0000000000000000
>>>    x0/zero  0000000000000000 x1/ra    0000000000000000 x2/sp
>>> 0000000000000000 x3/gp    0000000000000000
>>>    x4/tp    0000000000000000 x5/t0    0000000000000000 x6/t1
>>> 0000000000000000 x7/t2    0000000000000000
>>>    x8/s0    0000000000000000 x9/s1    0000000000000000 x10/a0
>>> 0000000000000000 x11/a1   00000000bf000000
>>>    x12/a2   0000000000000000 x13/a3   0000000000000000 x14/a4
>>> 0000000000000000 x15/a5   0000000000000000
>>>    x16/a6   0000000000000000 x17/a7   0000000000000000 x18/s2
>>> 0000000000000000 x19/s3   0000000000000000
>>>    x20/s4   0000000000000000 x21/s5   0000000000000000 x22/s6
>>> 0000000000000000 x23/s7   0000000000000000
>>>    x24/s8   0000000000000000 x25/s9   0000000000000000 x26/s10
>>> 0000000000000000 x27/s11  0000000000000000
>>>    x28/t3   0000000000000000 x29/t4   0000000000000000 x30/t5
>>> 0000000000000000 x31/t6   0000000000000000
>>>

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Anup Patel]

On Tue, Mar 15, 2022 at 5:47 PM Ralf Ramsauer
<ralf.ramsauer@oth-regensburg.de> wrote:
>
>
>
> On 15/03/2022 09:33, Anup Patel wrote:
> > On Tue, Mar 15, 2022 at 12:18 PM Alistair Francis <alistair23@gmail.com> wrote:
> >>
> >> On Sun, Mar 13, 2022 at 12:12 PM Ralf Ramsauer
> >> <ralf.ramsauer@oth-regensburg.de> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I'm trying to run Linux/QEMU+KVM inside an emulated
> >>> qemu-system-riscv64 VM (x86 host). On latest&greatest QEMU (1416688c53),
> >>> I run Linux inside QEMU. On host side:
> >>>
> >>> qemu-system-riscv64 -nographic \
> >>>           -machine virt \
> >>>           -cpu 'rv64,h=true' \
> >>>           -smp 8 -m 8G \
> >>>           -bios $latest_opensbi \
> >>>           -kernel $latest_upstream_linux \
> >>>          […snip…]
> >>>
> >>> Inside that machine, I boot Linux with KVM enabled:
> >>>
> >>> […]
> >>> [    0.228939] kvm [1]: hypervisor extension available
> >>> [    0.229000] kvm [1]: using Sv48x4 G-stage page table format
> >>> [    0.229033] kvm [1]: VMID 14 bits available
> >>> […]
> >>>
> >>> KVM seems to load correctly. Inside that machine, I compiled QEMU from
> >>> the same sources with the KVM accelerator enabled. When I try to start
> >>> QEMU with KVM enabled, I get
> >>>
> >>> ./qemu-system-riscv64 -nographic \
> >>>          -monitor none \
> >>>          -machine virt \
> >>>          -smp 1 -m 1G \
> >>>          -bios ./pc-bios/opensbi-riscv64-generic-fw_dynamic.bin \
> >>>          -serial stdio \
> >>>          -enable-kvm
> >
> > The QEMU RISC-V KVM support is quite new so I haven't got a chance
> > to play with it.
>
> Btw, the kernel's KVM selftests seem to run.
>
> >
> > Nevertheless, we should not pass any firmware when using KVM
> > so can you try "-bios none" ?
>
> I have no kernel specified in the guest, so I'd then expect to boot into
> an completely empty machine (besides the tiny startup stub in the MROM).
> However, I still get the exact same error as mentioned in my initial mail.

When KVM is enabled and the kernel is not provided, there is simply
nothing in Guest RAM (0x80000000) or ROM (0x00000000) which means
you are booting garbage.

>
> Again, without "-enable-kvm", everything works as expected (i.e., I
> don't see any output, because there's basically no guest code to execute).

Strange, but even in this case you might be running garbage.

>
> Just out of curiosity: if you don't pass a bios to the machine, what
> would then the boot sequence be?

For QEMU KVM:
1) There is no M-mode
2) Guest directly boots in S-mode
3) The SBI services are provided by in-kernel KVM module

You need to specify the Guest kernel using "-kernel" parameter when
using QEMU KVM.

Regards,
Anup

>
> Thanks
>    Ralf
>
> >
> > Regards,
> > Anup
> >
> >>>
> >>> [ 4860.559194] kvm [9942]: VCPU exit error -95
> >>> [ 4860.584262] kvm [9942]: SEPC=0x0 SSTATUS=0x200004120 HSTATUS=0x2002001c0
> >>> [ 4860.586839] kvm [9942]: SCAUSE=0x14 STVAL=0x0 HTVAL=0x0 HTINST=0x0
> >>>
> >>> on the ringbuffer, together with a register dump of qemu [1] on the
> >>> console. Needless to say, but without -enable-kvm, it works fine.
> >>>
> >>> As far as I see that, SCAUSE=0x14 reports a 'Instruction Guest PF',
> >>> which would be kind of correct, if the guest's PC really tries to
> >>> execute at phys 0x0. DRAM of the 'virt' machine definition should start
> >>> at 0x80000000, where OpenSBI resides. So I wonder if an erroneous reset
> >>> PC might be the culprit…
> >>>
> >>> Before digging deeper into that issue, I wanted to ask if Qemu/KVM
> >>> inside an emulated riscv64+H-extension is actually supported, or if this
> >>> is a known bug and has some ongoing work.
> >>
> >> RISC-V KVM should work. I haven't had a chance to try it myself though.
> >>
> >> I have CCed two people who hopefully can help.
> >>
> >> Alistair
> >>
> >>>
> >>> Thanks
> >>>     Ralf
> >>>
> >>> [1]
> >>>
> >>>    pc       0000000000000000
> >>>    mhartid  0000000000000000
> >>>    mstatus  0000000200000000
> >>>    mip      0000000000000000
> >>>    mie      0000000000000000
> >>>    mideleg  0000000000000000
> >>>    medeleg  0000000000000000
> >>>    mtvec    0000000000000000
> >>>    stvec    0000000000000000
> >>>    mepc     0000000000000000
> >>>    sepc     0000000000000000
> >>>    mcause   0000000000000000
> >>>    scause   0000000000000000
> >>>    mtval    0000000000000000
> >>>    stval    0000000000000000
> >>>    mscratch 0000000000000000
> >>>    sscratch 0000000000000000
> >>>    satp     0000000000000000
> >>>    x0/zero  0000000000000000 x1/ra    0000000000000000 x2/sp
> >>> 0000000000000000 x3/gp    0000000000000000
> >>>    x4/tp    0000000000000000 x5/t0    0000000000000000 x6/t1
> >>> 0000000000000000 x7/t2    0000000000000000
> >>>    x8/s0    0000000000000000 x9/s1    0000000000000000 x10/a0
> >>> 0000000000000000 x11/a1   00000000bf000000
> >>>    x12/a2   0000000000000000 x13/a3   0000000000000000 x14/a4
> >>> 0000000000000000 x15/a5   0000000000000000
> >>>    x16/a6   0000000000000000 x17/a7   0000000000000000 x18/s2
> >>> 0000000000000000 x19/s3   0000000000000000
> >>>    x20/s4   0000000000000000 x21/s5   0000000000000000 x22/s6
> >>> 0000000000000000 x23/s7   0000000000000000
> >>>    x24/s8   0000000000000000 x25/s9   0000000000000000 x26/s10
> >>> 0000000000000000 x27/s11  0000000000000000
> >>>    x28/t3   0000000000000000 x29/t4   0000000000000000 x30/t5
> >>> 0000000000000000 x31/t6   0000000000000000
> >>>

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Ralf Ramsauer]

On 15/03/2022 13:25, Anup Patel wrote:
> On Tue, Mar 15, 2022 at 5:47 PM Ralf Ramsauer
> <ralf.ramsauer@oth-regensburg.de> wrote:
>>
>>
>>
>> On 15/03/2022 09:33, Anup Patel wrote:
>>> On Tue, Mar 15, 2022 at 12:18 PM Alistair Francis <alistair23@gmail.com> wrote:
>>>>
>>>> On Sun, Mar 13, 2022 at 12:12 PM Ralf Ramsauer
>>>> <ralf.ramsauer@oth-regensburg.de> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to run Linux/QEMU+KVM inside an emulated
>>>>> qemu-system-riscv64 VM (x86 host). On latest&greatest QEMU (1416688c53),
>>>>> I run Linux inside QEMU. On host side:
>>>>>
>>>>> qemu-system-riscv64 -nographic \
>>>>>            -machine virt \
>>>>>            -cpu 'rv64,h=true' \
>>>>>            -smp 8 -m 8G \
>>>>>            -bios $latest_opensbi \
>>>>>            -kernel $latest_upstream_linux \
>>>>>           […snip…]
>>>>>
>>>>> Inside that machine, I boot Linux with KVM enabled:
>>>>>
>>>>> […]
>>>>> [    0.228939] kvm [1]: hypervisor extension available
>>>>> [    0.229000] kvm [1]: using Sv48x4 G-stage page table format
>>>>> [    0.229033] kvm [1]: VMID 14 bits available
>>>>> […]
>>>>>
>>>>> KVM seems to load correctly. Inside that machine, I compiled QEMU from
>>>>> the same sources with the KVM accelerator enabled. When I try to start
>>>>> QEMU with KVM enabled, I get
>>>>>
>>>>> ./qemu-system-riscv64 -nographic \
>>>>>           -monitor none \
>>>>>           -machine virt \
>>>>>           -smp 1 -m 1G \
>>>>>           -bios ./pc-bios/opensbi-riscv64-generic-fw_dynamic.bin \
>>>>>           -serial stdio \
>>>>>           -enable-kvm
>>>
>>> The QEMU RISC-V KVM support is quite new so I haven't got a chance
>>> to play with it.
>>
>> Btw, the kernel's KVM selftests seem to run.
>>
>>>
>>> Nevertheless, we should not pass any firmware when using KVM
>>> so can you try "-bios none" ?
>>
>> I have no kernel specified in the guest, so I'd then expect to boot into
>> an completely empty machine (besides the tiny startup stub in the MROM).
>> However, I still get the exact same error as mentioned in my initial mail.
> 
> When KVM is enabled and the kernel is not provided, there is simply
> nothing in Guest RAM (0x80000000) or ROM (0x00000000) which means
> you are booting garbage.

In all machine models ROM is at 0x1000, afaict. This is also the 
DEFAULT_RESETVEC of Qemu/RiscV. At least those few instructions should 
execute.

Cf.
https://github.com/qemu/qemu/blob/master/hw/riscv/boot.c#L285
and
https://github.com/qemu/qemu/blob/master/hw/riscv/virt.c#L1371

> 
>>
>> Again, without "-enable-kvm", everything works as expected (i.e., I
>> don't see any output, because there's basically no guest code to execute).
> 
> Strange, but even in this case you might be running garbage.

Yeah, and in case of !kvm, booting garbage results in no output. With 
kvm enabled, it faults.

> 
>>
>> Just out of curiosity: if you don't pass a bios to the machine, what
>> would then the boot sequence be?
> 
> For QEMU KVM:
> 1) There is no M-mode
> 2) Guest directly boots in S-mode
> 3) The SBI services are provided by in-kernel KVM module

Aaah! I understand. Thanks for the clarification.

> 
> You need to specify the Guest kernel using "-kernel" parameter when
> using QEMU KVM.

Yikes, that one works!

qemu-system-riscv64 -monitor none \
	-machine virt -smp 1 -m 1G \
	-bios none -nographic -serial stdio \
	-enable-kvm -kernel ./Image

[    0.000000] Linux version 5.17.0-rc7-00235-gaad611a868d1 
(ralf@redstone02) (riscv64-linux-gnu-gcc (Ubuntu 10.3.0-8ubuntu1) 
10.3.0, GNU ld (GNU Binutils for Ubuntu) 2.37) #10 SMP Sun Mar 13 
01:11:32 CET 2022
[    0.000000] Machine model: riscv-virtio,qemu
[    0.000000] efi: UEFI not found.
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 
0x0000000080000000-0x00000000bfffffff]
[    0.000000] SBI specification v0.2 detected
[    0.000000] SBI implementation ID=0x3 Version=0x51100
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI HSM extension detected
[…]


I thought it'd be better to get OpenSBI running before even trying to 
pass a kernel. I didn't know that SBI services are emulated/moderated(?) 
by KVM, and that there must be no bios specified when using KVM. Will 
have a closer look at the code.

Anup, thank you!

   Ralf

> 
> Regards,
> Anup
> 
>>
>> Thanks
>>     Ralf
>>
>>>
>>> Regards,
>>> Anup
>>>
>>>>>
>>>>> [ 4860.559194] kvm [9942]: VCPU exit error -95
>>>>> [ 4860.584262] kvm [9942]: SEPC=0x0 SSTATUS=0x200004120 HSTATUS=0x2002001c0
>>>>> [ 4860.586839] kvm [9942]: SCAUSE=0x14 STVAL=0x0 HTVAL=0x0 HTINST=0x0
>>>>>
>>>>> on the ringbuffer, together with a register dump of qemu [1] on the
>>>>> console. Needless to say, but without -enable-kvm, it works fine.
>>>>>
>>>>> As far as I see that, SCAUSE=0x14 reports a 'Instruction Guest PF',
>>>>> which would be kind of correct, if the guest's PC really tries to
>>>>> execute at phys 0x0. DRAM of the 'virt' machine definition should start
>>>>> at 0x80000000, where OpenSBI resides. So I wonder if an erroneous reset
>>>>> PC might be the culprit…
>>>>>
>>>>> Before digging deeper into that issue, I wanted to ask if Qemu/KVM
>>>>> inside an emulated riscv64+H-extension is actually supported, or if this
>>>>> is a known bug and has some ongoing work.
>>>>
>>>> RISC-V KVM should work. I haven't had a chance to try it myself though.
>>>>
>>>> I have CCed two people who hopefully can help.
>>>>
>>>> Alistair
>>>>
>>>>>
>>>>> Thanks
>>>>>      Ralf
>>>>>
>>>>> [1]
>>>>>
>>>>>     pc       0000000000000000
>>>>>     mhartid  0000000000000000
>>>>>     mstatus  0000000200000000
>>>>>     mip      0000000000000000
>>>>>     mie      0000000000000000
>>>>>     mideleg  0000000000000000
>>>>>     medeleg  0000000000000000
>>>>>     mtvec    0000000000000000
>>>>>     stvec    0000000000000000
>>>>>     mepc     0000000000000000
>>>>>     sepc     0000000000000000
>>>>>     mcause   0000000000000000
>>>>>     scause   0000000000000000
>>>>>     mtval    0000000000000000
>>>>>     stval    0000000000000000
>>>>>     mscratch 0000000000000000
>>>>>     sscratch 0000000000000000
>>>>>     satp     0000000000000000
>>>>>     x0/zero  0000000000000000 x1/ra    0000000000000000 x2/sp
>>>>> 0000000000000000 x3/gp    0000000000000000
>>>>>     x4/tp    0000000000000000 x5/t0    0000000000000000 x6/t1
>>>>> 0000000000000000 x7/t2    0000000000000000
>>>>>     x8/s0    0000000000000000 x9/s1    0000000000000000 x10/a0
>>>>> 0000000000000000 x11/a1   00000000bf000000
>>>>>     x12/a2   0000000000000000 x13/a3   0000000000000000 x14/a4
>>>>> 0000000000000000 x15/a5   0000000000000000
>>>>>     x16/a6   0000000000000000 x17/a7   0000000000000000 x18/s2
>>>>> 0000000000000000 x19/s3   0000000000000000
>>>>>     x20/s4   0000000000000000 x21/s5   0000000000000000 x22/s6
>>>>> 0000000000000000 x23/s7   0000000000000000
>>>>>     x24/s8   0000000000000000 x25/s9   0000000000000000 x26/s10
>>>>> 0000000000000000 x27/s11  0000000000000000
>>>>>     x28/t3   0000000000000000 x29/t4   0000000000000000 x30/t5
>>>>> 0000000000000000 x31/t6   0000000000000000
>>>>>

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Peter Maydell]

On Tue, 15 Mar 2022 at 12:29, Ralf Ramsauer
<ralf.ramsauer@oth-regensburg.de> wrote:
> I have no kernel specified in the guest, so I'd then expect to boot into
> an completely empty machine (besides the tiny startup stub in the MROM).
> However, I still get the exact same error as mentioned in my initial mail.
>
> Again, without "-enable-kvm", everything works as expected (i.e., I
> don't see any output, because there's basically no guest code to execute).

Well, you'll always be trying to execute something, even if it's
all-zeroes. I dunno what RISC-V does with its encodings, but you
typically either end up in an infinite loop of taking exceptions
(if all-0s isn't a valid instruction) or else you just execute
through all of guest RAM (if it is something similar to a no-op).
In the latter scenario it's quite plausible that the guest ends
up doing something KVM doesn't expect, such as trying to execute
from an area of the address space where there is no RAM, or a device.
If so, "return failure from VCPU_RUN and QEMU aborts" is what I
think I would expect to see.

If I were you I would try it with some actual guest code loaded
at a location where there is RAM...

-- PMM

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Ralf Ramsauer]

On 15/03/2022 13:42, Peter Maydell wrote:
> On Tue, 15 Mar 2022 at 12:29, Ralf Ramsauer
> <ralf.ramsauer@oth-regensburg.de> wrote:
>> I have no kernel specified in the guest, so I'd then expect to boot into
>> an completely empty machine (besides the tiny startup stub in the MROM).
>> However, I still get the exact same error as mentioned in my initial mail.
>>
>> Again, without "-enable-kvm", everything works as expected (i.e., I
>> don't see any output, because there's basically no guest code to execute).
> 
> Well, you'll always be trying to execute something, even if it's
> all-zeroes. I dunno what RISC-V does with its encodings, but you
> typically either end up in an infinite loop of taking exceptions
> (if all-0s isn't a valid instruction) or else you just execute
> through all of guest RAM (if it is something similar to a no-op).
> In the latter scenario it's quite plausible that the guest ends
> up doing something KVM doesn't expect, such as trying to execute
> from an area of the address space where there is no RAM, or a device.
> If so, "return failure from VCPU_RUN and QEMU aborts" is what I
> think I would expect to see.
> 
> If I were you I would try it with some actual guest code loaded
> at a location where there is RAM...

I guess you are right, and I think, together with the other mail, I now 
understand why I experienced a failure:

As Anup explained, there is no M-Mode, but I passed an OpenSBI BIOS. The 
initial MROM handed over to OpenSBI (located at RAM base 0x80000000). At 
some point, OpenSBI threw an exception (maybe it accessed an CSR it is 
not allowed to, whatever), because it is in S-Mode. Exception vectors 
were not (yet) installed, but initialized with 0x0.

This is how I ended up with the PC at 0x0. Behind 0x0, there's no 
physical backing, and KVM threw an Instruction Guest PF (SCAUSE=0x14).

Didn't verify this, but this somehow sounds plausible to me.

Thanks!
   Ralf

> 
> -- PMM

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Alistair Francis]

On Tue, Mar 15, 2022 at 10:51 PM Ralf Ramsauer
<ralf.ramsauer@oth-regensburg.de> wrote:
>
>
>
> On 15/03/2022 13:42, Peter Maydell wrote:
> > On Tue, 15 Mar 2022 at 12:29, Ralf Ramsauer
> > <ralf.ramsauer@oth-regensburg.de> wrote:
> >> I have no kernel specified in the guest, so I'd then expect to boot into
> >> an completely empty machine (besides the tiny startup stub in the MROM).
> >> However, I still get the exact same error as mentioned in my initial mail.
> >>
> >> Again, without "-enable-kvm", everything works as expected (i.e., I
> >> don't see any output, because there's basically no guest code to execute).
> >
> > Well, you'll always be trying to execute something, even if it's
> > all-zeroes. I dunno what RISC-V does with its encodings, but you
> > typically either end up in an infinite loop of taking exceptions
> > (if all-0s isn't a valid instruction) or else you just execute
> > through all of guest RAM (if it is something similar to a no-op).
> > In the latter scenario it's quite plausible that the guest ends
> > up doing something KVM doesn't expect, such as trying to execute
> > from an area of the address space where there is no RAM, or a device.
> > If so, "return failure from VCPU_RUN and QEMU aborts" is what I
> > think I would expect to see.
> >
> > If I were you I would try it with some actual guest code loaded
> > at a location where there is RAM...
>
> I guess you are right, and I think, together with the other mail, I now
> understand why I experienced a failure:
>
> As Anup explained, there is no M-Mode, but I passed an OpenSBI BIOS. The
> initial MROM handed over to OpenSBI (located at RAM base 0x80000000). At
> some point, OpenSBI threw an exception (maybe it accessed an CSR it is
> not allowed to, whatever), because it is in S-Mode. Exception vectors
> were not (yet) installed, but initialized with 0x0.

Hmm... This seems like a bug. We shouldn't allow the user to specify a
`-bios` option if using KVM. Would you mind preparing a patch to catch
this?

Alistair

>
> This is how I ended up with the PC at 0x0. Behind 0x0, there's no
> physical backing, and KVM threw an Instruction Guest PF (SCAUSE=0x14).
>
> Didn't verify this, but this somehow sounds plausible to me.
>
> Thanks!
>    Ralf
>
> >
> > -- PMM

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Peter Maydell]

On Wed, 16 Mar 2022 at 22:23, Alistair Francis <alistair23@gmail.com> wrote:
> Hmm... This seems like a bug. We shouldn't allow the user to specify a
> `-bios` option if using KVM. Would you mind preparing a patch to catch
> this?

You don't want to allow the possibility of a bios blob that expects
to run in S-mode, the way arm virt can run an EL1 UEFI BIOS ?

thanks
-- PMM

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Ralf Ramsauer]

On 17/03/2022 10:45, Peter Maydell wrote:
> On Wed, 16 Mar 2022 at 22:23, Alistair Francis <alistair23@gmail.com> wrote:
>> Hmm... This seems like a bug. We shouldn't allow the user to specify a
>> `-bios` option if using KVM. Would you mind preparing a patch to catch
>> this?
> 
> You don't want to allow the possibility of a bios blob that expects
> to run in S-mode, the way arm virt can run an EL1 UEFI BIOS ?

Valid point. In any case, if you wish, we could provide a patch for 
disallowing -bios in combination with -enable-kvm.

Thanks
   Ralf

> 
> thanks
> -- PMM

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Alistair Francis]

On Thu, Mar 17, 2022 at 7:46 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Wed, 16 Mar 2022 at 22:23, Alistair Francis <alistair23@gmail.com> wrote:
> > Hmm... This seems like a bug. We shouldn't allow the user to specify a
> > `-bios` option if using KVM. Would you mind preparing a patch to catch
> > this?
>
> You don't want to allow the possibility of a bios blob that expects
> to run in S-mode, the way arm virt can run an EL1 UEFI BIOS ?

Interesting. We could still allow that by using -device loader though.

If we load something in S mode it really is a kernel and not firmware,
so I think the -bios argument is a little misleading. We could make
-bios and -enable-kvm a warning instead, to discourage users from
providing OpenSBI, but still allow them to pass something.

To me not allowing it seems reasonable, and we can always re-allow it
in the future if there is a good use case.

Alistair

>
> thanks
> -- PMM

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Palmer Dabbelt]

On Sun, 20 Mar 2022 22:43:07 PDT (-0700), alistair23@gmail.com wrote:
> On Thu, Mar 17, 2022 at 7:46 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Wed, 16 Mar 2022 at 22:23, Alistair Francis <alistair23@gmail.com> wrote:
>> > Hmm... This seems like a bug. We shouldn't allow the user to specify a
>> > `-bios` option if using KVM. Would you mind preparing a patch to catch
>> > this?
>>
>> You don't want to allow the possibility of a bios blob that expects
>> to run in S-mode, the way arm virt can run an EL1 UEFI BIOS ?
>
> Interesting. We could still allow that by using -device loader though.
>
> If we load something in S mode it really is a kernel and not firmware,
> so I think the -bios argument is a little misleading. We could make
> -bios and -enable-kvm a warning instead, to discourage users from
> providing OpenSBI, but still allow them to pass something.

We could handle this in OpenSBI: have it just detect that it's been 
launched in S-mode and just jump to the payload.

> To me not allowing it seems reasonable, and we can always re-allow it
> in the future if there is a good use case.

That also seems fine to me.  There'd be a lot of work involved in doing 
anything useful with S-mode firmware, given that things aren't really 
designed to work that way.  Certainly a lot more work than re-enabling 
this sort of flag, and while it might be a bit of a headache coupling 
that to a QEMU source-level change my guess is that there'd need to be a 
bunch of work done on new HW interfaces to make this useful so there'd 
be QEMU changes to emulate those either way.

Having at least a warning seems prudent, as most users are going to end 
up with a hang here and a silent hang is a headache for everyone.

> Alistair
>
>>
>> thanks
>> -- PMM

Re: [EXT] Re: QEMU+KVM on RISC-V + Hypervisor Extension

[Ralf Ramsauer]

On 21/03/2022 21:47, Palmer Dabbelt wrote:
> On Sun, 20 Mar 2022 22:43:07 PDT (-0700), alistair23@gmail.com wrote:
>> On Thu, Mar 17, 2022 at 7:46 PM Peter Maydell 
>> <peter.maydell@linaro.org> wrote:
>>>
>>> On Wed, 16 Mar 2022 at 22:23, Alistair Francis <alistair23@gmail.com> 
>>> wrote:
>>> > Hmm... This seems like a bug. We shouldn't allow the user to specify a
>>> > `-bios` option if using KVM. Would you mind preparing a patch to catch
>>> > this?
>>>
>>> You don't want to allow the possibility of a bios blob that expects
>>> to run in S-mode, the way arm virt can run an EL1 UEFI BIOS ?
>>
>> Interesting. We could still allow that by using -device loader though.
>>
>> If we load something in S mode it really is a kernel and not firmware,
>> so I think the -bios argument is a little misleading. We could make
>> -bios and -enable-kvm a warning instead, to discourage users from
>> providing OpenSBI, but still allow them to pass something.
> 
> We could handle this in OpenSBI: have it just detect that it's been 
> launched in S-mode and just jump to the payload.
> 
>> To me not allowing it seems reasonable, and we can always re-allow it
>> in the future if there is a good use case.
> 
> That also seems fine to me.  There'd be a lot of work involved in doing 
> anything useful with S-mode firmware, given that things aren't really 
> designed to work that way.  Certainly a lot more work than re-enabling 
> this sort of flag, and while it might be a bit of a headache coupling 
> that to a QEMU source-level change my guess is that there'd need to be a 
> bunch of work done on new HW interfaces to make this useful so there'd 
> be QEMU changes to emulate those either way.
> 
> Having at least a warning seems prudent, as most users are going to end 
> up with a hang here and a silent hang is a headache for everyone.

Okay, my analysis was wrong, -bios is in fact already ignored when being 
used in combination with -enable-kvm on the virt machine model. See:

https://git.qemu.org/?p=qemu.git;a=blob;f=hw/riscv/virt.c;h=da50cbed43ec54777992d40dbf158ec63fccef03;hb=HEAD#l1314

However, the bios will be silently ignored and dropped, which led me to 
the mistake. I'll provide a patch that gives a warning to the user.

   Ralf

> 
>> Alistair
>>
>>>
>>> thanks
>>> -- PMM