kvm enable over cortex-a53?

kvm enable over cortex-a53?

[Lei Wen]

Hi,

I am current try to enable kvm over cortex-a53 over 3.10 kernel.
After some patches backporting, I manage to get the /dev/kvm showed
out, and build
pass for the kernel.

However while I try to use the lkvm to launch the guest, some error shows up:

  # lkvm run -k nfs/Image -m 256 -c 1 --name guest-143
  Warning: nfs/Image is not a bzImage. Trying to load it as a flat binary...
  Info: Loaded kernel to 0x100080000 (18072024 bytes)
  Info: Placing fdt at 0x10fe00000 - 0x10fffffff
  Info: virtio-mmio.devices=0x200@0x10000:36

  Info: virtio-mmio.devices=0x200@0x10200:37

  Info: virtio-mmio.devices=0x200@0x10400:39

Emulate result 0
KVM exit reason: 6 ("KVM_EXIT_MMIO")

 Registers:
 PC:    0xffffffc000091a3c
 PSTATE:    0x200002c5
 SP_EL1:    0xffffffc0010ebcb0
 LR:    0xffffffc000091a84

With further checking it seem that while it try to access the console
driver whose
base address is over 0xd4018000, but it failed to pass the checking by
arm_addr_in_ioport_region and arm_addr_in_virtio_mmio_region, thus panic_kvm
is invoked.

My question is whether ARM_IOPORT_AREA/ARM_MMIO_AREA/
ARM_AXI_AREA/ARM_MEMORY_AREA need to be reconfigured for different
SOCs?

While I try to hack those MACRO into some value to let 0xd4018000 checking
passed, but seems host kernel would hang after that, and if add debug message
in lkvm tool, I could see it endlessly calls mmio emulation.

Another finding is that  during the host kernel booting, I didn't see
hvc console is
registered, but I already enabled CONFIG_HVC_DRIVER and
CONFIG_VIRTIO_CONSOLE. Is there any other step needed to be done?


BTW, my lkvm tool is from:
https://github.com/virtualopensystems/linux-kvm-arm.git
with some modification to add cortex-a53 support.

Thanks,
Lei

Re: kvm enable over cortex-a53?

[Marc Zyngier]

On 02/06/14 10:09, Lei Wen wrote:
> Hi,
> 
> I am current try to enable kvm over cortex-a53 over 3.10 kernel.
> After some patches backporting, I manage to get the /dev/kvm showed
> out, and build
> pass for the kernel.
> 
> However while I try to use the lkvm to launch the guest, some error shows up:
> 
>   # lkvm run -k nfs/Image -m 256 -c 1 --name guest-143
>   Warning: nfs/Image is not a bzImage. Trying to load it as a flat binary...
>   Info: Loaded kernel to 0x100080000 (18072024 bytes)
>   Info: Placing fdt at 0x10fe00000 - 0x10fffffff
>   Info: virtio-mmio.devices=0x200@0x10000:36
> 
>   Info: virtio-mmio.devices=0x200@0x10200:37
> 
>   Info: virtio-mmio.devices=0x200@0x10400:39
> 
> Emulate result 0
> KVM exit reason: 6 ("KVM_EXIT_MMIO")
> 
>  Registers:
>  PC:    0xffffffc000091a3c
>  PSTATE:    0x200002c5
>  SP_EL1:    0xffffffc0010ebcb0
>  LR:    0xffffffc000091a84
> 
> With further checking it seem that while it try to access the console
> driver whose
> base address is over 0xd4018000, but it failed to pass the checking by
> arm_addr_in_ioport_region and arm_addr_in_virtio_mmio_region, thus panic_kvm
> is invoked.

kvmtool tells you where the console lives (it is one of the virtio-mmio
devices that it prints out when it starts). The address you quote
(assuming that this is an IPA) doesn't seem to have anything to do with
anything kvmtool emulates. Pretty sure your kernel is trying to poke at
some area without parsing the device tree...

> My question is whether ARM_IOPORT_AREA/ARM_MMIO_AREA/
> ARM_AXI_AREA/ARM_MEMORY_AREA need to be reconfigured for different
> SOCs?

No. That's the whole point of a virtual machine.

> While I try to hack those MACRO into some value to let 0xd4018000 checking
> passed, but seems host kernel would hang after that, and if add debug message
> in lkvm tool, I could see it endlessly calls mmio emulation.

Yes, because there is nothing there that kvmtool can emulate.

> Another finding is that  during the host kernel booting, I didn't see
> hvc console is
> registered, but I already enabled CONFIG_HVC_DRIVER and
> CONFIG_VIRTIO_CONSOLE. Is there any other step needed to be done?

Why would you expect the *host* to have a hvc console? This is a guest
feature, not a host one.

> 
> BTW, my lkvm tool is from:
> https://github.com/virtualopensystems/linux-kvm-arm.git
> with some modification to add cortex-a53 support.

What is wrong with *mainline* kvmtool?

	M.
-- 
Jazz is not dead. It just smells funny...

Re: kvm enable over cortex-a53?

[Lei Wen]

Hi Marc,

On Mon, Jun 2, 2014 at 5:27 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On 02/06/14 10:09, Lei Wen wrote:
>> Hi,
>>
>> I am current try to enable kvm over cortex-a53 over 3.10 kernel.
>> After some patches backporting, I manage to get the /dev/kvm showed
>> out, and build
>> pass for the kernel.
>>
>> However while I try to use the lkvm to launch the guest, some error shows up:
>>
>>   # lkvm run -k nfs/Image -m 256 -c 1 --name guest-143
>>   Warning: nfs/Image is not a bzImage. Trying to load it as a flat binary...
>>   Info: Loaded kernel to 0x100080000 (18072024 bytes)
>>   Info: Placing fdt at 0x10fe00000 - 0x10fffffff
>>   Info: virtio-mmio.devices=0x200@0x10000:36
>>
>>   Info: virtio-mmio.devices=0x200@0x10200:37
>>
>>   Info: virtio-mmio.devices=0x200@0x10400:39
>>
>> Emulate result 0
>> KVM exit reason: 6 ("KVM_EXIT_MMIO")
>>
>>  Registers:
>>  PC:    0xffffffc000091a3c
>>  PSTATE:    0x200002c5
>>  SP_EL1:    0xffffffc0010ebcb0
>>  LR:    0xffffffc000091a84
>>
>> With further checking it seem that while it try to access the console
>> driver whose
>> base address is over 0xd4018000, but it failed to pass the checking by
>> arm_addr_in_ioport_region and arm_addr_in_virtio_mmio_region, thus panic_kvm
>> is invoked.
>
> kvmtool tells you where the console lives (it is one of the virtio-mmio
> devices that it prints out when it starts). The address you quote
> (assuming that this is an IPA) doesn't seem to have anything to do with
> anything kvmtool emulates. Pretty sure your kernel is trying to poke at
> some area without parsing the device tree...

Do you have any suggestion for how to debug it?...
Would kvmtool auto load the device tree, or shall I put the device tree binary
into somewhere?

>
>> My question is whether ARM_IOPORT_AREA/ARM_MMIO_AREA/
>> ARM_AXI_AREA/ARM_MEMORY_AREA need to be reconfigured for different
>> SOCs?
>
> No. That's the whole point of a virtual machine.

Got it.

>
>> While I try to hack those MACRO into some value to let 0xd4018000 checking
>> passed, but seems host kernel would hang after that, and if add debug message
>> in lkvm tool, I could see it endlessly calls mmio emulation.
>
> Yes, because there is nothing there that kvmtool can emulate.
>
>> Another finding is that  during the host kernel booting, I didn't see
>> hvc console is
>> registered, but I already enabled CONFIG_HVC_DRIVER and
>> CONFIG_VIRTIO_CONSOLE. Is there any other step needed to be done?
>
> Why would you expect the *host* to have a hvc console? This is a guest
> feature, not a host one.
>
>>
>> BTW, my lkvm tool is from:
>> https://github.com/virtualopensystems/linux-kvm-arm.git
>> with some modification to add cortex-a53 support.
>
> What is wrong with *mainline* kvmtool?

The kvmtool seems not mainline yet? Am I wrong?
I also try below git, but nothing different...
http://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git

Thanks,
Lei

Re: kvm enable over cortex-a53?

[Marc Zyngier]

On Tue, Jun 03 2014 at  3:14:33 am BST, Lei Wen <adrian.wenl@gmail.com> wrote:
> Hi Marc,
>
> On Mon, Jun 2, 2014 at 5:27 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> kvmtool tells you where the console lives (it is one of the virtio-mmio
>> devices that it prints out when it starts). The address you quote
>> (assuming that this is an IPA) doesn't seem to have anything to do with
>> anything kvmtool emulates. Pretty sure your kernel is trying to poke at
>> some area without parsing the device tree...
>
> Do you have any suggestion for how to debug it?...

I suggest you start with a very simple guest kernel config (only virtio
drivers), no additional stuff. My gut feeling is that you have stuff in
your 3.10 kernel that shouldn't be there.

Your best bet would probably to use a recent tree for your guests.

> Would kvmtool auto load the device tree, or shall I put the device
> tree binary into somewhere?

kvmtool generates the device tree based on what option you give it.

>>> BTW, my lkvm tool is from:
>>> https://github.com/virtualopensystems/linux-kvm-arm.git
>>> with some modification to add cortex-a53 support.
>>
>> What is wrong with *mainline* kvmtool?
>
> The kvmtool seems not mainline yet? Am I wrong?

Mainline in this case means in Pekka Enberg's tree (he is the kvmtool
maintainer).

> I also try below git, but nothing different...
> http://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git

This is Will Deacon's staging tree, not the mainline tree.

The real thing is at:
git://git.kernel.org/pub/scm/linux/kernel/git/penberg/linux.git kvmtool/next

	M.
-- 
Jazz is not dead. It just smells funny.

Re: kvm enable over cortex-a53?

[Lei Wen]

On Tue, Jun 3, 2014 at 5:11 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On Tue, Jun 03 2014 at  3:14:33 am BST, Lei Wen <adrian.wenl@gmail.com> wrote:
>> Hi Marc,
>>
>> On Mon, Jun 2, 2014 at 5:27 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>> kvmtool tells you where the console lives (it is one of the virtio-mmio
>>> devices that it prints out when it starts). The address you quote
>>> (assuming that this is an IPA) doesn't seem to have anything to do with
>>> anything kvmtool emulates. Pretty sure your kernel is trying to poke at
>>> some area without parsing the device tree...
>>
>> Do you have any suggestion for how to debug it?...
>
> I suggest you start with a very simple guest kernel config (only virtio
> drivers), no additional stuff. My gut feeling is that you have stuff in
> your 3.10 kernel that shouldn't be there.
>
> Your best bet would probably to use a recent tree for your guests.

I see.
After remove the earlyprintk=uart8250-32bit,0xd4030000 in the kvmtool
command line, the KVM_EXIT_MMIO message is gone.
But the console is silence afterward, supposing there is something
wrong in boot the guest kernel.
Is there some method to access the guest kernel's log buffer?
Or some other way to output the guest kernel's message with earlyprintk?

I see virtio_block is defined in foundation-v8.dts, is it a must to have
such device node? Shall I map the uart phys register range by
this node?

>
>> Would kvmtool auto load the device tree, or shall I put the device
>> tree binary into somewhere?
>
> kvmtool generates the device tree based on what option you give it.
>
>>>> BTW, my lkvm tool is from:
>>>> https://github.com/virtualopensystems/linux-kvm-arm.git
>>>> with some modification to add cortex-a53 support.
>>>
>>> What is wrong with *mainline* kvmtool?
>>
>> The kvmtool seems not mainline yet? Am I wrong?
>
> Mainline in this case means in Pekka Enberg's tree (he is the kvmtool
> maintainer).
>
>> I also try below git, but nothing different...
>> http://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git
>
> This is Will Deacon's staging tree, not the mainline tree.
>
> The real thing is at:
> git://git.kernel.org/pub/scm/linux/kernel/git/penberg/linux.git kvmtool/next

Got it, very appreciate your great help.

Thanks,
Lei

Re: kvm enable over cortex-a53?

[Marc Zyngier]

On Wed, Jun 04 2014 at  7:19:42 am BST, Lei Wen <adrian.wenl@gmail.com> wrote:
> On Tue, Jun 3, 2014 at 5:11 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>> On Tue, Jun 03 2014 at  3:14:33 am BST, Lei Wen <adrian.wenl@gmail.com> wrote:
>>> Hi Marc,
>>>
>>> On Mon, Jun 2, 2014 at 5:27 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>> kvmtool tells you where the console lives (it is one of the virtio-mmio
>>>> devices that it prints out when it starts). The address you quote
>>>> (assuming that this is an IPA) doesn't seem to have anything to do with
>>>> anything kvmtool emulates. Pretty sure your kernel is trying to poke at
>>>> some area without parsing the device tree...
>>>
>>> Do you have any suggestion for how to debug it?...
>>
>> I suggest you start with a very simple guest kernel config (only virtio
>> drivers), no additional stuff. My gut feeling is that you have stuff in
>> your 3.10 kernel that shouldn't be there.
>>
>> Your best bet would probably to use a recent tree for your guests.
>
> I see.
> After remove the earlyprintk=uart8250-32bit,0xd4030000 in the kvmtool
> command line, the KVM_EXIT_MMIO message is gone.

Not surprising. Telling the kernel to poke random addresses that are
only relevant to the host is never going to have the effect you seem to
expect.

> But the console is silence afterward, supposing there is something
> wrong in boot the guest kernel.
> Is there some method to access the guest kernel's log buffer?
> Or some other way to output the guest kernel's message with earlyprintk?

If using kvmtool, the method is to use "earlyprintk=uart8250-32bit,0x3f8".

> I see virtio_block is defined in foundation-v8.dts, is it a must to have
> such device node? Shall I map the uart phys register range by
> this node?

No. The host devices are not delevant to the guest at all. The guest
never accesses any physical devices apart from GIC and timers. kvmtool
creates a completely virtual platform that has nothing to do with the
host.

You don't have to touch kvmtool at all, this is a self-contained
platform emulation. If it doesn't work properly, there is only two
options:
- your guest is not configured properly
- the KVM backport to your 3.10 kernel is less than perfect.

Overall, you'd be much better off using a mainline kernel with a model,
get familiar with the tools and configurations, and only then try to
replicate this on your HW.

	M.
-- 
Jazz is not dead. It just smells funny.

Re: kvm enable over cortex-a53?

[Lei Wen]

Marc,

On Wed, Jun 4, 2014 at 5:20 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> On Wed, Jun 04 2014 at  7:19:42 am BST, Lei Wen <adrian.wenl@gmail.com> wrote:
>> On Tue, Jun 3, 2014 at 5:11 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>> On Tue, Jun 03 2014 at  3:14:33 am BST, Lei Wen <adrian.wenl@gmail.com> wrote:
>>>> Hi Marc,
>>>>
>>>> On Mon, Jun 2, 2014 at 5:27 PM, Marc Zyngier <marc.zyngier@arm.com> wrote:
>>>>> kvmtool tells you where the console lives (it is one of the virtio-mmio
>>>>> devices that it prints out when it starts). The address you quote
>>>>> (assuming that this is an IPA) doesn't seem to have anything to do with
>>>>> anything kvmtool emulates. Pretty sure your kernel is trying to poke at
>>>>> some area without parsing the device tree...
>>>>
>>>> Do you have any suggestion for how to debug it?...
>>>
>>> I suggest you start with a very simple guest kernel config (only virtio
>>> drivers), no additional stuff. My gut feeling is that you have stuff in
>>> your 3.10 kernel that shouldn't be there.
>>>
>>> Your best bet would probably to use a recent tree for your guests.
>>
>> I see.
>> After remove the earlyprintk=uart8250-32bit,0xd4030000 in the kvmtool
>> command line, the KVM_EXIT_MMIO message is gone.
>
> Not surprising. Telling the kernel to poke random addresses that are
> only relevant to the host is never going to have the effect you seem to
> expect.
>
>> But the console is silence afterward, supposing there is something
>> wrong in boot the guest kernel.
>> Is there some method to access the guest kernel's log buffer?
>> Or some other way to output the guest kernel's message with earlyprintk?
>
> If using kvmtool, the method is to use "earlyprintk=uart8250-32bit,0x3f8".

I try this setting with real soc and foundation model, seems both cannot get
boot up after changing to this, seems get hang over somewhere, and kvmtool
would keep report that it emulate device message.
If with smh, foundation model could works for print out the early message.

>
>> I see virtio_block is defined in foundation-v8.dts, is it a must to have
>> such device node? Shall I map the uart phys register range by
>> this node?
>
> No. The host devices are not delevant to the guest at all. The guest
> never accesses any physical devices apart from GIC and timers. kvmtool
> creates a completely virtual platform that has nothing to do with the
> host.
>
> You don't have to touch kvmtool at all, this is a self-contained
> platform emulation. If it doesn't work properly, there is only two
> options:
> - your guest is not configured properly
> - the KVM backport to your 3.10 kernel is less than perfect.
>
> Overall, you'd be much better off using a mainline kernel with a model,
> get familiar with the tools and configurations, and only then try to
> replicate this on your HW.
>

It is true that something is not configured right, after modification,
the board with kvm could run now.

Thanks,
Lei