Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Jan Kiszka]

On 2017-04-19 05:48, Constantin Petra wrote:
> Hello,
> 
> I have tried the same binaries on a ZCU102 board and it seems to work.
> Cannot be sure though, the board is remote and I don't have access to
> the secondary UART as I understand it would be required for the uart-demo.

You can always reconfigure this to the primary one. We share it in other
setups as well.

> So from some reason, under QEMU something is not in place, but what can
> that be? It would be convenient from our perspective, if it would work...

I fully agree, and I'm waiting for official EL2 support in QEMU in order
to introduce a virtual target for ARM[64], just like we have on x86 already.

I suppose you are using Xilinx' version of QEMU, not upstream, correct?
I didn't track recently what the state of upstream is. Just checking the
list again, seeing some interesting patches from early this year...
Peter, can you comment on what is still missing? I'm reading that you
successfully booted KVM inside QEMU. Seems like we are pretty close now
and it could make sense to debug the remaining issues.

Jan

> 
> root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell list
> ID      Name                    State           Assigned CPUs          
> Failed CPUs
> 0       ZynqMP-ZCU102           running         0-2
> 1       gic-demo                running         3
> 
> 
> Best Regards,
> Constantin
> 
> On Tue, Apr 18, 2017 at 1:32 PM, Constantin Petra
> <constantin.petra@gmail.com <mailto:constantin.petra@gmail.com>> wrote:
> 
>     Hi,
> 
>     Yes, that's correct, I'm trying to run the whole stack in QEMU, not
>     hardware.
> 
>     Best Regards,
>     Constantin
> 
>     On Tue, Apr 18, 2017 at 12:52 PM, Jan Kiszka <jan.kiszka@web.de
>     <mailto:jan.kiszka@web.de>> wrote:
> 
>         On 2017-04-18 10:22, constantin.petra@gmail.com
>         <mailto:constantin.petra@gmail.com> wrote:
>         > Hi,
>         >
>         > I have made the changes to pci.c, but QEMU seems to freeze after enabling Jailhouse after the "opening" message is displayed (I have previously started a SSH connection which also freezes - to make sure the Serial port is not somehow hijacked by hypervisor as described in the documentaiton). I cannot start GDB using petalinux tools due to a Python 2.6 dependency (the system has Python 2.7) so I don't really know what happens. Any ideas? (Attached also a log file with the boot messages; tried relying on configured dtb and here also tried memmap for the reserved memory, with the same result).
>         >
> 
>         Oh, now I get it: You are running the whole stack emulated in
>         QEMU, not
>         on real hardware, right? Maybe there is a subtle difference in
>         the QEMU
>         system configuration compared to the board, or Jailhouse is
>         stressing
>         some aspect of virtualization that the QEMU model does not yet
>         handle
>         correctly or at all.
> 
>         Adding Edgar, maybe he can comment on the state of hypervisor
>         execution
>         in the Xilinx QEMU version. He will probably need some details
>         about the
>         setup (versions, configuration etc.).
> 
>         Jan
> 
>         > root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse enable
>         /home/root/jailhouse/configs/zynqmp-zcu102.cell
>         >
>         > Initializing Jailhouse hypervisor v0.6 (47-ge02403c-dirty) on
>         CPU 0
>         > Code location: 0x0000ffffc0200040
>         > Page pool usage after early setup: mem 33/996, remap 64/131072
>         > Initializing processors:
>         >  CPU 0... OK
>         >  CPU 1... OK
>         >  CPU 2... OK
>         >  CPU 3... OK
>         > Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
>         > Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
>         > Page pool usage after late setup: mem 44/996, remap 69/131072
>         > Activating hypervisor
>         > [   71.547900] OF: PCI: host bridge //vpci@0 ranges:
>         > [   71.549101] OF: PCI:   MEM 0xfc100000..0xfc103fff -> 0xfc100000
>         > [   71.552493] pci-host-generic fc000000.vpci: ECAM at [mem
>         0xfc000000-0xfc0fffff] for [bus 00]
>         > [   71.555644] pci-host-generic fc000000.vpci: PCI host bridge
>         to bus 0001:00
>         > [   71.556337] pci_bus 0001:00: root bus resource [bus 00]
>         > [   71.556698] pci_bus 0001:00: root bus resource [mem
>         0xfc100000-0xfc103fff]
>         > [   71.569008] pci 0001:00:00.0: BAR 0: assigned [mem
>         0xfc100000-0xfc1000ff 64bit]
>         > [   71.570284] pci 0001:00:01.0: BAR 0: assigned [mem
>         0xfc100100-0xfc1001ff 64bit]
>         > [   71.576654] The Jailhouse is opening.
>         > root@plnx_aarch64:~#
>         >
>         > Best Regards,
>         > Constantin
>         >
> 
> 
> 

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Constantin Petra]

Yes, I am using the QEMU version included by Xilinx in PetaLinux 2016.4.

On Wed, Apr 19, 2017 at 7:37 AM, Jan Kiszka <jan.kiszka@web.de> wrote:

> On 2017-04-19 05:48, Constantin Petra wrote:
> > Hello,
> >
> > I have tried the same binaries on a ZCU102 board and it seems to work.
> > Cannot be sure though, the board is remote and I don't have access to
> > the secondary UART as I understand it would be required for the
> uart-demo.
>
> You can always reconfigure this to the primary one. We share it in other
> setups as well.
>
> > So from some reason, under QEMU something is not in place, but what can
> > that be? It would be convenient from our perspective, if it would work...
>
> I fully agree, and I'm waiting for official EL2 support in QEMU in order
> to introduce a virtual target for ARM[64], just like we have on x86
> already.
>
> I suppose you are using Xilinx' version of QEMU, not upstream, correct?
> I didn't track recently what the state of upstream is. Just checking the
> list again, seeing some interesting patches from early this year...
> Peter, can you comment on what is still missing? I'm reading that you
> successfully booted KVM inside QEMU. Seems like we are pretty close now
> and it could make sense to debug the remaining issues.
>
> Jan
>
> >
> > root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell list
> > ID      Name                    State           Assigned CPUs
> > Failed CPUs
> > 0       ZynqMP-ZCU102           running         0-2
> > 1       gic-demo                running         3
> >
> >
> > Best Regards,
> > Constantin
> >
> > On Tue, Apr 18, 2017 at 1:32 PM, Constantin Petra
> > <constantin.petra@gmail.com <mailto:constantin.petra@gmail.com>> wrote:
> >
> >     Hi,
> >
> >     Yes, that's correct, I'm trying to run the whole stack in QEMU, not
> >     hardware.
> >
> >     Best Regards,
> >     Constantin
> >
> >     On Tue, Apr 18, 2017 at 12:52 PM, Jan Kiszka <jan.kiszka@web.de
> >     <mailto:jan.kiszka@web.de>> wrote:
> >
> >         On 2017-04-18 10:22, constantin.petra@gmail.com
> >         <mailto:constantin.petra@gmail.com> wrote:
> >         > Hi,
> >         >
> >         > I have made the changes to pci.c, but QEMU seems to freeze
> after enabling Jailhouse after the "opening" message is displayed (I have
> previously started a SSH connection which also freezes - to make sure the
> Serial port is not somehow hijacked by hypervisor as described in the
> documentaiton). I cannot start GDB using petalinux tools due to a Python
> 2.6 dependency (the system has Python 2.7) so I don't really know what
> happens. Any ideas? (Attached also a log file with the boot messages; tried
> relying on configured dtb and here also tried memmap for the reserved
> memory, with the same result).
> >         >
> >
> >         Oh, now I get it: You are running the whole stack emulated in
> >         QEMU, not
> >         on real hardware, right? Maybe there is a subtle difference in
> >         the QEMU
> >         system configuration compared to the board, or Jailhouse is
> >         stressing
> >         some aspect of virtualization that the QEMU model does not yet
> >         handle
> >         correctly or at all.
> >
> >         Adding Edgar, maybe he can comment on the state of hypervisor
> >         execution
> >         in the Xilinx QEMU version. He will probably need some details
> >         about the
> >         setup (versions, configuration etc.).
> >
> >         Jan
> >
> >         > root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse enable
> >         /home/root/jailhouse/configs/zynqmp-zcu102.cell
> >         >
> >         > Initializing Jailhouse hypervisor v0.6 (47-ge02403c-dirty) on
> >         CPU 0
> >         > Code location: 0x0000ffffc0200040
> >         > Page pool usage after early setup: mem 33/996, remap 64/131072
> >         > Initializing processors:
> >         >  CPU 0... OK
> >         >  CPU 1... OK
> >         >  CPU 2... OK
> >         >  CPU 3... OK
> >         > Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
> >         > Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
> >         > Page pool usage after late setup: mem 44/996, remap 69/131072
> >         > Activating hypervisor
> >         > [   71.547900] OF: PCI: host bridge //vpci@0 ranges:
> >         > [   71.549101] OF: PCI:   MEM 0xfc100000..0xfc103fff ->
> 0xfc100000
> >         > [   71.552493] pci-host-generic fc000000.vpci: ECAM at [mem
> >         0xfc000000-0xfc0fffff] for [bus 00]
> >         > [   71.555644] pci-host-generic fc000000.vpci: PCI host bridge
> >         to bus 0001:00
> >         > [   71.556337] pci_bus 0001:00: root bus resource [bus 00]
> >         > [   71.556698] pci_bus 0001:00: root bus resource [mem
> >         0xfc100000-0xfc103fff]
> >         > [   71.569008] pci 0001:00:00.0: BAR 0: assigned [mem
> >         0xfc100000-0xfc1000ff 64bit]
> >         > [   71.570284] pci 0001:00:01.0: BAR 0: assigned [mem
> >         0xfc100100-0xfc1001ff 64bit]
> >         > [   71.576654] The Jailhouse is opening.
> >         > root@plnx_aarch64:~#
> >         >
> >         > Best Regards,
> >         > Constantin
> >         >
> >
> >
> >
>
>

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Peter Maydell]

On 19 April 2017 at 05:37, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2017-04-19 05:48, Constantin Petra wrote:
>> So from some reason, under QEMU something is not in place, but what can
>> that be? It would be convenient from our perspective, if it would work...
>
> I fully agree, and I'm waiting for official EL2 support in QEMU in order
> to introduce a virtual target for ARM[64], just like we have on x86 already.

This has been in upstream QEMU for a little while now,
and will be in the upcoming 2.9 release. (Applies only for
AArch64 hosts using GICv3, needs some board support changes
which have currently only been done for 'virt'.)

thanks
-- PMM

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Jan Kiszka]

On 2017-04-19 10:45, Peter Maydell wrote:
> On 19 April 2017 at 05:37, Jan Kiszka <jan.kiszka@web.de> wrote:
>> On 2017-04-19 05:48, Constantin Petra wrote:
>>> So from some reason, under QEMU something is not in place, but what can
>>> that be? It would be convenient from our perspective, if it would work...
>>
>> I fully agree, and I'm waiting for official EL2 support in QEMU in order
>> to introduce a virtual target for ARM[64], just like we have on x86 already.
> 
> This has been in upstream QEMU for a little while now,
> and will be in the upcoming 2.9 release. (Applies only for
> AArch64 hosts using GICv3, needs some board support changes
> which have currently only been done for 'virt'.)

Ah, v3 - oversaw this "minor" detail. Too bad that there is still no
GICv3 silicon in our hands that we could test against (do you know some
recommendable board?). I'm pretty sure Jailhouse's GICv3 support is
broken by now (due to lacking tests), plus it is not yet enabled for our
ARM64 port. But I'm a bit reluctant to enable this only over QEMU...

Jan

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Edgar E. Iglesias]

On Wed, Apr 19, 2017 at 11:02:21AM +0200, Jan Kiszka wrote:
> On 2017-04-19 10:45, Peter Maydell wrote:
> > On 19 April 2017 at 05:37, Jan Kiszka <jan.kiszka@web.de> wrote:
> >> On 2017-04-19 05:48, Constantin Petra wrote:
> >>> So from some reason, under QEMU something is not in place, but what can
> >>> that be? It would be convenient from our perspective, if it would work...
> >>
> >> I fully agree, and I'm waiting for official EL2 support in QEMU in order
> >> to introduce a virtual target for ARM[64], just like we have on x86 already.
> > 
> > This has been in upstream QEMU for a little while now,
> > and will be in the upcoming 2.9 release. (Applies only for
> > AArch64 hosts using GICv3, needs some board support changes
> > which have currently only been done for 'virt'.)
> 
> Ah, v3 - oversaw this "minor" detail. Too bad that there is still no
> GICv3 silicon in our hands that we could test against (do you know some
> recommendable board?). I'm pretty sure Jailhouse's GICv3 support is
> broken by now (due to lacking tests), plus it is not yet enabled for our
> ARM64 port. But I'm a bit reluctant to enable this only over QEMU...

Hi,

Xilinx QEMU has support for the virtualization extensions in the GICv2 but
we're seeing issues with running hypervisors in multi-core setups.
It seems like guest migration between cores is causing trouble but we
haven't pinpointed the exact cause.

I'd be interested in debugging the problem seen with Jailhouse.
Can the images and instructions on howto run be shared with me?

Thanks,
Edgar

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Constantin Petra]

Hi,

Sure, I have added the archive containing the image files and jailhouse
build here (the archive contains the /tftpboot folder directly)
https://drive.google.com/file/d/0B2bhbcx8eg-caTBNdnZ4cWhJcWc/view?usp=sharing

I use the following sequence to start qemu / kernel / jailhouse, provided
the /tftpboot folder has the structure in the previously archive and, of
course, there is tftp access to the host)
petalinux-boot --qemu --uboot --qemu-args "-redir tcp:10022:10.0.2.15:22"

In QEMU:
setenv bootargs 'console=ttyPS0,115200'; tftpboot 0x10000000  image.ub;
bootm

tftp -g 10.0.2.2 jailhouse.tar.gz -r jailhouse.tar.gz

tar zvxf jailhouse.tar.gz

mv tftpboot/jailhouse jailhouse

mkdir -p  /lib/firmware

cp ./jailhouse/lib/firmware/jailhouse.bin /lib/firmware

insmod ./jailhouse/lib/modules/4.11.0-rc4/extra/driver/jailhouse.ko

sleep 2

jailhouse/usr/local/sbin/jailhouse enable
/home/root/jailhouse/configs/zynqmp-zcu102.cell


Thank you,

Constantin


On Apr 19, 2017 23:30, "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>
wrote:

> On Wed, Apr 19, 2017 at 11:02:21AM +0200, Jan Kiszka wrote:
> > On 2017-04-19 10:45, Peter Maydell wrote:
> > > On 19 April 2017 at 05:37, Jan Kiszka <jan.kiszka@web.de> wrote:
> > >> On 2017-04-19 05:48, Constantin Petra wrote:
> > >>> So from some reason, under QEMU something is not in place, but what
> can
> > >>> that be? It would be convenient from our perspective, if it would
> work...
> > >>
> > >> I fully agree, and I'm waiting for official EL2 support in QEMU in
> order
> > >> to introduce a virtual target for ARM[64], just like we have on x86
> already.
> > >
> > > This has been in upstream QEMU for a little while now,
> > > and will be in the upcoming 2.9 release. (Applies only for
> > > AArch64 hosts using GICv3, needs some board support changes
> > > which have currently only been done for 'virt'.)
> >
> > Ah, v3 - oversaw this "minor" detail. Too bad that there is still no
> > GICv3 silicon in our hands that we could test against (do you know some
> > recommendable board?). I'm pretty sure Jailhouse's GICv3 support is
> > broken by now (due to lacking tests), plus it is not yet enabled for our
> > ARM64 port. But I'm a bit reluctant to enable this only over QEMU...
>
> Hi,
>
> Xilinx QEMU has support for the virtualization extensions in the GICv2 but
> we're seeing issues with running hypervisors in multi-core setups.
> It seems like guest migration between cores is causing trouble but we
> haven't pinpointed the exact cause.
>
> I'd be interested in debugging the problem seen with Jailhouse.
> Can the images and instructions on howto run be shared with me?
>
> Thanks,
> Edgar
>

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Constantin Petra]

OK,

Changed configs for inmate to UART0, and the messages appeared on the
console (on the ZCU102 board, not QEMU), so it is indeed functional.
root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell start
ZynqMP-linux-demo
Started cell "ZynqMP-linux-demo"
Hello 1 from cell!
Hello 2 from cell!
Hello 3 from cell!
---
etc

Edgar, are the images I have sent of any help?
Let me know if you need any other information about the setup.

Best Regards,
Constantin
P.S. Sent the above previously by chance only to Edgar, reposted again for
the larger audience :)

On Thu, Apr 20, 2017 at 11:19 AM, Constantin Petra <
constantin.petra@gmail.com> wrote:

> Hi,
>
> Sure, I have added the archive containing the image files and jailhouse
> build here (the archive contains the /tftpboot folder directly)
> https://drive.google.com/file/d/0B2bhbcx8eg-caTBNdnZ4cWhJcWc/view?usp=
> sharing
>
> I use the following sequence to start qemu / kernel / jailhouse, provided
> the /tftpboot folder has the structure in the previously archive and, of
> course, there is tftp access to the host)
> petalinux-boot --qemu --uboot --qemu-args "-redir tcp:10022:10.0.2.15:22"
>
> In QEMU:
> setenv bootargs 'console=ttyPS0,115200'; tftpboot 0x10000000  image.ub;
> bootm
>
> tftp -g 10.0.2.2 jailhouse.tar.gz -r jailhouse.tar.gz
>
> tar zvxf jailhouse.tar.gz
>
> mv tftpboot/jailhouse jailhouse
>
> mkdir -p  /lib/firmware
>
> cp ./jailhouse/lib/firmware/jailhouse.bin /lib/firmware
>
> insmod ./jailhouse/lib/modules/4.11.0-rc4/extra/driver/jailhouse.ko
>
> sleep 2
>
> jailhouse/usr/local/sbin/jailhouse enable /home/root/jailhouse/configs/
> zynqmp-zcu102.cell
>
>
> Thank you,
>
> Constantin
>
>
> On Apr 19, 2017 23:30, "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>
> wrote:
>
>> On Wed, Apr 19, 2017 at 11:02:21AM +0200, Jan Kiszka wrote:
>> > On 2017-04-19 10:45, Peter Maydell wrote:
>> > > On 19 April 2017 at 05:37, Jan Kiszka <jan.kiszka@web.de> wrote:
>> > >> On 2017-04-19 05:48, Constantin Petra wrote:
>> > >>> So from some reason, under QEMU something is not in place, but what
>> can
>> > >>> that be? It would be convenient from our perspective, if it would
>> work...
>> > >>
>> > >> I fully agree, and I'm waiting for official EL2 support in QEMU in
>> order
>> > >> to introduce a virtual target for ARM[64], just like we have on x86
>> already.
>> > >
>> > > This has been in upstream QEMU for a little while now,
>> > > and will be in the upcoming 2.9 release. (Applies only for
>> > > AArch64 hosts using GICv3, needs some board support changes
>> > > which have currently only been done for 'virt'.)
>> >
>> > Ah, v3 - oversaw this "minor" detail. Too bad that there is still no
>> > GICv3 silicon in our hands that we could test against (do you know some
>> > recommendable board?). I'm pretty sure Jailhouse's GICv3 support is
>> > broken by now (due to lacking tests), plus it is not yet enabled for our
>> > ARM64 port. But I'm a bit reluctant to enable this only over QEMU...
>>
>> Hi,
>>
>> Xilinx QEMU has support for the virtualization extensions in the GICv2 but
>> we're seeing issues with running hypervisors in multi-core setups.
>> It seems like guest migration between cores is causing trouble but we
>> haven't pinpointed the exact cause.
>>
>> I'd be interested in debugging the problem seen with Jailhouse.
>> Can the images and instructions on howto run be shared with me?
>>
>> Thanks,
>> Edgar
>>
>

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Edgar E. Iglesias]

On Fri, Apr 21, 2017 at 10:04:56AM +0300, Constantin Petra wrote:
> OK,
> 
> Changed configs for inmate to UART0, and the messages appeared on the
> console (on the ZCU102 board, not QEMU), so it is indeed functional.
> root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell start
> ZynqMP-linux-demo
> Started cell "ZynqMP-linux-demo"
> Hello 1 from cell!
> Hello 2 from cell!
> Hello 3 from cell!
> ---
> etc
> 
> Edgar, are the images I have sent of any help?

Thanks Constantin, yes, I can reproduce a lockup.
This is what I see:
root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse enable /home/root/jailhouse/configs/zynqmp-zcu102.cell

Initializing Jailhouse hypervisor v0.6 (47-ge02403c-dirty) on CPU 2
Code location: 0x0000ffffc0200040
Page pool usage after early setup: mem 33/996, remap 64/131072
Initializing processors:
 CPU 2... OK
 CPU 3... OK
 CPU 0... OK
 CPU 1... OK
Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
Page pool usage after late setup: mem 44/996, remap 69/131072
Activating hypervisor
[   73.743265] OF: PCI: host bridge //vpci@0 ranges:
[   73.744161] OF: PCI:   MEM 0xfc100000..0xfc103fff -> 0xfc100000
[   73.747033] pci-host-generic fc000000.vpci: ECAM at [mem 0xfc000000-0xfc0fffff] for [bus 00]
[   73.748659] pci-host-generic fc000000.vpci: PCI host bridge to bus 0001:00
[   73.748972] pci_bus 0001:00: root bus resource [bus 00]
[   73.749644] pci_bus 0001:00: root bus resource [mem 0xfc100000-0xfc103fff]
[   73.758116] pci 0001:00:00.0: BAR 0: assigned [mem 0xfc100000-0xfc1000ff 64bit]
[   73.758928] pci 0001:00:01.0: BAR 0: assigned [mem 0xfc100100-0xfc1001ff 64bit]
[   73.761465] The Jailhouse is opening.
root@plnx_aarch64:~# [   74.710264] ivshmem-net 0001:00:00.0: enabling device (0000 -> 0002)
[   74.713514] ivshmem-net: probe of 0001:00:00.0 failed with error -16
[   74.714971] ivshmem-net 0001:00:01.0: enabling device (0000 -> 0002)
[   74.715904] ivshmem-net: probe of 0001:00:01.0 failed with error -16


After that, it all deadlocks.
I'll see what I can find.

Cheers,
Edgar

> Let me know if you need any other information about the setup.
> 
> Best Regards,
> Constantin
> P.S. Sent the above previously by chance only to Edgar, reposted again for
> the larger audience :)
> 
> On Thu, Apr 20, 2017 at 11:19 AM, Constantin Petra <
> constantin.petra@gmail.com> wrote:
> 
> > Hi,
> >
> > Sure, I have added the archive containing the image files and jailhouse
> > build here (the archive contains the /tftpboot folder directly)
> > https://drive.google.com/file/d/0B2bhbcx8eg-caTBNdnZ4cWhJcWc/view?usp=
> > sharing
> >
> > I use the following sequence to start qemu / kernel / jailhouse, provided
> > the /tftpboot folder has the structure in the previously archive and, of
> > course, there is tftp access to the host)
> > petalinux-boot --qemu --uboot --qemu-args "-redir tcp:10022:10.0.2.15:22"
> >
> > In QEMU:
> > setenv bootargs 'console=ttyPS0,115200'; tftpboot 0x10000000  image.ub;
> > bootm
> >
> > tftp -g 10.0.2.2 jailhouse.tar.gz -r jailhouse.tar.gz
> >
> > tar zvxf jailhouse.tar.gz
> >
> > mv tftpboot/jailhouse jailhouse
> >
> > mkdir -p  /lib/firmware
> >
> > cp ./jailhouse/lib/firmware/jailhouse.bin /lib/firmware
> >
> > insmod ./jailhouse/lib/modules/4.11.0-rc4/extra/driver/jailhouse.ko
> >
> > sleep 2
> >
> > jailhouse/usr/local/sbin/jailhouse enable /home/root/jailhouse/configs/
> > zynqmp-zcu102.cell
> >
> >
> > Thank you,
> >
> > Constantin
> >
> >
> > On Apr 19, 2017 23:30, "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>
> > wrote:
> >
> >> On Wed, Apr 19, 2017 at 11:02:21AM +0200, Jan Kiszka wrote:
> >> > On 2017-04-19 10:45, Peter Maydell wrote:
> >> > > On 19 April 2017 at 05:37, Jan Kiszka <jan.kiszka@web.de> wrote:
> >> > >> On 2017-04-19 05:48, Constantin Petra wrote:
> >> > >>> So from some reason, under QEMU something is not in place, but what
> >> can
> >> > >>> that be? It would be convenient from our perspective, if it would
> >> work...
> >> > >>
> >> > >> I fully agree, and I'm waiting for official EL2 support in QEMU in
> >> order
> >> > >> to introduce a virtual target for ARM[64], just like we have on x86
> >> already.
> >> > >
> >> > > This has been in upstream QEMU for a little while now,
> >> > > and will be in the upcoming 2.9 release. (Applies only for
> >> > > AArch64 hosts using GICv3, needs some board support changes
> >> > > which have currently only been done for 'virt'.)
> >> >
> >> > Ah, v3 - oversaw this "minor" detail. Too bad that there is still no
> >> > GICv3 silicon in our hands that we could test against (do you know some
> >> > recommendable board?). I'm pretty sure Jailhouse's GICv3 support is
> >> > broken by now (due to lacking tests), plus it is not yet enabled for our
> >> > ARM64 port. But I'm a bit reluctant to enable this only over QEMU...
> >>
> >> Hi,
> >>
> >> Xilinx QEMU has support for the virtualization extensions in the GICv2 but
> >> we're seeing issues with running hypervisors in multi-core setups.
> >> It seems like guest migration between cores is causing trouble but we
> >> haven't pinpointed the exact cause.
> >>
> >> I'd be interested in debugging the problem seen with Jailhouse.
> >> Can the images and instructions on howto run be shared with me?
> >>
> >> Thanks,
> >> Edgar
> >>
> >

Re: [Qemu-devel] Jailhouse on Ultrascale+ (ZCU102)

[Constantin Petra]

Hi,

Any findings on this case? Is there any chance PetaLinux 2017.1 addressed this issue?

Thank you,
Constantin


On Friday, April 21, 2017 at 10:23:07 PM UTC+3, Edgar E. Iglesias wrote:
> On Fri, Apr 21, 2017 at 10:04:56AM +0300, Constantin Petra wrote:
> > OK,
> > 
> > Changed configs for inmate to UART0, and the messages appeared on the
> > console (on the ZCU102 board, not QEMU), so it is indeed functional.
> > root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse cell start
> > ZynqMP-linux-demo
> > Started cell "ZynqMP-linux-demo"
> > Hello 1 from cell!
> > Hello 2 from cell!
> > Hello 3 from cell!
> > ---
> > etc
> > 
> > Edgar, are the images I have sent of any help?
> 
> Thanks Constantin, yes, I can reproduce a lockup.
> This is what I see:
> root@plnx_aarch64:~# jailhouse/usr/local/sbin/jailhouse enable /home/root/jailhouse/configs/zynqmp-zcu102.cell
> 
> Initializing Jailhouse hypervisor v0.6 (47-ge02403c-dirty) on CPU 2
> Code location: 0x0000ffffc0200040
> Page pool usage after early setup: mem 33/996, remap 64/131072
> Initializing processors:
>  CPU 2... OK
>  CPU 3... OK
>  CPU 0... OK
>  CPU 1... OK
> Adding virtual PCI device 00:00.0 to cell "ZynqMP-ZCU102"
> Adding virtual PCI device 00:01.0 to cell "ZynqMP-ZCU102"
> Page pool usage after late setup: mem 44/996, remap 69/131072
> Activating hypervisor
> [   73.743265] OF: PCI: host bridge //vpci@0 ranges:
> [   73.744161] OF: PCI:   MEM 0xfc100000..0xfc103fff -> 0xfc100000
> [   73.747033] pci-host-generic fc000000.vpci: ECAM at [mem 0xfc000000-0xfc0fffff] for [bus 00]
> [   73.748659] pci-host-generic fc000000.vpci: PCI host bridge to bus 0001:00
> [   73.748972] pci_bus 0001:00: root bus resource [bus 00]
> [   73.749644] pci_bus 0001:00: root bus resource [mem 0xfc100000-0xfc103fff]
> [   73.758116] pci 0001:00:00.0: BAR 0: assigned [mem 0xfc100000-0xfc1000ff 64bit]
> [   73.758928] pci 0001:00:01.0: BAR 0: assigned [mem 0xfc100100-0xfc1001ff 64bit]
> [   73.761465] The Jailhouse is opening.
> root@plnx_aarch64:~# [   74.710264] ivshmem-net 0001:00:00.0: enabling device (0000 -> 0002)
> [   74.713514] ivshmem-net: probe of 0001:00:00.0 failed with error -16
> [   74.714971] ivshmem-net 0001:00:01.0: enabling device (0000 -> 0002)
> [   74.715904] ivshmem-net: probe of 0001:00:01.0 failed with error -16
> 
> 
> After that, it all deadlocks.
> I'll see what I can find.
> 
> Cheers,
> Edgar
> 
> > Let me know if you need any other information about the setup.
> > 
> > Best Regards,
> > Constantin
> > P.S. Sent the above previously by chance only to Edgar, reposted again for
> > the larger audience :)
> > 
> > On Thu, Apr 20, 2017 at 11:19 AM, Constantin Petra <
> > constantin.petra@gmail.com> wrote:
> > 
> > > Hi,
> > >
> > > Sure, I have added the archive containing the image files and jailhouse
> > > build here (the archive contains the /tftpboot folder directly)
> > > https://drive.google.com/file/d/0B2bhbcx8eg-caTBNdnZ4cWhJcWc/view?usp=
> > > sharing
> > >
> > > I use the following sequence to start qemu / kernel / jailhouse, provided
> > > the /tftpboot folder has the structure in the previously archive and, of
> > > course, there is tftp access to the host)
> > > petalinux-boot --qemu --uboot --qemu-args "-redir tcp:10022:10.0.2.15:22"
> > >
> > > In QEMU:
> > > setenv bootargs 'console=ttyPS0,115200'; tftpboot 0x10000000  image.ub;
> > > bootm
> > >
> > > tftp -g 10.0.2.2 jailhouse.tar.gz -r jailhouse.tar.gz
> > >
> > > tar zvxf jailhouse.tar.gz
> > >
> > > mv tftpboot/jailhouse jailhouse
> > >
> > > mkdir -p  /lib/firmware
> > >
> > > cp ./jailhouse/lib/firmware/jailhouse.bin /lib/firmware
> > >
> > > insmod ./jailhouse/lib/modules/4.11.0-rc4/extra/driver/jailhouse.ko
> > >
> > > sleep 2
> > >
> > > jailhouse/usr/local/sbin/jailhouse enable /home/root/jailhouse/configs/
> > > zynqmp-zcu102.cell
> > >
> > >
> > > Thank you,
> > >
> > > Constantin
> > >
> > >
> > > On Apr 19, 2017 23:30, "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>
> > > wrote:
> > >
> > >> On Wed, Apr 19, 2017 at 11:02:21AM +0200, Jan Kiszka wrote:
> > >> > On 2017-04-19 10:45, Peter Maydell wrote:
> > >> > > On 19 April 2017 at 05:37, Jan Kiszka <jan.kiszka@web.de> wrote:
> > >> > >> On 2017-04-19 05:48, Constantin Petra wrote:
> > >> > >>> So from some reason, under QEMU something is not in place, but what
> > >> can
> > >> > >>> that be? It would be convenient from our perspective, if it would
> > >> work...
> > >> > >>
> > >> > >> I fully agree, and I'm waiting for official EL2 support in QEMU in
> > >> order
> > >> > >> to introduce a virtual target for ARM[64], just like we have on x86
> > >> already.
> > >> > >
> > >> > > This has been in upstream QEMU for a little while now,
> > >> > > and will be in the upcoming 2.9 release. (Applies only for
> > >> > > AArch64 hosts using GICv3, needs some board support changes
> > >> > > which have currently only been done for 'virt'.)
> > >> >
> > >> > Ah, v3 - oversaw this "minor" detail. Too bad that there is still no
> > >> > GICv3 silicon in our hands that we could test against (do you know some
> > >> > recommendable board?). I'm pretty sure Jailhouse's GICv3 support is
> > >> > broken by now (due to lacking tests), plus it is not yet enabled for our
> > >> > ARM64 port. But I'm a bit reluctant to enable this only over QEMU...
> > >>
> > >> Hi,
> > >>
> > >> Xilinx QEMU has support for the virtualization extensions in the GICv2 but
> > >> we're seeing issues with running hypervisors in multi-core setups.
> > >> It seems like guest migration between cores is causing trouble but we
> > >> haven't pinpointed the exact cause.
> > >>
> > >> I'd be interested in debugging the problem seen with Jailhouse.
> > >> Can the images and instructions on howto run be shared with me?
> > >>
> > >> Thanks,
> > >> Edgar
> > >>
> > >