[Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Andrew Baumann]

Hi all,

I have been working on Raspberry Pi 2 emulation, building on the previous work of Gregory Estrade, Stefan Weil and others on the original Raspberry Pi target. My current working tree (based off a recent master) is here:
    https://github.com/0xabu/qemu/tree/raspi

The present status is:
 * The original Raspberry Pi (-M raspi) support works only with older kernels from Raspbian releases up to 2014-09-09, for unknown reasons that I haven't investigated. [1]
 * Pi 2 (-M raspi2) supports both Raspbian and Windows.
 * Raspbian boots on pi2, but fails on an implemented SETEND instruction in early userspace. I'm told there are patches/workarounds for this floating around, but have not tried them.
 * The Windows IoT image [2] boots completely, but the USB emulation is broken, so the only IO devices are framebuffer and serial console (which can be used for a kernel debugger).
 * All four Pi2 cores must be enabled for Windows, since the bootloader panics if the expected cores are not present. Conversely, Raspbian fails to boot if multiple cores are enabled, but I haven't debugged this, since qemu is faster with a single-core guest.

I realise that getting this code integrated will require a series of smaller patches, but wanted to solicit any early feedback before I start doing that. I've verified that checkpatch.pl is happy, but if there are other stylistic or general problems with the code it would be helpful to know about them sooner rather than later :) If anyone has specific suggestions for how to split up the patch, that would also be welcome. If not, I will start submitting patches for the individual pieces, probably starting with the machine definition and then the larger devices, followed by the Windows-specific fixes described below.

There are also a couple of significant (non Pi-specific) changes that I made in order to boot a Windows on ARM guest, which also I plan to submit:
1. UEFI (TianoCore / EDK2) has a couple of bugs in its MMC support -- it assumes support for CMD23 (set multiple block count, which is supposed to be optional), and also fails to initialise correctly if the card is ready immediately upon issuing ACMD41. My tree includes workarounds for both issues -- I implemented CMD23, and modelled a small delay when initialising the SD card.
2. For better or worse, Windows relies on taking alignment faults on a misaligned LDREX, however qemu doesn't perform any alignment checks, leading to a bluescreen very early in boot. The changes in target-arm implement a generic infrastructure for testing/raising alignment exceptions, and add a suitable check to LDREX. (This is almost certainly applicable to arm64 as well, but I don't have a means to test it.)

Here's the current diffstat (ignore the minor unrelated bugfixes in lan9118 and tap-win32):
 default-configs/arm-softmmu.mak      |    1 +
 hw/arm/Makefile.objs                 |    1 +
 hw/arm/raspi.c                       |  514 ++++++++++++++++
 hw/char/Makefile.objs                |    1 +
 hw/char/bcm2835_aux.c                |  250 ++++++++
 hw/display/Makefile.objs             |    1 +
 hw/display/bcm2835_fb.c              |  384 ++++++++++++
 hw/dma/Makefile.objs                 |    1 +
 hw/dma/bcm2835_dma.c                 |  352 +++++++++++
 hw/intc/Makefile.objs                |    1 +
 hw/intc/bcm2835_ic.c                 |  248 ++++++++
 hw/intc/bcm2836_control.c            |  373 ++++++++++++
 hw/misc/Makefile.objs                |    5 +
 hw/misc/bcm2835_mphi.c               |  176 ++++++
 hw/misc/bcm2835_power.c              |  113 ++++
 hw/misc/bcm2835_property.c           |  409 +++++++++++++
 hw/misc/bcm2835_sbm.c                |  294 ++++++++++
 hw/misc/bcm2835_vchiq.c              |  113 ++++
 hw/net/lan9118.c                     |   18 +-
 hw/sd/Makefile.objs                  |    1 +
 hw/sd/bcm2835_emmc.c                 |  844 +++++++++++++++++++++++++++
 hw/sd/sd.c                           |   78 ++-
 hw/timer/Makefile.objs               |    2 +
 hw/timer/arm_timer.c                 |   39 ++
 hw/timer/bcm2835_st.c                |  201 +++++++
 hw/timer/bcm2835_timer.c             |  242 ++++++++
 hw/usb/Makefile.objs                 |    2 +
 hw/usb/bcm2835_usb.c                 |  655 +++++++++++++++++++++
 hw/usb/bcm2835_usb_regs.h            | 1061 ++++++++++++++++++++++++++++++++++
 include/hw/arm/bcm2835_arm_control.h |  481 +++++++++++++++
 include/hw/arm/bcm2835_common.h      |   35 ++
 include/hw/arm/raspi_platform.h      |  155 +++++
 net/tap-win32.c                      |   51 +-
 target-arm/helper.c                  |    8 +
 target-arm/helper.h                  |    1 +
 target-arm/internals.h               |    3 +
 target-arm/op_helper.c               |   21 +
 target-arm/translate.c               |   29 +
 38 files changed, 7141 insertions(+), 23 deletions(-)

The command line I've used for Raspbian is:
qemu-system-arm -M raspi2 -m 1024 -kernel kernel7.img -sd 2015-09-24-raspbian-jessie.img -append "rw earlyprintk loglevel=8 bcm2708_fb.fbwidth=1024 bcm2708_fb.fbheight=768 bcm2708.boardrev=0xf bcm2708.serial=0xcad0eedf vc_mem.mem_base=0x1c000000 vc_mem.mem_size=0x20000000 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2"

The command line for Windows is:
qemu-system-arm -M raspi2 -smp 4 -m 1024 -bios kernel.img -sd th2preview_pi2_iot.vhd
(where kernel.img is the EDK2 bootloader from the first partition of the Windows image).

Cheers,
Andrew

[1] https://www.raspberrypi.org/forums/viewtopic.php?f=72&t=26561&start=125
[2] http://ms-iot.github.io/content/en-US/Downloads.htm

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Stefan Weil]

Am 16.11.2015 um 23:22 schrieb Andrew Baumann:
> Hi all,
> 
> I have been working on Raspberry Pi 2 emulation, building on the previous work of Gregory Estrade, Stefan Weil and others on the original Raspberry Pi target. My current working tree (based off a recent master) is here:
>     https://github.com/0xabu/qemu/tree/raspi
> 
> The present status is:
>  * The original Raspberry Pi (-M raspi) support works only with older kernels from Raspbian releases up to 2014-09-09, for unknown reasons that I haven't investigated. [1]
>  * Pi 2 (-M raspi2) supports both Raspbian and Windows.
>  * Raspbian boots on pi2, but fails on an implemented SETEND instruction in early userspace. I'm told there are patches/workarounds for this floating around, but have not tried them.
>  * The Windows IoT image [2] boots completely, but the USB emulation is broken, so the only IO devices are framebuffer and serial console (which can be used for a kernel debugger).
>  * All four Pi2 cores must be enabled for Windows, since the bootloader panics if the expected cores are not present. Conversely, Raspbian fails to boot if multiple cores are enabled, but I haven't debugged this, since qemu is faster with a single-core guest.
> 
> I realise that getting this code integrated will require a series of smaller patches, but wanted to solicit any early feedback before I start doing that. I've verified that checkpatch.pl is happy, but if there are other stylistic or general problems with the code it would be helpful to know about them sooner rather than later :) If anyone has specific suggestions for how to split up the patch, that would also be welcome. If not, I will start submitting patches for the individual pieces, probably starting with the machine definition and then the larger devices, followed by the Windows-specific fixes described below.
> 
> There are also a couple of significant (non Pi-specific) changes that I made in order to boot a Windows on ARM guest, which also I plan to submit:
> 1. UEFI (TianoCore / EDK2) has a couple of bugs in its MMC support -- it assumes support for CMD23 (set multiple block count, which is supposed to be optional), and also fails to initialise correctly if the card is ready immediately upon issuing ACMD41. My tree includes workarounds for both issues -- I implemented CMD23, and modelled a small delay when initialising the SD card.
> 2. For better or worse, Windows relies on taking alignment faults on a misaligned LDREX, however qemu doesn't perform any alignment checks, leading to a bluescreen very early in boot. The changes in target-arm implement a generic infrastructure for testing/raising alignment exceptions, and add a suitable check to LDREX. (This is almost certainly applicable to arm64 as well, but I don't have a means to test it.)
> 
> Here's the current diffstat (ignore the minor unrelated bugfixes in lan9118 and tap-win32):
>  default-configs/arm-softmmu.mak      |    1 +
>  hw/arm/Makefile.objs                 |    1 +
>  hw/arm/raspi.c                       |  514 ++++++++++++++++
>  hw/char/Makefile.objs                |    1 +
>  hw/char/bcm2835_aux.c                |  250 ++++++++
>  hw/display/Makefile.objs             |    1 +
>  hw/display/bcm2835_fb.c              |  384 ++++++++++++
>  hw/dma/Makefile.objs                 |    1 +
>  hw/dma/bcm2835_dma.c                 |  352 +++++++++++
>  hw/intc/Makefile.objs                |    1 +
>  hw/intc/bcm2835_ic.c                 |  248 ++++++++
>  hw/intc/bcm2836_control.c            |  373 ++++++++++++
>  hw/misc/Makefile.objs                |    5 +
>  hw/misc/bcm2835_mphi.c               |  176 ++++++
>  hw/misc/bcm2835_power.c              |  113 ++++
>  hw/misc/bcm2835_property.c           |  409 +++++++++++++
>  hw/misc/bcm2835_sbm.c                |  294 ++++++++++
>  hw/misc/bcm2835_vchiq.c              |  113 ++++
>  hw/net/lan9118.c                     |   18 +-
>  hw/sd/Makefile.objs                  |    1 +
>  hw/sd/bcm2835_emmc.c                 |  844 +++++++++++++++++++++++++++
>  hw/sd/sd.c                           |   78 ++-
>  hw/timer/Makefile.objs               |    2 +
>  hw/timer/arm_timer.c                 |   39 ++
>  hw/timer/bcm2835_st.c                |  201 +++++++
>  hw/timer/bcm2835_timer.c             |  242 ++++++++
>  hw/usb/Makefile.objs                 |    2 +
>  hw/usb/bcm2835_usb.c                 |  655 +++++++++++++++++++++
>  hw/usb/bcm2835_usb_regs.h            | 1061 ++++++++++++++++++++++++++++++++++
>  include/hw/arm/bcm2835_arm_control.h |  481 +++++++++++++++
>  include/hw/arm/bcm2835_common.h      |   35 ++
>  include/hw/arm/raspi_platform.h      |  155 +++++
>  net/tap-win32.c                      |   51 +-
>  target-arm/helper.c                  |    8 +
>  target-arm/helper.h                  |    1 +
>  target-arm/internals.h               |    3 +
>  target-arm/op_helper.c               |   21 +
>  target-arm/translate.c               |   29 +
>  38 files changed, 7141 insertions(+), 23 deletions(-)
> 
> The command line I've used for Raspbian is:
> qemu-system-arm -M raspi2 -m 1024 -kernel kernel7.img -sd 2015-09-24-raspbian-jessie.img -append "rw earlyprintk loglevel=8 bcm2708_fb.fbwidth=1024 bcm2708_fb.fbheight=768 bcm2708.boardrev=0xf bcm2708.serial=0xcad0eedf vc_mem.mem_base=0x1c000000 vc_mem.mem_size=0x20000000 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2"
> 
> The command line for Windows is:
> qemu-system-arm -M raspi2 -smp 4 -m 1024 -bios kernel.img -sd th2preview_pi2_iot.vhd
> (where kernel.img is the EDK2 bootloader from the first partition of the Windows image).
> 
> Cheers,
> Andrew
> 
> [1] https://www.raspberrypi.org/forums/viewtopic.php?f=72&t=26561&start=125
> [2] http://ms-iot.github.io/content/en-US/Downloads.htm
> 


Hi Andrew,

thanks for this great overview of your activities regarding QEMU with
Raspberry Pi 2 support.

The QEMU development is currently preparing the new version 2.5 (see
http://wiki.qemu.org/Planning/2.5, http://wiki.qemu.org/ChangeLog/2.5).
This should be finished until 2015-12-10 (maybe a little later).
New patches for RPi can be discussed earlier, but won't be applied
to Git master before version 2.5 is finished.

I wish you good luck and hope RPi will be available to all QEMU users
soon. This target should be added to http://wiki.qemu.org/Planning/2.6
(or will it be http://wiki.qemu.org/Planning/3.0?), so nobody forgets
it. :-)

Kind regards,
Stefan

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Peter Crosthwaite]

On Mon, Nov 16, 2015 at 2:22 PM, Andrew Baumann
<Andrew.Baumann@microsoft.com> wrote:
> Hi all,
>
> I have been working on Raspberry Pi 2 emulation, building on the previous work of Gregory Estrade, Stefan Weil and others on the original Raspberry Pi target. My current working tree (based off a recent master) is here:
>     https://github.com/0xabu/qemu/tree/raspi
>
> The present status is:
>  * The original Raspberry Pi (-M raspi) support works only with older kernels from Raspbian releases up to 2014-09-09, for unknown reasons that I haven't investigated. [1]
>  * Pi 2 (-M raspi2) supports both Raspbian and Windows.
>  * Raspbian boots on pi2, but fails on an implemented SETEND instruction in early userspace. I'm told there are patches/workarounds for this floating around, but have not tried them.
>  * The Windows IoT image [2] boots completely, but the USB emulation is broken, so the only IO devices are framebuffer and serial console (which can be used for a kernel debugger).
>  * All four Pi2 cores must be enabled for Windows, since the bootloader panics if the expected cores are not present. Conversely, Raspbian fails to boot if multiple cores are enabled, but I haven't debugged this, since qemu is faster with a single-core guest.
>
> I realise that getting this code integrated will require a series of smaller patches, but wanted to solicit any early feedback before I start doing that. I've verified that checkpatch.pl is happy, but if there are other stylistic or general problems with the code it would be helpful to know about them sooner rather than later :) If anyone has specific suggestions for how to split up the patch, that would also be welcome. If not, I will start submitting patches for the individual pieces, probably starting with the machine definition and then the larger devices, followed by the Windows-specific fixes described below.
>

For splitting, there are some good examples in any of the recent ARM
SoC+Board bringups. The most recent full bringup from scratch would
probably be the STM/Netduino board. Myself and Jean-Christophe added
Xilinx ZynqMP and i.MX25 respectively, although in both those works we
used existing peripheral IP (with patches) rather than fresh.

https://lists.nongnu.org/archive/html/qemu-devel/2015-02/msg03398.html
https://lists.nongnu.org/archive/html/qemu-devel/2015-08/msg01194.html
https://lists.nongnu.org/archive/html/qemu-devel/2015-03/msg04683.html

I haven't looked beyond the diffstat yet, but a top level
architectural comment, I only see the one file in hw/arm. We are
promoting the split of board and SoC now as two separate objects. Each
of the patch series linked above demonstrates this. The BCM SoC would
be an object in its own right, then the board instantiates the SoC as
a composite device.

> There are also a couple of significant (non Pi-specific) changes that I made in order to boot a Windows on ARM guest, which also I plan to submit:
> 1. UEFI (TianoCore / EDK2) has a couple of bugs in its MMC support -- it assumes support for CMD23 (set multiple block count, which is supposed to be optional), and also fails to initialise correctly if the card is ready immediately upon issuing ACMD41. My tree includes workarounds for both issues -- I implemented CMD23, and modelled a small delay when initialising the SD card.
> 2. For better or worse, Windows relies on taking alignment faults on a misaligned LDREX, however qemu doesn't perform any alignment checks, leading to a bluescreen very early in boot. The changes in target-arm implement a generic infrastructure for testing/raising alignment exceptions, and add a suitable check to LDREX. (This is almost certainly applicable to arm64 as well, but I don't have a means to test it.)
>

I would send that as a single, and you would be able to do that
without any thing else as prerequisite.

> Here's the current diffstat (ignore the minor unrelated bugfixes in lan9118 and tap-win32):
>  default-configs/arm-softmmu.mak      |    1 +
>  hw/arm/Makefile.objs                 |    1 +
>  hw/arm/raspi.c                       |  514 ++++++++++++++++
>  hw/char/Makefile.objs                |    1 +
>  hw/char/bcm2835_aux.c                |  250 ++++++++
>  hw/display/Makefile.objs             |    1 +
>  hw/display/bcm2835_fb.c              |  384 ++++++++++++
>  hw/dma/Makefile.objs                 |    1 +
>  hw/dma/bcm2835_dma.c                 |  352 +++++++++++

New devices should have the state struct in a header file, the
referenced work gives an example on this.

>  hw/intc/Makefile.objs                |    1 +
>  hw/intc/bcm2835_ic.c                 |  248 ++++++++
>  hw/intc/bcm2836_control.c            |  373 ++++++++++++
>  hw/misc/Makefile.objs                |    5 +
>  hw/misc/bcm2835_mphi.c               |  176 ++++++
>  hw/misc/bcm2835_power.c              |  113 ++++
>  hw/misc/bcm2835_property.c           |  409 +++++++++++++
>  hw/misc/bcm2835_sbm.c                |  294 ++++++++++
>  hw/misc/bcm2835_vchiq.c              |  113 ++++
>  hw/net/lan9118.c                     |   18 +-
>  hw/sd/Makefile.objs                  |    1 +
>  hw/sd/bcm2835_emmc.c                 |  844 +++++++++++++++++++++++++++
>  hw/sd/sd.c                           |   78 ++-
>  hw/timer/Makefile.objs               |    2 +
>  hw/timer/arm_timer.c                 |   39 ++
>  hw/timer/bcm2835_st.c                |  201 +++++++
>  hw/timer/bcm2835_timer.c             |  242 ++++++++
>  hw/usb/Makefile.objs                 |    2 +
>  hw/usb/bcm2835_usb.c                 |  655 +++++++++++++++++++++
>  hw/usb/bcm2835_usb_regs.h            | 1061 ++++++++++++++++++++++++++++++++++
>  include/hw/arm/bcm2835_arm_control.h |  481 +++++++++++++++
>  include/hw/arm/bcm2835_common.h      |   35 ++
>  include/hw/arm/raspi_platform.h      |  155 +++++
>  net/tap-win32.c                      |   51 +-
>  target-arm/helper.c                  |    8 +
>  target-arm/helper.h                  |    1 +
>  target-arm/internals.h               |    3 +
>  target-arm/op_helper.c               |   21 +
>  target-arm/translate.c               |   29 +
>  38 files changed, 7141 insertions(+), 23 deletions(-)

It is big. Work of this magnitude is generally better managed as
multiple series (each of multiple patches). A general approach with
new ARM boards is the first series is the bare minimum needed to get a
linux boot over the UART console. This usually means any CPU
customisations, timer, UART, intc, SoC and boards, maybe the
system/power controllers (if the kernel touches them in boot) and
optionally network or emmc. With network you can do an NFS root (this
was the only option on Allwinner/cubieboard for quite some time) and
with emmc you can do your root=/dev/mmcblk0p2 style boot as normal. If
you are using existing lan9118, network first might be a clear win,
although emmc is more realistic. Can always do both.

The you can follow up with the USB, DMA, AUX, ... each independently
or as a follow up series'.

Any core code and ARM CPU work that is a bugfix or stands in its own
right, you should send straight up as a separate work.

>
> The command line I've used for Raspbian is:
> qemu-system-arm -M raspi2 -m 1024 -kernel kernel7.img -sd 2015-09-24-raspbian-jessie.img -append "rw earlyprintk loglevel=8 bcm2708_fb.fbwidth=1024 bcm2708_fb.fbheight=768 bcm2708.boardrev=0xf bcm2708.serial=0xcad0eedf vc_mem.mem_base=0x1c000000 vc_mem.mem_size=0x20000000 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2"
>

Curious, does rPi kernel support device tree boot? This would be nice
for testing with a generic kernel.

Regards,
Peter

> The command line for Windows is:
> qemu-system-arm -M raspi2 -smp 4 -m 1024 -bios kernel.img -sd th2preview_pi2_iot.vhd
> (where kernel.img is the EDK2 bootloader from the first partition of the Windows image).
>
> Cheers,
> Andrew
>
> [1] https://www.raspberrypi.org/forums/viewtopic.php?f=72&t=26561&start=125
> [2] http://ms-iot.github.io/content/en-US/Downloads.htm
>

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Andrew Baumann]

Hi Peter,

Thanks for your feedback!

From: Peter Crosthwaite [mailto:crosthwaitepeter@gmail.com]
Sent: Tuesday, 17 November 2015 23:51
> I haven't looked beyond the diffstat yet, but a top level
> architectural comment, I only see the one file in hw/arm. We are
> promoting the split of board and SoC now as two separate objects. Each
> of the patch series linked above demonstrates this. The BCM SoC would
> be an object in its own right, then the board instantiates the SoC as
> a composite device.

I understand the rationale here, but is there much value pursuing this for the Pi? As far as I know, the Pi1/2 SOCs (bcm2835 and 2836) are one-off designs used only in the Pi. Moreover, the 2836 is basically a copy of the 2835 with the addition of a new block for a multi-core interrupt controller and mailboxes, so trying to split those out into two separate SOC definition files would require either lots of duplication, or a tight coupling between the two. Finally, there is basically nothing on the board besides the SoC and RAM (just a USB-Ethernet adapter), so the actual board definition files would be almost empty. As it is, I've already factored out all the common devices within hw/arm/raspi.c -- my feeling is that this is a cleaner way to do it for the Pi.

> It is big. Work of this magnitude is generally better managed as
> multiple series (each of multiple patches). A general approach with
> new ARM boards is the first series is the bare minimum needed to get a
> linux boot over the UART console. This usually means any CPU
> customisations, timer, UART, intc, SoC and boards, maybe the
> system/power controllers (if the kernel touches them in boot) and
> optionally network or emmc. With network you can do an NFS root (this
> was the only option on Allwinner/cubieboard for quite some time) and
> with emmc you can do your root=/dev/mmcblk0p2 style boot as normal. If
> you are using existing lan9118, network first might be a clear win,
> although emmc is more realistic. Can always do both.
> 
> The you can follow up with the USB, DMA, AUX, ... each independently
> or as a follow up series'.
> 
> Any core code and ARM CPU work that is a bugfix or stands in its own
> right, you should send straight up as a separate work.

Ok, understood. Even following this guidance, the first patch will be big -- Pi has custom interrupt and DMA controllers, and the GPU gets itself involved in the boot process. I think we could probably leave out fb/usb and maybe also dma, mphi, vchiq and power, but most of the rest will be needed to boot anything.

BTW, I used lan9118 as a kludge for Windows, but it isn't present on a real Pi, and the device definition no longer instantiates it. At present, there is no usable network device.

> Curious, does rPi kernel support device tree boot? This would be nice
> for testing with a generic kernel.

I believe it does on recent Raspbian images (I see a number of dtb files in Raspbian's boot partition), but I haven't tried to figure out how to use them. In general, we have limitations where the bootloader is concerned. On Pi HW, the boot process is driven by the GPU, which reads a plethora of options from config files (config.txt and cmdline.txt) on the boot partition. We don't emulate any of that stuff... instead, the previous raspi code did just enough to boot Linux using qemu's Linux loader support, and I figured out how to make UEFI happy on raspi2 so it can boot Windows, but that's about all.

I will take a look at the patches you referred to, and try to refactor the devices accordingly.

BTW, my main goal here is Windows support (surprise! :), so if anyone else is willing to help out with Linux that would be much appreciated. The kernel currently limps into early userspace before crashing on a SETEND instruction, and is clearly not as happy as it was when the original Pi1 support was being actively developed ~2 years ago.

Thanks,
Andrew

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Peter Crosthwaite]

On Wed, Nov 18, 2015 at 10:16 AM, Andrew Baumann
<Andrew.Baumann@microsoft.com> wrote:
> Hi Peter,
>
> Thanks for your feedback!
>
> From: Peter Crosthwaite [mailto:crosthwaitepeter@gmail.com]
> Sent: Tuesday, 17 November 2015 23:51
>> I haven't looked beyond the diffstat yet, but a top level
>> architectural comment, I only see the one file in hw/arm. We are
>> promoting the split of board and SoC now as two separate objects. Each
>> of the patch series linked above demonstrates this. The BCM SoC would
>> be an object in its own right, then the board instantiates the SoC as
>> a composite device.
>
> I understand the rationale here, but is there much value pursuing this for the Pi? As far as I know, the Pi1/2 SOCs (bcm2835 and 2836) are one-off designs used only in the Pi. Moreover, the 2836 is basically a copy of the 2835 with the addition of a new block for a multi-core interrupt controller and mailboxes, so trying to split those out into two separate SOC definition files would require either lots of duplication, or a tight coupling between the two.

Duplication is not right. You can code share by either having a QOM
inheritance between the two SoCs (the QOM object oriented model allows
this) from a common abstraction, or a QOM property that drives the
needed iffery. Even if the board is thin, I would still do it split,
and several of the in-tree machine models are like this. If anyone
wants to model the lower level stuff like the power supplies or play
with the GPIOs etc there some framework there. If the board ever gets
cloned and modded then the SoC is ready to go.

> Finally, there is basically nothing on the board besides the SoC and RAM (just a USB-Ethernet adapter), so the actual board definition files would be almost empty.

USB-Ethernet is non trivial though isn't it? I'm looking at the
schematic and my gut is the lan9152 warrants a board/SoC split.

> As it is, I've already factored out all the common devices within hw/arm/raspi.c -- my feeling is that this is a cleaner way to do it for the Pi.
>

You will probably find that the way to do it is covert your board as
it stands to SoC, rebadging rPi as BCM SoC then put a thin rPI board
around it that sets just the RAM initially.

>> It is big. Work of this magnitude is generally better managed as
>> multiple series (each of multiple patches). A general approach with
>> new ARM boards is the first series is the bare minimum needed to get a
>> linux boot over the UART console. This usually means any CPU
>> customisations, timer, UART, intc, SoC and boards, maybe the
>> system/power controllers (if the kernel touches them in boot) and
>> optionally network or emmc. With network you can do an NFS root (this
>> was the only option on Allwinner/cubieboard for quite some time) and
>> with emmc you can do your root=/dev/mmcblk0p2 style boot as normal. If
>> you are using existing lan9118, network first might be a clear win,
>> although emmc is more realistic. Can always do both.
>>
>> The you can follow up with the USB, DMA, AUX, ... each independently
>> or as a follow up series'.
>>
>> Any core code and ARM CPU work that is a bugfix or stands in its own
>> right, you should send straight up as a separate work.
>
> Ok, understood. Even following this guidance, the first patch will be big -- Pi has custom interrupt and DMA controllers, and the GPU gets itself involved in the boot process. I think we could probably leave out fb/usb and maybe also dma, mphi, vchiq and power, but most of the rest will be needed to boot anything.
>
> BTW, I used lan9118 as a kludge for Windows, but it isn't present on a real Pi, and the device definition no longer instantiates it. At present, there is no usable network device.
>

Ok I guess just start with emmc. If that diff to sd.c is for emmc card
support (currently missing), I would start early and send that
straight away.

>> Curious, does rPi kernel support device tree boot? This would be nice
>> for testing with a generic kernel.
>
> I believe it does on recent Raspbian images (I see a number of dtb files in Raspbian's boot partition), but I haven't tried to figure out how to use them. In general, we have limitations where the bootloader is concerned. On Pi HW, the boot process is driven by the GPU, which reads a plethora of options from config files (config.txt and cmdline.txt) on the boot partition. We don't emulate any of that stuff... instead, the previous raspi code did just enough to boot Linux using qemu's Linux loader support, and I figured out how to make UEFI happy on raspi2 so it can boot Windows, but that's about all.
>
> I will take a look at the patches you referred to, and try to refactor the devices accordingly.
>
> BTW, my main goal here is Windows support (surprise! :), so if anyone else is willing to help out with Linux that would be much appreciated. The kernel currently limps into early userspace before crashing on a SETEND instruction, and is clearly not as happy as it was when the original Pi1 support was being actively developed ~2 years ago.
>

Yeh I think Paolo tried to fix that with the ARM BE support. I
reworked that recently but for v7+ ARM, so I didn't have the BE32
support anymore (I had no test cases). That BE work really needs to go
up as you are at least the 4th person to need it.

Regards,
Peter

> Thanks,
> Andrew

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Andrew Baumann]

Hi Peter (et al),

I am working on refactoring the Pi support code as you suggested. I have split the Pi SOCs into separate objects (bcm2835 and bcm2836) which both instantiate a third common bcm2835_peripherals device that in turn contains all the common devices. I have also switched the code to use object properties rather than global variables to communicate state where devices interact with each other.

Before I go further, I wanted to ask a couple of high-level questions about the desired code style for new devices.

1. Must every new device have a header file that declares its private state struct, or is it ok to keep those in the C file for leaf devices and refer to them through opaque pointers?
2. What's the status of the sysbus API? It is obviously a fairly thin layer over qdev/qom, but it's not clear whether the APIs are deprecated. Can new code instantiate children of TYPE_SYS_BUS_DEVICE and use the irq/mmio export features it provides, or should we be sticking only to lower-level (qdev?) APIs? If so, what's the right way to export interrupts and mmio space from a device?
3. Similarly, is using SysBusDeviceClass.init ok, or should we be using DeviceClass.realize? In general, what's the guidance on what belongs in object init vs. the realize method?
4. I have some property values (e.g. video ram size) that are set in the top-level machine definition code (raspi.c), and need to pass down through several layers of abstraction (e.g. raspi.c declares the machine class which instantiates a bcm2836 object which in turn instantiates bcm2835_peripherals, which in turn instantiates bcm2835_fb). Defining properties at each level just to pass these along is painful/fragile. Is there a programmatic way to have properties that are defined globally and passed through, roughly equivalent to -global on the command line? Is it acceptable to use that?

Forgive me if there is documentation/guidance on this stuff already -- I didn't manage to find very much.

If anyone wants to take a look, my working tree is at:
https://github.com/0xabu/qemu/tree/raspi

The relevant files are:
 hw/arm/bcm2835.c                     |   92 +++
 hw/arm/bcm2835_peripherals.c         |  368 ++++++++++++
 hw/arm/bcm2836.c                     |  149 +++++
 hw/arm/raspi.c                       |  228 ++++++++
 hw/char/bcm2835_aux.c                |  250 ++++++++
 hw/display/bcm2835_fb.c              |  427 ++++++++++++++
 hw/dma/bcm2835_dma.c                 |  366 ++++++++++++
 hw/intc/bcm2835_ic.c                 |  248 ++++++++
 hw/intc/bcm2836_control.c            |  373 ++++++++++++
 hw/misc/bcm2835_mphi.c               |  176 ++++++
 hw/misc/bcm2835_power.c              |  113 ++++
 hw/misc/bcm2835_property.c           |  417 +++++++++++++
 hw/misc/bcm2835_sbm.c                |  310 ++++++++++
 hw/misc/bcm2835_vchiq.c              |  114 ++++
 hw/sd/bcm2835_emmc.c                 |  844 +++++++++++++++++++++++++++
 hw/timer/arm_timer.c                 |   39 ++
 hw/timer/bcm2835_st.c                |  201 +++++++
 hw/timer/bcm2835_timer.c             |  242 ++++++++
 hw/usb/bcm2835_usb.c                 |  655 +++++++++++++++++++++
 hw/usb/bcm2835_usb_regs.h            | 1061 ++++++++++++++++++++++++++++++++++
 include/hw/arm/bcm2835.h             |   31 +
 include/hw/arm/bcm2835_arm_control.h |  481 +++++++++++++++
 include/hw/arm/bcm2835_mbox.h        |   14 +
 include/hw/arm/bcm2835_peripherals.h |   35 ++
 include/hw/arm/bcm2836.h             |   34 ++
 include/hw/arm/raspi_platform.h      |  160 +++++
 include/hw/display/bcm2835_fb.h      |   29 +

Thanks,
Andrew

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Peter Crosthwaite]

On Tue, Nov 24, 2015 at 4:00 PM, Andrew Baumann
<Andrew.Baumann@microsoft.com> wrote:
> Hi Peter (et al),
>
> I am working on refactoring the Pi support code as you suggested. I have split the Pi SOCs into separate objects (bcm2835 and bcm2836) which both instantiate a third common bcm2835_peripherals device that in turn contains all the common devices. I have also switched the code to use object properties rather than global variables to communicate state where devices interact with each other.
>

Should this be an inheritance of a common SoC rather than an
instantiation of a common container?

> Before I go further, I wanted to ask a couple of high-level questions about the desired code style for new devices.
>
> 1. Must every new device have a header file that declares its private state struct, or is it ok to keep those in the C file for leaf devices and refer to them through opaque pointers?

Headers. The idea is every device state is inline embed-able in a
container objects struct (requiring the header). The plan was to have
some checking mechanisms to ensure private fields remain private
implementing C file.

> 2. What's the status of the sysbus API? It is obviously a fairly thin layer over qdev/qom, but it's not clear whether the APIs are deprecated. Can new code instantiate children of TYPE_SYS_BUS_DEVICE and use the irq/mmio export features it provides, or should we be sticking only to lower-level (qdev?) APIs? If so, what's the right way to export interrupts and mmio space from a device?

Sysbus is fine to subclass and use for both MMIO and IRQs. Use of
sysbus IRQs should be limited to genuine interrupts. If there are
other GPIOs going around between peripherals, then named GPIOs are the
go. You can use unnamed GPIOs if there is a single homogenous set of
GPIOs (this is common for the input to interrupt controllers) but
anything more complex should go the named GPIOs.

> 3. Similarly, is using SysBusDeviceClass.init ok, or should we be using DeviceClass.realize?

That part of sysbus is deprecated. Use .realize and the instance_init.

>In general, what's the guidance on what belongs in object init vs. the realize method?

If it doesn't depend on property values, it goes in init. If it does
realize. Top level connectivity (e.g. connecting to UARTs are initing
drives) generally are realize.

> 4. I have some property values (e.g. video ram size) that are set in the top-level machine definition code (raspi.c), and need to pass down through several layers of abstraction (e.g. raspi.c declares the machine class which instantiates a bcm2836 object which in turn instantiates bcm2835_peripherals, which in turn instantiates bcm2835_fb).
> Defining properties at each level just to pass these along is painful/fragile. Is there a programmatic way to have properties that are defined globally and passed through, roughly equivalent to -global on the command line?

Yes. Alias properties. The SoC can alias a property of an IP. I had
patches on list to implement overloaded aliases if you are interested
in wrangling multiple properties together on the container level.

https://lists.nongnu.org/archive/html/qemu-devel/2015-06/msg03624.ht

>Is it acceptable to use that?
>
> Forgive me if there is documentation/guidance on this stuff already -- I didn't manage to find very much.
>

Not much! These are good questions.

Regards,
Peter

> If anyone wants to take a look, my working tree is at:
> https://github.com/0xabu/qemu/tree/raspi
>
> The relevant files are:
>  hw/arm/bcm2835.c                     |   92 +++
>  hw/arm/bcm2835_peripherals.c         |  368 ++++++++++++
>  hw/arm/bcm2836.c                     |  149 +++++
>  hw/arm/raspi.c                       |  228 ++++++++
>  hw/char/bcm2835_aux.c                |  250 ++++++++
>  hw/display/bcm2835_fb.c              |  427 ++++++++++++++
>  hw/dma/bcm2835_dma.c                 |  366 ++++++++++++
>  hw/intc/bcm2835_ic.c                 |  248 ++++++++
>  hw/intc/bcm2836_control.c            |  373 ++++++++++++
>  hw/misc/bcm2835_mphi.c               |  176 ++++++
>  hw/misc/bcm2835_power.c              |  113 ++++
>  hw/misc/bcm2835_property.c           |  417 +++++++++++++
>  hw/misc/bcm2835_sbm.c                |  310 ++++++++++
>  hw/misc/bcm2835_vchiq.c              |  114 ++++
>  hw/sd/bcm2835_emmc.c                 |  844 +++++++++++++++++++++++++++
>  hw/timer/arm_timer.c                 |   39 ++
>  hw/timer/bcm2835_st.c                |  201 +++++++
>  hw/timer/bcm2835_timer.c             |  242 ++++++++
>  hw/usb/bcm2835_usb.c                 |  655 +++++++++++++++++++++
>  hw/usb/bcm2835_usb_regs.h            | 1061 ++++++++++++++++++++++++++++++++++
>  include/hw/arm/bcm2835.h             |   31 +
>  include/hw/arm/bcm2835_arm_control.h |  481 +++++++++++++++
>  include/hw/arm/bcm2835_mbox.h        |   14 +
>  include/hw/arm/bcm2835_peripherals.h |   35 ++
>  include/hw/arm/bcm2836.h             |   34 ++
>  include/hw/arm/raspi_platform.h      |  160 +++++
>  include/hw/display/bcm2835_fb.h      |   29 +
>
> Thanks,
> Andrew

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Peter Maydell]

On 25 November 2015 at 06:04, Peter Crosthwaite
<crosthwaitepeter@gmail.com> wrote:
> On Tue, Nov 24, 2015 at 4:00 PM, Andrew Baumann
> <Andrew.Baumann@microsoft.com> wrote:
>>In general, what's the guidance on what belongs in object init vs. the realize method?
>
> If it doesn't depend on property values, it goes in init. If it does
> realize. Top level connectivity (e.g. connecting to UARTs are initing
> drives) generally are realize.

Also, realize has a mechanism for reporting errors (the Error** argument)
and instance init does not, so anything that might fail must go in
realize.

thanks
-- PMM

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Andrew Baumann]

> From: Peter Maydell [mailto:peter.maydell@linaro.org]
> Sent: Wednesday, 25 November 2015 01:11
> On 25 November 2015 at 06:04, Peter Crosthwaite
> <crosthwaitepeter@gmail.com> wrote:
> > On Tue, Nov 24, 2015 at 4:00 PM, Andrew Baumann
> > <Andrew.Baumann@microsoft.com> wrote:
> >>In general, what's the guidance on what belongs in object init vs. the
> realize method?
> >
> > If it doesn't depend on property values, it goes in init. If it does
> > realize. Top level connectivity (e.g. connecting to UARTs are initing
> > drives) generally are realize.
> 
> Also, realize has a mechanism for reporting errors (the Error** argument)
> and instance init does not, so anything that might fail must go in
> realize.

How does this work out with object_property_add_alias? It can fail and returns an error, so it should be called from realize, but it adds a new property on the object, so the property won't be useful if it can only be set after realize has added it.

It seems like everyone calling object_property_add_alias passes &error_abort, so I suppose that is the get-out-of-jail card for reporting errors from init...

Thanks,
Andrew

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Paolo Bonzini]

On 25/11/2015 19:14, Andrew Baumann wrote:
> How does this work out with object_property_add_alias? It can fail
> and returns an error, so it should be called from realize, but it
> adds a new property on the object, so the property won't be useful if
> it can only be set after realize has added it.

If you know it cannot fail, just pass &error_abort and call it from
instance_init.

Paolo

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Andrew Baumann]

> From: Peter Crosthwaite [mailto:crosthwaitepeter@gmail.com]
> Sent: Tuesday, 24 November 2015 22:04
> On Tue, Nov 24, 2015 at 4:00 PM, Andrew Baumann
> <Andrew.Baumann@microsoft.com> wrote:
> > I am working on refactoring the Pi support code as you suggested. I have
> split the Pi SOCs into separate objects (bcm2835 and bcm2836) which both
> instantiate a third common bcm2835_peripherals device that in turn contains
> all the common devices. I have also switched the code to use object
> properties rather than global variables to communicate state where devices
> interact with each other.
> >
> 
> Should this be an inheritance of a common SoC rather than an
> instantiation of a common container?

I considered that, but it doesn't seem to buy much. To avoid lots of conditional code in the common soc, I would still need three classes: the common stuff, bcm2835 (which inherits from the common stuff and adds the arm1176 cpu), and bcm2836 (which also inherits the common stuff, but adds up to four a7 cores, and a new interrupt controller with a more complex routing). I'd have to overload realize in both of those subclasses. And aside from the issue of properties (which aliases seem to solve -- thanks!) the code would be about the same complexity overall, with the downside that the interface between the common stuff and the parent soc device is now much wider through inheritance of all its internal stuff (like the private IO bus) that the parent doesn't need to see.

Andrew

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Peter Crosthwaite]

On Wed, Nov 18, 2015 at 10:16 AM, Andrew Baumann
<Andrew.Baumann@microsoft.com> wrote:
> Hi Peter,
>
> Thanks for your feedback!
>
> From: Peter Crosthwaite [mailto:crosthwaitepeter@gmail.com]
> Sent: Tuesday, 17 November 2015 23:51
>> I haven't looked beyond the diffstat yet, but a top level
>> architectural comment, I only see the one file in hw/arm. We are
>> promoting the split of board and SoC now as two separate objects. Each
>> of the patch series linked above demonstrates this. The BCM SoC would
>> be an object in its own right, then the board instantiates the SoC as
>> a composite device.
>
> I understand the rationale here, but is there much value pursuing this for the Pi? As far as I know, the Pi1/2 SOCs (bcm2835 and 2836) are one-off designs used only in the Pi. Moreover, the 2836 is basically a copy of the 2835 with the addition of a new block for a multi-core interrupt controller and mailboxes, so trying to split those out into two separate SOC definition files would require either lots of duplication, or a tight coupling between the two. Finally, there is basically nothing on the board besides the SoC and RAM (just a USB-Ethernet adapter), so the actual board definition files would be almost empty. As it is, I've already factored out all the common devices within hw/arm/raspi.c -- my feeling is that this is a cleaner way to do it for the Pi.
>

So I am looking at some recent news, and there is the RPi Zero which
reuses the bcm2835 as a different board. Can't find the schematic yet,
but from what I can see, this is a raw SoC (no USB ethernet). With
your SoC/board split we should be able pickup the Zero board quite
cheaply as follow up work.

Regards,
Peter

>> It is big. Work of this magnitude is generally better managed as
>> multiple series (each of multiple patches). A general approach with
>> new ARM boards is the first series is the bare minimum needed to get a

Re: [Qemu-devel] RFC: raspberry pi / pi2 / Windows-on-ARM support

[Andrew Baumann]

> From: Peter Crosthwaite [mailto:crosthwaitepeter@gmail.com]
> Sent: Thursday, 26 November 2015 11:39
> So I am looking at some recent news, and there is the RPi Zero which reuses
> the bcm2835 as a different board. Can't find the schematic yet, but from
> what I can see, this is a raw SoC (no USB ethernet). With your SoC/board split
> we should be able pickup the Zero board quite cheaply as follow up work.

Yes, in fact from what I've seen the Pi zero is functionally equivalent to our current emulation of the Pi1 -- we don't emulate the Pi's USB-Ethernet adapter either.

Andrew