Re: [Qemu-devel] QEMU+Linux ARMv7A current state

[John Snow <jsnow@redhat.com>]

On 10/03/2015 05:31 PM, Peter Crosthwaite wrote:
> Hi,
> 
> I have done an audit of the ARMv7 boards to see what can boot a
> vanilla linux kernel. The basic approach is to build ARM
> multi_v7_defconfig kernel and boot QEMU using the DTBs built out by
> the kernel. The intersection of what mainline Linux has a DTB for and
> what QEMU models is tested. The boards that do/should work include:
> 
>     vexpress-a9
>     vexpress-a15
>     smdkc210 (exynos)
>     xilinx-zynq-a9
>     cubieboard (allwinner)
>     highbank
>     midway
>     virt
> 
> Have I missed anything?
> 
> The basic rule is, no source code, config, or DTB edits allowed (i.e.
> what is a day0 user of QEMU+Linux going to experience?).
> 
> So going board-by-board, these are the issues.
> 
> highbank/midway:
> 
> The kernel expects a secure monitor to be present which does at-least
> two things:
> 
> * PSCI
> * Some cache-maintainance ops.
> 
> QEMU has PSCI support, but the 0.1 command codings specified in the
> DTB are different to QEMUs 0.1 encodings. So I switched monitor mode
> for the CPUs on (it was conservatively set to off by the original
> authors of the QEMU ARM monitor support) and turned PSCI on. I had to
> write some patches to remap the PSCI encodings to highbank's custom
> ones and then the kernel at least entry-points all 4 cpus. It then
> deadlocks on something that I am yet to figure out. So the PSCI SMP
> stuff is shelved.
> 
> The cache-maintenance is a bigger issue, that stops even a single core
> boot. The kernel cache controller driver is doing an smc that QEMU
> goes haywire on (as there simply is no firmware to catch the smc). But
> caches in QEMU are a nop, so I wrote some monitor firmware that just
> erets the smc without action and it works.
> 
> I am using Sata (sysbus-ahci) for the boot media which works straight
> out of the box.
> 
> So single core highbank works with the dummy monitor firmware and the
> monitor mode restrictor removed.
> 
> cubieboard:
> 
> The kernel expects a system level clock controller device that isn't
> modelled. A dummy model which hardcodes all registers to a know reset
> value and just lets the kernel read and write whatever it wants
> resolves. I'm not 100% sure whether this is needed but the kernel
> prints multiple errors with many pages of traces without. So it is at
> least needed to reduce the dmesg noise.
> 
> QEMU cubieboard has no usable storage media, but the real hardware
> does have AHCI sata. I added sysbus-ahci at the right place but turns
> out the SATA controller has some custom power/clock (not really
> sure??) registers specific to this SoC. It sets/clears bits then polls
> them back expecting them to change to the other value asynchronously.
> The kernel device probe then times-out. So I subclassed sysbus-ahci
> and added the missing registers and forced the polled registers to the
> "I'm done" state. It works.
> 

I'm looking into the cubieboard now. Is our emulation based on any
particular model? (1-4?)

I'm trying to see if I can find anything that resembles a spec to see
what kind of registers this SoC has for its SATA controller.

Feel free to point me towards your patches, too.

> I am using meta-sunxi Yocto-layer to build out the allwinner custom
> kernel/rootfs etc, and with the clock and Sata changes I get a boot.
> But when I change to the unedited kernel+dtb+rootfs I get stuck. RTC
> messages are around the point of failure which is not modelled in
> QEMU, so that is suspect.
> 
> So with a dummy clock-controller and a slightly modified Sata
> controller (as a new device) cubieboard progresses, but it is still
> stuck on something that looks RTC related.
> 
> xilinx-zynq-a9:
> 
> The kernel wants to use cpufreq to set the CPU frequency. The kernel
> assumes ownership of the CPU clock divider but not the PLLs. QEMU uses
> the HW default for the silicon (x26) for the PLL which is incompatible
> with the dtb-set CPU frequency. The kernel boot asserts this and
> crashes. The ideal solution is to provide some sort of pre-boot
> firmware to set it to a compatible value. In real HW, this is the FSBL
> bootloader.
> 
> TBH I think this is a kernel bug, as since cpufreq can be deconfigured
> it is ultimately optional. This means it shouldn't be a showstopper on
> a boot. The kernel should just switch off cpufreq and proceed and
> behave as if the feature was never configured in the kernel.
> 
> There is also a critical bug in the SLCR model (my bad) where write to
> registers are ignored. I have a patch.
> 
> More reading about the SLCR issues here (thanks to Guenter for getting
> the ball rolling here):
> 
> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03374.html
> 
> I have a patch with sets up the needed SLCR values from software but
> it needs more work.
> 
> The kernel also expect the ADC device which is absent. Thanks to
> Guenter who patched this in which is under review:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03375.html
> 
> I am using SD card for the rootfs media. SD has a bug where it cannot
> handle a backing file of a non-round size. I am just passing the ext3
> image in unchanged (so unpadded) to QEMU and QEMU rounds down the
> image to the SD block size (512k!!), so I am losing us to 512k off the
> end of my ext3 filesystem. I have a hacky patch that changes this to
> round-up.
> 
> With the minimal firmware, slcr bugfix, Guenters ADC, the SD hack a
> vanilla Zynq will be in good shape.
> 
> virt:
> 
> Virt largely works, but there are no immediately obvious
> storage/network options that are supported by the kernel. So I have to
> make some exceptions to the kernel config rule, that is, I add:
> 
> CONFIG_VIRTIO_PCI=y"
> CONFIG_VIRTIO_BLK=y"
> CONFIG_VIRTIO_NET=y"
> 
> To get disks and network for the virt board.
> 
> Could we add these configs to mainline Linux's defconfig? Note, we
> don't need to add a DTS for this board, as QEMU's bootloader will
> generate and provide it for us.
> 
> There is the known highmem PCI issue that is currently under discussion:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg06529.html
> 
> So aside from the missing VIRTIO kernel configs and highmem bug, virt
> is in good shape.
> 
> vexpress:
> 
> vexpress boots up to rootfs probing, however the only storage media
> that seems to be supported is flash, which doesn't have the needed
> kernel configs to boot from (not 100% sure on that but I gave up
> quickly). What are people who use this board in real hardware doing?
> It may just work best with initrd style boots.
> 
> exynos:
> 
> QEMU models an Exynos skdkc210 but there is only a 310 available in
> QEMU dtbs. QEMU exynos however does not model a lot on the board level
> so we can largely get away with booting 310 dtb on 210.
> 
> Similar to vexpress, the exynos board has a lack of usable boot media.
> The board does model USB (EHCI), however the DTB disables it as the
> smdk310 board does not support it. So the alternative would be to use
> one of the more full-featured exynos DTBs  (with way more peripherals)
> from mainline Linux, but the kernel is liable to then probe all sorts
> of missing hardware.
> 
> Exynos SDHCI support was on list for a few revs and quite some time,
> merging it and using SD may solve the problem.
> 
> I have not attempted the SMP support.
> 
> Unlike all other boards, Exynos is quite consuming of RAM. All other
> boards work with Yocto's default of 128MB but Exynos crashes unless
> you jack it up. So users need to be careful of the -m option.
> 
> realview:
> 
> Realview is not tested, but I discuss it here as it is a bit of a red
> herring. The kernel has dtb support for the realview-1176 board,
> however this is not modelled by QEMU. I creating a Realview variant
> with 1176 in QEMU, however the memory maps of all the peripherals were
> mismatched.
> 
> Turns, out, QEMU is is modelling the realview FPGA emulation platform
> which has a different peripheral map to the 1176 realview board
> proper. So QEMU doesn't support the realview boards, just the FPGA
> emulation platform, for which I can't find a kernel dtb. Should we
> upstream one with some dtsi's for the CPU variation supported in QEMU?
> 
> So Realview does not work within my testing parameters with no
> immediate prospects.
> 
> how I am testing:
> 
> So this is all powered by the Yocto project (Poky). I got some good
> help from Nathan Rossi. I have a poky fork which changes the qemuarm
> target to build my mainline (4.2.1) kernel (multi_v7_defconfig +
> VIRTIO), all the DTBs and a usable rootfs. You can then specify the
> type of machine (from the list above) to yocto's runqemu command. The
> command sets up boot media and network automatically.
> 
> To use it:
> 
> git clone https://github.com/pcrost/poky.git
> cd poky
> source ./oe-init-build-env
> MACHINE=qemuarm bitbake core-image-minimal #this takes a while
> QEMUBIN=/path/to/qemu-arm MACHINE_SUBTYPE=virt runqemu qemuarm slirp
> 
> The QEMUBIN is optional as Yocto does build QEMU for you, but this
> lets you BYO QEMU if you are doing qemu development outside of the
> Yocto flow. Change MACHINE_SUBTYPE to one of the supported boards to
> see results. I suggest starting with virt. Skip cubieboard, that
> assumes my SATA patches. Everything else you will see varying degrees
> of success. Pass qemuparams="-m 256" to runqemu for the smdkc200
> (Exynos) board. Highbank and Midway will blank screen as the
> outstanding issues happen pre-UART. A meaningful logbuf can be pulled
> from RAM over the monitor. Useful instructions here:
> 
> http://www.wiki.xilinx.com/QEMU-Linux+Kernel+logbuf+Extraction
> 
> Some QEMU patches to follow to fix a few of the things mentioned.
> 
> We should get this into regression testing. Yocto has some automated
> testing features in it's own right that we should be able to leverage.
> Yet to investigate (Richard pasted some stuff on an earlier thread).
> 
> The exercise should be redone for ARM64, ARMv5 and ARMv6 and ARMv7M,
> then other arches (many of which Poky supports).
> 
> Regards,
> Peter
>