Re: [Qemu-devel] QEMU+Linux ARMv7A current state

[Guenter Roeck <linux@roeck-us.net>]

On 10/03/2015 02:31 PM, Peter Crosthwaite wrote:
> Hi,
>
> I have done an audit of the ARMv7 boards to see what can boot a
> vanilla linux kernel. The basic approach is to build ARM
> multi_v7_defconfig kernel and boot QEMU using the DTBs built out by
> the kernel. The intersection of what mainline Linux has a DTB for and
> what QEMU models is tested. The boards that do/should work include:
>
>      vexpress-a9
>      vexpress-a15
>      smdkc210 (exynos)
>      xilinx-zynq-a9
>      cubieboard (allwinner)
>      highbank
>      midway
>      virt
>
> Have I missed anything?
>
> The basic rule is, no source code, config, or DTB edits allowed (i.e.
> what is a day0 user of QEMU+Linux going to experience?).
>
> So going board-by-board, these are the issues.
>
> highbank/midway:
>
> The kernel expects a secure monitor to be present which does at-least
> two things:
>
> * PSCI
> * Some cache-maintainance ops.
>
> QEMU has PSCI support, but the 0.1 command codings specified in the
> DTB are different to QEMUs 0.1 encodings. So I switched monitor mode
> for the CPUs on (it was conservatively set to off by the original
> authors of the QEMU ARM monitor support) and turned PSCI on. I had to
> write some patches to remap the PSCI encodings to highbank's custom
> ones and then the kernel at least entry-points all 4 cpus. It then
> deadlocks on something that I am yet to figure out. So the PSCI SMP
> stuff is shelved.
>
> The cache-maintenance is a bigger issue, that stops even a single core
> boot. The kernel cache controller driver is doing an smc that QEMU
> goes haywire on (as there simply is no firmware to catch the smc). But
> caches in QEMU are a nop, so I wrote some monitor firmware that just
> erets the smc without action and it works.
>
> I am using Sata (sysbus-ahci) for the boot media which works straight
> out of the box.
>
> So single core highbank works with the dummy monitor firmware and the
> monitor mode restrictor removed.
>
I have highbank working with multi_v7_defconfig. It generates a warning, though.

WARNING: CPU: 0 PID: 0 at arch/arm/mach-highbank/highbank.c:60 highbank_l2c310_write_sec+0x34/0x58()
[    0.000000] Highbank L2C310: ignoring write to reg 0x104

I think this is a kernel problem.

It also generates an SMC request.

SMC cmd 0x102 val 0x1

My "fix" for that is to simply log and ignore all SMC requests in qemu.
Not really feasible for production qemu, but after all I am testing the kernel,
not qemu. That also helped for various other HW. Maybe your monitor firmware
can help me get rid of that.

On top of that, I had to enable EL3. I don't remember why, just that it was
needed.

> cubieboard:
>
> The kernel expects a system level clock controller device that isn't
> modelled. A dummy model which hardcodes all registers to a know reset
> value and just lets the kernel read and write whatever it wants
> resolves. I'm not 100% sure whether this is needed but the kernel
> prints multiple errors with many pages of traces without. So it is at
> least needed to reduce the dmesg noise.
>
> QEMU cubieboard has no usable storage media, but the real hardware
> does have AHCI sata. I added sysbus-ahci at the right place but turns
> out the SATA controller has some custom power/clock (not really
> sure??) registers specific to this SoC. It sets/clears bits then polls
> them back expecting them to change to the other value asynchronously.
> The kernel device probe then times-out. So I subclassed sysbus-ahci
> and added the missing registers and forced the polled registers to the
> "I'm done" state. It works.
>
> I am using meta-sunxi Yocto-layer to build out the allwinner custom
> kernel/rootfs etc, and with the clock and Sata changes I get a boot.
> But when I change to the unedited kernel+dtb+rootfs I get stuck. RTC
> messages are around the point of failure which is not modelled in
> QEMU, so that is suspect.
>
> So with a dummy clock-controller and a slightly modified Sata
> controller (as a new device) cubieboard progresses, but it is still
> stuck on something that looks RTC related.
>
> xilinx-zynq-a9:
>
> The kernel wants to use cpufreq to set the CPU frequency. The kernel
> assumes ownership of the CPU clock divider but not the PLLs. QEMU uses
> the HW default for the silicon (x26) for the PLL which is incompatible
> with the dtb-set CPU frequency. The kernel boot asserts this and
> crashes. The ideal solution is to provide some sort of pre-boot
> firmware to set it to a compatible value. In real HW, this is the FSBL
> bootloader.
>
> TBH I think this is a kernel bug, as since cpufreq can be deconfigured
> it is ultimately optional. This means it shouldn't be a showstopper on
> a boot. The kernel should just switch off cpufreq and proceed and
> behave as if the feature was never configured in the kernel.
>
> There is also a critical bug in the SLCR model (my bad) where write to
> registers are ignored. I have a patch.
>
> More reading about the SLCR issues here (thanks to Guenter for getting
> the ball rolling here):
>
> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03374.html
>
Seemed to be going nowhere, though :-(.

> I have a patch with sets up the needed SLCR values from software but
> it needs more work.
>
> The kernel also expect the ADC device which is absent. Thanks to
> Guenter who patched this in which is under review:
>
> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg03375.html
>
Still working on the review comments.

> I am using SD card for the rootfs media. SD has a bug where it cannot
> handle a backing file of a non-round size. I am just passing the ext3
> image in unchanged (so unpadded) to QEMU and QEMU rounds down the
> image to the SD block size (512k!!), so I am losing us to 512k off the
> end of my ext3 filesystem. I have a hacky patch that changes this to
> round-up.
>
> With the minimal firmware, slcr bugfix, Guenters ADC, the SD hack a
> vanilla Zynq will be in good shape.
>
> virt:
>
> Virt largely works, but there are no immediately obvious
> storage/network options that are supported by the kernel. So I have to
> make some exceptions to the kernel config rule, that is, I add:
>
> CONFIG_VIRTIO_PCI=y"
> CONFIG_VIRTIO_BLK=y"
> CONFIG_VIRTIO_NET=y"
>
> To get disks and network for the virt board.
>
> Could we add these configs to mainline Linux's defconfig? Note, we
> don't need to add a DTS for this board, as QEMU's bootloader will
> generate and provide it for us.
>
> There is the known highmem PCI issue that is currently under discussion:
>
> https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg06529.html
>
> So aside from the missing VIRTIO kernel configs and highmem bug, virt
> is in good shape.
>
> vexpress:
>
> vexpress boots up to rootfs probing, however the only storage media
> that seems to be supported is flash, which doesn't have the needed
> kernel configs to boot from (not 100% sure on that but I gave up
> quickly). What are people who use this board in real hardware doing?
> It may just work best with initrd style boots.
>
I use it with
	-drive file=rootfs.ext3,format=raw,if=sd \
	-append "root=/dev/mmcblk0 ..."

with multi_v7_defconfig and vexpress-v2p-ca9.dtb. This works for both
vexpress-a9 and vexpress-a15 (with vexpress-v2p-ca15-tc1.dtb).

> exynos:
>
> QEMU models an Exynos skdkc210 but there is only a 310 available in
> QEMU dtbs. QEMU exynos however does not model a lot on the board level
> so we can largely get away with booting 310 dtb on 210.
>
I needed a couple of fixes to get the smdkc210 machine to work with
exynos4210-smdkv310.dtb.

- CMU support was submitted a long time ago by Maksim Kozlov, but never
   accepted. I took the code and merged the absolutely necessary parts of
   (I dropped clock support).
- exynos4210_mct needs a one-line bug fix to initialize a timer before
   it is used.
- dma support for exynos4 must be enabled (pl330 must be instantiated).
- If I recall correctly, smdkc210 also needed SMC support.

I have to change the configuration file, though. QEMU currently does not
implement exynos cpuidle, so I have to disable CONFIG_CPU_IDLE and
CONFIG_ARM_EXYNOS_CPUIDLE.

I test both multi_v7_defconfig and exynos_defconfig with -M smdkc210.

> Similar to vexpress, the exynos board has a lack of usable boot media.
> The board does model USB (EHCI), however the DTB disables it as the
> smdk310 board does not support it. So the alternative would be to use
> one of the more full-featured exynos DTBs  (with way more peripherals)
> from mainline Linux, but the kernel is liable to then probe all sorts
> of missing hardware.
>
> Exynos SDHCI support was on list for a few revs and quite some time,
> merging it and using SD may solve the problem.
>
> I have not attempted the SMP support.
>
I run smdkc210 with "-smp 2". If I recall correctly, this is mandatory,
but I may be wrong.

> Unlike all other boards, Exynos is quite consuming of RAM. All other
> boards work with Yocto's default of 128MB but Exynos crashes unless
> you jack it up. So users need to be careful of the -m option.
>
> realview:
>
> Realview is not tested, but I discuss it here as it is a bit of a red
> herring. The kernel has dtb support for the realview-1176 board,
> however this is not modelled by QEMU. I creating a Realview variant
> with 1176 in QEMU, however the memory maps of all the peripherals were
> mismatched.
>
> Turns, out, QEMU is is modelling the realview FPGA emulation platform
> which has a different peripheral map to the 1176 realview board
> proper. So QEMU doesn't support the realview boards, just the FPGA
> emulation platform, for which I can't find a kernel dtb. Should we
> upstream one with some dtsi's for the CPU variation supported in QEMU?
>
> So Realview does not work within my testing parameters with no
> immediate prospects.
>
I have realview-pb-a8, realview-eb-mpcore, and realview-eb working,
but not with a default configuration.

> how I am testing:
>
> So this is all powered by the Yocto project (Poky). I got some good
> help from Nathan Rossi. I have a poky fork which changes the qemuarm
> target to build my mainline (4.2.1) kernel (multi_v7_defconfig +
> VIRTIO), all the DTBs and a usable rootfs. You can then specify the
> type of machine (from the list above) to yocto's runqemu command. The
> command sets up boot media and network automatically.
>
> To use it:
>
> git clone https://github.com/pcrost/poky.git
> cd poky
> source ./oe-init-build-env
> MACHINE=qemuarm bitbake core-image-minimal #this takes a while
> QEMUBIN=/path/to/qemu-arm MACHINE_SUBTYPE=virt runqemu qemuarm slirp
>
Just for reference, my code is at github.

https://github.com/groeck/linux-build-test
https://github.com/groeck/qemu

Guenter