Re: Adding or1k support to mkroot.

[Stafford Horne <shorne@gmail.com>]

On Sat, Dec 16, 2023 at 07:00:23AM -0600, Rob Landley wrote:
> On 12/16/23 03:44, Stafford Horne wrote:
> > On Fri, Dec 15, 2023 at 07:33:42AM -0600, Rob Landley wrote:
> >> I've more or less gotten it to work. The trick was statically linking in the
> >> initramfs image:
> >> 
> >>   https://github.com/landley/toybox/commit/5647741f6687
> >> 
> >> I haven't added any of the OPENRISC_HAVE_INST_* symbols because it booted
> >> without them, and presumably the toolchain will automatically do the right thing
> >> for userspace:
> >> 
> >>   https://landley.net/notes-2023.html#10-12-2023
> >> 
> >> (I just specify an i486, i586, or i686 toolchain, or tell x86-64
> >> "CFLAGS=-mtune=nocona", and so on. I don't need to micromanage stuff in
> >> menuconfig to build for minor variations of a given target, I have no idea why
> >> openrisc would work differently.)
> >> 
> >> Nor did I add JUMP_UPON_UNHANDLED_EXCEPTION because I honestly don't know what
> >> that modifies or why you would ever NOT do it if it was at all useful? I err on
> >> the side of fewer symbols when the benefit's not clear...
> > 
> > I just use the defaults.
> 
> Because of the way I'm doing it, the "default" for any symbol I don't select is
> switched off. My config lists all the symbols I need that aren't in allnoconfig.
> (Although "need" means a dependency won't switch it on when the other symbol
> gets switched on.)
> 
> Conceptually it's the same as running "make allnoconfig" and then going through
> the list of symbols and switching them on by hand in menuconfig. "This is what's
> enabled in this kernel."

OK.

> > Those instruction specific switches will only be
> > helpful if you really want tune things.  Which may be fun for some people.
> > 
> > The OpenRISC kernel is not yet sophisticated enough to detect and replace,
> > emulate the missing instructions.  So we have these switches.  Some soft CPUs
> > may have these instructions enabled/or disabled and these switches help if you
> > know what you are doing.  Luckily QEMU has all the instruction enabled.
> 
> Isn't that a toolchain issue, though? Your userspace gets built with the same
> toolchain, "we have a multiply instruction" is kind of something it needs to
> know. Which means:
> 
> $ :| or1k-linux-musl-cc -E -dM -
> #define __DBL_MIN_EXP__ (-1021)
> #define __UINT_LEAST16_MAX__ 0xffff
> #define __ATOMIC_ACQUIRE 2
> ...
> #define __UINT_FAST8_TYPE__ unsigned char
> #define __ATOMIC_ACQ_REL 4
> #define __ATOMIC_RELEASE 3
> 
> Can't you have #defines in your toolchain saying what architecture variant it
> supports? And thus the code can #ifdef those without kconfig having to care?

Yes, the toolchain supports these, the kernel switches enable the toolchain
flags when building the kernel (mostly).  For example:

$ or1k-elf-gcc --target-help
The following options are target specific:
...
  -mcmov                      Enable generation of conditional move (l.cmov)
                              instructions.  By default the equivalent will
                              be generated using set and branch.
..
  -mhard-div                  Enable generation of hardware divide (l.div,
                              l.divu) instructions.  This is the default; use
                              -msoft-div to override.
  -mhard-mul                  Enable generation of hardware multiply
                              instructions (l.mul, l.muli) instructions. This
                              is the default; use -msoft-mul to override.
..

The exception is where we use these flags to enable bitops. In:

 arch/openrisc/include/asm/bitops/ffs.h
 arch/openrisc/include/asm/bitops/fls.h

Like other architectures the or1k toolchain doesn't expose which architecture is
being built. While architectures such as m68k define tunes and export i.e.
__mcfisaa__ to infer which instructions are enabled.

OpenRISC has no such architectures as within a softcpu architecture instructions
can be enabled or disabled as required.  I know, its a bit crazy.

However, you should be safe disabling everything.

> > In terms of JUMP_UPON_UNHANDLED_EXCEPTION, this was added before my time.  We
> > probably could get rid of it and do a more graceful power based halt.
> 
> I'm still not 100% clear on what it does, even after reading the help text. :)

This set's up an infinite loop on panic, rather than performing the default
kernel operation.

> >> >> I built an or1k musl+gcc toolchain with my wrapper script around Rich Felker's
> >> >> musl-cross-make project by adding "or1k::" to the TARGETS list in
> >> >> https://github.com/landley/toybox/blob/master/scripts/mcm-buildall.sh#L34 and
> >> >> that seems to have worked-ish. (Well, the kernel headers didn't install because
> >> >> I have to add or1k->openrisc to musl-cross-make's TARGET_ARCH_MANGLED in
> >> >> litecross/Makefile, but eh, close enough for the moment.)
> >> 
> >> I fixed that too, by the way:
> >> 
> >>   https://github.com/landley/toybox/commit/ab046139f9d8
> > 
> > OK.
> 
> Some glorious day Rich might apply the backlog of patches I've sent him to
> musl-cross-make (which last updated April 2022). Or I might scrape up the spoons
> to write a new linux-from-scratch style toolchain builder script so my
> mcm-buildall.sh isn't a wrapper around an external project. It's a toss-up which
> happens first...
> 
> (Mostly when I have spare toolchain brain I wrestle with llvm stuff instead...)
> 
> >> > This are my qemu arguments:
> >> > 
> >> > qemu-system-or1k -cpu or1200 -machine virt \
> >> >   -no-reboot -kernel /home/shorne/work/linux/vmlinux \
> >> >   -device virtio-net-pci,netdev=user \
> >> >   -netdev user,id=user,net=10.9.0.1/24,host=10.9.0.100,hostfwd=tcp::2222-:22 \
> >> >   -serial mon:stdio -nographic
> >> 
> >> If you're putting the monitor on stdio, where does the console go? (Is this
> >> going to pop up a graphics window or something? Or do you talk to it entirely
> >> through the virtual network?)
> > 
> > Console goes to stdio, I use ctrl-a C to drop into monitor which is also in
> > stdio.
> 
> Huh, I didn't know that was an option.

There are soo many options in QEMU.

> >> > -device virtio-blk-device,drive=d0 \
> >> >   -drive file=/home/shorne/work/openrisc/or1k-utils/buildroot/output/qemu-fs-or1k.qcow2,id=d0,if=none,format=qcow2 \
> >> >   -gdb tcp::10001 \
> >> >   -accel tcg,thread=multi \
> >> >   -smp cpus=4 -m 768 \
> >> 
> >> Do I need to micromanage -accel?
> > 
> > Probably not for your use cases.  I can not recall the defaults but I think it
> > was not multi threaded.  But I cannot be sure unless I go look.  You probably
> > can run with a single core and no SMP.
> 
> What I did years ago was set up distcc so qemu called out to the cross compiler
> running on loopback through the 10.0.2.2 emulated interface, thus moving the
> heavy lifting of compilation outside the emulator to where CPU was cheap without
> reintroducing all the problems of cross compiling (two compilers to keep
> straight, two sets of headers and two sets of libraries leaking into each other,
> can't run the binaries you generate, configure keeps asking questions about the
> host and applying the results to the target...)
> 
> It still maxed out around -j 3 because of the preprocessing, linking, network
> transfer, and make figuring out what to do next, but was way better than purely
> native single CPU builds.
> 
> No clue what the sweet spot is at today...
> 
> >> >   -append rootwait \
> >> >   -append boot=/dev/vda2
> >> 
> >> Did they ever fix qemu's problem that "append" didn't and you could only have
> >> one -append and it in fact completely replaced the existing command line on most
> >> architectures?
> >> 
> >> https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg03214.html
> > 
> > After I sent this message I noticed this looked strange.  It doesn't work, so my
> > command line ends up being: 'boot=/dev/vda2'.  Which was enough to get it to
> > work.
> 
> It's a misleading name.
> 
> >> Hmmm, I've got:
> >> 
> >> qemu-system-or1k -M or1k-sim -m 256 "$@" -nographic -no-reboot -kernel
> >> "$DIR"/linux-kernel -append "HOST=or1k console=FIXME $KARGS"
> >> 
> >> And it looks like the answer to "how do you enable a halt/reboot switch that can
> >> exit qemu" is "don't use the or1k-sim board, use the virt board".
> > 
> > Yeah, virt board is the only one that supports halt/reboot.  The 'or1ksim'
> > platform that we used as the model of qemu 'or1k-sim' supported reboot and halt
> > via special 'nop' instructions.  The QEMU maintainers did not feel these special
> > nop instructions should be supported in QEMU.
> 
> Sigh. Ok, I'll start the process over digesting the other config and trying to
> get that one to work instead...
> 
> >> It provides boot messages without any initrd at all. It eventually panics
> >> because there's no init, but early in board bringup I'm just looking for proof
> >> of life.
> >> 
> >> > Do you have a valid initrd that maybe I can try out?  I recently use qcow2
> >> > images to build, but I did used to use initrd images and it did work fine.
> >> 
> >> Sure, just remove the BUILTIN=1 from the if or1k stanza, and... it's 582k, I'll
> >> send it _not_ cc-ing the mailing list. :)
> > 
> > OK, I will try this out. But sorry busy this weekend I will probably not get
> > time.
> 
> No rush, I think you've answered my question above. It would be nice to get the
> bug fixed, but unless I can -device,poweroff or similar on the qemu command line
> to insert whatever it's poking to halt (and then tell the kernel about it), the
> sim board isn't useful to me for automation.

I tried to boot your initramfs with my virt_defconfig kernel, I just exchanged
driver for -initrd.  It seems to work, but panics with the following:

[    0.260000] 90000000.serial: ttyS0 at MMIO 0x90000000 (irq = 2, base_baud = 1250000) is a 16550A
[    0.270000] printk: console [ttyS0] enabled
..
[    0.420000] usbhid: USB HID core driver
[    0.420000] NET: Registered PF_PACKET protocol family
[    0.490000] clk: Disabling unused clocks
[    0.490000] ALSA device list:
[    0.490000]   No soundcards found.
[    0.520000] Freeing unused kernel image (initmem) memory: 232K
[    0.520000] This architecture does not have kernel memory protection.
[    0.520000] Run /init as init process
Type exit when done.
oneit: /dev/ttyS0ttyS0: No such file or directory
[    0.780000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[    0.780000] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 ]---

I am not sure about /dev/ttyS0ttyS0.

-Stafford

> >> >> Any idea what's going on here? That last one's kind of a blocker...
> >> > 
> >> > Have you tried to debug where it's getting hung up?  Gdb remote debug should
> >> > be working for or1k.
> >> 
> >> I have not yet tried to rustle up a gdb build that understands or1k machine
> >> code, no. I'm more likely to try to either get EARLY_PRINTK less broken or
> >> dredge up the 2 line spin-to-write-register version of 8250 output and cut and
> >> paste it in various places if I need to debug this myself.
> > 
> > That's OK, I usually use the console too to help debug too.
> 
> If I can reproduce it and stick a printf in it, I can generally debug it.
> 
> >> But I was hoping it was a known limitation or at least easily recognized...
> > 
> > OK, are things working for you now?  Or is the problem that the initrd still
> > does not work?
> 
> I can use initrd built-in to the kernel. I cannot supply it externally via
> qemu's -initrd. Which means I need to rebuild the kernel (7 minute clean
> rebuild) to tweak the filesystem (30 second clean rebuild).
> 
> That said, if I can supply an external -initrd to the virt board, then it works
> like the other targets. (It's a matter of workflow, I'm trying to regression
> test stuff against as many architectures as I can, which means I want them to
> work the same.)
> 
> One script I have doing this is my smoketest script at
> https://github.com/landley/toybox/blob/master/mkroot/testroot.sh which runs all
> the targets under qemu and makes sure they can get their clocks set (via ntp if
> necessary) and access a block device and do basic networking.
> 
> Eventually I want to get that running the toybox test suite inside qemu on each
> architecture, but that's still a ways off. (The shell isn't capable enough yet,
> working on it...)
> 
> I also want to get an automated Linux From Scratch build to compile and run on
> each target. I did that before with aboriginal linux years ago, but that was
> making busybox into a development environment. (Other projects like Alpine Linux
> and Adelie Linux took over from there when I stopped.) Now I'm trying to do it
> again with toybox, with the extra difficulty that I can't use any existing GPL
> code for anything because toybox is under a public domain equivalent license
> (Zero Clause BSD).
> 
> (Yes, this would mean building Linux From Scratch on or1k, under qemu. That's
> why I need I need a board with a working block device and at least 256 megs of
> ram, although I can use a network block device and a swap file in a pinch on
> systems with an MMU. I would someday LIKE to do a from-source debootstrap and
> populate a musl-backed package repository via automated cluster build, but
> there's still a certain amount of undocumented black magic in getting that to
> work. The old "people who know have known so long they've forgotten how to
> explain it" problem...)
> 
> > -Stafford
> 
> Rob