Some experimental layers and conclusions

Some experimental layers and conclusions

[cshored]

Hi all,

I've done some prototyping of some ideas I was working on with OpenEmbedded,
what I learned from the efforts (which I'm posting here for the benefit of
someone who might be interested in this or similar concepts).

I was initially interested in OE as a possible replacement for OpenWrt/LEDE
on my router.  I quickly discovered there are some issue with that, however:

* OE doesn't have good *router* networking support (i.e. nothing like
OpenWrt/LEDE's netifd and friends for a headless system, that one can
configure by editing a text file, or by using a WebUI for handling multiple
interfaces and VLANs and bridged (or not) wireless and wired networks, with
easy firewall configuration).
* The routers (ar71xx) I was interested in have a lot of OpenWrt/LEDE
technical debt and have drivers and BSP that has never been upstreamed.
* OE's default build can quickly exceed 16MB flash, which is the larger of
the flash sizes on the devices I was interested in.

I started on the latter by creating a poky-tiny inspired distro which called
CShore Embedded Device -- CShorED (cshore is a nick I've been using for a
while).  That layer can be viewed at
https://layers.openembedded.org/layerindex/branch/master/layer/meta-cshored/
.  In the process I created meta-runit-init
https://layers.openembedded.org/layerindex/branch/master/layer/meta-runit-in
it/ (which uses busybox's runit/daemontools-style applets a the main init
system, because systemd was clearly too large for my use case and I wanted
something more resilient to daemon failure than sysvinit or busybox-init)
and meta-earlyinit
(https://layers.openembedded.org/layerindex/branch/master/layer/meta-earlyin
it/) which allows initramfs like pre-init setup when not using an initramfs
and also adds what I think is a lot better support for read-only-rootfs that
doesn't require special handling after earlyinit.

Oh and cshored uses squashfs-xz read-only-rootfs with the objective of
having an easy 'factory-reset' capbability, however it doesn't take the
overlay approach to the entire filesystem, but only overlays a subset of
files, and uses a lot of symlinks for data that is either volatile (not kept
across boots) or which is persistent (and with earlyinit has handling of
auto-setup of persistent storage).

I was successful with cshored in getting build sizes into potentially useful
range (~8 MB which my smaller devices have for flash) while still having
useful commands/capabilities, but it compared to OpenWrt/LEDE OE is always
going to be bigger.  The lack of device support got me looking at x86/Pi
etc, but I discovered that when going that route there are already a
significant number of good players who take advantage of the power and
usually available storage on x86 to create high-grade
routers/firewalls/gateways/VPN gateways/switches/etc, so not much point in
being a latecomer to that party (as a new entry).

runit-init was an interesting experiment but the ultimately systemd is much
better on systems that are large enough (which is even most embedded devices
these days), or procd for smaller devices (from OpenWrt/LEDE).  The problem
with run-it is that it doesn't have meaningful dependency support one ends
up with a lot attempting to start, fail and restart during the boot process
and a lot of delay while trying to figure out of the dependency was actually
up or just hasn't reported down yet.

Earlyinit seems to me to be the most useful of these, but I've heard systemd
is wanting to take over the boot process too, so I'm not sure that earlyinit
is a useful road to go down for further development.

If anyone has comments on the layers, or wishes to see further work on any
of them, please ping me, otherwise at this point I consider it feasibility
research that led to no-go conclusion.

Regards,

Daniel

Re: Some experimental layers and conclusions

[Trevor Woerner]

Hi Daniel,

Very interesting report, thanks for the info!

On Thu 2017-04-27 @ 02:46:24 AM, cshored@cshore.thecshore.com wrote:
> * OE doesn't have good *router* networking support (i.e. nothing like
> OpenWrt/LEDE's netifd and friends for a headless system, that one can
> configure by editing a text file, or by using a WebUI for handling multiple
> interfaces and VLANs and bridged (or not) wireless and wired networks, with
> easy firewall configuration).

I wonder Khem's if meta-openwrt could have helped here? It looks like it has a
recipe for netifd and friends:

	https://layers.openembedded.org/layerindex/branch/master/layer/meta-openwrt/

Out of curiosity, did you try using musl? You don't mention which C library
you're using. I'd be curious to know if musl can help with size reduction.

Is toybox mature enough to be used in these situations?

	http://layers.openembedded.org/layerindex/recipe/28902/

Best regards,
	Trevor

Some experimental layers and conclusions

[cshored]

-----Original Message-----
From: Daniel Dickinson [mailto:cshore@thecshore.com] 
Sent: May 5, 2017 3:52 AM
To: 'openembedded-devel@lists.openembeeded.org' <openembedded-devel@lists.openembeeded.org>
Subject: [oe] Some experimental layers and conclusions

Apologies if you get this twice; I got a bounce due to sending with wrong email address.

> -----Original Message-----
> From: Trevor Woerner [mailto:twoerner@gmail.com]
> Sent: April 27, 2017 9:06 AM
> To: cshored@cshore.thecshore.com
> Cc: Openembedded-devel@lists.openembedded.org
> Subject: Re: [oe] Some experimental layers and conclusions
>
> Hi Daniel,
>
> Very interesting report, thanks for the info!
>
> On Thu 2017-04-27 @ 02:46:24 AM, cshored@cshore.thecshore.com wrote:
> > * OE doesn't have good *router* networking support (i.e. nothing 
> > like OpenWrt/LEDE's netifd and friends for a headless system, that 
> > one can configure by editing a text file, or by using a WebUI for 
> > handling multiple interfaces and VLANs and bridged (or not) wireless 
> > and wired networks, with easy firewall configuration).
>
> I wonder Khem's if meta-openwrt could have helped here? It looks like 
> it has a recipe for netifd and friends:

I hadn't actually gotten to the point of needing to worry about the networking yet. I was trying to avoid procd because of too many hard-coded openwrtisms (assumptions about how the system should loo) in a niche C project, and systemd for being too large.  From what I saw in the changelog the meta-openwrt is not working yet, and my guess it has to do with a lot assumptions about how the system is set up that are not well documented, and are openwrt-specific (I have some knowledge of how this setup works, so I can make it work, but I'm not sure it's worth it).

The real showstoppers where the lack of kernel (especially wireless support) without getting into e.g. wireless development which I don't have facilities to do legally and hence is off the table.

>
> 	https://layers.openembedded.org/layerindex/branch/master/layer/
> meta-openwrt/
>
> Out of curiosity, did you try using musl? You don't mention which C 
> library you're using. I'd be curious to know if musl can help with 
> size reduction.

Yes, using musl, that's part of how I managed to get the size down as far as I did without getting into major patches for debloating.

>
> Is toybox mature enough to be used in these situations?
>
> 	http://layers.openembedded.org/layerindex/recipe/28902/
>

Last I checked (about three months ago) it is not (I've peeked at it from time to time, but there are some important things like POSIX shell and working dhcpc that would have to come from busybox or a larger traditional sources.

I'm not sure how much size difference toybox will make over busybox, although I am interested for other reasons.

The biggest size challenge compared to openwrt is that openwrt has a lot of custom stuf to cut things down, mostly by excluding not needed functionality, or as with netifd, odhcd, uhttpd, etc, by writing minimalist versions of things that are needed.

Regards,

Daniel