Condensing and re-coding programs to save space

Condensing and re-coding programs to save space

[Jody Bruchon]

The recent flare-up of networking development has led to a lot of 
additional small programs on the full disk images and that forced me to 
eliminate other programs from being installed automatically to avoid 
image build failures. I'd like to suggest some things that can be done 
to reduce the amount of space these are taking up.

Since all programs are statically linked to C library components, it 
would be helpful to condense similar programs into a single binary where 
possible, have the Makefile's install section create relative symlinks, 
and change the program's behavior based on the name used to invoke it. 
The code to do this obviously has its own overhead as well, so it's only 
practical if there are enough binaries to justify the trouble. You can 
find useful stuff for this in the "busyelks" project I had been trying 
out and left in the code.

It may also make sense to condense the new non-standard programs like 
the "echo tests" into one binary and choose the specific functionality 
with a command parameter instead.

Another thing is the use of write() instead of [vf]printf(). If a 
program can completely avoid the use of the printf() family of functions 
and use write() instead, it'll make the program significantly smaller on 
disk and in memory. If you must use printf() even once then this isn't 
going to be helpful. If you can get away with write() everywhere 
instead, the program will shrink and we'll be able to fit more on disk.

I'm sure more suggestions could be made and they would be most welcome.

-Jody

Re: Condensing and re-coding programs to save space

[Marc-F. LUCCA-DANIAU]

The unit tests for networking that I grouped in /elkscmd/test should not 
be included in the default images (because "test"...), and would be 
removed in next commits to make room. But we should also provide a more 
convenient way to customize the image, for a developer to be able to 
quickly add them back when doing non-regression testing.

Today we have to modify the Makefile to include / exclude a program, 5 
different targets for partial / full images, multiplied by the available 
image formats. Plus now a build_hd_image script that is not consistent 
(and IMHO, poorly designed as it unpacks an existing image to build 
another one, in place of building it directly).

Would be far better to follow the Buildroot / Busybox way, and to select 
the applets to be installed on the target in a single location. We 
already have a quite nice kernel configuration tool, why not using it 
for that selection and the choice of the image format ?

MFLD


Le 27/02/2017 à 20:53, Jody Bruchon a écrit :
> The recent flare-up of networking development has led to a lot of 
> additional small programs on the full disk images and that forced me 
> to eliminate other programs from being installed automatically to 
> avoid image build failures. I'd like to suggest some things that can 
> be done to reduce the amount of space these are taking up.
>
> Since all programs are statically linked to C library components, it 
> would be helpful to condense similar programs into a single binary 
> where possible, have the Makefile's install section create relative 
> symlinks, and change the program's behavior based on the name used to 
> invoke it. The code to do this obviously has its own overhead as well, 
> so it's only practical if there are enough binaries to justify the 
> trouble. You can find useful stuff for this in the "busyelks" project 
> I had been trying out and left in the code.
>
> It may also make sense to condense the new non-standard programs like 
> the "echo tests" into one binary and choose the specific functionality 
> with a command parameter instead.
>
> Another thing is the use of write() instead of [vf]printf(). If a 
> program can completely avoid the use of the printf() family of 
> functions and use write() instead, it'll make the program 
> significantly smaller on disk and in memory. If you must use printf() 
> even once then this isn't going to be helpful. If you can get away 
> with write() everywhere instead, the program will shrink and we'll be 
> able to fit more on disk.
>
> I'm sure more suggestions could be made and they would be most welcome.
>
> -Jody
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-8086" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Condensing and re-coding programs to save space

[Jody Bruchon]

On 2017-02-27 17:02, Marc-F. LUCCA-DANIAU wrote:
> The unit tests for networking that I grouped in /elkscmd/test should 
> not be included in the default images (because "test"...), and would 
> be removed in next commits to make room. But we should also provide a 
> more convenient way to customize the image, for a developer to be able 
> to quickly add them back when doing non-regression testing.
I think that the current way of using Makefile stuff to decide what goes 
in an image does need to be replaced with something more flexible.
> Today we have to modify the Makefile to include / exclude a program, 5 
> different targets for partial / full images, multiplied by the 
> available image formats. Plus now a build_hd_image script that is not 
> consistent (and IMHO, poorly designed as it unpacks an existing image 
> to build another one, in place of building it directly).
I accepted it because it does the job of making a bootable hard drive 
image; it does need to be changed in the future, but it is a nice 
stop-gap measure to make more free space available on a root image while 
all this testing work is being done.
> Would be far better to follow the Buildroot / Busybox way, and to 
> select the applets to be installed on the target in a single location. 
> We already have a quite nice kernel configuration tool, why not using 
> it for that selection and the choice of the image format ?
Well, see...that's a problem. The ELKS kernel uses a really old copy of 
Kconfig from Linux and when I looked at extracting the one used in a 
more modern Linux source code base the task was far from trivial. A lot 
of ELKS was cobbled together well over a decade ago. I'm open to moving 
to a more all-encompassing build system if someone wants to do it, but 
it certainly will not be a simple task.

-Jody

Re: Condensing and re-coding programs to save space

[Marc-François LUCCA-DANIAU]

> Well, see...that's a problem. The ELKS kernel uses a really old copy of
> Kconfig from Linux and when I looked at extracting the one used in a more
> modern Linux source code base the task was far from trivial. A lot of ELKS
> was cobbled together well over a decade ago. I'm open to moving to a more
> all-encompassing build system if someone wants to do it, but it certainly
> will not be a simple task.

Mmm... let us be pragmatic and stop dreaming with stuff like "changing
the compiler", "moving to another build system", etc. ELKS is
definitively a "hobby" / "educational"  project, with a few active
contributors, and is intended to have fun, not for business (except
data erasing / recovery on antic machines ?). So our development
strategy should be consistent with that matter of fact : little steps
forward, and keep consistency and stability. Obsolescence is
absolutely NOT a concern here, as long we find compatible emulators
and embedded systems to play with.

Back to the current Kconfig: it works quite well, fulfills the need,
so why breaking it with an hazardous upgrade ? I only propose to use
it not only for the kernel, but also for the applets and image format
selection. Not to upgrade it.

MFLD

Re: Condensing and re-coding programs to save space

[Georg Potthast 2]

I have not tested it, but my understanding is that the Makefile puts different files on the floppy images depending on the type selected. So a 360 Kb image has fewer files than a 1.44 image. As long as the list of files for each format in the Makefile is maintained I think this would be sufficient for now.

With the new ethernet support one could set up a repository ;)

Georg

> Marc-François LUCCA-DANIAU <mfld.fr@gmail.com> hat am 28. Februar 2017 um 10:31 geschrieben:
> 
> 
> > Well, see...that's a problem. The ELKS kernel uses a really old copy of
> > Kconfig from Linux and when I looked at extracting the one used in a more
> > modern Linux source code base the task was far from trivial. A lot of ELKS
> > was cobbled together well over a decade ago. I'm open to moving to a more
> > all-encompassing build system if someone wants to do it, but it certainly
> > will not be a simple task.
> 
> Mmm... let us be pragmatic and stop dreaming with stuff like "changing
> the compiler", "moving to another build system", etc. ELKS is
> definitively a "hobby" / "educational"  project, with a few active
> contributors, and is intended to have fun, not for business (except
> data erasing / recovery on antic machines ?). So our development
> strategy should be consistent with that matter of fact : little steps
> forward, and keep consistency and stability. Obsolescence is
> absolutely NOT a concern here, as long we find compatible emulators
> and embedded systems to play with.
> 
> Back to the current Kconfig: it works quite well, fulfills the need,
> so why breaking it with an hazardous upgrade ? I only propose to use
> it not only for the kernel, but also for the applets and image format
> selection. Not to upgrade it.
> 
> MFLD
> --
> To unsubscribe from this list: send the line "unsubscribe linux-8086" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Condensing and re-coding programs to save space

[Jody Bruchon]

On 2017-02-28 6:54 AM, Georg Potthast 2 wrote:
> I have not tested it, but my understanding is that the Makefile puts different files on the floppy images depending on the type selected. So a 360 Kb image has fewer files than a 1.44 image. As long as the list of files for each format in the Makefile is maintained I think this would be sufficient for now.
The lists are really bad. Each individual Makefile in each program 
directory is responsible for deciding what binaries go into what disk 
image type. Adding a disk image type requires either making it a clone 
of one of the three "standard" types (thus full3 and full5 are both 
"full" types) or adding a new image type to every single Makefile in 
every program directory. This definitely needs to change.

Re: Condensing and re-coding programs to save space

[Alan Cox]

You could take a look at the FUZIX userspace. Much of it is far more
compact and a lot of it uses write() etc for size saving. The library
routines are also often a lot more memory efficient (eg the malloc is
much tinier as it was rewritten after I found the ELKS one was being
really buggy).

There shouldn't be too much churn involved as the FUZIX libc is based in
part on the ELKS one with other bits taken from MUSL and the like (and
the FP library largely comes from the Sun contributions)

A second option for the C library is to implement shared libraries not
necessarily by truely sharing it as a library with its own cs: but using
the callers ds:/ss:, as that involves some fun for fixing up callbacks,
but even something as simple as loading it into the process address space
from a different file when execve runs.

And the third of course is to implement support for something like
exomizer, that doesn't even need kernel support except for split I/D
binaries.

The compressor needs a big computer but the decompressor is tiny.

Alan