All of lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [PATCH 00/17] fs: support parallel filesystems build
@ 2018-01-14 14:16 Yann E. MORIN
  2018-01-14 14:16 ` [Buildroot] [PATCH 01/17] fs: pass EXTRA_ENV to post-fakeroot script Yann E. MORIN
                   ` (16 more replies)
  0 siblings, 17 replies; 22+ messages in thread
From: Yann E. MORIN @ 2018-01-14 14:16 UTC (permalink / raw)
  To: buildroot

Hello All!

TL;DR: filsystems are not parallel-safe, so we generate an intermediate
tarball from target/, which then serves as input for the various
filesystems; we also run pre-gen hooks under fakeroot, and get rid of
post-fs package-provided hooks.

Full explanations follows:

Currently, the filesystems are all built sequentially. This is
unfortunate, because each filesystem commands may modify the content of
target/, e.g. when systemd is enabled on a read-only filesystem, or cpio
to add its /init stub. cpio even changes the packages-supplied list of
devices, to shoe-horn /dev/console.

This has two issues:

  - the content of the generated rootfs depends on whether other
    filesystems are enabled or not;

  - it is not posible to build filesystems in parallel, becasue of
    concurency on target/.

A further (less critical) issue is that the fakeroot stuff is repeated
again and again for each filesystem: generating the user, groups and
device tables...

The only sane solution is to have each filesystem operate on its own
copy of target/, which is assembled once uner fakeroot and then re-used
by each filesystem, so that they can each do their own last-ditch tweaks
to the content of the filesystem before generating the image.

So we introduce an intermediate tarball that is assembled under
fakeroot, to run all the common fakeroot-needing setup. Each filesytem
then run a very simple fakeroot script that extracts that intermediate
tarball in a transient per-filesystem location and use that as input to
their image generation commands. This also allows to run the fakerooted,
common actions only once.

Today, the filesystems pre-/post-gen hooks are called very early,
outside of the fakeroot script, which means they can not do modifications
that require root access, like creating device nodes.

And as a last straw, the recently added (in August 2017) support for
package-provided pre-/post-fs hooks, and used solely by systemd on a
read-only filesystem, means that the target directory may be left in an
inconsistent state should the filesystem image generator fails, because
those pre-/post-fs hooks really need an atomic seciont which we can not
provide.

For this reason, the only solution is to act on a copy of target/.

So, now that all the issues have been exposed and their solutions
explained, the series is split as such:

  - the first few patches add fakeroot-script testing to the runtime
    test infra;

  - then we move the filesystemns pre-gen hooks under fakeroot, and we
    fix the cpio filesystem to correctly create its device node; the
    post-gen hooks need not run under fakeroot, because they are only
    supposed to act on the generated image, which does not require root
    rights (and modifying target/ after the iamge has been generated is
    pointless anyway;

  - then we introduce per-rootfs variables (per rootfs name and directory,
    and per rootfs target directory) and fakeroot scripts, and off-load
    the common dependencies to a common rule;

  - then we eventually split the filesystem generation in two, introducing
    the intermediate tarball, which filesystems re-use as input to
    generate their images. We also introduce a copy of target/ from which
    to generate the intermediate tarball, which allows us to get rid of
    the post-fs hooks provided by packages.

  - finally, the last patchs adds the removal of the transient copies of
    the target/ directory, as well as of the intermediate tarball, since
    they all can be rather large and are not meant to be user-visible.

As an aside, the pre-gen and post-gen hooks are not totally symetric
anymore: the pre-gen hooks run under fakeroot while the post-gen hooks
do not. A brief discussion on IRC suggested a change of name, since they
are no longer symetric, but in the end, I left this renaming as a
further exercise.

The series still runs the test suite with three failures, that are not
related to filesystems and are already present on master (as of today):

  - master: https://gitlab.com/buildroot.org/buildroot/pipelines/16106405/builds
  - series: https://gitlab.com/ymorin/buildroot-ci/pipelines/16132858/builds


Regards,
Yann E. MORIN.


The following changes since commit e4c2c68c3e1af04a3477a1777620f6021aa76065

  kvm-unit-tests: test for rdseed/rdrand (2018-01-14 15:08:37 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 2e287d91a65316ba4a8dc49feb963872b69a13b6

  fs: remove intermediate artefacts (2018-01-14 15:11:11 +0100)


----------------------------------------------------------------
Yann E. MORIN (17):
      fs: pass EXTRA_ENV to post-fakeroot script
      tests: add test for post-fakeroot script
      fs: run filesystem hooks under fakeroot
      fs/cpio: don't extend packages' permissions table
      fs/iso9660: rename internal variable
      core: introduce intermediate BASE_TARGET_DIR variable
      fs: set per-rootfs variable name
      fs: define per-rootfs directory
      fs: use a per-rootfs fakeroot script
      fs: split per-rootfs dependency line
      fs: introduce per-rootfs TARGET_DIR variable
      fs: new intermediate rootfs-common to gather common dependencies
      fs: introduce a macro for reproducible command
      fs: use a common tarball as base for the other filesystems
      fs: run packages' filesystem hooks in a copy of target/
      fs: get rid of package-provided post-fs hooks
      fs: remove intermediate artefacts

 Makefile                                           |  17 ++-
 fs/common.mk                                       | 114 +++++++++++++++------
 fs/cpio/cpio.mk                                    |   5 +-
 fs/iso9660/iso9660.mk                              |  30 +++---
 package/pkg-generic.mk                             |   6 +-
 .../skeleton-init-systemd/skeleton-init-systemd.mk |   6 --
 support/testing/tests/core/post-build.sh           |   2 +-
 support/testing/tests/core/post-fakeroot.sh        |   1 +
 support/testing/tests/core/post-image.sh           |  13 +--
 support/testing/tests/core/test_post_scripts.py    |  23 +++--
 10 files changed, 133 insertions(+), 84 deletions(-)
 create mode 120000 support/testing/tests/core/post-fakeroot.sh
 mode change 100755 => 120000 support/testing/tests/core/post-image.sh

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 22+ messages in thread
end of thread, other threads:[~2018-01-15 20:38 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-01-14 14:16 [Buildroot] [PATCH 00/17] fs: support parallel filesystems build Yann E. MORIN
2018-01-14 14:16 ` [Buildroot] [PATCH 01/17] fs: pass EXTRA_ENV to post-fakeroot script Yann E. MORIN
2018-01-15 20:37   ` Peter Korsgaard
2018-01-14 14:16 ` [Buildroot] [PATCH 02/17] tests: add test for " Yann E. MORIN
2018-01-14 23:53   ` Ricardo Martincoski
2018-01-15 20:38   ` Peter Korsgaard
2018-01-14 14:16 ` [Buildroot] [PATCH 03/17] fs: run filesystem hooks under fakeroot Yann E. MORIN
2018-01-15 10:36   ` Thomas Petazzoni
2018-01-14 14:17 ` [Buildroot] [PATCH 04/17] fs/cpio: don't extend packages' permissions table Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 05/17] fs/iso9660: rename internal variable Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 06/17] core: introduce intermediate BASE_TARGET_DIR variable Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 07/17] fs: set per-rootfs variable name Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 08/17] fs: define per-rootfs directory Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 09/17] fs: use a per-rootfs fakeroot script Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 10/17] fs: split per-rootfs dependency line Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 11/17] fs: introduce per-rootfs TARGET_DIR variable Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 12/17] fs: new intermediate rootfs-common to gather common dependencies Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 13/17] fs: introduce a macro for reproducible command Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 14/17] fs: use a common tarball as base for the other filesystems Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 15/17] fs: run packages' filesystem hooks in a copy of target/ Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 16/17] fs: get rid of package-provided post-fs hooks Yann E. MORIN
2018-01-14 14:17 ` [Buildroot] [PATCH 17/17] fs: remove intermediate artefacts Yann E. MORIN

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.