[Buildroot] [RFC] post-build scripts: limitations, and proposal

[Buildroot] [RFC] post-build scripts: limitations, and proposal

[Yann E. MORIN]

Hello All!

Currently, I'm using a set of post-build scripts, and it's becoming to
be slightly tedious.

For example, here are a few of them:

  - rpi-fixup-rootfs
    - adds /dev/mmcblk0p1 on /boot to fstab, don't mount _netdev filesystems
    - set root's PS1
    - store config's tree hash in /etc/rpi-config.version  [0]

  - rpi-fixup-rootfs-yem
    - stores my ssh pub key in root's ~/.ssh/authorized_keys

  - rpi-fixup-rootfs-tvheadend
    - add a script to wait for DVB devices
    - add a CIFS mount to fstab, to access my NAS

  - rpi-fixup-rootfs-wpa
    - add my wpa-supplicant conf file to connect to my LAN

So, when I build my tvheadend filesystem, I want to run scripts:
    rpi-fixup-rootfs  rpi-fixup-rootfs-yem  rpi-fixup-rootfs-tvheadend

But when I want to build my mpd filesystem, I want to run scripts:
    rpi-fixup-rootfs  rpi-fixup-rootfs-yem rpi-fixup-rootfs-wpa


Since the scripts lie out of the buildroot tree, I set (all on one line,
but broken-out for readability):

    BR2_ROOTFS_POST_BUILD_SCRIPT="
        $(CONFIG_DIR)/../scripts/rpi-fixup-rootfs
        $(CONFIG_DIR)/../scripts/rpi-fixup-rootfs-yem
        $(CONFIG_DIR)/../scripts/rpi-fixup-rootfs-tvheadend"

Now, I am adding a new script. This is starting to be ridiculous...

What I'd like is to be able to pass one or more options to each
post-build script, so I can have a wrapper and tell it what to run:

    BR2_ROOTFS_POST_BUILD_SCRIPT="
        $(CONFIG_DIR)/../scripts/fixup-rootfs rpi ssh-key rpi-tvheadend"

to call those scripts:
    fixup-rootfs-rpi
    fixup-rootfs-ssh-key
    fixup-rootfs-rpi-tvheadend

But it is not possible, as Buildroot interprets BR2_ROOTFS_POST_BUILD_SCRIPT
as a space-separated list, not a single command. So, it is not possible
to pass any option to the post-build scripts.

So, I was going to change it from space-separated to colon-separated. But
that would break existing configuration. Sigh... :-(

I can see only one way to solve this: rename the variable (and add to
legacy), and make it a colon-separated list.

Opinions?

[0] My config files (and scripts...) are stored in a git tree, so I want
    to know which to rebuild only by looking at a file on a running board.
    Having only the Buildroot tree hash is not enough.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [RFC] post-build scripts: limitations, and proposal

[Thomas De Schampheleire]

Hi Yann,

On Thu, Jun 20, 2013 at 11:36 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Hello All!
>
> Currently, I'm using a set of post-build scripts, and it's becoming to
> be slightly tedious.
>
> For example, here are a few of them:
>
>   - rpi-fixup-rootfs
>     - adds /dev/mmcblk0p1 on /boot to fstab, don't mount _netdev filesystems
>     - set root's PS1
>     - store config's tree hash in /etc/rpi-config.version  [0]
>
>   - rpi-fixup-rootfs-yem
>     - stores my ssh pub key in root's ~/.ssh/authorized_keys
>
>   - rpi-fixup-rootfs-tvheadend
>     - add a script to wait for DVB devices
>     - add a CIFS mount to fstab, to access my NAS
>
>   - rpi-fixup-rootfs-wpa
>     - add my wpa-supplicant conf file to connect to my LAN
>
> So, when I build my tvheadend filesystem, I want to run scripts:
>     rpi-fixup-rootfs  rpi-fixup-rootfs-yem  rpi-fixup-rootfs-tvheadend
>
> But when I want to build my mpd filesystem, I want to run scripts:
>     rpi-fixup-rootfs  rpi-fixup-rootfs-yem rpi-fixup-rootfs-wpa
>
>
> Since the scripts lie out of the buildroot tree, I set (all on one line,
> but broken-out for readability):
>
>     BR2_ROOTFS_POST_BUILD_SCRIPT="
>         $(CONFIG_DIR)/../scripts/rpi-fixup-rootfs
>         $(CONFIG_DIR)/../scripts/rpi-fixup-rootfs-yem
>         $(CONFIG_DIR)/../scripts/rpi-fixup-rootfs-tvheadend"
>
> Now, I am adding a new script. This is starting to be ridiculous...

Just to be clear: what you find ridiculous is the length and
non-readability of this option, is that correct?

Besides the proposal you are making below, here is an alternate one
(I'm not saying it is superior, just thinking out loud): if we add an
extra option like
BR2_ROOTFS_POST_BUILD_SCRIPT_BASE="$(CONFIG_DIR)/../scripts/", then:
BR2_ROOTFS_POST_BUILD_SCRIPT="rpi-fixup-rootfs rpi-fixup-rootfs-yem
rpi-fixup-rootfs-tvheadend".
This already is much more readable.
The logic I'd use for this is: when parsing the
BR2_ROOTFS_POST_BUILD_SCRIPT, prepend
BR2_ROOTFS_POST_BUILD_SCRIPT_BASE to each item. BASE can be empty
(legacy) in which case the entries in the normal variable should be
the full path, as is currently the case.

>
> What I'd like is to be able to pass one or more options to each
> post-build script, so I can have a wrapper and tell it what to run:
>
>     BR2_ROOTFS_POST_BUILD_SCRIPT="
>         $(CONFIG_DIR)/../scripts/fixup-rootfs rpi ssh-key rpi-tvheadend"
>
> to call those scripts:
>     fixup-rootfs-rpi
>     fixup-rootfs-ssh-key
>     fixup-rootfs-rpi-tvheadend
>
> But it is not possible, as Buildroot interprets BR2_ROOTFS_POST_BUILD_SCRIPT
> as a space-separated list, not a single command. So, it is not possible
> to pass any option to the post-build scripts.
>
> So, I was going to change it from space-separated to colon-separated. But
> that would break existing configuration. Sigh... :-(
>
> I can see only one way to solve this: rename the variable (and add to
> legacy), and make it a colon-separated list.
>
> Opinions?

It does indeed solve your use case. I'm not really against it, but
there certainly are other possibilities. Above I gave one, here's
another:
Instead of passing multiple scripts, you could just pass one wrapper
post build script, without options. This wrapper would be different in
your different configurations, and would call all relevant scripts in
turn. Adding a script for a certain configuration now no longer means
updating the buildroot configuration, but instead updating the wrapper
script.
So for example you'd have:

fixup-rootfs-tvheadend:
#!/bin/sh
targetdir=$1
CONFIG_DIR=<to be hardcoded here, or imported from elsewhere>
$CONFIG_DIR/../scripts/rpi-fixup-rootfs $targetdir
$CONFIG_DIR/../scripts/rpi-fixup-rootfs-yem $targetdir
$CONFIG_DIR/../scripts/rpi-fixup-rootfs-tvheadend $targetdir

fixup-rootfs-mpd:
#!/bin/sh
targetdir=$1
CONFIG_DIR=<to be hardcoded here, or imported from elsewhere>
$CONFIG_DIR/../scripts/rpi-fixup-rootfs $targetdir
$CONFIG_DIR/../scripts/rpi-fixup-rootfs-yem $targetdir
$CONFIG_DIR/../scripts/rpi-fixup-rootfs-wpa $targetdir

With this alternative, there is no need to change anything in
buildroot itself, there is just a change in your workflow.

Again, I'm not saying this is superior, just opening up the discussion...

Best regards,
Thomas

[Buildroot] [RFC] post-build scripts: limitations, and proposal

[Thomas Petazzoni]

Yann, Thomas,

On Fri, 21 Jun 2013 08:17:05 +0200, Thomas De Schampheleire wrote:

> Instead of passing multiple scripts, you could just pass one wrapper
> post build script, without options. This wrapper would be different in
> your different configurations, and would call all relevant scripts in
> turn. Adding a script for a certain configuration now no longer means
> updating the buildroot configuration, but instead updating the wrapper
> script.

This is absolutely what should be done in Yann's case. There's no need
to make the post-build script mechanism in Buildroot more complicated,
just handle the complexity in our own scripts, it is much more
flexible.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC] post-build scripts: limitations, and proposal

[Arnout Vandecappelle]

On 21/06/13 08:17, Thomas De Schampheleire wrote:
> Instead of passing multiple scripts, you could just pass one wrapper
> post build script, without options. This wrapper would be different in
> your different configurations, and would call all relevant scripts in
> turn.

  Or alternatively: a wrapper script that observes $0 and decides which 
other scripts to call based on that. So you'd have
rpi-fixup-rootfs at yem@tvheadend for the one configuration, and
rpi-fixup-rootfs at yem@wpa for the other one.

  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC] post-build scripts: limitations, and proposal

[Thomas De Schampheleire]

On Sun, Jun 23, 2013 at 7:06 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 21/06/13 08:17, Thomas De Schampheleire wrote:
>>
>> Instead of passing multiple scripts, you could just pass one wrapper
>> post build script, without options. This wrapper would be different in
>> your different configurations, and would call all relevant scripts in
>> turn.
>
>
>  Or alternatively: a wrapper script that observes $0 and decides which other
> scripts to call based on that. So you'd have
> rpi-fixup-rootfs at yem@tvheadend for the one configuration, and
> rpi-fixup-rootfs at yem@wpa for the other one.
>

But this requires a set of symbolic links matching each different
configuration, right?