[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Maxime Hadjinlian]

After bumping from a 2 years old Buildroot installation, I got the
following messages from the kernel at boot:

Failed to execute /init (error -2)

It appears that the overlay mechanisms was causing my issues.

Description:
The overlay mechanisms works by rsync'ing files from a folder specified
by the user into target.
Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are
symlinked into /usr.

Problem:
If you have an overlay that contains the directory /bin, /sbin or /lib, rsync
will erase the symlink and create a directory in its place.
The directory, say .../target/lib, would *only* contains the overlay's content.
Obviously, this causes a lot of troubles, starting with the kernel
failing to start init (because of missing shared libraries).

Solution:
Telling rsync to treat symlinked dir on the receiver side as directory
allow users to be blissfully ignorant of that problem.

Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 71735b5..2f33ddf 100644
--- a/Makefile
+++ b/Makefile
@@ -655,7 +655,7 @@ endif
 
 	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
 		$(call MESSAGE,"Copying overlay $(d)"); \
-		rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
+		rsync -a --keep-dirlinks --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
 			--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
 			$(d)/ $(TARGET_DIR)$(sep))
 
-- 
2.7.0

[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Arnout Vandecappelle]

On 10-02-16 16:58, Maxime Hadjinlian wrote:
> After bumping from a 2 years old Buildroot installation, I got the
> following messages from the kernel at boot:
> 
> Failed to execute /init (error -2)
> 
> It appears that the overlay mechanisms was causing my issues.
> 
> Description:
> The overlay mechanisms works by rsync'ing files from a folder specified
> by the user into target.
> Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are
> symlinked into /usr.
> 
> Problem:
> If you have an overlay that contains the directory /bin, /sbin or /lib, rsync
> will erase the symlink and create a directory in its place.
> The directory, say .../target/lib, would *only* contains the overlay's content.
> Obviously, this causes a lot of troubles, starting with the kernel
> failing to start init (because of missing shared libraries).
> 
> Solution:
> Telling rsync to treat symlinked dir on the receiver side as directory
> allow users to be blissfully ignorant of that problem.

 I'm not so sure of this one... It makes it impossible to override the symlinks
in the skeleton in the overlay. Say, you want to populate /var/spool with some
directories so it becomes useable.

 That said, it's probably better in that case to override the skeleton directly,
so that package installation can pick up the new directory.

 Still, you _are_ breaking a valid use case with this patch.

 What do the others think?

 Regards,
 Arnout

> 
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 71735b5..2f33ddf 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -655,7 +655,7 @@ endif
>  
>  	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
>  		$(call MESSAGE,"Copying overlay $(d)"); \
> -		rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
> +		rsync -a --keep-dirlinks --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
>  			--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
>  			$(d)/ $(TARGET_DIR)$(sep))
>  
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Maxime Hadjinlian]

Hi Arnout, all

On Wed, Feb 10, 2016 at 9:09 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 10-02-16 16:58, Maxime Hadjinlian wrote:
>> After bumping from a 2 years old Buildroot installation, I got the
>> following messages from the kernel at boot:
>>
>> Failed to execute /init (error -2)
>>
>> It appears that the overlay mechanisms was causing my issues.
>>
>> Description:
>> The overlay mechanisms works by rsync'ing files from a folder specified
>> by the user into target.
>> Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are
>> symlinked into /usr.
>>
>> Problem:
>> If you have an overlay that contains the directory /bin, /sbin or /lib, rsync
>> will erase the symlink and create a directory in its place.
>> The directory, say .../target/lib, would *only* contains the overlay's content.
>> Obviously, this causes a lot of troubles, starting with the kernel
>> failing to start init (because of missing shared libraries).
>>
>> Solution:
>> Telling rsync to treat symlinked dir on the receiver side as directory
>> allow users to be blissfully ignorant of that problem.
>
>  I'm not so sure of this one... It makes it impossible to override the symlinks
> in the skeleton in the overlay. Say, you want to populate /var/spool with some
> directories so it becomes useable.
Just a precision: it's not all the symlinks only the symlinks that
points to a directory, like /var/spool
>
>  That said, it's probably better in that case to override the skeleton directly,
> so that package installation can pick up the new directory.
>
>  Still, you _are_ breaking a valid use case with this patch.
Another way to deal with this, is to test if in the overlay, there's a
a folder that is a link is the default skeleton and in that case,
treat it differently.
In the spur of the moment it seems too complicated. But maybe it's the
right solution after all.
>
>  What do the others think?
>
>  Regards,
>  Arnout
>
>>
>> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>> ---
>>  Makefile | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/Makefile b/Makefile
>> index 71735b5..2f33ddf 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -655,7 +655,7 @@ endif
>>
>>       @$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
>>               $(call MESSAGE,"Copying overlay $(d)"); \
>> -             rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
>> +             rsync -a --keep-dirlinks --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
>>                       --chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
>>                       $(d)/ $(TARGET_DIR)$(sep))
>>
>>
>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Arnout Vandecappelle]

On 10-02-16 21:16, Maxime Hadjinlian wrote:
> Hi Arnout, all
> 
> On Wed, Feb 10, 2016 at 9:09 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>> On 10-02-16 16:58, Maxime Hadjinlian wrote:
>>> After bumping from a 2 years old Buildroot installation, I got the
>>> following messages from the kernel at boot:
>>>
>>> Failed to execute /init (error -2)
>>>
>>> It appears that the overlay mechanisms was causing my issues.
>>>
>>> Description:
>>> The overlay mechanisms works by rsync'ing files from a folder specified
>>> by the user into target.
>>> Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are
>>> symlinked into /usr.
>>>
>>> Problem:
>>> If you have an overlay that contains the directory /bin, /sbin or /lib, rsync
>>> will erase the symlink and create a directory in its place.
>>> The directory, say .../target/lib, would *only* contains the overlay's content.
>>> Obviously, this causes a lot of troubles, starting with the kernel
>>> failing to start init (because of missing shared libraries).
>>>
>>> Solution:
>>> Telling rsync to treat symlinked dir on the receiver side as directory
>>> allow users to be blissfully ignorant of that problem.
>>
>>  I'm not so sure of this one... It makes it impossible to override the symlinks
>> in the skeleton in the overlay. Say, you want to populate /var/spool with some
>> directories so it becomes useable.
> Just a precision: it's not all the symlinks only the symlinks that
> points to a directory, like /var/spool
>>
>>  That said, it's probably better in that case to override the skeleton directly,
>> so that package installation can pick up the new directory.
>>
>>  Still, you _are_ breaking a valid use case with this patch.
> Another way to deal with this, is to test if in the overlay, there's a
> a folder that is a link is the default skeleton and in that case,
> treat it differently.

 That is just dealing with the "accident" that the /bin and /lib symlinks are
created with an explicit ln instead of being present in the skeleton. It could
just as well have been the reverse, that they are symlinks in the skeleton and
mkdirs in the makefile. So I wouldn't rely on that.


 Regards,
 Arnout

> In the spur of the moment it seems too complicated. But maybe it's the
> right solution after all.
[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Maxime Hadjinlian]

On Wed, Feb 10, 2016 at 9:35 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 10-02-16 21:16, Maxime Hadjinlian wrote:
>> Hi Arnout, all
>>
>> On Wed, Feb 10, 2016 at 9:09 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>>> On 10-02-16 16:58, Maxime Hadjinlian wrote:
>>>> After bumping from a 2 years old Buildroot installation, I got the
>>>> following messages from the kernel at boot:
>>>>
>>>> Failed to execute /init (error -2)
>>>>
>>>> It appears that the overlay mechanisms was causing my issues.
>>>>
>>>> Description:
>>>> The overlay mechanisms works by rsync'ing files from a folder specified
>>>> by the user into target.
>>>> Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are
>>>> symlinked into /usr.
>>>>
>>>> Problem:
>>>> If you have an overlay that contains the directory /bin, /sbin or /lib, rsync
>>>> will erase the symlink and create a directory in its place.
>>>> The directory, say .../target/lib, would *only* contains the overlay's content.
>>>> Obviously, this causes a lot of troubles, starting with the kernel
>>>> failing to start init (because of missing shared libraries).
>>>>
>>>> Solution:
>>>> Telling rsync to treat symlinked dir on the receiver side as directory
>>>> allow users to be blissfully ignorant of that problem.
>>>
>>>  I'm not so sure of this one... It makes it impossible to override the symlinks
>>> in the skeleton in the overlay. Say, you want to populate /var/spool with some
>>> directories so it becomes useable.
>> Just a precision: it's not all the symlinks only the symlinks that
>> points to a directory, like /var/spool
>>>
>>>  That said, it's probably better in that case to override the skeleton directly,
>>> so that package installation can pick up the new directory.
>>>
>>>  Still, you _are_ breaking a valid use case with this patch.
>> Another way to deal with this, is to test if in the overlay, there's a
>> a folder that is a link is the default skeleton and in that case,
>> treat it differently.
>
>  That is just dealing with the "accident" that the /bin and /lib symlinks are
> created with an explicit ln instead of being present in the skeleton. It could
> just as well have been the reverse, that they are symlinks in the skeleton and
> mkdirs in the makefile. So I wouldn't rely on that.
Or we could do rsync twice, once where we exclude /sbin, /bin and /lib
from the overlay and do it as it was, and once again after which
include only /bin, /sbin and /lib from the overlay with the
--keep-dirlinks ?
>
>
>  Regards,
>  Arnout
>
>> In the spur of the moment it seems too complicated. But maybe it's the
>> right solution after all.
> [snip]
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

>> Solution:
 >> Telling rsync to treat symlinked dir on the receiver side as directory
 >> allow users to be blissfully ignorant of that problem.

 >  I'm not so sure of this one... It makes it impossible to override the symlinks
 > in the skeleton in the overlay. Say, you want to populate /var/spool with some
 > directories so it becomes useable.

 >  That said, it's probably better in that case to override the skeleton directly,
 > so that package installation can pick up the new directory.

 >  Still, you _are_ breaking a valid use case with this patch.

 >  What do the others think?

I'm also not really happy about it either. Another potential issue is if
the build machine is so old that rsync doesn't support --keep-dirlinks,
but according to:

https://download.samba.org/pub/unpacked/rsync/OLDNEWS

This was added in 2.6.3 (Sep 2004), so I guess even RHEL5 and similar
should have it.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Maxime Hadjinlian]

Hi Peter, all

On Thu, Feb 11, 2016 at 11:26 PM, Peter Korsgaard <peter@korsgaard.com> wrote:
>>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:
>
> Hi,
>
>>> Solution:
>  >> Telling rsync to treat symlinked dir on the receiver side as directory
>  >> allow users to be blissfully ignorant of that problem.
>
>  >  I'm not so sure of this one... It makes it impossible to override the symlinks
>  > in the skeleton in the overlay. Say, you want to populate /var/spool with some
>  > directories so it becomes useable.
>
>  >  That said, it's probably better in that case to override the skeleton directly,
>  > so that package installation can pick up the new directory.
>
>  >  Still, you _are_ breaking a valid use case with this patch.
>
>  >  What do the others think?
>
> I'm also not really happy about it either. Another potential issue is if
> the build machine is so old that rsync doesn't support --keep-dirlinks,
> but according to:
>
> https://download.samba.org/pub/unpacked/rsync/OLDNEWS
>
> This was added in 2.6.3 (Sep 2004), so I guess even RHEL5 and similar
> should have it.
So that's not really an issue then, is it ?

What do you think about the idea of doing rsync twice ? That should
solve every use case pretty nicely I think.
>
> --
> Bye, Peter Korsgaard

[Buildroot] [PATCH] Makefile: Fix overlay overwriting everything

[Yann E. MORIN]

Maxime, All,

On 2016-02-10 16:58 +0100, Maxime Hadjinlian spake thusly:
> After bumping from a 2 years old Buildroot installation, I got the
> following messages from the kernel at boot:
> 
> Failed to execute /init (error -2)
> 
> It appears that the overlay mechanisms was causing my issues.
> 
> Description:
> The overlay mechanisms works by rsync'ing files from a folder specified
> by the user into target.
> Since c5bd8af65e50a51735eb112fed9cbe6337f14e06, /bin, /sbin and /lib are
> symlinked into /usr.
> 
> Problem:
> If you have an overlay that contains the directory /bin, /sbin or /lib, rsync
> will erase the symlink and create a directory in its place.
> The directory, say .../target/lib, would *only* contains the overlay's content.
> Obviously, this causes a lot of troubles, starting with the kernel
> failing to start init (because of missing shared libraries).
> 
> Solution:
> Telling rsync to treat symlinked dir on the receiver side as directory
> allow users to be blissfully ignorant of that problem.

It seems that this has already been fixed by a patch by Peter Seiderer:

    0db3452 Makefile: fix rootfs overlay with BR2_ROOTFS_MERGED_USR
            enabled

Can you confirm that this also fixes your issue? If so, care to mark
your patch as superseded in Patchwork, please?

If not, just chime in!

Thanks!

Regards,
Yann E. MORIN.

> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 71735b5..2f33ddf 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -655,7 +655,7 @@ endif
>  
>  	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
>  		$(call MESSAGE,"Copying overlay $(d)"); \
> -		rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
> +		rsync -a --keep-dirlinks --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
>  			--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
>  			$(d)/ $(TARGET_DIR)$(sep))
>  
> -- 
> 2.7.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'