[Buildroot] [git commit] core/pkg-generic: check proper package installation

[Buildroot] [git commit] core/pkg-generic: check proper package installation

[Thomas Petazzoni]

commit: https://git.buildroot.net/buildroot/commit/?id=938f47c7e89362e7b824dbd0bd1ee9839d56d603
branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master

Some packages misbehave, and install files in either $(STAGING_DIR)/$(O)
or in $(TARGET_DIR)/$(O) .

One common reason for that is that pkgconf now prepends the sysroot path
to all the paths it returns. Other reasons vary, but are mostly due to
poorly writen generic-packages.

And a new step hooks to check that no file gets installed in either
location, called after the install-target and install-staging steps.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-generic.mk | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 3349092..827de62 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -95,6 +95,21 @@ define check_host_rpath
 endef
 GLOBAL_INSTRUMENTATION_HOOKS += check_host_rpath
 
+define step_check_build_dir_one
+	if [ -d $(2) ]; then \
+		printf "%s: installs files in %s\n" $(1) $(2) >&2; \
+		exit 1; \
+	fi
+endef
+
+define step_check_build_dir
+	$(if $(filter install-staging,$(2)),\
+		$(if $(filter end,$(1)),$(call step_check_build_dir_one,$(3),$(STAGING_DIR)/$(O))))
+	$(if $(filter install-target,$(2)),\
+		$(if $(filter end,$(1)),$(call step_check_build_dir_one,$(3),$(TARGET_DIR)/$(O))))
+endef
+GLOBAL_INSTRUMENTATION_HOOKS += step_check_build_dir
+
 # User-supplied script
 ifneq ($(BR2_INSTRUMENTATION_SCRIPTS),)
 define step_user

[Buildroot] [git commit] core/pkg-generic: check proper package installation

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > commit: https://git.buildroot.net/buildroot/commit/?id=938f47c7e89362e7b824dbd0bd1ee9839d56d603
 > branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master

 > Some packages misbehave, and install files in either $(STAGING_DIR)/$(O)
 > or in $(TARGET_DIR)/$(O) .

 > One common reason for that is that pkgconf now prepends the sysroot path
 > to all the paths it returns. Other reasons vary, but are mostly due to
 > poorly writen generic-packages.

 > And a new step hooks to check that no file gets installed in either
 > location, called after the install-target and install-staging steps.

Didn't we just discuss that this duplicates the proposed
leak-of-host-paths stuff Samuel is working on?

-- 
Bye, Peter Korsgaard

[Buildroot] [git commit] core/pkg-generic: check proper package installation

[Thomas Petazzoni]

Hello,

On Thu, 27 Oct 2016 10:38:48 +0200, Peter Korsgaard wrote:

>  > commit: https://git.buildroot.net/buildroot/commit/?id=938f47c7e89362e7b824dbd0bd1ee9839d56d603
>  > branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master  
> 
>  > Some packages misbehave, and install files in either $(STAGING_DIR)/$(O)
>  > or in $(TARGET_DIR)/$(O) .  
> 
>  > One common reason for that is that pkgconf now prepends the sysroot path
>  > to all the paths it returns. Other reasons vary, but are mostly due to
>  > poorly writen generic-packages.  
> 
>  > And a new step hooks to check that no file gets installed in either
>  > location, called after the install-target and install-staging steps.  
> 
> Didn't we just discuss that this duplicates the proposed
> leak-of-host-paths stuff Samuel is working on?

It does, but Samuel's work is not merged, and this is already useful by
itself. It can always be removed once Samuel's work is merged.

I guess Samuel is working first on the host rpath relativization, then
the target rpath cleanup, and finally the target "leak of host path"
stuff. So it might be a while before Samuel's patches are ready and
merged.

And Yann's patch has already uncovered an issue with the Bellagio
package.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [git commit] core/pkg-generic: check proper package installation

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 >> > And a new step hooks to check that no file gets installed in either
 >> > location, called after the install-target and install-staging steps.  
 >> 
 >> Didn't we just discuss that this duplicates the proposed
 >> leak-of-host-paths stuff Samuel is working on?

 > It does, but Samuel's work is not merged, and this is already useful by
 > itself. It can always be removed once Samuel's work is merged.

 > I guess Samuel is working first on the host rpath relativization, then
 > the target rpath cleanup, and finally the target "leak of host path"
 > stuff. So it might be a while before Samuel's patches are ready and
 > merged.

 > And Yann's patch has already uncovered an issue with the Bellagio
 > package.

Ok, fine by me. I just wondered as the patch had comments and you afaik
applied it without any notification.

-- 
Venlig hilsen,
Peter Korsgaard 

[Buildroot] [git commit] core/pkg-generic: check proper package installation

[Thomas Petazzoni]

Hello,

On Thu, 27 Oct 2016 10:56:30 +0200, Peter Korsgaard wrote:

>  > And Yann's patch has already uncovered an issue with the Bellagio
>  > package.  
> 
> Ok, fine by me. I just wondered as the patch had comments and you afaik
> applied it without any notification.

Yes, I should have replied to your comments after applying the patch,
sorry about that. To me, it felt like Yann's patch was useful today,
and was ready to be merged. It doesn't prevent from doing something
more general/better a later on, but when that "later" on is potentially
quite far in the future (Samuel's series on rpath has been around for a
very very long time), Yann's patch was a good thing to have right now.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [git commit] core/pkg-generic: check proper package installation

[Vicente Olivert Riera]

Hello,

this patch has broken MIPS Codescape toolchains. It fails like this:

>>> toolchain-external  Extracting
gzip -d -c
/br/dl/Codescape.GNU.Tools.Package.2016.05-03.for.MIPS.IMG.Linux.CentOS-5.x86.tar.gz
| tar --strip-components=2 -C /br/output/build/toolchain-external
--exclude='usr/lib/locale/*'   -xf -
rm -rf /br/output/host/opt/ext-toolchain
mkdir -p /br/output/host/opt/ext-toolchain
mv /br/output/build/toolchain-external/* /br/output/host/opt/ext-toolchain/
>>> toolchain-external  Patching
>>> toolchain-external  Configuring
Warning: Reading file '<stdin>' as free form
>>> toolchain-external  Building
/usr/bin/gcc -O2 -I/br/output/host/usr/include
-DBR_CROSS_PATH_SUFFIX='""'
-DBR_CROSS_PATH_REL='"opt/ext-toolchain/bin"' -DBR_ARCH='"mips64r6"'
-DBR_ABI='"64"' -DBR_MIPS_TARGET_LITTLE_ENDIAN
-DBR_SYSROOT='"usr/mips64el-buildroot-linux-gnu/sysroot"'
-DBR_ADDITIONAL_CFLAGS='' -s -Wl,--hash-style=both
toolchain/toolchain-wrapper.c -o
/br/output/build/toolchain-external/toolchain-wrapper
>>> toolchain-external  Installing to staging directory
/usr/bin/install -D -m 0755
/br/output/build/toolchain-external/toolchain-wrapper
/br/output/host/usr/bin/toolchain-wrapper
>>> toolchain-external  Copying external toolchain sysroot to staging...
ln: target
?/br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot//br/output/host/opt/ext-toolchain/sysroot/mipsel-r6-hard/?
is not a directory: No such file or directory
Symlinking
/br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot//br/output/host/opt/ext-toolchain/sysroot/mipsel-r6-hard/
-> ./../../../../../../../../../../../../
>>> toolchain-external  Installing gdbinit
rmdir /br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot/usr/bin
/br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot/usr/sbin
ln -sf bin-n64
/br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot/usr/bin
ln -sf sbin-n64
/br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot/usr/sbin
>>> toolchain-external  Fixing libtool files
toolchain-external: installs files in
/br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot//br/output
package/pkg-generic.mk:255: recipe for target
'/br/output/build/toolchain-external/.stamp_staging_installed' failed
make: *** [/br/output/build/toolchain-external/.stamp_staging_installed]
Error 1

I have done a git bisect and I've found that this is the first bad
commit. If I checkout to the previous commit it works fine as before.

I haven't done an in-depth investigation about why is actually
happening, but my knowledge on pkg-generic.mk (the file patched by this
commit) is not very advanced to be honest.

Could you please have a look as well?

Thanks in advance,

Vincent

On 27/10/16 10:04, Thomas Petazzoni wrote:
> Hello,
> 
> On Thu, 27 Oct 2016 10:56:30 +0200, Peter Korsgaard wrote:
> 
>>  > And Yann's patch has already uncovered an issue with the Bellagio
>>  > package.  
>>
>> Ok, fine by me. I just wondered as the patch had comments and you afaik
>> applied it without any notification.
> 
> Yes, I should have replied to your comments after applying the patch,
> sorry about that. To me, it felt like Yann's patch was useful today,
> and was ready to be merged. It doesn't prevent from doing something
> more general/better a later on, but when that "later" on is potentially
> quite far in the future (Samuel's series on rpath has been around for a
> very very long time), Yann's patch was a good thing to have right now.
> 
> Thomas
> 

[Buildroot] [git commit] core/pkg-generic: check proper package installation

[Vicente Olivert Riera]

This patch fixes the problem:

http://patchwork.ozlabs.org/patch/687750/

Vincent

On 27/10/16 10:39, Vicente Olivert Riera wrote:
> Hello,
> 
> this patch has broken MIPS Codescape toolchains. It fails like this:
> 
>>>> toolchain-external  Extracting
> gzip -d -c
> /br/dl/Codescape.GNU.Tools.Package.2016.05-03.for.MIPS.IMG.Linux.CentOS-5.x86.tar.gz
> | tar --strip-components=2 -C /br/output/build/toolchain-external
> --exclude='usr/lib/locale/*'   -xf -
> rm -rf /br/output/host/opt/ext-toolchain
> mkdir -p /br/output/host/opt/ext-toolchain
> mv /br/output/build/toolchain-external/* /br/output/host/opt/ext-toolchain/
>>>> toolchain-external  Patching
>>>> toolchain-external  Configuring
> Warning: Reading file '<stdin>' as free form
>>>> toolchain-external  Building
> /usr/bin/gcc -O2 -I/br/output/host/usr/include
> -DBR_CROSS_PATH_SUFFIX='""'
> -DBR_CROSS_PATH_REL='"opt/ext-toolchain/bin"' -DBR_ARCH='"mips64r6"'
> -DBR_ABI='"64"' -DBR_MIPS_TARGET_LITTLE_ENDIAN
> -DBR_SYSROOT='"usr/mips64el-buildroot-linux-gnu/sysroot"'
> -DBR_ADDITIONAL_CFLAGS='' -s -Wl,--hash-style=both
> toolchain/toolchain-wrapper.c -o
> /br/output/build/toolchain-external/toolchain-wrapper
>>>> toolchain-external  Installing to staging directory
> /usr/bin/install -D -m 0755
> /br/output/build/toolchain-external/toolchain-wrapper
> /br/output/host/usr/bin/toolchain-wrapper
>>>> toolchain-external  Copying external toolchain sysroot to staging...
> ln: target
> ?/br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot//br/output/host/opt/ext-toolchain/sysroot/mipsel-r6-hard/?
> is not a directory: No such file or directory
> Symlinking
> /br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot//br/output/host/opt/ext-toolchain/sysroot/mipsel-r6-hard/
> -> ./../../../../../../../../../../../../
>>>> toolchain-external  Installing gdbinit
> rmdir /br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot/usr/bin
> /br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot/usr/sbin
> ln -sf bin-n64
> /br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot/usr/bin
> ln -sf sbin-n64
> /br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot/usr/sbin
>>>> toolchain-external  Fixing libtool files
> toolchain-external: installs files in
> /br/output/host/usr/mips64el-buildroot-linux-gnu/sysroot//br/output
> package/pkg-generic.mk:255: recipe for target
> '/br/output/build/toolchain-external/.stamp_staging_installed' failed
> make: *** [/br/output/build/toolchain-external/.stamp_staging_installed]
> Error 1
> 
> I have done a git bisect and I've found that this is the first bad
> commit. If I checkout to the previous commit it works fine as before.
> 
> I haven't done an in-depth investigation about why is actually
> happening, but my knowledge on pkg-generic.mk (the file patched by this
> commit) is not very advanced to be honest.
> 
> Could you please have a look as well?
> 
> Thanks in advance,
> 
> Vincent
> 
> On 27/10/16 10:04, Thomas Petazzoni wrote:
>> Hello,
>>
>> On Thu, 27 Oct 2016 10:56:30 +0200, Peter Korsgaard wrote:
>>
>>>  > And Yann's patch has already uncovered an issue with the Bellagio
>>>  > package.  
>>>
>>> Ok, fine by me. I just wondered as the patch had comments and you afaik
>>> applied it without any notification.
>>
>> Yes, I should have replied to your comments after applying the patch,
>> sorry about that. To me, it felt like Yann's patch was useful today,
>> and was ready to be merged. It doesn't prevent from doing something
>> more general/better a later on, but when that "later" on is potentially
>> quite far in the future (Samuel's series on rpath has been around for a
>> very very long time), Yann's patch was a good thing to have right now.
>>
>> Thomas
>>