[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Thomas Petazzoni]

Currently, the cross-compilation.conf installed in
$(HOST_DIR)/etc/meson/cross-compilation.conf for use by the SDK is
generated in a post-install-staging hook of the toolchain package.

With per-package directory support enabled, this means that the
generated cross-compilation.conf contains references to the
per-package directory of the toolchain/ package, which is not want we
want:

[binaries]
c = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-gcc'
cpp = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-g++'
ar = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-ar'
strip = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-strip'
pkgconfig = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/usr/bin/pkg-config'

So instead, we generate this file in TOOLCHAIN_TARGET_FINALIZE_HOOKS,
so that the global paths are used:

[binaries]
c = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-gcc'
cpp = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-g++'
ar = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-ar'
strip = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-strip'
pkgconfig = '/home/thomas/projets/buildroot/output/host/usr/bin/pkg-config'

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-meson.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
index e7eea2aa58..085da219fe 100644
--- a/package/pkg-meson.mk
+++ b/package/pkg-meson.mk
@@ -205,4 +205,4 @@ define PKG_MESON_INSTALL_CROSS_CONF
 	    > $(HOST_DIR)/etc/meson/cross-compilation.conf
 endef
 
-TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
+TOOLCHAIN_TARGET_FINALIZE_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
-- 
2.23.0

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Peter Seiderer]

Hello Thomas,

On Wed,  4 Dec 2019 16:02:25 +0100, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> Currently, the cross-compilation.conf installed in
> $(HOST_DIR)/etc/meson/cross-compilation.conf for use by the SDK is
> generated in a post-install-staging hook of the toolchain package.
>
> With per-package directory support enabled, this means that the
> generated cross-compilation.conf contains references to the
> per-package directory of the toolchain/ package, which is not want we
> want:

Nitpick: s/not want we want/not what we want/

Reviewed-by: Peter Seiderer <ps.report@gmx.net>

Regards,
Peter

>
> [binaries]
> c = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-gcc'
> cpp = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-g++'
> ar = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-ar'
> strip = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-strip'
> pkgconfig = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/usr/bin/pkg-config'
>
> So instead, we generate this file in TOOLCHAIN_TARGET_FINALIZE_HOOKS,
> so that the global paths are used:
>
> [binaries]
> c = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-gcc'
> cpp = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-g++'
> ar = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-ar'
> strip = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-strip'
> pkgconfig = '/home/thomas/projets/buildroot/output/host/usr/bin/pkg-config'
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-meson.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> index e7eea2aa58..085da219fe 100644
> --- a/package/pkg-meson.mk
> +++ b/package/pkg-meson.mk
> @@ -205,4 +205,4 @@ define PKG_MESON_INSTALL_CROSS_CONF
>  	    > $(HOST_DIR)/etc/meson/cross-compilation.conf
>  endef
>
> -TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
> +TOOLCHAIN_TARGET_FINALIZE_HOOKS += PKG_MESON_INSTALL_CROSS_CONF

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Arnout Vandecappelle]

On 04/12/2019 16:02, Thomas Petazzoni wrote:
> Currently, the cross-compilation.conf installed in
> $(HOST_DIR)/etc/meson/cross-compilation.conf for use by the SDK is
> generated in a post-install-staging hook of the toolchain package.
> 
> With per-package directory support enabled, this means that the
> generated cross-compilation.conf contains references to the
> per-package directory of the toolchain/ package, which is not want we
> want:
> 
> [binaries]
> c = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-gcc'
> cpp = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-g++'
> ar = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-ar'
> strip = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-strip'
> pkgconfig = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/usr/bin/pkg-config'
> 
> So instead, we generate this file in TOOLCHAIN_TARGET_FINALIZE_HOOKS,
> so that the global paths are used:
> 
> [binaries]
> c = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-gcc'
> cpp = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-g++'
> ar = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-ar'
> strip = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-strip'
> pkgconfig = '/home/thomas/projets/buildroot/output/host/usr/bin/pkg-config'
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-meson.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> index e7eea2aa58..085da219fe 100644
> --- a/package/pkg-meson.mk
> +++ b/package/pkg-meson.mk
> @@ -205,4 +205,4 @@ define PKG_MESON_INSTALL_CROSS_CONF
>  	    > $(HOST_DIR)/etc/meson/cross-compilation.conf
>  endef
>  
> -TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
> +TOOLCHAIN_TARGET_FINALIZE_HOOKS += PKG_MESON_INSTALL_CROSS_CONF

 Note that this essentially reverts [1].

 The case for that patch was not terribly strong, so it should be OK. But then
again, the case for this patch is not that strong either.

 Note that pkg-cmake has the same issue. And a number of other packages as well,
e.g. the Python sysconfig refers to the Python per-package dirs.

 If we really want to do it fully correctly, we'd need to run something like the
relocate-sdk.sh on every per-package copy...

 Regards,
 Arnout

[1]
https://git.buildroot.org/buildroot/commit/?id=ccc9e05990a44d8db5825c07ed61d405ebb40caa

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Thomas Petazzoni]

Hello,

On Thu, 5 Dec 2019 23:13:42 +0100
Arnout Vandecappelle <arnout@mind.be> wrote:

> > -TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
> > +TOOLCHAIN_TARGET_FINALIZE_HOOKS += PKG_MESON_INSTALL_CROSS_CONF  
> 
>  Note that this essentially reverts [1].

Yes, true.

>  The case for that patch was not terribly strong, so it should be OK. But then
> again, the case for this patch is not that strong either.

Well, the case for my patch is that it allows to produce a meson
cross-compilation.conf that has correct and usable values at the end of
the build.

>  Note that pkg-cmake has the same issue. And a number of other packages as well,
> e.g. the Python sysconfig refers to the Python per-package dirs.

For sysconfig, I have a pending patch that I will submit soon to fix
the problem. It is in fact worse for sysconfig because if you don't
tweak it during the build, the build simply fails. While for the global
meson cross-compilation.conf, it is less dramatic as it is not used by
Buildroot itself, just provided as a convenience.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Arnout Vandecappelle]

On 06/12/2019 08:59, Thomas Petazzoni wrote:
> Hello,
> 
> On Thu, 5 Dec 2019 23:13:42 +0100
> Arnout Vandecappelle <arnout@mind.be> wrote:
> 
>>> -TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
>>> +TOOLCHAIN_TARGET_FINALIZE_HOOKS += PKG_MESON_INSTALL_CROSS_CONF  
>>
>>  Note that this essentially reverts [1].
> 
> Yes, true.
> 
>>  The case for that patch was not terribly strong, so it should be OK. But then
>> again, the case for this patch is not that strong either.
> 
> Well, the case for my patch is that it allows to produce a meson
> cross-compilation.conf that has correct and usable values at the end of
> the build.
> 
>>  Note that pkg-cmake has the same issue.

 Now that I'm awake again (<grin>) I see that it actually doesn't, because that
one uses relative paths. I don't suppose that meson's cross-compilation.conf can
use a similar trick? (Adding meson "experts" in Cc...)

>> And a number of other packages as well,
>> e.g. the Python sysconfig refers to the Python per-package dirs.
> 
> For sysconfig, I have a pending patch that I will submit soon to fix
> the problem. It is in fact worse for sysconfig because if you don't
> tweak it during the build, the build simply fails. While for the global
> meson cross-compilation.conf, it is less dramatic as it is not used by
> Buildroot itself, just provided as a convenience.

 Just to be entirely clear: it actually works as long as you keep the build
directory around, it just doesn't work in the context of the SDK, right?

 Regards,
 Arnout

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Thomas Petazzoni]

On Fri, 6 Dec 2019 09:58:15 +0100
Arnout Vandecappelle <arnout@mind.be> wrote:

> >> And a number of other packages as well,
> >> e.g. the Python sysconfig refers to the Python per-package dirs.  
> > 
> > For sysconfig, I have a pending patch that I will submit soon to fix
> > the problem. It is in fact worse for sysconfig because if you don't
> > tweak it during the build, the build simply fails. While for the global
> > meson cross-compilation.conf, it is less dramatic as it is not used by
> > Buildroot itself, just provided as a convenience.  
> 
>  Just to be entirely clear: it actually works as long as you keep the build
> directory around, it just doesn't work in the context of the SDK, right?

A final pass in staging-finalize will be needed indeed to fixup the
sysconfigdata for the SDK.

I think I tried to come up with a solution that involved using relative
paths, but didn't get it to work. But it's been a while I looked into
this, so I don't remember exactly.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Thomas De Schampheleire]

Thanks Arnout for looping me in.

Needless to say, I am opposed to moving the meson file back from the
target-finalize step :-)
I think most of the reasoning was already mentioned in that commit,
but it seems I did not explain that we actually have HOST_DIR mounted
read-only on subsequent 'make' commands after the initial make,
exactly to verify that no-one is changing directories they shouldn't
be changing. After all, a 'make' after the initial make will only
normally do the target-finalize step.

Can this problem of incorrect paths not be handled in another way,
e.g. by explicitly requesting the 'final' path variables outside the
per-package staging, or by doing a fixup afterwards (but not in the
scope of target-finalize), or by only creating the
cross-compilation.conf file if it does not already exist?

Thanks,
Thomas

El jue., 5 dic. 2019 a las 23:13, Arnout Vandecappelle
(<arnout@mind.be>) escribi?:
>
>
>
> On 04/12/2019 16:02, Thomas Petazzoni wrote:
> > Currently, the cross-compilation.conf installed in
> > $(HOST_DIR)/etc/meson/cross-compilation.conf for use by the SDK is
> > generated in a post-install-staging hook of the toolchain package.
> >
> > With per-package directory support enabled, this means that the
> > generated cross-compilation.conf contains references to the
> > per-package directory of the toolchain/ package, which is not want we
> > want:
> >
> > [binaries]
> > c = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-gcc'
> > cpp = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-g++'
> > ar = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-ar'
> > strip = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-strip'
> > pkgconfig = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/usr/bin/pkg-config'
> >
> > So instead, we generate this file in TOOLCHAIN_TARGET_FINALIZE_HOOKS,
> > so that the global paths are used:
> >
> > [binaries]
> > c = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-gcc'
> > cpp = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-g++'
> > ar = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-ar'
> > strip = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-strip'
> > pkgconfig = '/home/thomas/projets/buildroot/output/host/usr/bin/pkg-config'
> >
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > ---
> >  package/pkg-meson.mk | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> > index e7eea2aa58..085da219fe 100644
> > --- a/package/pkg-meson.mk
> > +++ b/package/pkg-meson.mk
> > @@ -205,4 +205,4 @@ define PKG_MESON_INSTALL_CROSS_CONF
> >           > $(HOST_DIR)/etc/meson/cross-compilation.conf
> >  endef
> >
> > -TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
> > +TOOLCHAIN_TARGET_FINALIZE_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
>
>  Note that this essentially reverts [1].
>
>  The case for that patch was not terribly strong, so it should be OK. But then
> again, the case for this patch is not that strong either.
>
>  Note that pkg-cmake has the same issue. And a number of other packages as well,
> e.g. the Python sysconfig refers to the Python per-package dirs.
>
>  If we really want to do it fully correctly, we'd need to run something like the
> relocate-sdk.sh on every per-package copy...
>
>  Regards,
>  Arnout
>
> [1]
> https://git.buildroot.org/buildroot/commit/?id=ccc9e05990a44d8db5825c07ed61d405ebb40caa
>

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Thomas De Schampheleire]

[oops, sorry for the top-posting]

El vie., 6 dic. 2019 a las 10:54, Thomas De Schampheleire
(<patrickdepinguin@gmail.com>) escribi?:
>
> Thanks Arnout for looping me in.
>
> Needless to say, I am opposed to moving the meson file back from the
> target-finalize step :-)
> I think most of the reasoning was already mentioned in that commit,
> but it seems I did not explain that we actually have HOST_DIR mounted
> read-only on subsequent 'make' commands after the initial make,
> exactly to verify that no-one is changing directories they shouldn't
> be changing. After all, a 'make' after the initial make will only
> normally do the target-finalize step.
>
> Can this problem of incorrect paths not be handled in another way,
> e.g. by explicitly requesting the 'final' path variables outside the
> per-package staging, or by doing a fixup afterwards (but not in the
> scope of target-finalize), or by only creating the
> cross-compilation.conf file if it does not already exist?
>
> Thanks,
> Thomas
>
> El jue., 5 dic. 2019 a las 23:13, Arnout Vandecappelle
> (<arnout@mind.be>) escribi?:
> >
> >
> >
> > On 04/12/2019 16:02, Thomas Petazzoni wrote:
> > > Currently, the cross-compilation.conf installed in
> > > $(HOST_DIR)/etc/meson/cross-compilation.conf for use by the SDK is
> > > generated in a post-install-staging hook of the toolchain package.
> > >
> > > With per-package directory support enabled, this means that the
> > > generated cross-compilation.conf contains references to the
> > > per-package directory of the toolchain/ package, which is not want we
> > > want:
> > >
> > > [binaries]
> > > c = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-gcc'
> > > cpp = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-g++'
> > > ar = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-ar'
> > > strip = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-strip'
> > > pkgconfig = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/usr/bin/pkg-config'
> > >
> > > So instead, we generate this file in TOOLCHAIN_TARGET_FINALIZE_HOOKS,
> > > so that the global paths are used:
> > >
> > > [binaries]
> > > c = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-gcc'
> > > cpp = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-g++'
> > > ar = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-ar'
> > > strip = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-strip'
> > > pkgconfig = '/home/thomas/projets/buildroot/output/host/usr/bin/pkg-config'
> > >
> > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > > ---
> > >  package/pkg-meson.mk | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> > > index e7eea2aa58..085da219fe 100644
> > > --- a/package/pkg-meson.mk
> > > +++ b/package/pkg-meson.mk
> > > @@ -205,4 +205,4 @@ define PKG_MESON_INSTALL_CROSS_CONF
> > >           > $(HOST_DIR)/etc/meson/cross-compilation.conf
> > >  endef
> > >
> > > -TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
> > > +TOOLCHAIN_TARGET_FINALIZE_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
> >
> >  Note that this essentially reverts [1].
> >
> >  The case for that patch was not terribly strong, so it should be OK. But then
> > again, the case for this patch is not that strong either.
> >
> >  Note that pkg-cmake has the same issue. And a number of other packages as well,
> > e.g. the Python sysconfig refers to the Python per-package dirs.
> >
> >  If we really want to do it fully correctly, we'd need to run something like the
> > relocate-sdk.sh on every per-package copy...
> >
> >  Regards,
> >  Arnout
> >
> > [1]
> > https://git.buildroot.org/buildroot/commit/?id=ccc9e05990a44d8db5825c07ed61d405ebb40caa
> >

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Thomas Petazzoni]

On Fri, 6 Dec 2019 10:54:13 +0100
Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote:

> Needless to say, I am opposed to moving the meson file back from the
> target-finalize step :-)
> I think most of the reasoning was already mentioned in that commit,
> but it seems I did not explain that we actually have HOST_DIR mounted
> read-only on subsequent 'make' commands after the initial make,
> exactly to verify that no-one is changing directories they shouldn't
> be changing. After all, a 'make' after the initial make will only
> normally do the target-finalize step.

No, it will do staging-finalize as well. I don't think we provide the
guarantee that HOST_DIR is unchanged/read-only between each "make"
invocation, and I'm not sure why we would want to provide this
guarantee.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Thomas De Schampheleire]

El vie., 6 dic. 2019 a las 10:57, Thomas Petazzoni
(<thomas.petazzoni@bootlin.com>) escribi?:
>
> On Fri, 6 Dec 2019 10:54:13 +0100
> Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote:
>
> > Needless to say, I am opposed to moving the meson file back from the
> > target-finalize step :-)
> > I think most of the reasoning was already mentioned in that commit,
> > but it seems I did not explain that we actually have HOST_DIR mounted
> > read-only on subsequent 'make' commands after the initial make,
> > exactly to verify that no-one is changing directories they shouldn't
> > be changing. After all, a 'make' after the initial make will only
> > normally do the target-finalize step.
>
> No, it will do staging-finalize as well. I don't think we provide the
> guarantee that HOST_DIR is unchanged/read-only between each "make"
> invocation, and I'm not sure why we would want to provide this
> guarantee.

The fact that staging-finalize is triggered from target-finalize is
also a problem in my use case, and I have cleared it in our local copy
(moving its action to another place). I think we talked about that
earlier too.

I think Yann suggested me to look into the sdk in this context. I need
to check it in more detail soon, to see whether it can really solve
this friction we have between our current needs and changes upstream.

To recap our use case: we make an initial Buildroot build, package it,
then in other machines but on the same virtual path (mounted to the
same location inside a docker container) extract it. Then,
applications are built outside of Buildroot, but stored to a copy of
the output/target directory. Then a subsequent 'make' is run to let
Buildroot package everything into an initramfs, setting TARGET_DIR to
the copy. During all these subsequent builds, the host directory and
some others are kept read-only to prevent corruption of that 'cache'.

Best regards,
Thomas

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Arnout Vandecappelle]

On 06/12/2019 11:16, Thomas De Schampheleire wrote:
> El vie., 6 dic. 2019 a las 10:57, Thomas Petazzoni
> (<thomas.petazzoni@bootlin.com>) escribi?:
>>
>> On Fri, 6 Dec 2019 10:54:13 +0100
>> Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote:
>>
>>> Needless to say, I am opposed to moving the meson file back from the
>>> target-finalize step :-)
>>> I think most of the reasoning was already mentioned in that commit,
>>> but it seems I did not explain that we actually have HOST_DIR mounted
>>> read-only on subsequent 'make' commands after the initial make,
>>> exactly to verify that no-one is changing directories they shouldn't
>>> be changing. After all, a 'make' after the initial make will only
>>> normally do the target-finalize step.
>>
>> No, it will do staging-finalize as well. I don't think we provide the
>> guarantee that HOST_DIR is unchanged/read-only between each "make"
>> invocation, and I'm not sure why we would want to provide this
>> guarantee.
> 
> The fact that staging-finalize is triggered from target-finalize is
> also a problem in my use case, and I have cleared it in our local copy
> (moving its action to another place). I think we talked about that
> earlier too.
> 
> I think Yann suggested me to look into the sdk in this context. I need
> to check it in more detail soon, to see whether it can really solve
> this friction we have between our current needs and changes upstream.
> 
> To recap our use case: we make an initial Buildroot build, package it,
> then in other machines but on the same virtual path (mounted to the
> same location inside a docker container) extract it. Then,
> applications are built outside of Buildroot, but stored to a copy of
> the output/target directory. Then a subsequent 'make' is run to let
> Buildroot package everything into an initramfs, setting TARGET_DIR to
> the copy.

 SDK is not going to work for that last bit.

 The problem is that overall, Buildroot doesn't have a great story for
rebuilding things from a partial build. To be honest, I think you're better off
not using Buildroot to package the initramfs, but just make a simple script that
you run in fakeroot:

tar -x rootfs.tar
rsync custom packages
cpio ...


 That is, assuming that the custom packages don't need to do things like adding
users etc. But if they're built outside of buildroot, that's not going to work
anyway.

 Regards,
 Arnout

> During all these subsequent builds, the host directory and
> some others are kept read-only to prevent corruption of that 'cache'.
> 
> Best regards,
> Thomas
> 

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Thomas De Schampheleire]

El vie., 6 dic. 2019 a las 12:31, Arnout Vandecappelle
(<arnout@mind.be>) escribi?:
>
>
>
> On 06/12/2019 11:16, Thomas De Schampheleire wrote:
> > El vie., 6 dic. 2019 a las 10:57, Thomas Petazzoni
> > (<thomas.petazzoni@bootlin.com>) escribi?:
> >>
> >> On Fri, 6 Dec 2019 10:54:13 +0100
> >> Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote:
> >>
> >>> Needless to say, I am opposed to moving the meson file back from the
> >>> target-finalize step :-)
> >>> I think most of the reasoning was already mentioned in that commit,
> >>> but it seems I did not explain that we actually have HOST_DIR mounted
> >>> read-only on subsequent 'make' commands after the initial make,
> >>> exactly to verify that no-one is changing directories they shouldn't
> >>> be changing. After all, a 'make' after the initial make will only
> >>> normally do the target-finalize step.
> >>
> >> No, it will do staging-finalize as well. I don't think we provide the
> >> guarantee that HOST_DIR is unchanged/read-only between each "make"
> >> invocation, and I'm not sure why we would want to provide this
> >> guarantee.
> >
> > The fact that staging-finalize is triggered from target-finalize is
> > also a problem in my use case, and I have cleared it in our local copy
> > (moving its action to another place). I think we talked about that
> > earlier too.
> >
> > I think Yann suggested me to look into the sdk in this context. I need
> > to check it in more detail soon, to see whether it can really solve
> > this friction we have between our current needs and changes upstream.
> >
> > To recap our use case: we make an initial Buildroot build, package it,
> > then in other machines but on the same virtual path (mounted to the
> > same location inside a docker container) extract it. Then,
> > applications are built outside of Buildroot, but stored to a copy of
> > the output/target directory. Then a subsequent 'make' is run to let
> > Buildroot package everything into an initramfs, setting TARGET_DIR to
> > the copy.
>
>  SDK is not going to work for that last bit.
>
>  The problem is that overall, Buildroot doesn't have a great story for
> rebuilding things from a partial build. To be honest, I think you're better off
> not using Buildroot to package the initramfs, but just make a simple script that
> you run in fakeroot:
>
> tar -x rootfs.tar
> rsync custom packages
> cpio ...
>
>
>  That is, assuming that the custom packages don't need to do things like adding
> users etc. But if they're built outside of buildroot, that's not going to work
> anyway.

Aside from the initramfs creation, we also rely on the other stuff
happening in target-finalize, like stripping of objects,
rootfs-overlays, post-build scripts, ... Duplicating this logic is not
ideal.

But perhaps I could solve my use case by making the dependency of
target-finalize on host-finalize and staging-finalize optional,
steerable via a variable. Then I could use the default flow for the
initial build, and pass the steer-variable to cut the dependency on
the subsequent make. I'd need to try it to make sure it fits what I
need.
This modification may or may not be acceptable for upstreaming, but
probably is easier than trying to force you in keeping
staging-finalize and host-finalize empty.

Best regards,
Thomas

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Yann E. MORIN]

Thomas?, All,

On 2019-12-04 16:02 +0100, Thomas Petazzoni spake thusly:
> Currently, the cross-compilation.conf installed in
> $(HOST_DIR)/etc/meson/cross-compilation.conf for use by the SDK is
> generated in a post-install-staging hook of the toolchain package.
> 
> With per-package directory support enabled, this means that the
> generated cross-compilation.conf contains references to the
> per-package directory of the toolchain/ package, which is not want we
> want:
> 
> [binaries]
> c = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-gcc'
> cpp = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-g++'
> ar = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-ar'
> strip = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-strip'
> pkgconfig = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/usr/bin/pkg-config'
> 
> So instead, we generate this file in TOOLCHAIN_TARGET_FINALIZE_HOOKS,
> so that the global paths are used:
> 
> [binaries]
> c = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-gcc'
> cpp = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-g++'
> ar = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-ar'
> strip = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-strip'
> pkgconfig = '/home/thomas/projets/buildroot/output/host/usr/bin/pkg-config'
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

After giving it quite some thoughts, I think we do not want to provide
any guarantee about the stableness of target/ vs. host/ on a partial
rebuild, like Thomas P. said.

As such, I have applied this patch now.

Regards,
Yann E. MORIN.

> ---
>  package/pkg-meson.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/pkg-meson.mk b/package/pkg-meson.mk
> index e7eea2aa58..085da219fe 100644
> --- a/package/pkg-meson.mk
> +++ b/package/pkg-meson.mk
> @@ -205,4 +205,4 @@ define PKG_MESON_INSTALL_CROSS_CONF
>  	    > $(HOST_DIR)/etc/meson/cross-compilation.conf
>  endef
>  
> -TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
> +TOOLCHAIN_TARGET_FINALIZE_HOOKS += PKG_MESON_INSTALL_CROSS_CONF
> -- 
> 2.23.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] package/pkg-meson: ensure the global cross-compilation.conf file is correct

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Thomas?, All,
 > On 2019-12-04 16:02 +0100, Thomas Petazzoni spake thusly:
 >> Currently, the cross-compilation.conf installed in
 >> $(HOST_DIR)/etc/meson/cross-compilation.conf for use by the SDK is
 >> generated in a post-install-staging hook of the toolchain package.
 >> 
 >> With per-package directory support enabled, this means that the
 >> generated cross-compilation.conf contains references to the
 >> per-package directory of the toolchain/ package, which is not want we
 >> want:
 >> 
 >> [binaries]
 >> c = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-gcc'
 >> cpp = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-g++'
 >> ar = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-ar'
 >> strip = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/bin/arm-linux-strip'
 >> pkgconfig = '/home/thomas/projets/buildroot/output/per-package/toolchain/host/usr/bin/pkg-config'
 >> 
 >> So instead, we generate this file in TOOLCHAIN_TARGET_FINALIZE_HOOKS,
 >> so that the global paths are used:
 >> 
 >> [binaries]
 >> c = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-gcc'
 >> cpp = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-g++'
 >> ar = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-ar'
 >> strip = '/home/thomas/projets/buildroot/output/host/bin/arm-linux-strip'
 >> pkgconfig = '/home/thomas/projets/buildroot/output/host/usr/bin/pkg-config'
 >> 
 >> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

 > After giving it quite some thoughts, I think we do not want to provide
 > any guarantee about the stableness of target/ vs. host/ on a partial
 > rebuild, like Thomas P. said.

 > As such, I have applied this patch now.

Committed to 2020.02.x, 2020.05.x and 2020.08.x, thanks.

-- 
Bye, Peter Korsgaard