[Buildroot] pre-built binary packages

[Buildroot] pre-built binary packages

[Marc Murphy]

Hello all,
I am wondering if it is possible to generate binary releases of packages to be included in the build tree.  For example I have a system that is rooted in 2017.02 and still has work done on some packages but when updating releases from other engineers the changes can get conflicted or messed up resulting in a fresh rebuild.  This normally isn't a problem until Qt is in the equation.  For me once Qt is configured built that's it I shouldn't need to build it again.  Is there any way to 
1) export or package the binaries and headers for module archiving
2) use the prebuilt binaries and headers in a new build to prevent having to rebuild Qt again

I think it could be a useful feature to create controlled releases or to speed up builds.

Thanks
Marc

[Buildroot] pre-built binary packages

[Thomas Petazzoni]

Hello,

On Wed, 11 Oct 2017 11:00:11 +0000, Marc Murphy wrote:

> I am wondering if it is possible to generate binary releases of
> packages to be included in the build tree.  For example I have a
> system that is rooted in 2017.02 and still has work done on some
> packages but when updating releases from other engineers the changes
> can get conflicted or messed up resulting in a fresh rebuild.  This
> normally isn't a problem until Qt is in the equation.  For me once Qt
> is configured built that's it I shouldn't need to build it again.  Is
> there any way to 1) export or package the binaries and headers for
> module archiving 2) use the prebuilt binaries and headers in a new
> build to prevent having to rebuild Qt again
> 
> I think it could be a useful feature to create controlled releases or
> to speed up builds.

Buildroot does not implement any mechanism to "cache" build results. It
is quite complicated to do it right, because you have to figure out
when an existing build result can be re-used from the cache or should
be rebuilt. Indeed, "building Qt" doesn't mean anything. With which
configuration was it built? For which host architecture and which
target architecture? With which target flags and optimization?

If you want to speed up your build with Buildroot, consider:

 - Using a faster build machine (more SSD, more RAM)
 - Using an external toolchain
 - Using ccache

If you want a build system that caches existing build artifacts, then
you should use OpenEmbedded/Yocto.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] pre-built binary packages

[Henrique Marks]

Hello,

> Hello,
> 
> On Wed, 11 Oct 2017 11:00:11 +0000, Marc Murphy wrote:
> 
>> I am wondering if it is possible to generate binary releases of
>> packages to be included in the build tree.  For example I have a
>> system that is rooted in 2017.02 and still has work done on some
>> packages but when updating releases from other engineers the changes
>> can get conflicted or messed up resulting in a fresh rebuild.  This
>> normally isn't a problem until Qt is in the equation.  For me once Qt
>> is configured built that's it I shouldn't need to build it again.  Is
>> there any way to 1) export or package the binaries and headers for
>> module archiving 2) use the prebuilt binaries and headers in a new
>> build to prevent having to rebuild Qt again
>> 
>> I think it could be a useful feature to create controlled releases or
>> to speed up builds.
> 
> Buildroot does not implement any mechanism to "cache" build results. It
> is quite complicated to do it right, because you have to figure out
> when an existing build result can be re-used from the cache or should
> be rebuilt. Indeed, "building Qt" doesn't mean anything. With which
> configuration was it built? For which host architecture and which
> target architecture? With which target flags and optimization?
> 
> If you want to speed up your build with Buildroot, consider:
> 
> - Using a faster build machine (more SSD, more RAM)
> - Using an external toolchain
> - Using ccache
> 
> If you want a build system that caches existing build artifacts, then
> you should use OpenEmbedded/Yocto.
> 

Here in Datacom we use "binary artifacts": when we build a package, we make a tarball of the artifacts. When we want to build the package, we try to use the tarball first, if the version (in the tarball) matches the package version.

We have not contributed this solution to buildroot because it doesn't seem to be possible to do this in a generic way. We have this binary artifact generation for our internal packages (and all of them use cmake to build). I think it is the only patch to buildroot that we consider to be "eternal".

You can do the same for any package: you can control what's installed in staging and target, and create a tarball containing these folders and the stamp files. Then, use this Binary, instead of the source, when the binary can be used (same source version, for instance).

So, a compilation in our systems always starts with a "build form binaries" and then "rebuild some parts from source". That's way we would like the python script that implements show-<pkg>-rrdepends to be accepted upstream: combined with binary artifacts, it is possible to build fast (from binaries) and rebuild fast (from source).

-- 
Dr. Henrique Marks
henrique.marks at datacom.ind.br
R. Am?rica, 1000 - Eldorado do Sul - RS
CEP: 92990-000 - Brasil
Fone: +55 51 3933 3000 - Ramal 3466

[Buildroot] pre-built binary packages

[Thomas Petazzoni]

Hello,

On Wed, 11 Oct 2017 18:36:14 -0300 (BRT), Henrique Marks wrote:

> Here in Datacom we use "binary artifacts": when we build a package,
> we make a tarball of the artifacts. When we want to build the
> package, we try to use the tarball first, if the version (in the
> tarball) matches the package version.
> 
> We have not contributed this solution to buildroot because it doesn't
> seem to be possible to do this in a generic way. We have this binary
> artifact generation for our internal packages (and all of them use
> cmake to build). I think it is the only patch to buildroot that we
> consider to be "eternal".
> 
> You can do the same for any package: you can control what's installed
> in staging and target, and create a tarball containing these folders
> and the stamp files. Then, use this Binary, instead of the source,
> when the binary can be used (same source version, for instance).
> 
> So, a compilation in our systems always starts with a "build form
> binaries" and then "rebuild some parts from source". That's way we
> would like the python script that implements show-<pkg>-rrdepends to
> be accepted upstream: combined with binary artifacts, it is possible
> to build fast (from binaries) and rebuild fast (from source).

The problem is that you don't have any reliable way to know if your
binary artifacts are valid for a given build or not.

Let's say you have a binary artifact of Qt5. But that artifact was
created with a given set of Qt5 configuration options. If you change
any Qt5 configuration option in Buildroot, then your binary artifact
should no longer be used.

The situation becomes worse with optional dependencies. You have built
package FOO, which optionally depends on BAR, but BAR was disabled. You
create a binary artifact of FOO, and re-use it to avoid rebuilding.
Later on, you enable BAR. How do you notice that your binary artifact
of FOO is no longer correct and should be rebuilt ?

So yes, you can use binary artifact, but only if you are very, very,
very, very, very careful, and you have a full and complete
understanding of what you're doing. You can shoot yourself in the foot
very easily.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] pre-built binary packages

[Marc Murphy]

>-----Original Message-----
>From: Henrique Marks [mailto:henrique.marks at datacom.ind.br]
>Sent: 11 October 2017 22:36
>To: Thomas Petazzoni
>Cc: Marc Murphy; buildroot at busybox.net
>Subject: Re: [Buildroot] pre-built binary packages
>
>Hello,
>
>> Hello,
>>
>> On Wed, 11 Oct 2017 11:00:11 +0000, Marc Murphy wrote:
>>
>>> I am wondering if it is possible to generate binary releases of
>>> packages to be included in the build tree.  For example I have a
>>> system that is rooted in 2017.02 and still has work done on some
>>> packages but when updating releases from other engineers the changes
>>> can get conflicted or messed up resulting in a fresh rebuild.  This
>>> normally isn't a problem until Qt is in the equation.  For me once Qt
>>> is configured built that's it I shouldn't need to build it again.  Is
>>> there any way to 1) export or package the binaries and headers for
>>> module archiving 2) use the prebuilt binaries and headers in a new
>>> build to prevent having to rebuild Qt again
>>>
>>> I think it could be a useful feature to create controlled releases or
>>> to speed up builds.
>>
>> Buildroot does not implement any mechanism to "cache" build results.
>> It is quite complicated to do it right, because you have to figure out
>> when an existing build result can be re-used from the cache or should
>> be rebuilt. Indeed, "building Qt" doesn't mean anything. With which
>> configuration was it built? For which host architecture and which
>> target architecture? With which target flags and optimization?
>>
>> If you want to speed up your build with Buildroot, consider:
>>
>> - Using a faster build machine (more SSD, more RAM)
>> - Using an external toolchain
>> - Using ccache
>>
>> If you want a build system that caches existing build artifacts, then
>> you should use OpenEmbedded/Yocto.
>>
>
>Here in Datacom we use "binary artifacts": when we build a package, we make
>a tarball of the artifacts. When we want to build the package, we try to use
>the tarball first, if the version (in the tarball) matches the package version.
>
>We have not contributed this solution to buildroot because it doesn't seem to
>be possible to do this in a generic way. We have this binary artifact generation
>for our internal packages (and all of them use cmake to build). I think it is the
>only patch to buildroot that we consider to be "eternal".
>
>You can do the same for any package: you can control what's installed in
>staging and target, and create a tarball containing these folders and the stamp
>files. Then, use this Binary, instead of the source, when the binary can be
>used (same source version, for instance).
>
>So, a compilation in our systems always starts with a "build form binaries" and
>then "rebuild some parts from source". That's way we would like the python
>script that implements show-<pkg>-rrdepends to be accepted upstream:
>combined with binary artifacts, it is possible to build fast (from binaries) and
>rebuild fast (from source).
>

Exactly what I am trying to achieve, if you have a number of products based on the same hardware base and some common components it makes sense to build one and reuse.
For me building Qt configured for the arm platform I am using and then deploying the binary tar for new configurations on the same platform makes sense.

Do you have a patch I could try and see if it works for me ?

Kind Regards
Marc

>--
>Dr. Henrique Marks
>henrique.marks at datacom.ind.br
>R. Am?rica, 1000 - Eldorado do Sul - RS
>CEP: 92990-000 - Brasil
>Fone: +55 51 3933 3000 - Ramal 3466

[Buildroot] pre-built binary packages

[Henrique Marks]

 
>> Here in Datacom we use "binary artifacts": when we build a package,
>> we make a tarball of the artifacts. When we want to build the
>> package, we try to use the tarball first, if the version (in the
>> tarball) matches the package version.
>> 
>> We have not contributed this solution to buildroot because it doesn't
>> seem to be possible to do this in a generic way. We have this binary
>> artifact generation for our internal packages (and all of them use
>> cmake to build). I think it is the only patch to buildroot that we
>> consider to be "eternal".
>> 
>> You can do the same for any package: you can control what's installed
>> in staging and target, and create a tarball containing these folders
>> and the stamp files. Then, use this Binary, instead of the source,
>> when the binary can be used (same source version, for instance).
>> 
>> So, a compilation in our systems always starts with a "build form
>> binaries" and then "rebuild some parts from source". That's way we
>> would like the python script that implements show-<pkg>-rrdepends to
>> be accepted upstream: combined with binary artifacts, it is possible
>> to build fast (from binaries) and rebuild fast (from source).
> 
> The problem is that you don't have any reliable way to know if your
> binary artifacts are valid for a given build or not.
> 
> Let's say you have a binary artifact of Qt5. But that artifact was
> created with a given set of Qt5 configuration options. If you change
> any Qt5 configuration option in Buildroot, then your binary artifact
> should no longer be used.
> 
> The situation becomes worse with optional dependencies. You have built
> package FOO, which optionally depends on BAR, but BAR was disabled. You
> create a binary artifact of FOO, and re-use it to avoid rebuilding.
> Later on, you enable BAR. How do you notice that your binary artifact
> of FOO is no longer correct and should be rebuilt ?
> 
> So yes, you can use binary artifact, but only if you are very, very,
> very, very, very careful, and you have a full and complete
> understanding of what you're doing. You can shoot yourself in the foot
> very easily.
> 
> Best regards,
> 

Yes, that's why it is not "upstreamable".

We have an integration process, that publish new binary artifacts for all packages and all platforms. It is extremely controlled.

But, i disagree about "shooting yourself in the foot easily": it never happened, in some YEARS using it. Or we are very special (very smart) or it is not so hard. The second option is the right one.

I would say to Marc try it. An example (qt5base.mk)

define QT5BASE_INSTALL_STAGING_CMDS
        $(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
        $(QT5_LA_PRL_FILES_FIXUP)
endef

define QT5BASE_INSTALL_TARGET_EXAMPLES
        if [ -d $(STAGING_DIR)/usr/lib/qt/examples/ ] ; then \
                mkdir -p $(TARGET_DIR)/usr/lib/qt/examples ; \
                cp -dpfr $(STAGING_DIR)/usr/lib/qt/examples/* $(TARGET_DIR)/usr/lib/qt/examples ; \
        fi
endef

define FAKE_STAGING
define FAKE_TARGET

and

use this FAKE_STAGING in place of (@D) in the STAGING part and FAKE_TARGET in the TARGET part. Something like this:

define QT5BASE_INSTALL_STAGING_CMDS
        $(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
        $(TARGET_MAKE_ENV) $(MAKE) -C $(FAKE_STAGING) install
        $(QT5_LA_PRL_FILES_FIXUP)
endef

define QT5BASE_INSTALL_TARGET_EXAMPLES
        if [ -d $(STAGING_DIR)/usr/lib/qt/examples/ ] ; then \
                mkdir -p $(TARGET_DIR)/usr/lib/qt/examples ; \
                cp -dpfr $(STAGING_DIR)/usr/lib/qt/examples/* $(TARGET_DIR)/usr/lib/qt/examples ; \
                mkdir -p $(FAKE_TARGET_DIR)/usr/lib/qt/examples ; \
                cp -dpfr $(STAGING_DIR)/usr/lib/qt/examples/* $(FAKE_TARGET)/usr/lib/qt/examples ; \
        fi
endef

When you build, you will have these artifacts in the fake staging and target. Create a tarball (in the recipe, when building from source), and use the tarball (in the recipe, when building from binary), and that's it (put the stamp files in the tarball also, of course).

-- 
Dr. Henrique Marks
henrique.marks at datacom.ind.br
R. Am?rica, 1000 - Eldorado do Sul - RS
CEP: 92990-000 - Brasil
Fone: +55 51 3933 3000 - Ramal 3466

[Buildroot] pre-built binary packages

[Arnout Vandecappelle]

 Hi all,

On 12-10-17 00:25, Henrique Marks wrote:
[snip]
>> Let's say you have a binary artifact of Qt5. But that artifact was
>> created with a given set of Qt5 configuration options. If you change
>> any Qt5 configuration option in Buildroot, then your binary artifact
>> should no longer be used.

 If I understand Henrique correctly, what they do in such a case is that they
"manually" remove the artifact in that case, and additionally remove any package
that depends on qt5 with
make qt5-rrdepends | sed -s 's/$/-artifact.tar.gz/' | xargs rm -f

 So if you follow the rules (manually clean the artifacts and their dependencies
whenever you change anything), it works fine.

 It is of course a problem when there is a dependency that Buildroot doesn't
know about. But that will be solved with per-package staging.


>> The situation becomes worse with optional dependencies. You have built
>> package FOO, which optionally depends on BAR, but BAR was disabled. You
>> create a binary artifact of FOO, and re-use it to avoid rebuilding.
>> Later on, you enable BAR. How do you notice that your binary artifact
>> of FOO is no longer correct and should be rebuilt ?

 This one is covered by bar-rrdepends, because after the config change foo will
be in bar-rrdepends. What doesn't work, however, is the reverse: you enable
openssl and botan, you build. Then you realize you don't need openssl so you
remove it and do a clean rebuild. Now botan-rrdepends does NOT contain openssl,
so the botan artifact will be reused. You start botan and it doesn't find
openssl.so...


>> So yes, you can use binary artifact, but only if you are very, very,
>> very, very, very careful, and you have a full and complete
>> understanding of what you're doing. You can shoot yourself in the foot
>> very easily.
>>
>> Best regards,
>>
> 
> Yes, that's why it is not "upstreamable".
> 
> We have an integration process, that publish new binary artifacts for all packages and all platforms. It is extremely controlled.
> 
> But, i disagree about "shooting yourself in the foot easily": it never happened, in some YEARS using it. Or we are very special (very smart) or it is not so hard. The second option is the right one.

 First of all, you are smart because you do properly clean artifacts when needed
(except in the cases I mentioned, but perhaps you cover even those). Second, I
think that when you do have an accident, you will indeed know what was the
problem and fixed it by just removing the artifact. It's so easy you hardly
would have noticed.


 There is one more element that is tricky that Thomas didn't mention:
STAGING_DIR is full of absolute paths, so you must always build in the same
directory. Sometimes it works because almost everything is installed in
$(STAGING_DIR)/usr/lib, but there is a reasonable risk that you get into trouble
when you change the build directory.


> I would say to Marc try it. An example (qt5base.mk)
> 
> define QT5BASE_INSTALL_STAGING_CMDS
>         $(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
>         $(QT5_LA_PRL_FILES_FIXUP)
> endef
> 
> define QT5BASE_INSTALL_TARGET_EXAMPLES
>         if [ -d $(STAGING_DIR)/usr/lib/qt/examples/ ] ; then \
>                 mkdir -p $(TARGET_DIR)/usr/lib/qt/examples ; \
>                 cp -dpfr $(STAGING_DIR)/usr/lib/qt/examples/* $(TARGET_DIR)/usr/lib/qt/examples ; \
>         fi
> endef
> 
> define FAKE_STAGING
> define FAKE_TARGET
> 
> and
> 
> use this FAKE_STAGING in place of (@D) in the STAGING part and FAKE_TARGET in the TARGET part. Something like this:
> 
> define QT5BASE_INSTALL_STAGING_CMDS
>         $(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
>         $(TARGET_MAKE_ENV) $(MAKE) -C $(FAKE_STAGING) install
>         $(QT5_LA_PRL_FILES_FIXUP)

 You'll want to do the QT5_LA_PRL_FILES_FIXUP as well, no?

> endef
> 
> define QT5BASE_INSTALL_TARGET_EXAMPLES
>         if [ -d $(STAGING_DIR)/usr/lib/qt/examples/ ] ; then \
>                 mkdir -p $(TARGET_DIR)/usr/lib/qt/examples ; \
>                 cp -dpfr $(STAGING_DIR)/usr/lib/qt/examples/* $(TARGET_DIR)/usr/lib/qt/examples ; \
>                 mkdir -p $(FAKE_TARGET_DIR)/usr/lib/qt/examples ; \
>                 cp -dpfr $(STAGING_DIR)/usr/lib/qt/examples/* $(FAKE_TARGET)/usr/lib/qt/examples ; \
>         fi
> endef

 You don't even need to do this. Just replace the definition of STAGING_DIR and
TARGET_DIR:

GLOBAL_STAGING_DIR = $(HOST_DIR)/$(STAGING_SUBDIR)
STAGING_DIR = $(if $(PKG),$($(PKG)_STAGING_DIR),$(GLOBAL_STAGING_DIR))

 Then add an install-staging-end and install-target-end step that makes a
tarball of $($(PKG)_STAGING_DIR) and extracts it into $(GLOBAL_STAGING_DIR).

 There's one problem with this global approach: there are some packages that do
tricky things, like installing stuff in host-dir during target-install.So these
will have to be worked around.

 The good news is that Thomas is (AFAIU) paid to do exactly that right now.

 Regards,
 Arnout


> When you build, you will have these artifacts in the fake staging and target. Create a tarball (in the recipe, when building from source), and use the tarball (in the recipe, when building from binary), and that's it (put the stamp files in the tarball also, of course).


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF