[Buildroot] [RFC] FAQ entry about why no binary packages

[Buildroot] [RFC] FAQ entry about why no binary packages

[Yann E. MORIN]

Hello All!

Now and then, someone will ask why Buildroot has no support for generating
binary packages.

Rather than re-hash the same answer every time, add a section in the FAQ
in the manual, that we can point users to.

This is an RFC, the text is not complete/accurate, and just a copy-paste
of a mail I sent a bit earlier on the list. The goal is that people
start adding to this section, to make it simple and complete, and not
too long.

    [PATCH] manual/faq: add section about why no binary packages

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] manual/faq: add section about why no binary packages

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

It somes up evry now and then on the list, so better be prepared to
point at the manual, rather than rehash the same evry time.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
 docs/manual/faq-troubleshooting.txt | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/docs/manual/faq-troubleshooting.txt b/docs/manual/faq-troubleshooting.txt
index 4e0612b..e97457a 100644
--- a/docs/manual/faq-troubleshooting.txt
+++ b/docs/manual/faq-troubleshooting.txt
@@ -111,3 +111,24 @@ directory as the new root, will most likely fail.
 If you want to run the target filesystem inside a chroot, or as an NFS
 root, then use the tarball image generated in +images/+ and extract it
 as root.
+
+[[faq-no-binary-packages]]
+Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
+----------------------------------------------------------------
+
+Buildroot explicitly avoids generating binary packages, on purpose.
+
+Buildroot strive at making it easy to generate a root filesystem (hence
+the name, by the way). That is what we want to make Buildroot good at:
+building root filesystems.
+
+Buildroot is not meant to be a distribution (or rather, a distribution
+generator). It is the opinion of most Buildroot developpers (me included)
+that this is not a goal we should pursue. We believe that there are other
+tools better suited to generate a distro than Buildroot is. For example,
+http://openembedded.org/[Open Embedded], or https://openwrt.org/[openWRT],
+are such tools.
+
+Rather, we prefer to push Buildroot in a direction that makes it easy
+(or even easier) to generate complete root filesystems. This is what
+makes Buildroot stands out in the crowd (among other things, of course!).
-- 
1.8.1.2

[Buildroot] [PATCH] manual/faq: add section about why no binary packages

[Samuel Martin]

Hi Yann, all,

2014/1/5 Yann E. MORIN <yann.morin.1998@free.fr>

> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> It somes up evry now and then on the list, so better be prepared to
> point at the manual, rather than rehash the same evry time.
>
s/evry/every/ (twice)
hmm... you typed faster than your keyboard can handle? ;-)


>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  docs/manual/faq-troubleshooting.txt | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
>
> diff --git a/docs/manual/faq-troubleshooting.txt
> b/docs/manual/faq-troubleshooting.txt
> index 4e0612b..e97457a 100644
> --- a/docs/manual/faq-troubleshooting.txt
> +++ b/docs/manual/faq-troubleshooting.txt
> @@ -111,3 +111,24 @@ directory as the new root, will most likely fail.
>  If you want to run the target filesystem inside a chroot, or as an NFS
>  root, then use the tarball image generated in +images/+ and extract it
>  as root.
> +
> +[[faq-no-binary-packages]]
> +Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
> +----------------------------------------------------------------
> +
> +Buildroot explicitly avoids generating binary packages, on purpose.
> +
> +Buildroot strive at making it easy to generate a root filesystem (hence
> +the name, by the way). That is what we want to make Buildroot good at:
> +building root filesystems.
> +
> +Buildroot is not meant to be a distribution (or rather, a distribution
> +generator). It is the opinion of most Buildroot developpers (me included)
>
s/(me included)//
Well, this is the manual, not a mail answer; so it'd be better to make this
impersonal.

+that this is not a goal we should pursue. We believe that there are other
> +tools better suited to generate a distro than Buildroot is. For example,
> +http://openembedded.org/[Open Embedded], or https://openwrt.org/[openWRT]
> ,
> +are such tools.
> +
> +Rather, we prefer to push Buildroot in a direction that makes it easy
> +(or even easier) to generate complete root filesystems. This is what
> +makes Buildroot stands out in the crowd (among other things, of course!).
> --
> 1.8.1.2
>
>
I think i have something roughly about this and other in one of my
doc-branches, but i lack time to work on it :-/
Meanwhile, an entry in the FAQ is good enough to me.

Regards,

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140106/14431e78/attachment.html>

[Buildroot] [PATCH] manual/faq: add section about why no binary packages

[Baruch Siach]

Hi Yann,

On Sun, Jan 05, 2014 at 09:44:23PM +0100, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> It somes up evry now and then on the list, so better be prepared to
> point at the manual, rather than rehash the same evry time.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  docs/manual/faq-troubleshooting.txt | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
> 
> diff --git a/docs/manual/faq-troubleshooting.txt b/docs/manual/faq-troubleshooting.txt
> index 4e0612b..e97457a 100644
> --- a/docs/manual/faq-troubleshooting.txt
> +++ b/docs/manual/faq-troubleshooting.txt
> @@ -111,3 +111,24 @@ directory as the new root, will most likely fail.
>  If you want to run the target filesystem inside a chroot, or as an NFS
>  root, then use the tarball image generated in +images/+ and extract it
>  as root.
> +
> +[[faq-no-binary-packages]]
> +Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
> +----------------------------------------------------------------
> +
> +Buildroot explicitly avoids generating binary packages, on purpose.
> +
> +Buildroot strive at making it easy to generate a root filesystem (hence
> +the name, by the way). That is what we want to make Buildroot good at:
> +building root filesystems.
> +
> +Buildroot is not meant to be a distribution (or rather, a distribution
> +generator). It is the opinion of most Buildroot developpers (me included)
> +that this is not a goal we should pursue. We believe that there are other
> +tools better suited to generate a distro than Buildroot is. For example,
> +http://openembedded.org/[Open Embedded], or https://openwrt.org/[openWRT],
> +are such tools.
> +
> +Rather, we prefer to push Buildroot in a direction that makes it easy
> +(or even easier) to generate complete root filesystems. This is what
> +makes Buildroot stands out in the crowd (among other things, of course!).

Thanks for working on this. I think it is also worth mentioning the rationale. 
By doing source only we avoid the complexity of handling installation time 
dependency tracking and resolution. We also don't need to track what files 
each package installs. This makes Buildroot simpler and easier to work with.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [PATCH] manual/faq: add section about why no binary packages

[Thomas Petazzoni]

Dear Baruch Siach,

On Mon, 6 Jan 2014 07:16:25 +0200, Baruch Siach wrote:

> Thanks for working on this. I think it is also worth mentioning the rationale. 
> By doing source only we avoid the complexity of handling installation time 
> dependency tracking and resolution. We also don't need to track what files 
> each package installs. This makes Buildroot simpler and easier to work with.

I would go even further, and explain why tracking what files each
package installs is by far not sufficient to support binary packages.
Several people have showed up throughout the project history, willing
to add support binary packages by assuming that simply tracking which
files "make install" installs will be sufficient. But that's forgetting
all the optional dependencies problems, and various other things.

We had a write-up about this in some report of a past Buildroot
Developers Meeting, with some good arguments. Would be nice to dig that
up and summarize these arguments in the doc.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH] manual/faq: add section about why no binary packages

[Thomas Petazzoni]

Hello,

On Mon, 6 Jan 2014 13:26:38 +0800, Thomas Petazzoni wrote:

> I would go even further, and explain why tracking what files each
> package installs is by far not sufficient to support binary packages.
> Several people have showed up throughout the project history, willing
> to add support binary packages by assuming that simply tracking which
> files "make install" installs will be sufficient. But that's forgetting
> all the optional dependencies problems, and various other things.
> 
> We had a write-up about this in some report of a past Buildroot
> Developers Meeting, with some good arguments. Would be nice to dig that
> up and summarize these arguments in the doc.

The report at
http://lists.busybox.net/pipermail/buildroot/2011-November/047229.html
has the details I was referring to. I'm copy/pasting them here:

===

Package management
------------------

On the feature that is often discussed on the Buildroot list, and
which was on the agenda for this meeting was the general topic of
"package management". To summarize, the idea would be to add some
tracking of which Buildroot package installs what files, with the
goals of :

 * Being able to remove files installed by a package when this package
   gets unselected from the menuconfig ;

 * Ultimately, be able to generate binary packages (ipk or other
   format) that can be installed on the target without re-generating a
   new root filesystem image.

In general, most people think it is easy to do: just track which
package installed what and remove it when the package is
unselected. However, it is much more complicated than that:

 * It is not only about the target/ directory, but also the sysroot in
   host/usr/<tuple>/sysroot and the host directory itself. All files
   installed in those directories by various packages must be tracked.

 * When a package is removed, it is not sufficient to remove just the
   files it installed. One must also remove all its reverse
   dependencies (i.e packages relying on it) and rebuild all those
   packages. For example, package A depends optionally on the OpenSSL
   library. Both are selected, and Buildroot is built. Package A is
   built with crypto support using OpenSSL. Later on, OpenSSL gets
   unselected from the configuration, but package A remains (since
   OpenSSL is an optional dependency, this is possible). If you just
   remove the OpenSSL files, then the files installed by package A are
   broken: they use a library that is no longer present on the
   target. Technically, it is possible to do this (the prototype that
   Lionel Landwerlin and Thomas Petazzoni have worked on started to do
   this), but it is difficult and adds a fair bit of complexity.

 * In addition to the previous problem, there is the case where the
   optional dependency is not even known to Buildroot. For example,
   package A in version 1.0 never used OpenSSL, but in version 2.0 it
   automatically uses OpenSSL if available. If the Buildroot .mk file
   hasn't been updated to take this into account, then package A will
   not be part of the reverse dependencies of OpenSSL and will not be
   removed and rebuilt when OpenSSL is removed. For sure, the .mk file
   of package A should be fixed to mention this optional dependency,
   but in the mean time, you can have non-reproducible behaviors.

 * The whole idea is also to allow changes in the menuconfig to be
   applied on the output directory without having to rebuild
   everything from scratch. However, this is very difficult to achieve
   in a reliable way: what happens when the suboptions of a package
   are changed (we would have to detect this, and rebuild the package
   from scratch and potentially all its reverse dependencies), what
   happens if toolchain options are changed, etc. At the moment, what
   Buildroot does is clear and simple so its behaviour is very
   reliable and it is easy to support users. If we start telling users
   that the configuration changes done in menuconfig are applied after
   the next make, then it has to work correctly and properly in all
   situations, and not have some bizarre corner cases. We fear bug
   reports like "I have enabled package A, B and C, then ran make,
   then disabled package C and enabled package D and ran make, then
   re-enabled package C and enabled package E and then there is a
   build failure". Or worse "I did some configuration, then built,
   then did some changes, built, some more changes, built, some more
   changes, built, and now it fails, but I don't remember all the
   changes I did and in which order". This will be impossible to
   support.

For all these reasons, the conclusion of the Buildroot Developer Day
was that adding tracking of installed files to remove them when the
package is unselected is something that is very hard to achieve
reliably and will add a lot of complexity.

In the morning, we had some discussion with Robert Schwebel about how
PTXdist does package management. They only do it for files installed
in the target filesystem. So for example the libraries/headers are
never removed from their sysroot, so you quickly end up with
inconsistencies. This is something we would like to avoid in
Buildroot, because it creates confusing behaviors in our
opinion. Esben from OE-lite also said that package management is very
difficult to do reliably and that it would add a lot of complexity to
a currently relatively simple Buildroot.

Buildroot focus is on simplicity and building relatively small
systems, and this is definitely something we want to preserve. Moving
away from this principle would make Buildroot more similar to existing
more complicated build systems and therefore less interesting.

For the time being, we'd prefer not to add such mechanisms in
Buildroot and keep its behavior as simple and easy to understand as it
is today. We think it's better to focus on other features than trying
to implement something that will never be completely reliable and will
add a very significant complexity to Buildroot.

===

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH] manual/faq: add section about why no binary packages

[Thomas De Schampheleire]

On Sun, Jan 5, 2014 at 9:44 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> It somes up evry now and then on the list, so better be prepared to
> point at the manual, rather than rehash the same evry time.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> ---
>  docs/manual/faq-troubleshooting.txt | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
>
> diff --git a/docs/manual/faq-troubleshooting.txt b/docs/manual/faq-troubleshooting.txt
> index 4e0612b..e97457a 100644
> --- a/docs/manual/faq-troubleshooting.txt
> +++ b/docs/manual/faq-troubleshooting.txt
> @@ -111,3 +111,24 @@ directory as the new root, will most likely fail.
>  If you want to run the target filesystem inside a chroot, or as an NFS
>  root, then use the tarball image generated in +images/+ and extract it
>  as root.
> +
> +[[faq-no-binary-packages]]
> +Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
> +----------------------------------------------------------------
> +
> +Buildroot explicitly avoids generating binary packages, on purpose.
> +
> +Buildroot strive at making it easy to generate a root filesystem (hence

strives

> +the name, by the way). That is what we want to make Buildroot good at:
> +building root filesystems.
> +
> +Buildroot is not meant to be a distribution (or rather, a distribution
> +generator). It is the opinion of most Buildroot developpers (me included)

developers

> +that this is not a goal we should pursue. We believe that there are other
> +tools better suited to generate a distro than Buildroot is. For example,
> +http://openembedded.org/[Open Embedded], or https://openwrt.org/[openWRT],
> +are such tools.
> +
> +Rather, we prefer to push Buildroot in a direction that makes it easy
> +(or even easier) to generate complete root filesystems. This is what
> +makes Buildroot stands out in the crowd (among other things, of course!).

Best regards,
Thomas