[Buildroot] Trouble with offline build

[Buildroot] Trouble with offline build

[A]

 I'm a long-time fan, but first-time user.  I'm using buildroot-2017.11.2
for an ARM target system.

My build system doesn't have internet access, so I need the offline build
capabilities, which I am having trouble with.  The build system also is
pretty old, and tends to lack modern packages.

I do a "make source" on a cygwin system that has internet and, and copy the
whole thing over to the build system.

Then, on the build system, I do a "make", and I get undesired internet
access attempts and an ultimate failure, which I show below.

I saw an archive post from 2007 here where the OP was told there are buggy
packages that need to be patched in order for this to happen.  They are
said to be buggy because "they lack a foo-source: target.  These need to be
fixed."  (there was an attachment to the forum post, but the attachment has
since been scrubbed).

Can someone please weigh in on whether this might be what is happening with
lzip (below), and how I might go about fixing this?

I'm new enough that I'm not even sure why this package is being
downloaded.  I must have it in one of the config files, but it's not an
obvious.

A brute-force find/grep shows that there might be something about the
BR2_LZCAT variable.  I've no idea what this means.  Is there an
undocumented host dependency on lzip?  I see it's not installed on my
system.


user at host:~/src/buildroot-2017.11.2$ make source
>>> host-lzip 1.19 Downloading
--2018-02-27 22:51:58--  http://download.savannah.gnu.
org/releases/lzip/lzip-1.19.tar.gz
Resolving download.savannah.gnu.org... 208.118.235.200
Connecting to download.savannah.gnu.org|208.118.235.200|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2018-02-27 22:52:03 ERROR 503: Service Unavailable.

--2018-02-27 22:52:04--  http://sources.buildroot.net/lzip-1.19.tar.gz
Resolving sources.buildroot.net... 104.27.166.48, 104.27.167.48,
2400:cb00:2048:1::681b:a730, ...
Connecting to sources.buildroot.net|104.27.166.48|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2018-02-27 22:52:04 ERROR 503: Service Unavailable.

make[1]: *** [/home/user/src/buildroot-2017.11.2/output/build/host-lzip-1.19/.stamp_downloaded]
Error 1
make: *** [_all] Error 2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20180227/4c4fd0eb/attachment.html>

[Buildroot] Trouble with offline build

[Baruch Siach]

Hi Aaron,

On Tue, Feb 27, 2018 at 06:34:53PM -0500, A wrote:
>  I'm a long-time fan, but first-time user.  I'm using buildroot-2017.11.2
> for an ARM target system.
> 
> My build system doesn't have internet access, so I need the offline build
> capabilities, which I am having trouble with.  The build system also is
> pretty old, and tends to lack modern packages.
> 
> I do a "make source" on a cygwin system that has internet and, and copy the
> whole thing over to the build system.
> 
> Then, on the build system, I do a "make", and I get undesired internet
> access attempts and an ultimate failure, which I show below.
> 
> I saw an archive post from 2007 here where the OP was told there are buggy
> packages that need to be patched in order for this to happen.  They are
> said to be buggy because "they lack a foo-source: target.  These need to be
> fixed."  (there was an attachment to the forum post, but the attachment has
> since been scrubbed).
> 
> Can someone please weigh in on whether this might be what is happening with
> lzip (below), and how I might go about fixing this?

Do you have lzip installed on your Internet connected machine?

> I'm new enough that I'm not even sure why this package is being
> downloaded.  I must have it in one of the config files, but it's not an
> obvious.

As you can see from the output below, this is a host package (host-lzip). lzip 
is needed to extract some packages. Buildroot builds this host package only 
when the host has no lzip installed. The same goes for a number of other 
dependencies. See support/dependencies/. So you can either install lzip on 
your build machine, or force the download of lzip with 'make lzip-source'

Hope this helps,
baruch

> A brute-force find/grep shows that there might be something about the
> BR2_LZCAT variable.  I've no idea what this means.  Is there an
> undocumented host dependency on lzip?  I see it's not installed on my
> system.
> 
> user at host:~/src/buildroot-2017.11.2$ make source
> >>> host-lzip 1.19 Downloading
> --2018-02-27 22:51:58--  http://download.savannah.gnu.
> org/releases/lzip/lzip-1.19.tar.gz
> Resolving download.savannah.gnu.org... 208.118.235.200
> Connecting to download.savannah.gnu.org|208.118.235.200|:80... connected.
> HTTP request sent, awaiting response... 503 Service Unavailable
> 2018-02-27 22:52:03 ERROR 503: Service Unavailable.
> 
> --2018-02-27 22:52:04--  http://sources.buildroot.net/lzip-1.19.tar.gz
> Resolving sources.buildroot.net... 104.27.166.48, 104.27.167.48,
> 2400:cb00:2048:1::681b:a730, ...
> Connecting to sources.buildroot.net|104.27.166.48|:80... connected.
> HTTP request sent, awaiting response... 503 Service Unavailable
> 2018-02-27 22:52:04 ERROR 503: Service Unavailable.
> 
> make[1]: *** [/home/user/src/buildroot-2017.11.2/output/build/host-lzip-1.19/.stamp_downloaded]
> Error 1
> make: *** [_all] Error 2

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -

[Buildroot] Trouble with offline build

[Thomas Petazzoni]

Hello Baruch, Hello Aaron,

On Wed, 28 Feb 2018 06:53:10 +0200, Baruch Siach wrote:

> > I'm new enough that I'm not even sure why this package is being
> > downloaded.  I must have it in one of the config files, but it's not an
> > obvious.  
> 
> As you can see from the output below, this is a host package (host-lzip). lzip 
> is needed to extract some packages. Buildroot builds this host package only 
> when the host has no lzip installed. The same goes for a number of other 
> dependencies. See support/dependencies/. So you can either install lzip on 
> your build machine, or force the download of lzip with 'make lzip-source'

I obviously entirely agree with Baruch analysis here. However, I am a
bit unhappy about the fact that Buildroot doesn't handle this in a
better way, even though I'm not sure what the better way would be.

I believe Aaron's expectation that given a Buildroot configuration, a
"make source" on machine A should download everything that is needed to
do a build of the same configuration on machine B is a reasonable
expectation.

However, there are several host machine details that prevent this from
happening today:

 - host architecture: we may download a x86 or x86-64 pre-built binary.
   External toolchains come to mind, but also things like pre-built
   rust compiler, or other things like that.

 - Things that we skip building if a suitable version if found on the
   host: host-cmake, host-lzip, host-xzcat, host-tar.

To be honest, I don't really see how to solve that problem. the
host-{cmake,lzip,xzcat,tar} issue could probably be solved by
downloading all of them even if some of them won't be needed (we may
have a "make all-source" that differs from "make source" to avoid "make
source" from downloading useless stuff). However, for the host
architecture difference, I'm not sure how to proceed, and I'm not sure
there is a reasonable solution.

Bottom line: should we document/make it clear that doing a "make
source" on machine A and then do a build on machine B is not
guaranteeing a fully offline build ? It's a bit of a pity, because I
know there are some environment where only a given machine has Internet
access, and the developer machines don't, but I don't see a good way of
solving this in Buildroot.

Any comments or suggestions ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

[Buildroot] Trouble with offline build

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:

Hi,

 > To be honest, I don't really see how to solve that problem. the
 > host-{cmake,lzip,xzcat,tar} issue could probably be solved by
 > downloading all of them even if some of them won't be needed (we may
 > have a "make all-source" that differs from "make source" to avoid "make
 > source" from downloading useless stuff). However, for the host
 > architecture difference, I'm not sure how to proceed, and I'm not sure
 > there is a reasonable solution.

 > Bottom line: should we document/make it clear that doing a "make
 > source" on machine A and then do a build on machine B is not
 > guaranteeing a fully offline build ? It's a bit of a pity, because I
 > know there are some environment where only a given machine has Internet
 > access, and the developer machines don't, but I don't see a good way of
 > solving this in Buildroot.

Me neither. Historically, make source was meant to ensure that
everything is downloaded now so you could go offline and continue
building _ON_THE_SAME_MACHINE_. This potentially does indeed break down
if you try to reuse the download directory on another machine, but I
don't quite see how we can solve this for all situations.

-- 
Bye, Peter Korsgaard

[Buildroot] Trouble with offline build

[Yann E. MORIN]

Peter, Thomas, All,

On 2018-02-28 09:29 +0100, Peter Korsgaard spake thusly:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes:
>  > To be honest, I don't really see how to solve that problem. the
>  > host-{cmake,lzip,xzcat,tar} issue could probably be solved by
>  > downloading all of them even if some of them won't be needed (we may
>  > have a "make all-source" that differs from "make source" to avoid "make
>  > source" from downloading useless stuff). However, for the host
>  > architecture difference, I'm not sure how to proceed, and I'm not sure
>  > there is a reasonable solution.

There is for example the case where we could download a different
archive depending on the host machine. For example, we can download a
different Linaro toolchain,depending on whether we're runnign on a 32-
or a 64-bit x86 host.

>  > Bottom line: should we document/make it clear that doing a "make
>  > source" on machine A and then do a build on machine B is not
>  > guaranteeing a fully offline build ?

Yes, because there is no way that machine A knows about what machine B
will be...

Heck, for this toolchain stuff, that would also break if machine A is
x86 but machine B is something else (e.g. PPC).

In fact, we're not even able to guarantee that a .config file from
machine A is buildable on machine B...

> > It's a bit of a pity, because I
>  > know there are some environment where only a given machine has Internet
>  > access, and the developer machines don't, but I don't see a good way of
>  > solving this in Buildroot.
> Me neither. Historically, make source was meant to ensure that
> everything is downloaded now so you could go offline and continue
> building _ON_THE_SAME_MACHINE_. This potentially does indeed break down
> if you try to reuse the download directory on another machine, but I
> don't quite see how we can solve this for all situations.

I don't have a good idea either. Yes, such are environment are more
widesoread than usually thought. For example, a CI infra may not have
internet access at all (real life experience!), so an intermediate
server is used as a filer...

Anyway, I'm afraid there is no technical solution, except maybe we could
have something like (names to refine later) running on the build machine:

    make prepare-scripted-source

then move the tree to the internet-facing machine, and run:

    make scripted-source

and finally move the tree back to the build machine to resume
building...

Short of doing something along those lines, there is not muh we can do,
I'm afraid... :-/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Trouble with offline build

[Bernd Kuhls]

Hi,

Am Tue, 27 Feb 2018 18:34:53 -0500 schrieb A:

> I do a "make source" on a cygwin system that has internet and, and copy 
the
> whole thing over to the build system.

does your cygwin system include a "lzip" binary?

> Then, on the build system, I do a "make", and I get undesired internet
> access attempts and an ultimate failure, which I show below.

[...]
> user at host:~/src/buildroot-2017.11.2$ make source
>>>> host-lzip 1.19 Downloading

Buildroot checks the host system for needed dependencies and, if they are 
missing, downloads and builds the missing package(s), for details check 
the scripts in support/dependencies/. Apparently cygwin includes lzip 
whereas your build system lacks it.

The solution is to call "make host-lzip-source" on your cygwin system and 
then copy lzip-1.19.tar.gz to your build system.

Regards, Bernd