[Buildroot] EXTRA_DOWNLOADS broken when SITE_METHOD = git

[Buildroot] EXTRA_DOWNLOADS broken when SITE_METHOD = git

[Luca Ceresoli]

Hi,

I noticed a problem with the EXTRA_DOWNLOADS for packages whose "main"
download happens using git (most likely applies to other SCMs too).

With the following defconfig:

BR2_TARGET_UBOOT=y
BR2_TARGET_UBOOT_BOARDNAME="foo"
BR2_TARGET_UBOOT_CUSTOM_GIT=y
BR2_TARGET_UBOOT_CUSTOM_REPO_URL="git://git.denx.de/u-boot.git"
BR2_TARGET_UBOOT_CUSTOM_REPO_VERSION="v2018.03"

and the following line added to boot/uboot/uboot.mk:

UBOOT_EXTRA_DOWNLOADS += https://buildroot.org/images/logo.png

With Buildroot 2018.02, 'make uboot-source' downloads only the main git
sources. The URL mentioned in UBOOT_EXTRA_DOWNLOADS is ignored.

With Buildroot master (2018.02-934-gf5b14df1104c), the same command
tries to download the UBOOT_EXTRA_DOWNLOADS using git and of course it
fails:

Reinitialized existing Git repository in <...>/dl/uboot/git/.git/
Doing a shallow fetch
fatal: repository 'https://buildroot.org/images/' not found
Shallow fetch failed, falling back to fetching all refs
Fetching all references
fatal: repository 'https://buildroot.org/images/' not found

I think both behaviors are wrong: EXTRA_DOWNLOADS should always use wget
or cp, just as the manual says, and common sense suggests.

Regards,
-- 
Luca

[Buildroot] EXTRA_DOWNLOADS broken when SITE_METHOD = git

[Yann E. MORIN]

Luca, All,

On 2018-04-25 10:56 +0200, Luca Ceresoli spake thusly:
> I noticed a problem with the EXTRA_DOWNLOADS for packages whose "main"
> download happens using git (most likely applies to other SCMs too).

Thanks for this report. I can indeed confirm that extra downloads are
broken on 2018.02.x and master, when the main download is from an SCM.

As I said on IRC, I think they have always been broken, but we silenly
failed so far. So, there is no "regresskion" per-se. Just a bug.

As for master, we no longer silently fail, but we do now error out.
This could be considered an improvement, no? ;-)

I am working on a fix, and I already have something that is workable
now. I jsut need a few tests, and I'll submit it later tonight (or by
the weeke-end at least).

Regards,
Yann E. MORIN.

> With the following defconfig:
> 
> BR2_TARGET_UBOOT=y
> BR2_TARGET_UBOOT_BOARDNAME="foo"
> BR2_TARGET_UBOOT_CUSTOM_GIT=y
> BR2_TARGET_UBOOT_CUSTOM_REPO_URL="git://git.denx.de/u-boot.git"
> BR2_TARGET_UBOOT_CUSTOM_REPO_VERSION="v2018.03"
> 
> and the following line added to boot/uboot/uboot.mk:
> 
> UBOOT_EXTRA_DOWNLOADS += https://buildroot.org/images/logo.png
> 
> With Buildroot 2018.02, 'make uboot-source' downloads only the main git
> sources. The URL mentioned in UBOOT_EXTRA_DOWNLOADS is ignored.
> 
> With Buildroot master (2018.02-934-gf5b14df1104c), the same command
> tries to download the UBOOT_EXTRA_DOWNLOADS using git and of course it
> fails:
> 
> Reinitialized existing Git repository in <...>/dl/uboot/git/.git/
> Doing a shallow fetch
> fatal: repository 'https://buildroot.org/images/' not found
> Shallow fetch failed, falling back to fetching all refs
> Fetching all references
> fatal: repository 'https://buildroot.org/images/' not found
> 
> I think both behaviors are wrong: EXTRA_DOWNLOADS should always use wget
> or cp, just as the manual says, and common sense suggests.
> 
> Regards,
> -- 
> Luca

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] EXTRA_DOWNLOADS broken when SITE_METHOD = git

[Yann E. MORIN]

Luca, All,

On 2018-04-25 19:22 +0200, Yann E. MORIN spake thusly:
> On 2018-04-25 10:56 +0200, Luca Ceresoli spake thusly:
> > I noticed a problem with the EXTRA_DOWNLOADS for packages whose "main"
> > download happens using git (most likely applies to other SCMs too).
> 
> Thanks for this report. I can indeed confirm that extra downloads are
> broken on 2018.02.x and master, when the main download is from an SCM.
> 
> As I said on IRC, I think they have always been broken, but we silenly
> failed so far. So, there is no "regresskion" per-se. Just a bug.
> 
> As for master, we no longer silently fail, but we do now error out.
> This could be considered an improvement, no? ;-)
> 
> I am working on a fix, and I already have something that is workable
> now. I jsut need a few tests, and I'll submit it later tonight (or by
> the weeke-end at least).

Wihtout much furhter ado, here's the series in a preliminary state:
    https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/dl

The commit logs are not entirely written (oif at all) yet, but at least,
it would nice if you could have a look and test if that fixes your
use-case.

This is for master, obviously. I'll see if I can do something for
2018.02.x later...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] EXTRA_DOWNLOADS broken when SITE_METHOD = git

[Thomas Petazzoni]

Hello,

On Wed, 25 Apr 2018 10:56:05 +0200, Luca Ceresoli wrote:

> I noticed a problem with the EXTRA_DOWNLOADS for packages whose "main"
> download happens using git (most likely applies to other SCMs too).

Thanks for the report.

I think this illustrates (once again) that we desperately need some
tests for the download infrastructure. I'm adding Ricardo in Cc, who
already started working on such topic.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] EXTRA_DOWNLOADS broken when SITE_METHOD = git

[Ricardo Martincoski]

Hello,

On Wed, Apr 25, 2018 at 04:27 PM, Thomas Petazzoni wrote:

> I think this illustrates (once again) that we desperately need some
> tests for the download infrastructure. I'm adding Ricardo in Cc, who
> already started working on such topic.

I am currently refreshing my series with test cases for the git download infra.

I think I can reuse the fixtures added by the series (static repo, br2-external)
in order to add a test case for EXTRA_DOWNLOADS with SITE_METHOD = git.

A simple solution probably will add one or few files to
http://autobuild.buildroot.net/artefacts/
to be used by the test case.

Another way is to add a http server to the docker image to serve files added to
support/testing to be used by the test case.

Yet another solution would be to build and run an image with a http server. The
files to be served would be added to support/testing and copied to the image.

I will try to include a RFC for the first solution at the end of the series, if
no one speaks against it (or in favor of other solution).

Regards,
Ricardo

[Buildroot] EXTRA_DOWNLOADS broken when SITE_METHOD = git

[Thomas Petazzoni]

Hello,

On Wed, 25 Apr 2018 18:01:10 -0300, Ricardo Martincoski wrote:

> I am currently refreshing my series with test cases for the git download infra.

Excellent! We will have to extend it to cover other download methods.
For example, the CVS backend got broken as a result of the recent
download infra changes.

> I think I can reuse the fixtures added by the series (static repo, br2-external)
> in order to add a test case for EXTRA_DOWNLOADS with SITE_METHOD = git.
> 
> A simple solution probably will add one or few files to
> http://autobuild.buildroot.net/artefacts/
> to be used by the test case.

That can easily be done. I have no idea how scalable and future proof
this "artefacts" location is going to be, but we can start by using
that, and move to a better solution in the future if needed.

> Another way is to add a http server to the docker image to serve files added to
> support/testing to be used by the test case.

This also works. Seems a bit more complicated, no ?

> Yet another solution would be to build and run an image with a http server. The
> files to be served would be added to support/testing and copied to the image.

I did not understand this solution.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] EXTRA_DOWNLOADS broken when SITE_METHOD = git

[Ricardo Martincoski]

Hello,

On Wed, Apr 25, 2018 at 06:06 PM, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> On Wed, 25 Apr 2018 18:01:10 -0300, Ricardo Martincoski wrote:
> 
>> I am currently refreshing my series with test cases for the git download infra.
> 
> Excellent! We will have to extend it to cover other download methods.
> For example, the CVS backend got broken as a result of the recent
> download infra changes.

ACK

>> A simple solution probably will add one or few files to
>> http://autobuild.buildroot.net/artefacts/
>> to be used by the test case.
> 
> That can easily be done. I have no idea how scalable and future proof
> this "artefacts" location is going to be, but we can start by using
> that, and move to a better solution in the future if needed.

Indeed it won't scale well if we need different files for different test cases.
I will try the docker image first. If it shows up too complex, then I fallback
to this one.

>> Another way is to add a http server to the docker image to serve files added to
>> support/testing to be used by the test case.
> 
> This also works. Seems a bit more complicated, no ?

A bit more complicated, yes. Maybe not too much.
I will play with the docker image to check.

The only downside is that we "lose" the ability to run few test cases in a local
computer without the docker image.
But the test cases are meant to be run in the GitLab CI anyway, and one can
always download the docker image and run the test cases inside it. It also
generates more reproducible results. So in the end we don't lose much IMO.

>> Yet another solution would be to build and run an image with a http server. The
>> files to be served would be added to support/testing and copied to the image.
> 
> I did not understand this solution.

We could have a defconfig on the test case with a http server enabled and adding
files from a subfolder of support/testing to the image. The test infra then
builds an image and runs it on qemu. A port forwarding must be set up on the
command that calls qemu in order to the docker image to have access to the http
server inside the qemu.
Then the test case would request a second build that only downloads things: the
main download from the git server and the extra download from the http server
inside qemu.
Solution 2 (server on the docker image) seems easier and more future-proof than
this one, as we could end up with a failure unrelated to the download infra
caused by a bug on the http server when its package version is bumped.


Regards,
Ricardo