[Buildroot] Download tag as archive from gitlab

[Buildroot] Download tag as archive from gitlab

[Angelo Compagnucci]

Hello guys,

I'm hitting some bumps trying to download a tag from gitlab as an
archive, the url format is like this:

https://gitlab.com/gitlab-org/gitlab-ce/repository/archive.tar.bz2?ref=v6.9.2

I can succesfully download the file, but then it's named
"archive.tar.bz2?ref=v6.9.2" but it cannot be extracted cause it lacks
an extensions and buildroot cannot choose the right helper to
decompress it.

How such an url can be downloaded in buildroot?

Thanks!

-- 
Profile: http://it.linkedin.com/in/compagnucciangelo

[Buildroot] Download tag as archive from gitlab

[Yann E. MORIN]

Angelo, All,

On 2017-03-07 00:28 +0100, Angelo Compagnucci spake thusly:
> I'm hitting some bumps trying to download a tag from gitlab as an
> archive, the url format is like this:
> 
> https://gitlab.com/gitlab-org/gitlab-ce/repository/archive.tar.bz2?ref=v6.9.2
> 
> I can succesfully download the file, but then it's named
> "archive.tar.bz2?ref=v6.9.2" but it cannot be extracted cause it lacks
> an extensions and buildroot cannot choose the right helper to
> decompress it.
> 
> How such an url can be downloaded in buildroot?

The only solution is to download it with a git clone:

    FOO_VERSION = TAG_OR_SHA1
    FOO_SITE = https://gitlab.com/gitlab-org/gitlab-ce
    FOO_SITE_METHOD = git

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Download tag as archive from gitlab

[Arnout Vandecappelle]

On 07-03-17 08:11, Yann E. MORIN wrote:
> Angelo, All,
> 
> On 2017-03-07 00:28 +0100, Angelo Compagnucci spake thusly:
>> I'm hitting some bumps trying to download a tag from gitlab as an
>> archive, the url format is like this:
>>
>> https://gitlab.com/gitlab-org/gitlab-ce/repository/archive.tar.bz2?ref=v6.9.2
>>
>> I can succesfully download the file, but then it's named
>> "archive.tar.bz2?ref=v6.9.2" but it cannot be extracted cause it lacks
>> an extensions and buildroot cannot choose the right helper to
>> decompress it.
>>
>> How such an url can be downloaded in buildroot?
> 
> The only solution is to download it with a git clone:

 The only solution you can think of :-)

 You can use the following hack:

FOO_VERSION = v6.9.2
FOO_SITE = https://.../archive.tar.bz2?ref=$(FOO_VERSION)&filename=
FOO_SOURCE = foo-$(FOO_VERSION).tar.bz2

 This will lead to a wget of
https://.../archive.tar.bz2?ref=v6.9.2&filename=/foo-v6.9.2.tar.bz2
to the file foo-v6.9.2.tar.bz2.

 It works because in a GET request, any unknown variables are typically ignored.
Last time I tried gitlab did ignore them.

 Of course you can use anything you like for FOO_SOURCE, but it's convenient to
use the canonical name.

 Regards,
 Arnout


> 
>     FOO_VERSION = TAG_OR_SHA1
>     FOO_SITE = https://gitlab.com/gitlab-org/gitlab-ce
>     FOO_SITE_METHOD = git
> 
> Regards,
> Yann E. MORIN.
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] Download tag as archive from gitlab

[Angelo Compagnucci]

Dear Arnout, Yann,

2017-03-07 9:36 GMT+01:00 Arnout Vandecappelle <arnout@mind.be>:
>
>
> On 07-03-17 08:11, Yann E. MORIN wrote:
>> Angelo, All,
>>
>> On 2017-03-07 00:28 +0100, Angelo Compagnucci spake thusly:
>>> I'm hitting some bumps trying to download a tag from gitlab as an
>>> archive, the url format is like this:
>>>
>>> https://gitlab.com/gitlab-org/gitlab-ce/repository/archive.tar.bz2?ref=v6.9.2
>>>
>>> I can succesfully download the file, but then it's named
>>> "archive.tar.bz2?ref=v6.9.2" but it cannot be extracted cause it lacks
>>> an extensions and buildroot cannot choose the right helper to
>>> decompress it.
>>>
>>> How such an url can be downloaded in buildroot?
>>
>> The only solution is to download it with a git clone:
>
>  The only solution you can think of :-)
>
>  You can use the following hack:
>
> FOO_VERSION = v6.9.2
> FOO_SITE = https://.../archive.tar.bz2?ref=$(FOO_VERSION)&filename=
> FOO_SOURCE = foo-$(FOO_VERSION).tar.bz2
>
>  This will lead to a wget of
> https://.../archive.tar.bz2?ref=v6.9.2&filename=/foo-v6.9.2.tar.bz2
> to the file foo-v6.9.2.tar.bz2.
>
>  It works because in a GET request, any unknown variables are typically ignored.
> Last time I tried gitlab did ignore them.
>
>  Of course you can use anything you like for FOO_SOURCE, but it's convenient to
> use the canonical name.

I was arrived at the same solution just minutes ago! I think just
adding a gitlab helper like the github one with the hack included!

Thanks!

Sincerely, Angelo

>
>  Regards,
>  Arnout
>
>
>>
>>     FOO_VERSION = TAG_OR_SHA1
>>     FOO_SITE = https://gitlab.com/gitlab-org/gitlab-ce
>>     FOO_SITE_METHOD = git
>>
>> Regards,
>> Yann E. MORIN.
>>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF



-- 
Profile: http://it.linkedin.com/in/compagnucciangelo

[Buildroot] Download tag as archive from gitlab

[Yann E. MORIN]

Arnout, All,

On 2017-03-07 09:36 +0100, Arnout Vandecappelle spake thusly:
> On 07-03-17 08:11, Yann E. MORIN wrote:
> > Angelo, All,
> > 
> > On 2017-03-07 00:28 +0100, Angelo Compagnucci spake thusly:
> >> I'm hitting some bumps trying to download a tag from gitlab as an
> >> archive, the url format is like this:
> >>
> >> https://gitlab.com/gitlab-org/gitlab-ce/repository/archive.tar.bz2?ref=v6.9.2
> >>
> >> I can succesfully download the file, but then it's named
> >> "archive.tar.bz2?ref=v6.9.2" but it cannot be extracted cause it lacks
> >> an extensions and buildroot cannot choose the right helper to
> >> decompress it.
> >>
> >> How such an url can be downloaded in buildroot?
> > 
> > The only solution is to download it with a git clone:
> 
>  The only solution you can think of :-)
> 
>  You can use the following hack:
> 
> FOO_VERSION = v6.9.2
> FOO_SITE = https://.../archive.tar.bz2?ref=$(FOO_VERSION)&filename=
> FOO_SOURCE = foo-$(FOO_VERSION).tar.bz2

Yeah, I know that trick. I think you even mentioned it not so long ago
for that very same issue.

Although this is a neat trick, it still is a trick, it's not obvious,
it's not documented, it's ugly. ;-)

So I think we should refrain from accepting such tricks.

If upstream does not have a sane archive download solution, then I
prefer that we do a git clone instead; it's much obvious what is going
on. Yes, this usually means bigger/slower downloads, but then people
should go complain to the culprit (here, gitlab).

Or we add a gitlab helper. Meh... No. ;-)

Regards,
Yann E. MORIN.

>  This will lead to a wget of
> https://.../archive.tar.bz2?ref=v6.9.2&filename=/foo-v6.9.2.tar.bz2
> to the file foo-v6.9.2.tar.bz2.
> 
>  It works because in a GET request, any unknown variables are typically ignored.
> Last time I tried gitlab did ignore them.
> 
>  Of course you can use anything you like for FOO_SOURCE, but it's convenient to
> use the canonical name.
> 
>  Regards,
>  Arnout
> 
> 
> > 
> >     FOO_VERSION = TAG_OR_SHA1
> >     FOO_SITE = https://gitlab.com/gitlab-org/gitlab-ce
> >     FOO_SITE_METHOD = git
> > 
> > Regards,
> > Yann E. MORIN.
> > 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Download tag as archive from gitlab

[Angelo Compagnucci]

Yann, All

2017-03-07 9:45 GMT+01:00 Yann E. MORIN <yann.morin.1998@free.fr>:
> Arnout, All,
>
> On 2017-03-07 09:36 +0100, Arnout Vandecappelle spake thusly:
>> On 07-03-17 08:11, Yann E. MORIN wrote:
>> > Angelo, All,
>> >
>> > On 2017-03-07 00:28 +0100, Angelo Compagnucci spake thusly:
>> >> I'm hitting some bumps trying to download a tag from gitlab as an
>> >> archive, the url format is like this:
>> >>
>> >> https://gitlab.com/gitlab-org/gitlab-ce/repository/archive.tar.bz2?ref=v6.9.2
>> >>
>> >> I can succesfully download the file, but then it's named
>> >> "archive.tar.bz2?ref=v6.9.2" but it cannot be extracted cause it lacks
>> >> an extensions and buildroot cannot choose the right helper to
>> >> decompress it.
>> >>
>> >> How such an url can be downloaded in buildroot?
>> >
>> > The only solution is to download it with a git clone:
>>
>>  The only solution you can think of :-)
>>
>>  You can use the following hack:
>>
>> FOO_VERSION = v6.9.2
>> FOO_SITE = https://.../archive.tar.bz2?ref=$(FOO_VERSION)&filename=
>> FOO_SOURCE = foo-$(FOO_VERSION).tar.bz2
>
> Yeah, I know that trick. I think you even mentioned it not so long ago
> for that very same issue.
>
> Although this is a neat trick, it still is a trick, it's not obvious,
> it's not documented, it's ugly. ;-)
>
> So I think we should refrain from accepting such tricks.
>
> If upstream does not have a sane archive download solution, then I
> prefer that we do a git clone instead; it's much obvious what is going
> on. Yes, this usually means bigger/slower downloads, but then people
> should go complain to the culprit (here, gitlab).
>
> Or we add a gitlab helper. Meh... No. ;-)

It's not entirely true: gitlab returns

Content-Disposition=attachment;
filename="gitlab-ce-v6.9.2-e46b644a8857a53ed3f6c3f64b224bb74b06fd8e.tar.gz"

In the header for the GET request, so it behaves correctly. Indeed if
you download the file with a browser you will obtain a sensible name.
The filename is in the form <project>-<version>-<sha>.ext .

Unfortunately, the buildroot download helper DOWNLOAD_WGET is a bit
rigid: it overwrites the filename and requires the url to be in a
certain form.

Content-Disposition is heavily used in modern web services where the
filename on the cloud usually has a form of a pointer to an object and
the real filename it's stored somewhere and retrieved via an API.

wget can use for example --content-disposition to get the filename.
This also makes the progress bar behave correctly cause it inspects
the header for the file size.

Sincerely, Angelo

>
> Regards,
> Yann E. MORIN.
>
>>  This will lead to a wget of
>> https://.../archive.tar.bz2?ref=v6.9.2&filename=/foo-v6.9.2.tar.bz2
>> to the file foo-v6.9.2.tar.bz2.
>>
>>  It works because in a GET request, any unknown variables are typically ignored.
>> Last time I tried gitlab did ignore them.
>>
>>  Of course you can use anything you like for FOO_SOURCE, but it's convenient to
>> use the canonical name.
>>
>>  Regards,
>>  Arnout
>>
>>
>> >
>> >     FOO_VERSION = TAG_OR_SHA1
>> >     FOO_SITE = https://gitlab.com/gitlab-org/gitlab-ce
>> >     FOO_SITE_METHOD = git
>> >
>> > Regards,
>> > Yann E. MORIN.
>> >
>>
>> --
>> Arnout Vandecappelle                          arnout at mind be
>> Senior Embedded Software Architect            +32-16-286500
>> Essensium/Mind                                http://www.mind.be
>> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
>> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
>> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'



-- 
Profile: http://it.linkedin.com/in/compagnucciangelo