[Buildroot] Problem using Linux kernel archive from Gitorious

[Buildroot] Problem using Linux kernel archive from Gitorious

[Aras Vaichas]

Hello,

I am configuring Buildroot to use an OMAP-L138/DaVinci CPU. I'd like
to use the Linux kernel version from
http://gitorious.org/linux-davinci/linux-davinci/trees/v3.7-davinci1
which is a specific tag in the git repository.

In Buildroot I set:
BR2_LINUX_KERNEL_CUSTOM_TARBALL=y
BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION=http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1

The file that is returned has the name "v3.7-davinci1", and it is a
"gzip compressed data, from Unix" according to file.

Gitorious calls it a "tarball" but it's obviously not a tarball.

Unfortunately Buildroot fails because, understandably, the file isn't
a tar archive.

>>> linux custom Extracting
/opt/br/dl/v3.7-davinci1 | tar --strip-components=1 -C
/opt/br/output/build/linux-custom  -xf -
/bin/bash: /opt/br/dl/v3.7-davinci1: Permission denied
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Should I add an option to Buildroot to handle gzipped files? Or is
there a better way?

regards,

Aras Vaichas

[Buildroot] Problem using Linux kernel archive from Gitorious

[Thomas Petazzoni]

Dear Aras Vaichas,

On Thu, 17 Jan 2013 14:47:47 +0000, Aras Vaichas wrote:

> I am configuring Buildroot to use an OMAP-L138/DaVinci CPU. I'd like
> to use the Linux kernel version from
> http://gitorious.org/linux-davinci/linux-davinci/trees/v3.7-davinci1
> which is a specific tag in the git repository.
> 
> In Buildroot I set:
> BR2_LINUX_KERNEL_CUSTOM_TARBALL=y
> BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION=http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1
> 
> The file that is returned has the name "v3.7-davinci1", and it is a
> "gzip compressed data, from Unix" according to file.
> 
> Gitorious calls it a "tarball" but it's obviously not a tarball.

It most likely is. When a tarball is gzipped compressed, "file" only
reports it as a gzipped compressed file, it doesn't know there is a
tarball inside.

> Unfortunately Buildroot fails because, understandably, the file isn't
> a tar archive.
> 
> >>> linux custom Extracting
> /opt/br/dl/v3.7-davinci1 | tar --strip-components=1 -C
> /opt/br/output/build/linux-custom  -xf -
> /bin/bash: /opt/br/dl/v3.7-davinci1: Permission denied
> tar: This does not look like a tar archive
> tar: Exiting with failure status due to previous errors
> 
> Should I add an option to Buildroot to handle gzipped files? Or is
> there a better way?

The problem here is that since the filename doesn't end with .tar.gz,
Buildroot doesn't know which extraction method to apply, and therefore
doesn't apply any, which leads Buildroot to try to execute your tarball.

Hum, I am not sure how to handle that. Should we try to extract simply
with tar something that does not have an extension?

Github sends a correct filename in the Content-Disposition header of
the HTTP reply, and wget --content-disposition is supposed to name the
output file according to this filename given in the HTTP reply, but it
doesn't work for me... and anyway, Buildroot would have no way to know
what is the new name of the file.

I don't have a lot of ideas for now, unfortunately :-(

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Problem using Linux kernel archive from Gitorious

[Stephan Hoffmann]

Am 17.01.2013 16:01, schrieb Thomas Petazzoni:
> Dear Aras Vaichas,
>
> On Thu, 17 Jan 2013 14:47:47 +0000, Aras Vaichas wrote:
>
>> I am configuring Buildroot to use an OMAP-L138/DaVinci CPU. I'd like
>> to use the Linux kernel version from
>> http://gitorious.org/linux-davinci/linux-davinci/trees/v3.7-davinci1
>> which is a specific tag in the git repository.
>>
>> In Buildroot I set:
>> BR2_LINUX_KERNEL_CUSTOM_TARBALL=y
>> BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION=http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1
>>
>> The file that is returned has the name "v3.7-davinci1", and it is a
>> "gzip compressed data, from Unix" according to file.
>>
>> Gitorious calls it a "tarball" but it's obviously not a tarball.
> It most likely is. When a tarball is gzipped compressed, "file" only
> reports it as a gzipped compressed file, it doesn't know there is a
> tarball inside.
>
>> Unfortunately Buildroot fails because, understandably, the file isn't
>> a tar archive.
>>
>>>>> linux custom Extracting
>> /opt/br/dl/v3.7-davinci1 | tar --strip-components=1 -C
>> /opt/br/output/build/linux-custom  -xf -
>> /bin/bash: /opt/br/dl/v3.7-davinci1: Permission denied
>> tar: This does not look like a tar archive
>> tar: Exiting with failure status due to previous errors
>>
>> Should I add an option to Buildroot to handle gzipped files? Or is
>> there a better way?
> The problem here is that since the filename doesn't end with .tar.gz,
> Buildroot doesn't know which extraction method to apply, and therefore
> doesn't apply any, which leads Buildroot to try to execute your tarball.
>
> Hum, I am not sure how to handle that. Should we try to extract simply
> with tar something that does not have an extension?
>
> Github sends a correct filename in the Content-Disposition header of
> the HTTP reply, and wget --content-disposition is supposed to name the
> output file according to this filename given in the HTTP reply, but it
> doesn't work for me... and anyway, Buildroot would have no way to know
> what is the new name of the file.
>
> I don't have a lot of ideas for now, unfortunately :-(
Hi Thomas,

what about using file instead of decoding the extension? If file says
"gzipped", then gunzip it and use file again to find out that it's tar?

This could be done until something useful comes out or, if not, a
descriptive error message could be generated.

Just my 2 cents

Stephan
>
> Thomas


-- 
reLinux     -    Stephan Hoffmann
Am Schmidtgrund 124    50765 K?ln
Tel. +49.221.95595-19    Fax: -64
www.reLinux.de     sho at reLinux.de

[Buildroot] Problem using Linux kernel archive from Gitorious

[Stefan Fröberg]

17.1.2013 17:01, Thomas Petazzoni kirjoitti:
> The problem here is that since the filename doesn't end with .tar.gz,
> Buildroot doesn't know which extraction method to apply, and therefore
> doesn't apply any, which leads Buildroot to try to execute your tarball.
>
> Hum, I am not sure how to handle that. Should we try to extract simply
> with tar something that does not have an extension?

if you use tar with -a switch it is smart enough to extract it
correctly. Even if wrong extension or no extension at all.
For example:

mv bison-2.5.tar.gz bison-2.5.tar.xyz
tar -xaSvf bison-2.5.tar.xyz

mv bison-2.5.tar.gz bison-2.5
tar -xaSvf bison-2.5

Regards
Stefan

[Buildroot] Problem using Linux kernel archive from Gitorious

[Arnout Vandecappelle]

On 17/01/13 15:47, Aras Vaichas wrote:
> Hello,
> 
> I am configuring Buildroot to use an OMAP-L138/DaVinci CPU. I'd like
> to use the Linux kernel version from
> http://gitorious.org/linux-davinci/linux-davinci/trees/v3.7-davinci1
> which is a specific tag in the git repository.
> 
> In Buildroot I set:
> BR2_LINUX_KERNEL_CUSTOM_TARBALL=y
> BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION=http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1

 It's a bit of a hack, but use this URL:

http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1?dummyname=/linux-v3.7-davinci.tar.gz

 The CGI script that is behind that URL will ignore any unknown variables
behind the ? and the download method of buildroot will consider the thing
behind the last / as the filename. So this URL will be split into

LINUX_SITE = http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1?dummyname=
LINUX_SOURCE = linux-v3.7-davinci.tar.gz

 So this gives you a nicely named file in the download directory.


 Note that the gitorious archive URLs have an important disadvantage:
if the archive is not already in the cache, than the site will respond
with a '202 Accepted'.  I think wget treats this as a success, so
buildroot will try to untar that placeholder file.

 Regards,
 Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Problem using Linux kernel archive from Gitorious

[Aras Vaichas]

On 19 January 2013 13:20, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 17/01/13 15:47, Aras Vaichas wrote:
>> Hello,
>>
>> I am configuring Buildroot to use an OMAP-L138/DaVinci CPU. I'd like
>> to use the Linux kernel version from
>> http://gitorious.org/linux-davinci/linux-davinci/trees/v3.7-davinci1
>> which is a specific tag in the git repository.
>>
>> In Buildroot I set:
>> BR2_LINUX_KERNEL_CUSTOM_TARBALL=y
>> BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION=http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1
>
>  It's a bit of a hack, but use this URL:
>
> http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1?dummyname=/linux-v3.7-davinci.tar.gz
>  The CGI script that is behind that URL will ignore any unknown variables
> behind the ? and the download method of buildroot will consider the thing
> behind the last / as the filename. So this URL will be split into
>
> LINUX_SITE = http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1?dummyname=
> LINUX_SOURCE = linux-v3.7-davinci.tar.gz
>
>  So this gives you a nicely named file in the download directory.
>
>
>  Note that the gitorious archive URLs have an important disadvantage:
> if the archive is not already in the cache, than the site will respond
> with a '202 Accepted'.  I think wget treats this as a success, so
> buildroot will try to untar that placeholder file.

Thank you. Yes, I did notice the delay the first time I tried to
download the tarball. I figured this would be a problem.

I'm currently configuring Buildroot for a company, so I want (need) to
provide links that will always work. I think what I will do is create
a davinci patch against a particular version of Linux and tell the
company to store the patch in their own source repository. That
guarantees that they can always get the file and re-recreate the same
same kernel - assuming kernel.org always has a copy of the original
kernel source for a particular version. :)

Aras

[Buildroot] Problem using Linux kernel archive from Gitorious

[Arnout Vandecappelle]

On 01/21/13 12:30, Aras Vaichas wrote:
> On 19 January 2013 13:20, Arnout Vandecappelle<arnout@mind.be>  wrote:
>> On 17/01/13 15:47, Aras Vaichas wrote:
>>> Hello,
>>>
>>> I am configuring Buildroot to use an OMAP-L138/DaVinci CPU. I'd like
>>> to use the Linux kernel version from
>>> http://gitorious.org/linux-davinci/linux-davinci/trees/v3.7-davinci1
>>> which is a specific tag in the git repository.
>>>
>>> In Buildroot I set:
>>> BR2_LINUX_KERNEL_CUSTOM_TARBALL=y
>>> BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION=http://gitorious.org/linux-davinci/linux-davinci/archive-tarball/v3.7-davinci1
[snip]
>>   Note that the gitorious archive URLs have an important disadvantage:
>> if the archive is not already in the cache, than the site will respond
>> with a '202 Accepted'.  I think wget treats this as a success, so
>> buildroot will try to untar that placeholder file.
>
> Thank you. Yes, I did notice the delay the first time I tried to
> download the tarball. I figured this would be a problem.
>
> I'm currently configuring Buildroot for a company, so I want (need) to
> provide links that will always work.


  You can of course use the git download method instead of the tarball. 
Since you're downloading a tag, a shallow clone should work, so the 
overhead compared to a tarball is limited.

> I think what I will do is create
> a davinci patch against a particular version of Linux and tell the
> company to store the patch in their own source repository. That
> guarantees that they can always get the file and re-recreate the same
> same kernel - assuming kernel.org always has a copy of the original
> kernel source for a particular version. :)

  To make things reproducible, it is anyway advisable to set up a 
BR2_PRIMARY_SITE where the tarballs are kept. It's a bit of a pain to 
maintain such a primary site, but it does solve a lot of potential issues 
(e.g. the kernel.org website going down :-).

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Problem using Linux kernel archive from Gitorious

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >> I think what I will do is create
 >> a davinci patch against a particular version of Linux and tell the
 >> company to store the patch in their own source repository. That
 >> guarantees that they can always get the file and re-recreate the same
 >> same kernel - assuming kernel.org always has a copy of the original
 >> kernel source for a particular version. :)

 Arnout>  To make things reproducible, it is anyway advisable to set up a
 Arnout> BR2_PRIMARY_SITE where the tarballs are kept. It's a bit of a pain to
 Arnout> maintain such a primary site, but it does solve a lot of potential
 Arnout> issues (e.g. the kernel.org website going down :-).

Pain to maintain? You can just add a cronjob doing:

make <blah>_defconfig
make BUILDROOT_DL_DIR=/path/to/primary/mirror source

That's basically how I maintain sources.buildroot.net (I use
allyesconfig instead)

-- 
Bye, Peter Korsgaard

[Buildroot] Problem using Linux kernel archive from Gitorious

[Aras Vaichas]

On 24 January 2013 06:32, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 01/21/13 12:30, Aras Vaichas wrote:
>>
>> On 19 January 2013 13:20, Arnout Vandecappelle<arnout@mind.be>  wrote:
>>>
>>> On 17/01/13 15:47, Aras Vaichas wrote:
>>>>
>>>> Hello,
>>>>
>>>> I am configuring Buildroot to use an OMAP-L138/DaVinci CPU. I'd like
>>>> to use the Linux kernel version from
>>>> http://gitorious.org/linux-davinci/linux-davinci/trees/v3.7-davinci1
>>>> which is a specific tag in the git repository.

... SNIP...

>  You can of course use the git download method instead of the tarball. Since
> you're downloading a tag, a shallow clone should work, so the overhead
> compared to a tarball is limited.

That is actually where I started, but it kept doing a full clone, so I
revisited what I did.

The tag, according to the davinci gitorious website is
"v3.7-davinci1", so I tried that first but Buildroot returned:

>>> linux v3.7-davinci1 Downloading
/bin/bash: line 0: test: refs/tags/v3.7-davinci1: unary operator expected
Doing full clone
Cloning into bare repository 'linux-v3.7-davinci1'...

I didn't notice the error before so I investigated package/pkg-download.mk.

$ git ls-remote git://gitorious.org/linux-davinci/linux-davinci.git |
cut -f 2- | grep v3.7-davinci1

refs/tags/v3.7-davinci1
refs/tags/v3.7-davinci1^{}

It returns two separate strings, and this causes the test function to
fail and drop through to a full clone.

Here is a fix:

--- a/package/pkg-download.mk	2013-01-25 11:02:03.491175888 +0000
+++ b/package/pkg-download.mk	2013-01-25 11:35:50.531175646 +0000
@@ -82,7 +82,7 @@
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
 	(pushd $(DL_DIR) > /dev/null && \
-	 ((test `git ls-remote  $($(PKG)_SITE) | cut -f 2- | grep
$($(PKG)_DL_VERSION)` && \
+	 ((test "`git ls-remote  $($(PKG)_SITE) | cut -f 2- | grep
$($(PKG)_DL_VERSION)`" && \
 	   echo "Doing shallow clone" && \
 	   $(GIT) clone --depth 1 -b $($(PKG)_DL_VERSION) --bare
$($(PKG)_SITE) $($(PKG)_BASE_NAME)) || \
 	  (echo "Doing full clone" && \

Shall I post a patch? (I use gmail, so it tends to word wrap the patch)

regards,

Aras Vaichas