[Buildroot] [Bug 6302] New: Versions of packages retrieved from github.com are wrong

* [Buildroot] [Bug 6302] New: Versions of packages retrieved from github.com are wrong
@ 2013-06-06  1:40 bugzilla at busybox.net
  2013-06-06 16:47 ` [Buildroot] [PATCH 01/14] jsmin: fix github tarball to use commit id Spenser Gilliland
                   ` (6 more replies)
  0 siblings, 7 replies; 29+ messages in thread
From: bugzilla at busybox.net @ 2013-06-06  1:40 UTC (permalink / raw)
  To: buildroot

https://bugs.busybox.net/show_bug.cgi?id=6302

           Summary: Versions of packages retrieved from github.com are
                    wrong
           Product: buildroot
           Version: unspecified
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Outdated package
        AssignedTo: unassigned at buildroot.uclibc.org
        ReportedBy: uclibc-bugzilla at loowis.durge.org
                CC: buildroot at uclibc.org
   Estimated Hours: 0.0

I'm no buildroot or github expert (so apologies if I get some of the
terminology wrong), but I think I've found a bit of a nasty problem with the
way buildroot downloads specific versions of packages from github.com. At first
I thought it was because the .mk files for the specific rpi-* packages I was
looking at were set up wrong, but I've been investigating the problem for much
of this afternoon and I think it actually affects the .mk files for many
packages :-(

According to
http://www.buildroot.org/downloads/manual/manual.html#github-download-url
packages in github with a specific tag can be downloaded with:
FOO_VERSION = v1.0
FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
(where here 'v1.0' is the tag in github)

And for those packages where you don't have a specific tag but instead want to
download a specific revision, you can use:
FOO_VERSION = 1234567
FOO_SITE = http://github.com/<user>/<package>/tarball/<branch>
(where here '1234567' is the first 7 characters of the github commit hash)

In my tests, the former (downloading a specific tag) works okay, but the latter
(downloading a specific revision) actually doesn't work as expected. I can't
tell if this used to work and has become broken recently by a change to
github.com, or if it's always been broken (but nobody ever noticed), or if I'm
simply misunderstanding things (to save me looking stupid, hopefully it's not
this last one!).

I'll stick with the two FOO examples above, but substituting 'fooauthor' for
'<user>', 'foopackage' for '<package>' and 'master' for <branch>...

When downloading a specific tag, the first example above would download
http://github.com/fooauthor/foopackage/tarball/v1.0/foo-v1.0.tar.gz to
dl/foo-v1.0.tar.gz and this is the part that works correctly.
When downloading a specific revision, the second example above would download
http://github.com/fooauthor/foopackage/tarball/master/foo-1234567.tar.gz to
dl/foo-1234567.tar.gz and this appears to work fine, but it actually downloads
the latest revision of the master branch, and not the 1234567 revision as you
might expect! So if 1234567 _is_ indeed the latest revision, then things all
look okay, which may be why nobody noticed this bug before, but things go
quickly wrong if there's commits after 1234567.

In my testing, it seems that _whatever_ filename you have after the .../master/
in the URL will get ignored, and you'll always get the latest commit from the
master branch. However it seems that you can actually specify a revision
instead of <branch> and it will then work as expected, i.e. AFAICT the
'download by revision' example above should be modified to:
FOO_VERSION = 1234567
FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)

and then buildroot will download
http://github.com/fooauthor/foopackage/tarball/1234567/foo-1234567.tar.gz to
dl/foo-1234567.tar.gz and this *will* then work as expected. Obviously you
could set FOO_VERSION to 'master' and it would go back to the previous
behaviour of downloading the latest commit on the master branch, but at least
this behaviour would then be explicit rather than implicit.

Where the <branch> has been used (which I've replaced with $(FOO_VERSION)) I've
found that you can either use a tag-name, a branch-name, or a (full or partial)
commit-id, and then following another slash you can include any filename you
want, which makes no difference to the file github.com serves to you.

To give a more concrete example, I found a random repository with lots of
commits, tags and branches: https://github.com/libgit2/libgit2
As of 2012-06-06 these URLs all result in the identical file being downloaded:
[by branch-name]
http://github.com/libgit2/libgit2/tarball/master  (wget saves to 'master')
http://github.com/libgit2/libgit2/tarball/master/  (wget saves to 'index.html')
http://github.com/libgit2/libgit2/tarball/master/libgit2-master.tar.gz  (wget
saves to 'libgit2-master.tar.gz')
http://github.com/libgit2/libgit2/tarball/master/libgit2-dummy-filename.tgz
[by tag-name]
http://github.com/libgit2/libgit2/tarball/v0.18.0
http://github.com/libgit2/libgit2/tarball/v0.18.0/libgit2-v0.18.0.tar.gz
[by commit-id]
http://github.com/libgit2/libgit2/tarball/a50086d
http://github.com/libgit2/libgit2/tarball/a50086d/libgit2-a50086d.tar.gz
http://github.com/libgit2/libgit2/tarball/a50086d174/libgit2-a50086d174.tar.gz
http://github.com/libgit2/libgit2/tarball/a50086d174658914d4d6462afbc83b02825b1f5b
http://github.com/libgit2/libgit2/tarball/a50086d174658914d4d6462afbc83b02825b1f5b/libgit2-a50086d174658914d4d6462afbc83b02825b1f5b.tar.gz
(I used 'sha1sum *' to verify all the downloaded files are identical. In each
case the files inside the tarfile are inside a 'libgit2-libgit2-a50086d'
subdirectory)

And these URLs result in identical files being downloaded:
[by commit-id]
http://github.com/libgit2/libgit2/tarball/36c2dfe
http://github.com/libgit2/libgit2/tarball/36c2dfe/libgit2-36c2dfe.tar.gz
http://github.com/libgit2/libgit2/tarball/36c2dfed69/libgit2-36c2dfed69.tar.gz
http://github.com/libgit2/libgit2/tarball/36c2dfed696f80a20ca1352f32ec8b136b800c30/libgit2-36c2dfed696f80a20ca1352f32ec8b136b800c30.tar.gz

And these URLs result in identical files being downloaded:
[by branch-name]
http://github.com/libgit2/libgit2/tarball/development
http://github.com/libgit2/libgit2/tarball/development/libgit2-development.tar.gz
[by commit-id]
http://github.com/libgit2/libgit2/tarball/2e1fa15
http://github.com/libgit2/libgit2/tarball/2e1fa15/libgit2-2e1fa15.tar.gz

etc. etc. you get the idea. The only (unlikely occurrence) I didn't test was a
branch-name having the same name as a tag-name. Obviously the match-ups between
branch-names and commit-ids shown above will go 'out of sync' once more commits
have been made to those branches.

In summary, I believe
http://www.buildroot.org/downloads/manual/manual.html#github-download-url
should be updated to say that FOO_SITE should always look like:
FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
and then mention that FOO_VERSION can either specify a tag-name or a commit-id
or a branch-name.

In the current git version of buildroot, by constructing a funky find command:
find package -exec grep -P
'^[A-Z0-9_]+_SITE\s*=\s*https?://github\.com/.*/tarball/(?!.*\$\(.+\))' {} \;
-print | grep \.mk | sort

...I believe these are the packages that need to have their _SITE setting
updated in their .mk file:
jsmin, libcofi, linenoise, lua-msgpack-native, mtdev2tuio, ne10,
omap-u-boot-utils, rpi-firmware, rpi-userland, socketcand, sunxi-boards,
sunxi-tools, ti-utils, tslib

-- 
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.

^ permalink raw reply	[flat|nested] 29+ messages in thread