* Re: Details on Bug #963 kernel tarball corruption.
2011-07-07 20:34 Details on Bug #963 kernel tarball corruption Flanagan, Elizabeth
@ 2011-07-07 20:42 ` Flanagan, Elizabeth
2011-07-07 21:16 ` Richard Purdie
2011-07-07 21:51 ` Tom Zanussi
2 siblings, 0 replies; 8+ messages in thread
From: Flanagan, Elizabeth @ 2011-07-07 20:42 UTC (permalink / raw)
To: yocto, Richard Purdie, Saul Wold, Zanussi, Tom, Bruce Ashfield,
Hart, Darren
On Thu, Jul 7, 2011 at 1:34 PM, Flanagan, Elizabeth
<elizabeth.flanagan@intel.com> wrote:
> So, do_kernel_checkout never even gets the chance to fix up the git repo.
Quick clarification here. It never gets the chance to do it because
do_unpack runs:
git clone -s -n
/srv/build/build/poky/build/downloads/git2/.srv.build.build.repo.git.pokylinux.org.linux-yocto-2.6.37
/srv/build/build/poky/build/tmp/work/qemux86-poky-linux/linux-yocto-2.6.37+git3+94bca94ff11276b4cfffc1818c1defa3051854b2_1+ea7bcd9e408ddfaf5ecd0bcc37956998add58e2d-r20/git/
which does not clone the remotes to the destination directory. So
do_kernel_checkout never even sees BAD_BRANCH.
-b
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Details on Bug #963 kernel tarball corruption.
2011-07-07 20:34 Details on Bug #963 kernel tarball corruption Flanagan, Elizabeth
2011-07-07 20:42 ` Flanagan, Elizabeth
@ 2011-07-07 21:16 ` Richard Purdie
2011-07-07 23:57 ` Darren Hart
2011-07-07 21:51 ` Tom Zanussi
2 siblings, 1 reply; 8+ messages in thread
From: Richard Purdie @ 2011-07-07 21:16 UTC (permalink / raw)
To: Flanagan, Elizabeth; +Cc: yocto, Hart, Darren, Saul Wold
On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
> I wanted to ping the list on this bug as I've drilled down into it and
> I see what's going on. These are the steps to reproduce it and I have
> some questions at the end....
>
> The issue: The autobuilder has been serving up bad kernel source
> tarballs. This is not an autobuilder issue. I've narrowed this down to
> being an issue with do_fetch and do_unpack....
>
> First, I created a local linux-yocto repo and modified
> meta/recipes-kernel/linux/linux-yocto_2.6.37.bb to point to it:
>
> SRCREV_machine = ${AUTOREV}
> SRCREV_meta = ${AUTOREV}
>
> SRC_URI = "git:///srv/build/build/repo/git.pokylinux.org.linux-yocto-2.6.37;protocol=file;nocheckout=1;branch=${KBRANCH},meta;name=machine,meta"
>
> I do bitbake virtual/kernel -c kernel_checkout -f and the repo is
> fetched into my DL_DIR, unpacked into
> tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/git and then
> do_kernel_checkout runs. At this point, everything is correct.
>
> At this point my local SRC_URI repo is:
>
> * master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
>
> My clone in DL_DIR is the same.
>
> My clone in tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/linux
> looks like this:
>
> master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
> remotes/origin/master
> remotes/origin/meta
> remotes/origin/yocto/base
> remotes/origin/yocto/eg20t
> remotes/origin/yocto/emgd
> remotes/origin/yocto/gma500
>
> With the local head branches and the remotes being the same.
>
> Now, on to where this gets ugly.
>
> I commit a new branch to my SRC_URI repo:
>
> My SRC_URI:
>
> BAD_BRANCH
> master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
>
> I re-run fetch and my DL_DIR looks like this:
>
> * master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
> remotes/origin/BAD_BRANCH
> remotes/origin/master
> remotes/origin/meta
> remotes/origin/yocto/base
> remotes/origin/yocto/eg20t
> remotes/origin/yocto/emgd
> remotes/origin/yocto/gma500
>
> Notice, the new BAD_BRANCH is now only in remotes. This *should* be
> ok, since after we do_unpack we run do_kernel_checkout which should
> copy the refs in remotes to heads
I think we hit game over at this point. BAD_BRANCH should be present and
isn't :(.
I talked quickly with Beth over jabber. I think this is a fetcher bug
and something like:
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index f3bc793..7954f66 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -170,7 +170,7 @@ class Git(FetchMethod):
# If the repo still doesn't exist, fallback to cloning it
if not os.path.exists(ud.clonedir):
- clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
+ clone_cmd = "%s clone --bare --mirror %s://%s%s%s %s" % \
(ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir)
bb.fetch2.check_network_access(d, clone_cmd)
runfetchcmd(clone_cmd, d)
@@ -188,7 +188,7 @@ class Git(FetchMethod):
except bb.fetch2.FetchError:
logger.debug(1, "No Origin")
- runfetchcmd("%s remote add origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
+ runfetchcmd("%s remote add --mirror origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
fetch_cmd = "%s fetch --all -t" % ud.basecmd
bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
runfetchcmd(fetch_cmd, d)
might just fix this (since it means a refspec is added to keep the
remote and local heads in sync).
Cheers,
Richard
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: Details on Bug #963 kernel tarball corruption.
2011-07-07 21:16 ` Richard Purdie
@ 2011-07-07 23:57 ` Darren Hart
2011-07-08 0:08 ` Richard Purdie
0 siblings, 1 reply; 8+ messages in thread
From: Darren Hart @ 2011-07-07 23:57 UTC (permalink / raw)
To: Richard Purdie; +Cc: yocto, Saul Wold
On 07/07/2011 02:16 PM, Richard Purdie wrote:
> On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
>> I wanted to ping the list on this bug as I've drilled down into it and
>> I see what's going on. These are the steps to reproduce it and I have
>> some questions at the end....
>>
>> The issue: The autobuilder has been serving up bad kernel source
>> tarballs. This is not an autobuilder issue. I've narrowed this down to
>> being an issue with do_fetch and do_unpack....
>>
>> First, I created a local linux-yocto repo and modified
>> meta/recipes-kernel/linux/linux-yocto_2.6.37.bb to point to it:
>>
>> SRCREV_machine = ${AUTOREV}
>> SRCREV_meta = ${AUTOREV}
>>
>> SRC_URI = "git:///srv/build/build/repo/git.pokylinux.org.linux-yocto-2.6.37;protocol=file;nocheckout=1;branch=${KBRANCH},meta;name=machine,meta"
>>
>> I do bitbake virtual/kernel -c kernel_checkout -f and the repo is
>> fetched into my DL_DIR, unpacked into
>> tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/git and then
>> do_kernel_checkout runs. At this point, everything is correct.
>>
>> At this point my local SRC_URI repo is:
>>
>> * master
>> meta
>> yocto/base
>> yocto/eg20t
>> yocto/emgd
>> yocto/gma500
>> ....
>>
>> My clone in DL_DIR is the same.
>>
>> My clone in tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/linux
>> looks like this:
>>
>> master
>> meta
>> yocto/base
>> yocto/eg20t
>> yocto/emgd
>> yocto/gma500
>> ....
>> remotes/origin/master
>> remotes/origin/meta
>> remotes/origin/yocto/base
>> remotes/origin/yocto/eg20t
>> remotes/origin/yocto/emgd
>> remotes/origin/yocto/gma500
>>
>> With the local head branches and the remotes being the same.
>>
>> Now, on to where this gets ugly.
>>
>> I commit a new branch to my SRC_URI repo:
>>
>> My SRC_URI:
>>
>> BAD_BRANCH
>> master
>> meta
>> yocto/base
>> yocto/eg20t
>> yocto/emgd
>> yocto/gma500
>> ....
>>
>> I re-run fetch and my DL_DIR looks like this:
>>
>> * master
>> meta
>> yocto/base
>> yocto/eg20t
>> yocto/emgd
>> yocto/gma500
>> ....
>> remotes/origin/BAD_BRANCH
>> remotes/origin/master
>> remotes/origin/meta
>> remotes/origin/yocto/base
>> remotes/origin/yocto/eg20t
>> remotes/origin/yocto/emgd
>> remotes/origin/yocto/gma500
>>
>> Notice, the new BAD_BRANCH is now only in remotes. This *should* be
>> ok, since after we do_unpack we run do_kernel_checkout which should
>> copy the refs in remotes to heads
>
> I think we hit game over at this point. BAD_BRANCH should be present and
> isn't :(.
>
> I talked quickly with Beth over jabber. I think this is a fetcher bug
> and something like:
>
> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> index f3bc793..7954f66 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -170,7 +170,7 @@ class Git(FetchMethod):
>
> # If the repo still doesn't exist, fallback to cloning it
> if not os.path.exists(ud.clonedir):
> - clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
> + clone_cmd = "%s clone --bare --mirror %s://%s%s%s %s" % \
--mirror implies --bare, so:
- clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
+ clone_cmd = "%s clone --mirror %s://%s%s%s %s" % \
should be adequate.
--
Darren
> (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir)
> bb.fetch2.check_network_access(d, clone_cmd)
> runfetchcmd(clone_cmd, d)
> @@ -188,7 +188,7 @@ class Git(FetchMethod):
> except bb.fetch2.FetchError:
> logger.debug(1, "No Origin")
>
> - runfetchcmd("%s remote add origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
> + runfetchcmd("%s remote add --mirror origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
> fetch_cmd = "%s fetch --all -t" % ud.basecmd
> bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
> runfetchcmd(fetch_cmd, d)
>
> might just fix this (since it means a refspec is added to keep the
> remote and local heads in sync).
>
> Cheers,
>
> Richard
>
--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Details on Bug #963 kernel tarball corruption.
2011-07-07 23:57 ` Darren Hart
@ 2011-07-08 0:08 ` Richard Purdie
0 siblings, 0 replies; 8+ messages in thread
From: Richard Purdie @ 2011-07-08 0:08 UTC (permalink / raw)
To: Darren Hart; +Cc: yocto, Saul Wold
On Thu, 2011-07-07 at 16:57 -0700, Darren Hart wrote:
> On 07/07/2011 02:16 PM, Richard Purdie wrote:
> > On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
> > diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> > index f3bc793..7954f66 100644
> > --- a/bitbake/lib/bb/fetch2/git.py
> > +++ b/bitbake/lib/bb/fetch2/git.py
> > @@ -170,7 +170,7 @@ class Git(FetchMethod):
> >
> > # If the repo still doesn't exist, fallback to cloning it
> > if not os.path.exists(ud.clonedir):
> > - clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
> > + clone_cmd = "%s clone --bare --mirror %s://%s%s%s %s" % \
>
>
> --mirror implies --bare, so:
> - clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
> + clone_cmd = "%s clone --mirror %s://%s%s%s %s" % \
>
> should be adequate.
Right, I left it in as it makes it clearer to anyone looking at the
command that we're really sure we want a bare clone...
Cheers,
Richard
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Details on Bug #963 kernel tarball corruption.
2011-07-07 20:34 Details on Bug #963 kernel tarball corruption Flanagan, Elizabeth
2011-07-07 20:42 ` Flanagan, Elizabeth
2011-07-07 21:16 ` Richard Purdie
@ 2011-07-07 21:51 ` Tom Zanussi
2011-07-07 22:31 ` Flanagan, Elizabeth
2 siblings, 1 reply; 8+ messages in thread
From: Tom Zanussi @ 2011-07-07 21:51 UTC (permalink / raw)
To: Flanagan, Elizabeth; +Cc: yocto@yoctoproject.org, Hart, Darren, Wold, Saul
On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
> I wanted to ping the list on this bug as I've drilled down into it and
> I see what's going on. These are the steps to reproduce it and I have
> some questions at the end....
>
> The issue: The autobuilder has been serving up bad kernel source
> tarballs. This is not an autobuilder issue. I've narrowed this down to
> being an issue with do_fetch and do_unpack....
>
> First, I created a local linux-yocto repo and modified
> meta/recipes-kernel/linux/linux-yocto_2.6.37.bb to point to it:
>
> SRCREV_machine = ${AUTOREV}
> SRCREV_meta = ${AUTOREV}
>
> SRC_URI = "git:///srv/build/build/repo/git.pokylinux.org.linux-yocto-2.6.37;protocol=file;nocheckout=1;branch=${KBRANCH},meta;name=machine,meta"
>
> I do bitbake virtual/kernel -c kernel_checkout -f and the repo is
> fetched into my DL_DIR, unpacked into
> tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/git and then
> do_kernel_checkout runs. At this point, everything is correct.
>
> At this point my local SRC_URI repo is:
>
> * master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
>
> My clone in DL_DIR is the same.
>
Shouldn't this always be the state of DL_DIR i.e. any change made to the
contents of SRC_URI be exactly reflected in the same thing in DL_DIR?
If that were the case, I think it would solve the problems below as well
as the original problem in the bug report where downstream is apparently
being served the resulting badness in DL_DIR...
Tom
> My clone in tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/linux
> looks like this:
>
> master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
> remotes/origin/master
> remotes/origin/meta
> remotes/origin/yocto/base
> remotes/origin/yocto/eg20t
> remotes/origin/yocto/emgd
> remotes/origin/yocto/gma500
>
> With the local head branches and the remotes being the same.
>
> Now, on to where this gets ugly.
>
> I commit a new branch to my SRC_URI repo:
>
> My SRC_URI:
>
> BAD_BRANCH
> master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
>
> I re-run fetch and my DL_DIR looks like this:
>
> * master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
> remotes/origin/BAD_BRANCH
> remotes/origin/master
> remotes/origin/meta
> remotes/origin/yocto/base
> remotes/origin/yocto/eg20t
> remotes/origin/yocto/emgd
> remotes/origin/yocto/gma500
>
> Notice, the new BAD_BRANCH is now only in remotes. This *should* be
> ok, since after we do_unpack we run do_kernel_checkout which should
> copy the refs in remotes to heads. However....
>
> I clean out my workdir and run unpack. A git branch -a of the git dir
> it unpacks to shows that BAD_BRANCH does not exist:
>
> ~/poky/build/tmp/work/qemux86-poky-linux/linux-yocto-2.6.37+git3+94bca94ff11276b4cfffc1818c1defa3051854b2_1+ea7bcd9e408ddfaf5ecd0bcc37956998add58e2d-r20/git>
> git branch -a
> * master
> remotes/origin/HEAD -> origin/master
> remotes/origin/master
> remotes/origin/meta
> remotes/origin/yocto/base
> remotes/origin/yocto/eg20t
> remotes/origin/yocto/emgd
> remotes/origin/yocto/gma500
>
> So, do_kernel_checkout never even gets the chance to fix up the git repo.
>
> My questions:
>
> 1. This appears to only be occurring with the kernel tarball, however,
> I'm concerned that it may also be happening elsewhere though. Without
> diving into the bb fetch code, should this be working like this?
> 2. This could easily be fixed by ripping out:
>
> cp -r ${S}/.git/refs/remotes/origin/* ${S}/.git/refs/heads
> rmdir ${S}/.git/refs/remotes/origin
>
> from do_kernel_checkout into a post_fetch target in
> kernel-yocto.bbclass, but if 1. is true, this obviously isn't the
> correct way to fix this.
>
> Ideas? Comments?
>
> -b
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Details on Bug #963 kernel tarball corruption.
2011-07-07 21:51 ` Tom Zanussi
@ 2011-07-07 22:31 ` Flanagan, Elizabeth
2011-07-07 22:46 ` Richard Purdie
0 siblings, 1 reply; 8+ messages in thread
From: Flanagan, Elizabeth @ 2011-07-07 22:31 UTC (permalink / raw)
To: Tom Zanussi; +Cc: yocto@yoctoproject.org, Hart, Darren, Wold, Saul
On Thu, Jul 7, 2011 at 2:51 PM, Tom Zanussi <tom.zanussi@intel.com> wrote:
> On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
>
> Shouldn't this always be the state of DL_DIR i.e. any change made to the
> contents of SRC_URI be exactly reflected in the same thing in DL_DIR?
> If that were the case, I think it would solve the problems below as well
> as the original problem in the bug report where downstream is apparently
> being served the resulting badness in DL_DIR...
It should and in a clean DL_DIR, do_fetch does, in fact, clone
correctly. But do_fetch with an already populated fetched repo in
DL_DIR causes it to get dirty by placing new branches in
remotes/origin. This is caused by fetch, however, looking at
do_kernel_checkout, it seems as if this was at one point, expected
behavior.
-b
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Details on Bug #963 kernel tarball corruption.
2011-07-07 22:31 ` Flanagan, Elizabeth
@ 2011-07-07 22:46 ` Richard Purdie
0 siblings, 0 replies; 8+ messages in thread
From: Richard Purdie @ 2011-07-07 22:46 UTC (permalink / raw)
To: Flanagan, Elizabeth; +Cc: Hart, Darren, Wold, Saul, yocto@yoctoproject.org
On Thu, 2011-07-07 at 15:31 -0700, Flanagan, Elizabeth wrote:
> On Thu, Jul 7, 2011 at 2:51 PM, Tom Zanussi <tom.zanussi@intel.com> wrote:
> > On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
> >
> > Shouldn't this always be the state of DL_DIR i.e. any change made to the
> > contents of SRC_URI be exactly reflected in the same thing in DL_DIR?
> > If that were the case, I think it would solve the problems below as well
> > as the original problem in the bug report where downstream is apparently
> > being served the resulting badness in DL_DIR...
>
> It should and in a clean DL_DIR, do_fetch does, in fact, clone
> correctly. But do_fetch with an already populated fetched repo in
> DL_DIR causes it to get dirty by placing new branches in
> remotes/origin. This is caused by fetch, however, looking at
> do_kernel_checkout, it seems as if this was at one point, expected
> behavior.
It is expected, *but* it should also update the local heads in sync with
the remotes.
The --mirror options claims to do this and should be exactly what is
missing...
Cheers,
Richard
^ permalink raw reply [flat|nested] 8+ messages in thread