Details on Bug #963 kernel tarball corruption.

Details on Bug #963 kernel tarball corruption.

[Flanagan, Elizabeth]

I wanted to ping the list on this bug as I've drilled down into it and
I see what's going on. These are the steps to reproduce it and I have
some questions at the end....

The issue: The autobuilder has been serving up bad kernel source
tarballs. This is not an autobuilder issue. I've narrowed this down to
being an issue with do_fetch and do_unpack....

First, I created a local linux-yocto repo and modified
meta/recipes-kernel/linux/linux-yocto_2.6.37.bb to point to it:

SRCREV_machine = ${AUTOREV}
SRCREV_meta = ${AUTOREV}

SRC_URI = "git:///srv/build/build/repo/git.pokylinux.org.linux-yocto-2.6.37;protocol=file;nocheckout=1;branch=${KBRANCH},meta;name=machine,meta"

I do bitbake virtual/kernel -c kernel_checkout -f and the repo is
fetched into my DL_DIR, unpacked into
tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/git and then
do_kernel_checkout runs. At this point, everything is correct.

At this point my local SRC_URI repo is:

* master
  meta
  yocto/base
  yocto/eg20t
  yocto/emgd
  yocto/gma500
....

My clone in DL_DIR is the same.

My clone in  tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/linux
looks like this:

  master
  meta
  yocto/base
  yocto/eg20t
  yocto/emgd
  yocto/gma500
  ....
  remotes/origin/master
  remotes/origin/meta
  remotes/origin/yocto/base
  remotes/origin/yocto/eg20t
  remotes/origin/yocto/emgd
  remotes/origin/yocto/gma500

With the local head branches and the remotes being the same.

Now, on to where this gets ugly.

I commit a new branch to my SRC_URI repo:

My SRC_URI:

BAD_BRANCH
master
meta
yocto/base
yocto/eg20t
yocto/emgd
yocto/gma500
....

I re-run fetch and my DL_DIR looks like this:

* master
  meta
  yocto/base
  yocto/eg20t
  yocto/emgd
  yocto/gma500
....
  remotes/origin/BAD_BRANCH
  remotes/origin/master
  remotes/origin/meta
  remotes/origin/yocto/base
  remotes/origin/yocto/eg20t
  remotes/origin/yocto/emgd
  remotes/origin/yocto/gma500

Notice, the new BAD_BRANCH is now only in remotes. This *should* be
ok, since after we do_unpack we run do_kernel_checkout which should
copy the refs in remotes to heads. However....

I clean out my workdir and run unpack. A git branch -a of the git dir
it unpacks to shows that BAD_BRANCH does not exist:

~/poky/build/tmp/work/qemux86-poky-linux/linux-yocto-2.6.37+git3+94bca94ff11276b4cfffc1818c1defa3051854b2_1+ea7bcd9e408ddfaf5ecd0bcc37956998add58e2d-r20/git>
git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/meta
  remotes/origin/yocto/base
  remotes/origin/yocto/eg20t
  remotes/origin/yocto/emgd
  remotes/origin/yocto/gma500

So, do_kernel_checkout never even gets the chance to fix up the git repo.

My questions:

1. This appears to only be occurring with the kernel tarball, however,
I'm concerned that it may also be happening elsewhere though. Without
diving into the bb fetch code, should this be working like this?
2. This could easily be fixed by ripping out:

cp -r ${S}/.git/refs/remotes/origin/* ${S}/.git/refs/heads
rmdir ${S}/.git/refs/remotes/origin

from do_kernel_checkout into a post_fetch target in
kernel-yocto.bbclass, but if 1. is true, this obviously isn't the
correct way to fix this.

Ideas? Comments?

-b

Re: Details on Bug #963 kernel tarball corruption.

[Flanagan, Elizabeth]

On Thu, Jul 7, 2011 at 1:34 PM, Flanagan, Elizabeth
<elizabeth.flanagan@intel.com> wrote:
> So, do_kernel_checkout never even gets the chance to fix up the git repo.

Quick clarification here. It never gets the chance to do it because
do_unpack runs:

git clone -s -n
/srv/build/build/poky/build/downloads/git2/.srv.build.build.repo.git.pokylinux.org.linux-yocto-2.6.37
/srv/build/build/poky/build/tmp/work/qemux86-poky-linux/linux-yocto-2.6.37+git3+94bca94ff11276b4cfffc1818c1defa3051854b2_1+ea7bcd9e408ddfaf5ecd0bcc37956998add58e2d-r20/git/

which does not clone the remotes to the destination directory. So
do_kernel_checkout never even sees BAD_BRANCH.

-b

Re: Details on Bug #963 kernel tarball corruption.

[Richard Purdie]

On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
> I wanted to ping the list on this bug as I've drilled down into it and
> I see what's going on. These are the steps to reproduce it and I have
> some questions at the end....
> 
> The issue: The autobuilder has been serving up bad kernel source
> tarballs. This is not an autobuilder issue. I've narrowed this down to
> being an issue with do_fetch and do_unpack....
> 
> First, I created a local linux-yocto repo and modified
> meta/recipes-kernel/linux/linux-yocto_2.6.37.bb to point to it:
> 
> SRCREV_machine = ${AUTOREV}
> SRCREV_meta = ${AUTOREV}
> 
> SRC_URI = "git:///srv/build/build/repo/git.pokylinux.org.linux-yocto-2.6.37;protocol=file;nocheckout=1;branch=${KBRANCH},meta;name=machine,meta"
> 
> I do bitbake virtual/kernel -c kernel_checkout -f and the repo is
> fetched into my DL_DIR, unpacked into
> tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/git and then
> do_kernel_checkout runs. At this point, everything is correct.
> 
> At this point my local SRC_URI repo is:
> 
> * master
>   meta
>   yocto/base
>   yocto/eg20t
>   yocto/emgd
>   yocto/gma500
> ....
> 
> My clone in DL_DIR is the same.
> 
> My clone in  tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/linux
> looks like this:
> 
>   master
>   meta
>   yocto/base
>   yocto/eg20t
>   yocto/emgd
>   yocto/gma500
>   ....
>   remotes/origin/master
>   remotes/origin/meta
>   remotes/origin/yocto/base
>   remotes/origin/yocto/eg20t
>   remotes/origin/yocto/emgd
>   remotes/origin/yocto/gma500
> 
> With the local head branches and the remotes being the same.
> 
> Now, on to where this gets ugly.
> 
> I commit a new branch to my SRC_URI repo:
> 
> My SRC_URI:
> 
> BAD_BRANCH
> master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
> 
> I re-run fetch and my DL_DIR looks like this:
> 
> * master
>   meta
>   yocto/base
>   yocto/eg20t
>   yocto/emgd
>   yocto/gma500
> ....
>   remotes/origin/BAD_BRANCH
>   remotes/origin/master
>   remotes/origin/meta
>   remotes/origin/yocto/base
>   remotes/origin/yocto/eg20t
>   remotes/origin/yocto/emgd
>   remotes/origin/yocto/gma500
> 
> Notice, the new BAD_BRANCH is now only in remotes. This *should* be
> ok, since after we do_unpack we run do_kernel_checkout which should
> copy the refs in remotes to heads

I think we hit game over at this point. BAD_BRANCH should be present and
isn't :(.

I talked quickly with Beth over jabber. I think this is a fetcher bug
and something like:

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index f3bc793..7954f66 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -170,7 +170,7 @@ class Git(FetchMethod):
 
         # If the repo still doesn't exist, fallback to cloning it
         if not os.path.exists(ud.clonedir):
-            clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
+            clone_cmd = "%s clone --bare --mirror %s://%s%s%s %s" % \
                   (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir)
             bb.fetch2.check_network_access(d, clone_cmd)
             runfetchcmd(clone_cmd, d)
@@ -188,7 +188,7 @@ class Git(FetchMethod):
             except bb.fetch2.FetchError:
                 logger.debug(1, "No Origin")
             
-            runfetchcmd("%s remote add origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
+            runfetchcmd("%s remote add --mirror origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
             fetch_cmd = "%s fetch --all -t" % ud.basecmd
             bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
             runfetchcmd(fetch_cmd, d)

might just fix this (since it means a refspec is added to keep the
remote and local heads in sync).

Cheers,

Richard

Re: Details on Bug #963 kernel tarball corruption.

[Tom Zanussi]

On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
> I wanted to ping the list on this bug as I've drilled down into it and
> I see what's going on. These are the steps to reproduce it and I have
> some questions at the end....
> 
> The issue: The autobuilder has been serving up bad kernel source
> tarballs. This is not an autobuilder issue. I've narrowed this down to
> being an issue with do_fetch and do_unpack....
> 
> First, I created a local linux-yocto repo and modified
> meta/recipes-kernel/linux/linux-yocto_2.6.37.bb to point to it:
> 
> SRCREV_machine = ${AUTOREV}
> SRCREV_meta = ${AUTOREV}
> 
> SRC_URI = "git:///srv/build/build/repo/git.pokylinux.org.linux-yocto-2.6.37;protocol=file;nocheckout=1;branch=${KBRANCH},meta;name=machine,meta"
> 
> I do bitbake virtual/kernel -c kernel_checkout -f and the repo is
> fetched into my DL_DIR, unpacked into
> tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/git and then
> do_kernel_checkout runs. At this point, everything is correct.
> 
> At this point my local SRC_URI repo is:
> 
> * master
>   meta
>   yocto/base
>   yocto/eg20t
>   yocto/emgd
>   yocto/gma500
> ....
> 
> My clone in DL_DIR is the same.
> 

Shouldn't this always be the state of DL_DIR i.e. any change made to the
contents of SRC_URI be exactly reflected in the same thing in DL_DIR?
If that were the case, I think it would solve the problems below as well
as the original problem in the bug report where downstream is apparently
being served the resulting badness in DL_DIR...

Tom

> My clone in  tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/linux
> looks like this:
> 
>   master
>   meta
>   yocto/base
>   yocto/eg20t
>   yocto/emgd
>   yocto/gma500
>   ....
>   remotes/origin/master
>   remotes/origin/meta
>   remotes/origin/yocto/base
>   remotes/origin/yocto/eg20t
>   remotes/origin/yocto/emgd
>   remotes/origin/yocto/gma500
> 
> With the local head branches and the remotes being the same.
> 
> Now, on to where this gets ugly.
> 
> I commit a new branch to my SRC_URI repo:
> 
> My SRC_URI:
> 
> BAD_BRANCH
> master
> meta
> yocto/base
> yocto/eg20t
> yocto/emgd
> yocto/gma500
> ....
> 
> I re-run fetch and my DL_DIR looks like this:
> 
> * master
>   meta
>   yocto/base
>   yocto/eg20t
>   yocto/emgd
>   yocto/gma500
> ....
>   remotes/origin/BAD_BRANCH
>   remotes/origin/master
>   remotes/origin/meta
>   remotes/origin/yocto/base
>   remotes/origin/yocto/eg20t
>   remotes/origin/yocto/emgd
>   remotes/origin/yocto/gma500
> 
> Notice, the new BAD_BRANCH is now only in remotes. This *should* be
> ok, since after we do_unpack we run do_kernel_checkout which should
> copy the refs in remotes to heads. However....
> 
> I clean out my workdir and run unpack. A git branch -a of the git dir
> it unpacks to shows that BAD_BRANCH does not exist:
> 
> ~/poky/build/tmp/work/qemux86-poky-linux/linux-yocto-2.6.37+git3+94bca94ff11276b4cfffc1818c1defa3051854b2_1+ea7bcd9e408ddfaf5ecd0bcc37956998add58e2d-r20/git>
> git branch -a
> * master
>   remotes/origin/HEAD -> origin/master
>   remotes/origin/master
>   remotes/origin/meta
>   remotes/origin/yocto/base
>   remotes/origin/yocto/eg20t
>   remotes/origin/yocto/emgd
>   remotes/origin/yocto/gma500
> 
> So, do_kernel_checkout never even gets the chance to fix up the git repo.
> 
> My questions:
> 
> 1. This appears to only be occurring with the kernel tarball, however,
> I'm concerned that it may also be happening elsewhere though. Without
> diving into the bb fetch code, should this be working like this?
> 2. This could easily be fixed by ripping out:
> 
> cp -r ${S}/.git/refs/remotes/origin/* ${S}/.git/refs/heads
> rmdir ${S}/.git/refs/remotes/origin
> 
> from do_kernel_checkout into a post_fetch target in
> kernel-yocto.bbclass, but if 1. is true, this obviously isn't the
> correct way to fix this.
> 
> Ideas? Comments?
> 
> -b

Re: Details on Bug #963 kernel tarball corruption.

[Flanagan, Elizabeth]

On Thu, Jul 7, 2011 at 2:51 PM, Tom Zanussi <tom.zanussi@intel.com> wrote:
> On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
>
> Shouldn't this always be the state of DL_DIR i.e. any change made to the
> contents of SRC_URI be exactly reflected in the same thing in DL_DIR?
> If that were the case, I think it would solve the problems below as well
> as the original problem in the bug report where downstream is apparently
> being served the resulting badness in DL_DIR...

It should and in a clean DL_DIR, do_fetch does, in fact, clone
correctly. But do_fetch with an already populated fetched repo in
DL_DIR causes it to get dirty by placing new branches in
remotes/origin. This is caused by fetch, however, looking at
do_kernel_checkout, it seems as if this was at one point, expected
behavior.

-b

Re: Details on Bug #963 kernel tarball corruption.

[Richard Purdie]

On Thu, 2011-07-07 at 15:31 -0700, Flanagan, Elizabeth wrote:
> On Thu, Jul 7, 2011 at 2:51 PM, Tom Zanussi <tom.zanussi@intel.com> wrote:
> > On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
> >
> > Shouldn't this always be the state of DL_DIR i.e. any change made to the
> > contents of SRC_URI be exactly reflected in the same thing in DL_DIR?
> > If that were the case, I think it would solve the problems below as well
> > as the original problem in the bug report where downstream is apparently
> > being served the resulting badness in DL_DIR...
> 
> It should and in a clean DL_DIR, do_fetch does, in fact, clone
> correctly. But do_fetch with an already populated fetched repo in
> DL_DIR causes it to get dirty by placing new branches in
> remotes/origin. This is caused by fetch, however, looking at
> do_kernel_checkout, it seems as if this was at one point, expected
> behavior.

It is expected, *but* it should also update the local heads in sync with
the remotes.

The --mirror options claims to do this and should be exactly what is
missing...

Cheers,

Richard

Re: Details on Bug #963 kernel tarball corruption.

[Darren Hart]

On 07/07/2011 02:16 PM, Richard Purdie wrote:
> On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
>> I wanted to ping the list on this bug as I've drilled down into it and
>> I see what's going on. These are the steps to reproduce it and I have
>> some questions at the end....
>>
>> The issue: The autobuilder has been serving up bad kernel source
>> tarballs. This is not an autobuilder issue. I've narrowed this down to
>> being an issue with do_fetch and do_unpack....
>>
>> First, I created a local linux-yocto repo and modified
>> meta/recipes-kernel/linux/linux-yocto_2.6.37.bb to point to it:
>>
>> SRCREV_machine = ${AUTOREV}
>> SRCREV_meta = ${AUTOREV}
>>
>> SRC_URI = "git:///srv/build/build/repo/git.pokylinux.org.linux-yocto-2.6.37;protocol=file;nocheckout=1;branch=${KBRANCH},meta;name=machine,meta"
>>
>> I do bitbake virtual/kernel -c kernel_checkout -f and the repo is
>> fetched into my DL_DIR, unpacked into
>> tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/git and then
>> do_kernel_checkout runs. At this point, everything is correct.
>>
>> At this point my local SRC_URI repo is:
>>
>> * master
>>   meta
>>   yocto/base
>>   yocto/eg20t
>>   yocto/emgd
>>   yocto/gma500
>> ....
>>
>> My clone in DL_DIR is the same.
>>
>> My clone in  tmp/work/qemux86-poky-linux/linux-yocto-2.6.37....r20/linux
>> looks like this:
>>
>>   master
>>   meta
>>   yocto/base
>>   yocto/eg20t
>>   yocto/emgd
>>   yocto/gma500
>>   ....
>>   remotes/origin/master
>>   remotes/origin/meta
>>   remotes/origin/yocto/base
>>   remotes/origin/yocto/eg20t
>>   remotes/origin/yocto/emgd
>>   remotes/origin/yocto/gma500
>>
>> With the local head branches and the remotes being the same.
>>
>> Now, on to where this gets ugly.
>>
>> I commit a new branch to my SRC_URI repo:
>>
>> My SRC_URI:
>>
>> BAD_BRANCH
>> master
>> meta
>> yocto/base
>> yocto/eg20t
>> yocto/emgd
>> yocto/gma500
>> ....
>>
>> I re-run fetch and my DL_DIR looks like this:
>>
>> * master
>>   meta
>>   yocto/base
>>   yocto/eg20t
>>   yocto/emgd
>>   yocto/gma500
>> ....
>>   remotes/origin/BAD_BRANCH
>>   remotes/origin/master
>>   remotes/origin/meta
>>   remotes/origin/yocto/base
>>   remotes/origin/yocto/eg20t
>>   remotes/origin/yocto/emgd
>>   remotes/origin/yocto/gma500
>>
>> Notice, the new BAD_BRANCH is now only in remotes. This *should* be
>> ok, since after we do_unpack we run do_kernel_checkout which should
>> copy the refs in remotes to heads
> 
> I think we hit game over at this point. BAD_BRANCH should be present and
> isn't :(.
> 
> I talked quickly with Beth over jabber. I think this is a fetcher bug
> and something like:
> 
> diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> index f3bc793..7954f66 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -170,7 +170,7 @@ class Git(FetchMethod):
>  
>          # If the repo still doesn't exist, fallback to cloning it
>          if not os.path.exists(ud.clonedir):
> -            clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
> +            clone_cmd = "%s clone --bare --mirror %s://%s%s%s %s" % \


--mirror implies --bare, so:
-            clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
+            clone_cmd = "%s clone --mirror %s://%s%s%s %s" % \

should be adequate.

--
Darren

>                    (ud.basecmd, ud.proto, username, ud.host, ud.path, ud.clonedir)
>              bb.fetch2.check_network_access(d, clone_cmd)
>              runfetchcmd(clone_cmd, d)
> @@ -188,7 +188,7 @@ class Git(FetchMethod):
>              except bb.fetch2.FetchError:
>                  logger.debug(1, "No Origin")
>              
> -            runfetchcmd("%s remote add origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
> +            runfetchcmd("%s remote add --mirror origin %s://%s%s%s" % (ud.basecmd, ud.proto, username, ud.host, ud.path), d)
>              fetch_cmd = "%s fetch --all -t" % ud.basecmd
>              bb.fetch2.check_network_access(d, fetch_cmd, ud.url)
>              runfetchcmd(fetch_cmd, d)
> 
> might just fix this (since it means a refspec is added to keep the
> remote and local heads in sync).
> 
> Cheers,
> 
> Richard
> 


-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Details on Bug #963 kernel tarball corruption.

[Richard Purdie]

On Thu, 2011-07-07 at 16:57 -0700, Darren Hart wrote:
> On 07/07/2011 02:16 PM, Richard Purdie wrote:
> > On Thu, 2011-07-07 at 13:34 -0700, Flanagan, Elizabeth wrote:
> > diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
> > index f3bc793..7954f66 100644
> > --- a/bitbake/lib/bb/fetch2/git.py
> > +++ b/bitbake/lib/bb/fetch2/git.py
> > @@ -170,7 +170,7 @@ class Git(FetchMethod):
> >  
> >          # If the repo still doesn't exist, fallback to cloning it
> >          if not os.path.exists(ud.clonedir):
> > -            clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
> > +            clone_cmd = "%s clone --bare --mirror %s://%s%s%s %s" % \
> 
> 
> --mirror implies --bare, so:
> -            clone_cmd = "%s clone --bare %s://%s%s%s %s" % \
> +            clone_cmd = "%s clone --mirror %s://%s%s%s %s" % \
> 
> should be adequate.

Right, I left it in as it makes it clearer to anyone looking at the
command that we're really sure we want a bare clone...

Cheers,

Richard