New issue with git fetcher?

New issue with git fetcher?

[Scott Garman]

Hi Richard,

This afternoon I performed a build from scratch using master, which 
included your git fetcher change to use mirror tarballs 
(1077021f7010388e889d85630b13c4c7901d7718).

When I performed my build from scratch (using a pre-populated DL_DIR 
from previous builds), linux-wrs failed on do_fetch, with the following 
in log.do_fetch:

 From git://git.pokylinux.org/linux-2.6-windriver
   * branch            common_pc-standard -> FETCH_HEAD
  From git://git.pokylinux.org/linux-2.6-windriver
   * [new tag]         v2.6.36-rc5 -> v2.6.36-rc5
   * [new tag]         v2.6.36-rc6 -> v2.6.36-rc6
  NOTE: Creating tarball of git repository
ERROR: Task failed: Unknown fetch Error: coercing to Unicode: need 
string or buffer, NoneType found

I performed a -c clean and then -c fetch, but ran into the same problem.

I then went and deleted the linux-wrs-related sources from DL_DIR, and 
ran -c clean and -c fetch again, and it looks to me like the fetcher is 
actually downloading tarballs from the source mirrors first, and then 
trying git directly:

NOTE: Running task 130 of 130 (ID: 0, 
/raid0/sources/poky/meta/recipes-kernel/linux/linux-wrs_git.bb, do_fetch)
NOTE: package 
linux-wrs-2.6.34+git0+b67e060194a38c6331da1532bd06446087a42b3b_0+0431115c9d720fee5bb105f6a7411efb4f851d26-r12: 
task do_fetch: Started
NOTE: fetch 
http://pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz
NOTE: fetch 
http://autobuilder.pokylinux.org/sources/git_git.pokylinux.org.linux-2.6-windriver.git.tar.gz

I confirmed this by seeing wget running in my process list, and then git 
eventually shows up.

Finally, I get the bitbake message:

NOTE: Creating tarball of git repository

...and I see gzip running, but the task ends up failing with the same 
error as the log.do_fetch example I mentioned above.

Scott

Re: New issue with git fetcher?

[Scott Garman]

On 10/18/2010 05:54 PM, Scott Garman wrote:
> Hi Richard,
>
> This afternoon I performed a build from scratch using master, which
> included your git fetcher change to use mirror tarballs
> (1077021f7010388e889d85630b13c4c7901d7718).

As a follow-up, when I revert this commit, -c clean and then -c fetch 
succeeds for linux-wrs.

Scott

Re: New issue with git fetcher?

[Richard Purdie]

On Mon, 2010-10-18 at 18:00 -0700, Scott Garman wrote:
> On 10/18/2010 05:54 PM, Scott Garman wrote:
> > Hi Richard,
> >
> > This afternoon I performed a build from scratch using master, which
> > included your git fetcher change to use mirror tarballs
> > (1077021f7010388e889d85630b13c4c7901d7718).

Thanks for testing and reporting this quickly, its appreciated.

> As a follow-up, when I revert this commit, -c clean and then -c fetch 
> succeeds for linux-wrs.

but fails for libzypp :/.

I started debugging the fetcher code and the whole thing basically fell
apart as I was finding all kinds of issues, some of which we're just not
hitting by pure luck. I had hoped that code would last us into the 1.0
development window when I've some significant fetcher changes in mind.
Short summary is that its not going to do that.

I've pushed some more invasive changes to the fetcher in the Poky master
branch. I'd appreciate the teams help in finding out if these work or
not, with both existing builds and with totally fresh from scratch
builds.

If they prove not to have major issues we're likely to merge these into
the release branch as what is there is going to bite people and look
rather bad.

Cheers,

Richard

Re: New issue with git fetcher?

[Richard Purdie]

On Tue, 2010-10-19 at 13:55 +0100, Richard Purdie wrote:
> I've pushed some more invasive changes to the fetcher in the Poky master
> branch. I'd appreciate the teams help in finding out if these work or
> not, with both existing builds and with totally fresh from scratch
> builds.
> 
> If they prove not to have major issues we're likely to merge these into
> the release branch as what is there is going to bite people and look
> rather bad.

There were some issues that Tom and Joshua were seeing which I debugged
and found that we weren't adding in new branches in the fetching code.

I was able to reproduce and then have added a patch to the git fetcher
which ensures these things are cloned correctly and any new branches are
created. I talked with Bruce to confirm my diagnosis and the solution.

This fix makes an awful lot of sense and explains various issues we've
been chasing for a while with the fetcher bugs so I'm very pleased to
have this fix checked in.

Could people continue to test the code and report any problems...

Thanks,

Richard

Re: New issue with git fetcher?

[Tian, Kevin]

>From: Richard Purdie
>Sent: Wednesday, October 20, 2010 12:29 AM
>
>On Tue, 2010-10-19 at 13:55 +0100, Richard Purdie wrote:
>> I've pushed some more invasive changes to the fetcher in the Poky master
>> branch. I'd appreciate the teams help in finding out if these work or
>> not, with both existing builds and with totally fresh from scratch
>> builds.
>>
>> If they prove not to have major issues we're likely to merge these into
>> the release branch as what is there is going to bite people and look
>> rather bad.
>
>There were some issues that Tom and Joshua were seeing which I debugged
>and found that we weren't adding in new branches in the fetching code.
>
>I was able to reproduce and then have added a patch to the git fetcher
>which ensures these things are cloned correctly and any new branches are
>created. I talked with Bruce to confirm my diagnosis and the solution.
>
>This fix makes an awful lot of sense and explains various issues we've
>been chasing for a while with the fetcher bugs so I'm very pleased to
>have this fix checked in.
>
>Could people continue to test the code and report any problems...
>

I tried "bitbake poky-image-sdk -c fetchall" both with an existing source directory
and with a fresh new download. Everything works fine. 

Thanks
Kevin

Re: New issue with git fetcher?

[Richard Purdie]

On Wed, 2010-10-20 at 16:20 +0800, Tian, Kevin wrote:
> I tried "bitbake poky-image-sdk -c fetchall" both with an existing source directory
> and with a fresh new download. Everything works fine. 

Thanks for testing and the feedback!

Cheers,

Richard