Git SRC_TARBALL_STASH filename

Git SRC_TARBALL_STASH filename

[Douglas Royds]

Working behind a firewall that doesn't allow Git access (except over 
http). I've set SRC_TARBALL_STASH so that we only have to download the 
appropriate source tarball (manually) once.

While building mtd-utils, BitBake is trying to fetch from the tarball stash:

    http://.../git_git.infradead.org.mtd-utils.git.tar.gz

Instead of:

    http://.../git_git.infradead.org.mtd-utils.git_ea429635388f7bb53f62c41ec3d5ccf5fa207370.tar.gz

The following patch appears to do the job, but I'm not confident this is 
the right fix. I notice that all the other fetchers use ud.localfile, 
which already has the correct filename.

Index: lib/bb/fetch/git.py
===================================================================
--- lib/bb/fetch/git.py    (revision 16639)
+++ lib/bb/fetch/git.py    (working copy)
@@ -73,7 +73,7 @@
 
         gitsrcname = '%s%s' % (ud.host, ud.path.replace('/', '.'))
 
-        repofilename = 'git_%s.tar.gz' % (gitsrcname)
+        repofilename = 'git_%s_%s.tar.gz' % (gitsrcname, ud.tag)
         repofile = os.path.join(data.getVar("DL_DIR", d, 1), repofilename)
         repodir = os.path.join(data.expand('${GITDIR}', d), gitsrcname)


Suggestions?
Douglas.






=======================================================================
This email, including any attachments, is only for the intended
addressee.  It is subject to copyright, is confidential and may be
the subject of legal or other privilege, none of which is waived or
lost by reason of this transmission.
If the receiver is not the intended addressee, please accept our
apologies, notify us by return, delete all copies and perform no
other act on the email.
Unfortunately, we cannot warrant that the email has not been
 altered or corrupted during transmission.
=======================================================================

Re: Git SRC_TARBALL_STASH filename

[Tom Rini]

On Tue, Mar 17, 2009 at 05:33:14PM +1300, Douglas Royds wrote:

> Working behind a firewall that doesn't allow Git access (except over  
> http). I've set SRC_TARBALL_STASH so that we only have to download the  
> appropriate source tarball (manually) once.
>
> While building mtd-utils, BitBake is trying to fetch from the tarball stash:
>
>    http://.../git_git.infradead.org.mtd-utils.git.tar.gz
>
> Instead of:
>
>    http://.../git_git.infradead.org.mtd-utils.git_ea429635388f7bb53f62c41ec3d5ccf5fa207370.tar.gz
>
> The following patch appears to do the job, but I'm not confident this is  
> the right fix. I notice that all the other fetchers use ud.localfile,  
> which already has the correct filename.

What's going on is that it's trying to fetch the raw git tree stash,
rather than the checkout stash, so you need to mirror the other.  OTOH,
perhaps it should try first the checkout then the raw?

-- 
Tom Rini

Re: Git SRC_TARBALL_STASH filename

[Shane Dixon]

Does SRC_TARBALL_STASH replace the PREMIRRORS variable?  I used to use a
variable in my local.conf that was similar to the following:

PREMIRRORS = "(ftp|https?)$://.*/.*/ http://server.com/openembedded/"

Is SRC_TARBALL_STASH the better way to do this?

--
Shane


On Mon, 2009-03-16 at 22:04 -0700, Tom Rini wrote:
> On Tue, Mar 17, 2009 at 05:33:14PM +1300, Douglas Royds wrote:
> 
> > Working behind a firewall that doesn't allow Git access (except over  
> > http). I've set SRC_TARBALL_STASH so that we only have to download the  
> > appropriate source tarball (manually) once.
> >
> > While building mtd-utils, BitBake is trying to fetch from the tarball stash:
> >
> >    http://.../git_git.infradead.org.mtd-utils.git.tar.gz
> >
> > Instead of:
> >
> >    http://.../git_git.infradead.org.mtd-utils.git_ea429635388f7bb53f62c41ec3d5ccf5fa207370.tar.gz
> >
> > The following patch appears to do the job, but I'm not confident this is  
> > the right fix. I notice that all the other fetchers use ud.localfile,  
> > which already has the correct filename.
> 
> What's going on is that it's trying to fetch the raw git tree stash,
> rather than the checkout stash, so you need to mirror the other.  OTOH,
> perhaps it should try first the checkout then the raw?
> 
-- 
Shane Dixon
Atmel Corporation
E-mail: shane.dixon@atmel.com

Re: Git SRC_TARBALL_STASH filename

[Tom Rini]

On Tue, Mar 17, 2009 at 04:40:09PM +0000, Shane Dixon wrote:

> Does SRC_TARBALL_STASH replace the PREMIRRORS variable?  I used to use a
> variable in my local.conf that was similar to the following:
> 
> PREMIRRORS = "(ftp|https?)$://.*/.*/ http://server.com/openembedded/"
> 
> Is SRC_TARBALL_STASH the better way to do this?

I believe the problem is that you need to set both to cover all cases.

-- 
Tom Rini

Re: Git SRC_TARBALL_STASH filename

[Douglas Royds]

SRC_TARBALL_STASH is used by all the VCS "fetchers" (CVS, Subversion, 
Git, ...) to try to download the desired VCS tagged version as a tarball 
before going to the VCS itself.

SOURCE_MIRROR_URL is used in own-mirrors.bbclass to set the PREMIRRORS 
variable, which is used by the wget.pyb BitBake "fetcher" before it has 
a go at the specified SRC_URI. SOURCE_MIRROR_URL will only apply to 
tarballs fetched by wget, not to VCS URLs.


Shane Dixon wrote:
> Does SRC_TARBALL_STASH replace the PREMIRRORS variable?  I used to use a
> variable in my local.conf that was similar to the following:
>
> PREMIRRORS = "(ftp|https?)$://.*/.*/ http://server.com/openembedded/"
>
> Is SRC_TARBALL_STASH the better way to do this?
>
> --
> Shane
>
>
> On Mon, 2009-03-16 at 22:04 -0700, Tom Rini wrote:
>   
>> On Tue, Mar 17, 2009 at 05:33:14PM +1300, Douglas Royds wrote:
>>
>>     
>>> Working behind a firewall that doesn't allow Git access (except over  
>>> http). I've set SRC_TARBALL_STASH so that we only have to download the  
>>> appropriate source tarball (manually) once.
>>>
>>> While building mtd-utils, BitBake is trying to fetch from the tarball stash:
>>>
>>>    http://.../git_git.infradead.org.mtd-utils.git.tar.gz
>>>
>>> Instead of:
>>>
>>>    http://.../git_git.infradead.org.mtd-utils.git_ea429635388f7bb53f62c41ec3d5ccf5fa207370.tar.gz
>>>
>>> The following patch appears to do the job, but I'm not confident this is  
>>> the right fix. I notice that all the other fetchers use ud.localfile,  
>>> which already has the correct filename.
>>>       
>> What's going on is that it's trying to fetch the raw git tree stash,
>> rather than the checkout stash, so you need to mirror the other.  OTOH,
>> perhaps it should try first the checkout then the raw?
>>
>>     


=======================================================================
This email, including any attachments, is only for the intended
addressee.  It is subject to copyright, is confidential and may be
the subject of legal or other privilege, none of which is waived or
lost by reason of this transmission.
If the receiver is not the intended addressee, please accept our
apologies, notify us by return, delete all copies and perform no
other act on the email.
Unfortunately, we cannot warrant that the email has not been
 altered or corrupted during transmission.
=======================================================================

Re: Git SRC_TARBALL_STASH filename

[Douglas Royds]

Tom Rini wrote:
> On Tue, Mar 17, 2009 at 05:33:14PM +1300, Douglas Royds wrote:
>
>   
>> Working behind a firewall that doesn't allow Git access (except over  
>> http). I've set SRC_TARBALL_STASH so that we only have to download the  
>> appropriate source tarball (manually) once.
>>
>> While building mtd-utils, BitBake is trying to fetch from the tarball stash:
>>
>>    http://.../git_git.infradead.org.mtd-utils.git.tar.gz
>>
>> Instead of:
>>
>>    http://.../git_git.infradead.org.mtd-utils.git_ea429635388f7bb53f62c41ec3d5ccf5fa207370.tar.gz
>>
>> The following patch appears to do the job, but I'm not confident this is  
>> the right fix. I notice that all the other fetchers use ud.localfile,  
>> which already has the correct filename.
>>     
>
> What's going on is that it's trying to fetch the raw git tree stash,
> rather than the checkout stash, so you need to mirror the other.  OTOH,
> perhaps it should try first the checkout then the raw?
>   

I don't have the raw git tree stash (firewall, remember)? Regrettably, 
mtd-utils don't appear to provide HTTP access to their repository - only 
git and ssh - so I don't appear to have any way of getting it.

The Subversion fetcher appears to happily pick up the checkout first. If 
someone has the enthusiasm to make that change for the git fetcher (as 
you suggest), that'd be great. I'm not going to get to it for some time.

Thanks,
Douglas.





=======================================================================
This email, including any attachments, is only for the intended
addressee.  It is subject to copyright, is confidential and may be
the subject of legal or other privilege, none of which is waived or
lost by reason of this transmission.
If the receiver is not the intended addressee, please accept our
apologies, notify us by return, delete all copies and perform no
other act on the email.
Unfortunately, we cannot warrant that the email has not been
 altered or corrupted during transmission.
=======================================================================

Re: Git SRC_TARBALL_STASH filename

[Tom Rini]

On Wed, Mar 18, 2009 at 03:11:15PM +1300, Douglas Royds wrote:
> Tom Rini wrote:
>> On Tue, Mar 17, 2009 at 05:33:14PM +1300, Douglas Royds wrote:
>>
>>   
>>> Working behind a firewall that doesn't allow Git access (except over  
>>> http). I've set SRC_TARBALL_STASH so that we only have to download 
>>> the  appropriate source tarball (manually) once.
>>>
>>> While building mtd-utils, BitBake is trying to fetch from the tarball stash:
>>>
>>>    http://.../git_git.infradead.org.mtd-utils.git.tar.gz
>>>
>>> Instead of:
>>>
>>>    http://.../git_git.infradead.org.mtd-utils.git_ea429635388f7bb53f62c41ec3d5ccf5fa207370.tar.gz
>>>
>>> The following patch appears to do the job, but I'm not confident this 
>>> is  the right fix. I notice that all the other fetchers use 
>>> ud.localfile,  which already has the correct filename.
>>>     
>>
>> What's going on is that it's trying to fetch the raw git tree stash,
>> rather than the checkout stash, so you need to mirror the other.  OTOH,
>> perhaps it should try first the checkout then the raw?
>>   
>
> I don't have the raw git tree stash (firewall, remember)? Regrettably,  
> mtd-utils don't appear to provide HTTP access to their repository - only  
> git and ssh - so I don't appear to have any way of getting it.

Right.  What I believe happens is that PREMIRRORS (etc) will work for
git_..GITHASH and SRC_TARBALL_STASH will work for git_...git.tar.gz as
(I believe) mirrors are, even for svn/cvs, the right version but
SRC_TARBALL_STASH will then make the checked-out tarball (ie populate
${DL_DIR}/cvs/pkg../ then make ${DL_DIR}/pkg...tar.gz).  I think :)

-- 
Tom Rini

Re: Git SRC_TARBALL_STASH filename

[Douglas Royds]

Douglas Royds wrote:
> Tom Rini wrote:
>> On Tue, Mar 17, 2009 at 05:33:14PM +1300, Douglas Royds wrote:
>>> Working behind a firewall that doesn't allow Git access (except 
>>> over  http). I've set SRC_TARBALL_STASH so that we only have to 
>>> download the  appropriate source tarball (manually) once.
>>>
>>> While building mtd-utils, BitBake is trying to fetch from the 
>>> tarball stash:
>>>
>>>    http://.../git_git.infradead.org.mtd-utils.git.tar.gz
>>
>> What's going on is that it's trying to fetch the raw git tree stash,
>> rather than the checkout stash, so you need to mirror the other. 
>
> I don't have the raw git tree stash (firewall, remember)? Regrettably, 
> mtd-utils don't appear to provide HTTP access to their repository - 
> only git and ssh - so I don't appear to have any way of getting it.

For the record, I was able to download the mtd-utils Git stash tarball 
from the Angstrom mirror:

    http://www.angstrom-distribution.org/unstable/sources/git_git.infradead.org.mtd-utils.git.tar.gz

Now I can set SRC_TARBALL_STASH, and people (here behind the firewall) 
can access the appropriate git revision.

Douglas.




=======================================================================
This email, including any attachments, is only for the intended
addressee.  It is subject to copyright, is confidential and may be
the subject of legal or other privilege, none of which is waived or
lost by reason of this transmission.
If the receiver is not the intended addressee, please accept our
apologies, notify us by return, delete all copies and perform no
other act on the email.
Unfortunately, we cannot warrant that the email has not been
 altered or corrupted during transmission.
=======================================================================