something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Robert P. J. Day]

  for a yocto talk to my local LUG later this week, i want to build a
simple core-image-sato image for qemux86 and, for quite some time,
i've kept a local download directory for tarballs, and taken advantage
of it with this in all of my local.conf files:

SOURCE_MIRROR_URL ?= "file:///home/rpjday/dl/"
INHERIT += "own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"
# BB_NO_NETWORK = "1"

  obviously, as i notice newer tarballs getting downloaded, i'll just
copy them into ~/dl so it doesn't happen again.  works fine, except
today with gtk+-2.24.8.tar.bz2 which, even after i made a copy of it
in ~/dl, the build still *insists* on downloading it from
download.gnome.org.

  i don't see any obvious problem -- the checksum of the tarball
appears correct, it's a regular file, the tarball downloaded is
*exactly* the same as the one i've saved in ~/dl and yet, of all the
tarballs i have stashed away there, that one is the *only* one that is
downloaded each time.

  am i missing something obvious?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Robert P. J. Day]

On Sun, 3 Jun 2012, Robert P. J. Day wrote:

>   for a yocto talk to my local LUG later this week, i want to build
> a simple core-image-sato image for qemux86 and, for quite some time,
> i've kept a local download directory for tarballs, and taken
> advantage of it with this in all of my local.conf files:
>
> SOURCE_MIRROR_URL ?= "file:///home/rpjday/dl/"
> INHERIT += "own-mirrors"
> BB_GENERATE_MIRROR_TARBALLS = "1"
> # BB_NO_NETWORK = "1"
>
>   obviously, as i notice newer tarballs getting downloaded, i'll
> just copy them into ~/dl so it doesn't happen again.  works fine,
> except today with gtk+-2.24.8.tar.bz2 which, even after i made a
> copy of it in ~/dl, the build still *insists* on downloading it from
> download.gnome.org.
>
>   i don't see any obvious problem -- the checksum of the tarball
> appears correct, it's a regular file, the tarball downloaded is
> *exactly* the same as the one i've saved in ~/dl and yet, of all the
> tarballs i have stashed away there, that one is the *only* one that
> is downloaded each time.

  ok, i'm baffled and getting thoroughly annoyed here.  i started from
absolute scratch, configured to build for qemux86, then did

  $ bitbake -c fetchall core-image-sato

while pointing at my massive directory of tarballs which includes
gtk+-2.24.8.tar.bz2.  every required tarball is accessed (via symlink)
from my ~/dl directory, *except* for that incredibly irritating
gtk+-2.24.8.tar.bz2 tarball, which is downloaded by the "fetchall"
command *every* *single* *time*.

  i've verified that tarball is there.  i've verified its md5 and
sha256 checksums.  i'm out of ideas.

  what am i doing wrong?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Wang, Shane]

Yeah, Robert,

You found a bug in bitbake.
In the fetch code, local directory /home/rpjday/dl/gtk+-2.24.8.tar.bz2 is decoded as /home/rpjday/dl/gtk%2B-2.24.8.tar.bz2 by
newuri = uri_replace(origud, find, replace, ld), which is line 480 in lib/bb/fetch2/__init__.py.
And later on it is checked whether or not existed by
if not os.path.exists(newpath) and path.find("*") == -1:, which is line 61 in lib/bb/fetch2/local.py.

Actually the directory /home/rpjday/dl/gtk+-2.24.8.tar.bz2 is there, but /home/rpjday/dl/gtk%2B-2.24.8.tar.bz2 can't be found by bitbake code. That is the problem. Therefore, Gtk+ shouldn't be decoded as gtk%2B.

I am not familiar with the whole fetch code and hope someone else can correct it, so can you please file a bug first?

--
Shane

Robert P. J. Day wrote on 2012-06-04:

> On Sun, 3 Jun 2012, Robert P. J. Day wrote:
> 
>>   for a yocto talk to my local LUG later this week, i want to build
>> a simple core-image-sato image for qemux86 and, for quite some time,
>> i've kept a local download directory for tarballs, and taken
>> advantage of it with this in all of my local.conf files:
>> 
>> SOURCE_MIRROR_URL ?= "file:///home/rpjday/dl/"
>> INHERIT += "own-mirrors"
>> BB_GENERATE_MIRROR_TARBALLS = "1"
>> # BB_NO_NETWORK = "1"
>> 
>>   obviously, as i notice newer tarballs getting downloaded, i'll
>> just copy them into ~/dl so it doesn't happen again.  works fine,
>> except today with gtk+-2.24.8.tar.bz2 which, even after i made a
>> copy of it in ~/dl, the build still *insists* on downloading it from
>> download.gnome.org.
>> 
>>   i don't see any obvious problem -- the checksum of the tarball
>> appears correct, it's a regular file, the tarball downloaded is
>> *exactly* the same as the one i've saved in ~/dl and yet, of all the
>> tarballs i have stashed away there, that one is the *only* one that
>> is downloaded each time.
> 
>   ok, i'm baffled and getting thoroughly annoyed here.  i started from
> absolute scratch, configured to build for qemux86, then did
> 
>   $ bitbake -c fetchall core-image-sato
> while pointing at my massive directory of tarballs which includes
> gtk+-2.24.8.tar.bz2.  every required tarball is accessed (via symlink)
> from my ~/dl directory, *except* for that incredibly irritating
> gtk+-2.24.8.tar.bz2 tarball, which is downloaded by the "fetchall"
> command *every* *single* *time*.
> 
>   i've verified that tarball is there.  i've verified its md5 and
> sha256 checksums.  i'm out of ideas.
> 
>   what am i doing wrong?
> rday
>

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Khem Raj]

On Mon, Jun 4, 2012 at 10:04 PM, Wang, Shane <shane.wang@intel.com> wrote:
> Yeah, Robert,
>
> You found a bug in bitbake.
> In the fetch code, local directory /home/rpjday/dl/gtk+-2.24.8.tar.bz2 is decoded as /home/rpjday/dl/gtk%2B-2.24.8.tar.bz2 by
> newuri = uri_replace(origud, find, replace, ld), which is line 480 in lib/bb/fetch2/__init__.py.
> And later on it is checked whether or not existed by
> if not os.path.exists(newpath) and path.find("*") == -1:, which is line 61 in lib/bb/fetch2/local.py.
>
> Actually the directory /home/rpjday/dl/gtk+-2.24.8.tar.bz2 is there, but /home/rpjday/dl/gtk%2B-2.24.8.tar.bz2 can't be found by bitbake code. That is the problem. Therefore, Gtk+ shouldn't be decoded as gtk%2B.
>
> I am not familiar with the whole fetch code and hope someone else can correct it, so can you please file a bug first?

its probably due to use of urllib.unquote in decodeurl and not keeping
the unquoted name for subsequent use
>
> --
> Shane
>
> Robert P. J. Day wrote on 2012-06-04:
>
>> On Sun, 3 Jun 2012, Robert P. J. Day wrote:
>>
>>>   for a yocto talk to my local LUG later this week, i want to build
>>> a simple core-image-sato image for qemux86 and, for quite some time,
>>> i've kept a local download directory for tarballs, and taken
>>> advantage of it with this in all of my local.conf files:
>>>
>>> SOURCE_MIRROR_URL ?= "file:///home/rpjday/dl/"
>>> INHERIT += "own-mirrors"
>>> BB_GENERATE_MIRROR_TARBALLS = "1"
>>> # BB_NO_NETWORK = "1"
>>>
>>>   obviously, as i notice newer tarballs getting downloaded, i'll
>>> just copy them into ~/dl so it doesn't happen again.  works fine,
>>> except today with gtk+-2.24.8.tar.bz2 which, even after i made a
>>> copy of it in ~/dl, the build still *insists* on downloading it from
>>> download.gnome.org.
>>>
>>>   i don't see any obvious problem -- the checksum of the tarball
>>> appears correct, it's a regular file, the tarball downloaded is
>>> *exactly* the same as the one i've saved in ~/dl and yet, of all the
>>> tarballs i have stashed away there, that one is the *only* one that
>>> is downloaded each time.
>>
>>   ok, i'm baffled and getting thoroughly annoyed here.  i started from
>> absolute scratch, configured to build for qemux86, then did
>>
>>   $ bitbake -c fetchall core-image-sato
>> while pointing at my massive directory of tarballs which includes
>> gtk+-2.24.8.tar.bz2.  every required tarball is accessed (via symlink)
>> from my ~/dl directory, *except* for that incredibly irritating
>> gtk+-2.24.8.tar.bz2 tarball, which is downloaded by the "fetchall"
>> command *every* *single* *time*.
>>
>>   i've verified that tarball is there.  i've verified its md5 and
>> sha256 checksums.  i'm out of ideas.
>>
>>   what am i doing wrong?
>> rday
>>
>
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Wang, Shane]

Khem Raj wrote on 2012-06-05:

> On Mon, Jun 4, 2012 at 10:04 PM, Wang, Shane <shane.wang@intel.com>
> wrote:
>> Yeah, Robert,
>> 
>> You found a bug in bitbake. In the fetch code, local directory
>> /home/rpjday/dl/gtk+-2.24.8.tar.bz2 is decoded as
>> /home/rpjday/dl/gtk%2B-2.24.8.tar.bz2 by newuri = uri_replace(origud,
>> find, replace, ld), which is line 480 in lib/bb/fetch2/__init__.py. And
>> later on it is checked whether or not existed by if not
>> os.path.exists(newpath) and path.find("*") == -1:, which is line 61 in
>> lib/bb/fetch2/local.py.
>> 
>> Actually the directory /home/rpjday/dl/gtk+-2.24.8.tar.bz2 is there, but
> /home/rpjday/dl/gtk%2B-2.24.8.tar.bz2 can't be found by bitbake code. That
> is the problem. Therefore, Gtk+ shouldn't be decoded as gtk%2B.
>> 
>> I am not familiar with the whole fetch code and hope someone else can
> correct it, so can you please file a bug first?
> 
> its probably due to use of urllib.unquote in decodeurl and not keeping
> the unquoted name for subsequent use
>> 
>> --
>> Shane

Correct.

--
Shane

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Robert P. J. Day]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1979 bytes --]

On Tue, 5 Jun 2012, Wang, Shane wrote:

>
> Khem Raj wrote on 2012-06-05:
>
> > On Mon, Jun 4, 2012 at 10:04 PM, Wang, Shane <shane.wang@intel.com>
> > wrote:
> >> Yeah, Robert,
> >>
> >> You found a bug in bitbake. In the fetch code, local directory
> >> /home/rpjday/dl/gtk+-2.24.8.tar.bz2 is decoded as
> >> /home/rpjday/dl/gtk%2B-2.24.8.tar.bz2 by newuri = uri_replace(origud,
> >> find, replace, ld), which is line 480 in lib/bb/fetch2/__init__.py. And
> >> later on it is checked whether or not existed by if not
> >> os.path.exists(newpath) and path.find("*") == -1:, which is line 61 in
> >> lib/bb/fetch2/local.py.
> >>
> >> Actually the directory /home/rpjday/dl/gtk+-2.24.8.tar.bz2 is there, but
> > /home/rpjday/dl/gtk%2B-2.24.8.tar.bz2 can't be found by bitbake code. That
> > is the problem. Therefore, Gtk+ shouldn't be decoded as gtk%2B.
> >>
> >> I am not familiar with the whole fetch code and hope someone else can
> > correct it, so can you please file a bug first?
> >
> > its probably due to use of urllib.unquote in decodeurl and not keeping
> > the unquoted name for subsequent use
> >>
> >> --
> >> Shane
>
> Correct.

  at this point, has the problem been identified?  or does this still
require a bugzilla submission?  it would appear that the issue
involves any package with a "+" in the package name which, for now,
appears to be just that gtk+ package.  would any other special
characters trigger this?  that's the only package i've tripped over
that caused this behaviour.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Wang, Shane]

Robert P. J. Day wrote on 2012-06-05:
>   at this point, has the problem been identified?  or does this still
> require a bugzilla submission?  it would appear that the issue
> involves any package with a "+" in the package name which, for now,
> appears to be just that gtk+ package.  would any other special
> characters trigger this?  that's the only package i've tripped over
> that caused this behaviour.
> 
> rday
>

Yes, it has been identified. If you can't submit a bug, I can do.
Yes, I believe other special characters will trigger that.

Just because I am not familiar with all fetch code, I can fix it for gtk+ now but I am not sure whether that fix will bring any regression bugs.
Give me more time.

--
Shane

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Robert P. J. Day]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1327 bytes --]

On Tue, 5 Jun 2012, Wang, Shane wrote:

> Robert P. J. Day wrote on 2012-06-05:

> >   at this point, has the problem been identified?  or does this still
> > require a bugzilla submission?  it would appear that the issue
> > involves any package with a "+" in the package name which, for now,
> > appears to be just that gtk+ package.  would any other special
> > characters trigger this?  that's the only package i've tripped over
> > that caused this behaviour.
> >
> > rday
> >
>
> Yes, it has been identified. If you can't submit a bug, I can do.
> Yes, I believe other special characters will trigger that.
>
> Just because I am not familiar with all fetch code, I can fix it for
> gtk+ now but I am not sure whether that fix will bring any
> regression bugs. Give me more time.

  ok, i'll leave it in your capable hands and go find something else
to nitpick.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Gary Thomas]

On 2012-06-05 09:01, Robert P. J. Day wrote:
> On Tue, 5 Jun 2012, Wang, Shane wrote:
>
>> Robert P. J. Day wrote on 2012-06-05:
>
>>>    at this point, has the problem been identified?  or does this still
>>> require a bugzilla submission?  it would appear that the issue
>>> involves any package with a "+" in the package name which, for now,
>>> appears to be just that gtk+ package.  would any other special
>>> characters trigger this?  that's the only package i've tripped over
>>> that caused this behaviour.
>>>
>>> rday
>>>
>>
>> Yes, it has been identified. If you can't submit a bug, I can do.
>> Yes, I believe other special characters will trigger that.
>>
>> Just because I am not familiar with all fetch code, I can fix it for
>> gtk+ now but I am not sure whether that fix will bring any
>> regression bugs. Give me more time.
>
>    ok, i'll leave it in your capable hands and go find something else
> to nitpick.

Has someone filed a bug on this yet?  It breaks all my builds now since
I always use BB_NO_NETWORK="1" :-(

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: something about gtk+-2.24.8.tar.bz2 that requires constant downloading?

[Paul Eggleton]

On Wednesday 06 June 2012 09:10:34 Gary Thomas wrote:
> On 2012-06-05 09:01, Robert P. J. Day wrote:
> > On Tue, 5 Jun 2012, Wang, Shane wrote:
> >> Robert P. J. Day wrote on 2012-06-05:
> >>>    at this point, has the problem been identified?  or does this still
> >>> 
> >>> require a bugzilla submission?  it would appear that the issue
> >>> involves any package with a "+" in the package name which, for now,
> >>> appears to be just that gtk+ package.  would any other special
> >>> characters trigger this?  that's the only package i've tripped over
> >>> that caused this behaviour.
> >>> 
> >>> rday
> >> 
> >> Yes, it has been identified. If you can't submit a bug, I can do.
> >> Yes, I believe other special characters will trigger that.
> >> 
> >> Just because I am not familiar with all fetch code, I can fix it for
> >> gtk+ now but I am not sure whether that fix will bring any
> >> regression bugs. Give me more time.
> >> 
> >    ok, i'll leave it in your capable hands and go find something else
> > 
> > to nitpick.
> 
> Has someone filed a bug on this yet?  It breaks all my builds now since
> I always use BB_NO_NETWORK="1" :-(

For reference, Gary has filed bug 2558 to track this.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre