hg fetcher failures for SRCREV set to a tag

hg fetcher failures for SRCREV set to a tag

[Jon Szymaniak]

tl;dr: Doesn't the "update" in lines 142-147 of hg.py cause issues when
SRCREV is a tag?


Hi all,

I didn't have any luck with this post on the Yocto mailing list. Since it seems
more like a bitbake issue (I'm very hesitant to cry "bug" at this point),
I hope I'm in the right place.

I've been running into "Fetcher failures" for my recipes using
mercurial recently, which report that I'm asking for an unknown revision,
despite it being valid. A trimmed log is at the end of this post.

After looking at hg.py, it seems the issue is that the
'hg update -r MY_TAG' that follows the 'hg clone -r MY_TAG ...' is doomed to
fail because that tag isn't visible until the next rev, where the action of
tagging it is committed.

For example, if I clone v1.1.0, I won't see that tag because the tag is added
to .hgtags in commit 41:58b..

      tip                             41:58b8f368be68
      v1.1.0                        40:a925b596c163
      v1.0.4                        39:6bf84da2571e

Therefore once I've cloned with "-r v1.1.0", I can't do an update
specifying tag v1.1.0.

Does this sound right, or am I way off here?

I'm not sure if using a tag rather than the revision number is unorthodox or
abnormal for bb's, so below's my use case, for a my-app_0.1.2.bb

require my-app.inc
PR = "${INC_PR}.0"
SRCREV = "v${PV}"

Therefore to bring in a new version, I can simply copy this file to a
my-app_1.1.0.bb, and adjust PR back to 0 if needed.

We have a policy not to move version tags, so tag-->revision should be 1:1. Is
what I'm seeing a non-issue simply because one shouldn't use tags in SRCREV?


Thank you,
Jon


Snipped Yocto/Poky log:
---------------------------
DEBUG: Fetch: checking for module directory
'/export/home/jon/projects/yocto/poky/build/test/downloads/hg/host/some_dir/my_package'
NOTE: Fetch hg://hg@host/some_dir;module=my_package;protocol=ssh
DEBUG: Running /usr/bin/env hg clone -r v1.1.0
ssh://hg@host/some_dir/my_package my_package
DEBUG: Fetcher accessed the network with the command /usr/bin/env hg
clone -r v1.1.0 ssh://hg@host/some_dir/my_package my_package
DEBUG: Running export HOME="/export/home/jon"; export
SSH_AGENT_PID="1234"; export SSH_AUTH_SOCK="<snipped>"; export
GIT_CONFIG="<snipped>/gitconfig"; export PATH="<snipped>";
/usr/bin/env hg clone -r v1.1.0 ssh://hg@host/some_dir/my_package
my_package
DEBUG: Running /usr/bin/env hg update -C -r v1.1.0
DEBUG: Running export HOME="/export/home/jon"; export
SSH_AGENT_PID="1234"; export SSH_AUTH_SOCK="<snipped>"; export
GIT_CONFIG="<snipped>/gitconfig"; export PATH="<snipped>";
/usr/bin/env hg update -C -r v1.1.0
WARNING: Failed to fetch URL
hg://hg@host/some_dir;module=my_package;protocol=ssh, attempting
MIRRORS if available
DEBUG: Fetcher failure: Fetch command failed with exit code 255, output:
  abort: unknown revision 'v1.1.0'!

DEBUG: Python function base_do_fetch finished
DEBUG: Python function do_fetch finished
ERROR: Function failed: Fetcher failure for URL:
'hg://hg@host/some_dir;module=my_package;protocol=ssh'. Unable to
fetch URL from any source.

Re: hg fetcher failures for SRCREV set to a tag

[Richard Purdie]

On Wed, 2012-12-05 at 19:17 -0500, Jon Szymaniak wrote:
> tl;dr: Doesn't the "update" in lines 142-147 of hg.py cause issues when
> SRCREV is a tag?
> 
> 
> Hi all,
> 
> I didn't have any luck with this post on the Yocto mailing list. Since it seems
> more like a bitbake issue (I'm very hesitant to cry "bug" at this point),
> I hope I'm in the right place.
> 
> I've been running into "Fetcher failures" for my recipes using
> mercurial recently, which report that I'm asking for an unknown revision,
> despite it being valid. A trimmed log is at the end of this post.
> 
> After looking at hg.py, it seems the issue is that the
> 'hg update -r MY_TAG' that follows the 'hg clone -r MY_TAG ...' is doomed to
> fail because that tag isn't visible until the next rev, where the action of
> tagging it is committed.
> 
> For example, if I clone v1.1.0, I won't see that tag because the tag is added
> to .hgtags in commit 41:58b..
> 
>       tip                             41:58b8f368be68
>       v1.1.0                        40:a925b596c163
>       v1.0.4                        39:6bf84da2571e
> 
> Therefore once I've cloned with "-r v1.1.0", I can't do an update
> specifying tag v1.1.0.
> 
> Does this sound right, or am I way off here?
> 
> I'm not sure if using a tag rather than the revision number is unorthodox or
> abnormal for bb's, so below's my use case, for a my-app_0.1.2.bb
> 
> require my-app.inc
> PR = "${INC_PR}.0"
> SRCREV = "v${PV}"
> 
> Therefore to bring in a new version, I can simply copy this file to a
> my-app_1.1.0.bb, and adjust PR back to 0 if needed.
> 
> We have a policy not to move version tags, so tag-->revision should be 1:1. Is
> what I'm seeing a non-issue simply because one shouldn't use tags in SRCREV?
> 

FWIW, the hg fetcher is a little unloved. We don't have many developers
who use it or are familiar with how hg works. If there are some ways we
can fix things I'd be interested in patches.

I'm afraid I haven't had a chance to look at the details of the above
yet though.

Cheers,

Richard

Re: hg fetcher failures for SRCREV set to a tag

[Jon Szymaniak]

>> tl;dr: Doesn't the "update" in lines 142-147 of hg.py cause issues when
>> SRCREV is a tag?
>>
>>
>> Hi all,
>>
>> I didn't have any luck with this post on the Yocto mailing list. Since it seems
>> more like a bitbake issue (I'm very hesitant to cry "bug" at this point),
>> I hope I'm in the right place.
>>
>> I've been running into "Fetcher failures" for my recipes using
>> mercurial recently, which report that I'm asking for an unknown revision,
>> despite it being valid. A trimmed log is at the end of this post.
>>
>> After looking at hg.py, it seems the issue is that the
>> 'hg update -r MY_TAG' that follows the 'hg clone -r MY_TAG ...' is doomed to
>> fail because that tag isn't visible until the next rev, where the action of
>> tagging it is committed.
>>
>> For example, if I clone v1.1.0, I won't see that tag because the tag is added
>> to .hgtags in commit 41:58b..
>>
>>       tip                             41:58b8f368be68
>>       v1.1.0                        40:a925b596c163
>>       v1.0.4                        39:6bf84da2571e
>>
>> Therefore once I've cloned with "-r v1.1.0", I can't do an update
>> specifying tag v1.1.0.
>>
>> Does this sound right, or am I way off here?
>>
>> I'm not sure if using a tag rather than the revision number is unorthodox or
>> abnormal for bb's, so below's my use case, for a my-app_0.1.2.bb
>>
>> require my-app.inc
>> PR = "${INC_PR}.0"
>> SRCREV = "v${PV}"
>>
>> Therefore to bring in a new version, I can simply copy this file to a
>> my-app_1.1.0.bb, and adjust PR back to 0 if needed.
>>
>> We have a policy not to move version tags, so tag-->revision should be 1:1. Is
>> what I'm seeing a non-issue simply because one shouldn't use tags in SRCREV?
>>
>
> FWIW, the hg fetcher is a little unloved. We don't have many developers
> who use it or are familiar with how hg works. If there are some ways we
> can fix things I'd be interested in patches.
>
> I'm afraid I haven't had a chance to look at the details of the above
> yet though.
>
> Cheers,
>
> Richard
>

Hi Richard,

Thanks for pinging back on this. (Sorry I just double-sent this to
you. Forgot to include the list! :P)

I think a fairly simple solution might be to just clone the entire
repo by not adding "-r REV" to options[] for an 'hg clone' invocation.
The 'hg update' that follows on lines 142-147 should set the repo to
the desired revision.

I suppose an argument against this solution may be that this is
overkill since we only need revision <tag> + 1; the fetch could
potentially take longer and result in greater disk space usage, but I
feel like this is a reasonable tradeoff. Then again, I'm not sure if
folks are using the hg fetcher to clone huge repos...

Thoughts? Just wanted to bounce this idea around, as I certainly don't
consider myself an hg expert and am still getting up to speed on OE,
Yocto, & bitbake. If this sounds reasonable, I'll give it a shot and
post a little patch.

- Jon

Re: hg fetcher failures for SRCREV set to a tag

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 2762 bytes --]

On Fri, Dec 14, 2012 at 12:09 PM, Jon Szymaniak <jon.szymaniak@gmail.com>wrote:

> >> tl;dr: Doesn't the "update" in lines 142-147 of hg.py cause issues when
> >> SRCREV is a tag?
> >>
> >>
> >> Hi all,
> >>
> >> I didn't have any luck with this post on the Yocto mailing list. Since
> it seems
> >> more like a bitbake issue (I'm very hesitant to cry "bug" at this
> point),
> >> I hope I'm in the right place.
> >>
> >> I've been running into "Fetcher failures" for my recipes using
> >> mercurial recently, which report that I'm asking for an unknown
> revision,
> >> despite it being valid. A trimmed log is at the end of this post.
> >>
> >> After looking at hg.py, it seems the issue is that the
> >> 'hg update -r MY_TAG' that follows the 'hg clone -r MY_TAG ...' is
> doomed to
> >> fail because that tag isn't visible until the next rev, where the
> action of
> >> tagging it is committed.
> >>
> >> For example, if I clone v1.1.0, I won't see that tag because the tag is
> added
> >> to .hgtags in commit 41:58b..
> >>
> >>       tip                             41:58b8f368be68
> >>       v1.1.0                        40:a925b596c163
> >>       v1.0.4                        39:6bf84da2571e
> >>
> >> Therefore once I've cloned with "-r v1.1.0", I can't do an update
> >> specifying tag v1.1.0.
> >>
> >> Does this sound right, or am I way off here?
> >>
> >> I'm not sure if using a tag rather than the revision number is
> unorthodox or
> >> abnormal for bb's, so below's my use case, for a my-app_0.1.2.bb
> >>
> >> require my-app.inc
> >> PR = "${INC_PR}.0"
> >> SRCREV = "v${PV}"
> >>
> >> Therefore to bring in a new version, I can simply copy this file to a
> >> my-app_1.1.0.bb, and adjust PR back to 0 if needed.
> >>
> >> We have a policy not to move version tags, so tag-->revision should be
> 1:1. Is
> >> what I'm seeing a non-issue simply because one shouldn't use tags in
> SRCREV?
> >>
> >
> > FWIW, the hg fetcher is a little unloved. We don't have many developers
> > who use it or are familiar with how hg works. If there are some ways we
> > can fix things I'd be interested in patches.
> >
> > I'm afraid I haven't had a chance to look at the details of the above
> > yet though.
> >
> > Cheers,
> >
> > Richard
> >
>
> Hi Richard,
>
> Thanks for pinging back on this. (Sorry I just double-sent this to
> you. Forgot to include the list! :P)
>
> I think a fairly simple solution might be to just clone the entire
> repo by not adding "-r REV" to options[] for an 'hg clone' invocation.
> The 'hg update' that follows on lines 142-147 should set the repo to
> the desired revision.


This does sound like the best bet, to me.
-- 
Christopher Larson

[-- Attachment #2: Type: text/html, Size: 3844 bytes --]

Re: hg fetcher failures for SRCREV set to a tag

[Jon Szymaniak]

>> Hi Richard,
>>
>> Thanks for pinging back on this. (Sorry I just double-sent this to
>> you. Forgot to include the list! :P)
>>
>> I think a fairly simple solution might be to just clone the entire
>> repo by not adding "-r REV" to options[] for an 'hg clone' invocation.
>> The 'hg update' that follows on lines 142-147 should set the repo to
>> the desired revision.
>
>
> This does sound like the best bet, to me.
> --
> Christopher Larson

Hi all,

Just wanted to touch base on this issue. I didn't hear anything back
on the little patch I posted...not sure if that's a silent rejection,
if I didn't include the right info in the subject line, or if folks
haven't had a chance to test it. If there's something I screwed up,
please do let me know for future reference. :)

http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-December/003962.html


Thanks,
Jon

Re: hg fetcher failures for SRCREV set to a tag

[Richard Purdie]

On Sat, 2012-12-22 at 12:36 -0500, Jon Szymaniak wrote:
> >> Hi Richard,
> >>
> >> Thanks for pinging back on this. (Sorry I just double-sent this to
> >> you. Forgot to include the list! :P)
> >>
> >> I think a fairly simple solution might be to just clone the entire
> >> repo by not adding "-r REV" to options[] for an 'hg clone' invocation.
> >> The 'hg update' that follows on lines 142-147 should set the repo to
> >> the desired revision.
> >
> >
> > This does sound like the best bet, to me.
> > --
> > Christopher Larson
> 
> Hi all,
> 
> Just wanted to touch base on this issue. I didn't hear anything back
> on the little patch I posted...not sure if that's a silent rejection,
> if I didn't include the right info in the subject line, or if folks
> haven't had a chance to test it. If there's something I screwed up,
> please do let me know for future reference. :)
> 
> http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-December/003962.html

Sorry about the delay, vacation and lots of other distractions. The
patch was find and I've merged it now, thanks. I did have trouble
applying it as it appeared the whitespace was messed up. I fixed it up
manually this time though.

Cheers,

Richard

Re: hg fetcher failures for SRCREV set to a tag

[Jon Szymaniak]

On Fri, Jan 18, 2013 at 7:51 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Sat, 2012-12-22 at 12:36 -0500, Jon Szymaniak wrote:
>> >> Hi Richard,
>> >>
>> >> Thanks for pinging back on this. (Sorry I just double-sent this to
>> >> you. Forgot to include the list! :P)
>> >>
>> >> I think a fairly simple solution might be to just clone the entire
>> >> repo by not adding "-r REV" to options[] for an 'hg clone' invocation.
>> >> The 'hg update' that follows on lines 142-147 should set the repo to
>> >> the desired revision.
>> >
>> >
>> > This does sound like the best bet, to me.
>> > --
>> > Christopher Larson
>>
>> Hi all,
>>
>> Just wanted to touch base on this issue. I didn't hear anything back
>> on the little patch I posted...not sure if that's a silent rejection,
>> if I didn't include the right info in the subject line, or if folks
>> haven't had a chance to test it. If there's something I screwed up,
>> please do let me know for future reference. :)
>>
>> http://lists.linuxtogo.org/pipermail/bitbake-devel/2012-December/003962.html
>
> Sorry about the delay, vacation and lots of other distractions. The
> patch was find and I've merged it now, thanks. I did have trouble
> applying it as it appeared the whitespace was messed up. I fixed it up
> manually this time though.
>
> Cheers,
>
> Richard
>

Richard,

Thanks for pinging back -- I know it's certainty that busy time of year. :)

If you happen to have any details on those whitespace issues, mind
shooting that my way? I'd like to get a feel of where I screwed up
there. (e.g. email client, stuff in my .vimrc, etc)

Thanks,
Jon