How to use externalsrc with git workspace

How to use externalsrc with git workspace

[Cliff Brake]

Hi,

In the past (say Dora era), I've used externalsrc like:

====================
DESCRIPTION = "MyApp"
LICENSE = "CLOSED"

inherit externalsrc

SRCREV = "${AUTOREV}"
PV = "1.4+gitr${SRCPV}"
PR = "r1"

SRC_URI = "git://${FILE_DIRNAME};protocol=file;branch=HEAD"
S = "${FILE_DIRNAME}"

...
====================

This was very nice in that it set SRCPV to to the git version of the HEAD of my
local workspace.  When building from a local git workspace, this makes sense to
me -- I want the Git version of the workspace, not a remote version.

Now with Jethro, I get errors like:

ERROR: ExpansionError during parsing myapp.bb: Failure expanding variable SRCPV,
expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Unable to resolve 'HEAD' in upstream git repository in git
ls-remote output for /scratch/externalsrc/MyApp

Do you have any suggestions how to use externalsrc + a local git workspace and
avoid the above error.  I can set branch=master, etc and bitbake will then parse
it, but the remote master version is not really what I'm building.

Thanks,
Cliff

Re: How to use externalsrc with git workspace

[Cliff Brake]

This change gets things parsing again, but still testing to see if it does what
I want it to do:

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 9bd87ad..fb44fdc 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -358,7 +358,7 @@ class Git(FetchMethod):
         """
         output = self._lsremote(ud, d, "")
         # Tags of the form ^{} may not work, need to fallback to other form
-        if ud.unresolvedrev[name][:5] == "refs/":
+        if ud.unresolvedrev[name][:5] == "refs/" or ud.unresolvedrev[name] ==
"HEAD":
             head = ud.unresolvedrev[name]
             tag = ud.unresolvedrev[name]
         else:

Cliff

On 03/25/2016 04:18 PM, Cliff Brake wrote:
> Hi,
>
> In the past (say Dora era), I've used externalsrc like:
>
> ====================
> DESCRIPTION = "MyApp"
> LICENSE = "CLOSED"
>
> inherit externalsrc
>
> SRCREV = "${AUTOREV}"
> PV = "1.4+gitr${SRCPV}"
> PR = "r1"
>
> SRC_URI = "git://${FILE_DIRNAME};protocol=file;branch=HEAD"
> S = "${FILE_DIRNAME}"
>
> ...
> ====================
>
> This was very nice in that it set SRCPV to to the git version of the HEAD of my
> local workspace.  When building from a local git workspace, this makes sense to
> me -- I want the Git version of the workspace, not a remote version.
>
> Now with Jethro, I get errors like:
>
> ERROR: ExpansionError during parsing myapp.bb: Failure expanding variable SRCPV,
> expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
> Fetcher failure: Unable to resolve 'HEAD' in upstream git repository in git
> ls-remote output for /scratch/externalsrc/MyApp
>
> Do you have any suggestions how to use externalsrc + a local git workspace and
> avoid the above error.  I can set branch=master, etc and bitbake will then parse
> it, but the remote master version is not really what I'm building.
>
> Thanks,
> Cliff
>

Re: How to use externalsrc with git workspace

[Richard Purdie]

On Fri, 2016-03-25 at 16:18 -0400, Cliff Brake wrote:
> Hi,
> 
> In the past (say Dora era), I've used externalsrc like:
> 
> ====================
> DESCRIPTION = "MyApp"
> LICENSE = "CLOSED"
> 
> inherit externalsrc
> 
> SRCREV = "${AUTOREV}"
> PV = "1.4+gitr${SRCPV}"
> PR = "r1"
> 
> SRC_URI = "git://${FILE_DIRNAME};protocol=file;branch=HEAD"
> S = "${FILE_DIRNAME}"
> 
> ...
> ====================
> 
> This was very nice in that it set SRCPV to to the git version of the
> HEAD of my
> local workspace.  When building from a local git workspace, this
> makes sense to
> me -- I want the Git version of the workspace, not a remote version.
> 
> Now with Jethro, I get errors like:
> 
> ERROR: ExpansionError during parsing myapp.bb: Failure expanding
> variable SRCPV,
> expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
> FetchError:
> Fetcher failure: Unable to resolve 'HEAD' in upstream git repository
> in git
> ls-remote output for /scratch/externalsrc/MyApp
> 
> Do you have any suggestions how to use externalsrc + a local git
> workspace and
> avoid the above error.  I can set branch=master, etc and bitbake will
> then parse
> it, but the remote master version is not really what I'm building.

HEAD is a revision, not a branch and the fetcher became a bit more
picky about that difference. Your local tree must have HEAD on some
branch which would usually be master so setting branch=master is likely
correct, or don't set that parameter at all? master isn't just a remote
thing.

The piece which says "use the HEAD revision" is the SRCREV =
"${AUTOREV}" line.

Cheers,

Richard

Re: How to use externalsrc with git workspace

[Cliff Brake]

On 03/26/2016 03:41 AM, Richard Purdie wrote:
> HEAD is a revision, not a branch and the fetcher became a bit more
> picky about that difference. Your local tree must have HEAD on some
> branch which would usually be master so setting branch=master is likely
> correct, or don't set that parameter at all? master isn't just a remote
> thing.
>
> The piece which says "use the HEAD revision" is the SRCREV =
> "${AUTOREV}" line.
>
> Cheers,
>
> Richard
>
We typically set up our projects to use git submodules, such that the top level
calls out a set of submodules which are the actively developed applications and
other bits in the system.  These submodules are externalsrc trees for bitbake,
and include a bitbake recipe (bitbake recipe is embedded directly in source
code).  Sometimes these submodules/externalsrc trees track branches (typically
release-xyz, or develop-xyz), but almost never master.  Sometimes they may be at
a tagged version that is not at the HEAD of a branch.  In the end, I want to be
able to build exactly what is in the externalsrc directory, and accurately
include the git version in the PV.  Branches/tags may change at any time and I
don't want to have to edit the recipe every time this happens.

Repo might be used in a similar fashion to manage a tree of components that get
built.  What we are trying to avoid is having to edit the git version in the
recipes to lock down versions for a set of components for a build -- git
submodules or repo work much better for doing this.

I tried nobranch=1;rev=HEAD, but then that conflicts with ${AUTOREV}.

The git fetcher could have an option like head=1 in the SRC_URI, that told it to
use whatever is in the git workspace, and allow AUTOREV to figure out version. 

Perhaps a better approach might be to figure out a function for PV that simply
extracts the git version of ${S} -- that is really all I want.  In this case, I
might even need a SRC_URI, as I'm not really fetching anything -- its already there.

Thanks,
Cliff

Re: How to use externalsrc with git workspace

[Richard Purdie]

On Mon, 2016-03-28 at 15:58 -0400, Cliff Brake wrote:
> On 03/26/2016 03:41 AM, Richard Purdie wrote:
> > HEAD is a revision, not a branch and the fetcher became a bit more
> > picky about that difference. Your local tree must have HEAD on some
> > branch which would usually be master so setting branch=master is
> > likely
> > correct, or don't set that parameter at all? master isn't just a
> > remote
> > thing.
> > 
> > The piece which says "use the HEAD revision" is the SRCREV =
> > "${AUTOREV}" line.
> > 
> > Cheers,
> > 
> > Richard
> > 
> We typically set up our projects to use git submodules, such that the
> top level
> calls out a set of submodules which are the actively developed
> applications and
> other bits in the system.  These submodules are externalsrc trees for
> bitbake,
> and include a bitbake recipe (bitbake recipe is embedded directly in
> source
> code).  Sometimes these submodules/externalsrc trees track branches
> (typically
> release-xyz, or develop-xyz), but almost never master.  Sometimes
> they may be at
> a tagged version that is not at the HEAD of a branch.  In the end, I
> want to be
> able to build exactly what is in the externalsrc directory, and
> accurately
> include the git version in the PV.  Branches/tags may change at any
> time and I
> don't want to have to edit the recipe every time this happens.
> 
> Repo might be used in a similar fashion to manage a tree of
> components that get
> built.  What we are trying to avoid is having to edit the git version
> in the
> recipes to lock down versions for a set of components for a build --
> git
> submodules or repo work much better for doing this.
> 
> I tried nobranch=1;rev=HEAD, but then that conflicts with ${AUTOREV}.

Did you just try nobranch=1 and leave rev to get set from SRCREV?

Cheers,

Richard

Re: How to use externalsrc with git workspace

[Cliff Brake]

On 03/28/2016 06:28 PM, Richard Purdie wrote:
>> We typically set up our projects to use git submodules, such that the
>> top level
>> calls out a set of submodules which are the actively developed
>> applications and
>> other bits in the system.  These submodules are externalsrc trees for
>> bitbake,
>> and include a bitbake recipe (bitbake recipe is embedded directly in
>> source
>> code).  Sometimes these submodules/externalsrc trees track branches
>> (typically
>> release-xyz, or develop-xyz), but almost never master.  Sometimes
>> they may be at
>> a tagged version that is not at the HEAD of a branch.  In the end, I
>> want to be
>> able to build exactly what is in the externalsrc directory, and
>> accurately
>> include the git version in the PV.  Branches/tags may change at any
>> time and I
>> don't want to have to edit the recipe every time this happens.
>>
>> Repo might be used in a similar fashion to manage a tree of
>> components that get
>> built.  What we are trying to avoid is having to edit the git version
>> in the
>> recipes to lock down versions for a set of components for a build --
>> git
>> submodules or repo work much better for doing this.
>>
>> I tried nobranch=1;rev=HEAD, but then that conflicts with ${AUTOREV}.
> Did you just try nobranch=1 and leave rev to get set from SRCREV?

That will parse, but it puts the wrong git rev in the PV:

The following package gets built:

blkqcl-app-tune_1.4+gitr1+a97fcb58e6-r1.0_cortexa9hf-vfp-neon.ipk

But, the actual version of the source that gets built is
0edb656fc9a1a77d75eaa49a17586deb4c986bb1.

a97... is the version of the branch (default is master) in the workspace.

So this would work if we did not care what gitv got put in the PV.  There is
also the issue of bitbake not rebuilding the source if you change the git
version of the workspace, but the branch happens to not change.

Does no one else use externalsrc with Git?

Thanks,
Cliff

Re: How to use externalsrc with git workspace

[Richard Purdie]

On Tue, 2016-03-29 at 08:25 -0400, Cliff Brake wrote:
> On 03/28/2016 06:28 PM, Richard Purdie wrote:
> > > I tried nobranch=1;rev=HEAD, but then that conflicts with
> > > ${AUTOREV}.
> > Did you just try nobranch=1 and leave rev to get set from SRCREV?
> 
> That will parse, but it puts the wrong git rev in the PV:
> 
> The following package gets built:
> 
> blkqcl-app-tune_1.4+gitr1+a97fcb58e6-r1.0_cortexa9hf-vfp-neon.ipk
> 
> But, the actual version of the source that gets built is
> 0edb656fc9a1a77d75eaa49a17586deb4c986bb1.
> 
> a97... is the version of the branch (default is master) in the
> workspace.
> 
> So this would work if we did not care what gitv got put in the PV. 
>  There is
> also the issue of bitbake not rebuilding the source if you change the
> git
> version of the workspace, but the branch happens to not change.

At least from my perspective, this is what *should* work and the fact
it doesn't is a bug somewhere. Adding extra things to the fetcher would
likely just workaround it. Filing a bug report for it would be the
likely next step.

Cheers,

Richard

Re: How to use externalsrc with git workspace

[Cliff Brake]

On 03/29/2016 08:28 AM, Richard Purdie wrote:
> On Tue, 2016-03-29 at 08:25 -0400, Cliff Brake wrote:
>> On 03/28/2016 06:28 PM, Richard Purdie wrote:
>>>> I tried nobranch=1;rev=HEAD, but then that conflicts with
>>>> ${AUTOREV}.
>>> Did you just try nobranch=1 and leave rev to get set from SRCREV?
>> That will parse, but it puts the wrong git rev in the PV:
>>
>> The following package gets built:
>>
>> blkqcl-app-tune_1.4+gitr1+a97fcb58e6-r1.0_cortexa9hf-vfp-neon.ipk
>>
>> But, the actual version of the source that gets built is
>> 0edb656fc9a1a77d75eaa49a17586deb4c986bb1.
>>
>> a97... is the version of the branch (default is master) in the
>> workspace.
>>
>> So this would work if we did not care what gitv got put in the PV. 
>>  There is
>> also the issue of bitbake not rebuilding the source if you change the
>> git
>> version of the workspace, but the branch happens to not change.
> At least from my perspective, this is what *should* work and the fact
> it doesn't is a bug somewhere. Adding extra things to the fetcher would
> likely just workaround it. Filing a bug report for it would be the
> likely next step.

Thanks, bug submitted: https://bugzilla.yoctoproject.org/show_bug.cgi?id=9351

Cliff