[PATCH] metadata-revs: provide more information

[PATCH] metadata-revs: provide more information

[Trevor Woerner]

Provide many more details concerning the repositories that are used in a
particular build: the remote information, the layer, the local branch, the
remote branch the local branch tracks (if any), and the HEAD commit.

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
 meta/classes/buildhistory.bbclass |  6 +++++-
 meta/classes/metadata_scm.bbclass | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index b6b4324..1b0ae0e 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -615,9 +615,13 @@ def buildhistory_get_build_id(d):
 
 def buildhistory_get_metadata_revs(d):
     # We want an easily machine-readable format here, so get_layers_branch_rev isn't quite what we want
+    import subprocess
     layers = (d.getVar("BBLAYERS", True) or "").split()
-    medadata_revs = ["%-17s = %s:%s" % (os.path.relpath(i, d.getVar('BBLAYERS_FETCH_DIR', True)), \
+    medadata_revs = ["%s\tlayer:  %s\n\tbranch: %s\n\tremote: %s\n\tHEAD:   %s\n" % ( \
+        base_get_metadata_git_remote(i, None), \
+        os.path.relpath(i, d.getVar('BBLAYERS_FETCH_DIR', True)), \
         base_get_metadata_git_branch(i, None).strip(), \
+        base_get_metadata_git_remote_branch(i, None).strip(), \
         base_get_metadata_git_revision(i, None)) \
             for i in layers]
     return '\n'.join(medadata_revs)
diff --git a/meta/classes/metadata_scm.bbclass b/meta/classes/metadata_scm.bbclass
index 0f7f423..31a2c54 100644
--- a/meta/classes/metadata_scm.bbclass
+++ b/meta/classes/metadata_scm.bbclass
@@ -73,6 +73,15 @@ def base_get_metadata_git_branch(path, d):
         rev = '<unknown>'
     return rev.strip()
 
+def base_get_metadata_git_remote_branch(path, d):
+    import bb.process
+
+    try:
+        rev, _ = bb.process.run('git rev-parse --abbrev-ref --symbolic-full-name @{u}', cwd=path)
+    except bb.process.ExecutionError:
+        rev = '(HEAD does not point to a remote branch)'
+    return rev.strip()
+
 def base_get_metadata_git_revision(path, d):
     import bb.process
 
@@ -81,3 +90,12 @@ def base_get_metadata_git_revision(path, d):
     except bb.process.ExecutionError:
         rev = '<unknown>'
     return rev.strip()
+
+def base_get_metadata_git_remote(path, d):
+    import bb.process
+
+    try:
+        lines, _ = bb.process.run('git remote -v', cwd=path)
+    except bb.process.ExecutionError:
+        return '<unknown>'
+    return lines
-- 
2.7.0.rc3

Re: [PATCH] metadata-revs: provide more information

[Paul Eggleton]

On Sat, 12 Mar 2016 21:35:29 Trevor Woerner wrote:
> Provide many more details concerning the repositories that are used in a
> particular build: the remote information, the layer, the local branch, the
> remote branch the local branch tracks (if any), and the HEAD commit.
> 
> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> ---
>  meta/classes/buildhistory.bbclass |  6 +++++-
>  meta/classes/metadata_scm.bbclass | 18 ++++++++++++++++++
>  2 files changed, 23 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/buildhistory.bbclass
> b/meta/classes/buildhistory.bbclass index b6b4324..1b0ae0e 100644
> --- a/meta/classes/buildhistory.bbclass
> +++ b/meta/classes/buildhistory.bbclass
> @@ -615,9 +615,13 @@ def buildhistory_get_build_id(d):
> 
>  def buildhistory_get_metadata_revs(d):
>      # We want an easily machine-readable format here, so
> get_layers_branch_rev isn't quite what we want +    import subprocess
>      layers = (d.getVar("BBLAYERS", True) or "").split()
> -    medadata_revs = ["%-17s = %s:%s" % (os.path.relpath(i,
> d.getVar('BBLAYERS_FETCH_DIR', True)), \ +    medadata_revs = ["%s\tlayer: 
> %s\n\tbranch: %s\n\tremote: %s\n\tHEAD:   %s\n" % ( \ +       
> base_get_metadata_git_remote(i, None), \
> +        os.path.relpath(i, d.getVar('BBLAYERS_FETCH_DIR', True)), \
>          base_get_metadata_git_branch(i, None).strip(), \
> +        base_get_metadata_git_remote_branch(i, None).strip(), \
>          base_get_metadata_git_revision(i, None)) \
>              for i in layers]
>      return '\n'.join(medadata_revs)
> diff --git a/meta/classes/metadata_scm.bbclass
> b/meta/classes/metadata_scm.bbclass index 0f7f423..31a2c54 100644
> --- a/meta/classes/metadata_scm.bbclass
> +++ b/meta/classes/metadata_scm.bbclass
> @@ -73,6 +73,15 @@ def base_get_metadata_git_branch(path, d):
>          rev = '<unknown>'
>      return rev.strip()
> 
> +def base_get_metadata_git_remote_branch(path, d):
> +    import bb.process
> +
> +    try:
> +        rev, _ = bb.process.run('git rev-parse --abbrev-ref
> --symbolic-full-name @{u}', cwd=path) +    except
> bb.process.ExecutionError:
> +        rev = '(HEAD does not point to a remote branch)'
> +    return rev.strip()
> +
>  def base_get_metadata_git_revision(path, d):
>      import bb.process
> 
> @@ -81,3 +90,12 @@ def base_get_metadata_git_revision(path, d):
>      except bb.process.ExecutionError:
>          rev = '<unknown>'
>      return rev.strip()
> +
> +def base_get_metadata_git_remote(path, d):
> +    import bb.process
> +
> +    try:
> +        lines, _ = bb.process.run('git remote -v', cwd=path)
> +    except bb.process.ExecutionError:
> +        return '<unknown>'
> +    return lines

As I mentioned in my other reply, metadata-revs was intended to be consumed by 
scripts rather than humans, so I'd rather not change its format unless 
absolutely necessary.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] metadata-revs: provide more information

[Trevor Woerner]

Hi Paul,

On 03/13/16 15:26, Paul Eggleton wrote:
> On Sat, 12 Mar 2016 21:35:29 Trevor Woerner wrote:
>> Provide many more details concerning the repositories that are used in a
>> particular build: the remote information, the layer, the local branch, the
>> remote branch the local branch tracks (if any), and the HEAD commit.
>>
>> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
>> ---
>>   meta/classes/buildhistory.bbclass |  6 +++++-
>>   meta/classes/metadata_scm.bbclass | 18 ++++++++++++++++++
>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/buildhistory.bbclass
>> b/meta/classes/buildhistory.bbclass index b6b4324..1b0ae0e 100644
>> --- a/meta/classes/buildhistory.bbclass
>> +++ b/meta/classes/buildhistory.bbclass
>> @@ -615,9 +615,13 @@ def buildhistory_get_build_id(d):
>>
>>   def buildhistory_get_metadata_revs(d):
>>       # We want an easily machine-readable format here, so
>> get_layers_branch_rev isn't quite what we want +    import subprocess
>>       layers = (d.getVar("BBLAYERS", True) or "").split()
>> -    medadata_revs = ["%-17s = %s:%s" % (os.path.relpath(i,
>> d.getVar('BBLAYERS_FETCH_DIR', True)), \ +    medadata_revs = ["%s\tlayer:
>> %s\n\tbranch: %s\n\tremote: %s\n\tHEAD:   %s\n" % ( \ +
>> base_get_metadata_git_remote(i, None), \
>> +        os.path.relpath(i, d.getVar('BBLAYERS_FETCH_DIR', True)), \
>>           base_get_metadata_git_branch(i, None).strip(), \
>> +        base_get_metadata_git_remote_branch(i, None).strip(), \
>>           base_get_metadata_git_revision(i, None)) \
>>               for i in layers]
>>       return '\n'.join(medadata_revs)
>> diff --git a/meta/classes/metadata_scm.bbclass
>> b/meta/classes/metadata_scm.bbclass index 0f7f423..31a2c54 100644
>> --- a/meta/classes/metadata_scm.bbclass
>> +++ b/meta/classes/metadata_scm.bbclass
>> @@ -73,6 +73,15 @@ def base_get_metadata_git_branch(path, d):
>>           rev = '<unknown>'
>>       return rev.strip()
>>
>> +def base_get_metadata_git_remote_branch(path, d):
>> +    import bb.process
>> +
>> +    try:
>> +        rev, _ = bb.process.run('git rev-parse --abbrev-ref
>> --symbolic-full-name @{u}', cwd=path) +    except
>> bb.process.ExecutionError:
>> +        rev = '(HEAD does not point to a remote branch)'
>> +    return rev.strip()
>> +
>>   def base_get_metadata_git_revision(path, d):
>>       import bb.process
>>
>> @@ -81,3 +90,12 @@ def base_get_metadata_git_revision(path, d):
>>       except bb.process.ExecutionError:
>>           rev = '<unknown>'
>>       return rev.strip()
>> +
>> +def base_get_metadata_git_remote(path, d):
>> +    import bb.process
>> +
>> +    try:
>> +        lines, _ = bb.process.run('git remote -v', cwd=path)
>> +    except bb.process.ExecutionError:
>> +        return '<unknown>'
>> +    return lines
> As I mentioned in my other reply, metadata-revs was intended to be consumed by
> scripts rather than humans, so I'd rather not change its format unless
> absolutely necessary.

No problem, I had guessed that might be the case.

Would you (or anyone) be open to the possibility of the buildhistory 
task creating a second file that would contain this information?

I had been thinking about doing something like this for a long time. 
Having the two "meta" listings is what finally prompted me to actually 
do something about it. The fact that Ross pointed out that it was really 
the result of my own mistake is a little bit secondary to what I'm 
trying to accomplish.

I like to keep my build artifacts for a very long time. But I find when 
I'm looking through my old artifacts that I wish I had also kept the 
various configurations that led to that artifact's creation. ["Wait... I 
successfully built something for MACHINE XYZ back in June, how did I do 
that?"] So along with the artifacts I've also been keeping local.conf, 
auto.conf, bblayers.conf... and metadata-revs.

However, as I've pointed out, there are lots of clones around. Have you 
taken a look at github recently? There are dozens of clones + tweaks to 
many of the repositories we all know and love: beaglebone, odroid, 
raspberrypi, browser. Sometimes it's one of these clones that works out 
better for a given situation (MACHINE, board...) than the canonical. For 
example, if I want to add chromium to my DragonBoard410c build, I need 
to use Linaro's clone of meta-browser and not the canonical OSSystems' one.

The information currently provided by buildhistory/metadata-revs doesn't 
provide the level of detail required to distinguish between these two 
situations. In six months I would look at what I've built today, grab 
git://github.com/OSSystems/meta-browser.git, and then say: "how come 
there's no commit 5c00d0114c5963a178cb33f6d06181c588c03ae0?". It's not 
like there's any requirement to make the names of all layers universally 
unique.

That's the problem I'm trying to solve: how can I easily keep all the 
information required to reproduce this build, exactly. The whole "two 
meta's" thing was just a speed bump.

Re: [PATCH] metadata-revs: provide more information

[Paul Eggleton]

On Sun, 13 Mar 2016 16:42:41 Trevor Woerner wrote:
> On 03/13/16 15:26, Paul Eggleton wrote:
> > On Sat, 12 Mar 2016 21:35:29 Trevor Woerner wrote:
> >> Provide many more details concerning the repositories that are used in a
> >> particular build: the remote information, the layer, the local branch,
> >> the
> >> remote branch the local branch tracks (if any), and the HEAD commit.
> >> 
> >> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> >> ---
> >> 
> >>   meta/classes/buildhistory.bbclass |  6 +++++-
> >>   meta/classes/metadata_scm.bbclass | 18 ++++++++++++++++++
> >>   2 files changed, 23 insertions(+), 1 deletion(-)
> >> 
> >> diff --git a/meta/classes/buildhistory.bbclass
> >> b/meta/classes/buildhistory.bbclass index b6b4324..1b0ae0e 100644
> >> --- a/meta/classes/buildhistory.bbclass
> >> +++ b/meta/classes/buildhistory.bbclass
> >> 
> >> @@ -615,9 +615,13 @@ def buildhistory_get_build_id(d):
> >>   def buildhistory_get_metadata_revs(d):
> >>       # We want an easily machine-readable format here, so
> >> 
> >> get_layers_branch_rev isn't quite what we want +    import subprocess
> >> 
> >>       layers = (d.getVar("BBLAYERS", True) or "").split()
> >> 
> >> -    medadata_revs = ["%-17s = %s:%s" % (os.path.relpath(i,
> >> d.getVar('BBLAYERS_FETCH_DIR', True)), \ +    medadata_revs =
> >> ["%s\tlayer:
> >> %s\n\tbranch: %s\n\tremote: %s\n\tHEAD:   %s\n" % ( \ +
> >> base_get_metadata_git_remote(i, None), \
> >> +        os.path.relpath(i, d.getVar('BBLAYERS_FETCH_DIR', True)), \
> >> 
> >>           base_get_metadata_git_branch(i, None).strip(), \
> >> 
> >> +        base_get_metadata_git_remote_branch(i, None).strip(), \
> >> 
> >>           base_get_metadata_git_revision(i, None)) \
> >>           
> >>               for i in layers]
> >>       
> >>       return '\n'.join(medadata_revs)
> >> 
> >> diff --git a/meta/classes/metadata_scm.bbclass
> >> b/meta/classes/metadata_scm.bbclass index 0f7f423..31a2c54 100644
> >> --- a/meta/classes/metadata_scm.bbclass
> >> +++ b/meta/classes/metadata_scm.bbclass
> >> 
> >> @@ -73,6 +73,15 @@ def base_get_metadata_git_branch(path, d):
> >>           rev = '<unknown>'
> >>       
> >>       return rev.strip()
> >> 
> >> +def base_get_metadata_git_remote_branch(path, d):
> >> +    import bb.process
> >> +
> >> +    try:
> >> +        rev, _ = bb.process.run('git rev-parse --abbrev-ref
> >> --symbolic-full-name @{u}', cwd=path) +    except
> >> bb.process.ExecutionError:
> >> +        rev = '(HEAD does not point to a remote branch)'
> >> +    return rev.strip()
> >> +
> >> 
> >>   def base_get_metadata_git_revision(path, d):
> >>       import bb.process
> >> 
> >> @@ -81,3 +90,12 @@ def base_get_metadata_git_revision(path, d):
> >>       except bb.process.ExecutionError:
> >>           rev = '<unknown>'
> >>       
> >>       return rev.strip()
> >> 
> >> +
> >> +def base_get_metadata_git_remote(path, d):
> >> +    import bb.process
> >> +
> >> +    try:
> >> +        lines, _ = bb.process.run('git remote -v', cwd=path)
> >> +    except bb.process.ExecutionError:
> >> +        return '<unknown>'
> >> +    return lines
> > 
> > As I mentioned in my other reply, metadata-revs was intended to be
> > consumed by scripts rather than humans, so I'd rather not change its
> > format unless absolutely necessary.
> 
> No problem, I had guessed that might be the case.
> 
> Would you (or anyone) be open to the possibility of the buildhistory
> task creating a second file that would contain this information?
> 
> I had been thinking about doing something like this for a long time.
> Having the two "meta" listings is what finally prompted me to actually
> do something about it. The fact that Ross pointed out that it was really
> the result of my own mistake is a little bit secondary to what I'm
> trying to accomplish.
> 
> I like to keep my build artifacts for a very long time. But I find when
> I'm looking through my old artifacts that I wish I had also kept the
> various configurations that led to that artifact's creation. ["Wait... I
> successfully built something for MACHINE XYZ back in June, how did I do
> that?"] So along with the artifacts I've also been keeping local.conf,
> auto.conf, bblayers.conf... and metadata-revs.

I'm not dead-set against it; I'm not madly keen on adding more noise to 
changelogs, but then they probably need to be looked at through buildhistory-
diff rather than git diff at the moment anyway.

> However, as I've pointed out, there are lots of clones around. Have you
> taken a look at github recently? There are dozens of clones + tweaks to
> many of the repositories we all know and love: beaglebone, odroid,
> raspberrypi, browser. Sometimes it's one of these clones that works out
> better for a given situation (MACHINE, board...) than the canonical. For
> example, if I want to add chromium to my DragonBoard410c build, I need
> to use Linaro's clone of meta-browser and not the canonical OSSystems' one.

Surely we really ought to be addressing that then? If this is coming up more 
and more we need to find a systematic approach to resolving the problem. The 
more we let this continue the harder it will be for people who don't have the 
knowledge or persistence to resolve these issues by swapping in forks to do 
so; they'll simply give up. The question is what can we do to address the 
issues in the canonical layers rather than people having to resort to using 
forks? Are you able to give some examples?

> The information currently provided by buildhistory/metadata-revs doesn't
> provide the level of detail required to distinguish between these two
> situations. In six months I would look at what I've built today, grab
> git://github.com/OSSystems/meta-browser.git, and then say: "how come
> there's no commit 5c00d0114c5963a178cb33f6d06181c588c03ae0?". It's not
> like there's any requirement to make the names of all layers universally
> unique.
> 
> That's the problem I'm trying to solve: how can I easily keep all the
> information required to reproduce this build, exactly. The whole "two
> meta's" thing was just a speed bump.

Right, understood, and it does make sense. I'm convinced there is an 
underlying problem to fix that this just papers over though.

This probably fits with what you said at OEDEM, which was that too often the 
system doesn't build properly out of the box for various boards. Have you had 
a chance to think about that problem further?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] metadata-revs: provide more information

[Trevor Woerner]

(sorry about my previous reply, somehow the quoting got messed up and 
was hard to read. hopefully this one is better)

Hi Paul,

On 03/13/16 15:26, Paul Eggleton wrote:
> On Sat, 12 Mar 2016 21:35:29 Trevor Woerner wrote:
>> Provide many more details concerning the repositories that are used in a
>> particular build: the remote information, the layer, the local branch, the
>> remote branch the local branch tracks (if any), and the HEAD commit.
>>
>> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
>> ---
>>   meta/classes/buildhistory.bbclass |  6 +++++-
>>   meta/classes/metadata_scm.bbclass | 18 ++++++++++++++++++
>>   2 files changed, 23 insertions(+), 1 deletion(-)
>>
>> diff --git a/meta/classes/buildhistory.bbclass
>> b/meta/classes/buildhistory.bbclass index b6b4324..1b0ae0e 100644
>> --- a/meta/classes/buildhistory.bbclass
>> +++ b/meta/classes/buildhistory.bbclass
>> @@ -615,9 +615,13 @@ def buildhistory_get_build_id(d):
>>
>>   def buildhistory_get_metadata_revs(d):
>>       # We want an easily machine-readable format here, so
>> get_layers_branch_rev isn't quite what we want +    import subprocess
>>       layers = (d.getVar("BBLAYERS", True) or "").split()
>> -    medadata_revs = ["%-17s = %s:%s" % (os.path.relpath(i,
>> d.getVar('BBLAYERS_FETCH_DIR', True)), \ +    medadata_revs = ["%s\tlayer:
>> %s\n\tbranch: %s\n\tremote: %s\n\tHEAD:   %s\n" % ( \ +
>> base_get_metadata_git_remote(i, None), \
>> +        os.path.relpath(i, d.getVar('BBLAYERS_FETCH_DIR', True)), \
>>           base_get_metadata_git_branch(i, None).strip(), \
>> +        base_get_metadata_git_remote_branch(i, None).strip(), \
>>           base_get_metadata_git_revision(i, None)) \
>>               for i in layers]
>>       return '\n'.join(medadata_revs)
>> diff --git a/meta/classes/metadata_scm.bbclass
>> b/meta/classes/metadata_scm.bbclass index 0f7f423..31a2c54 100644
>> --- a/meta/classes/metadata_scm.bbclass
>> +++ b/meta/classes/metadata_scm.bbclass
>> @@ -73,6 +73,15 @@ def base_get_metadata_git_branch(path, d):
>>           rev = '<unknown>'
>>       return rev.strip()
>>
>> +def base_get_metadata_git_remote_branch(path, d):
>> +    import bb.process
>> +
>> +    try:
>> +        rev, _ = bb.process.run('git rev-parse --abbrev-ref
>> --symbolic-full-name @{u}', cwd=path) +    except
>> bb.process.ExecutionError:
>> +        rev = '(HEAD does not point to a remote branch)'
>> +    return rev.strip()
>> +
>>   def base_get_metadata_git_revision(path, d):
>>       import bb.process
>>
>> @@ -81,3 +90,12 @@ def base_get_metadata_git_revision(path, d):
>>       except bb.process.ExecutionError:
>>           rev = '<unknown>'
>>       return rev.strip()
>> +
>> +def base_get_metadata_git_remote(path, d):
>> +    import bb.process
>> +
>> +    try:
>> +        lines, _ = bb.process.run('git remote -v', cwd=path)
>> +    except bb.process.ExecutionError:
>> +        return '<unknown>'
>> +    return lines
>
> As I mentioned in my other reply, metadata-revs was intended to be consumed by
> scripts rather than humans, so I'd rather not change its format unless
> absolutely necessary.


No problem, I had guessed that might be the case.

Would you (or anyone) be open to the possibility of the buildhistory 
task creating a second file that would contain this information?

I had been thinking about doing something like this for a long time. 
Having the two "meta" listings is what finally prompted me to actually 
do something about it. The fact that Ross pointed out that it was really 
the result of my own mistake is a little bit secondary to what I'm 
trying to accomplish.

I like to keep my build artifacts for a very long time. But I find when 
I'm looking through my old artifacts that I wish I had also kept the 
various configurations that led to that artifact's creation. ["Wait... I 
successfully built something for MACHINE XYZ back in June, how did I do 
that?"] So, along with the artifacts, I've also been keeping local.conf, 
auto.conf, bblayers.conf... and metadata-revs.

However, as I've pointed out, there are lots of clones around. Have you
taken a look at github recently? There are dozens of clones + tweaks to
many of the repositories we all know and love: beaglebone, odroid,
raspberrypi, browser. Sometimes it's one of these clones that works out
better for a given situation (MACHINE, board...) than the canonical. For 
example, if I want to add chromium to my DragonBoard410c build, I need 
to use Linaro's clone of meta-browser and not the canonical OSSystems' one.

The information currently provided by buildhistory/metadata-revs doesn't 
provide the level of detail required to distinguish between these two 
situations. In six months I would look at what I've built today, grab 
git://github.com/OSSystems/meta-browser.git, and then say: "how come 
there's no commit 5c00d0114c5963a178cb33f6d06181c588c03ae0?". It's not 
like there's any requirement to make the names of all layers universally 
unique.

That's the problem I'm trying to solve: how can I easily keep all the
information required to reproduce this build, exactly. The whole "two
meta's" thing was just a speed bump.

Re: [PATCH] metadata-revs: provide more information

[Trevor Woerner]

On 03/13/16 16:53, Paul Eggleton wrote:
> On Sun, 13 Mar 2016 16:42:41 Trevor Woerner wrote:
>> That's the problem I'm trying to solve: how can I easily keep all the
>> information required to reproduce this build, exactly. The whole "two
>> meta's" thing was just a speed bump.
>
> Right, understood, and it does make sense. I'm convinced there is an
> underlying problem to fix that this just papers over though.

Okay, I think I see your point, and I agree. There is a layer clone 
problem that should be addressed.

But again, I think this is a speed bump.

My underlying point is that we're purposefully leaving out information 
that is vital to reproducible builds.


> This probably fits with what you said at OEDEM, which was that too often the
> system doesn't build properly out of the box for various boards. Have you had
> a chance to think about that problem further?

Oh yes, but I'm rather sure nobody's going to like my suggestion, which 
makes me hesitant to suggest it ;-)

It should be possible for me to hand you _one_ configuration file and 
from that you could reproduce my entire build in _one_ step by issuing 
_one_ command. If I can get something to build successfully for a given 
board, you should too. But currently that's not always the case.

bblayers.conf and local.conf should somehow be rolled into one and the 
build should have some sort of "repo tool"-like functionality built in.

I hand you this one configuration file, you run "bitbake some-or-other" 
and the build starts by going through the provided list of layers 
checking out the exact ones I have used and potentially checking them 
out at the exact revisions I've used. This means the layer list will 
have to specify from where the layer was found, which branch, what the 
HEAD was, etc. In other words, exactly the information I'm trying to 
capture at the end of any build today.

I really think this might help everyone.

Newbies would have a better out-of-box experience. Chances are they just 
bought some board and want to make a build for it. I think it is rare 
that someone stumbles onto The Yocto Project and is happy just doing 
builds for qemu. My belief is that people find The Yocto Project as a 
secondary event, the main event is they want to perform a build for the 
hardware that just arrived and is sitting on their desk. The generic 
documentation doesn't target their board specifically, so in addition to 
trying to understand this new build system, they also have to figure out 
how to get a layer, where to get it from, and how to add it to their 
build. I know that The Yocto Project does have BSPs for beaglebone, 
raspberry pi 2, generic x86(-64), edgerouter, and some ppc board... and 
if this newbie happens to have one of those, things might go better. But 
if they have some other board...

I think experienced developers would be helped too by the combined 
config+layer-full-details thing too. For example, developers would swap 
configs between each other more easily, and build artifacts could be 
kept for future reference which is very important in a production 
environment.

The current way of doing things purposely hides information, and this is 
bad.

bblayers only tells you that on a given day on a specific machine there 
was a layer at a specific location that was used for this build. If that 
layer is something that was pulled from somewhere else (and not, as Khem 
was mentioning, something that was done locally) then there is no 
mechanism anywhere that preserves this information. There's a gap; a gap 
in the information. On my machine there's a meta-browser layer, on 
github there are hundreds of meta-browser layers, bblayers just says I 
used something called meta-browser... but the link is missing which 
points to from where I got this layer.

If I wanted to be really devious I could clone my repositories with any 
weird old name I wanted. I could clone meta-browser and name it 
openembedded-core and clone openembedded-core and name it meta-trevor. 
My build would still work 100%, but good luck with that! :-D

The genie's out of the bottle. There already are hundreds of clones. 
There is no way to police it, there is no way to force people to use 
globally unique names. I don't know how you're going to fix something 
like that. I think the best thing is to recognize this is going to 
happen and take steps to mitigate the issue... such as specifying where 
layers are coming from both in the config (for external layers) and 
after the build to record what happened.

I don't want to embarrass anyone by giving names, and I certainly don't 
want to be seen as a troll :-( I've been playing around with a large 
number of boards for the last while and there are lots of issues that 
I've come across. I already mentioned my thoughts on the out-of-box 
experience for newbies who have just bought a board and now have to 
figure out how to successfully add a layer.

Another thing I come across is the following: given a company (ACME) who 
produces a bunch of boards (ACME-1, ACME-2, and ACME-3). The ACME-1 has 
CPU-1, ACME-2 has CPU-2, and ACME-3 has CPU-3. On github you'll find 
several meta-acme's and a couple meta-cpu's. Some of the meta-acme's 
will have support for one or more of the ACME-n boards, but not 
necessarily all of them, or you might find your board is supported by 
one of the meta-cpu repositories you've found. You'll even find one of 
those meta-acme's in the layerindex, but the developer of that 
particular layer only has 2 out of the 3 boards so that's all their 
layer supports. If you happen to have the one that's not in layerindex's 
meta-acme then you'll need to google and use some other meta-acme you've 
found yourself. When you ask the maintainers of the various meta-acme's 
to please get together and make sure the layerindex layer fully supports 
all the ACME board, sadly that doesn't happen. When you provide patches 
and pull requests to add the missing support, you're at the mercy of the 
maintainer, who is happy to refuse your contribution for all manner of 
reasons ;-)

Simply having a bblayers file that says meta-acme or meta-cpu isn't good 
enough. And hoping that we can all standardize on one layer... well, you 
know what they say about standards! :-D

Re: [PATCH] metadata-revs: provide more information

[Paul Eggleton]

On Sun, 13 Mar 2016 17:54:08 Trevor Woerner wrote:
> On 03/13/16 16:53, Paul Eggleton wrote:
> > On Sun, 13 Mar 2016 16:42:41 Trevor Woerner wrote:
> >> That's the problem I'm trying to solve: how can I easily keep all the
> >> information required to reproduce this build, exactly. The whole "two
> >> meta's" thing was just a speed bump.
> > 
> > Right, understood, and it does make sense. I'm convinced there is an
> > underlying problem to fix that this just papers over though.
> 
> Okay, I think I see your point, and I agree. There is a layer clone
> problem that should be addressed.
> 
> But again, I think this is a speed bump.
> 
> My underlying point is that we're purposefully leaving out information
> that is vital to reproducible builds.
> 
> > This probably fits with what you said at OEDEM, which was that too often
> > the system doesn't build properly out of the box for various boards. Have
> > you had a chance to think about that problem further?
> 
> Oh yes, but I'm rather sure nobody's going to like my suggestion, which
> makes me hesitant to suggest it ;-)

You may be surprised ;)

> It should be possible for me to hand you _one_ configuration file and
> from that you could reproduce my entire build in _one_ step by issuing
> _one_ command. If I can get something to build successfully for a given
> board, you should too. But currently that's not always the case.
> 
> bblayers.conf and local.conf should somehow be rolled into one and the
> build should have some sort of "repo tool"-like functionality built in.
>
> I hand you this one configuration file, you run "bitbake some-or-other"
> and the build starts by going through the provided list of layers
> checking out the exact ones I have used and potentially checking them
> out at the exact revisions I've used. This means the layer list will
> have to specify from where the layer was found, which branch, what the
> HEAD was, etc. In other words, exactly the information I'm trying to
> capture at the end of any build today.

Actually, you're not the first to suggest this - at least it's come up in 
discussions about running builds from Toaster and again separately for running 
builds within fairly generic containers. For those purposes it really would be 
nice if we could have a single file that completely describes how to run a 
build (i.e. metadata revisions and configuration), and it would also be very 
useful for sending someone a build to test on their system - in the absence of 
local changes, of course. Like anything else though the devil is in the 
details. I suspect we will try to implement at least the basic capability in 
the next release.

> Newbies would have a better out-of-box experience. Chances are they just
> bought some board and want to make a build for it. I think it is rare
> that someone stumbles onto The Yocto Project and is happy just doing
> builds for qemu. My belief is that people find The Yocto Project as a
> secondary event, the main event is they want to perform a build for the
> hardware that just arrived and is sitting on their desk. The generic
> documentation doesn't target their board specifically, so in addition to
> trying to understand this new build system, they also have to figure out
> how to get a layer, where to get it from, and how to add it to their
> build. I know that The Yocto Project does have BSPs for beaglebone,
> raspberry pi 2, generic x86(-64), edgerouter, and some ppc board... and
> if this newbie happens to have one of those, things might go better. But
> if they have some other board...
> 
> I think experienced developers would be helped too by the combined
> config+layer-full-details thing too. For example, developers would swap
> configs between each other more easily, and build artifacts could be
> kept for future reference which is very important in a production
> environment.
> 
> The current way of doing things purposely hides information, and this is
> bad.
> 
> bblayers only tells you that on a given day on a specific machine there
> was a layer at a specific location that was used for this build. If that
> layer is something that was pulled from somewhere else (and not, as Khem
> was mentioning, something that was done locally) then there is no
> mechanism anywhere that preserves this information. There's a gap; a gap
> in the information. On my machine there's a meta-browser layer, on
> github there are hundreds of meta-browser layers, bblayers just says I
> used something called meta-browser... but the link is missing which
> points to from where I got this layer.

Right, shipping around bblayers.conf doesn't really help much I agree. I don't 
think bblayers.conf was meant to be used in that way though. If we want an 
interchange format I think the time has come to create one.
 
> If I wanted to be really devious I could clone my repositories with any
> weird old name I wanted. I could clone meta-browser and name it
> openembedded-core and clone openembedded-core and name it meta-trevor.
> My build would still work 100%, but good luck with that! :-D
> 
> The genie's out of the bottle. There already are hundreds of clones.
> There is no way to police it, there is no way to force people to use
> globally unique names. 

I wouldn't suggest trying to police it, as you say it would be impossible. 
What I think is at least theoretically possible is to address some of the 
issues we have that make people feel the need to create their own forks for 
anything beyond testing / distributing changes on a short-term basis.

> I don't want to embarrass anyone by giving names, and I certainly don't
> want to be seen as a troll :-( I've been playing around with a large
> number of boards for the last while and there are lots of issues that
> I've come across. I already mentioned my thoughts on the out-of-box
> experience for newbies who have just bought a board and now have to
> figure out how to successfully add a layer.

I don't see this kind of thing as trolling. I believe If we have a fundamental 
problem then we really need to address it head on rather than grumbling to 
ourselves and working around it - which isn't what you're doing right now, 
don't get me wrong - it's good that we're at least talking in general terms, 
but I believe we need to go deeper.

> Another thing I come across is the following: given a company (ACME) who
> produces a bunch of boards (ACME-1, ACME-2, and ACME-3). The ACME-1 has
> CPU-1, ACME-2 has CPU-2, and ACME-3 has CPU-3. On github you'll find
> several meta-acme's and a couple meta-cpu's. Some of the meta-acme's
> will have support for one or more of the ACME-n boards, but not
> necessarily all of them, or you might find your board is supported by
> one of the meta-cpu repositories you've found. You'll even find one of
> those meta-acme's in the layerindex, but the developer of that
> particular layer only has 2 out of the 3 boards so that's all their
> layer supports. If you happen to have the one that's not in layerindex's
> meta-acme then you'll need to google and use some other meta-acme you've
> found yourself. When you ask the maintainers of the various meta-acme's
> to please get together and make sure the layerindex layer fully supports
> all the ACME board, sadly that doesn't happen. When you provide patches
> and pull requests to add the missing support, you're at the mercy of the
> maintainer, who is happy to refuse your contribution for all manner of
> reasons ;-)

Maintainers generally have the final say on what goes into a layer and I'm not 
sure it could really be any other way. However, the OE TSC is supposed to be 
how we resolve any technical disputes or other problems that can't be resolved 
through normal channels, and I'd encourage everyone to contact the TSC if they 
feel like a problem is being ignored or isn't being addressed in a manner that 
benefits the wider community. As an aside I noticed the TSC page was somewhat 
out-of-date so I went ahead and updated it:

  http://www.openembedded.org/wiki/TSC

To clarify, I do understand people will fork things (and github encourages 
forking, and I'm definitely not saying that's a bad thing). However if OE needs 
to do better at testing or highlighting how to get a working build or 
something else entirely that's what we should strive to do; let's talk about 
that.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] metadata-revs: provide more information

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1444 bytes --]

On Sun, Mar 13, 2016 at 05:54:08PM -0400, Trevor Woerner wrote:
> On 03/13/16 16:53, Paul Eggleton wrote:
> > On Sun, 13 Mar 2016 16:42:41 Trevor Woerner wrote:
> >> That's the problem I'm trying to solve: how can I easily keep all the
> >> information required to reproduce this build, exactly. The whole "two
> >> meta's" thing was just a speed bump.
> >
> > Right, understood, and it does make sense. I'm convinced there is an
> > underlying problem to fix that this just papers over though.
> 
> Okay, I think I see your point, and I agree. There is a layer clone 
> problem that should be addressed.
> 
> But again, I think this is a speed bump.
> 
> My underlying point is that we're purposefully leaving out information 
> that is vital to reproducible builds.

FWIW: I think this information also useful (even as separate file in
buildhistory).

In most builds it shouldn't change very often (except the revision) so
the diff shouldn't grow significantly.

There are many reasons why people use different forks of some layers,
some of them are valid (like using own fork to verify some change before
submitting it upstream and showing buildhistory diff how you tested it),
some of them are sad, but still needed _to be shown somewhere_ (like
using fork with important fix, when upstream layer is slow or refusing
to accept the fix).

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [PATCH] metadata-revs: provide more information

[Jeremy Rosen]

----- Le 14 Mar 16, à 11:15, Martin Jansa martin.jansa@gmail.com a écrit :

> On Sun, Mar 13, 2016 at 05:54:08PM -0400, Trevor Woerner wrote:
>> On 03/13/16 16:53, Paul Eggleton wrote:
>> > On Sun, 13 Mar 2016 16:42:41 Trevor Woerner wrote:
>> >> That's the problem I'm trying to solve: how can I easily keep all the
>> >> information required to reproduce this build, exactly. The whole "two
>> >> meta's" thing was just a speed bump.
>> >
>> > Right, understood, and it does make sense. I'm convinced there is an
>> > underlying problem to fix that this just papers over though.
>> 
>> Okay, I think I see your point, and I agree. There is a layer clone
>> problem that should be addressed.
>> 
>> But again, I think this is a speed bump.
>> 
>> My underlying point is that we're purposefully leaving out information
>> that is vital to reproducible builds.
> 

I think there are two problems here and that they need to be solved
separately...


* buildhistory does not save all the information to do a reproducible build
  
  I think the easiest way to solve that would be to save the whole bitbake
  environement used for building the recipe in the buildhistory. We would
  loose the layout info (i.e what layer touched what variable, though that
  info is in bitbake -e) but we would have enough info to reproduce the 
  build itself. Moreover being able to diff the environement would be 
  a great way to figure out what the change in the build actually was

* We need a way to easily transmit a set of meta revisions + extra info

  that is a different problem, telling a user how to reproduce a build 
  from scratch, i.e starting from the download of the right meta. This
  calls for a tool similar to repo, but adapted to yocto (which might
  be repo itself, I don't know repo well enough to judge). What i've
  seen so far is that most board vendors will ship an ugly shell 
  script that will clone and set the meta for you, usually doing some
  ugly hacks like patching the meta themselves. Having an "official"
  way to pack/distribute that info in yocto would be a very usefull
  tool indeed

my 2 cents...

Cordialement 

Jérémy Rosen 
+33 (0)1 42 68 28 13

fight key loggers : write some perl using vim 


Open Wide Ingenierie

23, rue Daviel
75013 Paris - France
www.openwide.fr