[PATCH 0/2] Allow oe-pkgdata-util package-info to display more info

[PATCH 0/2] Allow oe-pkgdata-util package-info to display more info

[Peter Kjellerstedt]

After a build in our autobuilder, we use `oe-pkgdata-util package-info
-f package.manifest` to store a file with information about each
installed package. This is typically used to compare builds later
on. Since not every difference to a package is explained by its
version, we have found it beneficial to also include the SRC_URI in
the generated file.

This patch set adds SRC_URI to the pkgdata that is stored for each
package, and adds a new option to oe-pkgdata-util package-info,
-e <var>, that can be used to display extra variables from the
pkgdata.

//Peter

The following changes since commit 0fcbb4d0ec26ef24db78190dcdc5568af3beaa14:

  GNU_MIRROR: switch from ftp to https (2017-05-11 16:56:00 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib pkj/oe-pkgdata-util_extra
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/oe-pkgdata-util_extra

Peter Kjellerstedt (2):
  package.bbclass: Add SRC_URI to pkgdata
  oe-pkgdata-util: package-info: Allow extra variables to be displayed

 meta/classes/package.bbclass |  1 +
 scripts/oe-pkgdata-util      | 11 +++++++++--
 2 files changed, 10 insertions(+), 2 deletions(-)

-- 
2.12.0

[PATCH 1/2] package.bbclass: Add SRC_URI to pkgdata

[Peter Kjellerstedt]

Emit SRC_URI variable in pkgdata. Having this available is beneficial
when comparing the information generated for two different builds and
looking for changes not directly related to the package versions.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/package.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index cc466bd1b2..f6641f26e9 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1370,6 +1370,7 @@ python emit_pkgdata() {
         write_if_exists(sf, pkg, 'LICENSE')
         write_if_exists(sf, pkg, 'DESCRIPTION')
         write_if_exists(sf, pkg, 'SUMMARY')
+        write_if_exists(sf, pkg, 'SRC_URI')
         write_if_exists(sf, pkg, 'RDEPENDS')
         rprov = write_if_exists(sf, pkg, 'RPROVIDES')
         write_if_exists(sf, pkg, 'RRECOMMENDS')
-- 
2.12.0

[PATCH 2/2] oe-pkgdata-util: package-info: Allow extra variables to be displayed

[Peter Kjellerstedt]

By specifying the -e <var> option one or multiple times, extra
variables available in the pkgdata can be displayed, e.,g,
`oe-pkgdata-util package-info -e SUMMARY -e SRC_URI ...`. The extra
variables displayed are quoted since some of them may contain
whitespace.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 scripts/oe-pkgdata-util | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
index 6255662a4b..741bc7bf47 100755
--- a/scripts/oe-pkgdata-util
+++ b/scripts/oe-pkgdata-util
@@ -325,8 +325,14 @@ def package_info(args):
             recipe_version = recipe_version + "-" + mappings[pkg]['PR']
         pkg_size = mappings[pkg]['PKGSIZE']
 
-        items.append("%s %s %s %s %s" %
-                     (pkg, pkg_version, recipe, recipe_version, pkg_size))
+        line = "%s %s %s %s %s" % (pkg, pkg_version, recipe, recipe_version, pkg_size)
+
+        for val in args.extra:
+            src_uri = mappings[pkg][val].strip()
+            src_uri = re.sub(r'\s+', ' ', src_uri)
+            line += ' "%s"' % src_uri
+
+        items.append(line)
     print('\n'.join(items))
 
 def get_recipe_pkgs(pkgdata_dir, recipe, unpackaged):
@@ -530,6 +536,7 @@ def main():
                                           description='Looks up the specified runtime package(s) and display information')
     parser_package_info.add_argument('pkg', nargs='*', help='Runtime package name to look up')
     parser_package_info.add_argument('-f', '--file', help='Read package names from the specified file (one per line, first field only)')
+    parser_package_info.add_argument('-e', '--extra', help='Extra variables to display, e.g., SRC_URI (can be specified multiple times)', action='append')
     parser_package_info.set_defaults(func=package_info)
 
     parser_find_path = subparsers.add_parser('find-path',
-- 
2.12.0

Re: [PATCH 0/2] Allow oe-pkgdata-util package-info to display more info

[Richard Purdie]

On Tue, 2017-05-16 at 12:56 +0200, Peter Kjellerstedt wrote:
> After a build in our autobuilder, we use `oe-pkgdata-util package-
> info
> -f package.manifest` to store a file with information about each
> installed package. This is typically used to compare builds later
> on. Since not every difference to a package is explained by its
> version, we have found it beneficial to also include the SRC_URI in
> the generated file.
> 
> This patch set adds SRC_URI to the pkgdata that is stored for each
> package, and adds a new option to oe-pkgdata-util package-info,
> -e <var>, that can be used to display extra variables from the
> pkgdata.

I'm going to say no to this.

The reason is that pkgdata is not really about collecting up all build
information. If you want to know how two different builds differ, you'd
use the sigdata files. If I take this patch, more will follow where you
find some new difference you want to track and there are other
mechanisms I'd suggest (buildhistory and siginfo for starters). I don't
want to turn the pkgdata files into something they're not.

Cheers,

Richard

Re: [PATCH 0/2] Allow oe-pkgdata-util package-info to display more info

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: den 16 maj 2017 15:15
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 0/2] Allow oe-pkgdata-util package-info
> to display more info
> 
> On Tue, 2017-05-16 at 12:56 +0200, Peter Kjellerstedt wrote:
> > After a build in our autobuilder, we use `oe-pkgdata-util package-
> > info
> > -f package.manifest` to store a file with information about each
> > installed package. This is typically used to compare builds later
> > on. Since not every difference to a package is explained by its
> > version, we have found it beneficial to also include the SRC_URI in
> > the generated file.
> >
> > This patch set adds SRC_URI to the pkgdata that is stored for each
> > package, and adds a new option to oe-pkgdata-util package-info,
> > -e <var>, that can be used to display extra variables from the
> > pkgdata.
> 
> I'm going to say no to this.
> 
> The reason is that pkgdata is not really about collecting up all build
> information. If you want to know how two different builds differ, you'd
> use the sigdata files. If I take this patch, more will follow where you
> find some new difference you want to track and there are other
> mechanisms I'd suggest (buildhistory and siginfo for starters). I don't
> want to turn the pkgdata files into something they're not.
> 
> Cheers,
> 
> Richard

As you may have expected, that was not the response I had hoped for.
However, I will see if we can come to a working solution that you can 
accept.

Since I hope that the change to oe-pkgdata-util is acceptable to you, 
I will focus on the change in package.bbclass.

Regarding siginfo: I assume it has the information in there somewhere. 
However, I have been working with OE for five years now, and it is still 
basically a black hole to me. I have no idea how to do anything useful 
with it.

Regarding buildhistory: AFAICT there is nothing in there about neither 
recipe names nor SRC_URI. I guess that can be added, though, if really 
needed.

On the other hand, we have a tool, oe-pkgdata-util, that provides a 
simple interface to access the package information, and can produce a 
simple information file (oe-pkgdata-util package-info) with one line 
per package. This file is simple enough that I can give it to our 
maintenance team and they can look at it to see if there are any 
differences they need to know about. An additional benefit here is that 
we can run oe-pkgdata-util from within a bitbake task to generate the 
file together with the other artefacts we produce for a release.

That said, I can understand that you do not want to add information to 
the pkgdata that is not really needed to build. However, would you accept 
a way to add to this data, e.g., by specifying a BitBake variable such as 
EXTRA_PKGDATA_VARS with the names of the extra variables that should be 
stored in pkgdata? That way it would not affect anyone unless they 
actually need this extra data, and I would not have the burden of carrying 
a backported version of package.bbclass forever in our layers with all the 
extra maintenance that incurs.

//Peter

Re: [PATCH 0/2] Allow oe-pkgdata-util package-info to display more info

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Peter Kjellerstedt
> Sent: den 16 maj 2017 18:30
> To: Richard Purdie <richard.purdie@linuxfoundation.org>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 0/2] Allow oe-pkgdata-util package-info
> to display more info
> 
> > -----Original Message-----
> > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > Sent: den 16 maj 2017 15:15
> > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> > core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH 0/2] Allow oe-pkgdata-util package-info
> > to display more info
> >
> > On Tue, 2017-05-16 at 12:56 +0200, Peter Kjellerstedt wrote:
> > > After a build in our autobuilder, we use `oe-pkgdata-util package-
> > > info -f package.manifest` to store a file with information about 
> > > each installed package. This is typically used to compare builds 
> > > later on. Since not every difference to a package is explained by 
> > > its version, we have found it beneficial to also include the 
> > > SRC_URI in the generated file.
> > >
> > > This patch set adds SRC_URI to the pkgdata that is stored for each
> > > package, and adds a new option to oe-pkgdata-util package-info,
> > > -e <var>, that can be used to display extra variables from the
> > > pkgdata.
> >
> > I'm going to say no to this.
> >
> > The reason is that pkgdata is not really about collecting up all
> > build information. If you want to know how two different builds 
> > differ, you'd use the sigdata files. If I take this patch, more 
> > will follow where you find some new difference you want to track 
> > and there are other mechanisms I'd suggest (buildhistory and 
> > siginfo for starters). I don't want to turn the pkgdata files 
> > into something they're not.
> >
> > Cheers,
> >
> > Richard
> 
> As you may have expected, that was not the response I had hoped for.
> However, I will see if we can come to a working solution that you can
> accept.
> 
> Since I hope that the change to oe-pkgdata-util is acceptable to you,
> I will focus on the change in package.bbclass.
> 
> Regarding siginfo: I assume it has the information in there somewhere.
> However, I have been working with OE for five years now, and it is
> still basically a black hole to me. I have no idea how to do anything 
> useful with it.
> 
> Regarding buildhistory: AFAICT there is nothing in there about neither
> recipe names nor SRC_URI. I guess that can be added, though, if really
> needed.
> 
> On the other hand, we have a tool, oe-pkgdata-util, that provides a
> simple interface to access the package information, and can produce a
> simple information file (oe-pkgdata-util package-info) with one line
> per package. This file is simple enough that I can give it to our
> maintenance team and they can look at it to see if there are any
> differences they need to know about. An additional benefit here is that
> we can run oe-pkgdata-util from within a bitbake task to generate the
> file together with the other artefacts we produce for a release.
> 
> That said, I can understand that you do not want to add information to
> the pkgdata that is not really needed to build. However, would you
> accept a way to add to this data, e.g., by specifying a BitBake variable 
> such as EXTRA_PKGDATA_VARS with the names of the extra variables that 
> should be stored in pkgdata? That way it would not affect anyone unless 
> they actually need this extra data, and I would not have the burden of
> carrying a backported version of package.bbclass forever in our layers 
> with all the extra maintenance that incurs.
> 
> //Peter

It has been a week now. Any response would be appreciated.

//Peter

Re: [PATCH 0/2] Allow oe-pkgdata-util package-info to display more info

[Richard Purdie]

On Tue, 2017-05-16 at 16:29 +0000, Peter Kjellerstedt wrote:
> > 
> > -----Original Message-----
> > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > Sent: den 16 maj 2017 15:15
> > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> > core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH 0/2] Allow oe-pkgdata-util package-
> > info
> > to display more info
> > 
> > On Tue, 2017-05-16 at 12:56 +0200, Peter Kjellerstedt wrote:
> > > 
> > > After a build in our autobuilder, we use `oe-pkgdata-util
> > > package-
> > > info
> > > -f package.manifest` to store a file with information about each
> > > installed package. This is typically used to compare builds later
> > > on. Since not every difference to a package is explained by its
> > > version, we have found it beneficial to also include the SRC_URI
> > > in
> > > the generated file.
> > > 
> > > This patch set adds SRC_URI to the pkgdata that is stored for
> > > each
> > > package, and adds a new option to oe-pkgdata-util package-info,
> > > -e <var>, that can be used to display extra variables from the
> > > pkgdata.
> > I'm going to say no to this.
> > 
> > The reason is that pkgdata is not really about collecting up all
> > build
> > information. If you want to know how two different builds differ,
> > you'd
> > use the sigdata files. If I take this patch, more will follow where
> > you
> > find some new difference you want to track and there are other
> > mechanisms I'd suggest (buildhistory and siginfo for starters). I
> > don't
> > want to turn the pkgdata files into something they're not.
> > 
> As you may have expected, that was not the response I had hoped for.
> However, I will see if we can come to a working solution that you can
> accept.
> 
> Since I hope that the change to oe-pkgdata-util is acceptable to you,
> I will focus on the change in package.bbclass.

That change is ok in principle but I don't like the usage of SRC_URI in
the help text or src_uri as a variable name in the code. I could read
that as you're not even making an effort to think bout the bigger
picture or generic patches with a variable name like that :(.

> Regarding siginfo: I assume it has the information in there
> somewhere. However, I have been working with OE for five years now,
> and it is still basically a black hole to me. I have no idea how to
> do anything useful with it.

The siginfo files are pickled python data. You can look at the code in
siggen.py (called from bitbake-dumpsig) so an example of how to view
it. They contain all the information about what went into a given task.

If two siginfo files differ, they'll have different checksums. You can
then see from bitbake-diffsigs exactly what those differences were. On
a per variable basis. They therefore offer the ultimate in build
different analysis.

> Regarding buildhistory: AFAICT there is nothing in there about
> neither recipe names nor SRC_URI. I guess that can be added, though,
> if really needed.

buildhistory is about output comparison so it probably doesn't make
sense there. I do believe it would offer more generic mechanisms than
we can afford with pkgdata though. Your best bet for generic build
comparison is still the siginfo files.

> On the other hand, we have a tool, oe-pkgdata-util, that provides a 
> simple interface to access the package information, and can produce a
> simple information file (oe-pkgdata-util package-info) with one line 
> per package. This file is simple enough that I can give it to our 
> maintenance team and they can look at it to see if there are any 
> differences they need to know about. An additional benefit here is
> that we can run oe-pkgdata-util from within a bitbake task to
> generate the file together with the other artefacts we produce for a
> release.

SRC_URI is not a piece of metadata associated with a package, its
associated with a recipe. Yes, there is recipe metadata that feeds into
packages but we really do need to have data in the right places. We
could list every possible recipe variable in that list, for the reasons
you argue. For example, perhaps S should be listed (and B) since we
might want to locate the source or the built artefacts (and so on). I
believe pkgdata should be limited to package variables though. I
appreciate that isn't what you want to hear.

> That said, I can understand that you do not want to add information
> to the pkgdata that is not really needed to build. However, would you
> accept a way to add to this data, e.g., by specifying a BitBake
> variable such as EXTRA_PKGDATA_VARS with the names of the extra
> variables that should be stored in pkgdata? That way it would not
> affect anyone unless they actually need this extra data, and I would
> not have the burden of carrying a backported version of
> package.bbclass forever in our layers with all the extra maintenance
> that incurs.

The data from pkgdata is really meant to be used by the package_write_*
tasks as a mechanism to transfer data from the do_package task. oe-
pkgdata-util was created to solve problems of visibility into the same
*generated* data. We are planning to add a visualiser of that data too.

What I don't like is adding data that is:
  a) not package specific
  b) not generated
  c) available elsewhere

There are only a very specific set of variables which contain
appropriate data, so no, I don't think we do want to support usecases
which "abuse" this data.

I'm travelling, distracted and tired but you're pushing me for a reply
so there you have it, no, we're not doing this, sorry. 

Cheers,

Richard