* [PATCH] buildhistory: Add output file listing package information @ 2021-08-12 16:58 Andres Beltran 2021-08-13 7:38 ` [OE-core] " Mikko Rapeli 2021-08-26 21:33 ` Paul Eggleton 0 siblings, 2 replies; 4+ messages in thread From: Andres Beltran @ 2021-08-12 16:58 UTC (permalink / raw) To: openembedded-core; +Cc: Andres Beltran Currently, buildhistory does not produce a single file combining relevant information of installed packages. Produce an output file "installed-package-info.txt" listing a package's runtime name, buildtime name, its recipe, version, and size to avoid having to look up each package externally. Leave the existing package list files as-is for backwards compatibility. In order to support this efficiently, extend oe-pkgdata-util to accept multiple keys for its read-value argument. Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com> --- meta/classes/buildhistory.bbclass | 5 +++++ scripts/oe-pkgdata-util | 37 +++++++++++++++++++------------ 2 files changed, 28 insertions(+), 14 deletions(-) diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass index 8a1359acbe..134b56ab71 100644 --- a/meta/classes/buildhistory.bbclass +++ b/meta/classes/buildhistory.bbclass @@ -491,6 +491,11 @@ buildhistory_get_installed() { cat $1/installed-package-sizes.tmp | awk '{print $2 "\tKiB\t" $1}' | sort -n -r > $1/installed-package-sizes.txt rm $1/installed-package-sizes.tmp + # Produce package info: runtime_name, buildtime_name, recipe, version, size + oe-pkgdata-util -p ${PKGDATA_DIR} read-value "PACKAGE,PN,PV,PKGSIZE" -n -f $pkgcache > $1/installed-package-info.tmp + cat $1/installed-package-info.tmp | sort -n -r -k 5 > $1/installed-package-info.txt + rm $1/installed-package-info.tmp + # We're now done with the cache, delete it rm $pkgcache diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util index 75dd23efa3..c30baaa580 100755 --- a/scripts/oe-pkgdata-util +++ b/scripts/oe-pkgdata-util @@ -171,7 +171,7 @@ def read_value(args): val = line.split(': ', 1)[1].rstrip() return val - logger.debug("read-value('%s', '%s' '%s')" % (args.pkgdata_dir, args.valuename, packages)) + logger.debug("read-value('%s', '%s' '%s')" % (args.pkgdata_dir, args.valuenames, packages)) for package in packages: pkg_split = package.split('_') pkg_name = pkg_split[0] @@ -180,20 +180,29 @@ def read_value(args): logger.debug(revlink) if os.path.exists(revlink): mappedpkg = os.path.basename(os.readlink(revlink)) - qvar = args.valuename - value = readvar(revlink, qvar, mappedpkg) - if qvar == "PKGSIZE": - # PKGSIZE is now in bytes, but we we want it in KB - pkgsize = (int(value) + 1024 // 2) // 1024 - value = "%d" % pkgsize - if args.unescape: - import codecs - # escape_decode() unescapes backslash encodings in byte streams - value = codecs.escape_decode(bytes(value, "utf-8"))[0].decode("utf-8") + qvars = args.valuenames + val_names = qvars.split(',') + values = [] + for qvar in val_names: + if qvar == "PACKAGE": + value = mappedpkg + else: + value = readvar(revlink, qvar, mappedpkg) + if qvar == "PKGSIZE": + # PKGSIZE is now in bytes, but we we want it in KB + pkgsize = (int(value) + 1024 // 2) // 1024 + value = "%d" % pkgsize + if args.unescape: + import codecs + # escape_decode() unescapes backslash encodings in byte streams + value = codecs.escape_decode(bytes(value, "utf-8"))[0].decode("utf-8") + values.append(value) + + values_str = ' '.join(values) if args.prefix_name: - print('%s %s' % (pkg_name, value)) + print('%s %s' % (pkg_name, values_str)) else: - print(value) + print(values_str) else: logger.debug("revlink %s does not exist", revlink) @@ -570,7 +579,7 @@ def main(): parser_read_value = subparsers.add_parser('read-value', help='Read any pkgdata value for one or more packages', description='Reads the named value from the pkgdata files for the specified packages') - parser_read_value.add_argument('valuename', help='Name of the value to look up') + parser_read_value.add_argument('valuenames', help='Name of the value/s to look up (separated by commas, no spaces)') parser_read_value.add_argument('pkg', nargs='*', help='Runtime package name to look up') parser_read_value.add_argument('-f', '--file', help='Read package names from the specified file (one per line, first field only)') parser_read_value.add_argument('-n', '--prefix-name', help='Prefix output with package name', action='store_true') -- 2.17.1 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [OE-core] [PATCH] buildhistory: Add output file listing package information 2021-08-12 16:58 [PATCH] buildhistory: Add output file listing package information Andres Beltran @ 2021-08-13 7:38 ` Mikko Rapeli 2021-08-16 18:22 ` Andres Beltran 2021-08-26 21:33 ` Paul Eggleton 1 sibling, 1 reply; 4+ messages in thread From: Mikko Rapeli @ 2021-08-13 7:38 UTC (permalink / raw) To: abeltran; +Cc: openembedded-core Hi, On Thu, Aug 12, 2021 at 04:58:56PM +0000, Andres Beltran wrote: > Currently, buildhistory does not produce a single file combining relevant > information of installed packages. Produce an output file > "installed-package-info.txt" listing a package's runtime name, buildtime name, > its recipe, version, and size to avoid having to look up each package externally. > Leave the existing package list files as-is for backwards compatibility. Thanks for this patch, I find it useful! I have had similar needs and have created scripts which parse the same info from buildhistory. What I'd add here is the layer of the recipe. In large projects I have the need to also map binary packages into their maintainers and subprojects, and produce reports, e.g. rootfs usage reports per maintainer/subproject to be checked against an agreed budget. Thus I've used layer and MAINTAINER fields to map binary package to the subprojects and maintainers inside the projects. Would be nice to be able to extend the rootfs binary package reports with this info but a bit of scripting on the buildhistory output is fine too. Cheers, -Mikko > In order to support this efficiently, extend oe-pkgdata-util to accept multiple keys > for its read-value argument. > > Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com> > --- > meta/classes/buildhistory.bbclass | 5 +++++ > scripts/oe-pkgdata-util | 37 +++++++++++++++++++------------ > 2 files changed, 28 insertions(+), 14 deletions(-) > > diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass > index 8a1359acbe..134b56ab71 100644 > --- a/meta/classes/buildhistory.bbclass > +++ b/meta/classes/buildhistory.bbclass > @@ -491,6 +491,11 @@ buildhistory_get_installed() { > cat $1/installed-package-sizes.tmp | awk '{print $2 "\tKiB\t" $1}' | sort -n -r > $1/installed-package-sizes.txt > rm $1/installed-package-sizes.tmp > > + # Produce package info: runtime_name, buildtime_name, recipe, version, size > + oe-pkgdata-util -p ${PKGDATA_DIR} read-value "PACKAGE,PN,PV,PKGSIZE" -n -f $pkgcache > $1/installed-package-info.tmp > + cat $1/installed-package-info.tmp | sort -n -r -k 5 > $1/installed-package-info.txt > + rm $1/installed-package-info.tmp > + > # We're now done with the cache, delete it > rm $pkgcache > > diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util > index 75dd23efa3..c30baaa580 100755 > --- a/scripts/oe-pkgdata-util > +++ b/scripts/oe-pkgdata-util > @@ -171,7 +171,7 @@ def read_value(args): > val = line.split(': ', 1)[1].rstrip() > return val > > - logger.debug("read-value('%s', '%s' '%s')" % (args.pkgdata_dir, args.valuename, packages)) > + logger.debug("read-value('%s', '%s' '%s')" % (args.pkgdata_dir, args.valuenames, packages)) > for package in packages: > pkg_split = package.split('_') > pkg_name = pkg_split[0] > @@ -180,20 +180,29 @@ def read_value(args): > logger.debug(revlink) > if os.path.exists(revlink): > mappedpkg = os.path.basename(os.readlink(revlink)) > - qvar = args.valuename > - value = readvar(revlink, qvar, mappedpkg) > - if qvar == "PKGSIZE": > - # PKGSIZE is now in bytes, but we we want it in KB > - pkgsize = (int(value) + 1024 // 2) // 1024 > - value = "%d" % pkgsize > - if args.unescape: > - import codecs > - # escape_decode() unescapes backslash encodings in byte streams > - value = codecs.escape_decode(bytes(value, "utf-8"))[0].decode("utf-8") > + qvars = args.valuenames > + val_names = qvars.split(',') > + values = [] > + for qvar in val_names: > + if qvar == "PACKAGE": > + value = mappedpkg > + else: > + value = readvar(revlink, qvar, mappedpkg) > + if qvar == "PKGSIZE": > + # PKGSIZE is now in bytes, but we we want it in KB > + pkgsize = (int(value) + 1024 // 2) // 1024 > + value = "%d" % pkgsize > + if args.unescape: > + import codecs > + # escape_decode() unescapes backslash encodings in byte streams > + value = codecs.escape_decode(bytes(value, "utf-8"))[0].decode("utf-8") > + values.append(value) > + > + values_str = ' '.join(values) > if args.prefix_name: > - print('%s %s' % (pkg_name, value)) > + print('%s %s' % (pkg_name, values_str)) > else: > - print(value) > + print(values_str) > else: > logger.debug("revlink %s does not exist", revlink) > > @@ -570,7 +579,7 @@ def main(): > parser_read_value = subparsers.add_parser('read-value', > help='Read any pkgdata value for one or more packages', > description='Reads the named value from the pkgdata files for the specified packages') > - parser_read_value.add_argument('valuename', help='Name of the value to look up') > + parser_read_value.add_argument('valuenames', help='Name of the value/s to look up (separated by commas, no spaces)') > parser_read_value.add_argument('pkg', nargs='*', help='Runtime package name to look up') > parser_read_value.add_argument('-f', '--file', help='Read package names from the specified file (one per line, first field only)') > parser_read_value.add_argument('-n', '--prefix-name', help='Prefix output with package name', action='store_true') > -- > 2.17.1 > > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [OE-core] [PATCH] buildhistory: Add output file listing package information 2021-08-13 7:38 ` [OE-core] " Mikko Rapeli @ 2021-08-16 18:22 ` Andres Beltran 0 siblings, 0 replies; 4+ messages in thread From: Andres Beltran @ 2021-08-16 18:22 UTC (permalink / raw) To: Mikko Rapeli; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 6478 bytes --] Hi Mikko, This patch outputs information that is readily available in the context of the image, i.e. information that can be read directly from pkgdata. The information for the recipe's layer is not readily available, so it would have to be a separate task to add that information (or make it extensible to allow it). Best, -Andres On 8/13/2021 12:38 AM, Mikko Rapeli wrote: > Hi, > > On Thu, Aug 12, 2021 at 04:58:56PM +0000, Andres Beltran wrote: >> Currently, buildhistory does not produce a single file combining relevant >> information of installed packages. Produce an output file >> "installed-package-info.txt" listing a package's runtime name, buildtime name, >> its recipe, version, and size to avoid having to look up each package externally. >> Leave the existing package list files as-is for backwards compatibility. > Thanks for this patch, I find it useful! > > I have had similar needs and have created scripts which parse the same info > from buildhistory. What I'd add here is the layer of the recipe. > > In large projects I have the need to also map binary packages into > their maintainers and subprojects, and produce reports, e.g. rootfs usage > reports per maintainer/subproject to be checked against an agreed budget. > Thus I've used layer and MAINTAINER fields to map binary package to the > subprojects and maintainers inside the projects. > > Would be nice to be able to extend the rootfs binary package reports with > this info but a bit of scripting on the buildhistory output is fine too. > > Cheers, > > -Mikko > >> In order to support this efficiently, extend oe-pkgdata-util to accept multiple keys >> for its read-value argument. >> >> Signed-off-by: Andres Beltran <abeltran@linux.microsoft.com> >> --- >> meta/classes/buildhistory.bbclass | 5 +++++ >> scripts/oe-pkgdata-util | 37 +++++++++++++++++++------------ >> 2 files changed, 28 insertions(+), 14 deletions(-) >> >> diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass >> index 8a1359acbe..134b56ab71 100644 >> --- a/meta/classes/buildhistory.bbclass >> +++ b/meta/classes/buildhistory.bbclass >> @@ -491,6 +491,11 @@ buildhistory_get_installed() { >> cat $1/installed-package-sizes.tmp | awk '{print $2 "\tKiB\t" $1}' | sort -n -r > $1/installed-package-sizes.txt >> rm $1/installed-package-sizes.tmp >> >> + # Produce package info: runtime_name, buildtime_name, recipe, version, size >> + oe-pkgdata-util -p ${PKGDATA_DIR} read-value "PACKAGE,PN,PV,PKGSIZE" -n -f $pkgcache > $1/installed-package-info.tmp >> + cat $1/installed-package-info.tmp | sort -n -r -k 5 > $1/installed-package-info.txt >> + rm $1/installed-package-info.tmp >> + >> # We're now done with the cache, delete it >> rm $pkgcache >> >> diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util >> index 75dd23efa3..c30baaa580 100755 >> --- a/scripts/oe-pkgdata-util >> +++ b/scripts/oe-pkgdata-util >> @@ -171,7 +171,7 @@ def read_value(args): >> val = line.split(': ', 1)[1].rstrip() >> return val >> >> - logger.debug("read-value('%s', '%s' '%s')" % (args.pkgdata_dir, args.valuename, packages)) >> + logger.debug("read-value('%s', '%s' '%s')" % (args.pkgdata_dir, args.valuenames, packages)) >> for package in packages: >> pkg_split = package.split('_') >> pkg_name = pkg_split[0] >> @@ -180,20 +180,29 @@ def read_value(args): >> logger.debug(revlink) >> if os.path.exists(revlink): >> mappedpkg = os.path.basename(os.readlink(revlink)) >> - qvar = args.valuename >> - value = readvar(revlink, qvar, mappedpkg) >> - if qvar == "PKGSIZE": >> - # PKGSIZE is now in bytes, but we we want it in KB >> - pkgsize = (int(value) + 1024 // 2) // 1024 >> - value = "%d" % pkgsize >> - if args.unescape: >> - import codecs >> - # escape_decode() unescapes backslash encodings in byte streams >> - value = codecs.escape_decode(bytes(value, "utf-8"))[0].decode("utf-8") >> + qvars = args.valuenames >> + val_names = qvars.split(',') >> + values = [] >> + for qvar in val_names: >> + if qvar == "PACKAGE": >> + value = mappedpkg >> + else: >> + value = readvar(revlink, qvar, mappedpkg) >> + if qvar == "PKGSIZE": >> + # PKGSIZE is now in bytes, but we we want it in KB >> + pkgsize = (int(value) + 1024 // 2) // 1024 >> + value = "%d" % pkgsize >> + if args.unescape: >> + import codecs >> + # escape_decode() unescapes backslash encodings in byte streams >> + value = codecs.escape_decode(bytes(value, "utf-8"))[0].decode("utf-8") >> + values.append(value) >> + >> + values_str = ' '.join(values) >> if args.prefix_name: >> - print('%s %s' % (pkg_name, value)) >> + print('%s %s' % (pkg_name, values_str)) >> else: >> - print(value) >> + print(values_str) >> else: >> logger.debug("revlink %s does not exist", revlink) >> >> @@ -570,7 +579,7 @@ def main(): >> parser_read_value = subparsers.add_parser('read-value', >> help='Read any pkgdata value for one or more packages', >> description='Reads the named value from the pkgdata files for the specified packages') >> - parser_read_value.add_argument('valuename', help='Name of the value to look up') >> + parser_read_value.add_argument('valuenames', help='Name of the value/s to look up (separated by commas, no spaces)') >> parser_read_value.add_argument('pkg', nargs='*', help='Runtime package name to look up') >> parser_read_value.add_argument('-f', '--file', help='Read package names from the specified file (one per line, first field only)') >> parser_read_value.add_argument('-n', '--prefix-name', help='Prefix output with package name', action='store_true') >> -- >> 2.17.1 >> >> >> >> >> >> [-- Attachment #2: Type: text/html, Size: 7248 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [OE-core] [PATCH] buildhistory: Add output file listing package information 2021-08-12 16:58 [PATCH] buildhistory: Add output file listing package information Andres Beltran 2021-08-13 7:38 ` [OE-core] " Mikko Rapeli @ 2021-08-26 21:33 ` Paul Eggleton 1 sibling, 0 replies; 4+ messages in thread From: Paul Eggleton @ 2021-08-26 21:33 UTC (permalink / raw) To: openembedded-core; +Cc: Andres Beltran, Andres Beltran On Friday, 13 August 2021 04:58:56 NZST Andres Beltran wrote: > Currently, buildhistory does not produce a single file combining relevant > information of installed packages. Produce an output file > "installed-package-info.txt" listing a package's runtime name, buildtime > name, its recipe, version, and size to avoid having to look up each package > externally. Leave the existing package list files as-is for backwards > compatibility. > > In order to support this efficiently, extend oe-pkgdata-util to accept > multiple keys for its read-value argument. Acked-by: Paul Eggleton <paul.eggleton@linux.microsoft.com> (FWIW) Cheers Paul ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-08-26 21:33 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-08-12 16:58 [PATCH] buildhistory: Add output file listing package information Andres Beltran 2021-08-13 7:38 ` [OE-core] " Mikko Rapeli 2021-08-16 18:22 ` Andres Beltran 2021-08-26 21:33 ` Paul Eggleton
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox