* [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer
@ 2017-05-19 15:01 Haris Okanovic
2017-05-19 17:29 ` Martin Jansa
2017-08-31 18:46 ` Haris Okanovic
0 siblings, 2 replies; 6+ messages in thread
From: Haris Okanovic @ 2017-05-19 15:01 UTC (permalink / raw)
To: openembedded-core; +Cc: haris.okanovic
Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
`bitbake package-index`, which is index of files provided by each
IPK package in the feed. It's useful for figuring out which package
provides a particular file/program/library/etc.
Disabled by default since generating a filelist involves reading the
payload of every package in the feed, a time and IO intensive operation
many users won't want to run. Those who do may flip this switch.
Testing:
* Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
no Packages.filelist are generated.
* Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
has Packages.filelist; took ~3min longer for 8,200 IPKs.
Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
---
meta/lib/oe/package_manager.py | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index b4b359a8c6..5e1fc48500 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -161,6 +161,8 @@ class OpkgIndexer(Indexer):
else:
signer = None
+ enable_filelist = bb.utils.to_boolean(self.d.getVar('PACKAGE_ENABLE_FILELIST', True) or "False")
+
if not os.path.exists(os.path.join(self.deploy_dir, "Packages")):
open(os.path.join(self.deploy_dir, "Packages"), "w").close()
@@ -175,14 +177,18 @@ class OpkgIndexer(Indexer):
pkgs_dir = os.path.join(self.deploy_dir, arch)
pkgs_file = os.path.join(pkgs_dir, "Packages")
+ filelist_cmd = ""
+ if enable_filelist:
+ filelist_cmd = '-l %s.filelist' % (pkgs_file)
+
if not os.path.isdir(pkgs_dir):
continue
if not os.path.exists(pkgs_file):
open(pkgs_file, "w").close()
- index_cmds.add('%s -r %s -p %s -m %s' %
- (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir))
+ index_cmds.add('%s -r %s -p %s -m %s %s' %
+ (opkg_index_cmd, pkgs_file, pkgs_file, filelist_cmd, pkgs_dir))
index_sign_files.add(pkgs_file)
--
2.12.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer
2017-05-19 15:01 [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer Haris Okanovic
@ 2017-05-19 17:29 ` Martin Jansa
2017-05-19 21:34 ` Haris Okanovic
2017-08-31 18:46 ` Haris Okanovic
1 sibling, 1 reply; 6+ messages in thread
From: Martin Jansa @ 2017-05-19 17:29 UTC (permalink / raw)
To: Haris Okanovic; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 2985 bytes --]
On Fri, May 19, 2017 at 10:01:21AM -0500, Haris Okanovic wrote:
> Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
> `bitbake package-index`, which is index of files provided by each
> IPK package in the feed. It's useful for figuring out which package
> provides a particular file/program/library/etc.
>
> Disabled by default since generating a filelist involves reading the
> payload of every package in the feed, a time and IO intensive operation
> many users won't want to run. Those who do may flip this switch.
>
> Testing:
> * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
> no Packages.filelist are generated.
> * Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
> has Packages.filelist; took ~3min longer for 8,200 IPKs.
>
> Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
Why not use files-in-package.txt reports in buildhistory?
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5870
That should work for any package backend and better to generate it once
in one place.
> ---
> meta/lib/oe/package_manager.py | 10 ++++++++--
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
> index b4b359a8c6..5e1fc48500 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -161,6 +161,8 @@ class OpkgIndexer(Indexer):
> else:
> signer = None
>
> + enable_filelist = bb.utils.to_boolean(self.d.getVar('PACKAGE_ENABLE_FILELIST', True) or "False")
> +
> if not os.path.exists(os.path.join(self.deploy_dir, "Packages")):
> open(os.path.join(self.deploy_dir, "Packages"), "w").close()
>
> @@ -175,14 +177,18 @@ class OpkgIndexer(Indexer):
> pkgs_dir = os.path.join(self.deploy_dir, arch)
> pkgs_file = os.path.join(pkgs_dir, "Packages")
>
> + filelist_cmd = ""
> + if enable_filelist:
> + filelist_cmd = '-l %s.filelist' % (pkgs_file)
> +
> if not os.path.isdir(pkgs_dir):
> continue
>
> if not os.path.exists(pkgs_file):
> open(pkgs_file, "w").close()
>
> - index_cmds.add('%s -r %s -p %s -m %s' %
> - (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir))
> + index_cmds.add('%s -r %s -p %s -m %s %s' %
> + (opkg_index_cmd, pkgs_file, pkgs_file, filelist_cmd, pkgs_dir))
>
> index_sign_files.add(pkgs_file)
>
> --
> 2.12.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer
2017-05-19 17:29 ` Martin Jansa
@ 2017-05-19 21:34 ` Haris Okanovic
0 siblings, 0 replies; 6+ messages in thread
From: Haris Okanovic @ 2017-05-19 21:34 UTC (permalink / raw)
To: Martin Jansa; +Cc: Haris Okanovic, openembedded-core
> Why not use files-in-package.txt reports in buildhistory?
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5870
Practical reason: files-in-package.txt is an OE-ism. Our opkg-based
distribution provides software feeds built outside of OE. We use
opkg-utils to generate feed metadata like OE, so it makes sense to reuse
logic instead of rewriting it. Others may be in the same position.
Philosophical reason: I prefer to build the file list (and index) from
actual packages (I.e. exactly what we're shipping) instead of OE's
environment. This makes the file list and index more easily reproducible
and insulates that process from OE bugs.
-- Haris
On 05/19/2017 12:29 PM, Martin Jansa wrote:
> On Fri, May 19, 2017 at 10:01:21AM -0500, Haris Okanovic wrote:
>> Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
>> `bitbake package-index`, which is index of files provided by each
>> IPK package in the feed. It's useful for figuring out which package
>> provides a particular file/program/library/etc.
>>
>> Disabled by default since generating a filelist involves reading the
>> payload of every package in the feed, a time and IO intensive operation
>> many users won't want to run. Those who do may flip this switch.
>>
>> Testing:
>> * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
>> no Packages.filelist are generated.
>> * Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
>> has Packages.filelist; took ~3min longer for 8,200 IPKs.
>>
>> Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
>
> Why not use files-in-package.txt reports in buildhistory?
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5870
>
> That should work for any package backend and better to generate it once
> in one place.
>
>> ---
>> meta/lib/oe/package_manager.py | 10 ++++++++--
>> 1 file changed, 8 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
>> index b4b359a8c6..5e1fc48500 100644
>> --- a/meta/lib/oe/package_manager.py
>> +++ b/meta/lib/oe/package_manager.py
>> @@ -161,6 +161,8 @@ class OpkgIndexer(Indexer):
>> else:
>> signer = None
>>
>> + enable_filelist = bb.utils.to_boolean(self.d.getVar('PACKAGE_ENABLE_FILELIST', True) or "False")
>> +
>> if not os.path.exists(os.path.join(self.deploy_dir, "Packages")):
>> open(os.path.join(self.deploy_dir, "Packages"), "w").close()
>>
>> @@ -175,14 +177,18 @@ class OpkgIndexer(Indexer):
>> pkgs_dir = os.path.join(self.deploy_dir, arch)
>> pkgs_file = os.path.join(pkgs_dir, "Packages")
>>
>> + filelist_cmd = ""
>> + if enable_filelist:
>> + filelist_cmd = '-l %s.filelist' % (pkgs_file)
>> +
>> if not os.path.isdir(pkgs_dir):
>> continue
>>
>> if not os.path.exists(pkgs_file):
>> open(pkgs_file, "w").close()
>>
>> - index_cmds.add('%s -r %s -p %s -m %s' %
>> - (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir))
>> + index_cmds.add('%s -r %s -p %s -m %s %s' %
>> + (opkg_index_cmd, pkgs_file, pkgs_file, filelist_cmd, pkgs_dir))
>>
>> index_sign_files.add(pkgs_file)
>>
>> --
>> 2.12.1
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer
2017-05-19 15:01 [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer Haris Okanovic
2017-05-19 17:29 ` Martin Jansa
@ 2017-08-31 18:46 ` Haris Okanovic
2017-09-01 13:04 ` Alexander Kanavin
1 sibling, 1 reply; 6+ messages in thread
From: Haris Okanovic @ 2017-08-31 18:46 UTC (permalink / raw)
To: openembedded-core; +Cc: haris.okanovic, ken.sharp
Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
`bitbake package-index`, which is index of files provided by each
IPK package in the feed. It's useful for figuring out which package
provides a particular file/program/library/etc.
Disabled by default since generating a filelist involves reading the
payload of every package in the feed, a time and IO intensive operation
many users won't want to run. Those who do may flip this switch.
Testing:
* Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
no Packages.filelist are generated.
* Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
has Packages.filelist; took ~3min longer for 8,200 IPKs.
Signed-off-by: Haris Okanovic <haris.okanovic@ni.com>
---
meta/lib/oe/package_manager.py | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index b4b359a8c6..5e1fc48500 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -161,6 +161,8 @@ class OpkgIndexer(Indexer):
else:
signer = None
+ enable_filelist = bb.utils.to_boolean(self.d.getVar('PACKAGE_ENABLE_FILELIST', True) or "False")
+
if not os.path.exists(os.path.join(self.deploy_dir, "Packages")):
open(os.path.join(self.deploy_dir, "Packages"), "w").close()
@@ -175,14 +177,18 @@ class OpkgIndexer(Indexer):
pkgs_dir = os.path.join(self.deploy_dir, arch)
pkgs_file = os.path.join(pkgs_dir, "Packages")
+ filelist_cmd = ""
+ if enable_filelist:
+ filelist_cmd = '-l %s.filelist' % (pkgs_file)
+
if not os.path.isdir(pkgs_dir):
continue
if not os.path.exists(pkgs_file):
open(pkgs_file, "w").close()
- index_cmds.add('%s -r %s -p %s -m %s' %
- (opkg_index_cmd, pkgs_file, pkgs_file, pkgs_dir))
+ index_cmds.add('%s -r %s -p %s -m %s %s' %
+ (opkg_index_cmd, pkgs_file, pkgs_file, filelist_cmd, pkgs_dir))
index_sign_files.add(pkgs_file)
--
2.13.2
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer
2017-08-31 18:46 ` Haris Okanovic
@ 2017-09-01 13:04 ` Alexander Kanavin
2017-09-01 14:47 ` Richard Purdie
0 siblings, 1 reply; 6+ messages in thread
From: Alexander Kanavin @ 2017-09-01 13:04 UTC (permalink / raw)
To: openembedded-core, haris.okanovic
On 08/31/2017 09:46 PM, Haris Okanovic wrote:
> Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist on
> `bitbake package-index`, which is index of files provided by each
> IPK package in the feed. It's useful for figuring out which package
> provides a particular file/program/library/etc.
>
> Disabled by default since generating a filelist involves reading the
> payload of every package in the feed, a time and IO intensive operation
> many users won't want to run. Those who do may flip this switch.
>
> Testing:
> * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and verified
> no Packages.filelist are generated.
> * Built with PACKAGE_ENABLE_FILELIST="1" and verified each subfeed
> has Packages.filelist; took ~3min longer for 8,200 IPKs.
The only way to discover that this option exists is to read the code for
OpkgIndexer. And it's specific to opkg, which is not reflected in the
option name at all.
Can you instead just place the index generation into a custom recipe
that depends on package-index recipe?
Alex
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer
2017-09-01 13:04 ` Alexander Kanavin
@ 2017-09-01 14:47 ` Richard Purdie
0 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2017-09-01 14:47 UTC (permalink / raw)
To: Alexander Kanavin, openembedded-core, haris.okanovic
On Fri, 2017-09-01 at 16:04 +0300, Alexander Kanavin wrote:
> On 08/31/2017 09:46 PM, Haris Okanovic wrote:
> >
> > Setting PACKAGE_ENABLE_FILELIST option generates Packages.filelist
> > on
> > `bitbake package-index`, which is index of files provided by each
> > IPK package in the feed. It's useful for figuring out which package
> > provides a particular file/program/library/etc.
> >
> > Disabled by default since generating a filelist involves reading
> > the
> > payload of every package in the feed, a time and IO intensive
> > operation
> > many users won't want to run. Those who do may flip this switch.
> >
> > Testing:
> > * Built an opkg index with PACKAGE_ENABLE_FILELIST unset and
> > verified
> > no Packages.filelist are generated.
> > * Built with PACKAGE_ENABLE_FILELIST="1" and verified each
> > subfeed
> > has Packages.filelist; took ~3min longer for 8,200 IPKs.
>
>
> The only way to discover that this option exists is to read the code
> for OpkgIndexer. And it's specific to opkg, which is not reflected in
> the option name at all.
>
> Can you instead just place the index generation into a custom recipe
> that depends on package-index recipe?
I'm going to disagree with that, I don't want a separate recipe for
this and I'm not sure that makes sense.
I'm ok with adding an option but we need to come up with a better
namespace. How about OPKG_PACKAGE_INDEX_FILELIST_GEN?
(says its package index and opkg specific)
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-09-01 14:47 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-05-19 15:01 [PATCH] package_manager.py: Add PACKAGE_ENABLE_FILELIST option to OpkgIndexer Haris Okanovic
2017-05-19 17:29 ` Martin Jansa
2017-05-19 21:34 ` Haris Okanovic
2017-08-31 18:46 ` Haris Okanovic
2017-09-01 13:04 ` Alexander Kanavin
2017-09-01 14:47 ` Richard Purdie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox