Add package managers as a plugin

Add package managers as a plugin

[Fredrik Gustafsson]

OE-core today has three different package managers, the well-known formats deb
and rpm is supported as well as ipkg that is good for embedded devices.

When building and having a good cache hit, a significant amount of time is
spent in the phase of generating a rootfs, which is really about the
performance of the package manager. To save build time and also get a
package manager that is suitanle for use on targets where flash memory is a
concern, support for apk is suggested.

However, it might or might not be what's wanted for OE-core since it increases
the test matrix. Therefore I will refactor the package management code to
allow a layer to add a new package manager without editing the meta layer.

This refactor will be divided into multiple patch series to be easier to review,
this first one with a single patch.

[PATCH v3] nopackages.bbclass: Get tasks from variable

[Fredrik Gustafsson]

Get tasks from a variable instead of having them hardcoded. This will
enable other layers to add tasks that should be deleted when nopackages
is used.

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)

diff --git a/meta/classes/nopackages.bbclass b/meta/classes/nopackages.bbclass
index 559f5078bd..4c9fe510c5 100644
--- a/meta/classes/nopackages.bbclass
+++ b/meta/classes/nopackages.bbclass
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}
-- 
2.20.1

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

[Paul Barker]

On Wed, 1 Jul 2020 at 09:46, Fredrik Gustafsson
<fredrik.gustafsson@axis.com> wrote:
>
> Get tasks from a variable instead of having them hardcoded. This will
> enable other layers to add tasks that should be deleted when nopackages
> is used.
>
> Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
> ---
>  meta/classes/nopackages.bbclass | 19 +++++++------------
>  1 file changed, 7 insertions(+), 12 deletions(-)
>
> diff --git a/meta/classes/nopackages.bbclass b/meta/classes/nopackages.bbclass
> index 559f5078bd..4c9fe510c5 100644
> --- a/meta/classes/nopackages.bbclass
> +++ b/meta/classes/nopackages.bbclass
> @@ -1,12 +1,7 @@
> -deltask do_package
> -deltask do_package_write_rpm
> -deltask do_package_write_ipk
> -deltask do_package_write_deb
> -deltask do_package_qa
> -deltask do_packagedata
> -deltask do_package_setscene
> -deltask do_package_write_rpm_setscene
> -deltask do_package_write_ipk_setscene
> -deltask do_package_write_deb_setscene
> -deltask do_package_qa_setscene
> -deltask do_packagedata_setscene
> +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"

Not sure if `+=` was present in the previous versions, I only just
noticed it now. I think `?=` is a better choice so it can be
overridden easily. Other than that, this looks good.

> +
> +python () {
> +    list = d.getVar("NO_PACKAGES_DELTASKS").split()
> +    for task in list:
> +        bb.build.deltask(task, d)
> +}
> --
> 2.20.1


-- 
Paul Barker
Konsulko Group

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

[Richard Purdie]

On Wed, 2020-07-01 at 10:48 +0100, Paul Barker wrote:
> On Wed, 1 Jul 2020 at 09:46, Fredrik Gustafsson
> <fredrik.gustafsson@axis.com> wrote:
> > Get tasks from a variable instead of having them hardcoded. This
> > will
> > enable other layers to add tasks that should be deleted when
> > nopackages
> > is used.
> > 
> > Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
> > ---
> >  meta/classes/nopackages.bbclass | 19 +++++++------------
> >  1 file changed, 7 insertions(+), 12 deletions(-)
> > 
> > diff --git a/meta/classes/nopackages.bbclass
> > b/meta/classes/nopackages.bbclass
> > index 559f5078bd..4c9fe510c5 100644
> > --- a/meta/classes/nopackages.bbclass
> > +++ b/meta/classes/nopackages.bbclass
> > @@ -1,12 +1,7 @@
> > -deltask do_package
> > -deltask do_package_write_rpm
> > -deltask do_package_write_ipk
> > -deltask do_package_write_deb
> > -deltask do_package_qa
> > -deltask do_packagedata
> > -deltask do_package_setscene
> > -deltask do_package_write_rpm_setscene
> > -deltask do_package_write_ipk_setscene
> > -deltask do_package_write_deb_setscene
> > -deltask do_package_qa_setscene
> > -deltask do_packagedata_setscene
> > +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
> > do_package_write_ipk do_package_write_deb do_package_qa
> > do_packagedata do_package_setscene do_package_write_rpm_setscene
> > do_package_write_ipk_setscene do_package_write_deb_setscene
> > do_package_qa_setscene do_packagedata_setscene"
> 
> Not sure if `+=` was present in the previous versions, I only just
> noticed it now. I think `?=` is a better choice so it can be
> overridden easily. Other than that, this looks good.
> 
> > +
> > +python () {
> > +    list = d.getVar("NO_PACKAGES_DELTASKS").split()
> > +    for task in list:
> > +        bb.build.deltask(task, d)
> > +}

I'm just sad this is having to use anonymous python. Its slow and I'd
like to see less of it, not more, given people's complaints about
parsing speed.

I did make a suggestion about how this could be done but that isn't an
option. I might sort out such a patch.

Cheers,

Richard

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

[Richard Purdie]

On Wed, 2020-07-01 at 10:54 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Wed, 2020-07-01 at 10:48 +0100, Paul Barker wrote:
> > On Wed, 1 Jul 2020 at 09:46, Fredrik Gustafsson
> > <fredrik.gustafsson@axis.com> wrote:
> > > Get tasks from a variable instead of having them hardcoded. This
> > > will
> > > enable other layers to add tasks that should be deleted when
> > > nopackages
> > > is used.
> > > 
> > > Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
> > > ---
> > >  meta/classes/nopackages.bbclass | 19 +++++++------------
> > >  1 file changed, 7 insertions(+), 12 deletions(-)
> > > 
> > > diff --git a/meta/classes/nopackages.bbclass
> > > b/meta/classes/nopackages.bbclass
> > > index 559f5078bd..4c9fe510c5 100644
> > > --- a/meta/classes/nopackages.bbclass
> > > +++ b/meta/classes/nopackages.bbclass
> > > @@ -1,12 +1,7 @@
> > > -deltask do_package
> > > -deltask do_package_write_rpm
> > > -deltask do_package_write_ipk
> > > -deltask do_package_write_deb
> > > -deltask do_package_qa
> > > -deltask do_packagedata
> > > -deltask do_package_setscene
> > > -deltask do_package_write_rpm_setscene
> > > -deltask do_package_write_ipk_setscene
> > > -deltask do_package_write_deb_setscene
> > > -deltask do_package_qa_setscene
> > > -deltask do_packagedata_setscene
> > > +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
> > > do_package_write_ipk do_package_write_deb do_package_qa
> > > do_packagedata do_package_setscene do_package_write_rpm_setscene
> > > do_package_write_ipk_setscene do_package_write_deb_setscene
> > > do_package_qa_setscene do_packagedata_setscene"
> > 
> > Not sure if `+=` was present in the previous versions, I only just
> > noticed it now. I think `?=` is a better choice so it can be
> > overridden easily. Other than that, this looks good.
> > 
> > > +
> > > +python () {
> > > +    list = d.getVar("NO_PACKAGES_DELTASKS").split()
> > > +    for task in list:
> > > +        bb.build.deltask(task, d)
> > > +}
> 
> I'm just sad this is having to use anonymous python. Its slow and I'd
> like to see less of it, not more, given people's complaints about
> parsing speed.
> 
> I did make a suggestion about how this could be done but that isn't
> an option. I might sort out such a patch.

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46a01aef15b19431bd9f2924537723481e0151e5

is a patch which lets deltask become a bit more flexible. I've not
tested it much but see it if works/helps.

Cheers,

Richard

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

[Richard Purdie]

On Wed, 2020-07-01 at 11:11 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Wed, 2020-07-01 at 10:54 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > On Wed, 2020-07-01 at 10:48 +0100, Paul Barker wrote:
> > > On Wed, 1 Jul 2020 at 09:46, Fredrik Gustafsson
> > > <fredrik.gustafsson@axis.com> wrote:
> > > > Get tasks from a variable instead of having them hardcoded.
> > > > This
> > > > will
> > > > enable other layers to add tasks that should be deleted when
> > > > nopackages
> > > > is used.
> > > > 
> > > > Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
> > > > ---
> > > >  meta/classes/nopackages.bbclass | 19 +++++++------------
> > > >  1 file changed, 7 insertions(+), 12 deletions(-)
> > > > 
> > > > diff --git a/meta/classes/nopackages.bbclass
> > > > b/meta/classes/nopackages.bbclass
> > > > index 559f5078bd..4c9fe510c5 100644
> > > > --- a/meta/classes/nopackages.bbclass
> > > > +++ b/meta/classes/nopackages.bbclass
> > > > @@ -1,12 +1,7 @@
> > > > -deltask do_package
> > > > -deltask do_package_write_rpm
> > > > -deltask do_package_write_ipk
> > > > -deltask do_package_write_deb
> > > > -deltask do_package_qa
> > > > -deltask do_packagedata
> > > > -deltask do_package_setscene
> > > > -deltask do_package_write_rpm_setscene
> > > > -deltask do_package_write_ipk_setscene
> > > > -deltask do_package_write_deb_setscene
> > > > -deltask do_package_qa_setscene
> > > > -deltask do_packagedata_setscene
> > > > +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
> > > > do_package_write_ipk do_package_write_deb do_package_qa
> > > > do_packagedata do_package_setscene
> > > > do_package_write_rpm_setscene
> > > > do_package_write_ipk_setscene do_package_write_deb_setscene
> > > > do_package_qa_setscene do_packagedata_setscene"
> > > 
> > > Not sure if `+=` was present in the previous versions, I only
> > > just
> > > noticed it now. I think `?=` is a better choice so it can be
> > > overridden easily. Other than that, this looks good.
> > > 
> > > > +
> > > > +python () {
> > > > +    list = d.getVar("NO_PACKAGES_DELTASKS").split()
> > > > +    for task in list:
> > > > +        bb.build.deltask(task, d)
> > > > +}
> > 
> > I'm just sad this is having to use anonymous python. Its slow and
> > I'd
> > like to see less of it, not more, given people's complaints about
> > parsing speed.
> > 
> > I did make a suggestion about how this could be done but that isn't
> > an option. I might sort out such a patch.
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46a01aef15b19431bd9f2924537723481e0151e5
> 
> is a patch which lets deltask become a bit more flexible. I've not
> tested it much but see it if works/helps.

Sorry:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=fd24e59fc886e6642bc3c8fb2941a37a1cc4042a

Cheers,

Richard (in need of caffeine)

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

[Paul Barker]

On Wed, 1 Jul 2020 at 11:12, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2020-07-01 at 11:11 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > On Wed, 2020-07-01 at 10:54 +0100, Richard Purdie via
> > lists.openembedded.org wrote:
> > > On Wed, 2020-07-01 at 10:48 +0100, Paul Barker wrote:
> > > > On Wed, 1 Jul 2020 at 09:46, Fredrik Gustafsson
> > > > <fredrik.gustafsson@axis.com> wrote:
> > > > > Get tasks from a variable instead of having them hardcoded.
> > > > > This
> > > > > will
> > > > > enable other layers to add tasks that should be deleted when
> > > > > nopackages
> > > > > is used.
> > > > >
> > > > > Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
> > > > > ---
> > > > >  meta/classes/nopackages.bbclass | 19 +++++++------------
> > > > >  1 file changed, 7 insertions(+), 12 deletions(-)
> > > > >
> > > > > diff --git a/meta/classes/nopackages.bbclass
> > > > > b/meta/classes/nopackages.bbclass
> > > > > index 559f5078bd..4c9fe510c5 100644
> > > > > --- a/meta/classes/nopackages.bbclass
> > > > > +++ b/meta/classes/nopackages.bbclass
> > > > > @@ -1,12 +1,7 @@
> > > > > -deltask do_package
> > > > > -deltask do_package_write_rpm
> > > > > -deltask do_package_write_ipk
> > > > > -deltask do_package_write_deb
> > > > > -deltask do_package_qa
> > > > > -deltask do_packagedata
> > > > > -deltask do_package_setscene
> > > > > -deltask do_package_write_rpm_setscene
> > > > > -deltask do_package_write_ipk_setscene
> > > > > -deltask do_package_write_deb_setscene
> > > > > -deltask do_package_qa_setscene
> > > > > -deltask do_packagedata_setscene
> > > > > +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
> > > > > do_package_write_ipk do_package_write_deb do_package_qa
> > > > > do_packagedata do_package_setscene
> > > > > do_package_write_rpm_setscene
> > > > > do_package_write_ipk_setscene do_package_write_deb_setscene
> > > > > do_package_qa_setscene do_packagedata_setscene"
> > > >
> > > > Not sure if `+=` was present in the previous versions, I only
> > > > just
> > > > noticed it now. I think `?=` is a better choice so it can be
> > > > overridden easily. Other than that, this looks good.
> > > >
> > > > > +
> > > > > +python () {
> > > > > +    list = d.getVar("NO_PACKAGES_DELTASKS").split()
> > > > > +    for task in list:
> > > > > +        bb.build.deltask(task, d)
> > > > > +}
> > >
> > > I'm just sad this is having to use anonymous python. Its slow and
> > > I'd
> > > like to see less of it, not more, given people's complaints about
> > > parsing speed.
> > >
> > > I did make a suggestion about how this could be done but that isn't
> > > an option. I might sort out such a patch.
> >
> > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46a01aef15b19431bd9f2924537723481e0151e5
> >
> > is a patch which lets deltask become a bit more flexible. I've not
> > tested it much but see it if works/helps.
>
> Sorry:
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=fd24e59fc886e6642bc3c8fb2941a37a1cc4042a
>
> Cheers,
>
> Richard (in need of caffeine)

I like this approach.

Fredrik, what's the reason for using a variable here again? Is it just
to be able to add tasks to the list or do you also need the ability to
remove tasks from the list?

Thanks,

-- 
Paul Barker
Konsulko Group

Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

[Fredrik Gustafsson]

Paul, it's only for adding tasks to do deltask on. Not preventing current deltasks to be done.

Richard, I tried your suggestion but couldn't understand/judge the implications of changing
-__deltask_regexp__       = re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
+__deltask_regexp__       = re.compile(r"deltask\s+(.+)")

If you're fine with that change. Your solution works fine for my usecase, EXCEPT that
+EXTRA_NOPACKAGE_DELTASKS = ""
needs to be
+EXTRA_NOPACKAGE_DELTASKS ?= ""
right? Otherwise that variable would always be empty. Or would something like
EXTRA_NOPACKAGE_DELTASKS_append(....) be used?

/Fredrik
________________________________________
From: Paul Barker <pbarker@konsulko.com>
Sent: Wednesday, July 1, 2020 12:22 PM
To: Richard Purdie
Cc: Fredrik Gustafsson; openembedded-core; tools-cfpbuild-internal; Fredrik Gustafsson
Subject: Re: [OE-core] [PATCH v3] nopackages.bbclass: Get tasks from variable

On Wed, 1 Jul 2020 at 11:12, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2020-07-01 at 11:11 +0100, Richard Purdie via
> lists.openembedded.org wrote:
> > On Wed, 2020-07-01 at 10:54 +0100, Richard Purdie via
> > lists.openembedded.org wrote:
> > > On Wed, 2020-07-01 at 10:48 +0100, Paul Barker wrote:
> > > > On Wed, 1 Jul 2020 at 09:46, Fredrik Gustafsson
> > > > <fredrik.gustafsson@axis.com> wrote:
> > > > > Get tasks from a variable instead of having them hardcoded.
> > > > > This
> > > > > will
> > > > > enable other layers to add tasks that should be deleted when
> > > > > nopackages
> > > > > is used.
> > > > >
> > > > > Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
> > > > > ---
> > > > >  meta/classes/nopackages.bbclass | 19 +++++++------------
> > > > >  1 file changed, 7 insertions(+), 12 deletions(-)
> > > > >
> > > > > diff --git a/meta/classes/nopackages.bbclass
> > > > > b/meta/classes/nopackages.bbclass
> > > > > index 559f5078bd..4c9fe510c5 100644
> > > > > --- a/meta/classes/nopackages.bbclass
> > > > > +++ b/meta/classes/nopackages.bbclass
> > > > > @@ -1,12 +1,7 @@
> > > > > -deltask do_package
> > > > > -deltask do_package_write_rpm
> > > > > -deltask do_package_write_ipk
> > > > > -deltask do_package_write_deb
> > > > > -deltask do_package_qa
> > > > > -deltask do_packagedata
> > > > > -deltask do_package_setscene
> > > > > -deltask do_package_write_rpm_setscene
> > > > > -deltask do_package_write_ipk_setscene
> > > > > -deltask do_package_write_deb_setscene
> > > > > -deltask do_package_qa_setscene
> > > > > -deltask do_packagedata_setscene
> > > > > +NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm
> > > > > do_package_write_ipk do_package_write_deb do_package_qa
> > > > > do_packagedata do_package_setscene
> > > > > do_package_write_rpm_setscene
> > > > > do_package_write_ipk_setscene do_package_write_deb_setscene
> > > > > do_package_qa_setscene do_packagedata_setscene"
> > > >
> > > > Not sure if `+=` was present in the previous versions, I only
> > > > just
> > > > noticed it now. I think `?=` is a better choice so it can be
> > > > overridden easily. Other than that, this looks good.
> > > >
> > > > > +
> > > > > +python () {
> > > > > +    list = d.getVar("NO_PACKAGES_DELTASKS").split()
> > > > > +    for task in list:
> > > > > +        bb.build.deltask(task, d)
> > > > > +}
> > >
> > > I'm just sad this is having to use anonymous python. Its slow and
> > > I'd
> > > like to see less of it, not more, given people's complaints about
> > > parsing speed.
> > >
> > > I did make a suggestion about how this could be done but that isn't
> > > an option. I might sort out such a patch.
> >
> > http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=46a01aef15b19431bd9f2924537723481e0151e5
> >
> > is a patch which lets deltask become a bit more flexible. I've not
> > tested it much but see it if works/helps.
>
> Sorry:
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=fd24e59fc886e6642bc3c8fb2941a37a1cc4042a
>
> Cheers,
>
> Richard (in need of caffeine)

I like this approach.

Fredrik, what's the reason for using a variable here again? Is it just
to be able to add tasks to the list or do you also need the ability to
remove tasks from the list?

Thanks,

--
Paul Barker
Konsulko Group

Re: [PATCH v3] nopackages.bbclass: Get tasks from variable

[Fredrik Gustafsson]

Hi Richard,
is there something I can do to make this move forward?

/Fredrik
________________________________________
From: Fredrik Gustafsson
Sent: Wednesday, July 1, 2020 10:46 AM
To: openembedded-core@lists.openembedded.org
Cc: tools-cfpbuild-internal; Fredrik Gustafsson
Subject: [PATCH v3] nopackages.bbclass: Get tasks from variable

Get tasks from a variable instead of having them hardcoded. This will
enable other layers to add tasks that should be deleted when nopackages
is used.

Signed-off-by: Fredrik Gustafsson <fredrigu@axis.com>
---
 meta/classes/nopackages.bbclass | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)

diff --git a/meta/classes/nopackages.bbclass b/meta/classes/nopackages.bbclass
index 559f5078bd..4c9fe510c5 100644
--- a/meta/classes/nopackages.bbclass
+++ b/meta/classes/nopackages.bbclass
@@ -1,12 +1,7 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_DELTASKS += "do_package do_package_write_rpm do_package_write_ipk do_package_write_deb do_package_qa do_packagedata do_package_setscene do_package_write_rpm_setscene do_package_write_ipk_setscene do_package_write_deb_setscene do_package_qa_setscene do_packagedata_setscene"
+
+python () {
+    list = d.getVar("NO_PACKAGES_DELTASKS").split()
+    for task in list:
+        bb.build.deltask(task, d)
+}
--
2.20.1