[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[Christian Stewart]

Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243.
According to the systemd documentation:

  systemd now defaults to the "unified" cgroup hierarchy setup during
  build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default.
  Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream
  production distributions might want to continue to use
  -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately
  the popular container managers have not caught up with the kernel API changes.

Changing this option to "hybrid" or "legacy" fixes the Docker startup.

Reference: https://github.com/opencontainers/runc/issues/654

Signed-off-by: Christian Stewart <christian@paral.in>
---
 package/systemd/systemd.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk
index 94d5f703cd..92490eb86b 100644
--- a/package/systemd/systemd.mk
+++ b/package/systemd/systemd.mk
@@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \
 	-Dima=false \
 	-Dldconfig=false \
 	-Ddefault-dnssec=no \
+	-Ddefault-hierarchy=hybrid \
 	-Dtests=false \
 	-Dsplit-bin=true \
 	-Dsplit-usr=false \
-- 
2.24.0

[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[James Hilliard]

On Mon, Nov 11, 2019 at 1:33 AM Christian Stewart <christian@paral.in> wrote:
>
> Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243.
> According to the systemd documentation:
>
>   systemd now defaults to the "unified" cgroup hierarchy setup during
>   build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default.
>   Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream
>   production distributions might want to continue to use
>   -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately
>   the popular container managers have not caught up with the kernel API changes.
>
> Changing this option to "hybrid" or "legacy" fixes the Docker startup.
>
> Reference: https://github.com/opencontainers/runc/issues/654
>
> Signed-off-by: Christian Stewart <christian@paral.in>
> ---
>  package/systemd/systemd.mk | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk
> index 94d5f703cd..92490eb86b 100644
> --- a/package/systemd/systemd.mk
> +++ b/package/systemd/systemd.mk
> @@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \
>         -Dima=false \
>         -Dldconfig=false \
>         -Ddefault-dnssec=no \
> +       -Ddefault-hierarchy=hybrid \
This should probably be conditionally enabled only when packages that
don't support the unified cgroup hierarchy are enabled.
>         -Dtests=false \
>         -Dsplit-bin=true \
>         -Dsplit-usr=false \
> --
> 2.24.0
>

[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[Jérémy ROSEN]

Keeping a list like that is going to be tricky...
Let's just hope the pressure on docker is strong enough and in a version or
two we can move to unified
also:
Tested-by: J?r?my Rosen <jeremy.rosen@smile.fr>

Le lun. 11 nov. 2019 ? 10:05, James Hilliard <james.hilliard1@gmail.com> a
?crit :

> On Mon, Nov 11, 2019 at 1:33 AM Christian Stewart <christian@paral.in>
> wrote:
> >
> > Docker fails to start with "Devices cgroup isn't mounted" as of systemd
> 243.
> > According to the systemd documentation:
> >
> >   systemd now defaults to the "unified" cgroup hierarchy setup during
> >   build-time, i.e. -Ddefault-hierarchy=unified is now the build-time
> default.
> >   Previously, -Ddefault-hierarchy=hybrid was the default. [...]
> Downstream
> >   production distributions might want to continue to use
> >   -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as
> unfortunately
> >   the popular container managers have not caught up with the kernel API
> changes.
> >
> > Changing this option to "hybrid" or "legacy" fixes the Docker startup.
> >
> > Reference: https://github.com/opencontainers/runc/issues/654
> >
> > Signed-off-by: Christian Stewart <christian@paral.in>
> > ---
> >  package/systemd/systemd.mk | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk
> > index 94d5f703cd..92490eb86b 100644
> > --- a/package/systemd/systemd.mk
> > +++ b/package/systemd/systemd.mk
> > @@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \
> >         -Dima=false \
> >         -Dldconfig=false \
> >         -Ddefault-dnssec=no \
> > +       -Ddefault-hierarchy=hybrid \
> This should probably be conditionally enabled only when packages that
> don't support the unified cgroup hierarchy are enabled.
> >         -Dtests=false \
> >         -Dsplit-bin=true \
> >         -Dsplit-usr=false \
> > --
> > 2.24.0
> >
>


-- 
[image: SMILE]  <http://www.smile.eu/>

20 rue des Jardins
92600 Asni?res-sur-Seine
*J?r?my ROSEN*
Architecte technique

[image: email] jeremy.rosen at smile.fr
[image: phone]  +33 6 88 25 87 42
[image: url] http://www.smile.eu

[image: Twitter] <https://twitter.com/GroupeSmile> [image: Facebook]
<https://www.facebook.com/smileopensource> [image: LinkedIn]
<https://www.linkedin.com/company/smile> [image: Github]
<https://github.com/Smile-SA>

[image: D?couvrez l?univers Smile, rendez-vous sur smile.eu]
<https://www.smile.eu/fr/publications/livres-blancs/yocto?utm_source=signature&utm_medium=email&utm_campaign=signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191111/5ce8548b/attachment.html>

[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[Thomas Petazzoni]

Hello,

On Mon, 11 Nov 2019 11:09:27 +0100
J?r?my ROSEN <jeremy.rosen@smile.fr> wrote:

> Keeping a list like that is going to be tricky...

You don't really have to keep a list. You can have a blind option:

config BR2_PACKAGE_SYSTEMD_NEEDS_HYBRID_HIERARCHY
	bool

systemd.mk would observe this boolean to enable (or not) the hybrid
hierarchy, and packages that need it would select this boolean.

That being said, is there any drawback in unconditionally using the
hybrid hierarchy? James, since you suggested to make it conditional, do
you have some arguments/feedback?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[Jérémy ROSEN]

It's about what implementation of cgroup is used by systemd to maintain the
cgroup-tree of processes

* legacy is v1
* hybrid is both v1 and v2
* unified is v2 only
(note that only the default is changed, this can be changed via the kernel
command-line at boot time)

v1 is deprecated on the kernel side because it was... a mess.
The systemd people are migrating stuff from v1 to v2 as new controllers are
added to v2, but at this point there is little left that is v1 only thus
the change of the default value

Very few user-space projects care about cgroups : mainly systemd and
container-managers. Container-manager need to interact with systemd as
described in
https://systemd.io/CGROUP_DELEGATION.html which allows to drop v1 and to
correctly spread the responsibility between systemd and the container
managers without
stepping on each other's toes.
Also note that Fedora31 will use the unified cgroup hierarchy.
A good summary of adoption status :
https://medium.com/nttlabs/cgroup-v2-596d035be4d7

At that point, I think staying in hybrid mode is the safest bet :
everything in user-space hasn't migrated yet.
Let Fedora take the fire and force everybody to migrate, and let's revisit
the subject once (at least) Docker has migrated...

Le lun. 11 nov. 2019 ? 15:39, Thomas Petazzoni <thomas.petazzoni@bootlin.com>
a ?crit :

> Hello,
>
> On Mon, 11 Nov 2019 11:09:27 +0100
> J?r?my ROSEN <jeremy.rosen@smile.fr> wrote:
>
> > Keeping a list like that is going to be tricky...
>
> You don't really have to keep a list. You can have a blind option:
>
> config BR2_PACKAGE_SYSTEMD_NEEDS_HYBRID_HIERARCHY
>         bool
>
> systemd.mk would observe this boolean to enable (or not) the hybrid
> hierarchy, and packages that need it would select this boolean.
>
> That being said, is there any drawback in unconditionally using the
> hybrid hierarchy? James, since you suggested to make it conditional, do
> you have some arguments/feedback?
>
> Thanks,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>


-- 
[image: SMILE]  <http://www.smile.eu/>

20 rue des Jardins
92600 Asni?res-sur-Seine
*J?r?my ROSEN*
Architecte technique

[image: email] jeremy.rosen at smile.fr
[image: phone]  +33 6 88 25 87 42
[image: url] http://www.smile.eu

[image: Twitter] <https://twitter.com/GroupeSmile> [image: Facebook]
<https://www.facebook.com/smileopensource> [image: LinkedIn]
<https://www.linkedin.com/company/smile> [image: Github]
<https://github.com/Smile-SA>

[image: D?couvrez l?univers Smile, rendez-vous sur smile.eu]
<https://www.smile.eu/fr/publications/livres-blancs/yocto?utm_source=signature&utm_medium=email&utm_campaign=signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191111/9acb432c/attachment.html>

[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[Thomas Petazzoni]

Hello,

(If possible, you should bottom post instead of top post. Thanks!)

On Mon, 11 Nov 2019 15:56:04 +0100
J?r?my ROSEN <jeremy.rosen@smile.fr> wrote:

> It's about what implementation of cgroup is used by systemd to maintain the
> cgroup-tree of processes
> 
> * legacy is v1
> * hybrid is both v1 and v2
> * unified is v2 only
> (note that only the default is changed, this can be changed via the kernel
> command-line at boot time)
> 
> v1 is deprecated on the kernel side because it was... a mess.
> The systemd people are migrating stuff from v1 to v2 as new controllers are
> added to v2, but at this point there is little left that is v1 only thus
> the change of the default value
> 
> Very few user-space projects care about cgroups : mainly systemd and
> container-managers. Container-manager need to interact with systemd as
> described in
> https://systemd.io/CGROUP_DELEGATION.html which allows to drop v1 and to
> correctly spread the responsibility between systemd and the container
> managers without
> stepping on each other's toes.
> Also note that Fedora31 will use the unified cgroup hierarchy.
> A good summary of adoption status :
> https://medium.com/nttlabs/cgroup-v2-596d035be4d7
> 
> At that point, I think staying in hybrid mode is the safest bet :
> everything in user-space hasn't migrated yet.
> Let Fedora take the fire and force everybody to migrate, and let's revisit
> the subject once (at least) Docker has migrated...

OK, thanks a lot for this explanation, much clearer!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[Thomas Petazzoni]

On Mon, 11 Nov 2019 00:33:48 -0800
Christian Stewart <christian@paral.in> wrote:

> Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243.
> According to the systemd documentation:
> 
>   systemd now defaults to the "unified" cgroup hierarchy setup during
>   build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default.
>   Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream
>   production distributions might want to continue to use
>   -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately
>   the popular container managers have not caught up with the kernel API changes.
> 
> Changing this option to "hybrid" or "legacy" fixes the Docker startup.
> 
> Reference: https://github.com/opencontainers/runc/issues/654
> 
> Signed-off-by: Christian Stewart <christian@paral.in>
> ---
>  package/systemd/systemd.mk | 1 +
>  1 file changed, 1 insertion(+)

Applied to master, thanks. It would be nice to have some runtime tests
around Docker, to verify that it is still working properly at runtime,
so that we can catch this kind of issues.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility

[Christian Stewart]

Hi all,

(re-sending this from the correct email address)

Jeremy ROSEN <jeremy.rosen@smile.fr> writes:
> v1 is deprecated on the kernel side because it was... a mess.
> The systemd people are migrating stuff from v1 to v2 as new controllers a=
re
> added to v2, but at this point there is little left that is v1 only thus
> the change of the default value

Thomas Petazzoni writes:

>> systemd.mk would observe this boolean to enable (or not) the hybrid
>> hierarchy, and packages that need it would select this boolean.
>>
>> That being said, is there any drawback in unconditionally using the
>> hybrid hierarchy? James, since you suggested to make it conditional, do
>> you have some arguments/feedback?

I originally planned to submit this patch with a 3-state option for
which mode is desired (legacy, hybrid, merged) and then having Docker
and other packages depend on hybrid or legacy.

However, I realized that this would be quite a lot of work to then trace
down all of the packages that potentially need the hybrid configuration,
and that in the name of "not breaking userspace" it makes sense for
Buildroot to follow the direction in the systemd NEWS and just default
to "hybrid" for now.

If someone wants to do the above work to make the 3-state option, we
could merge that first, with "hybrid" as the default, and later add
dependencies from Docker -> systemd for that option.

> At that point, I think staying in hybrid mode is the safest bet :
> everything in user-space hasn't migrated yet.
> Let Fedora take the fire and force everybody to migrate, and let's revisit
> the subject once (at least) Docker has migrated...

Agreed, we can probably wait until Docker has finished their migration
to begin seriously considering moving off of "hybrid."

Best regards,
Christian

On Mon, Nov 11, 2019 at 7:29 AM Thomas Petazzoni <
thomas.petazzoni@bootlin.com> wrote:

> On Mon, 11 Nov 2019 00:33:48 -0800
> Christian Stewart <christian@paral.in> wrote:
>
> > Docker fails to start with "Devices cgroup isn't mounted" as of systemd
> 243.
> > According to the systemd documentation:
> >
> >   systemd now defaults to the "unified" cgroup hierarchy setup during
> >   build-time, i.e. -Ddefault-hierarchy=unified is now the build-time
> default.
> >   Previously, -Ddefault-hierarchy=hybrid was the default. [...]
> Downstream
> >   production distributions might want to continue to use
> >   -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as
> unfortunately
> >   the popular container managers have not caught up with the kernel API
> changes.
> >
> > Changing this option to "hybrid" or "legacy" fixes the Docker startup.
> >
> > Reference: https://github.com/opencontainers/runc/issues/654
> >
> > Signed-off-by: Christian Stewart <christian@paral.in>
> > ---
> >  package/systemd/systemd.mk | 1 +
> >  1 file changed, 1 insertion(+)
>
> Applied to master, thanks. It would be nice to have some runtime tests
> around Docker, to verify that it is still working properly at runtime,
> so that we can catch this kind of issues.
>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191111/4c017e02/attachment.html>