selinux_init_load_policy() and systemd switch-root

selinux_init_load_policy() and systemd switch-root

[Will Woods]

Hey all. I have a few questions for you! 

* Is it OK to load policy more than once? (e.g. load new policy after 
  chroot/switch-root to new system)

* If so (and I'm pretty sure that's OK), are we supposed to call
  selinux_init_load_policy for each switch-root, or is
  selinux_mkload_policy supposed to be sufficient?

* What should happen if we switch-root from a system with SELinux
  enabled/permissive to one that has SELINUX=disabled? 

Background on the problem follows. It's optional reading; sorry there's
so much of it.


So, I'm working on major-version upgrades in Fedora/RHEL (and, broadly,
anything that uses systemd + SELinux).

The important (and unusual) thing about upgrades is that they involve
*two* switch-roots - the normal one from initramfs to real root, and a
second one to go from real root to "upgrade.img"[1], which does the
upgrade itself.

So upgrades have *three* roots: initramfs, real root, and upgrade.img.
And initramfs/upgrade.img contain the new release's tools/policy/etc.

The problem: Currently, we try to load policy during initramfs. This
turns out to be a bad idea: nothing in initramfs is labeled, every
process ends up with kernel_t, and so files created in /run or /dev may
have the wrong labels.

Also, on systems where the admin has chosen to remove SELinux, systemd
freezes when it switches to the real root, because a) enforcing is on,
but b) policy can't be loaded.

So a saner plan would seem to be to let the system load policy as normal
(i.e.: do nothing during initramfs, load policy when entering real
root), then load the *new* policy when switching to upgrade.img.

But: systemd refuses to load policy more than once, for reasons that
seem unrelated to loading policy itself[2].

So: maybe it would make sense to have call selinux_init_load_policy()
after *every* switch-root?

We can't just use selinux_mkload_policy() because (as the comment in
selinux_init_load_policy() points out) we need to do things like reset
the config and re-mount selinuxfs after a chroot.

But it also seems like selinux_init_load_policy() assumes it will only
be called once; a later comment says:

        /*
         * If we failed to disable, SELinux will still be 
         * effectively permissive, because no policy is loaded. 
         * No need to call security_setenforce(0) here.
         */

Except: if we were enforcing in the previous root, and the new root
requests SELINUX=disabled.. then policy *is* loaded, and we *do* need
security_setenforce(0).

So: is that just a bug, or is selinux_init_load_policy() really not
intended to be called more than once? And if that's the case, how should
I re-load config/policy/etc. after a switch-root?

Any help / suggestions / comments / questions / expert guidance is very
welcome here. 

Thanks in advance,

-w

[1] initramfs and upgrade.img are actually the same image in the current
implementation, but that's not really important here.

[2] The original code[3] seems to be avoiding an infinite loop:
  * when systemd v12 starts, it tries to load policy
  * if policy loads OK, systemd restarts
So in *that* case, yes, systemd needed to load policy *only* once.
But these days, selinux_setup() is only called at system startup and
after each switch-root, and it doesn't restart systemd. So that's not a
problem.

[3] http://cgit.freedesktop.org/systemd/systemd/commit?id=c4dcdb9

Re: selinux_init_load_policy() and systemd switch-root

[Stephen Smalley]

On 04/17/2014 05:39 PM, Will Woods wrote:
> Hey all. I have a few questions for you! 
> 
> * Is it OK to load policy more than once? (e.g. load new policy after 
>   chroot/switch-root to new system)

It is certainly ok to load policy any number of times.

> * If so (and I'm pretty sure that's OK), are we supposed to call
>   selinux_init_load_policy for each switch-root, or is
>   selinux_mkload_policy supposed to be sufficient?

As you note below, only selinux_init_load_policy() does all of the
(re-)initialization required for reloading policy from a new root;
selinux_mkload_policy() is only for reloading policy afterward.

> * What should happen if we switch-root from a system with SELinux
>   enabled/permissive to one that has SELINUX=disabled? 

That's not a scenario we've ever considered/supported.  It is not
possible to disable SELinux at runtime (via /sys/fs/selinux/disable)
after policy has been loaded; that's a security feature.

> The problem: Currently, we try to load policy during initramfs. This
> turns out to be a bad idea: nothing in initramfs is labeled, every
> process ends up with kernel_t, and so files created in /run or /dev may
> have the wrong labels.

Yes, this can only truly work if you put the policy in the initramfs (as
in Android).

> So a saner plan would seem to be to let the system load policy as normal
> (i.e.: do nothing during initramfs, load policy when entering real
> root), then load the *new* policy when switching to upgrade.img.

Where does upgrade.img live?  Are you switching to the real root to get
access to it initially, mounting it, and then switching to it?

> But: systemd refuses to load policy more than once, for reasons that
> seem unrelated to loading policy itself[2].

Yes, we only ever considered/supported the scenario of one "initial"
policy load followed by subsequent reloads with an already established
root.  selinux_init_load_policy is actually what used to be directly in
the init program code, ripped out of it and taken to libselinux.

> So: maybe it would make sense to have call selinux_init_load_policy()
> after *every* switch-root?

Possibly, although obviously that hasn't been tested.

> We can't just use selinux_mkload_policy() because (as the comment in
> selinux_init_load_policy() points out) we need to do things like reset
> the config and re-mount selinuxfs after a chroot.

Agreed.

> But it also seems like selinux_init_load_policy() assumes it will only
> be called once; a later comment says:
> 
>         /*
>          * If we failed to disable, SELinux will still be 
>          * effectively permissive, because no policy is loaded. 
>          * No need to call security_setenforce(0) here.
>          */
> 
> Except: if we were enforcing in the previous root, and the new root
> requests SELINUX=disabled.. then policy *is* loaded, and we *do* need
> security_setenforce(0).
> 
> So: is that just a bug, or is selinux_init_load_policy() really not
> intended to be called more than once? And if that's the case, how should
> I re-load config/policy/etc. after a switch-root?

I guess it is a bug, but note that you cannot truly disable SELinux at
this point.  So while we could trivially add a security_setenforce(0)
call here, and that seems harmless, SELinux won't be disabled at that
point, just permissive and operating under the real root policy?  In
which case newly created files will be labeled in accordance with that
policy, and non-permission-related SELinux failures can still occur.

> Any help / suggestions / comments / questions / expert guidance is very
> welcome here. 
> 
> Thanks in advance,
> 
> -w
> 
> [1] initramfs and upgrade.img are actually the same image in the current
> implementation, but that's not really important here.

Oh, in that case, why do we need to switch to the real root before
switching to upgrade.img?  Why can't we just switch to upgrade.img directly?

> [2] The original code[3] seems to be avoiding an infinite loop:
>   * when systemd v12 starts, it tries to load policy
>   * if policy loads OK, systemd restarts
> So in *that* case, yes, systemd needed to load policy *only* once.
> But these days, selinux_setup() is only called at system startup and
> after each switch-root, and it doesn't restart systemd. So that's not a
> problem.
> 
> [3] http://cgit.freedesktop.org/systemd/systemd/commit?id=c4dcdb9

The original init model was to load policy and then re-exec so that init
would transition into the correct domain, but I think systemd (and
definitely Android init) now use setcon() to switch into the correct
security context directly and do not re-exec.  So, yes, we likely do not
need to restart systemd after policy load.

Re: selinux_init_load_policy() and systemd switch-root

[Stephen Smalley]

On 04/18/2014 08:53 AM, Stephen Smalley wrote:
> On 04/17/2014 05:39 PM, Will Woods wrote:
>> Hey all. I have a few questions for you! 
>>
>> * Is it OK to load policy more than once? (e.g. load new policy after 
>>   chroot/switch-root to new system)
> 
> It is certainly ok to load policy any number of times.
> 
>> * If so (and I'm pretty sure that's OK), are we supposed to call
>>   selinux_init_load_policy for each switch-root, or is
>>   selinux_mkload_policy supposed to be sufficient?
> 
> As you note below, only selinux_init_load_policy() does all of the
> (re-)initialization required for reloading policy from a new root;
> selinux_mkload_policy() is only for reloading policy afterward.
> 
>> * What should happen if we switch-root from a system with SELinux
>>   enabled/permissive to one that has SELINUX=disabled? 
> 
> That's not a scenario we've ever considered/supported.  It is not
> possible to disable SELinux at runtime (via /sys/fs/selinux/disable)
> after policy has been loaded; that's a security feature.
> 
>> The problem: Currently, we try to load policy during initramfs. This
>> turns out to be a bad idea: nothing in initramfs is labeled, every
>> process ends up with kernel_t, and so files created in /run or /dev may
>> have the wrong labels.
> 
> Yes, this can only truly work if you put the policy in the initramfs (as
> in Android).
> 
>> So a saner plan would seem to be to let the system load policy as normal
>> (i.e.: do nothing during initramfs, load policy when entering real
>> root), then load the *new* policy when switching to upgrade.img.
> 
> Where does upgrade.img live?  Are you switching to the real root to get
> access to it initially, mounting it, and then switching to it?
> 
>> But: systemd refuses to load policy more than once, for reasons that
>> seem unrelated to loading policy itself[2].
> 
> Yes, we only ever considered/supported the scenario of one "initial"
> policy load followed by subsequent reloads with an already established
> root.  selinux_init_load_policy is actually what used to be directly in
> the init program code, ripped out of it and taken to libselinux.
> 
>> So: maybe it would make sense to have call selinux_init_load_policy()
>> after *every* switch-root?
> 
> Possibly, although obviously that hasn't been tested.
> 
>> We can't just use selinux_mkload_policy() because (as the comment in
>> selinux_init_load_policy() points out) we need to do things like reset
>> the config and re-mount selinuxfs after a chroot.
> 
> Agreed.
> 
>> But it also seems like selinux_init_load_policy() assumes it will only
>> be called once; a later comment says:
>>
>>         /*
>>          * If we failed to disable, SELinux will still be 
>>          * effectively permissive, because no policy is loaded. 
>>          * No need to call security_setenforce(0) here.
>>          */
>>
>> Except: if we were enforcing in the previous root, and the new root
>> requests SELINUX=disabled.. then policy *is* loaded, and we *do* need
>> security_setenforce(0).
>>
>> So: is that just a bug, or is selinux_init_load_policy() really not
>> intended to be called more than once? And if that's the case, how should
>> I re-load config/policy/etc. after a switch-root?
> 
> I guess it is a bug, but note that you cannot truly disable SELinux at
> this point.  So while we could trivially add a security_setenforce(0)
> call here, and that seems harmless, SELinux won't be disabled at that
> point, just permissive and operating under the real root policy?  In
> which case newly created files will be labeled in accordance with that
> policy, and non-permission-related SELinux failures can still occur.
> 
>> Any help / suggestions / comments / questions / expert guidance is very
>> welcome here. 
>>
>> Thanks in advance,
>>
>> -w
>>
>> [1] initramfs and upgrade.img are actually the same image in the current
>> implementation, but that's not really important here.
> 
> Oh, in that case, why do we need to switch to the real root before
> switching to upgrade.img?  Why can't we just switch to upgrade.img directly?
> 
>> [2] The original code[3] seems to be avoiding an infinite loop:
>>   * when systemd v12 starts, it tries to load policy
>>   * if policy loads OK, systemd restarts
>> So in *that* case, yes, systemd needed to load policy *only* once.
>> But these days, selinux_setup() is only called at system startup and
>> after each switch-root, and it doesn't restart systemd. So that's not a
>> problem.
>>
>> [3] http://cgit.freedesktop.org/systemd/systemd/commit?id=c4dcdb9
> 
> The original init model was to load policy and then re-exec so that init
> would transition into the correct domain, but I think systemd (and
> definitely Android init) now use setcon() to switch into the correct
> security context directly and do not re-exec.  So, yes, we likely do not
> need to restart systemd after policy load.

Note however that there may be some policy implications there, e.g. init
may need kernel_t:fd use if it doesn't already have it to continue using
already open file descriptors originally opened while running in kernel_t.

Re: selinux_init_load_policy() and systemd switch-root

[Will Woods]

On Fri, 2014-04-18 at 08:53 -0400, Stephen Smalley wrote:
> On 04/17/2014 05:39 PM, Will Woods wrote:
> > * What should happen if we switch-root from a system with SELinux
> >   enabled/permissive to one that has SELINUX=disabled? 
> 
> That's not a scenario we've ever considered/supported.  It is not
> possible to disable SELinux at runtime (via /sys/fs/selinux/disable)
> after policy has been loaded; that's a security feature.

Indeed. I feel like "We can't disable SELinux, so just log a warning and
security_setenforce(0)" is a sensible way to handle that error, but I
wanted to see if the Collected Wisdom Of The List would agree with that.

> > The problem: Currently, we try to load policy during initramfs. This
> > turns out to be a bad idea: nothing in initramfs is labeled, every
> > process ends up with kernel_t, and so files created in /run or /dev may
> > have the wrong labels.
> 
> Yes, this can only truly work if you put the policy in the initramfs (as
> in Android).

Tangential, but: I assume you would have to relabel your initramfs after
policy is loaded? Or use an initrd[1], and a filesystem that supports
labels?

> > Except: if we were enforcing in the previous root, and the new root
> > requests SELINUX=disabled.. then policy *is* loaded, and we *do* need
> > security_setenforce(0).
> > 
> > So: is that just a bug, or is selinux_init_load_policy() really not
> > intended to be called more than once? And if that's the case, how should
> > I re-load config/policy/etc. after a switch-root?
> 
> I guess it is a bug, but note that you cannot truly disable SELinux at
> this point.  

Right, that's what the (quite thorough!) docs told me as well.

> So while we could trivially add a security_setenforce(0)
> call here, and that seems harmless, SELinux won't be disabled at that
> point, just permissive and operating under the real root policy?

As above, this seems like a reasonable course of action in this case.

> In which case newly created files will be labeled in accordance with
> that policy, and non-permission-related SELinux failures can still
> occur.

True, but I'm guessing this should amount mostly to noise; hopefully
anything that's checking to see if SELinux is *permanently* disabled is
using selinux_getenforcemode() or checking for the presence of policy or
something.

(And, in my inexpert opinion, people who disable SELinux deserve a
little noise now and then anyway. Heh.)

> > [1] initramfs and upgrade.img are actually the same image in the current
> > implementation, but that's not really important here.
> 
> Oh, in that case, why do we need to switch to the real root before
> switching to upgrade.img?  Why can't we just switch to upgrade.img directly?

Because we want the existing system to mount its disks for us. It turns
out to be very hard to examine a system from the outside and *correctly
and reliably* determine how it would mount its disks[2].

But obviously the system already knows how to mount its disks. So why
not let the system do that for us?

And indeed, in practice it's been much simpler[3] and more reliable to
just let the system do its initial setup as normal (i.e. sysinit.target
or /etc/rc.d/rc.sysinit) and then switch out to the upgrade.img.

Anyway - I may have a patch for selinux_init_load_policy() shortly, to
make sure it tries to do security_setenforce(0) if security_disable()
fails.

Thanks for the info,

-w

[1] reminder that initrd and initramfs are different things:
initramfs: cpio archive loaded into tmpfs (dynamic size, swappable)
initrd: filesystem image loaded into a ramdisk (fixed size, no swap)

[2] it's true that you could get it *mostly* right *some* of the time by
just parsing /etc/fstab. But what about systemd .mount units? What about
fancy RAID setups (e.g. for /var) that require mdadm.conf? What about
fancy LVM setups that require lvm.conf? What about multipath? What about
cryptsetup? &c. &c. Trust me, it's kind of a nightmare.

[3] Aside from the weird intricacies of the double-switch-root, which
are admittedly weird and tricky but far less complicated than dealing
with All The Storage Mechanisms That Ever Were Or Shall Be.

If someone wanted to design and enforce the use of some kind of
statically-analyzable storage configuration I'd be *delighted* to drop
the double switch-root. Until then, though...