WIP: verity support for overlayfs

WIP: verity support for overlayfs

[Alexander Larsson]

As was recently discussed in the various threads about composefs we
want the ability to specify a fs-verity digest for metacopy files,
such that the lower file used for the data is guaranteed to have the
specified digest.

I wrote an initial version of this here:

  https://github.com/alexlarsson/linux/tree/overlay-verity

I would like some feedback on this approach. Does it make sense?

For context, here is the main commit text:

This adds support for a new overlay xattr "overlay.verity", which
contains a fs-verity digest. This is used for metacopy files, and
whenever the lowerdata file is accessed overlayfs can verify that
the data file fs-verity digest matches the expected one.

By default this is ignored, but if the mount option "verity_policy" is
set to "validate" or "require", then all accesses validate any
specified digest. If you use "require" it additionally fails to access
metacopy file if the verity xattr is missing.

The digest is validated during ovl_open() as well as when the lower file
is copied up. Additionally the overlay.verity xattr is copied to the
upper file during a metacopy operation, in order to later do the validation
of the digest when the copy-up happens.

The primary usecase of this is to use a overlay mount with two lower
directories, the lower being a shared content-addressed-storage
containing fs-verity enabled files, and the upper being a read-only
filesystem (such as erofs) containing metacopy files with the redirect
xattr set pointing into the lower cas storage, as well as the verity
xattr. If this is combined with fs-verity or dm-verify for the
read-only filesystem then the entire mount is validated, even though
the backing files are shared between different images.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       alexl@redhat.com         alexander.larsson@gmail.com

Re: WIP: verity support for overlayfs

[Miklos Szeredi]

On Wed, 8 Mar 2023 at 16:29, Alexander Larsson <alexl@redhat.com> wrote:
>
> As was recently discussed in the various threads about composefs we
> want the ability to specify a fs-verity digest for metacopy files,
> such that the lower file used for the data is guaranteed to have the
> specified digest.
>
> I wrote an initial version of this here:
>
>   https://github.com/alexlarsson/linux/tree/overlay-verity
>
> I would like some feedback on this approach. Does it make sense?
>
> For context, here is the main commit text:
>
> This adds support for a new overlay xattr "overlay.verity", which
> contains a fs-verity digest. This is used for metacopy files, and
> whenever the lowerdata file is accessed overlayfs can verify that
> the data file fs-verity digest matches the expected one.
>
> By default this is ignored, but if the mount option "verity_policy" is
> set to "validate" or "require", then all accesses validate any
> specified digest. If you use "require" it additionally fails to access
> metacopy file if the verity xattr is missing.
>
> The digest is validated during ovl_open() as well as when the lower file
> is copied up. Additionally the overlay.verity xattr is copied to the
> upper file during a metacopy operation, in order to later do the validation
> of the digest when the copy-up happens.

Hmm, so what exactly happens if the file is copied up and then
modified?  The verification will fail, no?

Thanks,
Miklos

Re: WIP: verity support for overlayfs

[Colin Walters]

On Thu, Mar 9, 2023, at 9:59 AM, Miklos Szeredi wrote:
> On Wed, 8 Mar 2023 at 16:29, Alexander Larsson <alexl@redhat.com> wrote:
>>
>> As was recently discussed in the various threads about composefs we
>> want the ability to specify a fs-verity digest for metacopy files,
>> such that the lower file used for the data is guaranteed to have the
>> specified digest.
>>
>> I wrote an initial version of this here:
>>
>>   https://github.com/alexlarsson/linux/tree/overlay-verity
>>
>> I would like some feedback on this approach. Does it make sense?
>>
>> For context, here is the main commit text:
>>
>> This adds support for a new overlay xattr "overlay.verity", which
>> contains a fs-verity digest. This is used for metacopy files, and
>> whenever the lowerdata file is accessed overlayfs can verify that
>> the data file fs-verity digest matches the expected one.
>>
>> By default this is ignored, but if the mount option "verity_policy" is
>> set to "validate" or "require", then all accesses validate any
>> specified digest. If you use "require" it additionally fails to access
>> metacopy file if the verity xattr is missing.
>>
>> The digest is validated during ovl_open() as well as when the lower file
>> is copied up. Additionally the overlay.verity xattr is copied to the
>> upper file during a metacopy operation, in order to later do the validation
>> of the digest when the copy-up happens.
>
> Hmm, so what exactly happens if the file is copied up and then
> modified?  The verification will fail, no?

I believe the intention here is to deploy this without a writable upper dir by default, so there's no copy-up, the calling code just gets -EROFS.  The intention is to also use this to push the podman/docker/kube style ecosystem away from "mutable by default" container images i.e. to "readonlyRootFilesystem" by default (xref https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ )

But yes, some scenarios will still want a writable upper dir for default, as long as that writable upper dir is discarded across reboots (to aid in anti-persistence).  Maybe this needs to be configurable; I could imagine people wanting a writable upper dir, but to still enforce fs-verity for *existing* content.  Other cases may want the logic to just strip away the fsverity xattr across copy-up in this case.

Re: WIP: verity support for overlayfs

[Alexander Larsson]

On Thu, Mar 9, 2023 at 4:26 PM Colin Walters <walters@verbum.org> wrote:
>
>
>
> On Thu, Mar 9, 2023, at 9:59 AM, Miklos Szeredi wrote:
> > On Wed, 8 Mar 2023 at 16:29, Alexander Larsson <alexl@redhat.com> wrote:
> >>
> >> As was recently discussed in the various threads about composefs we
> >> want the ability to specify a fs-verity digest for metacopy files,
> >> such that the lower file used for the data is guaranteed to have the
> >> specified digest.
> >>
> >> I wrote an initial version of this here:
> >>
> >>   https://github.com/alexlarsson/linux/tree/overlay-verity
> >>
> >> I would like some feedback on this approach. Does it make sense?
> >>
> >> For context, here is the main commit text:
> >>
> >> This adds support for a new overlay xattr "overlay.verity", which
> >> contains a fs-verity digest. This is used for metacopy files, and
> >> whenever the lowerdata file is accessed overlayfs can verify that
> >> the data file fs-verity digest matches the expected one.
> >>
> >> By default this is ignored, but if the mount option "verity_policy" is
> >> set to "validate" or "require", then all accesses validate any
> >> specified digest. If you use "require" it additionally fails to access
> >> metacopy file if the verity xattr is missing.
> >>
> >> The digest is validated during ovl_open() as well as when the lower file
> >> is copied up. Additionally the overlay.verity xattr is copied to the
> >> upper file during a metacopy operation, in order to later do the validation
> >> of the digest when the copy-up happens.
> >
> > Hmm, so what exactly happens if the file is copied up and then
> > modified?  The verification will fail, no?
>
> I believe the intention here is to deploy this without a writable upper dir by default, so there's no copy-up, the calling code just gets -EROFS.  The intention is to also use this to push the podman/docker/kube style ecosystem away from "mutable by default" container images i.e. to "readonlyRootFilesystem" by default (xref https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ )

That is indeed some of the primary usecases for this. However, that
doesn't mean it is not useful also for other usecases.

> But yes, some scenarios will still want a writable upper dir for default, as long as that writable upper dir is discarded across reboots (to aid in anti-persistence).  Maybe this needs to be configurable; I could imagine people wanting a writable upper dir, but to still enforce fs-verity for *existing* content.  Other cases may want the logic to just strip away the fsverity xattr across copy-up in this case.

I've been chatting with amir in github about this, and yes, we can
have options that make this useful also with an upper. I'll try to
post a new version with this tomorrow.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       alexl@redhat.com         alexander.larsson@gmail.com

Re: WIP: verity support for overlayfs

[Alexander Larsson]

On Thu, Mar 9, 2023 at 3:59 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Wed, 8 Mar 2023 at 16:29, Alexander Larsson <alexl@redhat.com> wrote:
> >
> > As was recently discussed in the various threads about composefs we
> > want the ability to specify a fs-verity digest for metacopy files,
> > such that the lower file used for the data is guaranteed to have the
> > specified digest.
> >
> > I wrote an initial version of this here:
> >
> >   https://github.com/alexlarsson/linux/tree/overlay-verity
> >
> > I would like some feedback on this approach. Does it make sense?
> >
> > For context, here is the main commit text:
> >
> > This adds support for a new overlay xattr "overlay.verity", which
> > contains a fs-verity digest. This is used for metacopy files, and
> > whenever the lowerdata file is accessed overlayfs can verify that
> > the data file fs-verity digest matches the expected one.
> >
> > By default this is ignored, but if the mount option "verity_policy" is
> > set to "validate" or "require", then all accesses validate any
> > specified digest. If you use "require" it additionally fails to access
> > metacopy file if the verity xattr is missing.
> >
> > The digest is validated during ovl_open() as well as when the lower file
> > is copied up. Additionally the overlay.verity xattr is copied to the
> > upper file during a metacopy operation, in order to later do the validation
> > of the digest when the copy-up happens.
>
> Hmm, so what exactly happens if the file is copied up and then
> modified?  The verification will fail, no?

When we do a meta-copy-up we need to look at the data file and
synthesize an overlay.verity xattr in the upper dir based on the
existing fs-verity diges. At least if the file has fs-verity enabled.
And indeed, in the verify_policy=required case, if there is no
fs-verity in the lower file we should fall back to a full copy-up
instead of a metacopy-up, or we will end up with a metacopy we can't
validate.

However, if you actually modify a file I don't really see the problem,
you will get a non-verified upper layer file with the changes. It will
not fail validation because it is at that point not validated. Really
we can only expect to validate the lower layers.


--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       alexl@redhat.com         alexander.larsson@gmail.com

Re: WIP: verity support for overlayfs

[Alexander Larsson]

On Wed, Mar 8, 2023 at 4:28 PM Alexander Larsson <alexl@redhat.com> wrote:
>
> As was recently discussed in the various threads about composefs we
> want the ability to specify a fs-verity digest for metacopy files,
> such that the lower file used for the data is guaranteed to have the
> specified digest.
>
> I wrote an initial version of this here:
>
>   https://github.com/alexlarsson/linux/tree/overlay-verity

After some discussions with Amir in github I updated the branch. In
this new version there are four verity modes with this behaviour:

Unless you explicitly disable it ("verity=off") all existing xattrs
are validated before use. This is all that happens by default
("verity=validate"), but, if you turn on verity ("verity=on") then
during metacopy we generate verity xattr in the upper metacopy file (if
the source file has verity enabled). This means later accesses can
guarantee that the correct data is used.

Additionally you can use "verity=require". In this mode all metacopy
files must have a valid verity xattr. For this to work metadata
copy-up must be able to create a verity xattr (so that later accesses
are validated). Therefore, in this mode, if the lower data file
doesn't have fs-verity enabled we fall back to a full copy rather than
a metacopy.

In addition I changed the code so that validation of lowerdata happens
during lookup. Previously I was trying to do this lazily at use-time,
but that was only done partially right. Amir is doing some general
work on making lookups lazy, so the idea is to migrate the verity
validation to that later.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                Red Hat, Inc
       alexl@redhat.com         alexander.larsson@gmail.com