Feature proposal: support file content integrity verification based on fs-verity

Feature proposal: support file content integrity verification based on fs-verity

[Gmail]

Hello fuse-devel,

The fs-verity framework provides file content integrity verification services for filesystems. Currently ext4/btrfs/f2fs has enabled support for fs-verity. Here I would like to propose implementing FUSE file content integrity verification based on fs-verity.

Our current main use case is to support integrity verification for confidential containers using virtio-fs. With the new integrity verification feature, we can ensure that files from virtio-fs are trusted and fs-verity root digests are available for remote attestation. The integrity verification feature can also be used to support other FUSE based solutions.

Fs-verity supports generating and verifying file content hash values. For the sake of simplicity, we may only support hash value verification of file content in the first stage, and enable support for hash value generation in the later stage.

The following FUSE protocol changes are therefore proposed to support fs-verity:
1) add flag “FUSE_FS_VERITY” to negotiate fs-verity support 
2) add flag “FUSE_ATTR_FSVERITY” for fuse servers to mark that inodes have associated fs-verity meta data. 
3) add op “FUSE_FSVERITY” to get/set fs-verity descriptor and hash values.

The FUSE protocol does not specify how fuse servers store fs-verity metadata. The fuse server can store fs-verity metadata in its own ways.

I did a quick prototype and the changes seems moderate, about 250 lines of code changes.

Would love to hear about your feedback:)

Thanks,
Gerry

Re: Feature proposal: support file content integrity verification based on fs-verity

[Victor Hsieh]

On Thu, Nov 17, 2022 at 9:19 PM Gmail <liuj97@gmail.com> wrote:
>
> Hello fuse-devel,
>
> The fs-verity framework provides file content integrity verification services for filesystems. Currently ext4/btrfs/f2fs has enabled support for fs-verity. Here I would like to propose implementing FUSE file content integrity verification based on fs-verity.
>
> Our current main use case is to support integrity verification for confidential containers using virtio-fs. With the new integrity verification feature, we can ensure that files from virtio-fs are trusted and fs-verity root digests are available for remote attestation. The integrity verification feature can also be used to support other FUSE based solutions.
I'd argue FUSE isn't the right layer for supporting fs-verity
verification.  The verification can happen in the read path of
virtio-fs (or any FUSE-based filesystem).  In fact, Android is already
doing this in "authfs" fully in userspace.

Although FUSE lacks the support of "unrestricted" ioctl, which makes
it impossible for the filesystem to receive the fs-verity ioctls.
Same to statx.  I think that's where we'd need a change in FUSE
protocol.

>
> Fs-verity supports generating and verifying file content hash values. For the sake of simplicity, we may only support hash value verification of file content in the first stage, and enable support for hash value generation in the later stage.
>
> The following FUSE protocol changes are therefore proposed to support fs-verity:
> 1) add flag “FUSE_FS_VERITY” to negotiate fs-verity support
> 2) add flag “FUSE_ATTR_FSVERITY” for fuse servers to mark that inodes have associated fs-verity meta data.
> 3) add op “FUSE_FSVERITY” to get/set fs-verity descriptor and hash values.

>
> The FUSE protocol does not specify how fuse servers store fs-verity metadata. The fuse server can store fs-verity metadata in its own ways.
>
> I did a quick prototype and the changes seems moderate, about 250 lines of code changes.
>
> Would love to hear about your feedback:)
>
> Thanks,
> Gerry
>

Re: Feature proposal: support file content integrity verification based on fs-verity

[Gerry Liu]

> 2022年11月29日 08:44，Victor Hsieh <victorhsieh@google.com> 写道：
> 
> On Thu, Nov 17, 2022 at 9:19 PM Gmail <liuj97@gmail.com> wrote:
>> 
>> Hello fuse-devel,
>> 
>> The fs-verity framework provides file content integrity verification services for filesystems. Currently ext4/btrfs/f2fs has enabled support for fs-verity. Here I would like to propose implementing FUSE file content integrity verification based on fs-verity.
>> 
>> Our current main use case is to support integrity verification for confidential containers using virtio-fs. With the new integrity verification feature, we can ensure that files from virtio-fs are trusted and fs-verity root digests are available for remote attestation. The integrity verification feature can also be used to support other FUSE based solutions.
> I'd argue FUSE isn't the right layer for supporting fs-verity
> verification.  The verification can happen in the read path of
> virtio-fs (or any FUSE-based filesystem).  In fact, Android is already
> doing this in "authfs" fully in userspace.
Hi Victor,
Thanks for your comments:)

There’s a trust boundary problem here. There are two possible ways to verify data integrity:
1) verify data integrity in fuse kernel driver
2) verify data integrity in fuse server.

For hardware TEE(Trusted Execution Environment) based confidential vm/container with virtio-fs, the fuse server running on the host side is outside of trust domain, and the fuse driver is inside of trust domain. It is therefore recommended to verify data integrity in the fuse driver. The same situation may exist for fuse device based fuse server. The application trusts kernel but doesn’t trust the fuse server.

Thanks,
Gerry  

> 
> Although FUSE lacks the support of "unrestricted" ioctl, which makes
> it impossible for the filesystem to receive the fs-verity ioctls.
> Same to statx.  I think that's where we'd need a change in FUSE
> protocol.
> 
>> 
>> Fs-verity supports generating and verifying file content hash values. For the sake of simplicity, we may only support hash value verification of file content in the first stage, and enable support for hash value generation in the later stage.
>> 
>> The following FUSE protocol changes are therefore proposed to support fs-verity:
>> 1) add flag “FUSE_FS_VERITY” to negotiate fs-verity support
>> 2) add flag “FUSE_ATTR_FSVERITY” for fuse servers to mark that inodes have associated fs-verity meta data.
>> 3) add op “FUSE_FSVERITY” to get/set fs-verity descriptor and hash values.
> 
>> 
>> The FUSE protocol does not specify how fuse servers store fs-verity metadata. The fuse server can store fs-verity metadata in its own ways.
>> 
>> I did a quick prototype and the changes seems moderate, about 250 lines of code changes.
>> 
>> Would love to hear about your feedback:)
>> 
>> Thanks,
>> Gerry
>> 

Re: Feature proposal: support file content integrity verification based on fs-verity

[Victor Hsieh]

On Thu, Dec 1, 2022 at 1:51 AM Gerry Liu <liuj97@gmail.com> wrote:
>
>
>
> > 2022年11月29日 08:44，Victor Hsieh <victorhsieh@google.com> 写道：
> >
> > On Thu, Nov 17, 2022 at 9:19 PM Gmail <liuj97@gmail.com> wrote:
> >>
> >> Hello fuse-devel,
> >>
> >> The fs-verity framework provides file content integrity verification services for filesystems. Currently ext4/btrfs/f2fs has enabled support for fs-verity. Here I would like to propose implementing FUSE file content integrity verification based on fs-verity.
> >>
> >> Our current main use case is to support integrity verification for confidential containers using virtio-fs. With the new integrity verification feature, we can ensure that files from virtio-fs are trusted and fs-verity root digests are available for remote attestation. The integrity verification feature can also be used to support other FUSE based solutions.
> > I'd argue FUSE isn't the right layer for supporting fs-verity
> > verification.  The verification can happen in the read path of
> > virtio-fs (or any FUSE-based filesystem).  In fact, Android is already
> > doing this in "authfs" fully in userspace.
> Hi Victor,
> Thanks for your comments:)
>
> There’s a trust boundary problem here. There are two possible ways to verify data integrity:
> 1) verify data integrity in fuse kernel driver
> 2) verify data integrity in fuse server.
>
> For hardware TEE(Trusted Execution Environment) based confidential vm/container with virtio-fs, the fuse server running on the host side is outside of trust domain, and the fuse driver is inside of trust domain. It is therefore recommended to verify data integrity in the fuse driver. The same situation may exist for fuse device based fuse server. The application trusts kernel but doesn’t trust the fuse server.

It sounded like your case is similar to ours: the storage isn't
considered trusted (across the VM boundary).  Note that fs-verity can
only give you the consistent (and efficient) file measurement over the
file content.  If your storage is not trusted, you do have to ensure
the measurement of the *file paths* are the expected values, otherwise
the attacker can replace/rename one file with another.  For example,
the trusted process would have to know a mapping from file name to the
measurement, and check the measurement of the fs-verity-enabled files
before every open.   I can see how supporting fs-verity in virtio-fs
can be a stepping stone to solving your problem, but I'm not in a good
position to suggest whether it's a good idea or not.  But we did solve
our problem purely in userspace also using FUSE.

>
> Thanks,
> Gerry
>
> >
> > Although FUSE lacks the support of "unrestricted" ioctl, which makes
> > it impossible for the filesystem to receive the fs-verity ioctls.
> > Same to statx.  I think that's where we'd need a change in FUSE
> > protocol.
> >
> >>
> >> Fs-verity supports generating and verifying file content hash values. For the sake of simplicity, we may only support hash value verification of file content in the first stage, and enable support for hash value generation in the later stage.
> >>
> >> The following FUSE protocol changes are therefore proposed to support fs-verity:
> >> 1) add flag “FUSE_FS_VERITY” to negotiate fs-verity support
> >> 2) add flag “FUSE_ATTR_FSVERITY” for fuse servers to mark that inodes have associated fs-verity meta data.
> >> 3) add op “FUSE_FSVERITY” to get/set fs-verity descriptor and hash values.
> >
> >>
> >> The FUSE protocol does not specify how fuse servers store fs-verity metadata. The fuse server can store fs-verity metadata in its own ways.
> >>
> >> I did a quick prototype and the changes seems moderate, about 250 lines of code changes.
> >>
> >> Would love to hear about your feedback:)
> >>
> >> Thanks,
> >> Gerry
> >>
>