Re: Ceph and Netfslib

[David Howells <dhowells@redhat.com>]

Patrick Donnelly <pdonnell@redhat.com> wrote:

> On Wed, Dec 18, 2024 at 1:33 PM David Howells <dhowells@redhat.com> wrote:
> > Also, that would include doing things like content encryption, since that
> > is generally useful in filesystems and I have plans to support it in both
> > AFS and CIFS as well.
> 
> Would this be done with fscrypt? Can you expand on this part?

Since ceph already uses fscrypt, I would need to maintain that.

If you look here:

	https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=netfs-crypt

I have patches to provide content crypto for netfslib (not upstream yet).  If
you look at the afs patch at the top, you can see my hacked up AFS testing
implementation - though it isn't a complete solution as I don't currently have
a way to store the actual EOF there[*].

I add two new methods, encrypt_block and decrypt_block, for netfslib to call
out to the filesystem to actually do the encryption.

On the write/encryption side, netfslib creates one or more subrequests to do
the encryption, allowing it to be done asynchronously (I have it offloading to
my Intel QAT card), collecting the completed encryption subreqs as they're
done and dispatching the write I/O subrequests to the server and/or the local
cache as the ciphertext becomes available.

[Note: one reason I'm doing this in netfslib is so that the same
content-encrypted data is written to both the server and the local cache]

Currently, I create one subreq per fscrypt block to be encrypted (e.g. 4KiB),
but that feels a bit on the heavy side and some throttling is probably
necessary.

Netfslib takes care of encrypting multipage folios that are spanned by
multiple crypto blocks, calling out to the filesystem for each.

How and where the filesystem does the crypto is up to the filesystem - it can
offload it to fscrypt.  It may be possible to have the filesystem load what it
needs into netfs_io_request for fscrypt to pick up and then make the method
pointers point directly to fscrypt.  I don't really want to make the netfslib
module directly dependent on fscrypt, though this could be done to improve
performance.


On the read/decryption side, netfslib currently expects the filesystem to
synchronously decrypt the data, though I should also look at making that
asynchronous now that I have patches to do move read collection for a single
request to working in a single work item instead of a bunch of work items all
competing with each other.  One thing I want to avoid is scheduling async
decryption out of order just because the RPC ops finish out of order.

David

[*] Actually, I may have thought of a way to store the EOF in AFS today.  The
problem is that I have to round up the EOF to a full crypto block (e.g. 16 or
32 bytes for AES) but don't have anywhere to store the real EOF (no xattrs for
example).

So what I'm thinking is that I can just store a trailer of the crypto block
size of all zeros and then encrypt whatever of it that I need.  The trailer
will always be there (unless a zero-length file), always be right after the
EOF and always be the same length, so the AFS fs can trivially calculate the
real EOF size if it knows the algorithm used.

Further, AFS has a StoreData op that "atomically" does a truncate and write,
so maintaining a trailer is pretty straightforward.