Re: [RFC PATCH v10 00/48] ceph+fscrypt: full support

[Xiubo Li <xiubli@redhat.com>]

On 2/14/22 8:08 PM, Xiubo Li wrote:
>
> On 2/14/22 7:33 PM, Jeff Layton wrote:
>> On Mon, 2022-02-14 at 17:37 +0800, Xiubo Li wrote:
>>> Hi Jeff,
>>>
>>> I am using the 'wip-fscrypt' branch to test other issue and hit:
>>>
>>> cp: cannot access './dir___683': No buffer space available
>>> cp: cannot access './dir___686': No buffer space available
>>> cp: cannot access './dir___687': No buffer space available
>>> cp: cannot access './dir___688': No buffer space available
>>> cp: cannot access './dir___689': No buffer space available
>>> cp: cannot access './dir___693': No buffer space available
>>>
>>> ...
>>>
>>> [root@lxbceph1 kcephfs]# diff ./dir___997 /data/backup/kernel/dir___997
>>> diff: ./dir___997: No buffer space available
>>>
>>>
>>> The dmesg logs:
>>>
>>> <7>[ 1256.918228] ceph:  do_getattr inode 0000000089964a71 mask AsXsFs
>>> mode 040755
>>> <7>[ 1256.918232] ceph:  __ceph_caps_issued_mask ino 0x100000009be cap
>>> 0000000014f1c64b issued pAsLsXsFs (mask AsXsFs)
>>> <7>[ 1256.918237] ceph:  __touch_cap 0000000089964a71 cap
>>> 0000000014f1c64b mds0
>>> <7>[ 1256.918250] ceph:  readdir 0000000089964a71 file 00000000065cb689
>>> pos 0
>>> <7>[ 1256.918254] ceph:  readdir off 0 -> '.'
>>> <7>[ 1256.918258] ceph:  readdir off 1 -> '..'
>>> <4>[ 1256.918262] fscrypt (ceph, inode 1099511630270): Error -105
>>> getting encryption context
>>> <7>[ 1256.918269] ceph:  readdir 0000000089964a71 file 00000000065cb689
>>> pos 2
>>> <4>[ 1256.918273] fscrypt (ceph, inode 1099511630270): Error -105
>>> getting encryption context
>>> <7>[ 1256.918288] ceph:  release inode 0000000089964a71 dir file
>>> 00000000065cb689
>>> <7>[ 1256.918310] ceph:  __ceph_caps_issued_mask ino 0x1 cap
>>> 00000000aa2afb8b issued pAsLsXsFs (mask Fs)
>>> <7>[ 1257.574593] ceph:  mdsc delayed_work
>>>
>>> I did nothing about the fscrypt after mounting the kclient, just create
>>> 2000 directories and then made some snapshots on the root dir and then
>>> try to copy the root directory to the backup.
>>>
>>> - Xiubo
>>>
>> That means that ceph_crypt_get_context returned -ENODATA, which it can
>
> It should be -ENOBUFS.
>
> I am not sure it relates to the snapshot stuff. I will try without the 
> snapshot later.

Without snapshot, I also can see this error.

-- Xiubo

>
> I can debug it later, maybe in next week.
>
> -- Xiubo
>
>> do for several different reasons. We probably need to add in some
>> debugging there to see which one it is...
>>
>> TBH, I've done absolutely no testing with snapshots, so it's quite
>> possible there is some interaction there that is causing problems.
>>
>> -- Jeff
>>
>>> On 1/12/22 3:15 AM, Jeff Layton wrote:
>>>> This patchset represents a (mostly) complete rough draft of fscrypt
>>>> support for cephfs. The context, filename and symlink support is 
>>>> more or
>>>> less the same as the versions posted before, and comprise the first 
>>>> half
>>>> of the patches.
>>>>
>>>> The new bits here are the size handling changes and support for 
>>>> content
>>>> encryption, in buffered, direct and synchronous codepaths. Much of 
>>>> this
>>>> code is still very rough and needs a lot of cleanup work.
>>>>
>>>> fscrypt support relies on some MDS changes that are being tracked 
>>>> here:
>>>>
>>>>       https://github.com/ceph/ceph/pull/43588
>>>>
>>>> In particular, this PR adds some new opaque fields in the inode 
>>>> that we
>>>> use to store fscrypt-specific information, like the context and the 
>>>> real
>>>> size of a file. That is slated to be merged for the upcoming Quincy
>>>> release (which is sometime this northern spring).
>>>>
>>>> There are still some notable bugs:
>>>>
>>>> 1/ we've identified a few more potential races in truncate handling
>>>> which will probably necessitate a protocol change, as well as 
>>>> changes to
>>>> the MDS and kclient patchsets. The good news is that we think we have
>>>> an approach that will resolve this.
>>>>
>>>> 2/ the kclient doesn't handle reading sparse regions in OSD objects
>>>> properly yet. The client can end up writing to a non-zero offset in a
>>>> non-existent object. Then, if the client tries to read the written
>>>> region back later, it'll get back zeroes and give you garbage when you
>>>> try to decrypt them.
>>>>
>>>> It turns out that the OSD already supports a SPARSE_READ operation, so
>>>> I'm working on implementing that in the kclient to make it not try to
>>>> decrypt the sparse regions.
>>>>
>>>> Still, I was able to run xfstests on this set yesterday. Bug #2 above
>>>> prevented all of the tests from passing, but it didn't oops! I call 
>>>> that
>>>> progress! Given that, I figured this is a good time to post what I 
>>>> have
>>>> so far.
>>>>
>>>> Note that the buffered I/O changes in this set are not suitable for
>>>> merge and will likely end up being discarded. We need to plumb the
>>>> encryption in at the netfs layer, so that we can store encrypted data
>>>> in fscache.
>>>>
>>>> The non-buffered codepaths will likely also need substantial changes
>>>> before merging. It may be simpler to just move that into the netfs 
>>>> layer
>>>> too as cifs will need something similar anyway.
>>>>
>>>> My goal is to get most of this into v5.18, but v5.19 might be more
>>>> realistiv. Hopefully I'll have a non-RFC patchset to send in a few
>>>> weeks.
>>>>
>>>> Special thanks to Xiubo who came through with the MDS patches. Also,
>>>> thanks to everyone (especially Eric Biggers) for all of the previous
>>>> reviews. It's much appreciated!
>>>>
>>>> Jeff Layton (43):
>>>>     vfs: export new_inode_pseudo
>>>>     fscrypt: export fscrypt_base64url_encode and 
>>>> fscrypt_base64url_decode
>>>>     fscrypt: export fscrypt_fname_encrypt and 
>>>> fscrypt_fname_encrypted_size
>>>>     fscrypt: add fscrypt_context_for_new_inode
>>>>     ceph: preallocate inode for ops that may create one
>>>>     ceph: crypto context handling for ceph
>>>>     ceph: parse new fscrypt_auth and fscrypt_file fields in inode 
>>>> traces
>>>>     ceph: add fscrypt_* handling to caps.c
>>>>     ceph: add ability to set fscrypt_auth via setattr
>>>>     ceph: implement -o test_dummy_encryption mount option
>>>>     ceph: decode alternate_name in lease info
>>>>     ceph: add fscrypt ioctls
>>>>     ceph: make ceph_msdc_build_path use ref-walk
>>>>     ceph: add encrypted fname handling to ceph_mdsc_build_path
>>>>     ceph: send altname in MClientRequest
>>>>     ceph: encode encrypted name in dentry release
>>>>     ceph: properly set DCACHE_NOKEY_NAME flag in lookup
>>>>     ceph: make d_revalidate call fscrypt revalidator for encrypted
>>>>       dentries
>>>>     ceph: add helpers for converting names for userland presentation
>>>>     ceph: add fscrypt support to ceph_fill_trace
>>>>     ceph: add support to readdir for encrypted filenames
>>>>     ceph: create symlinks with encrypted and base64-encoded targets
>>>>     ceph: make ceph_get_name decrypt filenames
>>>>     ceph: add a new ceph.fscrypt.auth vxattr
>>>>     ceph: add some fscrypt guardrails
>>>>     libceph: add CEPH_OSD_OP_ASSERT_VER support
>>>>     ceph: size handling for encrypted inodes in cap updates
>>>>     ceph: fscrypt_file field handling in MClientRequest messages
>>>>     ceph: get file size from fscrypt_file when present in inode traces
>>>>     ceph: handle fscrypt fields in cap messages from MDS
>>>>     ceph: add infrastructure for file encryption and decryption
>>>>     libceph: allow ceph_osdc_new_request to accept a multi-op read
>>>>     ceph: disable fallocate for encrypted inodes
>>>>     ceph: disable copy offload on encrypted inodes
>>>>     ceph: don't use special DIO path for encrypted inodes
>>>>     ceph: set encryption context on open
>>>>     ceph: align data in pages in ceph_sync_write
>>>>     ceph: add read/modify/write to ceph_sync_write
>>>>     ceph: plumb in decryption during sync reads
>>>>     ceph: set i_blkbits to crypto block size for encrypted inodes
>>>>     ceph: add fscrypt decryption support to ceph_netfs_issue_op
>>>>     ceph: add encryption support to writepage
>>>>     ceph: fscrypt support for writepages
>>>>
>>>> Luis Henriques (1):
>>>>     ceph: don't allow changing layout on encrypted files/directories
>>>>
>>>> Xiubo Li (4):
>>>>     ceph: add __ceph_get_caps helper support
>>>>     ceph: add __ceph_sync_read helper support
>>>>     ceph: add object version support for sync read
>>>>     ceph: add truncate size handling support for fscrypt
>>>>
>>>>    fs/ceph/Makefile                |   1 +
>>>>    fs/ceph/acl.c                   |   4 +-
>>>>    fs/ceph/addr.c                  | 128 +++++--
>>>>    fs/ceph/caps.c                  | 211 ++++++++++--
>>>>    fs/ceph/crypto.c                | 374 +++++++++++++++++++++
>>>>    fs/ceph/crypto.h                | 237 +++++++++++++
>>>>    fs/ceph/dir.c                   | 209 +++++++++---
>>>>    fs/ceph/export.c                |  44 ++-
>>>>    fs/ceph/file.c                  | 476 +++++++++++++++++++++-----
>>>>    fs/ceph/inode.c                 | 576 
>>>> +++++++++++++++++++++++++++++---
>>>>    fs/ceph/ioctl.c                 |  87 +++++
>>>>    fs/ceph/mds_client.c            | 349 ++++++++++++++++---
>>>>    fs/ceph/mds_client.h            |  24 +-
>>>>    fs/ceph/super.c                 |  90 ++++-
>>>>    fs/ceph/super.h                 |  43 ++-
>>>>    fs/ceph/xattr.c                 |  29 ++
>>>>    fs/crypto/fname.c               |  44 ++-
>>>>    fs/crypto/fscrypt_private.h     |   9 +-
>>>>    fs/crypto/hooks.c               |   6 +-
>>>>    fs/crypto/policy.c              |  35 +-
>>>>    fs/inode.c                      |   1 +
>>>>    include/linux/ceph/ceph_fs.h    |  21 +-
>>>>    include/linux/ceph/osd_client.h |   6 +-
>>>>    include/linux/ceph/rados.h      |   4 +
>>>>    include/linux/fscrypt.h         |  10 +
>>>>    net/ceph/osd_client.c           |  32 +-
>>>>    26 files changed, 2700 insertions(+), 350 deletions(-)
>>>>    create mode 100644 fs/ceph/crypto.c
>>>>    create mode 100644 fs/ceph/crypto.h
>>>>