RE: [PATCH] ceph: fix kernel crash in ceph_open()

[Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>]

On Thu, 2025-11-20 at 19:50 +0530, Kotresh Hiremath Ravishankar wrote:
> Hi All,
> 
> I think the patch is necessary and fixes the crash. There is no harm
> in taking this patch as it behaves like an old kernel with this
> particular scenario.
> 
> When does the issue happen:
>    - The issue happens only when the old mount syntax is used where
> passing the file system name is optional in which case, it chooses the
> default mds namespace but doesn't get filled in the
> mdsc->fsc->mount_options->mds_namespace.
>    - Along with the above, the mount user should be non admin.
> Does it break the earlier fix ?
>    - Not fully!!! Though the open does succeed, the subsequent
> operation like write would get EPERM. I am not exactly able to
> recollect but this was discussed before writing the fix 22c73d52a6d0
> ("ceph: fix multifs mds auth caps issue"), it's guarded by another
> check before actual operation like write.
> 
> I think there are a couple of options to fix this cleanly.
>  1. Use the default fsname when
> mdsc->fsc->mount_options->mds_namespace is NULL during comparison.
>  2. Mandate passing the fsname with old syntax ?
> 

Anyway, we should be ready operate correctly if fsname or/and auth-
>match.fs_name are NULL. And if we need to make the fix more cleanly, then we
can introduce another patch with nicer fix.

I am not completely sure how default fsname can be applicable here. If I
understood the CephFS mount logic correctly, then fsname can be NULL during some
initial steps. But, finally, we will have the real fsname for comparison. But I
don't know if it's right of assuming that fsname == NULL is equal to fsname ==
default_name.

And I am not sure that we can mandate anyone to use the old syntax. If there is
some other opportunity, then someone could use it. But, maybe, I am missing the
point. :) What do you mean by "Mandate passing the fsname with old syntax"?

Thanks,
Slava.

> 
> Thanks,
> Kotresh H R
> 
> 
> 
> On Thu, Nov 20, 2025 at 4:47 AM Viacheslav Dubeyko
> <Slava.Dubeyko@ibm.com> wrote:
> > 
> > On Wed, 2025-11-19 at 15:02 -0800, Gregory Farnum wrote:
> > > 
> > > That doesn’t sound right — this is authentication code. If the authorization is supplied for a namespace and we are mounting without a namespace at all, isn’t that a jailbreak? So the NULL pointer should be accepted in one direction, but denied in the other?
> > 
> > What is your particular suggestion? I am simply fixing the kernel crash after
> > the 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"). We didn't have any
> > check before. Do you imply that 22c73d52a6d0 ("ceph: fix multifs mds auth caps
> > issue") fix is incorrect and we need to rework it somehow?
> > 
> > If we will not have any fix, then 6.18 release will have broken CephFS kernel
> > client.
> > 
> > Thanks,
> > Slava.
> > 
> > > 
> > > On Wed, Nov 19, 2025 at 2:54 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote:
> > > > On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote:
> > > > > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote:
> > > > > > 
> > > > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
> > > > > > 
> > > > > > The CephFS kernel client has regression starting from 6.18-rc1.
> > > > > > 
> > > > > > sudo ./check -g quick
> > > > > > FSTYP         -- ceph
> > > > > > PLATFORM      -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri
> > > > > > Nov 14 11:26:14 PST 2025
> > > > > > MKFS_OPTIONS  -- 192.168.1.213:3300:/scratch
> > > > > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch
> > > > > > /mnt/cephfs/scratch
> > > > > > 
> > > > > > Killed
> > > > > > 
> > > > > > Nov 14 11:48:10 ceph-0005 kernel: [  154.723902] libceph: mon0
> > > > > > (2)192.168.1.213:3300 session established
> > > > > > Nov 14 11:48:10 ceph-0005 kernel: [  154.727225] libceph: client167616
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.087260] BUG: kernel NULL pointer
> > > > > > dereference, address: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.087756] #PF: supervisor read access in
> > > > > > kernel mode
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.088043] #PF: error_code(0x0000) - not-
> > > > > > present page
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.088302] PGD 0 P4D 0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.088688] Oops: Oops: 0000 [#1] SMP KASAN
> > > > > > NOPTI
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.090080] CPU: 4 UID: 0 PID: 3453 Comm:
> > > > > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary)
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.091245] Hardware name: QEMU Standard PC
> > > > > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.092103] RIP: 0010:strcmp+0x1c/0x40
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.092493] Code: 90 90 90 90 90 90 90 90
> > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84
> > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc
> > > > > > cc cc 31
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.094057] RSP: 0018:ffff8881536875c0
> > > > > > EFLAGS: 00010246
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.094522] RAX: 0000000000000000 RBX:
> > > > > > ffff888116003200 RCX: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.095114] RDX: 0000000000000063 RSI:
> > > > > > 0000000000000000 RDI: ffff88810126c900
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.095714] RBP: ffff8881536876a8 R08:
> > > > > > 0000000000000000 R09: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.096297] R10: 0000000000000000 R11:
> > > > > > 0000000000000000 R12: dffffc0000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.096889] R13: ffff8881061d0000 R14:
> > > > > > 0000000000000000 R15: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.097490] FS:  000074a85c082840(0000)
> > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.098146] CS:  0010 DS: 0000 ES: 0000
> > > > > > CR0: 0000000080050033
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.098630] CR2: 0000000000000000 CR3:
> > > > > > 0000000110ebd001 CR4: 0000000000772ef0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.099219] PKRU: 55555554
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.099476] Call Trace:
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.099686]  <TASK>
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.099873]  ?
> > > > > > ceph_mds_check_access+0x348/0x1760
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.100267]  ?
> > > > > > __kasan_check_write+0x14/0x30
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.100671]  ? lockref_get+0xb1/0x170
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.100979]  ?
> > > > > > __pfx__raw_spin_lock+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.101372]  ceph_open+0x322/0xef0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.101669]  ? __pfx_ceph_open+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.101996]  ?
> > > > > > __pfx_apparmor_file_open+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.102434]  ?
> > > > > > __ceph_caps_issued_mask_metric+0xd6/0x180
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.102911]  do_dentry_open+0x7bf/0x10e0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.103249]  ? __pfx_ceph_open+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.103508]  vfs_open+0x6d/0x450
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.103697]  ? may_open+0xec/0x370
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.103893]  path_openat+0x2017/0x50a0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.104110]  ? __pfx_path_openat+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.104345]  ?
> > > > > > __pfx_stack_trace_save+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.104599]  ?
> > > > > > stack_depot_save_flags+0x28/0x8f0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.104865]  ? stack_depot_save+0xe/0x20
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105063]  do_filp_open+0x1b4/0x450
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105253]  ?
> > > > > > __pfx__raw_spin_lock_irqsave+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105538]  ? __pfx_do_filp_open+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105748]  ? __link_object+0x13d/0x2b0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105949]  ?
> > > > > > __pfx__raw_spin_lock+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.106169]  ?
> > > > > > __check_object_size+0x453/0x600
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.106428]  ? _raw_spin_unlock+0xe/0x40
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.106635]  do_sys_openat2+0xe6/0x180
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.106827]  ?
> > > > > > __pfx_do_sys_openat2+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.107052]  __x64_sys_openat+0x108/0x240
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.107258]  ?
> > > > > > __pfx___x64_sys_openat+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.107529]  ?
> > > > > > __pfx___handle_mm_fault+0x10/0x10
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.107783]  x64_sys_call+0x134f/0x2350
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108007]  do_syscall_64+0x82/0xd50
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108201]  ?
> > > > > > fpregs_assert_state_consistent+0x5c/0x100
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108467]  ? do_syscall_64+0xba/0xd50
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108626]  ? __kasan_check_read+0x11/0x20
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108801]  ?
> > > > > > count_memcg_events+0x25b/0x400
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109013]  ? handle_mm_fault+0x38b/0x6a0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109216]  ? __kasan_check_read+0x11/0x20
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109457]  ?
> > > > > > fpregs_assert_state_consistent+0x5c/0x100
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109724]  ?
> > > > > > irqentry_exit_to_user_mode+0x2e/0x2a0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109991]  ? irqentry_exit+0x43/0x50
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.110180]  ? exc_page_fault+0x95/0x100
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.110389]
> > > > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.110638] RIP: 0033:0x74a85bf145ab
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.110821] Code: 25 00 00 41 00 3d 00 00
> > > > > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff
> > > > > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48
> > > > > > 2b 14 25
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.111724] RSP: 002b:00007ffc77d316d0
> > > > > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.112080] RAX: ffffffffffffffda RBX:
> > > > > > 0000000000000002 RCX: 000074a85bf145ab
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.112442] RDX: 0000000000000000 RSI:
> > > > > > 00007ffc77d32789 RDI: 00000000ffffff9c
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.112790] RBP: 00007ffc77d32789 R08:
> > > > > > 00007ffc77d31980 R09: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.113125] R10: 0000000000000000 R11:
> > > > > > 0000000000000246 R12: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.113502] R13: 00000000ffffffff R14:
> > > > > > 0000000000000180 R15: 0000000000000001
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.113838]  </TASK>
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.113957] Modules linked in:
> > > > > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core
> > > > > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec
> > > > > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel
> > > > > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate
> > > > > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp
> > > > > > parport efi_pstore
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.116339] CR2: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.116574] ---[ end trace 0000000000000000
> > > > > > ]---
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.116826] RIP: 0010:strcmp+0x1c/0x40
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.117058] Code: 90 90 90 90 90 90 90 90
> > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84
> > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc
> > > > > > cc cc 31
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.118070] RSP: 0018:ffff8881536875c0
> > > > > > EFLAGS: 00010246
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.118362] RAX: 0000000000000000 RBX:
> > > > > > ffff888116003200 RCX: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.118748] RDX: 0000000000000063 RSI:
> > > > > > 0000000000000000 RDI: ffff88810126c900
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.119116] RBP: ffff8881536876a8 R08:
> > > > > > 0000000000000000 R09: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.119492] R10: 0000000000000000 R11:
> > > > > > 0000000000000000 R12: dffffc0000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.119865] R13: ffff8881061d0000 R14:
> > > > > > 0000000000000000 R15: 0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.120242] FS:  000074a85c082840(0000)
> > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.120704] CS:  0010 DS: 0000 ES: 0000
> > > > > > CR0: 0000000080050033
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.121008] CR2: 0000000000000000 CR3:
> > > > > > 0000000110ebd001 CR4: 0000000000772ef0
> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.121409] PKRU: 55555554
> > > > > > 
> > > > > > We have issue here [1] if fs_name == NULL:
> > > > > > 
> > > > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace;
> > > > > >      ...
> > > > > >      if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) {
> > > > > >              / fsname mismatch, try next one */
> > > > > >              return 0;
> > > > > >      }
> > > > > > 
> > > > > > The patch fixes the issue by introducing is_fsname_mismatch() method
> > > > > > that checks auth->match.fs_name and fs_name pointers validity, and
> > > > > > compares the file system names.
> > > > > > 
> > > > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666  
> > > > > > 
> > > > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue")
> > > > > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
> > > > > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
> > > > > > cc: Alex Markuze <amarkuze@redhat.com>
> > > > > > cc: Ilya Dryomov <idryomov@gmail.com>
> > > > > > cc: Ceph Development <ceph-devel@vger.kernel.org>
> > > > > > ---
> > > > > >   fs/ceph/mds_client.c | 20 +++++++++++++++++---
> > > > > >   1 file changed, 17 insertions(+), 3 deletions(-)
> > > > > > 
> > > > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> > > > > > index 1740047aef0f..19c75e206300 100644
> > > > > > --- a/fs/ceph/mds_client.c
> > > > > > +++ b/fs/ceph/mds_client.c
> > > > > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s)
> > > > > >          mutex_unlock(&s->s_mutex);
> > > > > >   }
> > > > > > 
> > > > > > +static inline
> > > > > > +bool is_fsname_mismatch(struct ceph_client *cl,
> > > > > > +                       const char *fs_name1, const char *fs_name2)
> > > > > > +{
> > > > > > +       if (!fs_name1 || !fs_name2)
> > > > > > +               return false;
> > > > > 
> > > > > Hi Slava,
> > > > > 
> > > > > It looks like this would declare a match (return false for "mismatch")
> > > > > in case ceph_mds_cap_auth is defined to require a particular fs_name but
> > > > > no mds_namespace was passed on mount.  Is that the desired behavior?
> > > > > 
> > > > 
> > > > Hi Ilya,
> > > > 
> > > > Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such
> > > > check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or
> > > > fs_name is NULL, then we cannot say that they match or not. It means that we
> > > > need to continue logic, this is why is_fsname_mismatch() returns false.
> > > > Otherwise, if we stop logic by returning true, then we have bunch of xfstests
> > > > failures.
> > > > 
> > > > Thanks,
> > > > Slava.
> > > > 
> > > > > > +
> > > > > > +       doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n",
> > > > > > +             fs_name1, fs_name2);
> > > > > > +
> > > > > > +       if (strcmp(fs_name1, fs_name2))
> > > > > > +               return true;
> > > > > > +
> > > > > > +       return false;
> > > > > > +}
> > > > > > +
> > > > > >   static int ceph_mds_auth_match(struct ceph_mds_client *mdsc,
> > > > > >                                 struct ceph_mds_cap_auth *auth,
> > > > > >                                 const struct cred *cred,
> > > > > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc,
> > > > > >          u32 gid, tlen, len;
> > > > > >          int i, j;
> > > > > > 
> > > > > > -       doutc(cl, "fsname check fs_name=%s  match.fs_name=%s\n",
> > > > > > -             fs_name, auth->match.fs_name ? auth->match.fs_name : "");
> > > > > > -       if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) {
> > > > > > +       if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) {
> > > > > >                  /* fsname mismatch, try next one */
> > > > > >                  return 0;
> > > > > >          }
> > > > > > --
> > > > > > 2.51.1
> > > > > > 
> > > > 
> >