RE: [PATCH] ceph: fix kernel crash in ceph_open()

[Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>]

On Wed, 2025-11-19 at 15:02 -0800, Gregory Farnum wrote: 
>  
> That doesn’t sound right — this is authentication code. If the authorization is supplied for a namespace and we are mounting without a namespace at all, isn’t that a jailbreak? So the NULL pointer should be accepted in one direction, but denied in the other?

What is your particular suggestion? I am simply fixing the kernel crash after
the 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"). We didn't have any
check before. Do you imply that 22c73d52a6d0 ("ceph: fix multifs mds auth caps
issue") fix is incorrect and we need to rework it somehow?

If we will not have any fix, then 6.18 release will have broken CephFS kernel
client.

Thanks,
Slava.

> 
> On Wed, Nov 19, 2025 at 2:54 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote:
> > On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote:
> > > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote:
> > > > 
> > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
> > > > 
> > > > The CephFS kernel client has regression starting from 6.18-rc1.
> > > > 
> > > > sudo ./check -g quick
> > > > FSTYP         -- ceph
> > > > PLATFORM      -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri
> > > > Nov 14 11:26:14 PST 2025
> > > > MKFS_OPTIONS  -- 192.168.1.213:3300:/scratch
> > > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch
> > > > /mnt/cephfs/scratch
> > > > 
> > > > Killed
> > > > 
> > > > Nov 14 11:48:10 ceph-0005 kernel: [  154.723902] libceph: mon0
> > > > (2)192.168.1.213:3300 session established
> > > > Nov 14 11:48:10 ceph-0005 kernel: [  154.727225] libceph: client167616
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.087260] BUG: kernel NULL pointer
> > > > dereference, address: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.087756] #PF: supervisor read access in
> > > > kernel mode
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.088043] #PF: error_code(0x0000) - not-
> > > > present page
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.088302] PGD 0 P4D 0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.088688] Oops: Oops: 0000 [#1] SMP KASAN
> > > > NOPTI
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.090080] CPU: 4 UID: 0 PID: 3453 Comm:
> > > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary)
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.091245] Hardware name: QEMU Standard PC
> > > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.092103] RIP: 0010:strcmp+0x1c/0x40
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.092493] Code: 90 90 90 90 90 90 90 90
> > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84
> > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc
> > > > cc cc 31
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.094057] RSP: 0018:ffff8881536875c0
> > > > EFLAGS: 00010246
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.094522] RAX: 0000000000000000 RBX:
> > > > ffff888116003200 RCX: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.095114] RDX: 0000000000000063 RSI:
> > > > 0000000000000000 RDI: ffff88810126c900
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.095714] RBP: ffff8881536876a8 R08:
> > > > 0000000000000000 R09: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.096297] R10: 0000000000000000 R11:
> > > > 0000000000000000 R12: dffffc0000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.096889] R13: ffff8881061d0000 R14:
> > > > 0000000000000000 R15: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.097490] FS:  000074a85c082840(0000)
> > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.098146] CS:  0010 DS: 0000 ES: 0000
> > > > CR0: 0000000080050033
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.098630] CR2: 0000000000000000 CR3:
> > > > 0000000110ebd001 CR4: 0000000000772ef0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.099219] PKRU: 55555554
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.099476] Call Trace:
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.099686]  <TASK>
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.099873]  ?
> > > > ceph_mds_check_access+0x348/0x1760
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.100267]  ?
> > > > __kasan_check_write+0x14/0x30
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.100671]  ? lockref_get+0xb1/0x170
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.100979]  ?
> > > > __pfx__raw_spin_lock+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.101372]  ceph_open+0x322/0xef0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.101669]  ? __pfx_ceph_open+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.101996]  ?
> > > > __pfx_apparmor_file_open+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.102434]  ?
> > > > __ceph_caps_issued_mask_metric+0xd6/0x180
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.102911]  do_dentry_open+0x7bf/0x10e0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.103249]  ? __pfx_ceph_open+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.103508]  vfs_open+0x6d/0x450
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.103697]  ? may_open+0xec/0x370
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.103893]  path_openat+0x2017/0x50a0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.104110]  ? __pfx_path_openat+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.104345]  ?
> > > > __pfx_stack_trace_save+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.104599]  ?
> > > > stack_depot_save_flags+0x28/0x8f0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.104865]  ? stack_depot_save+0xe/0x20
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105063]  do_filp_open+0x1b4/0x450
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105253]  ?
> > > > __pfx__raw_spin_lock_irqsave+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105538]  ? __pfx_do_filp_open+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105748]  ? __link_object+0x13d/0x2b0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.105949]  ?
> > > > __pfx__raw_spin_lock+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.106169]  ?
> > > > __check_object_size+0x453/0x600
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.106428]  ? _raw_spin_unlock+0xe/0x40
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.106635]  do_sys_openat2+0xe6/0x180
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.106827]  ?
> > > > __pfx_do_sys_openat2+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.107052]  __x64_sys_openat+0x108/0x240
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.107258]  ?
> > > > __pfx___x64_sys_openat+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.107529]  ?
> > > > __pfx___handle_mm_fault+0x10/0x10
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.107783]  x64_sys_call+0x134f/0x2350
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108007]  do_syscall_64+0x82/0xd50
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108201]  ?
> > > > fpregs_assert_state_consistent+0x5c/0x100
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108467]  ? do_syscall_64+0xba/0xd50
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108626]  ? __kasan_check_read+0x11/0x20
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.108801]  ?
> > > > count_memcg_events+0x25b/0x400
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109013]  ? handle_mm_fault+0x38b/0x6a0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109216]  ? __kasan_check_read+0x11/0x20
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109457]  ?
> > > > fpregs_assert_state_consistent+0x5c/0x100
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109724]  ?
> > > > irqentry_exit_to_user_mode+0x2e/0x2a0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.109991]  ? irqentry_exit+0x43/0x50
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.110180]  ? exc_page_fault+0x95/0x100
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.110389]
> > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.110638] RIP: 0033:0x74a85bf145ab
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.110821] Code: 25 00 00 41 00 3d 00 00
> > > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff
> > > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48
> > > > 2b 14 25
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.111724] RSP: 002b:00007ffc77d316d0
> > > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.112080] RAX: ffffffffffffffda RBX:
> > > > 0000000000000002 RCX: 000074a85bf145ab
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.112442] RDX: 0000000000000000 RSI:
> > > > 00007ffc77d32789 RDI: 00000000ffffff9c
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.112790] RBP: 00007ffc77d32789 R08:
> > > > 00007ffc77d31980 R09: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.113125] R10: 0000000000000000 R11:
> > > > 0000000000000246 R12: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.113502] R13: 00000000ffffffff R14:
> > > > 0000000000000180 R15: 0000000000000001
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.113838]  </TASK>
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.113957] Modules linked in:
> > > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core
> > > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec
> > > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel
> > > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate
> > > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp
> > > > parport efi_pstore
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.116339] CR2: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.116574] ---[ end trace 0000000000000000
> > > > ]---
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.116826] RIP: 0010:strcmp+0x1c/0x40
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.117058] Code: 90 90 90 90 90 90 90 90
> > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84
> > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc
> > > > cc cc 31
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.118070] RSP: 0018:ffff8881536875c0
> > > > EFLAGS: 00010246
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.118362] RAX: 0000000000000000 RBX:
> > > > ffff888116003200 RCX: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.118748] RDX: 0000000000000063 RSI:
> > > > 0000000000000000 RDI: ffff88810126c900
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.119116] RBP: ffff8881536876a8 R08:
> > > > 0000000000000000 R09: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.119492] R10: 0000000000000000 R11:
> > > > 0000000000000000 R12: dffffc0000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.119865] R13: ffff8881061d0000 R14:
> > > > 0000000000000000 R15: 0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.120242] FS:  000074a85c082840(0000)
> > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.120704] CS:  0010 DS: 0000 ES: 0000
> > > > CR0: 0000000080050033
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.121008] CR2: 0000000000000000 CR3:
> > > > 0000000110ebd001 CR4: 0000000000772ef0
> > > > Nov 14 11:48:11 ceph-0005 kernel: [  155.121409] PKRU: 55555554
> > > > 
> > > > We have issue here [1] if fs_name == NULL:
> > > > 
> > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace;
> > > >      ...
> > > >      if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) {
> > > >              / fsname mismatch, try next one */
> > > >              return 0;
> > > >      }
> > > > 
> > > > The patch fixes the issue by introducing is_fsname_mismatch() method
> > > > that checks auth->match.fs_name and fs_name pointers validity, and
> > > > compares the file system names.
> > > > 
> > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666  
> > > > 
> > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue")
> > > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
> > > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
> > > > cc: Alex Markuze <amarkuze@redhat.com>
> > > > cc: Ilya Dryomov <idryomov@gmail.com>
> > > > cc: Ceph Development <ceph-devel@vger.kernel.org>
> > > > ---
> > > >   fs/ceph/mds_client.c | 20 +++++++++++++++++---
> > > >   1 file changed, 17 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> > > > index 1740047aef0f..19c75e206300 100644
> > > > --- a/fs/ceph/mds_client.c
> > > > +++ b/fs/ceph/mds_client.c
> > > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s)
> > > >          mutex_unlock(&s->s_mutex);
> > > >   }
> > > > 
> > > > +static inline
> > > > +bool is_fsname_mismatch(struct ceph_client *cl,
> > > > +                       const char *fs_name1, const char *fs_name2)
> > > > +{
> > > > +       if (!fs_name1 || !fs_name2)
> > > > +               return false;
> > > 
> > > Hi Slava,
> > > 
> > > It looks like this would declare a match (return false for "mismatch")
> > > in case ceph_mds_cap_auth is defined to require a particular fs_name but
> > > no mds_namespace was passed on mount.  Is that the desired behavior?
> > > 
> > 
> > Hi Ilya,
> > 
> > Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such
> > check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or
> > fs_name is NULL, then we cannot say that they match or not. It means that we
> > need to continue logic, this is why is_fsname_mismatch() returns false.
> > Otherwise, if we stop logic by returning true, then we have bunch of xfstests
> > failures.
> > 
> > Thanks,
> > Slava.
> > 
> > > > +
> > > > +       doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n",
> > > > +             fs_name1, fs_name2);
> > > > +
> > > > +       if (strcmp(fs_name1, fs_name2))
> > > > +               return true;
> > > > +
> > > > +       return false;
> > > > +}
> > > > +
> > > >   static int ceph_mds_auth_match(struct ceph_mds_client *mdsc,
> > > >                                 struct ceph_mds_cap_auth *auth,
> > > >                                 const struct cred *cred,
> > > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc,
> > > >          u32 gid, tlen, len;
> > > >          int i, j;
> > > > 
> > > > -       doutc(cl, "fsname check fs_name=%s  match.fs_name=%s\n",
> > > > -             fs_name, auth->match.fs_name ? auth->match.fs_name : "");
> > > > -       if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) {
> > > > +       if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) {
> > > >                  /* fsname mismatch, try next one */
> > > >                  return 0;
> > > >          }
> > > > --
> > > > 2.51.1
> > > > 
> > 

-- 
Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>