* [PATCH] ceph: fix kernel crash in ceph_open()
@ 2025-11-19 19:37 Viacheslav Dubeyko
2025-11-19 22:40 ` Ilya Dryomov
0 siblings, 1 reply; 13+ messages in thread
From: Viacheslav Dubeyko @ 2025-11-19 19:37 UTC (permalink / raw)
To: ceph-devel
Cc: idryomov, linux-fsdevel, pdonnell, amarkuze, Slava.Dubeyko, slava,
vdubeyko, khiremat, Pavan.Rallabhandi
From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
The CephFS kernel client has regression starting from 6.18-rc1.
sudo ./check -g quick
FSTYP -- ceph
PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri
Nov 14 11:26:14 PST 2025
MKFS_OPTIONS -- 192.168.1.213:3300:/scratch
MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch
/mnt/cephfs/scratch
Killed
Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0
(2)192.168.1.213:3300 session established
Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616
Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer
dereference, address: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in
kernel mode
Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not-
present page
Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN
NOPTI
Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm:
xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary)
Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC
(i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014
Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40
Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90
90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84
d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc
cc cc 31
Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0
EFLAGS: 00010246
Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX:
ffff888116003200 RCX: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI:
0000000000000000 RDI: ffff88810126c900
Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08:
0000000000000000 R09: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11:
0000000000000000 R12: dffffc0000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14:
0000000000000000 R15: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000)
GS:ffff8882401a4000(0000) knlGS:0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000
CR0: 0000000080050033
Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3:
0000000110ebd001 CR4: 0000000000772ef0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554
Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace:
Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK>
Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ?
ceph_mds_check_access+0x348/0x1760
Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ?
__kasan_check_write+0x14/0x30
Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170
Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ?
__pfx__raw_spin_lock+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ?
__pfx_apparmor_file_open+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ?
__ceph_caps_issued_mask_metric+0xd6/0x180
Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450
Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370
Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ?
__pfx_stack_trace_save+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ?
stack_depot_save_flags+0x28/0x8f0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20
Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450
Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ?
__pfx__raw_spin_lock_irqsave+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ?
__pfx__raw_spin_lock+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ?
__check_object_size+0x453/0x600
Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40
Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180
Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ?
__pfx_do_sys_openat2+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240
Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ?
__pfx___x64_sys_openat+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ?
__pfx___handle_mm_fault+0x10/0x10
Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350
Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50
Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ?
fpregs_assert_state_consistent+0x5c/0x100
Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50
Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20
Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ?
count_memcg_events+0x25b/0x400
Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20
Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ?
fpregs_assert_state_consistent+0x5c/0x100
Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ?
irqentry_exit_to_user_mode+0x2e/0x2a0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50
Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100
Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389]
entry_SYSCALL_64_after_hwframe+0x76/0x7e
Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab
Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00
41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff
b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48
2b 14 25
Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0
EFLAGS: 00000246 ORIG_RAX: 0000000000000101
Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX:
0000000000000002 RCX: 000074a85bf145ab
Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI:
00007ffc77d32789 RDI: 00000000ffffff9c
Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08:
00007ffc77d31980 R09: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11:
0000000000000246 R12: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14:
0000000000000180 R15: 0000000000000001
Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK>
Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in:
intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core
pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec
kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel
rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate
serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp
parport efi_pstore
Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000
]---
Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40
Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90
90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84
d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc
cc cc 31
Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0
EFLAGS: 00010246
Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX:
ffff888116003200 RCX: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI:
0000000000000000 RDI: ffff88810126c900
Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08:
0000000000000000 R09: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11:
0000000000000000 R12: dffffc0000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14:
0000000000000000 R15: 0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000)
GS:ffff8882401a4000(0000) knlGS:0000000000000000
Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000
CR0: 0000000080050033
Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3:
0000000110ebd001 CR4: 0000000000772ef0
Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554
We have issue here [1] if fs_name == NULL:
const char fs_name = mdsc->fsc->mount_options->mds_namespace;
...
if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) {
/ fsname mismatch, try next one */
return 0;
}
The patch fixes the issue by introducing is_fsname_mismatch() method
that checks auth->match.fs_name and fs_name pointers validity, and
compares the file system names.
[1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666
Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue")
Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com>
cc: Alex Markuze <amarkuze@redhat.com>
cc: Ilya Dryomov <idryomov@gmail.com>
cc: Ceph Development <ceph-devel@vger.kernel.org>
---
fs/ceph/mds_client.c | 20 +++++++++++++++++---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 1740047aef0f..19c75e206300 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s)
mutex_unlock(&s->s_mutex);
}
+static inline
+bool is_fsname_mismatch(struct ceph_client *cl,
+ const char *fs_name1, const char *fs_name2)
+{
+ if (!fs_name1 || !fs_name2)
+ return false;
+
+ doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n",
+ fs_name1, fs_name2);
+
+ if (strcmp(fs_name1, fs_name2))
+ return true;
+
+ return false;
+}
+
static int ceph_mds_auth_match(struct ceph_mds_client *mdsc,
struct ceph_mds_cap_auth *auth,
const struct cred *cred,
@@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc,
u32 gid, tlen, len;
int i, j;
- doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n",
- fs_name, auth->match.fs_name ? auth->match.fs_name : "");
- if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) {
+ if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) {
/* fsname mismatch, try next one */
return 0;
}
--
2.51.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH] ceph: fix kernel crash in ceph_open() 2025-11-19 19:37 [PATCH] ceph: fix kernel crash in ceph_open() Viacheslav Dubeyko @ 2025-11-19 22:40 ` Ilya Dryomov 2025-11-19 22:50 ` Viacheslav Dubeyko 0 siblings, 1 reply; 13+ messages in thread From: Ilya Dryomov @ 2025-11-19 22:40 UTC (permalink / raw) To: Viacheslav Dubeyko Cc: ceph-devel, linux-fsdevel, pdonnell, amarkuze, Slava.Dubeyko, vdubeyko, khiremat, Pavan.Rallabhandi On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote: > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > The CephFS kernel client has regression starting from 6.18-rc1. > > sudo ./check -g quick > FSTYP -- ceph > PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri > Nov 14 11:26:14 PST 2025 > MKFS_OPTIONS -- 192.168.1.213:3300:/scratch > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch > /mnt/cephfs/scratch > > Killed > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0 > (2)192.168.1.213:3300 session established > Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer > dereference, address: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in > kernel mode > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not- > present page > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN > NOPTI > Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm: > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary) > Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90 > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > cc cc 31 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0 > EFLAGS: 00010246 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX: > ffff888116003200 RCX: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI: > 0000000000000000 RDI: ffff88810126c900 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08: > 0000000000000000 R09: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11: > 0000000000000000 R12: dffffc0000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14: > 0000000000000000 R15: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000) > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000 > CR0: 0000000080050033 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3: > 0000000110ebd001 CR4: 0000000000772ef0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace: > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK> > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ? > ceph_mds_check_access+0x348/0x1760 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ? > __kasan_check_write+0x14/0x30 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ? > __pfx__raw_spin_lock+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ? > __pfx_apparmor_file_open+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ? > __ceph_caps_issued_mask_metric+0xd6/0x180 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ? > __pfx_stack_trace_save+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ? > stack_depot_save_flags+0x28/0x8f0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ? > __pfx__raw_spin_lock_irqsave+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ? > __pfx__raw_spin_lock+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ? > __check_object_size+0x453/0x600 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ? > __pfx_do_sys_openat2+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ? > __pfx___x64_sys_openat+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ? > __pfx___handle_mm_fault+0x10/0x10 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ? > fpregs_assert_state_consistent+0x5c/0x100 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ? > count_memcg_events+0x25b/0x400 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ? > fpregs_assert_state_consistent+0x5c/0x100 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ? > irqentry_exit_to_user_mode+0x2e/0x2a0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389] > entry_SYSCALL_64_after_hwframe+0x76/0x7e > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00 > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > 2b 14 25 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0 > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX: > 0000000000000002 RCX: 000074a85bf145ab > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI: > 00007ffc77d32789 RDI: 00000000ffffff9c > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08: > 00007ffc77d31980 R09: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11: > 0000000000000246 R12: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14: > 0000000000000180 R15: 0000000000000001 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK> > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in: > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > parport efi_pstore > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000 > ]--- > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90 > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > cc cc 31 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0 > EFLAGS: 00010246 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX: > ffff888116003200 RCX: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI: > 0000000000000000 RDI: ffff88810126c900 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08: > 0000000000000000 R09: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11: > 0000000000000000 R12: dffffc0000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14: > 0000000000000000 R15: 0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000) > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000 > CR0: 0000000080050033 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3: > 0000000110ebd001 CR4: 0000000000772ef0 > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554 > > We have issue here [1] if fs_name == NULL: > > const char fs_name = mdsc->fsc->mount_options->mds_namespace; > ... > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > / fsname mismatch, try next one */ > return 0; > } > > The patch fixes the issue by introducing is_fsname_mismatch() method > that checks auth->match.fs_name and fs_name pointers validity, and > compares the file system names. > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666 > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com> > cc: Alex Markuze <amarkuze@redhat.com> > cc: Ilya Dryomov <idryomov@gmail.com> > cc: Ceph Development <ceph-devel@vger.kernel.org> > --- > fs/ceph/mds_client.c | 20 +++++++++++++++++--- > 1 file changed, 17 insertions(+), 3 deletions(-) > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c > index 1740047aef0f..19c75e206300 100644 > --- a/fs/ceph/mds_client.c > +++ b/fs/ceph/mds_client.c > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s) > mutex_unlock(&s->s_mutex); > } > > +static inline > +bool is_fsname_mismatch(struct ceph_client *cl, > + const char *fs_name1, const char *fs_name2) > +{ > + if (!fs_name1 || !fs_name2) > + return false; Hi Slava, It looks like this would declare a match (return false for "mismatch") in case ceph_mds_cap_auth is defined to require a particular fs_name but no mds_namespace was passed on mount. Is that the desired behavior? Thanks, Ilya > + > + doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n", > + fs_name1, fs_name2); > + > + if (strcmp(fs_name1, fs_name2)) > + return true; > + > + return false; > +} > + > static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > struct ceph_mds_cap_auth *auth, > const struct cred *cred, > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > u32 gid, tlen, len; > int i, j; > > - doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n", > - fs_name, auth->match.fs_name ? auth->match.fs_name : ""); > - if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > + if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) { > /* fsname mismatch, try next one */ > return 0; > } > -- > 2.51.1 > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [PATCH] ceph: fix kernel crash in ceph_open() 2025-11-19 22:40 ` Ilya Dryomov @ 2025-11-19 22:50 ` Viacheslav Dubeyko [not found] ` <CAJ4mKGZexNm--cKsT0sc0vmiAyWrA1a6FtmaGJ6WOsg8d_2R3w@mail.gmail.com> 0 siblings, 1 reply; 13+ messages in thread From: Viacheslav Dubeyko @ 2025-11-19 22:50 UTC (permalink / raw) To: idryomov@gmail.com, slava@dubeyko.com Cc: Pavan Rallabhandi, Viacheslav Dubeyko, ceph-devel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Patrick Donnelly, Alex Markuze, Kotresh Hiremath Ravishankar On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote: > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote: > > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > The CephFS kernel client has regression starting from 6.18-rc1. > > > > sudo ./check -g quick > > FSTYP -- ceph > > PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri > > Nov 14 11:26:14 PST 2025 > > MKFS_OPTIONS -- 192.168.1.213:3300:/scratch > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch > > /mnt/cephfs/scratch > > > > Killed > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0 > > (2)192.168.1.213:3300 session established > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer > > dereference, address: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in > > kernel mode > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not- > > present page > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN > > NOPTI > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm: > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary) > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90 > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > cc cc 31 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0 > > EFLAGS: 00010246 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX: > > ffff888116003200 RCX: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI: > > 0000000000000000 RDI: ffff88810126c900 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08: > > 0000000000000000 R09: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11: > > 0000000000000000 R12: dffffc0000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14: > > 0000000000000000 R15: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000) > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000 > > CR0: 0000000080050033 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3: > > 0000000110ebd001 CR4: 0000000000772ef0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace: > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK> > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ? > > ceph_mds_check_access+0x348/0x1760 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ? > > __kasan_check_write+0x14/0x30 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ? > > __pfx__raw_spin_lock+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ? > > __pfx_apparmor_file_open+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ? > > __ceph_caps_issued_mask_metric+0xd6/0x180 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ? > > __pfx_stack_trace_save+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ? > > stack_depot_save_flags+0x28/0x8f0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ? > > __pfx__raw_spin_lock_irqsave+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ? > > __pfx__raw_spin_lock+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ? > > __check_object_size+0x453/0x600 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ? > > __pfx_do_sys_openat2+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ? > > __pfx___x64_sys_openat+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ? > > __pfx___handle_mm_fault+0x10/0x10 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ? > > fpregs_assert_state_consistent+0x5c/0x100 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ? > > count_memcg_events+0x25b/0x400 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ? > > fpregs_assert_state_consistent+0x5c/0x100 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ? > > irqentry_exit_to_user_mode+0x2e/0x2a0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389] > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00 > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > > 2b 14 25 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0 > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX: > > 0000000000000002 RCX: 000074a85bf145ab > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI: > > 00007ffc77d32789 RDI: 00000000ffffff9c > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08: > > 00007ffc77d31980 R09: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11: > > 0000000000000246 R12: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14: > > 0000000000000180 R15: 0000000000000001 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK> > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in: > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > > parport efi_pstore > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000 > > ]--- > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90 > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > cc cc 31 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0 > > EFLAGS: 00010246 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX: > > ffff888116003200 RCX: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI: > > 0000000000000000 RDI: ffff88810126c900 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08: > > 0000000000000000 R09: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11: > > 0000000000000000 R12: dffffc0000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14: > > 0000000000000000 R15: 0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000) > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000 > > CR0: 0000000080050033 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3: > > 0000000110ebd001 CR4: 0000000000772ef0 > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554 > > > > We have issue here [1] if fs_name == NULL: > > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace; > > ... > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > / fsname mismatch, try next one */ > > return 0; > > } > > > > The patch fixes the issue by introducing is_fsname_mismatch() method > > that checks auth->match.fs_name and fs_name pointers validity, and > > compares the file system names. > > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666 > > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com> > > cc: Alex Markuze <amarkuze@redhat.com> > > cc: Ilya Dryomov <idryomov@gmail.com> > > cc: Ceph Development <ceph-devel@vger.kernel.org> > > --- > > fs/ceph/mds_client.c | 20 +++++++++++++++++--- > > 1 file changed, 17 insertions(+), 3 deletions(-) > > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c > > index 1740047aef0f..19c75e206300 100644 > > --- a/fs/ceph/mds_client.c > > +++ b/fs/ceph/mds_client.c > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s) > > mutex_unlock(&s->s_mutex); > > } > > > > +static inline > > +bool is_fsname_mismatch(struct ceph_client *cl, > > + const char *fs_name1, const char *fs_name2) > > +{ > > + if (!fs_name1 || !fs_name2) > > + return false; > > Hi Slava, > > It looks like this would declare a match (return false for "mismatch") > in case ceph_mds_cap_auth is defined to require a particular fs_name but > no mds_namespace was passed on mount. Is that the desired behavior? > Hi Ilya, Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or fs_name is NULL, then we cannot say that they match or not. It means that we need to continue logic, this is why is_fsname_mismatch() returns false. Otherwise, if we stop logic by returning true, then we have bunch of xfstests failures. Thanks, Slava. > > + > > + doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n", > > + fs_name1, fs_name2); > > + > > + if (strcmp(fs_name1, fs_name2)) > > + return true; > > + > > + return false; > > +} > > + > > static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > struct ceph_mds_cap_auth *auth, > > const struct cred *cred, > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > u32 gid, tlen, len; > > int i, j; > > > > - doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n", > > - fs_name, auth->match.fs_name ? auth->match.fs_name : ""); > > - if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > + if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) { > > /* fsname mismatch, try next one */ > > return 0; > > } > > -- > > 2.51.1 > > ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CAJ4mKGZexNm--cKsT0sc0vmiAyWrA1a6FtmaGJ6WOsg8d_2R3w@mail.gmail.com>]
* RE: [PATCH] ceph: fix kernel crash in ceph_open() [not found] ` <CAJ4mKGZexNm--cKsT0sc0vmiAyWrA1a6FtmaGJ6WOsg8d_2R3w@mail.gmail.com> @ 2025-11-19 23:16 ` Viacheslav Dubeyko 2025-11-20 14:20 ` Kotresh Hiremath Ravishankar 0 siblings, 1 reply; 13+ messages in thread From: Viacheslav Dubeyko @ 2025-11-19 23:16 UTC (permalink / raw) To: Gregory Farnum Cc: Viacheslav Dubeyko, linux-fsdevel@vger.kernel.org, slava@dubeyko.com, ceph-devel@vger.kernel.org, Patrick Donnelly, Alex Markuze, Kotresh Hiremath Ravishankar, idryomov@gmail.com, Pavan Rallabhandi On Wed, 2025-11-19 at 15:02 -0800, Gregory Farnum wrote: > > That doesn’t sound right — this is authentication code. If the authorization is supplied for a namespace and we are mounting without a namespace at all, isn’t that a jailbreak? So the NULL pointer should be accepted in one direction, but denied in the other? What is your particular suggestion? I am simply fixing the kernel crash after the 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"). We didn't have any check before. Do you imply that 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") fix is incorrect and we need to rework it somehow? If we will not have any fix, then 6.18 release will have broken CephFS kernel client. Thanks, Slava. > > On Wed, Nov 19, 2025 at 2:54 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote: > > > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote: > > > > > > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > > The CephFS kernel client has regression starting from 6.18-rc1. > > > > > > > > sudo ./check -g quick > > > > FSTYP -- ceph > > > > PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri > > > > Nov 14 11:26:14 PST 2025 > > > > MKFS_OPTIONS -- 192.168.1.213:3300:/scratch > > > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch > > > > /mnt/cephfs/scratch > > > > > > > > Killed > > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0 > > > > (2)192.168.1.213:3300 session established > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer > > > > dereference, address: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in > > > > kernel mode > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not- > > > > present page > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN > > > > NOPTI > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm: > > > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary) > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC > > > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90 > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > cc cc 31 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0 > > > > EFLAGS: 00010246 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX: > > > > ffff888116003200 RCX: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI: > > > > 0000000000000000 RDI: ffff88810126c900 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08: > > > > 0000000000000000 R09: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11: > > > > 0000000000000000 R12: dffffc0000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14: > > > > 0000000000000000 R15: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000) > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000 > > > > CR0: 0000000080050033 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3: > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace: > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK> > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ? > > > > ceph_mds_check_access+0x348/0x1760 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ? > > > > __kasan_check_write+0x14/0x30 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ? > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ? > > > > __pfx_apparmor_file_open+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ? > > > > __ceph_caps_issued_mask_metric+0xd6/0x180 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ? > > > > __pfx_stack_trace_save+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ? > > > > stack_depot_save_flags+0x28/0x8f0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ? > > > > __pfx__raw_spin_lock_irqsave+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ? > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ? > > > > __check_object_size+0x453/0x600 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ? > > > > __pfx_do_sys_openat2+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ? > > > > __pfx___x64_sys_openat+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ? > > > > __pfx___handle_mm_fault+0x10/0x10 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ? > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ? > > > > count_memcg_events+0x25b/0x400 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ? > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ? > > > > irqentry_exit_to_user_mode+0x2e/0x2a0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389] > > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00 > > > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > > > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > > > > 2b 14 25 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0 > > > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX: > > > > 0000000000000002 RCX: 000074a85bf145ab > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI: > > > > 00007ffc77d32789 RDI: 00000000ffffff9c > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08: > > > > 00007ffc77d31980 R09: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11: > > > > 0000000000000246 R12: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14: > > > > 0000000000000180 R15: 0000000000000001 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK> > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in: > > > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > > > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > > > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > > > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate > > > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > > > > parport efi_pstore > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000 > > > > ]--- > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90 > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > cc cc 31 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0 > > > > EFLAGS: 00010246 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX: > > > > ffff888116003200 RCX: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI: > > > > 0000000000000000 RDI: ffff88810126c900 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08: > > > > 0000000000000000 R09: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11: > > > > 0000000000000000 R12: dffffc0000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14: > > > > 0000000000000000 R15: 0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000) > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000 > > > > CR0: 0000000080050033 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3: > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554 > > > > > > > > We have issue here [1] if fs_name == NULL: > > > > > > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace; > > > > ... > > > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > / fsname mismatch, try next one */ > > > > return 0; > > > > } > > > > > > > > The patch fixes the issue by introducing is_fsname_mismatch() method > > > > that checks auth->match.fs_name and fs_name pointers validity, and > > > > compares the file system names. > > > > > > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666 > > > > > > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") > > > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com> > > > > cc: Alex Markuze <amarkuze@redhat.com> > > > > cc: Ilya Dryomov <idryomov@gmail.com> > > > > cc: Ceph Development <ceph-devel@vger.kernel.org> > > > > --- > > > > fs/ceph/mds_client.c | 20 +++++++++++++++++--- > > > > 1 file changed, 17 insertions(+), 3 deletions(-) > > > > > > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c > > > > index 1740047aef0f..19c75e206300 100644 > > > > --- a/fs/ceph/mds_client.c > > > > +++ b/fs/ceph/mds_client.c > > > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s) > > > > mutex_unlock(&s->s_mutex); > > > > } > > > > > > > > +static inline > > > > +bool is_fsname_mismatch(struct ceph_client *cl, > > > > + const char *fs_name1, const char *fs_name2) > > > > +{ > > > > + if (!fs_name1 || !fs_name2) > > > > + return false; > > > > > > Hi Slava, > > > > > > It looks like this would declare a match (return false for "mismatch") > > > in case ceph_mds_cap_auth is defined to require a particular fs_name but > > > no mds_namespace was passed on mount. Is that the desired behavior? > > > > > > > Hi Ilya, > > > > Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such > > check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or > > fs_name is NULL, then we cannot say that they match or not. It means that we > > need to continue logic, this is why is_fsname_mismatch() returns false. > > Otherwise, if we stop logic by returning true, then we have bunch of xfstests > > failures. > > > > Thanks, > > Slava. > > > > > > + > > > > + doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n", > > > > + fs_name1, fs_name2); > > > > + > > > > + if (strcmp(fs_name1, fs_name2)) > > > > + return true; > > > > + > > > > + return false; > > > > +} > > > > + > > > > static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > struct ceph_mds_cap_auth *auth, > > > > const struct cred *cred, > > > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > u32 gid, tlen, len; > > > > int i, j; > > > > > > > > - doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n", > > > > - fs_name, auth->match.fs_name ? auth->match.fs_name : ""); > > > > - if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > + if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) { > > > > /* fsname mismatch, try next one */ > > > > return 0; > > > > } > > > > -- > > > > 2.51.1 > > > > > > -- Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] ceph: fix kernel crash in ceph_open() 2025-11-19 23:16 ` Viacheslav Dubeyko @ 2025-11-20 14:20 ` Kotresh Hiremath Ravishankar 2025-11-20 20:17 ` Viacheslav Dubeyko 0 siblings, 1 reply; 13+ messages in thread From: Kotresh Hiremath Ravishankar @ 2025-11-20 14:20 UTC (permalink / raw) To: Viacheslav Dubeyko Cc: Gregory Farnum, Viacheslav Dubeyko, linux-fsdevel@vger.kernel.org, slava@dubeyko.com, ceph-devel@vger.kernel.org, Patrick Donnelly, Alex Markuze, idryomov@gmail.com, Pavan Rallabhandi Hi All, I think the patch is necessary and fixes the crash. There is no harm in taking this patch as it behaves like an old kernel with this particular scenario. When does the issue happen: - The issue happens only when the old mount syntax is used where passing the file system name is optional in which case, it chooses the default mds namespace but doesn't get filled in the mdsc->fsc->mount_options->mds_namespace. - Along with the above, the mount user should be non admin. Does it break the earlier fix ? - Not fully!!! Though the open does succeed, the subsequent operation like write would get EPERM. I am not exactly able to recollect but this was discussed before writing the fix 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), it's guarded by another check before actual operation like write. I think there are a couple of options to fix this cleanly. 1. Use the default fsname when mdsc->fsc->mount_options->mds_namespace is NULL during comparison. 2. Mandate passing the fsname with old syntax ? Thanks, Kotresh H R On Thu, Nov 20, 2025 at 4:47 AM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > On Wed, 2025-11-19 at 15:02 -0800, Gregory Farnum wrote: > > > > That doesn’t sound right — this is authentication code. If the authorization is supplied for a namespace and we are mounting without a namespace at all, isn’t that a jailbreak? So the NULL pointer should be accepted in one direction, but denied in the other? > > What is your particular suggestion? I am simply fixing the kernel crash after > the 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"). We didn't have any > check before. Do you imply that 22c73d52a6d0 ("ceph: fix multifs mds auth caps > issue") fix is incorrect and we need to rework it somehow? > > If we will not have any fix, then 6.18 release will have broken CephFS kernel > client. > > Thanks, > Slava. > > > > > On Wed, Nov 19, 2025 at 2:54 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > > On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote: > > > > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote: > > > > > > > > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > > > > The CephFS kernel client has regression starting from 6.18-rc1. > > > > > > > > > > sudo ./check -g quick > > > > > FSTYP -- ceph > > > > > PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri > > > > > Nov 14 11:26:14 PST 2025 > > > > > MKFS_OPTIONS -- 192.168.1.213:3300:/scratch > > > > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch > > > > > /mnt/cephfs/scratch > > > > > > > > > > Killed > > > > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0 > > > > > (2)192.168.1.213:3300 session established > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer > > > > > dereference, address: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in > > > > > kernel mode > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not- > > > > > present page > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN > > > > > NOPTI > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm: > > > > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary) > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC > > > > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90 > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > cc cc 31 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0 > > > > > EFLAGS: 00010246 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX: > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI: > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08: > > > > > 0000000000000000 R09: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11: > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14: > > > > > 0000000000000000 R15: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000) > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000 > > > > > CR0: 0000000080050033 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3: > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace: > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ? > > > > > ceph_mds_check_access+0x348/0x1760 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ? > > > > > __kasan_check_write+0x14/0x30 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ? > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ? > > > > > __pfx_apparmor_file_open+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ? > > > > > __ceph_caps_issued_mask_metric+0xd6/0x180 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ? > > > > > __pfx_stack_trace_save+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ? > > > > > stack_depot_save_flags+0x28/0x8f0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ? > > > > > __pfx__raw_spin_lock_irqsave+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ? > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ? > > > > > __check_object_size+0x453/0x600 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ? > > > > > __pfx_do_sys_openat2+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ? > > > > > __pfx___x64_sys_openat+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ? > > > > > __pfx___handle_mm_fault+0x10/0x10 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ? > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ? > > > > > count_memcg_events+0x25b/0x400 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ? > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ? > > > > > irqentry_exit_to_user_mode+0x2e/0x2a0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389] > > > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00 > > > > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > > > > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > > > > > 2b 14 25 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0 > > > > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX: > > > > > 0000000000000002 RCX: 000074a85bf145ab > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI: > > > > > 00007ffc77d32789 RDI: 00000000ffffff9c > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08: > > > > > 00007ffc77d31980 R09: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11: > > > > > 0000000000000246 R12: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14: > > > > > 0000000000000180 R15: 0000000000000001 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK> > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in: > > > > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > > > > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > > > > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > > > > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate > > > > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > > > > > parport efi_pstore > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000 > > > > > ]--- > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90 > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > cc cc 31 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0 > > > > > EFLAGS: 00010246 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX: > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI: > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08: > > > > > 0000000000000000 R09: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11: > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14: > > > > > 0000000000000000 R15: 0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000) > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000 > > > > > CR0: 0000000080050033 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3: > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554 > > > > > > > > > > We have issue here [1] if fs_name == NULL: > > > > > > > > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace; > > > > > ... > > > > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > / fsname mismatch, try next one */ > > > > > return 0; > > > > > } > > > > > > > > > > The patch fixes the issue by introducing is_fsname_mismatch() method > > > > > that checks auth->match.fs_name and fs_name pointers validity, and > > > > > compares the file system names. > > > > > > > > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666 > > > > > > > > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") > > > > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com> > > > > > cc: Alex Markuze <amarkuze@redhat.com> > > > > > cc: Ilya Dryomov <idryomov@gmail.com> > > > > > cc: Ceph Development <ceph-devel@vger.kernel.org> > > > > > --- > > > > > fs/ceph/mds_client.c | 20 +++++++++++++++++--- > > > > > 1 file changed, 17 insertions(+), 3 deletions(-) > > > > > > > > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c > > > > > index 1740047aef0f..19c75e206300 100644 > > > > > --- a/fs/ceph/mds_client.c > > > > > +++ b/fs/ceph/mds_client.c > > > > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s) > > > > > mutex_unlock(&s->s_mutex); > > > > > } > > > > > > > > > > +static inline > > > > > +bool is_fsname_mismatch(struct ceph_client *cl, > > > > > + const char *fs_name1, const char *fs_name2) > > > > > +{ > > > > > + if (!fs_name1 || !fs_name2) > > > > > + return false; > > > > > > > > Hi Slava, > > > > > > > > It looks like this would declare a match (return false for "mismatch") > > > > in case ceph_mds_cap_auth is defined to require a particular fs_name but > > > > no mds_namespace was passed on mount. Is that the desired behavior? > > > > > > > > > > Hi Ilya, > > > > > > Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such > > > check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or > > > fs_name is NULL, then we cannot say that they match or not. It means that we > > > need to continue logic, this is why is_fsname_mismatch() returns false. > > > Otherwise, if we stop logic by returning true, then we have bunch of xfstests > > > failures. > > > > > > Thanks, > > > Slava. > > > > > > > > + > > > > > + doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n", > > > > > + fs_name1, fs_name2); > > > > > + > > > > > + if (strcmp(fs_name1, fs_name2)) > > > > > + return true; > > > > > + > > > > > + return false; > > > > > +} > > > > > + > > > > > static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > struct ceph_mds_cap_auth *auth, > > > > > const struct cred *cred, > > > > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > u32 gid, tlen, len; > > > > > int i, j; > > > > > > > > > > - doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n", > > > > > - fs_name, auth->match.fs_name ? auth->match.fs_name : ""); > > > > > - if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > + if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) { > > > > > /* fsname mismatch, try next one */ > > > > > return 0; > > > > > } > > > > > -- > > > > > 2.51.1 > > > > > > > > > > -- > Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [PATCH] ceph: fix kernel crash in ceph_open() 2025-11-20 14:20 ` Kotresh Hiremath Ravishankar @ 2025-11-20 20:17 ` Viacheslav Dubeyko 2025-11-24 19:18 ` Kotresh Hiremath Ravishankar 0 siblings, 1 reply; 13+ messages in thread From: Viacheslav Dubeyko @ 2025-11-20 20:17 UTC (permalink / raw) To: Kotresh Hiremath Ravishankar Cc: Viacheslav Dubeyko, Patrick Donnelly, ceph-devel@vger.kernel.org, slava@dubeyko.com, linux-fsdevel@vger.kernel.org, Gregory Farnum, Alex Markuze, idryomov@gmail.com, Pavan Rallabhandi On Thu, 2025-11-20 at 19:50 +0530, Kotresh Hiremath Ravishankar wrote: > Hi All, > > I think the patch is necessary and fixes the crash. There is no harm > in taking this patch as it behaves like an old kernel with this > particular scenario. > > When does the issue happen: > - The issue happens only when the old mount syntax is used where > passing the file system name is optional in which case, it chooses the > default mds namespace but doesn't get filled in the > mdsc->fsc->mount_options->mds_namespace. > - Along with the above, the mount user should be non admin. > Does it break the earlier fix ? > - Not fully!!! Though the open does succeed, the subsequent > operation like write would get EPERM. I am not exactly able to > recollect but this was discussed before writing the fix 22c73d52a6d0 > ("ceph: fix multifs mds auth caps issue"), it's guarded by another > check before actual operation like write. > > I think there are a couple of options to fix this cleanly. > 1. Use the default fsname when > mdsc->fsc->mount_options->mds_namespace is NULL during comparison. > 2. Mandate passing the fsname with old syntax ? > Anyway, we should be ready operate correctly if fsname or/and auth- >match.fs_name are NULL. And if we need to make the fix more cleanly, then we can introduce another patch with nicer fix. I am not completely sure how default fsname can be applicable here. If I understood the CephFS mount logic correctly, then fsname can be NULL during some initial steps. But, finally, we will have the real fsname for comparison. But I don't know if it's right of assuming that fsname == NULL is equal to fsname == default_name. And I am not sure that we can mandate anyone to use the old syntax. If there is some other opportunity, then someone could use it. But, maybe, I am missing the point. :) What do you mean by "Mandate passing the fsname with old syntax"? Thanks, Slava. > > Thanks, > Kotresh H R > > > > On Thu, Nov 20, 2025 at 4:47 AM Viacheslav Dubeyko > <Slava.Dubeyko@ibm.com> wrote: > > > > On Wed, 2025-11-19 at 15:02 -0800, Gregory Farnum wrote: > > > > > > That doesn’t sound right — this is authentication code. If the authorization is supplied for a namespace and we are mounting without a namespace at all, isn’t that a jailbreak? So the NULL pointer should be accepted in one direction, but denied in the other? > > > > What is your particular suggestion? I am simply fixing the kernel crash after > > the 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"). We didn't have any > > check before. Do you imply that 22c73d52a6d0 ("ceph: fix multifs mds auth caps > > issue") fix is incorrect and we need to rework it somehow? > > > > If we will not have any fix, then 6.18 release will have broken CephFS kernel > > client. > > > > Thanks, > > Slava. > > > > > > > > On Wed, Nov 19, 2025 at 2:54 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > > > On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote: > > > > > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote: > > > > > > > > > > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > > > > > > The CephFS kernel client has regression starting from 6.18-rc1. > > > > > > > > > > > > sudo ./check -g quick > > > > > > FSTYP -- ceph > > > > > > PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri > > > > > > Nov 14 11:26:14 PST 2025 > > > > > > MKFS_OPTIONS -- 192.168.1.213:3300:/scratch > > > > > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch > > > > > > /mnt/cephfs/scratch > > > > > > > > > > > > Killed > > > > > > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0 > > > > > > (2)192.168.1.213:3300 session established > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer > > > > > > dereference, address: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in > > > > > > kernel mode > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not- > > > > > > present page > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN > > > > > > NOPTI > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm: > > > > > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary) > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC > > > > > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90 > > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > > cc cc 31 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0 > > > > > > EFLAGS: 00010246 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX: > > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI: > > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08: > > > > > > 0000000000000000 R09: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11: > > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14: > > > > > > 0000000000000000 R15: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000) > > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000 > > > > > > CR0: 0000000080050033 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3: > > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace: > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK> > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ? > > > > > > ceph_mds_check_access+0x348/0x1760 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ? > > > > > > __kasan_check_write+0x14/0x30 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ? > > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ? > > > > > > __pfx_apparmor_file_open+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ? > > > > > > __ceph_caps_issued_mask_metric+0xd6/0x180 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ? > > > > > > __pfx_stack_trace_save+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ? > > > > > > stack_depot_save_flags+0x28/0x8f0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ? > > > > > > __pfx__raw_spin_lock_irqsave+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ? > > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ? > > > > > > __check_object_size+0x453/0x600 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ? > > > > > > __pfx_do_sys_openat2+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ? > > > > > > __pfx___x64_sys_openat+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ? > > > > > > __pfx___handle_mm_fault+0x10/0x10 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ? > > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ? > > > > > > count_memcg_events+0x25b/0x400 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ? > > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ? > > > > > > irqentry_exit_to_user_mode+0x2e/0x2a0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389] > > > > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00 > > > > > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > > > > > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > > > > > > 2b 14 25 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0 > > > > > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX: > > > > > > 0000000000000002 RCX: 000074a85bf145ab > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI: > > > > > > 00007ffc77d32789 RDI: 00000000ffffff9c > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08: > > > > > > 00007ffc77d31980 R09: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11: > > > > > > 0000000000000246 R12: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14: > > > > > > 0000000000000180 R15: 0000000000000001 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK> > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in: > > > > > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > > > > > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > > > > > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > > > > > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate > > > > > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > > > > > > parport efi_pstore > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000 > > > > > > ]--- > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90 > > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > > cc cc 31 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0 > > > > > > EFLAGS: 00010246 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX: > > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI: > > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08: > > > > > > 0000000000000000 R09: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11: > > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14: > > > > > > 0000000000000000 R15: 0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000) > > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000 > > > > > > CR0: 0000000080050033 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3: > > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554 > > > > > > > > > > > > We have issue here [1] if fs_name == NULL: > > > > > > > > > > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace; > > > > > > ... > > > > > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > > / fsname mismatch, try next one */ > > > > > > return 0; > > > > > > } > > > > > > > > > > > > The patch fixes the issue by introducing is_fsname_mismatch() method > > > > > > that checks auth->match.fs_name and fs_name pointers validity, and > > > > > > compares the file system names. > > > > > > > > > > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666 > > > > > > > > > > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") > > > > > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com> > > > > > > cc: Alex Markuze <amarkuze@redhat.com> > > > > > > cc: Ilya Dryomov <idryomov@gmail.com> > > > > > > cc: Ceph Development <ceph-devel@vger.kernel.org> > > > > > > --- > > > > > > fs/ceph/mds_client.c | 20 +++++++++++++++++--- > > > > > > 1 file changed, 17 insertions(+), 3 deletions(-) > > > > > > > > > > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c > > > > > > index 1740047aef0f..19c75e206300 100644 > > > > > > --- a/fs/ceph/mds_client.c > > > > > > +++ b/fs/ceph/mds_client.c > > > > > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s) > > > > > > mutex_unlock(&s->s_mutex); > > > > > > } > > > > > > > > > > > > +static inline > > > > > > +bool is_fsname_mismatch(struct ceph_client *cl, > > > > > > + const char *fs_name1, const char *fs_name2) > > > > > > +{ > > > > > > + if (!fs_name1 || !fs_name2) > > > > > > + return false; > > > > > > > > > > Hi Slava, > > > > > > > > > > It looks like this would declare a match (return false for "mismatch") > > > > > in case ceph_mds_cap_auth is defined to require a particular fs_name but > > > > > no mds_namespace was passed on mount. Is that the desired behavior? > > > > > > > > > > > > > Hi Ilya, > > > > > > > > Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such > > > > check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or > > > > fs_name is NULL, then we cannot say that they match or not. It means that we > > > > need to continue logic, this is why is_fsname_mismatch() returns false. > > > > Otherwise, if we stop logic by returning true, then we have bunch of xfstests > > > > failures. > > > > > > > > Thanks, > > > > Slava. > > > > > > > > > > + > > > > > > + doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n", > > > > > > + fs_name1, fs_name2); > > > > > > + > > > > > > + if (strcmp(fs_name1, fs_name2)) > > > > > > + return true; > > > > > > + > > > > > > + return false; > > > > > > +} > > > > > > + > > > > > > static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > > struct ceph_mds_cap_auth *auth, > > > > > > const struct cred *cred, > > > > > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > > u32 gid, tlen, len; > > > > > > int i, j; > > > > > > > > > > > > - doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n", > > > > > > - fs_name, auth->match.fs_name ? auth->match.fs_name : ""); > > > > > > - if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > > + if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) { > > > > > > /* fsname mismatch, try next one */ > > > > > > return 0; > > > > > > } > > > > > > -- > > > > > > 2.51.1 > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] ceph: fix kernel crash in ceph_open() 2025-11-20 20:17 ` Viacheslav Dubeyko @ 2025-11-24 19:18 ` Kotresh Hiremath Ravishankar 2025-11-24 21:12 ` Viacheslav Dubeyko 0 siblings, 1 reply; 13+ messages in thread From: Kotresh Hiremath Ravishankar @ 2025-11-24 19:18 UTC (permalink / raw) To: Viacheslav Dubeyko Cc: Viacheslav Dubeyko, Patrick Donnelly, ceph-devel@vger.kernel.org, slava@dubeyko.com, linux-fsdevel@vger.kernel.org, Gregory Farnum, Alex Markuze, idryomov@gmail.com, Pavan Rallabhandi On Fri, Nov 21, 2025 at 1:47 AM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > On Thu, 2025-11-20 at 19:50 +0530, Kotresh Hiremath Ravishankar wrote: > > Hi All, > > > > I think the patch is necessary and fixes the crash. There is no harm > > in taking this patch as it behaves like an old kernel with this > > particular scenario. > > > > When does the issue happen: > > - The issue happens only when the old mount syntax is used where > > passing the file system name is optional in which case, it chooses the > > default mds namespace but doesn't get filled in the > > mdsc->fsc->mount_options->mds_namespace. > > - Along with the above, the mount user should be non admin. > > Does it break the earlier fix ? > > - Not fully!!! Though the open does succeed, the subsequent > > operation like write would get EPERM. I am not exactly able to > > recollect but this was discussed before writing the fix 22c73d52a6d0 > > ("ceph: fix multifs mds auth caps issue"), it's guarded by another > > check before actual operation like write. > > > > I think there are a couple of options to fix this cleanly. > > 1. Use the default fsname when > > mdsc->fsc->mount_options->mds_namespace is NULL during comparison. > > 2. Mandate passing the fsname with old syntax ? > > > > Anyway, we should be ready operate correctly if fsname or/and auth- > >match.fs_name are NULL. And if we need to make the fix more cleanly, then we > can introduce another patch with nicer fix. > > I am not completely sure how default fsname can be applicable here. If I > understood the CephFS mount logic correctly, then fsname can be NULL during some > initial steps. But, finally, we will have the real fsname for comparison. But I > don't know if it's right of assuming that fsname == NULL is equal to fsname == > default_name. We are pretty sure fsname is NULL only if the old mount syntax is used without providing the fsname in the optional arg. I believe kclient knows the fsname that's mounted somewhere in this case ? I am not sure though. If so, it can be used. If not, then can we rely on what mds sends as part of the mdsmap? With this fix, did the tests run fine ? Aren't you hitting this error https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mdsmap.c#L365 ? > > And I am not sure that we can mandate anyone to use the old syntax. If there is > some other opportunity, then someone could use it. But, maybe, I am missing the > point. :) What do you mean by "Mandate passing the fsname with old syntax"? In the old mount syntax, the fsname is passed as on optional argument using 'mds_namespace'. I was suggesting to mandate it if possible. But I guess it breaks backward compatibility. > > Thanks, > Slava. > > > > > Thanks, > > Kotresh H R > > > > > > > > On Thu, Nov 20, 2025 at 4:47 AM Viacheslav Dubeyko > > <Slava.Dubeyko@ibm.com> wrote: > > > > > > On Wed, 2025-11-19 at 15:02 -0800, Gregory Farnum wrote: > > > > > > > > That doesn’t sound right — this is authentication code. If the authorization is supplied for a namespace and we are mounting without a namespace at all, isn’t that a jailbreak? So the NULL pointer should be accepted in one direction, but denied in the other? > > > > > > What is your particular suggestion? I am simply fixing the kernel crash after > > > the 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"). We didn't have any > > > check before. Do you imply that 22c73d52a6d0 ("ceph: fix multifs mds auth caps > > > issue") fix is incorrect and we need to rework it somehow? > > > > > > If we will not have any fix, then 6.18 release will have broken CephFS kernel > > > client. > > > > > > Thanks, > > > Slava. > > > > > > > > > > > On Wed, Nov 19, 2025 at 2:54 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > > > > On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote: > > > > > > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote: > > > > > > > > > > > > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > > > > > > > > The CephFS kernel client has regression starting from 6.18-rc1. > > > > > > > > > > > > > > sudo ./check -g quick > > > > > > > FSTYP -- ceph > > > > > > > PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri > > > > > > > Nov 14 11:26:14 PST 2025 > > > > > > > MKFS_OPTIONS -- 192.168.1.213:3300:/scratch > > > > > > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch > > > > > > > /mnt/cephfs/scratch > > > > > > > > > > > > > > Killed > > > > > > > > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0 > > > > > > > (2)192.168.1.213:3300 session established > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer > > > > > > > dereference, address: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in > > > > > > > kernel mode > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not- > > > > > > > present page > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN > > > > > > > NOPTI > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm: > > > > > > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary) > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC > > > > > > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90 > > > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > > > cc cc 31 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0 > > > > > > > EFLAGS: 00010246 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX: > > > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI: > > > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08: > > > > > > > 0000000000000000 R09: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11: > > > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14: > > > > > > > 0000000000000000 R15: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000) > > > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000 > > > > > > > CR0: 0000000080050033 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3: > > > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace: > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK> > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ? > > > > > > > ceph_mds_check_access+0x348/0x1760 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ? > > > > > > > __kasan_check_write+0x14/0x30 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ? > > > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ? > > > > > > > __pfx_apparmor_file_open+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ? > > > > > > > __ceph_caps_issued_mask_metric+0xd6/0x180 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ? > > > > > > > __pfx_stack_trace_save+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ? > > > > > > > stack_depot_save_flags+0x28/0x8f0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ? > > > > > > > __pfx__raw_spin_lock_irqsave+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ? > > > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ? > > > > > > > __check_object_size+0x453/0x600 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ? > > > > > > > __pfx_do_sys_openat2+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ? > > > > > > > __pfx___x64_sys_openat+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ? > > > > > > > __pfx___handle_mm_fault+0x10/0x10 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ? > > > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ? > > > > > > > count_memcg_events+0x25b/0x400 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ? > > > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ? > > > > > > > irqentry_exit_to_user_mode+0x2e/0x2a0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389] > > > > > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00 > > > > > > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > > > > > > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > > > > > > > 2b 14 25 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0 > > > > > > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX: > > > > > > > 0000000000000002 RCX: 000074a85bf145ab > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI: > > > > > > > 00007ffc77d32789 RDI: 00000000ffffff9c > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08: > > > > > > > 00007ffc77d31980 R09: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11: > > > > > > > 0000000000000246 R12: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14: > > > > > > > 0000000000000180 R15: 0000000000000001 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK> > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in: > > > > > > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > > > > > > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > > > > > > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > > > > > > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate > > > > > > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > > > > > > > parport efi_pstore > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000 > > > > > > > ]--- > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90 > > > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > > > cc cc 31 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0 > > > > > > > EFLAGS: 00010246 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX: > > > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI: > > > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08: > > > > > > > 0000000000000000 R09: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11: > > > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14: > > > > > > > 0000000000000000 R15: 0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000) > > > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000 > > > > > > > CR0: 0000000080050033 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3: > > > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554 > > > > > > > > > > > > > > We have issue here [1] if fs_name == NULL: > > > > > > > > > > > > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace; > > > > > > > ... > > > > > > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > > > / fsname mismatch, try next one */ > > > > > > > return 0; > > > > > > > } > > > > > > > > > > > > > > The patch fixes the issue by introducing is_fsname_mismatch() method > > > > > > > that checks auth->match.fs_name and fs_name pointers validity, and > > > > > > > compares the file system names. > > > > > > > > > > > > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666 > > > > > > > > > > > > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") > > > > > > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com> > > > > > > > cc: Alex Markuze <amarkuze@redhat.com> > > > > > > > cc: Ilya Dryomov <idryomov@gmail.com> > > > > > > > cc: Ceph Development <ceph-devel@vger.kernel.org> > > > > > > > --- > > > > > > > fs/ceph/mds_client.c | 20 +++++++++++++++++--- > > > > > > > 1 file changed, 17 insertions(+), 3 deletions(-) > > > > > > > > > > > > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c > > > > > > > index 1740047aef0f..19c75e206300 100644 > > > > > > > --- a/fs/ceph/mds_client.c > > > > > > > +++ b/fs/ceph/mds_client.c > > > > > > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s) > > > > > > > mutex_unlock(&s->s_mutex); > > > > > > > } > > > > > > > > > > > > > > +static inline > > > > > > > +bool is_fsname_mismatch(struct ceph_client *cl, > > > > > > > + const char *fs_name1, const char *fs_name2) > > > > > > > +{ > > > > > > > + if (!fs_name1 || !fs_name2) > > > > > > > + return false; > > > > > > > > > > > > Hi Slava, > > > > > > > > > > > > It looks like this would declare a match (return false for "mismatch") > > > > > > in case ceph_mds_cap_auth is defined to require a particular fs_name but > > > > > > no mds_namespace was passed on mount. Is that the desired behavior? > > > > > > > > > > > > > > > > Hi Ilya, > > > > > > > > > > Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such > > > > > check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or > > > > > fs_name is NULL, then we cannot say that they match or not. It means that we > > > > > need to continue logic, this is why is_fsname_mismatch() returns false. > > > > > Otherwise, if we stop logic by returning true, then we have bunch of xfstests > > > > > failures. > > > > > > > > > > Thanks, > > > > > Slava. > > > > > > > > > > > > + > > > > > > > + doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n", > > > > > > > + fs_name1, fs_name2); > > > > > > > + > > > > > > > + if (strcmp(fs_name1, fs_name2)) > > > > > > > + return true; > > > > > > > + > > > > > > > + return false; > > > > > > > +} > > > > > > > + > > > > > > > static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > > > struct ceph_mds_cap_auth *auth, > > > > > > > const struct cred *cred, > > > > > > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > > > u32 gid, tlen, len; > > > > > > > int i, j; > > > > > > > > > > > > > > - doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n", > > > > > > > - fs_name, auth->match.fs_name ? auth->match.fs_name : ""); > > > > > > > - if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > > > + if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) { > > > > > > > /* fsname mismatch, try next one */ > > > > > > > return 0; > > > > > > > } > > > > > > > -- > > > > > > > 2.51.1 > > > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [PATCH] ceph: fix kernel crash in ceph_open() 2025-11-24 19:18 ` Kotresh Hiremath Ravishankar @ 2025-11-24 21:12 ` Viacheslav Dubeyko 2025-11-27 7:33 ` Kotresh Hiremath Ravishankar 0 siblings, 1 reply; 13+ messages in thread From: Viacheslav Dubeyko @ 2025-11-24 21:12 UTC (permalink / raw) To: Kotresh Hiremath Ravishankar Cc: Viacheslav Dubeyko, linux-fsdevel@vger.kernel.org, ceph-devel@vger.kernel.org, slava@dubeyko.com, Patrick Donnelly, Gregory Farnum, Alex Markuze, idryomov@gmail.com, Pavan Rallabhandi On Tue, 2025-11-25 at 00:48 +0530, Kotresh Hiremath Ravishankar wrote: > On Fri, Nov 21, 2025 at 1:47 AM Viacheslav Dubeyko > <Slava.Dubeyko@ibm.com> wrote: > > > > On Thu, 2025-11-20 at 19:50 +0530, Kotresh Hiremath Ravishankar wrote: > > > Hi All, > > > > > > I think the patch is necessary and fixes the crash. There is no harm > > > in taking this patch as it behaves like an old kernel with this > > > particular scenario. > > > > > > When does the issue happen: > > > - The issue happens only when the old mount syntax is used where > > > passing the file system name is optional in which case, it chooses the > > > default mds namespace but doesn't get filled in the > > > mdsc->fsc->mount_options->mds_namespace. > > > - Along with the above, the mount user should be non admin. > > > Does it break the earlier fix ? > > > - Not fully!!! Though the open does succeed, the subsequent > > > operation like write would get EPERM. I am not exactly able to > > > recollect but this was discussed before writing the fix 22c73d52a6d0 > > > ("ceph: fix multifs mds auth caps issue"), it's guarded by another > > > check before actual operation like write. > > > > > > I think there are a couple of options to fix this cleanly. > > > 1. Use the default fsname when > > > mdsc->fsc->mount_options->mds_namespace is NULL during comparison. > > > 2. Mandate passing the fsname with old syntax ? > > > > > > > Anyway, we should be ready operate correctly if fsname or/and auth- > > > match.fs_name are NULL. And if we need to make the fix more cleanly, then we > > can introduce another patch with nicer fix. > > > > I am not completely sure how default fsname can be applicable here. If I > > understood the CephFS mount logic correctly, then fsname can be NULL during some > > initial steps. But, finally, we will have the real fsname for comparison. But I > > don't know if it's right of assuming that fsname == NULL is equal to fsname == > > default_name. > > We are pretty sure fsname is NULL only if the old mount syntax is used > without providing the > fsname in the optional arg. I believe kclient knows the fsname that's > mounted somewhere in this case ? > I am not sure though. If so, it can be used. If not, then can we rely > on what mds sends as part > of the mdsmap? > > With this fix, did the tests run fine ? > The xfstests works fine with the fix. I don't see any issues with the it. > Aren't you hitting this error > https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mdsmap.c#L365 > ? > I am not sure how this error can be triggered. I see this sequence: Nov 24 12:51:10 ceph-0005 kernel: [ 89.621635] ceph: super.c:63 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] begin Nov 24 12:51:10 ceph-0005 kernel: [ 89.624691] ceph: super.c:117 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] done Nov 24 12:51:10 ceph-0005 kernel: [ 89.625349] ceph: super.c:63 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] begin Nov 24 12:51:10 ceph-0005 kernel: [ 89.627776] ceph: super.c:117 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] done Nov 24 12:51:10 ceph-0005 kernel: [ 89.645611] ceph: super.c:63 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] begin Nov 24 12:51:10 ceph-0005 kernel: [ 89.652534] ceph: super.c:117 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] done Nov 24 12:51:10 ceph-0005 kernel: [ 89.654695] ceph: super.c:63 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] begin Nov 24 12:51:10 ceph-0005 kernel: [ 89.656220] ceph: super.c:117 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] done Nov 24 12:51:10 ceph-0005 kernel: [ 89.678877] ceph: file.c:389 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] 000000000e172927 10000000000.fffffffffffffffe file 00000000c51edc48 flags 65536 (32768) Nov 24 12:51:10 ceph-0005 kernel: [ 89.680523] ceph: mds_client.c:2832 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] on 000000005248a4a0 41 built 10000000000 '/' Nov 24 12:51:10 ceph-0005 kernel: [ 89.681343] ceph: mds_client.c:5779 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] tpath '/', mask 4, caller_uid 0, caller_gid 0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.682296] ceph: mds_client.c:5664 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] fsname check fs_name=(null) match.fs_name=cephfs Nov 24 12:51:10 ceph-0005 kernel: [ 89.683134] BUG: kernel NULL pointer dereference, address: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.683603] #PF: supervisor read access in kernel mode Nov 24 12:51:10 ceph-0005 kernel: [ 89.683969] #PF: error_code(0x0000) - not- present page Nov 24 12:51:10 ceph-0005 kernel: [ 89.684311] PGD 0 P4D 0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.684496] Oops: Oops: 0000 [#2] SMP KASAN NOPTI Nov 24 12:51:10 ceph-0005 kernel: [ 89.684843] CPU: 1 UID: 0 PID: 3406 Comm: xfs_io Tainted: G D 6.18.0-rc6+ #64 PREEMPT(voluntary) Nov 24 12:51:10 ceph-0005 kernel: [ 89.685535] Tainted: [D]=DIE Nov 24 12:51:10 ceph-0005 kernel: [ 89.685738] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 Nov 24 12:51:10 ceph-0005 kernel: [ 89.686351] RIP: 0010:strcmp+0x1c/0x40 Nov 24 12:51:10 ceph-0005 kernel: [ 89.686578] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc cc cc 31 Nov 24 12:51:10 ceph-0005 kernel: [ 89.687550] RSP: 0018:ffff8881035c76e0 EFLAGS: 00010246 Nov 24 12:51:10 ceph-0005 kernel: [ 89.687870] RAX: 0000000000000000 RBX: ffff88810bc59600 RCX: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.688252] RDX: 0000000000000063 RSI: 0000000000000000 RDI: ffff888110739020 Nov 24 12:51:10 ceph-0005 kernel: [ 89.688654] RBP: ffff8881035c77c8 R08: 0000000000000000 R09: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.689126] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.689516] R13: ffff88811e104000 R14: 0000000000000000 R15: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.689918] FS: 000070f659aac840(0000) GS:ffff88825f422000(0000) knlGS:0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.690338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 24 12:51:10 ceph-0005 kernel: [ 89.690659] CR2: 0000000000000000 CR3: 00000001a0ebe003 CR4: 0000000000772ef0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.691092] PKRU: 55555554 Nov 24 12:51:10 ceph-0005 kernel: [ 89.691273] Call Trace: Nov 24 12:51:10 ceph-0005 kernel: [ 89.691433] <TASK> Nov 24 12:51:10 ceph-0005 kernel: [ 89.691575] ? ceph_mds_check_access+0x348/0x1760 Nov 24 12:51:10 ceph-0005 kernel: [ 89.691851] ? __kasan_check_write+0x14/0x30 Nov 24 12:51:10 ceph-0005 kernel: [ 89.692089] ? lockref_get+0xb1/0x170 Nov 24 12:51:10 ceph-0005 kernel: [ 89.692320] ceph_open+0x322/0xef0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.692557] ? __pfx_ceph_open+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.692801] ? __pfx_apparmor_file_open+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.693078] ? __ceph_caps_issued_mask_metric+0xd6/0x180 Nov 24 12:51:10 ceph-0005 kernel: [ 89.693405] do_dentry_open+0x7bf/0x10e0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.693649] ? __pfx_ceph_open+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.693870] vfs_open+0x6d/0x450 Nov 24 12:51:10 ceph-0005 kernel: [ 89.694097] ? may_open+0xec/0x370 Nov 24 12:51:10 ceph-0005 kernel: [ 89.694283] path_openat+0x2017/0x50a0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.694520] ? __pfx_path_openat+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.694918] ? __pfx_stack_trace_save+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.695284] ? stack_depot_save_flags+0x28/0x8f0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.695572] ? stack_depot_save+0xe/0x20 Nov 24 12:51:10 ceph-0005 kernel: [ 89.695864] do_filp_open+0x1b4/0x450 Nov 24 12:51:10 ceph-0005 kernel: [ 89.696271] ? __pfx__raw_spin_lock_irqsave+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.696712] ? __pfx_do_filp_open+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.697089] ? __link_object+0x13d/0x2b0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.697426] ? __pfx__raw_spin_lock+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.697801] ? __check_object_size+0x453/0x600 Nov 24 12:51:10 ceph-0005 kernel: [ 89.698216] ? _raw_spin_unlock+0xe/0x40 Nov 24 12:51:10 ceph-0005 kernel: [ 89.698556] do_sys_openat2+0xe6/0x180 Nov 24 12:51:10 ceph-0005 kernel: [ 89.698905] ? __pfx_do_sys_openat2+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.699280] __x64_sys_openat+0x108/0x240 Nov 24 12:51:10 ceph-0005 kernel: [ 89.699625] ? __pfx___x64_sys_openat+0x10/0x10 Nov 24 12:51:10 ceph-0005 kernel: [ 89.700038] x64_sys_call+0x134f/0x2350 Nov 24 12:51:10 ceph-0005 kernel: [ 89.700371] do_syscall_64+0x82/0xd50 Nov 24 12:51:10 ceph-0005 kernel: [ 89.700684] ? do_syscall_64+0xba/0xd50 Nov 24 12:51:10 ceph-0005 kernel: [ 89.701030] ? irqentry_exit_to_user_mode+0x2e/0x2a0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.701409] ? irqentry_exit+0x43/0x50 Nov 24 12:51:10 ceph-0005 kernel: [ 89.701595] ? exc_page_fault+0x95/0x100 Nov 24 12:51:10 ceph-0005 kernel: [ 89.701793] entry_SYSCALL_64_after_hwframe+0x76/0x7e Nov 24 12:51:10 ceph-0005 kernel: [ 89.702066] RIP: 0033:0x70f6599145ab Nov 24 12:51:10 ceph-0005 kernel: [ 89.702266] Code: 25 00 00 41 00 3d 00 00 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 2b 14 25 Nov 24 12:51:10 ceph-0005 kernel: [ 89.703191] RSP: 002b:00007ffc93ebcfc0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 Nov 24 12:51:10 ceph-0005 kernel: [ 89.703557] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 000070f6599145ab Nov 24 12:51:10 ceph-0005 kernel: [ 89.703940] RDX: 0000000000000000 RSI: 00007ffc93ebf786 RDI: 00000000ffffff9c Nov 24 12:51:10 ceph-0005 kernel: [ 89.704288] RBP: 00007ffc93ebf786 R08: 00007ffc93ebd270 R09: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.704624] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.704986] R13: 00000000ffffffff R14: 0000000000000180 R15: 0000000000000001 Nov 24 12:51:10 ceph-0005 kernel: [ 89.705329] </TASK> Nov 24 12:51:10 ceph-0005 kernel: [ 89.705453] Modules linked in: intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel input_leds psmouse rapl vga16fb serio_raw vgastate floppy i2c_piix4 mac_hid qemu_fw_cfg i2c_smbus bochs pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp parport efi_pstore Nov 24 12:51:10 ceph-0005 kernel: [ 89.707505] CR2: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.707692] ---[ end trace 0000000000000000 ]--- Nov 24 12:51:10 ceph-0005 kernel: [ 89.707965] RIP: 0010:strcmp+0x1c/0x40 Nov 24 12:51:10 ceph-0005 kernel: [ 89.708167] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc cc cc 31 Nov 24 12:51:10 ceph-0005 kernel: [ 89.709139] RSP: 0018:ffff8881b8faf600 EFLAGS: 00010246 Nov 24 12:51:10 ceph-0005 kernel: [ 89.709394] RAX: 0000000000000000 RBX: ffff88810bc59600 RCX: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.709760] RDX: 0000000000000063 RSI: 0000000000000000 RDI: ffff888110739020 Nov 24 12:51:10 ceph-0005 kernel: [ 89.710125] RBP: ffff8881b8faf6e8 R08: 0000000000000000 R09: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.710513] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.710874] R13: ffff88811e104000 R14: 0000000000000000 R15: 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.711244] FS: 000070f659aac840(0000) GS:ffff88825f422000(0000) knlGS:0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.711687] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 24 12:51:10 ceph-0005 kernel: [ 89.712009] CR2: 0000000000000000 CR3: 00000001a0ebe003 CR4: 0000000000772ef0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.712376] PKRU: 55555554 Nov 24 12:51:10 ceph-0005 kernel: [ 89.807353] ceph: super.c:414 : ceph_parse_mount_param: fs_parse 'source' token 12 Nov 24 12:51:10 ceph-0005 kernel: [ 89.807983] ceph: super.c:342 : '192.168.1.213:3300:/scratch' Nov 24 12:51:10 ceph-0005 kernel: [ 89.808750] ceph: super.c:366 : device name '192.168.1.213:3300' Nov 24 12:51:10 ceph-0005 kernel: [ 89.809361] ceph: super.c:368 : server path '/scratch' Nov 24 12:51:10 ceph-0005 kernel: [ 89.809813] ceph: super.c:370 : trying new device syntax Nov 24 12:51:10 ceph-0005 kernel: [ 89.809815] ceph: super.c:280 : separator '=' missing in source Nov 24 12:51:10 ceph-0005 kernel: [ 89.810219] ceph: super.c:375 : trying old device syntax Nov 24 12:51:10 ceph-0005 kernel: [ 89.810763] ceph: super.c:1299 : ceph_get_tree Nov 24 12:51:10 ceph-0005 kernel: [ 89.812515] ceph: super.c:1236 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] 0000000003c37076 Nov 24 12:51:10 ceph-0005 kernel: [ 89.813215] ceph: super.c:1239 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] monitor(s)/mount options don't match Nov 24 12:51:10 ceph-0005 kernel: [ 89.814018] ceph: super.c:1236 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] 000000005e011a18 Nov 24 12:51:10 ceph-0005 kernel: [ 89.814618] ceph: super.c:1239 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] monitor(s)/mount options don't match Nov 24 12:51:10 ceph-0005 kernel: [ 89.815493] ceph: super.c:1236 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] 0000000003c37076 Nov 24 12:51:10 ceph-0005 kernel: [ 89.816133] ceph: super.c:1239 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] monitor(s)/mount options don't match Nov 24 12:51:10 ceph-0005 kernel: [ 89.816830] ceph: super.c:1236 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] 000000005e011a18 Nov 24 12:51:10 ceph-0005 kernel: [ 89.817528] ceph: super.c:1239 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] monitor(s)/mount options don't match Nov 24 12:51:10 ceph-0005 kernel: [ 89.818320] ceph: super.c:1199 : [00000000-0000-0000-0000-000000000000 0] 00000000ce5cc894 Nov 24 12:51:10 ceph-0005 kernel: [ 89.818919] ceph: super.c:1335 : get_sb using new client 000000006de70127 Nov 24 12:51:10 ceph-0005 kernel: [ 89.819741] ceph: super.c:1145 : [00000000-0000-0000-0000-000000000000 0] mount start 000000006de70127 Nov 24 12:51:10 ceph-0005 kernel: [ 89.826429] libceph: mon0 (2)192.168.1.213:3300 session established Nov 24 12:51:10 ceph-0005 kernel: [ 89.829158] ceph: mds_client.c:6196 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] epoch 4346 len 1220 Nov 24 12:51:10 ceph-0005 kernel: [ 89.829955] ceph: mds_client.c:5060 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] new 4346 old 0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.831138] libceph: client175983 fsid 31977b06-8cdb-42a9-97ad-d6a7d59a42dd Nov 24 12:51:10 ceph-0005 kernel: [ 89.831780] ceph: super.c:1168 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mount opening path 'scratch' Nov 24 12:51:10 ceph-0005 kernel: [ 89.832570] ceph: super.c:1055 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] opening 'scratch' Nov 24 12:51:10 ceph-0005 kernel: [ 89.833170] ceph: mds_client.c:3796 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] do_request on 000000004bf8d029 Nov 24 12:51:10 ceph-0005 kernel: [ 89.833823] ceph: mds_client.c:3724 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] submit_request on 000000004bf8d029 for inode 0000000000000000 Nov 24 12:51:10 ceph-0005 kernel: [ 89.834644] ceph: mds_client.c:1183 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] 000000004bf8d029 tid 1 Nov 24 12:51:10 ceph-0005 kernel: [ 89.835278] ceph: mds_client.c:1434 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] chose random mds0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.835886] ceph: mds_client.c:984 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] realloc to 1 Nov 24 12:51:10 ceph-0005 kernel: [ 89.836390] ceph: mds_client.c:997 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 Nov 24 12:51:10 ceph-0005 kernel: [ 89.836969] ceph: mds_client.c:3509 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 session 00000000dd87f0dd state new Nov 24 12:51:10 ceph-0005 kernel: [ 89.837584] ceph: mds_client.c:1674 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] open_session to mds0 (up:active) Nov 24 12:51:10 ceph-0005 kernel: [ 89.838388] ceph: mds_client.c:3741 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] do_request waiting Nov 24 12:51:10 ceph-0005 kernel: [ 90.291635] ceph: mds_client.c:4210 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] cap_auths_num 1 Nov 24 12:51:10 ceph-0005 kernel: [ 90.292760] ceph: mds_client.c:4281 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] uid -1, num_gids 0, path (null), fs_name cephfs, root_squash 0, readable 1, writeable 1 Nov 24 12:51:10 ceph-0005 kernel: [ 90.294531] ceph: mds_client.c:4313 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 open 00000000dd87f0dd state opening seq 0 Nov 24 12:51:10 ceph-0005 kernel: [ 90.296268] ceph: mds_client.c:2090 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 ttl now 4294815743, was fresh, now stale Nov 24 12:51:10 ceph-0005 kernel: [ 90.297370] ceph: mds_client.c:3653 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] wake request 000000004bf8d029 tid 1 Nov 24 12:51:10 ceph-0005 kernel: [ 90.298127] ceph: mds_client.c:1306 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] using resend_mds mds0 Nov 24 12:51:10 ceph-0005 kernel: [ 90.298734] ceph: mds_client.c:3509 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 session 00000000dd87f0dd state open Nov 24 12:51:11 ceph-0005 kernel: [ 90.299523] ceph: mds_client.c:3340 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] 000000004bf8d029 tid 1 getattr (attempt 1) Nov 24 12:51:11 ceph-0005 kernel: [ 90.300184] ceph: mds_client.c:2923 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] path scratch Nov 24 12:51:11 ceph-0005 kernel: [ 90.300769] ceph: mds_client.c:3409 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] r_parent = 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.302465] ceph: mds_client.c:3863 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] handle_reply 000000004bf8d029 Nov 24 12:51:11 ceph-0005 kernel: [ 90.303109] ceph: mds_client.c:1208 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] 000000004bf8d029 tid 1 Nov 24 12:51:11 ceph-0005 kernel: [ 90.303661] ceph: mds_client.c:3917 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] tid 1 result 0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.304326] ceph: mds_client.c:3755 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] do_request waited, got 0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.305635] ceph: mds_client.c:3802 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] do_request 000000004bf8d029 done, result 0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.307048] ceph: super.c:1075 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] success Nov 24 12:51:11 ceph-0005 kernel: [ 90.308214] ceph: super.c:1081 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] success, root dentry is 00000000c9e255e3 Nov 24 12:51:11 ceph-0005 kernel: [ 90.309500] ceph: super.c:1183 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mount success Nov 24 12:51:11 ceph-0005 kernel: [ 90.310523] ceph: super.c:1347 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] root 00000000c9e255e3 inode 000000000946916c ino 10000000001.fffffffffffffffe Nov 24 12:51:11 ceph-0005 kernel: [ 90.312737] ceph: super.c:620 : destroy_mount_options 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.319198] ceph: file.c:389 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] 000000000946916c 10000000001.fffffffffffffffe file 0000000006cb5fc2 flags 65536 (100352) Nov 24 12:51:11 ceph-0005 kernel: [ 90.322405] ceph: mds_client.c:2832 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] on 00000000c9e255e3 38 built 10000000001 '/' Nov 24 12:51:11 ceph-0005 kernel: [ 90.324881] ceph: mds_client.c:5779 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] tpath '/', mask 4, caller_uid 0, caller_gid 0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.327514] ceph: mds_client.c:5664 : [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] fsname check fs_name=(null) match.fs_name=cephfs Nov 24 12:51:11 ceph-0005 kernel: [ 90.328688] BUG: kernel NULL pointer dereference, address: 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.329218] #PF: supervisor read access in kernel mode Nov 24 12:51:11 ceph-0005 kernel: [ 90.329547] #PF: error_code(0x0000) - not- present page Nov 24 12:51:11 ceph-0005 kernel: [ 90.330308] PGD 0 P4D 0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.330505] Oops: Oops: 0000 [#3] SMP KASAN NOPTI Nov 24 12:51:11 ceph-0005 kernel: [ 90.330776] CPU: 7 UID: 0 PID: 2530 Comm: check Tainted: G D 6.18.0-rc6+ #64 PREEMPT(voluntary) Nov 24 12:51:11 ceph-0005 kernel: [ 90.331723] Tainted: [D]=DIE Nov 24 12:51:11 ceph-0005 kernel: [ 90.331956] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 Nov 24 12:51:11 ceph-0005 kernel: [ 90.333363] RIP: 0010:strcmp+0x1c/0x40 Nov 24 12:51:11 ceph-0005 kernel: [ 90.333614] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc cc cc 31 Nov 24 12:51:11 ceph-0005 kernel: [ 90.334929] RSP: 0018:ffff8881b5f8f6d0 EFLAGS: 00010246 Nov 24 12:51:11 ceph-0005 kernel: [ 90.335267] RAX: 0000000000000000 RBX: ffff88810bc59200 RCX: 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.335694] RDX: 0000000000000063 RSI: 0000000000000000 RDI: ffff8881107393c0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.336176] RBP: ffff8881b5f8f7b8 R08: 0000000000000000 R09: 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.336688] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.337135] R13: ffff88811e218000 R14: 0000000000000000 R15: 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.337691] FS: 00007f6a1017c740(0000) GS:ffff88825f722000(0000) knlGS:0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.338600] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 24 12:51:11 ceph-0005 kernel: [ 90.339405] CR2: 0000000000000000 CR3: 000000010216e006 CR4: 0000000000772ef0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.339920] PKRU: 55555554 Nov 24 12:51:11 ceph-0005 kernel: [ 90.340105] Call Trace: Nov 24 12:51:11 ceph-0005 kernel: [ 90.340276] <TASK> Nov 24 12:51:11 ceph-0005 kernel: [ 90.340454] ? ceph_mds_check_access+0x348/0x1760 Nov 24 12:51:11 ceph-0005 kernel: [ 90.340775] ? __kasan_check_write+0x14/0x30 Nov 24 12:51:11 ceph-0005 kernel: [ 90.341447] ? lockref_get+0xb1/0x170 Nov 24 12:51:11 ceph-0005 kernel: [ 90.341729] ceph_open+0x322/0xef0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.341962] ? __kasan_check_write+0x14/0x30 Nov 24 12:51:11 ceph-0005 kernel: [ 90.342233] ? __pfx_ceph_open+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.342499] ? __pfx_apparmor_file_open+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.342763] do_dentry_open+0x7bf/0x10e0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.343646] ? __pfx_ceph_open+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.344196] vfs_open+0x6d/0x450 Nov 24 12:51:11 ceph-0005 kernel: [ 90.344409] ? may_open+0xec/0x370 Nov 24 12:51:11 ceph-0005 kernel: [ 90.344640] path_openat+0x2017/0x50a0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.344895] ? __pfx_path_openat+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.345198] ? __pfx_stack_trace_save+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.345566] ? __kasan_check_write+0x14/0x30 Nov 24 12:51:11 ceph-0005 kernel: [ 90.345911] ? stack_depot_save_flags+0x28/0x8f0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.346289] ? stack_depot_save+0xe/0x20 Nov 24 12:51:11 ceph-0005 kernel: [ 90.346597] do_filp_open+0x1b4/0x450 Nov 24 12:51:11 ceph-0005 kernel: [ 90.346899] ? __pfx__raw_spin_lock_irqsave+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.347276] ? __pfx_do_filp_open+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.347582] ? __link_object+0x13d/0x2b0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.347880] ? __pfx__raw_spin_lock+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.348140] ? __check_object_size+0x453/0x600 Nov 24 12:51:11 ceph-0005 kernel: [ 90.348385] ? _raw_spin_unlock+0xe/0x40 Nov 24 12:51:11 ceph-0005 kernel: [ 90.348597] do_sys_openat2+0xe6/0x180 Nov 24 12:51:11 ceph-0005 kernel: [ 90.348804] ? __pfx_do_sys_openat2+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.349047] ? __kasan_check_write+0x14/0x30 Nov 24 12:51:11 ceph-0005 kernel: [ 90.349302] ? lock_vma_under_rcu+0x2e9/0x730 Nov 24 12:51:11 ceph-0005 kernel: [ 90.349556] __x64_sys_openat+0x108/0x240 Nov 24 12:51:11 ceph-0005 kernel: [ 90.349772] ? __pfx___x64_sys_openat+0x10/0x10 Nov 24 12:51:11 ceph-0005 kernel: [ 90.350023] x64_sys_call+0x134f/0x2350 Nov 24 12:51:11 ceph-0005 kernel: [ 90.350239] do_syscall_64+0x82/0xd50 Nov 24 12:51:11 ceph-0005 kernel: [ 90.350440] ? __kasan_check_read+0x11/0x20 Nov 24 12:51:11 ceph-0005 kernel: [ 90.350660] ? fpregs_assert_state_consistent+0x5c/0x100 Nov 24 12:51:11 ceph-0005 kernel: [ 90.350968] ? irqentry_exit_to_user_mode+0x2e/0x2a0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.351240] ? irqentry_exit+0x43/0x50 Nov 24 12:51:11 ceph-0005 kernel: [ 90.351442] ? exc_page_fault+0x95/0x100 Nov 24 12:51:11 ceph-0005 kernel: [ 90.351647] entry_SYSCALL_64_after_hwframe+0x76/0x7e Nov 24 12:51:11 ceph-0005 kernel: [ 90.351933] RIP: 0033:0x7f6a0ff19a8c Nov 24 12:51:11 ceph-0005 kernel: [ 90.352129] Code: 24 18 31 c0 41 83 e2 40 75 44 89 f0 25 00 00 41 00 3d 00 00 41 00 74 36 44 89 c2 4c 89 ce bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 44 48 8b 54 24 18 64 48 2b 14 25 28 00 00 00 Nov 24 12:51:11 ceph-0005 kernel: [ 90.353095] RSP: 002b:00007ffe25b42190 EFLAGS: 00000287 ORIG_RAX: 0000000000000101 Nov 24 12:51:11 ceph-0005 kernel: [ 90.353939] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f6a0ff19a8c Nov 24 12:51:11 ceph-0005 kernel: [ 90.354382] RDX: 0000000000090800 RSI: 00005c7d3b043a90 RDI: 00000000ffffff9c Nov 24 12:51:11 ceph-0005 kernel: [ 90.354803] RBP: 00007ffe25b423f0 R08: 0000000000090800 R09: 00005c7d3b043a90 Nov 24 12:51:11 ceph-0005 kernel: [ 90.355276] R10: 0000000000000000 R11: 0000000000000287 R12: 00005c7d3ad24354 Nov 24 12:51:11 ceph-0005 kernel: [ 90.355686] R13: 00005c7d3b043a90 R14: 0000000000000000 R15: 00005c7d3ad24353 Nov 24 12:51:11 ceph-0005 kernel: [ 90.356091] </TASK> Nov 24 12:51:11 ceph-0005 kernel: [ 90.356231] Modules linked in: intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel input_leds psmouse rapl vga16fb serio_raw vgastate floppy i2c_piix4 mac_hid qemu_fw_cfg i2c_smbus bochs pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp parport efi_pstore Nov 24 12:51:11 ceph-0005 kernel: [ 90.358735] CR2: 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.358972] ---[ end trace 0000000000000000 ]--- Nov 24 12:51:11 ceph-0005 kernel: [ 90.359341] RIP: 0010:strcmp+0x1c/0x40 Nov 24 12:51:11 ceph-0005 kernel: [ 90.359581] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc cc cc 31 Nov 24 12:51:11 ceph-0005 kernel: [ 90.360687] RSP: 0018:ffff8881b8faf600 EFLAGS: 00010246 Nov 24 12:51:11 ceph-0005 kernel: [ 90.361008] RAX: 0000000000000000 RBX: ffff88810bc59600 RCX: 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.361488] RDX: 0000000000000063 RSI: 0000000000000000 RDI: ffff888110739020 Nov 24 12:51:11 ceph-0005 kernel: [ 90.362074] RBP: ffff8881b8faf6e8 R08: 0000000000000000 R09: 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.362498] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.362969] R13: ffff88811e104000 R14: 0000000000000000 R15: 0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.363442] FS: 00007f6a1017c740(0000) GS:ffff88825f722000(0000) knlGS:0000000000000000 Nov 24 12:51:11 ceph-0005 kernel: [ 90.363971] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 24 12:51:11 ceph-0005 kernel: [ 90.364338] CR2: 0000000000000000 CR3: 000000010216e006 CR4: 0000000000772ef0 Nov 24 12:51:11 ceph-0005 kernel: [ 90.364738] PKRU: 55555554 So, the main sequence is: ceph_open() -> ceph_mdsc_build_path() -> ceph_mds_check_access() -> ceph_mds_auth_match() -> crash happens > > > > And I am not sure that we can mandate anyone to use the old syntax. If there is > > some other opportunity, then someone could use it. But, maybe, I am missing the > > point. :) What do you mean by "Mandate passing the fsname with old syntax"? > > In the old mount syntax, the fsname is passed as on optional argument > using 'mds_namespace'. > I was suggesting to mandate it if possible. But I guess it breaks > backward compatibility. > > > > > We had a private discussion with Ilya. Yes, he also mentioned the breaking of backward compatibility for the case of mandating passing the fsname with old syntax. He believes that: "Use the default fsname when mdsc->fsc->mount_options- >mds_namespace is NULL during comparison seems like a sensible approach to me". Thanks, Slava. > > > > > > > > > > > > On Thu, Nov 20, 2025 at 4:47 AM Viacheslav Dubeyko > > > <Slava.Dubeyko@ibm.com> wrote: > > > > > > > > On Wed, 2025-11-19 at 15:02 -0800, Gregory Farnum wrote: > > > > > > > > > > That doesn’t sound right — this is authentication code. If the authorization is supplied for a namespace and we are mounting without a namespace at all, isn’t that a jailbreak? So the NULL pointer should be accepted in one direction, but denied in the other? > > > > > > > > What is your particular suggestion? I am simply fixing the kernel crash after > > > > the 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"). We didn't have any > > > > check before. Do you imply that 22c73d52a6d0 ("ceph: fix multifs mds auth caps > > > > issue") fix is incorrect and we need to rework it somehow? > > > > > > > > If we will not have any fix, then 6.18 release will have broken CephFS kernel > > > > client. > > > > > > > > Thanks, > > > > Slava. > > > > > > > > > > > > > > On Wed, Nov 19, 2025 at 2:54 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > > > > > On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote: > > > > > > > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote: > > > > > > > > > > > > > > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > > > > > > > > > > The CephFS kernel client has regression starting from 6.18-rc1. > > > > > > > > > > > > > > > > sudo ./check -g quick > > > > > > > > FSTYP -- ceph > > > > > > > > PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri > > > > > > > > Nov 14 11:26:14 PST 2025 > > > > > > > > MKFS_OPTIONS -- 192.168.1.213:3300:/scratch > > > > > > > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch > > > > > > > > /mnt/cephfs/scratch > > > > > > > > > > > > > > > > Killed > > > > > > > > > > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0 > > > > > > > > (2)192.168.1.213:3300 session established > > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer > > > > > > > > dereference, address: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in > > > > > > > > kernel mode > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not- > > > > > > > > present page > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN > > > > > > > > NOPTI > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm: > > > > > > > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary) > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC > > > > > > > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90 > > > > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > > > > cc cc 31 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0 > > > > > > > > EFLAGS: 00010246 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX: > > > > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI: > > > > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08: > > > > > > > > 0000000000000000 R09: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11: > > > > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14: > > > > > > > > 0000000000000000 R15: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000) > > > > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000 > > > > > > > > CR0: 0000000080050033 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3: > > > > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace: > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK> > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ? > > > > > > > > ceph_mds_check_access+0x348/0x1760 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ? > > > > > > > > __kasan_check_write+0x14/0x30 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ? > > > > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ? > > > > > > > > __pfx_apparmor_file_open+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ? > > > > > > > > __ceph_caps_issued_mask_metric+0xd6/0x180 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ? > > > > > > > > __pfx_stack_trace_save+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ? > > > > > > > > stack_depot_save_flags+0x28/0x8f0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ? > > > > > > > > __pfx__raw_spin_lock_irqsave+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ? > > > > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ? > > > > > > > > __check_object_size+0x453/0x600 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ? > > > > > > > > __pfx_do_sys_openat2+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ? > > > > > > > > __pfx___x64_sys_openat+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ? > > > > > > > > __pfx___handle_mm_fault+0x10/0x10 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ? > > > > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ? > > > > > > > > count_memcg_events+0x25b/0x400 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ? > > > > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ? > > > > > > > > irqentry_exit_to_user_mode+0x2e/0x2a0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389] > > > > > > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00 > > > > > > > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > > > > > > > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > > > > > > > > 2b 14 25 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0 > > > > > > > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX: > > > > > > > > 0000000000000002 RCX: 000074a85bf145ab > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI: > > > > > > > > 00007ffc77d32789 RDI: 00000000ffffff9c > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08: > > > > > > > > 00007ffc77d31980 R09: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11: > > > > > > > > 0000000000000246 R12: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14: > > > > > > > > 0000000000000180 R15: 0000000000000001 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK> > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in: > > > > > > > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > > > > > > > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > > > > > > > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > > > > > > > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate > > > > > > > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > > > > > > > > parport efi_pstore > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000 > > > > > > > > ]--- > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90 > > > > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > > > > cc cc 31 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0 > > > > > > > > EFLAGS: 00010246 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX: > > > > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI: > > > > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08: > > > > > > > > 0000000000000000 R09: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11: > > > > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14: > > > > > > > > 0000000000000000 R15: 0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000) > > > > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000 > > > > > > > > CR0: 0000000080050033 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3: > > > > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554 > > > > > > > > > > > > > > > > We have issue here [1] if fs_name == NULL: > > > > > > > > > > > > > > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace; > > > > > > > > ... > > > > > > > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > > > > / fsname mismatch, try next one */ > > > > > > > > return 0; > > > > > > > > } > > > > > > > > > > > > > > > > The patch fixes the issue by introducing is_fsname_mismatch() method > > > > > > > > that checks auth->match.fs_name and fs_name pointers validity, and > > > > > > > > compares the file system names. > > > > > > > > > > > > > > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666 > > > > > > > > > > > > > > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") > > > > > > > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com> > > > > > > > > cc: Alex Markuze <amarkuze@redhat.com> > > > > > > > > cc: Ilya Dryomov <idryomov@gmail.com> > > > > > > > > cc: Ceph Development <ceph-devel@vger.kernel.org> > > > > > > > > --- > > > > > > > > fs/ceph/mds_client.c | 20 +++++++++++++++++--- > > > > > > > > 1 file changed, 17 insertions(+), 3 deletions(-) > > > > > > > > > > > > > > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c > > > > > > > > index 1740047aef0f..19c75e206300 100644 > > > > > > > > --- a/fs/ceph/mds_client.c > > > > > > > > +++ b/fs/ceph/mds_client.c > > > > > > > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s) > > > > > > > > mutex_unlock(&s->s_mutex); > > > > > > > > } > > > > > > > > > > > > > > > > +static inline > > > > > > > > +bool is_fsname_mismatch(struct ceph_client *cl, > > > > > > > > + const char *fs_name1, const char *fs_name2) > > > > > > > > +{ > > > > > > > > + if (!fs_name1 || !fs_name2) > > > > > > > > + return false; > > > > > > > > > > > > > > Hi Slava, > > > > > > > > > > > > > > It looks like this would declare a match (return false for "mismatch") > > > > > > > in case ceph_mds_cap_auth is defined to require a particular fs_name but > > > > > > > no mds_namespace was passed on mount. Is that the desired behavior? > > > > > > > > > > > > > > > > > > > Hi Ilya, > > > > > > > > > > > > Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such > > > > > > check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or > > > > > > fs_name is NULL, then we cannot say that they match or not. It means that we > > > > > > need to continue logic, this is why is_fsname_mismatch() returns false. > > > > > > Otherwise, if we stop logic by returning true, then we have bunch of xfstests > > > > > > failures. > > > > > > > > > > > > Thanks, > > > > > > Slava. > > > > > > > > > > > > > > + > > > > > > > > + doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n", > > > > > > > > + fs_name1, fs_name2); > > > > > > > > + > > > > > > > > + if (strcmp(fs_name1, fs_name2)) > > > > > > > > + return true; > > > > > > > > + > > > > > > > > + return false; > > > > > > > > +} > > > > > > > > + > > > > > > > > static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > > > > struct ceph_mds_cap_auth *auth, > > > > > > > > const struct cred *cred, > > > > > > > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > > > > u32 gid, tlen, len; > > > > > > > > int i, j; > > > > > > > > > > > > > > > > - doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n", > > > > > > > > - fs_name, auth->match.fs_name ? auth->match.fs_name : ""); > > > > > > > > - if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > > > > + if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) { > > > > > > > > /* fsname mismatch, try next one */ > > > > > > > > return 0; > > > > > > > > } > > > > > > > > -- > > > > > > > > 2.51.1 > > > > > > > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] ceph: fix kernel crash in ceph_open() 2025-11-24 21:12 ` Viacheslav Dubeyko @ 2025-11-27 7:33 ` Kotresh Hiremath Ravishankar 2025-12-01 20:04 ` Viacheslav Dubeyko 0 siblings, 1 reply; 13+ messages in thread From: Kotresh Hiremath Ravishankar @ 2025-11-27 7:33 UTC (permalink / raw) To: Viacheslav Dubeyko Cc: Viacheslav Dubeyko, linux-fsdevel@vger.kernel.org, ceph-devel@vger.kernel.org, slava@dubeyko.com, Patrick Donnelly, Gregory Farnum, Alex Markuze, idryomov@gmail.com, Pavan Rallabhandi On Tue, Nov 25, 2025 at 2:42 AM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > On Tue, 2025-11-25 at 00:48 +0530, Kotresh Hiremath Ravishankar wrote: > > On Fri, Nov 21, 2025 at 1:47 AM Viacheslav Dubeyko > > <Slava.Dubeyko@ibm.com> wrote: > > > > > > On Thu, 2025-11-20 at 19:50 +0530, Kotresh Hiremath Ravishankar wrote: > > > > Hi All, > > > > > > > > I think the patch is necessary and fixes the crash. There is no harm > > > > in taking this patch as it behaves like an old kernel with this > > > > particular scenario. > > > > > > > > When does the issue happen: > > > > - The issue happens only when the old mount syntax is used where > > > > passing the file system name is optional in which case, it chooses the > > > > default mds namespace but doesn't get filled in the > > > > mdsc->fsc->mount_options->mds_namespace. > > > > - Along with the above, the mount user should be non admin. > > > > Does it break the earlier fix ? > > > > - Not fully!!! Though the open does succeed, the subsequent > > > > operation like write would get EPERM. I am not exactly able to > > > > recollect but this was discussed before writing the fix 22c73d52a6d0 > > > > ("ceph: fix multifs mds auth caps issue"), it's guarded by another > > > > check before actual operation like write. > > > > > > > > I think there are a couple of options to fix this cleanly. > > > > 1. Use the default fsname when > > > > mdsc->fsc->mount_options->mds_namespace is NULL during comparison. > > > > 2. Mandate passing the fsname with old syntax ? > > > > > > > > > > Anyway, we should be ready operate correctly if fsname or/and auth- > > > > match.fs_name are NULL. And if we need to make the fix more cleanly, then we > > > can introduce another patch with nicer fix. > > > > > > I am not completely sure how default fsname can be applicable here. If I > > > understood the CephFS mount logic correctly, then fsname can be NULL during some > > > initial steps. But, finally, we will have the real fsname for comparison. But I > > > don't know if it's right of assuming that fsname == NULL is equal to fsname == > > > default_name. > > > > We are pretty sure fsname is NULL only if the old mount syntax is used > > without providing the > > fsname in the optional arg. I believe kclient knows the fsname that's > > mounted somewhere in this case ? > > I am not sure though. If so, it can be used. If not, then can we rely > > on what mds sends as part > > of the mdsmap? > > > > With this fix, did the tests run fine ? > > > > The xfstests works fine with the fix. I don't see any issues with the it. > > > Aren't you hitting this error > > https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mdsmap.c#L365 > > ? > > > > I am not sure how this error can be triggered. I see this sequence: Ok. It doesn't crash because the null check is there at https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/super.h#L116 > > Nov 24 12:51:10 ceph-0005 kernel: [ 89.621635] ceph: super.c:63 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] begin > Nov 24 12:51:10 ceph-0005 kernel: [ 89.624691] ceph: super.c:117 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] done > Nov 24 12:51:10 ceph-0005 kernel: [ 89.625349] ceph: super.c:63 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] begin > Nov 24 12:51:10 ceph-0005 kernel: [ 89.627776] ceph: super.c:117 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] done > Nov 24 12:51:10 ceph-0005 kernel: [ 89.645611] ceph: super.c:63 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] begin > Nov 24 12:51:10 ceph-0005 kernel: [ 89.652534] ceph: super.c:117 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] done > Nov 24 12:51:10 ceph-0005 kernel: [ 89.654695] ceph: super.c:63 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] begin > Nov 24 12:51:10 ceph-0005 kernel: [ 89.656220] ceph: super.c:117 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] done > > Nov 24 12:51:10 ceph-0005 kernel: [ 89.678877] ceph: file.c:389 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] 000000000e172927 > 10000000000.fffffffffffffffe file 00000000c51edc48 flags 65536 (32768) > Nov 24 12:51:10 ceph-0005 kernel: [ 89.680523] ceph: mds_client.c:2832 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] on 000000005248a4a0 41 built > 10000000000 '/' > Nov 24 12:51:10 ceph-0005 kernel: [ 89.681343] ceph: mds_client.c:5779 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] tpath '/', mask 4, caller_uid 0, > caller_gid 0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.682296] ceph: mds_client.c:5664 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] fsname check fs_name=(null) > match.fs_name=cephfs > Nov 24 12:51:10 ceph-0005 kernel: [ 89.683134] BUG: kernel NULL pointer > dereference, address: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.683603] #PF: supervisor read access in > kernel mode > Nov 24 12:51:10 ceph-0005 kernel: [ 89.683969] #PF: error_code(0x0000) - not- > present page > Nov 24 12:51:10 ceph-0005 kernel: [ 89.684311] PGD 0 P4D 0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.684496] Oops: Oops: 0000 [#2] SMP KASAN > NOPTI > Nov 24 12:51:10 ceph-0005 kernel: [ 89.684843] CPU: 1 UID: 0 PID: 3406 Comm: > xfs_io Tainted: G D 6.18.0-rc6+ #64 PREEMPT(voluntary) > Nov 24 12:51:10 ceph-0005 kernel: [ 89.685535] Tainted: [D]=DIE > Nov 24 12:51:10 ceph-0005 kernel: [ 89.685738] Hardware name: QEMU Standard PC > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.686351] RIP: 0010:strcmp+0x1c/0x40 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.686578] Code: 90 90 90 90 90 90 90 90 > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > cc cc 31 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.687550] RSP: 0018:ffff8881035c76e0 > EFLAGS: 00010246 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.687870] RAX: 0000000000000000 RBX: > ffff88810bc59600 RCX: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.688252] RDX: 0000000000000063 RSI: > 0000000000000000 RDI: ffff888110739020 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.688654] RBP: ffff8881035c77c8 R08: > 0000000000000000 R09: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.689126] R10: 0000000000000000 R11: > 0000000000000000 R12: dffffc0000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.689516] R13: ffff88811e104000 R14: > 0000000000000000 R15: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.689918] FS: 000070f659aac840(0000) > GS:ffff88825f422000(0000) knlGS:0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.690338] CS: 0010 DS: 0000 ES: 0000 > CR0: 0000000080050033 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.690659] CR2: 0000000000000000 CR3: > 00000001a0ebe003 CR4: 0000000000772ef0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.691092] PKRU: 55555554 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.691273] Call Trace: > Nov 24 12:51:10 ceph-0005 kernel: [ 89.691433] <TASK> > Nov 24 12:51:10 ceph-0005 kernel: [ 89.691575] ? > ceph_mds_check_access+0x348/0x1760 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.691851] ? > __kasan_check_write+0x14/0x30 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.692089] ? lockref_get+0xb1/0x170 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.692320] ceph_open+0x322/0xef0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.692557] ? __pfx_ceph_open+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.692801] ? > __pfx_apparmor_file_open+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.693078] ? > __ceph_caps_issued_mask_metric+0xd6/0x180 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.693405] do_dentry_open+0x7bf/0x10e0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.693649] ? __pfx_ceph_open+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.693870] vfs_open+0x6d/0x450 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.694097] ? may_open+0xec/0x370 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.694283] path_openat+0x2017/0x50a0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.694520] ? __pfx_path_openat+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.694918] ? > __pfx_stack_trace_save+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.695284] ? > stack_depot_save_flags+0x28/0x8f0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.695572] ? stack_depot_save+0xe/0x20 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.695864] do_filp_open+0x1b4/0x450 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.696271] ? > __pfx__raw_spin_lock_irqsave+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.696712] ? __pfx_do_filp_open+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.697089] ? __link_object+0x13d/0x2b0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.697426] ? > __pfx__raw_spin_lock+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.697801] ? > __check_object_size+0x453/0x600 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.698216] ? _raw_spin_unlock+0xe/0x40 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.698556] do_sys_openat2+0xe6/0x180 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.698905] ? > __pfx_do_sys_openat2+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.699280] __x64_sys_openat+0x108/0x240 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.699625] ? > __pfx___x64_sys_openat+0x10/0x10 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.700038] x64_sys_call+0x134f/0x2350 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.700371] do_syscall_64+0x82/0xd50 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.700684] ? do_syscall_64+0xba/0xd50 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.701030] ? > irqentry_exit_to_user_mode+0x2e/0x2a0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.701409] ? irqentry_exit+0x43/0x50 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.701595] ? exc_page_fault+0x95/0x100 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.701793] > entry_SYSCALL_64_after_hwframe+0x76/0x7e > Nov 24 12:51:10 ceph-0005 kernel: [ 89.702066] RIP: 0033:0x70f6599145ab > Nov 24 12:51:10 ceph-0005 kernel: [ 89.702266] Code: 25 00 00 41 00 3d 00 00 > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > 2b 14 25 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.703191] RSP: 002b:00007ffc93ebcfc0 > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.703557] RAX: ffffffffffffffda RBX: > 0000000000000002 RCX: 000070f6599145ab > Nov 24 12:51:10 ceph-0005 kernel: [ 89.703940] RDX: 0000000000000000 RSI: > 00007ffc93ebf786 RDI: 00000000ffffff9c > Nov 24 12:51:10 ceph-0005 kernel: [ 89.704288] RBP: 00007ffc93ebf786 R08: > 00007ffc93ebd270 R09: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.704624] R10: 0000000000000000 R11: > 0000000000000246 R12: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.704986] R13: 00000000ffffffff R14: > 0000000000000180 R15: 0000000000000001 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.705329] </TASK> > Nov 24 12:51:10 ceph-0005 kernel: [ 89.705453] Modules linked in: > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > input_leds psmouse rapl vga16fb serio_raw vgastate floppy i2c_piix4 mac_hid > qemu_fw_cfg i2c_smbus bochs pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > parport efi_pstore > Nov 24 12:51:10 ceph-0005 kernel: [ 89.707505] CR2: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.707692] ---[ end trace 0000000000000000 > ]--- > Nov 24 12:51:10 ceph-0005 kernel: [ 89.707965] RIP: 0010:strcmp+0x1c/0x40 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.708167] Code: 90 90 90 90 90 90 90 90 > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > cc cc 31 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.709139] RSP: 0018:ffff8881b8faf600 > EFLAGS: 00010246 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.709394] RAX: 0000000000000000 RBX: > ffff88810bc59600 RCX: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.709760] RDX: 0000000000000063 RSI: > 0000000000000000 RDI: ffff888110739020 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.710125] RBP: ffff8881b8faf6e8 R08: > 0000000000000000 R09: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.710513] R10: 0000000000000000 R11: > 0000000000000000 R12: dffffc0000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.710874] R13: ffff88811e104000 R14: > 0000000000000000 R15: 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.711244] FS: 000070f659aac840(0000) > GS:ffff88825f422000(0000) knlGS:0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.711687] CS: 0010 DS: 0000 ES: 0000 > CR0: 0000000080050033 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.712009] CR2: 0000000000000000 CR3: > 00000001a0ebe003 CR4: 0000000000772ef0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.712376] PKRU: 55555554 > > Nov 24 12:51:10 ceph-0005 kernel: [ 89.807353] ceph: super.c:414 : > ceph_parse_mount_param: fs_parse 'source' token 12 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.807983] ceph: super.c:342 : > '192.168.1.213:3300:/scratch' > Nov 24 12:51:10 ceph-0005 kernel: [ 89.808750] ceph: super.c:366 : > device name '192.168.1.213:3300' > Nov 24 12:51:10 ceph-0005 kernel: [ 89.809361] ceph: super.c:368 : > server path '/scratch' > Nov 24 12:51:10 ceph-0005 kernel: [ 89.809813] ceph: super.c:370 : > trying new device syntax > Nov 24 12:51:10 ceph-0005 kernel: [ 89.809815] ceph: super.c:280 : > separator '=' missing in source > Nov 24 12:51:10 ceph-0005 kernel: [ 89.810219] ceph: super.c:375 : > trying old device syntax > Nov 24 12:51:10 ceph-0005 kernel: [ 89.810763] ceph: super.c:1299 : > ceph_get_tree > Nov 24 12:51:10 ceph-0005 kernel: [ 89.812515] ceph: super.c:1236 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] 0000000003c37076 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.813215] ceph: super.c:1239 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] monitor(s)/mount options don't > match > Nov 24 12:51:10 ceph-0005 kernel: [ 89.814018] ceph: super.c:1236 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] 000000005e011a18 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.814618] ceph: super.c:1239 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] monitor(s)/mount options don't > match > Nov 24 12:51:10 ceph-0005 kernel: [ 89.815493] ceph: super.c:1236 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] 0000000003c37076 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.816133] ceph: super.c:1239 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175980] monitor(s)/mount options don't > match > Nov 24 12:51:10 ceph-0005 kernel: [ 89.816830] ceph: super.c:1236 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] 000000005e011a18 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.817528] ceph: super.c:1239 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175977] monitor(s)/mount options don't > match > Nov 24 12:51:10 ceph-0005 kernel: [ 89.818320] ceph: super.c:1199 : > [00000000-0000-0000-0000-000000000000 0] 00000000ce5cc894 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.818919] ceph: super.c:1335 : > get_sb using new client 000000006de70127 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.819741] ceph: super.c:1145 : > [00000000-0000-0000-0000-000000000000 0] mount start 000000006de70127 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.826429] libceph: mon0 > (2)192.168.1.213:3300 session established > Nov 24 12:51:10 ceph-0005 kernel: [ 89.829158] ceph: mds_client.c:6196 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] epoch 4346 len 1220 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.829955] ceph: mds_client.c:5060 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] new 4346 old 0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.831138] libceph: client175983 fsid > 31977b06-8cdb-42a9-97ad-d6a7d59a42dd > Nov 24 12:51:10 ceph-0005 kernel: [ 89.831780] ceph: super.c:1168 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mount opening path 'scratch' > Nov 24 12:51:10 ceph-0005 kernel: [ 89.832570] ceph: super.c:1055 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] opening 'scratch' > Nov 24 12:51:10 ceph-0005 kernel: [ 89.833170] ceph: mds_client.c:3796 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] do_request on 000000004bf8d029 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.833823] ceph: mds_client.c:3724 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] submit_request on 000000004bf8d029 > for inode 0000000000000000 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.834644] ceph: mds_client.c:1183 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] 000000004bf8d029 tid 1 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.835278] ceph: mds_client.c:1434 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] chose random mds0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.835886] ceph: mds_client.c:984 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] realloc to 1 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.836390] ceph: mds_client.c:997 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 > Nov 24 12:51:10 ceph-0005 kernel: [ 89.836969] ceph: mds_client.c:3509 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 session 00000000dd87f0dd > state new > Nov 24 12:51:10 ceph-0005 kernel: [ 89.837584] ceph: mds_client.c:1674 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] open_session to mds0 (up:active) > Nov 24 12:51:10 ceph-0005 kernel: [ 89.838388] ceph: mds_client.c:3741 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] do_request waiting > > Nov 24 12:51:10 ceph-0005 kernel: [ 90.291635] ceph: mds_client.c:4210 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] cap_auths_num 1 > Nov 24 12:51:10 ceph-0005 kernel: [ 90.292760] ceph: mds_client.c:4281 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] uid -1, num_gids 0, path (null), > fs_name cephfs, root_squash 0, readable 1, writeable 1 > Nov 24 12:51:10 ceph-0005 kernel: [ 90.294531] ceph: mds_client.c:4313 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 open 00000000dd87f0dd state > opening seq 0 > Nov 24 12:51:10 ceph-0005 kernel: [ 90.296268] ceph: mds_client.c:2090 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 ttl now 4294815743, was > fresh, now stale > Nov 24 12:51:10 ceph-0005 kernel: [ 90.297370] ceph: mds_client.c:3653 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] wake request 000000004bf8d029 tid > 1 > Nov 24 12:51:10 ceph-0005 kernel: [ 90.298127] ceph: mds_client.c:1306 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] using resend_mds mds0 > Nov 24 12:51:10 ceph-0005 kernel: [ 90.298734] ceph: mds_client.c:3509 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mds0 session 00000000dd87f0dd > state open > Nov 24 12:51:11 ceph-0005 kernel: [ 90.299523] ceph: mds_client.c:3340 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] 000000004bf8d029 tid 1 getattr > (attempt 1) > Nov 24 12:51:11 ceph-0005 kernel: [ 90.300184] ceph: mds_client.c:2923 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] path scratch > Nov 24 12:51:11 ceph-0005 kernel: [ 90.300769] ceph: mds_client.c:3409 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] r_parent = 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.302465] ceph: mds_client.c:3863 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] handle_reply 000000004bf8d029 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.303109] ceph: mds_client.c:1208 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] 000000004bf8d029 tid 1 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.303661] ceph: mds_client.c:3917 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] tid 1 result 0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.304326] ceph: mds_client.c:3755 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] do_request waited, got 0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.305635] ceph: mds_client.c:3802 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] do_request 000000004bf8d029 done, > result 0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.307048] ceph: super.c:1075 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] success > Nov 24 12:51:11 ceph-0005 kernel: [ 90.308214] ceph: super.c:1081 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] success, root dentry is > 00000000c9e255e3 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.309500] ceph: super.c:1183 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] mount success > Nov 24 12:51:11 ceph-0005 kernel: [ 90.310523] ceph: super.c:1347 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] root 00000000c9e255e3 inode > 000000000946916c ino 10000000001.fffffffffffffffe > Nov 24 12:51:11 ceph-0005 kernel: [ 90.312737] ceph: super.c:620 : > destroy_mount_options 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.319198] ceph: file.c:389 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] 000000000946916c > 10000000001.fffffffffffffffe file 0000000006cb5fc2 flags 65536 (100352) > Nov 24 12:51:11 ceph-0005 kernel: [ 90.322405] ceph: mds_client.c:2832 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] on 00000000c9e255e3 38 built > 10000000001 '/' > Nov 24 12:51:11 ceph-0005 kernel: [ 90.324881] ceph: mds_client.c:5779 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] tpath '/', mask 4, caller_uid 0, > caller_gid 0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.327514] ceph: mds_client.c:5664 : > [31977b06-8cdb-42a9-97ad-d6a7d59a42dd 175983] fsname check fs_name=(null) > match.fs_name=cephfs > Nov 24 12:51:11 ceph-0005 kernel: [ 90.328688] BUG: kernel NULL pointer > dereference, address: 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.329218] #PF: supervisor read access in > kernel mode > Nov 24 12:51:11 ceph-0005 kernel: [ 90.329547] #PF: error_code(0x0000) - not- > present page > Nov 24 12:51:11 ceph-0005 kernel: [ 90.330308] PGD 0 P4D 0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.330505] Oops: Oops: 0000 [#3] SMP KASAN > NOPTI > Nov 24 12:51:11 ceph-0005 kernel: [ 90.330776] CPU: 7 UID: 0 PID: 2530 Comm: > check Tainted: G D 6.18.0-rc6+ #64 PREEMPT(voluntary) > Nov 24 12:51:11 ceph-0005 kernel: [ 90.331723] Tainted: [D]=DIE > Nov 24 12:51:11 ceph-0005 kernel: [ 90.331956] Hardware name: QEMU Standard PC > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.333363] RIP: 0010:strcmp+0x1c/0x40 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.333614] Code: 90 90 90 90 90 90 90 90 > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > cc cc 31 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.334929] RSP: 0018:ffff8881b5f8f6d0 > EFLAGS: 00010246 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.335267] RAX: 0000000000000000 RBX: > ffff88810bc59200 RCX: 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.335694] RDX: 0000000000000063 RSI: > 0000000000000000 RDI: ffff8881107393c0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.336176] RBP: ffff8881b5f8f7b8 R08: > 0000000000000000 R09: 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.336688] R10: 0000000000000000 R11: > 0000000000000000 R12: dffffc0000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.337135] R13: ffff88811e218000 R14: > 0000000000000000 R15: 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.337691] FS: 00007f6a1017c740(0000) > GS:ffff88825f722000(0000) knlGS:0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.338600] CS: 0010 DS: 0000 ES: 0000 > CR0: 0000000080050033 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.339405] CR2: 0000000000000000 CR3: > 000000010216e006 CR4: 0000000000772ef0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.339920] PKRU: 55555554 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.340105] Call Trace: > Nov 24 12:51:11 ceph-0005 kernel: [ 90.340276] <TASK> > Nov 24 12:51:11 ceph-0005 kernel: [ 90.340454] ? > ceph_mds_check_access+0x348/0x1760 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.340775] ? > __kasan_check_write+0x14/0x30 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.341447] ? lockref_get+0xb1/0x170 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.341729] ceph_open+0x322/0xef0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.341962] ? > __kasan_check_write+0x14/0x30 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.342233] ? __pfx_ceph_open+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.342499] ? > __pfx_apparmor_file_open+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.342763] do_dentry_open+0x7bf/0x10e0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.343646] ? __pfx_ceph_open+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.344196] vfs_open+0x6d/0x450 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.344409] ? may_open+0xec/0x370 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.344640] path_openat+0x2017/0x50a0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.344895] ? __pfx_path_openat+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.345198] ? > __pfx_stack_trace_save+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.345566] ? > __kasan_check_write+0x14/0x30 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.345911] ? > stack_depot_save_flags+0x28/0x8f0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.346289] ? stack_depot_save+0xe/0x20 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.346597] do_filp_open+0x1b4/0x450 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.346899] ? > __pfx__raw_spin_lock_irqsave+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.347276] ? __pfx_do_filp_open+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.347582] ? __link_object+0x13d/0x2b0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.347880] ? > __pfx__raw_spin_lock+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.348140] ? > __check_object_size+0x453/0x600 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.348385] ? _raw_spin_unlock+0xe/0x40 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.348597] do_sys_openat2+0xe6/0x180 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.348804] ? > __pfx_do_sys_openat2+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.349047] ? > __kasan_check_write+0x14/0x30 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.349302] ? > lock_vma_under_rcu+0x2e9/0x730 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.349556] __x64_sys_openat+0x108/0x240 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.349772] ? > __pfx___x64_sys_openat+0x10/0x10 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.350023] x64_sys_call+0x134f/0x2350 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.350239] do_syscall_64+0x82/0xd50 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.350440] ? __kasan_check_read+0x11/0x20 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.350660] ? > fpregs_assert_state_consistent+0x5c/0x100 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.350968] ? > irqentry_exit_to_user_mode+0x2e/0x2a0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.351240] ? irqentry_exit+0x43/0x50 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.351442] ? exc_page_fault+0x95/0x100 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.351647] > entry_SYSCALL_64_after_hwframe+0x76/0x7e > Nov 24 12:51:11 ceph-0005 kernel: [ 90.351933] RIP: 0033:0x7f6a0ff19a8c > Nov 24 12:51:11 ceph-0005 kernel: [ 90.352129] Code: 24 18 31 c0 41 83 e2 40 > 75 44 89 f0 25 00 00 41 00 3d 00 00 41 00 74 36 44 89 c2 4c 89 ce bf 9c ff ff ff > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 44 48 8b 54 24 18 64 48 2b 14 25 28 > 00 00 00 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.353095] RSP: 002b:00007ffe25b42190 > EFLAGS: 00000287 ORIG_RAX: 0000000000000101 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.353939] RAX: ffffffffffffffda RBX: > 0000000000000001 RCX: 00007f6a0ff19a8c > Nov 24 12:51:11 ceph-0005 kernel: [ 90.354382] RDX: 0000000000090800 RSI: > 00005c7d3b043a90 RDI: 00000000ffffff9c > Nov 24 12:51:11 ceph-0005 kernel: [ 90.354803] RBP: 00007ffe25b423f0 R08: > 0000000000090800 R09: 00005c7d3b043a90 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.355276] R10: 0000000000000000 R11: > 0000000000000287 R12: 00005c7d3ad24354 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.355686] R13: 00005c7d3b043a90 R14: > 0000000000000000 R15: 00005c7d3ad24353 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.356091] </TASK> > Nov 24 12:51:11 ceph-0005 kernel: [ 90.356231] Modules linked in: > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > input_leds psmouse rapl vga16fb serio_raw vgastate floppy i2c_piix4 mac_hid > qemu_fw_cfg i2c_smbus bochs pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > parport efi_pstore > Nov 24 12:51:11 ceph-0005 kernel: [ 90.358735] CR2: 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.358972] ---[ end trace 0000000000000000 > ]--- > Nov 24 12:51:11 ceph-0005 kernel: [ 90.359341] RIP: 0010:strcmp+0x1c/0x40 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.359581] Code: 90 90 90 90 90 90 90 90 > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > cc cc 31 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.360687] RSP: 0018:ffff8881b8faf600 > EFLAGS: 00010246 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.361008] RAX: 0000000000000000 RBX: > ffff88810bc59600 RCX: 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.361488] RDX: 0000000000000063 RSI: > 0000000000000000 RDI: ffff888110739020 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.362074] RBP: ffff8881b8faf6e8 R08: > 0000000000000000 R09: 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.362498] R10: 0000000000000000 R11: > 0000000000000000 R12: dffffc0000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.362969] R13: ffff88811e104000 R14: > 0000000000000000 R15: 0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.363442] FS: 00007f6a1017c740(0000) > GS:ffff88825f722000(0000) knlGS:0000000000000000 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.363971] CS: 0010 DS: 0000 ES: 0000 > CR0: 0000000080050033 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.364338] CR2: 0000000000000000 CR3: > 000000010216e006 CR4: 0000000000772ef0 > Nov 24 12:51:11 ceph-0005 kernel: [ 90.364738] PKRU: 55555554 > > So, the main sequence is: > > ceph_open() > -> ceph_mdsc_build_path() > -> ceph_mds_check_access() > -> ceph_mds_auth_match() > -> crash happens > > > > > > > And I am not sure that we can mandate anyone to use the old syntax. If there is > > > some other opportunity, then someone could use it. But, maybe, I am missing the > > > point. :) What do you mean by "Mandate passing the fsname with old syntax"? > > > > In the old mount syntax, the fsname is passed as on optional argument > > using 'mds_namespace'. > > I was suggesting to mandate it if possible. But I guess it breaks > > backward compatibility. > > > > > > > > > > We had a private discussion with Ilya. Yes, he also mentioned the breaking of > backward compatibility for the case of mandating passing the fsname with old > syntax. He believes that: "Use the default fsname when mdsc->fsc->mount_options- > >mds_namespace is NULL during comparison seems like a sensible approach to me". > > Thanks, > Slava. > > > > > > > > > > > > > > > > > On Thu, Nov 20, 2025 at 4:47 AM Viacheslav Dubeyko > > > > <Slava.Dubeyko@ibm.com> wrote: > > > > > > > > > > On Wed, 2025-11-19 at 15:02 -0800, Gregory Farnum wrote: > > > > > > > > > > > > That doesn’t sound right — this is authentication code. If the authorization is supplied for a namespace and we are mounting without a namespace at all, isn’t that a jailbreak? So the NULL pointer should be accepted in one direction, but denied in the other? > > > > > > > > > > What is your particular suggestion? I am simply fixing the kernel crash after > > > > > the 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"). We didn't have any > > > > > check before. Do you imply that 22c73d52a6d0 ("ceph: fix multifs mds auth caps > > > > > issue") fix is incorrect and we need to rework it somehow? > > > > > > > > > > If we will not have any fix, then 6.18 release will have broken CephFS kernel > > > > > client. > > > > > > > > > > Thanks, > > > > > Slava. > > > > > > > > > > > > > > > > > On Wed, Nov 19, 2025 at 2:54 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > > > > > > On Wed, 2025-11-19 at 23:40 +0100, Ilya Dryomov wrote: > > > > > > > > On Wed, Nov 19, 2025 at 8:38 PM Viacheslav Dubeyko <slava@dubeyko.com> wrote: > > > > > > > > > > > > > > > > > > From: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > > > > > > > > > > > > The CephFS kernel client has regression starting from 6.18-rc1. > > > > > > > > > > > > > > > > > > sudo ./check -g quick > > > > > > > > > FSTYP -- ceph > > > > > > > > > PLATFORM -- Linux/x86_64 ceph-0005 6.18.0-rc5+ #52 SMP PREEMPT_DYNAMIC Fri > > > > > > > > > Nov 14 11:26:14 PST 2025 > > > > > > > > > MKFS_OPTIONS -- 192.168.1.213:3300:/scratch > > > > > > > > > MOUNT_OPTIONS -- -o name=admin,ms_mode=secure 192.168.1.213:3300:/scratch > > > > > > > > > /mnt/cephfs/scratch > > > > > > > > > > > > > > > > > > Killed > > > > > > > > > > > > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.723902] libceph: mon0 > > > > > > > > > (2)192.168.1.213:3300 session established > > > > > > > > > Nov 14 11:48:10 ceph-0005 kernel: [ 154.727225] libceph: client167616 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087260] BUG: kernel NULL pointer > > > > > > > > > dereference, address: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.087756] #PF: supervisor read access in > > > > > > > > > kernel mode > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088043] #PF: error_code(0x0000) - not- > > > > > > > > > present page > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088302] PGD 0 P4D 0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.088688] Oops: Oops: 0000 [#1] SMP KASAN > > > > > > > > > NOPTI > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.090080] CPU: 4 UID: 0 PID: 3453 Comm: > > > > > > > > > xfs_io Not tainted 6.18.0-rc5+ #52 PREEMPT(voluntary) > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.091245] Hardware name: QEMU Standard PC > > > > > > > > > (i440FX + PIIX, 1996), BIOS 1.17.0-5.fc42 04/01/2014 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092103] RIP: 0010:strcmp+0x1c/0x40 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.092493] Code: 90 90 90 90 90 90 90 90 > > > > > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > > > > > cc cc 31 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094057] RSP: 0018:ffff8881536875c0 > > > > > > > > > EFLAGS: 00010246 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.094522] RAX: 0000000000000000 RBX: > > > > > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095114] RDX: 0000000000000063 RSI: > > > > > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.095714] RBP: ffff8881536876a8 R08: > > > > > > > > > 0000000000000000 R09: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096297] R10: 0000000000000000 R11: > > > > > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.096889] R13: ffff8881061d0000 R14: > > > > > > > > > 0000000000000000 R15: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.097490] FS: 000074a85c082840(0000) > > > > > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098146] CS: 0010 DS: 0000 ES: 0000 > > > > > > > > > CR0: 0000000080050033 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.098630] CR2: 0000000000000000 CR3: > > > > > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099219] PKRU: 55555554 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099476] Call Trace: > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099686] <TASK> > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.099873] ? > > > > > > > > > ceph_mds_check_access+0x348/0x1760 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100267] ? > > > > > > > > > __kasan_check_write+0x14/0x30 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100671] ? lockref_get+0xb1/0x170 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.100979] ? > > > > > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101372] ceph_open+0x322/0xef0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101669] ? __pfx_ceph_open+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.101996] ? > > > > > > > > > __pfx_apparmor_file_open+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102434] ? > > > > > > > > > __ceph_caps_issued_mask_metric+0xd6/0x180 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.102911] do_dentry_open+0x7bf/0x10e0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103249] ? __pfx_ceph_open+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103508] vfs_open+0x6d/0x450 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103697] ? may_open+0xec/0x370 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.103893] path_openat+0x2017/0x50a0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104110] ? __pfx_path_openat+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104345] ? > > > > > > > > > __pfx_stack_trace_save+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104599] ? > > > > > > > > > stack_depot_save_flags+0x28/0x8f0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.104865] ? stack_depot_save+0xe/0x20 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105063] do_filp_open+0x1b4/0x450 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105253] ? > > > > > > > > > __pfx__raw_spin_lock_irqsave+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105538] ? __pfx_do_filp_open+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105748] ? __link_object+0x13d/0x2b0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.105949] ? > > > > > > > > > __pfx__raw_spin_lock+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106169] ? > > > > > > > > > __check_object_size+0x453/0x600 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106428] ? _raw_spin_unlock+0xe/0x40 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106635] do_sys_openat2+0xe6/0x180 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.106827] ? > > > > > > > > > __pfx_do_sys_openat2+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107052] __x64_sys_openat+0x108/0x240 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107258] ? > > > > > > > > > __pfx___x64_sys_openat+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107529] ? > > > > > > > > > __pfx___handle_mm_fault+0x10/0x10 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.107783] x64_sys_call+0x134f/0x2350 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108007] do_syscall_64+0x82/0xd50 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108201] ? > > > > > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108467] ? do_syscall_64+0xba/0xd50 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108626] ? __kasan_check_read+0x11/0x20 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.108801] ? > > > > > > > > > count_memcg_events+0x25b/0x400 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109013] ? handle_mm_fault+0x38b/0x6a0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109216] ? __kasan_check_read+0x11/0x20 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109457] ? > > > > > > > > > fpregs_assert_state_consistent+0x5c/0x100 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109724] ? > > > > > > > > > irqentry_exit_to_user_mode+0x2e/0x2a0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.109991] ? irqentry_exit+0x43/0x50 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110180] ? exc_page_fault+0x95/0x100 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110389] > > > > > > > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110638] RIP: 0033:0x74a85bf145ab > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.110821] Code: 25 00 00 41 00 3d 00 00 > > > > > > > > > 41 00 74 4b 64 8b 04 25 18 00 00 00 85 c0 75 67 44 89 e2 48 89 ee bf 9c ff ff ff > > > > > > > > > b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 91 00 00 00 48 8b 54 24 28 64 48 > > > > > > > > > 2b 14 25 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.111724] RSP: 002b:00007ffc77d316d0 > > > > > > > > > EFLAGS: 00000246 ORIG_RAX: 0000000000000101 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112080] RAX: ffffffffffffffda RBX: > > > > > > > > > 0000000000000002 RCX: 000074a85bf145ab > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112442] RDX: 0000000000000000 RSI: > > > > > > > > > 00007ffc77d32789 RDI: 00000000ffffff9c > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.112790] RBP: 00007ffc77d32789 R08: > > > > > > > > > 00007ffc77d31980 R09: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113125] R10: 0000000000000000 R11: > > > > > > > > > 0000000000000246 R12: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113502] R13: 00000000ffffffff R14: > > > > > > > > > 0000000000000180 R15: 0000000000000001 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113838] </TASK> > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.113957] Modules linked in: > > > > > > > > > intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core > > > > > > > > > pmt_telemetry pmt_discovery pmt_class intel_pmc_ssram_telemetry intel_vsec > > > > > > > > > kvm_intel kvm joydev irqbypass polyval_clmulni ghash_clmulni_intel aesni_intel > > > > > > > > > rapl floppy input_leds psmouse i2c_piix4 vga16fb mac_hid i2c_smbus vgastate > > > > > > > > > serio_raw bochs qemu_fw_cfg pata_acpi sch_fq_codel rbd msr parport_pc ppdev lp > > > > > > > > > parport efi_pstore > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116339] CR2: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116574] ---[ end trace 0000000000000000 > > > > > > > > > ]--- > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.116826] RIP: 0010:strcmp+0x1c/0x40 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.117058] Code: 90 90 90 90 90 90 90 90 > > > > > > > > > 90 90 90 90 90 90 31 c0 eb 14 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 c0 01 84 > > > > > > > > > d2 74 19 0f b6 14 07 <3a> 14 06 74 ef 19 c0 83 c8 01 31 d2 31 f6 31 ff c3 cc cc > > > > > > > > > cc cc 31 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118070] RSP: 0018:ffff8881536875c0 > > > > > > > > > EFLAGS: 00010246 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118362] RAX: 0000000000000000 RBX: > > > > > > > > > ffff888116003200 RCX: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.118748] RDX: 0000000000000063 RSI: > > > > > > > > > 0000000000000000 RDI: ffff88810126c900 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119116] RBP: ffff8881536876a8 R08: > > > > > > > > > 0000000000000000 R09: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119492] R10: 0000000000000000 R11: > > > > > > > > > 0000000000000000 R12: dffffc0000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.119865] R13: ffff8881061d0000 R14: > > > > > > > > > 0000000000000000 R15: 0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120242] FS: 000074a85c082840(0000) > > > > > > > > > GS:ffff8882401a4000(0000) knlGS:0000000000000000 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.120704] CS: 0010 DS: 0000 ES: 0000 > > > > > > > > > CR0: 0000000080050033 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121008] CR2: 0000000000000000 CR3: > > > > > > > > > 0000000110ebd001 CR4: 0000000000772ef0 > > > > > > > > > Nov 14 11:48:11 ceph-0005 kernel: [ 155.121409] PKRU: 55555554 > > > > > > > > > > > > > > > > > > We have issue here [1] if fs_name == NULL: > > > > > > > > > > > > > > > > > > const char fs_name = mdsc->fsc->mount_options->mds_namespace; > > > > > > > > > ... > > > > > > > > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > > > > > / fsname mismatch, try next one */ > > > > > > > > > return 0; > > > > > > > > > } > > > > > > > > > > > > > > > > > > The patch fixes the issue by introducing is_fsname_mismatch() method > > > > > > > > > that checks auth->match.fs_name and fs_name pointers validity, and > > > > > > > > > compares the file system names. > > > > > > > > > > > > > > > > > > [1] https://elixir.bootlin.com/linux/v6.18-rc4/source/fs/ceph/mds_client.c#L5666 > > > > > > > > > > > > > > > > > > Fixes: 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue") > > > > > > > > > Signed-off-by: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> > > > > > > > > > cc: Kotresh Hiremath Ravishankar <khiremat@redhat.com> > > > > > > > > > cc: Alex Markuze <amarkuze@redhat.com> > > > > > > > > > cc: Ilya Dryomov <idryomov@gmail.com> > > > > > > > > > cc: Ceph Development <ceph-devel@vger.kernel.org> > > > > > > > > > --- > > > > > > > > > fs/ceph/mds_client.c | 20 +++++++++++++++++--- > > > > > > > > > 1 file changed, 17 insertions(+), 3 deletions(-) > > > > > > > > > > > > > > > > > > diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c > > > > > > > > > index 1740047aef0f..19c75e206300 100644 > > > > > > > > > --- a/fs/ceph/mds_client.c > > > > > > > > > +++ b/fs/ceph/mds_client.c > > > > > > > > > @@ -5647,6 +5647,22 @@ void send_flush_mdlog(struct ceph_mds_session *s) > > > > > > > > > mutex_unlock(&s->s_mutex); > > > > > > > > > } > > > > > > > > > > > > > > > > > > +static inline > > > > > > > > > +bool is_fsname_mismatch(struct ceph_client *cl, > > > > > > > > > + const char *fs_name1, const char *fs_name2) > > > > > > > > > +{ > > > > > > > > > + if (!fs_name1 || !fs_name2) > > > > > > > > > + return false; > > > > > > > > > > > > > > > > Hi Slava, > > > > > > > > > > > > > > > > It looks like this would declare a match (return false for "mismatch") > > > > > > > > in case ceph_mds_cap_auth is defined to require a particular fs_name but > > > > > > > > no mds_namespace was passed on mount. Is that the desired behavior? > > > > > > > > > > > > > > > > > > > > > > Hi Ilya, > > > > > > > > > > > > > > Before 22c73d52a6d0 ("ceph: fix multifs mds auth caps issue"), we had no such > > > > > > > check in the logic of ceph_mds_auth_match(). So, if auth->match.fs_name or > > > > > > > fs_name is NULL, then we cannot say that they match or not. It means that we > > > > > > > need to continue logic, this is why is_fsname_mismatch() returns false. > > > > > > > Otherwise, if we stop logic by returning true, then we have bunch of xfstests > > > > > > > failures. > > > > > > > > > > > > > > Thanks, > > > > > > > Slava. > > > > > > > > > > > > > > > > + > > > > > > > > > + doutc(cl, "fsname check fs_name1=%s fs_name2=%s\n", > > > > > > > > > + fs_name1, fs_name2); > > > > > > > > > + > > > > > > > > > + if (strcmp(fs_name1, fs_name2)) > > > > > > > > > + return true; > > > > > > > > > + > > > > > > > > > + return false; > > > > > > > > > +} > > > > > > > > > + > > > > > > > > > static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > > > > > struct ceph_mds_cap_auth *auth, > > > > > > > > > const struct cred *cred, > > > > > > > > > @@ -5661,9 +5677,7 @@ static int ceph_mds_auth_match(struct ceph_mds_client *mdsc, > > > > > > > > > u32 gid, tlen, len; > > > > > > > > > int i, j; > > > > > > > > > > > > > > > > > > - doutc(cl, "fsname check fs_name=%s match.fs_name=%s\n", > > > > > > > > > - fs_name, auth->match.fs_name ? auth->match.fs_name : ""); > > > > > > > > > - if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) { > > > > > > > > > + if (is_fsname_mismatch(cl, auth->match.fs_name, fs_name)) { > > > > > > > > > /* fsname mismatch, try next one */ > > > > > > > > > return 0; > > > > > > > > > } > > > > > > > > > -- > > > > > > > > > 2.51.1 > > > > > > > > > > > > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [PATCH] ceph: fix kernel crash in ceph_open() 2025-11-27 7:33 ` Kotresh Hiremath Ravishankar @ 2025-12-01 20:04 ` Viacheslav Dubeyko 2025-12-02 11:30 ` Ilya Dryomov 0 siblings, 1 reply; 13+ messages in thread From: Viacheslav Dubeyko @ 2025-12-01 20:04 UTC (permalink / raw) To: Kotresh Hiremath Ravishankar Cc: Viacheslav Dubeyko, Patrick Donnelly, ceph-devel@vger.kernel.org, slava@dubeyko.com, linux-fsdevel@vger.kernel.org, Gregory Farnum, Alex Markuze, idryomov@gmail.com, Pavan Rallabhandi On Thu, 2025-11-27 at 13:03 +0530, Kotresh Hiremath Ravishankar wrote: > On Tue, Nov 25, 2025 at 2:42 AM Viacheslav Dubeyko > <Slava.Dubeyko@ibm.com> wrote: > > > > On Tue, 2025-11-25 at 00:48 +0530, Kotresh Hiremath Ravishankar wrote: > > > On Fri, Nov 21, 2025 at 1:47 AM Viacheslav Dubeyko > > > <Slava.Dubeyko@ibm.com> wrote: > > > > > > > > On Thu, 2025-11-20 at 19:50 +0530, Kotresh Hiremath Ravishankar wrote: > > > > > Hi All, > > > > > > > > > > I think the patch is necessary and fixes the crash. There is no harm > > > > > in taking this patch as it behaves like an old kernel with this > > > > > particular scenario. > > > > > > > > > > When does the issue happen: > > > > > - The issue happens only when the old mount syntax is used where > > > > > passing the file system name is optional in which case, it chooses the > > > > > default mds namespace but doesn't get filled in the > > > > > mdsc->fsc->mount_options->mds_namespace. > > > > > - Along with the above, the mount user should be non admin. > > > > > Does it break the earlier fix ? > > > > > - Not fully!!! Though the open does succeed, the subsequent > > > > > operation like write would get EPERM. I am not exactly able to > > > > > recollect but this was discussed before writing the fix 22c73d52a6d0 > > > > > ("ceph: fix multifs mds auth caps issue"), it's guarded by another > > > > > check before actual operation like write. > > > > > > > > > > I think there are a couple of options to fix this cleanly. > > > > > 1. Use the default fsname when > > > > > mdsc->fsc->mount_options->mds_namespace is NULL during comparison. > > > > > 2. Mandate passing the fsname with old syntax ? > > > > > > > > > > > > > Anyway, we should be ready operate correctly if fsname or/and auth- > > > > > match.fs_name are NULL. And if we need to make the fix more cleanly, then we > > > > can introduce another patch with nicer fix. > > > > > > > > I am not completely sure how default fsname can be applicable here. If I > > > > understood the CephFS mount logic correctly, then fsname can be NULL during some > > > > initial steps. But, finally, we will have the real fsname for comparison. But I > > > > don't know if it's right of assuming that fsname == NULL is equal to fsname == > > > > default_name. > > > > > > We are pretty sure fsname is NULL only if the old mount syntax is used > > > without providing the > > > fsname in the optional arg. I believe kclient knows the fsname that's > > > mounted somewhere in this case ? > > > I am not sure though. If so, it can be used. If not, then can we rely > > > on what mds sends as part > > > of the mdsmap? > > > > > > <skipped> > > > > > > > > > > And I am not sure that we can mandate anyone to use the old syntax. If there is > > > > some other opportunity, then someone could use it. But, maybe, I am missing the > > > > point. :) What do you mean by "Mandate passing the fsname with old syntax"? > > > > > > In the old mount syntax, the fsname is passed as on optional argument > > > using 'mds_namespace'. > > > I was suggesting to mandate it if possible. But I guess it breaks > > > backward compatibility. > > > > > > > > > > > > > > > We had a private discussion with Ilya. Yes, he also mentioned the breaking of > > backward compatibility for the case of mandating passing the fsname with old > > syntax. He believes that: "Use the default fsname when mdsc->fsc->mount_options- > > > mds_namespace is NULL during comparison seems like a sensible approach to me". > > > > OK. So, what finally should we consider like a right solution/fix here? Thanks, Slava. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] ceph: fix kernel crash in ceph_open() 2025-12-01 20:04 ` Viacheslav Dubeyko @ 2025-12-02 11:30 ` Ilya Dryomov 2025-12-02 20:21 ` Patrick Donnelly 0 siblings, 1 reply; 13+ messages in thread From: Ilya Dryomov @ 2025-12-02 11:30 UTC (permalink / raw) To: Viacheslav Dubeyko Cc: Kotresh Hiremath Ravishankar, Viacheslav Dubeyko, Patrick Donnelly, ceph-devel@vger.kernel.org, slava@dubeyko.com, linux-fsdevel@vger.kernel.org, Gregory Farnum, Alex Markuze, Pavan Rallabhandi, Venky Shankar On Mon, Dec 1, 2025 at 9:04 PM Viacheslav Dubeyko <Slava.Dubeyko@ibm.com> wrote: > > On Thu, 2025-11-27 at 13:03 +0530, Kotresh Hiremath Ravishankar wrote: > > On Tue, Nov 25, 2025 at 2:42 AM Viacheslav Dubeyko > > <Slava.Dubeyko@ibm.com> wrote: > > > > > > On Tue, 2025-11-25 at 00:48 +0530, Kotresh Hiremath Ravishankar wrote: > > > > On Fri, Nov 21, 2025 at 1:47 AM Viacheslav Dubeyko > > > > <Slava.Dubeyko@ibm.com> wrote: > > > > > > > > > > On Thu, 2025-11-20 at 19:50 +0530, Kotresh Hiremath Ravishankar wrote: > > > > > > Hi All, > > > > > > > > > > > > I think the patch is necessary and fixes the crash. There is no harm > > > > > > in taking this patch as it behaves like an old kernel with this > > > > > > particular scenario. > > > > > > > > > > > > When does the issue happen: > > > > > > - The issue happens only when the old mount syntax is used where > > > > > > passing the file system name is optional in which case, it chooses the > > > > > > default mds namespace but doesn't get filled in the > > > > > > mdsc->fsc->mount_options->mds_namespace. > > > > > > - Along with the above, the mount user should be non admin. > > > > > > Does it break the earlier fix ? > > > > > > - Not fully!!! Though the open does succeed, the subsequent > > > > > > operation like write would get EPERM. I am not exactly able to > > > > > > recollect but this was discussed before writing the fix 22c73d52a6d0 > > > > > > ("ceph: fix multifs mds auth caps issue"), it's guarded by another > > > > > > check before actual operation like write. > > > > > > > > > > > > I think there are a couple of options to fix this cleanly. > > > > > > 1. Use the default fsname when > > > > > > mdsc->fsc->mount_options->mds_namespace is NULL during comparison. > > > > > > 2. Mandate passing the fsname with old syntax ? > > > > > > > > > > > > > > > > Anyway, we should be ready operate correctly if fsname or/and auth- > > > > > > match.fs_name are NULL. And if we need to make the fix more cleanly, then we > > > > > can introduce another patch with nicer fix. > > > > > > > > > > I am not completely sure how default fsname can be applicable here. If I > > > > > understood the CephFS mount logic correctly, then fsname can be NULL during some > > > > > initial steps. But, finally, we will have the real fsname for comparison. But I > > > > > don't know if it's right of assuming that fsname == NULL is equal to fsname == > > > > > default_name. > > > > > > > > We are pretty sure fsname is NULL only if the old mount syntax is used > > > > without providing the > > > > fsname in the optional arg. I believe kclient knows the fsname that's > > > > mounted somewhere in this case ? > > > > I am not sure though. If so, it can be used. If not, then can we rely > > > > on what mds sends as part > > > > of the mdsmap? > > > > > > > > > > <skipped> > > > > > > > > > > > > > > And I am not sure that we can mandate anyone to use the old syntax. If there is > > > > > some other opportunity, then someone could use it. But, maybe, I am missing the > > > > > point. :) What do you mean by "Mandate passing the fsname with old syntax"? > > > > > > > > In the old mount syntax, the fsname is passed as on optional argument > > > > using 'mds_namespace'. > > > > I was suggesting to mandate it if possible. But I guess it breaks > > > > backward compatibility. > > > > > > > > > > > > > > > > > > > > We had a private discussion with Ilya. Yes, he also mentioned the breaking of > > > backward compatibility for the case of mandating passing the fsname with old > > > syntax. He believes that: "Use the default fsname when mdsc->fsc->mount_options- > > > > mds_namespace is NULL during comparison seems like a sensible approach to me". > > > > > > > > OK. So, what finally should we consider like a right solution/fix here? Hi Slava, I think the right solution would be a patch that establishes consistency with the userspace client. What does ceph-fuse do when --client_fs option isn't passed? It's the exact equivalent of mds_namespace mount option (--client_mds_namespace is what it used to be named), so the kernel client just needs to be made to do exactly the same. After taking a deeper look I doubt that using the default fs_name for the comparison would be sufficient and not prone to edge cases. First, even putting the NULL dereference aside, both the existing check by Kotresh if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) /* mismatch */ and your proposed check if (!fs_name1 || !fs_name2) /* match */ if (strcmp(fs_name1, fs_name2)) /* mismatch */ aren't equivalent to bool match_fs(std::string_view target_fs) const { return fs_name == target_fs || fs_name.empty() || fs_name == "*"; } in src/mds/MDSAuthCaps.h -- "*" isn't handled at all. Second, I'm not following a reason to only "validate" fs_name against mds_namespace option in ceph_mdsmap_decode(). Why not hold onto it and actually use it in ceph_mds_auth_match() for the comparison as done in src/client/Client.cc? int Client::mds_check_access(std::string& path, const UserPerm& perms, int mask) { ... std::string_view fs_name = mdsmap->get_fs_name(); <--------- for (auto& s: cap_auths) { ... if (s.match.match(fs_name, path, perms.uid(), perms.gid(), &gid_list)) { /* match */ AFAIU the default fs_name would come into the picture only in case of a super ancient cluster with prior to mdsmap v8 encoding. I haven't really looked at this code before, so it's possible that there are other things that are missing/inconsistent here. I'd ask that the final patch is formally reviewed by Venky and Patrick as they were the approvers on https://github.com/ceph/ceph/pull/64550 in userspace. Thanks, Ilya ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] ceph: fix kernel crash in ceph_open() 2025-12-02 11:30 ` Ilya Dryomov @ 2025-12-02 20:21 ` Patrick Donnelly 2025-12-03 19:48 ` Viacheslav Dubeyko 0 siblings, 1 reply; 13+ messages in thread From: Patrick Donnelly @ 2025-12-02 20:21 UTC (permalink / raw) To: Ilya Dryomov Cc: Viacheslav Dubeyko, Kotresh Hiremath Ravishankar, Viacheslav Dubeyko, ceph-devel@vger.kernel.org, slava@dubeyko.com, linux-fsdevel@vger.kernel.org, Gregory Farnum, Alex Markuze, Pavan Rallabhandi, Venky Shankar [-- Attachment #1: Type: text/plain, Size: 2931 bytes --] I started work on a patch and it is largely in agreement with what Ilya suggested below. On Tue, Dec 2, 2025 at 6:30 AM Ilya Dryomov <idryomov@gmail.com> wrote: > Hi Slava, > > I think the right solution would be a patch that establishes > consistency with the userspace client. What does ceph-fuse do when > --client_fs option isn't passed? It's the exact equivalent of > mds_namespace mount option (--client_mds_namespace is what it used to > be named), so the kernel client just needs to be made to do exactly the > same. > > After taking a deeper look I doubt that using the default fs_name for > the comparison would be sufficient and not prone to edge cases. First, > even putting the NULL dereference aside, both the existing check by > Kotresh > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) > /* mismatch */ > > and your proposed check > > if (!fs_name1 || !fs_name2) > /* match */ > > if (strcmp(fs_name1, fs_name2)) > /* mismatch */ > > aren't equivalent to > > bool match_fs(std::string_view target_fs) const { > return fs_name == target_fs || fs_name.empty() || fs_name == "*"; > } > > in src/mds/MDSAuthCaps.h -- "*" isn't handled at all. > > Second, I'm not following a reason to only "validate" fs_name against > mds_namespace option in ceph_mdsmap_decode(). Why not hold onto it and > actually use it in ceph_mds_auth_match() for the comparison as done in > src/client/Client.cc? > > int Client::mds_check_access(std::string& path, const UserPerm& perms, int mask) > { > ... > std::string_view fs_name = mdsmap->get_fs_name(); <--------- > for (auto& s: cap_auths) { > ... > if (s.match.match(fs_name, path, perms.uid(), perms.gid(), &gid_list)) { > /* match */ > > AFAIU the default fs_name would come into the picture only in case of > a super ancient cluster with prior to mdsmap v8 encoding. > > I haven't really looked at this code before, so it's possible that > there are other things that are missing/inconsistent here. I'd ask > that the final patch is formally reviewed by Venky and Patrick as > they were the approvers on https://github.com/ceph/ceph/pull/64550 > in userspace. We should match the ceph-fuse client behavior. Attached is the patch (I've not built) which roughly gets us there. The missing bit will be the "*" glob matching. In summary, we should definitely start decoding `fs_name` from the MDSMap and do strict authorizations checks against it. Note that the `--mds_namespace` should only be used for selecting the file system to mount and nothing else. It's possible no mds_namespace is specified but the kernel will mount the only file system that exists which may have name "foo". -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D [-- Attachment #2: fs_name.patch --] [-- Type: application/x-patch, Size: 2299 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* RE: [PATCH] ceph: fix kernel crash in ceph_open() 2025-12-02 20:21 ` Patrick Donnelly @ 2025-12-03 19:48 ` Viacheslav Dubeyko 0 siblings, 0 replies; 13+ messages in thread From: Viacheslav Dubeyko @ 2025-12-03 19:48 UTC (permalink / raw) To: idryomov@gmail.com, Patrick Donnelly Cc: Venky Shankar, Viacheslav Dubeyko, Kotresh Hiremath Ravishankar, ceph-devel@vger.kernel.org, slava@dubeyko.com, linux-fsdevel@vger.kernel.org, Gregory Farnum, Alex Markuze, Pavan Rallabhandi Hi Patrick, On Tue, 2025-12-02 at 15:21 -0500, Patrick Donnelly wrote: > I started work on a patch and it is largely in agreement with what > Ilya suggested below. > > On Tue, Dec 2, 2025 at 6:30 AM Ilya Dryomov <idryomov@gmail.com> wrote: > > Hi Slava, > > > > I think the right solution would be a patch that establishes > > consistency with the userspace client. What does ceph-fuse do when > > --client_fs option isn't passed? It's the exact equivalent of > > mds_namespace mount option (--client_mds_namespace is what it used to > > be named), so the kernel client just needs to be made to do exactly the > > same. > > > > After taking a deeper look I doubt that using the default fs_name for > > the comparison would be sufficient and not prone to edge cases. First, > > even putting the NULL dereference aside, both the existing check by > > Kotresh > > > > if (auth->match.fs_name && strcmp(auth->match.fs_name, fs_name)) > > /* mismatch */ > > > > and your proposed check > > > > if (!fs_name1 || !fs_name2) > > /* match */ > > > > if (strcmp(fs_name1, fs_name2)) > > /* mismatch */ > > > > aren't equivalent to > > > > bool match_fs(std::string_view target_fs) const { > > return fs_name == target_fs || fs_name.empty() || fs_name == "*"; > > } > > > > in src/mds/MDSAuthCaps.h -- "*" isn't handled at all. > > > > Second, I'm not following a reason to only "validate" fs_name against > > mds_namespace option in ceph_mdsmap_decode(). Why not hold onto it and > > actually use it in ceph_mds_auth_match() for the comparison as done in > > src/client/Client.cc? > > > > int Client::mds_check_access(std::string& path, const UserPerm& perms, int mask) > > { > > ... > > std::string_view fs_name = mdsmap->get_fs_name(); <--------- > > for (auto& s: cap_auths) { > > ... > > if (s.match.match(fs_name, path, perms.uid(), perms.gid(), &gid_list)) { > > /* match */ > > > > AFAIU the default fs_name would come into the picture only in case of > > a super ancient cluster with prior to mdsmap v8 encoding. > > > > I haven't really looked at this code before, so it's possible that > > there are other things that are missing/inconsistent here. I'd ask > > that the final patch is formally reviewed by Venky and Patrick as > > they were the approvers on https://github.com/ceph/ceph/pull/64550 > > in userspace. > > We should match the ceph-fuse client behavior. > > Attached is the patch (I've not built) which roughly gets us there. > The missing bit will be the "*" glob matching. > > In summary, we should definitely start decoding `fs_name` from the > MDSMap and do strict authorizations checks against it. Note that the > `--mds_namespace` should only be used for selecting the file system to > mount and nothing else. It's possible no mds_namespace is specified > but the kernel will mount the only file system that exists which may > have name "foo". > > Is the attached patch ready for review and testing? Should we wait the final version of the patch? Thanks, Slava. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-12-03 19:48 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-11-19 19:37 [PATCH] ceph: fix kernel crash in ceph_open() Viacheslav Dubeyko
2025-11-19 22:40 ` Ilya Dryomov
2025-11-19 22:50 ` Viacheslav Dubeyko
[not found] ` <CAJ4mKGZexNm--cKsT0sc0vmiAyWrA1a6FtmaGJ6WOsg8d_2R3w@mail.gmail.com>
2025-11-19 23:16 ` Viacheslav Dubeyko
2025-11-20 14:20 ` Kotresh Hiremath Ravishankar
2025-11-20 20:17 ` Viacheslav Dubeyko
2025-11-24 19:18 ` Kotresh Hiremath Ravishankar
2025-11-24 21:12 ` Viacheslav Dubeyko
2025-11-27 7:33 ` Kotresh Hiremath Ravishankar
2025-12-01 20:04 ` Viacheslav Dubeyko
2025-12-02 11:30 ` Ilya Dryomov
2025-12-02 20:21 ` Patrick Donnelly
2025-12-03 19:48 ` Viacheslav Dubeyko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).