[Bug 220669] New: Drive issues cause system and coredump and reboot

[Bug 220669] New: Drive issues cause system and coredump and reboot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=220669

            Bug ID: 220669
           Summary: Drive issues cause system and coredump and reboot
           Product: File System
           Version: 2.5
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P3
         Component: XFS
          Assignee: filesystem_xfs@kernel-bugs.kernel.org
          Reporter: bshephar@bne-home.net
        Regression: No

We have a number of drives across our clusters that don't present as failing,
but seem to have read errors that cause the system to coredump and reboot.
Failing drives is obviously not the fault of XFS, but my expectation would be
that it doesn't completely cause the system to hang and need to reboot.

Kernel Version:
$ uname -r
4.18.0-553.69.1.el8_10.x86_64




Ideally, I think XFS should be able to more gracefully handle this situation.

We can see the drive at /dev/sdv have issues here before the Call Trace and
coredump:

[  998.356008] sd 0:0:21:0: [sdv] tag#415 FAILED Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE cmd_age=3s
[  998.365528] sd 0:0:21:0: [sdv] tag#415 Sense Key : Medium Error [current]
[descriptor]
[  998.373544] sd 0:0:21:0: [sdv] tag#415 Add. Sense: Unrecovered read error
[  998.380347] sd 0:0:21:0: [sdv] tag#415 CDB: Read(16) 88 00 00 00 00 05 01 1f
6d 90 00 00 00 08 00 00
[  998.389524] blk_update_request: critical medium error, dev sdv, sector
21493673360 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0
[  998.404141] XFS (sdv): metadata I/O error in "xfs_da_read_buf+0xd3/0x120
[xfs]" at daddr 0x5011f6d90 len 8 error 61
[  998.414706] BUG: unable to handle kernel NULL pointer dereference at
0000000000000020
[  998.422554] PGD 1218fc067 P4D 1218fc067 PUD 1d0139067 PMD 0
[  998.428247] Oops: 0002 [#1] SMP NOPTI
[  998.431931] CPU: 72 PID: 4802 Comm: swift-object-au Kdump: loaded Not
tainted 4.18.0-553.69.1.el8_10.x86_64 #1
[  998.441958] Hardware name: Dell Inc. PowerEdge R740xd2/0VNGN1, BIOS 2.14.2
03/23/2022
[  998.449818] RIP: 0010:_raw_spin_lock_irqsave+0x1e/0x40
[  998.454975] Code: cf 80 0b 08 eb 88 90 90 90 90 90 90 0f 1f 44 00 00 53 9c
58 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1
17 75 09 48 89 d8 5b c3 cc cc cc cc 89 c6 e8 1c 7b 73 ff
[  998.473799] RSP: 0018:ffff99455ccafbe0 EFLAGS: 00010046
[  998.479344] RAX: 0000000000000000 RBX: 0000000000000286 RCX:
0000000000000000
[  998.486695] RDX: 0000000000000001 RSI: 0000000000000000 RDI:
0000000000000020
[  998.494044] RBP: 0000000000000000 R08: 0000000080000000 R09:
ffff8e3abffd5000
[  998.501360] R10: 0000000000000001 R11: ffff99455ccaf848 R12:
0000000000000000
[  998.508690] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[  998.516004] FS:  00007f74aa03e740(0000) GS:ffff8e3941100000(0000)
knlGS:0000000000000000
[  998.524333] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  998.530270] CR2: 0000000000000020 CR3: 00000001cfbb6003 CR4:
00000000007706e0
[  998.537585] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[  998.544901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[  998.552224] PKRU: 55555554
[  998.555136] Call Trace:
[  998.557765]  ? __die_body+0x1a/0x60
[  998.561435]  ? no_context+0x1ba/0x3f0
[  998.565276]  ? __bad_area_nosemaphore+0x157/0x180
[  998.570202]  ? xfs_trans_read_buf_map+0x128/0x360 [xfs]
[  998.575693]  ? do_page_fault+0x37/0x12d
[  998.579713]  ? page_fault+0x1e/0x30
[  998.583390]  ? _raw_spin_lock_irqsave+0x1e/0x40
[  998.588113]  up+0x12/0x50
[  998.590913]  xfs_buf_unlock+0x15/0x70 [xfs]
[  998.595350]  xfs_trans_brelse+0xc6/0xe0 [xfs]
[  998.599960]  xfs_attr_leaf_get+0xb0/0xc0 [xfs]
[  998.604639]  xfs_attr_get+0x9e/0xc0 [xfs]
[  998.608888]  xfs_xattr_get+0x75/0xb0 [xfs]
[  998.613238]  __vfs_getxattr+0x54/0x70
[  998.617104]  vfs_getxattr+0x118/0x140
[  998.620959]  getxattr+0x187/0x1b0
[  998.624470]  ? cp_new_stat+0x150/0x190
[  998.628411]  ? do_vfs_ioctl+0xa4/0x690
[  998.632356]  ? __do_sys_newfstat+0x5e/0x70
[  998.636644]  ? syscall_trace_enter+0x1ff/0x2d0
[  998.641283]  __x64_sys_fgetxattr+0x5f/0xb0
[  998.645573]  do_syscall_64+0x5b/0x1a0
[  998.649429]  entry_SYSCALL_64_after_hwframe+0x66/0xcb
[  998.654672] RIP: 0033:0x7f74a88dc5fe
[  998.658434] Code: 48 8b 0d 8d 78 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e
0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 c1 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 5a 78 39 00 f7 d8 64 89 01 48
[  998.677570] RSP: 002b:00007ffe2289a9f8 EFLAGS: 00000246 ORIG_RAX:
00000000000000c1
[  998.685351] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
00007f74a88dc5fe
[  998.692683] RDX: 0000000000000000 RSI: 00007f749538e7d0 RDI:
0000000000000003
[  998.700024] RBP: 00007ffe2289aa80 R08: 0000000000000000 R09:
00007f74a9f400d0
[  998.707357] R10: 0000000000000000 R11: 0000000000000246 R12:
0000000000000000
[  998.714680] R13: 0000000000000003 R14: 0000000000000000 R15:
0000000000000000
[  998.722013] Modules linked in: dm_mod tun xfs ipt_MASQUERADE
nf_conntrack_netlink xt_addrtype br_netfilter bridge stp llc tcp_diag inet_diag
overlay xt_multiport ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT
nf_reject_ipv4 xt_conntrack nft_counter nft_chain_nat nf_nat nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 ip6_tables ip_tables nft_compat ip_set nf_tables
libcrc32c nfnetlink dell_rbu vfat fat ipmi_ssif intel_rapl_msr
intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common
isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel irdma kvm i40e irqbypass crc32_pclmul ib_uverbs rapl
dell_smbios iTCO_wdt intel_cstate iTCO_vendor_support acpi_ipmi
dell_wmi_descriptor wmi_bmof mei_me dcdbas ib_core intel_uncore pcspkr mei
i2c_i801 ipmi_si lpc_ich wmi ipmi_devintf ipmi_msghandler ext4 mbcache jbd2
sd_mod sg mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt i2c_algo_bit
crct10dif_pclmul drm_shmem_helper ahci ice nvme
[  998.722079]  crc32c_intel libahci drm nvme_core ghash_clmulni_intel libata
megaraid_sas tg3 t10_pi fuse


crash> sys | grep -e CPUS -e LOAD -e RELEASE -e PANIC
        CPUS: 80
LOAD AVERAGE: 79.36, 197.55, 179.00
     RELEASE: 4.18.0-553.69.1.el8_10.x86_64
       PANIC: "BUG: unable to handle kernel NULL pointer dereference at
0000000000000020"

crash> set -p
    PID: 4802
COMMAND: "swift-object-au"
   TASK: ffff8e3ad2990000  [THREAD_INFO: ffff8e3ad2990000]
    CPU: 72
  STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 4802     TASK: ffff8e3ad2990000  CPU: 72   COMMAND: "swift-object-au"
 #0 [ffff99455ccaf910] machine_kexec at ffffffff9b86de53
 #1 [ffff99455ccaf968] __crash_kexec at ffffffff9b9b9e1a
 #2 [ffff99455ccafa28] crash_kexec at ffffffff9b9bad51
 #3 [ffff99455ccafa40] oops_end at ffffffff9b82c131
 #4 [ffff99455ccafa60] no_context at ffffffff9b880da3
 #5 [ffff99455ccafab8] __bad_area_nosemaphore at ffffffff9b881107
 #6 [ffff99455ccafb00] do_page_fault at ffffffff9b881dc7
 #7 [ffff99455ccafb30] page_fault at ffffffff9c4011fe
    [exception RIP: _raw_spin_lock_irqsave+30]
    RIP: ffffffff9c2270be  RSP: ffff99455ccafbe0  RFLAGS: 00010046
    RAX: 0000000000000000  RBX: 0000000000000286  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: 0000000000000000  RDI: 0000000000000020
    RBP: 0000000000000000   R8: 0000000080000000   R9: ffff8e3abffd5000
    R10: 0000000000000001  R11: ffff99455ccaf848  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffff99455ccafbe8] up at ffffffff9c2230e2
 #9 [ffff99455ccafc00] xfs_buf_unlock at ffffffffc0d95455 [xfs]
#10 [ffff99455ccafc20] xfs_trans_brelse at ffffffffc0dcf3b6 [xfs]
#11 [ffff99455ccafc40] xfs_attr_leaf_get at ffffffffc0d49e80 [xfs]
#12 [ffff99455ccafc70] xfs_attr_get at ffffffffc0d4afae [xfs]
#13 [ffff99455ccafc90] xfs_xattr_get at ffffffffc0dbaa05 [xfs]
#14 [ffff99455ccafd40] __vfs_getxattr at ffffffff9bb9a064
#15 [ffff99455ccafd70] vfs_getxattr at ffffffff9bb9a5a8
#16 [ffff99455ccafdb8] getxattr at ffffffff9bb9aa07
#17 [ffff99455ccaff00] __x64_sys_fgetxattr at ffffffff9bb9b17f
#18 [ffff99455ccaff38] do_syscall_64 at ffffffff9b803cab
#19 [ffff99455ccaff50] entry_SYSCALL_64_after_hwframe at ffffffff9c40012e
    RIP: 00007f74a88dc5fe  RSP: 00007ffe2289a9f8  RFLAGS: 00000246
    RAX: ffffffffffffffda  RBX: 0000000000000000  RCX: 00007f74a88dc5fe
    RDX: 0000000000000000  RSI: 00007f749538e7d0  RDI: 0000000000000003
    RBP: 00007ffe2289aa80   R8: 0000000000000000   R9: 00007f74a9f400d0
    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000000
    R13: 0000000000000003  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: 00000000000000c1  CS: 0033  SS: 002b

crash> dis _raw_spin_lock_irqsave+30
0xffffffff9c2270be <_raw_spin_lock_irqsave+30>: lock cmpxchg %edx,(%rdi)

RDI From the above 
    [exception RIP: _raw_spin_lock_irqsave+30]
    RIP: ffffffff9c2270be  RSP: ffff99455ccafbe0  RFLAGS: 00010046
    RAX: 0000000000000000  RBX: 0000000000000286  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: 0000000000000000  RDI: 0000000000000020
    RBP: 0000000000000000   R8: 0000000080000000   R9: ffff8e3abffd5000
    R10: 0000000000000001  R11: ffff99455ccaf848  R12: 0000000000000000
    R13: 0000000000000000  R14: 0000000000000000  R15: 0000000000000000

Seems like that memory address would be incorrect for RDI?
https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt


The `vmcore` is quite large. But I can probably get any information that you
need to investigate further if required.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 220669] Drive issues cause system and coredump and reboot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=220669

--- Comment #1 from Brendan Shephard (bshephar@bne-home.net) ---
Created attachment 308812
  --> https://bugzilla.kernel.org/attachment.cgi?id=308812&action=edit
vmcore-dmesg.txt

The generated vmcore-dmesg.txt file

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

Re: [Bug 220669] New: Drive issues cause system and coredump and reboot

[Dave Chinner]

On Thu, Oct 16, 2025 at 12:10:43AM +0000, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=220669
> 
>             Bug ID: 220669
>            Summary: Drive issues cause system and coredump and reboot
>            Product: File System
>            Version: 2.5
>           Hardware: All
>                 OS: Linux
>             Status: NEW
>           Severity: normal
>           Priority: P3
>          Component: XFS
>           Assignee: filesystem_xfs@kernel-bugs.kernel.org
>           Reporter: bshephar@bne-home.net
>         Regression: No
> 
> We have a number of drives across our clusters that don't present as failing,
> but seem to have read errors that cause the system to coredump and reboot.
> Failing drives is obviously not the fault of XFS, but my expectation would be
> that it doesn't completely cause the system to hang and need to reboot.

Fixed upstream a couple of months back by commit ae668cd567a6 ("xfs:
do not propagate ENODATA disk errors into xattr code"). You'll need
to raise a distro bug (Rocky Linux) to get them to back port it for
you.

Please close this bz.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

[Bug 220669] Drive issues cause system and coredump and reboot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=220669

--- Comment #2 from Dave Chinner (david@fromorbit.com) ---
On Thu, Oct 16, 2025 at 12:10:43AM +0000, bugzilla-daemon@kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=220669
> 
>             Bug ID: 220669
>            Summary: Drive issues cause system and coredump and reboot
>            Product: File System
>            Version: 2.5
>           Hardware: All
>                 OS: Linux
>             Status: NEW
>           Severity: normal
>           Priority: P3
>          Component: XFS
>           Assignee: filesystem_xfs@kernel-bugs.kernel.org
>           Reporter: bshephar@bne-home.net
>         Regression: No
> 
> We have a number of drives across our clusters that don't present as failing,
> but seem to have read errors that cause the system to coredump and reboot.
> Failing drives is obviously not the fault of XFS, but my expectation would be
> that it doesn't completely cause the system to hang and need to reboot.

Fixed upstream a couple of months back by commit ae668cd567a6 ("xfs:
do not propagate ENODATA disk errors into xattr code"). You'll need
to raise a distro bug (Rocky Linux) to get them to back port it for
you.

Please close this bz.

-Dave.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 220669] Drive issues cause system and coredump and reboot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=220669

Artem S. Tashkinov (aros@gmx.com) changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |WILL_NOT_FIX

--- Comment #3 from Artem S. Tashkinov (aros@gmx.com) ---
Please request support from RedHat.

Vendor kernels are not supported here.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 220669] Drive issues cause system and coredump and reboot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=220669

--- Comment #4 from Artem S. Tashkinov (aros@gmx.com) ---
> [  998.356008] sd 0:0:21:0: [sdv] tag#415 FAILED Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE cmd_age=3s
> [  998.365528] sd 0:0:21:0: [sdv] tag#415 Sense Key : Medium Error [current]
> [descriptor]
> [  998.373544] sd 0:0:21:0: [sdv] tag#415 Add. Sense: Unrecovered read error
> [  998.380347] sd 0:0:21:0: [sdv] tag#415 CDB: Read(16) 88 00 00 00 00 05 01
> 1f 6d 90 00 00 00 08 00 00
> [  998.389524] blk_update_request: critical medium error, dev sdv, sector
> 21493673360 op 0x0:(READ) flags 0x1000 phys_seg 1 prio class 0

You may need to replace your failing hardware first.

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 220669] Drive issues cause system and coredump and reboot

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=220669

--- Comment #5 from Brendan Shephard (bshephar@bne-home.net) ---
(In reply to Dave Chinner from comment #2)
> On Thu, Oct 16, 2025 at 12:10:43AM +0000, bugzilla-daemon@kernel.org wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=220669
> > 
> >             Bug ID: 220669
> >            Summary: Drive issues cause system and coredump and reboot
> >            Product: File System
> >            Version: 2.5
> >           Hardware: All
> >                 OS: Linux
> >             Status: NEW
> >           Severity: normal
> >           Priority: P3
> >          Component: XFS
> >           Assignee: filesystem_xfs@kernel-bugs.kernel.org
> >           Reporter: bshephar@bne-home.net
> >         Regression: No
> > 
> > We have a number of drives across our clusters that don't present as
> failing,
> > but seem to have read errors that cause the system to coredump and reboot.
> > Failing drives is obviously not the fault of XFS, but my expectation would
> be
> > that it doesn't completely cause the system to hang and need to reboot.
> 
> Fixed upstream a couple of months back by commit ae668cd567a6 ("xfs:
> do not propagate ENODATA disk errors into xattr code"). You'll need
> to raise a distro bug (Rocky Linux) to get them to back port it for
> you.
> 
> Please close this bz.
> 
> -Dave.

Legend! Thanks for the pointer to the commit. I really appreciate you taking a
look so quickly!

All the best

-- 
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.