Re: Question on xfs related kernel panic

[Jianan Wang <wangjianan.zju@gmail.com>]

Hi Darrick,

Thanks for your response. I will open a case to Ubuntu on this issue. However, can you give me a hint on what could be wrong? Failed to kmalloc seems to be a pretty severe issue, and is that related to any kind of kernel memory corruption by certain kernel modules or so?

Thanks.
Jianan.

On 11/10/23 11:34, Darrick J. Wong wrote:
> On Fri, Nov 10, 2023 at 12:14:45AM -0800, Jianan Wang wrote:
>> Hi all,
>>
>> I have a question regarding a kernel panic leading to our server reboot issue, which has its stack-trace like the following (copied from /var/lib/systemd/pstore/*):
>>
>> <4>[888969.888666] general protection fault, probably for non-canonical address 0xbf5bc9c369fd38ba: 0000 [#1] SMP PTI
>> <4>[888969.891355] CPU: 47 PID: 2662145 Comm: find Tainted: P           OE     5.15.0-46-generic #49~20.04.1-Ubuntu
> Please open a support case with your vendor for this issue with their
> kernel.
>
> --D
>
>> <4>[888969.894004] Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.8b 01/17/2023
>> <4>[888969.896608] RIP: 0010:__kmalloc+0xfc/0x4b0
>> <4>[888969.899170] Code: ca 2b ad 56 49 8b 50 08 49 83 78 10 00 4d 8b 30 0f 84 67 03 00 00 4d 85 f6 0f 84 5e 03 00 00 41 8b 45 28 49 8b 7d 00 4c 01 f0 <48> 8b 18 48 89 c1 49 33 9d b8 00 00 00 4c 89 f0 48 0f c9 48 31 cb
>> <4>[888969.904329] RSP: 0018:ffffba69b18a78c0 EFLAGS: 00010282
>> <4>[888969.906872] RAX: bf5bc9c369fd38ba RBX: 0000000000002c40 RCX: ffffffffc4d3ea92
>> <4>[888969.909420] RDX: 0000000004d3b836 RSI: 0000000000002c40 RDI: 00000000000350a0
>> <4>[888969.911952] RBP: ffffba69b18a7900 R08: ffff979effef50a0 R09: 000000000000002c
>> <4>[888969.914471] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
>> <4>[888969.916976] R13: ffff976080042500 R14: bf5bc9c369fd389a R15: ffffffffc4d80b0e
>> <4>[888969.919594] FS:  00007fdbf10dd800(0000) GS:ffff979effec0000(0000) knlGS:0000000000000000
>> <4>[888969.922109] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> <4>[888969.924601] CR2: 00007f236f3419f0 CR3: 00000050e6e62001 CR4: 00000000007706e0
>> <4>[888969.927099] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> <4>[888969.929579] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> <4>[888969.932029] PKRU: 55555554
>> <4>[888969.934445] Call Trace:
>> <4>[888969.936827]  <TASK>
>> <4>[888969.939269]  kmem_alloc+0x6e/0x110 [xfs]
>> <4>[888969.941882]  xfs_init_local_fork+0x72/0xf0 [xfs]
>> <4>[888969.944418]  xfs_iformat_local+0xac/0x180 [xfs]
>> <4>[888969.946921]  xfs_iformat_data_fork+0x105/0x130 [xfs]
>> <4>[888969.949405]  xfs_inode_from_disk+0x2be/0x470 [xfs]
>> <4>[888969.951869]  xfs_iget+0x334/0xbd0 [xfs]
>> <4>[888969.954319]  ? kvfree+0x2c/0x40
>> <4>[888969.956529]  xfs_lookup+0xd2/0x100 [xfs]
>> <4>[888969.958930]  xfs_vn_lookup+0x76/0xb0 [xfs]
>> <4>[888969.961310]  __lookup_slow+0x85/0x150
>> <4>[888969.963443]  walk_component+0x145/0x1c0
>> <4>[888969.965637]  ? __fdget_raw+0x10/0x20
>> <4>[888969.967747]  ? path_init+0x1e5/0x390
>> <4>[888969.969888]  path_lookupat.isra.0+0x6e/0x150
>> <4>[888969.971927]  filename_lookup+0xcf/0x1a0
>> <4>[888969.973943]  ? __check_object_size+0x14f/0x160
>> <4>[888969.975937]  ? strncpy_from_user+0x44/0x160
>> <4>[888969.977879]  ? getname_flags+0x6f/0x1f0
>> <4>[888969.979769]  user_path_at_empty+0x3f/0x60
>> <4>[888969.981604]  vfs_statx+0x73/0x110
>> <4>[888969.983390]  __do_sys_newfstatat+0x36/0x70
>> <4>[888969.985125]  ? alloc_fd+0x58/0x190
>> <4>[888969.986806]  ? f_dupfd+0x4b/0x70
>> <4>[888969.988513]  ? do_fcntl+0x3af/0x5b0
>> <4>[888969.990090]  __x64_sys_newfstatat+0x1e/0x30
>> <4>[888969.991649]  do_syscall_64+0x59/0xc0
>> <4>[888969.993146]  ? syscall_exit_to_user_mode+0x27/0x50
>> <4>[888969.994611]  ? do_syscall_64+0x69/0xc0
>> <4>[888969.996020]  ? exit_to_user_mode_prepare+0x3d/0x1c0
>> <4>[888969.997404]  ? filp_close+0x60/0x70
>> <4>[888969.998752]  ? syscall_exit_to_user_mode+0x27/0x50
>> <4>[888970.000084]  ? __x64_sys_close+0x12/0x50
>> <4>[888970.001371]  ? do_syscall_64+0x69/0xc0
>> <4>[888970.002605]  ? do_syscall_64+0x69/0xc0
>> <4>[888970.003793]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
>>
>> Our xfs version, config, OS and kernel version are the following:
>>
>> Linux$ xfs_info -V /data/
>> xfs_info version 5.9.0
>>
>> Linux$ xfs_info /data
>> meta-data=/dev/md127p1           isize=512    agcount=32, agsize=117206400 blks
>>          =                       sectsz=4096  attr=2, projid32bit=1
>>          =                       crc=1        finobt=1, sparse=1, rmapbt=0
>>          =                       reflink=1
>> data     =                       bsize=4096   blocks=3750604800, imaxpct=5
>>          =                       sunit=128    swidth=512 blks
>> naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
>> log      =internal log           bsize=4096   blocks=521728, version=2
>>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
>> realtime =none                   extsz=4096   blocks=0, rtextents=0
>>
>> Linux$ cat /etc/*-release
>> DISTRIB_ID=Ubuntu
>> DISTRIB_RELEASE=20.04
>> DISTRIB_CODENAME=focal
>> DISTRIB_DESCRIPTION="Ubuntu-Server 20.04.6 2023.05.30 (Cubic 2023-05-30 13:13)"
>> NAME="Ubuntu"
>> VERSION="20.04.6 LTS (Focal Fossa)"
>> ID=ubuntu
>> ID_LIKE=debian
>> PRETTY_NAME="Ubuntu-Server 20.04.6 2023.05.30 (Cubic 2023-05-30 13:13)"
>> VERSION_ID="20.04"
>> HOME_URL="https://www.ubuntu.com/"
>> SUPPORT_URL="https://help.ubuntu.com/"
>> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
>> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
>> VERSION_CODENAME=focal
>> UBUNTU_CODENAME=focal
>>
>> Linux$ uname -a
>> Linux abc-server-001 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
>>
>> It would be great if any insight could be provided on whether this is a known issue or how we could troubleshoot further.
>>
>> Best Regards.
>>
>> Jianan
>>