Re: Question on xfs related kernel panic

[Jianan Wang <wangjianan.zju@gmail.com>]

Another note is that during the kernel panic and reboot situation, system still has plenty of free memory. More detailed memory stats harvested from the monitoring system could be found as the following:

Active_anon    413437952
Active_bytes    71427657728
Active_file    71014219776
AnonHugePages_bytes    0
AnonPages_bytes    8740204544
Bounce_bytes    0
Buffers_bytes    72273920
Cached_bytes    181501706240
CommitLimit_bytes    270333632512
Committed_AS    32973795328
DirectMap1G_bytes    9663676416
DirectMap2M_bytes    41498443776
DirectMap4k_bytes    500373483520
Dirty_bytes    32768
FileHugePages_bytes    0
FilePmdMapped_bytes    0
HardwareCorrupted_bytes    0
HugePages_Free    0
HugePages_Rsvd    0
HugePages_Surp    0
HugePages_Total    0
Hugepagesize_bytes    2097152
Hugetlb_bytes    0
Inactive_anon    9190891520
Inactive_bytes    119080124416
Inactive_file    109889232896
KReclaimable_bytes    35071422464
KernelStack_bytes    66338816
Mapped_bytes    3630841856
MemAvailable_bytes    348949958656
MemFree_bytes    136491732992
MemTotal_bytes    540667269120
Mlocked_bytes    19243008
NFS_Unstable    0
PageTables_bytes    72445952
Percpu_bytes    253624320
SReclaimable_bytes    35071422464
SUnreclaim_bytes    9349984256
ShmemHugePages_bytes    0
ShmemPmdMapped_bytes    0
Shmem_bytes    741265408
Slab_bytes    44421406720
SwapCached_bytes    0
SwapFree_bytes    0
SwapTotal_bytes    0
Unevictable_bytes    19243008
VmallocChunk_bytes    0
VmallocTotal_bytes    35184372087808
VmallocUsed_bytes    739246080
WritebackTmp_bytes    0
Writeback_bytes    0

On 11/10/23 00:14, Jianan Wang wrote:
> Hi all,
>
> I have a question regarding a kernel panic leading to our server reboot issue, which has its stack-trace like the following (copied from /var/lib/systemd/pstore/*):
>
> <4>[888969.888666] general protection fault, probably for non-canonical address 0xbf5bc9c369fd38ba: 0000 [#1] SMP PTI
> <4>[888969.891355] CPU: 47 PID: 2662145 Comm: find Tainted: P           OE     5.15.0-46-generic #49~20.04.1-Ubuntu
> <4>[888969.894004] Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.8b 01/17/2023
> <4>[888969.896608] RIP: 0010:__kmalloc+0xfc/0x4b0
> <4>[888969.899170] Code: ca 2b ad 56 49 8b 50 08 49 83 78 10 00 4d 8b 30 0f 84 67 03 00 00 4d 85 f6 0f 84 5e 03 00 00 41 8b 45 28 49 8b 7d 00 4c 01 f0 <48> 8b 18 48 89 c1 49 33 9d b8 00 00 00 4c 89 f0 48 0f c9 48 31 cb
> <4>[888969.904329] RSP: 0018:ffffba69b18a78c0 EFLAGS: 00010282
> <4>[888969.906872] RAX: bf5bc9c369fd38ba RBX: 0000000000002c40 RCX: ffffffffc4d3ea92
> <4>[888969.909420] RDX: 0000000004d3b836 RSI: 0000000000002c40 RDI: 00000000000350a0
> <4>[888969.911952] RBP: ffffba69b18a7900 R08: ffff979effef50a0 R09: 000000000000002c
> <4>[888969.914471] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
> <4>[888969.916976] R13: ffff976080042500 R14: bf5bc9c369fd389a R15: ffffffffc4d80b0e
> <4>[888969.919594] FS:  00007fdbf10dd800(0000) GS:ffff979effec0000(0000) knlGS:0000000000000000
> <4>[888969.922109] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> <4>[888969.924601] CR2: 00007f236f3419f0 CR3: 00000050e6e62001 CR4: 00000000007706e0
> <4>[888969.927099] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> <4>[888969.929579] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> <4>[888969.932029] PKRU: 55555554
> <4>[888969.934445] Call Trace:
> <4>[888969.936827]  <TASK>
> <4>[888969.939269]  kmem_alloc+0x6e/0x110 [xfs]
> <4>[888969.941882]  xfs_init_local_fork+0x72/0xf0 [xfs]
> <4>[888969.944418]  xfs_iformat_local+0xac/0x180 [xfs]
> <4>[888969.946921]  xfs_iformat_data_fork+0x105/0x130 [xfs]
> <4>[888969.949405]  xfs_inode_from_disk+0x2be/0x470 [xfs]
> <4>[888969.951869]  xfs_iget+0x334/0xbd0 [xfs]
> <4>[888969.954319]  ? kvfree+0x2c/0x40
> <4>[888969.956529]  xfs_lookup+0xd2/0x100 [xfs]
> <4>[888969.958930]  xfs_vn_lookup+0x76/0xb0 [xfs]
> <4>[888969.961310]  __lookup_slow+0x85/0x150
> <4>[888969.963443]  walk_component+0x145/0x1c0
> <4>[888969.965637]  ? __fdget_raw+0x10/0x20
> <4>[888969.967747]  ? path_init+0x1e5/0x390
> <4>[888969.969888]  path_lookupat.isra.0+0x6e/0x150
> <4>[888969.971927]  filename_lookup+0xcf/0x1a0
> <4>[888969.973943]  ? __check_object_size+0x14f/0x160
> <4>[888969.975937]  ? strncpy_from_user+0x44/0x160
> <4>[888969.977879]  ? getname_flags+0x6f/0x1f0
> <4>[888969.979769]  user_path_at_empty+0x3f/0x60
> <4>[888969.981604]  vfs_statx+0x73/0x110
> <4>[888969.983390]  __do_sys_newfstatat+0x36/0x70
> <4>[888969.985125]  ? alloc_fd+0x58/0x190
> <4>[888969.986806]  ? f_dupfd+0x4b/0x70
> <4>[888969.988513]  ? do_fcntl+0x3af/0x5b0
> <4>[888969.990090]  __x64_sys_newfstatat+0x1e/0x30
> <4>[888969.991649]  do_syscall_64+0x59/0xc0
> <4>[888969.993146]  ? syscall_exit_to_user_mode+0x27/0x50
> <4>[888969.994611]  ? do_syscall_64+0x69/0xc0
> <4>[888969.996020]  ? exit_to_user_mode_prepare+0x3d/0x1c0
> <4>[888969.997404]  ? filp_close+0x60/0x70
> <4>[888969.998752]  ? syscall_exit_to_user_mode+0x27/0x50
> <4>[888970.000084]  ? __x64_sys_close+0x12/0x50
> <4>[888970.001371]  ? do_syscall_64+0x69/0xc0
> <4>[888970.002605]  ? do_syscall_64+0x69/0xc0
> <4>[888970.003793]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
>
> Our xfs version, config, OS and kernel version are the following:
>
> Linux$ xfs_info -V /data/
> xfs_info version 5.9.0
>
> Linux$ xfs_info /data
> meta-data=/dev/md127p1           isize=512    agcount=32, agsize=117206400 blks
>          =                       sectsz=4096  attr=2, projid32bit=1
>          =                       crc=1        finobt=1, sparse=1, rmapbt=0
>          =                       reflink=1
> data     =                       bsize=4096   blocks=3750604800, imaxpct=5
>          =                       sunit=128    swidth=512 blks
> naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
> log      =internal log           bsize=4096   blocks=521728, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
>
> Linux$ cat /etc/*-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=20.04
> DISTRIB_CODENAME=focal
> DISTRIB_DESCRIPTION="Ubuntu-Server 20.04.6 2023.05.30 (Cubic 2023-05-30 13:13)"
> NAME="Ubuntu"
> VERSION="20.04.6 LTS (Focal Fossa)"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu-Server 20.04.6 2023.05.30 (Cubic 2023-05-30 13:13)"
> VERSION_ID="20.04"
> HOME_URL="https://www.ubuntu.com/"
> SUPPORT_URL="https://help.ubuntu.com/"
> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> VERSION_CODENAME=focal
> UBUNTU_CODENAME=focal
>
> Linux$ uname -a
> Linux abc-server-001 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
>
> It would be great if any insight could be provided on whether this is a known issue or how we could troubleshoot further.
>
> Best Regards.
>
> Jianan
>