[next-20220225][Oops][ppc] lvm snapshot merge results kernel panics (throtl_pending_timer_fn)

Abdul Haleem
2022-03-02T08:05:04Z

Greeting's

Linux next kernel 5.17.0-rc5-next-20220225 crashed on my power 10 LPAR
when merge lvm snapshot on nvme disk

console logs
------------

fdisk -l /dev/nvme1n1

Disk /dev/nvme1n1: 372.6 GiB, 400088457216 bytes, 97677846 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
vgcreate avocado_vg /dev/nvme1n1
Volume group "avocado_vg" successfully created

lvcreate --name avocado_lv --size 18432.0 avocado_vg -y
Wiping ext2 signature on /dev/avocado_vg/avocado_lv.
Logical volume "avocado_lv" created.

lvdisplay avocado_vg
--- Logical volume ---
LV Path                /dev/avocado_vg/avocado_lv
LV Name                avocado_lv
VG Name                avocado_vg
LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
LV Write Access        read/write
LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02
01:32:19 -0600
LV Status              available
# open                 0
LV Size                18.00 GiB
Current LE             4608
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     8192
Block device           253:0

mkfs.ext2 /dev/avocado_vg/avocado_lv

mke2fs 1.44.6 (5-Mar-2019)
Creating filesystem with 4718592 4k blocks and 1179648 inodes
Filesystem UUID: 5ed3c335-bac2-4b64-a827-222d287af0b2
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
4096000

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

mount /dev/avocado_vg/avocado_lv /mnt

[ 1053.517019] EXT4-fs (dm-0): mounting ext2 file system using the ext4
subsystem
[ 1053.518270] EXT4-fs (dm-0): mounted filesystem without journal. Quota
mode: none.

umount /dev/avocado_vg/avocado_lv

lvdisplay avocado_vg
--- Logical volume ---
LV Path                /dev/avocado_vg/avocado_lv
LV Name                avocado_lv
VG Name                avocado_vg
LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
LV Write Access        read/write
LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02
01:32:19 -0600
LV Status              available
# open                 0
LV Size                18.00 GiB
Current LE             4608
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     8192
Block device           253:0

lvcreate --snapshot --name avocado_sn /dev/avocado_vg/avocado_lv
--ignoreactivationskip --size 18432.0

Logical volume "avocado_sn" created.

vgdisplay avocado_vg

--- Volume group ---
VG Name               avocado_vg
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  4
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                2
Open LV               0
Max PV                0
Cur PV                1
Act PV                1
VG Size               <372.61 GiB
PE Size               4.00 MiB
Total PE              95388
Alloc PE / Size       9216 / 36.00 GiB
Free  PE / Size       86172 / <336.61 GiB
VG UUID               jobi1f-kHw4-2ovw-nR3E-Eml5-tFYR-mJc3WT

lvdisplay avocado_vg
--- Logical volume ---
LV Path                /dev/avocado_vg/avocado_lv
LV Name                avocado_lv
VG Name                avocado_vg
LV UUID nkhkFh-Oofl-GKH1-1055-3B47-0gup-yQtI1s
LV Write Access        read/write
LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02
01:32:19 -0600
LV snapshot status     source of
avocado_sn [active]
LV Status              available
# open                 0
LV Size                18.00 GiB
Current LE             4608
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           253:0

--- Logical volume ---
LV Path                /dev/avocado_vg/avocado_sn
LV Name                avocado_sn
VG Name                avocado_vg
LV UUID QL14R9-aVa9-nD8z-DTyg-87qL-b2yY-O30G5P
LV Write Access        read/write
LV Creation host, time ltc-zz1b-lp4.aus.stglabs.ibm.com, 2022-03-02
01:32:20 -0600
LV snapshot status     active destination for avocado_lv
LV Status              available
# open                 0
LV Size                18.00 GiB
Current LE             4608
COW-table size         18.00 GiB
COW-table LE           4608
Allocated to snapshot  0.00%
Snapshot chunk size    4.00 KiB
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     8192
Block device           253:3

lvconvert --merge --interval 1 /dev/avocado_vg/avocado_sn
Merging of volume avocado_vg/avocado_sn started.
avocado_vg/avocado_lv: Merged: 100.00%
BUG: Unable to handle kernel data access on write at 0x3335323d524f4ab1
Faulting instruction address: 0xc000000000cb3a2c
Oops: Kernel access of bad area, sig: 11 [#1]
LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries
Modules linked in: dm_snapshot dm_bufio rpadlpar_io rpaphp kvm_pr kvm
nf_tables libcrc32c
nfnetlink tcp_diag udp_diag inet_diag unix_diag af_packet_diag
netlink_diag rfkill dm_mod sg pseries_rn
g xts uio_pdrv_genirq vmx_crypto gf128mul uio nfsd auth_rpcgss nfs_acl
lockd grace sunrpc binfmt_misc sc
h_fq_codel ip_tables ext4 mbcache jbd2 sd_mod ibmvscsi ibmvnic
scsi_transport_srp ibmveth nvme nvme_core
t10_pi
CPU: 17 PID: 0 Comm: swapper/17 Kdump: loaded Not tainted
5.17.0-rc5-next-20220225-autote

NIP:  c000000000cb3a2c LR: c0000000006740b4 CTR: c000000000674030
REGS: c0000000034ff390 TRAP: 0380   Not tainted
(5.17.0-rc5-next-20220225-autotest)
MSR:  8000000000009033 <SF,EE,ME,IR,DR,RI,LE>  CR: 82008228  XER: 20040000
CFAR: c0000000006740b0 IRQMASK: 1
GPR00: c000000000220290 c0000000034ff630 c0000000028d3200
3335323d524f4ab1
GPR04: c000000000674030 00000001000126d5 0000000000000200
0000000000000000
GPR08: 0000000000000000 0000000000000000 0000000000000001
0000000000000001
GPR12: 0000000022000222 c00000000ffa6680 0000000000000100
0000000000000010
GPR16: 0000000000000100 0000000000000002 c000000002902108
c0000000020e7880
GPR20: c000000000674030 5deadbeef0000122 c00000077c908668
c0000000b4dce000
GPR24: c0000000acb6a800 c0000000034ff7c8 3335323d524f4a41
0000000000000100
GPR28: c000000000674030 c0000000acb6a880 3335323d524f4ab1
c0000000acb6a830
NIP [c000000000cb3a2c] _raw_spin_lock_irq+0x1c/0xa0
LR [c0000000006740b4] throtl_pending_timer_fn+0x84/0x450
Call Trace:
[c0000000034ff630] [c0000000034ff670] 0xc0000000034ff670 (unreliable)
[c0000000034ff650] [c000000002903a00] jiffies+0x0/0x80
[c0000000034ff710] [c000000000220290] call_timer_fn+0x50/0x200
[c0000000034ff7a0] [c000000000221c30] run_timer_softirq+0x340/0x7c0
[c0000000034ff870] [c000000000cb3cb8] __do_softirq+0x158/0x3e0
[c0000000034ff970] [c00000000014ef78] irq_exit+0x168/0x1b0
[c0000000034ff9a0] [c0000000000271c4] timer_interrupt+0x1a4/0x3e0
[c0000000034ffa00] [c000000000009a08] decrementer_common_virt+0x208/0x210
--- interrupt: 900 at plpar_hcall_norets_notrace+0x18/0x2c
NIP:  c0000000000ece58 LR: c0000000009d58c8 CTR: 0000000000000000
REGS: c0000000034ffa70 TRAP: 0900   Not tainted
(5.17.0-rc5-next-20220225-autotest)
MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR:
22000224  XER: 20040000
CFAR: 0000000000000000 IRQMASK: 0
GPR00: 0000000000000000 c0000000034ffd10 c0000000028d3200
0000000000000000
GPR04: 000000000000ffff 0000000000000000 0000000000000000
0000000000000000
GPR08: 0000000000000000 0000000000000000 800000140fc3fc00
ffffffffffffffff
GPR12: 0000000000000000 c00000000ffa6680 0000000000010000
0000000000000800
GPR16: c0000000016a20e8 c00000077fff1678 ffffffffffffffff
0000000000000001
GPR20: 0000000000000001 0000000000000002 0000000000000010
c00000000282ff88
GPR24: 0000000000000001 000000f59a421dce 0000000000000000
0000000000000000
GPR28: 0000000000000001 0000000000000000 c0000000020e2098
c0000000020e20a0
NIP [c0000000000ece58] plpar_hcall_norets_notrace+0x18/0x2c
LR [c0000000009d58c8] check_and_cede_processor+0x48/0x60
--- interrupt: 900
[c0000000034ffd10] [c0000000034ffd40] 0xc0000000034ffd40 (unreliable)
[c0000000034ffd70] [c0000000009d5d04] dedicated_cede_loop+0x94/0x1a0
[c0000000034ffdc0] [c0000000009d2634] cpuidle_enter_state+0x2d4/0x4e0
[c0000000034ffe20] [c0000000009d28d8] cpuidle_enter+0x48/0x70
[c0000000034ffe60] [c00000000019db74] call_cpuidle+0x44/0x80
[c0000000034ffe80] [c00000000019e150] do_idle+0x340/0x390
[c0000000034fff00] [c00000000019e3d4] cpu_startup_entry+0x34/0x40
[c0000000034fff30] [c0000000000607e0] start_secondary+0x290/0x2b0
[c0000000034fff90] [c00000000000d154] start_secondary_prolog+0x10/0x14
Instruction dump:
7c0803a6 4bffffa4 60000000 60000000 60000000 3c4c01c2 3842f7f0 f821ffe1
39200001 992d0932 39200000 39400001 <7c801829> 7c044800 40c20010 7d40192d
---[ end trace 0000000000000000 ]---

--
Regard's

Abdul Haleem
IBM Linux Technology Center