Re: [linux-next][mainline/master] [IPR] [Function could be = "__mutex_lock_slowpath(lock)"]OOPs kernel crash while performing IPR test

[Tasmiya Nalatwad <tasmiya@linux.vnet.ibm.com>]

Greetings,

Thank you Waiman Long for your analysis. The issue is seen consistently 
on every build of linux-next and mainline/master

On 8/27/23 21:39, Waiman Long wrote:
>
> On 8/27/23 04:26, Tasmiya Nalatwad wrote:
>> Greetings,
>>
>> [linux-next][mainline/master] [IPR] [Function could be = 
>> "__mutex_lock_slowpath(lock)"]OOPs kernel crash while performing IPR 
>> test
>>
>> --- Traces ---
>>
>> --- Traces ---
>> [65818.211823] Kernel attempted to read user page (380) - exploit 
>> attempt? (uid: 0)
>> [65818.211836] BUG: Kernel NULL pointer dereference on read at 
>> 0x00000380
>> [65818.211840] Faulting instruction address: 0xc000000000f5f2e4
>> [65818.211844] Oops: Kernel access of bad area, sig: 11 [#1]
>> [65818.211846] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=8192 NUMA pSeries
>> [65818.211850] Modules linked in: rpadlpar_io rpaphp nfnetlink 
>> xsk_diag bonding tls rfkill sunrpc ses enclosure scsi_transport_sas 
>> vmx_crypto pseries_rng binfmt_misc ip_tables ext4 mbcache jbd2 
>> dm_service_time sd_mod t10_pi crc64_rocksoft crc64 sg ibmvfc 
>> scsi_transport_fc ibmveth ipr dm_multipath dm_mirror dm_region_hash 
>> dm_log dm_mod fuse
>> [65818.211879] CPU: 16 PID: 613 Comm: kworker/16:3 Kdump: loaded Not 
>> tainted 6.5.0-rc7-next-20230824-auto #1
>> [65818.211883] Hardware name: IBM,9080-HEX POWER10 (raw) 0x800200 
>> 0xf000006 of:IBM,FW1030.30 (NH1030_062) hv:phyp pSeries
>> [65818.211887] Workqueue: events sg_remove_sfp_usercontext [sg]
>> [65818.211894] NIP:  c000000000f5f2e4 LR: c000000000f5f2d8 CTR: 
>> c00000000032df70
>> [65818.211897] REGS: c0000000081c7a10 TRAP: 0300   Not tainted 
>> (6.5.0-rc7-next-20230824-auto)
>> [65818.211900] MSR:  800000000280b033 
>> <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  CR: 28000882  XER: 20040000
>> [65818.211909] CFAR: c000000000f5b0a4 DAR: 0000000000000380 DSISR: 
>> 40000000 IRQMASK: 0
>> [65818.211909] GPR00: c000000000f5f2d8 c0000000081c7cb0 
>> c000000001451300 0000000000000000
>> [65818.211909] GPR04: 00000000000000c0 00000000c0000000 
>> c000000006c5a298 98a2c506000000c0
>> [65818.211909] GPR08: c00000006408ab00 c0000000022a3515 
>> 0000000000000000 c008000000327d60
>> [65818.211909] GPR12: c00000000032df70 c000000c1bc93f00 
>> c000000000197cc8 c000000008797500
>> [65818.211909] GPR16: 0000000000000000 0000000000000000 
>> 0000000000000000 c000000003071ab0
>> [65818.211909] GPR20: c000000003494c05 c000000c11340040 
>> 0000000000000000 c0000000b9bb4030
>> [65818.211909] GPR24: c0000000b9bb4000 c00000005e8627c0 
>> 0000000000000000 c000000c19b91e00
>> [65818.211909] GPR28: c0000000b9bb5328 c00000005e8627c0 
>> 0000000000000380 0000000000000380
>> [65818.211946] NIP [c000000000f5f2e4] mutex_lock+0x34/0x90
>> [65818.211953] LR [c000000000f5f2d8] mutex_lock+0x28/0x90
>> [65818.211957] Call Trace:
>> [65818.211959] [c0000000081c7cb0] [c000000000f5f2d8] 
>> mutex_lock+0x28/0x90 (unreliable)
>> [65818.211966] [c0000000081c7ce0] [c00000000032df9c] 
>> blk_trace_remove+0x2c/0x80
>> [65818.211971] [c0000000081c7d10] [c0080000003205fc] 
>> sg_device_destroy+0x44/0x110 [sg]
>> [65818.211976] [c0000000081c7d90] [c008000000322988] 
>> sg_remove_sfp_usercontext+0x1d0/0x2c0 [sg]
>> [65818.211981] [c0000000081c7e40] [c000000000188010] 
>> process_scheduled_works+0x230/0x4f0
>> [65818.211987] [c0000000081c7f10] [c00000000018b044] 
>> worker_thread+0x1e4/0x500
>> [65818.211992] [c0000000081c7f90] [c000000000197df8] kthread+0x138/0x140
>> [65818.211996] [c0000000081c7fe0] [c00000000000df98] 
>> start_kernel_thread+0x14/0x18 
>
> The panic happens when a device is being removed. Maybe it is a 
> use-after-free problem. The mutex lock itself seems to be in an area 
> that is no longer accessible. It is not a problem in the locking code 
> itself.
>
> Cheers,
> Longman
>
-- 
Regards,
Tasmiya Nalatwad
IBM Linux Technology Center