kernel NULL pointer dereference on hotplug

kernel NULL pointer dereference on hotplug

[Olliver Schinagl]

Hey guys,

I noticed this nasty kernel NULL pointer dereference yesterday on 
6.7.2-arch1-1 (haven't done this in a while) but also today, after 
updating to 6.7.3-arch1-1 it's still there, so dumping the panic here. 
Hopefully it'll be resolved by 6.7.4-arch1-1.

The thunderbolt gbit adapter always worked in the past, so this seems 
like a regression. Anyway, here's the log.

[  224.924021] BUG: kernel NULL pointer dereference, address: 
0000000000000020
[  224.924026] #PF: supervisor read access in kernel mode
[  224.924028] #PF: error_code(0x0000) - not-present page
[  224.924030] PGD 0 P4D 0
[  224.924032] Oops: 0000 [#1] PREEMPT SMP NOPTI
[  224.924035] CPU: 2 PID: 282 Comm: kworker/u8:9 Tainted: G 
OE      6.7.3-arch1-1 #1 b8291227ebee24f0bec9b3471a94151938512264
[  224.924038] Hardware name: Apple Inc. 
MacBookPro12,1/Mac-E43C1C25D4880AD6, BIOS 481.0.0.0.0 01/12/2023
[  224.924039] Workqueue: thunderbolt0 tb_handle_hotplug [thunderbolt]
[  224.924060] RIP: 0010:tb_port_do_update_credits+0x1b/0x130 [thunderbolt]
[  224.924079] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 
44 00 00 55 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 
c0 <48> 8b 47 20 c7 44 24 0c 00 00 00 00 80 b8 78 03 00 00 00 0f 85 ec
[  224.924081] RSP: 0018:ffffae7e4195bd70 EFLAGS: 00010246
[  224.924083] RAX: 0000000000000000 RBX: ffff99392dfa1c00 RCX: 
0000000003c00000
[  224.924084] RDX: 0000000000000000 RSI: ffffae7e4195bbc0 RDI: 
0000000000000000
[  224.924086] RBP: 0000000000000000 R08: 0000000003c00000 R09: 
0000000002080004
[  224.924087] R10: 0000000000000002 R11: 0000000000000000 R12: 
ffff99392dfa04c8
[  224.924088] R13: ffff99388c5ef258 R14: 0000000000000000 R15: 
ffff99392dfa1c00
[  224.924089] FS:  0000000000000000(0000) GS:ffff993bded00000(0000) 
knlGS:0000000000000000
[  224.924091] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  224.924092] CR2: 0000000000000020 CR3: 0000000190620003 CR4: 
00000000003706f0
[  224.924093] Call Trace:
[  224.924095]  <TASK>
[  224.924097]  ? __die+0x23/0x70
[  224.924101]  ? page_fault_oops+0x171/0x4e0
[  224.924105]  ? exc_page_fault+0x7f/0x180
[  224.924109]  ? asm_exc_page_fault+0x26/0x30
[  224.924112]  ? tb_port_do_update_credits+0x1b/0x130 [thunderbolt 
11ca615e403f4fd9365fcc050909dff1175dc0c5]
[  224.924131]  ? tb_switch_update_link_attributes+0x83/0xd0 
[thunderbolt 11ca615e403f4fd9365fcc050909dff1175dc0c5]
[  224.924150]  tb_switch_add+0x7a2/0xfe0 [thunderbolt 
11ca615e403f4fd9365fcc050909dff1175dc0c5]
[  224.924170]  tb_scan_port+0x236/0x6f0 [thunderbolt 
11ca615e403f4fd9365fcc050909dff1175dc0c5]
[  224.924188]  tb_handle_hotplug+0x6db/0x900 [thunderbolt 
11ca615e403f4fd9365fcc050909dff1175dc0c5]
[  224.924206]  process_one_work+0x171/0x340
[  224.924209]  worker_thread+0x27b/0x3a0
[  224.924211]  ? __pfx_worker_thread+0x10/0x10
[  224.924213]  kthread+0xe5/0x120
[  224.924216]  ? __pfx_kthread+0x10/0x10
[  224.924219]  ret_from_fork+0x31/0x50
[  224.924221]  ? __pfx_kthread+0x10/0x10
[  224.924224]  ret_from_fork_asm+0x1b/0x30
[  224.924227]  </TASK>
[  224.924227] Modules linked in: facetimehd(OE) videobuf2_dma_sg 
videobuf2_memops videobuf2_v4l2 videodev videobuf2_common mc rfcomm 
xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_policy iptable_mangle 
xt_mark xt_bpf xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink 
iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 
xt_addrtype iptable_filter br_netfilter bridge stp llc overlay cmac 
algif_hash algif_skcipher af_alg bnep snd_hda_codec_hdmi 
snd_hda_codec_cirrus snd_hda_codec_generic ledtrig_audio intel_rapl_msr 
intel_rapl_common snd_hda_intel x86_pkg_temp_thermal intel_powerclamp 
snd_intel_dspcfg coretemp snd_intel_sdw_acpi brcmfmac_wcc snd_hda_codec 
kvm_intel i915 spi_nor snd_hda_core btusb drm_buddy kvm brcmfmac btrtl 
mtd i2c_algo_bit snd_hwdep btintel brcmutil btbcm spi_intel_platform 
irqbypass ttm btmtk mei_pxp mei_hdcp cfg80211 iTCO_wdt spi_intel snd_pcm 
rapl bluetooth intel_pmc_bxt drm_display_helper snd_timer intel_cstate 
iTCO_vendor_support joydev acpi_als mei_me industrialio_triggered_buffer
[  224.924275]  applesmc mmc_core snd ecdh_generic cec kfifo_buf 
i2c_i801 intel_uncore thunderbolt pcspkr sbs intel_pch_thermal mei 
lpc_ich rfkill soundcore intel_gtt i2c_smbus industrialio bcm5974 sbshc 
video wmi mousedev apple_mfi_fastcharge mac_hid sg crypto_user fuse loop 
nfnetlink ip_tables x_tables uas usb_storage usbhid crct10dif_pclmul 
crc32_pclmul polyval_clmulni polyval_generic gf128mul 
ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 applespi 
xhci_pci crc16 spi_pxa2xx_pci xhci_pci_renesas spi_pxa2xx_platform 
dw_dmac hid_apple tg3 libphy vfat fat btrfs blake2b_generic libcrc32c 
crc32c_generic crc32c_intel xor raid6_pq dm_crypt aesni_intel 
crypto_simd cryptd cbc encrypted_keys trusted asn1_encoder tee dm_mod
[  224.924313] CR2: 0000000000000020
[  224.924315] ---[ end trace 0000000000000000 ]---
[  224.924316] RIP: 0010:tb_port_do_update_credits+0x1b/0x130 [thunderbolt]
[  224.924335] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 
44 00 00 55 53 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 44 24 10 31 
c0 <48> 8b 47 20 c7 44 24 0c 00 00 00 00 80 b8 78 03 00 00 00 0f 85 ec
[  224.924336] RSP: 0018:ffffae7e4195bd70 EFLAGS: 00010246
[  224.924338] RAX: 0000000000000000 RBX: ffff99392dfa1c00 RCX: 
0000000003c00000
[  224.924339] RDX: 0000000000000000 RSI: ffffae7e4195bbc0 RDI: 
0000000000000000
[  224.924340] RBP: 0000000000000000 R08: 0000000003c00000 R09: 
0000000002080004
[  224.924341] R10: 0000000000000002 R11: 0000000000000000 R12: 
ffff99392dfa04c8
[  224.924342] R13: ffff99388c5ef258 R14: 0000000000000000 R15: 
ffff99392dfa1c00
[  224.924343] FS:  0000000000000000(0000) GS:ffff993bded00000(0000) 
knlGS:0000000000000000
[  224.924344] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  224.924346] CR2: 0000000000000020 CR3: 0000000190620003 CR4: 
00000000003706f0
[  224.924347] note: kworker/u8:9[282] exited with irqs disabled


Thanks,

Olliver

Re: kernel NULL pointer dereference on hotplug

[Mika Westerberg]

Hi,

On Fri, Feb 02, 2024 at 05:47:01PM +0100, Olliver Schinagl wrote:
> Hey guys,
> 
> I noticed this nasty kernel NULL pointer dereference yesterday on
> 6.7.2-arch1-1 (haven't done this in a while) but also today, after updating
> to 6.7.3-arch1-1 it's still there, so dumping the panic here. Hopefully
> it'll be resolved by 6.7.4-arch1-1.

Thanks for the report.

> The thunderbolt gbit adapter always worked in the past, so this seems like a
> regression. Anyway, here's the log.

Can you try to bisect this, preferably using the mainline kernel? Let me
know if you need instructions how to do this.

Re: kernel NULL pointer dereference on hotplug

[Linux regression tracking (Thorsten Leemhuis)]

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

On 04.02.24 07:40, Mika Westerberg wrote:
> On Fri, Feb 02, 2024 at 05:47:01PM +0100, Olliver Schinagl wrote:
>>
>> I noticed this nasty kernel NULL pointer dereference yesterday on
>> 6.7.2-arch1-1 (haven't done this in a while) but also today, after updating
>> to 6.7.3-arch1-1 it's still there, so dumping the panic here. Hopefully
>> it'll be resolved by 6.7.4-arch1-1.
> 
> Thanks for the report.
> 
>> The thunderbolt gbit adapter always worked in the past, so this seems like a
>> regression. Anyway, here's the log.
> 
> Can you try to bisect this, preferably using the mainline kernel? Let me
> know if you need instructions how to do this.

Olliver, did you try a bisection?

BTW, I'm working on a document for the Linux kernel sources that
explains a Linux kernel bisection. Might be helpful:

https://www.leemhuis.info/files/misc/How%20to%20bisect%20a%20Linux%20kernel%20regression%20%e2%80%94%20The%20Linux%20Kernel%20documentation.html

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: kernel NULL pointer dereference on hotplug

[Thorsten Leemhuis]

On 07.02.24 07:13, Linux regression tracking (Thorsten Leemhuis) wrote:
> 
> On 04.02.24 07:40, Mika Westerberg wrote:
>> On Fri, Feb 02, 2024 at 05:47:01PM +0100, Olliver Schinagl wrote:
>>>
>>> I noticed this nasty kernel NULL pointer dereference yesterday on
>>> 6.7.2-arch1-1 (haven't done this in a while) but also today, after updating
>>> to 6.7.3-arch1-1 it's still there, so dumping the panic here. Hopefully
>>> it'll be resolved by 6.7.4-arch1-1.
>>
>> Thanks for the report.
>>
>>> The thunderbolt gbit adapter always worked in the past, so this seems like a
>>> regression. Anyway, here's the log.

FWIW, Leon Weiß (BCCed) just posted another NULL pointer deference issue
in 6.7 and found the culprit; for details see:
https://lore.kernel.org/all/38c253ea42072cc825dc969ac4e6b9b600371cc8.camel@ruhr-uni-bochum.de/

Sadly Leon's report lacks a backtrace we had in this thread (
https://lore.kernel.org/all/c24c7882-6254-4e68-8f22-f3e8f65dc84f@schinagl.nl/
), so it might be something totally different. Leon's problem afaics
also happens on unplug while this one on hotplug. But well, I thought I
better quickly mention it here anyway so everyone is aware of it.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: kernel NULL pointer dereference on hotplug

[Mika Westerberg]

Hi,

On Wed, Feb 07, 2024 at 10:12:07AM +0100, Thorsten Leemhuis wrote:
> On 07.02.24 07:13, Linux regression tracking (Thorsten Leemhuis) wrote:
> > 
> > On 04.02.24 07:40, Mika Westerberg wrote:
> >> On Fri, Feb 02, 2024 at 05:47:01PM +0100, Olliver Schinagl wrote:
> >>>
> >>> I noticed this nasty kernel NULL pointer dereference yesterday on
> >>> 6.7.2-arch1-1 (haven't done this in a while) but also today, after updating
> >>> to 6.7.3-arch1-1 it's still there, so dumping the panic here. Hopefully
> >>> it'll be resolved by 6.7.4-arch1-1.
> >>
> >> Thanks for the report.
> >>
> >>> The thunderbolt gbit adapter always worked in the past, so this seems like a
> >>> regression. Anyway, here's the log.
> 
> FWIW, Leon Weiß (BCCed) just posted another NULL pointer deference issue
> in 6.7 and found the culprit; for details see:
> https://lore.kernel.org/all/38c253ea42072cc825dc969ac4e6b9b600371cc8.camel@ruhr-uni-bochum.de/
>
> Sadly Leon's report lacks a backtrace we had in this thread (
> https://lore.kernel.org/all/c24c7882-6254-4e68-8f22-f3e8f65dc84f@schinagl.nl/
> ), so it might be something totally different. Leon's problem afaics
> also happens on unplug while this one on hotplug. But well, I thought I
> better quickly mention it here anyway so everyone is aware of it.

Thanks for the information. 

As you mention, this seems to be unrelated. The one above is about DRM
and graphics, the one Olliver reported is about Thunderbolt (okay there
is Thunderbolt device involved in both but this one seems to be purely
DP thing).