System hang with latest kernel v6.16.0-rc1 (rc2 & rc3)

System hang with latest kernel v6.16.0-rc1 (rc2 & rc3)

[Himanshu Madhani]

Hi Folks, 

We are seeing kernel hang while booting after new 6.16-rc1 kernel is installed.

Here’s stack track that shows up 

[  297.656683] systemd-shutdown[1]: Rebooting with kexec.
[  513.790993] INFO: task kexec:19038 blocked for more than 122 seconds.
[  513.868087]       Not tainted 6.16.0-rc1.master.20250611.ol9.x86_64 #1
[  513.946210] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  514.039923] task:kexec           state:D stack:0     pid:19038 tgid:19038 ppid:1      task_flags:0x400100 flags:0x00004002
[  514.172122] Call Trace:
[  514.201356]  <TASK>
[  514.226438]  __schedule+0x2d1/0x730
[  514.268161]  schedule+0x27/0x80
[  514.305717]  schedule_preempt_disabled+0x15/0x30
[  514.360954]  __mutex_lock.constprop.0+0x4be/0x8a0
[  514.417232]  msi_domain_get_virq+0xcc/0x110
[  514.467279]  pci_msix_write_tph_tag+0x3c/0x100
[  514.520441]  pcie_tph_set_st_entry+0x125/0x1d0
[  514.573605]  bnxt_irq_affinity_release+0x35/0x50 [bnxt_en]
[  514.639258]  irq_set_affinity_notifier+0xdd/0x130
[  514.695534]  bnxt_free_irq+0x6e/0x110 [bnxt_en]
[  514.749746]  __bnxt_close_nic.isra.0+0x1eb/0x220 [bnxt_en]
[  514.815404]  bnxt_close+0x3a/0x100 [bnxt_en]
[  514.866498]  __dev_close_many+0xab/0x220
[  514.913423]  __dev_change_flags+0x102/0x240
[  514.963464]  netif_change_flags+0x26/0x70
[  515.011424]  dev_change_flags+0x40/0xc0
[  515.057304]  devinet_ioctl+0x3aa/0x7a0
[  515.102142]  inet_ioctl+0x1d3/0x1f0
[  515.143863]  sock_do_ioctl+0x7a/0x140
[  515.187667]  __x64_sys_ioctl+0x9b/0x100
[  515.233545]  ? syscall_trace_enter+0x10c/0x1d0
[  515.286704]  do_syscall_64+0x84/0x940
[  515.330502]  ? refill_obj_stock+0x143/0x240
[  515.380543]  ? __dentry_kill+0x12e/0x190
[  515.427459]  ? __memcg_slab_free_hook+0xf4/0x150
[  515.482698]  ? __x64_sys_close+0x3d/0x80
[  515.529616]  ? kmem_cache_free+0x3fe/0x460
[  515.578614]  ? syscall_exit_work+0x118/0x150
[  515.629695]  ? arch_exit_to_user_mode_prepare.isra.0+0x9/0xb0
[  515.698453]  ? do_syscall_64+0xba/0x940
[  515.744330]  ? mod_memcg_lruvec_state+0x1a2/0x1f0
[  515.800608]  ? __lruvec_stat_mod_folio+0x83/0xd0
[  515.855843]  ? __folio_mod_stat+0x26/0x80
[  515.903801]  ? set_ptes.isra.0+0x36/0x90
[  515.950723]  ? do_anonymous_page+0x103/0x4b0
[  516.001802]  ? __handle_mm_fault+0x394/0x6f0
[  516.052886]  ? count_memcg_events+0x15a/0x1a0
[  516.105008]  ? handle_mm_fault+0x24a/0x350
[  516.154003]  ? do_user_addr_fault+0x221/0x690
[  516.206122]  ? arch_exit_to_user_mode_prepare.isra.0+0x9/0xb0
[  516.274887]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  516.335330] RIP: 0033:0x7fc96e903bcb
[  516.378086] RSP: 002b:00007ffcc7f78518 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
[  516.468683] RAX: ffffffffffffffda RBX: 000055dc432d8f80 RCX: 00007fc96e903bcb
[  516.554080] RDX: 00007ffcc7f78680 RSI: 0000000000008914 RDI: 0000000000000003
[  516.639482] RBP: 0000000000000000 R08: 0000000000000007 R09: 0000000000000007
[  516.724882] R10: 000000000000005e R11: 0000000000000202 R12: 000055dc095468dd
[  516.810278] R13: 000055dc095468e4 R14: 00007ffcc7f78680 R15: 000055dc432d9020
[  516.895676]  </TASK>
[  516.921808] INFO: task kexec:19038 is blocked on a mutex likely owned by task kexec:19038.
[  517.020728] task:kexec           state:D stack:0     pid:19038 tgid:19038 ppid:1      task_flags:0x400100 flags:0x00004002


Git-bisect point to this merge commit 

commit 6376c0770656f3bdf7f411faf068371b6932aeca
Merge: 5e8bbb2caa4e 29857e6f4e30
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Tue May 27 09:01:26 2025 -0700

    Merge tag 'timers-clocksource-2025-05-25' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
    
    Pull clocksource updates from Thomas Gleixner:
     "Updates for clocksource/clockevent drivers:
    
       - The final conversion of text formatted device tree binding to
         schemas
    
       - A new driver fot the System Timer Module on S32G NXP SoCs
    
       - A new driver fot the Econet HPT timer
    
       - The usual improvements and device tree binding updates"
    
    * tag 'timers-clocksource-2025-05-25' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (31 commits)
      clocksource/drivers/renesas-ostm: Unconditionally enable reprobe support
      dt-bindings: timer: renesas,ostm: Document RZ/V2N (R9A09G056) support
      dt-bindings: timer: Convert marvell,armada-370-timer to DT schema
      dt-bindings: timer: Convert ti,keystone-timer to DT schema
      dt-bindings: timer: Convert st,spear-timer to DT schema
      dt-bindings: timer: Convert socionext,milbeaut-timer to DT schema
      dt-bindings: timer: Convert snps,arc-timer to DT schema
      dt-bindings: timer: Convert snps,archs-rtc to DT schema
      dt-bindings: timer: Convert snps,archs-gfrc to DT schema
      dt-bindings: timer: Convert lsi,zevio-timer to DT schema
      dt-bindings: timer: Convert jcore,pit to DT schema
      dt-bindings: timer: Convert img,pistachio-gptimer to DT schema
      dt-bindings: timer: Convert ezchip,nps400-timer to DT schema
      dt-bindings: timer: Convert cirrus,clps711x-timer to DT schema
      dt-bindings: timer: Convert altr,timer-1.0 to DT schema
      dt-bindings: timer: Add ESWIN EIC7700 CLINT
      clocksource/drivers: Add EcoNet Timer HPT driver
      dt-bindings: timer: Add EcoNet EN751221 "HPT" CPU Timer
      dt-bindings: timer: Convert arm,mps2-timer to DT schema
      dt-bindings: timer: Add Sophgo SG2044 ACLINT timer
      …

Following further in this commit, I only see this following series that had changes which may or may not be related to hang. 

https://lore.kernel.org/all/20250429065337.117370076@linutronix.de/

I am not very familiar with this subsystem and was hoping if somebody can spot the offending commit and possibly provide fix for this hang. 

Note that we tried with rc3 as well to see if there was fix applied in later RC and still see same issue. 

[  525.390801] INFO: task systemd-shutdow:1 blocked for more than 122 seconds.
[  525.474133]       Tainted: G S                  6.16.0-rc3.master.20250625.ol9.x86_64 #1
[  525.570969] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  525.664681] task:systemd-shutdow state:D stack:0     pid:1     tgid:1     ppid:0      task_flags:0x400100 flags:0x00004002
[  525.796878] Call Trace:
[  525.826116]  <TASK>
[  525.851195]  __schedule+0x2d1/0x730
[  525.892917]  schedule+0x27/0x80
[  525.930478]  schedule_preempt_disabled+0x15/0x30
[  525.985718]  __mutex_lock.constprop.0+0x4be/0x8a0
[  526.041993]  msi_domain_get_virq+0xcc/0x110
[  526.092031]  pci_msix_write_tph_tag+0x3c/0x100
[  526.145186]  pcie_tph_set_st_entry+0x125/0x1d0
[  526.198346]  bnxt_irq_affinity_release+0x35/0x50 [bnxt_en]
[  526.264015]  irq_set_affinity_notifier+0xe0/0x130
[  526.320291]  bnxt_free_irq+0x6e/0x110 [bnxt_en]
[  526.374507]  __bnxt_close_nic.isra.0+0x1eb/0x220 [bnxt_en]
[  526.440175]  bnxt_close+0x3a/0x100 [bnxt_en]
[  526.491264]  __dev_close_many+0xae/0x220
[  526.538179]  dev_close_many+0xc2/0x1b0
[  526.583014]  netif_close+0x9d/0xd0
[  526.623693]  bnxt_shutdown+0xb1/0xe0 [bnxt_en]
[  526.676874]  pci_device_shutdown+0x35/0x70
[  526.725871]  device_shutdown+0x118/0x1a0
[  526.772788]  kernel_restart+0x3a/0x70
[  526.816588]  __do_sys_reboot+0x150/0x250
[  526.863504]  do_syscall_64+0x84/0x940
[  526.907300]  ? __put_user_8+0xd/0x20
[  526.950059]  ? rseq_ip_fixup+0x90/0x1e0
[  526.995937]  ? task_mm_cid_work+0x1ad/0x220
[  527.045971]  ? __rseq_handle_notify_resume+0x35/0x90
[  527.105367]  ? arch_exit_to_user_mode_prepare.isra.0+0x98/0xb0
[  527.175166]  ? do_syscall_64+0xba/0x940
[  527.221040]  ? do_filp_open+0xd7/0x1a0
[  527.265882]  ? alloc_fd+0xba/0x110
[  527.306556]  ? do_sys_openat2+0xa4/0xf0
[  527.352434]  ? __x64_sys_openat+0x54/0xb0
[  527.400389]  ? arch_exit_to_user_mode_prepare.isra.0+0x9/0xb0
[  527.469150]  ? do_syscall_64+0xba/0x940
[  527.515023]  ? do_user_addr_fault+0x221/0x690
[  527.567141]  ? clear_bhb_loop+0x30/0x80
[  527.613017]  ? clear_bhb_loop+0x30/0x80
[  527.658895]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  527.719332] RIP: 0033:0x7fc3ec504777
[  527.762091] RSP: 002b:00007ffecd62c4f8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a9
[  527.852685] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fc3ec504777
[  527.938085] RDX: 0000000001234567 RSI: 0000000028121969 RDI: 00000000fee1dead
[  528.023485] RBP: 00007ffecd62c700 R08: 0000000000000000 R09: 00007ffecd62b8e0
[  528.108878] R10: 0000000000000001 R11: 0000000000000202 R12: 00007ffecd62c568
[  528.194273] R13: 00007ffecd62c548 R14: 00007ffecd62c568 R15: 0000000000000000
[  528.279672]  </TASK>

-- 
Himanshu Madhani	Oracle Linux Engineering

Re: System hang with latest kernel v6.16.0-rc1 (rc2 & rc3)

[Thomas Gleixner]

On Thu, Jul 03 2025 at 18:32, Himanshu Madhani wrote:
> On Jul 3, 2025, at 11:27, Himanshu Madhani <himanshu.madhani@oracle.com> wrote:
> Git-bisect point to this merge commit
>
> commit 6376c0770656f3bdf7f411faf068371b6932aeca
> Merge: 5e8bbb2caa4e 29857e6f4e30
> Author: Linus Torvalds <torvalds@linux-foundation.org>
> Date:   Tue May 27 09:01:26 2025 -0700
>
>    Merge tag 'timers-clocksource-2025-05-25' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>
>    Pull clocksource updates from Thomas Gleixner:
>     "Updates for clocksource/clockevent drivers:
>
>       - The final conversion of text formatted device tree binding to
>         schemas
>
>       - A new driver fot the System Timer Module on S32G NXP SoCs
>
>       - A new driver fot the Econet HPT timer
>
>       - The usual improvements and device tree binding updates"

That obviously does not make sense, so your bisect got side ways.

> Following further in this commit, I only see this following series
> that had changes which may or may not be related to hang.
>
> https://lore.kernel.org/all/20250429065337.117370076@linutronix.de/

They are not. There is a hint in both backtraces:

> [  514.305717]  schedule_preempt_disabled+0x15/0x30
> [  514.360954]  __mutex_lock.constprop.0+0x4be/0x8a0
> [  514.417232]  msi_domain_get_virq+0xcc/0x110
> [  514.467279]  pci_msix_write_tph_tag+0x3c/0x100

and

> [  525.930478]  schedule_preempt_disabled+0x15/0x30
> [  525.985718]  __mutex_lock.constprop.0+0x4be/0x8a0
> [  526.041993]  msi_domain_get_virq+0xcc/0x110
> [  526.092031]  pci_msix_write_tph_tag+0x3c/0x100

pci_msix_write_tph_tag() is the function which ends up trying to lock
the mutex and gets stuck. This function was introduced with commit

  d5124a9957b2 ("PCI/MSI: Provide a sane mechanism for TPH")

and the subsequent commit

  71296eae5887 ("PCI/TPH: Replace the broken MSI-X control word update")

flipped the TPH code over to use that.

The problem is obvious and if you would have enabled
CONFIG_PROVE_LOCKING then you would have got the reason presented on a
silver tablet in dmesg. I encourage you to do so nevertheless.

I definitely screwed that one up in the most stupid way.

As I had no idea how to exercise that code path I did not test it. It
seems this code is not really tested by any of the CI stuff either
before it hits Linus tree and as some folks start testing only post rc1
it takes some time to surface :( 

The fix is as obvious as the problem. See uncompiled and untested patch
below. If it solves the problem, which it should, feel free to take it
and create a proper patch with changelog and Fixes tag yourself (Adding
Suggested-by: Thomas ... is good enough). Otherwise let me know, and I
take care of it in my copious spare time :)

Thanks,

        tglx
---
diff --git a/drivers/pci/msi/msi.c b/drivers/pci/msi/msi.c
index 6ede55a7c5e6..eb26f3816922 100644
--- a/drivers/pci/msi/msi.c
+++ b/drivers/pci/msi/msi.c
@@ -934,10 +934,11 @@ int pci_msix_write_tph_tag(struct pci_dev *pdev, unsigned int index, u16 tag)
 	if (!pdev->msix_enabled)
 		return -ENXIO;
 
-	guard(msi_descs_lock)(&pdev->dev);
 	virq = msi_get_virq(&pdev->dev, index);
 	if (!virq)
 		return -ENXIO;
+
+	guard(msi_descs_lock)(&pdev->dev);
 	/*
 	 * This is a horrible hack, but short of implementing a PCI
 	 * specific interrupt chip callback and a huge pile of

Re: System hang with latest kernel v6.16.0-rc1 (rc2 & rc3)

[Himanshu Madhani]

> On Jul 3, 2025, at 13:21, Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> On Thu, Jul 03 2025 at 18:32, Himanshu Madhani wrote:
>> On Jul 3, 2025, at 11:27, Himanshu Madhani <himanshu.madhani@oracle.com> wrote:
>> Git-bisect point to this merge commit
>> 
>> commit 6376c0770656f3bdf7f411faf068371b6932aeca
>> Merge: 5e8bbb2caa4e 29857e6f4e30
>> Author: Linus Torvalds <torvalds@linux-foundation.org>
>> Date:   Tue May 27 09:01:26 2025 -0700
>> 
>>   Merge tag 'timers-clocksource-2025-05-25' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>> 
>>   Pull clocksource updates from Thomas Gleixner:
>>    "Updates for clocksource/clockevent drivers:
>> 
>>      - The final conversion of text formatted device tree binding to
>>        schemas
>> 
>>      - A new driver fot the System Timer Module on S32G NXP SoCs
>> 
>>      - A new driver fot the Econet HPT timer
>> 
>>      - The usual improvements and device tree binding updates"
> 
> That obviously does not make sense, so your bisect got side ways.
> 
>> Following further in this commit, I only see this following series
>> that had changes which may or may not be related to hang.
>> 
>> https://lore.kernel.org/all/20250429065337.117370076@linutronix.de/
> 
> They are not. There is a hint in both backtraces:
> 
>> [  514.305717]  schedule_preempt_disabled+0x15/0x30
>> [  514.360954]  __mutex_lock.constprop.0+0x4be/0x8a0
>> [  514.417232]  msi_domain_get_virq+0xcc/0x110
>> [  514.467279]  pci_msix_write_tph_tag+0x3c/0x100
> 
> and
> 
>> [  525.930478]  schedule_preempt_disabled+0x15/0x30
>> [  525.985718]  __mutex_lock.constprop.0+0x4be/0x8a0
>> [  526.041993]  msi_domain_get_virq+0xcc/0x110
>> [  526.092031]  pci_msix_write_tph_tag+0x3c/0x100
> 
> pci_msix_write_tph_tag() is the function which ends up trying to lock
> the mutex and gets stuck. This function was introduced with commit
> 
>  d5124a9957b2 ("PCI/MSI: Provide a sane mechanism for TPH")
> 
> and the subsequent commit
> 
>  71296eae5887 ("PCI/TPH: Replace the broken MSI-X control word update")
> 
> flipped the TPH code over to use that.
> 
> The problem is obvious and if you would have enabled
> CONFIG_PROVE_LOCKING then you would have got the reason presented on a
> silver tablet in dmesg. I encourage you to do so nevertheless.
> 
Great tip on this. I’ll keep that in mind for future debugging efforts. 

> I definitely screwed that one up in the most stupid way.
> 
> As I had no idea how to exercise that code path I did not test it. It
> seems this code is not really tested by any of the CI stuff either
> before it hits Linus tree and as some folks start testing only post rc1
> it takes some time to surface :( 
> 
> The fix is as obvious as the problem. See uncompiled and untested patch
> below. If it solves the problem, which it should, feel free to take it
> and create a proper patch with changelog and Fixes tag yourself (Adding
> Suggested-by: Thomas ... is good enough). Otherwise let me know, and I
> take care of it in my copious spare time :)
> 

Sure. I’ll get this testing in our test bed and report back in couple days.

> Thanks,
> 
>        tglx
> ---
> diff --git a/drivers/pci/msi/msi.c b/drivers/pci/msi/msi.c
> index 6ede55a7c5e6..eb26f3816922 100644
> --- a/drivers/pci/msi/msi.c
> +++ b/drivers/pci/msi/msi.c
> @@ -934,10 +934,11 @@ int pci_msix_write_tph_tag(struct pci_dev *pdev, unsigned int index, u16 tag)
> if (!pdev->msix_enabled)
> return -ENXIO;
> 
> - guard(msi_descs_lock)(&pdev->dev);
> virq = msi_get_virq(&pdev->dev, index);
> if (!virq)
> return -ENXIO;
> +
> + guard(msi_descs_lock)(&pdev->dev);
> /*
> * This is a horrible hack, but short of implementing a PCI
> * specific interrupt chip callback and a huge pile of



-- 
Himanshu Madhani	Oracle Linux Engineering

Re: System hang with latest kernel v6.16.0-rc1 (rc2 & rc3)

[Thomas Gleixner]

On Thu, Jul 03 2025 at 20:34, Himanshu Madhani wrote:
>> On Jul 3, 2025, at 13:21, Thomas Gleixner <tglx@linutronix.de> wrote:
>> The problem is obvious and if you would have enabled
>> CONFIG_PROVE_LOCKING then you would have got the reason presented on a
>> silver tablet in dmesg. I encourage you to do so nevertheless.
>> 
> Great tip on this. I’ll keep that in mind for future debugging efforts. 

Actually the very first thing in testing of a new kernel should be to
run it with a copious amount of debug options. That avoids all the
headaches of chasing fallout caught by them, in painful ways later.

I'm truly surprised that this is not done already and testing blindly
assumes that rc1 has already been objected to such tests completely.

It's bloody obvious that with a code base of the complexity of the
kernel and the gazillion of drivers, the CI coverage is far from
complete and only best effort based.

Obviously the companies, who have access to and care about their
specialized hardware, should run CI against linux-next to begin
with. Then such problems would be caught way before they hit Linus tree.

I know there is no budget for this kind of effort. Companies rather
waste their budget on chasing problems, which could have been avoided
upfront. That's a huge cost saving, which is proven by applying magic to
the relevant Excel-sh*ts.

Not your decision, I know.

Shrug,

        tglx

System hang with latest kernel v6.16.0-rc1 (rc2 & rc3)

[Himanshu Madhani]

Hi Folks,

We are seeing kernel hang while booting after new 6.16-rc1 kernel is 
installed.

Here’s stack track that shows up

[  297.656683] systemd-shutdown[1]: Rebooting with kexec.
[  513.790993] INFO: task kexec:19038 blocked for more than 122 seconds.
[  513.868087]       Not tainted 6.16.0-rc1.master.20250611.ol9.x86_64 #1
[  513.946210] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  514.039923] task:kexec           state:D stack:0     pid:19038 
tgid:19038 ppid:1      task_flags:0x400100 flags:0x00004002
[  514.172122] Call Trace:
[  514.201356]  <TASK>
[  514.226438]  __schedule+0x2d1/0x730
[  514.268161]  schedule+0x27/0x80
[  514.305717]  schedule_preempt_disabled+0x15/0x30
[  514.360954]  __mutex_lock.constprop.0+0x4be/0x8a0
[  514.417232]  msi_domain_get_virq+0xcc/0x110
[  514.467279]  pci_msix_write_tph_tag+0x3c/0x100
[  514.520441]  pcie_tph_set_st_entry+0x125/0x1d0
[  514.573605]  bnxt_irq_affinity_release+0x35/0x50 [bnxt_en]
[  514.639258]  irq_set_affinity_notifier+0xdd/0x130
[  514.695534]  bnxt_free_irq+0x6e/0x110 [bnxt_en]
[  514.749746]  __bnxt_close_nic.isra.0+0x1eb/0x220 [bnxt_en]
[  514.815404]  bnxt_close+0x3a/0x100 [bnxt_en]
[  514.866498]  __dev_close_many+0xab/0x220
[  514.913423]  __dev_change_flags+0x102/0x240
[  514.963464]  netif_change_flags+0x26/0x70
[  515.011424]  dev_change_flags+0x40/0xc0
[  515.057304]  devinet_ioctl+0x3aa/0x7a0
[  515.102142]  inet_ioctl+0x1d3/0x1f0
[  515.143863]  sock_do_ioctl+0x7a/0x140
[  515.187667]  __x64_sys_ioctl+0x9b/0x100
[  515.233545]  ? syscall_trace_enter+0x10c/0x1d0
[  515.286704]  do_syscall_64+0x84/0x940
[  515.330502]  ? refill_obj_stock+0x143/0x240
[  515.380543]  ? __dentry_kill+0x12e/0x190
[  515.427459]  ? __memcg_slab_free_hook+0xf4/0x150
[  515.482698]  ? __x64_sys_close+0x3d/0x80
[  515.529616]  ? kmem_cache_free+0x3fe/0x460
[  515.578614]  ? syscall_exit_work+0x118/0x150
[  515.629695]  ? arch_exit_to_user_mode_prepare.isra.0+0x9/0xb0
[  515.698453]  ? do_syscall_64+0xba/0x940
[  515.744330]  ? mod_memcg_lruvec_state+0x1a2/0x1f0
[  515.800608]  ? __lruvec_stat_mod_folio+0x83/0xd0
[  515.855843]  ? __folio_mod_stat+0x26/0x80
[  515.903801]  ? set_ptes.isra.0+0x36/0x90
[  515.950723]  ? do_anonymous_page+0x103/0x4b0
[  516.001802]  ? __handle_mm_fault+0x394/0x6f0
[  516.052886]  ? count_memcg_events+0x15a/0x1a0
[  516.105008]  ? handle_mm_fault+0x24a/0x350
[  516.154003]  ? do_user_addr_fault+0x221/0x690
[  516.206122]  ? arch_exit_to_user_mode_prepare.isra.0+0x9/0xb0
[  516.274887]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  516.335330] RIP: 0033:0x7fc96e903bcb
[  516.378086] RSP: 002b:00007ffcc7f78518 EFLAGS: 00000202 ORIG_RAX: 
0000000000000010
[  516.468683] RAX: ffffffffffffffda RBX: 000055dc432d8f80 RCX: 
00007fc96e903bcb
[  516.554080] RDX: 00007ffcc7f78680 RSI: 0000000000008914 RDI: 
0000000000000003
[  516.639482] RBP: 0000000000000000 R08: 0000000000000007 R09: 
0000000000000007
[  516.724882] R10: 000000000000005e R11: 0000000000000202 R12: 
000055dc095468dd
[  516.810278] R13: 000055dc095468e4 R14: 00007ffcc7f78680 R15: 
000055dc432d9020
[  516.895676]  </TASK>
[  516.921808] INFO: task kexec:19038 is blocked on a mutex likely owned 
by task kexec:19038.
[  517.020728] task:kexec           state:D stack:0     pid:19038 
tgid:19038 ppid:1      task_flags:0x400100 flags:0x00004002


Git-bisect point to this merge commit

commit 6376c0770656f3bdf7f411faf068371b6932aeca
Merge: 5e8bbb2caa4e 29857e6f4e30
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Tue May 27 09:01:26 2025 -0700

    Merge tag 'timers-clocksource-2025-05-25' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

    Pull clocksource updates from Thomas Gleixner:
     "Updates for clocksource/clockevent drivers:

       - The final conversion of text formatted device tree binding to
         schemas

       - A new driver fot the System Timer Module on S32G NXP SoCs

       - A new driver fot the Econet HPT timer

       - The usual improvements and device tree binding updates"

    * tag 'timers-clocksource-2025-05-25' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (31 commits)
      clocksource/drivers/renesas-ostm: Unconditionally enable reprobe 
support
      dt-bindings: timer: renesas,ostm: Document RZ/V2N (R9A09G056) support
      dt-bindings: timer: Convert marvell,armada-370-timer to DT schema
      dt-bindings: timer: Convert ti,keystone-timer to DT schema
      dt-bindings: timer: Convert st,spear-timer to DT schema
      dt-bindings: timer: Convert socionext,milbeaut-timer to DT schema
      dt-bindings: timer: Convert snps,arc-timer to DT schema
      dt-bindings: timer: Convert snps,archs-rtc to DT schema
      dt-bindings: timer: Convert snps,archs-gfrc to DT schema
      dt-bindings: timer: Convert lsi,zevio-timer to DT schema
      dt-bindings: timer: Convert jcore,pit to DT schema
      dt-bindings: timer: Convert img,pistachio-gptimer to DT schema
      dt-bindings: timer: Convert ezchip,nps400-timer to DT schema
      dt-bindings: timer: Convert cirrus,clps711x-timer to DT schema
      dt-bindings: timer: Convert altr,timer-1.0 to DT schema
      dt-bindings: timer: Add ESWIN EIC7700 CLINT
      clocksource/drivers: Add EcoNet Timer HPT driver
      dt-bindings: timer: Add EcoNet EN751221 "HPT" CPU Timer
      dt-bindings: timer: Convert arm,mps2-timer to DT schema
      dt-bindings: timer: Add Sophgo SG2044 ACLINT timer
      …

Following further in this commit, I only see this following series that 
had changes which may or may not be related to hang.

https://lore.kernel.org/all/20250429065337.117370076@linutronix.de/

I am not very familiar with this subsystem and was hoping if somebody 
can spot the offending commit and possibly provide fix for this hang.

Note that we tried with rc3 as well to see if there was fix applied in 
later RC and still see same issue.

[  525.390801] INFO: task systemd-shutdow:1 blocked for more than 122 
seconds.
[  525.474133]       Tainted: G S 
6.16.0-rc3.master.20250625.ol9.x86_64 #1
[  525.570969] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  525.664681] task:systemd-shutdow state:D stack:0     pid:1     tgid:1 
     ppid:0      task_flags:0x400100 flags:0x00004002
[  525.796878] Call Trace:
[  525.826116]  <TASK>
[  525.851195]  __schedule+0x2d1/0x730
[  525.892917]  schedule+0x27/0x80
[  525.930478]  schedule_preempt_disabled+0x15/0x30
[  525.985718]  __mutex_lock.constprop.0+0x4be/0x8a0
[  526.041993]  msi_domain_get_virq+0xcc/0x110
[  526.092031]  pci_msix_write_tph_tag+0x3c/0x100
[  526.145186]  pcie_tph_set_st_entry+0x125/0x1d0
[  526.198346]  bnxt_irq_affinity_release+0x35/0x50 [bnxt_en]
[  526.264015]  irq_set_affinity_notifier+0xe0/0x130
[  526.320291]  bnxt_free_irq+0x6e/0x110 [bnxt_en]
[  526.374507]  __bnxt_close_nic.isra.0+0x1eb/0x220 [bnxt_en]
[  526.440175]  bnxt_close+0x3a/0x100 [bnxt_en]
[  526.491264]  __dev_close_many+0xae/0x220
[  526.538179]  dev_close_many+0xc2/0x1b0
[  526.583014]  netif_close+0x9d/0xd0
[  526.623693]  bnxt_shutdown+0xb1/0xe0 [bnxt_en]
[  526.676874]  pci_device_shutdown+0x35/0x70
[  526.725871]  device_shutdown+0x118/0x1a0
[  526.772788]  kernel_restart+0x3a/0x70
[  526.816588]  __do_sys_reboot+0x150/0x250
[  526.863504]  do_syscall_64+0x84/0x940
[  526.907300]  ? __put_user_8+0xd/0x20
[  526.950059]  ? rseq_ip_fixup+0x90/0x1e0
[  526.995937]  ? task_mm_cid_work+0x1ad/0x220
[  527.045971]  ? __rseq_handle_notify_resume+0x35/0x90
[  527.105367]  ? arch_exit_to_user_mode_prepare.isra.0+0x98/0xb0
[  527.175166]  ? do_syscall_64+0xba/0x940
[  527.221040]  ? do_filp_open+0xd7/0x1a0
[  527.265882]  ? alloc_fd+0xba/0x110
[  527.306556]  ? do_sys_openat2+0xa4/0xf0
[  527.352434]  ? __x64_sys_openat+0x54/0xb0
[  527.400389]  ? arch_exit_to_user_mode_prepare.isra.0+0x9/0xb0
[  527.469150]  ? do_syscall_64+0xba/0x940
[  527.515023]  ? do_user_addr_fault+0x221/0x690
[  527.567141]  ? clear_bhb_loop+0x30/0x80
[  527.613017]  ? clear_bhb_loop+0x30/0x80
[  527.658895]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  527.719332] RIP: 0033:0x7fc3ec504777
[  527.762091] RSP: 002b:00007ffecd62c4f8 EFLAGS: 00000202 ORIG_RAX: 
00000000000000a9
[  527.852685] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 
00007fc3ec504777
[  527.938085] RDX: 0000000001234567 RSI: 0000000028121969 RDI: 
00000000fee1dead
[  528.023485] RBP: 00007ffecd62c700 R08: 0000000000000000 R09: 
00007ffecd62b8e0
[  528.108878] R10: 0000000000000001 R11: 0000000000000202 R12: 
00007ffecd62c568
[  528.194273] R13: 00007ffecd62c548 R14: 00007ffecd62c568 R15: 
0000000000000000
[  528.279672]  </TASK>


-- 
Himanshu Madhani                                Oracle Linux Engineering

Re: System hang with latest kernel v6.16.0-rc1 (rc2 & rc3)

[Thomas Gleixner]

On Thu, Jul 03 2025 at 12:31, Himanshu Madhani wrote:
> We are seeing kernel hang while booting after new 6.16-rc1 kernel is 
> installed.

Please don't resend stuff within a few hours just because nobody
replies. People live in different time zones and are not waiting in
front of their computer for your important messages.

Re: System hang with latest kernel v6.16.0-rc1 (rc2 & rc3)

[Himanshu Madhani]

> On Jul 3, 2025, at 13:24, Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> On Thu, Jul 03 2025 at 12:31, Himanshu Madhani wrote:
>> We are seeing kernel hang while booting after new 6.16-rc1 kernel is 
>> installed.
> 
> Please don't resend stuff within a few hours just because nobody
> replies. People live in different time zones and are not waiting in
> front of their computer for your important messages.

I got mail rejection notice from kernel mailing list complaining about HTML format. 
So I fixed it as plain text and resent. Did not realize that both mails made it to
Mailing list.  Apologies for noise. 

Thanks,
Himanshu