From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A00A70830; Tue, 13 Jan 2026 15:24:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768317890; cv=none; b=XdegkTWPinoSs4zuGe3Wp8HwcJjHo6L4MZaD22vdtzqkw0YY8umok1101Z+xwxVBxQLp+Wzk9ygeaFtZT+0hXbrJWGyI570T2Nlu/6uBIV2WtW99kighKwH6ce0b4lTIIOaudjxw6dtQ3kvFcopwXwGILaf2NXwrWitYadXssuw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1768317890; c=relaxed/simple; bh=taak2UygMZbdacgSedIXzlFkiSjhrP80UZpBg3+A9T4=; h=From:To:Cc:Subject:In-Reply-To:References:Date:Message-ID: MIME-Version:Content-Type; b=uKXTvmE5KTGkqPCVLoOGTPkma+MkZEPTRbdD2om7mf9QskYtaoc0d7f4ICyUmbcgMPwF4/nEgpSpUuR56u4sq8avld39mnCnmW0ZV3XKfB2A7R3m8D9YQh1fepebnWItR0z07A8mVoKv0476oS4Ipa0TETk11QbKO8rX/h4UuBM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AhP/y80K; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AhP/y80K" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 22D46C116C6; Tue, 13 Jan 2026 15:24:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1768317889; bh=taak2UygMZbdacgSedIXzlFkiSjhrP80UZpBg3+A9T4=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=AhP/y80KVpZcmRV1+NMKLTI7A3S3B1azBgUpQ1DJUoaoH/TdnGwIdSpNdf4iOtRl5 VMjPXhRqRohFjpEbWulvcJ9JjRwozK8fucaQwQiM8u0Gja1ik0Q3sTcW/6IIEq+3d5 hHD4PWcpVpfgVDOn0SV7bboOAc09bk7OwyDufmZublxhYqPbm49WVLbdvElM/VF8he HvtLSZJ2jXKm2fI/o+QkcMPrelVltHLBYAFKn1iSoIrObiMteKseVxFxZCfeELV8Ah IWAlUHyqaY0szCLSQBji+FnqKNi/kZEfK8gaMPepsMkMAyl1jzdeApqJdvxGLgQf5K W731h5A9eUm3w== From: Thomas Gleixner To: Bert Karwatzki , linux-kernel@vger.kernel.org Cc: Bert Karwatzki , linux-next@vger.kernel.org, Mario Limonciello , Sebastian Andrzej Siewior , Clark Williams , Steven Rostedt , Christian =?utf-8?Q?K=C3=B6nig?= , regressions@lists.linux.dev, linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org, "Rafael J . Wysocki" , acpica-devel@lists.linux.dev, Robert Moore , Saket Dumbre , Bjorn Helgaas , Clemens Ladisch , Jinchao Wang , Yury Norov , Anna Schumaker , Baoquan He , "Darrick J. Wong" , Dave Young , Doug Anderson , "Guilherme G. Piccoli" , Helge Deller , Ingo Molnar , Jason Gunthorpe , Joanthan Cameron , Joel Granados , John Ogness , Kees Cook , Li Huafei , "Luck, Tony" , Luo Gengkun , Max Kellermann , Nam Cao , oushixiong , Petr Mladek , Qianqiang Liu , Sergey Senozhatsky , Sohil Mehta , Tejun Heo , Thomas Zimemrmann , Thorsten Blum , Ville Syrjala , Vivek Goyal , Yicong Yang , Yunhui Cui , Andrew Morton , W_Armin@gmx.de Subject: Re: NMI stack overflow during resume of PCIe bridge with CONFIG_HARDLOCKUP_DETECTOR=y In-Reply-To: <20260113094129.3357-1-spasswolf@web.de> References: <20260113094129.3357-1-spasswolf@web.de> Date: Tue, 13 Jan 2026 16:24:46 +0100 Message-ID: <87h5spk01t.ffs@tglx> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain On Tue, Jan 13 2026 at 10:41, Bert Karwatzki wrote: > Here's the result in case of the crash: > 2026-01-12T04:24:36.809904+01:00 T1510;acpi_ex_system_memory_space_handler 255: logical_addr_ptr = ffffc066977b3000 > 2026-01-12T04:24:36.846170+01:00 C14;exc_nmi: 0 Here the NMI triggers in non-task context on CPU14 > 2026-01-12T04:24:36.960760+01:00 C14;exc_nmi: 10.3 > 2026-01-12T04:24:36.960760+01:00 C14;default_do_nmi > 2026-01-12T04:24:36.960760+01:00 C14;nmi_handle: type=0x0 > 2026-01-12T04:24:36.960760+01:00 C14;nmi_handle: a=0xffffffffa1612de0 > 2026-01-12T04:24:36.960760+01:00 C14;nmi_handle: a->handler=perf_event_nmi_handler+0x0/0xa6 > 2026-01-12T04:24:36.960760+01:00 C14;perf_event_nmi_handler: 0 > 2026-01-12T04:24:36.960760+01:00 C14;perf_event_nmi_handler: 1 > 2026-01-12T04:24:36.960760+01:00 C14;perf_event_nmi_handler: 2 > 2026-01-12T04:24:36.960760+01:00 C14;x86_pmu_handle_irq: 2 > 2026-01-12T04:24:36.960760+01:00 C14;x86_pmu_handle_irq: 2.6 > 2026-01-12T04:24:36.960760+01:00 C14;__perf_event_overflow: 0 > 2026-01-12T04:24:36.960760+01:00 C14;__perf_event_overflow: 6.99: overflow_handler=watchdog_overflow_callback+0x0/0x10d > 2026-01-12T04:24:36.960760+01:00 C14;watchdog_overflow_callback: 0 > 2026-01-12T04:24:36.960760+01:00 C14;__ktime_get_fast_ns_debug: 0.1 > 2026-01-12T04:24:36.960760+01:00 C14;tk_clock_read_debug: read=read_hpet+0x0/0xf0 > 2026-01-12T04:24:36.960760+01:00 C14;read_hpet: 0 > 2026-01-12T04:24:36.960760+01:00 C14;read_hpet: 0.1 > 2026-01-12T04:24:36.960760+01:00 T0;exc_nmi: 0 This one triggers in task context of PID0, aka idle task, but it's not clear on which CPU that happens. It's probably CPU13 as that continues with the expected 10.3 output, but that's almost ~1.71 seconds later. > 2026-01-12T04:24:38.674625+01:00 C13;exc_nmi: 10.3 > 2026-01-12T04:24:38.674625+01:00 C13;default_do_nmi > 2026-01-12T04:24:38.674625+01:00 C13;nmi_handle: type=0x0 > 2026-01-12T04:24:38.674625+01:00 C13;nmi_handle: a=0xffffffffa1612de0 > 2026-01-12T04:24:38.674625+01:00 C13;nmi_handle: a->handler=perf_event_nmi_handler+0x0/0xa6 > 2026-01-12T04:24:38.674625+01:00 C13;perf_event_nmi_handler: 0 > 2026-01-12T04:24:38.674625+01:00 C13;perf_event_nmi_handler: 1 > 2026-01-12T04:24:38.674625+01:00 C13;perf_event_nmi_handler: 2 > 2026-01-12T04:24:38.674625+01:00 C13;x86_pmu_handle_irq: 2 > 2026-01-12T04:24:38.674625+01:00 C13;x86_pmu_handle_irq: 2.6 > 2026-01-12T04:24:38.674625+01:00 C13;__perf_event_overflow: 0 > 2026-01-12T04:24:38.674625+01:00 C13;__perf_event_overflow: 6.99: overflow_handler=watchdog_overflow_callback+0x0/0x10d > 2026-01-12T04:24:38.674625+01:00 C13;watchdog_overflow_callback: 0 > 2026-01-12T04:24:38.674625+01:00 C13;__ktime_get_fast_ns_debug: 0.1 > 2026-01-12T04:24:38.674625+01:00 C13;tk_clock_read_debug: read=read_hpet+0x0/0xf0 > 2026-01-12T04:24:38.674625+01:00 C13;read_hpet: 0 > 2026-01-12T04:24:38.674625+01:00 C13;read_hpet: 0.1 > 2026-01-12T04:24:38.674625+01:00 T0;exc_nmi: 0 Same picture as above, but this time on CPU2 with a delay of 0.68 seconds > 2026-01-12T04:24:39.355101+01:00 C2;exc_nmi: 10.3 > 2026-01-12T04:24:39.355101+01:00 C2;default_do_nmi > 2026-01-12T04:24:39.355101+01:00 C2;nmi_handle: type=0x0 > 2026-01-12T04:24:39.355101+01:00 C2;nmi_handle: a=0xffffffffa1612de0 > 2026-01-12T04:24:39.355101+01:00 C2;nmi_handle: a->handler=perf_event_nmi_handler+0x0/0xa6 > 2026-01-12T04:24:39.355101+01:00 C2;perf_event_nmi_handler: 0 > 2026-01-12T04:24:39.355101+01:00 C2;perf_event_nmi_handler: 1 > 2026-01-12T04:24:39.355101+01:00 C2;perf_event_nmi_handler: 2 > 2026-01-12T04:24:39.355101+01:00 C2;x86_pmu_handle_irq: 2 > 2026-01-12T04:24:39.355101+01:00 C2;x86_pmu_handle_irq: 2.6 > 2026-01-12T04:24:39.355101+01:00 C2;__perf_event_overflow: 0 > 2026-01-12T04:24:39.355101+01:00 C2;__perf_event_overflow: 6.99: overflow_handler=watchdog_overflow_callback+0x0/0x10d > 2026-01-12T04:24:39.355101+01:00 C2;watchdog_overflow_callback: 0 > 2026-01-12T04:24:39.355101+01:00 C2;__ktime_get_fast_ns_debug: 0.1 > 2026-01-12T04:24:39.355101+01:00 C2;tk_clock_read_debug: read=read_hpet+0x0/0xf0 > 2026-01-12T04:24:39.355101+01:00 C2;read_hpet: 0 > 2026-01-12T04:24:39.355101+01:00 C2;read_hpet: 0.1 > 2026-01-12T04:24:39.355101+01:00 T0;exc_nmi: 0 Again on CPU0 with a delay of 0.06 seconds > 2026-01-12T04:24:39.410207+01:00 C0;exc_nmi: 10.3 > 2026-01-12T04:24:39.410207+01:00 C0;default_do_nmi > 2026-01-12T04:24:39.410207+01:00 C0;nmi_handle: type=0x0 > 2026-01-12T04:24:39.410207+01:00 C0;nmi_handle: a=0xffffffffa1612de0 > 2026-01-12T04:24:39.410207+01:00 C0;nmi_handle: a->handler=perf_event_nmi_handler+0x0/0xa6 > 2026-01-12T04:24:39.410207+01:00 C0;perf_event_nmi_handler: 0 > 2026-01-12T04:24:39.410207+01:00 C0;perf_event_nmi_handler: 1 > 2026-01-12T04:24:39.410207+01:00 C0;perf_event_nmi_handler: 2 > 2026-01-12T04:24:39.410207+01:00 C0;x86_pmu_handle_irq: 2 > 2026-01-12T04:24:39.410207+01:00 C0;x86_pmu_handle_irq: 2.6 > 2026-01-12T04:24:39.410207+01:00 C0;__perf_event_overflow: 0 > 2026-01-12T04:24:39.410207+01:00 C0;__perf_event_overflow: 6.99: overflow_handler=watchdog_overflow_callback+0x0/0x10d > 2026-01-12T04:24:39.410207+01:00 C0;watchdog_overflow_callback: 0 > 2026-01-12T04:24:39.410207+01:00 C0;__ktime_get_fast_ns_debug: 0.1 > 2026-01-12T04:24:39.410207+01:00 C0;tk_clock_read_debug: read=read_hpet+0x0/0xf0 > 2026-01-12T04:24:39.410207+01:00 C0;read_hpet: 0 > 2026-01-12T04:24:39.410207+01:00 C0;read_hpet: 0.1 > 2026-01-12T04:24:39.410207+01:00 T0;exc_nmi: 0 .... > In the case of the crash the interrupt handler never returns because when accessing > the HPET another NMI is triggered. This goes on until a crash happens, probably because > of stack overflow. No. NMI nesting is only one level deep and immediately returns: if (this_cpu_read(nmi_state) != NMI_NOT_RUNNING) { this_cpu_write(nmi_state, NMI_LATCHED); return; } So it's not a stack overflow. What's more likely is that after a while _ALL_ CPUs are hung up in the NMI handler after they tripped over the HPET read. > The behaviour described here seems to be similar to the bug that commit > 3d5f4f15b778 ("watchdog: skip checks when panic is in progress") is fixing, but > this is actually a different bug as kernel 6.18 (which contains 3d5f4f15b778) > is also affected (I've conducted 5 tests with 6.18 so far and got 4 crashes (crashes occured > after (0.5h, 1h, 4.5h, 1.5h) of testing)). > Nevertheless these look similar enough to CC the involved people. There is nothing similar. Your problem originates from a screwed up hardware state which in turn causes the HPET to go haywire for unknown reasons. What is the physical address of this ACPI handler access: logical_addr_ptr = ffffc066977b3000 along with the full output of /proc/iomem Thanks, tglx