From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7D54175D53 for ; Mon, 23 Feb 2026 08:23:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=216.40.44.11 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771834987; cv=none; b=MaCkaLMy6UvMFxSeyHqCQ1xlU4CbthZANTJQCeAzx81s8KYBkY4vFw/e6AUl/5eXPQpTNke5AQSih0pF/IKwwue+5VEzDkrs+JBSTaqoX/+eZTmt10Xx/UWYI5PKYZ+Oi4I9rfBW5o9OMS4QlUNBXzvSwg8UDGVsNNGpZSB8QIg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771834987; c=relaxed/simple; bh=EU8i9QBfwsUMNHFeFxO63qU8YXBcS3pxny0rK3Gx+hI=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=RAo4JeS6PqVQ1Pn9z7DDkLYijCtyK3EhvCX13qRJRjZvjuPgbaIqT6a2S+ryNC1oLdaCsvwGly39Rkxh3X6oKdBr9xnUrRoFHda8zlcE0qt7H/vSKKpyuPj+10eLqp4cF5BCryLBGG8Tz2yT7L6FYRrcAIzB/PqtjO23j/2OMtU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=goodmis.org; spf=pass smtp.mailfrom=goodmis.org; arc=none smtp.client-ip=216.40.44.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=goodmis.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=goodmis.org Received: from omf06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0707A1A0813; Mon, 23 Feb 2026 08:23:03 +0000 (UTC) Received: from [HIDDEN] (Authenticated sender: rostedt@goodmis.org) by omf06.hostedemail.com (Postfix) with ESMTPA id 6752E20014; Mon, 23 Feb 2026 08:23:01 +0000 (UTC) Date: Mon, 23 Feb 2026 03:22:57 -0500 From: Steven Rostedt To: Bert Karwatzki Cc: Calvin Owens , Tejun Heo , Sebastian Andrzej Siewior , Thomas Gleixner , dschatzberg@meta.com, peterz@infradead.org, linux-kernel@vger.kernel.org, linux-rt-devel@lists.linux.dev Subject: Re: ~90s shutdown delay with v6.19 and PREEMPT_RT Message-ID: <20260223032257.4e6db467@fedora> In-Reply-To: References: <20260219164648.3014-1-spasswolf@web.de> <594b9094ab76acf9e73cad8fca9c9efd6f75a980.camel@web.de> <20260219195851.7a7bd1b2@fedora> <1d56c93ee93334b9d320c93d255d7548f4497836.camel@web.de> <20260220104420.54610b04@gandalf.local.home> X-Mailer: Claws Mail 4.3.1 (GTK 3.24.51; x86_64-redhat-linux-gnu) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Stat-Signature: 6ko3cihughryo1ifmq6yxoxnbbbwfdpj X-Rspamd-Server: rspamout02 X-Rspamd-Queue-Id: 6752E20014 X-Session-Marker: 726F737465647440676F6F646D69732E6F7267 X-Session-ID: U2FsdGVkX1+LBVeR0ORAs0qlpk21GQbSQMQtK1CFO4U= X-HE-Tag: 1771834981-433130 X-HE-Meta: U2FsdGVkX1+y+D7edhI/4QL8pglANvRAYC9c6ueOnFOoJ082NKv18lggheSBu+8Cm1Xizxkicl6Pu9hYNMbcX97BqWECfK6PSJXM+gz5N/w1UaQmJzjXfdUrF4GRE6Au2zmszcwwrrz1X0wHlYrXmlOKbZXYjPfbAdeqzl49hUo90IgUbpM4Fra1dmxRZ3a7iH2MR6zxfnFjiIhLfCifYDyw+V8V327Amx5bL2G65gZ8zRl6ChMyWgJoOTZyowESrH9/z7Tjxp8KdyPNUZH6D4NaNGWht60pKmzhMHRYuyu3o79y/ZfED0jf9sRIXbti On Mon, 23 Feb 2026 01:35:36 +0100 Bert Karwatzki wrote: > So the time to was is ~120s with PREEMPT_RT and 7s without. > > The interesting difference between these two traces is that the second one only > contains messages with "status" d..2. while the first also contains some with different status > (191 of 265126). Could these be the reason for the delay. > > $ grep -v d..2. trace.txt > > # tracer: nop > # > # entries-in-buffer/entries-written: 265126/265126 #P:16 > # > # _-----=> irqs-off/BH-disabled > # / _----=> need-resched > # | / _---=> hardirq/softirq > # || / _--=> preempt-depth > # ||| / _-=> migrate-disable > # |||| / delay > # TASK-PID CPU# ||||| TIMESTAMP FUNCTION > # | | | ||||| | | > <...>-1584 [011] D..22 62.779670: sched_switch: prev_comm=ntpd prev_pid=0x630 (1584) prev_prio=0x78 (120) prev_state=0x100 (256) > next_comm=mt76-tx phy0 next_pid=0x5fb (1531) next_prio=0x62 (98) The 'D' means both interrupts 'd' and softirqs 'b' are disabled. The last number is migrate disable which means the task is pinned to a CPU. That may be an issue if the system is trying to take down a CPU and there's a task pinned to it. Now that we know that the persistent ring buffer works, we can add even more debugging. We could see where things are stuck... cd /sys/kernel/tracing/instances/boot_map echo 'stacktrace if prev_state & 3' > events/sched/sched_switch/trigger That will do a stacktrace at every location that schedules out in a non-running state. That way we can see what is waiting for something to finish. Then in a separate boot, we may want to see where things are pinned. echo 'stacktrace if common_flags & 0xf00' > events/sched/sched_switch/trigger That will do a stacktrace every time a task schedules out with migration disabled. -- Steve