From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C3A22C00144 for ; Wed, 27 Jul 2022 02:05:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rbUvO4ayvj3m2b+4bP0fMurlcntvM5ItEj/c1G99kl4=; b=CpsLzbpMUJm9x4 rMfisi1/bKu3wrt4enDT9Meqahd1LsUYlf9tYAiAd+wrq9nBSSkXyYm/t9rcbuMQlgPrPbjxkLIDr Pc6p6p5W/wYwowQrv2Cxlt9k9O7nAwci2orlG8Mq4tStXmBou355jVFNb+HzTW91r1W8mk/klzRFj 2WZ7Ioo3JdWlfpnX3iJ3lSDKTzAApvO/cXdeGK1ZUDEQiBnwmgivTKCY/bwGPjKHNvevXk2Db4f4O S5ARglQWIFuIkFkaIwOiTq/8TC6BVwFrFw9hpnCuCgzqAHLXpfmjIklqDJw5wco5If4dpf0lZCuxd 1qLdQsuHirj5GBoC0JiA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oGWP6-007ZbI-JS; Wed, 27 Jul 2022 02:04:04 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oGWP3-007ZXm-6V for linux-arm-kernel@lists.infradead.org; Wed, 27 Jul 2022 02:04:02 +0000 Received: by mail-pf1-x430.google.com with SMTP id o12so14816209pfp.5 for ; Tue, 26 Jul 2022 19:03:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=yd/l1Y2v+nNxV0KyGFNfQlfxUt2Z9k1HtIC4DEOBQIU=; b=yf93Bqh4B7NAb5zzYjgyRtPl5sG47SRE+ksECQ5hxpl8IOazvvey1c0bfIuyUoPpjz Lt9XU76+4iqUu0sDqEZUbnccnCEdaOYUUqCpsDBHaagc+yJL0WTlSaJnwDoEfb8rW/D+ IZExgYmmpTc8iwpYcu9lujNolDDLwlfqfdoMcfBxxbSy5cjMsF5RA92Db8qyhXxik/6I uUuNIgWXiQ6p0sVKMEeB3GaSs+D9JoNK/iULHq6yCIPfdCrk4geRD+W8gw88AVSeBGmX kENgXRKBaUiqZrgOiyuA8TClKTCKB/B/9xqrqB7hK3FfZI6P7nGn/P9bJuzM6gPVXAgZ h1nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=yd/l1Y2v+nNxV0KyGFNfQlfxUt2Z9k1HtIC4DEOBQIU=; b=7XCB+3+gXhscairlTyB9/NPQarj2lBJ+52k6aciHg/kfoUicaTC27VFH8G7k1XlSN/ 4Xb+h3jYH6yg/cy0JohVTYGfz+iXxIEhNZ9hx3FqlCW0mvKt1OeKjaRVEBTGsUe3s2CL v5SRTOm2VHonBpFi0jM2RK0K9A+DF1q2jucRdM4qHIpSr9ZLhvR7hxSf0ae80eAeLqOm bKx9YrRHQb+LsrOZ1p9Z/88tiW0K0PRwRNYtqOfUqvugEOklNWBiHIzOePpym/TjxaOS l+gApYGF9IAtWp0hMLzjKMvua0njRP3Ae4oiEtzBXGFUO6TX+Mk+ykW/HXMSPCLlyMNw Cyuw== X-Gm-Message-State: AJIora/jgkrSusZKcPeSEnxVrU9uTpD6wIqKb4qfcDP5KDBu2d795/kO ukG566VvZ0MW4rq94//pPZF3dg== X-Google-Smtp-Source: AGRyM1tDkPTK1dhSVBWeIzbIo15BGWiupCCg42sIRLr/tTeciL8rBbq/r2ksgg2jhug6UlZ7PcO3QQ== X-Received: by 2002:aa7:84c1:0:b0:52a:e11a:f5e9 with SMTP id x1-20020aa784c1000000b0052ae11af5e9mr19626590pfn.55.1658887439049; Tue, 26 Jul 2022 19:03:59 -0700 (PDT) Received: from leoy-ThinkPad-X240s (n058152077182.netvigator.com. [58.152.77.182]) by smtp.gmail.com with ESMTPSA id o63-20020a625a42000000b0052ba70ea97esm12330831pfb.30.2022.07.26.19.03.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Jul 2022 19:03:58 -0700 (PDT) Date: Wed, 27 Jul 2022 10:03:54 +0800 From: Leo Yan To: Mark Rutland , Mike Leach , Coresight ML Cc: Sudeep Holla , linux-kernel@vger.kernel.org, linux-pm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Peter Zijlstra Subject: Re: [-next] Lockdep warnings Message-ID: <20220727020354.GE36862@leoy-ThinkPad-X240s> References: <20220726104134.3b3awfphvafljdgp@bogus> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220726_190401_267583_D3471674 X-CRM114-Status: GOOD ( 25.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Jul 26, 2022 at 01:50:06PM +0100, Mark Rutland wrote: > On Tue, Jul 26, 2022 at 01:40:40PM +0100, Mark Rutland wrote: > > [Adding Peter; I suspect this is due to the cpuidle rework] > > Looking again I see the cpuidle rework isn't in next, so evidently not... > > Sorry for the noise! I'd like to loop in Mike.L and CoreSight ML for CTI PM callbacks. Please see below a comment for CTI spinlock usage. > > I'll go give next a spin in a VM, but I suspect I might need real HW to see > > this due to the way PSCI idle states work. > > > > Mark. > > > > On Tue, Jul 26, 2022 at 11:41:34AM +0100, Sudeep Holla wrote: > > > I was seeing the below lockdep warnings on my arm64 Juno development > > > platform almost 2 weeks back with -next. I wanted to check for similar > > > reports before post and forgot. > > > > > > --->8 > > > > > > DEBUG_LOCKS_WARN_ON(lockdep_hardirqs_enabled()) > > > hardirqs last enabled at (46157): cpuidle_enter_state+0x174/0x2b4 > > > WARNING: CPU: 5 PID: 0 at kernel/locking/lockdep.c:5506 check_flags+0x90/0x1e8 > > > hardirqs last disabled at (46158): el1_interrupt+0x2c/0xc8 > > > Modules linked in: > > > softirqs last enabled at (46154): __do_softirq+0x2c0/0x388 > > > softirqs last disabled at (46139): __irq_exit_rcu+0x118/0x18c > > > CPU: 5 PID: 0 Comm: swapper/5 Not tainted 5.19.0-rc6-next-20220714 #9 > > > pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) > > > pc : check_flags+0x90/0x1e8 > > > lr : check_flags+0x90/0x1e8 > > > Call trace: > > > check_flags+0x90/0x1e8 > > > lock_is_held_type+0x80/0x164 > > > rcu_read_lock_sched_held+0x40/0x7c > > > trace_rcu_dyntick+0x5c/0x140 > > > ct_kernel_enter+0x78/0xd4 > > > ct_idle_exit+0x1c/0x44 > > > cpu_idle_poll+0x74/0xb8 > > > do_idle+0x90/0x2c4 > > > cpu_startup_entry+0x30/0x34 > > > secondary_start_kernel+0x130/0x144 > > > __secondary_switched+0xb0/0xb4 > > > irq event stamp: 64229 > > > hardirqs last enabled at (64229): cpu_idle_poll+0x40/0xb8 > > > hardirqs last disabled at (64228): do_idle+0xbc/0x2c4 > > > softirqs last enabled at (64190): __do_softirq+0x2c0/0x388 > > > softirqs last disabled at (64185): __irq_exit_rcu+0x118/0x18c > > > ---[ end trace 0000000000000000 ]--- > > > possible reason: unannotated irqs-off. > > > irq event stamp: 64229 > > > hardirqs last enabled at (64229): cpu_idle_poll+0x40/0xb8 > > > hardirqs last disabled at (64228): do_idle+0xbc/0x2c4 > > > softirqs last enabled at (64190): __do_softirq+0x2c0/0x388 > > > softirqs last disabled at (64185): __irq_exit_rcu+0x118/0x18c > > > > > > ---- > > > > > > However I don't see the above warning with the latest -next. When I tried > > > yesterday's -next now, I see a different warning. Not sure if they are > > > related. I haven't tried to bisect. > > > > > > --->8 > > > ============================= > > > [ BUG: Invalid wait context ] > > > 5.19.0-rc8-next-20220725 #38 Not tainted > > > ----------------------------- > > > swapper/0/0 is trying to lock: > > > (&drvdata->spinlock){....}-{3:3}, at: cti_cpu_pm_notify+0x54/0x114 > > > other info that might help us debug this: > > > context-{5:5} > > > 1 lock held by swapper/0/0: > > > #0: (cpu_pm_notifier.lock){....}-{2:2}, at: cpu_pm_enter+0x2c/0x80 > > > stack backtrace: > > > CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.19.0-rc8-next-20220725-00004-g599e6691ed8c #38 > > > Call trace: > > > dump_backtrace+0xe8/0x108 > > > show_stack+0x18/0x4c > > > dump_stack_lvl+0x90/0xc8 > > > dump_stack+0x18/0x54 > > > __lock_acquire+0xa70/0x32d0 > > > lock_acquire+0x160/0x308 > > > _raw_spin_lock+0x60/0xa0 > > > cti_cpu_pm_notify+0x54/0x114 > > > raw_notifier_call_chain_robust+0x50/0xd4 > > > cpu_pm_enter+0x48/0x80 > > > psci_enter_idle_state+0x34/0x74 > > > cpuidle_enter_state+0x120/0x2a8 > > > cpuidle_enter+0x38/0x50 > > > do_idle+0x1e8/0x2b8 > > > cpu_startup_entry+0x24/0x28 > > > kernel_init+0x0/0x1a0 > > > start_kernel+0x0/0x470 > > > start_kernel+0x34c/0x470 > > > __primary_switched+0xbc/0xc4 If we look into for this callback, we can see the lock sequence is: cti_cpu_pm_notify() `> cpu_pm_notify_robust(): `> raw_spin_lock_irqsave(cpu_pm_notifier.lock, flag) -> a raw spinlock `> cti_cpu_pm_notify() `> spin_lock(&drvdata->spinlock) -> a normal spinlock A raw spinlock is not a sleepable lock, and normal spinlock can be a sleepable lock (e.g. it can be a mutex after enabled PREEMPT_RT). One solution is we can change to a raw spinlock in CTI driver, so this can dismiss the lockdep warning. Actually, I am a bit suspect if it's really necessary to use spinlock in CTI PM callbacks, the reason is in CPU's idle flow, it will run into idle thread context and disable the local IRQ, which means it likely has no race condition with thread context and interrupt handler, so we can remove the locking in PM callbacks. Mike, could you check for this? Thanks! Leo _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel