From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8782BC433F5 for ; Tue, 18 Jan 2022 08:01:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1EF86B0072; Tue, 18 Jan 2022 03:01:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DCF176B0073; Tue, 18 Jan 2022 03:01:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CBE796B0074; Tue, 18 Jan 2022 03:01:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0197.hostedemail.com [216.40.44.197]) by kanga.kvack.org (Postfix) with ESMTP id BCAF26B0072 for ; Tue, 18 Jan 2022 03:01:04 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7A17A1809CDE4 for ; Tue, 18 Jan 2022 08:01:04 +0000 (UTC) X-FDA: 79042662048.21.33646FC Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) by imf20.hostedemail.com (Postfix) with ESMTP id 21A281C000F for ; Tue, 18 Jan 2022 08:01:03 +0000 (UTC) Received: by mail-oi1-f175.google.com with SMTP id w188so27198981oiw.13 for ; Tue, 18 Jan 2022 00:01:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Al3F+2ViLnm9/9qxjHoxeFPrd9kF19aHZnlQsAdIPRg=; b=FCysJPty2mrcMf6v3VRjTFbhweCJRR3WyXaxWMp3y2v3HgQYlh1B9roeLBgjjTLRrF q0EveM9wLKiqTMkN1DW5Bwx6oBRbVXaR3Yor5+pZHw/GeYeVfg7jaml0nA3ZTYTbDPf8 /M5uRa5UkC50Ris5+mVE2ymR7zWkKQSwm1M6tEpnSjw+FA+oE8BdNjJc5IU+QJ45WFS1 1eYPiEg8mweB/z9UP6dqcOFv/DMTLw15D/CbD2dDV5ue9EUg5bCwcw/h3jULi4lZQdms qsgFgsm0r2f8iqGy0dIX/Ep+vHUUwgup0pUjPtXrxHk/w8ZNsQE+Ez0HsgQDhj6KnjWf h9hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Al3F+2ViLnm9/9qxjHoxeFPrd9kF19aHZnlQsAdIPRg=; b=VHpgIBtKydhpt1c7WyLZl0Eove6MbbJhOMQLmuL6uWQ0bJ3wcffUB61p6okNryO2RF M2BQyW28YzPwHQfA+01YzmVbdROeJc218prpDRfdnmfnHxNvC3OV/IulFNzivcyh201U spCoQJ2JFldUhr/ED1rMOXAZJI1YApStjcUJusKpI0Ncun9f/Eg5GQ9CSnsBlJUCeXzF IVr8YXg/HiBVPRlxs+bKNE46htx1fPE2h/fKCRzjgT75JqMJb0EOhYXrn91c3aLD88G9 G7VNZFPapExJtmRxEZPsw3rQF/efb6OWxcT7HaczDQ+s9RkYhYSGgYY2YB1omDlqHvoe TOTQ== X-Gm-Message-State: AOAM5324u23weKvGmckVbPHrmzPFHZuIIOpnn4t46QnqPGbjjF/haPgY +xggPLhEvDkf+vM8tKvZrfwU+1Loq0UxIxI2Fk0= X-Google-Smtp-Source: ABdhPJzvGwPY3OUJ5cr+YDP3ryIKCL5St1eOUXqyHiBMdGlYGVftPHZ71UYHtXtQprVP0IFc5O3S9yOgkr3fuXzNLkU= X-Received: by 2002:aca:2802:: with SMTP id 2mr7601418oix.23.1642492863253; Tue, 18 Jan 2022 00:01:03 -0800 (PST) MIME-Version: 1.0 References: <388098b2c03fbf0a732834fc01b2d875c335bc49.1642170196.git.lucien.xin@gmail.com> <39a3470f-06ab-cf41-32e4-80edb249c7d3@suse.cz> <20220117131304.pdc3mfdowkzovw6q@localhost.localdomain> In-Reply-To: <20220117131304.pdc3mfdowkzovw6q@localhost.localdomain> From: Xin Long Date: Tue, 18 Jan 2022 16:00:51 +0800 Message-ID: Subject: Re: [PATCH] mm: slub: fix a deadlock warning in kmem_cache_destroy To: Juri Lelli Cc: Vlastimil Babka , Sebastian Andrzej Siewior , Hyeonggon Yoo <42.hyeyoo@gmail.com>, LKML , linux-mm@kvack.org, Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Antoine Tenart , Clark Williams Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 21A281C000F X-Stat-Signature: 6g1f4um837aw9d7fm5xsrqhhu57jinrq Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FCysJPty; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of lucien.xin@gmail.com designates 209.85.167.175 as permitted sender) smtp.mailfrom=lucien.xin@gmail.com X-HE-Tag: 1642492863-475641 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jan 17, 2022 at 9:13 PM Juri Lelli wrote: > > Hi, > > On 17/01/22 13:40, Vlastimil Babka wrote: > > +CC Clark > > > > On 1/17/22 10:33, Sebastian Andrzej Siewior wrote: > > > On 2022-01-17 16:32:46 [+0800], Xin Long wrote: > > >> another issue. From the code analysis, this issue does exist on the > > >> upstream kernel, though I couldn't build an upstream RT kernel for the > > >> testing. > > > > > > This should also reproduce in v5.16 since the commit in question is > > > there. > > > > Yeah. I remember we had some issues with the commit during development, but > > I'd hope those were resolved and the commit that's ultimately merged got the > > fixes, see this subthread: > > > > https://lore.kernel.org/all/0b36128c-3e12-77df-85fe-a153a714569b@quicinc.com/ > > > > >> > > CPU0 CPU1 > > >> > > ---- ---- > > >> > > cpus_read_lock() > > >> > > kn->active++ > > >> > > cpus_read_lock() [a] > > >> > > wait until kn->active == 0 > > >> > > > > >> > > Although cpu_hotplug_lock is a RWSEM, [a] will not block in there. But as > > >> > > lockdep annotations are added for cpu_hotplug_lock, a deadlock warning > > >> > > would be detected: > > > > > > The cpu_hotplug_lock is a per-CPU RWSEM. The lock in [a] will block if > > > there is a writer pending. > > > > > >> > > ====================================================== > > >> > > WARNING: possible circular locking dependency detected > > >> > > ------------------------------------------------------ > > >> > > dmsetup/1832 is trying to acquire lock: > > >> > > ffff986f5a0f9f20 (kn->count#144){++++}-{0:0}, at: kernfs_remove+0x1d/0x30 > > >> > > > > >> > > but task is already holding lock: > > >> > > ffffffffa43817c0 (slab_mutex){+.+.}-{3:3}, at: kmem_cache_destroy+0x2a/0x120 > > >> > > > > > > > > I tried to create & destroy a cryptarget which creates/destroy a cache > > > via bio_put_slab(). Either the callchain is different or something else > > > is but I didn't see a lockdep warning. > > > > RHEL-8 kernel seems to be 4.18, unless RT uses a newer one. Could be some > > silently relevant backport is missing? How about e.g. 59450bbc12be ("mm, > > slab, slub: stop taking cpu hotplug lock") ? > > Hummm, looks like we have backported commit 59450bbc12be in RHEL-8. > > Xin Long, would you be able to check if you still see the lockdep splat > with latest upstream RT? > > git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git linux-5.16.y-rt Hi, Juri, Thanks for sharing the RT kernel repo. I just tried with this kernel, and I couldn't reproduce it on my env. But I don't see how the upstream RT kernel can avoid the call trace. As this warning was triggered when the system was shutting down, it might not be reproduced on it due to some timing change. > > Thanks! > Juri >