From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2A0B0C43458 for ; Mon, 6 Jul 2026 06:21:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC13A6B00D6; Mon, 6 Jul 2026 02:20:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D98C66B00D7; Mon, 6 Jul 2026 02:20:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3B816B00D8; Mon, 6 Jul 2026 02:20:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 88D816B00D6 for ; Mon, 6 Jul 2026 02:20:39 -0400 (EDT) Received: from smtpin01.hostedemail.com (lb01a-stub [10.200.18.249]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EA07B16870E for ; Mon, 6 Jul 2026 06:20:38 +0000 (UTC) X-FDA: 84957352956.01.46FD402 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf13.hostedemail.com (Postfix) with ESMTP id DBED220002 for ; Mon, 6 Jul 2026 06:20:36 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=sk.com; spf=pass (imf13.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1783318837; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=MstF8XFF3bnboLjOfugxBpZu1QiAVXIcKGowHBZxBjE=; b=zECt+6osFu4O5+ipctQ+JtmdzA+G+gMfuIBYpBs0cNXAYID8/0sjWYQJMiUgeb3TBn5uI6 W8+Dlm7ksoyhiBToqhb62P/BeFVlhsK7rN2nv95znYCa25FSVWX29+Jd2Jmz0GmOwdcItk buYIExljxlNzBWhmidmyJ2/tSehaofQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=sk.com; spf=pass (imf13.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; a=rsa-sha256; d=hostedemail.com; s=arc-20220608; cv=none; t=1783318837; b=T3uFO3YvmnxNb6jFbecGDyfW3oKwiimiXyhgnNm2ybo7nYJRQsk8udk81mxb6wceavL3z0 mcVB89QqJGHioo9IlaqGJwNXRHJ7blVxxPbKunz5tNcTQqIgftkD7su6dcIxGdOpxGwHr9 tBnyczjVrcGH8h3si9xQhDaSzwfEjOg= X-AuditID: a67dfc5b-c45ff70000001609-da-6a4b4900ec5c From: Byungchul Park To: linux-kernel@vger.kernel.org Cc: max.byungchul.park@gmail.com, kernel_team@skhynix.com, torvalds@linux-foundation.org, damien.lemoal@opensource.wdc.com, linux-ide@vger.kernel.org, adilger.kernel@dilger.ca, linux-ext4@vger.kernel.org, mingo@redhat.com, peterz@infradead.org, will@kernel.org, tglx@linutronix.de, rostedt@goodmis.org, joel@joelfernandes.org, sashal@kernel.org, daniel.vetter@ffwll.ch, duyuyang@gmail.com, johannes.berg@intel.com, tj@kernel.org, tytso@mit.edu, willy@infradead.org, david@fromorbit.com, amir73il@gmail.com, gregkh@linuxfoundation.org, kernel-team@lge.com, linux-mm@kvack.org, akpm@linux-foundation.org, mhocko@kernel.org, minchan@kernel.org, hannes@cmpxchg.org, vdavydov.dev@gmail.com, sj@kernel.org, jglisse@redhat.com, dennis@kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, vbabka@suse.cz, ngupta@vflare.org, linux-block@vger.kernel.org, josef@toxicpanda.com, linux-fsdevel@vger.kernel.org, jack@suse.cz, jlayton@kernel.org, dan.j.williams@intel.com, hch@infradead.org, djwong@kernel.org, dri-devel@lists.freedesktop.org, rodrigosiqueiramelo@gmail.com, melissa.srw@gmail.com, hamohammed.sa@gmail.com, harry.yoo@oracle.com, chris.p.wilson@intel.com, gwan-gyeong.mun@intel.com, boqun.feng@gmail.com, longman@redhat.com, yunseong.kim@ericsson.com, ysk@kzalloc.com, yeoreum.yun@arm.com, netdev@vger.kernel.org, matthew.brost@intel.com, her0gyugyu@gmail.com, corbet@lwn.net, catalin.marinas@arm.com, bp@alien8.de, x86@kernel.org, hpa@zytor.com, luto@kernel.org, sumit.semwal@linaro.org, gustavo@padovan.org, christian.koenig@amd.com, andi.shyti@kernel.org, arnd@arndb.de, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, rppt@kernel.org, surenb@google.com, mcgrof@kernel.org, petr.pavlu@suse.com, da.gomez@kernel.org, samitolvanen@google.com, paulmck@kernel.org, frederic@kernel.org, neeraj.upadhyay@kernel.org, joelagnelf@nvidia.com, josh@joshtriplett.org, urezki@gmail.com, mathieu.desnoyers@efficios.com, jiangshanlai@gmail.com, qiang.zhang@linux.dev, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, chuck.lever@oracle.com, neil@brown.name, okorniev@redhat.com, Dai.Ngo@oracle.com, tom@talpey.com, trondmy@kernel.org, anna@kernel.org, kees@kernel.org, bigeasy@linutronix.de, clrkwllms@kernel.org, mark.rutland@arm.com, ada.coupriediaz@arm.com, kristina.martsenko@arm.com, wangkefeng.wang@huawei.com, broonie@kernel.org, kevin.brodsky@arm.com, dwmw@amazon.co.uk, shakeel.butt@linux.dev, ast@kernel.org, ziy@nvidia.com, yuzhao@google.com, baolin.wang@linux.alibaba.com, usamaarif642@gmail.com, joel.granados@kernel.org, richard.weiyang@gmail.com, geert+renesas@glider.be, tim.c.chen@linux.intel.com, linux@treblig.org, alexander.shishkin@linux.intel.com, lillian@star-ark.net, chenhuacai@kernel.org, francesco@valla.it, guoweikang.kernel@gmail.com, link@vivo.com, jpoimboe@kernel.org, masahiroy@kernel.org, brauner@kernel.org, thomas.weissschuh@linutronix.de, oleg@redhat.com, mjguzik@gmail.com, andrii@kernel.org, wangfushuai@baidu.com, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org, linux-i2c@vger.kernel.org, linux-arch@vger.kernel.org, linux-modules@vger.kernel.org, rcu@vger.kernel.org, linux-nfs@vger.kernel.org, linux-rt-devel@lists.linux.dev, 2407018371@qq.com, dakr@kernel.org, miguel.ojeda.sandonis@gmail.com, neilb@ownmail.net, bagasdotme@gmail.com, wsa+renesas@sang-engineering.com, dave.hansen@intel.com, geert@linux-m68k.org, ojeda@kernel.org, alex.gaynor@gmail.com, gary@garyguo.net, bjorn3_gh@protonmail.com, lossin@kernel.org, a.hindborg@kernel.org, aliceryhl@google.com, tmgross@umich.edu, rust-for-linux@vger.kernel.org Subject: [PATCH v19 22/40] dept: track PG_locked with dept Date: Mon, 6 Jul 2026 15:19:10 +0900 Message-Id: <20260706061928.66713-23-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20260706061928.66713-1-byungchul@sk.com> References: <20260706061928.66713-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAAzWSbUxTZxTH99z73Jd2NrmpGJ9hNrYiw+h0c3HL2YuL2ZLt2TTLNqcx84NW eiNlUExRhC3LQFG6ChNYCsFaV4p2WCp27Rgvs8oI4mpGQBShvAhMUyUMMawtqUAYYPxy8sv5 n/M7X47Iqnu4eFFvOCgbDdp0Da/EyolljvXsx1vTXgv+TcB07HsYHL7Hwe38FgyRsAnD6Ytu HuasDQKYvJUc/NV7BENXXS2C6RkrC+HYgACWfAQVY/kYeqP/IrjXUojgZ4ePh/HyRzzMdHSy UDU6xMLU2AiC39rvIPDXHOHhVmSSh4DlBA8TN04z8NDDg81ahiDU72eg3ObF0DTSLMBgeRkD w84QXjjBg7XiKLNQHjBgufAHAzGnSwBnXhLMjm6EeXsmtNfeF2DopAVD3UQnB+OhMh6Grx3n oCFvRABv8CqC8K1RBtxFIRZMzREM/v51UHlmkIdL/gAG01wYQXvjPwwUeeo5uOOe56C7tgtD 4NR5DAMdJwU413uDgeiPq6CrtJiDnx6G0BYddfl+Z6j7jBvR8LmjLC3wHaaPIz089UftmF53 EFrasZ42nRoSqN17iBa0TXDUV7OWVl8aY2jVVISj/eObqdf1A//ZK18p39XJ6fps2fjqe3uV qZXHr+IDVzJzRksgDxXuNiOFSKRNpPFyFD/l0nrPEvNSMunri7GLHCe9SHzFIc6MlCIr3Uwg Rc7ipaHl0tskVNgtLDKWkohtLsaYkSiqpDfJtIM+cSaQWk/Lkkex0K6btTGLrJbeIAOT1/Ci k0jVCvKgMsA8WXiO/FnTh0uQyo6ecSG13pCdodWnb9qQmmvQ52xIyczwooVnc343u7sRTXVt b0WSiDTLVLDlkzQ1p83Oys1oRURkNXGql9dsTVOrdNrcb2Rj5h7joXQ5qxWtErFmper16GGd WtqvPSh/LcsHZOPTlBEV8XnIfuXEmvh1L+mnYjMrUpOfn0/BL5SKvc9+u8ui++gdUm/eZzIl Xv+l0Nb4ZcbqlA8mm3fsStoX3B63t6cqkmxbvafNE0wMrnirs6pO+18F02a+P/eoYWX22ffv fpHTQ4Y/9CV+Wn35V2uw6TwuufD5ePfZbYr9CdN3A1LHsQLiMjh2NmhwVqp241rWmKX9H1rs 1bBoAwAA X-Brightmail-Tracker: H4sIAAAAAAAAAzWSe2xLcRTH/e799fauUbmpJW6IR5pgiMeEOd6TCL8Ri4REgqDsWrsnLbN5 xLqp1QxdaZetG9NZI9swK2Ok2TI2896MWTDdpMpsVGoPe3TTEf+cfM73fM8354/D0rJy0XhW FXdAUMcpYuSMBEvCl6bOHhW2Pmqe90Iw6HXH4YPTJYImbRWG7i49htwbJQz4LHfEoC/LFkHd 2xQM9deLETi79Qh6Byw06CqGMfiMtWLo6nsvBpMWwbCjFoG5wUhDc30lDSW3tBT8Kh1ioOOB F4GpzcVAVrsWg8eWgSDHbRFDe81a+O68L4Lhli8UvO3pRGBzDVHgqkpD4DNHwyWr3b9u/snA wPOXNGSZ6hFcbmuhwdveiuBW7UcEjqspDHw23Kah0TUGXnd7GHhsOs3A94ZcCn6UMpCf4hBB w7MOBHkWIwL3OwcFqQU3GDDnlWGoaL0nhoaOQQo+mI0UFJdtAKfNjeGpwUr5z/W7bo4DS1Yq 5S9fKTBdu09Bn61IHFqISK/uLCZF9nKK6F75GFJysQSRgX4jIl2FqTTRGfztg04PTU7YD5HC p50M6e9+wxBHTz4mT6w8uXKqnyKZz2eTipwW8cZVWyXLIoQYVYKgnrtil0SZfbIG76uMT2wz QDJK25aOAlieW8Bn3i7FI8xw0/nm5j56hAO5Kbz9jFuUjiQszTVO5jNsZ/6axnJLeHfaK/EI Y24qn+fro9IRy0q5EL7XSv5lTuaLS6v+5gT45euDedQIy7iF/HvPI2xAknw0qggFquISYhWq mIVzNNHKpDhV4pw98bFlyP9NtmODmXdRV+PaasSxSD5aCqHromQiRYImKbYa8SwtD5ROC1of JZNGKJIOC+r4neqDMYKmGk1gsXycdN0WYZeMi1QcEKIFYZ+g/j+l2IDxyWjlw5f911Z+QrvX aMNqyIvlXpsnhHbanbP4HbnZ58InjP3C7Z6ywdV71Bm6+ePPucvtVcq6kO2nzk/aGWy6a206 frKuwBZWEBn5bWJ4ZYayFfNBPcmJh35vayGbElYnFcTWT0o8+3iRtZxWtc04d2T+UNjiwgFv 0/nOvTq5PiJof6Qca5SK4Jm0WqP4AymxBXBJAwAA X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: DBED220002 X-Stat-Signature: jsbmu7ug1nx1jthdem5bya5jh8jzrag6 X-HE-Tag: 1783318836-213015 X-HE-Meta: U2FsdGVkX1/yRH36VuZkd5lyrAXZd1xFgSQLw0PwfBTQWMsnooUmxGvTNbvxgnwJr4xeiLCdya/Bo1YGEvqfqtIAF3bfVvyyQmmeXf6Isnjbf5dpR24O2FgdZUWhRGk1bFLm8Gx47wHxDxXvUlFoUlsqoqs9hyZNgLc9tPMTL9DxGONrrRoGSn9dMbT4bdRM3Yq0hz3txid9qWqB92Giqwty3iNBrvbwrjBXmttKg4Q9rZfwk/DVi4gkk2YKAtHvAL5EAfI100TT7CErUp7xOrV7Ir6xWwjkccp5ru52wEOA7vLOagjsKZTqDot2AC3qKqpKXB+BB65Swh8aE0bJLk3IV7m+QOf+rbS8Ai9vob0mzi6/U/jBagmEaTqPfMeOYq/xwHsEI3xw1cN8BChIIKNYRdSJoV+4s3QagnSIEnBwkcCHCAt33MD+nkJO8au+RaA1COewz9PRwwlekGxvSfxHLzbNmD9Xo3gNpLIe3abaFO04A05iplp4DPmy5l+t3voB967H7VbCFBv+teb9DV/yhn8W//mldstQcwdqmV+5Ji1qd19UqtgllZyxJyVylSjYLiPMkDyOceyXVMT3/qvHcYPHO4Hh4s09I13UmGyOyyl72V1uZBNejIq6ft2X/o1pD0pJg9W9kpKzGbIJCfkTf8j8fhEBSgJAl+fd1Zc9oiDoGtqCUOl+D1uO4wg4zF96R3+X7slXPuj2wM9QqkCRsD3zdFND6aJMqYQeh6/8pbt6Svw98x38hEapgFwxm+qXfCjjtJXKmaNqt8MRNEH+scJMSe/t2rL1ZdAXNEVMKUsrP+2Fa3PnS/HaP+fw8GvGzmxB8Ozt94qLozu3A9TAvChlFiLD+dEUlEaCYeLe3aXMQVFkdrUp7Celkx2SYbIZl2iqGqMzbRReYHhw1AQAFWZ7fvX+/byPJ9qmqPWmq+5vyA9wpS7Hnd73GWVgrWkiYmAc1xxYppDhJNE GuCByM5/ NdPs2/HITqGsywra1cMmkL6UBun5w74P65NFiTH2p4ZnEb3kT759fE94sqKzbGZt3Atw2Jk2y7WNUfyGEKq4jUWzCIsb3ED/UeIz2bcwFd/ybIUKbbtunst/kPYW4S7va21JeHFOzyMYAh/PJjMZfp1l8Y2zQwbJORnMQC8HJ28PQRQfsTop62ukLwgH56QtgAVY9iRlh2m/3F4f6P2Y7KrtYTV7c4pDs2ZptWPeDyZ8yC6jHO+ytgFndYEKBBWaG/loYI6jOrM5ndh2LBK0tQNhv+tV+xf//Q43A Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Makes dept able to track PG_locked waits and events, which will be useful in practice. See the following link that shows dept worked with PG_locked and detected real issues in practice: https://lore.kernel.org/lkml/1674268856-31807-1-git-send-email-byungchul.park@lge.com/ Signed-off-by: Byungchul Park --- include/linux/mm_types.h | 2 + include/linux/page-flags.h | 125 +++++++++++++++++++++++++++++++++---- include/linux/pagemap.h | 37 ++++++++++- mm/filemap.c | 34 ++++++++++ mm/mm_init.c | 2 + 5 files changed, 187 insertions(+), 13 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3cc8ae722886..5b3f54ee0d38 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -22,6 +22,7 @@ #include #include #include +#include #include @@ -219,6 +220,7 @@ struct page { struct page *kmsan_shadow; struct page *kmsan_origin; #endif + struct dept_ext_wgen pg_locked_wgen; } _struct_page_alignment; /* diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index f7a0e4af0c73..8ab39823ea31 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -198,6 +198,61 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H +#ifdef CONFIG_DEPT +#include +#include + +extern struct dept_map pg_locked_map; + +/* + * Place the following annotations in its suitable point in code: + * + * Annotate dept_page_set_bit() around firstly set_bit*() + * Annotate dept_page_clear_bit() around clear_bit*() + * Annotate dept_page_wait_on_bit() around wait_on_bit*() + */ + +static inline void dept_page_set_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_request_event(&pg_locked_map, &p->pg_locked_wgen); +} + +static inline void dept_page_clear_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_event(&pg_locked_map, 1UL, _RET_IP_, __func__, &p->pg_locked_wgen); +} + +static inline void dept_page_wait_on_bit(struct page *p, int bit_nr) +{ + if (bit_nr == PG_locked) + dept_wait(&pg_locked_map, 1UL, _RET_IP_, __func__, 0, -1L); +} + +static inline void dept_folio_set_bit(struct folio *f, int bit_nr) +{ + dept_page_set_bit(&f->page, bit_nr); +} + +static inline void dept_folio_clear_bit(struct folio *f, int bit_nr) +{ + dept_page_clear_bit(&f->page, bit_nr); +} + +static inline void dept_folio_wait_on_bit(struct folio *f, int bit_nr) +{ + dept_page_wait_on_bit(&f->page, bit_nr); +} +#else +#define dept_page_set_bit(p, bit_nr) do { } while (0) +#define dept_page_clear_bit(p, bit_nr) do { } while (0) +#define dept_page_wait_on_bit(p, bit_nr) do { } while (0) +#define dept_folio_set_bit(f, bit_nr) do { } while (0) +#define dept_folio_clear_bit(f, bit_nr) do { } while (0) +#define dept_folio_wait_on_bit(f, bit_nr) do { } while (0) +#endif + #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP DECLARE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); @@ -419,27 +474,51 @@ static __always_inline bool folio_test_##name(const struct folio *folio) \ #define FOLIO_SET_FLAG(name, page) \ static __always_inline void folio_set_##name(struct folio *folio) \ -{ set_bit(PG_##name, folio_flags(folio, page)); } +{ \ + set_bit(PG_##name, folio_flags(folio, page)); \ + dept_folio_set_bit(folio, PG_##name); \ +} #define FOLIO_CLEAR_FLAG(name, page) \ static __always_inline void folio_clear_##name(struct folio *folio) \ -{ clear_bit(PG_##name, folio_flags(folio, page)); } +{ \ + clear_bit(PG_##name, folio_flags(folio, page)); \ + dept_folio_clear_bit(folio, PG_##name); \ +} #define __FOLIO_SET_FLAG(name, page) \ static __always_inline void __folio_set_##name(struct folio *folio) \ -{ __set_bit(PG_##name, folio_flags(folio, page)); } +{ \ + __set_bit(PG_##name, folio_flags(folio, page)); \ + dept_folio_set_bit(folio, PG_##name); \ +} #define __FOLIO_CLEAR_FLAG(name, page) \ static __always_inline void __folio_clear_##name(struct folio *folio) \ -{ __clear_bit(PG_##name, folio_flags(folio, page)); } +{ \ + __clear_bit(PG_##name, folio_flags(folio, page)); \ + dept_folio_clear_bit(folio, PG_##name); \ +} #define FOLIO_TEST_SET_FLAG(name, page) \ static __always_inline bool folio_test_set_##name(struct folio *folio) \ -{ return test_and_set_bit(PG_##name, folio_flags(folio, page)); } +{ \ + bool __ret = test_and_set_bit(PG_##name, folio_flags(folio, page)); \ + \ + if (!__ret) \ + dept_folio_set_bit(folio, PG_##name); \ + return __ret; \ +} #define FOLIO_TEST_CLEAR_FLAG(name, page) \ static __always_inline bool folio_test_clear_##name(struct folio *folio) \ -{ return test_and_clear_bit(PG_##name, folio_flags(folio, page)); } +{ \ + bool __ret = test_and_clear_bit(PG_##name, folio_flags(folio, page)); \ + \ + if (__ret) \ + dept_folio_clear_bit(folio, PG_##name); \ + return __ret; \ +} #define FOLIO_FLAG(name, page) \ FOLIO_TEST_FLAG(name, page) \ @@ -454,32 +533,54 @@ static __always_inline int Page##uname(const struct page *page) \ #define SETPAGEFLAG(uname, lname, policy) \ FOLIO_SET_FLAG(lname, FOLIO_##policy) \ static __always_inline void SetPage##uname(struct page *page) \ -{ set_bit(PG_##lname, &policy(page, 1)->flags.f); } +{ \ + set_bit(PG_##lname, &policy(page, 1)->flags.f); \ + dept_page_set_bit(page, PG_##lname); \ +} #define CLEARPAGEFLAG(uname, lname, policy) \ FOLIO_CLEAR_FLAG(lname, FOLIO_##policy) \ static __always_inline void ClearPage##uname(struct page *page) \ -{ clear_bit(PG_##lname, &policy(page, 1)->flags.f); } +{ \ + clear_bit(PG_##lname, &policy(page, 1)->flags.f); \ + dept_page_clear_bit(page, PG_##lname); \ +} #define __SETPAGEFLAG(uname, lname, policy) \ __FOLIO_SET_FLAG(lname, FOLIO_##policy) \ static __always_inline void __SetPage##uname(struct page *page) \ -{ __set_bit(PG_##lname, &policy(page, 1)->flags.f); } +{ \ + __set_bit(PG_##lname, &policy(page, 1)->flags.f); \ + dept_page_set_bit(page, PG_##lname); \ +} #define __CLEARPAGEFLAG(uname, lname, policy) \ __FOLIO_CLEAR_FLAG(lname, FOLIO_##policy) \ static __always_inline void __ClearPage##uname(struct page *page) \ -{ __clear_bit(PG_##lname, &policy(page, 1)->flags.f); } +{ \ + __clear_bit(PG_##lname, &policy(page, 1)->flags.f); \ + dept_page_clear_bit(page, PG_##lname); \ +} #define TESTSETFLAG(uname, lname, policy) \ FOLIO_TEST_SET_FLAG(lname, FOLIO_##policy) \ static __always_inline int TestSetPage##uname(struct page *page) \ -{ return test_and_set_bit(PG_##lname, &policy(page, 1)->flags.f); } +{ \ + bool ret = test_and_set_bit(PG_##lname, &policy(page, 1)->flags.f);\ + if (!ret) \ + dept_page_set_bit(page, PG_##lname); \ + return ret; \ +} #define TESTCLEARFLAG(uname, lname, policy) \ FOLIO_TEST_CLEAR_FLAG(lname, FOLIO_##policy) \ static __always_inline int TestClearPage##uname(struct page *page) \ -{ return test_and_clear_bit(PG_##lname, &policy(page, 1)->flags.f); } +{ \ + bool ret = test_and_clear_bit(PG_##lname, &policy(page, 1)->flags.f);\ + if (ret) \ + dept_page_clear_bit(page, PG_##lname); \ + return ret; \ +} #define PAGEFLAG(uname, lname, policy) \ TESTPAGEFLAG(uname, lname, policy) \ diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 31a848485ad9..6605800ba3ad 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1119,7 +1119,12 @@ void folio_unlock(struct folio *folio); */ static inline bool folio_trylock(struct folio *folio) { - return likely(!test_and_set_bit_lock(PG_locked, folio_flags(folio, 0))); + bool ret = !test_and_set_bit_lock(PG_locked, folio_flags(folio, 0)); + + if (ret) + dept_page_set_bit(&folio->page, PG_locked); + + return likely(ret); } /* @@ -1155,6 +1160,16 @@ static inline bool trylock_page(struct page *page) static inline void folio_lock(struct folio *folio) { might_sleep(); + + /* + * dept_page_wait_on_bit() will be called if __folio_lock() goes + * through a real wait path. However, for better job to detect + * *potential* deadlocks, let's assume that folio_lock() always + * goes through wait so that dept can take into account all the + * potential cases. + */ + dept_page_wait_on_bit(&folio->page, PG_locked); + if (!folio_trylock(folio)) __folio_lock(folio); } @@ -1175,6 +1190,15 @@ static inline void lock_page(struct page *page) struct folio *folio; might_sleep(); + /* + * dept_page_wait_on_bit() will be called if __folio_lock() goes + * through a real wait path. However, for better job to detect + * *potential* deadlocks, let's assume that lock_page() always + * goes through wait so that dept can take into account all the + * potential cases. + */ + dept_page_wait_on_bit(page, PG_locked); + folio = page_folio(page); if (!folio_trylock(folio)) __folio_lock(folio); @@ -1193,6 +1217,17 @@ static inline void lock_page(struct page *page) static inline int folio_lock_killable(struct folio *folio) { might_sleep(); + + /* + * dept_page_wait_on_bit() will be called if + * __folio_lock_killable() goes through a real wait path. + * However, for better job to detect *potential* deadlocks, + * let's assume that folio_lock_killable() always goes through + * wait so that dept can take into account all the potential + * cases. + */ + dept_page_wait_on_bit(&folio->page, PG_locked); + if (!folio_trylock(folio)) return __folio_lock_killable(folio); return 0; diff --git a/mm/filemap.c b/mm/filemap.c index 3c1e785542dd..c0bff1e08a28 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include "internal.h" @@ -1151,6 +1152,7 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, if (flags & WQ_FLAG_CUSTOM) { if (test_and_set_bit(key->bit_nr, &key->folio->flags.f)) return -1; + dept_page_set_bit(&key->folio->page, key->bit_nr); flags |= WQ_FLAG_DONE; } } @@ -1191,6 +1193,13 @@ static void folio_wake_bit(struct folio *folio, int bit_nr) key.bit_nr = bit_nr; key.page_match = 0; + /* + * dept_page_clear_bit() being called multiple times is harmless. + * The worst case is to miss some dependencies but it's okay. + */ + if (bit_nr == PG_locked) + dept_page_clear_bit(&folio->page, bit_nr); + spin_lock_irqsave(&q->lock, flags); __wake_up_locked_key(q, TASK_NORMAL, &key); @@ -1234,6 +1243,7 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, if (wait->flags & WQ_FLAG_EXCLUSIVE) { if (test_and_set_bit(bit_nr, &folio->flags.f)) return false; + dept_page_set_bit(&folio->page, bit_nr); } else if (test_bit(bit_nr, &folio->flags.f)) return false; @@ -1241,6 +1251,9 @@ static inline bool folio_trylock_flag(struct folio *folio, int bit_nr, return true; } +struct dept_map __maybe_unused pg_locked_map = DEPT_MAP_INITIALIZER(pg_locked_map, NULL); +EXPORT_SYMBOL(pg_locked_map); + static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, int state, enum behavior behavior) { @@ -1252,6 +1265,8 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, unsigned long pflags; bool in_thrashing; + dept_page_wait_on_bit(&folio->page, bit_nr); + if (bit_nr == PG_locked && !folio_test_uptodate(folio) && folio_test_workingset(folio)) { delayacct_thrashing_start(&in_thrashing); @@ -1345,6 +1360,23 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, break; } + /* + * dept_page_set_bit() might have been called already in + * folio_trylock_flag(), wake_page_function() or somewhere. + * However, call it again to reset the wgen of dept to ensure + * dept_page_wait_on_bit() is called prior to + * dept_page_set_bit(). + * + * Remind dept considers all the waits between + * dept_page_set_bit() and dept_page_clear_bit() as potential + * event disturbers. Ensure the correct sequence so that dept + * can make correct decisions: + * + * wait -> acquire(set bit) -> release(clear bit) + */ + if (wait->flags & WQ_FLAG_DONE) + dept_page_set_bit(&folio->page, bit_nr); + /* * If a signal happened, this 'finish_wait()' may remove the last * waiter from the wait-queues, but the folio waiters bit will remain @@ -1507,6 +1539,7 @@ void folio_unlock(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); BUILD_BUG_ON(PG_locked > 7); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + dept_page_clear_bit(&folio->page, PG_locked); if (folio_xor_flags_has_waiters(folio, 1 << PG_locked)) folio_wake_bit(folio, PG_locked); } @@ -1537,6 +1570,7 @@ void folio_end_read(struct folio *folio, bool success) if (likely(success)) mask |= 1 << PG_uptodate; + dept_page_clear_bit(&folio->page, PG_locked); if (folio_xor_flags_has_waiters(folio, mask)) folio_wake_bit(folio, PG_locked); } diff --git a/mm/mm_init.c b/mm/mm_init.c index df34797691bd..66b68c02d8d4 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -32,6 +32,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -587,6 +588,7 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn, atomic_set(&page->_mapcount, -1); page_cpupid_reset_last(page); page_kasan_tag_reset(page); + dept_ext_wgen_init(&page->pg_locked_wgen); INIT_LIST_HEAD(&page->lru); #ifdef WANT_PAGE_VIRTUAL -- 2.17.1