From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80A36C433EF for ; Thu, 19 May 2022 18:30:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1467A6B007B; Thu, 19 May 2022 14:30:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CF256B007D; Thu, 19 May 2022 14:30:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E66336B007E; Thu, 19 May 2022 14:30:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D737B6B007B for ; Thu, 19 May 2022 14:30:37 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id AD5FF120D3C for ; Thu, 19 May 2022 18:30:37 +0000 (UTC) X-FDA: 79483333314.19.5EF0EA9 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf04.hostedemail.com (Postfix) with ESMTP id 79CF44000E for ; Thu, 19 May 2022 18:30:23 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id pq9-20020a17090b3d8900b001df622bf81dso5973530pjb.3 for ; Thu, 19 May 2022 11:30:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UmD3lcFeTPDWHVGorCCwoEXIVzcsZGCT2BZYAichU1Y=; b=oY+dcEgNEIo9g6YqyXfxa63f/ZoyciRGzpdYhqxPmlGdmu4UQhDp5GSrZVEekt9cBe aleWBsFQEgfuiiGNK9Qy/YGRavjN8QNttJmZTKDfcyUN77iJOC8n1IEfzfWwmYR4QSkb A5dbTP0JAKWckixtpadpAGtfgOrUbdXiEJHAiYu5FtYDQ4zIWzth5qNKm+dTvlId4ow+ NyVzELN+Ux+WowpkTTgAvo4QYSQ3XbPWUIdix32vBd5x4b6Dzo/bdu4zBw2V6JZvoko5 b6kZYT5eK2OWI8AshzsQXn3D7kp8hnzwbYXgt7chaULMqKqU5qcjZT4ujoTvz/2JD9/d ucnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UmD3lcFeTPDWHVGorCCwoEXIVzcsZGCT2BZYAichU1Y=; b=URPFKLApdlPdN9GK9lfbyShCbANfdX/vmwgnPO133Q/ZqFo8Y6jsHLq6LZUWrPcrsW q9SXLullIKc1QQW1PNpPOv0bPnX13c6ArBsElFxExJqlPLG6QcWCzVz8o3EVQyT+gM75 2cEasbjdZNg0QQldjBtB5lxvT4zRi0x48s8Kd08mzkn3U5mIRyOtSptz7zGntnO+/PxH MlwYE1veTV0LsH2w4h9R3D8IHMIDJ9mLIYN0WAJlVD2cjYZbINgDs4ZfMeEWa0V62H5B 4NaabJ1dOJuzU/WwJiXnvO4D/tz3U6uGKsDn7tY2vzjaihc1U4247m0ex9hkWx31Gmx5 Oohg== X-Gm-Message-State: AOAM532KObSEH/zZpyXqjoAehW2BXW7D6MbKwoXNC1evuX1Zb7Y5cvRR v92De4wts7EnU+3bEAZ80Pw= X-Google-Smtp-Source: ABdhPJzcgtA40HpIUmFEvX1Y8bcQpMq8LvSrxZ0XWDgxCPWU8IRvzIFvnpxFPWLY1xszYrbDzx6oEA== X-Received: by 2002:a17:90a:7441:b0:1df:5f54:502c with SMTP id o1-20020a17090a744100b001df5f54502cmr7132196pjk.129.1652985036178; Thu, 19 May 2022 11:30:36 -0700 (PDT) Received: from archlinux.localdomain ([140.121.198.213]) by smtp.googlemail.com with ESMTPSA id z5-20020a63e105000000b003c14af505f6sm3884674pgh.14.2022.05.19.11.30.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:30:35 -0700 (PDT) From: Chih-En Lin To: Andrew Morton , linux-mm@kvack.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Christian Brauner , "Matthew Wilcox (Oracle)" , Vlastimil Babka , William Kucharski , John Hubbard , Yunsheng Lin , Arnd Bergmann , Suren Baghdasaryan , Chih-En Lin , Colin Cross , Feng Tang , "Eric W. Biederman" , Mike Rapoport , Geert Uytterhoeven , Anshuman Khandual , "Aneesh Kumar K.V" , Daniel Axtens , Jonathan Marek , Christophe Leroy , Pasha Tatashin , Peter Xu , Andrea Arcangeli , Thomas Gleixner , Andy Lutomirski , Sebastian Andrzej Siewior , Fenghua Yu , David Hildenbrand , linux-kernel@vger.kernel.org, Kaiyang Zhao , Huichun Feng , Jim Huang Subject: [RFC PATCH 5/6] mm, pgtable: Add the reference counter for COW PTE Date: Fri, 20 May 2022 02:31:26 +0800 Message-Id: <20220519183127.3909598-6-shiyn.lin@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220519183127.3909598-1-shiyn.lin@gmail.com> References: <20220519183127.3909598-1-shiyn.lin@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=oY+dcEgN; spf=pass (imf04.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.216.45 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 79CF44000E X-Stat-Signature: akzwnbi39q7rydtjh9zwiwht79hf9mgx X-HE-Tag: 1652985023-376778 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add the reference counter cow_pgtable_refcount to maintain the number of process references to COW PTE. Before decreasing the reference count, it will check whether the counter is one or not for reusing COW PTE when the counter is one. Signed-off-by: Chih-En Lin --- include/linux/mm.h | 1 + include/linux/mm_types.h | 1 + include/linux/pgtable.h | 27 +++++++++++++++++++++++++++ mm/memory.c | 1 + 4 files changed, 30 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 221926a3d818..e48bb3fbc33c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2329,6 +2329,7 @@ static inline bool pgtable_pte_page_ctor(struct page *page) __SetPageTable(page); inc_lruvec_page_state(page, NR_PAGETABLE); page->cow_pte_owner = NULL; + atomic_set(&page->cow_pgtable_refcount, 1); return true; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5dcbd7f6c361..984d81e47d53 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -221,6 +221,7 @@ struct page { #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS int _last_cpupid; #endif + atomic_t cow_pgtable_refcount; /* COW page table */ pmd_t *cow_pte_owner; /* cow pte: pmd */ } _struct_page_alignment; diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index faca57af332e..33c01fec7b92 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -604,6 +604,33 @@ static inline bool cow_pte_owner_is_same(pmd_t *pmd, pmd_t *owner) true : false; } +extern void cow_pte_fallback(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr); + +static inline int pmd_get_pte(pmd_t *pmd) +{ + return atomic_inc_return(&pmd_page(*pmd)->cow_pgtable_refcount); +} + +/* If the COW PTE page->cow_pgtable_refcount is 1, instead of decreasing the + * counter, clear write protection of the corresponding PMD entry and reset + * the COW PTE owner to reuse the table. + */ +static inline int pmd_put_pte(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr) +{ + if (!atomic_add_unless(&pmd_page(*pmd)->cow_pgtable_refcount, -1, 1)) { + cow_pte_fallback(vma, pmd, addr); + return 1; + } + return 0; +} + +static inline int cow_pte_refcount_read(pmd_t *pmd) +{ + return atomic_read(&pmd_page(*pmd)->cow_pgtable_refcount); +} + #ifndef pte_access_permitted #define pte_access_permitted(pte, write) \ (pte_present(pte) && (!(write) || pte_write(pte))) diff --git a/mm/memory.c b/mm/memory.c index dcb678cbb051..aa66af76e214 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4597,6 +4597,7 @@ void cow_pte_fallback(struct vm_area_struct *vma, pmd_t *pmd, pmd_t new; BUG_ON(pmd_write(*pmd)); + BUG_ON(cow_pte_refcount_read(pmd) != 1); start = addr & PMD_MASK; end = (addr + PMD_SIZE) & PMD_MASK; -- 2.36.1