From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C12A8C35274 for ; Mon, 18 Dec 2023 16:26:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B9476B0075; Mon, 18 Dec 2023 11:26:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5422D6B0078; Mon, 18 Dec 2023 11:26:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BC356B007B; Mon, 18 Dec 2023 11:26:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 263F06B0075 for ; Mon, 18 Dec 2023 11:26:49 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CAEAEC08D6 for ; Mon, 18 Dec 2023 16:26:48 +0000 (UTC) X-FDA: 81580467696.25.A7011C1 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id EF125180016 for ; Mon, 18 Dec 2023 16:26:46 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702916807; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+tM1/bMdGeKYKRzZy4Vk7VhGJqk0w6SKI2oJigto+gw=; b=ewAQWZgHirt8Vn1PvR/e0Ad3CLOvC1QGyU1kXwqAG08Y675hiwmBuE/D922oXxDLh9hpik Nu/i6yjNzXfyZVkiiY3loryZGQbTBQ851GyrLCEZOw2l90hmOVapKOjuBAhARBMke53lY2 kLBYGNhvcZsh6toS8/XtzXYwlVQ+SQY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702916807; a=rsa-sha256; cv=none; b=HbpAxlgDzXxpL2wzbDVcHQhq96lbjOEWunOeWNp8UDwDJlQSmGojXfAromJyLjcn1Sc48T mibaUrvp60DOZASYCStgl0bow3ZRtAYllVTF4QkC5uJU+ygCKNVgCf2orJ4K0CIqyin4ST TC2NoXKMvJpCWz4o66uf/IxjQ5WkZNk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 93AD02F4; Mon, 18 Dec 2023 08:27:30 -0800 (PST) Received: from [10.57.75.230] (unknown [10.57.75.230]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C852E3F5A1; Mon, 18 Dec 2023 08:26:44 -0800 (PST) Message-ID: <593c4eb0-e430-4186-a95c-9d2ebd91235c@arm.com> Date: Mon, 18 Dec 2023 16:26:43 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1 14/39] mm/rmap: introduce folio_add_anon_rmap_[pte|ptes|pmd]() Content-Language: en-GB To: David Hildenbrand , linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu References: <20231211155652.131054-1-david@redhat.com> <20231211155652.131054-15-david@redhat.com> From: Ryan Roberts In-Reply-To: <20231211155652.131054-15-david@redhat.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: EF125180016 X-Rspam-User: X-Stat-Signature: y9go9rjdx3ohrkjwuiuqnppyc5zpwfsf X-Rspamd-Server: rspam01 X-HE-Tag: 1702916806-295459 X-HE-Meta: U2FsdGVkX1/RXDfzBYBI4CE8wFqwUMBhLoqe3Fk6+6f/312Ns0TIwPnTu6w3PicZ2He1FZd7v0yomPSxLDAdm4Xb59No4rumnAhMIp0ui/1r9IEDBtmJTwSPe3vrB18e9StFfRLKDL2ewcFTPex4/zFatuUWK3ldGhJPhwDUreXKZxuJXjS+Pwmpe841ZavbIOrq/vBQnjOkNgQlwWfzB2tvh7NcpTdEsK0nS7ZcjrSGU8k8D9LGTyic1XYhYcZGX5HzjiHGbCQTxYnSyFRQFblvudpKOzQZUR87Qy925zVV464SDOk0VnuDuFLvBjFu++PB5th+QLcoRh/GNwRsgxgb0L+rFfN/5/nHGyfOKrb4hpiEhy6/2mVDPYdR8KOcChRVToRs4u5UDCSvBi4cPo446CcVpPf30HBZCl8ng261YNqPdpcuS7t8vxLuZfrmoB6zBYjNMftV6yHdDaGvuUq5KJgYWoFf9cvBwkg2WPZXFFQ1DoXWi4YcYiRqFVs9HRoSfFtp0aYKoykHUcqhyNkNNk4vPaslJaA44YX/8v+dPCGOqzzObWfTDsLHZ2U+MTxwyCLp41ncoLG1rklNWagb2tekn3D/Swm2cjfjRzeXuQqeZBu+Fi/UvpaK5RibnY5TQ651D0+HURrkvZFWeHQZoMhY6fyZ3fLI2NAmUWhfYf7Q4LvJj9/O5oIHt67YWK37rZzfQyeaD+r+T853y06vvjku6oZmH8BkG6Hh9TdhmB/exVE6LhDZdZL2/VlmQ9TuW06sT54I9z/Zhx6uQ0hEJ2AN0sicQON2lJfmpn5c9Yf5oxSXyLL+KgWTZGgrW/A8TBbtw2Nh+cspBrNeLMpCUyfnaBfTKXbQ4W6Ou/zCW+LMe3/NsyULlBDbx1eIiHYtr4a2+5GvulVy7su6kc9XPa2IgIerYD3AysV39mxaY7xIHVUhuVR3Aq56DWHS5YfyPCsfhAizEXyvdwC uo2njVaD wwdhWFtK0DzjI1yJjBxM2iPWetuflXzBOZ9Te1GYj2MxoXS31G7Ng4P62AaulgPYle8xtNoQcZ18HJ4VIVHp38626y/yoZ7w71yUYo33wV/fm4cwCLxZPp9XEpXuHXS+GeCHC9lmV/ud3mrooEq7PdjUznFIUbdrIwWQSBwjhpxhAR26KwtQPAVj65TiqBRJx8/uHCJdtjmbGU/m9lX4WzWbE30HUyJ2Nvgy4y6WJnZH+kYyzUlhkE7i2GhZY5BiAw7VA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 11/12/2023 15:56, David Hildenbrand wrote: > Let's mimic what we did with folio_add_file_rmap_*() so we can similarly > replace page_add_anon_rmap() next. > > Make the compiler always special-case on the granularity by using > __always_inline. > > Note that the new functions ignore the RMAP_COMPOUND flag, which we will > remove as soon as page_add_anon_rmap() is gone. > > Signed-off-by: David Hildenbrand > --- > include/linux/rmap.h | 6 +++ > mm/rmap.c | 118 ++++++++++++++++++++++++++++++------------- > 2 files changed, 88 insertions(+), 36 deletions(-) > > diff --git a/include/linux/rmap.h b/include/linux/rmap.h > index 7198905dc8be..3b5357cb1c09 100644 > --- a/include/linux/rmap.h > +++ b/include/linux/rmap.h > @@ -234,6 +234,12 @@ static inline void __folio_rmap_sanity_checks(struct folio *folio, > * rmap interfaces called when adding or removing pte of page > */ > void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); > +void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, > + struct vm_area_struct *, unsigned long address, rmap_t flags); > +#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \ > + folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) > +void folio_add_anon_rmap_pmd(struct folio *, struct page *, > + struct vm_area_struct *, unsigned long address, rmap_t flags); > void page_add_anon_rmap(struct page *, struct vm_area_struct *, > unsigned long address, rmap_t flags); > void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, > diff --git a/mm/rmap.c b/mm/rmap.c > index c5761986a411..7787499fa2ad 100644 > --- a/mm/rmap.c > +++ b/mm/rmap.c > @@ -1300,38 +1300,20 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, > unsigned long address, rmap_t flags) > { > struct folio *folio = page_folio(page); > - atomic_t *mapped = &folio->_nr_pages_mapped; > - int nr = 0, nr_pmdmapped = 0; > - bool compound = flags & RMAP_COMPOUND; > - bool first; > > - /* Is page being mapped by PTE? Is this its first map to be added? */ > - if (likely(!compound)) { > - first = atomic_inc_and_test(&page->_mapcount); > - nr = first; > - if (first && folio_test_large(folio)) { > - nr = atomic_inc_return_relaxed(mapped); > - nr = (nr < COMPOUND_MAPPED); > - } > - } else if (folio_test_pmd_mappable(folio)) { > - /* That test is redundant: it's for safety or to optimize out */ > + if (likely(!(flags & RMAP_COMPOUND))) > + folio_add_anon_rmap_pte(folio, page, vma, address, flags); > + else > + folio_add_anon_rmap_pmd(folio, page, vma, address, flags); > +} > > - first = atomic_inc_and_test(&folio->_entire_mapcount); > - if (first) { > - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); > - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { > - nr_pmdmapped = folio_nr_pages(folio); > - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); > - /* Raced ahead of a remove and another add? */ > - if (unlikely(nr < 0)) > - nr = 0; > - } else { > - /* Raced ahead of a remove of COMPOUND_MAPPED */ > - nr = 0; > - } > - } > - } > +static __always_inline void __folio_add_anon_rmap(struct folio *folio, > + struct page *page, int nr_pages, struct vm_area_struct *vma, > + unsigned long address, rmap_t flags, enum rmap_mode mode) > +{ > + unsigned int i, nr, nr_pmdmapped = 0; > > + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); > if (nr_pmdmapped) > __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped); > if (nr) > @@ -1345,18 +1327,34 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, > * folio->index right when not given the address of the head > * page. > */ > - VM_WARN_ON_FOLIO(folio_test_large(folio) && !compound, folio); > + VM_WARN_ON_FOLIO(folio_test_large(folio) && > + mode != RMAP_MODE_PMD, folio); > __folio_set_anon(folio, vma, address, > !!(flags & RMAP_EXCLUSIVE)); > } else if (likely(!folio_test_ksm(folio))) { > __page_check_anon_rmap(folio, page, vma, address); > } > - if (flags & RMAP_EXCLUSIVE) > - SetPageAnonExclusive(page); > - /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ > - VM_WARN_ON_FOLIO((atomic_read(&page->_mapcount) > 0 || > - (folio_test_large(folio) && folio_entire_mapcount(folio) > 1)) && > - PageAnonExclusive(page), folio); > + > + if (flags & RMAP_EXCLUSIVE) { > + switch (mode) { > + case RMAP_MODE_PTE: > + for (i = 0; i < nr_pages; i++) > + SetPageAnonExclusive(page + i); > + break; > + case RMAP_MODE_PMD: > + SetPageAnonExclusive(page); Just to check; I suppose only setting this on the head is ok, because it's an exclusive mapping and therefore by definition it can only be mapped by pmd? > + break; > + } > + } > + for (i = 0; i < nr_pages; i++) { > + struct page *cur_page = page + i; > + > + /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ > + VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 || > + (folio_test_large(folio) && > + folio_entire_mapcount(folio) > 1)) && > + PageAnonExclusive(cur_page), folio); > + } > > /* > * For large folio, only mlock it if it's fully mapped to VMA. It's > @@ -1368,6 +1366,54 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, > mlock_vma_folio(folio, vma); > } > > +/** > + * folio_add_anon_rmap_ptes - add PTE mappings to a page range of an anon folio > + * @folio: The folio to add the mappings to > + * @page: The first page to add > + * @nr_pages: The number of pages which will be mapped > + * @vma: The vm area in which the mappings are added > + * @address: The user virtual address of the first page to map > + * @flags: The rmap flags > + * > + * The page range of folio is defined by [first_page, first_page + nr_pages) > + * > + * The caller needs to hold the page table lock, and the page must be locked in > + * the anon_vma case: to serialize mapping,index checking after setting, > + * and to ensure that an anon folio is not being upgraded racily to a KSM folio > + * (but KSM folios are never downgraded). > + */ > +void folio_add_anon_rmap_ptes(struct folio *folio, struct page *page, > + int nr_pages, struct vm_area_struct *vma, unsigned long address, > + rmap_t flags) > +{ > + __folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags, > + RMAP_MODE_PTE); > +} > + > +/** > + * folio_add_anon_rmap_pmd - add a PMD mapping to a page range of an anon folio > + * @folio: The folio to add the mapping to > + * @page: The first page to add > + * @vma: The vm area in which the mapping is added > + * @address: The user virtual address of the first page to map > + * @flags: The rmap flags > + * > + * The page range of folio is defined by [first_page, first_page + HPAGE_PMD_NR) > + * > + * The caller needs to hold the page table lock, and the page must be locked in > + * the anon_vma case: to serialize mapping,index checking after setting. > + */ > +void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, > + struct vm_area_struct *vma, unsigned long address, rmap_t flags) > +{ > +#ifdef CONFIG_TRANSPARENT_HUGEPAGE > + __folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags, > + RMAP_MODE_PMD); > +#else > + WARN_ON_ONCE(true); > +#endif > +} > + > /** > * folio_add_new_anon_rmap - Add mapping to a new anonymous folio. > * @folio: The folio to add the mapping to.