From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FEAAEB64DA for ; Tue, 18 Jul 2023 10:28:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 145E06B0075; Tue, 18 Jul 2023 06:28:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F6056B0078; Tue, 18 Jul 2023 06:28:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F26A58D0001; Tue, 18 Jul 2023 06:27:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E17926B0075 for ; Tue, 18 Jul 2023 06:27:59 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AA338A030A for ; Tue, 18 Jul 2023 10:27:59 +0000 (UTC) X-FDA: 81024357078.06.A6E5747 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id A1F85C000D for ; Tue, 18 Jul 2023 10:27:57 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689676077; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+OmtNA5dTIRcBq6K9ybdhEHme+Qg6Wq7IHKlPZ8epXs=; b=D/OYo0jCvaX+6mGZU5ww3CC7pmY3l237qQ7sQOpfmHaaxOR5x2ibw2Lo3z0Y7aK+R0lSG3 +dJxDRDkr1UnGOBjjVfwGD6nR7guZvOPk8SVzIfESLDkGOEPxQ0mfFfMjZiEHpE9GqRqDw 1th9Y/8FoS7WmQ1nBZ9W/gDyXAB8BHY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689676077; a=rsa-sha256; cv=none; b=T0iSjRYZens0N5x06qKK3qCH2/bBXqFCMVLM0aP29WwEz0DD81T29TUsNpiQ+D3NSsfubo Ul9+Fm682qQa1AzzqnQeEdkx9r5SI+slYtQe5vbaBI9ZvAg9o/70ep/3+gNsgxQRlu3qak 1o3AUvri9uwBA63LJJL1Sd+ziYynbMo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 20BB92F4; Tue, 18 Jul 2023 03:28:40 -0700 (PDT) Received: from [10.1.34.52] (C02Z41KALVDN.cambridge.arm.com [10.1.34.52]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 408E43F67D; Tue, 18 Jul 2023 03:27:55 -0700 (PDT) Message-ID: Date: Tue, 18 Jul 2023 11:27:53 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH v1 3/3] mm: Batch-zap large anonymous folio PTE mappings To: Yin Fengwei , Andrew Morton , Matthew Wilcox , David Hildenbrand , Yu Zhao , Yang Shi , "Huang, Ying" , Zi Yan Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20230717143110.260162-1-ryan.roberts@arm.com> <20230717143110.260162-4-ryan.roberts@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: A1F85C000D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: sczp75umod6rernyu6rgrhb9pjwg7317 X-HE-Tag: 1689676077-923795 X-HE-Meta: U2FsdGVkX1+pHCuYbcaoVPmagsxQbi36LRdZuNC7c4K3Jp924CAQ9KTY5j/wLVMjIt909LvEGUHDYltfhzRq7HWN6i2P2kXWJSoS94OVTK/3NqeMgSDZ4lyaaCNF2mZ2Ai1UXcEckoi2SkAsuE28vjupEh84i8KzDpGbKQGO6SxXMwFLQBIcZ0NhJFE+FHtaslbPd07XFtldBloq5Qt9SBDqdbaJKuy/VDRHhSal31UPBX2BIfzvgs6reWOebReF7y1n5nzlzyFYk4ryJfVXO1UHLkTgIZ73xqeJ+r/3B8EpuomORxI3mHvigdv/StQ+Km5fFRV5rk+w/ANCQM9xcvtokA3AYTOBLUIzHH1HcLAvotvVDK/fJR3JLKdJExhgI1pdZnqWvdpIBDos+8mT6OnW+Y6Z6V+U92lAuqQlKPtXNyXlvf1qgawEW2mtWRGbPZHwcarGS6jPrZwzSJtPo6EXFFjxd8AYEV6zv+pPJFj+yRd11hoO7EmmYloYWdAov4U7tRGbpfrC0WWSrg/CjpWt7IegFOeNWcOksE93a4dIyQ5wOVFr8og1mr6he7+tIHvbfEO8M68kQaQ++/qCTxe+GoqF/9+K4FqYRLVlwQt8T0Vs1HRwLHU7kmPow8trPbKCVlRb4Ay0+e3gImIKXBdPWWYfEAa3Co2M8nOHImkyCOzOd+NBqaE1Ae8th3bjytGcmQsPSirHF+kUfxiQWAuwowikcRJBeZgOSmteqgjov9BdD3W1LmfF/gzsahR39VhvYfwR78ylxFjk9Z/VR96ZOJ9xrgcwbv5myoUKrPL4to2G9V0ZOPVkm7zkjSghFdRgQj5XHvnoZArJ+sTMz6as9+loU0a4WaiPzeCnZTwBsE4eR4jKBm3GJncy2YiElzQdoziRMfcDyqoEe9s7DMAxlDsfZIhhQtY0CYdWdA+PkwCezqK1XSBn9SUOEgBU1TcDPpiRQFMG08EM3g0 Wuh0O3uZ WHbLb5gYoS638y/C9MuEIGqSnyJEtA+D4QHi7sDV7RkF58uJBqtC57poT4bP+jglmLJDv82YL0yVDB2NEEWKSJtgqc3HtP6BGRq4rCSBbiyZGCFC8SzuclJZwLDKTCcPvGMH81cO2RyQw0iR0nZkHzJw5DFDsX6yJGZ55CJXv4hcJCu4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 18/07/2023 00:27, Yin Fengwei wrote: > > > On 7/17/23 22:31, Ryan Roberts wrote: >> This allows batching the rmap removal with folio_remove_rmap_range(), >> which means we avoid spuriously adding a partially unmapped folio to the >> deferrred split queue in the common case, which reduces split queue lock >> contention. >> >> Previously each page was removed from the rmap individually with >> page_remove_rmap(). If the first page belonged to a large folio, this >> would cause page_remove_rmap() to conclude that the folio was now >> partially mapped and add the folio to the deferred split queue. But >> subsequent calls would cause the folio to become fully unmapped, meaning >> there is no value to adding it to the split queue. >> >> Signed-off-by: Ryan Roberts >> --- >> mm/memory.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 119 insertions(+) >> >> diff --git a/mm/memory.c b/mm/memory.c >> index 01f39e8144ef..6facb8c8807a 100644 >> --- a/mm/memory.c >> +++ b/mm/memory.c >> @@ -1391,6 +1391,95 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, >> pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); >> } >> >> +static inline unsigned long page_addr(struct page *page, >> + struct page *anchor, unsigned long anchor_addr) >> +{ >> + unsigned long offset; >> + unsigned long addr; >> + >> + offset = (page_to_pfn(page) - page_to_pfn(anchor)) << PAGE_SHIFT; >> + addr = anchor_addr + offset; >> + >> + if (anchor > page) { >> + if (addr > anchor_addr) >> + return 0; >> + } else { >> + if (addr < anchor_addr) >> + return ULONG_MAX; >> + } >> + >> + return addr; >> +} >> + >> +static int calc_anon_folio_map_pgcount(struct folio *folio, >> + struct page *page, pte_t *pte, >> + unsigned long addr, unsigned long end) >> +{ >> + pte_t ptent; >> + int floops; >> + int i; >> + unsigned long pfn; >> + >> + end = min(page_addr(&folio->page + folio_nr_pages(folio), page, addr), >> + end); >> + floops = (end - addr) >> PAGE_SHIFT; >> + pfn = page_to_pfn(page); >> + pfn++; >> + pte++; >> + >> + for (i = 1; i < floops; i++) { >> + ptent = ptep_get(pte); >> + >> + if (!pte_present(ptent) || >> + pte_pfn(ptent) != pfn) { >> + return i; >> + } >> + >> + pfn++; >> + pte++; >> + } >> + >> + return floops; >> +} >> + >> +static unsigned long zap_anon_pte_range(struct mmu_gather *tlb, >> + struct vm_area_struct *vma, >> + struct page *page, pte_t *pte, >> + unsigned long addr, unsigned long end, >> + bool *full_out) >> +{ >> + struct folio *folio = page_folio(page); >> + struct mm_struct *mm = tlb->mm; >> + pte_t ptent; >> + int pgcount; >> + int i; >> + bool full; >> + >> + pgcount = calc_anon_folio_map_pgcount(folio, page, pte, addr, end); >> + >> + for (i = 0; i < pgcount;) { >> + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); >> + tlb_remove_tlb_entry(tlb, pte, addr); >> + full = __tlb_remove_page(tlb, page, 0); >> + >> + if (unlikely(page_mapcount(page) < 1)) >> + print_bad_pte(vma, addr, ptent, page); >> + >> + i++; >> + page++; >> + pte++; >> + addr += PAGE_SIZE; >> + >> + if (unlikely(full)) >> + break; >> + } >> + >> + folio_remove_rmap_range(folio, page - i, i, vma); >> + >> + *full_out = full; >> + return i; >> +} >> + >> static unsigned long zap_pte_range(struct mmu_gather *tlb, >> struct vm_area_struct *vma, pmd_t *pmd, >> unsigned long addr, unsigned long end, >> @@ -1428,6 +1517,36 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, >> page = vm_normal_page(vma, addr, ptent); >> if (unlikely(!should_zap_page(details, page))) >> continue; >> + >> + /* >> + * Batch zap large anonymous folio mappings. This allows >> + * batching the rmap removal, which means we avoid >> + * spuriously adding a partially unmapped folio to the >> + * deferrred split queue in the common case, which >> + * reduces split queue lock contention. Require the VMA >> + * to be anonymous to ensure that none of the PTEs in >> + * the range require zap_install_uffd_wp_if_needed(). >> + */ >> + if (page && PageAnon(page) && vma_is_anonymous(vma)) { > Why this is only for anonymous page? I suppose it can support file mapping also. I was trying to avoid the complexity. For file-backed pages, there is a bunch of dirty and access management stuff that needs to happen (see "if (!PageAnon(page)) {" a bit further down). And for file-backed VMAs (even if the page is anon, I think?) zap_install_uffd_wp_if_needed() might do some extra work, which again I didn't want to have to drag into zap_anon_pte_range(). I guess it's implementable, but given only anon folios will be deferred-split and anon folios in a file-backed vma will all be single page, I didn't feel that the extra complexity would add anything performance-wise. > > > Regards > Yin, Fengwei > >> + bool full; >> + int pgcount; >> + >> + pgcount = zap_anon_pte_range(tlb, vma, >> + page, pte, addr, end, &full); >> + >> + rss[mm_counter(page)] -= pgcount; >> + pgcount--; >> + pte += pgcount; >> + addr += pgcount << PAGE_SHIFT; >> + >> + if (unlikely(full)) { >> + force_flush = 1; >> + addr += PAGE_SIZE; >> + break; >> + } >> + continue; >> + } >> + >> ptent = ptep_get_and_clear_full(mm, addr, pte, >> tlb->fullmm); >> tlb_remove_tlb_entry(tlb, pte, addr);