From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AF83EC6FD1F for ; Thu, 21 Mar 2024 22:10:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0r/O7papj+oui8wyVXtigDiLfbSDFjB/plndC0hIfmI=; b=zKf2fzDry86+S8 q6cGShZrCpZVnJ3hEIHZR6RlzlTM9aYlpshoP1lDPAIwiFxtyRD7g8y6IuQvc+mXtFo4bcjA0UY/6 TTN8S9pwIEXSg+9Y0I/YA96GnnpukHvcc9AuAcTEg9/m6e5jB6h+IG9dNiiiFdiGoeX3wi8RF5X7A EURs37HGBNd75l1e4MxyR6uswsYrPbBGhw2wFmZC92QX1JTajH+FU1fidpLHWeyurCc1wmuAjyMvm WHsGnePTuTnqDZSkIFc2RKCwDnbkuWan6bpjBUR1BRerMIlXAyQrG0xhWJCJBQuSSrSYnp7rgGft7 qksi2VKVMB6Tvhzn+D6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQcg-00000004sFP-4487; Thu, 21 Mar 2024 22:10:54 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaU-00000004qac-0Dqt for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:43 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058912; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=fKM4VqYAhMay1/b4NGyU6dSXL/rrOm2sU1lH2xiw8x5BjFUKE7kpClPgNf8K1RMEsgeP95 0x0lmTu+mQNBLKRflDVJnz9GjVxn5yCzjBcsFr1WuwGCJQyxqJEMIlehapF07wKLdKIUAR Tm8lU6Nyz+Q5OSk+4S4GH2BS38x71Oo= Received: from mail-ot1-f71.google.com (mail-ot1-f71.google.com [209.85.210.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-204-uUz_8c9zNW2IZi-rs-s0Tw-1; Thu, 21 Mar 2024 18:08:30 -0400 X-MC-Unique: uUz_8c9zNW2IZi-rs-s0Tw-1 Received: by mail-ot1-f71.google.com with SMTP id 46e09a7af769-6e691df3cd6so567557a34.0 for ; Thu, 21 Mar 2024 15:08:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058909; x=1711663709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=OBJWYkbNsTxk4oM4smaX9heOcj7hhxZJypJbDO9mty5RbRf78H81NsaUf3Fa9UXc4E eJ920eBKMiTVTwxhz52IE6rbe9CH4Pf6B611yK1vSTWUbG2AwdEMLMoKnUUiZJ7+QsAT zIjkc5aWLmiGurLwmvA0xlNyDzWVGfc0JzfQqXEVKca5nksUMgl1F4wjEaiCp2YxI7Q3 g+y2UoBf/+bjppDzpx7S643Hz/ISl3vPefdqh5lIt3PjXpbsepg066TgwcN7YhSFWEs6 UmV+9VpJJgY1wa7iZyqOn/SfGGg4K4YEPVjJkg0/37ju9lN6w9ULzimjfnbbHbDzSjxR wIfw== X-Forwarded-Encrypted: i=1; AJvYcCWJa6SuuyDINyBSCTNqCsJt+T/wqKbqoEULHHLZe7ovrR8Owa14QdD+7vToQVW1Qbsj93SLzrEbJHaocGjqxmQV3Fb7pLoJE2wOL1BqK5FZ X-Gm-Message-State: AOJu0YyKLr/TaWOOP1u5nA0ZML7k1JyoFlmOl/fpuN9MjlRd6K8IgOEj BrluBut+8bZxn/IlKsn2HmTOFVNfwtMSlBt5PnE2UQs3YIxHFy83Um3bwosnW7eoiZCEG5jJQE6 zHJZwbc5rE9qs8MoWPmbnSqZTj0mbcpfDtd6+133LyNRbQIAvOlBPicBmx3TC1o/ieQ== X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr912009oot.0.1711058909577; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH7bJa0sTEC4z3o8Ei9oO8UNdi6+GHr5ekBXamqtFS26QUeqg4nthASLLtCNNWPAVCCH5AJXA== X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr911987oot.0.1711058909121; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:28 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 12/12] mm/gup: Handle hugetlb in the generic follow_page_mask code Date: Thu, 21 Mar 2024 18:08:02 -0400 Message-ID: <20240321220802.679544-13-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150838_915304_16480EE5 X-CRM114-Status: GOOD ( 23.06 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing slow GUP over a large hugetlb range on cont_pte/cont_pmd supported archs: each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. A quick performance test on an aarch64 VM on M1 chip shows 15% degrade over a tight loop of slow gup after the path switched. That shouldn't be a problem because slow-gup should not be a hot path for GUP in general: when page is commonly present, fast-gup will already succeed, while when the page is indeed missing and require a follow up page fault, the slow gup degrade will probably buried in the fault paths anyway. It also explains why slow gup for THP used to be very slow before 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL"") lands, the latter not part of a performance analysis but a side benefit. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(). Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 52d9efcf1edf..85e1c9931ae5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -328,13 +328,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 43a2e0a203cd..2eb5911ba849 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -997,18 +997,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -1020,6 +1013,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index abec04575c89..2e320757501b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6883,77 +6883,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) -- 2.44.0 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A1D22C6FD1F for ; Thu, 21 Mar 2024 22:18:10 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dpm03JTq; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=D+txLCow; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4V10G92FYPz30fm for ; Fri, 22 Mar 2024 09:18:09 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dpm03JTq; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=D+txLCow; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4V103929vBz3dVK for ; Fri, 22 Mar 2024 09:08:37 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058914; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=dpm03JTqZbWMlokdHc6F6ZZHRyRyMyRMdGBBsBggvlTMwKFiPMnRCnpdib4lIjhCuHizK9 TlKjkFCd4Ov11JaK4FkNDVpOx2AHKeJYmHgpTeHyv0EkGltQ1YibXD9MID0Unlv/+znN++ g1KairYkgwowQ9oTGzh7eG0+cbmgvN4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=D+txLCow8aZDP8APvKjZSlXwbiuMa8rFEUDJoO0SV9fEuTN+qjHBgnlEyV614BVBNKwcez vAMslSuaK51lHajGql9mWDvJCOJ4SFtLwNzE44QL4DAqMBuH17iKy5tAC0KLWk+ZPGoxlp PgLjt6JiEkyEBFL7cNZgcLK1+2alAXM= Received: from mail-oa1-f70.google.com (mail-oa1-f70.google.com [209.85.160.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-84-CuE9EcBlOnaBslUxg1wNOA-1; Thu, 21 Mar 2024 18:08:33 -0400 X-MC-Unique: CuE9EcBlOnaBslUxg1wNOA-1 Received: by mail-oa1-f70.google.com with SMTP id 586e51a60fabf-221a566a435so551719fac.0 for ; Thu, 21 Mar 2024 15:08:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058910; x=1711663710; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=hC3S9XVA6n1axCkM6nwimZjOOcZBszPqD5zv1VoVe1FJiwR9oNcwBPPJIaD65aHSIm 9nfJWww5j0E4t/uDroau8R4UhIvgRikI/oF++/mHELtyyA0g122Cb5zlVGwFprV0Zdbu 8VVYujUv67F7ScjzGrZYDgbr7bjk7zC1uFMbpkvk4gHKEc4k/YTCHFmmgF0dRINXF9Ma 1M9MgxBGCJtuaI62Z3I9FhHNfc7qiCBjb5r7f5Aoy+ZIjHp3ZUEql13Ew3kbK6Ecv/AY 8X4HpavgtwgVpW2rsLDfo2aLbqXRnKvlg2sL9S2fh1g5DD4wiG2W9PayUvXFri/43vil zT1Q== X-Gm-Message-State: AOJu0YxPee7jr+V/XzSuPHawJYF7fxodjWlE1viUif4oILxsTJPQLGgP XpUKDX0eta/NElW2+dSZUX6n24J1NDQlck/jAAztN9VfwQvJuFnmcKwafKA6kjbs1gZYz6HE8sW zGN7CBlPEO4CIByzZwlDfpo0KnBlYjlEnPUeceptUUjW+oSwtj/nYgZa4doUueCE= X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr912014oot.0.1711058909580; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH7bJa0sTEC4z3o8Ei9oO8UNdi6+GHr5ekBXamqtFS26QUeqg4nthASLLtCNNWPAVCCH5AJXA== X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr911987oot.0.1711058909121; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:28 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 12/12] mm/gup: Handle hugetlb in the generic follow_page_mask code Date: Thu, 21 Mar 2024 18:08:02 -0400 Message-ID: <20240321220802.679544-13-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , David Hildenbrand , Yang Shi , peterx@redhat.com, Andrew Jones , linux-riscv@lists.infradead.org, Andrea Arcangeli , "Aneesh Kumar K . V" , Matthew Wilcox , Christoph Hellwig , linux-arm-kernel@lists.infradead.org, Jason Gunthorpe , Axel Rasmussen , Rik van Riel , John Hubbard , "Kirill A . Shutemov" , Vlastimil Babka , Lorenzo Stoakes , Muchun Song , Andrew Morton , linuxppc-dev@lists.ozlabs.org, Mike Rapoport , Mike Kravetz Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Peter Xu Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing slow GUP over a large hugetlb range on cont_pte/cont_pmd supported archs: each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. A quick performance test on an aarch64 VM on M1 chip shows 15% degrade over a tight loop of slow gup after the path switched. That shouldn't be a problem because slow-gup should not be a hot path for GUP in general: when page is commonly present, fast-gup will already succeed, while when the page is indeed missing and require a follow up page fault, the slow gup degrade will probably buried in the fault paths anyway. It also explains why slow gup for THP used to be very slow before 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL"") lands, the latter not part of a performance analysis but a side benefit. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(). Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 52d9efcf1edf..85e1c9931ae5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -328,13 +328,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 43a2e0a203cd..2eb5911ba849 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -997,18 +997,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -1020,6 +1013,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index abec04575c89..2e320757501b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6883,77 +6883,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) -- 2.44.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5C838C6FD1F for ; Thu, 21 Mar 2024 22:12:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HVHi32SITw6wibU2xf2ss1wciuHo5/EMqOtA44m7cVw=; b=sQfhehDhHUv0jC 6WjBuVflEXhPFkzLGWqtW2sbNgJsMiX5NLMrHpg9OhmHC3C0z0uq6F9D6cYsLks6H4ZRRcWuTI857 6CnsV+EPMf7dlGJFfPM3Y4xJ7AYf/k3Cv1X6thVCOuMQ0+dGaQJjtD1BiNtEUxTf24Fj9WTk7LlOC SUwbD/vlyXVBnr3h5aTufDg2JId9mtpfCO96i628FeAdFr0Q7w5Ms20SwpKulYf7z7x7Z/zyxinaA lOkRLKWLnrYLbLPC8K+vLKLdJ22M2eSSiKuFQHBS5iFy2wHApCjJKKY8I9aFSznDo6J8/YBpfdZoK +H2AuP1HTN13sxQROpRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQde-00000004svU-1wLG; Thu, 21 Mar 2024 22:11:54 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaV-00000004qeS-37k7 for linux-arm-kernel@bombadil.infradead.org; Thu, 21 Mar 2024 22:08:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Content-Transfer-Encoding :MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Sender:Reply-To:Content-ID:Content-Description; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=aZnIgW+1tUf3zVpgaUiUWA/wzB DkOGe/+4mdwGxos+9zQpD1KL/GM9u8NsRmlJq6lIFhkiA4K10vkVmww71JKYpacM+hEgiXrbWQ+9J 5ICZ/+UlA8PjsISW34xmj9tZe0uX3qSb3Whw2w0158K4CoSQj1W10T8Ef8kwzaE0IJlVtsJXs7pXx XkHD1v4WE2atnUdWgTadIRRGNL5F91TZMdIOx5t6wOt4Vzw+Ogk+g4NfYQItgxWXZH9wudLRBkA7b cgzNY5RnzNkvT1eq0Oxm1UOc16WdLIQDGLxu6/C9SQCa1zHFlNiN1ToKmVfUxyBVQDm2C3mRLslMw pM335vEg==; Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaQ-0000000ESiH-3uiy for linux-arm-kernel@lists.infradead.org; Thu, 21 Mar 2024 22:08:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058913; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=DLuefRc/C7fDu9il6UOdghrM5U2tSGjN0bLkJLUiZZ4njEb9ZKBoHNWANWYnf/oScgCrra jK+LbfSEMYM3MtckD13rLpuV4rzvBwt2gL0kOmj3alcFEDvubRqOC5Bi/P68PMRtrRK7yh vP/iHZUWUkBOpigWD+WwmR7KxaZnq+k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058914; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=dpm03JTqZbWMlokdHc6F6ZZHRyRyMyRMdGBBsBggvlTMwKFiPMnRCnpdib4lIjhCuHizK9 TlKjkFCd4Ov11JaK4FkNDVpOx2AHKeJYmHgpTeHyv0EkGltQ1YibXD9MID0Unlv/+znN++ g1KairYkgwowQ9oTGzh7eG0+cbmgvN4= Received: from mail-ot1-f71.google.com (mail-ot1-f71.google.com [209.85.210.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-516-HXPnxa1JNESrhTrBByoqeQ-1; Thu, 21 Mar 2024 18:08:32 -0400 X-MC-Unique: HXPnxa1JNESrhTrBByoqeQ-1 Received: by mail-ot1-f71.google.com with SMTP id 46e09a7af769-6e67ee38ce7so413894a34.1 for ; Thu, 21 Mar 2024 15:08:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058910; x=1711663710; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=LhupLEskDaQ8oaMKFUL5cDmwOp4YXUtlCSi3g/VBDnALylqoClWGNdxdSisLpAECmY xmyCll/aBsTlIWGH1LTTr+CjcBRqJUZcw6cxX83hEXvocz/0QLTKiHD+owX+MFg6EQZJ QU7qyqRxguxwsRWgqEi/8662l4XRMqTST6wxz0mvw1hT6ePo++H4nzKAC4VJf6MiAka8 0dcjO/fMLtwrXQaksEMRhG4G6o/O/vWOiAHXyDyGwEJZJJQ/g1rvz/fuB5BGGP1Vfh9r tKBY93p96IHeGqMqnKNL3inoQI5IM9FGekWu1GXLIVioxrrSR/oqas5L+Vn9G07BqPTM mpgg== X-Forwarded-Encrypted: i=1; AJvYcCW9Um4AZovFPMF062X7Plt1WTd/vAxWWMSAH7c5MakMW32WYdX7qDPfZC2kAZYdDsVinKD+o+mNiO0Q9looTe4Q9j9kPZmoUZf8ZMtAoy1cuBChWS8= X-Gm-Message-State: AOJu0YwvqbZaJeBfIhoKQQmj9jGfY+vJco4wicdCRpTS50bfCOONwvg7 2QAcATJO39mfT2p1MuZX0A9uGgk7nAnpYXD1DFmNzyti1HpSgzp6AhMKNAHlwo2wZngs+1aa4CX dEQ865ILb9snX/i9FhKTIC98rdOARcGPOEk2vVweP/sPjdZxhZr05HEZAO31VqJZw69f6oU/G X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr912022oot.0.1711058909583; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH7bJa0sTEC4z3o8Ei9oO8UNdi6+GHr5ekBXamqtFS26QUeqg4nthASLLtCNNWPAVCCH5AJXA== X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr911987oot.0.1711058909121; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:28 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 12/12] mm/gup: Handle hugetlb in the generic follow_page_mask code Date: Thu, 21 Mar 2024 18:08:02 -0400 Message-ID: <20240321220802.679544-13-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_220835_284489_DFA8B5DC X-CRM114-Status: GOOD ( 24.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Peter Xu Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing slow GUP over a large hugetlb range on cont_pte/cont_pmd supported archs: each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. A quick performance test on an aarch64 VM on M1 chip shows 15% degrade over a tight loop of slow gup after the path switched. That shouldn't be a problem because slow-gup should not be a hot path for GUP in general: when page is commonly present, fast-gup will already succeed, while when the page is indeed missing and require a follow up page fault, the slow gup degrade will probably buried in the fault paths anyway. It also explains why slow gup for THP used to be very slow before 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL"") lands, the latter not part of a performance analysis but a side benefit. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(). Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 52d9efcf1edf..85e1c9931ae5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -328,13 +328,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 43a2e0a203cd..2eb5911ba849 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -997,18 +997,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -1020,6 +1013,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index abec04575c89..2e320757501b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6883,77 +6883,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) -- 2.44.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C399BC6FD1F for ; Thu, 21 Mar 2024 22:08:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50E9C6B00A7; Thu, 21 Mar 2024 18:08:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 497B76B00A8; Thu, 21 Mar 2024 18:08:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 311656B00A9; Thu, 21 Mar 2024 18:08:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 19C856B00A7 for ; Thu, 21 Mar 2024 18:08:37 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E9DBEA08B8 for ; Thu, 21 Mar 2024 22:08:36 +0000 (UTC) X-FDA: 81922436232.28.DAA57D8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id DC04BA000A for ; Thu, 21 Mar 2024 22:08:34 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dpm03JTq; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711058914; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=liXYTUzk8K0zbFF+RvEsoTOzaBBjTOA7Y1uLHeU+uQqSOxBIgvVNOyiuMEKiK7/cUB0oZL MctyzJH0zn+ON3LY4SlUDVToNXlfYk8gLdFv0GF3vEVlzmP7a2XBeBVjsHO4gQryjGAyOi SDjjkWjl2ExNcr/KZPkxR/jq3PUw5d0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dpm03JTq; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711058914; a=rsa-sha256; cv=none; b=5BQ6kYjYRtT7H+XDCB3W4/EokovZljsp+ZMhia5T/uMvs11YYpsUMpJ++Q9Ip8lqQyUj7U U78z8oi4JLraVFka8AfoUDcNXx+ue7FN+rrbJu9XN5VRs4HfBtItuSIa6xSNI1wSwGYQe9 VI8FtQ93OnOLgey5dwucKCXd7uK9leE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058914; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=dpm03JTqZbWMlokdHc6F6ZZHRyRyMyRMdGBBsBggvlTMwKFiPMnRCnpdib4lIjhCuHizK9 TlKjkFCd4Ov11JaK4FkNDVpOx2AHKeJYmHgpTeHyv0EkGltQ1YibXD9MID0Unlv/+znN++ g1KairYkgwowQ9oTGzh7eG0+cbmgvN4= Received: from mail-oa1-f72.google.com (mail-oa1-f72.google.com [209.85.160.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-131-6QKRsGpGP_qWtzPt9ffSdg-1; Thu, 21 Mar 2024 18:08:32 -0400 X-MC-Unique: 6QKRsGpGP_qWtzPt9ffSdg-1 Received: by mail-oa1-f72.google.com with SMTP id 586e51a60fabf-222b55202f4so525986fac.1 for ; Thu, 21 Mar 2024 15:08:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058910; x=1711663710; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kKAkS039JEm9eSlNjnxGInSB8mOTV3IHPzDUFjGHTVk=; b=qe4uLPZh6ZctT7TvWyH3Tu+0377jeM6v5uo0KK3FgesCi8u3/m5SRf0XHVxLEMLx3j LccZu9Dln2XlAZFa1nsgtIC597jHwl/rg+jlFKvJV56I21hrbE4P8vP+/KwG2SBWoen1 38SnP+aBkwr/e0WZrB4wSWtX9gemyeEgKgwtSSzpFBetV5moNGtLMshEarHcNmK7z8f7 6ckPnmLAFHxsl15+Gu03WwkCOqYv1kj6b8iLbKTOvNa+Ht4tedhjv8w53i3Yj4g3+nQA HPI/YtNCGvyq0UnfkCMZyrQAj2GVXGua6IKChBgJHtKT54eu4jpHYwecskIYv1P8+hQ8 BAAg== X-Gm-Message-State: AOJu0YyKmYR/rNto2il4Apqt6ubAYQJIk8qEkJP/dS9nV9A7ze5MTLuW lr0HWuIUOLskUJgNDLONK1U/OqZOrhEx2PMhqO1gG4CXZcV8JDPyIF9s3r77C7QxmuRbsu3bvOm Hif8eF/P6AmyJHcYurNwtVdi8//9+9ixfcjbEs6MpF3Nu2j9Yg6UF8cmOcdS++o8JhHjhXxWYIO i5QWnMK7dyDaAY7qFgyXX1rk+QYdQp9A== X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr912040oot.0.1711058909673; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH7bJa0sTEC4z3o8Ei9oO8UNdi6+GHr5ekBXamqtFS26QUeqg4nthASLLtCNNWPAVCCH5AJXA== X-Received: by 2002:a4a:d5ce:0:b0:5a2:26c7:397b with SMTP id a14-20020a4ad5ce000000b005a226c7397bmr911987oot.0.1711058909121; Thu, 21 Mar 2024 15:08:29 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:28 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 12/12] mm/gup: Handle hugetlb in the generic follow_page_mask code Date: Thu, 21 Mar 2024 18:08:02 -0400 Message-ID: <20240321220802.679544-13-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: DC04BA000A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: pojzzgeuei3m7crd8a3pd6qp1b9c186m X-HE-Tag: 1711058914-328012 X-HE-Meta: U2FsdGVkX19MRoa7amfWNuXIIMrnaj0shhGJT2gUzdw9Xx+9L7MhGFvoYpdwTtIhHpH7nbQz9FhaGaRVtIxaiP7zn6y1hr8jmQoW0nozKXu+3eYIJW8YugXM6VH12dlyqmqeo9tyd6n+ESCDE4csfRBgJ3UkKWT8+KkaRSyEw818TjgwRqHIAokf0vPSl5qXZJPe0XFzOMlr1efq76AGWOPxxvy1Kplc4hc59k8QYVba/MPx2MionSveHKjsWngjZhdQDLL9xFUmnK0XnbsSzPNWLh852Keoh4WoT4Oq5Ng9TQueC9vRA3nUDDsyxSpwTt5wH3DrN6FCvocCYgsxJsnP76juyGZVBdyXTj93KjFZAra+8mADoZX6l5/hFi+otYpoUkPvgp69ETO+3scTcWA3EshMMAEJYJFj6vW1Y5U8jGZmQWbfVSMpQ1qh5cjQje2hyc72BE/HyXUyHjA0pzjjFb0f5FgAztETAgXSg7ENbNVn45+5JdO/n/jNg8yYW/9ZUgmqhDBetMjGIz+eXM8wxDioM3zMc2i73GDaV9QTf8+smKyOvWtl04LyQ5Y4oxzujCDeBypyD3xUEH0Ojy+YjSWHYGIKfHayXdCENN/oT0KPu3fdR0XRY3qb+KpErbuWfReq3iA/M+DbwJ9SSAuoF6rEhefX/HL4XsQuM+9uZNIgaHaIe1M9zyc+UHGFM4uCV4aeov5OtI1ju0iM+O9jsW4Lot2P8X2gJE7SOwWDb4LO5E4XJO6BtbsikqdbRO1ANuLQHMUX4XWTO8mA9WLkzPWsf8jPXn5vEphyoXeICe1dWO+We0dgWrsJ9FJD9yjumxgqqGd8spYWPYRwI6kYyb5tCmOriVM9G/5E4gFq9/sO8rI/FiovikCfuTMeMnCoVjW30133JFCbEtC9MJGQgm+EaXaTktAE1h+7VI/HLX+cQdpzW7F9rkgu8itPfqKDBo4tzhfCywpatIW vYeQ04zZ 26GFWawhfl7qOWAmNpiiEXgAEjS3dvGXR4JArydGsqAIX3lM5635iDEGS4vojH4V9Ml6d24ys0npb7UsjVr/8Yj5FBoNYPTXp74v7PzgVGGA8eNPdOcn2z0FEMHGyfQh0vK36ozvVt+t0/zMFdLYazYEuLx7DWSOHbmGM+rXcDCCNTU/q9fbnxPUt5RJkgfNqba+hyrwikUUheSKakupFCOWGvg0bZFBv4JgIvpj9aiXieFRU5F+i7eaM1JzkbOjgJoNTUHUAgdBQ0oRL2JOMDe4TnFb+fxMZRoqyqEqeX0QLDN7axBflsx+nJrZkhGy7fBT7G3vDbggp2a0UIGbulgjsrWQcYlHgY4yXxw5mqDkGU5Ohu1xZmldmXFc10/ZlGXSTe0eZcDZ9AF3G4+xgxIGS9V/qC3O9zks5SUOhMFRkzb6+uOqNE6xxuw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing slow GUP over a large hugetlb range on cont_pte/cont_pmd supported archs: each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. A quick performance test on an aarch64 VM on M1 chip shows 15% degrade over a tight loop of slow gup after the path switched. That shouldn't be a problem because slow-gup should not be a hot path for GUP in general: when page is commonly present, fast-gup will already succeed, while when the page is indeed missing and require a follow up page fault, the slow gup degrade will probably buried in the fault paths anyway. It also explains why slow gup for THP used to be very slow before 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL"") lands, the latter not part of a performance analysis but a side benefit. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(). Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 52d9efcf1edf..85e1c9931ae5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -328,13 +328,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 43a2e0a203cd..2eb5911ba849 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -997,18 +997,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -1020,6 +1013,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index abec04575c89..2e320757501b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6883,77 +6883,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) -- 2.44.0