From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5AF8AC6FD1F for ; Thu, 21 Mar 2024 22:09:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TIdSYgnG+6Rt5e6kRhKaO2Jjf5GkDLKg5iwaJO4mQCc=; b=eRYC1mNsqzjdmr 5jyxIQIolawJ21KNQ8jfKHKxvVLWKHgj1sWqrFO4b/mV1hdIUFhLAYpNGl/SMvYqvZd+JIcMc519X vO9tLosunPJEiKS/Jn5lQ2/iIUBvFrS4bKM1/SM/bK7C1Oyk8K10xpkY36QkpKbfyVTApC81UHhGH WXWNRiyAuDxbW16SXmtsgebnHIzbgyIL29krHk0v2BOCRAP6t5cESmXrlMOTMva5VBsR0xIGYQCEL tlk5ywJCQAEvJcYXqXHTiB6bGnxOwa/kefoaLACx5M20S4fimFcYFzxMHVnEaJ16joes5QZ7qEDp+ Qg16RrL039TkDHh1EQJQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQbS-00000004rFi-1Mex; Thu, 21 Mar 2024 22:09:38 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaC-00000004qOT-2vvy for linux-riscv@lists.infradead.org; Thu, 21 Mar 2024 22:08:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=Gq0YrZaZJDba3/oiLHQ5k0QmoQm24cQ+2uRUbyM7sQ1AQDcS7Dy2L/3pkR7wVjlqpxGD3z mcAbI1PWYLUrjrMR4xJKymVdTk2i//qVxoi6PSNXD+zK4MGzMyJGLIS+lin8FeDhv0Hmcl 3dxdDIBhC+tRORZawc/LBLVRrZXzBG8= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-342-kRwCnjsYOpqLXS7UsPO25A-1; Thu, 21 Mar 2024 18:08:17 -0400 X-MC-Unique: kRwCnjsYOpqLXS7UsPO25A-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7830635331bso30344785a.1 for ; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058897; x=1711663697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=QKZGcb0LrD1wE3N2jKJJLVse+UdTbLq4CeiIvoWXfWZRN8ln7Yu+nSjsvr4JiZyFET tQzWBbGhyQA1/WvfQUO4Gk10wsmNWVhplSGhdoQgR1oxkNkyEtPkNYdobaNyRfFwDxH0 J7A0HulcIGLKtaXwUrFmR2wjteTQI2BUSn2G/Hz3EZ7Y0akn3Ulmfw0JkPxdkbWGY3kD bF+VcPsw7sieblfRJ9wadyIUZDZSR6DRvfv+BS4Z/VvAATloglKK9WRmmj5a0ZOxRKvk NN3MrrDCsEZcCjE5uk7HBS2SBrAW8rXdlBExdCAYkil0QNl0iWXqm82jL4b7HqMB0ofc mV0A== X-Forwarded-Encrypted: i=1; AJvYcCULMZ/ZlYtFEuZF2w+qgMuaZ3CZr3N1F8EDgH/2rzIX0XuwotI0IYtihnftREQx8Z9uy8KNa2ke1XZptM8n6Gn2CltvdBm2gqt3W1kiPfNg X-Gm-Message-State: AOJu0Yy1ImjQEwlbYjjj5ASk5mRDToU9aK8LrDi3P0binHq+cc6CZjyP RJCM+7g0tNrY/K7rVR3FF75xq8J7m0ckcs7KT9PE6Jo2WKPkam8uXWc0pBJN5GO9k7dZwfxXbDZ PqKWxj3oh1c6a0m3WdH9qlfPC0wAgnhQQZKaYDwikjKS2qExspHs20bboUbX1ov4mLQ== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443538qvb.3.1711058897151; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHGuIUGmtGDilXkO+Txm1IkzpifWGQBAYBHvb3QP196cSPWL36y2snARbq2rVPrLWqoQw6FQA== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443515qvb.3.1711058896774; Thu, 21 Mar 2024 15:08:16 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:16 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 06/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Thu, 21 Mar 2024 18:07:56 -0400 Message-ID: <20240321220802.679544-7-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150820_997565_0E6EAAAC X-CRM114-Status: GOOD ( 10.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9127ec5515ac..f3ae8f6ce8a4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2778,13 +2778,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2819,8 +2822,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2893,8 +2896,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2937,8 +2940,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2977,8 +2980,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) -- 2.44.0 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2D755C6FD1F for ; Thu, 21 Mar 2024 22:13:26 +0000 (UTC) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Gq0YrZaZ; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Gq0YrZaZ; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4V108h5Xhvz3vcM for ; Fri, 22 Mar 2024 09:13:24 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Gq0YrZaZ; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Gq0YrZaZ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=peterx@redhat.com; receiver=lists.ozlabs.org) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4V102t4MK8z3dBn for ; Fri, 22 Mar 2024 09:08:22 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=Gq0YrZaZJDba3/oiLHQ5k0QmoQm24cQ+2uRUbyM7sQ1AQDcS7Dy2L/3pkR7wVjlqpxGD3z mcAbI1PWYLUrjrMR4xJKymVdTk2i//qVxoi6PSNXD+zK4MGzMyJGLIS+lin8FeDhv0Hmcl 3dxdDIBhC+tRORZawc/LBLVRrZXzBG8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=Gq0YrZaZJDba3/oiLHQ5k0QmoQm24cQ+2uRUbyM7sQ1AQDcS7Dy2L/3pkR7wVjlqpxGD3z mcAbI1PWYLUrjrMR4xJKymVdTk2i//qVxoi6PSNXD+zK4MGzMyJGLIS+lin8FeDhv0Hmcl 3dxdDIBhC+tRORZawc/LBLVRrZXzBG8= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-354-VVGXq9lNPFyIhDamVp6vTw-1; Thu, 21 Mar 2024 18:08:18 -0400 X-MC-Unique: VVGXq9lNPFyIhDamVp6vTw-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-78a0e8a60f8so22227285a.0 for ; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058897; x=1711663697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=mpFvQ8H+0jXUPvrIsQKRpbAbxpTe2/CjZxLTQbbPKt3NnnDdrf7wKzSt+yYt40cS9U 8Z5/cjmrRLtF4Hf7IzmhZzdeOFlAXANpMmQlUruWqKLg7Vl+7/Ey04w6PAKAxHfowpOS sEmSN5JECuw7BLecc1C6/YsCK8I4/G9jCzAWKlh+zACuzchwaOzsZWQErIdImeWYLhAt NdOQHmCz6EuYjr/QmOA/VKe8bF+LdYJDjbqqbSxc/KRhe/B9xzrlYBdeiD5RADID5eja xUec2lGCWcVK227arohOmJ+3prZWi4l/v1ouvHoin3nZGEKuAV5mt3NleqvuAySdfTDv Elvg== X-Gm-Message-State: AOJu0YzGdG2oapdSxu3AjEd/jIYOfENj8ac0pvUPsBOSRSF+6B5aIrOZ z0L4GcPv5QlCmndfikA2D/+p5K29/CxaXn4AJ5mRubHo2gI2llsz5NbK+qpeOTw2rqthVmnfNZp vTv9QC3T/bWjwVZX1I1I3jwAVk7O2gk7cn0avlWwH9QHhE23YCehgTnvasrsa0XE= X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443546qvb.3.1711058897153; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHGuIUGmtGDilXkO+Txm1IkzpifWGQBAYBHvb3QP196cSPWL36y2snARbq2rVPrLWqoQw6FQA== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443515qvb.3.1711058896774; Thu, 21 Mar 2024 15:08:16 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:16 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 06/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Thu, 21 Mar 2024 18:07:56 -0400 Message-ID: <20240321220802.679544-7-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Houghton , David Hildenbrand , Yang Shi , peterx@redhat.com, Andrew Jones , linux-riscv@lists.infradead.org, Andrea Arcangeli , "Aneesh Kumar K . V" , Matthew Wilcox , Christoph Hellwig , linux-arm-kernel@lists.infradead.org, Jason Gunthorpe , Axel Rasmussen , Rik van Riel , John Hubbard , "Kirill A . Shutemov" , Vlastimil Babka , Lorenzo Stoakes , Muchun Song , Andrew Morton , linuxppc-dev@lists.ozlabs.org, Mike Rapoport , Mike Kravetz Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9127ec5515ac..f3ae8f6ce8a4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2778,13 +2778,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2819,8 +2822,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2893,8 +2896,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2937,8 +2940,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2977,8 +2980,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) -- 2.44.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6BF35C6FD1F for ; Thu, 21 Mar 2024 22:10:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mjwrf7mA2q0r7UF8raW3xTFaaBtdgzCZl12WSdh4WKo=; b=g7bNlmxsjj9jDX zSP/TPEGS1QCU9COUUlW86rkAAOxnm4pYKdskX5jAbNoWYNhp6bSkn17yEcyI2lcEAIHvmGlr4CKy 0FOYjYof8tt+a7z9CFdEk0fQgMreg5uNvp7nG+WEM0cm4It+q7cwfAsTVONb5cYBh1YrM4Q+/ZBzV W/IkzTtrHYbidwG+eJyJ9Dwe8TwmYEYfpC1OBaVddSJ6M/Q0qn1f7KHlq22qEE0P4t436j3UQJjeg Jqe9cWn4XuJdSx1D06l/a3xZOOr7vwHHsQUFu/tHHNkSvBrCxAP4rN23UaSU/ky7WzGepv8BaMtOR vuGc5NtFvmJRSyPtboBQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQbg-00000004rQU-2Zv9; Thu, 21 Mar 2024 22:09:52 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnQaB-00000004qOC-45CP for linux-arm-kernel@lists.infradead.org; Thu, 21 Mar 2024 22:08:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=Gq0YrZaZJDba3/oiLHQ5k0QmoQm24cQ+2uRUbyM7sQ1AQDcS7Dy2L/3pkR7wVjlqpxGD3z mcAbI1PWYLUrjrMR4xJKymVdTk2i//qVxoi6PSNXD+zK4MGzMyJGLIS+lin8FeDhv0Hmcl 3dxdDIBhC+tRORZawc/LBLVRrZXzBG8= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-447-SZiH64wNPdSE8fRiHlRCzw-1; Thu, 21 Mar 2024 18:08:18 -0400 X-MC-Unique: SZiH64wNPdSE8fRiHlRCzw-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-78a0e8a60f8so22227685a.0 for ; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058897; x=1711663697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=DBxdCQ/dNsu5Uag9YGE5W8uMe1e0j7aig3XBypRqon6UNgCJ+y/G9yxtRuYj1x+2Gn FvLjPfC0U6V7KtqZhiLmVXud7na4YUAP0wcU5a1r36WjwON9VV6ouhJconOPUq6J9e52 AxfwNgJzTSFtgBkeRz0XSwViD6Io0hj/55gux17RcZ6/pnTsoZyHH3pwmo3SuQImEa8S H0fqJ6Gq2/6vxjPblM7PL4fibJjTSmYrDO/g8gKHitvA2CaYZ6sXwC0uxKTKTyeeDlyL QXOO3jhJqWb4pqCsExDJOWecumbGKa2YulgG21bfnwmsWrLuNyszwZT/9yU1xPYaxDf6 e/eA== X-Forwarded-Encrypted: i=1; AJvYcCWp8iFARQCpw0Ucc4JuZE52EVxqBUyKpke0rv6n8O7q6yk4vx7lPbmW+pKUj3Hrbrwd1MoP/pde1PgM/cG1ypdQ1/ZQZ1XR8leeTlY1jAc+c5O70/o= X-Gm-Message-State: AOJu0YwpE7yZv5kKTjBVJkrC4lwchuPRqyepoLNyMg8guo6pmEBO+yLv wqhYltx7a+XXuThYEWvrp6ic4rVjbJIvZe2abHBJikOAv7dM8wJBr+7gDpEGlf+SnESr19Gx/pZ Q1VO2nYfSOy0iXkl7R7rfRCKe3aFyX9FDnsq7ZEo55njmx953ZTTxrpR4n+BpLm02W8P/UJ88 X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443528qvb.3.1711058897149; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHGuIUGmtGDilXkO+Txm1IkzpifWGQBAYBHvb3QP196cSPWL36y2snARbq2rVPrLWqoQw6FQA== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443515qvb.3.1711058896774; Thu, 21 Mar 2024 15:08:16 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:16 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 06/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Thu, 21 Mar 2024 18:07:56 -0400 Message-ID: <20240321220802.679544-7-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_150821_017036_CDC3AAE2 X-CRM114-Status: GOOD ( 12.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9127ec5515ac..f3ae8f6ce8a4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2778,13 +2778,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2819,8 +2822,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2893,8 +2896,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2937,8 +2940,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2977,8 +2980,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) -- 2.44.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92D89CD11DC for ; Thu, 21 Mar 2024 22:08:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B1FE6B009D; Thu, 21 Mar 2024 18:08:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 13D186B009E; Thu, 21 Mar 2024 18:08:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA7EB6B009F; Thu, 21 Mar 2024 18:08:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D2B6B6B009D for ; Thu, 21 Mar 2024 18:08:22 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9A3B7C048F for ; Thu, 21 Mar 2024 22:08:22 +0000 (UTC) X-FDA: 81922435644.28.0CD2B56 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 803C540010 for ; Thu, 21 Mar 2024 22:08:20 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gq0YrZaZ; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711058900; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=npzWGjXQqCHUMrbyOPIIcY5ddBDMMhTlA75BoWRdlmeERWgEyXKN3ctjkFeiPn7Px7olNJ sKDvj8s99FoChA2oQMyYMLaHSMrfjYqCWMRnCc2oN1nFsR7vQWX1RU2Hc5L6R1me3IhI+5 kj1dtIX2qvMHDTWlp989/9G0EYROL1Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711058900; a=rsa-sha256; cv=none; b=HgUFm+EHn6x65tpD3ucHXPBu9A3v+ff6nO9i/hOpojnwXcf0OK82XRFJOa4bRN9OJQp8Un uayC1VCr06riV1xnR6ZFYQiS+zs9UNQTVBXAQ3exszMTqaVo0aOyWxwaF2E1JQFg9YxgtZ Nu7pW3T2Q+Vja9Rd43wshS/HPvwHl/o= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Gq0YrZaZ; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1711058899; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=Gq0YrZaZJDba3/oiLHQ5k0QmoQm24cQ+2uRUbyM7sQ1AQDcS7Dy2L/3pkR7wVjlqpxGD3z mcAbI1PWYLUrjrMR4xJKymVdTk2i//qVxoi6PSNXD+zK4MGzMyJGLIS+lin8FeDhv0Hmcl 3dxdDIBhC+tRORZawc/LBLVRrZXzBG8= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-588-Ym7j8uxwPQSKPiE1AwZfuA-1; Thu, 21 Mar 2024 18:08:18 -0400 X-MC-Unique: Ym7j8uxwPQSKPiE1AwZfuA-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-78a0e8a60f8so22227885a.0 for ; Thu, 21 Mar 2024 15:08:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711058897; x=1711663697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dI3BkO2nCtQEnBDt3PLoa+NL/Gig0J3ZsiByK6YiGEU=; b=GVMBSlbSIJTU5EbCGCzA/6FC739WyTM7iffblImKGFKqj4jF5XrTvnjIjwhNP9DquF wBjhYyixKgMr+ouADDTVqsYvLn0TOPSlNBjJ3pOKWCTBp6aJzTUytbGoNcJghmY3Ulm+ vRmc3m6QSLRLCi8i03Jqwbn96tjLnNxwHyUZZ2xb/Xj+poHm07IMopp5NIJ1SjMoTDPE MJbWcbDPgfCaY+IkfrYMamEMb/cPI13Fxryq2ecgGsW8QEtPN8g9zg6PoYFViV4MEMf4 BWKYy+t/hTnWDG7YmXRj73DkYW5bUdYFcYsahZfnPaNeofTk2dpOsG3yaW95rCpY9L67 Lkhw== X-Gm-Message-State: AOJu0YyTK/9SYMdS43GBu/V3rmo95k1iQdN2heePLs8v2tudsOKHuKYQ bd9Iv7hhKeOdLF7cnfzBfbzbw9fmWbx4iB6zlLjlU1msejbrb5YULsyf8G/qWntxyVbVZSadyIN vSwvHuuluLBFsnoyKXo/85szSM7H6VM0vvr9Izqw3nEcclwM7zRdeYAEjKs/z4s9q+Cpf37B4Zv xtVMByhVo7pmcssWNlPh6muImAkr5zNA== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443561qvb.3.1711058897286; Thu, 21 Mar 2024 15:08:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHGuIUGmtGDilXkO+Txm1IkzpifWGQBAYBHvb3QP196cSPWL36y2snARbq2rVPrLWqoQw6FQA== X-Received: by 2002:a05:6214:4489:b0:68f:e779:70f2 with SMTP id on9-20020a056214448900b0068fe77970f2mr443515qvb.3.1711058896774; Thu, 21 Mar 2024 15:08:16 -0700 (PDT) Received: from x1n.redhat.com ([99.254.121.117]) by smtp.gmail.com with ESMTPSA id o6-20020a0562140e4600b00690baf5cde9sm351663qvc.118.2024.03.21.15.08.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Mar 2024 15:08:16 -0700 (PDT) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org, Michael Ellerman , Christophe Leroy , Matthew Wilcox , Rik van Riel , Lorenzo Stoakes , Axel Rasmussen , peterx@redhat.com, Yang Shi , John Hubbard , linux-arm-kernel@lists.infradead.org, "Kirill A . Shutemov" , Andrew Jones , Vlastimil Babka , Mike Rapoport , Andrew Morton , Muchun Song , Christoph Hellwig , linux-riscv@lists.infradead.org, James Houghton , David Hildenbrand , Jason Gunthorpe , Andrea Arcangeli , "Aneesh Kumar K . V" , Mike Kravetz Subject: [PATCH v3 06/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Thu, 21 Mar 2024 18:07:56 -0400 Message-ID: <20240321220802.679544-7-peterx@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240321220802.679544-1-peterx@redhat.com> References: <20240321220802.679544-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: 803C540010 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 39ehau4kxu7nmjczfh8nfwt77ojsq1em X-HE-Tag: 1711058900-207467 X-HE-Meta: U2FsdGVkX19UdH039RSD684lfw4/zkoZxSSsm6FuonffrkS/ImjUnS4YTnBQXktpel37RRaOxK356XtnxwOyNUKQmlPM15+4wQYBrdf8E1zWqxeWwux0O/z6D2XCjmY3G96RlJmQ5buuDfsopP9GLqmE0DMwYqsromnhdcdapooHCz1hlfquGFvW4WQn7nYTgKiaAOQfRV44J8233Q709yNk3e9Tt+UOmWoFI3dmLnQ04CD7iJm6sarhMhD0IY3cVWpe26WOOQw/IHPcV2A906qiKwJkseS+IojVyDp436egjc0d6l+SdcTg0BeIaHehLumMGbKDuTMNmKja8KzFPwjGxirFIm8p3AmGWzRvP4OmAo1SfWMpKTFFZlQrpOrhylbVOZVL6MnnQ2FCzsFheBd9HIW39G1KNfwTXwNa+yzjknX53T4/gUXxAXd02DNMvw2OIjokH3MkN+vOPf0bkDqMNMhANyi5rB9M7MEp8agYEx0qyRJVY1Yr1tq1tgD7leXUlwqgvHwvY+FHMSIw2qwIZTu1k5JmBjDHbiSe3WSHfN4mYmHBNaToeb19DPcbl41RtyXztzFyXC1yhNd7VHqe4GcVdNszDovpC7K80s8xC6DjXsukYKkOcbPVc1soz49L0SLV9aa0qrW7mhiysWfBnigAKuDlmsUH+dqTag0oZe5wXxxPADERKT/rtwSW6CE7ASYBSNSdXO+nYMUO0OsW5Z1kfPDjJj4CSUxRl9Kd8mUjZF550RVr3+9ChWeZ1meFfr8+CsipxC72b54gR++CY0I4w2HDyqfVtgtpczesB4OF1vGrMwzj5DKcXhtaAQ5Xu05ollOf1LSNt4CL1TnQeXeTvXasHJVHVHX7/QRQUg1PX1yBRlxGivI8dfcf7QYD0t8ccvEynOAzU8PuuG9mmbAJg9677i9Wqu0lttRCIGiklOwn/7QkljfOx/ScS8O2TnUMa5IKex6T5iN osETQJsE HUrj0jND/xb69qXoGEeHB6Gg6s9rEz5bXYQFsHARGcIeb2dFkw55KdQVwnmTkQmod9yoHl0o5wcFR9+dBI6ClDrQ4gHa1UpFrXKCTV5uPjLo1rnR43rkqWqZyo9fNQDk9j4FJhlHfMXph3PMCYBgY/qPAF0h6jwcsR7mgTphkSjntcGFkvGrKDwsDpf2mcU2X3UYdUIBBOkrwNGs5KiK6mloP7NwrcNe+EgmRm2Eo6krsixJVV2UqB+L4u/FjNfOBpB9XFttLSDb1hfDBl0dSYR22zF14CIuUAC3LWHve8SKt4jqRzAOgOWjX5LAcXctHgtNgfFpInLmenpH5+6eNbu2LvkwTYOoe3DXRJlAgEOmHrBSmLYnlyXUk3Q4Yw+q+WDIVv8W5TZgZ2kXu/j4mwchv2DmIBdiu+eFYl+08ENg1ocJ//lrOmN0fky6AZVWuQCOdxK6txA7OMQo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Reviewed-by: Jason Gunthorpe Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9127ec5515ac..f3ae8f6ce8a4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2778,13 +2778,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2819,8 +2822,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2893,8 +2896,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2937,8 +2940,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2977,8 +2980,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) -- 2.44.0