From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09723ECAAD1 for ; Thu, 1 Sep 2022 17:48:46 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MJT704l9Tz3fJS for ; Fri, 2 Sep 2022 03:48:44 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=eEZ4BqXk; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--surenb.bounces.google.com (client-ip=2607:f8b0:4864:20::1149; helo=mail-yw1-x1149.google.com; envelope-from=3ko0qywykdm0bdax6uz77z4x.v75416dg88v-wxe41bcb.7i4tub.7az@flex--surenb.bounces.google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=eEZ4BqXk; dkim-atps=neutral Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MJSrj6fQbz2yxc for ; Fri, 2 Sep 2022 03:36:17 +1000 (AEST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-335420c7bfeso234912657b3.16 for ; Thu, 01 Sep 2022 10:36:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=eEZ4BqXkAmPCIUFHBPa2tHlcApvv3HMidc3eCYWOXuAzHrpAN0HS0lhYeZxby9RzBR 0w6QoRpV+iKQz37VUGml6Y4ETbNA7OPStJ8QApy5HYv7GzeLDO7+EvEq5dUOIcaM6UVC 7rLgM8Qnok81T8RLLnPTDqNW3fzyiOCIgOk9erIOBMCvX1CoI/RbBp7yoBH88B/Pqkwt H8ojE/1F974lnPVAUpGISzyJjuOBuorgNT9VfkfOyqVqOEVMNKxpaqT01PsyXu4rsbpt stRow1K+x3Cpkks4NKMTKIZbGi9T2v1db4rNZMcbp4CaDTkAzPc4JsqrDHDLe5K5oO1C a6Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=4Az7TNo1+PoR8s6MuhwLo/jNB1WIrddYOi25/0AM732bawG/bG9Zv9coUEYgZaxTdD GFWQsMvUeZmhj7o6AFUHDGjzG0vWHB05gGzyW8g8fngiBJtftWAO+4aolQpqLQAu+awK fjMCZRKHQeDYFkypazriKUPGaG8JPjquzSC1SJKTpy4Nhl+/H3UEu7jaftGdKlsZKgTb qaiPcViutapCBjrn2EsC3PAKM/ID8/acFRvQ7ykWSwgXo3JbS4ao/15Jack9lxb+D4Ez yXO0n9gI7yIvoNhNFfmr1bYtw0HEx28WUBaHA4Gjn2HX1uAdQVpEfdgpc68c0pITeqh+ oC5A== X-Gm-Message-State: ACgBeo0hHxbTOlfJgSpjLUf+4GCewUUUNkTjeTpwZhYoAKJeP+u7k6SX qghAm5gY6rQjcCAcpkSWAVVC/eBX0LI= X-Google-Smtp-Source: AA6agR7n//3fxGZjvfyML08bNSpeGfau+aTgkZu5QxUJzLJc9oUWAiawZA1UWbNFrnWa2xxdQVaE06YdPF8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:18c:b0:695:9d04:c3e with SMTP id t12-20020a056902018c00b006959d040c3emr19907526ybh.58.1662053776523; Thu, 01 Sep 2022 10:36:16 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:09 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-22-surenb@google.com> Subject: [RFC PATCH RESEND 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: michel@lespinasse.org, joelaf@google.com, songliubraving@fb.com, mhocko@suse.com, david@redhat.com, peterz@infradead.org, bigeasy@linutronix.de, peterx@redhat.com, dhowells@redhat.com, linux-mm@kvack.org, jglisse@google.com, dave@stgolabs.net, minchan@google.com, x86@kernel.org, hughd@google.com, willy@infradead.org, laurent.dufour@fr.ibm.com, mgorman@suse.de, rientjes@google.com, axelrasmussen@google.com, kernel-team@android.com, paulmck@kernel.org, liam.howlett@oracle.com, luto@kernel.org, ldufour@linux.ibm.com, surenb@google.com, vbabka@suse.cz, linux-arm-kernel@lists.infradead.org, kent.overstreet@linux.dev, linux-kernel@vger.kernel.org, hannes@cmpxchg.org, linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma = __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be stable + * and not isolated. If the VMA is not found of is being modified the function + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma = find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. -- 2.37.2.789.g6183377224-goog From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 47B66ECAAD3 for ; Thu, 1 Sep 2022 18:05:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=hoXnndhZXLo5aIgaycyCxO4b5nSxCgy9dzkYspZxDgo=; b=NzikOf7m9mRXF/IQSyIEzC2zYQ RnIhF0wzE1vp93LqfJlynB7V51cuGMzDZU4votLQWoQeLUP/WiMmeIkEIzsqsHwoefzibBkDUmqDR 2388GwYUvEcuKJ08oEiFg//SIfH77U8IygiiZ+1OhTV1PmG9fvTdsdnHjd2FHfzG01jZSGa1ogvVF IjRsPOmZkuAnrjI9RyEDmGP8dOJYKQU15jw+Txp9yv87kI8duw0POhHQ44Yq1lBLTVxo1sgUPZqV6 B5sBQfOhXonz/2Kmidk55wIl6x9MmlpMHP6ZL2iybK2s4tfKPl7arAoRX0YfieN4C4CLS32WUevAo Ofed+BNA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToY3-00DmZ3-Db; Thu, 01 Sep 2022 18:04:16 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oToXM-00Dm5L-Ne for linux-arm-kernel@bombadil.infradead.org; Thu, 01 Sep 2022 18:03:32 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=U17mw/lvD8Omk8eo/K7cBBouOy DF8X3ISxhWY+q2LrktzRLii7q0m2IUTzJ4Uzwv5J3maVdkItQS+Dvo0E/57y2xMoV31M/eCaYEa/0 rVgR1veAjIiSJTcUfQuZbWIoBuvMggpUwy0QGHpnTrn/3kmP0CTLpwYLWY5XpS+TlyECjUmfkN0iK Xm+tKSMgpQ8K4iEOJ7wSGfHskVzcCHIOu+XsE8rpj5vdK7wjLSRLxhjudj1kCe2NeU4koA9+KClWQ ianpQHd4RwGupDrwNYn9RBj0Cc5JA62PEe9uaapsaiu6v9Mv1YHuveod1w33+9+0Z+3jajw76t0zR Cr8UbaHQ==; Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTo72-008Rn1-2T for linux-arm-kernel@lists.infradead.org; Thu, 01 Sep 2022 17:36:22 +0000 Received: by mail-yb1-xb49.google.com with SMTP id y81-20020a253254000000b0069339437485so4864878yby.5 for ; Thu, 01 Sep 2022 10:36:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=eEZ4BqXkAmPCIUFHBPa2tHlcApvv3HMidc3eCYWOXuAzHrpAN0HS0lhYeZxby9RzBR 0w6QoRpV+iKQz37VUGml6Y4ETbNA7OPStJ8QApy5HYv7GzeLDO7+EvEq5dUOIcaM6UVC 7rLgM8Qnok81T8RLLnPTDqNW3fzyiOCIgOk9erIOBMCvX1CoI/RbBp7yoBH88B/Pqkwt H8ojE/1F974lnPVAUpGISzyJjuOBuorgNT9VfkfOyqVqOEVMNKxpaqT01PsyXu4rsbpt stRow1K+x3Cpkks4NKMTKIZbGi9T2v1db4rNZMcbp4CaDTkAzPc4JsqrDHDLe5K5oO1C a6Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=2Fz0JyGUCaaQ8O4rHOkTIk9V/hZXoJRCCDqiYpYQhbwx+JYeX2jklHOKIpMatgWF4a kcWG+vsKDHNbiICxUrZAeJXjmgF+YYXQbOOKjkf01Hd0LgUk1Y+HmfSRjmU5YvVv3E/C JpSaQYIkapjq4QlKiCnoF2m/49HYAFG9FmkEKjnmXLWEun3SaLBPtSC/bhOzXdQnRAoX UoUtGyybs5+3Q143iTsYZBNIilPTV5kxuzWwz+yuVt28rcYCpWcRuSN+A2dLrbWmKWPG 9ktVHUKufPmd/vwN3dYRHpSeJhXxSOPT4v1C3XblkhISNnMUQc5fma+3ttwgxZ5lFE34 5Xog== X-Gm-Message-State: ACgBeo096JrzQ9fCsjvuVBywZsrDEfPeSrhT1lfvS5PAQVwznaXmAVK+ 5l7qtfhVxO6p2LsxVxo+jveVVL8msAg= X-Google-Smtp-Source: AA6agR7n//3fxGZjvfyML08bNSpeGfau+aTgkZu5QxUJzLJc9oUWAiawZA1UWbNFrnWa2xxdQVaE06YdPF8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:18c:b0:695:9d04:c3e with SMTP id t12-20020a056902018c00b006959d040c3emr19907526ybh.58.1662053776523; Thu, 01 Sep 2022 10:36:16 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:09 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-22-surenb@google.com> Subject: [RFC PATCH RESEND 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_183620_864639_04897B54 X-CRM114-Status: GOOD ( 16.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma = __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be stable + * and not isolated. If the VMA is not found of is being modified the function + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma = find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. -- 2.37.2.789.g6183377224-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66247ECAAD3 for ; Thu, 1 Sep 2022 17:36:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFCEE8002B; Thu, 1 Sep 2022 13:36:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E85808000D; Thu, 1 Sep 2022 13:36:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD7C78002B; Thu, 1 Sep 2022 13:36:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B64338000D for ; Thu, 1 Sep 2022 13:36:17 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 80C7EAABF2 for ; Thu, 1 Sep 2022 17:36:17 +0000 (UTC) X-FDA: 79864220394.14.EE159C0 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 343D3C0055 for ; Thu, 1 Sep 2022 17:36:17 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id s15-20020a5b044f000000b00680c4eb89f1so4880039ybp.7 for ; Thu, 01 Sep 2022 10:36:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=eEZ4BqXkAmPCIUFHBPa2tHlcApvv3HMidc3eCYWOXuAzHrpAN0HS0lhYeZxby9RzBR 0w6QoRpV+iKQz37VUGml6Y4ETbNA7OPStJ8QApy5HYv7GzeLDO7+EvEq5dUOIcaM6UVC 7rLgM8Qnok81T8RLLnPTDqNW3fzyiOCIgOk9erIOBMCvX1CoI/RbBp7yoBH88B/Pqkwt H8ojE/1F974lnPVAUpGISzyJjuOBuorgNT9VfkfOyqVqOEVMNKxpaqT01PsyXu4rsbpt stRow1K+x3Cpkks4NKMTKIZbGi9T2v1db4rNZMcbp4CaDTkAzPc4JsqrDHDLe5K5oO1C a6Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=Shuuzb1OefFNIm2DGEK32sMQkCT3xrZoo3q/+H11bIE0tSCJOrJyj9Shr0LNUBzsa6 HXqWoKz8aYiF25kJN9fxDtTX0WMJ5oavVkXLPq8czzwEmM+D6ctnPrqxt8uuGg6s/anD exCPG2qYI5b/uR2St9X8ZaUWOnHQOCfEzWh/Gs8AojfDKArCi/rkSnMz6fQiLAhIDDvy 6R4nhIfNG2dCgfWUm0hnGzzTEnKvFzH3ttN+vtfuT8ooAaP5BfSobtghRSlUX2U7z6XB QSElQt9Vb8P9s9A6VOOpE0YWSgUnr1c4rLgwOzYFl/MWIsvBuhUP1sYeuvYj5kBsEuun jIzw== X-Gm-Message-State: ACgBeo38gaxfTQOs+x3EmqebATwEAD0pBnTZLdfhDwjdHPMfj6vexKDH LBVG/Kpv0uDsKmwCVn4S9H/Z6DdWvW8= X-Google-Smtp-Source: AA6agR7n//3fxGZjvfyML08bNSpeGfau+aTgkZu5QxUJzLJc9oUWAiawZA1UWbNFrnWa2xxdQVaE06YdPF8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:1bfc:e7ee:6530:4449]) (user=surenb job=sendgmr) by 2002:a05:6902:18c:b0:695:9d04:c3e with SMTP id t12-20020a056902018c00b006959d040c3emr19907526ybh.58.1662053776523; Thu, 01 Sep 2022 10:36:16 -0700 (PDT) Date: Thu, 1 Sep 2022 10:35:09 -0700 In-Reply-To: <20220901173516.702122-1-surenb@google.com> Mime-Version: 1.0 References: <20220901173516.702122-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.789.g6183377224-goog Message-ID: <20220901173516.702122-22-surenb@google.com> Subject: [RFC PATCH RESEND 21/28] mm: introduce find_and_lock_anon_vma to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662053777; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BJfct7EED8CP836t+WrOyNhtwjRjF3r9ZgkUbpnfoug=; b=wInUPXcMW90PaWqfO2Xhd/HUF3Vi9mskVH7u9Dydu070zEU1/s3jNv/LdAmak9hmHE26Tp vrzC0YwhEWQyH1849dKIE0ZY22U0iyF6Xx2GvlCFzGQJXO8xrrl6PTg5HUdW/AbgYqCMoS 2kDeZngvQGHVjHCJJylk9zjO2xa+39s= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=eEZ4BqXk; spf=pass (imf22.hostedemail.com: domain of 3kO0QYwYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3kO0QYwYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662053777; a=rsa-sha256; cv=none; b=oP4dL7CPIJuCoeeOFt4oX9tPRemrSqGZsCqt9wrRvrF/ubdczZYoziw1urrArhseX2hd7I Z2AJ6SnlLTc3hKtuoaY4OGmZTRtmv7hfHmXlMUKcrRc6Bn5SIS9jxv5acOIUvGCtIM0x6x wW/vfnGIWfwVSSypafNuBnY+m89/Fvo= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=eEZ4BqXk; spf=pass (imf22.hostedemail.com: domain of 3kO0QYwYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3kO0QYwYKCM0BDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Stat-Signature: b4oc38kuuswsgrpjdf545nen13xxbiby X-Rspamd-Queue-Id: 343D3C0055 X-Rspamd-Server: rspam05 X-HE-Tag: 1662053777-458351 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce find_and_lock_anon_vma function to lookup and lock an anonymous VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7c3190eaabd7..a3cbaa7b9119 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 29d2f49f922a..bf557f7056de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5183,6 +5183,51 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +static inline struct vm_area_struct *find_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma = __find_vma(mm, address); + + if (!vma || vma->vm_start > address) + return NULL; + + if (!vma_is_anonymous(vma)) + return NULL; + + if (!vma_read_trylock(vma)) { + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; + } + + /* Check if the VMA got isolated after we found it */ + if (RB_EMPTY_NODE(&vma->vm_rb)) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + return NULL; + } + + return vma; +} + +/* + * Lookup and lock and anonymous VMA. Returned VMA is guaranteed to be stable + * and not isolated. If the VMA is not found of is being modified the function + * returns NULL. + */ +struct vm_area_struct *find_and_lock_anon_vma(struct mm_struct *mm, + unsigned long address) +{ + struct vm_area_struct *vma; + + rcu_read_lock(); + vma = find_vma_under_rcu(mm, address); + rcu_read_unlock(); + + return vma; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. -- 2.37.2.789.g6183377224-goog