From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94205C6FD1D for ; Tue, 4 Apr 2023 14:04:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sinsqnISBNUUYaIkwWos4L6jfxlbkL/WcPo038wDha8=; b=l4mXAKV7DKZKP9 vIlL1GcVr3jhl8dBtucT8abW4oXAG8hVg88UItT/OG3eIxp2mHdNIncqs/hacVCkflf6Mge/VWwNO XoFuW60zWZHGZh0Bz3CsZ0XM0Ig6mewflTM+fhA0pETBrt2EyLYqj71uJgVepOw1cE2uCYu9eIboD IIULHeZmP/Vts0VslNNDf8SyITxpKavqTjjF15WZ3kRhpbVtYvLIN7unhm72WDLiXxTzhb5igoyHB Klm2utlGTOfIOD5kPMXxZnqHJ2DIxS9iJTeg560He7ki7BVg6tQyjf2LTcGJuu+62+1Jsbw7lUlUF uhL/i6cikLU5OfB5x9wQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pjhGM-001ih8-0g; Tue, 04 Apr 2023 14:03:54 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pjhGG-001iew-30 for linux-arm-kernel@lists.infradead.org; Tue, 04 Apr 2023 14:03:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680617027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+eiQuy0297qofRfcSxKqcQyipv7GSPC3sFo1rtx/hWs=; b=CrZTCzenPYvnJaYQ50fTbNEDOARQ7I/WnFf5Oxf/bEsMeHV0XoU6pKnGkzSpRiIX/jczD8 s+cgPaEst6P43BZt/9OITdHHdFC0OV9JEF+ff6gj3KypZIel0W3XCB6Jw9qv9U8SGaspy1 P/MRBEy8upsV0iUMqaVce6UloDCqatw= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-528-8GHL4VfjN7eZVnFyNeBw6g-1; Tue, 04 Apr 2023 10:03:46 -0400 X-MC-Unique: 8GHL4VfjN7eZVnFyNeBw6g-1 Received: by mail-wm1-f69.google.com with SMTP id m5-20020a05600c4f4500b003ee8db23ef9so17668108wmq.8 for ; Tue, 04 Apr 2023 07:03:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680617022; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+eiQuy0297qofRfcSxKqcQyipv7GSPC3sFo1rtx/hWs=; b=HLQuK87cajNyK8w5wpSd7Y4Jtzdm7dJINTgb7/MvVFaoD1s+FFc89AMBIGlUDrsQv5 yTWoWTpjGUW1uWSP9WcrItHk2o2KLEV7CgzUW170MPHLqGoLzRnudtsWaSmCI/temB8t 72hXMipP+JohQnFrruTf0fQT8JqrF0mxTP4esvnmwRr59rGdWrYMrhbF7/OPKEQ4Vc7S ZDkUW8GPGxZpD8Y0Lw3hrtg6Gq1McvZGUPXZv1kgC6mtCS9sX8Y572JZMs1aQ55cE/k7 VZXprvEmpMQYHD3z/SoKoJZDyQ+k/lWarb6Cf0w3EC0A8r52yewEPvpKb9kj9ipqylJV gh8g== X-Gm-Message-State: AAQBX9f0plMjfurav+yGhXYXttexYj5vlZ7VZP0jsLVzgRN/VjkRUbJY xKb+JX8KqbjYnWCKbai7Fhinn8xxzSL8dMmrw2ALBBhtQqQEqrEEsQnlAFp90dyIsjETzdHI4Sp X/XKjrj/IRw3X9SRAb/u6x13v6MzLovFin0s= X-Received: by 2002:a5d:6291:0:b0:2d6:5afe:7b99 with SMTP id k17-20020a5d6291000000b002d65afe7b99mr1501600wru.10.1680617021965; Tue, 04 Apr 2023 07:03:41 -0700 (PDT) X-Google-Smtp-Source: AKy350ZgF89ZubYczhCwXf0n6e5MiZ9Gbs+vNGbCQed6l+DJFU8EvLia/3dXAX7rxTgsIMUPc3yreQ== X-Received: by 2002:a5d:6291:0:b0:2d6:5afe:7b99 with SMTP id k17-20020a5d6291000000b002d65afe7b99mr1501576wru.10.1680617021501; Tue, 04 Apr 2023 07:03:41 -0700 (PDT) Received: from ?IPV6:2003:cb:c709:b600:e63:6c3b:7b5d:f439? (p200300cbc709b6000e636c3b7b5df439.dip0.t-ipconnect.de. [2003:cb:c709:b600:e63:6c3b:7b5d:f439]) by smtp.gmail.com with ESMTPSA id i17-20020a5d5591000000b002eaac3a9beesm2694936wrv.8.2023.04.04.07.03.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 04 Apr 2023 07:03:40 -0700 (PDT) Message-ID: Date: Tue, 4 Apr 2023 16:03:38 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 Subject: Re: [PATCH 3/3] mm/mmu_gather: send tlb_remove_table_smp_sync IPI only to CPUs in kernel mode To: Yair Podemsky , linux@armlinux.org.uk, mpe@ellerman.id.au, npiggin@gmail.com, christophe.leroy@csgroup.eu, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, davem@davemloft.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, will@kernel.org, aneesh.kumar@linux.ibm.com, akpm@linux-foundation.org, peterz@infradead.org, arnd@arndb.de, keescook@chromium.org, paulmck@kernel.org, jpoimboe@kernel.org, samitolvanen@google.com, frederic@kernel.org, ardb@kernel.org, juerg.haefliger@canonical.com, rmk+kernel@armlinux.org.uk, geert+renesas@glider.be, tony@atomide.com, linus.walleij@linaro.org, sebastian.reichel@collabora.com, nick.hawkins@hpe.com, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, mtosatti@redhat.com, vschneid@redhat.com, dhildenb@redhat.com Cc: alougovs@redhat.com References: <20230404134224.137038-1-ypodemsk@redhat.com> <20230404134224.137038-4-ypodemsk@redhat.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: <20230404134224.137038-4-ypodemsk@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230404_070349_072185_90286395 X-CRM114-Status: GOOD ( 35.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 04.04.23 15:42, Yair Podemsky wrote: > The tlb_remove_table_smp_sync IPI is used to ensure the outdated tlb page > is not currently being accessed and can be cleared. > This occurs once all CPUs have left the lockless gup code section. > If they reenter the page table walk, the pointers will be to the new > pages. > Therefore the IPI is only needed for CPUs in kernel mode. > By preventing the IPI from being sent to CPUs not in kernel mode, > Latencies are reduced. > > Race conditions considerations: > The context state check is vulnerable to race conditions between the > moment the context state is read to when the IPI is sent (or not). > > Here are these scenarios. > case 1: > CPU-A CPU-B > > state == CONTEXT_KERNEL > int state = atomic_read(&ct->state); > Kernel-exit: > state == CONTEXT_USER > if (state & CT_STATE_MASK == CONTEXT_KERNEL) > > In this case, the IPI will be sent to CPU-B despite it is no longer in > the kernel. The consequence of which would be an unnecessary IPI being > handled by CPU-B, causing a reduction in latency. > This would have been the case every time without this patch. > > case 2: > CPU-A CPU-B > > modify pagetables > tlb_flush (memory barrier) > state == CONTEXT_USER > int state = atomic_read(&ct->state); > Kernel-enter: > state == CONTEXT_KERNEL > READ(pagetable values) > if (state & CT_STATE_MASK == CONTEXT_USER) > > In this case, the IPI will not be sent to CPU-B despite it returning to > the kernel and even reading the pagetable. > However since this CPU-B has entered the pagetable after the > modification it is reading the new, safe values. > > The only case when this IPI is truly necessary is when CPU-B has entered > the lockless gup code section before the pagetable modifications and > has yet to exit them, in which case it is still in the kernel. > > Signed-off-by: Yair Podemsky > --- > mm/mmu_gather.c | 19 +++++++++++++++++-- > 1 file changed, 17 insertions(+), 2 deletions(-) > > diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c > index 5ea9be6fb87c..731d955e152d 100644 > --- a/mm/mmu_gather.c > +++ b/mm/mmu_gather.c > @@ -9,6 +9,7 @@ > #include > #include > #include > +#include > > #include > #include > @@ -191,6 +192,20 @@ static void tlb_remove_table_smp_sync(void *arg) > /* Simply deliver the interrupt */ > } > > + > +#ifdef CONFIG_CONTEXT_TRACKING > +static bool cpu_in_kernel(int cpu, void *info) > +{ > + struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); > + int state = atomic_read(&ct->state); > + /* will return true only for cpus in kernel space */ > + return state & CT_STATE_MASK == CONTEXT_KERNEL; > +} > +#define CONTEXT_PREDICATE cpu_in_kernel > +#else > +#define CONTEXT_PREDICATE NULL > +#endif /* CONFIG_CONTEXT_TRACKING */ > + > #ifdef CONFIG_ARCH_HAS_CPUMASK_BITS > #define REMOVE_TABLE_IPI_MASK mm_cpumask(mm) > #else > @@ -206,8 +221,8 @@ void tlb_remove_table_sync_one(struct mm_struct *mm) > * It is however sufficient for software page-table walkers that rely on > * IRQ disabling. > */ > - on_each_cpu_mask(REMOVE_TABLE_IPI_MASK, tlb_remove_table_smp_sync, > - NULL, true); > + on_each_cpu_cond_mask(CONTEXT_PREDICATE, tlb_remove_table_smp_sync, > + NULL, true, REMOVE_TABLE_IPI_MASK); > } > > static void tlb_remove_table_rcu(struct rcu_head *head) Maybe a bit cleaner by avoiding CONTEXT_PREDICATE, still not completely nice (an empty dummy function "cpu_maybe_in_kernel" might be cleanest but would be slightly slower for !CONFIG_CONTEXT_TRACKING): #ifdef CONFIG_CONTEXT_TRACKING static bool cpu_in_kernel(int cpu, void *info) { struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); int state = atomic_read(&ct->state); /* will return true only for cpus in kernel space */ return state & CT_STATE_MASK == CONTEXT_KERNEL; } #endif /* CONFIG_CONTEXT_TRACKING */ ... #ifdef CONFIG_CONTEXT_TRACKING on_each_cpu_mask(REMOVE_TABLE_IPI_MASK, tlb_remove_table_smp_sync, NULL, true); #else /* CONFIG_CONTEXT_TRACKING */ on_each_cpu_cond_mask(cpu_in_kernel, tlb_remove_table_smp_sync, NULL, true, REMOVE_TABLE_IPI_MASK); #endif /* CONFIG_CONTEXT_TRACKING */ -- Thanks, David / dhildenb _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel