From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26181C7EE24 for ; Tue, 16 May 2023 17:56:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:References:Message-Id:Cc:Date: In-Reply-To:From:Subject:Mime-Version:Reply-To:Content-ID:Content-Description :Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DmiIwxEH2f/vx2ikjI3rzD4f6AemihYaY51pChRIzUk=; b=g3Dta7gNFRwGwX txKGiJQlj9/QQ1Q7cC4MO4+eWIpUzETdaEB2WfqhsLW/L1QIqu81xxkwlxhhGbKpMpYAq/N30qhHP TOhNwm30E8BX8ziTm8tQBpRo470mIrawsLXKGPcSflAbpFnqKvdhR+mGTFkyv/usdfmj3Z173wNyF UZOxHjBNT90hWDIuRMalUyO3FJp44uVn4nfi4TqD1zHiQT23CPnnSfRk3eghYksmW+yW0eBt8LyZ8 XGEtYRxd5uag2XXB9PNFcG1gwjatqiQmn/aefFtubMTFPNIvDuCiJ3S5BgHauqTBlFFKStXgdey5C sdJ16D8cLmB7KSmCUuTg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pyyuS-006hSo-09; Tue, 16 May 2023 17:56:28 +0000 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pyyuP-006hR9-0n for linux-arm-kernel@lists.infradead.org; Tue, 16 May 2023 17:56:26 +0000 Received: by mail-pg1-x52d.google.com with SMTP id 41be03b00d2f7-51b4ef5378bso13076254a12.1 for ; Tue, 16 May 2023 10:56:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684259782; x=1686851782; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=j/iFIFR/+4isEeFtiLhNrK6Bv+z1ClXHKbQ3ARD5xuk=; b=dGUyYABOLXdUYYVdLg6OFvUawGxNb7Mo6cTILUP40mjazexDA8y1Ym4PywRxYTCwUr 3ceafX9F13xg5M/zY7dQgwHGcaKCwtxikaPO5kO2z8VM1Sb9BfQ6xDg4+3dNtAVtN7V7 N0APiaxFopQagPmr7eYHduuN5G2gGShKbNka0vDjYN0i+2D9X/64Ora41BpfKanhd+/W aa8WDIxy/zY0oKrrzpt7VRpS6nBdkRZj1lJQqRlZw40h7+A/ISS6v3kKQfFG4j80uVkP iICEKqiGFFxQw9F98skN/KjX60V8CQgRgjV22LUlZYC3pfQlpRbZtGUamDLnBfu60JQf wiFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684259782; x=1686851782; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j/iFIFR/+4isEeFtiLhNrK6Bv+z1ClXHKbQ3ARD5xuk=; b=UBfVMOZRCrjkr63ARWsBH2ZCf+TQEPgSx/lqP2Gfj8WkCQ/3lH7b5VITYlXUlhWqHO CV+2p4WOynhM919SCMk0fF5+2rdMUdmDw47WhBipVpmthIcfXri9q8mnh0bDEJxao9HV m/YxF9qbHYsvgfyztHAmlfkLzCKhMv0zmxkhR8MSS/arOBaK+fGKs8Exw+5SPblvrNQh Xq2m30dJpP8Z/ZTUD83cACxK1zkD5du9fW1ZTjo1s8Ce8RLQmnhCT6pt9qzORZmhXYc0 QILkOIUtP9ftz2rCfR7kAHnxk8sDPkSIjdFbymCnsTL3vkS81sH2r9ejqxNXOmqt1KnX ZjfA== X-Gm-Message-State: AC+VfDxQw8C/rN7ZXVsB2EAdjed2Nz1rLvb4RPEiwPY4PRqrsCC1TMFX W8V8galsoMZ/4UmbtQwe7CA= X-Google-Smtp-Source: ACHHUZ7iNSPXkRz8NXxhyTXGh/qhWMr6tVaXBfw8FD0k542CNtX3yI3G/dysbJXUHk/S9n+YBqD+2A== X-Received: by 2002:a17:902:8691:b0:19a:9890:eac6 with SMTP id g17-20020a170902869100b0019a9890eac6mr36061539plo.24.1684259781861; Tue, 16 May 2023 10:56:21 -0700 (PDT) Received: from smtpclient.apple (c-24-6-216-183.hsd1.ca.comcast.net. [24.6.216.183]) by smtp.gmail.com with ESMTPSA id u4-20020a170902e5c400b001ae197fdbb2sm4713277plf.274.2023.05.16.10.56.19 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 May 2023 10:56:21 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: Excessive TLB flush ranges From: Nadav Amit In-Reply-To: <87o7mk733x.ffs@tglx> Date: Tue, 16 May 2023 10:56:08 -0700 Cc: Uladzislau Rezki , "Russell King (Oracle)" , Andrew Morton , linux-mm , Christoph Hellwig , Lorenzo Stoakes , Peter Zijlstra , Baoquan He , John Ogness , linux-arm-kernel@lists.infradead.org, Mark Rutland , Marc Zyngier , x86@kernel.org Message-Id: <7ED917BC-420F-47D4-8956-8984205A75F0@gmail.com> References: <87a5y5a6kj.ffs@tglx> <87353x9y3l.ffs@tglx> <87zg658fla.ffs@tglx> <87r0rg93z5.ffs@tglx> <87cz308y3s.ffs@tglx> <87y1lo7a0z.ffs@tglx> <87o7mk733x.ffs@tglx> To: Thomas Gleixner X-Mailer: Apple Mail (2.3731.500.231) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230516_105625_284771_7E733B09 X-CRM114-Status: GOOD ( 20.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org > On May 16, 2023, at 7:38 AM, Thomas Gleixner wrote: > > There is a world outside of x86, but even on x86 it's borderline silly > to take the whole TLB out when you can flush 3 TLB entries one by one > with exactly the same number of IPIs, i.e. _one_. No? I just want to re-raise points that were made in the past, including in the discussion that I sent before and match my experience. Feel free to reject them, but I think you should not ignore them. In a nutshell, there is a tradeoff which is non-trivial. Controlling the exact ranges that need to be flushed might require, especially in IPI-based TLB invalidation systems, additional logic and more cache lines that need to traverse between the caches. The latter - the cache-lines that hold the ranges that need to be flushed - are the main issue. They might induce overhead that negates the benefits if in most cases it turns out that many pages are flushed. Data structures such as linked-lists might therefore not be suitable to hold the ranges that need to be flushed, as they are not cache-friendly. The data that is transferred between the cores to indicate which ranges should be flushed would ideally be cache line aligned and fit into a single cache-line. It is possible that for kernel ranges, where the stride is always a base-page size (4KB on x86) you might come with more condense way of communicating TLB flushing ranges of kernel pages than userspace pages. Perhaps the workload characteristics are different. But it should be noticed that major parts of the rationale behind the changes that you suggest could also apply to TLB invalidations of userspace mapping, as done in tlb_gather and UBC mechanisms. But in those cases the rationale, at least for x86, was that since the CPU knows to do TLB refills very efficiently, the extra complexity and overheads are likely not to worth the trouble. I hope my feedback is useful. Here is again a link to a discussion from 2015 about this subject: https://lore.kernel.org/all/CA+55aFwVUkdaf0_rBk7uJHQjWXu+OcLTHc6FKuCn0Cb2Kvg9NA@mail.gmail.com/ There are several patches that showed the benefit of reducing cache contention during TLB shootdown. Here is one for example: https://lore.kernel.org/all/20190423065706.15430-1-namit@vmware.com/ _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel