From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3564C433DF for ; Wed, 8 Jul 2020 18:46:50 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 96B1D2065D for ; Wed, 8 Jul 2020 18:46:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="27xpiEhD"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="FtcQGpNT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 96B1D2065D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=0CF7qMciLTsFPhGH7rBc3J0/pQwz3GUzrXUH/q+1iWk=; b=27xpiEhDE8Mb/BgihPhBXseIR 9wLmKVGNlOqh4jEFdZut9Go5XkGAyMOhZbst0VFrzZXuqUIlU1a6D+PfU+h6fxsPlA+r7re6WBYrE cZCsEXFtV1Gd0HTGu5U2/188bu5H/HaAEAYAywuRFooAEv7YQnxt/2+KZd4mHFZT/62TNlsW0IVLN keKYmcds02R9j8R1qgBmFwYcw/7xcTAgLLikMG6RsMeV3zUdFqBe/l7kHIu6LrXsicxLzPkOfilPd bOX6i2ZXvjPWf5/81APa0ihL/iUz/tnGOp3BLB3J8SpOG101LWD8jlw41xNgPrUAkjBqMxPZ4nfD8 tQwpwojhA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jtF4Q-00049S-II; Wed, 08 Jul 2020 18:45:26 +0000 Received: from casper.infradead.org ([90.155.50.34]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jtF4O-00044L-99 for linux-arm-kernel@merlin.infradead.org; Wed, 08 Jul 2020 18:45:24 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=s2pKaGL9kvlrbTL+GPpRHVEkdupodvLZZslUErCcW6Y=; b=FtcQGpNTiyIp5+Z0JRN/rrRGtH AtYeYCkg3ktGQ5qMov1qX/ohExd8NdQBdZQF75DUwdezDbNE2rXfRxtNDmCtss7Hyfq3ec7S2OG/g zchIb/dcn9cZNTA5tSy3FusS0A2YtnlkxtXa0B4IgYSx+4/1v701v6BqGQRb6JvSKiQwjgZp+ulGn 8I8ihC+vEOU/Zp4Mg9GoLjqk3n0Ai1SgSsfeDpo1L7A7caVApIMHxSPzBR9kc3F/8HSF1JgKk3/Tc YAK1JNs2WU72GTC4wzYYBf9OhEU1uIoccrHthAdlu+FT0n4NRbINk+mJd7EJbiVnWnkVHh9VimP+F aGS8VZHw==; Received: from mail.kernel.org ([198.145.29.99]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jtEkg-0003GQ-23 for linux-arm-kernel@lists.infradead.org; Wed, 08 Jul 2020 18:25:15 +0000 Received: from gaia (unknown [95.146.230.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 06BF2206F6; Wed, 8 Jul 2020 18:24:53 +0000 (UTC) Date: Wed, 8 Jul 2020 19:24:51 +0100 From: Catalin Marinas To: Zhenyu Ye Subject: Re: [RFC PATCH v5 2/2] arm64: tlb: Use the TLBI RANGE feature in arm64 Message-ID: <20200708182451.GF6308@gaia> References: <20200708124031.1414-1-yezhenyu2@huawei.com> <20200708124031.1414-3-yezhenyu2@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200708124031.1414-3-yezhenyu2@huawei.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200708_192503_742197_28C295DB X-CRM114-Status: GOOD ( 42.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, suzuki.poulose@arm.com, maz@kernel.org, linux-kernel@vger.kernel.org, xiexiangyou@huawei.com, steven.price@arm.com, zhangshaokun@hisilicon.com, linux-mm@kvack.org, arm@kernel.org, prime.zeng@hisilicon.com, guohanjun@huawei.com, olof@lixom.net, kuhn.chenqun@huawei.com, will@kernel.org, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jul 08, 2020 at 08:40:31PM +0800, Zhenyu Ye wrote: > Add __TLBI_VADDR_RANGE macro and rewrite __flush_tlb_range(). > > In this patch, we only use the TLBI RANGE feature if the stride == PAGE_SIZE, > because when stride > PAGE_SIZE, usually only a small number of pages need > to be flushed and classic tlbi intructions are more effective. Why are they more effective? I guess a range op would work on this as well, say unmapping a large THP range. If we ignore this stride == PAGE_SIZE, it could make the code easier to read. > We can also use 'end - start < threshold number' to decide which way > to go, however, different hardware may have different thresholds, so > I'm not sure if this is feasible. > > Signed-off-by: Zhenyu Ye > --- > arch/arm64/include/asm/tlbflush.h | 104 ++++++++++++++++++++++++++---- > 1 file changed, 90 insertions(+), 14 deletions(-) Could you please rebase these patches on top of the arm64 for-next/tlbi branch: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-next/tlbi > diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h > index bc3949064725..30975ddb8f06 100644 > --- a/arch/arm64/include/asm/tlbflush.h > +++ b/arch/arm64/include/asm/tlbflush.h > @@ -50,6 +50,16 @@ > __tlbi(op, (arg) | USER_ASID_FLAG); \ > } while (0) > > +#define __tlbi_last_level(op1, op2, arg, last_level) do { \ > + if (last_level) { \ > + __tlbi(op1, arg); \ > + __tlbi_user(op1, arg); \ > + } else { \ > + __tlbi(op2, arg); \ > + __tlbi_user(op2, arg); \ > + } \ > +} while (0) > + > /* This macro creates a properly formatted VA operand for the TLBI */ > #define __TLBI_VADDR(addr, asid) \ > ({ \ > @@ -59,6 +69,60 @@ > __ta; \ > }) > > +/* > + * Get translation granule of the system, which is decided by > + * PAGE_SIZE. Used by TTL. > + * - 4KB : 1 > + * - 16KB : 2 > + * - 64KB : 3 > + */ > +static inline unsigned long get_trans_granule(void) > +{ > + switch (PAGE_SIZE) { > + case SZ_4K: > + return 1; > + case SZ_16K: > + return 2; > + case SZ_64K: > + return 3; > + default: > + return 0; > + } > +} Maybe you can factor out this switch statement in the for-next/tlbi branch to be shared with TTL. > +/* > + * This macro creates a properly formatted VA operand for the TLBI RANGE. > + * The value bit assignments are: > + * > + * +----------+------+-------+-------+-------+----------------------+ > + * | ASID | TG | SCALE | NUM | TTL | BADDR | > + * +-----------------+-------+-------+-------+----------------------+ > + * |63 48|47 46|45 44|43 39|38 37|36 0| > + * > + * The address range is determined by below formula: > + * [BADDR, BADDR + (NUM + 1) * 2^(5*SCALE + 1) * PAGESIZE) > + * > + */ > +#define __TLBI_VADDR_RANGE(addr, asid, scale, num, ttl) \ I don't see a non-zero ttl passed to this macro but I suspect this would change if based on top of the TTL patches. > + ({ \ > + unsigned long __ta = (addr) >> PAGE_SHIFT; \ > + __ta &= GENMASK_ULL(36, 0); \ > + __ta |= (unsigned long)(ttl) << 37; \ > + __ta |= (unsigned long)(num) << 39; \ > + __ta |= (unsigned long)(scale) << 44; \ > + __ta |= get_trans_granule() << 46; \ > + __ta |= (unsigned long)(asid) << 48; \ > + __ta; \ > + }) > + > +/* These macros are used by the TLBI RANGE feature. */ > +#define __TLBI_RANGE_PAGES(num, scale) (((num) + 1) << (5 * (scale) + 1)) > +#define MAX_TLBI_RANGE_PAGES __TLBI_RANGE_PAGES(31, 3) > + > +#define TLBI_RANGE_MASK GENMASK_ULL(4, 0) > +#define __TLBI_RANGE_NUM(range, scale) \ > + (((range) >> (5 * (scale) + 1)) & TLBI_RANGE_MASK) > + > /* > * TLB Invalidation > * ================ > @@ -181,32 +245,44 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, > unsigned long start, unsigned long end, > unsigned long stride, bool last_level) > { > + int num = 0; > + int scale = 0; > unsigned long asid = ASID(vma->vm_mm); > unsigned long addr; > + unsigned long range_pages; > > start = round_down(start, stride); > end = round_up(end, stride); > + range_pages = (end - start) >> PAGE_SHIFT; > > - if ((end - start) >= (MAX_TLBI_OPS * stride)) { > + if ((!cpus_have_const_cap(ARM64_HAS_TLBI_RANGE) && > + (end - start) >= (MAX_TLBI_OPS * stride)) || > + range_pages >= MAX_TLBI_RANGE_PAGES) { > flush_tlb_mm(vma->vm_mm); > return; > } Is there any value in this range_pages check here? What's the value of MAX_TLBI_RANGE_PAGES? If we have TLBI range ops, we make a decision here but without including the stride. Further down we use the stride to skip the TLBI range ops. > > - /* Convert the stride into units of 4k */ > - stride >>= 12; > - > - start = __TLBI_VADDR(start, asid); > - end = __TLBI_VADDR(end, asid); > - > dsb(ishst); > - for (addr = start; addr < end; addr += stride) { > - if (last_level) { > - __tlbi(vale1is, addr); > - __tlbi_user(vale1is, addr); > - } else { > - __tlbi(vae1is, addr); > - __tlbi_user(vae1is, addr); > + while (range_pages > 0) { BTW, I think we can even drop the "range_" from range_pages, it's just the number of pages. > + if (cpus_have_const_cap(ARM64_HAS_TLBI_RANGE) && > + stride == PAGE_SIZE && range_pages % 2 == 0) { > + num = __TLBI_RANGE_NUM(range_pages, scale) - 1; > + if (num >= 0) { > + addr = __TLBI_VADDR_RANGE(start, asid, scale, > + num, 0); > + __tlbi_last_level(rvale1is, rvae1is, addr, > + last_level); > + start += __TLBI_RANGE_PAGES(num, scale) << PAGE_SHIFT; > + range_pages -= __TLBI_RANGE_PAGES(num, scale); > + } > + scale++; > + continue; > } > + > + addr = __TLBI_VADDR(start, asid); > + __tlbi_last_level(vale1is, vae1is, addr, last_level); > + start += stride; > + range_pages -= stride >> PAGE_SHIFT; > } > dsb(ish); > } I think the algorithm is correct, though I need to work it out on a piece of paper. The code could benefit from some comments (above the loop) on how the range is built and the right scale found. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel