From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2EA6BC4332F for ; Mon, 28 Nov 2022 17:01:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=N40Ah+dAKlerD630zJdgIO9WXtfq5Lty9lrCeGilYLY=; b=P/PZIv3Iqb4Rm5 duLn0vHNYSY6kjZ5rcxjnQf0TlzLt8QxGCV7wHcYeULcHV8IiC4u5iLfLN5jGsQv4ZG2S29g7sdkJ 7Cm9+GobcBirV7oUM7ntBDEekEYQNLljzJSw8ni/mxJit7Gu97SIalXIA3iA/ydA6hgrxDa8n/yXo PpwmPeMP3LYM9fPrTqxJJ8BQy9UtT7EuqSeJoXYM8H67AMLrYvaOEq5WJciUiurdNKCnUzL7ffqKg F1gOfzxkJTeQokbMiFX+58cPuV1oV93NVFdvvF7AohzgyPM85SKetMHRF0Z2gqX/xX8oOYwUvRe0H 4zC+FrcF9nEctrpP3Gbw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ozhUo-002xzG-AM; Mon, 28 Nov 2022 17:00:42 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ozhUl-002xy2-OM for linux-arm-kernel@lists.infradead.org; Mon, 28 Nov 2022 17:00:41 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 565F6B80E57; Mon, 28 Nov 2022 17:00:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9379C433C1; Mon, 28 Nov 2022 17:00:34 +0000 (UTC) Date: Mon, 28 Nov 2022 17:00:31 +0000 From: Catalin Marinas To: Yunfeng Ye Cc: will@kernel.org, wangkefeng.wang@huawei.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linfeilong@huawei.com Subject: Re: [PATCH 4/5] arm64: mm: Support ASID isolation feature Message-ID: References: <20221017083203.3690346-1-yeyunfeng@huawei.com> <20221017083203.3690346-5-yeyunfeng@huawei.com> <3607b658-304a-ecc8-b07a-530f4a6365e8@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <3607b658-304a-ecc8-b07a-530f4a6365e8@huawei.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221128_090039_980949_58A43150 X-CRM114-Status: GOOD ( 29.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Nov 10, 2022 at 03:07:53PM +0800, Yunfeng Ye wrote: > On 2022/11/9 20:43, Catalin Marinas wrote: > > On Mon, Oct 17, 2022 at 04:32:02PM +0800, Yunfeng Ye wrote: > >> After a rollover, the global generation will be flushed, which will > >> cause the process mm->context.id on all CPUs do not match the > >> generation. Thus, the process will compete for the global spinlock lock > >> to reallocate a new ASID and refresh the TLBs of all CPUs on context > >> switch. This will lead to the increase of scheduling delay and TLB miss. > >> > >> In some delay-sensitive scenarios, for example, part of CPUs are > >> isolated, only a limited number of processes are deployed to run on the > >> isolated CPUs. In this case, we do not want these key processes to be > >> affected by the rollover of ASID. > > > > Part of this commit log should also go in the cover letter and it would> help to back this up by some numbers, e.g. what percentage improvement > > you get with this patchset by running hackbench on an isolated CPU. > > > > In theory it looks like CPU isolation would benefit from this patchset > > but we try not to touch this code often, so any modification should come > > with proper justification, backed by numbers. > > > Yes, CPU isolation will benefit from this patchset. We use cyclictest tool > to test the maximum scheduling and interrupt delays, found that the > sched_switch process takes several microseconds sometimes, The analysis > result shows that the delay is caused by the ASID refresh. Do you know whether it's predominantly the spinlock or the TLBI that's causing this (or just a combination of the two)? I was talking to Will and concluded we should try to reuse the ASID pinning code that's already in that file rather than adding a new bitmap. At a high level, a thread migrating to an isolated CPU can have its ASID pinned. If context switching only happens between pinned ASIDs on an isolated CPU, we may be able to avoid the lock even if the generation rolled over on another CPU. I think the tricky problem is when a pinned ASID task eventually dies, possibly after migrating to another CPU. If we avoided the TLBI on generation roll-over for the isolated CPU, it will have stale entries. One option would be to broadcast a TLBI for the pinned ASID when the task dies, though this would introduce some jitter. An alternative may be to track whether a pinned ASID ever run on a CPU and do a local TLBI for that ASID when a pinned thread is migrated. All these need a lot more thinking and (formal) modelling. I have a TLA+ model but I haven't updated it to cover the pinned ASIDs. Or, alternatively, make the current code stand-alone and get it through CBMC (faking the spinlock as pthread mutexes and implementing some of the atomics in plain C with __CPROVER_atomic_begin/end). -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel