Re: [PATCH v2] iommu/arm-smmu-v3: Add a user-configurable tlb_invalidate_threshold

From: Nicolin Chen <nicolinc@nvidia.com>
To: Will Deacon <will@kernel.org>
Cc: <robin.murphy@arm.com>, <jgg@nvidia.com>, <joro@8bytes.org>,
	<jean-philippe@linaro.org>, <apopple@nvidia.com>,
	<linux-kernel@vger.kernel.org>,
	<linux-arm-kernel@lists.infradead.org>, <iommu@lists.linux.dev>
Subject: Re: [PATCH v2] iommu/arm-smmu-v3: Add a user-configurable tlb_invalidate_threshold
Date: Fri, 18 Aug 2023 10:18:50 -0700	[thread overview]
Message-ID: <ZN+n+lwM655TVeJp@Asurada-Nvidia> (raw)
In-Reply-To: <20230818161119.GA16216@willie-the-truck>

On Fri, Aug 18, 2023 at 05:11:19PM +0100, Will Deacon wrote:

> On Thu, Aug 17, 2023 at 11:36:18AM -0700, Nicolin Chen wrote:
> > On Wed, Aug 16, 2023 at 01:43:50PM -0700, Nicolin Chen wrote:
> >
> > > When receiving an __arm_smmu_tlb_inv_range() call with a large size, there
> > > could be a long latency at this function call: one part is coming from a
> > > large software overhead in the routine of building commands, and the other
> > > part is coming from CMDQ hardware consuming the large number of commands.
> > > This latency could be significantly large on an SMMU that does not support
> > > range invalidation commands, i.e. no ARM_SMMU_FEAT_RANGE_INV.
> > >
> > > One way to optimize this is to replace a large number of VA invalidation
> > > commands with one single per-asid invalidation command, when the requested
> > > size reaches a threshold. This threshold can be configurable depending on
> > > the SMMU implementaion.
> >
> > I'm rethinking about this size-based threshold, since what really
> > affects the latency is the number of the invalidation commands in
> > the request. So having an npages-based threshold might be optimal,
> > though the idea and implementation would be similar.
> 
> On the CPU side, we just have:
> 
> #define MAX_TLBI_OPS    PTRS_PER_PTE
> 
> in asm/tlbflush.h
> 
> Can we start off with something similar for the SMMU? I'm not massively
> keen on exposing this as a knob to userspace, because I don't think most
> people will have a clue about how to tune it.

Yes! I was hesitating about an arbitrary threshold setup that actually
fits our situation better. Now it makes sense. Thanks for the input!

What I would do was to pick a number based on our test results. Yet, it
seems that 1024 was chosen at the beginning to fix a softlock bug, and
it changed to PTRS_PER_PTE for a better perf, IIUIC. Should we use the
similar setup for SMMU? I found it is stored in data->bits_per_level,
so perhaps pass it back to the driver via "struct io_pgtable_cfg".

Nicolin

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel