From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31079ECE564 for ; Tue, 10 Sep 2024 10:58:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To: Content-Transfer-Encoding:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vHmk114QIZCjlFSKd2nx+MtW7wZ0G3Zn0tQIsPkVXhc=; b=yoXYeVfNt5hxjcdgTgTLJz1RiI kaih1n2YiXkoIxA64qVzPUsNKtn96Onj2M1cKsROiubVUb3iWUau2uf+VT8I8mXdqwURLntc7vQWf pAoKgcVUXx3HejhNGEC1QHH/izmcgHc4jhhVaEf2pfK14LCrLfdQzRXNs66RNBkMID23FPsLN7pUG a8ui6T5WUiyp/+mVyY0jiVxg/Vq8NKcviljvGERhg4T1DPYoSefzsJgF/ioSPgXTR41siyC+9/X3D YDj4blmjEqSLEnSk05+Wyo9QU0S/lNKnoE+Gvm2A7KOuzGGEo/lkzWqIvZt2i4JVDv+FEMyIkrJVI 1zqG7R4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1snyZq-00000005K1i-2zIg; Tue, 10 Sep 2024 10:58:30 +0000 Received: from mail-ed1-x535.google.com ([2a00:1450:4864:20::535]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1snyXS-00000005JZd-1f8I for linux-arm-kernel@lists.infradead.org; Tue, 10 Sep 2024 10:56:06 +0000 Received: by mail-ed1-x535.google.com with SMTP id 4fb4d7f45d1cf-5c2460e885dso11598a12.0 for ; Tue, 10 Sep 2024 03:56:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725965759; x=1726570559; darn=lists.infradead.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=vHmk114QIZCjlFSKd2nx+MtW7wZ0G3Zn0tQIsPkVXhc=; b=R2h4fIBzLl5ySkkDa+4DkiedDbUbYvein6akVxn9i//R8Q8O+xVSkWIOySPRd1K0l3 SvjOmej+BqVfURWHf6DsAsZaMmwf2SIItFEXGHXN1MoEa5Q9+P2xf/ZhAJ2i/WAYmZb0 oiatdnB3tUK8fQ0iiVvcogjdwOM4IwIZCCXy1gCYOZcxOmZSl/rsooDM9ZnLelL49j3j XKCTSZfYjNVDKmyMOGrNYVZf3RlfaM7g7k72RioDQ6BTdKb0jfBqyOr2W8L52NZjyu0M /KGlb5s+dbELeDgDuBKRW1QOr4BaLQUQWEPv8wNP0MffXCuJvcEekXiaGo4WZYuGWjJc VYUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725965759; x=1726570559; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vHmk114QIZCjlFSKd2nx+MtW7wZ0G3Zn0tQIsPkVXhc=; b=B6Hfnx/+TBYrMKPMLg2CQz3C0w3wVX7NmgcYrod/0rRFepr3mLm99fzK6xFLB1Ezjg XpDdhKr4K6spXeH561y6Gd0G77iEIlcwFJRCQ4GCeAlxOoq5DN5/JsS8M26pQI6S8JCY Xnc62cGKFPj747OWPONik/ywtuAnBDjhmTk8OawJPFz+Ns5cUbVBtCeIDbFEqfdzhvaC MnyOPGEmwZpMcnvtROfRyX5ksIXHC/eiY9j33vgzeVV723GBZxM+mCTY7pZp9/B0f5SK WaOTJ11FJKnHjs+LdcAKNASwwwUkX/6/TAHVpiiI0vU9utF7ulcFVB/pyeXClOjg0jfq B7Hg== X-Forwarded-Encrypted: i=1; AJvYcCVyEYwqTxOo2VySNhVElTldfdJcZYJeuicVTyOLjeYlELLroA3rahVZOJpxKhNrUMJ3RRbnSGJntlZgkLR2pz8G@lists.infradead.org X-Gm-Message-State: AOJu0Yzn4ILXnsj1WSCg8b0gAI8hGGM9/lvFVf91cx+YOtBZ7zYjKe8Q xvp268IvL/sde9v/m54NVh7AvX+ftSeapDeXng9GsYt1Byz07HNMQNsKTu/zww== X-Google-Smtp-Source: AGHT+IH5Z6CO+MKfDYUG1iOU31Fa2e8Ac+77rN2VKHvSd6OZvtfGFA8hOAHmo3xhEkofbaoqsCkAhQ== X-Received: by 2002:a05:6402:34d5:b0:58b:15e4:d786 with SMTP id 4fb4d7f45d1cf-5c4040d9d24mr152948a12.5.1725965758834; Tue, 10 Sep 2024 03:55:58 -0700 (PDT) Received: from google.com (205.215.190.35.bc.googleusercontent.com. [35.190.215.205]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42caeb8afc9sm108460105e9.44.2024.09.10.03.55.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Sep 2024 03:55:58 -0700 (PDT) Date: Tue, 10 Sep 2024 10:55:51 +0000 From: Mostafa Saleh To: Jason Gunthorpe Cc: acpica-devel@lists.linux.dev, Hanjun Guo , iommu@lists.linux.dev, Joerg Roedel , Kevin Tian , kvm@vger.kernel.org, Len Brown , linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Lorenzo Pieralisi , "Rafael J. Wysocki" , Robert Moore , Robin Murphy , Sudeep Holla , Will Deacon , Alex Williamson , Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: Re: [PATCH v2 2/8] iommu/arm-smmu-v3: Use S2FWB when available Message-ID: References: <0-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com> <2-v2-621370057090+91fec-smmuv3_nesting_jgg@nvidia.com> <20240830164019.GU3773488@nvidia.com> <20240903000546.GD3773488@nvidia.com> <20240903233340.GH3773488@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240903233340.GH3773488@nvidia.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240910_035602_487661_58B68C48 X-CRM114-Status: GOOD ( 30.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Sep 03, 2024 at 08:33:40PM -0300, Jason Gunthorpe wrote: > On Tue, Sep 03, 2024 at 07:57:01AM +0000, Mostafa Saleh wrote: > > > Basically, I believe we shouldn’t set FWB blindly just because it’s supported, > > I don’t see how it’s useful for stage-2 only domains. > > And the only problem we can see is some niche scenario where incoming > memory attributes that are already requesting cachable combine to a > different kind of cachable? No, it’s not about the niche scenario, as I mentioned I don’t think we should enable FWB because it just exists. One can argue the opposite, if S2FWB is no different why enable it? AFAIU, FWB would be useful in cases where the hypervisor(or VMM) knows better than the VM, for example some devices MMIO space are emulated so they are normal memory and it’s more efficient to use memory attributes. Taking into consideration all the hassle that can happen if non-coherent devices use the wrong attribute, I’d suggest either set FWB only for coherent devices (I know it’s not easy to define, but maybe be should?) or we have a new CAP where the caller is aware of that. But I don’t think the driver should decide that on behalf of the caller. > > > And I believe making assumptions about VFIO (which actually is not correctly > > enforced at the moment) is fragile. > > VFIO requiring cachable is definately not fragile, and it also sets > the IOMMU_CACHE flag to indicate this. Revising VFIO to allow > non-cachable would be a signficant change and would also change what > IOMMU_CACHE flag it sets. > I meant the driver shouldn't assume the caller behaviour, if it's VFIO or something new. > > and we should only set FWB for coherent > > devices in nested setup only where the VMM(or hypervisor) knows better than > > the VM. > > I don't want to touch the 'only coherent devices' question. Last time > I tried to do that I got told every option was wrong. > > I would be fine to only enable for nesting parent domains. It is > mandatory here and we definitely don't support non-cachable nesting > today. Can we agree on that? > Why is it mandatory? I think a supporting point for this, is that KVM does the same for the CPU, where it enables FWB for VMs if supported. I have this on my list to study if that can be improved. But may be if we are out of options that would be a start. > Keep in mind SMMU S2FWB is really new and probably very little HW > supports it right now. So we are not breaking anything existing > here. IMHO it is better to always enable the stricter features going > forward, and then evaluate an in-kernel opt-out if someone comes with > a concrete use case. > I agree, it’s unlikely that this breaks existing hardware, but I’d be concerned if FWB is enabled unconditionally it breaks devices in the future and we end up restricting it more. Thanks, Mostafa > Jason