From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A9ADC52D6F for ; Wed, 21 Aug 2024 12:07:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To: Content-Transfer-Encoding:Content-Type:References:Message-ID:Subject:Cc:To: From:Date:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=V7+v8nb/xfVy5mAqLlpzuv0u+fOXWKRqbEKZz6c2OE0=; b=Sr4AFmzgpCOMo1NgCyNu2etz3y ntpk7yeCslfCZ5T019WKpzLYHEgavWNrYvWcrELL+lgF6hHXOoFabgJV9Oig3D1aDYFexlKqZ7TzV NAb0yMBntfsObMgLK0O5jP+Dw1ZoZEglDtj70BDwNM6y3bXCGOHbx+S56AHsTaypaRYKIbGyAhcXq z1952rRTIbxkFkKYxRtYzsay844hV7F+1errWcPDubo6hUSRTqj5nLrDFrCqHR+xrKBqkzn+N5op9 39UF/IwZfAI7rKcZmddcSbMCg4mQskV+zwhbip9wJcsm7sya0H7V+MqArlY3l67Z4UkCfxMvIA4/w ibzhVvPw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sgk7I-00000008qa1-2wPF; Wed, 21 Aug 2024 12:07:08 +0000 Received: from mail-co1nam11on20631.outbound.protection.outlook.com ([2a01:111:f403:2416::631] helo=NAM11-CO1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sgk6Y-00000008qLZ-37Ew for linux-arm-kernel@lists.infradead.org; Wed, 21 Aug 2024 12:06:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=QB5NHoTM45Lk7Z7xsXfJAkK0XNQ3V4gkvJe4gwnzVNPh1HkrPprHuZwzUVQLlZD5BW83MEb4ib8nG0e5BT26iIjpA2cMPbs9+c2TYDC8PxUp1FGZ/8DV47DDs4DfTzqRUDJ4CRsvPw0Gmt61iKQq7lliBJkE/C8bwlIHUVW3t8vam78ktNHHDm273/3jqDFEvSUnHTRQQ3i49uSec+YsP0T/QziGLfWzpIdp6hJ+6thSIwzRdwHtfGFJrugoYlZR4hkXuwOXvuvmt9kH/sqKleFzZiyJD2aYTM/dr9F705aNmYrwmzGhu3xnRW1w6sVzJxSYY7Wr4BbAMktOmL+OhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=V7+v8nb/xfVy5mAqLlpzuv0u+fOXWKRqbEKZz6c2OE0=; b=fTh0xTtMLqC28wHk28IilnYSHMH7+hPLaBLEJXMF8l40qn5GK3wecI3q2cux7efBveDkpYJ/vPh+HpzVvQaxGml/1VKHtZVu+IvXfYh1cy+pae0aTIaPoJAAJd59hx1mNaCw/pIeAyZIx4MFpvlFXDJ8VDKYEyDy3dSsB35dkK4E0Sj+z1DtwrRo1RUKngsGj5EB508RIUqKWG20KafZRYqDHkNyUqr5Iw9z2S5f2an3wPl9RaBzuMs/1tZJMd850qdKCfqdaPLRf09/GOcev2+MRbiZbUN4xNRtEHfBrvLvdV/iOaOVlbZqt/jzj7iKb3aqPm3Hhh7uwJIc1AU/Tg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V7+v8nb/xfVy5mAqLlpzuv0u+fOXWKRqbEKZz6c2OE0=; b=XHzYi9r3qq7wJbsF/Rh38F4s84wjhKWGkssP8mD8UxUlk+FXnCHY5a2SyQHpJAbrfBWCE586kNJmMzloCoQAXInKADFV1mvnIlsQWCOiZLRWYOyF8VFBcdbdJx/LWNNCY4X/tjHhcUfuWc/6kIuQlfZ6/MHnYuFglJ/nzP8sroVk1jogwo5d2u+wS4E2HJ8gPiwFASB0qBeRLMiEoB7Ze1c9AZxARAr6slf2KvWmgz2IuqIjMAsPCwFA0BbhDTYg+DVt0aOFTw3QXRafWv+WLcWRSzm+LhKSCpm9AjqAGqJsPpECbETTUi1iDLhlrvvuD6PrE1nFw6oUTg1LYOuRQA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) by SN7PR12MB6887.namprd12.prod.outlook.com (2603:10b6:806:261::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.21; Wed, 21 Aug 2024 12:06:09 +0000 Received: from CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8]) by CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8%3]) with mapi id 15.20.7875.023; Wed, 21 Aug 2024 12:06:09 +0000 Date: Wed, 21 Aug 2024 09:06:07 -0300 From: Jason Gunthorpe To: Mostafa Saleh Cc: acpica-devel@lists.linux.dev, Alex Williamson , Hanjun Guo , iommu@lists.linux.dev, Joerg Roedel , Kevin Tian , kvm@vger.kernel.org, Len Brown , linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Lorenzo Pieralisi , "Rafael J. Wysocki" , Robert Moore , Robin Murphy , Sudeep Holla , Will Deacon , Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi , Marc Zyngier Subject: Re: [PATCH 2/8] iommu/arm-smmu-v3: Use S2FWB when available Message-ID: <20240821120607.GI3773488@nvidia.com> References: <0-v1-54e734311a7f+14f72-smmuv3_nesting_jgg@nvidia.com> <2-v1-54e734311a7f+14f72-smmuv3_nesting_jgg@nvidia.com> <20240820120102.GB3773488@nvidia.com> <20240820202138.GH3773488@nvidia.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: BN0PR10CA0016.namprd10.prod.outlook.com (2603:10b6:408:143::23) To CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB7763:EE_|SN7PR12MB6887:EE_ X-MS-Office365-Filtering-Correlation-Id: 13128dca-b254-4f75-89fd-08dcc1d9a440 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: =?utf-8?B?djI1TUpoUUZOY2VZMDBvMGVFRFBsdlJSdncrWnhJNGVseFNTRGVWSTE5WVRk?= =?utf-8?B?UUlCWDJvSFRNRU1TUkRYSWcyb01EVitPcFFFRGVPekhtdXA3Q2ZsVjI1MHNQ?= =?utf-8?B?TWxPWFdVNlZuOHNxZ0pwQ0pZaWE2MUVLWk5WRnpEUkYrUjBLdEV3Y0F4cU55?= =?utf-8?B?Qm9hM3diSkg1RklGTklXaDRXRTQyVlJwRUczZFVqNmZESWxQVzRNc2hmMlJn?= =?utf-8?B?WnVuMEtiUFBoazhCWkVKeG10VU9pN1AvVW1QcGJEZFhVa2VieTVTNGhuam44?= =?utf-8?B?cFR6MGlENkRuZHJ0Y0NVUjZZMVNPMFdCZC9CRUtMQWVDbmJ4dWh0ejhCNEU0?= =?utf-8?B?MXI4bGZHS3BNSE1QNm9ROTY3QTNlVThDUUM5MGRKTDdQcDVRYUNMRkYxUHk5?= =?utf-8?B?TlF5eW9JVkVqMXZUeExvNHhodnBvZ3JpK3JyWFpyVDI3WXZzWUtRc1RXcTFT?= =?utf-8?B?Q0pPeEcwMitya3A0M3JJckhMUVFIYjYyalUzM1U1Ny9qN3l6S0ptYTA5MlZz?= =?utf-8?B?WW01c3JteTZ0UngrVXZPU3hxYmhzRmtnd29USHhFcmhNZHdQNXBFeDNwM3NU?= =?utf-8?B?bUpQTjlOUmpKZGV5biswREt3RjNnaEM3WXk2YkIrQURJQWl2MExyNEpsMXVG?= =?utf-8?B?ajhCY3gvQWwvamdWcU1pUWQweVFaRFNuaHZiOFZ4OGJ0Z3FtNkh0TXZYOWdw?= =?utf-8?B?M3dIU0FEOFZNdzZJZ3JZS1p2Wk04U3dDdGMxcXViU2FHT1p0RDJwS3AwbmV3?= =?utf-8?B?NE04dDZ0ZUtFNHJxL2szNWxwWjZxd2Q3Qk9zenhhZEZzbE9keldEZkF2OWRs?= =?utf-8?B?Mzhwa0hVdC9ScmRXVm1BdXROakJwS3lTR2pYVU9HVGhKN2t0anUyN1U4dk1C?= =?utf-8?B?U3BWUG9GTU9va2lLbmNhUGtJcXIzZTZVd05mWFVUSk5oNXNKNU10Szd0c1NC?= =?utf-8?B?N2ttTGxucW9JTVA3QjNrUUpXTk1iU2EzMHhmb0I1Tk84ZnBXN2szekdFQ1F3?= =?utf-8?B?ZHFjODhiZWhCQzM5bHFadmw1K0tPSXB1NHorYTR4cndXQmNTYk1wNVdaWG9P?= =?utf-8?B?MDVPbXUzaDdPK2JJOW1OUnhheHVXL1J3dVFBM0F2cC9sMEw3UmdBYldYRU5x?= =?utf-8?B?cW9xdmdCanh0UFgzRE91UjQyQnFZNWN5d2xYVSt0WWRVbFl3NkhRemVJVktL?= =?utf-8?B?YUpaczIzZWs4RFloT0VUdkM3Tm1RS2RBQXU5YjlTTElSTG1MaldSdWZYSDY3?= =?utf-8?B?b0svK3loVVhzMlpuL3pna0VkNkhNYUhlVHZvMVg5SjZ1U1VhcmNQcFdnMXpX?= =?utf-8?B?SGNDdS9QWVlMaU9NRzVJaFhOazRMTDB2SDY2bkdOeXVzN0NWMmFkTkZTcDFh?= =?utf-8?B?VVVMVzhwK1hJeUVsUFdsK0tWeUdMNzY0WG5GZ2VKMEdYbDVwTm12clgrcnNu?= =?utf-8?B?TjhtM0RQdWlEOGsza0kyOVVYWElpckZqanZkallWdW5UTGdZZHJsS2k5R3oz?= =?utf-8?B?MmVMWFpQbXJFbmdTSjJmaUo5bEwrLzhhOFpONXB1dmpPdGROUG1hbStwZkt3?= =?utf-8?B?Nnhmc3Jqd1ZuSkdsdTRRdDB0eHNDZUtLbU9JQUNNSjlraUFDMDNaY3Nnb3JB?= =?utf-8?B?YTMrY0F4TkZGaVFIc25Lcm1yOHdHVHNGMmtQMlduOHhieWZISkgrZmh2ZE9U?= =?utf-8?B?aVEwQks4R29YUWt3dzdISXh0Q0JtR1ZEU2haOEZZQWN1NG1Ga2E4cDFaUWF2?= =?utf-8?B?QldZLzFXYnRSSHMxSi85ME5iRmgvOE5RV1RmTDV5RnFHU2tGZlVMOEI4eWNZ?= =?utf-8?B?eDA0Nk5IS09jbGhXNXNBUT09?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB7763.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ajJGVCtqWWhkM3ZNejlKNmJwc3BNRk5LcGh5N3lKSGVFM3lnYlhmRE92T3BF?= =?utf-8?B?RTFxZFkreUw0KzlsVC9uZ1M4bkNpRXF1bWhLVTBtZWlyRS94UGFQSUd0RWR3?= =?utf-8?B?dDA1QnUzOTJaVTFsM3pSdUNzcGdDYkY4UzFFa1RYMGRNSEcwUFJxcUkwVTE3?= =?utf-8?B?UW1uL0VtRWV5N09MTGI1c0Nid2dlNHk5dFVzWGF6WE1XOEoxREl5UTA1WmJH?= =?utf-8?B?SzZ5bVBkSEhvdVBCSllzRThKNmhVWFFOUFdEYnljMXhnQTZ3blZCM05XaStX?= =?utf-8?B?dnBkblNRelFNN1g4b0FPMjVTSmw2UitxRHJ4MzFnUVgxV04xQmF6N1hIVURB?= =?utf-8?B?WDlDUjcxR210dDR0TzU1eHR5REY3L3d1c2ZJTTJkQ1lRNEF4Z0Zsbk1aT2E0?= =?utf-8?B?ejJyUERkd0g1cTBFeFcyNWNFTDIrdHR2STU2QklqTDZkVlVmRWpLUitWNjJB?= =?utf-8?B?NitVV29laFY1aVpBZ0ovenZKUTZIWFJxQzBoQ3IxM1Y2czRlSHZveGgxQnBL?= =?utf-8?B?QWhYK2xXK3ZzQ2pTeUdoWU9IL2FHckthTG9sRUZycXhkRVhMTDhOMUpBZXc5?= =?utf-8?B?aFhoSmJ2SUhpNnFjc3dBM2lRdnlhb2dVNEZ6bGFFaTQxZXY3TVJMMWttTzgv?= =?utf-8?B?VkpieGRhQ010S25uYkhmMXh3cEU2ckdkZ1lKYkpDYUJReGtmUEw1dlFNU00r?= =?utf-8?B?RlFIYTJObzYxdTNzRGwvM2h6eE1ia1UwOGJBUFpIS3RMdWpMT3Q4cGgvOGxh?= =?utf-8?B?azUyWEQ2RUVmU0lvQVFkc25MeStZd2I0S0xXRGY2VzIvSFRDLzQvVVo2S2RG?= =?utf-8?B?OXlIc3Q4NFk2UmRaeHNPZ1FjRWcxTVZYQkxRWGZtck95VW1SejF1anFFbDF4?= =?utf-8?B?NVNCM3NMa3JQTk9maXRRVWpZWGF2STgzL3dNMHpBYkNoVWlhMVJSRm96clEz?= =?utf-8?B?bzh5K0hvaytsSzhURC96RzNtVlpLMDIxWHU2K0Z4UWptVFU4VUwrbEJUZERl?= =?utf-8?B?LzBGRGd6bW9pTkFsTWdQMUZ6bWY3cFN6MTY4VWsyaWtzbzZ0TVd3VkM0TE9y?= =?utf-8?B?TUVjQ0Ntc0tnc1dPc2I2WlBTSjFCSWFOaGo4Ky95UU1sdGo3ZDV2SU14OEh0?= =?utf-8?B?WDRpWHBoMkJ6dDR3RnI2SW5lZ0VvaG00QWhrU3ZUUHZWN2VBL0FKZFJvRlI0?= =?utf-8?B?V1U2bVFhUjhlVUNHUkxtckYwL2grcjNyZ09YZmNvSmR2Sk1QR2NWOFNVaW1w?= =?utf-8?B?aTIvMjRUOTZzWDloZGpJQTZuem5McDM2N2tBaitac0lsUUdCYnBqM3JSZkll?= =?utf-8?B?aC9rTmtZYW1rSXBzcGwyUEU5TlFBMXNKLzdWMUlhS0Y1SUJTUEhBemJWWWxJ?= =?utf-8?B?a1BwQzMxNitjRTBYVVNFTE0zMnBFNHlWdnIrK00rcTRyN21mNzR3ZDNGbXoy?= =?utf-8?B?RWRLNWNEdkV4T1dHVkFjck9ZVlVBODFBZVdhVFFWaFJ4OHQvV00yMDA5YTMx?= =?utf-8?B?TUsrYTgzQUpERERyN0FiQWdXMDlEK2Z1eWQveFpVTThrWVk2RkZLZHJ0TVJn?= =?utf-8?B?dWl1bnVkd1dVZUovNUtrYjJoSkR5bGRJbkdPY09kcnUvUUJVWTFLL1M0aGll?= =?utf-8?B?eGVaRkxYL29jcDJMa2doREJYdm5wakk5Wnp1bEFib09IcXlhUDFtbVdhSU1k?= =?utf-8?B?emQrdU1WNGpSOEJ1cE9uQ3B0RnFSOENMM08ra2txZndacTgxbzhISVB1bnRM?= =?utf-8?B?NVptNzdjS3hKbkhzV284aC9CSkwrQzlXeGhqcHlxVnRpY2FjekRhM0VWelNk?= =?utf-8?B?MFFDdEFSNThqRHhFVWt4SU5WdEV5cmdRd2c0WnpkYzVKS1RsL2dFT0dUM093?= =?utf-8?B?UE5BUi9OR3Jrekp2U3R6dWQ5ZW5YUlowcE5CN0dwL0FnYmRNWW1CeWZYa2Uz?= =?utf-8?B?dEZ4OUZhOC9kK1IxMG0wcnJFRjNLZklocko3RGdybk9pblNaWjZHeUloZnRq?= =?utf-8?B?R1VlbTN3OTZGd0R3NUt1RC9iT0dlVHlrSjR5UVlIVnc1MWxBQzQzaFR0OTZO?= =?utf-8?B?MWQyWTFLR1h0KzI0dXNLV3FmR0N3OWpjcXI0S3BTTk5RRlB3dHQ5UGJiZmZU?= =?utf-8?Q?mq6joiOUXF3ERlYdBhRXvW4Hs?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 13128dca-b254-4f75-89fd-08dcc1d9a440 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB7763.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Aug 2024 12:06:08.8992 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8u4xQe6aPdPmB6kV86Mdk/DYpN7dQozJdHUs5Ld49EuGufoug1h19UmvVhZkMP0r X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6887 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240821_050622_874032_94A3FE4C X-CRM114-Status: GOOD ( 25.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Aug 21, 2024 at 09:53:33AM +0000, Mostafa Saleh wrote: > > Oh, from that perspective yes, but the entire point of S2FWB is that > > VM's can not create non-coherent access so it is a bit nonsense to ask > > for both S2FWB and try to assign a non-DMA coherent device. > > Yes, but KVM sets FWB unconditionally and would use cacheable mapping > for stage-2, and I expect the same for the nested SMMU. Yes, you'd need some kind of handshake like Intel built for their GPU cache incoherence to turn that off. > > > What I mean is the master itself not the SMMU (the SID basically), > > > so in that case the STE shouldn’t have FWB enabled. > > > > That doesn't matter, those cases will not pass in IOMMU_CACHE and they > > will work fine with S2FWB turned on. > > But that won’t be the case in nested? Otherwise why we use FWB in the > first place. Right, without KVM support for guest cachability selection and cache flushing in VFIO, is infeasible to allow non-coherent devices. It is a medium sized problem if someone wants to tackle it. > Maybe the SMMUv3 .capable, should be changed to check if the device is > coherent (instead of using dev_is_dma_coherent, it can use lower level > functions from the supported buses) That would be the fix I expect. Either SMMUv3 does it, or the core code adds it on top in the .capable wrapper. It makes sense to me that the iommu driver should be aware of per-master coherence capability. > Also, I think supporting IOMMU_CACHE is not enough, as the SMMU can > support it but the device is still not coherent. IOMMU_CACHE is defined as requiring no cache maintenance on that memory. If specific devices can't guarentee that then IOMMU_CACHE should not be used on those devices and IOMMU_CAP_CACHE_COHERENCY for that device should be false. That is what I mean by support. Anyhow, I'm going to continue to leave this problem alone for nesting. Nothing gets worse by adding nesting on top of this. Even if we wrongly permit VFIO to open non-coherent devices they won't actually work correctly (VFIO forces IOMMU_CACHE and S2FWB). Most likely anything trying to use them will just crash/malfunction due to missing cache flushing. Jason