From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85BD621170F for ; Wed, 22 Jan 2025 11:04:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737543874; cv=none; b=aJ1cxqro+iaDQvmrOZF0kHDvybAQnbTwMXw7o1Fp0x6BcDZBIUWnr0kX3guDsZ/selzB1bQFGo6h0YyaKIUpfZnGpArF5+da8THk8toy9UJbpXIaCEu4SB9BzeMH5ab6PQrwJcasTXZ7rgHL/M45DHqswtxqBM+hXDmAMKqXzjo= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737543874; c=relaxed/simple; bh=utqVEO5I67Qn1SxggtlBg2uyccJsm9B3vF55yUEBW0Q=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=YuISGqvGSOtj8Vxf9afhjxb60lPUa0TjsuYe+6CCT0DRBMLH9r+wNFkjKdlGUojpAqMdR4SVu9yFTBSZtUNkLgxYRzS/XOQkaIYJ2CM2zL2HCLhaOHvblinCEsTdXzMaVflAPVAeMBQ5GAxUPlBiFa0OAjaGkemVH5/Wu/fN5Fc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=W7ovZj5K; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="W7ovZj5K" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-4368a290e0dso36365e9.1 for ; Wed, 22 Jan 2025 03:04:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737543871; x=1738148671; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=6x8SiYYYUQFFMCokfouR/mswZBMHq350sgLQtsTuQB0=; b=W7ovZj5K7WpcdysHRh2TWJvFP0J1xvEg4/2+JrFfAUMNEOs11TdRsF3ZLQ80E77Ttk /nUkhHSOtP+HaBAqkaBTYy9wSz2gh1nNwIDRiBcWDjJaU7ePHXwcVkA/XTA2bpANOqPf jKsOa/Po1T+GBa+aFdEqMvLRPA5IwN81Yi5ukFDppL1LbOH+O8Gzji77t8HRqwfGBpSo 58rM76/9nildknZAXbGui7a5mp4E5XMM+kqgx9b50xidhZbcfjAO14bPKQFRopZjojtk aYCzpTtqVJOemWQWcO69lZFYl32/pKXFXXMp6Ru70c0lZ+Umm36/n5REgnzJTA0eVAB+ /f5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737543871; x=1738148671; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6x8SiYYYUQFFMCokfouR/mswZBMHq350sgLQtsTuQB0=; b=PghnoWuueL4Xzq9dxYGoQ08OBCIPr8mbZ1tMiG2I1f6rYmqagFxuqe8CWV1j6gT6T+ gXg4Hi/eu0/A0/75QdtlRe6neNGKGZrdYKswMa4USsCDhXBHGEI2KEw9OZqnQ0j5yAEe Y3bZmmPMZgscVBDGycGF0fOkEEp156QmUmEQQO+XTbGMb8iYwcNYZA4ihffPgQNJfoLd 8gPwA/tGJUU3alZeHc7lH4uw7HqOIBDSyZ8mqWg7uIlRQRglJ52yPVFLef6tjwmKQJ0l +8LAbdlS3+uyOFKa9wWDv5KoA8/stzLWl503I01d0t2uWDGhvC/PWapUiyJn+CBvWQab PePQ== X-Forwarded-Encrypted: i=1; AJvYcCVHgh/asUbHTaQNl4jqonw/G2zHuS43/M20LVIDVyBGMi8Olb+sMr4mXE6VFZXpT/wyojds57yi2LoY6OA=@vger.kernel.org X-Gm-Message-State: AOJu0YwxEFF1jtUdaR1PhrBjK9h24z57C4ZnUDsTTfnQSB0M5Aa2qHse WUOld+AWsQ3Lxx7A/pSXgi3Rh49th/tRQgP8KAkAz1zVusr3JnaczaWh/3syhqZNDlDJ8MPCFUA s0A== X-Gm-Gg: ASbGnctqvOQFCRn3NZ9iMIj1cSEUX9C6lL8v2oL02YCPuMjCRmr9CUaY04IJKbKvv5H SDh/wi3I0y/CoxuveUI4Mx/NqPslcU74ONmmjvtC55REzh/AMKWYEi2w46Tsds2AIr8yTzwbTtM RiW3S2vblzc3nRiWy6tk3Zr2aEU33HKR0AF7FOj13R/7ufeS5fe1D3s9uxWRWORn3rAaFlh4bw0 9CVz10GrR1BKj1+TdCPmCyta4Fr+lbw9W1A2cu9unhwEJ0AQN/uailC4QX1+LwpyVF5VbvLaVlt WVUe7PN3EfSKWg9WXnaH359BfM4vPw== X-Google-Smtp-Source: AGHT+IHCcPrZhJYSQbULZrw9Pp5790wT0aW6WD0dYCBGNGReHretNQSCRXdvXPT2A8uBJ9bQ0Dv2SQ== X-Received: by 2002:a05:600c:1d8f:b0:436:ed00:9e3 with SMTP id 5b1f17b1804b1-438b21c931dmr1151165e9.6.1737543870607; Wed, 22 Jan 2025 03:04:30 -0800 (PST) Received: from google.com (88.140.78.34.bc.googleusercontent.com. [34.78.140.88]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38bf3221baasm16122838f8f.35.2025.01.22.03.04.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jan 2025 03:04:30 -0800 (PST) Date: Wed, 22 Jan 2025 11:04:24 +0000 From: Mostafa Saleh To: "Tian, Kevin" Cc: Jason Gunthorpe , "iommu@lists.linux.dev" , "kvmarm@lists.linux.dev" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "catalin.marinas@arm.com" , "will@kernel.org" , "maz@kernel.org" , "oliver.upton@linux.dev" , "joey.gouly@arm.com" , "suzuki.poulose@arm.com" , "yuzenghui@huawei.com" , "robdclark@gmail.com" , "joro@8bytes.org" , "robin.murphy@arm.com" , "jean-philippe@linaro.org" , "nicolinc@nvidia.com" , "vdonnefort@google.com" , "qperret@google.com" , "tabba@google.com" , "danielmentz@google.com" , "tzukui@google.com" Subject: Re: [RFC PATCH v2 00/58] KVM: Arm SMMUv3 driver for pKVM Message-ID: References: <20241212180423.1578358-1-smostafa@google.com> <20241212194119.GA4679@ziepe.ca> <20250102201614.GA26854@ziepe.ca> <20250116191455.GC674319@ziepe.ca> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Fri, Jan 17, 2025 at 06:57:12AM +0000, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Friday, January 17, 2025 3:15 AM > > > > On Thu, Jan 16, 2025 at 06:39:31AM +0000, Tian, Kevin wrote: > > > > From: Mostafa Saleh > > > > Sent: Wednesday, January 8, 2025 8:10 PM > > > > > > > > On Thu, Jan 02, 2025 at 04:16:14PM -0400, Jason Gunthorpe wrote: > > > > > On Fri, Dec 13, 2024 at 07:39:04PM +0000, Mostafa Saleh wrote: > > > > > > Yeah, SVA is tricky, I guess for that we would have to use nesting, > > > > > > but tbh, I don’t think it’s a deal breaker for now. > > > > > > > > > > Again, it depends what your actual use case for translation is inside > > > > > the host/guest environments. It would be good to clearly spell this out.. > > > > > There are few drivers that directly manpulate the iommu_domains of a > > > > > device. a few gpus, ath1x wireless, some tegra stuff, "venus". Which > > > > > of those are you targetting? > > > > > > > > > > > > > Not sure I understand this point about manipulating domains. > > > > AFAIK, SVA is not that common, including mobile spaces but I can be > > wrong, > > > > that’s why it’s not a priority here. > > > > > > Nested translation is required beyond SVA. A scenario which requires > > > a vIOMMU and multiple device domains within the guest would like to > > > embrace nesting. Especially for ARM vSMMU nesting is a must. We can still do para-virtualization for guests the same way we do for the host and use a single stage IOMMU. > > > > Right, if you need an iommu domain in the guest there are only three > > mainstream ways to get this in Linux: > > 1) Use the DMA API and have the iommu group be translating. This is > > optional in that the DMA API usually supports identity as an option. > > 2) A driver directly calls iommu_paging_domain_alloc() and manually > > attaches it to some device, and does not use the DMA API. My list > > above of ath1x/etc are examples doing this > > 3) Use VFIO > > > > My remark to Mostafa is to be specific, which of the above do you want > > to do in your mobile guest (and what driver exactly if #2) and why. > > > > This will help inform what the performance profile looks like and > > guide if nesting/para virt is appropriate. > AFAIK, the most common use cases would be: - Devices using DMA API because it requires a lot of memory to be contiguous in IOVA, which is hard to do with identity - Devices with security requirements/constraints to be isolated from the rest of the system, also using DMA API - VFIO is something we are looking at the moment and have prototyped with pKVM, and it should be supported soon in Android (only for platform devices for now) > Yeah that part would be critical to help decide which route to pursue > first. Even when all options might be required in the end when pKVM > is scaled to more scenarios, as you mentioned in another mail, a staging > approach would be much preferrable to evolve. I agree that would probably be the case. I will work on more staging approach for v3, mostly without the pv part as Jason suggested. > > The pros/cons between nesting/para virt is clear - more static the > mapping is, more gain from the para approach due to less paging > walking and smaller tlb footprint, while vice versa nesting performs > much better by avoiding frequent para calls on page table mgmt. 😊 I am also working to get the numbers for both cases so we know the order of magnitude of each case, as I guess it won't be as clear for large systems with many DMA initiators what approach is best. Thanks, Mostafa > > > > > > But I'm not sure that I got Jason's point about " there is no way to get > > > SVA support with para-virtualization." virtio-iommu is a para-virtualized > > > model and SVA support is in its plan. The main requirement is to pass > > > the base pointer of the guest CPU page table to backend and PRI faults/ > > > responses back forth. > > > > That's nesting, you have a full page table under the control of the > > guest, and the guest needs to have a level of HW-specific > > knowledge. It is just an alternative to using the native nesting > > vIOMMU. > > > > What I mean by "para-virtualization" is the guest does map/unmap calls > > to the hypervisor and has no page tbale. > > > > Yes, that should never happen for SVA.