From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 17A5AC2D0D1 for ; Mon, 24 Jun 2024 19:29:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TVuLYAHX8gXy6iGoDlxWfTLH0m/dbWsxwJkE9QnlL8o=; b=h4aeevpdWZJwXFiBx3S9FKuium I9pTJ2ohony0Cm82No+LODu7HruuE1OES3NfRpECNXC/cZwqvjAC57c7aoSJEqBLDdoYDtC1Q5iz0 c3O91R4Fl4rgmbSbgI7PFMMwzuDunhWHbkJEKwBTqV9TUPNRV3CZ9hbYPPriJEgshPIwhwTQLe+o7 QxOL3IR76XeU7bIBCrha/uw1BvEELBLlDBTmWSB3cUYvJ8ST+9Oua+VS1zBtDGkU00BY8ukNCpYjr 05br5J79BB1m7FSBhr7/NkGInfjjfZWvvm0o9lz1uupzUEKl4EYVM3IOBGXkq4aoN6eEUEXNhBH3V 8NU9KYlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sLpNM-00000000Qh4-01Ac; Mon, 24 Jun 2024 19:29:16 +0000 Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sLpNG-00000000Qe4-1hUR for linux-arm-kernel@lists.infradead.org; Mon, 24 Jun 2024 19:29:11 +0000 Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-718c62ad099so3580379a12.3 for ; Mon, 24 Jun 2024 12:29:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1719257349; x=1719862149; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TVuLYAHX8gXy6iGoDlxWfTLH0m/dbWsxwJkE9QnlL8o=; b=xgl+1CxAyu+EVP5U4AY/4qCu8dsR7eP3uNsWa3JeIhraxLhvNcqDSC30cF84eFSK0Q /JRohYv+6X2teCZTzJL0PUtr7MAgU75oDgBBngWpgGI7dYZtsau0IsFITzBESOXaRDSV wF7Bls2LIzRttkRhyJMxNoxy8r8lyy0ruY51VhWNoQeG/IoZ68XsfFX/nuzJzhm6AEq8 QNVStspHQM7aGmCV/avFCNkJOTtJgp2YvXAQiYHkhwq7omz5yI+ubnLmfAztgZ84itXw 85GtMJogmHkA4YcVJSatdgVvVrPhkmUw0wE23tnjvlt6Lowa/ZsG/xXYu4hZhfiXdYO3 0SDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719257349; x=1719862149; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TVuLYAHX8gXy6iGoDlxWfTLH0m/dbWsxwJkE9QnlL8o=; b=atg5W3YeMEAzOMdK+bL7IWjhlT0uYO9LgJb/YSsx97kUo/O8DQ1uCkbxGA9arVKAIp SBiSm0mZf+W9gzkOqLaXk04xr+gIPKszSG2gq3M4IbrYHanIR46qwS+WZwFHIyjQeWJ1 3l0CM+yx9Zhap5eMpjVWpZN9Y6C52gbz62YNCdJ+2RNhKKRPe4Y76EgbPbPMm6ej9KAi eAIxeihTVJDg8e6g5t07knvVd6CExRlVyZfbq1olzPw2szhjTt4B4L+FOK7WpO7kqjiV cua80mna7zXFzrXW1a8HIaT2GrZAsA5KsctfeB/EEJiIysXLW4VPkH53kGPa6iK+7Yjx jgvg== X-Forwarded-Encrypted: i=1; AJvYcCXYQIV3/JZmqAPogxTJt/m65mFsXA2Nk0uF/QWdMgIvtpdq1CE5n2QAWbDny5zLwAvuEAbwxtEs2stYHVf6tnrmzbiRTUSC7K7TOFoSR4ZWNqjT4+o= X-Gm-Message-State: AOJu0YzI371zmBkNujUj+JiYsrZIVeYz507/HWAGAnxkFYydpRayRtB3 s+HJsT4AijvNJP+4/efcj8Z3SlsbIVXjjaxfZMC5vkZ7DN8+La3GuPY/ihglYe8v1tervkXREvY Nyg== X-Google-Smtp-Source: AGHT+IEJKp/82XGN5zTzJceOYTl1iz+gFaSHnIG6fY86rBs+MqpT0OdjuXJwtSsYk3DKzCheG/u8hCds3Mk= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6a02:51f:b0:6ea:320b:a2af with SMTP id 41be03b00d2f7-71a363d4e8dmr18982a12.5.1719257348896; Mon, 24 Jun 2024 12:29:08 -0700 (PDT) Date: Mon, 24 Jun 2024 12:29:07 -0700 In-Reply-To: Mime-Version: 1.0 References: <20240208151837.35068-1-shameerali.kolothum.thodi@huawei.com> <20240208151837.35068-5-shameerali.kolothum.thodi@huawei.com> <20240208154210.GP31743@ziepe.ca> <20240624170747.GA1515249@ziepe.ca> <20240624180148.GV791043@ziepe.ca> Message-ID: Subject: Re: [RFC PATCH v2 4/7] iommufd: Associate kvm pointer to iommufd ctx From: Sean Christopherson To: Oliver Upton Cc: Jason Gunthorpe , Shameer Kolothum , kvmarm@lists.linux.dev, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linuxarm@huawei.com, kevin.tian@intel.com, alex.williamson@redhat.com, maz@kernel.org, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, jonathan.cameron@huawei.com Content-Type: text/plain; charset="us-ascii" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240624_122910_471896_D7D1CD89 X-CRM114-Status: GOOD ( 36.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Jun 24, 2024, Oliver Upton wrote: > On Mon, Jun 24, 2024 at 03:01:48PM -0300, Jason Gunthorpe wrote: > > On Mon, Jun 24, 2024 at 10:54:37AM -0700, Sean Christopherson wrote: > > > > > And assuming that pinnable VMIDs are a somewhat scarce resource, it wouldn't > > > > > suprise me if someone wanted to add cgroup integration, e.g. similar to the > > > > > misc cgroup that's used to manage SEV(-ES) ASIDs on KVM AMD (IIUC, an SEV ASID > > > > > is analagous to an ARM VMID). > > > > > > > > Yeah, but if someone is using such a cgroup then I expect they will > > > > also have an up to date VMM that doesn't trigger this VMID allocation > > > > in the first place... > > > > > > I suspect we're talking about two different things. Either that, or I am really > > > lost. > > > > I mean KVM will have already allocated and charged the cgroup for it's > > use of the VMID. The IOMMU side just has to match it, no second > > allocation of a VMID. > > > > We wouldn't charge a cgroup for iommu and kvm sharing the same vmid. > > I think the concern remains that an operator may want to limit the blast > radius of some runaway VMID allocation in a system. But you're right, a > well-intentioned VMM should wind up with a single charge for all the > stage-2's that used the VMID allocation. > > > > > When a KVM is present then the iommu needs to adopt the VMID of KVM, > > > > and that should have a mechanism to ensure the VMID is valid so long > > > > as the IOMMU is using it (eg because the KVM FD is open) > > > > > > Right, and that's what I'm referring to as "on-demand pinning". For the IOMMU > > > to adopt a KVM VMID, the VMID needs to be pinned (or KVM would need to notify > > > the IOMMU every time the VMID changed), i.e. every KVM+IOMMU pair pins a VMID > > > that is managed by KVM. > > > > Ok, right, yes, the expectation is that KVM allocates a VMID at some > > point and it stays fixed for the life of that kvm. > > > > If KVM can change VMID on the fly then that is a further complication > > :\ Ya, as written today, KVM doesn't assign a VMID when the VM is created, and instead allocates VMIDs on-demand when a vCPU is run. The KVM changes in this series allow "pinning" the currently assigned VMID, i.e. tries to address that further complication. But because of the on-demand allocation, there might not be a currently assigned VMID for VM, or the VMID might be stale, i.e. re-assigned to a different VM. Thus, kvm_arm_pinned_vmid_get() can effectively trigger VMID allocations, and thus cgroup charging and failure. If I'm reading the ARM code correctly, the intent is to cycle through VMIDs as necessary so that it's possible for every actively running VM to have a VMID. And maybe also to also minimize the number of TLB + I$ invalidations? > > > > > Hmm, kvm_arm_pinned_vmid_get() doesn't fail, it just falls back to VMID=0. Which > > > seems odd. > > This is bleeding a bit of implementation detail where VMID=0 is known to > be reserved (thus invalid), it'd probably be better if the > implementation actually just returned an error. Oof, I assumed using VMID=0 just caused a loss of performance, but this makes it sound like the IOMMU mappings will fault? > VMID=0 is associated with the host's MMU context, which is relevant when > running {n,h}VHE mode, as the VMID tags TLB entries even if stage-2 > translation is disabled (HCR_EL2.VM = 0). Heh, I assumed VMID=0 is the host MMU, Intel and AMD have the same effective behavior :-)