From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF4C02EB5CD for ; Tue, 10 Mar 2026 21:47:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773179246; cv=none; b=pp0sAAmC+hidhZTlYrkjbYGR61gK1nE7J8Fx/iwqTp+ivUjo608ih+D1QZGYwKb1Ckl+j60R+VAPqob8w+nwotHWGSVhwOoRRBzceGzbJQ2LAjyR11FRHTvU5cg/GO/cW6C1Pz604GLWezxv0qIlzcHRC2x3GNOWAoHj7UAMA9Q= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773179246; c=relaxed/simple; bh=mQuTQCXgyBXZGtOv+LuZDeRQk0twCEJR67xLMDOKk44=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=kea/L4NfcWcSuqOBvLhWnvyUl5I34nIlxB/I6DyH38ZLQgHliy85uBc/sXv2kQkkFGxhMHfDlV43FMXDWWq0zcDd7SNQ63VTEZbrcfPiAX0+xQVHrvrfP6a5vjKig0LMM8UmWZAZodOlEcRCoZ0cjqCQ6NL9IFXtQh9lGrTpObo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LdirLuc3; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LdirLuc3" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2aeab6ff148so18645ad.1 for ; Tue, 10 Mar 2026 14:47:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1773179243; x=1773784043; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=JdVc8T+6UNGh/VsXP1dXf3PthPQA075yGDRasc1aV+I=; b=LdirLuc3hgtswf8m7+qGTVLVSquOW0aVec52It6MP7TcBpv9MjUb4KtMCJE0c5woIJ R08Tl8KmybFjTOZf3iaFq8QaWxkT2ZhH83A84kgVCdPwtaGej16VbaIl06tq+smLIxt3 OGBBUd681d/fMUANbr/bvkE6nYs5B3cyuCVG7fy7bOdw1eABZ9zWN5NhB6Yc3bPTJcuI 4j5eNO9tMlDvlEorbZL2kmBAKPzsYk9terZYrni9qdXEvbaa4+l5N1Sef7h52b5P/VMj SE0mkeaJ0gf1xHTmYWAFYcJ76L7oewav0oeaeoOpJiI54v9WRq/tJC43827FdKFRVO/K BMVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773179243; x=1773784043; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JdVc8T+6UNGh/VsXP1dXf3PthPQA075yGDRasc1aV+I=; b=WtroOqBmWZpOnatXCZr/WkS2tTp7s1gvJW4eOtg3sDWiWqLCRWjI58KhlqE8DvNUEz Kajj8mTae/2RuwO+sgTG9yyLl8H+qXH7iLexJIL8LqK3PvYteRZWtZkj2oms68gHB2Vf p+2FhF0elPUR0wZikShO6Y1Ubj79nleRB3OU+uA2F7dEEhnVyUqDa4/jRK3ZuwjdYVa3 HGi9nDzDzqXafjryVFcSFkrHr2g1Sp56fIBRreJvAjHjJ+HgII4D4Xr/b6giQJrJZYTX No7GeTbdqCz5aDCljzxQqxFUt0XhY7YmP1nMozoocVjRXTqPKlXbQIlAJstGYh5EoAEh JzzQ== X-Forwarded-Encrypted: i=1; AJvYcCW+eUx/S9LThq8giPYsEvEkTGUz3kDSKpIqgtQYeMaW3FM+y3WGAFATILERuiFtbk9c2IwQsSkKG0varW8=@vger.kernel.org X-Gm-Message-State: AOJu0Yyba5qfgyXVmvwEFYHqLuOh1CZubjYZqhUq22/d5S2QapfPmdoR Kn644A1yzpEQqZRRnnc+lYz4L1J1L/2ujX/9rn7JfxPKqWrEPHiSCVFZOv7sRN9UVA== X-Gm-Gg: ATEYQzzUrkitB3TM6V9KhSbepwvJ4Ztwa8e5mb9HfUgLNGITaxPCTFbfvDa5NWBVToX dbPfQpJheltajsCTZ2e9o+BgeL9WNa/tC3ySRmPW3rI1aEeeSGZWSnFds3z7q7J5sHeYaJJqRWM bmDRzoifbZg+s0or6FKeF0RGT0VqPjy5RuofPumAWjb80W7Mm8WnZiME7Y8j6xGpoNa2FphMASH dlO+WgngoOlu2Xf3m0UjnWD96+/MQQlza3u7w8sSsezsL8z/jBpDWEMMhybFERxMZ4rbBAu8HZ/ /jFoPPp29XxCycVqv9/OjTEptbiZNF2YN7s1v/OMsBe5D5DO3p58/p8pXZYMH5T5yOEvzVk2e/G b8ZrP9M3k+6wmI53yj3kKhdWNXXroPxPDrcN9JO/hRyXKrvtCsaufr8HFFCEU2pYm7wY1P210fy u/X5gdBxYvzpA9+t5+YkVwbxp2IsVSAXORZh4IMvF5Tuv2vtyPd3MAOpsddXfjXA== X-Received: by 2002:a17:903:2ca:b0:2ae:5d57:c94f with SMTP id d9443c01a7336-2aead3e31a0mr1133255ad.16.1773179242632; Tue, 10 Mar 2026 14:47:22 -0700 (PDT) Received: from google.com (168.136.83.34.bc.googleusercontent.com. [34.83.136.168]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-829f703a7casm213874b3a.59.2026.03.10.14.47.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Mar 2026 14:47:22 -0700 (PDT) Date: Tue, 10 Mar 2026 21:47:18 +0000 From: Samiullah Khawaja To: Ankit Soni Cc: David Woodhouse , Lu Baolu , Joerg Roedel , Will Deacon , Jason Gunthorpe , Robin Murphy , Kevin Tian , Alex Williamson , Shuah Khan , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Saeed Mahameed , Adithya Jayachandran , Parav Pandit , Leon Romanovsky , William Tu , Pratyush Yadav , Pasha Tatashin , David Matlack , Andrew Morton , Chris Li , Pranjal Shrivastava , Vipin Sharma , YiFei Zhu Subject: Re: [PATCH 08/14] iommu: Restore and reattach preserved domains to devices Message-ID: References: <20260203220948.2176157-1-skhawaja@google.com> <20260203220948.2176157-9-skhawaja@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: On Tue, Mar 10, 2026 at 05:16:35AM +0000, Ankit Soni wrote: >On Tue, Feb 03, 2026 at 10:09:42PM +0000, Samiullah Khawaja wrote: >> Restore the preserved domains by restoring the page tables using restore >> IOMMU domain op. Reattach the preserved domain to the device during >> default domain setup. While attaching, reuse the domain ID that was used >> in the previous kernel. The context entry setup is not needed as that is >> preserved during liveupdate. >> >> Signed-off-by: Samiullah Khawaja >> --- >> drivers/iommu/intel/iommu.c | 40 ++++++++++++++++++------------ >> drivers/iommu/intel/iommu.h | 3 ++- >> drivers/iommu/intel/nested.c | 2 +- >> drivers/iommu/iommu.c | 47 ++++++++++++++++++++++++++++++++++-- >> drivers/iommu/liveupdate.c | 31 ++++++++++++++++++++++++ >> include/linux/iommu-lu.h | 8 ++++++ >> 6 files changed, 112 insertions(+), 19 deletions(-) >> >> diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c >> index 8acb7f8a7627..83faad53f247 100644 >> --- a/drivers/iommu/intel/iommu.c >> +++ b/drivers/iommu/intel/iommu.c >> @@ -1029,7 +1029,8 @@ static bool first_level_by_default(struct intel_iommu *iommu) >> return true; >> } >> >> -int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu) >> +int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu, >> + int restore_did) >> { >> struct iommu_domain_info *info, *curr; >> int num, ret = -ENOSPC; >> @@ -1049,8 +1050,11 @@ int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu) >> return 0; >> } >> >> - num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID, >> - cap_ndoms(iommu->cap) - 1, GFP_KERNEL); >> + if (restore_did >= 0) >> + num = restore_did; >> + else >> + num = ida_alloc_range(&iommu->domain_ida, IDA_START_DID, >> + cap_ndoms(iommu->cap) - 1, GFP_KERNEL); >> if (num < 0) { >> pr_err("%s: No free domain ids\n", iommu->name); >> goto err_unlock; >> @@ -1321,10 +1325,14 @@ static int dmar_domain_attach_device(struct dmar_domain *domain, >> { >> struct device_domain_info *info = dev_iommu_priv_get(dev); >> struct intel_iommu *iommu = info->iommu; >> + struct device_ser *device_ser = NULL; >> unsigned long flags; >> int ret; >> >> - ret = domain_attach_iommu(domain, iommu); >> + device_ser = dev_iommu_restored_state(dev); >> + >> + ret = domain_attach_iommu(domain, iommu, >> + dev_iommu_restore_did(dev, &domain->domain)); >> if (ret) >> return ret; >> >> @@ -1337,16 +1345,18 @@ static int dmar_domain_attach_device(struct dmar_domain *domain, >> if (dev_is_real_dma_subdevice(dev)) >> return 0; >> >> - if (!sm_supported(iommu)) >> - ret = domain_context_mapping(domain, dev); >> - else if (intel_domain_is_fs_paging(domain)) >> - ret = domain_setup_first_level(iommu, domain, dev, >> - IOMMU_NO_PASID, NULL); >> - else if (intel_domain_is_ss_paging(domain)) >> - ret = domain_setup_second_level(iommu, domain, dev, >> - IOMMU_NO_PASID, NULL); >> - else if (WARN_ON(true)) >> - ret = -EINVAL; >> + if (!device_ser) { >> + if (!sm_supported(iommu)) >> + ret = domain_context_mapping(domain, dev); >> + else if (intel_domain_is_fs_paging(domain)) >> + ret = domain_setup_first_level(iommu, domain, dev, >> + IOMMU_NO_PASID, NULL); >> + else if (intel_domain_is_ss_paging(domain)) >> + ret = domain_setup_second_level(iommu, domain, dev, >> + IOMMU_NO_PASID, NULL); >> + else if (WARN_ON(true)) >> + ret = -EINVAL; >> + } >> >> if (ret) >> goto out_block_translation; >> @@ -3630,7 +3640,7 @@ domain_add_dev_pasid(struct iommu_domain *domain, >> if (!dev_pasid) >> return ERR_PTR(-ENOMEM); >> >> - ret = domain_attach_iommu(dmar_domain, iommu); >> + ret = domain_attach_iommu(dmar_domain, iommu, -1); >> if (ret) >> goto out_free; >> >> diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h >> index d7bf63aff17d..057bd6035d85 100644 >> --- a/drivers/iommu/intel/iommu.h >> +++ b/drivers/iommu/intel/iommu.h >> @@ -1174,7 +1174,8 @@ void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr, >> */ >> #define QI_OPT_WAIT_DRAIN BIT(0) >> >> -int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu); >> +int domain_attach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu, >> + int restore_did); >> void domain_detach_iommu(struct dmar_domain *domain, struct intel_iommu *iommu); >> void device_block_translation(struct device *dev); >> int paging_domain_compatible(struct iommu_domain *domain, struct device *dev); >> diff --git a/drivers/iommu/intel/nested.c b/drivers/iommu/intel/nested.c >> index a3fb8c193ca6..4fed9f5981e5 100644 >> --- a/drivers/iommu/intel/nested.c >> +++ b/drivers/iommu/intel/nested.c >> @@ -40,7 +40,7 @@ static int intel_nested_attach_dev(struct iommu_domain *domain, >> return ret; >> } >> >> - ret = domain_attach_iommu(dmar_domain, iommu); >> + ret = domain_attach_iommu(dmar_domain, iommu, -1); >> if (ret) { >> dev_err_ratelimited(dev, "Failed to attach domain to iommu\n"); >> return ret; >> diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c >> index c0632cb5b570..8103b5372364 100644 >> --- a/drivers/iommu/iommu.c >> +++ b/drivers/iommu/iommu.c >> @@ -18,6 +18,7 @@ >> #include >> #include >> #include >> +#include >> #include >> #include >> #include >> @@ -489,6 +490,10 @@ static int iommu_init_device(struct device *dev) >> goto err_free; >> } >> >> +#ifdef CONFIG_IOMMU_LIVEUPDATE >> + dev->iommu->device_ser = iommu_get_device_preserved_data(dev); >> +#endif >> + >> iommu_dev = ops->probe_device(dev); >> if (IS_ERR(iommu_dev)) { >> ret = PTR_ERR(iommu_dev); >> @@ -2149,6 +2154,13 @@ static int __iommu_attach_device(struct iommu_domain *domain, >> ret = domain->ops->attach_dev(domain, dev, old); >> if (ret) >> return ret; >> + >> +#ifdef CONFIG_IOMMU_LIVEUPDATE >> + /* The associated state can be unset once restored. */ >> + if (dev_iommu_restored_state(dev)) >> + WRITE_ONCE(dev->iommu->device_ser, NULL); >> +#endif >> + >> dev->iommu->attach_deferred = 0; >> trace_attach_device_to_domain(dev); >> return 0; >> @@ -3061,6 +3073,34 @@ int iommu_fwspec_add_ids(struct device *dev, const u32 *ids, int num_ids) >> } >> EXPORT_SYMBOL_GPL(iommu_fwspec_add_ids); >> >> +static struct iommu_domain *__iommu_group_maybe_restore_domain(struct iommu_group *group) >> +{ >> + struct device_ser *device_ser; >> + struct iommu_domain *domain; >> + struct device *dev; >> + >> + dev = iommu_group_first_dev(group); >> + if (!dev_is_pci(dev)) >> + return NULL; >> + >> + device_ser = dev_iommu_restored_state(dev); >> + if (!device_ser) >> + return NULL; >> + >> + domain = iommu_restore_domain(dev, device_ser); >> + if (WARN_ON(IS_ERR(domain))) >> + return NULL; >> + >> + /* >> + * The group is owned by the entity (a preserved iommufd) that provided >> + * this token in the previous kernel. It will be used to reclaim it >> + * later. >> + */ >> + group->owner = (void *)device_ser->token; >> + group->owner_cnt = 1; >> + return domain; >> +} >> + >> /** >> * iommu_setup_default_domain - Set the default_domain for the group >> * @group: Group to change >> @@ -3075,8 +3115,8 @@ static int iommu_setup_default_domain(struct iommu_group *group, >> int target_type) >> { >> struct iommu_domain *old_dom = group->default_domain; >> + struct iommu_domain *dom, *restored_domain; >> struct group_device *gdev; >> - struct iommu_domain *dom; >> bool direct_failed; >> int req_type; >> int ret; >> @@ -3120,6 +3160,9 @@ static int iommu_setup_default_domain(struct iommu_group *group, >> /* We must set default_domain early for __iommu_device_set_domain */ >> group->default_domain = dom; >> if (!group->domain) { >> + restored_domain = __iommu_group_maybe_restore_domain(group); >> + if (!restored_domain) >> + restored_domain = dom; >> /* >> * Drivers are not allowed to fail the first domain attach. >> * The only way to recover from this is to fail attaching the >> @@ -3127,7 +3170,7 @@ static int iommu_setup_default_domain(struct iommu_group *group, >> * in group->default_domain so it is freed after. >> */ >> ret = __iommu_group_set_domain_internal( >> - group, dom, IOMMU_SET_DOMAIN_MUST_SUCCEED); >> + group, restored_domain, IOMMU_SET_DOMAIN_MUST_SUCCEED); >> if (WARN_ON(ret)) >> goto out_free_old; >> } else { >> diff --git a/drivers/iommu/liveupdate.c b/drivers/iommu/liveupdate.c >> index 83eb609b3fd7..6b211436ad25 100644 >> --- a/drivers/iommu/liveupdate.c >> +++ b/drivers/iommu/liveupdate.c >> @@ -501,3 +501,34 @@ void iommu_unpreserve_device(struct iommu_domain *domain, struct device *dev) >> >> iommu_unpreserve_locked(iommu->iommu_dev); >> } >> + >> +struct iommu_domain *iommu_restore_domain(struct device *dev, struct device_ser *ser) >> +{ >> + struct iommu_domain_ser *domain_ser; >> + struct iommu_lu_flb_obj *flb_obj; >> + struct iommu_domain *domain; >> + int ret; >> + >> + domain_ser = __va(ser->domain_iommu_ser.domain_phys); >> + >> + ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj); >> + if (ret) >> + return ERR_PTR(ret); >> + >> + guard(mutex)(&flb_obj->lock); >> + if (domain_ser->restored_domain) >> + return domain_ser->restored_domain; >> + >> + domain_ser->obj.incoming = true; >> + domain = iommu_paging_domain_alloc(dev); >> + if (IS_ERR(domain)) >> + return domain; >> + >> + ret = domain->ops->restore(domain, domain_ser); >> + if (ret) > >'domain' will leak here. Agreed. It should be cleaned here if restore fails. I will fix this in the next revision. > >> + return ERR_PTR(ret); >> + >> + domain->preserved_state = domain_ser; >> + domain_ser->restored_domain = domain; >> + return domain; >> +} >> diff --git a/include/linux/iommu-lu.h b/include/linux/iommu-lu.h >> index 48c07514a776..4879abaf83d3 100644 >> --- a/include/linux/iommu-lu.h >> +++ b/include/linux/iommu-lu.h >> @@ -65,6 +65,8 @@ static inline int dev_iommu_restore_did(struct device *dev, struct iommu_domain >> return -1; >> } >> >> +struct iommu_domain *iommu_restore_domain(struct device *dev, >> + struct device_ser *ser); >> int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn, >> void *arg); >> struct device_ser *iommu_get_device_preserved_data(struct device *dev); >> @@ -95,6 +97,12 @@ static inline void *iommu_domain_restored_state(struct iommu_domain *domain) >> return NULL; >> } >> >> +static inline struct iommu_domain *iommu_restore_domain(struct device *dev, >> + struct device_ser *ser) >> +{ >> + return NULL; >> +} >> + >> static inline int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn, void *arg) >> { >> return -EOPNOTSUPP; >> -- >> 2.53.0.rc2.204.g2597b5adb4-goog >> Sami