From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F11A936607D for ; Mon, 22 Jun 2026 23:27:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782170875; cv=none; b=dleObnKAgIbXDvVSQIVK3S9pwjdqF441GzzwmJYDBPqcPrnUt4mtvpGjAXhmfBlhtwLHS6X30maI/N2TtovuNiqiNGYNL2Hm6r+Rx+I010ph8ZlRYd+8/8ETlUakqvVuvIdxSQYeVuhisnLTiKR8YfFdaMOqvqc0wD2sPFOcF/A= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782170875; c=relaxed/simple; bh=lwVDuCan41sfTJTnrOrdE1tC3q2gBSoCWyBPW5RUdWE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=X3Q70RqcjKhoDaT8/uJWEsO16zk3DOfAdlxARonEXNYsibr0BdzgdJG2S8XADK9kYHdctf+VQzYdtiAREkNEltOUBV+UJzMgevZw665ap3cwyZOm5GJayhWpcllil1huDtr9o36mm3UVOX7Xvg7bH5h56+8tYenAM2ftzTAbkHg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vI8N5A2d; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vI8N5A2d" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2c6b7bd4e8dso18385ad.0 for ; Mon, 22 Jun 2026 16:27:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1782170872; x=1782775672; darn=vger.kernel.org; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=Wp+dObOPQcCrEViDJa1TZgNuuO73gOCATlytODnl2CU=; b=vI8N5A2dkuuC6ZlMglc1nJv4JSKiYLHkIVecLahyg96otfvreNAoaguj+tu8rx7qXP HtC49j+CCjL3Q3agyzgPdsOh1F8xO8xYXGfGerCBG7hlKDcJIVry1UdgiX76xQbEkkIi 59G0k6g/BXR8H4SWaEh6ZSllFTz9cgs3xQrNRLz6j1nEcDSNY/GiWEYwIxYh13DSk1aJ PrpRLSia4t9gdlU2qounMYkX4jQ9x5k6DzCPxt+JknJVeSEjfC4JoE1zodUX7qvUJCsX caGnY8f4OcLcIa3eFqq/fCq3xAZpfbaXFvPePK3TAr5IB3rwDW0A7nb7STyi0KLpig2k 2Qfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782170872; x=1782775672; h=in-reply-to:content-disposition:content-type:mime-version :references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Wp+dObOPQcCrEViDJa1TZgNuuO73gOCATlytODnl2CU=; b=MGP7Lf/92anK+TsqaSYFw0BA+WnK6oEOr3qXqm+LliZpUD43fgwL1gvXcIIt8MwZuY uYM6I2STkSz8lCzDHDaF05xb5MpJHaP5Ja5XIWTKblj8QIp+odJK0hfey6Ih5hxL1fdB E0jmtb6pbk0a0QvYG34VDuCQOfjITdUK6SZhVxBvdp5MoLXaP05dFJriqY2o74GVVfiA kttsSGrXu7Gau7a9276yqwAzMoUcznawkfhONwwNiG5N86IvkS+U7DwRjs5x22H9fY3F ahk9dTTIqdJUVIxzgrtkMhvooAeiI/CdUsMGMsMF+wtdbpC6q0Jr15a6EG5uiKU8+39C Mc7Q== X-Forwarded-Encrypted: i=1; AHgh+RrCFjv2gHD52qJQ4Z92U2dR9k/Az+gtPwj9yzy4Clf5HmmqvhqISb1xc2KLuWBOjw3ws7ewG0+0L609IWw=@vger.kernel.org X-Gm-Message-State: AOJu0Yw62kIrPLBiQGHJezj/syYQj0hV/aoUbAdEBPO7AirVoWfvDFCP 49XsVMMbGJxqOcqozlbUSP8B+2yt3K3pntOoAZkAk9J5vRKEuDLzbpBZcYQnBnPOgg== X-Gm-Gg: AfdE7cnCzxhJgeVsnm8FmO1Li9ISbAGeM2UK3/U9yzZixqnKehYozObOt8w9UWp7SJk pbzO3ZjFdpGijPP3HlpfyJzD899sxGoZ5nsEPU71y69etGOqAMdrQ9G3ebS0rksOFic6ocfxRfn +R+Q29y1gvbPTYQ+VuD9izFaEyfUr9nFAbsI38UJ5d5nbUbYALcgHQcSI7uoJx3zlQIynckYji1 3Br5EuP0s0BJUDaA7OPU8N9+nOmB/EaiM0vkW1aD5YCN5A4y47L8og1QlAjYgLId9dhiStQvkec OSkUGeIDD6hdlpDqAwcrSdPAau/SaY9itx2MjahVM7Qvj69aW7jpTMPWsqLNn73jP9tkbFa6OAV r01WJ59L+SZbtyJX5lUJg7Pm/dyIkAi7Icoeek7/6zfyLcnC9RC6nzsQ9z2sxN9TXWOPa6olKAp Aj49ySmqXprP1rwTAzNgtMb58RJBKBxKSfxcV0Ym07HjUi+7ezQx6WK79+XJDYdq5jx7+n05JMU VluW2KRVx7GA9RxkMk= X-Received: by 2002:a17:902:ec83:b0:2b4:641a:6b7c with SMTP id d9443c01a7336-2c7c70ec5damr540285ad.13.1782170871493; Mon, 22 Jun 2026 16:27:51 -0700 (PDT) Received: from google.com (25.75.145.34.bc.googleusercontent.com. [34.145.75.25]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2c7436af57asm97554325ad.13.2026.06.22.16.27.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jun 2026 16:27:50 -0700 (PDT) Date: Mon, 22 Jun 2026 23:27:47 +0000 From: Samiullah Khawaja To: Baolu Lu Cc: David Woodhouse , Joerg Roedel , Will Deacon , Jason Gunthorpe , Robin Murphy , Kevin Tian , Alex Williamson , Shuah Khan , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Pratyush Yadav , Pasha Tatashin , David Matlack , Andrew Morton , Pranjal Shrivastava , Vipin Sharma Subject: Re: [PATCH v3 09/18] iommu: Add APIs to get iommu and device preserved state Message-ID: References: <20260614233728.2212104-1-skhawaja@google.com> <20260614233728.2212104-10-skhawaja@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: On Mon, Jun 22, 2026 at 11:10:55AM +0800, Baolu Lu wrote: >On 6/15/26 07:37, Samiullah Khawaja wrote: >>The preserved state of the device and IOMMU needs to be fetched during >>shutdown and boot in the next kernel. Add APIs that can be used to fetch >>the preserved state of a device and IOMMU. The APIs will only be used >>during shutdown and after liveupdate so no locking needed. >> >>Reviewed-by: Pranjal Shrivastava >>Signed-off-by: Samiullah Khawaja >>--- >> drivers/iommu/liveupdate.c | 75 ++++++++++++++++++++++++++++++++ >> include/linux/iommu-liveupdate.h | 31 +++++++++++++ >> 2 files changed, 106 insertions(+) >> >>diff --git a/drivers/iommu/liveupdate.c b/drivers/iommu/liveupdate.c >>index 2d782023407d..90750e78cd2a 100644 >>--- a/drivers/iommu/liveupdate.c >>+++ b/drivers/iommu/liveupdate.c >>@@ -18,6 +18,17 @@ >> #define iommu_max_objs_per_page(_array) \ >> ((PAGE_SIZE - sizeof(struct iommu_array_hdr_ser)) / sizeof((_array)->objects[0])) >>+#define iommu_liveupdate_for_each_obj(_arr, _obj, _idx) \ >>+ for ((_idx) = 0, (_obj) = (_arr)->objects; \ >>+ (_idx) < (_arr)->hdr.nr_objects; (_idx)++, (_obj)++) \ >>+ if (((_obj)->hdr.flags & IOMMU_SER_FLAG_DELETED)) \ >>+ continue; \ >>+ else >>+ >>+#define iommu_liveupdate_for_each_arr(_arr) \ >>+ for (; (_arr); (_arr) = (_arr)->hdr.next_array_phys ? \ >>+ phys_to_virt((_arr)->hdr.next_array_phys) : NULL) >>+ >> struct iommu_flb_obj { >> struct mutex lock; >> struct iommu_flb_ser *ser; >>@@ -216,6 +227,70 @@ void iommu_liveupdate_unregister_flb(struct liveupdate_file_handler *handler) >> } >> EXPORT_SYMBOL(iommu_liveupdate_unregister_flb); >>+int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn, >>+ void *arg) >>+{ >>+ struct iommu_flb_obj *flb_obj; >>+ struct iommu_device_array_ser *array; >>+ struct iommu_device_ser *device_ser; >>+ int ret, idx; >>+ >>+ ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj); >>+ if (ret) >>+ return -ENOENT; >>+ >>+ if (!flb_obj->ser->device_array_phys) >>+ return -ENOENT; >>+ >>+ array = phys_to_virt(flb_obj->ser->device_array_phys); >>+ iommu_liveupdate_for_each_arr(array) { >>+ iommu_liveupdate_for_each_obj(array, device_ser, idx) { >>+ ret = fn(device_ser, arg); >>+ if (ret) >>+ goto out; >>+ } >>+ } >>+ >>+out: >>+ liveupdate_flb_put_incoming(&iommu_flb); >>+ return ret; >>+} >>+EXPORT_SYMBOL(iommu_for_each_preserved_device); >>+ >>+struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type) >>+{ >>+ struct iommu_hw_ser *iommu_ser = NULL; >>+ struct iommu_hw_array_ser *array; >>+ struct iommu_flb_obj *flb_obj; >>+ int ret, idx; >>+ >>+ ret = liveupdate_flb_get_incoming(&iommu_flb, (void **)&flb_obj); >>+ if (ret == -ENODATA || ret == -ENOENT) >>+ return NULL; >>+ >>+ if (ret) >>+ return ERR_PTR(ret); > >Why does this helper return both NULL and an ERR_PTR()? Is NULL >considered an error case here? Also, why are -ENODATA and -ENOENT >treated differently? NULL indicates that no preserved state exists for this iommu HW, so setup normally as cold boot. ERR_PTR() indicates a fatal parsing or corruption error in the KHO tree that the driver needs to explicitly handle or abort on. liveupdate_flb_get_incoming() returns two different errors for two different missing-data scenarios: -ENODATA: The system did not perform a live update. -ENOENT: The system did perform a live update, but no data was preserved for this specific FLB. From the IOMMU driver's perspective, both scenarios simply mean "proceed with a normal cold boot," which is why we intercept them and return NULL. But as you suggested later, I will add kdocs for these to document all these things. > >>+ >>+ if (!flb_obj->ser->iommu_array_phys) { >>+ iommu_ser = ERR_PTR(-EINVAL); >>+ goto out; >>+ } >>+ >>+ array = phys_to_virt(flb_obj->ser->iommu_array_phys); >>+ iommu_liveupdate_for_each_arr(array) { >>+ iommu_liveupdate_for_each_obj(array, iommu_ser, idx) { >>+ if (iommu_ser->token == token && iommu_ser->type == type) >>+ goto out; >>+ } >>+ } >>+ >>+ iommu_ser = NULL; >>+out: >>+ liveupdate_flb_put_incoming(&iommu_flb); >>+ return iommu_ser; >>+} >>+EXPORT_SYMBOL(iommu_get_preserved_data); > >These two helpers are intended for use by IOMMU drivers, correct? If so, >please add kernel-doc style comments to document their usage and guide >how drivers should use them. Agreed. Will do. > >>+ >> static int alloc_object_ser(void **curr_array_ptr, u64 max_objs) >> { >> struct iommu_array_hdr_ser *curr_array = *curr_array_ptr; >>diff --git a/include/linux/iommu-liveupdate.h b/include/linux/iommu-liveupdate.h >>index f9528c94979d..5ad006892cbd 100644 >>--- a/include/linux/iommu-liveupdate.h >>+++ b/include/linux/iommu-liveupdate.h >>@@ -13,6 +13,8 @@ >> #include >> #include >>+typedef int (*iommu_preserved_device_iter_fn)(struct iommu_device_ser *ser, >>+ void *arg); >> #ifdef CONFIG_IOMMU_LIVEUPDATE >> static inline void *dev_iommu_preserved_state(struct device *dev) >> { >>@@ -28,6 +30,20 @@ static inline void *dev_iommu_preserved_state(struct device *dev) >> return NULL; >> } >>+static inline void *iommu_domain_restored_state(struct iommu_domain *domain) >>+{ >>+ struct iommu_domain_ser *ser; >>+ >>+ ser = domain->preserved_state; >>+ if (ser && (ser->hdr.flags & IOMMU_SER_FLAG_INCOMING)) >>+ return ser; >>+ >>+ return NULL; >>+} >>+ >>+int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn, >>+ void *arg); >>+struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type); >> int iommu_preserve_domain(struct iommu_domain *domain, struct iommu_domain_ser **ser); >> void iommu_unpreserve_domain(struct iommu_domain *domain); >> int iommu_preserve_device(struct iommu_domain *domain, >>@@ -44,6 +60,21 @@ static inline void *dev_iommu_preserved_state(struct device *dev) >> return NULL; >> } >>+static inline void *iommu_domain_restored_state(struct iommu_domain *domain) >>+{ >>+ return NULL; >>+} >>+ >>+static inline int iommu_for_each_preserved_device(iommu_preserved_device_iter_fn fn, void *arg) >>+{ >>+ return -EOPNOTSUPP; >>+} >>+ >>+static inline struct iommu_hw_ser *iommu_get_preserved_data(u64 token, enum iommu_type_ser type) >>+{ >>+ return NULL; >>+} >>+ >> static inline int iommu_preserve_domain(struct iommu_domain *domain, struct iommu_domain_ser **ser) >> { >> return -EOPNOTSUPP; > >Thanks, >baolu > Thanks, Sami