From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 815A6C98302 for ; Wed, 23 Sep 2026 14:16:34 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x9Nkz-0007fg-89; Wed, 23 Sep 2026 10:15:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x9Nkw-0007eK-Hb; Wed, 23 Sep 2026 10:15:30 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x9Nku-0005v0-JN; Wed, 23 Sep 2026 10:15:30 -0400 Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 68N8ZZno2940636; Wed, 23 Sep 2026 14:15:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=bAYeAYaWL8yxEIWEW 4KkE9OfTrRyrN2oieXC0sNnVsc=; b=ETFKPGBNkb+ZGBPWbadMFPR/FyFOguV2K 4ZNoaBRoH2nDBItK5p4IPV2ksyroWXzN4hUU11f+FArhIDf2jaVNHLIgTjR1aaVA 9Bx0cz5HPJCyEkxwJX0MF+EN+zmNaPgmTgAseL5R2/Lt/zaCFkQTBmJcC5v8cMX7 jTT1OV+66p7OZrj2QM7mMAdAdd49eqkuKj9HCDeG+DEU518NPD7ApKj4pRZCrC/C y2+bJy/NiJcqS+82xILxfbRgdntIZ+JI8u3zofpm+ngSO/5tBzlqZjoJBY2NujtZ K9hwrRY3NZuI27HZ05Xk4P8tO4PeWe/ap1IuDHfd4hAIZx5OTL6pQ== Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4gskdvba6d-1 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 23 Sep 2026 14:15:24 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.11/8.18.1.11) with ESMTP id 68NCSo7i008014; Wed, 23 Sep 2026 14:15:23 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([172.16.1.7]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4gvbe1sb1n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 23 Sep 2026 14:15:23 +0000 (GMT) Received: from smtpav02.wdc07v.mail.ibm.com (smtpav02.wdc07v.mail.ibm.com [10.39.53.229]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 68NEFMT332375442 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Sep 2026 14:15:22 GMT Received: from smtpav02.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 329A95805C; Wed, 23 Sep 2026 14:15:22 +0000 (GMT) Received: from smtpav02.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8342F58059; Wed, 23 Sep 2026 14:15:21 +0000 (GMT) Received: from WIN-DU0DFC9G5VV.atx-us.ibm.com (unknown [9.16.60.171]) by smtpav02.wdc07v.mail.ibm.com (Postfix) with ESMTP; Wed, 23 Sep 2026 14:15:21 +0000 (GMT) From: Konstantin Shkolnyy To: mjrosato@linux.ibm.com Cc: alifm@linux.ibm.com, farman@linux.ibm.com, richard.henderson@linaro.org, iii@linux.ibm.com, david@kernel.org, cohuck@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, qemu-s390x@nongnu.org, qemu-devel@nongnu.org, Konstantin Shkolnyy Subject: [PATCH v10 01/16] s390x/pci: implement IOMMU replay Date: Wed, 23 Sep 2026 09:15:03 -0500 Message-Id: <20260923141518.960364-2-kshk@linux.ibm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260923141518.960364-1-kshk@linux.ibm.com> References: <20260923141518.960364-1-kshk@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: MGy8GuPOzInytWn-i08VqM5VFOvt8CHB X-Proofpoint-Spam-Info: AW1haW4tMjYwOTIzMDA1NiBTYWx0ZWRfX2+1jKaAEyrjR VtlEoJ20p5qqQFpKcXp+ChVqlUp2ZNkRic+7sERsn/RAxsfXaOtwcsI158Rw2G+3hiI8MEGQ2Qf +YbjYSShc2iFj+LWao+iRAh/2QsIg68= X-Authority-Analysis: v=2.4 cv=FLiOVOos c=1 sm=1 tr=0 ts=6ab3defc cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=VdqzKS8jKosA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=RzCfie-kr_QcCd8fBx8p:22 a=VnNF1IyMAAAA:8 a=uoNow8biE-CX8mucm5YA:9 X-Proofpoint-GUID: MGy8GuPOzInytWn-i08VqM5VFOvt8CHB X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwOTIzMDA1NiBTYWx0ZWRfX1gdCpp2xrJgX PC4rmy6iXyZ4mxJ7+I6zVk6kiNAzR57aN1PH7h9PnjZumLK69/srRXHCGbmQ4BVYrLQD1kko47s jXKeJ+RSDuVmqeQDKQFNtyZnmke73TSUjra9vYH2Gg+Nmpel5x+StZPtjR9fNUMa9EB0DSeMOGv 44PU6sF94b7bE6vCz6LP/BgQh0ROgx37L6jgI4jf3sPP9UPp45QM7kkxCdDMUyOmXORZrRqUtXC tPJqqrkYIhBodZBjYx+2WPumza9w9Boedok7Q51omftrxKPDaPoAjM+chRTy1SsjxpJVnP2u9YW DGveFWY2bvPoASl4HAnMVLJBnjDkmG0qXMlngXwkwfXxCOruTUsCkYLX7bkqZMtVYMG2ZY/e/6j cnFc8OGFbXbDD650w6iZhTRZNpVEHvwNF425Xq5fzujiB0TO1Amf2S/spRTGG3G1HXdScQ4YscZ Gr51PzCON4586hiaTCA== X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1176,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-09-23_04,2026-09-21_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 impostorscore=0 phishscore=0 spamscore=0 clxscore=1015 suspectscore=0 bulkscore=0 lowpriorityscore=0 priorityscore=1501 malwarescore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2609040000 definitions=main-2609230056 Received-SPF: pass client-ip=148.163.158.5; envelope-from=kshk@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Matthew Rosato There are a few scenarios where IOMMU replay can potentially be needed for zPCI device, namely VFIO device reset scenarios where the guest continues running and expects the contents of its IOMMU to be replayed upon IOAT re-registration and migration scenarios where the destination must reconstruct the IOMMU on the destination. zPCI migration is not supported yet, but the IOMMU replay function is implemented so that it can be called both from IOMMUMemoryRegionClass now and migration post_load later. Signed-off-by: Matthew Rosato Signed-off-by: Konstantin Shkolnyy Reviewed-by: Farhan Ali --- hw/s390x/s390-pci-bus.c | 68 +++++++++++++++++++++++++++++--- hw/s390x/s390-pci-inst.c | 4 +- include/hw/s390x/s390-pci-inst.h | 1 + 3 files changed, 65 insertions(+), 8 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index eff980fdfe..10b417a779 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -593,14 +593,70 @@ err: return ret; } -static void s390_pci_iommu_replay(IOMMUMemoryRegion *iommu, +static void s390_pci_ioat_replay(S390PCIIOMMU *iommu) +{ + S390PCIBusDevice *pbdev = iommu->pbdev; + S390IOTLBEntry entry; + uint16_t error = 0; + uint32_t dma_avail; + hwaddr curr, end; + + curr = iommu->pba; + end = iommu->pal; + + if (iommu->dm_mr || !pbdev) { + /* If direct mapping is used, there are no guest tables to replay */ + return; + } + + if (iommu->dma_limit) { + dma_avail = iommu->dma_limit->avail; + } else { + dma_avail = 1; + } + + while (curr < end) { + error = s390_guest_io_table_walk(iommu->g_iota, curr, &entry); + if (error) { + pbdev->state = ZPCI_FS_ERROR; + s390_pci_generate_error_event(error, pbdev->fh, pbdev->fid, curr, + 0); + error_report("Failure to walk table during iommu remap"); + return; + } + + curr += entry.len; + if (entry.perm != IOMMU_NONE) { + while (entry.iova < curr && entry.iova < end) { + if (dma_avail > 0) { + dma_avail = s390_pci_update_iotlb(iommu, &entry); + } else { + /* + * There is no reliable method to request the guest to + * release mappings other than in response to a RPCIT + * instruction; generate a permanent error condition and + * require the device to be completely re-initialized from + * the guest side. + */ + pbdev->state = ZPCI_FS_ERROR; + s390_pci_generate_error_event(ERR_EVENT_PERMERR, pbdev->fh, + pbdev->fid, 0, 0); + error_report("DMA mappings exhausted: iommu remap failed"); + return; + } + entry.iova += TARGET_PAGE_SIZE; + entry.translated_addr += TARGET_PAGE_SIZE; + } + } + } +} + +static void s390_pci_iommu_replay(IOMMUMemoryRegion *mr, IOMMUNotifier *notifier) { - /* It's impossible to plug a pci device on s390x that already has iommu - * mappings which need to be replayed, that is due to the "one iommu per - * zpci device" construct. But when we support migration of vfio-pci - * devices in future, we need to revisit this. - */ + S390PCIIOMMU *iommu = container_of(mr, S390PCIIOMMU, iommu_mr); + + s390_pci_ioat_replay(iommu); } static S390PCIIOMMU *s390_pci_get_iommu(S390pciState *s, PCIBus *bus, diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index b7de23c7d2..37ca70dcf1 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -630,8 +630,8 @@ int pcistg_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) return 0; } -static uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, - S390IOTLBEntry *entry) +uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, + S390IOTLBEntry *entry) { S390IOTLBEntry *cache = g_hash_table_lookup(iommu->iotlb, &entry->iova); IOMMUTLBEvent event = { diff --git a/include/hw/s390x/s390-pci-inst.h b/include/hw/s390x/s390-pci-inst.h index 5cb8da540b..c782990e3b 100644 --- a/include/hw/s390x/s390-pci-inst.h +++ b/include/hw/s390x/s390-pci-inst.h @@ -111,6 +111,7 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, int stpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, uintptr_t ra); void fmb_timer_free(S390PCIBusDevice *pbdev); +uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, S390IOTLBEntry *entry); #define ZPCI_IO_BAR_MIN 0 #define ZPCI_IO_BAR_MAX 5 -- 2.34.1