From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A648CD6E7C for ; Fri, 5 Jun 2026 02:19:10 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1wVK84-0005NI-75; Thu, 04 Jun 2026 22:17:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wVK80-0005Hd-4t; Thu, 04 Jun 2026 22:17:44 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1wVK7w-0001lw-VS; Thu, 04 Jun 2026 22:17:43 -0400 Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 654LdXTD867987; Fri, 5 Jun 2026 02:17:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=xKaH8/VylfBSI/rIS APTq6UY2nsuNOVzBdGFpBFZyxg=; b=q4OIOu4qKXIfod1iZ35I+1fS36br+bf7o r866ZOfg1r+TVZqoLN9RxBraYNt2jE0xa7Hg7WDTN9IEjGuYlxw2bpSPMn8x6YSK MGcY4YKqaPKbBqIU4u1oZZ+1Z0mPrShn91sXX/nQYhix7mQD1WDUbSPGmJ3N/o8m rGRwQeGROP4a6Z8s2m0gYcGgjt1JpFXnW0dDlXCwCx/PT/93msxRg6hdukEWjGdp Ppl7vT2GdUylR7zvG7C9umQh4m2/kYtsPybmZFeWJuR0ycZWn2Cv2nnXhxbfxrhA yxdbSRdBKuxiq3Qg9YE9gvf/X1La5TzKi2VdjwpIwsMa/bLaG4lNw== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4efqjqjf1j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 05 Jun 2026 02:17:36 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 65529DbS025313; Fri, 5 Jun 2026 02:17:35 GMT Received: from smtprelay07.wdc07v.mail.ibm.com ([172.16.1.74]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4egcegyh0u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 05 Jun 2026 02:17:35 +0000 (GMT) Received: from smtpav05.dal12v.mail.ibm.com (smtpav05.dal12v.mail.ibm.com [10.241.53.104]) by smtprelay07.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 6552HXmx22610566 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 5 Jun 2026 02:17:33 GMT Received: from smtpav05.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5C48A58067; Fri, 5 Jun 2026 02:17:33 +0000 (GMT) Received: from smtpav05.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3083B5805D; Fri, 5 Jun 2026 02:17:33 +0000 (GMT) Received: from WIN-DU0DFC9G5VV.ibm.com (unknown [9.61.240.22]) by smtpav05.dal12v.mail.ibm.com (Postfix) with ESMTP; Fri, 5 Jun 2026 02:17:33 +0000 (GMT) From: Konstantin Shkolnyy To: mjrosato@linux.ibm.com Cc: alifm@linux.ibm.com, richard.henderson@linaro.org, iii@linux.ibm.com, david@kernel.org, cohuck@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, qemu-s390x@nongnu.org, qemu-devel@nongnu.org, Konstantin Shkolnyy Subject: [PATCH v3 01/15] s390x/pci: implement IOMMU replay Date: Thu, 4 Jun 2026 21:17:14 -0500 Message-Id: <20260605021728.1125090-2-kshk@linux.ibm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260605021728.1125090-1-kshk@linux.ibm.com> References: <20260605021728.1125090-1-kshk@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 53QjjMJwpMqH9KgNUSb6pym-MS6aX4pM X-Proofpoint-GUID: 53QjjMJwpMqH9KgNUSb6pym-MS6aX4pM X-Authority-Analysis: v=2.4 cv=bcVbluPB c=1 sm=1 tr=0 ts=6a2231c0 cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=FelO9ux0wxsA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=Ne9W6HA6cR8Tgt2TMNYA:9 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNjA1MDAxOCBTYWx0ZWRfX7wx2y3bPrLBd o+Y+T8RfL2zCH2TINGNeCNEe4y9vuYDnW0EuXPAB+15Y6o+Q9G6AT+ZgPcMwLR2xxHNJg0Pgibv v3kYRZr98Fw+wu+5kRVMZ1YJvNTQVScQHY4EkPvepT5RBrfg2XkktcuUuJohtX60ahmUuOoEMWt ACVl5xnW/hgOmb+AQc5lXZKAr+CoDliSlFyqpj0SSjgAcz/KybWRQf2gPvQDWVuJou/hV1rfBgM MMowEcI6sYAKmhwa5eyQLz74JA2OnvOPEWAAuHnIXeNL50x/69VhWzOxepFvX9xZgIYZQKrxBzP uA9gBsSwznBZFhsyVohQ1hzAKDTb5z8CZqNd3rBltTFx3HDwKLLV8yMex1e8VqUpw9lGFpMNnmX 6gQsoxThXwpycnns/PE7WnMb3B58/U7KOlYlyTHqDZcdBPpV8NzhlMxizJHtU371Qb6uTaf5rDs kDO8y7VljzDX7V7lcFg== X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.125,FMLib:17.12.100.49 definitions=2026-06-04_07,2026-05-28_03,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 clxscore=1015 adultscore=0 priorityscore=1501 phishscore=0 lowpriorityscore=0 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2605210000 definitions=main-2606050018 Received-SPF: pass client-ip=148.163.156.1; envelope-from=kshk@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Matthew Rosato There are a few scenarios where IOMMU replay can potentially be needed for zPCI device, namely VFIO device reset scenarios where the guest continues running and expects the contents of its IOMMU to be replayed upon IOAT re-registration and migration scenarios where the destination must reconstruct the IOMMU on the destination. zPCI migration is not supported yet, but the IOMMU replay function is implemented so that it can be called both from IOMMUMemoryRegionClass now and migration post_load later. Signed-off-by: Matthew Rosato Signed-off-by: Konstantin Shkolnyy --- hw/s390x/s390-pci-bus.c | 62 ++++++++++++++++++++++++++++---- hw/s390x/s390-pci-inst.c | 4 +-- include/hw/s390x/s390-pci-inst.h | 1 + 3 files changed, 59 insertions(+), 8 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 4de7b587e8..a104e550b1 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -592,14 +592,64 @@ err: return ret; } -static void s390_pci_iommu_replay(IOMMUMemoryRegion *iommu, +static void s390_pci_ioat_replay(S390PCIIOMMU *iommu) +{ + S390IOTLBEntry entry; + uint16_t error = 0; + uint32_t dma_avail; + hwaddr curr, end; + + curr = iommu->pba; + end = iommu->pal; + + if (iommu->dm_mr) { + /* If direct mapping is used, there are no guest tables to replay */ + return; + } + + if (iommu->dma_limit) { + dma_avail = iommu->dma_limit->avail; + } else { + dma_avail = 1; + } + + while (curr < end) { + error = s390_guest_io_table_walk(iommu->g_iota, curr, &entry); + if (error) { + pbdev->state = ZPCI_FS_ERROR; + s390_pci_generate_error_event(error, pbdev->fh, pbdev->fid, curr, + 0); + error_report("Failure to walk table during iommu remap"); + return; + } + + if (entry.perm != IOMMU_NONE) { + if (dma_avail > 0) { + dma_avail = s390_pci_update_iotlb(iommu, &entry); + } else { + /* + * There is no reliable method to request the guest to release + * mappings other than in response to a RPCIT instruction; + * generate a permanent error condition and require the device + * to be completely re-initialized from the guest side. + */ + pbdev->state = ZPCI_FS_ERROR; + s390_pci_generate_error_event(ERR_EVENT_PERMERR, pbdev->fh, + pbdev->fid, 0, 0); + error_report("DMA mappings exhausted: iommu remap failed"); + return; + } + } + curr += entry.len; + } +} + +static void s390_pci_iommu_replay(IOMMUMemoryRegion *mr, IOMMUNotifier *notifier) { - /* It's impossible to plug a pci device on s390x that already has iommu - * mappings which need to be replayed, that is due to the "one iommu per - * zpci device" construct. But when we support migration of vfio-pci - * devices in future, we need to revisit this. - */ + S390PCIIOMMU *iommu = container_of(mr, S390PCIIOMMU, iommu_mr); + + s390_pci_ioat_replay(iommu); } static S390PCIIOMMU *s390_pci_get_iommu(S390pciState *s, PCIBus *bus, diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 10066ca618..1834596076 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -613,8 +613,8 @@ int pcistg_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) return 0; } -static uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, - S390IOTLBEntry *entry) +uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, + S390IOTLBEntry *entry) { S390IOTLBEntry *cache = g_hash_table_lookup(iommu->iotlb, &entry->iova); IOMMUTLBEvent event = { diff --git a/include/hw/s390x/s390-pci-inst.h b/include/hw/s390x/s390-pci-inst.h index 5cb8da540b..c782990e3b 100644 --- a/include/hw/s390x/s390-pci-inst.h +++ b/include/hw/s390x/s390-pci-inst.h @@ -111,6 +111,7 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, int stpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, uintptr_t ra); void fmb_timer_free(S390PCIBusDevice *pbdev); +uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, S390IOTLBEntry *entry); #define ZPCI_IO_BAR_MIN 0 #define ZPCI_IO_BAR_MAX 5 -- 2.34.1