From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9DCC8CA5FF0 for ; Tue, 6 Oct 2026 00:17:58 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1xDssE-000444-GJ; Mon, 05 Oct 2026 20:17:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1xDsrZ-0003xf-57; Mon, 05 Oct 2026 20:17:01 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1xDsrW-0001Et-JA; Mon, 05 Oct 2026 20:16:56 -0400 Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 695HZS0I057919; Tue, 6 Oct 2026 00:16:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=QHiOv1MkqyHrpv6By 88GPVqHwIC8issXAPZKoIdlNTY=; b=pGviy9VJLA+LIuBg0+oLEZ813rZsH37Ps b9+gE4GkjHW/wvU5hG3b4uuwt7wE8eHNBG1Gu+eJLDBsC0sPuMfDiYjeXwh6I3+/ 9x3iIu89yy3M0Gm4QnPuM2oK6RFVDtM1Pkk2aydUO/dw69cvicWr5t3u2iBbgCCg O9NhUzJQtQj38i1ZEAHZA4Q598jSezHajHKtNOdvXWvrHTuv5UOE+IpEvMgc71oV yBgYL5ZF5cfZKaeDU2fIf3Cx570LLD6ryO4LyARgeNQ/jQ++MvZ5QiQUvNQ3DNft 1sw2VyFHngZXnbALUFsCZosDBFYuJlquchv7wnp/5TK+iwb2u+GcQ== Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4h2r4fvgg6-1 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Tue, 06 Oct 2026 00:16:50 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.11/8.18.1.11) with ESMTP id 695M7Kdj3231089; Tue, 6 Oct 2026 00:16:50 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4h3cdvqu13-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 06 Oct 2026 00:16:50 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (smtpav05.wdc07v.mail.ibm.com [10.39.53.232]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 6960GmGY31916682 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 6 Oct 2026 00:16:49 GMT Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A6E2758053; Tue, 6 Oct 2026 00:16:48 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BDD2258059; Tue, 6 Oct 2026 00:16:47 +0000 (GMT) Received: from WIN-DU0DFC9G5VV.ibm.com (unknown [9.61.249.221]) by smtpav05.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 6 Oct 2026 00:16:47 +0000 (GMT) From: Konstantin Shkolnyy To: mjrosato@linux.ibm.com Cc: alifm@linux.ibm.com, farman@linux.ibm.com, richard.henderson@linaro.org, iii@linux.ibm.com, david@kernel.org, cohuck@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, qemu-s390x@nongnu.org, qemu-devel@nongnu.org, Konstantin Shkolnyy Subject: [PATCH v12 01/17] s390x/pci: implement IOMMU replay Date: Mon, 5 Oct 2026 19:16:27 -0500 Message-Id: <20261006001643.420929-2-kshk@linux.ibm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20261006001643.420929-1-kshk@linux.ibm.com> References: <20261006001643.420929-1-kshk@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=TOPQ2Fla c=1 sm=1 tr=0 ts=6ac43df2 cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=660iZSQnnn4A:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=V8glGbnc2Ofi9Qvn3v5h:22 a=VnNF1IyMAAAA:8 a=uoNow8biE-CX8mucm5YA:9 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYxMDA2MDAwMCBTYWx0ZWRfX/6K3V0ULpdmy +71POHFk9Qipis1xDB1kZ/NY9ojeNXeB+wRnj96wmSEU3unRhim82W3PCRcuMGGTkd0Q6sEPG2H RoNNy9Cy0hVRnO4W7qqeoL2ze0k/FoKgrUlqkOoIxxLzOcpSemDqmm6P3mCU3bAJnqUQM4OV5QJ kRLQsRElSyoPYI9yYoYfjsuKIn676BWxAU5ubhZ51M7OIrktEhLkimmuaoJPO0jZbCHNWFYp/Ur jBrEVofbpxqB1EYpdHTQ5PpoSeRrI0qrZPAZpXdefgvQ62JF03jkCvzvTS/ZIXIgo0g9/UmGwNo 3Z5hDesL9IdDVVeOliKDS60y5P+jmxKKbCIlZS9hBLeqvs4cqNCyn2ZR9RnpbvCpcy4/0+wMaXV 7OQ9WnHFNQoPcywo5vRafHdja9pcoS9aRrE1px0Hr0rKYhMiJD/sA78MyabU0zMlksR0C0JdinV r4/wITzJ15ZJmd3Ukqg== X-Proofpoint-GUID: jUJ8YUmOYh5Wx1jG1IIDY_OqYS7fMhNR X-Proofpoint-ORIG-GUID: jUJ8YUmOYh5Wx1jG1IIDY_OqYS7fMhNR X-Proofpoint-Spam-Info: AW1haW4tMjYxMDA2MDAwMCBTYWx0ZWRfX8HaG5IPiESXW jUBiLysXc1qDAz9yLgvxgAD+ZpuaVaMw7ZWUCGIf7UeP6WpxOkacftVG1rM/0FbfUrHnhQA0Mtb pJsNvULdvvoIbcaAC75JN1deZ105DSo= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1176,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-10-05_05,2026-10-05_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 spamscore=0 adultscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2609040000 definitions=main-2610060000 Received-SPF: pass client-ip=148.163.158.5; envelope-from=kshk@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Matthew Rosato There are a few scenarios where IOMMU replay can potentially be needed for zPCI device, namely VFIO device reset scenarios where the guest continues running and expects the contents of its IOMMU to be replayed upon IOAT re-registration and migration scenarios where the destination must reconstruct the IOMMU on the destination. zPCI migration is not supported yet, but the IOMMU replay function is implemented so that it can be called both from IOMMUMemoryRegionClass now and migration post_load later. Signed-off-by: Matthew Rosato Signed-off-by: Konstantin Shkolnyy Reviewed-by: Farhan Ali --- hw/s390x/s390-pci-bus.c | 71 +++++++++++++++++++++++++++++--- hw/s390x/s390-pci-inst.c | 4 +- include/hw/s390x/s390-pci-inst.h | 1 + 3 files changed, 69 insertions(+), 7 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 2eb4e8cec4..b3f7b59422 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -593,14 +593,75 @@ err: return ret; } -static void s390_pci_iommu_replay(IOMMUMemoryRegion *iommu, +static void s390_pci_ioat_replay(S390PCIIOMMU *iommu) +{ + S390PCIBusDevice *pbdev = iommu->pbdev; + S390IOTLBEntry entry; + uint16_t error = 0; + uint32_t dma_avail; + hwaddr curr, end; + + curr = iommu->pba; + end = iommu->pal; + + if (iommu->dm_mr || !pbdev) { + /* If direct mapping is used, there are no guest tables to replay */ + return; + } + + if (iommu->dma_limit) { + dma_avail = iommu->dma_limit->avail; + } else { + dma_avail = 1; + } + + while (curr < end) { + error = s390_guest_io_table_walk(iommu->g_iota, curr, &entry); + if (error) { + pbdev->state = ZPCI_FS_ERROR; + s390_pci_generate_error_event(error, pbdev->fh, pbdev->fid, curr, + 0); + error_report("Failure to walk table during iommu remap"); + return; + } + + /* Advance to next frame boundary if start was not frame-aligned */ + curr = QEMU_ALIGN_UP(curr + 1, entry.len); + if (entry.perm != IOMMU_NONE) { + while (entry.iova < curr && entry.iova < end) { + if (dma_avail > 0) { + dma_avail = s390_pci_update_iotlb(iommu, &entry); + } else { + /* + * There is no reliable method to request the guest to + * release mappings other than in response to a RPCIT + * instruction; generate a permanent error condition and + * require the device to be completely re-initialized from + * the guest side. + */ + pbdev->state = ZPCI_FS_ERROR; + s390_pci_generate_error_event(ERR_EVENT_PERMERR, pbdev->fh, + pbdev->fid, 0, 0); + error_report("DMA mappings exhausted: iommu remap failed"); + return; + } + entry.iova += TARGET_PAGE_SIZE; + entry.translated_addr += TARGET_PAGE_SIZE; + } + } + } +} + +static void s390_pci_iommu_replay(IOMMUMemoryRegion *mr, IOMMUNotifier *notifier) { - /* It's impossible to plug a pci device on s390x that already has iommu - * mappings which need to be replayed, that is due to the "one iommu per - * zpci device" construct. But when we support migration of vfio-pci - * devices in future, we need to revisit this. + S390PCIIOMMU *iommu = container_of(mr, S390PCIIOMMU, iommu_mr); + + /* + * The notifier argument is not used directly; s390_pci_ioat_replay() + * broadcasts via memory_region_notify_iommu() instead. */ + s390_pci_ioat_replay(iommu); } static S390PCIIOMMU *s390_pci_get_iommu(S390pciState *s, PCIBus *bus, diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 09026360d4..285ff0e089 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -630,8 +630,8 @@ int pcistg_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) return 0; } -static uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, - S390IOTLBEntry *entry) +uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, + S390IOTLBEntry *entry) { S390IOTLBEntry *cache = g_hash_table_lookup(iommu->iotlb, &entry->iova); IOMMUTLBEvent event = { diff --git a/include/hw/s390x/s390-pci-inst.h b/include/hw/s390x/s390-pci-inst.h index 5cb8da540b..c782990e3b 100644 --- a/include/hw/s390x/s390-pci-inst.h +++ b/include/hw/s390x/s390-pci-inst.h @@ -111,6 +111,7 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, int stpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, uintptr_t ra); void fmb_timer_free(S390PCIBusDevice *pbdev); +uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, S390IOTLBEntry *entry); #define ZPCI_IO_BAR_MIN 0 #define ZPCI_IO_BAR_MAX 5 -- 2.34.1