From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E40171112276 for ; Thu, 2 Apr 2026 02:31:44 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1w87pU-0004kQ-E2; Wed, 01 Apr 2026 22:30:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1w87ov-0004Eh-6k; Wed, 01 Apr 2026 22:30:10 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1w87ot-0001yn-1z; Wed, 01 Apr 2026 22:30:08 -0400 Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 631M4hsB560642; Thu, 2 Apr 2026 02:30:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=t5cVYkxBAOpmPSSXd FdsVxwtAJBjRxBXKstytrtL+qc=; b=U9h/5jsV5fZ38DkQM8nGtw9voZEy30wuf 0l+2LXyUOLfC7OoA7MIhgM2zZf+CgBRyqfWru2PoItC7qkI+MoY6FRB7zkNyVBIK xVSmnP/7N9YxHae6gySPuRhmPsllgxCG1Pq8J1jrU7+nvWu0mBjGyD33xulJ/3Ly LqhNfXVJkFyCz/fnUzTaQxfDjRIHWKcsf8OjYZEKnxzjZDiBdCakNdJLknSCxGPe 4Id87xLeVdJhOTAuFKjfD8YFvQOMxVYM5RqFL5ZNyCAXoh09tzRU40HyxIpokV+F ePuoJ0puOmtIG/Kn/K5auhVOH7ZljY2wD3UBnvqDhfuz6smy3xiZA== Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4d66g231pu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 02 Apr 2026 02:30:02 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 6321iueg008698; Thu, 2 Apr 2026 02:30:02 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 4d6v11r449-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 02 Apr 2026 02:30:02 +0000 Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 6322U03N29491796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Apr 2026 02:30:01 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C26C8580F4; Thu, 2 Apr 2026 02:30:00 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 39DB3580F6; Thu, 2 Apr 2026 02:30:00 +0000 (GMT) Received: from WIN-DU0DFC9G5VV.ibm.com (unknown [9.61.247.119]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Thu, 2 Apr 2026 02:30:00 +0000 (GMT) From: Konstantin Shkolnyy To: mjrosato@linux.ibm.com Cc: alifm@linux.ibm.com, richard.henderson@linaro.org, iii@linux.ibm.com, david@kernel.org, cohuck@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, qemu-s390x@nongnu.org, qemu-devel@nongnu.org, Konstantin Shkolnyy Subject: [PATCH 14/15] s390x/pci: Implement migration for emulated devices Date: Wed, 1 Apr 2026 21:29:20 -0500 Message-Id: <20260402022921.298818-15-kshk@linux.ibm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260402022921.298818-1-kshk@linux.ibm.com> References: <20260402022921.298818-1-kshk@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=Fdo6BZ+6 c=1 sm=1 tr=0 ts=69cdd4aa cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=A5OVakUREuEA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=1WZqjCYO3q-P4G2YYo4A:9 X-Proofpoint-ORIG-GUID: oRdHDHgOv2BWWHXZ973_1EsBb1UfJMUD X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNDAyMDAxOSBTYWx0ZWRfX/Z0hZxiogMlK A07AO9zBdtKJuQnFveUdp1kP61DHQXVuKKNktKgcnvY/TMphbSfrJNa1j42thT6A1xFJ+cAZ1P2 egPQiQNpwJw+JkjEbO+OFZGztnMme9pfTR3CscTyxIBkz6ylcmQAi/vC1zG4+62SFlBaolL0VS3 HVkSteYwL9CvIJJvWxL2B5efwizzmveWZDUY6xp/XTvhZjyN2DXaRG4ETAEKmdC65+DKMFh9UxF jHV6zik3fGoxkewFMgfiP562g1y06s0TRA87+yj1Y3Jx6MV9gxsDayG7yA4lU4GneHORj6PIpwg ObekpDNmtwolZk0lcLUOz97F5ZLOP6vOk4MsRn6b51bWwroK+9Xvr26+Jp3OC6MufqoOZKBZF76 k0BHOplINqx+neIQgE/9egWJ9JshsvQNg0DXJcNyFFumCb/Fs6iB2xvPQAebnfeej3ptD8fiakj nMY2Eh28lzuZVB1U0HA== X-Proofpoint-GUID: oRdHDHgOv2BWWHXZ973_1EsBb1UfJMUD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-04-01_05,2026-04-01_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 clxscore=1015 bulkscore=0 suspectscore=0 priorityscore=1501 adultscore=0 malwarescore=0 phishscore=0 spamscore=0 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2603050001 definitions=main-2604020019 Received-SPF: pass client-ip=148.163.156.1; envelope-from=kshk@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -6 X-Spam_score: -0.7 X-Spam_bar: / X-Spam_report: (-0.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=1, RCVD_IN_VALIDITY_RPBL_BLOCKED=1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Implement zPCI device state migration, consequently enabling migration of VMs that have emulated PCI devices, whether virtio or not. Migration is allowed for devices whose function handle has the FH_SHM_EMUL bit set. For these devices QEMU will save and restore the state of its zPCI emulator. Passthrough devices will continue to block migration. Signed-off-by: Konstantin Shkolnyy --- hw/s390x/s390-pci-bus.c | 112 ++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-inst.c | 2 +- include/hw/s390x/s390-pci-bus.h | 2 + 3 files changed, 110 insertions(+), 6 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index f339a0fd94..b3037b9a8a 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -26,6 +26,7 @@ #include "hw/pci/pci_bridge.h" #include "hw/pci/msi.h" #include "exec/cpu-common.h" +#include "migration/blocker.h" #include "qemu/error-report.h" #include "qemu/module.h" #include "system/reset.h" @@ -1116,6 +1117,28 @@ static int s390_pci_interp_plug(S390pciState *s, S390PCIBusDevice *pbdev) return 0; } +static int s390_set_passthrough_migration_blocker(S390PCIBusDevice *pbdev, + Error **errp) +{ + pbdev->passthrough_migr_blocker = NULL; + + if (pbdev->fh & FH_SHM_EMUL) { + return 0; + } + error_setg(&pbdev->passthrough_migr_blocker, + "Migration blocked by passthrough zPCI device " + "fh 0x%x uid %d fid %d", pbdev->fh, pbdev->uid, pbdev->fid); + + return migrate_add_blocker(&pbdev->passthrough_migr_blocker, errp); +} + +static void s390_clear_passthrough_migration_blocker(S390PCIBusDevice *pbdev) +{ + if (pbdev->passthrough_migr_blocker) { + migrate_del_blocker(&pbdev->passthrough_migr_blocker); + } +} + static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { @@ -1235,6 +1258,11 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, return; } + if (s390_set_passthrough_migration_blocker(pbdev, errp)) { + s390_pci_msix_free(pbdev); + return; + } + if (dev->hotplugged) { s390_pci_generate_plug_event(HP_EVENT_TO_CONFIGURED , pbdev->fh, pbdev->fid); @@ -1271,6 +1299,8 @@ static void s390_pcihost_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, return; } + s390_clear_passthrough_migration_blocker(pbdev); + s390_pci_generate_plug_event(HP_EVENT_STANDBY_TO_RESERVED, pbdev->fh, pbdev->fid); bus = pci_get_bus(pci_dev); @@ -1600,13 +1630,85 @@ static const Property s390_pci_device_properties[] = { true), }; -static const VMStateDescription s390_pci_device_vmstate = { - .name = TYPE_S390_PCI_DEVICE, +static int s390_pci_device_post_load(void *opaque, int version_id) +{ + S390PCIBusDevice *pbdev = S390_PCI_DEVICE(opaque); + + /* + * Now that S390PCIBusDevice fields have been restored, regenerate IOMMU + * state - that includes IOTLB contents and QEMU memory regions. + */ + if (pbdev->iommu_enabled) { + assert(pbdev->iommu); + if (s390_pci_is_translation_enabled(pbdev->g_iota)) { + s390_pci_iommu_enable(pbdev); + s390_pci_ioat_replay(pbdev); + } else { + s390_pci_iommu_direct_map_enable(pbdev); + } + } + /* - * TODO: add state handling here, so migration works at least with - * emulated pci devices on s390x + * Guest sets fmb_addr by mpcifc.ZPCI_MOD_FC_SET_MEASURE instruction. + * The handler consequently starts fmb_timer. Now that fmb_addr has been + * restored, we may need to restart fmb_timer. */ - .unmigratable = 1, + if (pbdev->fmb_addr) { + assert(!pbdev->fmb_timer); + assert(pbdev->pci_group); + pbdev->fmb_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, + fmb_update, pbdev); + timer_mod(pbdev->fmb_timer, + qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + + pbdev->pci_group->zpci_group.mui); + } + return 0; +} + +static const VMStateDescription s390_pci_device_vmstate = { + .name = TYPE_S390_PCI_DEVICE, + .version_id = 1, + .minimum_version_id = 1, + .post_load = s390_pci_device_post_load, + .fields = (const VMStateField[]) { + VMSTATE_UINT32(state, S390PCIBusDevice), + VMSTATE_UINT16(uid, S390PCIBusDevice), + VMSTATE_UINT32(idx, S390PCIBusDevice), + VMSTATE_UINT32(fh, S390PCIBusDevice), + VMSTATE_UINT32(fid, S390PCIBusDevice), + VMSTATE_BOOL(fid_defined, S390PCIBusDevice), + VMSTATE_UINT64(fmb_addr, S390PCIBusDevice), + VMSTATE_UINT32(fmb.format, S390PCIBusDevice), + VMSTATE_UINT32(fmb.sample, S390PCIBusDevice), + VMSTATE_UINT64(fmb.last_update, S390PCIBusDevice), + VMSTATE_UINT64_ARRAY(fmb.counter, S390PCIBusDevice, + ARRAY_SIZE(((S390PCIBusDevice *)0)->fmb.counter)), + VMSTATE_UINT64(fmb.fmt0.dma_rbytes, S390PCIBusDevice), + VMSTATE_UINT64(fmb.fmt0.dma_wbytes, S390PCIBusDevice), + VMSTATE_UINT8(isc, S390PCIBusDevice), + VMSTATE_UINT16(noi, S390PCIBusDevice), + VMSTATE_UINT8(sum, S390PCIBusDevice), + VMSTATE_UINT8(pft, S390PCIBusDevice), + VMSTATE_UINT64(routes.adapter.ind_addr, S390PCIBusDevice), + VMSTATE_UINT64(routes.adapter.summary_addr, S390PCIBusDevice), + VMSTATE_UINT64(routes.adapter.ind_offset, S390PCIBusDevice), + VMSTATE_UINT32(routes.adapter.summary_offset, S390PCIBusDevice), + VMSTATE_UINT32(routes.adapter.adapter_id, S390PCIBusDevice), + VMSTATE_BOOL(iommu_enabled, S390PCIBusDevice), + VMSTATE_UINT64(g_iota, S390PCIBusDevice), + VMSTATE_UINT64(pba, S390PCIBusDevice), + VMSTATE_UINT64(pal, S390PCIBusDevice), + VMSTATE_UINT64(max_dma_limit, S390PCIBusDevice), + VMSTATE_PTR_TO_IND_ADDR(summary_ind, S390PCIBusDevice), + VMSTATE_PTR_TO_IND_ADDR(indicator, S390PCIBusDevice), + VMSTATE_BOOL(pci_unplug_request_processed, S390PCIBusDevice), + VMSTATE_BOOL(unplug_requested, S390PCIBusDevice), + VMSTATE_BOOL(interp, S390PCIBusDevice), + VMSTATE_BOOL(forwarding_assist, S390PCIBusDevice), + VMSTATE_BOOL(aif, S390PCIBusDevice), + VMSTATE_BOOL(rtr_avail, S390PCIBusDevice), + VMSTATE_END_OF_LIST() + } }; static void s390_pci_device_class_init(ObjectClass *klass, const void *data) diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 191f549ba7..a9c0a4effb 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -1094,7 +1094,7 @@ static int fmb_do_update(S390PCIBusDevice *pbdev, int offset, uint64_t val, return ret; } -static void fmb_update(void *opaque) +void fmb_update(void *opaque) { S390PCIBusDevice *pbdev = opaque; int64_t t = qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h index 2fab28e6e0..2edb023112 100644 --- a/include/hw/s390x/s390-pci-bus.h +++ b/include/hw/s390x/s390-pci-bus.h @@ -337,6 +337,7 @@ struct S390PCIBusDevice { uint16_t uid; uint32_t idx; uint32_t fh; + Error *passthrough_migr_blocker; uint32_t fid; bool fid_defined; uint64_t fmb_addr; @@ -415,5 +416,6 @@ S390PCIBusDevice *s390_pci_find_dev_by_pci(S390pciState *s, S390PCIBusDevice *s390_pci_find_next_avail_dev(S390pciState *s, S390PCIBusDevice *pbdev); void s390_pci_ism_reset(void); +void fmb_update(void *opaque); #endif -- 2.34.1