From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3428B4CA273 for ; Tue, 14 Jul 2026 15:15:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784042137; cv=none; b=pSUZxS8+99gwaRcTx75LUhlNh2GdSbk40pzdtHIHQ7vXyn2hw18r8x39xIZ3h/7wBFfDBrYXvEK+r1XxbPERJ4ah+DKU3QIu6mDs0vfxSuSAXjS4+/qGZ33ibHKHnapzwjwJMM+ySN/VrWC2QsN5d5sjmz4KLQOyriV2vQOV7HI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784042137; c=relaxed/simple; bh=8CBMauhZZenpn+a2yQ87Rr2EfyqC5lFfzuVr5bte59U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=U6jre/oN8Vpyhu5IiJxSgDes1AJqEb9gJzEZVsW5j/1sbybacLdInXaHBD8PqfNb9hAGCn/Nt2KHvlEay42u7UxZ1HurNTldRqH0Et32QmvTlklsXtjFjqtd2pj2ayRBEUzkYXMgaWXUliQqHqL/lPNBfQ6Htzg9+Cqq8nlGQEQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--vipinsh.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CbeLsf7O; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--vipinsh.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CbeLsf7O" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2cc8bde6318so82381165ad.3 for ; Tue, 14 Jul 2026 08:15:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784042131; x=1784646931; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=xjEe04qswGeo8Cvf7F1uxuMmAPxZARk/R3E4HFO0u5g=; b=CbeLsf7ODiv/CAWg1Rr/dD33QZT/TGVzG4/d5rqy2hjIksaoqkfjs9in7QEiV4flMZ RvIcM4ZQ06NW5ZyxK+AcRRryMs5MC4jN0kCBTT90OQmhpY+luRhXRfxyJdjO0G/Pshul 7n0HfAM5GKt8OAtuk/DaIWQIHJaWuQvX5dljn9BLw/6HQ3dKWO+GdlRf1XhMkw4bnu8a Qv2dySadfBIrnjObPZ9r5fmGX4UKxrd68Nx9QHxAlWTjrZyJewymgHL2HRJyP0f3j5An q6DaHfZW1SBGtNFTrb3Pmm9Jo+9Bgl45BXyWpvtcc4UIMg+3jYoHzARufkmN+awQ8vmJ JgZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784042131; x=1784646931; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xjEe04qswGeo8Cvf7F1uxuMmAPxZARk/R3E4HFO0u5g=; b=eUunrEes8FOxsZMAap2z/ic6q2SshvYUmBy9khSebH5Zqm2CTci2Q0jlRMYb057B5I Keg216eH54coPVeSK0WICC0Zwbogwm7cvmJdVcK07elPkVhACu+CkAAuEujvVXv1plot fDhpyO7W8gNqMcpuDs25fEnZptKg6DFZfRhzoUMYXzt2kcjI3+uJUpldKAcCHPRtuUhu oLDyJf/ZGUdpG2mZNYvY6cXHw3sijvbuatccGwEG1Vl/K9k2WDZZzi0DxA1derfFjaZu 7A/wvwwClrHTZPcVKB+OPOxoFCn2cm37vPkZ5bAKw/FgCa+j+cP+vP2K3GEZjlmR1+Vh RcPQ== X-Forwarded-Encrypted: i=1; AHgh+RpEJ/ed6EiFAlBoj6+9PZ76zth86NwyPMMpsdACD9o7tFF9ehYPpVf8zEmhV/oSVgVjmyzMRKZPpct1EUTyV3M=@vger.kernel.org X-Gm-Message-State: AOJu0YwjMSJaLMbOZQv1OFowAqxcib/fnjQYi1I00ULijsSjU4C2AGFO D7MrpfK7J/HBFf87GTfV2OxKvFPQkgkEx5HLpYBuWEfGpYCwh4yHKwbNx+o5bEQbNECsGWiM/4/ HwJNVXO060A== X-Received: from plbl21.prod.google.com ([2002:a17:902:eb15:b0:2cc:da2c:8085]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:120c:b0:2ce:d1dc:1e1 with SMTP id d9443c01a7336-2cef1332689mr29905365ad.39.1784042130777; Tue, 14 Jul 2026 08:15:30 -0700 (PDT) Date: Tue, 14 Jul 2026 08:15:05 -0700 In-Reply-To: <20260714151505.3466855-1-vipinsh@google.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260714151505.3466855-1-vipinsh@google.com> X-Mailer: git-send-email 2.55.0.795.g602f6c329a-goog Message-ID: <20260714151505.3466855-21-vipinsh@google.com> Subject: [PATCH v5 20/20] vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test From: Vipin Sharma To: kexec@lists.infradead.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org Cc: ajayachandra@nvidia.com, alex@shazbot.org, amastro@fb.com, ankita@nvidia.com, apopple@nvidia.com, bhelgaas@google.com, chrisl@kernel.org, christian.koenig@amd.com, corbet@lwn.net, dmatlack@google.com, graf@amazon.com, jacob.pan@linux.microsoft.com, jgg@nvidia.com, jgg@ziepe.ca, jrhilke@google.com, julianr@linux.ibm.com, kees@kernel.org, kevin.tian@intel.com, leon@kernel.org, leonro@nvidia.com, lukas@wunner.de, mattev@meta.com, michal.winiarski@intel.com, parav@nvidia.com, pasha.tatashin@soleen.com, praan@google.com, pratyush@kernel.org, rananta@google.com, rientjes@google.com, rodrigo.vivi@intel.com, rppt@kernel.org, saeedm@nvidia.com, schnelle@linux.ibm.com, skhan@linuxfoundation.org, skhawaja@google.com, vipinsh@google.com, vivek.kasireddy@intel.com, witu@nvidia.com, yanjun.zhu@linux.dev, yi.l.liu@intel.com Content-Type: text/plain; charset="UTF-8" From: David Matlack Add a long-running DMA memcpy operation to vfio_pci_liveupdate_kexec_test so that the device attempts to perform DMAs continuously during the Live Update. At this point iommufd preservation is not supported and bus mastering is not kept enabled on the device across the kexec, so most of these DMAs will be dropped. However this test ensures that the current device preservation support does not lead to system instability or crashes if the device is active. And once iommufd and bus mastering are preserved, this test can be relaxed to check that the DMA operations completed successfully. Signed-off-by: David Matlack Signed-off-by: Vipin Sharma --- .../vfio/vfio_pci_liveupdate_kexec_test.c | 129 ++++++++++++++++++ 1 file changed, 129 insertions(+) diff --git a/tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test.c b/tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test.c index ec3fe8f19502..51b2f24f352c 100644 --- a/tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test.c +++ b/tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test.c @@ -1,8 +1,16 @@ // SPDX-License-Identifier: GPL-2.0-only +#include +#include + #include #include +#define MEMCPY_SIZE SZ_1G +#define DRIVER_SIZE SZ_1M +#define MEMFD_SIZE (MEMCPY_SIZE + DRIVER_SIZE) + +static struct dma_region memcpy_region; static const char *device_bdf; static char state_session[LIVEUPDATE_SESSION_NAME_LENGTH]; @@ -11,8 +19,89 @@ static char device_session[LIVEUPDATE_SESSION_NAME_LENGTH]; enum { STATE_TOKEN, DEVICE_TOKEN, + MEMFD_TOKEN, }; +static void dma_memcpy_one(struct vfio_pci_device *device) +{ + void *src = memcpy_region.vaddr, *dst; + u64 size; + + size = min_t(u64, memcpy_region.size / 2, device->driver.max_memcpy_size); + dst = src + size; + + memset(src, 1, size); + memset(dst, 0, size); + + printf("Kicking off 1 DMA memcpy operations of size 0x%lx...\n", size); + vfio_pci_driver_memcpy(device, + to_iova(device, src), + to_iova(device, dst), + size); + + VFIO_ASSERT_EQ(memcmp(src, dst, size), 0); +} + +static void dma_memcpy_start(struct vfio_pci_device *device) +{ + void *src = memcpy_region.vaddr, *dst; + u64 count, size; + + size = min_t(u64, memcpy_region.size / 2, device->driver.max_memcpy_size); + dst = src + size; + + /* + * Rough Math: If we assume the device will perform memcpy at a rate of + * 30GB/s then 7200GB of transfers will run for about 4 minutes. + */ + count = (u64)7200 * SZ_1G / size; + count = min_t(u64, count, device->driver.max_memcpy_count); + + memset(src, 1, size / 2); + memset(dst, 0, size / 2); + + printf("Kicking off %lu DMA memcpy operations of size 0x%lx...\n", count, size); + vfio_pci_driver_memcpy_start(device, + to_iova(device, src), + to_iova(device, dst), + size, count); +} + +static void dma_memfd_map(struct vfio_pci_device *device, int fd) +{ + void *vaddr; + + vaddr = mmap(NULL, MEMFD_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); + VFIO_ASSERT_NE(vaddr, MAP_FAILED); + + memcpy_region.iova = SZ_4G; + memcpy_region.size = MEMCPY_SIZE; + memcpy_region.vaddr = vaddr; + iommu_map(device->iommu, &memcpy_region); + + device->driver.region.iova = memcpy_region.iova + memcpy_region.size; + device->driver.region.size = DRIVER_SIZE; + device->driver.region.vaddr = vaddr + memcpy_region.size; + iommu_map(device->iommu, &device->driver.region); +} + +static void dma_memfd_setup(struct vfio_pci_device *device, int session_fd) +{ + int fd, ret; + + fd = memfd_create("dma-buffer", 0); + VFIO_ASSERT_GE(fd, 0); + + ret = fallocate(fd, 0, 0, MEMFD_SIZE); + VFIO_ASSERT_EQ(ret, 0); + + printf("Preserving memfd of size 0x%x in session\n", MEMFD_SIZE); + ret = luo_session_preserve_fd(session_fd, fd, MEMFD_TOKEN); + VFIO_ASSERT_EQ(ret, 0); + + dma_memfd_map(device, fd); +} + static void before_kexec(int luo_fd) { struct vfio_pci_device *device; @@ -32,6 +121,27 @@ static void before_kexec(int luo_fd) ret = luo_session_preserve_fd(session_fd, device->fd, DEVICE_TOKEN); VFIO_ASSERT_EQ(ret, 0); + dma_memfd_setup(device, session_fd); + + /* + * If the device has a selftests driver, kick off a long-running DMA + * operation to exercise the device trying to DMA during a Live Update. + * Since iommufd preservation is not supported yet, these DMAs should be + * dropped. So this is just looking to verify that the system does not + * fall over and crash as a result of a busy device being preserved. + */ + if (device->driver.ops) { + vfio_pci_driver_init(device); + dma_memcpy_start(device); + + /* + * Disable interrupts on the device or freeze() will fail. + * Unfortunately there isn't a way to easily have a test for + * that here since the check happens during shutdown. + */ + vfio_pci_msix_disable(device); + } + close(luo_fd); daemonize_and_wait(); } @@ -78,6 +188,7 @@ static void after_kexec(int luo_fd, int state_session_fd) struct iommu *iommu; int session_fd; int device_fd; + int memfd; int stage; check_open_vfio_device_fails(); @@ -88,6 +199,10 @@ static void after_kexec(int luo_fd, int state_session_fd) session_fd = luo_retrieve_session(luo_fd, device_session); VFIO_ASSERT_GE(session_fd, 0); + printf("Retrieving memfd from LUO\n"); + memfd = luo_session_retrieve_fd(session_fd, MEMFD_TOKEN); + VFIO_ASSERT_GE(memfd, 0); + printf("Finishing the session before retrieving the device (should fail)\n"); VFIO_ASSERT_NE(luo_session_finish(session_fd), 0); @@ -109,9 +224,23 @@ static void after_kexec(int luo_fd, int state_session_fd) */ device = __vfio_pci_device_init(device_bdf, iommu, device_fd); + dma_memfd_map(device, memfd); + printf("Finishing the session\n"); VFIO_ASSERT_EQ(luo_session_finish(session_fd), 0); + /* + * Once iommufd preservation is supported and the device is kept fully + * running across the Live Update, this should wait for the long- + * running DMA memcpy operation kicked off in before_kexec() to + * complete. But for now we expect the device to be reset so just + * trigger a single memcpy to make sure it's still functional. + */ + if (device->driver.ops) { + vfio_pci_driver_init(device); + dma_memcpy_one(device); + } + vfio_pci_device_cleanup(device); iommu_cleanup(iommu); } -- 2.55.0.795.g602f6c329a-goog