From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from PH0PR06CU001.outbound.protection.outlook.com (mail-westus3azon11011050.outbound.protection.outlook.com [40.107.208.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7096C3C1973 for ; Tue, 28 Apr 2026 15:54:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.208.50 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777391666; cv=fail; b=XrAiO2pcI/Wq31AjVbNJnMXNFUB7zNFdAXQI3fjtQ2p2CbRYmKWJ7sXWulgX8pQKU312fZ1b19xgIdvYmD9C51bu7bHchDc57XLdgbuKkhD/UiuPwUOBnvRzwZHDT5uM+uEJu27wvWsJElRohkNCcWFE6qqP28gWqwJhneT7wUg= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777391666; c=relaxed/simple; bh=nUsTCoeyfbKl1XTS5ddQErKxBMek4VAW+NuyJ43vSbE=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=AosWs7FpC9qUra9IhfOXQJ9dwub44kbTwAs3SafUTZjSGVR+fjen42w/8XdkypLbdmyQDhEZv/5dgJnmLjlTBKJSy4R+HTTJa+QSDbKO4C11svhTdMMFK3YN8AYlech9Qch4n98NVFK81nRcrdHBgppAI2H74Zyyy1H4LZZN20g= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=RJDWg4Xq; arc=fail smtp.client-ip=40.107.208.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="RJDWg4Xq" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tk2N2YPRRJkeyLINY0Dgw+YRVDCnH0oZi5J9Ul6QhsYTjl+GPeNw2HCDiNa755+kNausuR3MM7+BH7LQTzfRtRXU9JMpH/ULCjl3OCQMeQr3nXQA/D4YETBYfrTPssP3hMKNWdns2+1JkknN5GKQrcg3PjwO3bc4L75EgVLMRP5QqBbbd0wLNq6wOxDjefNA79n+4alt9vpa99ODj6zihxUCX3BStVWR5Bzm2xABxSwn/eefjfvtSKg4BiITWyLqwjUP2deZqhjXurJ87ll6jp8bc6BbiNpPJKqqFVZOewgQhDtrsypa5VyEvlPWez6rdSPwr8rnS8CmXJBlJDISKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nljV61LSXhzqhBC1urFr4hQnCE/si+5O1JBTyx/owO8=; b=mlcBbRRCnWaH+AzveqUGMyev+PCfjzwu15Ek+JYC+stwfq0sSdyN0aYPc2u01pyFYl5UEyuIRS2C5S6H32+ACY7f06CDRwVl+nGxyOodY+UHVuywvQJIguuqBcvWLtVgTFUIOWJ2g706xwitSXJaNtwRXplQkMIncdGbDcIDjzFZRAoBgO+D9NCNCXafcGpCF6SSHSLEjTP2eRZ/zRvxNu0G5hfNjjKY0wkdhFYZ5jsoITRJavGpAk5WB0rHh6DUiNqMAzCKOXfJWhHps6f+KiJZnzxWHIGWQVHdjuFpua3HC0SlPWNs0BzR/JrJrPUzDxAonj99uZ5z5j1hYWDPRQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nljV61LSXhzqhBC1urFr4hQnCE/si+5O1JBTyx/owO8=; b=RJDWg4Xqs2jAUTs5to12Fhb+8wnOHduzJMusemPjtLYmQCuLvtBL4KRndvt79/ABt3Vt8Eg2/BPVZTgUaVJlixQ327B78Cp41aIx144QbYL5dIylZolwH5hnEH9BtTtjaw891ct+1YD5v92YBxvYPjmGX+bYDRmfGEMIC6K9SNk= Received: from BL1PR13CA0071.namprd13.prod.outlook.com (2603:10b6:208:2b8::16) by CH3PR12MB7715.namprd12.prod.outlook.com (2603:10b6:610:151::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.16; Tue, 28 Apr 2026 15:54:10 +0000 Received: from MN1PEPF0000F0DF.namprd04.prod.outlook.com (2603:10b6:208:2b8:cafe::ec) by BL1PR13CA0071.outlook.office365.com (2603:10b6:208:2b8::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9846.26 via Frontend Transport; Tue, 28 Apr 2026 15:54:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by MN1PEPF0000F0DF.mail.protection.outlook.com (10.167.242.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.18 via Frontend Transport; Tue, 28 Apr 2026 15:54:09 +0000 Received: from kaveri.amd.com (10.180.168.240) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Tue, 28 Apr 2026 10:53:56 -0500 From: Shivank Garg To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Shivank Garg , Mike Day Subject: [PATCH 5/7] mm/migrate: add copy offload registration infrastructure Date: Tue, 28 Apr 2026 15:50:47 +0000 Message-ID: <20260428155043.39251-12-shivankg@amd.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260428155043.39251-2-shivankg@amd.com> References: <20260428155043.39251-2-shivankg@amd.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: satlexmb07.amd.com (10.181.42.216) To satlexmb07.amd.com (10.181.42.216) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000F0DF:EE_|CH3PR12MB7715:EE_ X-MS-Office365-Filtering-Correlation-Id: e7e449f8-45ac-431d-565c-08dea53e62f3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|82310400026|36860700016|376014|56012099003|22082099003|18002099003; X-Microsoft-Antispam-Message-Info: XH6ktHeLnbIFMXhfQn36RfXHuma+Xr7Mf59sdHcUlKE4XvKV6ZJY+vYE7ZgF9Bal6jl/Nu08IjX+4AHAaeFgQn6zmSoUtynBMHgEmgPfLgnJjzYOYInOoiDC481DQABG7OuvJFgxJx9WaP7FZjAnKcvHgcSdKogqMFbVF/jYgUZXQA3BZZZOCUdM9uJzuKONeuqgQasYuOFK2aRvPehWDREVK1dz6fqkGgrRiA9OVVkkeBdVkSkyr6OZ21qKbFyixRugpS+/CD6YUhnhdD+Li2C/EQKz4bWlQ6MMN56cwI8lOVTXEhQ1/qnMEEQJ08pyxEhlE5MklaS4RqaxTiFJ6+SrW/CcJt+PZ8rXUJeZLrIjPV4dh/mVAz71DNBeRUXjQM8EbWQt2Ns7A/6X5EXDFddcJJnOAYl5YvZn8aUUcbyUVNLXDKsbdJemj7L/MI2dZayKyD/YMd/4JNpiE/8+GkzN1sW31pRaMzhtbBKzPg/NbJDwbSFCMC4L7Hd2jrekOpr8VjM5VSsvHjtppLadwahne+qxECVk3RAZzJL9OcvGbbjtm+Ih8Cf8BtDyw3w6ADII82QKp+plCPjDCjENPdGB5cBB+Hr9o4yGs4zH1mIhZjZWB24i9pfG0WJ2Cu77D5T5LvWA0We4rRcti9zswD7QAvR6LrqFICWebENl53etw6u7rhV5VizdfO6itj4lGtloRwTxb+f0NWWsBkcC23WiAePfFFoqKjZKs4IfqvP+61Q7o7g+LcyHku7ahesd6qg2JXAhCYVfWK/MempoEw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(82310400026)(36860700016)(376014)(56012099003)(22082099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qZZDn8hzIxRrMr6Jzp7YyBRcFOi0STRPK8HPTUVamIFOQkkqYxnHI9e0+v92FMrn4vx8hi5YNrVi5PDZjxCrkDH4/UXcpilBiEWkNJTOh6eUpZez3s1OpLggrfj0VquPTH8CDgyKzTdk8JyWqLpF4Q6xxkT1+DI77ehtqHpIyr9gfeamPbO+tFXqqoWZ2ri+CzMzdAkpmiqmlPZpXwXnMHhbPgvT/q9y8i5WXLkvKx+gDva7J/k5mcEpR/I8e/N/6bSPEcrf/uB3fFGs81YOa5Qxu47dy1TAdd7zIzmPrrWkShbAvroIEwI87DhMmSiuOqcJX3mJ1EgJ87YBsSC/fmK8yHYZZpZzJOlLtRKBR/ggkxncH71qhUAS0ThjfOsG9GgE7VuQTNcVnl4Jm3VMcgGZqDzBO8fYT36s1XMq64IgVZivNhMo2Zbm7O2fr+fs X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2026 15:54:09.9422 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e7e449f8-45ac-431d-565c-08dea53e62f3 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000F0DF.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7715 Add a registration interface that lets a single offload provider (DMA, multi-threaded CPU copy, etc) take over the batch folio copy performed by migrate_pages_batch(). The provider fills in a struct migrator with an offload_copy() callback and calls migrate_offload_register(). Registration patches the migrate_offload_copy() static_call and flips the migrate_offload_enabled static branch. The migrate_offload_unregister() reverts both. Whether a migration reason is batch-copy eligible is decided by the core in migrate_offload_do_batch(). A migrator may decline a particular batch (e.g. when nr_batch is too small to amortize setup) by returning -EOPNOTSUPP, and the move phase falls back to per-folio CPU copy. Only one migrator can be active at a time. A second registration returns -EBUSY, and only the active migrator can unregister itself. The static_call dispatch is protected by SRCU so that the synchronize_srcu() in unregister waits for all in-flight copy before the module reference is dropped. Co-developed-by: Mike Day Signed-off-by: Mike Day Signed-off-by: Shivank Garg --- include/linux/migrate_copy_offload.h | 44 +++++++++++++ mm/Kconfig | 6 ++ mm/Makefile | 1 + mm/migrate.c | 57 +++++++++++++++-- mm/migrate_copy_offload.c | 94 ++++++++++++++++++++++++++++ 5 files changed, 198 insertions(+), 4 deletions(-) create mode 100644 include/linux/migrate_copy_offload.h create mode 100644 mm/migrate_copy_offload.c diff --git a/include/linux/migrate_copy_offload.h b/include/linux/migrate_copy_offload.h new file mode 100644 index 000000000000..d68b10a84743 --- /dev/null +++ b/include/linux/migrate_copy_offload.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_MIGRATE_COPY_OFFLOAD_H +#define _LINUX_MIGRATE_COPY_OFFLOAD_H + +#include +#include +#include +#include + +struct list_head; +struct module; + +#define MIGRATOR_NAME_LEN 32 + +/** + * struct migrator - batch-copy provider for page migration. + * @name: name of the provider. + * @offload_copy: copy @folio_cnt folios from @src_list to @dst_list. + * + * The migrator may inspect @folio_cnt to decide whether the batch + * is worth offloading, e.g. skip when the batch is too small to + * amortize setup cost. If returns error, the core falls back to CPU copy. + * + * @owner: module providing the migrator. + */ +struct migrator { + char name[MIGRATOR_NAME_LEN]; + int (*offload_copy)(struct list_head *dst_list, + struct list_head *src_list, + unsigned int folio_cnt); + struct module *owner; +}; + +#ifdef CONFIG_MIGRATION_COPY_OFFLOAD +extern struct static_key_false migrate_offload_enabled; +extern struct srcu_struct migrate_offload_srcu; +int migrate_offload_register(struct migrator *m); +int migrate_offload_unregister(struct migrator *m); +#else +static inline int migrate_offload_register(struct migrator *m) { return -EOPNOTSUPP; } +static inline int migrate_offload_unregister(struct migrator *m) { return -EOPNOTSUPP; } +#endif + +#endif /* _LINUX_MIGRATE_COPY_OFFLOAD_H */ diff --git a/mm/Kconfig b/mm/Kconfig index e8bf1e9e6ad9..325d79619680 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -647,6 +647,12 @@ config MIGRATION config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE +# Page-migration batch-copy offload infrastructure. +# Selected by migrator drivers (e.g. CONFIG_DCBM_DMA). +config MIGRATION_COPY_OFFLOAD + bool + depends on MIGRATION + config ARCH_ENABLE_HUGEPAGE_MIGRATION bool diff --git a/mm/Makefile b/mm/Makefile index 8ad2ab08244e..db1ac8097089 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -96,6 +96,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_MIGRATION_COPY_OFFLOAD) += migrate_copy_offload.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o diff --git a/mm/migrate.c b/mm/migrate.c index 6c2f1cb66f96..9af070f9a1f2 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -44,6 +44,8 @@ #include #include #include +#include +#include #include @@ -54,6 +56,51 @@ DEFINE_STATIC_KEY_FALSE(migrate_offload_enabled); +#ifdef CONFIG_MIGRATION_COPY_OFFLOAD +DEFINE_SRCU(migrate_offload_srcu); +DEFINE_STATIC_CALL(migrate_offload_copy, folios_mc_copy); + +static bool migrate_offload_do_batch(int reason) +{ + if (!static_branch_unlikely(&migrate_offload_enabled)) + return false; + + switch (reason) { + case MR_COMPACTION: + case MR_SYSCALL: + case MR_DEMOTION: + case MR_NUMA_MISPLACED: + return true; + default: + return false; + } +} + +static int migrate_offload_batch_copy(struct list_head *dst_batch, + struct list_head *src_batch, + unsigned int nr_batch) +{ + int idx, rc; + + idx = srcu_read_lock(&migrate_offload_srcu); + rc = static_call(migrate_offload_copy)(dst_batch, src_batch, nr_batch); + srcu_read_unlock(&migrate_offload_srcu, idx); + return rc; +} +#else +static bool migrate_offload_do_batch(int reason) +{ + return false; +} + +static int migrate_offload_batch_copy(struct list_head *dst_batch, + struct list_head *src_batch, + unsigned int nr_batch) +{ + return -EOPNOTSUPP; +} +#endif + static const struct movable_operations *offline_movable_ops; static const struct movable_operations *zsmalloc_movable_ops; @@ -1833,7 +1880,7 @@ static int migrate_pages_batch(struct list_head *from, struct folio *folio, *folio2, *dst = NULL; int rc, rc_saved = 0, nr_pages; unsigned int nr_batch = 0; - bool batch_copied = false; + bool do_batch = false, batch_copied = false; LIST_HEAD(unmap_batch); LIST_HEAD(dst_batch); LIST_HEAD(unmap_single); @@ -1843,6 +1890,8 @@ static int migrate_pages_batch(struct list_head *from, VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); + do_batch = migrate_offload_do_batch(reason); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; @@ -1984,8 +2033,7 @@ static int migrate_pages_batch(struct list_head *from, nr_retry_pages += nr_pages; break; case 0: - if (static_branch_unlikely(&migrate_offload_enabled) && - folio_supports_batch_copy(folio)) { + if (do_batch && folio_supports_batch_copy(folio)) { list_move_tail(&folio->lru, &unmap_batch); list_add_tail(&dst->lru, &dst_batch); nr_batch++; @@ -2017,7 +2065,8 @@ static int migrate_pages_batch(struct list_head *from, /* Batch-copy eligible folios before the move phase */ if (!list_empty(&unmap_batch)) { - rc = folios_mc_copy(&dst_batch, &unmap_batch, nr_batch); + rc = migrate_offload_batch_copy(&dst_batch, &unmap_batch, + nr_batch); batch_copied = (rc == 0); } diff --git a/mm/migrate_copy_offload.c b/mm/migrate_copy_offload.c new file mode 100644 index 000000000000..6f837c725239 --- /dev/null +++ b/mm/migrate_copy_offload.c @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include + +static DEFINE_MUTEX(migrator_mutex); +static struct migrator *active_migrator; + +DECLARE_STATIC_CALL(migrate_offload_copy, folios_mc_copy); + +/** + * migrate_offload_register - register a batch-copy provider for page migration. + * @m: migrator to install. + * + * Only one provider can be active at a time, returns -EBUSY if another migrator + * is already registered. + * + * Return: 0 on success, negative errno on failure. + */ +int migrate_offload_register(struct migrator *m) +{ + int ret = 0; + + if (!m || !m->offload_copy || !m->owner) + return -EINVAL; + + mutex_lock(&migrator_mutex); + if (active_migrator) { + ret = -EBUSY; + goto unlock; + } + + if (!try_module_get(m->owner)) { + ret = -ENODEV; + goto unlock; + } + + static_call_update(migrate_offload_copy, m->offload_copy); + active_migrator = m; + static_branch_enable(&migrate_offload_enabled); + +unlock: + mutex_unlock(&migrator_mutex); + + if (ret) + pr_err("migrate_offload: %s: failed to register (%d)\n", + m->name, ret); + else + pr_info("migrate_offload: enabled by %s\n", m->name); + return ret; +} +EXPORT_SYMBOL_GPL(migrate_offload_register); + +/** + * migrate_offload_unregister - unregister the active batch-copy provider. + * @m: migrator to remove (must be the currently active one). + * + * Reverts static_call targets and waits for SRCU grace period so that + * no in-flight migration is still calling the driver functions before + * releasing the module. + * + * Return: 0 on success, negative errno on failure. + */ +int migrate_offload_unregister(struct migrator *m) +{ + struct module *owner; + + mutex_lock(&migrator_mutex); + if (active_migrator != m) { + mutex_unlock(&migrator_mutex); + return -EINVAL; + } + + /* + * Disable the static branch first so new migrate_pages_batch calls + * won't enter the batch copy path. + */ + static_branch_disable(&migrate_offload_enabled); + static_call_update(migrate_offload_copy, folios_mc_copy); + owner = active_migrator->owner; + active_migrator = NULL; + mutex_unlock(&migrator_mutex); + + /* Wait for all in-flight callers to finish before module_put(). */ + synchronize_srcu(&migrate_offload_srcu); + module_put(owner); + + pr_info("migrate_offload: disabled by %s\n", m->name); + return 0; +} +EXPORT_SYMBOL_GPL(migrate_offload_unregister); -- 2.43.0