From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51983C71136 for ; Thu, 12 Jun 2025 18:15:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EAAF26B00A2; Thu, 12 Jun 2025 14:15:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E5A936B00A3; Thu, 12 Jun 2025 14:15:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D22B36B00A4; Thu, 12 Jun 2025 14:15:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B00546B00A2 for ; Thu, 12 Jun 2025 14:15:34 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5F6E8C17B6 for ; Thu, 12 Jun 2025 18:15:34 +0000 (UTC) X-FDA: 83547551388.12.CD02AC3 Received: from mail-yb1-f177.google.com (mail-yb1-f177.google.com [209.85.219.177]) by imf07.hostedemail.com (Postfix) with ESMTP id 5BFE440016 for ; Thu, 12 Jun 2025 18:15:32 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VrALpk3L; spf=pass (imf07.hostedemail.com: domain of bijan311@gmail.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=bijan311@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749752132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yJgUDGFlTqLrMXyweY6vXETHiIGKHN448RSe2G/RQd4=; b=AkNcKLxcwc9LE6iNMrePuTZhZ6RDt/sU55Uda5DExMjUibPOx7x0RhffJV37rR8k4gyR6a Y7QnOHbmtnUmBPIV7yLE5wQt2lRw9fIbRDjrCR44z45glgiPrhrL28ypxEMM/XWIC+gOc8 O3Q00M/X8uwPeTTMco9/lROU2k0dZhA= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VrALpk3L; spf=pass (imf07.hostedemail.com: domain of bijan311@gmail.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=bijan311@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749752132; a=rsa-sha256; cv=none; b=W6u4kQEKodMoGHMjxkRFwShcS925J9U0t9exiDVkXNyNy72dYcjABBiBGW0ICQ0aLldoF8 hr88pTu134uP66eh/aDMz5hXEifIjshIXouXXsqJvRgle5xHdR5bdpf6v7PaxonGuIfHSn wNAFwb9AbIkSx79GLVTygJb+sa3QuPo= Received: by mail-yb1-f177.google.com with SMTP id 3f1490d57ef6-e7db6aaef22so1024855276.0 for ; Thu, 12 Jun 2025 11:15:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749752131; x=1750356931; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yJgUDGFlTqLrMXyweY6vXETHiIGKHN448RSe2G/RQd4=; b=VrALpk3LA2ijXyGeX/dUaEysfNlQcdirmn/xP/wVGNX0bRvcNF0D82PPbqnDPfyYtZ PvLBJiPgdaELUz95mjNogySpJgK4ZcRdxpy1yDVyIroiU2D+yatHG+hYVEOBqqLiyye2 LzaGY9/XQpwFIt5yHZ0UeQxHOw80RQke1n5PxANHnvNYoZkLC5zdOxq5ONWr3HTmqWkL XFHFNd5tG1bw/DGT0MqT2bd8VLdZiQoaoGl0BVRIr2w2CeVTS1svW2QJDH5ClPVyxfEd Zn5T9/1K6NrxkpwqgoHvLL7LD5dmoAPh1icZbW9Nvb8qqwVUokrQakceRzMKo94uExTv jwhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749752131; x=1750356931; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yJgUDGFlTqLrMXyweY6vXETHiIGKHN448RSe2G/RQd4=; b=Ui79evQ/xKiyvRBHNIRj14h6brLny1Fc0Kx52WRYH1O4fYijf5d3CQ7pApoo7JAJks IkVtxWCaD5dEVJabidv9v2RmgKa15py2CHwUJY82Vi0JofVWATO2LvJMRZooO5WC1tLF hliDuM1foW0rM+3vkhf/MRCrL3IlShUwXZrg2NkIRCa4U7uvGHO8hwC4lw6Ookee1al6 E7zdE7GRBjkhe8gwUMtFjDrstLQrffsSB0d0umQRtAw2hzYu0OH4YCGHWOjUeB7JhLBo uBPqNoMOucl+/+nrKAwVE0+Zo2F1OHsKIRKuDmkdjTY37ouhVOz1VcAduGoJXIEFKagh SdBg== X-Forwarded-Encrypted: i=1; AJvYcCUODdd3wwdRKzzdIgg5JbswSV84f09zvCKiRsVkcrNneui8KxX5Hbp3jodyBf56Dh+XDQtbw8x/SA==@kvack.org X-Gm-Message-State: AOJu0Yxvansd8oY9USdl/OlXgvRSRsgKUSZB5dq51zFJc+nvkpCCDozl +p1scNkrvMQVoYgi80dU5qkCjPZDESbif9dmhSWcpqyOvX1wSMsuul5/ X-Gm-Gg: ASbGncvzIaxrpSmhdeTGZAu1u6UqUubGVXUIgRvIn15uTRqkO/6NTToG/1Y1GeTkn9z in7aYMOm4rEQRlYXNTP+g0EImJuyWboSzMiAmgzABsg+iORKqaBIsIlwm22jd0+phlnnnJ18iJI NSoT+BQ4Tejxva+dqXq0i9q8AWL2pquEwXXCsmm+pFo0Dpb+VPw4mqgjgNU//bh7huZCT1cjzWV p3xonHeR4aYugaDabXIWX/WjWag9fYDnehFhlHx6NV/aV91hdXLiZQK6DTEUAxK/r40qq18V/fX dI/pwxnAGOZPlMX3Cejf/t2WKfKiangBF5Dm25QKcRsWGcet12BMW8cCINptxlwZLeRdBxNTQHG 3nr4YylU= X-Google-Smtp-Source: AGHT+IGmai82U6it5jDvpRdZAOh/erEuNSBZ/HFRj2fOcXstR7G50ztztivaNFHDIHN9lND4uANkQQ== X-Received: by 2002:a05:6902:4790:b0:e7d:d162:f15a with SMTP id 3f1490d57ef6-e820b719f47mr6337384276.40.1749752131219; Thu, 12 Jun 2025 11:15:31 -0700 (PDT) Received: from bijan-laptop.attlocal.net ([2600:1700:680e:c000:dd1b:d4ae:15de:11db]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e820e312452sm592480276.40.2025.06.12.11.15.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jun 2025 11:15:30 -0700 (PDT) From: Bijan Tabatabai To: damon@lists.linux.com, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Cc: sj@kernel.org, akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Subject: [RFC PATCH 4/4] mm/damon/vaddr: Add vaddr version of DAMOS_INTERLEAVE Date: Thu, 12 Jun 2025 13:13:30 -0500 Message-ID: <20250612181330.31236-5-bijan311@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250612181330.31236-1-bijan311@gmail.com> References: <20250612181330.31236-1-bijan311@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5BFE440016 X-Stat-Signature: 8pzspzz5szghqs9msi1rytouextrih5r X-Rspam-User: X-HE-Tag: 1749752132-598349 X-HE-Meta: U2FsdGVkX19rgp6TG4fg7tQIItqqD2F1JLYWF+i2h2jl4GLWcQFb2qpHr68AyoonDiwshRYebfY9nZfwn/ql76YFubMiG1t8ODZJA2z8GTMe8cuW8nZHi5vy5zsGsdTtU9Kgll98deG4gcUfqakxp1Ja+7pJDB/dN7BiqLlj+qq2ydZ7pH4wzT1rFH5hgjrYwYajvBSuqHtPmAVw8rDWBOZY68wbO4NVU02CMYq6cXpw/Pz6Ipc1JCqWzv5QsBxf3itZqMCnCvW5K4Z3391+R+C9E8uJBkd5XonE850uZ/37j4ASm2CFLFA20OY/6WfUrAirWT8O1aj+X3CveKf2rSm5pnycVOl7z/qofIEI6s/Z8M264G7NPc2NzCmcvu7oSsll69m8bmXd5ob9Fr/7Gc7J8Zq++6x5ThrIskFzfOrjwE47503uqecKKmtPTh0mx+US/uNnEnKTNZ61miRdAmhq+0QFXx8u2doX/zQRHvz3MFtU1ZL4Ea2avY/48/YnQ/iuzO1cwaGXLNqJvb7UiwZ3hAP1agzpb3/OXFrLbbXnGII3zukiQcrNrH5aWDPzrTBpkLSY23eurrZWMgYmcXpMQnHmvS0IxEmQ9Let7oE+5aSaRflzlc9EhmwkRkVhdLoph1HXz31yVh8TDsuM9Ww+PtKWhRTWaALn5v6avq8OtGwNIML9GM0ZfEAQ+TTodPd6NUMUWRnToRCaouznS3UlcItwhb0wgtkpL1Ywlmr6Mp3+4QtwknFAUAeki7zrQrbWuRscsvMdham2CdhhWoS9kl9bwPf4RepZ30ntHMyZxLthy4EtGxwBhnFb0rQ7Y1CSl/xqK4PpzgOBp69Ow1NM0x2Ppc8iBbS3MfM1nRMxAAa8nnoz4IBFG9YEziMBtLnus6vCirMXEnMyvDfkjQZO/OHNm9YgtRw77STL1q+AlDbdsKxjC8g31Ez4BSnzsYnWqId3Mpa00qPkBl2 V6gOsNnK tM0YuwiRt8dBuafQaUWaUTH3ljZY0MIB57bDGpt4+I76AwpiMgONNoRKgiOBC2UXjdpnvffJv3OACzIFrveZKapzH3hQYnb8Zw7L9wSljprLFMDQeJeBOMeWGaNzyl1kuFn1Dk3vV6iaoXxTKyGyjovzymN6WOG0z9qvjN1QdIa4WWa8TKO88z6QH0MQuZm5TDmzAchoWaAZaCtTNloOgPkySx/CqpMoKNlV59tSnfTvWZoaC/dIyp+E/DkIOwRZI7tntnqpn+5ZbJs9kD70YtRHsVCvua0rHIlnhfiMEytpsFU9bx+yhaqt/UPYt7oAKBAgx+Kik07wD1zgTigZfa0z+UUABzWUCX4CioQ2FvdLAQeJtYCB7q//zC4ZPkFylYV1Ufm/UNsuFNvzeoOdbohewC6NeXXRpSauOmmZFI3bhNcvC1UCP/g/aAG/I+waVF5LoxIaNc1dk4DweGNTrq+Zs+7DyNB9E3a7eHmsO4lMidd2pc2VHXve7s7qavVViVJl6tKnOciBs9kjknSnfKdNb/z2odp5PrRXTsGStQzlo9mc6oF1vQxgTWCd+3+mDZaIV5q/da7TB2NiA1e6snyaXSe3XPwv/yCl9gjPvydgKVGE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Bijan Tabatabai This patch adds a vaddr implementation of the DAMOS_INTERLEAVE action. Below is an example of its usage where pages are initially interleaved at a 1:1 ratio and then changed to be interleaved at a 2:1 ratio. The alloc_data program simply allocates 1GB of data then sleeps. $ echo 1 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node0 $ echo 1 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node1 $ numactl -w 0,1 ./alloc_data 1G& [1] 11447 $ cat interleave_vaddr.yaml kdamonds: - state: null pid: null contexts: - ops: vaddr targets: - pid: 11447 regions: [] intervals: sample_us: 200 ms aggr_us: 5 s ops_update_us: 10 s nr_regions: min: 200 max: 500 schemes: - action: interleave access_pattern: sz_bytes: min: 0 B max: max nr_accesses: min: 0 % max: 100 % age: min: 0 ns max: max $ sudo ./damo/damo start interleave_vaddr.yaml $ numastat -c -p 11447 Per-node process memory usage (in MBs) for PID 11447 (alloc_data) Node 0 Node 1 Total ------ ------ ----- Huge 0 0 0 Heap 0 0 0 Stack 0 0 0 Private 514 514 1027 ------- ------ ------ ----- Total 514 514 1027 $ echo 2 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node0 $ numastat -c -p 11447 Per-node process memory usage (in MBs) for PID 11447 (alloc_data) Node 0 Node 1 Total ------ ------ ----- Huge 0 0 0 Heap 0 0 0 Stack 0 0 0 Private 684 343 1027 ------- ------ ------ ----- Total 684 343 1027 Signed-off-by: Bijan Tabatabai --- Documentation/mm/damon/design.rst | 2 +- mm/damon/ops-common.c | 13 +++ mm/damon/ops-common.h | 2 + mm/damon/paddr.c | 11 +-- mm/damon/vaddr.c | 135 ++++++++++++++++++++++++++++++ 5 files changed, 155 insertions(+), 8 deletions(-) diff --git a/Documentation/mm/damon/design.rst b/Documentation/mm/damon/design.rst index c50d2105cea0..a79ba62f820b 100644 --- a/Documentation/mm/damon/design.rst +++ b/Documentation/mm/damon/design.rst @@ -456,7 +456,7 @@ that supports each action are as below. - ``migrate_cold``: Migrate the regions prioritizing colder regions. Supported by ``paddr`` operations set. - ``interleave``: Interleave the regions according to the weighted interleave weights. - Supported by ``paddr`` operations set. + Supported by ``vaddr``, ``fvaddr`` and ``paddr`` operations set. - ``stat``: Do nothing but count the statistics. Supported by all operations sets. diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 2c4fb274b7f6..59d92404fc8f 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -261,3 +261,16 @@ unsigned long damon_migrate_pages(struct list_head *folio_list, return nr_migrated; } + +int damon_interleave_target_nid(unsigned long addr, struct vm_area_struct *vma, + struct mempolicy *pol, struct folio *folio) +{ + pgoff_t ilx; + int target_nid; + + ilx = vma->vm_pgoff >> folio_order(folio); + ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + folio_order(folio)); + policy_nodemask(0, pol, ilx, &target_nid); + + return target_nid; +} diff --git a/mm/damon/ops-common.h b/mm/damon/ops-common.h index 54209a7e70e6..bacb4de92dc9 100644 --- a/mm/damon/ops-common.h +++ b/mm/damon/ops-common.h @@ -18,3 +18,5 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r, struct damos *s); unsigned long damon_migrate_pages(struct list_head *folio_list, int target_nid); +int damon_interleave_target_nid(unsigned long addr, struct vm_area_struct *vma, + struct mempolicy *pol, struct folio *folio); diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 722d69f26e37..93e3c72b54c7 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -415,7 +415,7 @@ static unsigned long damon_pa_migrate(struct damon_region *r, struct damos *s, } #if defined(CONFIG_MEMCG) && defined(CONFIG_NUMA) -struct damos_interleave_private { +struct damos_pa_interleave_private { struct list_head *folio_migration_list; bool putback_lru; }; @@ -425,9 +425,8 @@ static bool damon_pa_interleave_rmap(struct folio *folio, struct vm_area_struct { struct mempolicy *pol; struct task_struct *task; - pgoff_t ilx; int target_nid; - struct damos_interleave_private *priv = arg; + struct damos_pa_interleave_private *priv = arg; task = rcu_dereference(vma->vm_mm->owner); if (!task) @@ -443,9 +442,7 @@ static bool damon_pa_interleave_rmap(struct folio *folio, struct vm_area_struct return true; } - ilx = vma->vm_pgoff >> folio_order(folio); - ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + folio_order(folio)); - policy_nodemask(0, pol, ilx, &target_nid); + target_nid = damon_interleave_target_nid(addr, vma, pol, folio); if (target_nid != NUMA_NO_NODE && folio_nid(folio) != target_nid) { list_add(&folio->lru, &priv->folio_migration_list[target_nid]); @@ -459,7 +456,7 @@ static bool damon_pa_interleave_rmap(struct folio *folio, struct vm_area_struct static unsigned long damon_pa_interleave(struct damon_region *r, struct damos *s, unsigned long *sz_filter_passed) { - struct damos_interleave_private priv; + struct damos_pa_interleave_private priv; struct rmap_walk_control rwc; unsigned long addr, applied; struct folio *folio; diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 46554e49a478..1d1170f49317 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -9,12 +9,14 @@ #include #include +#include #include #include #include #include #include +#include "../internal.h" #include "ops-common.h" #ifdef CONFIG_DAMON_VADDR_KUNIT_TEST @@ -653,6 +655,137 @@ static unsigned long damos_madvise(struct damon_target *target, } #endif /* CONFIG_ADVISE_SYSCALLS */ +#ifdef CONFIG_NUMA +struct damos_va_interleave_private { + struct list_head *folio_migration_list; + struct mempolicy *pol; +}; + +static void damos_va_interleave_folio(unsigned long addr, struct folio *folio, + struct vm_area_struct *vma, struct damos_va_interleave_private *priv) +{ + int target_nid; + + if (!folio_isolate_lru(folio)) + return; + + target_nid = damon_interleave_target_nid(addr, vma, priv->pol, folio); + + if (target_nid != NUMA_NO_NODE && folio_nid(folio) != target_nid) + list_add(&folio->lru, &priv->folio_migration_list[target_nid]); + else + folio_putback_lru(folio); + +} + +static int damos_va_interleave_pmd(pmd_t *pmd, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + struct damos_va_interleave_private *priv = walk->private; + struct folio *folio; + spinlock_t *ptl; + pmd_t pmde; + + ptl = pmd_lock(walk->mm, pmd); + pmde = pmdp_get(pmd); + + if (!pmd_present(pmde) || !pmd_trans_huge(pmde)) + goto unlock; + + folio = damon_get_folio(pmd_pfn(pmde)); + if (!folio) + goto unlock; + + damos_va_interleave_folio(addr, folio, walk->vma, priv); + + folio_put(folio); +unlock: + spin_unlock(ptl); + return 0; +} + +static int damos_va_interleave_pte(pte_t *pte, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + struct damos_va_interleave_private *priv = walk->private; + struct folio *folio; + + if (pte_none(*pte) || !pte_present(*pte)) + return 0; + + folio = vm_normal_folio(walk->vma, addr, *pte); + if (!folio) + return 0; + folio_get(folio); + + damos_va_interleave_folio(addr, folio, walk->vma, priv); + + folio_put(folio); + return 0; +} + +static unsigned long damos_va_interleave(struct damon_target *target, + struct damon_region *r, struct damos *s) +{ + struct damos_va_interleave_private priv; + struct task_struct *task; + struct mm_struct *mm; + int ret; + unsigned long applied = 0; + struct mm_walk_ops walk_ops = { + .pmd_entry = damos_va_interleave_pmd, + .pte_entry = damos_va_interleave_pte, + }; + + task = damon_get_task_struct(target); + if (!task) + return 0; + + priv.pol = get_task_policy(task); + if (!priv.pol) + goto put_task; + + if (priv.pol->mode != MPOL_WEIGHTED_INTERLEAVE) + goto put_pol; + + priv.folio_migration_list = kmalloc_array(nr_node_ids, sizeof(struct list_head), + GFP_KERNEL); + if (!priv.folio_migration_list) + goto put_pol; + + for (int i = 0; i < nr_node_ids; i++) + INIT_LIST_HEAD(&priv.folio_migration_list[i]); + + mm = damon_get_mm(target); + if (!mm) + goto free_folio_list; + + mmap_read_lock(mm); + ret = walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv); + mmap_read_unlock(mm); + mmput(mm); + + for (int i = 0; i < nr_node_ids; i++) { + applied += damon_migrate_pages(&priv.folio_migration_list[i], i); + cond_resched(); + } + +free_folio_list: + kfree(priv.folio_migration_list); +put_pol: + mpol_cond_put(priv.pol); +put_task: + put_task_struct(task); + return applied * PAGE_SIZE; +} +#else +static unsigned long damos_va_interleave(struct damon_target *target, + struct damon_region *r, struct damos *s) +{ + return 0; +} +#endif /* CONFIG_NUMA */ + static unsigned long damon_va_apply_scheme(struct damon_ctx *ctx, struct damon_target *t, struct damon_region *r, struct damos *scheme, unsigned long *sz_filter_passed) @@ -675,6 +808,8 @@ static unsigned long damon_va_apply_scheme(struct damon_ctx *ctx, case DAMOS_NOHUGEPAGE: madv_action = MADV_NOHUGEPAGE; break; + case DAMOS_INTERLEAVE: + return damos_va_interleave(t, r, scheme); case DAMOS_STAT: return 0; default: -- 2.43.5