From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F6053D3D02 for ; Fri, 24 Apr 2026 13:06:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777035988; cv=none; b=AymWtyNZlMmkrEJZ8oBpYeBNKkbcEa/aa9GxojRSDYeLxFxMGHoOeoVaNBw0ChNvKkGrGM/zFuJwl2az3SqR0lRn49A8BMYCQF8ThKKPoBYN6qfddBCa28blnrIpLnfZEV3aJMZSaP7RJJO4DzO28/eGXa2uVSwqUwBrYZnxQaE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777035988; c=relaxed/simple; bh=WpBAPCL3BiX3KQ0D/UZVGNGmtqxWijBriUGn1lyIeGw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ZCVObBRpy5ddmO+YODru7GeeAmY323rP57HrpwGsrCneJwxzTaGtv4Ot0C3N10qyDLEaBpLQRCS28FgBLr9J92+ZEgHvB/BfhLXpIZOrEWT+9Hfb7pMUBDO4RRh9uBvwFT+al2jPSzANlyUoyFW/pvyhM1WOStV+LAfyOLA+7II= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca; spf=pass smtp.mailfrom=ziepe.ca; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b=UN19Yhtk; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="UN19Yhtk" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-8cb3bae8d3eso702404985a.1 for ; Fri, 24 Apr 2026 06:06:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1777035978; x=1777640778; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=LdRRvkEqDaXRhj1wK9ax0Kl7CerOjC6Wl0sGdeJ62/E=; b=UN19Yhtk49j1+ZRTncnhuui3cSk5YjrEFPtRVBmi7TtKrkqAwbJS4Q+dMhO5Z0OtHY GKhcA++7dw7O8GrmiHKE0/Ugjvg4XSCQlFRp0savq3BfmTvCEhJYn4Xrb1EqmHAv3rFX Q1+qznfZUDK7qUFvU3W+qCLKkSWzTAENmOsrSaa2WNHJIzavfWdWIMF3IkwwqRoZ0h1J 7BjgBGTA77CWUdkCzF+pGwsv3qR9220cyGLxwRHMsysYs6kWD/IeH1KEzTODZUVXDcKT tJp+BEL64iD5ePS3bAR5tCil+3dxd2yRwQ3XL0z/iIW10nLZXN7Nf3cuNE5y/mw6LX2z cFOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777035978; x=1777640778; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LdRRvkEqDaXRhj1wK9ax0Kl7CerOjC6Wl0sGdeJ62/E=; b=C6eIgtOjAD2pRujsUqBBOdiPxVj8xErqRs/ATARjq+3txGn3lSkwonmoUu6cHe56Rn RSC4Ebq7gvCYEgO7pfNzJUI5HAfPsum5jASVzv/dcbiaS57Ly88FgDOI61A/cPthwv4T kTnYJAyi0RBo5GFjG2Lo4axT/FIcRSO7my9n+/0++sCCM6b1GlQj5Gxu5H5z5itJARDH GVXyIxg+hXe9W4i+/jySNekVEMJajBgQLFj34tjpZuNtLEBFPSaQeDuwma4UJUvuQTqZ ADKuR+MjIF8Xfdn0N7QMXPUj20bh343eye3r0wfME18RNbfysIPCgH6vXVgFwFlSpMAB QTxg== X-Forwarded-Encrypted: i=1; AFNElJ8pyN5xBY+8xWhPcwDzayTCZZ+7S0niFEDFr6mJMB3fRDngA7akRYz1pWlwKWN+KRN/ZLuUzpctV5O/Fe4=@vger.kernel.org X-Gm-Message-State: AOJu0YwTz0KoS5QKIW4AXQ9oIxYJsYlP5B8uHClQCp/RNhlbhNmk/DeE ULIfDX6U4rn/YF6Wpp0Fz+Ro9n/t3Y/N/H33d+cgrr7lIsT8kW/gwceLERtoIlqDQF0= X-Gm-Gg: AeBDieuQX0hN4/OL6LohJy7q++QyLaGFAkW/r41m89BkgFyIfXLOp2qGGEzZfjlOn1p BpWjlIMvrrA/ADIK44E7VY1v5F6QfN5cKkx4o/4yI/BBBCD/ngl63APaPTYC4iRHBOiO6qru4ae xEN/6XQbLux5Nzz4AIWylP7w5MBcKmhBGN4SJYqAGbiqcJzczD7bRLDxGnv+JOcPoIh9n/zqmvb GWYLiHb0kKfbkjhWB+qlpGAKMC7hjx/V0LORe93p09vYN2QnxjQfkLHaiiHpj3cwDyygLDEv5Wd c5kPj+G+Q0xokRWggMgIWkG/EUFmXRpCH33HzOH9FX6st3IPtKIWa2sTFiigfPRtyr4x3TRkH3r Uap6dqLrrGse+GH4UfM53qsP3rR7YS+t/Bs2+o5gCKwUM3zx8ZA30SzZYiUbCqOPZyZeOqUi+Fg w1q1m+3lOI+0H4Na279pu3Ttn4b1Bc6qenF0lHcQxMRMdkXfIpYAZDVVaiqL30HREquBuKFoZ6P hCBHfZzZx3LWr/e X-Received: by 2002:a05:620a:4397:b0:8eb:6f04:f97f with SMTP id af79cd13be357-8eb6f04fb1amr3242464285a.44.1777035978468; Fri, 24 Apr 2026 06:06:18 -0700 (PDT) Received: from ziepe.ca (crbknf0213w-47-54-130-67.pppoe-dynamic.high-speed.nl.bellaliant.net. [47.54.130.67]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8e7d5fe8ed1sm1911934485a.8.2026.04.24.06.06.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Apr 2026 06:06:16 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.97) (envelope-from ) id 1wGGEZ-00000002Ovx-2693; Fri, 24 Apr 2026 10:06:15 -0300 Date: Fri, 24 Apr 2026 10:06:15 -0300 From: Jason Gunthorpe To: Joel Granados Cc: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH RFC 0/5] nvme: Controller Data Queue (CDQ) support Message-ID: <20260424130615.GW3611611@ziepe.ca> References: <20260424-jag-cdq-lkml-v1-0-d773343a717c@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20260424-jag-cdq-lkml-v1-0-d773343a717c@kernel.org> On Fri, Apr 24, 2026 at 01:37:50PM +0200, Joel Granados wrote: > There is however, no clear consensus on how NVMe Live Migration should > land in the Linux kernel. The 2022 discussion [1] explored a VFIO-based > approach but reached no conclusion, likely because the specification was > not yet mature. Yes it was paused until the spec matures, then I expect it to go forward. > To move CDQ forward, I would like to understand where the LM logic belongs. I > currently see two options (of which I have no particular preference): > > 1. VFIO: Implement NVMe LM following the VFIO state machine, similar to what > was proposed in 2022. > 2. VM manager interface: Bypass VFIO and implement LM logic in the interface > between the VM manager (e.g., QEMU) and the NVMe driver. I imagined it to be split between VFIO for the pci and volatile guest state and something else for the namespace setup and media migration. Media migration is only needed for local drive so there use cases that don't need this component. We have many drivers fitting into the VFIO scheme now and good VMM coverage, I don't see a reason to throw it out. > One aspect that has not received much attention in previous discussions > is namespace migration as prior work focused on migrating state and not > the actual data. Migrating potential terabytes is IMO a distinct use > case worth considering. Yes Though IDK if just plumbing the entire CDQ to userspace is the right choice for NVMe.. We don't know what future specs will add to CDQ, it may not be appropriate to treat it so insecurely. Jason