From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8479C2FC0F for ; Thu, 17 Aug 2023 18:05:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354146AbjHQSEk (ORCPT ); Thu, 17 Aug 2023 14:04:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354139AbjHQSE1 (ORCPT ); Thu, 17 Aug 2023 14:04:27 -0400 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AFE519A1; Thu, 17 Aug 2023 11:04:26 -0700 (PDT) Received: by mail-ed1-x52d.google.com with SMTP id 4fb4d7f45d1cf-52889bc61b6so95376a12.0; Thu, 17 Aug 2023 11:04:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692295465; x=1692900265; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=UxvCrw/069KltoPZkQIYth4BBOJdgJ8sNaI0SyyMDnM=; b=LlNp+0S95wa2DHWIhxZumDFmianXTCRMJjOitpuiuyKYp0AAZb4zHQT8ZXqFwgTr3M TAcGjKr7A/7LbmvxCY8pqvLWzcywa017/uoZ3dhPRVfKQBpxe2Rx4JQVzmQMUv/lluNQ 8t/PjmueudVyVsA4+vkTHFn9Wg1EnP9GTqy6bCt1AIDvGSTkLDX8uzmkFWqf6cdFJtPf KRm02KV0hef2hr/xa3gZ+j3wmJGjbj9dQaAlasQmWzrxH3IJ5JCWJI2/kGrxl9CivxpV AvKdmfKLPT+4igVvanygECsKarn9iz35Qwe/ar6m/vTrTtCGRNAG2OY42t5fatoyv1PR bXLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692295465; x=1692900265; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UxvCrw/069KltoPZkQIYth4BBOJdgJ8sNaI0SyyMDnM=; b=JhsFPpmYOuj4rYlE4N7b7MtGbrs0UOyyoUx/1bTrk8QwjNM8sQQvhEKDUnCtLIFRjP dY6N+XWk4gTdEIs5/N2eUTHwlH/MIVHfJPpAhDoeuZaZxS4Gcb8IKUavge67hSu9oRW6 d1FuIXUx9mEXgnHnUUHidq316zlLklnxKsZ3fJq821CGg7hU7XQBSOhJXBeCaoHCRQ+s xYknktqvERx6kD+ctmMhGQMFRh//7qX4SR5IKGDWkg6HModTy0OjDgAYKUUX5JvOHT+e /+Ngdx+JgorUFfFBHh8Wg+phL47o6IyrFAMVabF1ZKz4hDXWWJAhGYBzSWWs8YjtCCwL nRmg== X-Gm-Message-State: AOJu0YzyqJMlqryaVNhyuAu+NLcsWYOFRdj4Scdnyu72wBTPT9C4MKKr vAk5H1TEcRxq6bAYA/GiHSk= X-Google-Smtp-Source: AGHT+IG0Woy+0POxg4sPHGGIdfrc456TaHlsCCBtEoEEY8ldQE+9Fdkgkm7V8IaZV79Aen7WtSv5Lw== X-Received: by 2002:a17:906:8a72:b0:99c:6692:7f76 with SMTP id hy18-20020a1709068a7200b0099c66927f76mr153247ejc.16.1692295464443; Thu, 17 Aug 2023 11:04:24 -0700 (PDT) Received: from [192.168.8.100] ([148.252.129.159]) by smtp.gmail.com with ESMTPSA id h6-20020a1709063c0600b0099b921de301sm27628ejg.159.2023.08.17.11.04.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 17 Aug 2023 11:04:23 -0700 (PDT) Message-ID: <7889b4f8-78d9-9a0a-e2cc-aae4ed8a80fd@gmail.com> Date: Thu, 17 Aug 2023 19:00:35 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH v2 00/11] Device Memory TCP Content-Language: en-US To: David Ahern , Mina Almasry , netdev@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , Willem de Bruijn , Sumit Semwal , =?UTF-8?Q?Christian_K=C3=B6nig?= , Jason Gunthorpe , Hari Ramakrishnan , Dan Williams , Andy Lutomirski , stephen@networkplumber.org, sdf@google.com, David Wei References: <20230810015751.3297321-1-almasrymina@google.com> <7dc4427f-ee99-e401-9ff8-d554999e60ca@kernel.org> From: Pavel Begunkov In-Reply-To: <7dc4427f-ee99-e401-9ff8-d554999e60ca@kernel.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org On 8/14/23 02:12, David Ahern wrote: > On 8/9/23 7:57 PM, Mina Almasry wrote: >> Changes in RFC v2: >> ------------------ ... >> ** Test Setup >> >> Kernel: net-next with this RFC and memory provider API cherry-picked >> locally. >> >> Hardware: Google Cloud A3 VMs. >> >> NIC: GVE with header split & RSS & flow steering support. > > This set seems to depend on Jakub's memory provider patches and a netdev > driver change which is not included. For the testing mentioned here, you > must have a tree + branch with all of the patches. Is it publicly available? > > It would be interesting to see how well (easy) this integrates with > io_uring. Besides avoiding all of the syscalls for receiving the iov and > releasing the buffers back to the pool, io_uring also brings in the > ability to seed a page_pool with registered buffers which provides a > means to get simpler Rx ZC for host memory. The patchset sounds pretty interesting. I've been working with David Wei (CC'ing) on io_uring zc rx (prototype polishing stage) all that is old similar approaches based on allocating an rx queue. It targets host memory and device memory as an extra feature, uapi is different, lifetimes are managed/bound to io_uring. Completions/buffers are returned to user via a separate queue instead of cmsg, and pushed back granularly to the kernel via another queue. I'll leave it to David to elaborate It sounds like we have space for collaboration here, if not merging then reusing internals as much as we can, but we'd need to look into the details deeper. > Overall I like the intent and possibilities for extensions, but a lot of > details are missing - perhaps some are answered by seeing an end-to-end > implementation. -- Pavel Begunkov From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B35D0C678DC for ; Fri, 18 Aug 2023 10:15:05 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9B57A10E080; Fri, 18 Aug 2023 10:15:04 +0000 (UTC) Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by gabe.freedesktop.org (Postfix) with ESMTPS id 605A210E545 for ; Thu, 17 Aug 2023 18:04:26 +0000 (UTC) Received: by mail-ed1-x533.google.com with SMTP id 4fb4d7f45d1cf-52256241c50so68747a12.3 for ; Thu, 17 Aug 2023 11:04:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692295465; x=1692900265; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=UxvCrw/069KltoPZkQIYth4BBOJdgJ8sNaI0SyyMDnM=; b=LlNp+0S95wa2DHWIhxZumDFmianXTCRMJjOitpuiuyKYp0AAZb4zHQT8ZXqFwgTr3M TAcGjKr7A/7LbmvxCY8pqvLWzcywa017/uoZ3dhPRVfKQBpxe2Rx4JQVzmQMUv/lluNQ 8t/PjmueudVyVsA4+vkTHFn9Wg1EnP9GTqy6bCt1AIDvGSTkLDX8uzmkFWqf6cdFJtPf KRm02KV0hef2hr/xa3gZ+j3wmJGjbj9dQaAlasQmWzrxH3IJ5JCWJI2/kGrxl9CivxpV AvKdmfKLPT+4igVvanygECsKarn9iz35Qwe/ar6m/vTrTtCGRNAG2OY42t5fatoyv1PR bXLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692295465; x=1692900265; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UxvCrw/069KltoPZkQIYth4BBOJdgJ8sNaI0SyyMDnM=; b=TN9J2OQyEIh5qUf3JjR+bezpzgVtFPWymIEU7YudtYrKA4d0Qsa+swjvWdIQG8Ic0s kZGojJC00cp1LJe+dYzvobcNLkt1WjK9/ZmVKVdwh8c87i8XsmsXRQG0id+PFSa+/FvK WFNWEiml4QE7/wC08LBxqvSROF8a57DV6icgwX53/gCXntg68wPmuDwJ4W+xLcYZXu4u 0flMy/PQc3QeZf2BNry+SWHAqPFoN7ub1hyEaqAfdjb9ZIZ1uG4SHX86lZfoWZ/WMbob BXT9rjnyYneveG8wW12NRAaW9LEQFlCrV/uMt1bte9OQBnvAmXqoaU+Mij1KN7/4H5Zm x3aA== X-Gm-Message-State: AOJu0YzTkmc6I5hpdzPTCYOYqGxdb+JyfFfqIGxPoEzWm5ZdWX2MzPjC GJJIc42eSyqDHeERqsPJ4fw= X-Google-Smtp-Source: AGHT+IG0Woy+0POxg4sPHGGIdfrc456TaHlsCCBtEoEEY8ldQE+9Fdkgkm7V8IaZV79Aen7WtSv5Lw== X-Received: by 2002:a17:906:8a72:b0:99c:6692:7f76 with SMTP id hy18-20020a1709068a7200b0099c66927f76mr153247ejc.16.1692295464443; Thu, 17 Aug 2023 11:04:24 -0700 (PDT) Received: from [192.168.8.100] ([148.252.129.159]) by smtp.gmail.com with ESMTPSA id h6-20020a1709063c0600b0099b921de301sm27628ejg.159.2023.08.17.11.04.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 17 Aug 2023 11:04:23 -0700 (PDT) Message-ID: <7889b4f8-78d9-9a0a-e2cc-aae4ed8a80fd@gmail.com> Date: Thu, 17 Aug 2023 19:00:35 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH v2 00/11] Device Memory TCP Content-Language: en-US To: David Ahern , Mina Almasry , netdev@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org References: <20230810015751.3297321-1-almasrymina@google.com> <7dc4427f-ee99-e401-9ff8-d554999e60ca@kernel.org> From: Pavel Begunkov In-Reply-To: <7dc4427f-ee99-e401-9ff8-d554999e60ca@kernel.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Fri, 18 Aug 2023 10:15:02 +0000 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Willem de Bruijn , Jesper Dangaard Brouer , Arnd Bergmann , =?UTF-8?Q?Christian_K=C3=B6nig?= , Ilias Apalodimas , David Wei , Sumit Semwal , stephen@networkplumber.org, Jason Gunthorpe , Eric Dumazet , sdf@google.com, Andy Lutomirski , Hari Ramakrishnan , Jakub Kicinski , Dan Williams , Paolo Abeni , "David S. Miller" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On 8/14/23 02:12, David Ahern wrote: > On 8/9/23 7:57 PM, Mina Almasry wrote: >> Changes in RFC v2: >> ------------------ ... >> ** Test Setup >> >> Kernel: net-next with this RFC and memory provider API cherry-picked >> locally. >> >> Hardware: Google Cloud A3 VMs. >> >> NIC: GVE with header split & RSS & flow steering support. > > This set seems to depend on Jakub's memory provider patches and a netdev > driver change which is not included. For the testing mentioned here, you > must have a tree + branch with all of the patches. Is it publicly available? > > It would be interesting to see how well (easy) this integrates with > io_uring. Besides avoiding all of the syscalls for receiving the iov and > releasing the buffers back to the pool, io_uring also brings in the > ability to seed a page_pool with registered buffers which provides a > means to get simpler Rx ZC for host memory. The patchset sounds pretty interesting. I've been working with David Wei (CC'ing) on io_uring zc rx (prototype polishing stage) all that is old similar approaches based on allocating an rx queue. It targets host memory and device memory as an extra feature, uapi is different, lifetimes are managed/bound to io_uring. Completions/buffers are returned to user via a separate queue instead of cmsg, and pushed back granularly to the kernel via another queue. I'll leave it to David to elaborate It sounds like we have space for collaboration here, if not merging then reusing internals as much as we can, but we'd need to look into the details deeper. > Overall I like the intent and possibilities for extensions, but a lot of > details are missing - perhaps some are answered by seeing an end-to-end > implementation. -- Pavel Begunkov