From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18D98EB64DD for ; Tue, 18 Jul 2023 17:32:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233129AbjGRRci (ORCPT ); Tue, 18 Jul 2023 13:32:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231719AbjGRRch (ORCPT ); Tue, 18 Jul 2023 13:32:37 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02C40E4C; Tue, 18 Jul 2023 10:32:37 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8CB196169B; Tue, 18 Jul 2023 17:32:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B41EC433C8; Tue, 18 Jul 2023 17:32:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1689701556; bh=5sr+yjwLVU6f+lqMOfWoGq5NUJvs6qsaMrWv5HRb0zE=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=idOxW7Wfi/M22qWhbX0ZuMGzGLMpVBG26F9PGqW02cBDx+a8kPduaXPCNyv2gJLAO YIOzIg2DhjzuNdLVitTUITjJC45H8l1WtTyq8YlyoDV2wLe9w+3JKxi6+JhB5+ZsKZ 9wq43VR+kOlkL3WhN7OM8HWIv3hpicw2TuPpxSe7Nbwo9qllmCwRpgp+ILSXD1jsQ3 LuBZYWJJMChc1JlbmkOyU1EVCZhj+II1qEFbMJMMf/l+EmZIulzA5dxUNwwo+q+91y 93fOXY6mg7tjd6adh1mHOcqT/q72z1++d3bJbyeWA/p6lEm9COrpwzKTq+VKWBFhOg DU4JRoJhnIYPQ== Date: Tue, 18 Jul 2023 10:32:34 -0700 From: Jakub Kicinski To: Andy Lutomirski Cc: Mina Almasry , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, netdev@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, Sumit Semwal , Christian =?UTF-8?B?S8O2bmln?= , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , jgg@ziepe.ca Subject: Re: [RFC PATCH 00/10] Device Memory TCP Message-ID: <20230718103234.711d7e4f@kernel.org> In-Reply-To: <12393cd2-4b09-4956-fff0-93ef3929ee37@kernel.org> References: <20230710223304.1174642-1-almasrymina@google.com> <12393cd2-4b09-4956-fff0-93ef3929ee37@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-arch@vger.kernel.org On Sun, 16 Jul 2023 19:41:28 -0700 Andy Lutomirski wrote: > I'm wondering if a more capable if somewhat higher latency model could > work where the NIC stores received packets in its own device memory. > Then userspace (or the kernel or a driver or whatever) could initiate a > separate DMA from the NIC to the final target *after* reading the > headers. Can the hardware support this? No, no, that's impossible. SW response times are in 100s of usec (at best) which at 200Gbps already means megabytes of data _per-queue_. Way more than the amount of buffer NICs will have. The Rx application can bind to a IP addr + Port and then install a one-sided-3-tuple (dst IP+proto+port) rule in the HW. Worst case a full 5-tuple per flow. Most NICs support OvS offloads with 100s of thousands of flows. The steering should be bread and butter. It does require splitting flows into separate data and control channels, but it's the right trade-off - complexity should be on the SW side.