From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F09A1FB5E for ; Wed, 19 Jul 2023 17:57:17 +0000 (UTC) Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E54D91FF1 for ; Wed, 19 Jul 2023 10:57:14 -0700 (PDT) Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-668709767b1so5383938b3a.2 for ; Wed, 19 Jul 2023 10:57:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20221208.gappssmtp.com; s=20221208; t=1689789434; x=1690394234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=4FVEMGAeC4A4acfilWsGklHz88eR5z5EWRCCV9Kst0I=; b=0Q1tzUOb9vYJXHmPeDryrYB5A4pEESrmEqXYj9Hg+90Z5+QCPv2ecrkofZgQVHrh2Q 2SmV6ESktQAtsjCYVJZhJojTRHsrQLvYLqNH0EoTgXcN4+EyUGH6ZSXKFYb02Y0hiirh icie1/r4QjaO+Xr/8VrDJJ+t8iIcRUo2snH46DM7aXAbB12rMZS46rJ2h5Z2q3coLAKN GMT7GgUwcUUW4SpZr+6zG8X6fnKROWImTBv7u2v/1CdFXwNuMSzJvhzr6uNXEVsoU8p0 jr3ynj+/29rinW5TGGhTFa+uF7yLH222PS2w2a9Iu7qAlX2Am1gjapUEui6PcMe/HT1i pwCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689789434; x=1690394234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4FVEMGAeC4A4acfilWsGklHz88eR5z5EWRCCV9Kst0I=; b=fgTISInie+fgwUvpXPb+PoH0GoxzQ5z9LlqVoywN6DIX3HjgDYKSX7EiUBjizDsXdJ fhWkv52Bw+ASs7Nto/yNfUFcWu6OS2ZmiZrsm9A3ta6904fkhvD5nx2CnUDlBSJZnRWs d0mtsRKYAM+pLBNAbVwIuNy2KdgDTU50LEWQG7ni9sAp2dT/SAvuMdUZnJdzcSRUxdcp NJ8qWVhAClybkZazYmUsHqJE+5v0w77rzNptta+O+1w1fow6jnzpo5UM810ZBX2pWEfy 608BEeHOpH2d7JdWZs23iY3xQXWMIQeHmfEEai9XHIWaQmpcFqnOko8IMvGITkf59Zox FFng== X-Gm-Message-State: ABy/qLZFSF3cdzgX0sRxg7mCoM0EnNOlDllc2DHiHMEjiDOEhrZ0/n8r 3841k6hEnA2t4vGWwGQo4IEutS38hODEXm9bOa7Wgd5F X-Google-Smtp-Source: APBJJlEsVXT/riHHb217/IRnC4FkTdrL+WEIRgGEaqW1QkKPQqx7J9SZ2+4YPKCL/Z0Wz9JzjQeyoQ== X-Received: by 2002:a17:90a:d598:b0:264:97a:2ba6 with SMTP id v24-20020a17090ad59800b00264097a2ba6mr4560333pju.7.1689789434001; Wed, 19 Jul 2023 10:57:14 -0700 (PDT) Received: from hermes.local (204-195-127-207.wavecable.com. [204.195.127.207]) by smtp.gmail.com with ESMTPSA id nw17-20020a17090b255100b00263f41a655esm1415304pjb.43.2023.07.19.10.57.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 10:57:13 -0700 (PDT) Date: Wed, 19 Jul 2023 10:57:11 -0700 From: Stephen Hemminger To: Mina Almasry Cc: Jakub Kicinski , David Ahern , Jason Gunthorpe , Andy Lutomirski , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, netdev@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, Sumit Semwal , Christian =?UTF-8?B?S8O2bmln?= , "David S. Miller" , Eric Dumazet , Paolo Abeni , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , Willem de Bruijn , Shuah Khan Subject: Re: [RFC PATCH 00/10] Device Memory TCP Message-ID: <20230719105711.448f8cad@hermes.local> In-Reply-To: References: <20230710223304.1174642-1-almasrymina@google.com> <12393cd2-4b09-4956-fff0-93ef3929ee37@kernel.org> <20230718111508.6f0b9a83@kernel.org> <35f3ec37-11fe-19c8-9d6f-ae5a789843cb@kernel.org> <20230718112940.2c126677@kernel.org> <20230718154503.0421b4cd@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net On Wed, 19 Jul 2023 08:10:58 -0700 Mina Almasry wrote: > On Tue, Jul 18, 2023 at 3:45=E2=80=AFPM Jakub Kicinski = wrote: > > > > On Tue, 18 Jul 2023 16:35:17 -0600 David Ahern wrote: =20 > > > I do not see how 1 RSS context (or more specifically a h/w Rx queue) = can > > > be used properly with memory from different processes (or dma-buf > > > references). =20 >=20 > Right, my experience with dma-bufs from GPUs are that they're > allocated from the userspace and owned by the process that allocated > the backing GPU memory and generated the dma-buf from it. I.e., we're > limited to 1 dma-buf per RX queue. If we enable binding multiple > dma-bufs to the same RX queue, we have a problem, because AFAIU the > NIC can't decide which dma-buf to put the packet into (it hasn't > parsed the packet's destination yet). >=20 > > > When the process dies, that memory needs to be flushed from > > > the H/W queues. Queues with interlaced submissions make that more > > > complicated. =20 > > =20 >=20 > When the process dies, do we really want to flush the memory from the > hardware queues? The drivers I looked at don't seem to have a function > to flush the rx queues alone, they usually do an entire driver reset > to achieve that. Not sure if that's just convenience or there is some > technical limitation there. Do we really want to trigger a driver > reset at the event a userspace process crashes? Naive idea. Would it be possible for process to use mmap() on the GPU memory and then do zero copy TCP receive some how? Or is this what is being proposed.