From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 836D8C433FE for ; Tue, 14 Dec 2021 16:07:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235515AbhLNQHB (ORCPT ); Tue, 14 Dec 2021 11:07:01 -0500 Received: from mga02.intel.com ([134.134.136.20]:23548 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231515AbhLNQHB (ORCPT ); Tue, 14 Dec 2021 11:07:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1639498021; x=1671034021; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=z/iMOPZEzcSCY4ZnO9/RwV+Zjt0Rc/4MCkd9NNblJxY=; b=duAVjdlsUAzQnwBKqVIWZlklhfo2hI4gMomm0I0TGjs+f3lYtSrJbFFe x04xqO2NQA6hLM6aA6F1A3rjzKX3cGH9ppwb19DorpUV2gF2gOdGcoE74 r3nELYFSYqROVb6sMAKI9YiZtMgAm22aeH0ROu2RYtjoOxicDJNBZ1SRY +hP8A5BQDwzUpJ/H23/MYfYnUCgutrLt5Quy4MAbPu6s4ndsoN93CvhZk JSdbLzsEkoyhBwFk9+5UaUfdi3tm8PVGMxdGzu3Bln1s2gv6u9LZhBqAY C9ynk1Wmy5CT2KvIZd3rvHpVGu8UbmguO1FqNI273S9JkPeag0RgbMTk3 A==; X-IronPort-AV: E=McAfee;i="6200,9189,10197"; a="226286479" X-IronPort-AV: E=Sophos;i="5.88,205,1635231600"; d="scan'208";a="226286479" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Dec 2021 08:05:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,205,1635231600"; d="scan'208";a="505408768" Received: from boxer.igk.intel.com (HELO boxer) ([10.102.20.173]) by orsmga007.jf.intel.com with ESMTP; 14 Dec 2021 08:05:44 -0800 Date: Tue, 14 Dec 2021 17:05:44 +0100 From: Maciej Fijalkowski To: Jesper Dangaard Brouer Cc: "Ong, Boon Leong" , "Karlsson, Magnus" , brouer@redhat.com, =?iso-8859-1?Q?Bj=F6rn_T=F6pel?= , "Desouza, Ederson" , Xdp , Joao Pedro Barros Silva , Diogo Alexandre Da Silva Lima Subject: Re: AF_XDP not transmitting frames immediately Message-ID: References: <65b16496-1b4e-55d9-ea17-610cfd923792@redhat.com> <7a888c6c-1f54-6e82-a4d0-794465cf17c3@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7a888c6c-1f54-6e82-a4d0-794465cf17c3@redhat.com> Precedence: bulk List-ID: X-Mailing-List: xdp-newbies@vger.kernel.org On Tue, Dec 14, 2021 at 04:42:18PM +0100, Jesper Dangaard Brouer wrote: > > On 14/12/2021 15.57, Ong, Boon Leong wrote: > > > I suspected Boon Leong (cc) would have this hardware. > > > > Unfortunately, my current setup in lab does not have I225 hooked-up > > and I am working remotely due to control access to intel facility. > > Perhaps, Ederson may have ready system to test? > > > > For ZC mode, the igc driver (also true to stmmac) depends on the XSK wakeup > > to trigger the NAPI poll (igc_poll) to first clean-up Tx ring and eventually call > > igc_xdp_xmit_zc() to start submitting Tx frame into DMA engine. We have > > used busy-poll to ensure in smaller Tx frame latency/jitter. > > > > There was another issue in stmmac that was patched [1] recently to ensure > > the driver does not perform MAC reset whenever XDP program is added > > so that between XDP socket creation, the Tx transmit does not take extra > > 2-3s due to link down/up. Jesper, are you seeing something similar in your > > app? > > Yes, and it is quite annoying. > > In my setup, if I AF_XDP transmit packets too early they are simply lost... > that confused me a bit. > > I wanted to ask AF_XDP maintainers: > - What is the best way to know when AF_XDP is ready to Tx packets? > > E.g. what API should I call, e.g. that blocks, until XSK socket is ready to > transmit on? Not a maintainer, but anyway. >From a driver POV xsk_tx_peek_desc() (or batching variant) is used to make sure that user space produced entries in the XSK Tx ring so that driver can consume it and place it onto HW descriptors. >From the top of my head I'm not aware of any blocking calls, maybe you could spin on xsk_tx_peek_desc. For igc maybe it would be worth returning some status from igc_xdp_xmit_zc(). Like, imagine that you consumed all the budget but there are still descriptors in the XSK Tx ring. You'd like to signal to NAPI that there is still work to be done. > > > > If yes, then it is likely because of the implementation of igc driver in mainline > > that is doing igc_down(), a little bit too aggressive in reseting MAC completely. > > > > It would be good to fix igc too, like[1]. > BUT afaik it will only make the window smaller when XSK is not ready for TX > packets. > > > > [1]https://patchwork.kernel.org/project/netdevbpf/patch/20211111143949.2806049-1-boon.leong.ong@intel.com/ > > > > Thanks for the link > --Jesper >