From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99C6BC3A59B for ; Mon, 2 Sep 2019 09:08:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7B5E021874 for ; Mon, 2 Sep 2019 09:08:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730928AbfIBJI3 (ORCPT ); Mon, 2 Sep 2019 05:08:29 -0400 Received: from mx1.redhat.com ([209.132.183.28]:16380 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729804AbfIBJI1 (ORCPT ); Mon, 2 Sep 2019 05:08:27 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 739483084212; Mon, 2 Sep 2019 09:08:26 +0000 (UTC) Received: from carbon (ovpn-200-29.brq.redhat.com [10.40.200.29]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5244819C5B; Mon, 2 Sep 2019 09:08:19 +0000 (UTC) Date: Mon, 2 Sep 2019 11:08:18 +0200 From: Jesper Dangaard Brouer To: Kal Cutter Conley Cc: brouer@redhat.com, "Karlsson, Magnus" , =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= , Maxim Mikityanskiy , Jakub Kicinski , Toke =?UTF-8?B?SMO4aWxhbmQtSsO4cmdlbnNl?= =?UTF-8?B?bg==?= , Andy Gospodarek , "xdp-newbies@vger.kernel.org" , Saeed Mahameed , Tariq Toukan , "netdev@vger.kernel.org" Subject: Re: net/mlx5e: bind() always returns EINVAL with XDP_ZEROCOPY Message-ID: <20190902110818.2f6a8894@carbon> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.40]); Mon, 02 Sep 2019 09:08:26 +0000 (UTC) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Sun, 1 Sep 2019 18:47:15 +0200 Kal Cutter Conley wrote: > Hi, > I figured out the problem. Let me document the issue here for others > and hopefully start a discussion. > > The mlx5 driver uses special queue ids for ZC. If N is the number of > configured queues, then for XDP_ZEROCOPY the queue ids start at N. So > queue ids [0..N) can only be used with XDP_COPY and queue ids [N..2N) > can only be used with XDP_ZEROCOPY. Thanks for the followup and explanation on how mlx5 AF_XDP queue implementation is different from other vendors. > sudo ethtool -L eth0 combined 16 > sudo samples/bpf/xdpsock -r -i eth0 -c -q 0 # OK > sudo samples/bpf/xdpsock -r -i eth0 -z -q 0 # ERROR > sudo samples/bpf/xdpsock -r -i eth0 -c -q 16 # ERROR > sudo samples/bpf/xdpsock -r -i eth0 -z -q 16 # OK > > Why was this done? To use zerocopy if available and fallback on copy > mode normally you would set sxdp_flags=0. However, here this is no > longer possible. To support this driver, you have to first try binding > with XDP_ZEROCOPY and the special queue id, then if that fails, you > have to try binding again with a normal queue id. Peculiarities like > this complicate the XDP user api. Maybe someone can explain the > benefits? Thanks for complaining, it is actually valuable. It really illustrate the kernel need to improve in this area, which is what our talk[1] at LPC2019 (Sep 10) is about. Title: "Making Networking Queues a First Class Citizen in the Kernel" [1] https://linuxplumbersconf.org/event/4/contributions/462/ As you can see, several vendors are actually involved. Kudos to Magnus for taking initiative here! It's unfortunately not solved "tomorrow", as first we have to agree this is needed (facility to register queues), then agree on API and get commitment from vendors, as this requires drivers changes. There is a long road ahead, but I think it will be worthwhile in the end, as effective use of dedicated hardware queues (both RX and TX) is key to performance. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer > On Wed, Aug 7, 2019 at 2:49 PM Kal Cutter Conley wrote: > > > > Hello, > > I am testing the mlx5e driver with AF_XDP. When I specify > > XDP_ZEROCOPY, bind() always returns EINVAL. I observe the same problem > > with the xdpsock sample: > > > > sudo samples/bpf/xdpsock -r -i dcb1-port1 -z > > samples/bpf/xdpsock_user.c:xsk_configure_socket:322: errno: > > 22/"Invalid argument" > > > > Without XDP_ZEROCOPY, everything works as expected. Is this a known > > issue/limitation? I expected this to be supported looking at the > > code/commit history. > > > > Thanks, > > Kal