From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: [net-next PATCH 0/3] Symmetric queue selection using XPS for Rx queues Date: Tue, 8 May 2018 10:07:46 -0700 Message-ID: <439f729f-8a1d-1d2b-b59a-3f13786aab77@gmail.com> References: <152418597668.5832.5150463027149101930.stgit@anamdev.jf.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: Eric Dumazet , Amritha Nambiar , netdev , David Miller , Alexander Duyck , "Samudrala, Sridhar" , Hannes Frederic Sowa To: Alexander Duyck , Tom Herbert Return-path: Received: from mail-pl0-f44.google.com ([209.85.160.44]:35361 "EHLO mail-pl0-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752123AbeEHRHs (ORCPT ); Tue, 8 May 2018 13:07:48 -0400 Received: by mail-pl0-f44.google.com with SMTP id i5-v6so2594793plt.2 for ; Tue, 08 May 2018 10:07:48 -0700 (PDT) In-Reply-To: Content-Language: en-US Sender: netdev-owner@vger.kernel.org List-ID: On 05/08/2018 09:02 AM, Alexander Duyck wrote: > On Tue, May 8, 2018 at 8:15 AM, Tom Herbert wrote: >> On Thu, Apr 19, 2018 at 7:41 PM, Eric Dumazet wrote: >>> On Thu, Apr 19, 2018 at 6:07 PM Amritha Nambiar >>> wrote: >>> >>>> This patch series implements support for Tx queue selection based on >>>> Rx queue map. This is done by configuring Rx queue map per Tx-queue >>>> using sysfs attribute. If the user configuration for Rx queues does >>>> not apply, then the Tx queue selection falls back to XPS using CPUs and >>>> finally to hashing. >>> >>>> XPS is refactored to support Tx queue selection based on either the >>>> CPU map or the Rx-queue map. The config option CONFIG_XPS needs to be >>>> enabled. By default no receive queues are configured for the Tx queue. >>> >>>> - /sys/class/net/eth0/queues/tx-*/xps_rxqs >>> >>>> This is to enable sending packets on the same Tx-Rx queue pair as this >>>> is useful for busy polling multi-threaded workloads where it is not >>>> possible to pin the threads to a CPU. This is a rework of Sridhar's >>>> patch for symmetric queueing via socket option: >>>> https://www.spinics.net/lists/netdev/msg453106.html >>> >> I suspect this is an artifact of flow director which I believe >> required queue pairs to be able to work (i.e. receive queue chose >> hardware is determined send queue). But that was only required because >> of hardware design, I don't see the rationale for introducing queue >> pairs in the software stack. There's no need to correlate the transmit >> path with receive path, no need to enforce a 1-1 mapping between RX >> and TX queues, and the OOO mitigations should be sufficient when TX >> queue changes for a flow. >> >> Tom > > If I am not mistaken I think there are benefits to doing this sort of > thing with polling as it keeps the Tx work locked into the same queue > pair that a given application is polling on. So as a result you can > keep the interrupts contained to the queue pair that is being busy > polled on and if the application cleans up the packets during the busy > poll it ends up being a net savings in terms of both latency and power > since the Tx clean-up happens sooner, and it happens on the queue that > is already busy polling instead of possibly triggering an interrupt on > another CPU. > > So for example in the case of routing and bridging workloads we > already had code that would take the Rx queue and associate it to a Tx > queue. One of the ideas behind doing this is to try and keep the CPU > overhead low by having a 1:1 mapping. In the case of this code we > allow for a little more flexibility in that you could have > many-to-many mappings but the general idea and common use case is the > same which is a 1:1 mapping. I thought we had everything in place to be able to have this already. Setting IRQ affinities and XPS is certainly something doable. This is why I wanted a proper documentation of yet another way to reach the same behavior.