From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2EA9C43387 for ; Fri, 18 Jan 2019 03:59:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A0AC72086D for ; Fri, 18 Jan 2019 03:59:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727060AbfARD7f (ORCPT ); Thu, 17 Jan 2019 22:59:35 -0500 Received: from mx1.redhat.com ([209.132.183.28]:35976 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726834AbfARD7f (ORCPT ); Thu, 17 Jan 2019 22:59:35 -0500 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 499357D0D3; Fri, 18 Jan 2019 03:59:35 +0000 (UTC) Received: from redhat.com (ovpn-120-92.rdu2.redhat.com [10.10.120.92]) by smtp.corp.redhat.com (Postfix) with SMTP id 07D5B101963A; Fri, 18 Jan 2019 03:59:33 +0000 (UTC) Date: Thu, 17 Jan 2019 22:59:32 -0500 From: "Michael S. Tsirkin" To: Willem de Bruijn Cc: Network Development , David Miller , Jason Wang , Willem de Bruijn , Mark Hlady Subject: Re: [PATCH net-next] virtio-net: per-queue RPS config Message-ID: <20190117225900-mutt-send-email-mst@kernel.org> References: <20190118010853.47325-1-willemdebruijn.kernel@gmail.com> <20190117204248-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Fri, 18 Jan 2019 03:59:35 +0000 (UTC) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Thu, Jan 17, 2019 at 10:37:26PM -0500, Willem de Bruijn wrote: > On Thu, Jan 17, 2019 at 8:43 PM Michael S. Tsirkin wrote: > > > > On Thu, Jan 17, 2019 at 08:08:53PM -0500, Willem de Bruijn wrote: > > > From: Willem de Bruijn > > > > > > On multiqueue network devices, RPS maps are configured independently > > > for each receive queue through /sys/class/net/$DEV/queues/rx-*. > > > > > > On virtio-net currently all packets use the map from rx-0, because the > > > real rx queue is not known at time of map lookup by get_rps_cpu. > > > > > > Call skb_record_rx_queue in the driver rx path to make lookup work. > > > > > > Recording the receive queue has ramifications beyond RPS, such as in > > > sticky load balancing decisions for sockets (skb_tx_hash) and XPS. > > > > > > Reported-by: Mark Hlady > > > Signed-off-by: Willem de Bruijn > > > > And any examples how to see the benefit of this? > > When there are fewer queues than cpus and rps is used to spread load > across all cpus, it can be preferable to setup disjoint sets, such > that each cpu handling an rxq interrupt spreads to an exclusive set of > neighbors instead of having all interrupt handling cores contend on > all other cores' softnet_data. > > More subtly, even if the policy is to spread uniformly, it can be > preferable to set the RPS map to all cores except the core that > handled the interrupt, as it already had to do some work in the > initial receive path. > > It is also simply expected behavior for network devices to be able to > configure rxq rps maps individually, so the current silent fallback to > rx0 is confusing, especially since rx-1/rps_cpus, .. rx-n/rps_cpus > files do exist and can be configured. OK I think I got it. Acked-by: Michael S. Tsirkin