From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DAC4AC3DA4A for ; Tue, 20 Aug 2024 08:33:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 85948608A9; Tue, 20 Aug 2024 08:33:10 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id nA-YHokRo8cB; Tue, 20 Aug 2024 08:33:09 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.34; helo=ash.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 9E60C60626 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1724142789; bh=XNQ9tyNfl4ljX/BPGLUOjjoGD0Ou+TIYM+qPExssLvc=; h=Date:From:To:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=7diapXBg4jn6kBcdMhTEBN/Rd3L+unioEh0DeblZd1ZOJI2UVcIxk/b3cYCUaRzNp TIbQT2UIgAry6dHuCbtDgD0CmjryQYVCJ6WXpIWugWY+emex0rgy/KrfFoozQdjsV9 nXwFzaDNUzEnIws/WsKNMmRX/yEgBClvgMhJrnwqvwtiwzexuGF3J6xnTPtzKnOE5J jGaiMemxXMJpp1fB/1olVO116MUyaNkrqqw9Ux2xd8y71XjySvRVAEo4+JyTddWeqX Bol0nlO3U+/Ef+N5y9BCL+evCwBgmZa+S6Iw85JUDRHS2NOM7m3xu6VTIiAE9bRtkX uAW0dSbbpSQ/w== Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id 9E60C60626; Tue, 20 Aug 2024 08:33:09 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 3A5F41BF341 for ; Tue, 20 Aug 2024 08:33:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 26E6F408E5 for ; Tue, 20 Aug 2024 08:33:08 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id eZ7X5BK5VNhJ for ; Tue, 20 Aug 2024 08:33:07 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::636; helo=mail-ej1-x636.google.com; envelope-from=jdamato@fastly.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 8F980408C8 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 8F980408C8 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by smtp2.osuosl.org (Postfix) with ESMTPS id 8F980408C8 for ; Tue, 20 Aug 2024 08:33:06 +0000 (UTC) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a7a8caef11fso615442366b.0 for ; Tue, 20 Aug 2024 01:33:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724142784; x=1724747584; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XNQ9tyNfl4ljX/BPGLUOjjoGD0Ou+TIYM+qPExssLvc=; b=pRCKRPcoElji+lV8gVatO3/ChmEMO+ocRQc4gVGTF9Rc47MhREkTnsLmk/MrSkmcNw onlVSgqyQ7Olm0N7IxW55xUF1cEUzHcBFGrT8wT2NpJ+eGBRqL3p1DWW2mFLSKwWj/4w MBLXtb/3n859rHYQ0EUh8aQKLMwCYvRAjakW16Ylcm1KuZ190kYsamWAAInCHjyvIx0O ih0IFex+z8IP6EbtgyKss6HMyii9K7IHwoevsA/KuzKq6j+ybDRICPGplqU/B54sedSk Y0vpIU4wlsWVJYc2RJG1+B1jkgm3eMhwsq9o84j09kR89K3h0Zly2dmrLsc9nzjcKNA+ Vgbw== X-Forwarded-Encrypted: i=1; AJvYcCWSRw567YPpsxJhfoOl+CAWiTD+5m5IDZ4Y+jAhf0z+FEXqMMDHLmbalyCDdvGwOErcSAYn9thX0tdJKYww7Vz9Ax3GkAsGcSIDgVDBxBOksg== X-Gm-Message-State: AOJu0YzYU8TOzhOd6JAXAOnF+v3GqKiro6370yYs5sqAg8L05z+avRKj MDBw76u43Nl9vV+PpX30gQt92IWiy2krAb5vZU7a9g2YnqfRKNzpHxcUvKq7HuE= X-Google-Smtp-Source: AGHT+IHdCu1gJlayHwi4drLhhfQxyfMUzMC+JYbFPtj9dlU1JF6GxmGI/OuqVO5LTmVnsfb9hkmppA== X-Received: by 2002:a17:907:97cf:b0:a6f:59dc:4ece with SMTP id a640c23a62f3a-a83928a4023mr845774466b.2.1724142783295; Tue, 20 Aug 2024 01:33:03 -0700 (PDT) Received: from LQ3V64L9R2.home ([2a02:c7c:f016:fc00:3906:31c:255a:bf09]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83aeb6eb4dsm435559766b.35.2024.08.20.01.33.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Aug 2024 01:33:03 -0700 (PDT) Date: Tue, 20 Aug 2024 09:33:01 +0100 From: Joe Damato To: Shay Drori Message-ID: Mail-Followup-To: Joe Damato , Shay Drori , Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao , Thomas Gleixner References: <20240813171710.599d3f01@kernel.org> <20240814080915.005cb9ac@kernel.org> <701eb84c-8d26-4945-8af3-55a70e05b09c@nvidia.com> <20240814172046.7753a62c@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724142784; x=1724747584; darn=lists.osuosl.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=XNQ9tyNfl4ljX/BPGLUOjjoGD0Ou+TIYM+qPExssLvc=; b=bxXBID7KvVMhIuBzCscqf9DRbL1Ui2/4AYP8xfsSifOpcUW0rD57OiReDQ9aHMPH3n ERyVzuct82O0b5TcBxiCKfSad6QdhhKyKaLuBmQ9mtH0W5qLdzuuh2aoKVMxqRUDlAt3 WzUktNsl5OJEA/kB+hrHPsAuG22tRxbszctCw= X-Mailman-Original-Authentication-Results: smtp2.osuosl.org; dmarc=pass (p=reject dis=none) header.from=fastly.com X-Mailman-Original-Authentication-Results: smtp2.osuosl.org; dkim=pass (1024-bit key, unprotected) header.d=fastly.com header.i=@fastly.com header.a=rsa-sha256 header.s=google header.b=bxXBID7K Subject: Re: [Intel-wired-lan] [RFC net-next 0/6] Cleanup IRQ affinity checks in several drivers X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sebastian Andrzej Siewior , Eric Dumazet , Tony Nguyen , Ziwei Xiao , Jeroen de Borst , Leon Romanovsky , "open list:MELLANOX MLX4 core VPI driver" , Przemek Kitszel , "moderated list:INTEL ETHERNET DRIVERS" , Jakub Kicinski , Paolo Abeni , Lorenzo Bianconi , Jiri Pirko , Shailend Chand , Harshitha Ramamurthy , Thomas Gleixner , Willem de Bruijn , Yishai Hadas , Daniel Borkmann , netdev@vger.kernel.org, open list , Tariq Toukan , Praveen Kaligineedi , Saeed Mahameed , "David S. Miller" Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" On Tue, Aug 20, 2024 at 09:40:31AM +0300, Shay Drori wrote: > > > On 15/08/2024 13:22, Joe Damato wrote: > > External email: Use caution opening links or attachments > > > > > > On Wed, Aug 14, 2024 at 05:20:46PM -0700, Jakub Kicinski wrote: > > > On Wed, 14 Aug 2024 19:01:40 +0100 Joe Damato wrote: > > > > If it is, then the only option is to have the drivers pass in their > > > > IRQ affinity masks, as Stanislav suggested, to avoid adding that > > > > call to the hot path. > > > > > > > > If not, then the IRQ from napi_struct can be used and the affinity > > > > mask can be generated on every napi poll. i40e/gve/iavf would need > > > > calls to netif_napi_set_irq to set the IRQ mapping, which seems to > > > > be straightforward. > > > > > > It's a bit sad to have the generic solution blocked. > > > cpu_rmap_update() is exported. Maybe we can call it from our notifier? > > > rmap lives in struct net_device > > > > I agree on the sadness. I will take a look today. > > > > I guess if we were being really ambitious, we'd try to move ARFS > > stuff into the core (as RSS was moved into the core). > > > Sorry for the late reply. Maybe we can modify affinity notifier infra to > support more than a single notifier per IRQ. > @Thomas, do you know why only a single notifier per IRQ is supported? Sorry for the delayed response as well on my side; I've been in between lots of different kernel RFCs :) Jakub: the issue seems to be that the internals in lib/cpu_rmap.c are needed to call cpu_rmap_update. It's probably possible to expose them somehow so that a generic IRQ notifier could call cpu_rmap_update, as you mentioned, but some rewiring is going to be needed, I think. I had a couple ideas for rewiring stuff, but I haven't had time to context switch back on to this work as I've been busy with a few other things (the IRQ suspension stuff and another mlx5 thing I have yet to send upstream). I hope to take another look at it this week, but I welcome any suggestions from Shay/Thomas in the meantime. - Joe From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DB8C18A6B6 for ; Tue, 20 Aug 2024 08:33:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724142787; cv=none; b=a6SWCqiAJnbdH5zabNqO/lhxzx7FjnlyeQhK4s/VxPv0MuZrI5N5A3k8kbZf4LNXkFL8du5dF10lKGp+ExGF/or+RqkGwLx7u5fKiwFzX3T6HFwd1/ttksgcxbFqffUhPmDOrmw/tdxcQLKbATbt0hskuxq7pbUcfTCHPLzS0UA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724142787; c=relaxed/simple; bh=sSUYkM8HYmaJN7xqG+tfsAHvEYV5RAq6szVJelU+X+U=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=k5JRV1hap3wL3btXHOzAKEeC6t+Al7n02haYDVEeJFzB7KzesL3eNWsEoD6Ebth5ezTRoq074vSsBbbKSLYWFn2ZVMwS5nR7IjM1e8CElCnaJvk7d1KfQN8/ZkTlvcnZtP0Gxg3KylyyxcR8LcWpdLbegbRvpEicxPWldCS3DzM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=VG8ajf6B; arc=none smtp.client-ip=209.85.208.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="VG8ajf6B" Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-5bed83488b3so3850815a12.0 for ; Tue, 20 Aug 2024 01:33:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724142784; x=1724747584; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=XNQ9tyNfl4ljX/BPGLUOjjoGD0Ou+TIYM+qPExssLvc=; b=VG8ajf6BuTNmkSG0citgtyRIslAki+3nxKaQreOE/2Jt+XtQq867Hbt6IZQKqYcZIy Jah6u7gnrTVuJMlWiAlwcqZxMRsbpB8G9sGpIMHVAIwMhDn6Chc2+I4tonuf1y1MhNdL F8BFC3GhlV619NSvDG4KxYAMFyyT5dW/57ES4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724142784; x=1724747584; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XNQ9tyNfl4ljX/BPGLUOjjoGD0Ou+TIYM+qPExssLvc=; b=C7Oyk6lKSsM1YZnWi9W/0+n5GfvOkE3OZuHK8oFL9pa300LuCPtMVzWlY699lqfoXO Dyvk5uaex3WQ3fVjRxAfTDPjIgI1jiSwqD043+v0mr3X8LsKy2B887Jbh9/GPMhgbkmc mLKhtcVXKQgXlKYMQ7gW6ityVVyJ7yaYHCiH2+/EaI5nV35LDy6Jk9faFL4IcRA6JjGL QrMdVr9dYo5lzsZ8DgOOxv6wiwdJpi0hbBp335p1NW7/btKKh1UlK9O1/+2kDr5INoqS EODd92qHgBJwlX9LInN4f295lpYE80N/rLfjNNNCb9YBwn2zOwXnF+DziFIMC1cwPJdi VcTA== X-Forwarded-Encrypted: i=1; AJvYcCVbjzSfFlmkMsOmMJRNdjM0XxqDVpLroKug4TmUcP9gHDHoK0cFEsnlMhuTNg/zCRqB+tUX60uDgSDbKtg3GczJgtJjGm2BaSmgvg== X-Gm-Message-State: AOJu0YzHVUXFo0mTui0ar7RGGwzRI6vwdqE1u2SZ05bMHp1rQ7hJdvXm qjA+iIKnzU77BUaZUuQgyJnnbgdBeYBmWs8GAYNmveFZxKw+CxS2aWneTSAyE2c= X-Google-Smtp-Source: AGHT+IHdCu1gJlayHwi4drLhhfQxyfMUzMC+JYbFPtj9dlU1JF6GxmGI/OuqVO5LTmVnsfb9hkmppA== X-Received: by 2002:a17:907:97cf:b0:a6f:59dc:4ece with SMTP id a640c23a62f3a-a83928a4023mr845774466b.2.1724142783295; Tue, 20 Aug 2024 01:33:03 -0700 (PDT) Received: from LQ3V64L9R2.home ([2a02:c7c:f016:fc00:3906:31c:255a:bf09]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83aeb6eb4dsm435559766b.35.2024.08.20.01.33.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Aug 2024 01:33:03 -0700 (PDT) Date: Tue, 20 Aug 2024 09:33:01 +0100 From: Joe Damato To: Shay Drori Cc: Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao , Thomas Gleixner Subject: Re: [RFC net-next 0/6] Cleanup IRQ affinity checks in several drivers Message-ID: Mail-Followup-To: Joe Damato , Shay Drori , Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao , Thomas Gleixner References: <20240813171710.599d3f01@kernel.org> <20240814080915.005cb9ac@kernel.org> <701eb84c-8d26-4945-8af3-55a70e05b09c@nvidia.com> <20240814172046.7753a62c@kernel.org> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Tue, Aug 20, 2024 at 09:40:31AM +0300, Shay Drori wrote: > > > On 15/08/2024 13:22, Joe Damato wrote: > > External email: Use caution opening links or attachments > > > > > > On Wed, Aug 14, 2024 at 05:20:46PM -0700, Jakub Kicinski wrote: > > > On Wed, 14 Aug 2024 19:01:40 +0100 Joe Damato wrote: > > > > If it is, then the only option is to have the drivers pass in their > > > > IRQ affinity masks, as Stanislav suggested, to avoid adding that > > > > call to the hot path. > > > > > > > > If not, then the IRQ from napi_struct can be used and the affinity > > > > mask can be generated on every napi poll. i40e/gve/iavf would need > > > > calls to netif_napi_set_irq to set the IRQ mapping, which seems to > > > > be straightforward. > > > > > > It's a bit sad to have the generic solution blocked. > > > cpu_rmap_update() is exported. Maybe we can call it from our notifier? > > > rmap lives in struct net_device > > > > I agree on the sadness. I will take a look today. > > > > I guess if we were being really ambitious, we'd try to move ARFS > > stuff into the core (as RSS was moved into the core). > > > Sorry for the late reply. Maybe we can modify affinity notifier infra to > support more than a single notifier per IRQ. > @Thomas, do you know why only a single notifier per IRQ is supported? Sorry for the delayed response as well on my side; I've been in between lots of different kernel RFCs :) Jakub: the issue seems to be that the internals in lib/cpu_rmap.c are needed to call cpu_rmap_update. It's probably possible to expose them somehow so that a generic IRQ notifier could call cpu_rmap_update, as you mentioned, but some rewiring is going to be needed, I think. I had a couple ideas for rewiring stuff, but I haven't had time to context switch back on to this work as I've been busy with a few other things (the IRQ suspension stuff and another mlx5 thing I have yet to send upstream). I hope to take another look at it this week, but I welcome any suggestions from Shay/Thomas in the meantime. - Joe