From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8B6D1C3DA4A for ; Wed, 14 Aug 2024 15:19:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 30388402C4; Wed, 14 Aug 2024 15:19:55 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 2Fm1Z_hJ8kXe; Wed, 14 Aug 2024 15:19:54 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.166.34; helo=ash.osuosl.org; envelope-from=intel-wired-lan-bounces@osuosl.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 0BBF340097 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1723648794; bh=gTi7bYjYTGMEWGbyKlWoq2Paz7kIbeWFkJUp3rdie2E=; h=Date:From:To:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=uX9R8QBNbZTqR1tYzlHdCLIN7ttSFVMO+zpG1HwknVqQLPFhV4zJE+ZUJKzlUdv/G DMQSGaeBRILBPbEun4j6k8uDSlyO3rlU1iMZeaCgaTXBUZGyb3AfNrmfghCCBCQJLR JFqlw2VsmyKUhQl8i4sofPCRIw8z5/8xIQjEkCxGpktD0o1n2/lFKR2SXIV9vX+lCQ hvhgNF4TeRhctIVFxwdrr6rW9MXYhj+ZUPsZD+COEG9olJYDTTIjEQu92dIShnG82B Yo9YzQ7/0EMAfi5+3bN+F9kECxnsZVWWsUA4scMuYeEZZrPehkZMC+HUQ28sgB2IT3 ffHM7mCF8dndw== Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 0BBF340097; Wed, 14 Aug 2024 15:19:54 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 1C22E1BF588 for ; Wed, 14 Aug 2024 15:19:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 089FF40068 for ; Wed, 14 Aug 2024 15:19:53 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id lkmncyFFaZTF for ; Wed, 14 Aug 2024 15:19:52 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::635; helo=mail-ej1-x635.google.com; envelope-from=jdamato@fastly.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp4.osuosl.org 0F7D140097 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 0F7D140097 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by smtp4.osuosl.org (Postfix) with ESMTPS id 0F7D140097 for ; Wed, 14 Aug 2024 15:19:50 +0000 (UTC) Received: by mail-ej1-x635.google.com with SMTP id a640c23a62f3a-a7a8caef11fso1635666b.0 for ; Wed, 14 Aug 2024 08:19:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723648788; x=1724253588; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gTi7bYjYTGMEWGbyKlWoq2Paz7kIbeWFkJUp3rdie2E=; b=TlB5OPqxMjoBs0z6sUQn7npSzBfaEGMI6zGKOSaOLhTtgH0nzbth9ZZ6WamwGzGVOX DPq16EVdxSaLo/TYUl6BjPtkQE5JR4VLY7SNA+MFnzC3QXQQPfzilldBsoJmPVQ173qk bUXiQyboRzY6oTeE5zJRJPu7mmDdnJAfuz/DMjc7EJKp3ghe1T/h9aqlC+MSoJc8xbJ0 j+g9BbHS/ODfBJzxwO4hYVgyNmEp6oG+oDHcg+NctLpbH5hRLK9sC/yAUqoo7uxCRI4F JZ41Po99Dg2A0bBM1H7MiXZoGVS0fUMqqekPouKly0Y68RqXainclJPeXBQPmJilkdkE 8+wQ== X-Forwarded-Encrypted: i=1; AJvYcCXiWVJyZwk7koH2teTA+Pxhza0Uy03r/o15ArIse8DW50T74ONmNYsGb3lFsYG6yzK1nJD2HbqqDIsM6/nYuWmPYrwsARpo5tYUv7dcjjcCHg== X-Gm-Message-State: AOJu0Yzrn0IRcyNHAxmdA5lJta5sFh4hkKDuKzPUIlx6vAN2vSncz03z znCMOpsKx2v9ZsWzuWHQxGInrq1tsDznvfwOBLtEJq1KSPlPC+q6nnjYjbDjEyw= X-Google-Smtp-Source: AGHT+IFvDOsbuKJOT0XL/TM+iXA5EFj0BhRg1ts5WGvw7A2g2cgRk9L2pdTDRxwV8rDbcUIbgQvSeA== X-Received: by 2002:a17:907:d5a4:b0:a72:40b4:c845 with SMTP id a640c23a62f3a-a8367006615mr221312666b.51.1723648788188; Wed, 14 Aug 2024 08:19:48 -0700 (PDT) Received: from LQ3V64L9R2 ([80.208.222.2]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a80f414e127sm183820466b.150.2024.08.14.08.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 08:19:47 -0700 (PDT) Date: Wed, 14 Aug 2024 16:19:45 +0100 From: Joe Damato To: Jakub Kicinski Message-ID: Mail-Followup-To: Joe Damato , Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao References: <20240812145633.52911-1-jdamato@fastly.com> <20240813171710.599d3f01@kernel.org> <20240814080915.005cb9ac@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240814080915.005cb9ac@kernel.org> X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1723648788; x=1724253588; darn=lists.osuosl.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=gTi7bYjYTGMEWGbyKlWoq2Paz7kIbeWFkJUp3rdie2E=; b=sJSDWt7yN/2RgHfH3ItiAegdCsP2PyUyWif1uxlVtKsJHhv6UGubGkB60cRW+FknIr hh06hdsB55Jp4daKUN3qMdrhzHZXasF8qnlnsMsr7tt0jdeBz3SoFVBKR5Fe8KowKNLO JzP8++fLWIl8RPL55aIZOeyZtl/DJWH7OivhA= X-Mailman-Original-Authentication-Results: smtp4.osuosl.org; dmarc=pass (p=reject dis=none) header.from=fastly.com X-Mailman-Original-Authentication-Results: smtp4.osuosl.org; dkim=pass (1024-bit key, unprotected) header.d=fastly.com header.i=@fastly.com header.a=rsa-sha256 header.s=google header.b=sJSDWt7y Subject: Re: [Intel-wired-lan] [RFC net-next 0/6] Cleanup IRQ affinity checks in several drivers X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sebastian Andrzej Siewior , Eric Dumazet , Tony Nguyen , Ziwei Xiao , Jeroen de Borst , Leon Romanovsky , "open list:MELLANOX MLX4 core VPI driver" , Przemek Kitszel , "moderated list:INTEL ETHERNET DRIVERS" , Paolo Abeni , Lorenzo Bianconi , Jiri Pirko , Shailend Chand , Harshitha Ramamurthy , Willem de Bruijn , Yishai Hadas , Daniel Borkmann , netdev@vger.kernel.org, open list , Tariq Toukan , Praveen Kaligineedi , Saeed Mahameed , "David S. Miller" Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" On Wed, Aug 14, 2024 at 08:09:15AM -0700, Jakub Kicinski wrote: > On Wed, 14 Aug 2024 13:12:08 +0100 Joe Damato wrote: > > Actually... how about a slightly different approach, which caches > > the affinity mask in the core? > > I was gonna say :) > > > 0. Extend napi struct to have a struct cpumask * field > > > > 1. extend netif_napi_set_irq to: > > a. store the IRQ number in the napi struct (as you suggested) > > b. call irq_get_effective_affinity_mask to store the mask in the > > napi struct > > c. set up generic affinity_notify.notify and > > affinity_notify.release callbacks to update the in core mask > > when it changes > > This part I'm not an export on. > > > 2. add napi_affinity_no_change which now takes a napi_struct > > > > 3. cleanup all 5 drivers: > > a. add calls to netif_napi_set_irq for all 5 (I think no RTNL > > is needed, so I think this would be straight forward?) > > b. remove all affinity_mask caching code in 4 of 5 drivers > > c. update all 5 drivers to call napi_affinity_no_change in poll > > > > Then ... anyone who adds support for netif_napi_set_irq to their > > driver in the future gets automatic support in-core for > > caching/updating of the mask? And in the future netdev-genl could > > dump the mask since its in-core? > > > > I'll mess around with that locally to see how it looks, but let me > > know if that sounds like a better overall approach. I ended up going with the approach laid out above; moving the IRQ affinity mask updating code into the core (which adds that ability to gve/mlx4/mlx5... it seems mlx4/5 cached but didn't have notifiers setup to update the cached copy?) and adding calls to netif_napi_set_irq in i40e/iavf and deleting their custom notifier code. It's almost ready for rfcv2; I think this approach is probably better ? > Could we even handle this directly as part of __napi_poll(), > once the driver gives core all of the relevant pieces of information ? I had been thinking the same thing, too, but it seems like at least one driver (mlx5) counts the number of affinity changes to export as a stat, so moving all of this to core would break that. So, I may avoid attempting that for this series. I'm still messing around with this but will send an rfcv2 in a bit. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F2A12D7B8 for ; Wed, 14 Aug 2024 15:19:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.48 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723648791; cv=none; b=GeYarfq40X/efRnqFNnk8aCI5cWZ3afiMe3dmjysiKrdMQKAX141+srk+4+e1SGYePy6UrLPSwlQZGyquzHHe8YbfxmTzvsmzjkKcye8hDNok/5hk07UhhCjhRySxjohqzgzCFvTYwVYv+A6Fn3lZ34j+4qIK9bRLi008MdB/u0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723648791; c=relaxed/simple; bh=XGuhBnv/MQQll+NCKqoo61KE0fgOnlHZWsz6TSQEtNY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=J317K7FLvy5ExMZD66ZclGfMVsY4U3AzF7W53wDh0t3v2c4QyqFHmngz6dTdJt7Cx6yx4ukzrd6i8XLC0+YRovlkg318mWU9/yhxjJBZ4W4U6dcwqmV76CTUsnGnjs8T0BmAYv//OFg9szMcO6LD4nD4AVcWCYbMunIJrSWkgjI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=DGCAjS/H; arc=none smtp.client-ip=209.85.208.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="DGCAjS/H" Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-5a15692b6f6so66520a12.0 for ; Wed, 14 Aug 2024 08:19:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1723648788; x=1724253588; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=gTi7bYjYTGMEWGbyKlWoq2Paz7kIbeWFkJUp3rdie2E=; b=DGCAjS/HoeJPconZgbWhtB1sEzhgKIS8J9nDlWQYlHCplwHkTW5/kfTAWzTdbEuYZX UK/Q4AuOTeXeYJgtkK3yUmOPwvqsUNYb4si6qkuhzppSk3GtX7LZPcdT7To6LmIfDbKW 6ZwYNN2DOkNPXJj/YAcxAl2da9Vch5fbYQgPA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723648788; x=1724253588; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gTi7bYjYTGMEWGbyKlWoq2Paz7kIbeWFkJUp3rdie2E=; b=WTmsdN3QI50kxBGd2vQdYClhOcD8v4W7Edhn/b62wNsimceSSmPkWNAnlt1NW5SsGI 60pdOyn+/3j5+GZpI0t0eXH/4yI1hoqpn52mBlo5aANVr7bLt/OWIcDzDOQR9q/wlrPZ /aA7LNar4GeObvaVNlmHmLIDihlt97IQHadPOkPSuKXgAcOI76DHEGIO7fZ/Afxyhyz+ j+NSNCjj0++nbXkcQpwFctMQAOKgWWmD45noBkgH+OmZBmunpT9twFZDYZ3nB2ykVG2s vKazQ+HOtM2s/La+FZmzuTdQ9amEM7MBD+IWtClwfQqRVvbZ/nbt5UjI8kHziOHj/o43 mjEg== X-Forwarded-Encrypted: i=1; AJvYcCXrda6zpBRH0Fp49RIZ62ucUbR3AuCZQbNNhidteFoRumuYODR4VNiStWeFxlb9nlT2fDEOOU0ptRsqRMLaMCm1ucuA/NnBwpoV1A== X-Gm-Message-State: AOJu0YxD9OPtkJfTCVO1QRvCvTWv7CoKsJEe/zOdww3sesOIHg4Ua2ZE IOEkGetNXO7nTar6YjEruIuVSi6wLdHOE6ekNmJYj2d7rRwASUbCd/jN2AQsU/A= X-Google-Smtp-Source: AGHT+IFvDOsbuKJOT0XL/TM+iXA5EFj0BhRg1ts5WGvw7A2g2cgRk9L2pdTDRxwV8rDbcUIbgQvSeA== X-Received: by 2002:a17:907:d5a4:b0:a72:40b4:c845 with SMTP id a640c23a62f3a-a8367006615mr221312666b.51.1723648788188; Wed, 14 Aug 2024 08:19:48 -0700 (PDT) Received: from LQ3V64L9R2 ([80.208.222.2]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a80f414e127sm183820466b.150.2024.08.14.08.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 08:19:47 -0700 (PDT) Date: Wed, 14 Aug 2024 16:19:45 +0100 From: Joe Damato To: Jakub Kicinski Cc: netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao Subject: Re: [RFC net-next 0/6] Cleanup IRQ affinity checks in several drivers Message-ID: Mail-Followup-To: Joe Damato , Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao References: <20240812145633.52911-1-jdamato@fastly.com> <20240813171710.599d3f01@kernel.org> <20240814080915.005cb9ac@kernel.org> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240814080915.005cb9ac@kernel.org> On Wed, Aug 14, 2024 at 08:09:15AM -0700, Jakub Kicinski wrote: > On Wed, 14 Aug 2024 13:12:08 +0100 Joe Damato wrote: > > Actually... how about a slightly different approach, which caches > > the affinity mask in the core? > > I was gonna say :) > > > 0. Extend napi struct to have a struct cpumask * field > > > > 1. extend netif_napi_set_irq to: > > a. store the IRQ number in the napi struct (as you suggested) > > b. call irq_get_effective_affinity_mask to store the mask in the > > napi struct > > c. set up generic affinity_notify.notify and > > affinity_notify.release callbacks to update the in core mask > > when it changes > > This part I'm not an export on. > > > 2. add napi_affinity_no_change which now takes a napi_struct > > > > 3. cleanup all 5 drivers: > > a. add calls to netif_napi_set_irq for all 5 (I think no RTNL > > is needed, so I think this would be straight forward?) > > b. remove all affinity_mask caching code in 4 of 5 drivers > > c. update all 5 drivers to call napi_affinity_no_change in poll > > > > Then ... anyone who adds support for netif_napi_set_irq to their > > driver in the future gets automatic support in-core for > > caching/updating of the mask? And in the future netdev-genl could > > dump the mask since its in-core? > > > > I'll mess around with that locally to see how it looks, but let me > > know if that sounds like a better overall approach. I ended up going with the approach laid out above; moving the IRQ affinity mask updating code into the core (which adds that ability to gve/mlx4/mlx5... it seems mlx4/5 cached but didn't have notifiers setup to update the cached copy?) and adding calls to netif_napi_set_irq in i40e/iavf and deleting their custom notifier code. It's almost ready for rfcv2; I think this approach is probably better ? > Could we even handle this directly as part of __napi_poll(), > once the driver gives core all of the relevant pieces of information ? I had been thinking the same thing, too, but it seems like at least one driver (mlx5) counts the number of affinity changes to export as a stat, so moving all of this to core would break that. So, I may avoid attempting that for this series. I'm still messing around with this but will send an rfcv2 in a bit.