From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E497A1B010D for ; Wed, 14 Aug 2024 12:12:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723637534; cv=none; b=sAscyhtvi1GeilGVD7fkgfGInD4D/PngcenBKopCOYxmZlPFeWRNi0ZK1HZNIaJ/4KpFBRHNMbdP5e958Onom78noQFM0dnmIamMo+x2r+A3eUZvY3FkDQ35NI2obX51XzuWL9gAWVv01rNgSART3Orx3EO1xJL48R1cH2x38e0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723637534; c=relaxed/simple; bh=MQ57xcNRnx2hbtvSnDnW9ALhWFVxGf3zZIIpvn2IVqk=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=YF0LzcWRh54JYt6Nm36StsYsGgHUMUXKYiY9JaqIr/IOlaA//8rWV0tspdzi4GmPWJUh/OcvR4bm3+DgRjEFtJg64KZTPdGI2ymSxEw+1zC33lXqZL+CFZPnOHSWFZvFI99FtweWh3BXWrjlZRXfrA+XAVUD16PIND1UJ4c3EMM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=ER91I/pA; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="ER91I/pA" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-42819654737so48297825e9.1 for ; Wed, 14 Aug 2024 05:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1723637531; x=1724242331; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=PG9e6JitvYXoqzSTP8p6u30qXecQWEzNyRf8++iwfzg=; b=ER91I/pAUCKFn6ndSuySZNLG422z4clo57ycN76odiFyCjcH0QMFJQp2MN2KmlXIo9 /UIISX3qwXpFZT0GEMPLg8kDii+imqwHiSvzq3KPDu+rHcXwilLPLGqqpGZR/7qSjLJ8 Xns+oN5HGUECJ2zb3vkHnQT3nPtPVY03X0LNE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723637531; x=1724242331; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=PG9e6JitvYXoqzSTP8p6u30qXecQWEzNyRf8++iwfzg=; b=VcJGRAavZ9kDq42UL9eF89GXt0BFSxXjPgdeefYdfjFVn57bADJmGXeAtc46zjue/u +vnz6Pq9SnUWBcEZrUKIFoFb3udOxPWb30R5re0D4Zcfg1KbV4lWVKt/Jkb4sHQlP7mW hZIZ0cXZJcvENpdrvrPgyP/MJId7MWLaU7OG7g5VhulI5rv4Vc9TVI3O5kKL/IK/ONsN IBBgvaWmHpdrxZDLFuy2qwQIc4Stzdxka3l2GfcU5W5X3I+YqlrnL8yoCKy5MMmoHFA6 xHEE37vApo7+98UmBraXYJIL0dqPF2JVA/+QanhdZfQOepElYCw2nF+dqcre52ctPozZ iUCQ== X-Forwarded-Encrypted: i=1; AJvYcCUyJzTAGwCEzCNLNTU+/18oFvl6F0u9VUPf8eiSRRwB3ry0iYa9qOOU/H8MCpxQ2NCJb697a5e/G3P77rKHAkyEO1qXQpjM X-Gm-Message-State: AOJu0YxZq0E/FYv5hgCBBvMtfjUYcbI3iKAWcxUheIkzPTAym4hB4dh2 TMJ5EjPS/Is/R9qslZXjO8IFKhTXWXPCMZK5ItOK9MeY9cMFvmQ40vBlfdHDOT0= X-Google-Smtp-Source: AGHT+IEXQhc1DRLmbpucvwdmK1QHHySaNdm9iIXPW0P1dXOgYRVT3R+EtoisglPADmrSMdzG+PAwVw== X-Received: by 2002:a05:600c:4710:b0:426:62c5:4741 with SMTP id 5b1f17b1804b1-429dd22fe4dmr18421295e9.2.1723637531117; Wed, 14 Aug 2024 05:12:11 -0700 (PDT) Received: from LQ3V64L9R2.home ([80.208.222.2]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ded1e3f7sm18267235e9.4.2024.08.14.05.12.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 05:12:10 -0700 (PDT) Date: Wed, 14 Aug 2024 13:12:08 +0100 From: Joe Damato To: Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao Subject: Re: [RFC net-next 0/6] Cleanup IRQ affinity checks in several drivers Message-ID: Mail-Followup-To: Joe Damato , Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao References: <20240812145633.52911-1-jdamato@fastly.com> <20240813171710.599d3f01@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Wed, Aug 14, 2024 at 08:14:48AM +0100, Joe Damato wrote: > On Tue, Aug 13, 2024 at 05:17:10PM -0700, Jakub Kicinski wrote: > > On Mon, 12 Aug 2024 14:56:21 +0000 Joe Damato wrote: > > > Several drivers make a check in their napi poll functions to determine > > > if the CPU affinity of the IRQ has changed. If it has, the napi poll > > > function returns a value less than the budget to force polling mode to > > > be disabled, so that it can be rescheduled on the correct CPU next time > > > the softirq is raised. > > > > Any reason not to use the irq number already stored in napi_struct ? > > Thanks for taking a look. > > IIUC, that's possible if i40e, iavf, and gve are updated to call > netif_napi_set_irq first, which I could certainly do. > > But as Stanislav points out, I would be adding a call to > irq_get_effective_affinity_mask in the hot path where one did not > exist before for 4 of 5 drivers. > > In that case, it might make more sense to introduce: > > bool napi_affinity_no_change(const struct cpumask *aff_mask) > > instead and the drivers which have a cached mask can pass it in and > gve can be updated later to cache it. > > Not sure how crucial avoiding the irq_get_effective_affinity_mask > call is; I would guess maybe some driver owners would object to > adding a new call in the hot path where one didn't exist before. > > What do you think? Actually... how about a slightly different approach, which caches the affinity mask in the core? 0. Extend napi struct to have a struct cpumask * field 1. extend netif_napi_set_irq to: a. store the IRQ number in the napi struct (as you suggested) b. call irq_get_effective_affinity_mask to store the mask in the napi struct c. set up generic affinity_notify.notify and affinity_notify.release callbacks to update the in core mask when it changes 2. add napi_affinity_no_change which now takes a napi_struct 3. cleanup all 5 drivers: a. add calls to netif_napi_set_irq for all 5 (I think no RTNL is needed, so I think this would be straight forward?) b. remove all affinity_mask caching code in 4 of 5 drivers c. update all 5 drivers to call napi_affinity_no_change in poll Then ... anyone who adds support for netif_napi_set_irq to their driver in the future gets automatic support in-core for caching/updating of the mask? And in the future netdev-genl could dump the mask since its in-core? I'll mess around with that locally to see how it looks, but let me know if that sounds like a better overall approach. - Joe