From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com [209.85.218.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C2FB18A6A8 for ; Tue, 20 Aug 2024 08:33:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.50 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724142787; cv=none; b=pPX9gOogM4Mmz0ytS7N3jPWvv2DdCiZTTzqQSIIzywxhFguIESq2ppL5KfCnb5aSfni5Xdr6h/+9MZyRnO2oRpZKIi4GwX8bWnMnniYEHpbicZhHn+27ea/5Hp2D9Y9k1xHMyJMaGS2dl8/ACgLi092R7okoBlRolfuc9tHXT/w= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724142787; c=relaxed/simple; bh=sSUYkM8HYmaJN7xqG+tfsAHvEYV5RAq6szVJelU+X+U=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=k5JRV1hap3wL3btXHOzAKEeC6t+Al7n02haYDVEeJFzB7KzesL3eNWsEoD6Ebth5ezTRoq074vSsBbbKSLYWFn2ZVMwS5nR7IjM1e8CElCnaJvk7d1KfQN8/ZkTlvcnZtP0Gxg3KylyyxcR8LcWpdLbegbRvpEicxPWldCS3DzM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=VG8ajf6B; arc=none smtp.client-ip=209.85.218.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="VG8ajf6B" Received: by mail-ej1-f50.google.com with SMTP id a640c23a62f3a-a7d26c2297eso598942666b.2 for ; Tue, 20 Aug 2024 01:33:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1724142784; x=1724747584; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date:from:to:cc :subject:date:message-id:reply-to; bh=XNQ9tyNfl4ljX/BPGLUOjjoGD0Ou+TIYM+qPExssLvc=; b=VG8ajf6BuTNmkSG0citgtyRIslAki+3nxKaQreOE/2Jt+XtQq867Hbt6IZQKqYcZIy Jah6u7gnrTVuJMlWiAlwcqZxMRsbpB8G9sGpIMHVAIwMhDn6Chc2+I4tonuf1y1MhNdL F8BFC3GhlV619NSvDG4KxYAMFyyT5dW/57ES4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724142784; x=1724747584; h=in-reply-to:content-disposition:mime-version:references :mail-followup-to:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XNQ9tyNfl4ljX/BPGLUOjjoGD0Ou+TIYM+qPExssLvc=; b=n/fx5zjIwpHFkL1Bl02FbxG97kluQ4SFQeS+qkiTX/3aPmqpyw8kRkibEvpgou2keA +4IIbfzWTt74Oe8MMw1VKdNSz7KuVp4Mc8NfytcAtTJCe4Yh1X5QbCnbVnFaIM1wDDmV 893bf0KhLmRlIo8H6pJejnfF3xx49qWwFEFF1VQvJuTM5ohyPt5hY4SZ+BrQMDS3xPWF S3y6wYAgz5mMc99w8kqB6QMlqxwKhMpGAG7MDCjXvbCDfACMLOSnjSADIgJP8AL39EWn aEqp0tZtyGP5oGNJGabPErpnSacxpzegqCGKxzzbQ3HgpdYTvY5S5vpBJc0/VL7WETgK ZL0Q== X-Forwarded-Encrypted: i=1; AJvYcCWeYUOqr+Jy+O050C7I6LkaaIcZA/QiEhKdl7+0Lw01K+fHrvuXucKub9LWEsfmQjbDaKvjfnfygKMQTyzgrdG/8LTDaAQX X-Gm-Message-State: AOJu0Yy58ETfaSnuroqNwrT3HkhSa4NaUgyEenDPx9rcJkjNoN7qihix 6puGUj+OjKJrKz2JHbgDG49+k+cKTgWzLdeSWAe9hPBF4sGW4lG53G1P8cL7980= X-Google-Smtp-Source: AGHT+IHdCu1gJlayHwi4drLhhfQxyfMUzMC+JYbFPtj9dlU1JF6GxmGI/OuqVO5LTmVnsfb9hkmppA== X-Received: by 2002:a17:907:97cf:b0:a6f:59dc:4ece with SMTP id a640c23a62f3a-a83928a4023mr845774466b.2.1724142783295; Tue, 20 Aug 2024 01:33:03 -0700 (PDT) Received: from LQ3V64L9R2.home ([2a02:c7c:f016:fc00:3906:31c:255a:bf09]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a83aeb6eb4dsm435559766b.35.2024.08.20.01.33.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Aug 2024 01:33:03 -0700 (PDT) Date: Tue, 20 Aug 2024 09:33:01 +0100 From: Joe Damato To: Shay Drori Cc: Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao , Thomas Gleixner Subject: Re: [RFC net-next 0/6] Cleanup IRQ affinity checks in several drivers Message-ID: Mail-Followup-To: Joe Damato , Shay Drori , Jakub Kicinski , netdev@vger.kernel.org, Daniel Borkmann , "David S. Miller" , Eric Dumazet , Harshitha Ramamurthy , "moderated list:INTEL ETHERNET DRIVERS" , Jeroen de Borst , Jiri Pirko , Leon Romanovsky , open list , "open list:MELLANOX MLX4 core VPI driver" , Lorenzo Bianconi , Paolo Abeni , Praveen Kaligineedi , Przemek Kitszel , Saeed Mahameed , Sebastian Andrzej Siewior , Shailend Chand , Tariq Toukan , Tony Nguyen , Willem de Bruijn , Yishai Hadas , Ziwei Xiao , Thomas Gleixner References: <20240813171710.599d3f01@kernel.org> <20240814080915.005cb9ac@kernel.org> <701eb84c-8d26-4945-8af3-55a70e05b09c@nvidia.com> <20240814172046.7753a62c@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Tue, Aug 20, 2024 at 09:40:31AM +0300, Shay Drori wrote: > > > On 15/08/2024 13:22, Joe Damato wrote: > > External email: Use caution opening links or attachments > > > > > > On Wed, Aug 14, 2024 at 05:20:46PM -0700, Jakub Kicinski wrote: > > > On Wed, 14 Aug 2024 19:01:40 +0100 Joe Damato wrote: > > > > If it is, then the only option is to have the drivers pass in their > > > > IRQ affinity masks, as Stanislav suggested, to avoid adding that > > > > call to the hot path. > > > > > > > > If not, then the IRQ from napi_struct can be used and the affinity > > > > mask can be generated on every napi poll. i40e/gve/iavf would need > > > > calls to netif_napi_set_irq to set the IRQ mapping, which seems to > > > > be straightforward. > > > > > > It's a bit sad to have the generic solution blocked. > > > cpu_rmap_update() is exported. Maybe we can call it from our notifier? > > > rmap lives in struct net_device > > > > I agree on the sadness. I will take a look today. > > > > I guess if we were being really ambitious, we'd try to move ARFS > > stuff into the core (as RSS was moved into the core). > > > Sorry for the late reply. Maybe we can modify affinity notifier infra to > support more than a single notifier per IRQ. > @Thomas, do you know why only a single notifier per IRQ is supported? Sorry for the delayed response as well on my side; I've been in between lots of different kernel RFCs :) Jakub: the issue seems to be that the internals in lib/cpu_rmap.c are needed to call cpu_rmap_update. It's probably possible to expose them somehow so that a generic IRQ notifier could call cpu_rmap_update, as you mentioned, but some rewiring is going to be needed, I think. I had a couple ideas for rewiring stuff, but I haven't had time to context switch back on to this work as I've been busy with a few other things (the IRQ suspension stuff and another mlx5 thing I have yet to send upstream). I hope to take another look at it this week, but I welcome any suggestions from Shay/Thomas in the meantime. - Joe