From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81FD9270022 for ; Thu, 27 Feb 2025 04:48:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740631689; cv=none; b=eKrYBZvzdAdGmiFYRbP7yPPGqn4VFhBfdz7sSwtflR3Jn86JZ1ORRZexQ36xfE5scQTYj3U4hDf9UGNeV0TWfaJgFykhz9HDEzr26R4mqk+LkX4OASwAzJ2eelPED3ALzU8BMn6pulCGc+t+fdO5xjj5Urhylo6ZizLhYeX0IUs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740631689; c=relaxed/simple; bh=Dz5CLJbJtxhty0xg8roq/GxXUaSa9CenQ02zc3yTaa8=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=TJoz0cAFdnixKR7llikX9EXUzl5A2vdQ8QybGwyYBYbOIb7K0CnlBj9CoYa76jQQwOX5EDgv/Gw7PfGhyGXbQLPXQ1+o/yaFm3rqP53OmNvL0rNwBcwZNx3lkMDMOXqt9MEz8igDi2mTpawI464QxxcIVfl9hTBbUU64yJr0a1c= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=daEsRvgc; arc=none smtp.client-ip=209.85.160.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="daEsRvgc" Received: by mail-qt1-f174.google.com with SMTP id d75a77b69052e-4720a115d15so14513641cf.1 for ; Wed, 26 Feb 2025 20:48:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1740631685; x=1741236485; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:from:to:cc:subject:date:message-id:reply-to; bh=j9RrWBQnGmr7pfCKfDKI3Or5JbQRE5y4Ygu5ENY7AJw=; b=daEsRvgcoYYCEMrJEdGNv1L2iPUswcd8MuchKmHWiGo77AukfOWpM3zn76hYBsn75p vSYeu3jT0Z0w0j56n3L1k5GplIb4WQpXsXQ7zPLMtXAid+N+Q41Fd0fUMQqSVPcjmQbj ax3mR/XFjPjXsndmfR8qfgjIzrFmrrrVM0EuI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740631685; x=1741236485; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:mail-followup-to:message-id:subject:cc:to :from:date:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=j9RrWBQnGmr7pfCKfDKI3Or5JbQRE5y4Ygu5ENY7AJw=; b=bwBkv9wqeXxwh3+CjNkaEraBYrQ1Zd/CFxOSh+RLey0CyUStXrwBL1K/OR0grJF2CN Kpc513MtPYIfH4/4n8mkYq/x0FPJSUYKuKDy6gssK6P1h9QaZ73gM9OnGZpke5kSg/Gx F4rdPnsHquqNTP3jwWRLO1X892wtYcpIPBfvmNZd1t5++uI5evVLvp/j1a6NoapTot51 LSeN4lg/bb+KQAd76owiHru0VWLjpeK6ITARLo9tDNN3oKaznPMBuY0NDmLCZkXcbhEi 6uQ6nVJjEBaqg9qt15ee58U9f0l9+tInLu+vCW8kQ/jiR91i6GBf3/q52q9Xe+RtR5Lt FnKw== X-Forwarded-Encrypted: i=1; AJvYcCUNhO7pSyCY3woXFg/CNd0hRM+le/Wb8ugKUlIdAKbY19FKa8Vaz5mPzuZDYM+mojXaMABAfXfzABi7lq0=@vger.kernel.org X-Gm-Message-State: AOJu0Yyu7vrFGzX7JOy3mUPK6ksqUHCcMTOMLTQMmr+z34/aSIKv1Siu ao2k/DWBnagXGlU4VQF8JnVx7b6Rd6TGGMb6CtsPEUHUN1gE+k0nr/6hNd2KrZ1Db2fPmDX4YSb 6 X-Gm-Gg: ASbGnctRzP77QtEzUgyMj1oz6kBnN4Wp1Cll0jpibig/23YfA5KGxSMk4o1K/yBVoqz 3fwq/NlURv5fGQU0od5gvVhYWCIXwQqO+E5oipM6TcZkuKVvVKU3jhfLCkDic0CVcn3YbXvV+el e0NCgM3v+McEC+PRDK5VrYvgjBKjCYUE/YirqNa8+wEjRqclC9KjEIHwoCIvEq1UhU0bSpu6pE7 MTQkIzRRz2tpQyxBS+JAj1WBAGuNQx9B7oVOd0g+aVYNqKktlAiIs0T+mJNwapLlGkAH6Sglr1I iIw/EuoWK8Uth57TolVQyOU74hoN1Nr/huy2KdjygpLIoO1aZGn16CAmf4x1Dsaz X-Google-Smtp-Source: AGHT+IG6AMaCEtuGDNAymqPMHPgx7iO0OQVDvaKjbVaa0iyhzZP9jJJbU8LowBwmScd9NGrAxjr9OQ== X-Received: by 2002:a05:622a:1187:b0:46f:d6c3:2ddc with SMTP id d75a77b69052e-473d8f7f11emr31340551cf.1.1740631685293; Wed, 26 Feb 2025 20:48:05 -0800 (PST) Received: from LQ3V64L9R2 (ool-44c5a22e.dyn.optonline.net. [68.197.162.46]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-47472409ac2sm6469901cf.58.2025.02.26.20.48.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 20:48:04 -0800 (PST) Date: Wed, 26 Feb 2025 23:48:01 -0500 From: Joe Damato To: Jason Wang Cc: "Michael S. Tsirkin" , netdev@vger.kernel.org, mkarsten@uwaterloo.ca, gerhard@engleder-embedded.com, xuanzhuo@linux.alibaba.com, kuba@kernel.org, Eugenio =?iso-8859-1?Q?P=E9rez?= , Andrew Lunn , "David S. Miller" , Eric Dumazet , Paolo Abeni , "open list:VIRTIO CORE AND NET DRIVERS" , open list Subject: Re: [PATCH net-next v4 3/4] virtio-net: Map NAPIs to queues Message-ID: Mail-Followup-To: Joe Damato , Jason Wang , "Michael S. Tsirkin" , netdev@vger.kernel.org, mkarsten@uwaterloo.ca, gerhard@engleder-embedded.com, xuanzhuo@linux.alibaba.com, kuba@kernel.org, Eugenio =?iso-8859-1?Q?P=E9rez?= , Andrew Lunn , "David S. Miller" , Eric Dumazet , Paolo Abeni , "open list:VIRTIO CORE AND NET DRIVERS" , open list References: <20250225020455.212895-1-jdamato@fastly.com> <20250225020455.212895-4-jdamato@fastly.com> <20250226171252-mutt-send-email-mst@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Thu, Feb 27, 2025 at 12:18:33PM +0800, Jason Wang wrote: > On Thu, Feb 27, 2025 at 6:13 AM Michael S. Tsirkin wrote: > > > > On Wed, Feb 26, 2025 at 03:27:42PM -0500, Joe Damato wrote: > > > On Wed, Feb 26, 2025 at 01:08:49PM -0500, Joe Damato wrote: > > > > On Wed, Feb 26, 2025 at 01:03:09PM -0500, Joe Damato wrote: > > > > > On Wed, Feb 26, 2025 at 01:48:50PM +0800, Jason Wang wrote: > > > > > > On Tue, Feb 25, 2025 at 10:05 AM Joe Damato wrote: > > > > > > > > > > > > > > Use netif_queue_set_napi to map NAPIs to queue IDs so that the mapping > > > > > > > can be accessed by user apps, taking care to hold RTNL as needed. > > > > > > > > > > > > I may miss something but I wonder whether letting the caller hold the > > > > > > lock is better. > > > > > > > > > > Hmm... > > > > > > > > > > Double checking all the paths over again, here's what I see: > > > > > - refill_work, delayed work that needs RTNL so this change seems > > > > > right? > > > > > > > > > > - virtnet_disable_queue_pair, called from virtnet_open and > > > > > virtnet_close. When called via NDO these are safe and hold RTNL, > > > > > but they can be called from power management and need RTNL. > > > > > > > > > > - virtnet_enable_queue_pair called from virtnet_open, safe when > > > > > used via NDO but needs RTNL when used via power management. > > > > > > > > > > - virtnet_rx_pause called in both paths as you mentioned, one > > > > > which needs RTNL and one which doesn't. > > > > > > > > Sorry, I missed more paths: > > > > > > > > - virtnet_rx_resume > > > > - virtnet_tx_pause and virtnet_tx_resume > > > > > > > > which are similar to path you mentioned (virtnet_rx_pause) and need > > > > rtnl in one of two different paths. > > > > > > > > Let me know if I missed any paths and what your preferred way to fix > > > > this would be? > > > > > > > > I think both options below are possible and I have no strong > > > > preference. > > > > > > OK, my apologies. I read your message and the code wrong. Sorry for > > > the back-to-back emails from me. > > > > > > Please ignore my message above... I think after re-reading the code, > > > here's where I've arrived: > > > > > > - refill_work needs to hold RTNL (as in the existing patch) > > > > > > - virtnet_rx_pause, virtnet_rx_resume, virtnet_tx_pause, > > > virtnet_tx_resume -- all do NOT need to hold RTNL because it is > > > already held in the ethtool resize path and the XSK path, as you > > > explained, but I mis-read (sorry). > > > > > > - virtnet_disable_queue_pair and virtnet_enable_queue_pair both > > > need to hold RTNL only when called via power management, but not > > > when called via ndo_open or ndo_close > > > > > > Is my understanding correct and does it match your understanding? > > > > > > If so, that means there are two issues: > > > > > > 1. Fixing the hardcoded bools in rx_pause, rx_resume, tx_pause, > > > tx_resume (all should be false, RTNL is not needed). > > > > > > 2. Handling the power management case which calls virtnet_open and > > > virtnet_close. > > > > > > I made a small diff included below as an example of a possible > > > solution: > > > > > > 1. Modify virtnet_disable_queue_pair and virtnet_enable_queue_pair > > > to take a "bool need_rtnl" and pass it through to the helpers > > > they call. > > > > > > 2. Create two helpers, virtnet_do_open and virt_do_close both of > > > which take struct net_device *dev, bool need_rtnl. virtnet_open > > > and virtnet_close are modified to call the helpers and pass > > > false for need_rtnl. The power management paths call the > > > helpers and pass true for need_rtnl. (fixes issue 2 above) > > > > > > 3. Fix the bools for rx_pause, rx_resume, tx_pause, tx_resume to > > > pass false since all paths that I could find that lead to these > > > functions hold RTNL. (fixes issue 1 above) > > > > > > See the diff below (which can be applied on top of patch 3) to see > > > what it looks like. > > > > > > If you are OK with this approach, I will send a v5 where patch 3 > > > includes the changes shown in this diff. > > > > > > Please let me know what you think: > > > > > > > > Looks ok I think. > > > > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > > > index 13bb4a563073..76ecb8f3ce9a 100644 > > > --- a/drivers/net/virtio_net.c > > > +++ b/drivers/net/virtio_net.c > > > @@ -3098,14 +3098,16 @@ static int virtnet_poll(struct napi_struct *napi, int budget) > > > return received; > > > } > > > > > > -static void virtnet_disable_queue_pair(struct virtnet_info *vi, int qp_index) > > > +static void virtnet_disable_queue_pair(struct virtnet_info *vi, int qp_index, > > > + bool need_rtnl) > > > { > > > - virtnet_napi_tx_disable(&vi->sq[qp_index], false); > > > - virtnet_napi_disable(&vi->rq[qp_index], false); > > > + virtnet_napi_tx_disable(&vi->sq[qp_index], need_rtnl); > > > + virtnet_napi_disable(&vi->rq[qp_index], need_rtnl); > > > xdp_rxq_info_unreg(&vi->rq[qp_index].xdp_rxq); > > > } > > > > > > -static int virtnet_enable_queue_pair(struct virtnet_info *vi, int qp_index) > > > +static int virtnet_enable_queue_pair(struct virtnet_info *vi, int qp_index, > > > + bool need_rtnl) > > > { > > > struct net_device *dev = vi->dev; > > > int err; > > > @@ -3120,8 +3122,8 @@ static int virtnet_enable_queue_pair(struct virtnet_info *vi, int qp_index) > > > if (err < 0) > > > goto err_xdp_reg_mem_model; > > > > > > - virtnet_napi_enable(&vi->rq[qp_index], false); > > > - virtnet_napi_tx_enable(&vi->sq[qp_index], false); > > > + virtnet_napi_enable(&vi->rq[qp_index], need_rtnl); > > > + virtnet_napi_tx_enable(&vi->sq[qp_index], need_rtnl); > > > > > > return 0; > > > > > > @@ -3156,7 +3158,7 @@ static void virtnet_update_settings(struct virtnet_info *vi) > > > vi->duplex = duplex; > > > } > > > > > > -static int virtnet_open(struct net_device *dev) > > > +static int virtnet_do_open(struct net_device *dev, bool need_rtnl) > > > { > > > struct virtnet_info *vi = netdev_priv(dev); > > > int i, err; > > > @@ -3169,7 +3171,7 @@ static int virtnet_open(struct net_device *dev) > > > if (!try_fill_recv(vi, &vi->rq[i], GFP_KERNEL)) > > > schedule_delayed_work(&vi->refill, 0); > > > > > > - err = virtnet_enable_queue_pair(vi, i); > > > + err = virtnet_enable_queue_pair(vi, i, need_rtnl); > > > if (err < 0) > > > goto err_enable_qp; > > > } > > > @@ -3190,13 +3192,18 @@ static int virtnet_open(struct net_device *dev) > > > cancel_delayed_work_sync(&vi->refill); > > > > > > for (i--; i >= 0; i--) { > > > - virtnet_disable_queue_pair(vi, i); > > > + virtnet_disable_queue_pair(vi, i, need_rtnl); > > > virtnet_cancel_dim(vi, &vi->rq[i].dim); > > > } > > > > > > return err; > > > } > > > > > > +static int virtnet_open(struct net_device *dev) > > > +{ > > > + return virtnet_do_open(dev, false); > > > +} > > > + > > > static int virtnet_poll_tx(struct napi_struct *napi, int budget) > > > { > > > struct send_queue *sq = container_of(napi, struct send_queue, napi); > > > @@ -3373,7 +3380,7 @@ static void virtnet_rx_pause(struct virtnet_info *vi, struct receive_queue *rq) > > > bool running = netif_running(vi->dev); > > > > > > if (running) { > > > - virtnet_napi_disable(rq, true); > > > + virtnet_napi_disable(rq, false); > > > virtnet_cancel_dim(vi, &rq->dim); > > > } > > > } > > > @@ -3386,7 +3393,7 @@ static void virtnet_rx_resume(struct virtnet_info *vi, struct receive_queue *rq) > > > schedule_delayed_work(&vi->refill, 0); > > > > > > if (running) > > > - virtnet_napi_enable(rq, true); > > > + virtnet_napi_enable(rq, false); > > > } > > > > > > static int virtnet_rx_resize(struct virtnet_info *vi, > > > @@ -3415,7 +3422,7 @@ static void virtnet_tx_pause(struct virtnet_info *vi, struct send_queue *sq) > > > qindex = sq - vi->sq; > > > > > > if (running) > > > - virtnet_napi_tx_disable(sq, true); > > > + virtnet_napi_tx_disable(sq, false); > > > > > > txq = netdev_get_tx_queue(vi->dev, qindex); > > > > > > @@ -3449,7 +3456,7 @@ static void virtnet_tx_resume(struct virtnet_info *vi, struct send_queue *sq) > > > __netif_tx_unlock_bh(txq); > > > > > > if (running) > > > - virtnet_napi_tx_enable(sq, true); > > > + virtnet_napi_tx_enable(sq, false); > > Instead of this, it looks to me it would be much simpler if we can > just hold the rtnl lock in freeze/restore. I disagree. Holding RTNL for all of open and close instead of just the 1 API call that needs it has the possibility of introducing other lock ordering bugs now or in the future. We only need RTNL for 1 API, why hold it for all of open or close?