From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27C5A2DF13E for ; Tue, 9 Dec 2025 15:23:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765293819; cv=none; b=Ou7RWf8jmKJVBKs7aogrWCathA6YbBY6ZcvaxF8g7JzR8oBrSqF7f0fQvZbDdJH32AuJRKov+NpKBUxUrmWe9FcAmSzIr4wzFzQmxu9epjQEwqbNJwaLWLOzHoIc1loGZWdI4KpTu6eXaHd/jOpr771GzEdVpBBRYCxA4l7rf1g= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1765293819; c=relaxed/simple; bh=JD5hyBpJAaNJ7sYgl/UrQWHJjtabTdl0L4UaYY83N+s=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=F+I9nxuU8ebC/Epk6mh92XvBJJAIiOIsVIhSKDWSAnDwlOCnclkO/rk44Nf1+rbTLE57HhyG9z6uOfu7pNIWgcd+gT8HEYJIjiaoHrb0mcS+QbSAtoasxQ2WWmleOVLlkAsFZ319l3CaHSTCoDTbF93RGkn4igGSzlTK8T9m5lk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dq659yd9; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dq659yd9" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-7aad4823079so5238736b3a.0 for ; Tue, 09 Dec 2025 07:23:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1765293817; x=1765898617; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=W7DbZ+8N23FwouofrnDLigkDHoe3CUpkkU2pOJ+ToVI=; b=dq659yd9spEH0AGlJjSXLFg6yLammOn6b7Fash88SM1EoPff0OUXVC+Ah/d/EPfwqm YE8eERQNhpwarsHnbSftuNxhg54x80TAfSHqYgMqe/pGXRHo/XmVDBto3IaQdG0XROzn vKIr8nyQbVjI3fxamkeWdzYSi7qzNWRd2/u48xE8O18sY2T88KQBMJjwXOP1x4hcm/le Wrttm7XMiFG7E6THyTguFmi3pSdYEZSOfyo/++qQD4c7QIhaTGHRH1kOMN9w6AQk3EzV Uljw8GebCdczwbGV2kp0WQW3xe2T8DF4JN2wa3Ohfmb1PbWK61JiNvrr3vqXlGRYsJ6k BpDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765293817; x=1765898617; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=W7DbZ+8N23FwouofrnDLigkDHoe3CUpkkU2pOJ+ToVI=; b=Ov6ECgBZs2O7fnee8tNHYI6fQFUBj4oHjWCM/zfa6EUrVaSsz9gsjfg2VkGSTKW7X5 T1g//JHkcFhmaXX7REt4ds609w93pkuZyDNsXLo7/sBy1Vq99wy1K3W7r9ZNUJ7hURj8 B5bYEVYUyUbfz5vOjgrrxwrKKDI3NtzHE52Ltzb8Buntu/cH0QCvRJsJ5ispynJLLtUt jnjJUMJEKH/Gf16+EBqwmef3gbqIOzDWqFb7uTXqpdhq9paI0QZ9QDETGKRI/V6Rz05V epw0cVVX2kxYdAu8DEUKCLAPOglYqKmWmh/5oyp0CjmYn3mmAiAlG/W0lCNQb5wkbHbA 2peQ== X-Forwarded-Encrypted: i=1; AJvYcCUxKnFh4vwO6AakLxwIbbxfqzreb+rLnEQ5GIrG0Whqo7ZB/vrXtzLATW9c65V15fopZMCm8Y9p+h9Br6U=@vger.kernel.org X-Gm-Message-State: AOJu0YwYPj0Ho52gVG7/ik9M9q5sOQy1N2tIe/FGgAtxuffSMZ+1Ux4I craQrH1zY9l5v7ihA1UaPjij1tWoMqw9ZutvCQDv558D6iDwyS5x+ieb X-Gm-Gg: ASbGncsmaZV+dusGJ1s9+E9QW0sta1zSFif0iNpb4z26+WMT8vtDC8RZD1ikUbFztQm CYv+hkfry7u4GaIEu9887Etx8wj/Tgx9w0WUA6a6QTQsZDzSyMFDiPwF7DU88PM9cdRdGxYXIiz xO5feOoxHv5L89FnvoMZoUB20BQUCZhQG/ADvm/EuO/JarV6gNzDoYMotcZZEjh1LD/bFtI9MtO 2O5VuRA4wLCfkZGFPKi7BGkP5Qpa6ZTJkOWLDehYZvLGm3+5Dv0aW2W8vJ2in7qoAWB7weQBIom ckOraNfIYZ++rgye9M6GIdA8UT6wzdI87IpWmgZ83lPjXWridx7Q9ubSvknhLAfESdlSaJzPa2A JtfIW/be/Nj2skPFtJ0+5s4gYUykTNGJWCWM/rOPJh2cdmXVoaVV9MDSV7W1oDLNjUp0VEZwp5r dad+EPtJZ2St49is3RbS1A1AnoPklFKR40w7+XvhKcx88QYIyQrfbDHu+5r79v X-Google-Smtp-Source: AGHT+IFN5sdylXJLxCHNNsv6xbK8xt8uPVU4i306NX8HPuK5V8M26O/oWPXK67LxkxgltqZUNhCZqw== X-Received: by 2002:a05:6a00:1703:b0:7e8:4471:8d1 with SMTP id d2e1a72fcca58-7e8c786662bmr9491026b3a.50.1765293817283; Tue, 09 Dec 2025 07:23:37 -0800 (PST) Received: from ?IPV6:2001:ee0:4f4c:210:e7d6:7fa4:50a3:fa14? ([2001:ee0:4f4c:210:e7d6:7fa4:50a3:fa14]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7ed2cd65ad1sm6229403b3a.56.2025.12.09.07.23.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 09 Dec 2025 07:23:36 -0800 (PST) Message-ID: <66d9f44c-295e-4b62-86ae-a0aff5f062bb@gmail.com> Date: Tue, 9 Dec 2025 22:23:30 +0700 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH net] virtio-net: enable all napis before scheduling refill work To: Jason Wang Cc: netdev@vger.kernel.org, "Michael S. Tsirkin" , Xuan Zhuo , =?UTF-8?Q?Eugenio_P=C3=A9rez?= , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Stanislav Fomichev , virtualization@lists.linux.dev, linux-kernel@vger.kernel.org, bpf@vger.kernel.org References: <20251208153419.18196-1-minhquangbui99@gmail.com> Content-Language: en-US From: Bui Quang Minh In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 12/9/25 11:30, Jason Wang wrote: > On Mon, Dec 8, 2025 at 11:35 PM Bui Quang Minh wrote: >> Calling napi_disable() on an already disabled napi can cause the >> deadlock. In commit 4bc12818b363 ("virtio-net: disable delayed refill >> when pausing rx"), to avoid the deadlock, when pausing the RX in >> virtnet_rx_pause[_all](), we disable and cancel the delayed refill work. >> However, in the virtnet_rx_resume_all(), we enable the delayed refill >> work too early before enabling all the receive queue napis. >> >> The deadlock can be reproduced by running >> selftests/drivers/net/hw/xsk_reconfig.py with multiqueue virtio-net >> device and inserting a cond_resched() inside the for loop in >> virtnet_rx_resume_all() to increase the success rate. Because the worker >> processing the delayed refilled work runs on the same CPU as >> virtnet_rx_resume_all(), a reschedule is needed to cause the deadlock. >> In real scenario, the contention on netdev_lock can cause the >> reschedule. >> >> This fixes the deadlock by ensuring all receive queue's napis are >> enabled before we enable the delayed refill work in >> virtnet_rx_resume_all() and virtnet_open(). >> >> Fixes: 4bc12818b363 ("virtio-net: disable delayed refill when pausing rx") >> Reported-by: Paolo Abeni >> Closes: https://netdev-ctrl.bots.linux.dev/logs/vmksft/drv-hw-dbg/results/400961/3-xdp-py/stderr >> Signed-off-by: Bui Quang Minh >> --- >> drivers/net/virtio_net.c | 59 +++++++++++++++++++--------------------- >> 1 file changed, 28 insertions(+), 31 deletions(-) >> >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >> index 8e04adb57f52..f2b1ea65767d 100644 >> --- a/drivers/net/virtio_net.c >> +++ b/drivers/net/virtio_net.c >> @@ -2858,6 +2858,20 @@ static bool try_fill_recv(struct virtnet_info *vi, struct receive_queue *rq, >> return err != -ENOMEM; >> } >> >> +static void virtnet_rx_refill_all(struct virtnet_info *vi) >> +{ >> + bool schedule_refill = false; >> + int i; >> + >> + enable_delayed_refill(vi); > This seems to be still racy? > > For example, in virtnet_open() we had: > > static int virtnet_open(struct net_device *dev) > { > struct virtnet_info *vi = netdev_priv(dev); > int i, err; > > for (i = 0; i < vi->max_queue_pairs; i++) { > err = virtnet_enable_queue_pair(vi, i); > if (err < 0) > goto err_enable_qp; > } > > virtnet_rx_refill_all(vi); > > So NAPI and refill work is enabled in this case, so the refill work > could be scheduled and run at the same time? Yes, that's what we expect. We must ensure that refill work is scheduled only when all NAPIs are enabled. The deadlock happens when refill work is scheduled but there are still disabled RX NAPIs. Thanks, Quang Minh.