From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 935E935FF7B for ; Sat, 31 Jan 2026 17:46:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769881589; cv=none; b=ALYxmWnXLXRDBvRTHPCf859+tSw0G72intjRtuQ28pzxgoLBWE7wNch43FEmQGwucyF+gYphWIFjMJcC5wwxrXnnCnR9cZQi2ieo1UyKyMGBFMrhVsHzumo/hOrtyDMRh55wWuj2p2AuQKxcyeYbz4qpZrw7Me7uXIXAjl/o8QY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769881589; c=relaxed/simple; bh=FO0XxGiW7YqfVtzLltyctsCxTlWqT0+V7VxlKQHFZjY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=hd/6P0cdeOQqqR3qwMZPBJkJ0MtTpkGIFpl+tV0Sjs76V2jWkplcjTf3X3bQ5cpW/BGoxyUDcCvLRKA66l9LPwud5kINcyeUZtZaMJctj18Ol/NY+TLhza9DtjVSIPTeCc0DSZJSh5+GMgtVV4XOvvbJ0m7/9n3KqxX6msQXzfk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=LXzVlO8v; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=Q+TLzkC1; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LXzVlO8v"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="Q+TLzkC1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1769881586; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=5XwIcJJrN88f5EmnbhwkMpwuoOxnaOad2/ndcUFOcSM=; b=LXzVlO8vs4axWQd715chtDsx9UvKnJxT9ejNqnRJLzH+1wlomTjR9l4v/DwGcWR2syUTUZ 7SoI7z2mQakNK1L4juxewjagoUFVB3PTQKzGEuxCZInxhxIOa6GUToHA2E2TMby1WtC/37 pp2TsIxAyT+jkZs3LfxI3TNjjFUnt0Q= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-209-G1rZcVC7O5aBusjDza-YDQ-1; Sat, 31 Jan 2026 12:46:24 -0500 X-MC-Unique: G1rZcVC7O5aBusjDza-YDQ-1 X-Mimecast-MFC-AGG-ID: G1rZcVC7O5aBusjDza-YDQ_1769881583 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4806b12ad3fso27984075e9.0 for ; Sat, 31 Jan 2026 09:46:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1769881583; x=1770486383; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=5XwIcJJrN88f5EmnbhwkMpwuoOxnaOad2/ndcUFOcSM=; b=Q+TLzkC1DP5bITFBGDhnXcm3vNcnsid/WBjU1A7k4o/7Uxb5FBFQXwvJvMkur0i4b9 MXinKAebcQS9GRtj6/4TQhOsu0+6HOAgvxrEBB9AxSk3IDY+N1+X6AampNv9hsYm37Sb /pV3p6/+yJFzT5IEVAvmtFYP6J60iJN2GL73sv1RUrcC+STZr1xDeO2UMXl4L6acP+PW 9qitcI4EGi8INZqT+onEY5WjZl+FX6hvlEIm2Rc8Fx6Nie1q9c8E93G4edHq1+z+AIX9 i0ERkCW7WfA5AHPaKYlXdEJdmAv0jTlSgE90yjCjX+pOlJ3eo5Q9KctiCKUVEd0QyRTl WT4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769881583; x=1770486383; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5XwIcJJrN88f5EmnbhwkMpwuoOxnaOad2/ndcUFOcSM=; b=Jzir50NtME0TPXZFk5PDQHbX+HC5+N0Nxpzr98RjPVd0MhAAEREmzjdLyvS7NbS/rW 6ccwsQpNhTq+Iyb+aYu7PTvVmONlt0VK5+ctqtR5jxHdq77Wtf24hdb5ENjU6a795JvG xBKoUkV69VB9ie74sTZ0k2d8tuTRd3BiVthBfN7QAYtMqQKpQSZNJqqCp+C+5a6xfOF7 t7vyE/DCZwMcmhiNa6SqorEakvInTSpshbh/zuU+Z8pgW1G0z1urOa3HulnjGCykUFeI Ng1IDJ4tq5xNTFsAsnkkTn29E8OOjYSY375PhK+6+8+6IeWtoeBfYZS2cHDjEb+I/Cab uGMA== X-Forwarded-Encrypted: i=1; AJvYcCUhETy/6zWi5ARh/JPMBLcG9iQqtSxRZ41Eca00+tasSB6tinIsOHtH0RcLNdvYTwtkl9rHD+STPl7JEqE=@vger.kernel.org X-Gm-Message-State: AOJu0Yw32ZGPgyP3sLqZmH8bnij5g/kdTaAEV5zAm3sASYQ7VDffOI5J qrUiidaVMhPZYj2gxiPFxsdbwjdP//z/FIp7/rDH94R4rMBQyDE14KHZeTeSRT10Gg4d/1kZcVq hpLoWfTusE24R4BLtYwV/yRNL/PHWQOtSnIXeoCaV37OPn8my0eUjXNDXxhd3pphb8p+Xho/x3Q == X-Gm-Gg: AZuq6aLFv2peIGPj2ixZ0FZA7b/5PGkIfgD1d+J/GyLkQGN/YQhYSIzEt0BNBy+ZNvO tQ4md4yJRR9XOU8FZzVXv5xcVRUMM+DirVvyiYt7QHDoBSOsrZHXo4pfWRgxQE3te6tNnYRIvwK QSeftZZVinKGiX1W2E180vLBsu60sCzgjfJgv022Hdc/AXX9ccDYZ3dkpUve5eymtRS6ouBgJ3Z Xzl1VryW0ra2QY1unZSmhl9BRtVYnVSqMlxOpI21gTwuCAU7EJ75d4vj4Mc2EY+whaQCSAMCMfy zjY5WrYr/ya4KU0S02EPg2t+e6OGICySl8k01Ci/Pgjw4UhH88tiyV2Ts7ltKlAOrBsuLkSX3Xd rud1GAw== X-Received: by 2002:a05:600c:34c4:b0:480:52fd:d2e4 with SMTP id 5b1f17b1804b1-482db012dd7mr87513095e9.0.1769881583161; Sat, 31 Jan 2026 09:46:23 -0800 (PST) X-Received: by 2002:a05:600c:34c4:b0:480:52fd:d2e4 with SMTP id 5b1f17b1804b1-482db012dd7mr87512805e9.0.1769881582675; Sat, 31 Jan 2026 09:46:22 -0800 (PST) Received: from redhat.com ([2a06:c701:73e3:8f00:866c:5eeb:fc46:7674]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4806ce4c3d1sm289628525e9.9.2026.01.31.09.46.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 31 Jan 2026 09:46:21 -0800 (PST) Date: Sat, 31 Jan 2026 12:46:19 -0500 From: "Michael S. Tsirkin" To: Ira Weiny Cc: Li Chen , Dan Williams , Vishal Verma , Dave Jiang , Pankaj Gupta , Cornelia Huck , Jakub Staron , nvdimm@lists.linux.dev, virtualization@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH] nvdimm: virtio_pmem: serialize flush requests Message-ID: <20260131124554-mutt-send-email-mst@kernel.org> References: <20260113034552.62805-1-me@linux.beauty> <697d19fc772ad_f6311008@iweiny-mobl.notmuch> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <697d19fc772ad_f6311008@iweiny-mobl.notmuch> On Fri, Jan 30, 2026 at 02:52:12PM -0600, Ira Weiny wrote: > Li Chen wrote: > > Under heavy concurrent flush traffic, virtio-pmem can overflow its request > > virtqueue (req_vq): virtqueue_add_sgs() starts returning -ENOSPC and the > > driver logs "no free slots in the virtqueue". Shortly after that the > > device enters VIRTIO_CONFIG_S_NEEDS_RESET and flush requests fail with > > "virtio pmem device needs a reset". > > > > Serialize virtio_pmem_flush() with a per-device mutex so only one flush > > request is in-flight at a time. This prevents req_vq descriptor overflow > > under high concurrency. > > > > Reproducer (guest with virtio-pmem): > > - mkfs.ext4 -F /dev/pmem0 > > - mount -t ext4 -o dax,noatime /dev/pmem0 /mnt/bench > > - fio: ioengine=io_uring rw=randwrite bs=4k iodepth=64 numjobs=64 > > direct=1 fsync=1 runtime=30s time_based=1 > > I don't see this error. > > > 13:28:50 > cat foo.fio > # test http://lore.kernel.org/20260113034552.62805-1-me@linux.beauty > > [global] > filename=/mnt/bench/foo > ioengine=io_uring > size=1G > bs=4K > iodepth=64 > numjobs=64 > direct=1 > fsync=1 > runtime=30s > time_based=1 > > [rand-write] > rw=randwrite > > > It's possible I'm doing something wrong. Can you share your qemu cmdline > or more details on the bug yall see. > > > - dmesg: "no free slots in the virtqueue" > > "virtio pmem device needs a reset" > > > > Fixes: 6e84200c0a29 ("virtio-pmem: Add virtio pmem driver") > > Signed-off-by: Li Chen > > --- > > drivers/nvdimm/nd_virtio.c | 15 +++++++++++---- > > drivers/nvdimm/virtio_pmem.c | 1 + > > drivers/nvdimm/virtio_pmem.h | 4 ++++ > > 3 files changed, 16 insertions(+), 4 deletions(-) > > > > diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c > > index c3f07be4aa22..827a17fe7c71 100644 > > --- a/drivers/nvdimm/nd_virtio.c > > +++ b/drivers/nvdimm/nd_virtio.c > > @@ -44,19 +44,24 @@ static int virtio_pmem_flush(struct nd_region *nd_region) > > unsigned long flags; > > int err, err1; > > > > + might_sleep(); for that matter might_sleep not really needed near mutex_lock. > > + mutex_lock(&vpmem->flush_lock); > > Assuming this does fix a bug I'd rather use guard here. > > guard(mutex)(&vpmem->flush_lock); > > Then skip all the gotos and out_unlock stuff. > > Also, does this affect performance at all? > > Ira > > > + > > /* > > * Don't bother to submit the request to the device if the device is > > * not activated. > > */ > > if (vdev->config->get_status(vdev) & VIRTIO_CONFIG_S_NEEDS_RESET) { > > dev_info(&vdev->dev, "virtio pmem device needs a reset\n"); > > - return -EIO; > > + err = -EIO; > > + goto out_unlock; > > } > > > > - might_sleep(); > > req_data = kmalloc(sizeof(*req_data), GFP_KERNEL); > > - if (!req_data) > > - return -ENOMEM; > > + if (!req_data) { > > + err = -ENOMEM; > > + goto out_unlock; > > + } > > > > req_data->done = false; > > init_waitqueue_head(&req_data->host_acked); > > @@ -103,6 +108,8 @@ static int virtio_pmem_flush(struct nd_region *nd_region) > > } > > > > kfree(req_data); > > +out_unlock: > > + mutex_unlock(&vpmem->flush_lock); > > return err; > > }; > > [snip]