Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter

From: Leon Romanovsky <leon@kernel.org>
To: Max Gurtovoy <mgurtovoy@nvidia.com>
Cc: Chaitanya Kulkarni <ckulkarnilinux@gmail.com>,
	hch@infradead.org, mst@redhat.com,
	virtualization@lists.linux-foundation.org, kvm@vger.kernel.org,
	stefanha@redhat.com, israelr@nvidia.com, nitzanc@nvidia.com,
	oren@nvidia.com, linux-block@vger.kernel.org, axboe@kernel.dk
Subject: Re: [PATCH v3 1/1] virtio-blk: add num_request_queues module parameter
Date: Sun, 5 Sep 2021 16:10:04 +0300	[thread overview]
Message-ID: <YTTBrD/0bHcyfNGm@unreal> (raw)
In-Reply-To: <5aada544-356f-5363-c6e4-6885e9812f82@nvidia.com>

On Sun, Sep 05, 2021 at 02:16:58PM +0300, Max Gurtovoy wrote:
> 
> On 9/5/2021 1:19 PM, Leon Romanovsky wrote:
> > On Sun, Sep 05, 2021 at 12:19:09PM +0300, Max Gurtovoy wrote:
> > > On 9/5/2021 11:49 AM, Chaitanya Kulkarni wrote:
> > > > On 9/5/2021 12:46 AM, Leon Romanovsky wrote:
> > > > > > +static unsigned int num_request_queues;
> > > > > > +module_param_cb(num_request_queues, &queue_count_ops,
> > > > > > &num_request_queues,
> > > > > > +        0644);
> > > > > > +MODULE_PARM_DESC(num_request_queues,
> > > > > > +         "Number of request queues to use for blk device.
> > > > > > Should > 0");
> > > > > > +
> > > > > Won't it limit all virtio block devices to the same limit?
> > > > > 
> > > > > It is very common to see multiple virtio-blk devices on the same system
> > > > > and they probably need different limits.
> > > > > 
> > > > > Thanks
> > > > 
> > > > Without looking into the code, that can be done adding a configfs
> > > > 
> > > > interface and overriding a global value (module param) when it is set
> > > > from
> > > > 
> > > > configfs.
> > > > 
> > > > 
> > > You have many ways to overcome this issue.
> > > 
> > > For example:
> > > 
> > > # ls -l /sys/block/vda/mq/
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 0
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 1
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 2
> > > drwxr-xr-x 18 root root 0 Sep  5 12:14 3
> > > 
> > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/unbind
> > > 
> > > # echo 8 > /sys/module/virtio_blk/parameters/num_request_queues
> > This is global to all virtio-blk devices.
> 
> You define a global module param but you bind/unbind a specific device.
> 
> Do you have a better way to control it ?

One of the possible solutions will be to extend virtio bus to allow
setting of such pre-probe parameters. However I don't know if it is
really worth doing it,

> 
> if the device is already probed, it's too late to change the queue_num.
> 
> 
> > 
> > > # echo virtio0 > /sys/bus/virtio/drivers/virtio_blk/bind
> > > 
> > > # ls -l /sys/block/vda/mq/
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 0
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 1
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 2
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 3
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 4
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 5
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 6
> > > drwxr-xr-x 10 root root 0 Sep  5 12:17 7
> > > 
> > > -Max.
> > > 
> > > 