[PATCH] virtio_blk: merge S/G list entries by default

[PATCH] virtio_blk: merge S/G list entries by default

[Christoph Hellwig]

Most virtio setups have a fairly limited number of ring entries available.
Enable S/G entry merging by default to fit into less of them.  This restores
the behavior at time of the virtio-blk blk-mq conversion, which was changed
by commit "block: add queue flag for disabling SG merging" which made the
behavior optional, but didn't update the existing drivers to keep their
previous behavior.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/block/virtio_blk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index 0a58140..311b857 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
 	vblk->tag_set.ops = &virtio_mq_ops;
 	vblk->tag_set.queue_depth = virtblk_queue_depth;
 	vblk->tag_set.numa_node = NUMA_NO_NODE;
-	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
+	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
 	vblk->tag_set.cmd_size =
 		sizeof(struct virtblk_req) +
 		sizeof(struct scatterlist) * sg_elems;
-- 
1.9.1

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Paolo Bonzini]

Il 07/09/2014 01:09, Christoph Hellwig ha scritto:
> Most virtio setups have a fairly limited number of ring entries available.

Are you disabling indirect descriptors?  With indirect descriptors entry
merging doesn't buy you any more space, so perhaps you can key the flag
off the availability of VIRTIO_RING_F_INDIRECT_DESC.

Paolo

> Enable S/G entry merging by default to fit into less of them.  This restores
> the behavior at time of the virtio-blk blk-mq conversion, which was changed
> by commit "block: add queue flag for disabling SG merging" which made the
> behavior optional, but didn't update the existing drivers to keep their
> previous behavior.

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Ming Lei]

On Sun, Sep 7, 2014 at 7:09 AM, Christoph Hellwig <hch@lst.de> wrote:
> Most virtio setups have a fairly limited number of ring entries available.
> Enable S/G entry merging by default to fit into less of them.  This restores
> the behavior at time of the virtio-blk blk-mq conversion, which was changed
> by commit "block: add queue flag for disabling SG merging" which made the
> behavior optional, but didn't update the existing drivers to keep their
> previous behavior.

It is a good idea to disable SG merge for vq incapable of indirect because
there are very limited direct descriptors.

For vq capable of indirect, it should be better to not do SG merge at default
because:

- from hypervisor view, no matter how many segments one req has, all are
submitted to host kernel by one syscall, such as readv/io_submit

- host kernel still need to do the same merge again

>From my test(virtio-blk over null_blk), looks enabling SG merge may cause
throughput a little drop(~3%).

>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  drivers/block/virtio_blk.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 0a58140..311b857 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>         vblk->tag_set.ops = &virtio_mq_ops;
>         vblk->tag_set.queue_depth = virtblk_queue_depth;
>         vblk->tag_set.numa_node = NUMA_NO_NODE;
> -       vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
> +       vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
>         vblk->tag_set.cmd_size =
>                 sizeof(struct virtblk_req) +
>                 sizeof(struct scatterlist) * sg_elems;
> --
> 1.9.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


Thanks,
-- 
Ming Lei

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Michael S. Tsirkin]

On Sat, Sep 06, 2014 at 04:09:54PM -0700, Christoph Hellwig wrote:
> Most virtio setups have a fairly limited number of ring entries available.

Seems a bit vague: QEMU at least has pretty large queues.
Which hypervisor do you have in mind?
This could be a gain everywhere if you manage to make descriptors
completely linear, so they fit in a single s/g.
ATM __virtblk_add_req always adds an s/g for the header:
is there a chance linux can pre-allocate a bit of memory
in front of the buffer to stick the header in?

> Enable S/G entry merging by default to fit into less of them.  This restores
> the behavior at time of the virtio-blk blk-mq conversion, which was changed
> by commit "block: add queue flag for disabling SG merging" which made the
> behavior optional, but didn't update the existing drivers to keep their
> previous behavior.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>

OK so this is an optimization patch right?
What kind of performance gain is observed with it?

> ---
>  drivers/block/virtio_blk.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
> index 0a58140..311b857 100644
> --- a/drivers/block/virtio_blk.c
> +++ b/drivers/block/virtio_blk.c
> @@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>  	vblk->tag_set.ops = &virtio_mq_ops;
>  	vblk->tag_set.queue_depth = virtblk_queue_depth;
>  	vblk->tag_set.numa_node = NUMA_NO_NODE;
> -	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
> +	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
>  	vblk->tag_set.cmd_size =
>  		sizeof(struct virtblk_req) +
>  		sizeof(struct scatterlist) * sg_elems;
> -- 
> 1.9.1

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Christoph Hellwig]

On Sun, Sep 07, 2014 at 02:41:53PM +0300, Michael S. Tsirkin wrote:
> > Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> OK so this is an optimization patch right?
> What kind of performance gain is observed with it?

None.  I actually wrote it when the block layer had a bug when dm was
used on top of the !merge case, and I decided to send it out as there had been
no discussion about disabling this by default on the existing blk-mq
drivers.

At least for my qemu/kvm setup it doesn't make a difference either way,
although not doing the cheap merge doesn't like the right kind of optimization
to me.

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Michael S. Tsirkin]

On Sun, Sep 07, 2014 at 08:47:45PM +0200, Christoph Hellwig wrote:
> On Sun, Sep 07, 2014 at 02:41:53PM +0300, Michael S. Tsirkin wrote:
> > > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > 
> > OK so this is an optimization patch right?
> > What kind of performance gain is observed with it?
> 
> None.  I actually wrote it when the block layer had a bug when dm was
> used on top of the !merge case, and I decided to send it out as there had been
> no discussion about disabling this by default on the existing blk-mq
> drivers.
> 
> At least for my qemu/kvm setup it doesn't make a difference either way,
> although not doing the cheap merge doesn't like the right kind of optimization
> to me.

Could you respond to Ming Lei's mail, who benchmarked the patch, please?

-- 
MST

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Paolo Bonzini]

Il 07/09/2014 13:41, Michael S. Tsirkin ha scritto:
> On Sat, Sep 06, 2014 at 04:09:54PM -0700, Christoph Hellwig wrote:
>> Most virtio setups have a fairly limited number of ring entries available.
> 
> Seems a bit vague: QEMU at least has pretty large queues.
> Which hypervisor do you have in mind?
> This could be a gain everywhere if you manage to make descriptors
> completely linear, so they fit in a single s/g.
> ATM __virtblk_add_req always adds an s/g for the header:
> is there a chance linux can pre-allocate a bit of memory
> in front of the buffer to stick the header in?

Nope, the buffer usually comes directly from the page cache and will be
page aligned.

Paolo

>> Enable S/G entry merging by default to fit into less of them.  This restores
>> the behavior at time of the virtio-blk blk-mq conversion, which was changed
>> by commit "block: add queue flag for disabling SG merging" which made the
>> behavior optional, but didn't update the existing drivers to keep their
>> previous behavior.
>>
>> Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> OK so this is an optimization patch right?
> What kind of performance gain is observed with it?
> 
>> ---
>>  drivers/block/virtio_blk.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
>> index 0a58140..311b857 100644
>> --- a/drivers/block/virtio_blk.c
>> +++ b/drivers/block/virtio_blk.c
>> @@ -634,7 +634,7 @@ static int virtblk_probe(struct virtio_device *vdev)
>>  	vblk->tag_set.ops = &virtio_mq_ops;
>>  	vblk->tag_set.queue_depth = virtblk_queue_depth;
>>  	vblk->tag_set.numa_node = NUMA_NO_NODE;
>> -	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
>> +	vblk->tag_set.flags = BLK_MQ_F_SHOULD_MERGE | BLK_MQ_F_SG_MERGE;
>>  	vblk->tag_set.cmd_size =
>>  		sizeof(struct virtblk_req) +
>>  		sizeof(struct scatterlist) * sg_elems;
>> -- 
>> 1.9.1

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Christoph Hellwig]

On Mon, Sep 08, 2014 at 11:18:30AM +0300, Michael S. Tsirkin wrote:
> Could you respond to Ming Lei's mail, who benchmarked the patch, please?

I don't really have any additional data or disagreement, not sure what
I should respond there.

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Paolo Bonzini]

Il 07/09/2014 12:32, Ming Lei ha scritto:
> It is a good idea to disable SG merge for vq incapable of indirect because
> there are very limited direct descriptors.

I think you mean _enabling_ SG merge if indirect descriptors are not there.

> For vq capable of indirect, it should be better to not do SG merge at default
> because:
> 
> - from hypervisor view, no matter how many segments one req has, all are
> submitted to host kernel by one syscall, such as readv/io_submit
> 
> - host kernel still need to do the same merge again

Here we agree.

Paolo

> From my test(virtio-blk over null_blk), looks enabling SG merge may cause
> throughput a little drop(~3%).

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Ming Lei]

On Wed, Sep 10, 2014 at 11:18 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> Il 07/09/2014 12:32, Ming Lei ha scritto:
>> It is a good idea to disable SG merge for vq incapable of indirect because
>> there are very limited direct descriptors.
>
> I think you mean _enabling_ SG merge if indirect descriptors are not there.

You are right, sorry for the typo.

>> For vq capable of indirect, it should be better to not do SG merge at default
>> because:
>>
>> - from hypervisor view, no matter how many segments one req has, all are
>> submitted to host kernel by one syscall, such as readv/io_submit
>>
>> - host kernel still need to do the same merge again
>
> Here we agree.
>
> Paolo
>
>> From my test(virtio-blk over null_blk), looks enabling SG merge may cause
>> throughput a little drop(~3%).
>


Thanks,
-- 
Ming Lei

Re: [PATCH] virtio_blk: merge S/G list entries by default

[Michael S. Tsirkin]

On Mon, Sep 08, 2014 at 10:15:57PM +0200, Christoph Hellwig wrote:
> On Mon, Sep 08, 2014 at 11:18:30AM +0300, Michael S. Tsirkin wrote:
> > Could you respond to Ming Lei's mail, who benchmarked the patch, please?
> 
> I don't really have any additional data or disagreement, not sure what
> I should respond there.

Hmm Ming Lei measured a slowdown with your patch.
Is that expected?