[Qemu-devel] Re: [RFC] vhost-blk implementation

[Badari Pulavarty <pbadari@us.ibm.com>]

Michael S. Tsirkin wrote:
> On Tue, Mar 23, 2010 at 12:55:07PM -0700, Badari Pulavarty wrote:
>   
>> Michael S. Tsirkin wrote:
>>     
>>> On Tue, Mar 23, 2010 at 10:57:33AM -0700, Badari Pulavarty wrote:
>>>   
>>>       
>>>> Michael S. Tsirkin wrote:
>>>>     
>>>>         
>>>>> On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote:
>>>>>         
>>>>>           
>>>>>> Write Results:
>>>>>> ==============
>>>>>>
>>>>>> I see degraded IO performance when doing sequential IO write
>>>>>> tests with vhost-blk compared to virtio-blk.
>>>>>>
>>>>>> # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct
>>>>>>
>>>>>> I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with
>>>>>> vhost-blk. Wondering why ?
>>>>>>             
>>>>>>             
>>>>> Try to look and number of interrupts and/or number of exits.
>>>>>         
>>>>>           
>>>> I checked interrupts and IO exits - there is no major noticeable   
>>>> difference between
>>>> vhost-blk and virtio-blk scenerios.
>>>>     
>>>>         
>>>>> It could also be that you are overrunning some queue.
>>>>>
>>>>> I don't see any exit mitigation strategy in your patch:
>>>>> when there are already lots of requests in a queue, it's usually
>>>>> a good idea to disable notifications and poll the
>>>>> queue as requests complete. That could help performance.
>>>>>         
>>>>>           
>>>> Do you mean poll eventfd for new requests instead of waiting for new  
>>>> notifications ?
>>>> Where do you do that in vhost-net code ?
>>>>     
>>>>         
>>> vhost_disable_notify does this.
>>>
>>>   
>>>       
>>>> Unlike network socket, since we are dealing with a file, there is no  
>>>> ->poll support for it.
>>>> So I can't poll for the data. And also, Issue I am having is on the   
>>>> write() side.
>>>>     
>>>>         
>>> Not sure I understand.
>>>
>>>   
>>>       
>>>> I looked at it some more - I see 512K write requests on the
>>>> virtio-queue  in both vhost-blk and virtio-blk cases. Both qemu or
>>>> vhost is doing synchronous  writes to page cache (there is no write
>>>> batching in qemu that is affecting this  case).  I still puzzled on
>>>> why virtio-blk outperforms vhost-blk.
>>>>
>>>> Thanks,
>>>> Badari
>>>>     
>>>>         
>>> If you say the number of requests is the same, we are left with:
>>> - requests are smaller for some reason?
>>> - something is causing retries?
>>>   
>>>       
>> No. IO requests sizes are exactly same (512K) in both cases. There are  
>> no retries or
>> errors in both cases. One thing I am not clear is - for some reason  
>> guest kernel
>> could push more data into virtio-ring in case of virtio-blk vs  
>> vhost-blk. Is this possible ?
>> Does guest gets to run much sooner in virtio-blk case than vhost-blk ?  
>> Sorry, if its dumb question -
>> I don't understand  all the vhost details :(
>>
>> Thanks,
>> Badari
>>     
>
> BTW, did you put the backend in non-blocking mode?
> As I said, vhost net passes non-blocking flag to
> socket backend, but vfs_write/read that you use does
> not have an option to do this.
>
> So we'll need to extend the backend to fix that,
> but just for demo purposes, you could set non-blocking
> mode on the file from userspace.
>
>   
Michael,

Atleast I understand why the performance difference now.. My debug
code is changed the behaviour of virtio-blk which confused me.

1) virtio-blk is able to batch up writes from various requests.
2) virtio-blk offloads the writes to different thread

Where as vhost-blk, I do each request syncrhonously. Hence
the difference. You are right - i have to make backend async.
I will working on handing off work to in-kernel threads.
I am not sure about IO completion handling and calling
vhost_add_used_and_signal() out of context. But let
me take a stab at it and ask your help if I run into
issues.

Thanks,
Badari