Re: [Qemu-devel] Performance problem and improvement about block drive on NFS shares with libnfs

[Stefan Hajnoczi <stefanha@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2576 bytes --]

On Sat, Apr 01, 2017 at 01:23:46PM +0800, Jaden Liang wrote:
> Hello,
> 
> I ran qemu with drive file via libnfs recently, and found some performance
> problem and an improvement idea.
> 
> I started qemu with 6 drives parameter like nfs://127.0.0.1/dir/vm-disk-x.qcow2
> which linked to a local NFS server, then used iometer in guest machine to test
> the 4K random read or random write IO performance. I found that while the IO
> depth go up, the IOPS hit a bottleneck. I looked into the causes, found that the
> main thread of qemu used 100% CPU. From the perf data, it show the CPU heats are
> send / recv calls in libnfs. By reading the source code of libnfs and qemu block
> drive of nfs.c, libnfs only support single work thread, and the network events
> of nfs interface in qemu are all registered in the epoll of main thread. That is
> the cause why main thread uses 100% CPU.
> 
> After the analysis above, there is an improvement idea comes up. I start a
> thread for every drive while libnfs open drive file, then create an epoll in
> every drive thread to handle all of the network events. I have finished an demo
> modification in block/nfs.c, then rerun iometer in the guest machine, the
> performance increased a lot. Random read IOPS increases almost 100%, random
> write IOPS increases about 68%.
> 
> Test model details
> VM configure: 6 vdisks in 1 VM
> Test tool and parameter: iometer with 4K random read and randwrite
> Backend physical drive: 2 SSDs, 6 vdisks are seperated in 2 SSDs
> 
> Before modified:
> IO Depth           1        2          4           8       16         32
> 4K randread  16659  28387   42932   46868   52108   55760
> 4K randwrite  12212   19456   30447   30574   35788   39015
> 
> After modified:
> IO Depth            1         2          4          8        16          32
> 4K randread   17661   33115   57138   82016   99369   109410
> 4K randwrite  12669   21492   36017   51532   61475   65577
> 
> I could put a up to coding standard patch later. Now I want to get some advise
> about this modification. Is this a reasonable solution to improve performance in
> NFS shares? Or there is another better way?
> 
> Any suggestions would be great! Also please feel free to ask question.

Did you try using -object iothread,id=iothread1 -device
virtio-blk-pci,iothread=iothread1,... to define IOThreads for each
virtio-blk-pci device?

The block/nfs.c code already supports IOThread so you can run multiple
threads and don't need to use 100% CPU in the main loop.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]