Re: [Virtio-fs] [PATCH][RFC] Support multiqueue mode by setting cpu affinity

[piaojun <piaojun@huawei.com>]

On 2019/8/21 23:38, Stefan Hajnoczi wrote:
> On Fri, Aug 09, 2019 at 02:04:54PM +0800, piaojun wrote:
>> Set cpu affinity for each queue in multiqueue mode to improve the iops
>> performance.
>>
>> >From my test, the iops is increased by adding multiqueues as below,
>> but it has not achieved my expect yet due to some reason. So I'm
>> considering if we could drop some locks when operating vq as it is
>> binded to one vCPU. I'm very glad to have a discuss with other
>> developers.
>>
>> Further more, I modified virtiofsd to support multiqueue which just for
>> testing.
>>
>> Test Environment:
>> Guest configuration:
>> 8 vCPU
>> 8GB RAM
>> Linux 5.1 (vivek-aug-06-2019)
>>
>> Host configuration:
>> Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (8 cores x 4 threads)
>> 32GB RAM
>> Linux 3.10.0
>> EXT4 + 4G Ramdisk
>>
>> ---
>> Single-queue:
>> # fio -direct=1 -time_based -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=1G -numjob=8 -runtime=30 -group_reporting -name=file -filename=/mnt/virtiofs/file
>> file: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> ...
>> fio-2.13
>> Starting 8 processes
>> Jobs: 8 (f=8): [w(8)] [100.0% done] [0KB/316.5MB/0KB /s] [0/81.2K/0 iops] [eta 00m:00s]
>> file: (groupid=0, jobs=8): err= 0: pid=5808: Fri Aug  9 20:35:22 2019
>>   write: io=9499.9MB, bw=324251KB/s, iops=81062, runt= 30001msec
>>
>> Multi-queues:
>> # fio -direct=1 -time_based -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=1G -numjob=8 -runtime=30 -group_reporting -name=file -filename=/mnt/virtiofs/file
>> file: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> ...
>> fio-2.13
>> Starting 8 processes
>> Jobs: 8 (f=8): [w(8)] [100.0% done] [0KB/444.6MB/0KB /s] [0/114K/0 iops] [eta 00m:00s]
>> file: (groupid=0, jobs=8): err= 0: pid=5704: Fri Aug  9 20:38:47 2019
>>   write: io=12967MB, bw=442582KB/s, iops=110645, runt= 30001msec
>> ---
> 
> How does the same fio command-line perform on the host when bound to 8
> CPUs?

fio has great performance on host side, so the bottleneck should be at virtiofsd.

---
Run status group 0 (all jobs):
  WRITE: bw=12.7GiB/s (13.6GB/s), 12.7GiB/s-12.7GiB/s (13.6GB/s-13.6GB/s), io=381GiB (409GB), run=30001-30001msec

> 
> What about the virtiofsd changes?  Did you implement host CPU affinity
> for the virtqueue processing threads and their workqueues?
> 
> I wonder if numbers are better if you use 8 files instead of 1 file.
> 
I implement host CPU affinity and re-design the testcase with 8 files,
the result looks better:

---
[global]
runtime=30
time_based
group_reporting
direct=1
bs=1M
size=1G
ioengine=libaio
rw=write
numjobs=8
iodepth=128
thread=1

[file1]
filename=/mnt/virtiofs/file1
numjobs=1
[file2]
filename=/mnt/virtiofs/file2
numjobs=1
[file3]
filename=/mnt/virtiofs/file3
numjobs=1
[file4]
filename=/mnt/virtiofs/file4
numjobs=1
[file5]
filename=/mnt/virtiofs/file5
numjobs=1
[file6]
filename=/mnt/virtiofs/file6
numjobs=1
[file7]
filename=/mnt/virtiofs/file7
numjobs=1
[file8]
filename=/mnt/virtiofs/file8
numjobs=1

Single-Queue:
Jobs: 8 (f=8): [W(8)] [100.0% done] [0KB/1594MB/0KB /s] [0/1594/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=8): err= 0: pid=6379: Mon Aug 26 16:24:10 2019
  write: io=46676MB, bw=1555.6MB/s, iops=1555, runt= 30007msec

Multi-Queues(8):
Jobs: 8 (f=8): [W(8)] [100.0% done] [0KB/4064MB/0KB /s] [0/4064/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=8): err= 0: pid=5785: Mon Aug 26 16:26:46 2019
  write: io=115421MB, bw=3847.2MB/s, iops=3847, runt= 30002msec

I write a draft patch for virtiofsd, but the sandbox make it hard to
set affinity for each vq, as _SC_NPROCESSORS_ONLN always equals 1. So I
just delete the related code for testing. Maybe we could create a
thread pool before setup_sandbox() or some effective way. I'm glad to
help finding out the solution.

Thanks,
Jun

---
 contrib/virtiofsd/fuse_virtio.c    | 23 ++++++++++++++++++-----
 contrib/virtiofsd/passthrough_ll.c |  4 ++--
 contrib/virtiofsd/seccomp.c        |  2 ++
 3 files changed, 22 insertions(+), 7 deletions(-)

diff --git a/contrib/virtiofsd/fuse_virtio.c b/contrib/virtiofsd/fuse_virtio.c
index bd50723..efc4ba7 100644
--- a/contrib/virtiofsd/fuse_virtio.c
+++ b/contrib/virtiofsd/fuse_virtio.c
@@ -748,8 +748,11 @@ static void fv_queue_set_started(VuDev *dev, int qidx, bool started)
 {
         struct fv_VuDev *vud = container_of(dev, struct fv_VuDev, dev);
         struct fv_QueueInfo *ourqi;
+        cpu_set_t mask;
+        int num = sysconf(_SC_NPROCESSORS_ONLN);

-        fuse_info("%s: qidx=%d started=%d\n", __func__, qidx, started);
+        fuse_info("%s: nqueues %lu, qidx=%d, started=%d, cpunum %d\n",
+                  __func__, vud->nqueues, qidx, started, num);
         assert(qidx>=0);

         /*
@@ -759,9 +762,9 @@ static void fv_queue_set_started(VuDev *dev, int qidx, bool started)
          * races yet.
          */
         if (qidx > 1) {
-                fuse_err("%s: multiple request queues not yet implemented, please only configure 1 request queue\n",
-                         __func__);
-                exit(EXIT_FAILURE);
+                //fuse_err("%s: multiple request queues not yet implemented, please only configure 1 request queue\n",
+                //         __func__);
+                //exit(EXIT_FAILURE);
         }

         if (started) {
@@ -798,6 +801,16 @@ static void fv_queue_set_started(VuDev *dev, int qidx, bool started)
                                  __func__, qidx);
                         assert(0);
                 }
+		if (qidx > 0) {
+			fuse_info("%s: thread[%ld], set CPU[%d] affinity for vq[%d]\n", __func__, ourqi->thread, qidx, qidx);
+			/* set CPU affinity for vqs */
+			CPU_ZERO(&mask);
+			CPU_SET(qidx, &mask);
+			if (pthread_setaffinity_np(ourqi->thread, sizeof(mask), &mask) < 0) {
+				fuse_err("%s: Failed to setaffinity for vq[%d]\n", __func__, qidx);
+                        	assert(0);
+			}
+		}
         } else {
                 int ret;
                 assert(qidx < vud->nqueues);
@@ -962,7 +975,7 @@ int virtio_session_mount(struct fuse_session *se)
         se->virtio_dev = calloc(sizeof(struct fv_VuDev), 1);
         se->virtio_dev->se = se;
         pthread_rwlock_init(&se->virtio_dev->vu_dispatch_rwlock, NULL);
-        vu_init(&se->virtio_dev->dev, 2, se->vu_socketfd,
+        vu_init(&se->virtio_dev->dev, 16, se->vu_socketfd,
                 fv_panic,
                 fv_set_watch, fv_remove_watch,
                 &fv_iface);
diff --git a/contrib/virtiofsd/passthrough_ll.c b/contrib/virtiofsd/passthrough_ll.c
index ca11764..7eabe73 100644
--- a/contrib/virtiofsd/passthrough_ll.c
+++ b/contrib/virtiofsd/passthrough_ll.c
@@ -2773,7 +2773,7 @@ static void setup_root(struct lo_data *lo, struct lo_inode *root)
 	int fd, res;
 	struct stat stat;

-	fd = open("/", O_PATH);
+	fd = open(lo->source, O_PATH);
 	if (fd == -1)
 		err(1, "open(%s, O_PATH)", lo->source);

@@ -2990,7 +2990,7 @@ int main(int argc, char *argv[])
 	/* Must be after daemonize to get the right /proc/self/fd */
 	setup_proc_self_fd(&lo);

-	setup_sandbox(&lo, opts.syslog);
+	//setup_sandbox(&lo, opts.syslog);

 	setup_root(&lo, &lo.root);

diff --git a/contrib/virtiofsd/seccomp.c b/contrib/virtiofsd/seccomp.c
index 3b92c6e..e9f0737 100644
--- a/contrib/virtiofsd/seccomp.c
+++ b/contrib/virtiofsd/seccomp.c
@@ -82,6 +82,8 @@ static const int syscall_whitelist[] = {
 	SCMP_SYS(writev),
 	SCMP_SYS(capget),
 	SCMP_SYS(capset),
+	SCMP_SYS(sched_setaffinity),
+	SCMP_SYS(sched_getaffinity),
 };

 /* Syscalls used when --syslog is enabled */
--