[Virtio-fs] Achieving parallelism in virtiofsd

[Virtio-fs] Achieving parallelism in virtiofsd

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1361 bytes --]

Hi,
Here are my plans for achieving parallelism in virtiofsd.  This will
improve performance for workloads that keep more than one request in
flight at a time.

Today virtiofsd performance is limited because it only processes 1
request at a time.  This can be improved in two independent ways:
parallel request processing and multiqueue.

Parallel request processing means working on more than one request at a
time.  A request that blocks should not prevent the next request from
executing.  The FUSE protocol is asynchronous so it's just a question of
adjusting virtiofsd.

Multiqueue means providing several request virtqueues instead of just
one.  This can be used with CPU and NUMA pinning so that request
processing takes place on a core and NUMA node.  Better locality can
result in higher performance.

virtiofsd needs to offer both of these features.  The model I'm
proposing is one thread per virtqueue which distributes requests to a
thread pool for execution.  Each virtqueue thread and its thread pool
can be bound to a subset of CPUs.

Separate optimizations such as virtqueue polling could be added later to
reduce latency.

I plan to use the glib thread pool, which offers the basic functionality
that virtiofsd requires.  In the process of this work I will also audit
and fix passthrough_ll.c's thread-safety.

Feedback is appreciated!

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[piaojun]

Hi Stefan,

>From your description, virtiofsd does the same work as SPDK, and I
wonder if we could adapt it to SPDK-vhost-fs which is widely used.

Thanks,
Jun

On 2019/7/4 18:46, Stefan Hajnoczi wrote:
> Hi,
> Here are my plans for achieving parallelism in virtiofsd.  This will
> improve performance for workloads that keep more than one request in
> flight at a time.
> 
> Today virtiofsd performance is limited because it only processes 1
> request at a time.  This can be improved in two independent ways:
> parallel request processing and multiqueue.
> 
> Parallel request processing means working on more than one request at a
> time.  A request that blocks should not prevent the next request from
> executing.  The FUSE protocol is asynchronous so it's just a question of
> adjusting virtiofsd.
> 
> Multiqueue means providing several request virtqueues instead of just
> one.  This can be used with CPU and NUMA pinning so that request
> processing takes place on a core and NUMA node.  Better locality can
> result in higher performance.
> 
> virtiofsd needs to offer both of these features.  The model I'm
> proposing is one thread per virtqueue which distributes requests to a
> thread pool for execution.  Each virtqueue thread and its thread pool
> can be bound to a subset of CPUs.
> 
> Separate optimizations such as virtqueue polling could be added later to
> reduce latency.
> 
> I plan to use the glib thread pool, which offers the basic functionality
> that virtiofsd requires.  In the process of this work I will also audit
> and fix passthrough_ll.c's thread-safety.
> 
> Feedback is appreciated!
> 
> Stefan
> 
> 
> 
> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://www.redhat.com/mailman/listinfo/virtio-fs
> 

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

On Fri, Jul 05, 2019 at 12:43:16PM +0800, piaojun wrote:
> From your description, virtiofsd does the same work as SPDK, and I
> wonder if we could adapt it to SPDK-vhost-fs which is widely used.

I don't see 9P code in spdk.git/master.  How does the guest access
SPDK's vhost-fs device and where is the code?

A quick look at blobfs suggests additional work is needed to introduce
the concept of inodes instead of performing path name traversal on most
operations.  This also changes the semantics of the file system since
POSIX files can still be accessed after they are unlinked or renamed.

In theory SPDK can implement the vhost-user-fs device interface and
replace virtiofsd but SPDK needs a POSIX file system implementation
first.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[Peng Tao]

On Fri, Jul 5, 2019 at 4:04 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Fri, Jul 05, 2019 at 12:43:16PM +0800, piaojun wrote:
> > From your description, virtiofsd does the same work as SPDK, and I
> > wonder if we could adapt it to SPDK-vhost-fs which is widely used.
>
> I don't see 9P code in spdk.git/master.  How does the guest access
> SPDK's vhost-fs device and where is the code?
>
> A quick look at blobfs suggests additional work is needed to introduce
> the concept of inodes instead of performing path name traversal on most
> operations.  This also changes the semantics of the file system since
> POSIX files can still be accessed after they are unlinked or renamed.
>
> In theory SPDK can implement the vhost-user-fs device interface and
> replace virtiofsd but SPDK needs a POSIX file system implementation
> first.
Intel has presented virtiofsd+blobfs+SPDK in KubeCon Shanghai last
month. As you have spotted, the main problem is on blobfs side and
virtio-fs DAX sharing is not working because of blobfs limitation.

Cheers,
Tao
-- 
Into something rich and strange.

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[Liu, Changpeng]

> -----Original Message-----
> From: virtio-fs-bounces@redhat.com [mailto:virtio-fs-bounces@redhat.com] On
> Behalf Of Peng Tao
> Sent: Friday, July 5, 2019 4:14 PM
> To: Stefan Hajnoczi <stefanha@redhat.com>
> Cc: virtio-fs@redhat.com
> Subject: Re: [Virtio-fs] Achieving parallelism in virtiofsd
> 
> On Fri, Jul 5, 2019 at 4:04 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > On Fri, Jul 05, 2019 at 12:43:16PM +0800, piaojun wrote:
> > > From your description, virtiofsd does the same work as SPDK, and I
> > > wonder if we could adapt it to SPDK-vhost-fs which is widely used.
> >
> > I don't see 9P code in spdk.git/master.  How does the guest access
> > SPDK's vhost-fs device and where is the code?
> >
> > A quick look at blobfs suggests additional work is needed to introduce
> > the concept of inodes instead of performing path name traversal on most
> > operations.  This also changes the semantics of the file system since
> > POSIX files can still be accessed after they are unlinked or renamed.
> >
> > In theory SPDK can implement the vhost-user-fs device interface and
> > replace virtiofsd but SPDK needs a POSIX file system implementation
> > first.
> Intel has presented virtiofsd+blobfs+SPDK in KubeCon Shanghai last
> month. As you have spotted, the main problem is on blobfs side and
> virtio-fs DAX sharing is not working because of blobfs limitation.
> 
I'm planning to add similar mmap API in SPDK, hopefully it can work in next few weeks.
> Cheers,
> Tao
> --
> Into something rich and strange.
> 
> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://www.redhat.com/mailman/listinfo/virtio-fs

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[piaojun]

On 2019/7/5 16:04, Stefan Hajnoczi wrote:
> On Fri, Jul 05, 2019 at 12:43:16PM +0800, piaojun wrote:
>> From your description, virtiofsd does the same work as SPDK, and I
>> wonder if we could adapt it to SPDK-vhost-fs which is widely used.
> 
> I don't see 9P code in spdk.git/master.  How does the guest access
> SPDK's vhost-fs device and where is the code?

Yes, 9P code is not in spdk yet, and they can comunicate though share
memory. But it needs some extra work.

> 
> A quick look at blobfs suggests additional work is needed to introduce
> the concept of inodes instead of performing path name traversal on most
> operations.  This also changes the semantics of the file system since
> POSIX files can still be accessed after they are unlinked or renamed.
> 
> In theory SPDK can implement the vhost-user-fs device interface and
> replace virtiofsd but SPDK needs a POSIX file system implementation
> first.

Right, SPDK could only support blobfs now, and I wish it supporting any
local file system as backend in the future. SPDK vhost-fs patches is in
developing as below:

https://review.gerrithub.io/c/spdk/spdk/+/449163/5
https://review.gerrithub.io/c/spdk/spdk/+/449162/5

Thanks,
Jun

> 
> Stefan
> 

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[Vivek Goyal]

On Thu, Jul 04, 2019 at 11:46:50AM +0100, Stefan Hajnoczi wrote:
> Hi,
> Here are my plans for achieving parallelism in virtiofsd.  This will
> improve performance for workloads that keep more than one request in
> flight at a time.
> 
> Today virtiofsd performance is limited because it only processes 1
> request at a time.  This can be improved in two independent ways:
> parallel request processing and multiqueue.
> 
> Parallel request processing means working on more than one request at a
> time.  A request that blocks should not prevent the next request from
> executing.  The FUSE protocol is asynchronous so it's just a question of
> adjusting virtiofsd.
> 
> Multiqueue means providing several request virtqueues instead of just
> one.  This can be used with CPU and NUMA pinning so that request
> processing takes place on a core and NUMA node.  Better locality can
> result in higher performance.
> 
> virtiofsd needs to offer both of these features.  The model I'm
> proposing is one thread per virtqueue which distributes requests to a
> thread pool for execution.  Each virtqueue thread and its thread pool
> can be bound to a subset of CPUs.
> 
> Separate optimizations such as virtqueue polling could be added later to
> reduce latency.
> 
> I plan to use the glib thread pool, which offers the basic functionality
> that virtiofsd requires.  In the process of this work I will also audit
> and fix passthrough_ll.c's thread-safety.
> 
> Feedback is appreciated!

Hi Stefan,

I agree that virtiofsd needs to probably offer both the models. It
will be nice if we implement processing multiple parallel requests
on same queue now. And offer multiqueue support later (that will
require kernel changes as well).

/me is hoping that allowing processing multiple requests in parallel
will give us decent performance improvement, and also help solve the
issue of implementing *waiting* remote posix locks.

Thanks
Vivek

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@redhat.com) wrote:
> Hi,
> Here are my plans for achieving parallelism in virtiofsd.  This will
> improve performance for workloads that keep more than one request in
> flight at a time.
> 
> Today virtiofsd performance is limited because it only processes 1
> request at a time.  This can be improved in two independent ways:
> parallel request processing and multiqueue.
> 
> Parallel request processing means working on more than one request at a
> time.  A request that blocks should not prevent the next request from
> executing.  The FUSE protocol is asynchronous so it's just a question of
> adjusting virtiofsd.

Are there any ordering constraints? Or barriers etc?

> Multiqueue means providing several request virtqueues instead of just
> one.  This can be used with CPU and NUMA pinning so that request
> processing takes place on a core and NUMA node.  Better locality can
> result in higher performance.
> 
> virtiofsd needs to offer both of these features.  The model I'm
> proposing is one thread per virtqueue which distributes requests to a
> thread pool for execution.  Each virtqueue thread and its thread pool
> can be bound to a subset of CPUs.
> 
> Separate optimizations such as virtqueue polling could be added later to
> reduce latency.
> 
> I plan to use the glib thread pool, which offers the basic functionality
> that virtiofsd requires.  In the process of this work I will also audit
> and fix passthrough_ll.c's thread-safety.
> 
> Feedback is appreciated!

I think that should work; it probably needs a bit more abstraction for
some concept of a current command; 

I think currently we have that:

  fuse_req_t req
has  fuse_chan *ch
has     fv_QueueInfo *qi
has       VuVirtqElement *qe

(There's also reply_sent and elem_bad_in that relate to the current
request)

and the filesystem code just passes 'req' in to calls where it wants
to return data; with multiple threads you could have multiple
fuse_chan's and thus multiple fv_QueueInfo's; but for a thread pool
you really need to tie the qe directly to the req.

It's a bit structurally differnt from normal fuse, since they just
write on an fd, there's no concept of having to use particularly
storage to return the result of a particular request.

Perhaps the easiest thing would be to keep a hash of virtqelement's
based on the unique id in each request?

Dave

> 
> Stefan


--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3074 bytes --]

On Mon, Jul 08, 2019 at 04:08:22PM +0100, Dr. David Alan Gilbert wrote:
> * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > Hi,
> > Here are my plans for achieving parallelism in virtiofsd.  This will
> > improve performance for workloads that keep more than one request in
> > flight at a time.
> > 
> > Today virtiofsd performance is limited because it only processes 1
> > request at a time.  This can be improved in two independent ways:
> > parallel request processing and multiqueue.
> > 
> > Parallel request processing means working on more than one request at a
> > time.  A request that blocks should not prevent the next request from
> > executing.  The FUSE protocol is asynchronous so it's just a question of
> > adjusting virtiofsd.
> 
> Are there any ordering constraints? Or barriers etc?

FUSE_INIT must be processed before the session is usable for
general-purpose requests.  So there is at least one ordering constraint
in the FUSE protocol :).

Miklos: Can you confirm that fuse.ko does not assume ordering
constraints on general-purpose requests (lookup, create, read, write,
etc)?  For example, does it ever queue up requests and expect them to be
processed in order?

> > Multiqueue means providing several request virtqueues instead of just
> > one.  This can be used with CPU and NUMA pinning so that request
> > processing takes place on a core and NUMA node.  Better locality can
> > result in higher performance.
> > 
> > virtiofsd needs to offer both of these features.  The model I'm
> > proposing is one thread per virtqueue which distributes requests to a
> > thread pool for execution.  Each virtqueue thread and its thread pool
> > can be bound to a subset of CPUs.
> > 
> > Separate optimizations such as virtqueue polling could be added later to
> > reduce latency.
> > 
> > I plan to use the glib thread pool, which offers the basic functionality
> > that virtiofsd requires.  In the process of this work I will also audit
> > and fix passthrough_ll.c's thread-safety.
> > 
> > Feedback is appreciated!
> 
> I think that should work; it probably needs a bit more abstraction for
> some concept of a current command; 
> 
> I think currently we have that:
> 
>   fuse_req_t req
> has  fuse_chan *ch
> has     fv_QueueInfo *qi
> has       VuVirtqElement *qe
> 
> (There's also reply_sent and elem_bad_in that relate to the current
> request)
> 
> and the filesystem code just passes 'req' in to calls where it wants
> to return data; with multiple threads you could have multiple
> fuse_chan's and thus multiple fv_QueueInfo's; but for a thread pool
> you really need to tie the qe directly to the req.
> 
> It's a bit structurally differnt from normal fuse, since they just
> write on an fd, there's no concept of having to use particularly
> storage to return the result of a particular request.
> 
> Perhaps the easiest thing would be to keep a hash of virtqelement's
> based on the unique id in each request?

My plan was 1 fuse_chan per worker thread.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Virtio-fs] Achieving parallelism in virtiofsd

[Miklos Szeredi]

On Tue, Jul 9, 2019 at 10:15 AM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> On Mon, Jul 08, 2019 at 04:08:22PM +0100, Dr. David Alan Gilbert wrote:
> > * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > > Hi,
> > > Here are my plans for achieving parallelism in virtiofsd.  This will
> > > improve performance for workloads that keep more than one request in
> > > flight at a time.
> > >
> > > Today virtiofsd performance is limited because it only processes 1
> > > request at a time.  This can be improved in two independent ways:
> > > parallel request processing and multiqueue.
> > >
> > > Parallel request processing means working on more than one request at a
> > > time.  A request that blocks should not prevent the next request from
> > > executing.  The FUSE protocol is asynchronous so it's just a question of
> > > adjusting virtiofsd.
> >
> > Are there any ordering constraints? Or barriers etc?
>
> FUSE_INIT must be processed before the session is usable for
> general-purpose requests.  So there is at least one ordering constraint
> in the FUSE protocol :).
>
> Miklos: Can you confirm that fuse.ko does not assume ordering
> constraints on general-purpose requests (lookup, create, read, write,
> etc)?  For example, does it ever queue up requests and expect them to be
> processed in order?

Fuse does not assume that requests will be processed in order of queuing.

If request B is dependent on request A, then B is only queued if reply
for A is received.

Thanks,
Miklos