From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=55326 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P8Wi6-0002B3-AG for qemu-devel@nongnu.org; Wed, 20 Oct 2010 07:19:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P8Wi4-0002RW-JQ for qemu-devel@nongnu.org; Wed, 20 Oct 2010 07:19:58 -0400 Received: from mail-gy0-f173.google.com ([209.85.160.173]:44733) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P8Wi4-0002RH-G4 for qemu-devel@nongnu.org; Wed, 20 Oct 2010 07:19:56 -0400 Received: by gyb11 with SMTP id 11so2061345gyb.4 for ; Wed, 20 Oct 2010 04:19:55 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20101019174350.16514.66426.stgit@localhost6.localdomain6> References: <20101019173946.16514.62027.stgit@localhost6.localdomain6> <20101019174350.16514.66426.stgit@localhost6.localdomain6> Date: Wed, 20 Oct 2010 12:19:55 +0100 Message-ID: Subject: Re: [Qemu-devel] [PATCH 3/3] Add helper functions for virtio-9p to use threadlets From: Stefan Hajnoczi Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Arun R Bharadwaj Cc: qemu-devel@nongnu.org On Tue, Oct 19, 2010 at 6:43 PM, Arun R Bharadwaj wrote: > From: Gautham R Shenoy > > Add helper functions to enable virtio-9p make use of the threadlets > infrastructure for offloading blocking tasks such as making posix calls o= n > to the helper threads and handle the post_posix_operations() from the > context of the iothread. This frees the vcpu thread to process any other = guest > operations while the processing of v9fs_io is in progress. > > Signed-off-by: Gautham R Shenoy > Signed-off-by: Sripathi Kodi > Signed-off-by: Arun R Bharadwaj > --- > =A0hw/virtio-9p.c =A0 =A0 | =A0165 ++++++++++++++++++++++++++++++++++++++= ++++++++++++++ > =A0posix-aio-compat.c | =A0 33 +++------- > =A0qemu-threadlets.c =A0| =A0 21 +++++++ > =A0qemu-threadlets.h =A0| =A0 =A01 > =A0vl.c =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A03 + > =A05 files changed, 200 insertions(+), 23 deletions(-) I wish --enable-io-thread was the default and only model. The signals and pipes are ugly and should be hidden behind a QEMU eventfd abstraction, which would also reduce the code duplication between posix-aio-compat.c and virtio-9p. I'm not asking you to do this but I hope we'll get there eventually. Does anyone know why non-io-thread still exists and is used by default? > diff --git a/hw/virtio-9p.c b/hw/virtio-9p.c > index a871685..174300d 100644 > --- a/hw/virtio-9p.c > +++ b/hw/virtio-9p.c > @@ -18,6 +18,7 @@ > =A0#include "fsdev/qemu-fsdev.h" > =A0#include "virtio-9p-debug.h" > =A0#include "virtio-9p-xattr.h" > +#include "qemu-threadlets.h" > > =A0int debug_9p_pdu; > > @@ -33,6 +34,146 @@ enum { > =A0 =A0 Oappend =3D 0x80, > =A0}; > > +struct v9fs_post_op { > + =A0 =A0QTAILQ_ENTRY(v9fs_post_op) node; > + =A0 =A0void (*func)(void *arg); > + =A0 =A0void *arg; > +}; > + > +static struct { > + =A0 =A0int rfd; > + =A0 =A0int wfd; > + =A0 =A0QemuMutex lock; > + =A0 =A0QTAILQ_HEAD(, v9fs_post_op) post_op_list; > +} v9fs_async_struct; > + > +static void die2(int err, const char *what) > +{ > + =A0 =A0fprintf(stderr, "%s failed: %s\n", what, strerror(err)); > + =A0 =A0abort(); > +} > + > +static void die(const char *what) > +{ > + =A0 =A0die2(errno, what); > +} > + > +#define ASYNC_MAX_PROCESS =A0 5 What does this constant define? I think it is an arbitrary limit on the amount of work you want to do per v9fs_process_post_ops() call? > + > +/** > + * v9fs_process_post_ops: Process any pending v9fs_post_posix_operation > + * @arg: Not used. > + * > + * This function serves as a callback to the iothread to be called into = whenever > + * the v9fs_async_struct.wfd is written into. This thread goes through t= he list > + * of v9fs_post_posix_operations() and executes them. In the process, it= might > + * queue more job on the asynchronous thread pool. > + */ > +static void v9fs_process_post_ops(void *arg) > +{ > + =A0 =A0int count =3D 0; > + =A0 =A0struct v9fs_post_op *post_op; > + =A0 =A0int ret; > + =A0 =A0char byte; > + > + =A0 =A0qemu_mutex_lock(&v9fs_async_struct.lock); > + =A0 =A0do { > + =A0 =A0 =A0 =A0ret =3D read(v9fs_async_struct.rfd, &byte, sizeof(byte))= ; > + =A0 =A0} while (ret >=3D 0 && errno !=3D EAGAIN); ret >=3D 0 && errno !=3D EAGAIN looks odd to me. Should && be ||? > + > + =A0 =A0for (count =3D 0; count < ASYNC_MAX_PROCESS; count++) { > + =A0 =A0 =A0 =A0if (QTAILQ_EMPTY(&(v9fs_async_struct.post_op_list))) { > + =A0 =A0 =A0 =A0 =A0 break; > + =A0 =A0 =A0 =A0} > + =A0 =A0 =A0 =A0post_op =3D QTAILQ_FIRST(&(v9fs_async_struct.post_op_lis= t)); > + =A0 =A0 =A0 =A0QTAILQ_REMOVE(&(v9fs_async_struct.post_op_list), post_op= , node); > + > + =A0 =A0 =A0 =A0qemu_mutex_unlock(&v9fs_async_struct.lock); > + =A0 =A0 =A0 =A0post_op->func(post_op->arg); > + =A0 =A0 =A0 =A0qemu_free(post_op); > + =A0 =A0 =A0 =A0qemu_mutex_lock(&v9fs_async_struct.lock); > + =A0 =A0} > + =A0 =A0qemu_mutex_unlock(&v9fs_async_struct.lock); > +} > + > +/** > + * v9fs_async_signal: Inform the io-thread of completion of async job. > + * > + * This function is used to inform the iothread that a particular > + * async-operation pertaining to v9fs has been completed and that the io= thread > + * can handle the v9fs_post_posix_operation. > + * > + * This is based on the aio_signal_handler > + */ > +static inline void v9fs_async_signal(void) > +{ > + =A0 =A0char byte =3D 0; > + =A0 =A0ssize_t ret; > + =A0 =A0int tries =3D 0; > + > + =A0 =A0qemu_mutex_lock(&v9fs_async_struct.lock); > + =A0 =A0do { > + =A0 =A0 =A0 =A0assert(tries !=3D 100); > + =A0 =A0 =A0 ret =3D write(v9fs_async_struct.wfd, &byte, sizeof(byte)); > + =A0 =A0 =A0 =A0tries++; > + =A0 =A0} while (ret < 0 && errno =3D=3D EAGAIN); > + =A0 =A0qemu_mutex_unlock(&v9fs_async_struct.lock); > + > + =A0 =A0if (ret < 0 && errno !=3D EAGAIN) > + =A0 =A0 =A0 =A0die("write() in v9fs"); > + > + =A0 =A0if (kill(getpid(), SIGUSR2)) die("kill failed"); > +} > + > +/** > + * v9fs_async_helper_done: Marks the completion of the v9fs_async job > + * @func: v9fs_post_posix_func() for post-processing invoked in the cont= ext of > + * =A0 =A0 =A0 =A0the io-thread > + * @arg: Argument to func. > + * > + * This function is called from the context of one of the asynchronous t= hreads > + * in the thread pool. This is called when the asynchronous thread has f= inished > + * executing a v9fs_posix_operation. It's purpose is to initiate the pro= cess of > + * informing the io-thread that the v9fs_posix_operation has completed. > + */ > +static void v9fs_async_helper_done(void (*func)(void *arg), void *arg) > +{ > + =A0 =A0struct v9fs_post_op *post_op; > + > + =A0 =A0post_op =3D qemu_mallocz(sizeof(*post_op)); > + =A0 =A0post_op->func =3D func; > + =A0 =A0post_op->arg =3D arg; > + > + =A0 =A0qemu_mutex_lock(&v9fs_async_struct.lock); > + =A0 =A0QTAILQ_INSERT_TAIL(&(v9fs_async_struct.post_op_list), post_op, n= ode); > + =A0 =A0qemu_mutex_unlock(&v9fs_async_struct.lock); > + > + =A0 =A0v9fs_async_signal(); > +} > + > +/** > + * v9fs_do_async_posix: Offload v9fs_posix_operation onto async thread. > + * @vs: V9fsOPState variable for the OP operation. > + * @posix_fn: The posix function which has to be offloaded onto async th= read. > + * @post_fn_ptr: Address of the location to hold the post_fn correspondi= ng to > + * the posix_fn > + * @post_fn: The post processing function corresponding to the posix_fn. > + * > + * This function is a helper to offload posix_operation on to the asynch= ronous > + * thread pool. It sets up the associations with the post_function that = needs to > + * be invoked by from the context of the iothread once the posix_fn has = been > + * executed. > + */ > +static void v9fs_do_async_posix(ThreadletWork *work , > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0void (*p= osix_fn)(ThreadletWork *work), > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0void (**= post_fn_ptr)(void *arg), > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0void (*p= ost_fn)(void *arg)) > +{ > + =A0 =A0*post_fn_ptr =3D post_fn; > + =A0 =A0work->func =3D posix_fn; > + =A0 =A0submit_threadletwork(work); > +} > + > =A0static int omode_to_uflags(int8_t mode) > =A0{ > =A0 =A0 int ret =3D 0; > @@ -3639,7 +3780,7 @@ VirtIODevice *virtio_9p_init(DeviceState *dev, V9fs= Conf *conf) > =A0 =A0 int i, len; > =A0 =A0 struct stat stat; > =A0 =A0 FsTypeEntry *fse; > - > + =A0 =A0int fds[2]; > > =A0 =A0 s =3D (V9fsState *)virtio_common_init("virtio-9p", > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 V= IRTIO_ID_9P, > @@ -3722,5 +3863,27 @@ VirtIODevice *virtio_9p_init(DeviceState *dev, V9f= sConf *conf) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 s->tag_len; > =A0 =A0 s->vdev.get_config =3D virtio_9p_get_config; > > + =A0 =A0if (qemu_pipe(fds) =3D=3D -1) { > + =A0 =A0 =A0 =A0fprintf(stderr, "failed to create fd's for virtio-9p\n")= ; > + =A0 =A0 =A0 =A0exit(1); > + =A0 =A0} > + > + =A0 =A0v9fs_async_struct.rfd =3D fds[0]; > + =A0 =A0v9fs_async_struct.wfd =3D fds[1]; > + > + =A0 =A0printf("v9fs: rfd: %d\n", v9fs_async_struct.rfd); > + =A0 =A0printf("v9fs: wfd: %d\n", v9fs_async_struct.wfd); This looks like debug code. > + > + =A0 =A0fcntl(fds[0], F_SETFL, O_NONBLOCK); > + =A0 =A0fcntl(fds[1], F_SETFL, O_NONBLOCK); > + > + =A0 =A0qemu_set_fd_handler(fds[0], v9fs_process_post_ops, NULL, NULL); > + =A0 =A0QTAILQ_INIT(&v9fs_async_struct.post_op_list); > + =A0 =A0qemu_mutex_init(&(v9fs_async_struct.lock)); > + =A0 =A0/* Create async queue. */ Not sure what the purpose of this comment is. > + > + =A0 =A0(void)v9fs_do_async_posix; > + =A0 =A0(void)v9fs_async_helper_done; These will disappear once there is a user for these functions? > + > =A0 =A0 return &s->vdev; > =A0} > diff --git a/posix-aio-compat.c b/posix-aio-compat.c > index 6977c18..dbf9321 100644 > --- a/posix-aio-compat.c > +++ b/posix-aio-compat.c > @@ -261,6 +261,8 @@ static ssize_t handle_aiocb_rw(struct qemu_paiocb *ai= ocb) > =A0 =A0 return nbytes; > =A0} > > +static PosixAioState *posix_aio_state; > + > =A0static void aio_thread(ThreadletWork *work) > =A0{ > =A0 =A0 pid_t pid; > @@ -289,6 +291,15 @@ static void aio_thread(ThreadletWork *work) > > =A0 =A0 aiocb->ret =3D ret; > > + =A0 =A0if (posix_aio_state) { The posix_aio_state check was necessary to guard the signal handler before posix aio was fully set up. In aio_thread() posix_aio_state must always be set up. There's no need to check. Stefan