From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from [140.186.70.92] (port=41090 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1PGEUd-0000oI-L0
	for qemu-devel@nongnu.org; Wed, 10 Nov 2010 12:29:58 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <blauwirbel@gmail.com>) id 1PGEUa-0000aW-FH
	for qemu-devel@nongnu.org; Wed, 10 Nov 2010 12:29:55 -0500
Received: from mail-qw0-f45.google.com ([209.85.216.45]:42688)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <blauwirbel@gmail.com>) id 1PGEUa-0000aE-9R
	for qemu-devel@nongnu.org; Wed, 10 Nov 2010 12:29:52 -0500
Received: by qwf6 with SMTP id 6so956131qwf.4
	for <qemu-devel@nongnu.org>; Wed, 10 Nov 2010 09:29:51 -0800 (PST)
MIME-Version: 1.0
In-Reply-To: <20101110131953.29714.21740.stgit@localhost6.localdomain6>
References: <20101110131822.29714.34137.stgit@localhost6.localdomain6>
	<20101110131953.29714.21740.stgit@localhost6.localdomain6>
From: Blue Swirl <blauwirbel@gmail.com>
Date: Wed, 10 Nov 2010 17:29:30 +0000
Message-ID: <AANLkTik4Va_OEEAofJtaHew4jG6CGRnahwNJNAWLmtj_@mail.gmail.com>
Subject: Re: [Qemu-devel] [PATCH 2/3] Move threadlets infrastructure to
	qemu-threadlets.c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
List-Id: qemu-devel.nongnu.org
List-Unsubscribe: <http://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <http://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Arun R Bharadwaj <arun@linux.vnet.ibm.com>
Cc: qemu-devel@nongnu.org

On Wed, Nov 10, 2010 at 1:19 PM, Arun R Bharadwaj
<arun@linux.vnet.ibm.com> wrote:
> The reason for creating this generic infrastructure is so that other subs=
ystems,
> such as virtio-9p could make use of it for offloading tasks that could bl=
ock.
>
> Signed-off-by: Arun R Bharadwaj <arun@linux.vnet.ibm.com>
> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
> Signed-off-by: Gautham R Shenoy <gautham.shenoy@gmail.com>
> Signed-off-by: Sripathi Kodi <sripathik@in.ibm.com>
> ---
> =C2=A0Makefile.objs =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A01
> =C2=A0docs/async-support.txt | =C2=A0141 ++++++++++++++++++++++++++++++++=
+++++++
> =C2=A0posix-aio-compat.c =C2=A0 =C2=A0 | =C2=A0173 ----------------------=
--------------------------
> =C2=A0qemu-threadlets.c =C2=A0 =C2=A0 =C2=A0| =C2=A0168 +++++++++++++++++=
++++++++++++++++++++++++++++++
> =C2=A0qemu-threadlets.h =C2=A0 =C2=A0 =C2=A0| =C2=A0 48 +++++++++++++
> =C2=A05 files changed, 359 insertions(+), 172 deletions(-)
> =C2=A0create mode 100644 docs/async-support.txt
> =C2=A0create mode 100644 qemu-threadlets.c
> =C2=A0create mode 100644 qemu-threadlets.h
>
> diff --git a/Makefile.objs b/Makefile.objs
> index 3b7ec27..2cf8aba 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -10,6 +10,7 @@ qobject-obj-y +=3D qerror.o
> =C2=A0block-obj-y =3D cutils.o cache-utils.o qemu-malloc.o qemu-option.o =
module.o
> =C2=A0block-obj-y +=3D nbd.o block.o aio.o aes.o osdep.o qemu-config.o
> =C2=A0block-obj-$(CONFIG_POSIX) +=3D qemu-thread.o
> +block-obj-$(CONFIG_POSIX) +=3D qemu-threadlets.o
> =C2=A0block-obj-$(CONFIG_POSIX) +=3D posix-aio-compat.o
> =C2=A0block-obj-$(CONFIG_LINUX_AIO) +=3D linux-aio.o
>
> diff --git a/docs/async-support.txt b/docs/async-support.txt
> new file mode 100644
> index 0000000..9f22b9a
> --- /dev/null
> +++ b/docs/async-support.txt
> @@ -0,0 +1,141 @@
> +=3D=3D How to use the threadlets infrastructure supported in Qemu =3D=3D
> +
> +=3D=3D Threadlets =3D=3D
> +
> +Q.1: What are threadlets ?
> +A.1: Threadlets is an infrastructure within QEMU that allows other subsy=
stems
> + =C2=A0 =C2=A0 to offload possibly blocking work to a queue to be proces=
sed by a pool
> + =C2=A0 =C2=A0 of threads asynchronously.
> +
> +Q.2: When would one want to use threadlets ?
> +A.2: Threadlets are useful when there are operations that can be perform=
ed
> + =C2=A0 =C2=A0 outside the context of the VCPU/IO threads inorder to fre=
e these latter
> + =C2=A0 =C2=A0 to service any other guest requests.
> +
> +Q.3: I have some work that can be executed in an asynchronous context. H=
ow
> + =C2=A0 =C2=A0 should I go about it ?
> +A.3: One could follow the steps listed below:
> +
> + =C2=A0 =C2=A0 - Define a function which would do the asynchronous work.
> + =C2=A0 =C2=A0 =C2=A0 static void my_threadlet_func(ThreadletWork *work)
> + =C2=A0 =C2=A0 =C2=A0 {
> + =C2=A0 =C2=A0 =C2=A0 }
> +
> + =C2=A0 =C2=A0 - Declare an object of type ThreadletWork;
> + =C2=A0 =C2=A0 =C2=A0 ThreadletWork work;
> +
> +
> + =C2=A0 =C2=A0 - Assign a value to the "func" member of ThreadletWork ob=
ject.
> + =C2=A0 =C2=A0 =C2=A0 work.func =3D my_threadlet_func;
> +
> + =C2=A0 =C2=A0 - Submit the threadlet to the global queue.
> + =C2=A0 =C2=A0 =C2=A0 submit_threadletwork(&work);
> +
> + =C2=A0 =C2=A0 - Continue servicing some other guest operations.
> +
> +Q.4: I want to my_threadlet_func to access some non-global data. How do =
I do
> + =C2=A0 =C2=A0 that ?
> +A.4: Suppose you want my_threadlet_func to access some non-global data-o=
bject
> + =C2=A0 =C2=A0 of type myPrivateData. In that case one could follow the =
following steps.
> +
> + =C2=A0 =C2=A0 - Define a member of the type ThreadletWork within myPriv=
ateData.
> + =C2=A0 =C2=A0 =C2=A0 typedef struct MyPrivateData {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ...;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ...;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ...;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ThreadletWork work;
> + =C2=A0 =C2=A0 =C2=A0 } MyPrivateData;
> +
> + =C2=A0 =C2=A0 =C2=A0 MyPrivateData my_data;
> +
> + =C2=A0 =C2=A0 - Initialize myData.work as described in A.3
> + =C2=A0 =C2=A0 =C2=A0 myData.work.func =3D my_threadlet_func;
> + =C2=A0 =C2=A0 =C2=A0 submit_threadletwork(&myData.work);
> +
> + =C2=A0 =C2=A0 - Access the myData object inside my_threadlet_func() usi=
ng container_of
> + =C2=A0 =C2=A0 =C2=A0 primitive
> + =C2=A0 =C2=A0 =C2=A0 static void my_threadlet_func(ThreadletWork *work)
> + =C2=A0 =C2=A0 =C2=A0 {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 myPrivateData *mydata_=
ptr;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mydata_ptr =3D contain=
er_of(work, myPrivateData, work);
> +
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* mydata_ptr now poin=
ts to myData object */
> + =C2=A0 =C2=A0 =C2=A0 }
> +
> +Q.5: Are there any precautions one must take while sharing data with the
> + =C2=A0 =C2=A0 Asynchronous thread-pool ?
> +A.5: Yes, make sure that the helper function of the type my_threadlet_fu=
nc()
> + =C2=A0 =C2=A0 does not access/modify data when it can be accessed or mo=
dified in the
> + =C2=A0 =C2=A0 context of VCPU thread or IO thread. This is because the =
asynchronous
> + =C2=A0 =C2=A0 threads in the pool can run in parallel with the VCPU/IOT=
hreads as shown
> + =C2=A0 =C2=A0 in the figure.
> +
> + =C2=A0 =C2=A0 A typical workflow is as follows:
> +
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0VCPU/IOThread
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | (1)
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 V
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Offload work =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(2)
> + =C2=A0 =C2=A0 =C2=A0|-------> to threadlets ---------------------------=
--> Helper thread
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| (3) =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=
 (4)
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 Handle other Guest re=
quests =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0|
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 V
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| (3) =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Signal the I/O Thread
> + =C2=A0 =C2=A0 =C2=A0|(6) =C2=A0 =C2=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0/
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0V =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Do the post <--=
-------------------------------/
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0processing =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (5)
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| (6)
> + =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0V
> + =C2=A0 =C2=A0 =C2=A0|-Yes------ More async work?
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | (7)
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0No
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .
> +
> + =C2=A0 =C2=A0Hence one needs to make sure that in the steps (3) and (4)=
 which run in
> + =C2=A0 =C2=A0parallel, any global data is accessed within only one cont=
ext.
> +
> +Q.6: I have queued a threadlet which I want to cancel. How do I do that =
?
> +A.6: Threadlets framework provides the API cancel_threadlet:
> + =C2=A0 =C2=A0 =C2=A0 - int cancel_threadletwork(ThreadletWork *work)
> +
> + =C2=A0 =C2=A0 The API scans the ThreadletQueue to see if (work) is pres=
ent. If it finds
> + =C2=A0 =C2=A0 work, it'll dequeue work and return 0.
> +
> + =C2=A0 =C2=A0 On the other hand, if it does not find the (work) in the =
ThreadletQueue,
> + =C2=A0 =C2=A0 then it'll return 1. This can imply two things. Either th=
e work is being
> + =C2=A0 =C2=A0 processed by one of the helper threads or it has been pro=
cessed. The
> + =C2=A0 =C2=A0 threadlet infrastructure currently _does_not_ distinguish=
 between these
> + =C2=A0 =C2=A0 two and the onus is on the caller to do that.
> +
> +Q.7: Apart from the global pool of threads, can I have my own private Qu=
eue ?
> +A.7: Yes, the threadlets framework allows subsystems to create their own=
 private
> + =C2=A0 =C2=A0 queues with associated pools of threads.
> +
> + =C2=A0 =C2=A0 - Define a PrivateQueue
> + =C2=A0 =C2=A0 =C2=A0 ThreadletQueue myQueue;
> +
> + =C2=A0 =C2=A0 - Initialize it:
> + =C2=A0 =C2=A0 =C2=A0 threadlet_queue_init(&myQueue, my_max_threads, my_=
min_threads);
> + =C2=A0 =C2=A0 =C2=A0 where my_max_threads is the maximum number of thre=
ads that can be in the
> + =C2=A0 =C2=A0 =C2=A0 thread pool and my_min_threads is the minimum numb=
er of active threads
> + =C2=A0 =C2=A0 =C2=A0 that can be in the thread-pool.
> +
> + =C2=A0 =C2=A0 - Submit work:
> + =C2=A0 =C2=A0 =C2=A0 submit_threadletwork_to_queue(&myQueue, &my_work);
> +
> + =C2=A0 =C2=A0 - Cancel work:
> + =C2=A0 =C2=A0 =C2=A0 cancel_threadletwork_on_queue(&myQueue, &my_work);
> diff --git a/posix-aio-compat.c b/posix-aio-compat.c
> index dc5f8ec..b357449 100644
> --- a/posix-aio-compat.c
> +++ b/posix-aio-compat.c
> @@ -29,34 +29,11 @@
> =C2=A0#include "block_int.h"
>
> =C2=A0#include "block/raw-posix-aio.h"
> -#include "qemu-thread.h"
> -
> -#define MAX_GLOBAL_THREADS =C2=A064
> -#define MIN_GLOBAL_THREADS =C2=A0 8
> +#include "qemu-threadlets.h"
>
> =C2=A0static QemuMutex aiocb_mutex;
> =C2=A0static QemuCond aiocb_completion;
>
> -typedef struct ThreadletQueue
> -{
> - =C2=A0 =C2=A0QemuMutex lock;
> - =C2=A0 =C2=A0QemuCond cond;
> - =C2=A0 =C2=A0int max_threads;
> - =C2=A0 =C2=A0int min_threads;
> - =C2=A0 =C2=A0int cur_threads;
> - =C2=A0 =C2=A0int idle_threads;
> - =C2=A0 =C2=A0QTAILQ_HEAD(, ThreadletWork) request_list;
> -} ThreadletQueue;
> -
> -typedef struct ThreadletWork
> -{
> - =C2=A0 =C2=A0QTAILQ_ENTRY(ThreadletWork) node;
> - =C2=A0 =C2=A0void (*func)(struct ThreadletWork *work);
> -} ThreadletWork;
> -
> -static ThreadletQueue globalqueue;
> -static int globalqueue_init;
> -
> =C2=A0struct qemu_paiocb {
> =C2=A0 =C2=A0 BlockDriverAIOCB common;
> =C2=A0 =C2=A0 int aio_fildes;
> @@ -83,154 +60,6 @@ typedef struct PosixAioState {
> =C2=A0 =C2=A0 struct qemu_paiocb *first_aio;
> =C2=A0} PosixAioState;
>
> -static void *threadlet_worker(void *data)
> -{
> - =C2=A0 =C2=A0ThreadletQueue *queue =3D data;
> -
> - =C2=A0 =C2=A0qemu_mutex_lock(&queue->lock);
> - =C2=A0 =C2=A0while (1) {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0ThreadletWork *work;
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0int ret =3D 0;
> -
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0while (QTAILQ_EMPTY(&queue->request_list) &&
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (ret !=3D ETIMEDOUT)) =
{
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* wait for cond to be signall=
ed or broadcast for 1000s */
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ret =3D qemu_cond_timedwait((&=
queue->cond),
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &(que=
ue->lock), 10*100000);
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0}
> -
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0assert(queue->idle_threads !=3D 0);
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0if (QTAILQ_EMPTY(&queue->request_list)) {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (queue->cur_threads > queue=
->min_threads) {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* We retain the=
 minimum number of threads */
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0break;
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0} else {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0work =3D QTAILQ_FIRST(&queue->=
request_list);
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0QTAILQ_REMOVE(&queue->request_=
list, work, node);
> -
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0queue->idle_threads--;
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0qemu_mutex_unlock(&queue->lock=
);
> -
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* execute the work function *=
/
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0work->func(work);
> -
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0qemu_mutex_lock(&queue->lock);
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0queue->idle_threads++;
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0}
> - =C2=A0 =C2=A0}
> -
> - =C2=A0 =C2=A0queue->idle_threads--;
> - =C2=A0 =C2=A0queue->cur_threads--;
> - =C2=A0 =C2=A0qemu_mutex_unlock(&queue->lock);
> -
> - =C2=A0 =C2=A0return NULL;
> -}
> -
> -static void spawn_threadlet(ThreadletQueue *queue)
> -{
> - =C2=A0 =C2=A0QemuThread thread;
> -
> - =C2=A0 =C2=A0queue->cur_threads++;
> - =C2=A0 =C2=A0queue->idle_threads++;
> -
> - =C2=A0 =C2=A0qemu_thread_create(&thread, threadlet_worker, queue);
> -}
> -
> -/**
> - * submit_work_to_queue: Submit a new task to a private queue to be
> - * =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0executed asynchronously.
> - * @queue: Per-subsystem private queue to which the new task needs
> - * =C2=A0 =C2=A0 =C2=A0 =C2=A0 to be submitted.
> - * @work: Contains information about the task that needs to be submitted=
.
> - */
> -static void submit_work_to_queue(ThreadletQueue *queue, ThreadletWork *w=
ork)
> -{
> - =C2=A0 =C2=A0qemu_mutex_lock(&queue->lock);
> - =C2=A0 =C2=A0if (queue->idle_threads =3D=3D 0 && queue->cur_threads < q=
ueue->max_threads) {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0spawn_threadlet(queue);
> - =C2=A0 =C2=A0} else {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0qemu_cond_signal(&queue->cond);
> - =C2=A0 =C2=A0}
> - =C2=A0 =C2=A0QTAILQ_INSERT_TAIL(&queue->request_list, work, node);
> - =C2=A0 =C2=A0qemu_mutex_unlock(&queue->lock);
> -}
> -
> -static void threadlet_queue_init(ThreadletQueue *queue,
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0int max_threads, int min_threads);
> -
> -/**
> - * submit_work: Submit to the global queue a new task to be executed
> - * =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 asynch=
ronously.
> - * @work: Contains information about the task that needs to be submitted=
.
> - */
> -static void submit_work(ThreadletWork *work)
> -{
> - =C2=A0 =C2=A0if (!globalqueue_init) {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0threadlet_queue_init(&globalqueue, MAX_GLOBA=
L_THREADS,
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MIN_GLOBAL_THREADS);
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0globalqueue_init =3D 1;
> - =C2=A0 =C2=A0}
> -
> - =C2=A0 =C2=A0submit_work_to_queue(&globalqueue, work);
> -}
> -
> -/**
> - * dequeue_work_on_queue: Cancel a task queued on a Queue.
> - * @queue: The queue containing the task to be cancelled.
> - * @work: Contains the information of the task that needs to be cancelle=
d.
> - *
> - * Returns: 0 if the task is successfully cancelled.
> - * =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01 otherwise.
> - */
> -static int dequeue_work_on_queue(ThreadletQueue *queue, ThreadletWork *w=
ork)
> -{
> - =C2=A0 =C2=A0ThreadletWork *ret_work;
> - =C2=A0 =C2=A0int ret =3D 1;
> -
> - =C2=A0 =C2=A0qemu_mutex_lock(&queue->lock);
> - =C2=A0 =C2=A0QTAILQ_FOREACH(ret_work, &(queue->request_list), node) {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0if (ret_work =3D=3D work) {
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0QTAILQ_REMOVE(&queue->request_=
list, ret_work, node);
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ret =3D 0;
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0break;
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0}
> - =C2=A0 =C2=A0}
> - =C2=A0 =C2=A0qemu_mutex_unlock(&queue->lock);
> -
> - =C2=A0 =C2=A0return ret;
> -}
> -
> -/**
> - * dequeue_work: Cancel a task queued on the global queue.
> - * @work: Contains the information of the task that needs to be cancelle=
d.
> - *
> - * Returns: 0 if the task is successfully cancelled.
> - * =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01 otherwise.
> - */
> -static int dequeue_work(ThreadletWork *work)
> -{
> - =C2=A0 =C2=A0return dequeue_work_on_queue(&globalqueue, work);
> -}
> -
> -/**
> - * threadlet_queue_init: Initialize a threadlet queue.
> - * @queue: The threadlet queue to be initialized.
> - * @max_threads: Maximum number of threads processing the queue.
> - * @min_threads: Minimum number of threads processing the queue.
> - */
> -static void threadlet_queue_init(ThreadletQueue *queue,
> - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0int max_threads, int=
 min_threads)
> -{
> - =C2=A0 =C2=A0queue->cur_threads =C2=A0=3D 0;
> - =C2=A0 =C2=A0queue->idle_threads =3D 0;
> - =C2=A0 =C2=A0queue->max_threads =C2=A0=3D max_threads;
> - =C2=A0 =C2=A0queue->min_threads =C2=A0=3D min_threads;
> - =C2=A0 =C2=A0QTAILQ_INIT(&queue->request_list);
> - =C2=A0 =C2=A0qemu_mutex_init(&queue->lock);
> - =C2=A0 =C2=A0qemu_cond_init(&queue->cond);
> -}
> -
> =C2=A0#ifdef CONFIG_PREADV
> =C2=A0static int preadv_present =3D 1;
> =C2=A0#else
> diff --git a/qemu-threadlets.c b/qemu-threadlets.c
> new file mode 100644
> index 0000000..23b4ecf
> --- /dev/null
> +++ b/qemu-threadlets.c
> @@ -0,0 +1,168 @@
> +/*
> + * Threadlet support for offloading tasks to be executed asynchronously
> + *
> + * Copyright IBM, Corp. 2008
> + * Copyright IBM, Corp. 2010
> + *
> + * Authors:
> + * =C2=A0Anthony Liguori =C2=A0 =C2=A0 <aliguori@us.ibm.com>
> + * =C2=A0Aneesh Kumar K.V =C2=A0 =C2=A0<aneesh.kumar@linux.vnet.ibm.com>
> + * =C2=A0Gautham R Shenoy =C2=A0 =C2=A0<gautham.shenoy@gmail.com>
> + * =C2=A0Arun R Bharadwaj =C2=A0 =C2=A0<arun@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2. =C2=
=A0See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu-threadlets.h"
> +#include "osdep.h"
> +
> +#define MAX_GLOBAL_THREADS =C2=A064
> +#define MIN_GLOBAL_THREADS =C2=A0 8
> +static ThreadletQueue globalqueue;
> +static int globalqueue_init;
> +
> +static void *threadlet_worker(void *data)
> +{
> + =C2=A0 =C2=A0ThreadletQueue *queue =3D data;
> +
> + =C2=A0 =C2=A0qemu_mutex_lock(&queue->lock);
> + =C2=A0 =C2=A0while (1) {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0ThreadletWork *work;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0int ret =3D 0;
> +
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0while (QTAILQ_EMPTY(&queue->request_list) &&
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (ret !=3D ETIMEDOUT)) =
{
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* wait for cond to be signall=
ed or broadcast for 1000s */
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ret =3D qemu_cond_timedwait((&=
queue->cond),
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 &(que=
ue->lock), 10*100000);
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0}
> +
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0assert(queue->idle_threads !=3D 0);
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0if (QTAILQ_EMPTY(&queue->request_list)) {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (queue->cur_threads > queue=
->min_threads) {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* We retain the=
 minimum number of threads */
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0break;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0} else {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0work =3D QTAILQ_FIRST(&queue->=
request_list);
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0QTAILQ_REMOVE(&queue->request_=
list, work, node);
> +
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0queue->idle_threads--;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0qemu_mutex_unlock(&queue->lock=
);
> +
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0/* execute the work function *=
/
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0work->func(work);
> +
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0qemu_mutex_lock(&queue->lock);
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0queue->idle_threads++;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0}
> + =C2=A0 =C2=A0}
> +
> + =C2=A0 =C2=A0queue->idle_threads--;
> + =C2=A0 =C2=A0queue->cur_threads--;
> + =C2=A0 =C2=A0qemu_mutex_unlock(&queue->lock);
> +
> + =C2=A0 =C2=A0return NULL;
> +}
> +
> +static void spawn_threadlet(ThreadletQueue *queue)
> +{
> + =C2=A0 =C2=A0QemuThread thread;
> +
> + =C2=A0 =C2=A0queue->cur_threads++;
> + =C2=A0 =C2=A0queue->idle_threads++;
> +
> + =C2=A0 =C2=A0qemu_thread_create(&thread, threadlet_worker, queue);
> +}
> +
> +/**
> + * submit_work_to_queue: Submit a new task to a private queue to be
> + * =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0executed asynchronously.
> + * @queue: Per-subsystem private queue to which the new task needs
> + * =C2=A0 =C2=A0 =C2=A0 =C2=A0 to be submitted.
> + * @work: Contains information about the task that needs to be submitted=
.
> + */
> +void submit_work_to_queue(ThreadletQueue *queue, ThreadletWork *work)
> +{
> + =C2=A0 =C2=A0qemu_mutex_lock(&queue->lock);
> + =C2=A0 =C2=A0if (queue->idle_threads =3D=3D 0 && queue->cur_threads < q=
ueue->max_threads) {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0spawn_threadlet(queue);
> + =C2=A0 =C2=A0} else {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0qemu_cond_signal(&queue->cond);
> + =C2=A0 =C2=A0}
> + =C2=A0 =C2=A0QTAILQ_INSERT_TAIL(&queue->request_list, work, node);
> + =C2=A0 =C2=A0qemu_mutex_unlock(&queue->lock);
> +}
> +
> +/**
> + * submit_work: Submit to the global queue a new task to be executed
> + * =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 asynch=
ronously.
> + * @work: Contains information about the task that needs to be submitted=
.
> + */
> +void submit_work(ThreadletWork *work)
> +{
> + =C2=A0 =C2=A0if (!globalqueue_init) {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0threadlet_queue_init(&globalqueue, MAX_GLOBA=
L_THREADS,
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MIN_GLOBAL_THREADS);
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0globalqueue_init =3D 1;
> + =C2=A0 =C2=A0}
> +
> + =C2=A0 =C2=A0submit_work_to_queue(&globalqueue, work);
> +}
> +
> +/**
> + * dequeue_work_on_queue: Cancel a task queued on a Queue.
> + * @queue: The queue containing the task to be cancelled.
> + * @work: Contains the information of the task that needs to be cancelle=
d.
> + *
> + * Returns: 0 if the task is successfully cancelled.
> + * =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01 otherwise.
> + */
> +int dequeue_work_on_queue(ThreadletQueue *queue, ThreadletWork *work)
> +{
> + =C2=A0 =C2=A0ThreadletWork *ret_work;
> + =C2=A0 =C2=A0int ret =3D 1;
> +
> + =C2=A0 =C2=A0qemu_mutex_lock(&queue->lock);
> + =C2=A0 =C2=A0QTAILQ_FOREACH(ret_work, &(queue->request_list), node) {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0if (ret_work =3D=3D work) {
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0QTAILQ_REMOVE(&queue->request_=
list, ret_work, node);
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ret =3D 0;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0break;
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0}
> + =C2=A0 =C2=A0}
> + =C2=A0 =C2=A0qemu_mutex_unlock(&queue->lock);
> +
> + =C2=A0 =C2=A0return ret;
> +}
> +
> +/**
> + * dequeue_work: Cancel a task queued on the global queue.
> + * @work: Contains the information of the task that needs to be cancelle=
d.
> + *
> + * Returns: 0 if the task is successfully cancelled.
> + * =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01 otherwise.
> + */
> +int dequeue_work(ThreadletWork *work)
> +{
> + =C2=A0 =C2=A0return dequeue_work_on_queue(&globalqueue, work);
> +}
> +
> +/**
> + * threadlet_queue_init: Initialize a threadlet queue.
> + * @queue: The threadlet queue to be initialized.
> + * @max_threads: Maximum number of threads processing the queue.
> + * @min_threads: Minimum number of threads processing the queue.
> + */
> +void threadlet_queue_init(ThreadletQueue *queue,
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0int max_threads, int=
 min_threads)
> +{
> + =C2=A0 =C2=A0queue->cur_threads =C2=A0=3D 0;
> + =C2=A0 =C2=A0queue->idle_threads =3D 0;
> + =C2=A0 =C2=A0queue->max_threads =C2=A0=3D max_threads;
> + =C2=A0 =C2=A0queue->min_threads =C2=A0=3D min_threads;
> + =C2=A0 =C2=A0QTAILQ_INIT(&queue->request_list);
> + =C2=A0 =C2=A0qemu_mutex_init(&queue->lock);
> + =C2=A0 =C2=A0qemu_cond_init(&queue->cond);
> +}
> diff --git a/qemu-threadlets.h b/qemu-threadlets.h
> new file mode 100644
> index 0000000..f869304
> --- /dev/null
> +++ b/qemu-threadlets.h
> @@ -0,0 +1,48 @@
> +/*
> + * Threadlet support for offloading tasks to be executed asynchronously
> + *
> + * Copyright IBM, Corp. 2008
> + * Copyright IBM, Corp. 2010
> + *
> + * Authors:
> + * =C2=A0Anthony Liguori =C2=A0 =C2=A0 <aliguori@us.ibm.com>
> + * =C2=A0Aneesh Kumar K.V =C2=A0 =C2=A0<aneesh.kumar@linux.vnet.ibm.com>
> + * =C2=A0Gautham R Shenoy =C2=A0 =C2=A0<gautham.shenoy@gmail.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2. =C2=
=A0See
> + * the COPYING file in the top-level directory.
> + */
> +
> +#ifndef QEMU_ASYNC_WORK_H
> +#define QEMU_ASYNC_WORK_H
> +
> +#include "qemu-queue.h"
> +#include "qemu-common.h"
> +#include "qemu-thread.h"
> +
> +typedef struct ThreadletQueue
> +{
> + =C2=A0 =C2=A0QemuMutex lock;
> + =C2=A0 =C2=A0QemuCond cond;
> + =C2=A0 =C2=A0int max_threads;
> + =C2=A0 =C2=A0int min_threads;
> + =C2=A0 =C2=A0int cur_threads;
> + =C2=A0 =C2=A0int idle_threads;
> + =C2=A0 =C2=A0QTAILQ_HEAD(, ThreadletWork) request_list;
> +} ThreadletQueue;
> +
> +typedef struct ThreadletWork
> +{
> + =C2=A0 =C2=A0QTAILQ_ENTRY(ThreadletWork) node;
> + =C2=A0 =C2=A0void (*func)(struct ThreadletWork *work);
> +} ThreadletWork;
> +
> +extern void submit_work_to_queue(ThreadletQueue *queue,
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ThreadletWork *work);
> +extern void submit_work(ThreadletWork *work);
> +extern int dequeue_work_on_queue(ThreadletQueue *queue,
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ThreadletWork *work);
> +extern int dequeue_work(ThreadletWork *work);
> +extern void threadlet_queue_init(ThreadletQueue *queue, int max_threads,
> + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 int min_threads);

Please remove 'extern' qualifiers, they are useless for function
declarations. In most cases they are not used in QEMU.