[Qemu-devel] qemu_cond_signal() taking a long time to complete.

[Arun R Bharadwaj <arun@linux.vnet.ibm.com>]

Hi,

I am working on introducing threading model into Qemu. This introduces
the Threadlets infrastructure which allows subsystems to offload possibly 
blocking work to a queue to be processed by a pool of threads asynchrnonously.
Threadlets are useful when there are operations that can be performed
outside the context of the VCPU/IO threads inorder to free these latter
to service any other guest requests. As of now we have converted a few
v9fs calls like v9fs_read, v9fs_write etc to this model to test the
working nature of this model.

I observed that performance is degrading in the threading model for the
following reason:

Taking the example of v9fs_read call: We submit the blocking work in
v9fs_read to a queue and do a qemu_cond_signal(). the work will be picked 
up by a worker thread which is waiting on the condition variable to go
true. I measured the time taken to execute the v9fs_read call; in the
case without the threading model, it takes around 15microseconds to
complete, while in the threading model, it takes around 30microsends
to complete. Most of this extra time (around 22microsends) is spent in
completing the qemu_cond_signal() call. I suspect this is the reason why
I am seeing performance hit with the threading model, because this
time is much more than the time needed to complete the entire
v9fs_read call in the non threading model case.

I need advice on how to proceed from this situation. Pasting relevant
code snippets below.

thanks
arun.
---

/* Code to sumbit work to the queue */
void submit_threadlet_to_queue(ThreadletQueue *queue, ThreadletWork
*work)
{
    qemu_mutex_lock(&(queue->lock));
    if (queue->idle_threads == 0 && queue->cur_threads <
queue->max_threads) {
        spawn_threadlet(queue);
    }
    QTAILQ_INSERT_TAIL(&(queue->request_list), work, node);
    qemu_cond_signal(&(queue->cond));
    qemu_mutex_unlock(&(queue->lock));
}

/* Worker thread code */
static void *threadlet_worker(void *data)
{
    ThreadletQueue *queue = data;

    while (1) {
        ThreadletWork *work;
        int ret = 0;
        qemu_mutex_lock(&(queue->lock));

        while (QTAILQ_EMPTY(&(queue->request_list)) &&
               (ret != ETIMEDOUT)) {
            ret = qemu_cond_timedwait(&(queue->cond),
                                         &(queue->lock), 10*100000);
        }

        assert(queue->idle_threads != 0);
        if (QTAILQ_EMPTY(&(queue->request_list))) {
            if (queue->cur_threads > queue->min_threads) {
                /* We retain the minimum number of threads */
                break;
            }
        } else {
            work = QTAILQ_FIRST(&(queue->request_list));
            QTAILQ_REMOVE(&(queue->request_list), work, node);

            queue->idle_threads--;
            qemu_mutex_unlock(&(queue->lock));

            /* execute the work function */
            work->func(work);

            qemu_mutex_lock(&(queue->lock));
            queue->idle_threads++;
        }
        qemu_mutex_unlock(&(queue->lock));
    }

    queue->idle_threads--;
    queue->cur_threads--;

    return NULL;
}