Re: [PATCH] kvm tools: Process virito blk requests in separate thread

[Asias He <asias.hejun@gmail.com>]

On 11/29/2011 10:36 PM, Sasha Levin wrote:
> On Tue, 2011-11-29 at 22:28 +0800, Asias He wrote:
>> Currently, all blk requests are processed in notify_vq() which is in
>> the context of ioeventfd thread: ioeventfd__thread(). The processing
>> in notify_vq() may take a long time to complete.
>>
>> We should make notify_vq() return as soon as possible, since all devices
>> are sharing the single ioeventfd thread. Otherwise, it will block other
>> device's notify_vq() being called and starve other devices.
>>
>> In virtio net's notify_vq(), we simply signal the tx/rx handle thread
>> and return.
> 
> Why not use the threadpool?

No.

1) In thread pool model, each job handling operation:
thread_pool__do_job() takes about 6 or 7 mutex_{lock,unlock} ops. Most
of the mutex are global (job_mutex) which are contented by the threads
in the pool. It's fine for the non performance critical virtio devices,
such as console, rng, etc. But it's not optimal for net and blk devices.

2) Using dedicated threads to handle blk requests opens the door for
user to set different IO priority for the blk threads.

3) It also reduces the contentions between net and blk devices if they
do not share the thead pool.

--->> the thread pool lock flow <<---

thread_pool__do_job() {

        mutex_lock(&jobinfo->mutex);
        if (jobinfo->signalcount++ == 0)
                thread_pool__job_push_locked(job) {
			        mutex_lock(&job_mutex);
			        thread_pool__job_push(job);
		        	mutex_unlock(&job_mutex);

		}
        mutex_unlock(&jobinfo->mutex);

        mutex_lock(&job_muex);
        pthread_cond_signal(&job_cond);
        mutex_unlock(&job_mutex);

}

thread_pool__threadfunc()
{

        for (;;) {
                mutex_lock(&job_mutex);
                pthread_cond_wait(&job_cond, &job_mutex);
                mutex_unlock(&job_mutex);

                if (curjob)
                        thread_pool__handle_job(curjob);
        }
}

thread_pool__handle_job()
{

        while (job) {
                job->callback(job->kvm, job->data);

                mutex_lock(&job->mutex);
                        thread_pool__job_push_locked(job);
                mutex_unlock(&job->mutex);

                job = thread_pool__job_pop_locked();
        }

}


-- 
Asias He