qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
From: "Daniel P. Berrange" <berrange@redhat.com>
To: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: liang.z.li@intel.com, qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] Hang with migration multi-thread compression under high load
Date: Wed, 27 Apr 2016 16:02:49 +0100	[thread overview]
Message-ID: <20160427150249.GH17937@redhat.com> (raw)
In-Reply-To: <20160427142929.GC2290@work-vm>

On Wed, Apr 27, 2016 at 03:29:30PM +0100, Dr. David Alan Gilbert wrote:
> ccing in Liang Li
> 
> * Daniel P. Berrange (berrange@redhat.com) wrote:
> > for some reason it isn't shown in the stack thrace for thread
> > 1 above, when initially connecting GDB it says the main thread
> > is at:
> > 
> > decompress_data_with_multi_threads (len=702, host=0x7fd78fe06000, f=0x55901af09950) at /home/berrange/src/virt/qemu/migration/ram.c:2254
> > 2254	        for (idx = 0; idx < thread_count; idx++) {
> > 
> > 
> > Looking at the target QEMU, we see  do_data_decompress method
> > is waiting in a condition var:
> > 
> >         while (!param->start && !quit_decomp_thread) {
> > 	    qemu_cond_wait(&param->cond, &param->mutex);
> >             ....do stuff..
> > 	    param->start = false
> >         }
> > 
> > 
> > Now the decompress_data_with_multi_threads is checking param->start without
> > holding the param->mutex lock.
> > 
> > Changing decompress_data_with_multi_threads to acquire param->mutex
> > lock makes it work, but isn't ideal, since that now blocks the
> > decompress_data_with_multi_threads() method on the completion of
> > each thread, which defeats the point of having multiple threads.

FWIW, the following patch also appears to "fix" the issue, presumably
by just making the race much less likely to hit:

diff --git a/migration/ram.c b/migration/ram.c
index 3f05738..be0233f 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2271,6 +2271,7 @@ static void decompress_data_with_multi_threads(QEMUFile *f,
         if (idx < thread_count) {
             break;
         }
+        sched_yield();
     }
 }
 

Incidentally IIUC, this decompress_data_with_multi_threads is just
busy waiting for a thread to become free, which seems pretty wasteful
of CPU resources. I wonder if there's a more effective way to structure
this, so that instead of having decompress_data_with_multi_threads()
choose which thread to pass the decompression job to, it just puts
the job into a queue, and then let all the threads pull from that
shared queue. IOW whichever thread the kerenl decides to wakeup would
get the job, without us having to explicitly assign a thread to the
job.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

  reply	other threads:[~2016-04-27 15:02 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-04-27 14:20 [Qemu-devel] Hang with migration multi-thread compression under high load Daniel P. Berrange
2016-04-27 14:29 ` Dr. David Alan Gilbert
2016-04-27 15:02   ` Daniel P. Berrange [this message]
2016-04-28  2:41     ` Li, Liang Z
2016-04-28  3:27 ` Li, Liang Z
2016-04-28  8:22   ` Daniel P. Berrange
2016-04-29  1:42     ` Li, Liang Z
2016-04-28 10:15 ` Daniel P. Berrange
2016-04-29  4:51   ` Li, Liang Z

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160427150249.GH17937@redhat.com \
    --to=berrange@redhat.com \
    --cc=dgilbert@redhat.com \
    --cc=liang.z.li@intel.com \
    --cc=qemu-devel@nongnu.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).