From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33704) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVHU4-0001d9-5Z for qemu-devel@nongnu.org; Tue, 10 Mar 2015 06:33:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YVHTz-000121-4V for qemu-devel@nongnu.org; Tue, 10 Mar 2015 06:33:56 -0400 Received: from mail-wi0-x231.google.com ([2a00:1450:400c:c05::231]:40999) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVHTy-00011l-QM for qemu-devel@nongnu.org; Tue, 10 Mar 2015 06:33:51 -0400 Received: by wiwl15 with SMTP id l15so1724256wiw.0 for ; Tue, 10 Mar 2015 03:33:50 -0700 (PDT) Sender: Paolo Bonzini Message-ID: <54FEC889.2000408@redhat.com> Date: Tue, 10 Mar 2015 11:33:45 +0100 From: Paolo Bonzini MIME-Version: 1.0 References: <54EEE795.3050501@de.ibm.com> <20150306172324.GF2431@stefanha-thinkpad.redhat.com> <54FE0492.5080305@de.ibm.com> <54FEA34C.4030600@de.ibm.com> In-Reply-To: <54FEA34C.4030600@de.ibm.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] qemu crash in coroutine bdrv_co_do_rw List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Christian Borntraeger , Stefan Hajnoczi Cc: qemu-devel , Stefan Hajnoczi On 10/03/2015 08:54, Christian Borntraeger wrote: > Am 09.03.2015 um 21:37 schrieb Christian Borntraeger: >> Am 06.03.2015 um 18:23 schrieb Stefan Hajnoczi: >>> On Thu, Feb 26, 2015 at 10:29:57AM +0100, Christian Borntraeger wrote: >>>> is this some know issue? Under heavy load with lots of dataplane devices I sometimes get a segfault in the bdrc_co_do_rw routine: >>>> >>>> #0 bdrv_co_do_rw (opaque=0x0) at /home/cborntra/REPOS/qemu/block.c:4791 >>>> 4791 if (!acb->is_write) { >>>> (gdb) bt >>>> #0 bdrv_co_do_rw (opaque=0x0) at /home/cborntra/REPOS/qemu/block.c:4791 >>>> #1 0x00000000801aeb78 in coroutine_trampoline (i0=, i1=-725099072) at /home/cborntra/REPOS/qemu/coroutine-ucontext.c:80 >>>> #2 0x000003fffbe1cca2 in __makecontext_ret () from /lib64/libc.so.6 >>>> Backtrace stopped: previous frame identical to this frame (corrupt stack?) >>>> (gdb) up >>>> #1 0x00000000801aeb78 in coroutine_trampoline (i0=, i1=-725099072) at /home/cborntra/REPOS/qemu/coroutine-ucontext.c:80 >>>> 80 co->entry(co->entry_arg); >>>> (gdb) print *co >>>> $1 = {entry = 0x801a3c28 , entry_arg = 0x0, caller = 0x3ffe2fff788, pool_next = {sle_next = 0x3ffd2287990}, co_queue_wakeup = {tqh_first = 0x0, >>>> tqh_last = 0x3ffd4c7dde0}, co_queue_next = {tqe_next = 0x0, tqe_prev = 0x0}} >>>> >>>> As you can see enty_arg is 0, causing the problem. Do you have any quick idea before I start debugging? >>> >>> No, I haven't seen this bug before. Are you running qemu.git/master? >>> >>> Have you tried disabling the coroutine pool (freelist)? >>> >>> Stefan >>> >> >> I was able to increase the likelyhood of hitting this (more vCPUs, less guests). >> >> bisect thinks that this makes this shaky: >> >> 4d68e86bb10159099da0798f74e7512955f15eec is the first bad commit >> commit 4d68e86bb10159099da0798f74e7512955f15eec >> Author: Paolo Bonzini >> Date: Tue Dec 2 12:05:48 2014 +0100 >> >> coroutine: rewrite pool to avoid mutex >> >> >> Christian >> > > Yes, reverting these 3 makes the problem go away during an overnight run. Let's see if a quick hack helps isolate the problem (either in the lockless magic or in the algorithm itself): diff --git a/qemu-coroutine.c b/qemu-coroutine.c index 525247b..38e1a32 100644 --- a/qemu-coroutine.c +++ b/qemu-coroutine.c @@ -29,6 +29,7 @@ static unsigned int release_pool_size; static __thread QSLIST_HEAD(, Coroutine) alloc_pool = QSLIST_HEAD_INITIALIZER(pool); static __thread unsigned int alloc_pool_size; static __thread Notifier coroutine_pool_cleanup_notifier; +static QemuMutex pool_lock; static void coroutine_pool_cleanup(Notifier *n, void *value) { @@ -59,8 +60,10 @@ Coroutine *qemu_coroutine_create(CoroutineEntry *entry) * release_pool_size and the actual size of release_pool. But * it is just a heuristic, it does not need to be perfect. */ + qemu_mutex_lock(&pool_lock); alloc_pool_size = atomic_xchg(&release_pool_size, 0); QSLIST_MOVE_ATOMIC(&alloc_pool, &release_pool); + qemu_mutex_unlock(&pool_lock); co = QSLIST_FIRST(&alloc_pool); } } @@ -85,8 +88,10 @@ static void coroutine_delete(Coroutine *co) if (CONFIG_COROUTINE_POOL) { if (release_pool_size < POOL_BATCH_SIZE * 2) { + qemu_mutex_lock(&pool_lock); QSLIST_INSERT_HEAD_ATOMIC(&release_pool, co, pool_next); atomic_inc(&release_pool_size); + qemu_mutex_unlock(&pool_lock); return; } if (alloc_pool_size < POOL_BATCH_SIZE) { And if it doesn't help, on top: diff --git a/qemu-coroutine.c b/qemu-coroutine.c index 525247b..38e1a32 100644 --- a/qemu-coroutine.c +++ b/qemu-coroutine.c @@ -85,7 +88,7 @@ static void coroutine_delete(Coroutine *co) if (CONFIG_COROUTINE_POOL) { if (release_pool_size < POOL_BATCH_SIZE * 2) { qemu_mutex_lock(&pool_lock); - QSLIST_INSERT_HEAD_ATOMIC(&release_pool, co, pool_next); + QSLIST_INSERT_HEAD(&release_pool, co, pool_next); atomic_inc(&release_pool_size); qemu_mutex_unlock(&pool_lock); return; Thanks, Paolo