From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43483) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1V2CMR-0005iY-KP for qemu-devel@nongnu.org; Wed, 24 Jul 2013 23:37:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1V2CMQ-0000pn-Jk for qemu-devel@nongnu.org; Wed, 24 Jul 2013 23:37:03 -0400 Received: from mail-pa0-x230.google.com ([2607:f8b0:400e:c03::230]:57871) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1V2CMQ-0000pg-D3 for qemu-devel@nongnu.org; Wed, 24 Jul 2013 23:37:02 -0400 Received: by mail-pa0-f48.google.com with SMTP id kp13so224970pab.35 for ; Wed, 24 Jul 2013 20:37:01 -0700 (PDT) Date: Thu, 25 Jul 2013 11:36:51 +0800 From: Liu Yuan Message-ID: <20130725033651.GB24069@ubuntu-precise> References: <1374652593-7242-1-git-send-email-morita.kazutaka@lab.ntt.co.jp> <20130724082830.GA11913@ubuntu-precise> <87ip00gwc6.wl%morita.kazutaka@lab.ntt.co.jp> <20130724154249.GA16838@ubuntu-precise> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130724154249.GA16838@ubuntu-precise> Subject: Re: [Qemu-devel] [sheepdog] [PATCH v2 0/9] sheepdog: reconnect server after connection failure List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: MORITA Kazutaka Cc: Kevin Wolf , Paolo Bonzini , sheepdog@lists.wpkg.org, qemu-devel@nongnu.org, Stefan Hajnoczi On Wed, Jul 24, 2013 at 11:42:49PM +0800, Liu Yuan wrote: > On Wed, Jul 24, 2013 at 06:07:21PM +0900, MORITA Kazutaka wrote: > > At Wed, 24 Jul 2013 16:28:30 +0800, > > Liu Yuan wrote: > > > > > > On Wed, Jul 24, 2013 at 04:56:24PM +0900, MORITA Kazutaka wrote: > > > > Currently, if a sheepdog server exits, all the connecting VMs need to > > > > be restarted. This series implements a feature to reconnect the > > > > server, and enables us to do online sheepdog upgrade and avoid > > > > restarting VMs when sheepdog servers crash unexpectedly. > > > > > > > > > > It doesn't work on my test. I tried start linux-0.2.img stored in sheepdog > > > cluster and then > > > > > > 1. did some buffered writes > > > 2. restart sheep that this QEMU VM connected to. > > > 3. $ sync > > > > > > I got following error: > > > > > > $ ../qemu/x86_64-softmmu/qemu-system-x86_64 --enable-kvm -m 1024 -hda sheepdog:test > > > qemu-system-x86_64: failed to get the header, Resource temporarily unavailable > > > qemu-system-x86_64: Failed to connect to socket: Connection refused > > > qemu-system-x86_64: Failed to connect to socket: Connection refused > > > qemu-system-x86_64: Failed to connect to socket: Connection refused > > > qemu-system-x86_64: Failed to connect to socket: Connection refused > > > qemu-system-x86_64: Failed to connect to socket: Connection refused > > > ...repeat... > > > > > > QEMU version is master tip > > > > Your sheep daemon looks like unreachable from qemu. I tried the same > > procedure, but couldn't reproduce it. > > > > Is the problem reproducible? Can you make sure that you can connect > > to the sheep daemon from collie while the error message shows up? > > > > Yesh. Well I try to repeat it with following process: > > 1. did some buffered write > 2. kill the sheep > 3. $ sync # at guest, now 'sync' hang for response > 4. restart sheep > > After 4 'sync' still hangs until timeout with a message > "hda:dma_timer_expiry: dma status == 0x21" > > Guest end up freeze. > > QEMU output is the same: > qemu-system-x86_64: failed to get the header, Resource temporarily unavailable > qemu-system-x86_64: Failed to connect to socket: Connection refused > qemu-system-x86_64: Failed to connect to socket: Connection refused > qemu-system-x86_64: Failed to connect to socket: Connection refused > qemu-system-x86_64: Failed to connect to socket: Connection refused > > But notice, if I did restart sheep with guest doing nothing, your patch set work > like a charm. I have debug it a bit. The problem is that at stage 3, 'sync' invoke add_aio_request() in the sheepdog driver and add_aio_request *succeed* with aio put on the inflight_aio_head list, *not* on the failed_aio_head list. So in the reconnect_to_sdog(), we have no way to resend the targeted aio and 'sync' wait for ever. Thanks Yuan