From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:36881)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiecl.fnst@cn.fujitsu.com>) id 1ao7h1-0002qd-4r
	for qemu-devel@nongnu.org; Thu, 07 Apr 2016 07:01:47 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiecl.fnst@cn.fujitsu.com>) id 1ao7gx-0006ZL-V7
	for qemu-devel@nongnu.org; Thu, 07 Apr 2016 07:01:43 -0400
Received: from [59.151.112.132] (port=4917 helo=heian.cn.fujitsu.com)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiecl.fnst@cn.fujitsu.com>) id 1ao7gx-0006RW-D3
	for qemu-devel@nongnu.org; Thu, 07 Apr 2016 07:01:39 -0400
Message-ID: <57063EA4.6050402@cn.fujitsu.com>
Date: Thu, 7 Apr 2016 19:04:04 +0800
From: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
Subject: [Qemu-devel] Questions about nbd with QIOChannel
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: qemu devel <qemu-devel@nongnu.org>, Paolo Bonzini <pbonzini@redhat.com>, "Daniel P. Berrange" <berrange@redhat.com>
Cc: Li Zhijian <lizhijian@cn.fujitsu.com>, =?UTF-8?B?IlhpZSwgQ2hhbmdsb25nL+iwog==?= =?UTF-8?B?IOaYjOm+mSI=?= <xiecl.fnst@cn.fujitsu.com>, zhanghailiang <zhang.zhanghailiang@huawei.com>

Hi all

Recently during test COLO, i found sometimes the client goes to hung on 
Primary side. First i thought it maybe a COLO revelant issue, but after 
ton of tests i doubt that this maybe a NBD issue (athough i'm not sure). 
So i'd like to share what i found:

Since commit 1c778ef7, we convert to using QIOChannel APIs for actual 
socket I/O.

Let foucus on nbd_reply_ready() here:

Before commit 1c778ef7
nbd_reply_ready()
   nbd_receive_reply()
     nbd_wr_sync()
     {
      ...
      while (offset < size) {
          if (do_read) {
              len = qemu_recv(fd, buffer + offset, size - offset, 0);
          } else {
              ...
          }
          if (len < 0) {
              err = socket_error();
              if (err == EINTR || (offset > 0 && (err == EAGAIN || err 
== EWOULDBLOCK))) {
                  continue;
              }
              return -err;
          }
          ...
      }
      ....
     }

if len < 0 && error == EAGAIN. we have two choice
1) continue to recv until finished.
2) return -EAGAIN, nbd_receive_reply() will check this return value and 
will return *Successfully*.
			
After commit 1c778ef7:
nbd_reply_ready()
   read_sync()
     nbd_wr_syncv()
     {
      ...
      while (nlocal_iov > 0) {
          ...
          if (do_read) {
              len = qio_channel_readv(ioc, local_iov, nlocal_iov, 
&local_err);
          } else {
              ...
          }
          if (len == QIO_CHANNEL_ERR_BLOCK) {
              if (qemu_in_coroutine()) {
                  qemu_coroutine_yield();
              } else {
                  qio_channel_wait(ioc,
                                   do_read ? G_IO_IN : G_IO_OUT);
              }
              continue;
          }
          ...
      }
     }

For NBD,
qio_channel_readv()
   qio_channel_readv_full
     klass->io_readv()
      qio_channel_socket_readv()
      {
         for(..) {
             ret = recv(xxx);
             if (ret < 0) {
                 if (errno == EAGAIN) {
                     if (done) {
                         return done;
                     } else {
                         return QIO_CHANNEL_ERR_BLOCK;
                     }
                 }

             }
             ...
         }
      }

Here, if ret < 0 && error == EAGAIN && !done, we'll return 
QIO_CHANNEL_ERR_BLOCK. Then nbd_wr_syncv() will invoke 
qio_channel_wait() and the guest will *HUNG* until i kill
nbd server service.

It's easy to reproduce. My question: If the scenario i describe above is 
what we expected?

Thanks
     -Xie