From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46118) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W2ytD-0000P8-34 for qemu-devel@nongnu.org; Tue, 14 Jan 2014 02:58:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1W2yt8-0006NI-Ub for qemu-devel@nongnu.org; Tue, 14 Jan 2014 02:58:23 -0500 Received: from mail-yh0-f41.google.com ([209.85.213.41]:48172) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1W2yt8-0006N8-Px for qemu-devel@nongnu.org; Tue, 14 Jan 2014 02:58:18 -0500 Received: by mail-yh0-f41.google.com with SMTP id i57so9830yha.0 for ; Mon, 13 Jan 2014 23:58:18 -0800 (PST) Message-ID: <52D4EE16.5060703@inktank.com> Date: Mon, 13 Jan 2014 23:58:14 -0800 From: Josh Durgin MIME-Version: 1.0 References: <20140108044407.GG30570@stefanha-thinkpad.redhat.com> <52D4D83B.4040803@inktank.com> In-Reply-To: Content-Type: text/plain; charset=gb18030; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Qemu-devel] =?gb18030?b?u9i4tKO6ILvYuLSjuiC72Li0o7ogu9i4tKO6ICBx?= =?gb18030?q?emu_vm_big_network_latency_when_met_heavy_io?= List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: =?gb18030?Q?=D2=B6=C9=DC=E8=A1?= Cc: qemu-devel On 01/13/2014 10:39 PM, 叶绍琛 wrote: > Hi Josh > > Thanks for your reply. > > librbd version : 0.67.5-1 > > /etc/ceph/ceph.conf config file: > the ip and hostname was hiddened. > [global] > ;open auth. > auth cluster required = cephx > auth service required = cephx > auth client required = cephx > ;global pid & log setting. > admin socket = /home/ceph/var/run/$cluster-$name.asok > > [mon] > keyring = /home/ceph/var/lib/$type/$cluster-$id/keyring > mon data = /home/ceph/var/lib/$type/$cluster-$id > mon cluster log file = /home/ceph/log/$cluster.log > [mon.a] > host = cld-xx > mon addr = x.x.x.x:6789 > user = ceph > [mon.b] > host = cld-xx > mon addr = x.x.x.x:6789 > user = ceph > [mon.c] > host = cld-xx > mon addr = x.x.x.x:6789 > user = ceph > [mon.d] > host = cld-xx > mon addr = x.x.x.x:6789 > user = ceph > [mon.e] > host = cld-xx > mon addr = x.x.x.x:6789 > user = ceph > > [osd] > keyring = /home/ceph/var/lib/$type/$cluster-$id/keyring > osd data = /home/ceph/var/lib/$type/$cluster-$id > osd journal = /home/ceph/var/lib/$type/$cluster-$id/journal > osd journal size = 1000 > osd mkfs type = xfs > osd mount options xfs = rw,noatime,inode64 > [osd.0] > host = cld-xx > addr = x.x.x.x > user = ceph > devs = /dev/sdb1 > [osd.1] > host = cld-xx > addr = x.x.x.x > user = ceph > devs = /dev/sda1 > [osd.2] > host = cld-xx > addr = x.x.x.x > user = ceph > devs = /dev/sdb1 > [osd.3] > host = cld-xx > addr = x.x.x.x > user = ceph > devs = /dev/sda1 > [osd.4] > host = cld-xx > addr = x.x.x.x > user = ceph > devs = /dev/sdb1 > [osd.5] > host = cld-xx > addr = x.x.x.x > user = ceph > devs = /dev/sda1 > > [client.libvirt] > rbd cache = true Turning on rbd caching in ceph.conf, while telling QEMU it is not writeback in libvirt's xml like: is not safe since QEMU won't propagate flush requests from the guest to librbd's cache. Since QEMU 1.3 setting the QEMU cache mode will also set the librbd cache mode appropriately, so you don't need to have the setting in your ceph.conf file at all. Can you verify that your QEMU binary was built against librbd v0.60 or later? It would also help to see whether you can reproduce the problem when QEMU is sending flushes (cache=writeback), and with the cache disabled (cache=none, no 'rbd cache' setting in ceph.conf). The next step would be gathering a log from librbd (debug rbd = 20, debug objectcacher=20, debug objecter=20, debug ms = 1) when this problem is occurring, and correlating it with a timestamped log of the network latency. Thanks, Josh > auth support = cephx none > mon host = > x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789;x.x.x.x:6789 > > Regards > > Alan ye > > ------------------ > 叶绍琛 > Alan Ye > > > > ------------------ 原始邮件 ------------------ > *发件人:* "Josh Durgin";; > *发送时间:* 2014年1月14日(星期二) 下午2:24 > *收件人:* "Stefan Hajnoczi"; "叶绍琛" > ; > *抄送:* "qemu-devel"; > *主题:* Re: 回复： 回复： 回复： [Qemu-devel] qemu vm big network > latency when met heavy io > > On 01/12/2014 06:39 PM, Stefan Hajnoczi wrote: > > On Fri, Jan 10, 2014 at 11:50 AM, 叶绍琛 wrote: > > > > Please use Reply-all to keep the CC list in tact. That way the > > conversation stays on the mailing list and others can participate. > > > >>> Is the sum of guests' RAM less than the total physical RAM on the host > >> The host run 3 vms, each vm use one vcpu core and 1G ram. > >> # free -m > >> > >> total used free shared buffers > cached > >> > >> Mem: 32242 4808 27434 0 278 > 2058 > >> > >> -/+ buffers/cache: 2471 29771 > >> > >> Swap: 4095 0 4095 > >> > >> > >> The host has 8 cores. > >> # cat /proc/cpuinfo | grep processor > >> processor : 0 > >> processor : 1 > >> processor : 2 > >> processor : 3 > >> processor : 4 > >> processor : 5 > >> processor : 6 > >> processor : 7 > >> > >> so, both of two question's answer is 'yes'. > >> When I runing random write test, the host use 0 swap. > > > > Great. That means the host is not overcommitted. > > > > It's likely that the problem is a bug in QEMU's rbd driver or librados. > > > > Josh: Perhaps something you're interested in looking into? > > Yes, thanks for bringing it to my attention. It does sound like a bug in > QEMU's rbd driver or ceph's userspace libraries. > > Could you share what version of librbd you're using, and your > /etc/ceph/ceph.conf? > > Thanks, > Josh