From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eugene Istomin Subject: Read IOPS storm in case of reflinking running Xen VM disk on OCFS2 Date: Thu, 09 Jul 2015 12:01:59 +0300 Message-ID: <2256638.BpbQTyUdDT@evis> Reply-To: E.Istomin@edss.ee Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6843859827721289267==" Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1ZD7iV-0000Nd-Ni for xen-devel@lists.xenproject.org; Thu, 09 Jul 2015 09:02:03 +0000 List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xenproject.org List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============6843859827721289267== Content-Type: multipart/alternative; boundary="nextPart1640288.uDjPQscg7v" Content-Transfer-Encoding: 7Bit This is a multi-part message in MIME format. --nextPart1640288.uDjPQscg7v Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Hello, after deploying OCFS2 reflink-based VM snapshots to production servers we discovered a performace degradation: OS: Opensuse 13.1, 13.2 Hypervisors: Xen 4.4, 4.5, 4.5.1 Dom0 kernels: 3.12, 3.16, 3.18, 4.1 DomU kernels: 3.12, 3.16, 3.18, 4.1 Tested DomU disk backends: tapdisk2, qdisk 1) on DomU (VM) #dd if=/dev/zero of=test2 bs=1M count=6000 2) atop on Dom0: sdb - busy:92% - read:375 - write:130902 Reads are from others VMs, seems OK 3) DomU dd finished: 6291456000 bytes (6.3 GB) copied, 16.6265 s, 378 MB/s 4) Lets start dd again & do a snapshot: #dd if=/dev/zero of=test2 bs=1M count=6000 #reflink test.raw ref/ 5) atop on Dom0: sdb - busy:97% - read:112740 - write:28037 So, Read IOPS = 112740, why? 6) DomU dd finished: 6291456000 bytes (6.3 GB) copied, 175.45 s, 35.9 MB/s 7) Second & further reflinks do not change the atop stat & dd time #dd if=/dev/zero of=test2 bs=1M count=6000 #reflink --backup=t test.raw ref/ \\ * n times ~ 6291456000 bytes (6.3 GB) copied, 162.959 s, 38.6 MB/s Working perfectly If reflink is done as fully Dom0 operations (dd & reflink are in Dom0) - so, this is not (or not only) OCFS2 problem. The question is why reflinking a running Xen VM disk leads to read IOPS storm? -- Best regards, Eugene Istomin --nextPart1640288.uDjPQscg7v Content-Transfer-Encoding: 7Bit Content-Type: text/html; charset="us-ascii"

Hello,

 

after deploying OCFS2 reflink-based VM snapshots to production servers we discovered a performace degradation:

 

OS: Opensuse 13.1, 13.2

Hypervisors: Xen 4.4, 4.5, 4.5.1

Dom0 kernels: 3.12, 3.16, 3.18, 4.1

DomU kernels: 3.12, 3.16, 3.18, 4.1

Tested DomU disk backends: tapdisk2, qdisk

 

 

1) on DomU (VM)

#dd if=/dev/zero of=test2 bs=1M count=6000

 

2) atop on Dom0:

sdb - busy:92% - read:375 - write:130902

Reads are from others VMs, seems OK

 

3) DomU dd finished:

6291456000 bytes (6.3 GB) copied, 16.6265 s, 378 MB/s

 

4) Lets start dd again & do a snapshot:

#dd if=/dev/zero of=test2 bs=1M count=6000

#reflink test.raw ref/

 

5) atop on Dom0:

sdb - busy:97% - read:112740 - write:28037

So, Read IOPS = 112740, why?

 

6) DomU dd finished:

6291456000 bytes (6.3 GB) copied, 175.45 s, 35.9 MB/s

 

7) Second & further reflinks do not change the atop stat & dd time

#dd if=/dev/zero of=test2 bs=1M count=6000

#reflink --backup=t test.raw ref/ \\ * n times

~ 6291456000 bytes (6.3 GB) copied, 162.959 s, 38.6 MB/s

 

Working perfectly If reflink is done as fully Dom0 operations (dd & reflink are in Dom0) - so, this is not (or not only) OCFS2 problem.

 

The question is why reflinking a running Xen VM disk leads to read IOPS storm?

--

Best regards,

Eugene Istomin

 

--nextPart1640288.uDjPQscg7v-- --===============6843859827721289267== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============6843859827721289267==--