From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steven Haigh Subject: Re: IO speed limited by size of IO request (for RBD driver) Date: Tue, 30 Apr 2013 05:47:48 +1000 Message-ID: <517ECE64.6000503@crc.id.au> References: <51768FA5.6090609@crc.id.au> <5176957E.6010306@citrix.com> <51769B9D.4000708@crc.id.au> <51769CFD.7020907@citrix.com> <51769E1E.6040902@crc.id.au> <5176A19A.2010802@citrix.com> <5176A440.8040303@crc.id.au> <5176A520.5030503@citrix.com> <5176A61F.6050607@crc.id.au> <5176A6DD.5000404@citrix.com> <5176AFF9.4020003@crc.id.au> <5176B237.8020803@citrix.com> <5176C073.3050409@crc.id.au> <5176CF56.8000505@citrix.com> <5176DB88.1070200@crc.id.au> <517A89DA.3030804@citrix.com> <517A8C44.5020103@crc.id.au> <517B3088.7070809@crc.id.au> <517B790A.3020009@citrix.com> <517B838C.9040607@crc.id.au> <517B8DE3.90306@crc.id.au> <517E3195.8090204@citrix.com> <517EC975.7030807@crc.id.au> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <517EC975.7030807@crc.id.au> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= Cc: Felipe Franciosi , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On 30/04/2013 5:26 AM, Steven Haigh wrote: > On 29/04/2013 6:38 PM, Roger Pau Monn=E9 wrote: >> Did you also copy xen-blkfront? > > Dammit! No, no I didn't. I tried to just copy this back over to the > 3.8.8 and 3.8.10 kernel versions, but it came up with too many errors - > so I just rebuilt/packages the checkout of your git based on 3.8.0-rc7. > >> It seems you are missing some pieces, you should see something like: >> >> blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; >> indirect descriptors: enabled; > > Now I'm running 3.8.0-rc7 from your git on both DomU and Dom0. In the > DomU, I now see: > > blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; > indirect descriptors: enabled; > > From what you say, this should be what I'd expect. > > From the DomU: > # dd if=3D/dev/zero of=3Doutput.zero bs=3D1M count=3D2048 > 2048+0 records in > 2048+0 records out > 2147483648 bytes (2.1 GB) copied, 32.9252 s, 65.2 MB/s > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.23 0.00 9.61 0.00 0.46 89.70 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdd 1071.40 7914.87 67.28 479.18 4.40 32.64 > 138.82 17.45 31.65 2.00 109.36 > sde 1151.72 7943.71 68.65 486.73 4.79 33.20 > 140.10 13.18 23.87 1.93 107.14 > sdc 1123.34 7921.05 66.36 482.84 4.66 32.86 > 139.89 8.80 15.96 1.86 102.31 > sdf 1091.53 7937.30 70.02 483.30 4.54 32.97 > 138.84 18.98 34.31 1.98 109.45 > md2 0.00 0.00 0.00 1003.66 0.00 65.31 > 133.27 0.00 0.00 0.00 0.00 > > # dd if=3D/dev/zero of=3Doutput.zero bs=3D1M count=3D2048 oflag=3Ddirect > 2048+0 records in > 2048+0 records out > 2147483648 bytes (2.1 GB) copied, 25.4705 s, 84.3 MB/s > > avg-cpu: %user %nice %system %iowait %steal %idle > 0.22 0.00 10.94 0.00 0.22 88.62 > > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s > avgrq-sz avgqu-sz await svctm %util > sdd 20.35 13703.72 1.75 258.64 0.10 54.54 > 429.75 0.47 1.81 1.13 29.34 > sde 1858.64 11655.36 61.05 199.56 7.51 46.36 > 423.36 1.54 5.89 3.27 85.27 > sdc 142.45 11824.07 5.47 254.70 0.59 47.18 > 376.03 0.42 1.61 1.02 26.59 > sdf 332.39 13489.72 11.38 248.80 1.35 53.72 > 433.47 1.06 4.10 2.50 65.16 > md2 0.00 0.00 3.72 733.48 0.06 91.68 > 254.86 0.00 0.00 0.00 0.00 I just thought - I should probably include a baseline by mounting the = same LV in the Dom0 and doing the exact same tests. # dd if=3D/dev/zero of=3Doutput.zero bs=3D1M count=3D2048 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 19.1554 s, 112 MB/s avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 23.18 76.60 0.22 0.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s = avgrq-sz avgqu-sz await svctm %util sdd 139.07 14785.43 11.92 286.98 0.59 58.88 = 407.50 2.60 8.71 1.84 54.92 sde 83.44 14846.58 8.39 292.05 0.36 59.09 = 405.23 4.12 13.69 2.56 76.84 sdc 98.23 14828.04 9.93 289.18 0.42 58.84 = 405.73 2.55 8.45 1.75 52.43 sdf 77.04 14816.78 8.61 289.40 0.33 58.96 = 407.51 3.89 13.05 2.52 75.14 md2 0.00 0.00 0.00 973.51 0.00 116.72 = 245.55 0.00 0.00 0.00 0.00 # dd if=3D/dev/zero of=3Doutput.zero bs=3D1M count=3D2048 oflag=3Ddirect 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 24.8914 s, 86.3 MB/s avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 12.22 87.58 0.21 0.00 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s = avgrq-sz avgqu-sz await svctm %util sdd 32.09 12310.14 1.04 291.10 0.13 49.22 = 345.99 0.48 1.66 0.91 26.71 sde 1225.88 9931.88 39.54 224.84 4.94 39.70 = 345.81 1.20 4.53 2.44 64.55 sdc 19.25 11116.15 0.62 266.05 0.08 44.46 = 342.06 0.41 1.53 0.86 22.94 sdf 1206.63 11122.77 38.92 253.21 4.87 44.51 = 346.17 1.39 4.78 2.46 71.97 md2 0.00 0.00 0.00 634.37 0.00 79.30 = 256.00 0.00 0.00 0.00 0.00 This is running the same kernel - 3.8.0-rc7 from your git. And also for the sake completeness, the Dom0 grub.conf: title Scientific Linux (3.8.0-1.el6xen.x86_64) root (hd0,0) kernel /xen.gz dom0_mem=3D1024M cpufreq=3Dxen dom0_max_vcpus=3D1 = dom0_vcpus_pin module /vmlinuz-3.8.0-1.el6xen.x86_64 ro = root=3D/dev/vg_raid1/xenhost rd_LVM_LV=3Dvg_raid1/xenhost = rd_MD_UUID=3Dafb92c19:b9b1e3ae:07af315d:738e38be rd_NO_LUKS rd_NO_DM = LANG=3Den_US.UTF-8 SYSFONT=3Dlatarcyrheb-sun16 KEYBOARDTYPE=3Dpc KEYTABLE= =3Dus = crashkernel=3Dauto quiet panic=3D5 i915.i915_enable_rc6=3D7 = i915.i915_enable_fbc=3D1 i915.lvds_downclock=3D1 drm.vblankoffdelay=3D1 module /initramfs-3.8.0-1.el6xen.x86_64.img and the DomU config: # cat /etc/xen/zeus.vm name =3D "zeus.vm" memory =3D 1024 vcpus =3D 2 cpus =3D "1-3" disk =3D [ 'phy:/dev/vg_raid1/zeus.vm,xvda,w' , = 'phy:/dev/md2,xvdb,w' ] vif =3D [ "mac=3D02:16:36:35:35:09, bridge=3Dbr203, = vifname=3Dvm.zeus.203", "mac=3D10:16:36:35:35:09, bridge=3Dbr10, = vifname=3Dvm.zeus.10" ] bootloader =3D "pygrub" on_poweroff =3D 'destroy' on_reboot =3D 'restart' on_crash =3D 'restart' All the tests are being done on /dev/md2 (from Dom0) presented as xvdb = on the DomU. # cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] md2 : active raid6 sdd[5] sdc[4] sdf[1] sde[0] 3906766592 blocks super 1.2 level 6, 128k chunk, algorithm 2 = [4/4] [UUUU] -- = Steven Haigh Email: netwiz@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299