* Inode and dentry cache behavior
@ 2015-04-23 19:50 Shrinand Javadekar
2015-04-23 22:43 ` Dave Chinner
0 siblings, 1 reply; 9+ messages in thread
From: Shrinand Javadekar @ 2015-04-23 19:50 UTC (permalink / raw)
To: xfs
[-- Attachment #1: Type: text/plain, Size: 4822 bytes --]
Hi,
I am running Openstack Swift on a single server with 8 disks. All
these 8 disks are formatted with default XFS parameters. Each disk has
a capacity of 3TB. The machine has 64GB of data.
Here's what Openstack Swift does:
1. The file-system is mounted at /srv/node/r0.
2. Creates a temp file: /srv/node/r0/tmp/tmp_sdfsdf
3. Writes to this file: 4 writes of 64K each and does an fsync and
close. Final size of the file is 256K.
4. Create the path: /srv/node/r0/1004/eef/deadbeef. The directory
/srv/node/r0/objects/1004 already existed before. So it only needs to
create "eef" and "deadbeef". Before creating each directory, it
verifies that the directory does not exist.
5. Rename the file /srv/node/r0/tmp/tmp_sdfsdf to
/srv/node/r0/objects/1004/eef/deadbeef/foo.data.
6. fsync /srv/node/r0/objects/1004/eef/deadbeef/foo.data.
7. It then does a directory listing for /srv/node/r0/objects/1004/eef.
8. Opens the file /srv/node/r0/objects/1004/hashes.pkl
9. Writes to the file /srv/node/r0/objects/1004/hashes.pkl
10. Closes the file /srv/node/r0/objects/1004/hashes.pkl.
Writes are getting sharded across ~1024 directories. Essentially,
there are 0000-1024 directories under /srv/node/r0/objects/. 1004
above is one of them in the example above.
This works great when the filesystem is newly formatted and mounted.
However, as more and more data get's written to the system, the above
sequence of events progressively gets slower.
* We observe that the time for fsync remains pretty much constant throughout.
* What seems to be causing the performance to nosedive, is that inode
and dentry caching doesn't seem to be working.
* For experiment sake, we set vfs_cache_pressure to 0 so there would
be no reclaiming of inode and dentry cache entries. However, that does
not seem to help.
* We see openat() calls taking close to 1 second.
Any ideas, what might be causing this behavior? Are there other
params, specifically, xfs params that can be tuned for this workload.
The sequence of events above is the typical workload, at high
concurrency.
Here are the answers to other questions requested from the XFS wiki page:
* kernel version (uname -a)
3.13.0-39-generic #66-Ubuntu SMP Tue Oct 28 13:30:27 UTC 2014 x86_64
x86_64 x86_64 GNU/Linux
* xfsprogs version
xfs_repair version 3.1.7
* number of CPUs
16
* contents of /proc/meminfo
See attached file mem_info.
* contents of /proc/mounts
/dev/mapper/troll_data_vg_23578621012a_1-troll_data_lv_1 /srv/node/r0
xfs rw,nosuid,nodev,noexec,noatime,nodiratime,attr2,inode64,logbufs=8,noquota
0 0
/dev/mapper/troll_data_vg_23578621012a_2-troll_data_lv_2 /srv/node/r1
xfs rw,nosuid,nodev,noexec,noatime,nodiratime,attr2,inode64,logbufs=8,noquota
0 0
/dev/mapper/troll_data_vg_23578621012a_3-troll_data_lv_3 /srv/node/r2
xfs rw,nosuid,nodev,noexec,noatime,nodiratime,attr2,inode64,logbufs=8,noquota
0 0
/dev/mapper/troll_data_vg_23578621012a_4-troll_data_lv_4 /srv/node/r3
xfs rw,nosuid,nodev,noexec,noatime,nodiratime,attr2,inode64,logbufs=8,noquota
0 0
/dev/mapper/troll_data_vg_23578621012a_5-troll_data_lv_5 /srv/node/r4
xfs rw,nosuid,nodev,noexec,noatime,nodiratime,attr2,inode64,logbufs=8,noquota
0 0
/dev/mapper/troll_data_vg_23578621012a_6-troll_data_lv_6 /srv/node/r5
xfs rw,nosuid,nodev,noexec,noatime,nodiratime,attr2,inode64,logbufs=8,noquota
0 0
/dev/mapper/troll_data_vg_23578621012a_7-troll_data_lv_7 /srv/node/r6
xfs rw,nosuid,nodev,noexec,noatime,nodiratime,attr2,inode64,logbufs=8,noquota
0 0
/dev/mapper/troll_data_vg_23578621012a_8-troll_data_lv_8 /srv/node/r7
xfs rw,nosuid,nodev,noexec,noatime,nodiratime,attr2,inode64,logbufs=8,noquota
0 0
* contents of /proc/partitions
See attached file partitions_info.
* RAID layout (hardware and/or software)
NO RAID!!
* LVM configuration
See attached file lvm_info. Use lvdisplay to obtain it.
* type of disks you are using
sdm disk 2.7T ST3000NXCLAR3000
sdm1 part 1M
sdm2 part 2.7T
dm-1 lvm 2.7T
* write cache status of drives
Drives have no write cache.
* size of BBWC and mode it is running in
No BBWC
* xfs_info output on the filesystem in question
meta-data=/dev/mapper/troll_data_vg_23578621012a_8-troll_data_lv_8
isize=256 agcount=4, agsize=183141376 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=732565504, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=357698, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
* dmesg output showing all error messages and stack traces
No errors.
* IOStat and VMStat output.
See the attached files iostat_log and vmstat_log.
-Shri
[-- Attachment #2: mem_info --]
[-- Type: application/octet-stream, Size: 1198 bytes --]
MemTotal: 65954316 kB
MemFree: 38121680 kB
Buffers: 82884 kB
Cached: 7181368 kB
SwapCached: 205040 kB
Active: 9116896 kB
Inactive: 6713328 kB
Active(anon): 7450984 kB
Inactive(anon): 1115664 kB
Active(file): 1665912 kB
Inactive(file): 5597664 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16777212 kB
SwapFree: 15746448 kB
Dirty: 2848 kB
Writeback: 0 kB
AnonPages: 8445388 kB
Mapped: 29240 kB
Shmem: 676 kB
Slab: 8900756 kB
SReclaimable: 8034432 kB
SUnreclaim: 866324 kB
KernelStack: 11456 kB
PageTables: 55408 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49754368 kB
Committed_AS: 35197292 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 1973428 kB
VmallocChunk: 34357624832 kB
HardwareCorrupted: 0 kB
AnonHugePages: 6385664 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 137956 kB
DirectMap2M: 11372544 kB
DirectMap1G: 57671680 kB
[-- Attachment #3: partitions_info --]
[-- Type: application/octet-stream, Size: 1549 bytes --]
major minor #blocks name
11 0 1048575 sr0
8 32 2930266584 sdc
8 33 1024 sdc1
8 34 2930264064 sdc2
8 48 2930266584 sdd
8 49 1024 sdd1
8 50 2930264064 sdd2
8 64 2930266584 sde
8 65 1024 sde1
8 66 2930264064 sde2
8 80 2930266584 sdf
8 81 1024 sdf1
8 82 2930264064 sdf2
8 96 2930266584 sdg
8 97 1024 sdg1
8 98 2930264064 sdg2
8 112 2930266584 sdh
8 113 1024 sdh1
8 114 2930264064 sdh2
8 128 2930266584 sdi
8 129 1024 sdi1
8 130 2930264064 sdi2
8 144 2930266584 sdj
8 145 1024 sdj1
8 146 2930264064 sdj2
8 160 2930266584 sdk
8 161 1024 sdk1
8 162 2930264064 sdk2
8 176 2930266584 sdl
8 177 1024 sdl1
8 178 2930264064 sdl2
8 192 2930266584 sdm
8 193 1024 sdm1
8 194 2930264064 sdm2
8 208 2930266584 sdn
8 209 1024 sdn1
8 210 2930264064 sdn2
9 127 2930132800 md127
9 126 2930132800 md126
252 0 2930262016 dm-0
252 1 2930262016 dm-1
252 2 2930262016 dm-2
252 3 2930262016 dm-3
252 4 2930262016 dm-4
252 5 2930262016 dm-5
252 6 52428800 dm-6
252 7 5242880 dm-7
252 8 16777216 dm-8
252 9 1465065472 dm-9
252 10 2930262016 dm-10
252 11 2930262016 dm-11
[-- Attachment #4: iostat_log --]
[-- Type: application/octet-stream, Size: 75862 bytes --]
Linux 3.13.0-39-generic (tie-fighter-bottom) 04/23/15 _x86_64_ (16 CPU)
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 1.17 8.34 1.00 25.02 0.03 0.19 16.83 18.17 698.17 14.07 725.56 7.30 18.99
sdd 0.25 8.34 0.32 25.02 0.00 0.19 15.37 19.22 758.42 128.93 766.38 7.59 19.22
sde 0.04 0.00 0.03 0.00 0.00 0.00 41.07 0.00 1.24 0.70 11.65 0.88 0.00
sdf 0.04 0.00 0.01 0.00 0.00 0.00 50.55 0.00 2.40 0.59 13.31 1.88 0.00
sdg 53.59 17.14 3.81 37.84 0.04 2.25 112.47 21.87 525.02 38.80 573.99 6.14 25.56
sdh 29.27 16.86 2.87 34.83 0.03 2.21 121.63 17.71 469.76 29.50 506.07 6.13 23.12
sdi 32.16 17.35 3.42 39.10 0.03 2.25 109.88 20.98 493.43 43.78 532.75 5.92 25.17
sdj 30.56 17.38 3.41 38.71 0.03 2.25 110.73 22.45 532.99 51.77 575.40 6.07 25.56
sdk 0.04 17.06 1.37 34.99 0.01 2.22 125.67 19.09 525.05 61.21 543.22 6.25 22.73
sdl 0.04 17.21 2.37 36.41 0.01 2.23 118.76 18.45 475.65 39.40 504.09 6.06 23.49
sdm 0.04 16.94 2.09 34.52 0.01 2.23 125.19 16.86 460.47 33.40 486.29 6.05 22.13
sdn 15.53 17.08 2.58 35.26 0.02 2.22 121.25 17.84 471.32 33.35 503.37 6.05 22.90
md127 0.00 0.00 2.66 32.93 0.03 0.18 12.55 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.02 0.00 0.00 0.00 36.83 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 18.07 52.34 0.02 2.22 65.16 25.88 367.56 7.38 491.88 3.25 22.90
dm-1 0.00 0.00 2.08 51.46 0.01 2.23 85.59 24.69 461.19 34.58 478.41 4.13 22.13
dm-2 0.00 0.00 2.36 53.62 0.01 2.23 82.26 26.97 481.81 41.32 501.24 4.20 23.49
dm-3 0.00 0.00 1.37 52.06 0.01 2.22 85.54 28.14 526.69 64.21 538.82 4.26 22.73
dm-4 0.00 0.00 33.92 56.09 0.03 2.25 51.81 31.84 353.70 7.88 562.83 2.84 25.56
dm-5 0.00 0.00 35.53 56.45 0.03 2.25 50.79 29.90 325.08 6.76 525.46 2.74 25.17
dm-6 0.00 0.00 0.96 25.96 0.03 0.16 14.05 20.27 753.15 38.99 779.47 6.98 18.78
dm-7 0.00 0.00 0.07 1.23 0.00 0.01 9.35 0.21 162.41 20.59 170.25 51.14 6.64
dm-8 0.00 0.00 1.63 5.40 0.01 0.02 8.00 10.55 1500.84 52.78 1937.41 2.76 1.94
dm-9 0.00 0.00 0.01 0.00 0.00 0.00 54.05 0.00 2.85 1.17 20.10 1.61 0.00
dm-10 0.00 0.00 32.10 51.70 0.03 2.21 54.73 25.96 309.85 5.29 498.95 2.76 23.12
dm-11 0.00 0.00 57.35 54.99 0.04 2.25 41.70 31.36 279.15 5.31 564.76 2.28 25.56
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 9.60 0.00 112.40 0.00 0.48 8.66 103.30 919.03 0.00 919.03 7.67 86.16
sdd 0.00 9.60 0.00 112.40 0.00 0.48 8.66 109.84 977.22 0.00 977.22 7.74 86.96
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md127 0.00 0.00 0.00 121.80 0.00 0.47 7.99 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 121.00 0.00 0.47 8.00 124.74 1030.95 0.00 1030.95 7.17 86.80
dm-7 0.00 0.00 0.00 0.60 0.00 0.00 8.00 0.72 1204.00 0.00 1204.00 784.00 47.04
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 1.00 0.60 1.60 0.01 0.01 13.45 0.02 9.09 1.33 12.00 8.00 1.76
sdd 0.00 1.00 0.20 1.60 0.00 0.01 9.33 0.04 20.44 8.00 22.00 19.56 3.52
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md127 0.00 0.00 0.80 2.00 0.01 0.01 10.86 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.80 1.40 0.01 0.01 12.36 0.04 19.64 3.00 29.14 13.82 3.04
dm-7 0.00 0.00 0.00 0.40 0.00 0.00 8.00 0.01 24.00 0.00 24.00 18.00 0.72
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 9.80 1.80 13.20 2.40 0.16 0.01 23.05 0.06 3.59 2.97 7.00 2.92 4.56
sdd 0.40 1.80 5.20 2.40 0.07 0.01 21.63 0.04 5.16 4.92 5.67 3.68 2.80
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 4.20 0.00 0.06 29.81 0.10 22.67 0.00 22.67 11.05 4.64
sdh 0.00 1.00 1.00 9.20 0.00 0.08 16.86 0.18 17.57 9.60 18.43 8.71 8.88
sdi 0.00 1.00 1.00 8.80 0.00 0.07 15.29 0.18 17.06 14.40 17.36 9.71 9.52
sdj 0.00 0.00 0.00 3.20 0.00 0.04 25.12 0.04 14.00 0.00 14.00 11.75 3.76
sdk 0.00 0.00 0.00 2.20 0.00 0.03 25.55 0.03 14.55 0.00 14.55 13.45 2.96
sdl 0.00 0.00 0.00 1.80 0.00 0.02 26.78 0.03 15.56 0.00 15.56 10.67 1.92
sdm 0.00 0.00 0.00 1.60 0.00 0.02 29.38 0.03 16.00 0.00 16.00 15.00 2.40
sdn 0.00 0.00 0.00 2.80 0.00 0.03 24.43 0.10 37.43 0.00 37.43 18.86 5.28
md127 0.00 0.00 28.60 3.40 0.23 0.01 15.55 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 2.80 0.00 0.03 24.43 0.10 37.43 0.00 37.43 18.86 5.28
dm-1 0.00 0.00 0.00 1.60 0.00 0.02 29.38 0.03 16.00 0.00 16.00 15.00 2.40
dm-2 0.00 0.00 0.00 1.80 0.00 0.02 26.78 0.03 15.56 0.00 15.56 10.67 1.92
dm-3 0.00 0.00 0.00 2.40 0.00 0.03 24.33 0.03 13.33 0.00 13.33 12.67 3.04
dm-4 0.00 0.00 0.00 3.20 0.00 0.04 25.12 0.04 14.00 0.00 14.00 11.75 3.76
dm-5 0.00 0.00 1.00 10.20 0.00 0.07 13.95 0.22 17.43 14.40 17.73 8.57 9.60
dm-6 0.00 0.00 16.20 1.80 0.18 0.01 21.60 0.08 4.18 3.26 12.44 1.87 3.36
dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00 0.01 10.67 0.00 10.67 7.33 0.88
dm-8 0.00 0.00 12.40 0.00 0.05 0.00 8.00 0.06 5.16 5.16 0.00 0.58 0.72
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 1.00 10.60 0.00 0.08 15.14 0.20 17.31 9.60 18.04 7.72 8.96
dm-11 0.00 0.00 0.00 4.20 0.00 0.06 29.81 0.10 22.67 0.00 22.67 11.05 4.64
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 10.20 5.40 59.00 0.05 0.27 9.93 55.52 614.15 0.89 670.28 6.63 42.72
sdd 0.00 9.20 1.00 66.00 0.01 0.29 9.04 45.54 539.15 4.80 547.25 6.41 42.96
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 9.40 0.80 39.40 0.01 2.01 102.97 4.48 111.36 97.00 111.65 7.00 28.16
sdh 0.00 17.40 0.20 85.00 0.00 2.78 66.89 6.89 80.88 96.00 80.85 7.30 62.16
sdi 0.00 13.80 0.40 65.60 0.01 2.01 62.48 2.47 37.71 6.00 37.90 8.59 56.72
sdj 0.00 12.00 0.60 40.60 0.01 2.40 119.77 4.10 99.57 14.67 100.83 6.43 26.48
sdk 0.00 11.40 0.40 33.80 0.00 2.52 151.23 3.98 116.42 26.00 117.49 7.09 24.24
sdl 0.00 14.20 0.80 51.20 0.01 2.31 91.13 10.08 97.12 48.00 97.89 6.18 32.16
sdm 0.00 13.80 1.60 53.40 0.02 2.22 83.13 9.69 103.56 16.00 106.19 6.33 34.80
sdn 0.00 13.80 0.80 53.60 0.01 1.77 67.01 6.56 120.65 28.00 122.03 5.68 30.88
md127 0.00 0.00 6.40 259.00 0.05 1.01 8.20 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.80 67.40 0.01 1.77 53.45 8.12 119.00 28.00 120.08 4.53 30.88
dm-1 0.00 0.00 1.60 78.80 0.02 2.28 58.56 11.13 82.80 16.00 84.15 4.33 34.80
dm-2 0.00 0.00 0.80 78.60 0.01 2.38 61.54 11.62 79.16 48.00 79.48 4.05 32.16
dm-3 0.00 0.00 0.40 45.00 0.00 2.52 113.87 4.82 106.29 26.00 107.00 5.32 24.16
dm-4 0.00 0.00 0.60 52.60 0.01 2.40 92.76 4.77 89.58 14.67 90.43 4.98 26.48
dm-5 0.00 0.00 0.40 79.00 0.01 2.00 51.86 3.11 39.49 6.00 39.66 7.13 56.64
dm-6 0.00 0.00 6.40 2.00 0.05 0.01 14.67 0.30 3.81 1.50 11.20 33.33 28.00
dm-7 0.00 0.00 0.00 1.60 0.00 0.01 8.00 0.04 23.00 0.00 23.00 5.00 0.80
dm-8 0.00 0.00 0.00 257.20 0.00 1.00 8.00 450.62 225.62 0.00 225.62 1.59 40.96
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.20 102.00 0.00 2.78 55.73 8.70 85.16 96.00 85.14 6.08 62.16
dm-11 0.00 0.00 0.80 48.80 0.01 2.01 83.45 5.26 106.05 97.00 106.20 5.68 28.16
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 29.00 0.20 120.00 0.00 0.55 9.41 135.73 1132.21 1128.00 1132.22 8.33 100.08
sdd 0.00 29.00 0.00 114.80 0.00 0.53 9.49 119.56 999.95 0.00 999.95 8.72 100.08
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 58.00 2.20 111.00 0.03 13.23 239.81 32.11 283.39 121.82 286.59 5.72 64.80
sdh 0.00 58.40 1.80 118.00 0.02 13.41 229.70 14.07 117.30 47.11 118.37 6.05 72.48
sdi 0.00 52.80 1.60 124.00 0.02 12.69 207.29 17.52 139.26 40.50 140.54 5.65 70.96
sdj 0.00 55.00 2.40 119.20 0.03 12.81 216.24 33.63 276.17 105.00 279.62 5.22 63.44
sdk 0.00 55.80 3.00 113.40 0.04 12.69 224.04 21.16 180.12 110.13 181.97 5.90 68.72
sdl 0.00 52.60 1.80 107.20 0.03 13.55 255.08 7.65 116.23 64.89 117.09 5.91 64.40
sdm 0.00 45.40 1.80 95.20 0.02 11.47 242.72 6.82 111.33 61.78 112.27 6.43 62.40
sdn 0.00 48.20 2.20 98.00 0.03 12.64 259.01 5.60 55.70 29.45 56.29 5.80 58.08
md127 0.00 0.00 0.40 216.00 0.00 0.84 8.00 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 2.20 146.60 0.03 12.65 174.49 6.93 46.47 29.45 46.72 3.90 58.08
dm-1 0.00 0.00 1.80 129.40 0.02 11.51 179.97 8.61 99.57 61.78 100.09 4.76 62.40
dm-2 0.00 0.00 1.80 147.40 0.03 13.59 186.89 9.69 100.54 64.89 100.98 4.32 64.40
dm-3 0.00 0.00 3.20 171.60 0.04 12.94 152.12 27.22 154.03 103.25 154.98 3.93 68.72
dm-4 0.00 0.00 2.60 174.80 0.03 12.82 148.41 45.20 254.54 96.92 256.89 3.58 63.44
dm-5 0.00 0.00 1.80 177.40 0.02 12.84 147.01 21.90 121.89 36.00 122.76 3.96 70.96
dm-6 0.00 0.00 0.40 1.00 0.00 0.00 8.00 3.01 1254.29 564.00 1530.40 714.86 100.08
dm-7 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.72 0.00 0.00 0.00 3576.00 71.52
dm-8 0.00 0.00 0.00 215.80 0.00 0.84 8.00 1564.20 2913.07 0.00 2913.07 4.64 100.08
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 1.80 177.60 0.02 13.54 154.82 17.49 97.33 47.11 97.84 4.04 72.48
dm-11 0.00 0.00 2.20 170.00 0.03 13.30 158.53 44.29 257.05 121.82 258.80 3.76 64.80
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 42.20 0.40 120.00 0.00 0.65 11.02 135.26 1128.65 612.00 1130.37 8.31 100.00
sdd 0.00 42.20 0.00 120.40 0.00 0.65 11.03 128.65 1076.28 0.00 1076.28 8.31 100.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 76.80 2.40 133.80 0.03 10.31 155.34 68.96 390.07 79.00 395.65 6.35 86.48
sdh 0.00 36.80 0.60 61.60 0.01 9.96 328.13 1.81 29.34 50.67 29.13 5.80 36.08
sdi 0.00 40.00 2.40 69.40 0.03 10.90 311.97 2.73 38.38 37.67 38.41 5.04 36.16
sdj 0.00 93.00 1.60 161.80 0.02 9.28 116.57 106.47 602.55 45.00 608.06 5.25 85.76
sdk 0.00 40.80 1.00 70.20 0.01 11.20 322.66 3.61 53.39 68.80 53.17 5.57 39.68
sdl 0.00 98.00 1.80 171.00 0.02 10.07 119.54 79.45 372.29 24.44 375.95 5.00 86.40
sdm 0.00 94.60 2.40 154.40 0.03 12.36 161.84 79.33 446.40 95.00 451.86 5.43 85.12
sdn 0.00 42.60 1.20 70.40 0.02 11.39 326.35 2.76 38.77 81.33 38.05 4.91 35.12
md127 0.00 0.00 0.20 222.00 0.00 0.87 7.99 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 1.20 112.60 0.02 11.39 205.24 4.12 36.29 81.33 35.81 3.09 35.12
dm-1 0.00 0.00 2.40 276.60 0.03 12.49 91.86 107.85 326.80 95.00 328.81 3.05 85.12
dm-2 0.00 0.00 1.80 294.60 0.02 10.12 70.09 118.62 328.19 24.44 330.04 2.91 86.40
dm-3 0.00 0.00 0.80 108.60 0.01 10.96 205.31 5.05 48.92 86.00 48.65 3.63 39.68
dm-4 0.00 0.00 1.60 285.40 0.02 9.48 67.81 153.44 494.64 45.50 497.15 2.99 85.84
dm-5 0.00 0.00 2.20 108.80 0.03 10.75 199.01 3.88 35.43 41.09 35.32 3.26 36.16
dm-6 0.00 0.00 0.20 22.40 0.00 0.09 8.00 22.28 839.01 1224.00 835.57 44.25 100.00
dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00 2.82 1517.33 0.00 1517.33 833.33 100.00
dm-8 0.00 0.00 0.00 195.20 0.00 0.76 8.00 1727.51 4178.89 0.00 4178.89 5.12 100.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.60 97.20 0.01 9.83 206.07 2.57 26.47 50.67 26.32 3.70 36.16
dm-11 0.00 0.00 2.40 236.60 0.03 10.44 89.63 97.26 297.77 79.00 299.99 3.62 86.48
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 86.60 0.00 121.40 0.00 0.83 14.04 116.16 1065.27 0.00 1065.27 8.24 100.00
sdd 0.00 86.60 0.00 101.00 0.00 0.73 14.89 150.17 1448.80 0.00 1448.80 9.90 100.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 62.40 0.00 146.80 0.00 1.02 14.23 116.97 904.81 0.00 904.81 5.99 87.92
sdh 0.00 68.20 1.40 115.80 0.01 0.74 13.20 72.71 505.64 13.14 511.59 4.48 52.56
sdi 0.00 37.00 1.60 80.40 0.01 0.56 14.18 71.13 707.57 18.50 721.28 6.37 52.24
sdj 0.00 16.20 0.20 96.40 0.00 0.67 14.29 84.29 956.07 160.00 957.72 7.24 69.92
sdk 0.00 37.00 0.80 73.60 0.01 0.51 14.22 70.09 708.34 14.00 715.89 6.82 50.72
sdl 0.00 24.60 0.00 118.80 0.00 0.70 12.10 99.02 960.82 0.00 960.82 7.39 87.76
sdm 0.00 36.00 0.80 166.00 0.01 1.01 12.49 108.72 707.78 100.00 710.71 5.96 99.44
sdn 0.00 29.40 1.00 62.80 0.01 0.43 14.21 58.74 726.42 16.00 737.73 6.68 42.64
md127 0.00 0.00 0.00 37.20 0.00 0.15 8.00 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 1.00 119.80 0.01 0.55 9.44 78.64 543.09 16.00 547.49 3.53 42.64
dm-1 0.00 0.00 0.80 174.00 0.01 0.79 9.31 142.50 910.71 100.00 914.44 5.69 99.44
dm-2 0.00 0.00 0.00 117.00 0.00 0.53 9.34 130.56 1298.49 0.00 1298.49 7.50 87.76
dm-3 0.00 0.00 0.80 139.80 0.01 0.69 10.12 97.54 528.37 14.00 531.31 3.61 50.72
dm-4 0.00 0.00 0.00 81.40 0.00 0.46 11.47 109.51 1486.97 0.00 1486.57 8.59 69.92
dm-5 0.00 0.00 1.60 147.60 0.01 0.77 10.75 90.35 463.13 18.50 467.95 3.50 52.24
dm-6 0.00 0.00 0.00 37.20 0.00 0.15 8.00 51.49 1203.33 0.00 1203.33 26.88 100.00
dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1178.85 0.00 0.00 0.00 0.00 100.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 1.40 215.80 0.01 0.95 9.06 109.29 417.79 13.14 420.42 2.42 52.56
dm-11 0.00 0.00 0.00 182.20 0.00 0.82 9.17 174.39 1100.53 0.00 1100.53 4.83 87.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 15.00 5.80 2.60 131.00 0.07 0.54 9.26 15.23 124.68 11.69 126.93 6.99 93.36
sdd 0.00 6.00 0.00 126.60 0.00 0.53 8.64 135.44 1090.60 0.00 1090.60 7.90 100.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 20.40 0.40 31.00 0.01 5.35 349.62 1.22 38.85 24.00 39.05 6.29 19.76
sdh 0.00 51.20 0.40 141.00 0.00 6.87 99.53 106.66 849.40 172.00 851.32 7.06 99.76
sdi 0.00 78.00 0.00 169.40 0.00 4.66 56.31 122.11 797.38 0.00 797.38 5.90 100.00
sdj 0.00 26.80 0.40 45.80 0.01 7.15 317.24 1.97 42.60 16.00 42.83 5.45 25.20
sdk 0.00 77.00 0.00 152.00 0.00 2.35 31.72 145.96 997.58 0.00 997.58 6.58 100.00
sdl 0.00 19.80 0.80 31.20 0.01 5.21 333.94 1.48 46.25 36.00 46.51 6.40 20.48
sdm 0.00 22.20 1.00 34.40 0.01 5.80 336.13 1.50 42.49 20.00 43.14 6.03 21.36
sdn 0.00 86.60 0.00 172.00 0.00 1.85 22.08 132.23 807.73 0.00 807.73 5.81 100.00
md127 0.00 0.00 17.60 64.00 0.07 0.25 8.00 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.20 242.00 0.00 4.14 35.02 196.76 820.22 0.00 820.90 4.13 100.00
dm-1 0.00 0.00 1.00 56.80 0.01 5.85 207.64 2.22 38.38 20.00 38.70 3.70 21.36
dm-2 0.00 0.00 1.00 51.20 0.01 5.26 206.70 2.21 42.30 28.80 42.56 3.92 20.48
dm-3 0.00 0.00 0.00 218.80 0.00 3.87 36.27 215.95 984.25 0.00 984.25 4.57 100.00
dm-4 0.00 0.00 0.40 73.00 0.01 7.21 201.25 2.95 40.21 16.00 40.34 3.43 25.20
dm-5 0.00 0.00 0.00 223.00 0.00 5.45 50.04 185.59 926.58 0.00 926.58 4.48 100.00
dm-6 0.00 0.00 0.00 64.00 0.00 0.25 8.00 69.61 1085.12 0.00 1085.12 15.62 100.00
dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
dm-8 0.00 0.00 17.60 0.00 0.07 0.00 8.00 586.68 70627.14 6.82 0.00 56.82 100.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.40 160.80 0.00 6.76 85.97 151.90 1057.35 172.00 1059.56 6.19 99.76
dm-11 0.00 0.00 0.40 51.80 0.01 5.40 212.27 1.91 36.55 24.00 36.65 3.79 19.76
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 9.40 0.60 78.80 0.01 0.36 9.37 5.61 75.67 40.00 75.94 7.68 60.96
sdd 0.00 9.40 0.00 95.20 0.00 0.42 9.06 118.76 1268.11 0.00 1268.11 10.50 100.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 77.60 1.20 124.20 0.02 20.43 333.99 4.34 34.56 20.00 34.71 7.29 91.36
sdh 0.00 85.00 0.60 134.60 0.01 22.51 341.14 4.68 34.59 53.33 34.51 6.69 90.48
sdi 0.00 81.40 1.40 149.80 0.02 22.69 307.65 4.29 29.16 17.71 29.27 5.80 87.76
sdj 0.00 88.40 1.00 147.60 0.01 23.44 323.28 5.57 37.31 92.80 36.93 5.77 85.68
sdk 0.00 89.60 1.80 166.60 0.02 25.67 312.40 14.88 157.98 78.22 158.84 5.72 96.32
sdl 0.00 84.60 1.80 141.20 0.02 22.59 323.83 4.33 30.30 27.56 30.34 6.51 93.12
sdm 0.00 94.40 1.40 191.60 0.01 18.77 199.30 94.20 486.22 178.86 488.46 5.13 99.04
sdn 0.00 90.80 0.80 168.20 0.01 26.58 322.20 7.17 76.09 222.00 75.40 5.31 89.68
md127 0.00 0.00 0.60 15.20 0.01 0.07 10.13 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.60 248.80 0.01 24.33 199.88 11.29 89.91 296.00 89.41 3.60 89.68
dm-1 0.00 0.00 1.40 290.40 0.01 19.43 136.43 122.03 416.44 178.86 417.59 3.39 99.04
dm-2 0.00 0.00 1.60 225.80 0.02 22.55 203.23 6.45 28.36 31.00 28.34 4.09 93.12
dm-3 0.00 0.00 1.80 237.60 0.02 23.97 205.26 25.28 205.19 78.22 206.15 4.02 96.32
dm-4 0.00 0.00 1.00 237.00 0.01 23.60 203.18 8.38 35.04 92.80 34.80 3.60 85.68
dm-5 0.00 0.00 1.40 226.20 0.02 21.84 196.72 6.58 29.76 17.71 29.83 3.86 87.84
dm-6 0.00 0.00 0.60 14.80 0.01 0.07 10.18 26.06 2433.77 40.00 2530.81 64.94 100.00
dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 226.59 0.00 0.00 0.00 0.00 100.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.60 219.60 0.01 22.51 209.46 7.25 32.92 53.33 32.86 4.12 90.64
dm-11 0.00 0.00 1.20 201.80 0.02 20.44 206.38 6.64 32.66 20.00 32.74 4.50 91.44
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 15.80 0.00 98.00 0.00 0.44 9.20 101.43 894.84 0.00 894.84 8.13 79.68
sdd 0.00 15.80 0.00 107.80 0.00 0.48 9.12 94.85 896.87 0.00 896.87 8.04 86.72
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 115.20 1.20 181.20 0.01 30.59 343.58 9.45 51.68 106.67 51.32 5.27 96.08
sdh 0.00 135.20 0.60 212.20 0.00 36.01 346.59 9.85 46.27 37.33 46.30 4.53 96.48
sdi 0.00 123.40 0.20 202.20 0.00 32.90 332.91 8.24 40.74 48.00 40.73 4.55 92.08
sdj 0.00 125.00 0.80 215.80 0.01 33.83 319.88 8.65 40.04 45.00 40.02 4.47 96.72
sdk 0.00 126.40 1.40 204.00 0.01 33.76 336.74 12.34 60.01 134.86 59.50 4.73 97.12
sdl 0.00 128.60 1.80 207.40 0.01 34.32 336.05 9.70 45.46 41.33 45.50 4.70 98.24
sdm 0.00 133.20 2.40 213.20 0.02 35.81 340.29 11.34 54.07 90.00 53.67 4.52 97.52
sdn 0.00 122.80 3.80 205.20 0.03 32.82 321.92 8.68 41.34 87.16 40.49 4.55 95.04
md127 0.00 0.00 0.00 141.80 0.00 0.57 8.28 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 4.00 328.40 0.03 32.92 203.05 12.38 37.02 82.80 36.46 2.86 95.04
dm-1 0.00 0.00 2.60 343.40 0.02 35.45 209.97 16.47 48.92 83.08 48.66 2.82 97.52
dm-2 0.00 0.00 2.00 337.40 0.01 34.36 207.42 14.22 41.34 37.20 41.37 2.89 98.24
dm-3 0.00 0.00 1.40 331.20 0.01 33.88 208.70 18.57 55.78 134.86 55.45 2.92 96.96
dm-4 0.00 0.00 0.80 340.00 0.01 33.76 202.94 12.23 35.97 45.00 35.95 2.84 96.72
dm-5 0.00 0.00 0.20 326.20 0.00 33.01 207.11 12.49 38.25 48.00 38.24 2.82 92.08
dm-6 0.00 0.00 0.00 140.20 0.00 0.57 8.31 108.66 664.22 0.00 664.22 6.32 88.56
dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00 0.24 458.00 0.00 458.00 112.67 13.52
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.72 0.00 0.00 0.00 0.00 10.32
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.60 347.60 0.00 36.01 211.85 14.28 40.98 37.33 40.98 2.77 96.40
dm-11 0.00 0.00 1.20 296.40 0.01 30.59 210.58 14.22 47.66 106.67 47.42 3.23 96.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 9.60 0.00 67.40 0.00 0.32 9.85 53.85 1002.71 0.00 1002.71 7.41 49.92
sdd 0.00 9.60 0.00 64.20 0.00 0.31 9.95 47.00 898.01 0.00 898.01 7.50 48.16
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 115.60 4.60 154.60 0.04 7.09 91.75 110.90 590.67 32.35 607.28 6.28 99.92
sdh 0.00 176.80 4.20 217.00 0.03 6.42 59.77 114.46 480.49 26.86 489.27 4.52 99.92
sdi 0.00 100.80 3.00 162.80 0.02 6.79 84.20 114.90 617.01 37.07 627.70 5.96 98.88
sdj 0.00 83.80 4.40 151.40 0.03 7.06 93.28 113.21 645.08 28.00 663.01 6.41 99.92
sdk 0.00 96.20 2.00 151.60 0.01 7.08 94.60 119.10 664.76 108.80 672.09 6.51 100.00
sdl 0.00 180.00 5.20 239.00 0.04 7.30 61.56 116.15 427.78 50.62 435.99 4.10 100.00
sdm 0.00 126.20 3.40 160.20 0.03 6.09 76.58 119.35 659.97 59.76 672.70 6.11 100.00
sdn 0.00 100.60 0.80 157.60 0.01 6.05 78.28 144.16 831.12 112.00 834.77 6.31 100.00
md127 0.00 0.00 0.00 49.20 0.00 0.19 7.93 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.60 283.00 0.00 6.05 43.75 217.24 656.59 149.33 657.67 3.53 100.00
dm-1 0.00 0.00 3.20 311.00 0.03 6.13 40.15 186.62 475.64 63.50 479.88 3.18 100.00
dm-2 0.00 0.00 5.00 446.60 0.04 7.45 33.95 202.44 401.82 52.64 405.73 2.21 100.00
dm-3 0.00 0.00 2.00 274.40 0.01 7.17 53.22 180.52 545.41 108.80 548.59 3.62 100.00
dm-4 0.00 0.00 4.40 262.80 0.03 7.05 54.32 168.54 575.42 28.00 584.58 3.74 99.92
dm-5 0.00 0.00 3.00 291.80 0.02 6.77 47.16 168.79 503.40 37.07 508.19 3.35 98.88
dm-6 0.00 0.00 0.00 48.20 0.00 0.19 8.00 66.26 1703.35 0.00 1703.35 10.34 49.84
dm-7 0.00 0.00 0.00 0.60 0.00 0.00 8.00 0.44 738.67 0.00 738.67 392.00 23.52
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 4.20 424.80 0.03 6.59 31.61 199.20 423.35 26.86 427.27 2.33 99.92
dm-11 0.00 0.00 4.60 299.20 0.04 7.27 49.29 181.25 493.27 32.35 500.36 3.29 99.92
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 7.60 0.00 30.20 0.00 0.15 9.92 17.85 591.10 0.00 591.10 8.00 24.16
sdd 0.00 7.60 0.00 30.20 0.00 0.15 9.92 13.98 463.07 0.00 463.07 6.81 20.56
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 73.40 0.00 146.40 0.00 0.92 12.81 144.67 1018.18 0.00 1018.18 6.83 100.00
sdh 0.00 87.80 0.00 152.60 0.00 1.07 14.39 143.37 883.83 0.00 883.83 6.55 100.00
sdi 0.00 184.00 0.00 195.40 0.00 1.67 17.47 142.90 722.81 0.00 722.81 5.12 100.00
sdj 0.00 214.00 0.00 219.00 0.00 1.81 16.95 143.13 633.37 0.00 633.37 4.57 100.00
sdk 0.00 81.60 0.00 140.60 0.00 1.12 16.26 144.54 1058.36 0.00 1058.36 7.11 100.00
sdl 0.00 63.00 0.00 124.60 0.00 0.87 14.33 143.12 1114.01 0.00 1114.01 8.03 100.00
sdm 0.00 127.80 0.00 166.60 0.00 1.49 18.37 141.54 829.65 0.00 829.65 6.00 100.00
sdn 0.00 134.80 0.00 178.80 0.00 1.45 16.65 143.58 781.05 0.00 781.05 5.59 100.00
md127 0.00 0.00 0.00 37.60 0.00 0.15 7.96 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 314.00 0.00 1.35 8.78 245.30 801.14 0.00 801.14 3.18 100.00
dm-1 0.00 0.00 0.00 287.40 0.00 1.24 8.81 251.82 916.00 0.00 916.00 3.48 100.00
dm-2 0.00 0.00 0.00 184.60 0.00 0.82 9.10 219.24 1175.71 0.00 1175.71 5.42 100.00
dm-3 0.00 0.00 0.00 224.60 0.00 1.27 11.57 210.28 954.65 0.00 954.65 4.45 100.00
dm-4 0.00 0.00 0.00 431.60 0.00 1.84 8.72 283.81 638.66 0.00 638.66 2.32 100.00
dm-5 0.00 0.00 0.00 376.00 0.00 1.60 8.73 283.28 751.75 0.00 751.75 2.66 100.00
dm-6 0.00 0.00 0.00 36.80 0.00 0.14 8.00 20.42 554.87 0.00 554.87 6.54 24.08
dm-7 0.00 0.00 0.00 0.60 0.00 0.00 8.00 0.01 14.67 0.00 14.67 6.67 0.40
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.00 235.00 0.00 1.01 8.78 232.17 929.60 0.00 929.60 4.26 100.00
dm-11 0.00 0.00 0.00 221.00 0.00 1.00 9.29 213.95 1001.02 0.00 1001.02 4.52 100.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 1.40 1.00 1.00 1.20 0.02 0.01 23.45 0.02 7.64 5.60 9.33 6.18 1.36
sdd 0.00 1.00 1.00 1.20 0.01 0.01 16.91 0.01 5.82 4.00 7.33 5.82 1.28
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 118.40 0.60 177.20 0.00 4.94 56.91 92.34 589.53 402.67 590.16 5.16 91.68
sdh 0.00 56.20 0.40 127.40 0.00 5.01 80.40 90.17 835.92 152.00 838.07 7.27 92.96
sdi 0.00 47.40 0.40 133.20 0.00 5.61 86.07 68.18 616.99 90.00 618.58 6.77 90.48
sdj 0.00 54.20 0.20 134.60 0.00 5.56 84.51 100.57 873.23 1292.00 872.61 7.04 94.96
sdk 0.00 128.60 0.20 180.20 0.00 4.45 50.50 141.94 761.77 700.00 761.84 5.54 100.00
sdl 0.00 42.20 1.00 152.00 0.01 5.44 72.90 107.96 811.41 25.60 816.58 6.54 100.00
sdm 0.00 41.40 1.80 149.60 0.01 5.55 75.23 78.41 615.20 90.67 621.51 6.13 92.80
sdn 0.00 63.40 1.00 174.80 0.01 4.97 57.95 111.35 694.38 310.40 696.58 5.69 100.00
md127 0.00 0.00 3.40 2.20 0.03 0.01 13.14 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 1.00 222.40 0.01 4.87 44.70 160.24 800.57 310.40 802.77 4.48 100.00
dm-1 0.00 0.00 1.80 171.80 0.01 5.41 63.97 109.18 777.48 90.67 784.68 5.35 92.80
dm-2 0.00 0.00 1.00 168.20 0.01 5.29 64.15 144.94 994.77 25.60 1000.53 5.91 100.00
dm-3 0.00 0.00 0.20 302.00 0.00 5.08 34.44 251.93 824.08 700.00 824.16 3.31 100.00
dm-4 0.00 0.00 0.20 162.00 0.00 5.39 68.13 153.72 1089.29 1292.00 1089.04 5.85 94.96
dm-5 0.00 0.00 0.40 154.60 0.00 5.44 71.93 99.85 779.83 90.00 781.62 5.84 90.48
dm-6 0.00 0.00 1.80 0.60 0.02 0.00 21.33 0.02 7.67 5.33 14.67 4.33 1.04
dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00 0.01 11.33 0.00 11.33 3.33 0.40
dm-8 0.00 0.00 1.60 0.00 0.01 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.40 157.60 0.00 4.81 62.43 150.52 1150.71 152.00 1153.24 5.88 92.96
dm-11 0.00 0.00 0.60 265.20 0.00 4.62 35.59 180.33 769.22 402.67 770.05 3.45 91.76
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 23.60 0.20 62.20 0.01 0.37 12.45 39.25 628.96 4.00 630.97 7.41 46.24
sdd 0.00 23.60 0.00 62.20 0.00 0.37 12.21 44.06 708.37 0.00 708.37 7.58 47.12
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 76.00 1.20 155.20 0.01 9.00 117.97 97.43 536.94 52.00 540.69 6.26 97.92
sdh 0.00 34.80 0.20 60.40 0.00 8.97 303.25 2.40 40.03 60.00 39.96 7.08 42.88
sdi 0.00 82.20 0.60 156.20 0.00 9.35 122.23 112.97 624.44 277.33 625.77 5.93 93.04
sdj 0.00 40.20 0.20 69.00 0.00 10.52 311.37 3.28 47.35 28.00 47.41 5.88 40.72
sdk 0.00 33.60 0.20 100.00 0.00 9.44 192.95 23.66 408.50 576.00 408.17 5.33 53.36
sdl 0.00 102.20 1.20 164.40 0.01 9.63 119.17 87.33 421.80 28.00 424.68 5.45 90.32
sdm 0.00 38.80 0.40 62.20 0.00 10.08 329.83 2.55 40.77 30.00 40.84 6.21 38.88
sdn 0.00 49.20 0.00 95.40 0.00 12.91 277.08 6.75 131.11 0.00 131.11 5.43 51.84
md127 0.00 0.00 0.20 87.00 0.01 0.37 8.90 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 134.00 0.00 12.85 196.40 9.29 115.93 0.00 115.93 3.87 51.84
dm-1 0.00 0.00 0.40 101.00 0.00 10.08 203.62 3.82 37.71 30.00 37.74 3.83 38.88
dm-2 0.00 0.00 1.20 294.40 0.01 10.34 71.71 133.32 368.16 28.00 369.55 3.06 90.32
dm-3 0.00 0.00 0.20 110.20 0.00 8.44 156.60 39.45 615.83 576.00 615.90 4.83 53.36
dm-4 0.00 0.00 0.20 109.20 0.00 10.52 196.95 5.01 45.83 28.00 45.86 3.72 40.72
dm-5 0.00 0.00 0.60 259.60 0.00 9.57 75.36 164.79 562.42 277.33 563.08 3.57 92.96
dm-6 0.00 0.00 0.20 84.40 0.01 0.37 9.12 58.91 696.29 4.00 697.93 5.61 47.44
dm-7 0.00 0.00 0.00 0.60 0.00 0.00 8.00 0.01 17.33 0.00 17.33 9.33 0.56
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.20 95.00 0.00 8.97 192.92 3.78 40.01 60.00 39.97 4.50 42.88
dm-11 0.00 0.00 1.20 250.80 0.01 9.54 77.62 133.58 411.71 52.00 413.44 3.88 97.84
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 3.79 0.00 16.57 0.00 0.08 10.51 2.66 160.58 0.00 160.58 5.01 8.30
sdd 0.00 3.79 0.00 16.57 0.00 0.08 10.51 2.48 149.54 0.00 149.54 5.54 9.18
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 86.23 0.20 162.87 0.00 22.76 285.90 12.97 161.80 108.00 161.87 6.00 97.88
sdh 0.00 90.42 0.20 142.32 0.00 23.74 341.23 6.58 46.12 96.00 46.05 5.70 81.28
sdi 0.00 86.63 0.40 170.66 0.00 23.21 277.89 14.26 170.88 16.00 171.25 5.61 96.05
sdj 0.00 88.42 0.40 151.90 0.00 23.50 315.99 5.71 37.33 10.00 37.40 5.48 83.43
sdk 0.00 100.40 2.20 139.72 0.02 23.43 338.38 12.62 67.57 64.73 67.62 5.89 83.59
sdl 0.00 91.82 0.60 187.82 0.00 23.63 256.89 66.23 443.95 458.67 443.90 5.30 99.80
sdm 0.00 93.81 0.40 148.90 0.00 24.75 339.47 4.84 32.41 82.00 32.27 5.55 82.79
sdn 0.00 82.24 0.20 136.33 0.00 21.86 327.94 5.66 41.44 108.00 41.34 5.68 77.60
md127 0.00 0.00 0.00 19.16 0.00 0.08 9.00 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.20 218.76 0.00 21.91 204.94 8.76 39.96 108.00 39.90 3.54 77.60
dm-1 0.00 0.00 0.40 242.71 0.00 24.75 208.47 7.11 29.26 82.00 29.17 3.41 82.79
dm-2 0.00 0.00 0.60 252.30 0.00 23.02 186.46 96.45 477.93 458.67 477.97 3.95 99.80
dm-3 0.00 0.00 2.20 275.85 0.02 24.29 179.00 18.59 52.28 64.73 52.18 3.01 83.59
dm-4 0.00 0.00 0.40 241.32 0.00 23.61 200.10 8.38 34.52 10.00 34.56 3.45 83.43
dm-5 0.00 0.00 0.40 237.13 0.00 23.20 200.01 19.20 157.94 16.00 158.18 4.05 96.13
dm-6 0.00 0.00 0.00 16.97 0.00 0.08 9.32 3.13 184.28 0.00 184.28 4.66 7.90
dm-7 0.00 0.00 0.00 1.80 0.00 0.01 8.00 0.02 9.78 0.00 9.78 4.44 0.80
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.20 233.33 0.00 23.81 208.77 9.91 42.36 96.00 42.31 3.48 81.28
dm-11 0.00 0.00 0.20 230.34 0.00 22.38 198.79 22.48 226.55 108.00 226.65 4.25 97.88
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 7.60 0.00 5.40 0.00 0.06 22.89 0.11 21.19 0.00 21.19 7.11 3.84
sdd 0.00 7.60 0.00 5.40 0.00 0.06 22.89 0.09 17.33 0.00 17.33 6.52 3.52
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 1.40 0.00 13.40 0.00 0.47 71.24 0.29 22.21 0.00 22.21 11.46 15.36
sdh 0.00 136.40 2.00 191.00 0.02 2.31 24.70 130.75 588.34 34.80 594.13 5.18 100.00
sdi 0.00 2.20 0.00 5.80 0.00 0.81 286.10 0.12 30.76 0.00 30.76 5.79 3.36
sdj 0.00 131.40 2.00 173.40 0.02 1.71 20.19 135.27 722.63 20.80 730.73 5.70 99.92
sdk 0.00 53.20 0.00 135.00 0.00 1.80 27.36 151.00 1008.27 0.00 1008.27 7.41 100.00
sdl 0.00 6.40 1.40 16.80 0.01 0.89 101.66 4.68 49.01 11.43 52.14 7.74 14.08
sdm 0.00 65.00 3.00 119.40 0.02 1.19 20.27 107.29 777.80 12.80 797.03 6.78 83.04
sdn 0.00 121.20 3.20 151.40 0.03 1.47 19.86 131.98 796.25 14.25 812.78 6.46 99.92
md127 0.00 0.00 0.00 13.60 0.00 0.06 9.06 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 3.20 302.20 0.03 1.72 11.71 214.04 621.61 14.25 628.04 3.27 99.92
dm-1 0.00 0.00 3.00 216.80 0.02 1.44 13.60 145.96 588.48 12.80 596.44 3.78 83.04
dm-2 0.00 0.00 1.40 53.60 0.01 0.92 34.60 5.11 20.65 11.43 20.90 2.56 14.08
dm-3 0.00 0.00 0.00 184.20 0.00 1.14 12.70 216.31 1059.53 0.00 1059.53 5.43 100.00
dm-4 0.00 0.00 2.00 330.00 0.02 1.80 11.21 221.52 616.64 20.80 620.25 3.01 99.92
dm-5 0.00 0.00 0.00 6.80 0.00 0.61 182.76 0.19 44.12 0.00 44.12 4.94 3.36
dm-6 0.00 0.00 0.00 9.80 0.00 0.05 10.29 0.14 14.53 0.00 14.53 3.84 3.76
dm-7 0.00 0.00 0.00 2.80 0.00 0.01 8.00 0.13 44.86 0.00 44.86 6.86 1.92
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 2.00 354.00 0.02 2.48 14.38 222.84 526.82 34.80 529.60 2.81 100.00
dm-11 0.00 0.00 0.00 13.80 0.00 0.31 46.23 0.32 25.04 0.00 25.04 11.13 15.36
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 3.40 0.00 1.20 0.00 0.02 28.33 0.01 9.33 0.00 9.33 9.33 1.12
sdd 0.00 3.40 0.00 1.20 0.00 0.02 28.33 0.01 7.33 0.00 7.33 7.33 0.88
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 37.00 2.20 68.80 0.02 0.78 23.05 55.80 556.30 12.36 573.70 6.25 44.40
sdh 0.00 46.60 0.20 126.20 0.00 1.54 24.91 122.78 1100.97 108.00 1102.55 7.91 100.00
sdi 0.00 101.20 1.40 148.80 0.01 1.60 22.01 124.34 757.14 12.57 764.15 5.78 86.80
sdj 0.00 56.80 0.00 137.20 0.00 1.61 24.00 133.07 1011.15 0.00 1011.15 7.29 100.00
sdk 0.00 96.60 0.00 180.80 0.00 2.00 22.63 129.32 790.77 0.00 790.77 5.53 100.00
sdl 0.00 120.40 0.00 175.60 0.00 1.83 21.40 147.67 761.28 0.00 761.28 5.69 100.00
sdm 0.00 136.40 0.00 175.60 0.00 1.77 20.64 146.09 808.67 0.00 808.67 5.69 100.00
sdn 0.00 72.40 0.00 157.20 0.00 1.71 22.29 130.47 875.54 0.00 875.54 6.36 100.00
md127 0.00 0.00 0.00 4.60 0.00 0.02 7.30 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 201.80 0.00 1.72 17.47 211.67 1154.55 0.00 1154.55 4.96 100.00
dm-1 0.00 0.00 0.00 306.20 0.00 1.85 12.40 263.02 834.76 0.00 834.76 3.27 100.00
dm-2 0.00 0.00 0.00 297.40 0.00 2.17 14.97 243.07 726.71 0.00 726.71 3.36 100.00
dm-3 0.00 0.00 0.00 250.80 0.00 2.08 16.95 208.56 885.29 0.00 885.29 3.99 100.00
dm-4 0.00 0.00 0.00 170.20 0.00 1.57 18.89 201.92 1251.22 0.00 1251.22 5.88 100.00
dm-5 0.00 0.00 1.40 279.40 0.01 1.89 13.85 192.01 636.03 12.57 639.16 3.09 86.80
dm-6 0.00 0.00 0.00 3.60 0.00 0.01 8.00 0.04 12.22 0.00 12.22 1.56 0.56
dm-7 0.00 0.00 0.00 0.60 0.00 0.00 8.00 0.01 13.33 0.00 13.33 8.00 0.48
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.20 149.20 0.00 1.32 18.16 184.72 1467.20 108.00 1469.02 6.69 100.00
dm-11 0.00 0.00 2.20 136.00 0.02 1.06 15.93 76.68 401.73 12.36 408.02 3.21 44.40
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 8.60 0.00 6.00 0.00 0.06 22.00 0.25 41.07 0.00 41.07 6.00 3.60
sdd 0.00 8.60 0.00 6.00 0.00 0.06 22.00 0.77 128.67 0.00 128.67 11.73 7.04
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 65.20 0.00 139.00 0.00 1.40 20.66 150.56 1105.70 0.00 1105.70 7.19 100.00
sdh 0.00 9.40 0.20 37.80 0.00 1.32 71.02 8.06 233.66 28.00 234.75 7.94 30.16
sdi 0.00 36.80 0.20 120.00 0.00 2.24 38.24 103.54 949.73 24.00 951.27 7.75 93.12
sdj 0.00 9.00 0.00 57.80 0.00 1.72 60.98 10.41 230.02 0.00 230.02 7.02 40.56
sdk 0.00 12.00 0.20 68.60 0.00 1.75 52.13 22.96 395.76 4.00 396.90 7.14 49.12
sdl 0.00 11.80 0.00 65.20 0.00 1.81 56.89 48.62 1018.75 0.00 1018.75 8.83 57.60
sdm 0.00 5.20 0.00 42.60 0.00 1.70 81.77 16.09 757.46 0.00 757.46 10.54 44.88
sdn 0.00 7.80 0.20 40.00 0.00 1.51 77.04 5.55 180.88 40.00 181.58 7.20 28.96
md127 0.00 0.00 0.00 14.20 0.00 0.06 9.24 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.20 45.80 0.00 1.20 53.65 6.48 203.58 40.00 204.30 6.30 28.96
dm-1 0.00 0.00 0.00 21.20 0.00 1.37 132.22 32.83 2682.19 0.00 2682.19 21.17 44.88
dm-2 0.00 0.00 0.00 44.60 0.00 1.34 61.63 67.85 2215.12 0.00 2215.12 12.91 57.60
dm-3 0.00 0.00 0.20 75.40 0.00 1.48 40.03 28.23 528.52 4.00 529.91 6.50 49.12
dm-4 0.00 0.00 0.00 64.40 0.00 1.55 49.37 11.63 270.39 0.00 270.39 6.30 40.56
dm-5 0.00 0.00 0.20 127.40 0.00 1.96 31.45 154.61 1316.82 24.00 1318.84 7.30 93.12
dm-6 0.00 0.00 0.00 13.20 0.00 0.06 9.58 2.60 197.27 0.00 197.27 4.18 5.52
dm-7 0.00 0.00 0.00 0.60 0.00 0.00 8.00 0.02 25.33 0.00 25.33 22.67 1.36
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.20 43.60 0.00 1.30 60.63 10.39 256.02 28.00 257.06 6.89 30.16
dm-11 0.00 0.00 0.00 201.00 0.00 1.39 14.18 213.12 1018.81 0.00 1018.81 4.98 100.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 11.40 0.00 53.60 0.00 0.25 9.71 44.59 775.46 0.00 775.46 7.87 42.16
sdd 0.00 11.40 0.00 51.40 0.00 0.25 9.79 49.11 845.37 0.00 845.37 8.14 41.84
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 34.40 0.00 91.80 0.00 2.80 62.49 26.25 429.39 0.00 429.39 6.39 58.64
sdh 0.00 3.40 0.20 17.00 0.00 0.82 97.20 0.47 27.49 32.00 27.44 13.67 23.52
sdi 0.00 7.20 0.20 20.20 0.00 1.93 194.30 0.58 28.47 44.00 28.32 17.33 35.36
sdj 0.00 10.80 0.20 47.00 0.00 2.22 96.24 7.92 167.71 136.00 167.85 8.31 39.20
sdk 0.00 4.40 0.20 16.60 0.00 1.16 141.70 0.54 32.33 112.00 31.37 12.10 20.32
sdl 0.00 5.80 0.00 18.60 0.00 1.61 177.61 0.42 22.32 0.00 22.32 13.12 24.40
sdm 0.00 3.40 0.20 14.20 0.00 0.86 122.60 0.34 23.89 32.00 23.77 14.28 20.56
sdn 0.00 6.20 0.00 19.60 0.00 1.74 182.19 0.60 30.78 0.00 30.78 15.80 30.96
md127 0.00 0.00 0.20 67.60 0.00 0.27 8.05 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 25.80 0.00 1.74 138.41 0.91 35.13 0.00 35.13 12.00 30.96
dm-1 0.00 0.00 0.20 17.60 0.00 0.86 99.18 0.43 24.40 32.00 24.32 11.55 20.56
dm-2 0.00 0.00 0.00 24.40 0.00 1.61 135.39 0.56 22.92 0.00 22.92 10.00 24.40
dm-3 0.00 0.00 0.20 21.00 0.00 1.16 112.29 0.70 33.09 112.00 32.34 9.58 20.32
dm-4 0.00 0.00 0.20 57.80 0.00 2.22 78.32 9.42 162.47 136.00 162.56 6.76 39.20
dm-5 0.00 0.00 0.20 27.40 0.00 1.93 143.62 0.87 31.51 44.00 31.42 12.81 35.36
dm-6 0.00 0.00 0.20 67.20 0.00 0.26 8.07 61.64 829.88 0.00 832.35 6.18 41.68
dm-7 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 8.00 0.00 8.00 8.00 0.16
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.20 20.40 0.00 0.82 81.16 0.66 32.12 32.00 32.12 11.42 23.52
dm-11 0.00 0.00 0.00 99.20 0.00 2.54 52.35 55.43 856.07 0.00 856.07 5.91 58.64
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 2.40 1.40 4.80 0.01 0.02 10.48 0.16 514.32 2.29 663.67 5.68 3.52
sdd 0.00 2.40 2.20 7.00 0.02 0.03 12.46 0.60 680.43 153.09 846.17 8.26 7.60
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 12.00 0.00 12.00 12.00 0.24
sdi 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 20.00 0.00 20.00 20.00 0.40
sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md127 0.00 0.00 3.40 3.60 0.03 0.01 12.11 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 20.00 0.00 20.00 20.00 0.40
dm-6 0.00 0.00 3.40 2.20 0.03 0.01 13.71 0.61 1128.71 100.00 2718.55 12.43 6.96
dm-7 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.01 8.80 0.00 8.80 4.00 0.40
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 12.00 0.00 12.00 12.00 0.24
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 0.20 0.00 1.60 0.00 0.00 5.50 0.01 6.50 0.00 6.50 6.00 0.96
sdd 0.00 0.20 0.00 1.60 0.00 0.00 5.50 0.01 6.00 0.00 6.00 5.00 0.80
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdm 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdn 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md127 0.00 0.00 0.00 1.20 0.00 0.00 6.67 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-7 0.00 0.00 0.00 1.00 0.00 0.00 8.00 0.01 12.00 0.00 12.00 8.80 0.88
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sdc 0.00 0.20 0.20 1.40 0.00 0.00 6.50 0.01 5.50 0.00 6.29 5.50 0.88
sdd 0.00 0.20 0.00 1.40 0.00 0.00 5.14 0.01 6.29 0.00 6.29 6.29 0.88
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdl 0.00 13.00 0.40 7.80 0.00 0.04 10.49 4.34 80.10 6.00 83.90 4.00 3.28
sdm 0.00 10.00 0.40 6.40 0.00 0.04 13.62 4.02 64.00 12.00 67.25 4.82 3.28
sdn 0.00 7.20 0.60 37.20 0.00 0.22 12.04 16.48 362.58 17.33 368.15 4.93 18.64
md127 0.00 0.00 0.20 1.00 0.00 0.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00
md126 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.60 48.60 0.00 0.23 9.93 19.06 330.55 17.33 334.42 3.79 18.64
dm-1 0.00 0.00 0.40 42.20 0.00 0.20 9.62 5.34 11.53 12.00 11.53 0.77 3.28
dm-2 0.00 0.00 0.40 51.60 0.00 0.25 9.82 6.20 13.62 6.00 13.67 0.63 3.28
dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-6 0.00 0.00 0.20 0.60 0.00 0.00 10.00 0.01 7.00 0.00 9.33 5.00 0.40
dm-7 0.00 0.00 0.00 0.20 0.00 0.00 8.00 0.00 20.00 0.00 20.00 20.00 0.40
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-9 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
[-- Attachment #5: vmstat_log --]
[-- Type: application/octet-stream, Size: 1935 bytes --]
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 1123900 908848 6675332 235880 0 1 14 1157 16 11 6 1 78 15 0
1 0 1123900 895000 6675348 235944 0 0 2 13 315 725 0 0 100 0 0
0 1 1123864 747792 6675364 238260 50 0 246 865 1229 3895 7 1 88 4 0
3 4 1133272 886456 6630516 239840 0 1883 127 26284 3652 26584 12 4 65 19 0
31 58 1137216 861152 6620676 276552 0 790 336 157334 13896 87023 43 9 23 25 0
1 9 1137212 926880 6620640 243296 0 0 58 31686 3987 18218 8 2 44 46 0
7 20 1137212 902476 6620616 255004 0 0 48 7690 1949 6154 2 1 55 43 0
18 85 1136856 762876 6620616 273112 70 0 176 111425 9192 62244 28 6 27 38 0
16 89 1136844 627232 6620632 308760 0 0 94 223518 16958 112242 58 12 7 23 0
0 37 1136840 646588 6595504 268540 0 0 278 216830 17245 88749 59 13 1 27 0
0 8 1136840 660628 6595520 256788 0 0 0 12362 2516 5400 1 1 64 34 0
23 135 1136836 624888 6589576 309560 0 0 30 20616 2709 10042 4 1 49 46 0
3 46 1136832 655468 6589224 270260 6 0 76 101402 9301 55967 26 7 6 62 0
9 50 1136828 479640 6589288 278512 0 0 6 74871 7801 39989 28 6 25 40 0
1 7 1136824 619044 6588616 262072 0 0 103 131523 11077 52600 45 9 24 22 0
1 64 1136824 614120 6588616 264172 0 0 22 10813 1879 5802 2 0 61 36 0
0 13 1136820 638912 6587888 252584 0 0 24 14556 3085 12514 6 1 15 78 0
3 3 1136816 687108 6587904 243424 0 0 6 14452 2657 12784 8 2 58 33 0
0 0 1136812 1356840 6587912 242400 0 0 30 2953 2006 6159 3 1 90 7 0
0 0 1136808 1357728 6587928 242444 0 0 0 16 457 1219 0 0 100 0 0
0 0 1136808 1357728 6587944 242444 0 0 0 7 540 1364 0 0 100 0 0
[-- Attachment #6: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Inode and dentry cache behavior
2015-04-23 19:50 Inode and dentry cache behavior Shrinand Javadekar
@ 2015-04-23 22:43 ` Dave Chinner
2015-04-23 23:48 ` Shrinand Javadekar
0 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2015-04-23 22:43 UTC (permalink / raw)
To: Shrinand Javadekar; +Cc: xfs
On Thu, Apr 23, 2015 at 12:50:15PM -0700, Shrinand Javadekar wrote:
> Hi,
>
> I am running Openstack Swift on a single server with 8 disks. All
> these 8 disks are formatted with default XFS parameters. Each disk has
> a capacity of 3TB. The machine has 64GB of data.
>
> Here's what Openstack Swift does:
....
> * We observe that the time for fsync remains pretty much constant throughout.
> * What seems to be causing the performance to nosedive, is that inode
> and dentry caching doesn't seem to be working.
> * For experiment sake, we set vfs_cache_pressure to 0 so there would
> be no reclaiming of inode and dentry cache entries. However, that does
> not seem to help.
> * We see openat() calls taking close to 1 second.
>
> Any ideas, what might be causing this behavior? Are there other
> params, specifically, xfs params that can be tuned for this workload.
> The sequence of events above is the typical workload, at high
> concurrency.
Work out why you're disks are reporting 100% utilisation when they
have little or no IO being issued to them.
> See the attached files iostat_log and vmstat_log.
from the iostat log:
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
.....
dm-6 0.00 0.00 0.20 22.40 0.00 0.09 8.00 22.28 839.01 1224.00 835.57 44.25 100.00
dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00 2.82 1517.33 0.00 1517.33 833.33 100.00
dm-8 0.00 0.00 0.00 195.20 0.00 0.76 8.00 1727.51 4178.89 0.00 4178.89 5.12 100.00
...
dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1178.85 0.00 0.00 0.00 0.00 100.00
dm-7 is showing almost a second for single IO wait times, when it is
actually completing IO. dm-8 has a massive queue depth - I can only
assume you've tuned sys/block/*/queue/nr_requests to something
really large? But like dm-7, it's showing very long IO times, and
that's likely the source of your latency problems.
i.e. this looks like a storage problem, not an XFS problem.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Inode and dentry cache behavior
2015-04-23 22:43 ` Dave Chinner
@ 2015-04-23 23:48 ` Shrinand Javadekar
2015-04-24 6:15 ` Dave Chinner
0 siblings, 1 reply; 9+ messages in thread
From: Shrinand Javadekar @ 2015-04-23 23:48 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
> from the iostat log:
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> .....
> dm-6 0.00 0.00 0.20 22.40 0.00 0.09 8.00 22.28 839.01 1224.00 835.57 44.25 100.00
> dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00 2.82 1517.33 0.00 1517.33 833.33 100.00
> dm-8 0.00 0.00 0.00 195.20 0.00 0.76 8.00 1727.51 4178.89 0.00 4178.89 5.12 100.00
> ...
> dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
> dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1178.85 0.00 0.00 0.00 0.00 100.00
>
> dm-7 is showing almost a second for single IO wait times, when it is
> actually completing IO. dm-8 has a massive queue depth - I can only
> assume you've tuned sys/block/*/queue/nr_requests to something
> really large? But like dm-7, it's showing very long IO times, and
> that's likely the source of your latency problems.
I see that /sys/block/*/queue/nr_requests is set to 128 which is way
less than the queue depth shown in the iostat numbers. What gives?
One other observation we had was that xfs shows a large amount of
directory fragmentation. Directory fragmentation was shown at ~40%
whereas file fragmentation was very low at 0.1%.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Inode and dentry cache behavior
2015-04-23 23:48 ` Shrinand Javadekar
@ 2015-04-24 6:15 ` Dave Chinner
2015-04-29 0:17 ` Shrinand Javadekar
0 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2015-04-24 6:15 UTC (permalink / raw)
To: Shrinand Javadekar; +Cc: xfs
On Thu, Apr 23, 2015 at 04:48:51PM -0700, Shrinand Javadekar wrote:
> > from the iostat log:
> >
> > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
> > .....
> > dm-6 0.00 0.00 0.20 22.40 0.00 0.09 8.00 22.28 839.01 1224.00 835.57 44.25 100.00
> > dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00 2.82 1517.33 0.00 1517.33 833.33 100.00
> > dm-8 0.00 0.00 0.00 195.20 0.00 0.76 8.00 1727.51 4178.89 0.00 4178.89 5.12 100.00
> > ...
> > dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
> > dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1178.85 0.00 0.00 0.00 0.00 100.00
> >
> > dm-7 is showing almost a second for single IO wait times, when it is
> > actually completing IO. dm-8 has a massive queue depth - I can only
> > assume you've tuned sys/block/*/queue/nr_requests to something
> > really large? But like dm-7, it's showing very long IO times, and
> > that's likely the source of your latency problems.
>
> I see that /sys/block/*/queue/nr_requests is set to 128 which is way
> less than the queue depth shown in the iostat numbers. What gives?
No idea, but it's indicative of a problem below XFS. Work out what
is happening with your storage hardware first, then work your way up
the stack...
> One other observation we had was that xfs shows a large amount of
> directory fragmentation. Directory fragmentation was shown at ~40%
> whereas file fragmentation was very low at 0.1%.
Pretty common. Directories are only accessed a single block at a
time, and sequential offset reads are pretty rare, so fragmentation
makes little difference to performance. You're seeing almost zero
read IO load, so the directory layout is not a concern for this
workload.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Inode and dentry cache behavior
2015-04-24 6:15 ` Dave Chinner
@ 2015-04-29 0:17 ` Shrinand Javadekar
2015-04-29 1:30 ` Dave Chinner
0 siblings, 1 reply; 9+ messages in thread
From: Shrinand Javadekar @ 2015-04-29 0:17 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
I will look at the hardware. But, I think, there's also a possible
software problem here.
If you look at the sequence of events, first a tmp file is created in
<mount-point>/tmp/tmp_blah. After a few writes, this file is renamed
to a different path in the filesystem.
rename(<mount-point>/tmp/tmp_blah,
<mount-point>/objects/1004/eef/deadbeef/foo.data).
The "tmp" directory above is created only once. Temp files get created
inside it and then get renamed. We wondered if this causes disk layout
issues resulting in slower performance. And then, we stumbled upon
this[1]. Someone complaining about the exact same problem.
One quick way to validate this was to delete the "tmp" directory
periodically and see what numbers we get. And they do. With 15 runs of
writing 80K objects in each run, our performance was dropping from
~100MB/s to 30MB/s. With deleting the tmp directory after each run, we
saw the performance only drop from ~100MB/s to 80MB/s.
The explanation in the link below says that when xfs does not find
free extents in an existing allocation group, it frees up the extents
by copying data from existing extents to their target allocation group
(which happens because of renames). Is that explanation still valid?
Thanks in advance.
-Shri
[1] http://xfs.9218.n7.nabble.com/Performance-degradation-over-time-td28514.html
On Thu, Apr 23, 2015 at 11:15 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Thu, Apr 23, 2015 at 04:48:51PM -0700, Shrinand Javadekar wrote:
>> > from the iostat log:
>> >
>> > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> > .....
>> > dm-6 0.00 0.00 0.20 22.40 0.00 0.09 8.00 22.28 839.01 1224.00 835.57 44.25 100.00
>> > dm-7 0.00 0.00 0.00 1.20 0.00 0.00 8.00 2.82 1517.33 0.00 1517.33 833.33 100.00
>> > dm-8 0.00 0.00 0.00 195.20 0.00 0.76 8.00 1727.51 4178.89 0.00 4178.89 5.12 100.00
>> > ...
>> > dm-7 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 100.00
>> > dm-8 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1178.85 0.00 0.00 0.00 0.00 100.00
>> >
>> > dm-7 is showing almost a second for single IO wait times, when it is
>> > actually completing IO. dm-8 has a massive queue depth - I can only
>> > assume you've tuned sys/block/*/queue/nr_requests to something
>> > really large? But like dm-7, it's showing very long IO times, and
>> > that's likely the source of your latency problems.
>>
>> I see that /sys/block/*/queue/nr_requests is set to 128 which is way
>> less than the queue depth shown in the iostat numbers. What gives?
>
> No idea, but it's indicative of a problem below XFS. Work out what
> is happening with your storage hardware first, then work your way up
> the stack...
>
>> One other observation we had was that xfs shows a large amount of
>> directory fragmentation. Directory fragmentation was shown at ~40%
>> whereas file fragmentation was very low at 0.1%.
>
> Pretty common. Directories are only accessed a single block at a
> time, and sequential offset reads are pretty rare, so fragmentation
> makes little difference to performance. You're seeing almost zero
> read IO load, so the directory layout is not a concern for this
> workload.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Inode and dentry cache behavior
2015-04-29 0:17 ` Shrinand Javadekar
@ 2015-04-29 1:30 ` Dave Chinner
2015-04-29 17:46 ` Shrinand Javadekar
0 siblings, 1 reply; 9+ messages in thread
From: Dave Chinner @ 2015-04-29 1:30 UTC (permalink / raw)
To: Shrinand Javadekar; +Cc: xfs
On Tue, Apr 28, 2015 at 05:17:14PM -0700, Shrinand Javadekar wrote:
> I will look at the hardware. But, I think, there's also a possible
> software problem here.
>
> If you look at the sequence of events, first a tmp file is created in
> <mount-point>/tmp/tmp_blah. After a few writes, this file is renamed
> to a different path in the filesystem.
>
> rename(<mount-point>/tmp/tmp_blah,
> <mount-point>/objects/1004/eef/deadbeef/foo.data).
>
> The "tmp" directory above is created only once. Temp files get created
> inside it and then get renamed. We wondered if this causes disk layout
> issues resulting in slower performance. And then, we stumbled upon
> this[1]. Someone complaining about the exact same problem.
That's pretty braindead behaviour. That will screw performance and
locality on any filesystem you do that on, not to mention age it
extremely quickly.
In the case of XFS, it forces allocation of all the inodes in one
AG, rather than allowing XFs to distribute and balance inode
allocation around the filesystem and keeping good
directory/inode/data locality for all your data.
Best way to do this is to create your tmp files using O_TMPFILE,
with the source directory being the destination directory and then
use linkat() rather than rename to make them visible in the
directory.
> One quick way to validate this was to delete the "tmp" directory
> periodically and see what numbers we get. And they do. With 15 runs of
> writing 80K objects in each run, our performance was dropping from
> ~100MB/s to 30MB/s. With deleting the tmp directory after each run, we
> saw the performance only drop from ~100MB/s to 80MB/s.
>
> The explanation in the link below says that when xfs does not find
> free extents in an existing allocation group, it frees up the extents
> by copying data from existing extents to their target allocation group
> (which happens because of renames). Is that explanation still valid?
No, it wasn't correct even back then. XFS does not move data around
once it has been allocated and is on disk. Indeed, rename() does not
move data, it only modifies directory entries.
The problem is that the locality of a new inode is determined by the
parent inode, and so if all new inodes are created in the same
directory, then they are all created in the same AG. If you have
millions of inodes, then you have a btree will millions on inodes in
it in one AG, and pretty much none in any other AG. Hence inode
allocation, which has to search for free inodes in a btree
containing millions of records, can be extremely IO and CPU
intensive and therefore slow. And the larger the number of inodes,
the slower it will go....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Inode and dentry cache behavior
2015-04-29 1:30 ` Dave Chinner
@ 2015-04-29 17:46 ` Shrinand Javadekar
2015-05-11 21:07 ` Shrinand Javadekar
0 siblings, 1 reply; 9+ messages in thread
From: Shrinand Javadekar @ 2015-04-29 17:46 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
Awesome!! Thanks Dave!
On Tue, Apr 28, 2015 at 6:30 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Tue, Apr 28, 2015 at 05:17:14PM -0700, Shrinand Javadekar wrote:
>> I will look at the hardware. But, I think, there's also a possible
>> software problem here.
>>
>> If you look at the sequence of events, first a tmp file is created in
>> <mount-point>/tmp/tmp_blah. After a few writes, this file is renamed
>> to a different path in the filesystem.
>>
>> rename(<mount-point>/tmp/tmp_blah,
>> <mount-point>/objects/1004/eef/deadbeef/foo.data).
>>
>> The "tmp" directory above is created only once. Temp files get created
>> inside it and then get renamed. We wondered if this causes disk layout
>> issues resulting in slower performance. And then, we stumbled upon
>> this[1]. Someone complaining about the exact same problem.
>
> That's pretty braindead behaviour. That will screw performance and
> locality on any filesystem you do that on, not to mention age it
> extremely quickly.
>
> In the case of XFS, it forces allocation of all the inodes in one
> AG, rather than allowing XFs to distribute and balance inode
> allocation around the filesystem and keeping good
> directory/inode/data locality for all your data.
>
> Best way to do this is to create your tmp files using O_TMPFILE,
> with the source directory being the destination directory and then
> use linkat() rather than rename to make them visible in the
> directory.
>
>> One quick way to validate this was to delete the "tmp" directory
>> periodically and see what numbers we get. And they do. With 15 runs of
>> writing 80K objects in each run, our performance was dropping from
>> ~100MB/s to 30MB/s. With deleting the tmp directory after each run, we
>> saw the performance only drop from ~100MB/s to 80MB/s.
>>
>> The explanation in the link below says that when xfs does not find
>> free extents in an existing allocation group, it frees up the extents
>> by copying data from existing extents to their target allocation group
>> (which happens because of renames). Is that explanation still valid?
>
> No, it wasn't correct even back then. XFS does not move data around
> once it has been allocated and is on disk. Indeed, rename() does not
> move data, it only modifies directory entries.
>
> The problem is that the locality of a new inode is determined by the
> parent inode, and so if all new inodes are created in the same
> directory, then they are all created in the same AG. If you have
> millions of inodes, then you have a btree will millions on inodes in
> it in one AG, and pretty much none in any other AG. Hence inode
> allocation, which has to search for free inodes in a btree
> containing millions of records, can be extremely IO and CPU
> intensive and therefore slow. And the larger the number of inodes,
> the slower it will go....
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Inode and dentry cache behavior
2015-04-29 17:46 ` Shrinand Javadekar
@ 2015-05-11 21:07 ` Shrinand Javadekar
2015-05-11 21:44 ` Dave Chinner
0 siblings, 1 reply; 9+ messages in thread
From: Shrinand Javadekar @ 2015-05-11 21:07 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
In case you're curious, the solutions to this problem are being
discussed at [1] on the Swift side.
O_TMPFILE and linkat() aren't available in all kernels (and python)
and therefore not a viable option right away. Among few others, one of
the proposals is to shard the tmp directory into say 256 dirs and
create these files inside those dirs. That way files won't get created
in a single AG. Are there any other problems if this is done?
[1] https://bugs.launchpad.net/swift/+bug/1450656
On Wed, Apr 29, 2015 at 10:46 AM, Shrinand Javadekar
<shrinand@maginatics.com> wrote:
> Awesome!! Thanks Dave!
>
> On Tue, Apr 28, 2015 at 6:30 PM, Dave Chinner <david@fromorbit.com> wrote:
>> On Tue, Apr 28, 2015 at 05:17:14PM -0700, Shrinand Javadekar wrote:
>>> I will look at the hardware. But, I think, there's also a possible
>>> software problem here.
>>>
>>> If you look at the sequence of events, first a tmp file is created in
>>> <mount-point>/tmp/tmp_blah. After a few writes, this file is renamed
>>> to a different path in the filesystem.
>>>
>>> rename(<mount-point>/tmp/tmp_blah,
>>> <mount-point>/objects/1004/eef/deadbeef/foo.data).
>>>
>>> The "tmp" directory above is created only once. Temp files get created
>>> inside it and then get renamed. We wondered if this causes disk layout
>>> issues resulting in slower performance. And then, we stumbled upon
>>> this[1]. Someone complaining about the exact same problem.
>>
>> That's pretty braindead behaviour. That will screw performance and
>> locality on any filesystem you do that on, not to mention age it
>> extremely quickly.
>>
>> In the case of XFS, it forces allocation of all the inodes in one
>> AG, rather than allowing XFs to distribute and balance inode
>> allocation around the filesystem and keeping good
>> directory/inode/data locality for all your data.
>>
>> Best way to do this is to create your tmp files using O_TMPFILE,
>> with the source directory being the destination directory and then
>> use linkat() rather than rename to make them visible in the
>> directory.
>>
>>> One quick way to validate this was to delete the "tmp" directory
>>> periodically and see what numbers we get. And they do. With 15 runs of
>>> writing 80K objects in each run, our performance was dropping from
>>> ~100MB/s to 30MB/s. With deleting the tmp directory after each run, we
>>> saw the performance only drop from ~100MB/s to 80MB/s.
>>>
>>> The explanation in the link below says that when xfs does not find
>>> free extents in an existing allocation group, it frees up the extents
>>> by copying data from existing extents to their target allocation group
>>> (which happens because of renames). Is that explanation still valid?
>>
>> No, it wasn't correct even back then. XFS does not move data around
>> once it has been allocated and is on disk. Indeed, rename() does not
>> move data, it only modifies directory entries.
>>
>> The problem is that the locality of a new inode is determined by the
>> parent inode, and so if all new inodes are created in the same
>> directory, then they are all created in the same AG. If you have
>> millions of inodes, then you have a btree will millions on inodes in
>> it in one AG, and pretty much none in any other AG. Hence inode
>> allocation, which has to search for free inodes in a btree
>> containing millions of records, can be extremely IO and CPU
>> intensive and therefore slow. And the larger the number of inodes,
>> the slower it will go....
>>
>> Cheers,
>>
>> Dave.
>> --
>> Dave Chinner
>> david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Inode and dentry cache behavior
2015-05-11 21:07 ` Shrinand Javadekar
@ 2015-05-11 21:44 ` Dave Chinner
0 siblings, 0 replies; 9+ messages in thread
From: Dave Chinner @ 2015-05-11 21:44 UTC (permalink / raw)
To: Shrinand Javadekar; +Cc: xfs
On Mon, May 11, 2015 at 02:07:39PM -0700, Shrinand Javadekar wrote:
> In case you're curious, the solutions to this problem are being
> discussed at [1] on the Swift side.
>
> O_TMPFILE and linkat() aren't available in all kernels (and python)
> and therefore not a viable option right away. Among few others, one of
> the proposals is to shard the tmp directory into say 256 dirs and
> create these files inside those dirs. That way files won't get created
> in a single AG. Are there any other problems if this is done?
Yes, you don't get any locality between files in the same directory
once you move them to their final location in the hash tree. You'll
end up with the same problems you are trying to solve now as the
object count goes up - you'll simply have taken a different path to
the same destination....
Create your temp files in their eventual destination directory -
doing anything else is pretty much guaranteed to compromise all the
locality algorithms the filesystem uses. Without good locality
within the data set, performance of lookups and writeback will
degrade to large seek time random IO pretty quickly.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2015-05-11 21:45 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-04-23 19:50 Inode and dentry cache behavior Shrinand Javadekar
2015-04-23 22:43 ` Dave Chinner
2015-04-23 23:48 ` Shrinand Javadekar
2015-04-24 6:15 ` Dave Chinner
2015-04-29 0:17 ` Shrinand Javadekar
2015-04-29 1:30 ` Dave Chinner
2015-04-29 17:46 ` Shrinand Javadekar
2015-05-11 21:07 ` Shrinand Javadekar
2015-05-11 21:44 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox