* Raid 10 LVM JFS Seeking performance help @ 2009-12-17 23:49 Chris 2009-12-21 12:56 ` Goswin von Brederlow 0 siblings, 1 reply; 3+ messages in thread From: Chris @ 2009-12-17 23:49 UTC (permalink / raw) To: linux-raid I have a pair of servers serving 10MB-100MB files. Each server has 12x 7200 SaS 750GB Drives. When I look at iostat I see the avgrq-sz is 8.0 always. I think this has to do with the fact my LVM PE Size is 4096 with JFS on top of that. Best I can tell the fact I have so many rrqm/s is not great and the reason I have that many is because my avgrq-sz is 8.0. I have been trying to grasp how I should come up with the best chunk and PE for more performance. Switch from n2 to f2 raid10? How do I calculate where I need to go from here with Chunk Size and PE size? Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdf 1934.00 0.00 123.00 0.00 8.00 0.00 133.27 3.66 29.76 7.05 86.70 sdg 1765.00 0.00 117.00 0.00 7.45 0.00 130.32 3.09 29.61 7.46 87.30 sdh 1744.00 0.00 83.00 0.00 6.50 0.00 160.48 3.31 38.47 10.89 90.40 sdi 2369.00 0.00 109.00 0.00 9.50 0.00 178.42 5.30 47.83 8.65 94.30 sdj 1867.00 0.00 90.00 0.00 6.89 0.00 156.89 1.89 21.70 8.83 79.50 sdk 1574.00 0.00 74.00 0.00 6.49 0.00 179.57 2.45 34.11 11.74 86.90 sdl 2437.00 0.00 105.00 0.00 9.10 0.00 177.52 4.66 41.79 8.35 87.70 sdm 1259.00 0.00 102.00 0.00 5.22 0.00 104.86 2.19 21.34 7.99 81.50 sdn 2096.00 0.00 114.00 0.00 8.88 0.00 159.51 4.95 45.06 8.33 95.00 sdo 1835.00 0.00 106.00 0.00 7.09 0.00 137.06 2.71 24.00 8.19 86.80 sdp 1431.00 0.00 113.00 0.00 5.92 0.00 107.33 4.32 38.82 8.24 93.10 sdq 2068.00 0.00 138.00 0.00 8.39 0.00 124.46 10.18 71.28 7.04 97.10 md2 0.00 0.00 23671.00 0.00 92.46 0.00 8.00 0.00 0.00 0.00 0.00 dm-7 0.00 0.00 23671.00 0.00 92.46 0.00 8.00 1006.46 42.07 0.04 100.10 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sdf 112.00 10784.00 0.00 10784 0 sdg 111.00 10464.00 0.00 10464 0 sdh 104.00 11520.00 0.00 11520 0 sdi 98.00 14280.00 0.00 14280 0 sdj 89.00 14200.00 0.00 14200 0 sdk 79.00 6328.00 0.00 6328 0 sdl 113.00 11296.00 0.00 11296 0 sdm 74.00 7504.00 0.00 7504 0 sdn 109.00 11840.00 0.00 11840 0 sdo 113.00 15488.00 0.00 15488 0 sdp 107.00 9928.00 0.00 9928 0 sdq 109.00 10656.00 0.00 10656 0 md2 16937.00 135496.00 0.00 135496 0 dm-7 16937.00 135496.00 0.00 135496 0 Personalities : [raid10] md2 : active raid10 sdf[0] sdq[11] sdp[10] sdo[9] sdn[8] sdm[7] sdl[6] sdk[5] sdj[4] sdi[3] sdh[2] sdg[1] 4395442176 blocks super 1.2 1024K chunks 2 near-copies [12/12] [UUUUUUUUUUUU] --- Physical volume --- PV Name /dev/md2 VG Name storage PV Size 4.09 TB / not usable 4.00 MB Allocatable yes (but full) PE Size (KByte) 4096 Total PE 1073105 Free PE 0 Allocated PE 1073105 PV UUID SwNPeb-QHqH-evb3-sdDM-el7V- NfQl-HJijzQ 12x Vendor: SEAGATE Model: ST3750630SS ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Raid 10 LVM JFS Seeking performance help 2009-12-17 23:49 Raid 10 LVM JFS Seeking performance help Chris @ 2009-12-21 12:56 ` Goswin von Brederlow 2009-12-28 21:23 ` Chris 0 siblings, 1 reply; 3+ messages in thread From: Goswin von Brederlow @ 2009-12-21 12:56 UTC (permalink / raw) To: Chris; +Cc: linux-raid Chris <cmtimegn@gmail.com> writes: > I have a pair of servers serving 10MB-100MB files. Each server has > 12x 7200 SaS 750GB Drives. When I look at iostat I see the avgrq-sz > is 8.0 always. I think this has to do with the fact my LVM PE Size is > 4096 with JFS on top of that.   Best I can tell the fact I have so > many rrqm/s is not great and the reason I have that many is because my > avgrq-sz is 8.0. I have been trying to grasp how I should come up > with the best chunk and PE for more performance. > > Switch from n2 to f2 raid10? > How do I calculate where I need to go from here with Chunk Size and PE size? 2 far copies means each disk is split into 2 partitions, lets call them sda1/2, sdb1/2, ... Then sda1, sdb2 form a raid1 (md1) and sdb1, sdc2 form a second raid1 (md2), ..... Last md1, md2, ... are combined as raid0. That all is done internaly and more flexible. The above is just so you can visualize the layout. Writes will always go to sdX1 and sd(X+1)2. Reads should always go to sdX1, which is usualy the faster part on rotating disks. You need to optimize the raid0 part and, probably way more important, the alignment of your data access. If everything is aligned nicely each request should be fully serviced by a single disk given your small request size. And the seeks should be evenly spread out between the disks with each disk seeking every 12 reads or twice every 6 writes (or less). Check if you are seeking more than is expected. Also, on a lower level, make sure your raid does not start on a partition starting at sector 63 (which is still the default in many partitioning progs). That easily results in bad alignment causing 4k chunks to land on 2 sectors. But you need to test that with your specific drive to see if it really is a problem. MfG Goswin -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Raid 10 LVM JFS Seeking performance help 2009-12-21 12:56 ` Goswin von Brederlow @ 2009-12-28 21:23 ` Chris 0 siblings, 0 replies; 3+ messages in thread From: Chris @ 2009-12-28 21:23 UTC (permalink / raw) To: linux-raid Right now we use no partitions so I MD the full disk. Since my avrgq-sz is 8.0 What should I make my Chunk Size how do I look into this more? Over the holidays I did a Chunk Size of 32000 in f2 for testing but that did not seem to work very well. 512 and 1024 Chunk size is what I had before. No matter the PE size and Chunk size my avgrq-sz is 8. My real problem is just to even test one setup is about a week of copying data and building the array. That's why I am trying to get anything to help me make a better guess to how to set this all up right. How do I check if I am seeking more than expected? On Mon, Dec 21, 2009 at 4:56 AM, Goswin von Brederlow <goswin-v-b@web.de> wrote: > Chris <cmtimegn@gmail.com> writes: > >> I have a pair of servers serving 10MB-100MB files. Each server has >> 12x 7200 SaS 750GB Drives. When I look at iostat I see the avgrq-sz >> is 8.0 always. I think this has to do with the fact my LVM PE Size is >> 4096 with JFS on top of that. Best I can tell the fact I have so >> many rrqm/s is not great and the reason I have that many is because my >> avgrq-sz is 8.0. I have been trying to grasp how I should come up >> with the best chunk and PE for more performance. >> >> Switch from n2 to f2 raid10? >> How do I calculate where I need to go from here with Chunk Size and PE size? > > 2 far copies means each disk is split into 2 partitions, lets call > them sda1/2, sdb1/2, ... Then sda1, sdb2 form a raid1 (md1) and sdb1, > sdc2 form a second raid1 (md2), ..... Last md1, md2, ... are combined > as raid0. That all is done internaly and more flexible. The above is > just so you can visualize the layout. Writes will always go to sdX1 > and sd(X+1)2. Reads should always go to sdX1, which is usualy the > faster part on rotating disks. > > You need to optimize the raid0 part and, probably way more important, > the alignment of your data access. If everything is aligned nicely > each request should be fully serviced by a single disk given your > small request size. And the seeks should be evenly spread out between > the disks with each disk seeking every 12 reads or twice every 6 > writes (or less). Check if you are seeking more than is expected. > > Also, on a lower level, make sure your raid does not start on a > partition starting at sector 63 (which is still the default in many > partitioning progs). That easily results in bad alignment causing 4k > chunks to land on 2 sectors. But you need to test that with your > specific drive to see if it really is a problem. > > MfG > Goswin > > -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2009-12-28 21:23 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-12-17 23:49 Raid 10 LVM JFS Seeking performance help Chris 2009-12-21 12:56 ` Goswin von Brederlow 2009-12-28 21:23 ` Chris
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).