Raid 10 LVM JFS Seeking performance help

Raid 10 LVM JFS Seeking performance help

[Chris]

I have a pair of servers serving 10MB-100MB files.  Each server has
12x 7200 SaS 750GB Drives.  When I look at iostat I see the avgrq-sz
is 8.0 always.  I think this has to do with the fact my LVM PE Size is
4096 with JFS on top of that.    Best I can tell the fact I have so
many rrqm/s is not great and the reason I have that many is because my
avgrq-sz is 8.0.  I have been trying to grasp how I should come up
with the best chunk and PE for more performance.

Switch from n2 to f2 raid10?
How do I calculate where I need to go from here with Chunk Size and PE size?


Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz
avgqu-sz   await  svctm  %util
sdf            1934.00     0.00 123.00  0.00     8.00     0.00
133.27     3.66   29.76   7.05  86.70
sdg            1765.00     0.00 117.00  0.00     7.45     0.00
130.32     3.09   29.61   7.46  87.30
sdh            1744.00     0.00 83.00  0.00     6.50     0.00   160.48
    3.31   38.47  10.89  90.40
sdi            2369.00     0.00 109.00  0.00     9.50     0.00
178.42     5.30   47.83   8.65  94.30
sdj            1867.00     0.00 90.00  0.00     6.89     0.00   156.89
    1.89   21.70   8.83  79.50
sdk            1574.00     0.00 74.00  0.00     6.49     0.00   179.57
    2.45   34.11  11.74  86.90
sdl            2437.00     0.00 105.00  0.00     9.10     0.00
177.52     4.66   41.79   8.35  87.70
sdm            1259.00     0.00 102.00  0.00     5.22     0.00
104.86     2.19   21.34   7.99  81.50
sdn            2096.00     0.00 114.00  0.00     8.88     0.00
159.51     4.95   45.06   8.33  95.00
sdo            1835.00     0.00 106.00  0.00     7.09     0.00
137.06     2.71   24.00   8.19  86.80
sdp            1431.00     0.00 113.00  0.00     5.92     0.00
107.33     4.32   38.82   8.24  93.10
sdq            2068.00     0.00 138.00  0.00     8.39     0.00
124.46    10.18   71.28   7.04  97.10
md2               0.00     0.00 23671.00  0.00    92.46     0.00
8.00     0.00    0.00   0.00   0.00
dm-7              0.00     0.00 23671.00  0.00    92.46     0.00
8.00  1006.46   42.07   0.04 100.10

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdf             112.00     10784.00         0.00      10784          0
sdg             111.00     10464.00         0.00      10464          0
sdh             104.00     11520.00         0.00      11520          0
sdi              98.00     14280.00         0.00      14280          0
sdj              89.00     14200.00         0.00      14200          0
sdk              79.00      6328.00         0.00       6328          0
sdl             113.00     11296.00         0.00      11296          0
sdm              74.00      7504.00         0.00       7504          0
sdn             109.00     11840.00         0.00      11840          0
sdo             113.00     15488.00         0.00      15488          0
sdp             107.00      9928.00         0.00       9928          0
sdq             109.00     10656.00         0.00      10656          0
md2           16937.00    135496.00         0.00     135496          0
dm-7          16937.00    135496.00         0.00     135496          0


Personalities : [raid10]
md2 : active raid10 sdf[0] sdq[11] sdp[10] sdo[9] sdn[8] sdm[7] sdl[6]
sdk[5] sdj[4] sdi[3] sdh[2] sdg[1]
      4395442176 blocks super 1.2 1024K chunks 2 near-copies [12/12]
[UUUUUUUUUUUU]


  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               storage
  PV Size               4.09 TB / not usable 4.00 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              1073105
  Free PE               0
  Allocated PE          1073105
  PV UUID               SwNPeb-QHqH-evb3-sdDM-el7V-
NfQl-HJijzQ


12x   Vendor: SEAGATE  Model: ST3750630SS

Re: Raid 10 LVM JFS Seeking performance help

[Goswin von Brederlow]

Chris <cmtimegn@gmail.com> writes:

> I have a pair of servers serving 10MB-100MB files.  Each server has
> 12x 7200 SaS 750GB Drives.  When I look at iostat I see the avgrq-sz
> is 8.0 always.  I think this has to do with the fact my LVM PE Size is
> 4096 with JFS on top of that.    Best I can tell the fact I have so
> many rrqm/s is not great and the reason I have that many is because my
> avgrq-sz is 8.0.  I have been trying to grasp how I should come up
> with the best chunk and PE for more performance.
>
> Switch from n2 to f2 raid10?
> How do I calculate where I need to go from here with Chunk Size and PE size?

2 far copies means each disk is split into 2 partitions, lets call
them sda1/2, sdb1/2, ... Then sda1, sdb2 form a raid1 (md1) and sdb1,
sdc2 form a second raid1 (md2), ..... Last md1, md2, ... are combined
as raid0. That all is done internaly and more flexible. The above is
just so you can visualize the layout. Writes will always go to sdX1
and sd(X+1)2. Reads should always go to sdX1, which is usualy the
faster part on rotating disks.

You need to optimize the raid0 part and, probably way more important,
the alignment of your data access. If everything is aligned nicely
each request should be fully serviced by a single disk given your
small request size. And the seeks should be evenly spread out between
the disks with each disk seeking every 12 reads or twice every 6
writes (or less). Check if you are seeking more than is expected.

Also, on a lower level, make sure your raid does not start on a
partition starting at sector 63 (which is still the default in many
partitioning progs). That easily results in bad alignment causing 4k
chunks to land on 2 sectors. But you need to test that with your
specific drive to see if it really is a problem.

MfG
        Goswin

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 10 LVM JFS Seeking performance help

[Chris]

Right now we use no partitions so I MD the full disk.
Since my avrgq-sz is 8.0  What should I make my Chunk Size how do I
look into this more?
Over the holidays I did a Chunk Size of 32000 in f2 for testing but
that did not seem to work very well.  512 and 1024 Chunk size is what
I had before.  No matter the PE size and Chunk size my avgrq-sz is 8.
My real problem is just to even test one setup is about a week of
copying data and building the array.  That's why I am trying to get
anything to help me make a better guess to how to set this all up
right.

How do I check if I am seeking more than expected?

On Mon, Dec 21, 2009 at 4:56 AM, Goswin von Brederlow <goswin-v-b@web.de> wrote:
> Chris <cmtimegn@gmail.com> writes:
>
>> I have a pair of servers serving 10MB-100MB files.  Each server has
>> 12x 7200 SaS 750GB Drives.  When I look at iostat I see the avgrq-sz
>> is 8.0 always.  I think this has to do with the fact my LVM PE Size is
>> 4096 with JFS on top of that.    Best I can tell the fact I have so
>> many rrqm/s is not great and the reason I have that many is because my
>> avgrq-sz is 8.0.  I have been trying to grasp how I should come up
>> with the best chunk and PE for more performance.
>>
>> Switch from n2 to f2 raid10?
>> How do I calculate where I need to go from here with Chunk Size and PE size?
>
> 2 far copies means each disk is split into 2 partitions, lets call
> them sda1/2, sdb1/2, ... Then sda1, sdb2 form a raid1 (md1) and sdb1,
> sdc2 form a second raid1 (md2), ..... Last md1, md2, ... are combined
> as raid0. That all is done internaly and more flexible. The above is
> just so you can visualize the layout. Writes will always go to sdX1
> and sd(X+1)2. Reads should always go to sdX1, which is usualy the
> faster part on rotating disks.
>
> You need to optimize the raid0 part and, probably way more important,
> the alignment of your data access. If everything is aligned nicely
> each request should be fully serviced by a single disk given your
> small request size. And the seeks should be evenly spread out between
> the disks with each disk seeking every 12 reads or twice every 6
> writes (or less). Check if you are seeking more than is expected.
>
> Also, on a lower level, make sure your raid does not start on a
> partition starting at sector 63 (which is still the default in many
> partitioning progs). That easily results in bad alignment causing 4k
> chunks to land on 2 sectors. But you need to test that with your
> specific drive to see if it really is a problem.
>
> MfG
>        Goswin
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html