RAID-10 unbalanced reads

RAID-10 unbalanced reads

[Piergiorgio Sartor]

Hi all.

I've setup a system with two RAID volumes.
One is a small RAID-1, /dev/md0, with the /boot
content, the other is a RAID-10 f2, with LVM,
for the rest of the system.

I just noticed that /dev/sdb, which is not the
boot disk, I assume, has more reads than /dev/sda.
Writes are the same, here is the output of "iostat":

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              10.08       183.15       408.74     755018    1684972
sda1              0.04         1.24         0.02       5118         66
sda2             10.03       181.71       408.72     749071    1684880
sdb              10.49       187.64       408.74     773532    1684972
sdb1              0.03         1.17         0.02       4837         66
sdb2             10.44       186.26       408.72     767832    1684880
md1              62.95       367.26       391.79    1513976    1615112
md0               0.02         0.18         0.00        724          8

Note that sda1 has more reads than sdb1.
This is possibly due to the fact that it is
used during boot.

What is strange, is sdb2, which has by far
more reads than sda2. Both belong to /dev/md1.

Note also the /dev/sdb is the slowest of the
two drives.

Is that somehow normal?
If not, is it possible to find out what or
why the reads are unbalanced?

Thanks a lot in advance,

bye,

-- 

piergiorgio

Re: RAID-10 unbalanced reads

[Keld Jørn Simonsen]

On Mon, Feb 09, 2009 at 09:27:28PM +0100, Piergiorgio Sartor wrote:
> Hi all.
> 
> I've setup a system with two RAID volumes.
> One is a small RAID-1, /dev/md0, with the /boot
> content, the other is a RAID-10 f2, with LVM,
> for the rest of the system.
> 
> I just noticed that /dev/sdb, which is not the
> boot disk, I assume, has more reads than /dev/sda.
> Writes are the same, here is the output of "iostat":
> 
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda              10.08       183.15       408.74     755018    1684972
> sda1              0.04         1.24         0.02       5118         66
> sda2             10.03       181.71       408.72     749071    1684880
> sdb              10.49       187.64       408.74     773532    1684972
> sdb1              0.03         1.17         0.02       4837         66
> sdb2             10.44       186.26       408.72     767832    1684880
> md1              62.95       367.26       391.79    1513976    1615112
> md0               0.02         0.18         0.00        724          8
> 
> Note that sda1 has more reads than sdb1.
> This is possibly due to the fact that it is
> used during boot.
> 
> What is strange, is sdb2, which has by far
> more reads than sda2. Both belong to /dev/md1.
> 
> Note also the /dev/sdb is the slowest of the
> two drives.
> 
> Is that somehow normal?
> If not, is it possible to find out what or
> why the reads are unbalanced?

What kernel are you using?
Before 2.6.25 you could have differences in usage.

Anyway, I think the difference is too small to be of inportance
for the performance. Or did you notice a difference?

best regards
keld

Re: RAID-10 unbalanced reads

[Piergiorgio Sartor]

Hi,

thanks for the answer.

The kernel is the latest in Fedora 10, i.e.
the 2.6.27.12-170.2.5.fc10.x86_64, the array
has metadata 1.1 and bitmap.

There are two issues, which I noticed, which
have led to the iostat discovery.

One is that the reading speed of the array is
less than double the speed of the slowest drive.
Accordingly to hdparm, /dev/sda2 transfers about
100~110 MB/s, /dev/sdb2 about 75~85 MB/s, while
/dev/md1 goes about 140~150 MB/s.
I would have expected a bit more, let's say
around 150~160 MB/s for the RAID.

Second, I ran a smart long test on both drives.
/dev/sda finished more or less in one hour, as
declared by the smart info (the system was
barely in use).
/dev/sdb finished in maybe 3~4 hours, by far more
than the (about) 1h40m declared by the smart info.

The only reason it took so long could be the drive
was somehow in use, but it belongs completely to
the RAID, so it should be is use as much as /dev/sda.

Or I have some virus...

Any idea?

Thanks again,

pg

On Tue, Feb 10, 2009 at 12:11:39AM +0100, Keld Jørn Simonsen wrote:
> On Mon, Feb 09, 2009 at 09:27:28PM +0100, Piergiorgio Sartor wrote:
> > Hi all.
> > 
> > I've setup a system with two RAID volumes.
> > One is a small RAID-1, /dev/md0, with the /boot
> > content, the other is a RAID-10 f2, with LVM,
> > for the rest of the system.
> > 
> > I just noticed that /dev/sdb, which is not the
> > boot disk, I assume, has more reads than /dev/sda.
> > Writes are the same, here is the output of "iostat":
> > 
> > Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> > sda              10.08       183.15       408.74     755018    1684972
> > sda1              0.04         1.24         0.02       5118         66
> > sda2             10.03       181.71       408.72     749071    1684880
> > sdb              10.49       187.64       408.74     773532    1684972
> > sdb1              0.03         1.17         0.02       4837         66
> > sdb2             10.44       186.26       408.72     767832    1684880
> > md1              62.95       367.26       391.79    1513976    1615112
> > md0               0.02         0.18         0.00        724          8
> > 
> > Note that sda1 has more reads than sdb1.
> > This is possibly due to the fact that it is
> > used during boot.
> > 
> > What is strange, is sdb2, which has by far
> > more reads than sda2. Both belong to /dev/md1.
> > 
> > Note also the /dev/sdb is the slowest of the
> > two drives.
> > 
> > Is that somehow normal?
> > If not, is it possible to find out what or
> > why the reads are unbalanced?
> 
> What kernel are you using?
> Before 2.6.25 you could have differences in usage.
> 
> Anyway, I think the difference is too small to be of inportance
> for the performance. Or did you notice a difference?
> 
> best regards
> keld
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 

piergiorgio
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID-10 unbalanced reads

[Piergiorgio Sartor]

Hi,

some updates.

> > Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> > sda              10.08       183.15       408.74     755018    1684972
> > sda1              0.04         1.24         0.02       5118         66
> > sda2             10.03       181.71       408.72     749071    1684880
> > sdb              10.49       187.64       408.74     773532    1684972
> > sdb1              0.03         1.17         0.02       4837         66
> > sdb2             10.44       186.26       408.72     767832    1684880
> > md1              62.95       367.26       391.79    1513976    1615112
> > md0               0.02         0.18         0.00        724          8
[...]
> Anyway, I think the difference is too small to be of inportance
> for the performance. Or did you notice a difference?

What about this:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               5.43        64.15        71.33     598692     665656
sda1              0.01         0.15         0.01       1421         64
sda2              5.42        63.95        71.32     596823     665592
sdb               6.22       219.19        71.33    2045590     665656
sdb1              0.29        67.16         0.01     626774         64
sdb2              5.93       151.87        71.32    1417344     665592
md1              12.56       129.09        57.25    1204696     534320
md0               0.01         0.08         0.00        724          8

Now, I do not recall any particular reason why
it should be like this (i.e. no resync or else),
but it seems to me a bit too unbalanced.

Could it be a problem with /dev/sdb?
I run smart long tests weekly, but nothing special
was reported, until now.

bye,

-- 

piergiorgio

Re: RAID-10 unbalanced reads

[Keld Jørn Simonsen]

On Mon, Feb 23, 2009 at 11:15:32PM +0100, Piergiorgio Sartor wrote:
> Hi,
> 
> some updates.
> 
> > > Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> > > sda              10.08       183.15       408.74     755018    1684972
> > > sda1              0.04         1.24         0.02       5118         66
> > > sda2             10.03       181.71       408.72     749071    1684880
> > > sdb              10.49       187.64       408.74     773532    1684972
> > > sdb1              0.03         1.17         0.02       4837         66
> > > sdb2             10.44       186.26       408.72     767832    1684880
> > > md1              62.95       367.26       391.79    1513976    1615112
> > > md0               0.02         0.18         0.00        724          8
> [...]
> > Anyway, I think the difference is too small to be of inportance
> > for the performance. Or did you notice a difference?
> 
> What about this:
> 
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sda               5.43        64.15        71.33     598692     665656
> sda1              0.01         0.15         0.01       1421         64
> sda2              5.42        63.95        71.32     596823     665592
> sdb               6.22       219.19        71.33    2045590     665656
> sdb1              0.29        67.16         0.01     626774         64
> sdb2              5.93       151.87        71.32    1417344     665592
> md1              12.56       129.09        57.25    1204696     534320
> md0               0.01         0.08         0.00        724          8
> 
> Now, I do not recall any particular reason why
> it should be like this (i.e. no resync or else),
> but it seems to me a bit too unbalanced.
> 
> Could it be a problem with /dev/sdb?
> I run smart long tests weekly, but nothing special
> was reported, until now.

You think the sdb2 read rate of 151 blks/s vs sda2 63 blks/s is strange?
Well, it does look strange. OTOH the tps is 

sdb2   5.93
sda2   5.42 

Which is not so big a difference. 

what is sdb1 and sda1 used for? the fugures there seems even stranger.
It also looks strnge that the sdb disk is the slower about 80 MB/s
while sda is around 110 MB/s - and then sdb produces more than sda...

Best regards
keld

Re: RAID-10 unbalanced reads

[Piergiorgio Sartor]

Hi,

> You think the sdb2 read rate of 151 blks/s vs sda2 63 blks/s is strange?
> Well, it does look strange. OTOH the tps is 
> 
> sdb2   5.93
> sda2   5.42 
> 
> Which is not so big a difference. 

Also the whole RAID tps somehow matches
the (sum of the) single devices.

> what is sdb1 and sda1 used for? the fugures there seems even stranger.
> It also looks strnge that the sdb disk is the slower about 80 MB/s
> while sda is around 110 MB/s - and then sdb produces more than sda...

The first partion (sda1, sdb1) is the /boot,
around 100MB, the rest (sda2, sdb2) is the LVM
physical volume, around 320GB.

Also this looks strange:

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb2              4.98       109.30        64.28    1444432     849528
md1              10.45        95.23        51.27    1258536     677608

sdb2 has more reads than the whole RAID it belongs,
but tps are OK...

The only thing that *could* do something on the
raw devices is "smartd", I cannot think anything
else. Maybe I will disable it, for now...

bye,

-- 

piergiorgio

Re: RAID-10 unbalanced reads

[Piergiorgio Sartor]

Hi,

one more thing I see right now:

mdadm --examine-bitmap for sd[ab]2

mdadm --examine-bitmap /dev/sda2
        Filename : /dev/sda2
           Magic : 6d746962
         Version : 4
            UUID : 54db81a7:b47e9253:7291055e:4953c163
          Events : 2772
  Events Cleared : 2772
           State : OK
       Chunksize : 512 KB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 312464000 (297.99 GiB 319.96 GB)
          Bitmap : 610282 bits (chunks), 0 dirty (0.0%)

mdadm --examine-bitmap /dev/sdb2
        Filename : /dev/sdb2
           Magic : 6d746962
         Version : 4
            UUID : 54db81a7:b47e9253:7291055e:4953c163
          Events : 2772
  Events Cleared : 2772
           State : OK
       Chunksize : 512 KB
          Daemon : 5s flush period
      Write Mode : Normal
       Sync Size : 312464000 (297.99 GiB 319.96 GB)
          Bitmap : 610282 bits (chunks), 22506 dirty (3.7%)

Is it the last line a bit strange?
And this is somehow persistent, i.e. after consecutive
"examinations" the dirty status of sdb2 does not
change, while sda2 goes a little bit up and down.

This, BTW, is after a reboot, so a "clean" machine,
and without "smartd".

bye,

-- 

piergiorgio

Re: RAID-10 unbalanced reads

[Keld Jørn Simonsen]

On Tue, Feb 24, 2009 at 12:24:15AM +0100, Piergiorgio Sartor wrote:
> Hi,
> 
> > You think the sdb2 read rate of 151 blks/s vs sda2 63 blks/s is strange?
> > Well, it does look strange. OTOH the tps is 
> > 
> > sdb2   5.93
> > sda2   5.42 
> > 
> > Which is not so big a difference. 
> 
> Also the whole RAID tps somehow matches
> the (sum of the) single devices.

And is about equal +- 10 % for the two devices.

The boot partition is most likely a raid1 partition, which is quite
oriented towards a very uneven usage, theory will tell that if you are
mostly reading sequentially, you are better off just reading from one
device, as you can always read all content in the same time as it would
take to only read hlaf of the content, because you have to skip data
otherwise.


> > what is sdb1 and sda1 used for? the fugures there seems even stranger.
> > It also looks strnge that the sdb disk is the slower about 80 MB/s
> > while sda is around 110 MB/s - and then sdb produces more than sda...
> 
> The first partion (sda1, sdb1) is the /boot,
> around 100MB, the rest (sda2, sdb2) is the LVM
> physical volume, around 320GB.
> 
> Also this looks strange:
> 
> Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
> sdb2              4.98       109.30        64.28    1444432     849528
> md1              10.45        95.23        51.27    1258536     677608
> 
> sdb2 has more reads than the whole RAID it belongs,
> but tps are OK...

Depends on how you measure things. It could be that sdb2 has delivered
some blocks that md1 did not use, and thus did not register as read.

Are these numbers taken like just after a boot?
Activity on /boot indicates this. You seldomly use /boot after booting.
If so, it is a bit interesting that you have about 1/3 of the IO done in
writing. Why is there so much wroting? I thought that booting was 
almost only reading.

And sdb2 reading should only be about 1/2 of total md1 reading, 
the rest should be on sda2.
> 
> The only thing that *could* do something on the
> raw devices is "smartd", I cannot think anything
> else. Maybe I will disable it, for now...

I think that would be a good idea for your testing.

best regards
keld

Re: RAID-10 unbalanced reads

[Piergiorgio Sartor]

Hi,

> The boot partition is most likely a raid1 partition, which is quite

yep, it is.

> Depends on how you measure things. It could be that sdb2 has delivered
> some blocks that md1 did not use, and thus did not register as read.

OK, good to know.
 
> Are these numbers taken like just after a boot?

No, after some usage.
And this makes me doubt of myself.
Maybe I started something which was causing this.

> Activity on /boot indicates this. You seldomly use /boot after booting.
> If so, it is a bit interesting that you have about 1/3 of the IO done in
> writing. Why is there so much wroting? I thought that booting was 
> almost only reading.

I think /boot was OK, it is on /dev/sd[ab]1, not
so many writes.
 
> > The only thing that *could* do something on the
> > raw devices is "smartd", I cannot think anything
> > else. Maybe I will disable it, for now...
> 
> I think that would be a good idea for your testing.

I'll try to enable/disable "special" services and see.

If I can reproduce clearly the issue, I'll come back.
Hope not...

Thanks again for your support,

bye,

-- 

piergiorgio