RAID5/6 slow due to a member device -- how to diagnose?

RAID5/6 slow due to a member device -- how to diagnose?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

Hello,

Let's suppose I am seeing slow write performance on a RAID5 or RAID6, and I am
suspecting that it's because of one of its devices operating slower than
others (maybe a soft-failing or just low-performing hard drive, in case with
non-identical member drives). Nothing in dmesg, no timeouts or ATA errors or
anything like that.

How would I go about diagnosing which device that is (if any) is slowing down
the array, aside from performance-testing each individual device (which seems
rather difficult if it's only slow on writes, not reads).

In 'top' there's of course high iowait, 50-70% all the time during writes. But
is that broken down somewhere to show which particular block device is being
the most 'waited for'?

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: RAID5/6 slow due to a member device -- how to diagnose?

[Mateusz Korniak]

On Thursday 03 March 2016 19:30:26 Roman Mamedov wrote:
> But is that broken down somewhere to show which particular block device is
> being the most 'waited for'?

iostat  ?

r_await w_await  svctm  %util  columns?

Regards,

-- 
Mateusz Korniak
"(...) mam brata - poważny, domator, liczykrupa, hipokryta, pobożniś,
 	krótko mówiąc - podpora społeczeństwa."
				Nikos Kazantzakis - "Grek Zorba"

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RAID5/6 slow due to a member device -- how to diagnose?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 875 bytes --]

On Thu, 03 Mar 2016 15:35:21 +0100
Mateusz Korniak <mateusz-lists@ant.gliwice.pl> wrote:

> On Thursday 03 March 2016 19:30:26 Roman Mamedov wrote:
> > But is that broken down somewhere to show which particular block device is
> > being the most 'waited for'?
> 
> iostat  ?
> 
> r_await w_await  svctm  %util  columns?

Cheers -- never knew there was this 'extended' output mode in iostat.
Indeed it pinpoints the culprit nicely.

Device: [...]    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sde1    [...] 53748.00   969.01     1.95   17.46   68.00   17.00   4.09  45.40
sdf1    [...] 49410.75  1176.97    32.09  393.95  106.00  397.42  11.90 100.00
sdg1    [...] 50230.00   995.17     7.12   66.81   88.00   66.60   6.08  61.40
sdh1    [...] 53110.00   970.89     2.46   22.21   46.40   21.64   4.64  50.80

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: RAID5/6 slow due to a member device -- how to diagnose?

[Carlos Carvalho]

Roman Mamedov (rm@romanrm.net) wrote on Thu, Mar 03, 2016 at 11:30:26AM BRT:
> Let's suppose I am seeing slow write performance on a RAID5 or RAID6, and I am
> suspecting that it's because of one of its devices operating slower than
> others (maybe a soft-failing or just low-performing hard drive, in case with
> non-identical member drives). Nothing in dmesg, no timeouts or ATA errors or
> anything like that.
> 
> How would I go about diagnosing which device that is (if any) is slowing down
> the array, aside from performance-testing each individual device (which seems
> rather difficult if it's only slow on writes, not reads).

I use sar -d -p 5 |grep -v md