read errors not corrected when doing check on RAID6

read errors not corrected when doing check on RAID6

[Mikael Abrahamsson]

Hello.

Running Debian with 3.11-0.bpo.2-amd64 with mdadm 3.2.5 (later upgraded to 
3.3 to do the replace function described).

/dev/sds on the system had 9 pending sectors according to smartctl. I 
issued a check and got several read during the check. I started the check 
by taking what debian has in its crontab:

/usr/share/mdadm/checkarray --cron --all --idle --quiet

This resulted in this in dmesg (there are two arrays), I don't know why it 
didn't do the RAID conf printout for md2.

[  569.034227] RAID conf printout:
[  569.034233]  --- level:6 rd:10 wd:10
[  569.034236]  disk 0, o:1, dev:sdn
[  569.034238]  disk 1, o:1, dev:sdh
[  569.034241]  disk 2, o:1, dev:sdj
[  569.034243]  disk 3, o:1, dev:sdd
[  569.034245]  disk 4, o:1, dev:sds
[  569.034248]  disk 5, o:1, dev:sdv
[  569.034250]  disk 6, o:1, dev:sdb
[  569.034252]  disk 7, o:1, dev:sdc
[  569.034254]  disk 8, o:1, dev:sdf
[  569.034257]  disk 9, o:1, dev:sde
[  569.034292] md: data-check of RAID array md0
[  569.034337] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  569.034384] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[  569.034440] md: using 128k window, over a total of 1953512960k.
[  569.520326] md: data-check of RAID array md2
[  569.520379] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  569.520427] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[  569.520488] md: using 128k window, over a total of 2930265408k.
[  662.413059] perf samples too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
[  714.577123] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[  714.577187] ata13.01: failed command: READ DMA EXT
[  714.577245] ata13.01: cmd 25/00:00:58:3a:7c/00:04:00:00:00/f0 tag 0 dma 524288 in
[  714.577245]          res 51/40:28:30:3c:7c/40:02:00:00:00/f0 Emask 0x9 (media error)
[  714.577330] ata13.01: status: { DRDY ERR }
[  714.577381] ata13.01: error: { UNC }
[  714.796272] ata13.00: configured for UDMA/133
[  714.804347] ata13.01: configured for UDMA/133
[  714.804767] sd 13:0:1:0: [sds] Unhandled sense code
[  714.804887] sd 13:0:1:0: [sds]
[  714.804988] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  714.805094] sd 13:0:1:0: [sds]
[  714.805194] Sense Key : Medium Error [current] [descriptor]
[  714.805491] Descriptor sense data with sense descriptors (in hex):
[  714.805665]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[  714.806845]         00 7c 3c 30
[  714.807257] sd 13:0:1:0: [sds]
[  714.807357] Add. Sense: Unrecovered read error - auto reallocate failed
[  714.807531] sd 13:0:1:0: [sds] CDB:
[  714.807632] Read(10): 28 00 00 7c 3a 58 00 04 00 00
[  714.808494] end_request: I/O error, dev sds, sector 8141872
[  714.808638] ata13: EH complete

There were even more errors (not included), but md didn't seem to correct 
them. After the check, the number of pending sectors on /dev/sds was still 
9.

The reason to upgrade fmor kernel 3.2 to 3.11 was to gain the use of the 
replace function. I then issued:

mdadm /dev/md0 --replace /dev/sds

dmesg:

[74791.213852] RAID conf printout:
[74791.213858]  --- level:6 rd:10 wd:10
[74791.213861]  disk 0, o:1, dev:sdn
[74791.213864]  disk 1, o:1, dev:sdh
[74791.213866]  disk 2, o:1, dev:sdj
[74791.213869]  disk 3, o:1, dev:sdd
[74791.213871]  disk 4, o:1, dev:sds
[74791.213874]  disk 5, o:1, dev:sdv
[74791.213876]  disk 6, o:1, dev:sdb
[74791.213879]  disk 7, o:1, dev:sdc
[74791.213881]  disk 8, o:1, dev:sdf
[74791.213884]  disk 9, o:1, dev:sde
[74791.252331] md: recovery of RAID array md0
[74791.252371] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[74791.252405] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[74791.252466] md: using 128k window, over a total of 1953512960k.
[74872.590157] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[74872.590193] ata13.01: failed command: READ DMA EXT
[74872.590226] ata13.01: cmd 25/00:00:38:fd:7b/00:04:00:00:00/f0 tag 0 dma 524288 in
[74872.590226]          res 51/40:48:f0:fe:7b/40:02:00:00:00/f0 Emask 0x9 (media error)
[74872.590315] ata13.01: status: { DRDY ERR }
[74872.590341] ata13.01: error: { UNC }
[74872.820239] ata13.00: configured for UDMA/133
[74872.828258] ata13.01: configured for UDMA/133
[74872.828571] sd 13:0:1:0: [sds] Unhandled sense code
[74872.828601] sd 13:0:1:0: [sds]
[74872.828624] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74872.828652] sd 13:0:1:0: [sds]
[74872.828675] Sense Key : Medium Error [current] [descriptor]
[74872.828710] Descriptor sense data with sense descriptors (in hex):
[74872.828753]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[74872.828816]         00 7b fe f0
[74872.828850] sd 13:0:1:0: [sds]
[74872.828874] Add. Sense: Unrecovered read error - auto reallocate failed
[74872.828904] sd 13:0:1:0: [sds] CDB:
[74872.828928] Read(10): 28 00 00 7b fd 38 00 04 00 00
[74872.828979] end_request: I/O error, dev sds, sector 8126192
[74872.829055] ata13: EH complete
[74877.699161] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[74877.699197] ata13.01: failed command: READ DMA
[74877.699228] ata13.01: cmd c8/00:08:f0:fe:7b/00:00:00:00:00/f0 tag 0 dma 4096 in
[74877.699228]          res 51/40:08:f0:fe:7b/40:02:00:00:00/f0 Emask 0x9 (media error)
[74877.699317] ata13.01: status: { DRDY ERR }
[74877.699342] ata13.01: error: { UNC }
[74877.928227] ata13.00: configured for UDMA/133
[74877.936278] ata13.01: configured for UDMA/133
[74877.936317] sd 13:0:1:0: [sds] Unhandled sense code
[74877.936343] sd 13:0:1:0: [sds]
[74877.936367] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74877.936395] sd 13:0:1:0: [sds]
[74877.936417] Sense Key : Medium Error [current] [descriptor]
[74877.936450] Descriptor sense data with sense descriptors (in hex):
[74877.936479]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[74877.936541]         00 7b fe f0
[74877.936573] sd 13:0:1:0: [sds]
[74877.936597] Add. Sense: Unrecovered read error - auto reallocate failed
[74877.936628] sd 13:0:1:0: [sds] CDB:
[74877.936651] Read(10): 28 00 00 7b fe f0 00 00 08 00
[74877.936701] end_request: I/O error, dev sds, sector 8126192
[74877.936730] end_request: I/O error, dev sds, sector 8126192
[74877.936771] ata13: EH complete
[74884.058379] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[74884.058416] ata13.01: failed command: READ DMA EXT
[74884.058447] ata13.01: cmd 25/00:00:38:39:7c/00:04:00:00:00/f0 tag 0 dma 524288 in
[74884.058447]          res 51/40:08:30:3c:7c/40:01:00:00:00/f0 Emask 0x9 (media error)
[74884.058535] ata13.01: status: { DRDY ERR }
[74884.058561] ata13.01: error: { UNC }
[74884.288226] ata13.00: configured for UDMA/133
[74884.296277] ata13.01: configured for UDMA/133
[74884.296570] sd 13:0:1:0: [sds] Unhandled sense code
[74884.296598] sd 13:0:1:0: [sds]
[74884.296622] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74884.296650] sd 13:0:1:0: [sds]
[74884.296673] Sense Key : Medium Error [current] [descriptor]
[74884.296708] Descriptor sense data with sense descriptors (in hex):
[74884.296739]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[74884.296805]         00 7c 3c 30
[74884.296838] sd 13:0:1:0: [sds]
[74884.296862] Add. Sense: Unrecovered read error - auto reallocate failed
[74884.296892] sd 13:0:1:0: [sds] CDB:
[74884.296916] Read(10): 28 00 00 7c 39 38 00 04 00 00
[74884.296967] end_request: I/O error, dev sds, sector 8141872
[74884.297020] ata13: EH complete
[74889.334006] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[74889.334041] ata13.01: failed command: READ DMA
[74889.334072] ata13.01: cmd c8/00:08:30:3c:7c/00:00:00:00:00/f0 tag 0 dma 4096 in
[74889.334072]          res 51/40:08:30:3c:7c/40:01:00:00:00/f0 Emask 0x9 (media error)
[74889.334159] ata13.01: status: { DRDY ERR }
[74889.334184] ata13.01: error: { UNC }
[74889.564227] ata13.00: configured for UDMA/133
[74889.572286] ata13.01: configured for UDMA/133
[74889.572322] sd 13:0:1:0: [sds] Unhandled sense code
[74889.572349] sd 13:0:1:0: [sds]
[74889.572372] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74889.572399] sd 13:0:1:0: [sds]
[74889.572422] Sense Key : Medium Error [current] [descriptor]
[74889.572455] Descriptor sense data with sense descriptors (in hex):
[74889.572484]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[74889.572546]         00 7c 3c 30
[74889.572579] sd 13:0:1:0: [sds]
[74889.572603] Add. Sense: Unrecovered read error - auto reallocate failed
[74889.572633] sd 13:0:1:0: [sds] CDB:
[74889.572657] Read(10): 28 00 00 7c 3c 30 00 00 08 00
[74889.572708] end_request: I/O error, dev sds, sector 8141872
[74889.572737] end_request: I/O error, dev sds, sector 8141872
[74889.572774] ata13: EH complete
[74891.334029] md/raid:md0: read error corrected (8 sectors at 8126192 on sds)
[74891.353112] md/raid:md0: read error corrected (8 sectors at 8141872 on sds)

So here now in the replace function, md decided to correct the read errors 
on /dev/sds. Now the pending sectors are down to 7 on /dev/sds. There were 
some more errors which were also corrected, and now the pending count is 
down to 5. Recovery is still ongoing.

So my question is, what did I do wrong the first time when doing the 
check? I expected it to correct the read errors on the drive then, but it 
seems it didn't. Should I have issued a "repair" instead?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: read errors not corrected when doing check on RAID6

[Peter Grandi]

[ ... ]
> So here now in the replace function, md decided to correct the
> read errors on /dev/sds. Now the pending sectors are down to 7
> on /dev/sds. There were some more errors which were also
> corrected, and now the pending count is down to 5. Recovery is
> still ongoing. [ ... ]  I expected it to correct the read
> errors on the drive then, but it seems it didn't. [ ... ]

That seems to me a big misunderstanding (or delusion).

Re: read errors not corrected when doing check on RAID6

[Mikael Abrahamsson]

On Sun, 12 Jan 2014, Peter Grandi wrote:

> [ ... ]
>> So here now in the replace function, md decided to correct the
>> read errors on /dev/sds. Now the pending sectors are down to 7
>> on /dev/sds. There were some more errors which were also
>> corrected, and now the pending count is down to 5. Recovery is
>> still ongoing. [ ... ]  I expected it to correct the read
>> errors on the drive then, but it seems it didn't. [ ... ]
>
> That seems to me a big misunderstanding (or delusion).

Which of it? Pending sectors (which from my understanding is the same as 
UNC when reading) on a component drive in RAID6 wouldn't be re-calculated 
from parity and written to fix the problem?

But from my other email, it seems "pending sector" doesn't mean it always 
gives read errors, but just sometimes. The read errors I got from the 
check doesn't match up with the read errors I got when I did replace, and 
it also seems the read errors didn't always occur.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: read errors not corrected when doing check on RAID6

[Mikael Abrahamsson]

On Sun, 12 Jan 2014, Mikael Abrahamsson wrote:

After the replace event is done, here is an excerpt (at 569 I initiated 
the inital "check". At 74791 I initiated the replace of sds.

# dmesg | egrep -i 'end_request|md0'
[  103.321478] md: md0 stopped.
[  103.697351] md/raid:md0: device sdn operational as raid disk 0
[  103.697408] md/raid:md0: device sde operational as raid disk 9
[  103.697464] md/raid:md0: device sdf operational as raid disk 8
[  103.697520] md/raid:md0: device sdc operational as raid disk 7
[  103.697575] md/raid:md0: device sdb operational as raid disk 6
[  103.697631] md/raid:md0: device sdv operational as raid disk 5
[  103.697687] md/raid:md0: device sds operational as raid disk 4
[  103.697742] md/raid:md0: device sdd operational as raid disk 3
[  103.699136] md/raid:md0: device sdj operational as raid disk 2
[  103.699191] md/raid:md0: device sdh operational as raid disk 1
[  103.699925] md/raid:md0: allocated 10674kB
[  103.700000] md/raid:md0: raid level 6 active with 10 out of 10 devices, algorithm 2
[  103.700233] created bitmap (15 pages) for device md0
[  103.700714] md0: bitmap initialized from disk: read 1 pages, set 0 of 29809 bits
[  103.785552] md0: detected capacity change from 0 to 16003178168320
[  103.791690]  md0: unknown partition table
[  569.034292] md: data-check of RAID array md0
[  714.808494] end_request: I/O error, dev sds, sector 8141872
[  868.466729] end_request: I/O error, dev sds, sector 16075040
[ 1095.400603] end_request: I/O error, dev sds, sector 28157152
[ 1119.427166] end_request: I/O error, dev sds, sector 29280528
[45411.209327] md: md0: data-check done.
[74791.252331] md: recovery of RAID array md0
[74872.828979] end_request: I/O error, dev sds, sector 8126192
[74877.936701] end_request: I/O error, dev sds, sector 8126192
[74877.936730] end_request: I/O error, dev sds, sector 8126192
[74884.296967] end_request: I/O error, dev sds, sector 8141872
[74889.572708] end_request: I/O error, dev sds, sector 8141872
[74889.572737] end_request: I/O error, dev sds, sector 8141872
[74891.334029] md/raid:md0: read error corrected (8 sectors at 8126192 on sds)
[74891.353112] md/raid:md0: read error corrected (8 sectors at 8141872 on sds)
[75038.596998] end_request: I/O error, dev sds, sector 29280528
[75043.278096] end_request: I/O error, dev sds, sector 29280528
[75043.278124] end_request: I/O error, dev sds, sector 29280528
[75043.464460] md/raid:md0: read error corrected (8 sectors at 29280528 on sds)
[75055.565033] end_request: I/O error, dev sds, sector 30348408
[75060.840703] end_request: I/O error, dev sds, sector 30348408
[75060.840731] end_request: I/O error, dev sds, sector 30348408
[75061.051075] md/raid:md0: read error corrected (8 sectors at 30348408 on sds)
[75067.796988] end_request: I/O error, dev sds, sector 30733328
[113272.067198] md: md0: recovery done.

# smartctl -a /dev/sds | grep -i pending
197 Current_Pending_Sector  0x0032   100   100   000    Old_age   Always       -       4

So sds has gone down from 9 to 4 pending sectors during the replace 
operation. This doesn't make sense to me at all. Above seems to indicate 
that md wants 3 read errors in order to correct?

# smartctl -a /dev/sds | less
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.11-0.bpo.2-amd64] (local 
build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     SAMSUNG SpinPoint F4 EG (AFT)
Device Model:     SAMSUNG HD204UI
LU WWN Device Id: 5 0024e9 004b27bb0
Firmware Version: 1AQ10001
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 6

From dmesg as well:

[113272.067198] md: md0: recovery done.
[113272.528813] RAID conf printout:
[113272.528818]  --- level:6 rd:10 wd:10
[113272.528821]  disk 0, o:1, dev:sdn
[113272.528824]  disk 1, o:1, dev:sdh
[113272.528827]  disk 2, o:1, dev:sdj
[113272.528829]  disk 3, o:1, dev:sdd
[113272.528831]  disk 4, o:0, dev:sds
[113272.528834]  disk 5, o:1, dev:sdv
[113272.528836]  disk 6, o:1, dev:sdb
[113272.528839]  disk 7, o:1, dev:sdc
[113272.528841]  disk 8, o:1, dev:sdf
[113272.528844]  disk 9, o:1, dev:sde
[113272.661106] RAID conf printout:
[113272.661111]  --- level:6 rd:10 wd:10
[113272.661113]  disk 0, o:1, dev:sdn
[113272.661114]  disk 1, o:1, dev:sdh
[113272.661116]  disk 2, o:1, dev:sdj
[113272.661118]  disk 3, o:1, dev:sdd
[113272.661119]  disk 4, o:0, dev:sds
[113272.661121]  disk 5, o:1, dev:sdv
[113272.661123]  disk 6, o:1, dev:sdb
[113272.661124]  disk 7, o:1, dev:sdc
[113272.661126]  disk 8, o:1, dev:sdf
[113272.661127]  disk 9, o:1, dev:sde
[113272.668116] RAID conf printout:
[113272.668120]  --- level:6 rd:10 wd:10
[113272.668123]  disk 0, o:1, dev:sdn
[113272.668126]  disk 1, o:1, dev:sdh
[113272.668129]  disk 2, o:1, dev:sdj
[113272.668132]  disk 3, o:1, dev:sdd
[113272.668134]  disk 4, o:1, dev:sdk
[113272.668137]  disk 5, o:1, dev:sdv
[113272.668139]  disk 6, o:1, dev:sdb
[113272.668142]  disk 7, o:1, dev:sdc
[113272.668145]  disk 8, o:1, dev:sdf
[113272.668147]  disk 9, o:1, dev:sde

So the operation was successful it seems, it's just that I don't 
udnerstand why the initial "check" didn't find and fix all the pending 
sectors?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: read errors not corrected when doing check on RAID6

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 11506 bytes --]

On Sun, 12 Jan 2014 07:41:14 +0100 (CET) Mikael Abrahamsson
<swmike@swm.pp.se> wrote:

> 
> Hello.
> 
> Running Debian with 3.11-0.bpo.2-amd64 with mdadm 3.2.5 (later upgraded to 
> 3.3 to do the replace function described).
> 
> /dev/sds on the system had 9 pending sectors according to smartctl. I 
> issued a check and got several read during the check. I started the check 
> by taking what debian has in its crontab:
> 
> /usr/share/mdadm/checkarray --cron --all --idle --quiet
> 
> This resulted in this in dmesg (there are two arrays), I don't know why it 
> didn't do the RAID conf printout for md2.
> 
> [  569.034227] RAID conf printout:
> [  569.034233]  --- level:6 rd:10 wd:10
> [  569.034236]  disk 0, o:1, dev:sdn
> [  569.034238]  disk 1, o:1, dev:sdh
> [  569.034241]  disk 2, o:1, dev:sdj
> [  569.034243]  disk 3, o:1, dev:sdd
> [  569.034245]  disk 4, o:1, dev:sds
> [  569.034248]  disk 5, o:1, dev:sdv
> [  569.034250]  disk 6, o:1, dev:sdb
> [  569.034252]  disk 7, o:1, dev:sdc
> [  569.034254]  disk 8, o:1, dev:sdf
> [  569.034257]  disk 9, o:1, dev:sde
> [  569.034292] md: data-check of RAID array md0
> [  569.034337] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
> [  569.034384] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> [  569.034440] md: using 128k window, over a total of 1953512960k.
> [  569.520326] md: data-check of RAID array md2
> [  569.520379] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
> [  569.520427] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> [  569.520488] md: using 128k window, over a total of 2930265408k.
> [  662.413059] perf samples too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
> [  714.577123] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [  714.577187] ata13.01: failed command: READ DMA EXT
> [  714.577245] ata13.01: cmd 25/00:00:58:3a:7c/00:04:00:00:00/f0 tag 0 dma 524288 in
> [  714.577245]          res 51/40:28:30:3c:7c/40:02:00:00:00/f0 Emask 0x9 (media error)
> [  714.577330] ata13.01: status: { DRDY ERR }
> [  714.577381] ata13.01: error: { UNC }
> [  714.796272] ata13.00: configured for UDMA/133
> [  714.804347] ata13.01: configured for UDMA/133
> [  714.804767] sd 13:0:1:0: [sds] Unhandled sense code
> [  714.804887] sd 13:0:1:0: [sds]
> [  714.804988] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [  714.805094] sd 13:0:1:0: [sds]
> [  714.805194] Sense Key : Medium Error [current] [descriptor]
> [  714.805491] Descriptor sense data with sense descriptors (in hex):
> [  714.805665]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
> [  714.806845]         00 7c 3c 30
> [  714.807257] sd 13:0:1:0: [sds]
> [  714.807357] Add. Sense: Unrecovered read error - auto reallocate failed
> [  714.807531] sd 13:0:1:0: [sds] CDB:
> [  714.807632] Read(10): 28 00 00 7c 3a 58 00 04 00 00
> [  714.808494] end_request: I/O error, dev sds, sector 8141872
> [  714.808638] ata13: EH complete
> 
> There were even more errors (not included), but md didn't seem to correct 
> them. After the check, the number of pending sectors on /dev/sds was still 
> 9.
> 
> The reason to upgrade fmor kernel 3.2 to 3.11 was to gain the use of the 
> replace function. I then issued:
> 
> mdadm /dev/md0 --replace /dev/sds
> 
> dmesg:
> 
> [74791.213852] RAID conf printout:
> [74791.213858]  --- level:6 rd:10 wd:10
> [74791.213861]  disk 0, o:1, dev:sdn
> [74791.213864]  disk 1, o:1, dev:sdh
> [74791.213866]  disk 2, o:1, dev:sdj
> [74791.213869]  disk 3, o:1, dev:sdd
> [74791.213871]  disk 4, o:1, dev:sds
> [74791.213874]  disk 5, o:1, dev:sdv
> [74791.213876]  disk 6, o:1, dev:sdb
> [74791.213879]  disk 7, o:1, dev:sdc
> [74791.213881]  disk 8, o:1, dev:sdf
> [74791.213884]  disk 9, o:1, dev:sde
> [74791.252331] md: recovery of RAID array md0
> [74791.252371] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
> [74791.252405] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
> [74791.252466] md: using 128k window, over a total of 1953512960k.
> [74872.590157] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [74872.590193] ata13.01: failed command: READ DMA EXT
> [74872.590226] ata13.01: cmd 25/00:00:38:fd:7b/00:04:00:00:00/f0 tag 0 dma 524288 in
> [74872.590226]          res 51/40:48:f0:fe:7b/40:02:00:00:00/f0 Emask 0x9 (media error)
> [74872.590315] ata13.01: status: { DRDY ERR }
> [74872.590341] ata13.01: error: { UNC }
> [74872.820239] ata13.00: configured for UDMA/133
> [74872.828258] ata13.01: configured for UDMA/133
> [74872.828571] sd 13:0:1:0: [sds] Unhandled sense code
> [74872.828601] sd 13:0:1:0: [sds]
> [74872.828624] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [74872.828652] sd 13:0:1:0: [sds]
> [74872.828675] Sense Key : Medium Error [current] [descriptor]
> [74872.828710] Descriptor sense data with sense descriptors (in hex):
> [74872.828753]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
> [74872.828816]         00 7b fe f0
> [74872.828850] sd 13:0:1:0: [sds]
> [74872.828874] Add. Sense: Unrecovered read error - auto reallocate failed
> [74872.828904] sd 13:0:1:0: [sds] CDB:
> [74872.828928] Read(10): 28 00 00 7b fd 38 00 04 00 00
> [74872.828979] end_request: I/O error, dev sds, sector 8126192
> [74872.829055] ata13: EH complete
> [74877.699161] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [74877.699197] ata13.01: failed command: READ DMA
> [74877.699228] ata13.01: cmd c8/00:08:f0:fe:7b/00:00:00:00:00/f0 tag 0 dma 4096 in
> [74877.699228]          res 51/40:08:f0:fe:7b/40:02:00:00:00/f0 Emask 0x9 (media error)
> [74877.699317] ata13.01: status: { DRDY ERR }
> [74877.699342] ata13.01: error: { UNC }
> [74877.928227] ata13.00: configured for UDMA/133
> [74877.936278] ata13.01: configured for UDMA/133
> [74877.936317] sd 13:0:1:0: [sds] Unhandled sense code
> [74877.936343] sd 13:0:1:0: [sds]
> [74877.936367] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [74877.936395] sd 13:0:1:0: [sds]
> [74877.936417] Sense Key : Medium Error [current] [descriptor]
> [74877.936450] Descriptor sense data with sense descriptors (in hex):
> [74877.936479]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
> [74877.936541]         00 7b fe f0
> [74877.936573] sd 13:0:1:0: [sds]
> [74877.936597] Add. Sense: Unrecovered read error - auto reallocate failed
> [74877.936628] sd 13:0:1:0: [sds] CDB:
> [74877.936651] Read(10): 28 00 00 7b fe f0 00 00 08 00
> [74877.936701] end_request: I/O error, dev sds, sector 8126192
> [74877.936730] end_request: I/O error, dev sds, sector 8126192
> [74877.936771] ata13: EH complete
> [74884.058379] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [74884.058416] ata13.01: failed command: READ DMA EXT
> [74884.058447] ata13.01: cmd 25/00:00:38:39:7c/00:04:00:00:00/f0 tag 0 dma 524288 in
> [74884.058447]          res 51/40:08:30:3c:7c/40:01:00:00:00/f0 Emask 0x9 (media error)
> [74884.058535] ata13.01: status: { DRDY ERR }
> [74884.058561] ata13.01: error: { UNC }
> [74884.288226] ata13.00: configured for UDMA/133
> [74884.296277] ata13.01: configured for UDMA/133
> [74884.296570] sd 13:0:1:0: [sds] Unhandled sense code
> [74884.296598] sd 13:0:1:0: [sds]
> [74884.296622] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [74884.296650] sd 13:0:1:0: [sds]
> [74884.296673] Sense Key : Medium Error [current] [descriptor]
> [74884.296708] Descriptor sense data with sense descriptors (in hex):
> [74884.296739]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
> [74884.296805]         00 7c 3c 30
> [74884.296838] sd 13:0:1:0: [sds]
> [74884.296862] Add. Sense: Unrecovered read error - auto reallocate failed
> [74884.296892] sd 13:0:1:0: [sds] CDB:
> [74884.296916] Read(10): 28 00 00 7c 39 38 00 04 00 00
> [74884.296967] end_request: I/O error, dev sds, sector 8141872
> [74884.297020] ata13: EH complete
> [74889.334006] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
> [74889.334041] ata13.01: failed command: READ DMA
> [74889.334072] ata13.01: cmd c8/00:08:30:3c:7c/00:00:00:00:00/f0 tag 0 dma 4096 in
> [74889.334072]          res 51/40:08:30:3c:7c/40:01:00:00:00/f0 Emask 0x9 (media error)
> [74889.334159] ata13.01: status: { DRDY ERR }
> [74889.334184] ata13.01: error: { UNC }
> [74889.564227] ata13.00: configured for UDMA/133
> [74889.572286] ata13.01: configured for UDMA/133
> [74889.572322] sd 13:0:1:0: [sds] Unhandled sense code
> [74889.572349] sd 13:0:1:0: [sds]
> [74889.572372] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> [74889.572399] sd 13:0:1:0: [sds]
> [74889.572422] Sense Key : Medium Error [current] [descriptor]
> [74889.572455] Descriptor sense data with sense descriptors (in hex):
> [74889.572484]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
> [74889.572546]         00 7c 3c 30
> [74889.572579] sd 13:0:1:0: [sds]
> [74889.572603] Add. Sense: Unrecovered read error - auto reallocate failed
> [74889.572633] sd 13:0:1:0: [sds] CDB:
> [74889.572657] Read(10): 28 00 00 7c 3c 30 00 00 08 00
> [74889.572708] end_request: I/O error, dev sds, sector 8141872
> [74889.572737] end_request: I/O error, dev sds, sector 8141872
> [74889.572774] ata13: EH complete
> [74891.334029] md/raid:md0: read error corrected (8 sectors at 8126192 on sds)
> [74891.353112] md/raid:md0: read error corrected (8 sectors at 8141872 on sds)
> 
> So here now in the replace function, md decided to correct the read errors 
> on /dev/sds. Now the pending sectors are down to 7 on /dev/sds. There were 
> some more errors which were also corrected, and now the pending count is 
> down to 5. Recovery is still ongoing.
> 
> So my question is, what did I do wrong the first time when doing the 
> check? I expected it to correct the read errors on the drive then, but it 
> seems it didn't. Should I have issued a "repair" instead?
> 

Don't be too hasty in assuming you did something wrong.  When working with
computers, it is usually the computer that did something wrongly :-)

A 'check' certainly should correct read errors.  The code for handling read
errors makes no distinction between 'check' or 'repair' or regular IO.

When md/raid[56] gets a read error, the first thing it does is retry the read
with REQ_FLUSH (or in later kernels: REQ_NOMERGE) set.  This ensure that each
1-page read goes down to the drive as a stand-alone request so we know with
more precision which sector had the error (usually multiple blocks are merged
so an error might mean "one block in a 128K region near here failed").
If this second read fails, then that required data is generated from
elsewhere and a write is scheduled.

My only guess is that in your case the second read succeeded when you were
doing a 'check', but it failed when you were doing a 'replace'.

To check this you can looked for "end_request: I/O error" in the kernel logs.
In the fragments you included there is just one error during the 'check', but
3 for each block during the 'replace'.  Given the timestamps, the 2nd and 3rd
are probably for the same request - some logging weirdness maybe.
Obviously you only provided fragments so we can't read too much into what we
see.  Looking at the complete logs could be more helpful.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: read errors not corrected when doing check on RAID6

[Mikael Abrahamsson]

On Thu, 16 Jan 2014, NeilBrown wrote:

> To check this you can looked for "end_request: I/O error" in the kernel 
> logs. In the fragments you included there is just one error during the 
> 'check', but 3 for each block during the 'replace'.  Given the 
> timestamps, the 2nd and 3rd are probably for the same request - some 
> logging weirdness maybe. Obviously you only provided fragments so we 
> can't read too much into what we see.  Looking at the complete logs 
> could be more helpful.

Sure, here it is. I cut off the boot stuff, I don't think there was that 
much interesting there

[  103.321478] md: md0 stopped.
[  103.326136] md: bind<sdh>
[  103.326481] md: bind<sdj>
[  103.326678] md: bind<sdd>
[  103.326921] md: bind<sds>
[  103.327120] md: bind<sdv>
[  103.327295] md: bind<sdb>
[  103.327466] md: bind<sdc>
[  103.327639] md: bind<sdf>
[  103.327850] md: bind<sde>
[  103.328076] md: bind<sdk>
[  103.328251] md: bind<sdn>
[  103.403226] async_tx: api initialized (async)
[  103.404441] xor: measuring software checksum speed
[  103.444005]    prefetch64-sse:  9607.000 MB/sec
[  103.484002]    generic_sse:  8565.000 MB/sec
[  103.484055] xor: using function: prefetch64-sse (9607.000 MB/sec)
[  103.552005] raid6: sse2x1    4263 MB/s
[  103.620015] raid6: sse2x2    5114 MB/s
[  103.688011] raid6: sse2x4    7205 MB/s
[  103.688063] raid6: using algorithm sse2x4 (7205 MB/s)
[  103.688116] raid6: using ssse3x2 recovery algorithm
[  103.688936] w83627ehf: Found W83627DHG chip at 0x290
[  103.689018] ACPI Warning: 0x0000000000000295-0x0000000000000296 SystemIO conflicts with Region \_SB_.PCI0.SBRG.SIOR.HWRE 1 (20130517/utaddress-251)
[  103.689172] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[  103.696793] md: raid6 personality registered for level 6
[  103.696854] md: raid5 personality registered for level 5
[  103.696910] md: raid4 personality registered for level 4
[  103.697351] md/raid:md0: device sdn operational as raid disk 0
[  103.697408] md/raid:md0: device sde operational as raid disk 9
[  103.697464] md/raid:md0: device sdf operational as raid disk 8
[  103.697520] md/raid:md0: device sdc operational as raid disk 7
[  103.697575] md/raid:md0: device sdb operational as raid disk 6
[  103.697631] md/raid:md0: device sdv operational as raid disk 5
[  103.697687] md/raid:md0: device sds operational as raid disk 4
[  103.697742] md/raid:md0: device sdd operational as raid disk 3
[  103.699136] md/raid:md0: device sdj operational as raid disk 2
[  103.699191] md/raid:md0: device sdh operational as raid disk 1
[  103.699925] md/raid:md0: allocated 10674kB
[  103.700000] md/raid:md0: raid level 6 active with 10 out of 10 devices, algorithm 2
[  103.700081] RAID conf printout:
[  103.700083]  --- level:6 rd:10 wd:10
[  103.700085]  disk 0, o:1, dev:sdn
[  103.700086]  disk 1, o:1, dev:sdh
[  103.700088]  disk 2, o:1, dev:sdj
[  103.700089]  disk 3, o:1, dev:sdd
[  103.700091]  disk 4, o:1, dev:sds
[  103.700092]  disk 5, o:1, dev:sdv
[  103.700094]  disk 6, o:1, dev:sdb
[  103.700095]  disk 7, o:1, dev:sdc
[  103.700097]  disk 8, o:1, dev:sdf
[  103.700098]  disk 9, o:1, dev:sde
[  103.700233] created bitmap (15 pages) for device md0
[  103.700714] md0: bitmap initialized from disk: read 1 pages, set 0 of 29809 bits
[  103.785552] md0: detected capacity change from 0 to 16003178168320
[  103.785641] RAID conf printout:
[  103.785644]  --- level:6 rd:10 wd:10
[  103.785646]  disk 0, o:1, dev:sdn
[  103.785648]  disk 1, o:1, dev:sdh
[  103.785650]  disk 2, o:1, dev:sdj
[  103.785651]  disk 3, o:1, dev:sdd
[  103.785653]  disk 4, o:1, dev:sds
[  103.785655]  disk 5, o:1, dev:sdv
[  103.785657]  disk 6, o:1, dev:sdb
[  103.785658]  disk 7, o:1, dev:sdc
[  103.785660]  disk 8, o:1, dev:sdf
[  103.785662]  disk 9, o:1, dev:sde
[  103.791690]  md0: unknown partition table
[  104.031804] md: md2 stopped.
[  104.035115] md: bind<sdm>
[  104.035312] md: bind<sdo>
[  104.035483] md: bind<sdp>
[  104.035661] md: bind<sdq>
[  104.035879] md: bind<sdw>
[  104.036102] md: bind<sdu>
[  104.036281] md: bind<sdl>
[  104.139144] md/raid:md2: device sdl operational as raid disk 0
[  104.139205] md/raid:md2: device sdu operational as raid disk 6
[  104.139262] md/raid:md2: device sdw operational as raid disk 5
[  104.139318] md/raid:md2: device sdq operational as raid disk 4
[  104.139374] md/raid:md2: device sdp operational as raid disk 3
[  104.139430] md/raid:md2: device sdo operational as raid disk 2
[  104.139486] md/raid:md2: device sdm operational as raid disk 1
[  104.140104] md/raid:md2: allocated 7506kB
[  104.141433] md/raid:md2: raid level 6 active with 7 out of 7 devices, algorithm 2
[  104.141514] RAID conf printout:
[  104.141515]  --- level:6 rd:7 wd:7
[  104.141517]  disk 0, o:1, dev:sdl
[  104.141519]  disk 1, o:1, dev:sdm
[  104.141521]  disk 2, o:1, dev:sdo
[  104.141523]  disk 3, o:1, dev:sdp
[  104.141524]  disk 4, o:1, dev:sdq
[  104.141526]  disk 5, o:1, dev:sdw
[  104.141528]  disk 6, o:1, dev:sdu
[  104.141658] created bitmap (22 pages) for device md2
[  104.142227] md2: bitmap initialized from disk: read 2 pages, set 0 of 44713 bits
[  104.284166] md2: detected capacity change from 0 to 15002958888960
[  104.320646]  md2: unknown partition table
[  114.003127] Adding 3905528k swap on /dev/mapper/swap.  Priority:-1 extents:1 across:3905528k SS
[  114.118236] EXT4-fs (sdr1): mounting ext2 file system using the ext4 subsystem
[  114.119773] EXT4-fs (sdr1): mounted filesystem without journal. Opts: (null)
[  114.151684] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[  114.152897] XFS (dm-5): Mounting Filesystem
[  114.673690] XFS (dm-5): Ending clean mount
[  114.685733] XFS (dm-4): Mounting Filesystem
[  115.508942] XFS (dm-4): Ending clean mount
[  116.233038] RPC: Registered named UNIX socket transport module.
[  116.233115] RPC: Registered udp transport module.
[  116.233170] RPC: Registered tcp transport module.
[  116.233226] RPC: Registered tcp NFSv4.1 backchannel transport module.
[  116.239399] FS-Cache: Loaded
[  116.254219] FS-Cache: Netfs 'nfs' registered for caching
[  116.269545] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[  116.366618] fuse init (API version 7.22)
[  117.208184] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[  117.208705] NFSD: starting 90-second grace period (net ffffffff81882080)
[  117.507070] ip_tables: (C) 2000-2006 Netfilter Core Team
[  117.519316] ip6_tables: (C) 2000-2006 Netfilter Core Team
[  117.542465] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)

Here I started the check:

[  569.034227] RAID conf printout:
[  569.034233]  --- level:6 rd:10 wd:10
[  569.034236]  disk 0, o:1, dev:sdn
[  569.034238]  disk 1, o:1, dev:sdh
[  569.034241]  disk 2, o:1, dev:sdj
[  569.034243]  disk 3, o:1, dev:sdd
[  569.034245]  disk 4, o:1, dev:sds
[  569.034248]  disk 5, o:1, dev:sdv
[  569.034250]  disk 6, o:1, dev:sdb
[  569.034252]  disk 7, o:1, dev:sdc
[  569.034254]  disk 8, o:1, dev:sdf
[  569.034257]  disk 9, o:1, dev:sde
[  569.034292] md: data-check of RAID array md0
[  569.034337] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  569.034384] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[  569.034440] md: using 128k window, over a total of 1953512960k.
[  569.520326] md: data-check of RAID array md2
[  569.520379] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  569.520427] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
[  569.520488] md: using 128k window, over a total of 2930265408k.
[  662.413059] perf samples too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 50000
[  714.577123] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[  714.577187] ata13.01: failed command: READ DMA EXT
[  714.577245] ata13.01: cmd 25/00:00:58:3a:7c/00:04:00:00:00/f0 tag 0 dma 524288 in
[  714.577245]          res 51/40:28:30:3c:7c/40:02:00:00:00/f0 Emask 0x9 (media error)
[  714.577330] ata13.01: status: { DRDY ERR }
[  714.577381] ata13.01: error: { UNC }
[  714.796272] ata13.00: configured for UDMA/133
[  714.804347] ata13.01: configured for UDMA/133
[  714.804767] sd 13:0:1:0: [sds] Unhandled sense code
[  714.804887] sd 13:0:1:0: [sds] 
[  714.804988] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  714.805094] sd 13:0:1:0: [sds] 
[  714.805194] Sense Key : Medium Error [current] [descriptor]
[  714.805491] Descriptor sense data with sense descriptors (in hex):
[  714.805665]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[  714.806845]         00 7c 3c 30 
[  714.807257] sd 13:0:1:0: [sds] 
[  714.807357] Add. Sense: Unrecovered read error - auto reallocate failed
[  714.807531] sd 13:0:1:0: [sds] CDB: 
[  714.807632] Read(10): 28 00 00 7c 3a 58 00 04 00 00
[  714.808494] end_request: I/O error, dev sds, sector 8141872
[  714.808638] ata13: EH complete
[  868.240036] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[  868.240113] ata13.01: failed command: READ DMA EXT
[  868.240183] ata13.01: cmd 25/00:00:18:49:f5/00:04:00:00:00/f0 tag 0 dma 524288 in
[  868.240183]          res 51/40:f8:20:49:f5/40:03:00:00:00/f0 Emask 0x9 (media error)
[  868.240312] ata13.01: status: { DRDY ERR }
[  868.240374] ata13.01: error: { UNC }
[  868.456254] ata13.00: configured for UDMA/133
[  868.464333] ata13.01: configured for UDMA/133
[  868.464754] sd 13:0:1:0: [sds] Unhandled sense code
[  868.464812] sd 13:0:1:0: [sds] 
[  868.464866] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.464924] sd 13:0:1:0: [sds] 
[  868.464978] Sense Key : Medium Error [current] [descriptor]
[  868.465135] Descriptor sense data with sense descriptors (in hex):
[  868.465226]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[  868.465848]         00 f5 49 20 
[  868.466068] sd 13:0:1:0: [sds] 
[  868.466123] Add. Sense: Unrecovered read error - auto reallocate failed
[  868.466215] sd 13:0:1:0: [sds] CDB: 
[  868.466269] Read(10): 28 00 00 f5 49 18 00 04 00 00
[  868.466729] end_request: I/O error, dev sds, sector 16075040
[  868.466884] ata13: EH complete
[ 1095.166247] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1095.166327] ata13.01: failed command: READ DMA EXT
[ 1095.166400] ata13.01: cmd 25/00:00:10:a4:ad/00:04:01:00:00/f0 tag 0 dma 524288 in
[ 1095.166400]          res 51/40:30:e0:a4:ad/40:03:01:00:00/f0 Emask 0x9 (media error)
[ 1095.166546] ata13.01: status: { DRDY ERR }
[ 1095.166612] ata13.01: error: { UNC }
[ 1095.388269] ata13.00: configured for UDMA/133
[ 1095.396331] ata13.01: configured for UDMA/133
[ 1095.396938] sd 13:0:1:0: [sds] Unhandled sense code
[ 1095.397004] sd 13:0:1:0: [sds] 
[ 1095.397064] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1095.397129] sd 13:0:1:0: [sds] 
[ 1095.397190] Sense Key : Medium Error [current] [descriptor]
[ 1095.397370] Descriptor sense data with sense descriptors (in hex):
[ 1095.397473]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[ 1095.398154]         01 ad a4 e0 
[ 1095.399853] sd 13:0:1:0: [sds] 
[ 1095.399910] Add. Sense: Unrecovered read error - auto reallocate failed
[ 1095.400019] sd 13:0:1:0: [sds] CDB: 
[ 1095.400079] Read(10): 28 00 01 ad a4 10 00 04 00 00
[ 1095.400603] end_request: I/O error, dev sds, sector 28157152
[ 1095.400736] ata13: EH complete
[ 1119.195283] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[ 1119.195363] ata13.01: failed command: READ DMA EXT
[ 1119.195436] ata13.01: cmd 25/00:00:50:c6:be/00:04:01:00:00/f0 tag 0 dma 524288 in
[ 1119.195436]          res 51/40:40:10:c9:be/40:01:01:00:00/f0 Emask 0x9 (media error)
[ 1119.195583] ata13.01: status: { DRDY ERR }
[ 1119.195654] ata13.01: error: { UNC }
[ 1119.416277] ata13.00: configured for UDMA/133
[ 1119.424349] ata13.01: configured for UDMA/133
[ 1119.424803] sd 13:0:1:0: [sds] Unhandled sense code
[ 1119.424874] sd 13:0:1:0: [sds] 
[ 1119.424940] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 1119.425013] sd 13:0:1:0: [sds] 
[ 1119.425081] Sense Key : Medium Error [current] [descriptor]
[ 1119.425284] Descriptor sense data with sense descriptors (in hex):
[ 1119.425399]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[ 1119.426181]         01 be c9 10 
[ 1119.426431] sd 13:0:1:0: [sds] 
[ 1119.426492] Add. Sense: Unrecovered read error - auto reallocate failed
[ 1119.426595] sd 13:0:1:0: [sds] CDB: 
[ 1119.426654] Read(10): 28 00 01 be c6 50 00 04 00 00
[ 1119.427166] end_request: I/O error, dev sds, sector 29280528
[ 1119.427272] ata13: EH complete
[45411.209327] md: md0: data-check done.
[45412.147602] RAID conf printout:
[45412.147609]  --- level:6 rd:10 wd:10
[45412.147612]  disk 0, o:1, dev:sdn
[45412.147614]  disk 1, o:1, dev:sdh
[45412.147617]  disk 2, o:1, dev:sdj
[45412.147620]  disk 3, o:1, dev:sdd
[45412.147623]  disk 4, o:1, dev:sds
[45412.147625]  disk 5, o:1, dev:sdv
[45412.147628]  disk 6, o:1, dev:sdb
[45412.147631]  disk 7, o:1, dev:sdc
[45412.147634]  disk 8, o:1, dev:sdf
[45412.147636]  disk 9, o:1, dev:sde
[58772.330150] md: md2: data-check done.

Here I started the replace:

[74791.213852] RAID conf printout:
[74791.213858]  --- level:6 rd:10 wd:10
[74791.213861]  disk 0, o:1, dev:sdn
[74791.213864]  disk 1, o:1, dev:sdh
[74791.213866]  disk 2, o:1, dev:sdj
[74791.213869]  disk 3, o:1, dev:sdd
[74791.213871]  disk 4, o:1, dev:sds
[74791.213874]  disk 5, o:1, dev:sdv
[74791.213876]  disk 6, o:1, dev:sdb
[74791.213879]  disk 7, o:1, dev:sdc
[74791.213881]  disk 8, o:1, dev:sdf
[74791.213884]  disk 9, o:1, dev:sde
[74791.252331] md: recovery of RAID array md0
[74791.252371] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[74791.252405] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[74791.252466] md: using 128k window, over a total of 1953512960k.
[74872.590157] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[74872.590193] ata13.01: failed command: READ DMA EXT
[74872.590226] ata13.01: cmd 25/00:00:38:fd:7b/00:04:00:00:00/f0 tag 0 dma 524288 in
[74872.590226]          res 51/40:48:f0:fe:7b/40:02:00:00:00/f0 Emask 0x9 (media error)
[74872.590315] ata13.01: status: { DRDY ERR }
[74872.590341] ata13.01: error: { UNC }
[74872.820239] ata13.00: configured for UDMA/133
[74872.828258] ata13.01: configured for UDMA/133
[74872.828571] sd 13:0:1:0: [sds] Unhandled sense code
[74872.828601] sd 13:0:1:0: [sds] 
[74872.828624] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74872.828652] sd 13:0:1:0: [sds] 
[74872.828675] Sense Key : Medium Error [current] [descriptor]
[74872.828710] Descriptor sense data with sense descriptors (in hex):
[74872.828753]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[74872.828816]         00 7b fe f0 
[74872.828850] sd 13:0:1:0: [sds] 
[74872.828874] Add. Sense: Unrecovered read error - auto reallocate failed
[74872.828904] sd 13:0:1:0: [sds] CDB: 
[74872.828928] Read(10): 28 00 00 7b fd 38 00 04 00 00
[74872.828979] end_request: I/O error, dev sds, sector 8126192
[74872.829055] ata13: EH complete
[74877.699161] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[74877.699197] ata13.01: failed command: READ DMA
[74877.699228] ata13.01: cmd c8/00:08:f0:fe:7b/00:00:00:00:00/f0 tag 0 dma 4096 in
[74877.699228]          res 51/40:08:f0:fe:7b/40:02:00:00:00/f0 Emask 0x9 (media error)
[74877.699317] ata13.01: status: { DRDY ERR }
[74877.699342] ata13.01: error: { UNC }
[74877.928227] ata13.00: configured for UDMA/133
[74877.936278] ata13.01: configured for UDMA/133
[74877.936317] sd 13:0:1:0: [sds] Unhandled sense code
[74877.936343] sd 13:0:1:0: [sds] 
[74877.936367] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74877.936395] sd 13:0:1:0: [sds] 
[74877.936417] Sense Key : Medium Error [current] [descriptor]
[74877.936450] Descriptor sense data with sense descriptors (in hex):
[74877.936479]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[74877.936541]         00 7b fe f0 
[74877.936573] sd 13:0:1:0: [sds] 
[74877.936597] Add. Sense: Unrecovered read error - auto reallocate failed
[74877.936628] sd 13:0:1:0: [sds] CDB: 
[74877.936651] Read(10): 28 00 00 7b fe f0 00 00 08 00
[74877.936701] end_request: I/O error, dev sds, sector 8126192
[74877.936730] end_request: I/O error, dev sds, sector 8126192
[74877.936771] ata13: EH complete
[74884.058379] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[74884.058416] ata13.01: failed command: READ DMA EXT
[74884.058447] ata13.01: cmd 25/00:00:38:39:7c/00:04:00:00:00/f0 tag 0 dma 524288 in
[74884.058447]          res 51/40:08:30:3c:7c/40:01:00:00:00/f0 Emask 0x9 (media error)
[74884.058535] ata13.01: status: { DRDY ERR }
[74884.058561] ata13.01: error: { UNC }
[74884.288226] ata13.00: configured for UDMA/133
[74884.296277] ata13.01: configured for UDMA/133
[74884.296570] sd 13:0:1:0: [sds] Unhandled sense code
[74884.296598] sd 13:0:1:0: [sds] 
[74884.296622] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74884.296650] sd 13:0:1:0: [sds] 
[74884.296673] Sense Key : Medium Error [current] [descriptor]
[74884.296708] Descriptor sense data with sense descriptors (in hex):
[74884.296739]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[74884.296805]         00 7c 3c 30 
[74884.296838] sd 13:0:1:0: [sds] 
[74884.296862] Add. Sense: Unrecovered read error - auto reallocate failed
[74884.296892] sd 13:0:1:0: [sds] CDB: 
[74884.296916] Read(10): 28 00 00 7c 39 38 00 04 00 00
[74884.296967] end_request: I/O error, dev sds, sector 8141872
[74884.297020] ata13: EH complete
[74889.334006] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[74889.334041] ata13.01: failed command: READ DMA
[74889.334072] ata13.01: cmd c8/00:08:30:3c:7c/00:00:00:00:00/f0 tag 0 dma 4096 in
[74889.334072]          res 51/40:08:30:3c:7c/40:01:00:00:00/f0 Emask 0x9 (media error)
[74889.334159] ata13.01: status: { DRDY ERR }
[74889.334184] ata13.01: error: { UNC }
[74889.564227] ata13.00: configured for UDMA/133
[74889.572286] ata13.01: configured for UDMA/133
[74889.572322] sd 13:0:1:0: [sds] Unhandled sense code
[74889.572349] sd 13:0:1:0: [sds] 
[74889.572372] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[74889.572399] sd 13:0:1:0: [sds] 
[74889.572422] Sense Key : Medium Error [current] [descriptor]
[74889.572455] Descriptor sense data with sense descriptors (in hex):
[74889.572484]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[74889.572546]         00 7c 3c 30 
[74889.572579] sd 13:0:1:0: [sds] 
[74889.572603] Add. Sense: Unrecovered read error - auto reallocate failed
[74889.572633] sd 13:0:1:0: [sds] CDB: 
[74889.572657] Read(10): 28 00 00 7c 3c 30 00 00 08 00
[74889.572708] end_request: I/O error, dev sds, sector 8141872
[74889.572737] end_request: I/O error, dev sds, sector 8141872
[74889.572774] ata13: EH complete
[74891.334029] md/raid:md0: read error corrected (8 sectors at 8126192 on sds)
[74891.353112] md/raid:md0: read error corrected (8 sectors at 8141872 on sds)
[75038.358845] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[75038.358881] ata13.01: failed command: READ DMA EXT
[75038.358913] ata13.01: cmd 25/00:00:48:c5:be/00:04:01:00:00/f0 tag 0 dma 524288 in
[75038.358913]          res 51/40:38:10:c9:be/40:00:01:00:00/f0 Emask 0x9 (media error)
[75038.359001] ata13.01: status: { DRDY ERR }
[75038.359026] ata13.01: error: { UNC }
[75038.588211] ata13.00: configured for UDMA/133
[75038.596261] ata13.01: configured for UDMA/133
[75038.596603] sd 13:0:1:0: [sds] Unhandled sense code
[75038.596631] sd 13:0:1:0: [sds] 
[75038.596655] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[75038.596682] sd 13:0:1:0: [sds] 
[75038.596705] Sense Key : Medium Error [current] [descriptor]
[75038.596739] Descriptor sense data with sense descriptors (in hex):
[75038.596769]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[75038.596833]         01 be c9 10 
[75038.596867] sd 13:0:1:0: [sds] 
[75038.596891] Add. Sense: Unrecovered read error - auto reallocate failed
[75038.596924] sd 13:0:1:0: [sds] CDB: 
[75038.596948] Read(10): 28 00 01 be c5 48 00 04 00 00
[75038.596998] end_request: I/O error, dev sds, sector 29280528
[75038.597036] ata13: EH complete
[75043.039158] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[75043.039193] ata13.01: failed command: READ DMA
[75043.039224] ata13.01: cmd c8/00:08:10:c9:be/00:00:00:00:00/f1 tag 0 dma 4096 in
[75043.039224]          res 51/40:08:10:c9:be/40:00:01:00:00/f1 Emask 0x9 (media error)
[75043.039312] ata13.01: status: { DRDY ERR }
[75043.039337] ata13.01: error: { UNC }
[75043.268249] ata13.00: configured for UDMA/133
[75043.276261] ata13.01: configured for UDMA/133
[75043.276309] sd 13:0:1:0: [sds] Unhandled sense code
[75043.276335] sd 13:0:1:0: [sds] 
[75043.276359] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[75043.276387] sd 13:0:1:0: [sds] 
[75043.276409] Sense Key : Medium Error [current] [descriptor]
[75043.276442] Descriptor sense data with sense descriptors (in hex):
[75043.276471]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[75043.276532]         01 be c9 10 
[75043.276565] sd 13:0:1:0: [sds] 
[75043.277990] Add. Sense: Unrecovered read error - auto reallocate failed
[75043.278021] sd 13:0:1:0: [sds] CDB: 
[75043.278045] Read(10): 28 00 01 be c9 10 00 00 08 00
[75043.278096] end_request: I/O error, dev sds, sector 29280528
[75043.278124] end_request: I/O error, dev sds, sector 29280528
[75043.278160] ata13: EH complete
[75043.464460] md/raid:md0: read error corrected (8 sectors at 29280528 on sds)
[75055.326046] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[75055.326082] ata13.01: failed command: READ DMA EXT
[75055.326114] ata13.01: cmd 25/00:00:58:11:cf/00:04:01:00:00/f0 tag 0 dma 524288 in
[75055.326114]          res 51/40:e0:78:14:cf/40:00:01:00:00/f0 Emask 0x9 (media error)
[75055.326202] ata13.01: status: { DRDY ERR }
[75055.326228] ata13.01: error: { UNC }
[75055.556225] ata13.00: configured for UDMA/133
[75055.564263] ata13.01: configured for UDMA/133
[75055.564635] sd 13:0:1:0: [sds] Unhandled sense code
[75055.564663] sd 13:0:1:0: [sds] 
[75055.564688] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[75055.564715] sd 13:0:1:0: [sds] 
[75055.564738] Sense Key : Medium Error [current] [descriptor]
[75055.564771] Descriptor sense data with sense descriptors (in hex):
[75055.564802]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[75055.564867]         01 cf 14 78 
[75055.564903] sd 13:0:1:0: [sds] 
[75055.564927] Add. Sense: Unrecovered read error - auto reallocate failed
[75055.564958] sd 13:0:1:0: [sds] CDB: 
[75055.564982] Read(10): 28 00 01 cf 11 58 00 04 00 00
[75055.565033] end_request: I/O error, dev sds, sector 30348408
[75055.565082] ata13: EH complete
[75060.601853] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[75060.601887] ata13.01: failed command: READ DMA
[75060.601918] ata13.01: cmd c8/00:08:78:14:cf/00:00:00:00:00/f1 tag 0 dma 4096 in
[75060.601918]          res 51/40:08:78:14:cf/40:00:01:00:00/f1 Emask 0x9 (media error)
[75060.602005] ata13.01: status: { DRDY ERR }
[75060.602032] ata13.01: error: { UNC }
[75060.832227] ata13.00: configured for UDMA/133
[75060.840279] ata13.01: configured for UDMA/133
[75060.840315] sd 13:0:1:0: [sds] Unhandled sense code
[75060.840342] sd 13:0:1:0: [sds] 
[75060.840365] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[75060.840393] sd 13:0:1:0: [sds] 
[75060.840415] Sense Key : Medium Error [current] [descriptor]
[75060.840449] Descriptor sense data with sense descriptors (in hex):
[75060.840478]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[75060.840542]         01 cf 14 78 
[75060.840575] sd 13:0:1:0: [sds] 
[75060.840598] Add. Sense: Unrecovered read error - auto reallocate failed
[75060.840629] sd 13:0:1:0: [sds] CDB: 
[75060.840652] Read(10): 28 00 01 cf 14 78 00 00 08 00
[75060.840703] end_request: I/O error, dev sds, sector 30348408
[75060.840731] end_request: I/O error, dev sds, sector 30348408
[75060.840768] ata13: EH complete
[75061.051075] md/raid:md0: read error corrected (8 sectors at 30348408 on sds)
[75067.557510] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[75067.557547] ata13.01: failed command: READ DMA EXT
[75067.557579] ata13.01: cmd 25/00:00:60:f1:d4/00:04:01:00:00/f0 tag 0 dma 524288 in
[75067.557579]          res 51/40:50:10:f4:d4/40:01:01:00:00/f0 Emask 0x9 (media error)
[75067.557666] ata13.01: status: { DRDY ERR }
[75067.557692] ata13.01: error: { UNC }
[75067.788249] ata13.00: configured for UDMA/133
[75067.796276] ata13.01: configured for UDMA/133
[75067.796591] sd 13:0:1:0: [sds] Unhandled sense code
[75067.796619] sd 13:0:1:0: [sds] 
[75067.796642] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[75067.796671] sd 13:0:1:0: [sds] 
[75067.796694] Sense Key : Medium Error [current] [descriptor]
[75067.796727] Descriptor sense data with sense descriptors (in hex):
[75067.796757]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[75067.796825]         01 d4 f4 10 
[75067.796858] sd 13:0:1:0: [sds] 
[75067.796882] Add. Sense: Unrecovered read error - auto reallocate failed
[75067.796913] sd 13:0:1:0: [sds] CDB: 
[75067.796937] Read(10): 28 00 01 d4 f1 60 00 04 00 00
[75067.796988] end_request: I/O error, dev sds, sector 30733328
[75067.797047] ata13: EH complete
[113272.067198] md: md0: recovery done.
[113272.528813] RAID conf printout:
[113272.528818]  --- level:6 rd:10 wd:10
[113272.528821]  disk 0, o:1, dev:sdn
[113272.528824]  disk 1, o:1, dev:sdh
[113272.528827]  disk 2, o:1, dev:sdj
[113272.528829]  disk 3, o:1, dev:sdd
[113272.528831]  disk 4, o:0, dev:sds
[113272.528834]  disk 5, o:1, dev:sdv
[113272.528836]  disk 6, o:1, dev:sdb
[113272.528839]  disk 7, o:1, dev:sdc
[113272.528841]  disk 8, o:1, dev:sdf
[113272.528844]  disk 9, o:1, dev:sde
[113272.661106] RAID conf printout:
[113272.661111]  --- level:6 rd:10 wd:10
[113272.661113]  disk 0, o:1, dev:sdn
[113272.661114]  disk 1, o:1, dev:sdh
[113272.661116]  disk 2, o:1, dev:sdj
[113272.661118]  disk 3, o:1, dev:sdd
[113272.661119]  disk 4, o:0, dev:sds
[113272.661121]  disk 5, o:1, dev:sdv
[113272.661123]  disk 6, o:1, dev:sdb
[113272.661124]  disk 7, o:1, dev:sdc
[113272.661126]  disk 8, o:1, dev:sdf
[113272.661127]  disk 9, o:1, dev:sde
[113272.668116] RAID conf printout:
[113272.668120]  --- level:6 rd:10 wd:10
[113272.668123]  disk 0, o:1, dev:sdn
[113272.668126]  disk 1, o:1, dev:sdh
[113272.668129]  disk 2, o:1, dev:sdj
[113272.668132]  disk 3, o:1, dev:sdd
[113272.668134]  disk 4, o:1, dev:sdk
[113272.668137]  disk 5, o:1, dev:sdv
[113272.668139]  disk 6, o:1, dev:sdb
[113272.668142]  disk 7, o:1, dev:sdc
[113272.668145]  disk 8, o:1, dev:sdf
[113272.668147]  disk 9, o:1, dev:sde
[115328.025801] md: unbind<sds>
[115328.036030] md: export_rdev(sds)

Here I started a badblocks test on sds by means of writing 0x00 to the 
entire drive, this should be less relevant though:

[171479.512259] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[171479.512305] ata13.01: failed command: READ DMA
[171479.512335] ata13.01: cmd c8/00:80:80:01:00/00:00:00:00:00/f0 tag 0 dma 65536 in
[171479.512335]          res 51/40:10:f0:01:00/40:01:01:00:00/f0 Emask 0x9 (media error)
[171479.512423] ata13.01: status: { DRDY ERR }
[171479.512448] ata13.01: error: { UNC }
[171479.800230] ata13.00: configured for UDMA/133
[171479.808259] ata13.01: configured for UDMA/133
[171479.808340] sd 13:0:1:0: [sds] Unhandled sense code
[171479.808367] sd 13:0:1:0: [sds] 
[171479.808391] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[171479.808418] sd 13:0:1:0: [sds] 
[171479.808441] Sense Key : Medium Error [current] [descriptor]
[171479.808475] Descriptor sense data with sense descriptors (in hex):
[171479.808504]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[171479.808566]         00 00 01 f0 
[171479.808599] sd 13:0:1:0: [sds] 
[171479.808623] Add. Sense: Unrecovered read error - auto reallocate failed
[171479.808653] sd 13:0:1:0: [sds] CDB: 
[171479.808676] Read(10): 28 00 00 00 01 80 00 00 80 00
[171479.808727] end_request: I/O error, dev sds, sector 496
[171479.808761] ata13: EH complete
[171484.954258] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[171484.954292] ata13.01: failed command: READ DMA
[171484.954324] ata13.01: cmd c8/00:80:e8:01:00/00:00:00:00:00/f0 tag 0 dma 65536 in
[171484.954324]          res 51/40:78:f0:01:00/40:01:01:00:00/f0 Emask 0x9 (media error)
[171484.954413] ata13.01: status: { DRDY ERR }
[171484.954438] ata13.01: error: { UNC }
[171485.184249] ata13.00: configured for UDMA/133
[171485.192259] ata13.01: configured for UDMA/133
[171485.192339] sd 13:0:1:0: [sds] Unhandled sense code
[171485.192367] sd 13:0:1:0: [sds] 
[171485.192390] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[171485.192418] sd 13:0:1:0: [sds] 
[171485.192441] Sense Key : Medium Error [current] [descriptor]
[171485.192475] Descriptor sense data with sense descriptors (in hex):
[171485.192504]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[171485.192566]         00 00 01 f0 
[171485.192599] sd 13:0:1:0: [sds] 
[171485.192623] Add. Sense: Unrecovered read error - auto reallocate failed
[171485.192654] sd 13:0:1:0: [sds] CDB: 
[171485.192677] Read(10): 28 00 00 00 01 e8 00 00 80 00
[171485.192727] end_request: I/O error, dev sds, sector 496
[171485.192755] Buffer I/O error on device sds, logical block 62
[171485.192784] Buffer I/O error on device sds, logical block 63
[171485.192812] Buffer I/O error on device sds, logical block 64
[171485.192840] Buffer I/O error on device sds, logical block 65
[171485.192867] Buffer I/O error on device sds, logical block 66
[171485.192895] Buffer I/O error on device sds, logical block 67
[171485.192923] Buffer I/O error on device sds, logical block 68
[171485.192950] Buffer I/O error on device sds, logical block 69
[171485.192977] Buffer I/O error on device sds, logical block 70
[171485.193005] Buffer I/O error on device sds, logical block 71
[171485.193039] ata13: EH complete
[171496.884843] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[171496.884879] ata13.01: failed command: READ DMA
[171496.884910] ata13.01: cmd c8/00:08:f0:01:00/00:00:00:00:00/f0 tag 0 dma 4096 in
[171496.884910]          res 51/40:08:f0:01:00/40:01:01:00:00/f0 Emask 0x9 (media error)
[171496.885000] ata13.01: status: { DRDY ERR }
[171496.885026] ata13.01: error: { UNC }
[171497.188226] ata13.00: configured for UDMA/133
[171497.196279] ata13.01: configured for UDMA/133
[171497.196317] sd 13:0:1:0: [sds] Unhandled sense code
[171497.196344] sd 13:0:1:0: [sds] 
[171497.196367] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[171497.196394] sd 13:0:1:0: [sds] 
[171497.196417] Sense Key : Medium Error [current] [descriptor]
[171497.196450] Descriptor sense data with sense descriptors (in hex):
[171497.196480]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[171497.196541]         00 00 01 f0 
[171497.196574] sd 13:0:1:0: [sds] 
[171497.196598] Add. Sense: Unrecovered read error - auto reallocate failed
[171497.196629] sd 13:0:1:0: [sds] CDB: 
[171497.196652] Read(10): 28 00 00 00 01 f0 00 00 08 00
[171497.196704] end_request: I/O error, dev sds, sector 496
[171497.196732] quiet_error: 5 callbacks suppressed
[171497.196757] Buffer I/O error on device sds, logical block 62
[171497.196794] ata13: EH complete
[171502.605051] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[171502.605085] ata13.01: failed command: READ DMA
[171502.605114] ata13.01: cmd c8/00:08:f0:01:00/00:00:00:00:00/f0 tag 0 dma 4096 in
[171502.605114]          res 51/40:08:f0:01:00/40:01:01:00:00/f0 Emask 0x9 (media error)
[171502.605202] ata13.01: status: { DRDY ERR }
[171502.605227] ata13.01: error: { UNC }
[171502.868224] ata13.00: configured for UDMA/133
[171502.876225] ata13.01: configured for UDMA/133
[171502.876268] sd 13:0:1:0: [sds] Unhandled sense code
[171502.876295] sd 13:0:1:0: [sds] 
[171502.877696] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[171502.877724] sd 13:0:1:0: [sds] 
[171502.877747] Sense Key : Medium Error [current] [descriptor]
[171502.877782] Descriptor sense data with sense descriptors (in hex):
[171502.877811]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[171502.877874]         00 00 01 f0 
[171502.877908] sd 13:0:1:0: [sds] 
[171502.877932] Add. Sense: Unrecovered read error - auto reallocate failed
[171502.877963] sd 13:0:1:0: [sds] CDB: 
[171502.877987] Read(10): 28 00 00 00 01 f0 00 00 08 00
[171502.878038] end_request: I/O error, dev sds, sector 496
[171502.878066] Buffer I/O error on device sds, logical block 62
[171502.878104] ata13: EH complete
[190261.434197] ata13.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[190261.434233] ata13.01: failed command: READ DMA
[190261.434264] ata13.01: cmd c8/00:08:f0:01:00/00:00:00:00:00/f0 tag 0 dma 4096 in
[190261.434264]          res 51/40:08:f0:01:00/40:01:01:00:00/f0 Emask 0x9 (media error)
[190261.434353] ata13.01: status: { DRDY ERR }
[190261.434378] ata13.01: error: { UNC }
[190261.688229] ata13.00: configured for UDMA/133
[190261.696263] ata13.01: configured for UDMA/133
[190261.696310] sd 13:0:1:0: [sds] Unhandled sense code
[190261.696337] sd 13:0:1:0: [sds] 
[190261.696360] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[190261.696388] sd 13:0:1:0: [sds] 
[190261.696411] Sense Key : Medium Error [current] [descriptor]
[190261.696443] Descriptor sense data with sense descriptors (in hex):
[190261.696473]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[190261.696536]         00 00 01 f0 
[190261.696569] sd 13:0:1:0: [sds] 
[190261.696592] Add. Sense: Unrecovered read error - auto reallocate failed
[190261.696623] sd 13:0:1:0: [sds] CDB: 
[190261.696647] Read(10): 28 00 00 00 01 f0 00 00 08 00
[190261.696698] end_request: I/O error, dev sds, sector 496
[190261.696726] Buffer I/O error on device sds, logical block 62
[190261.696759] ata13: EH complete

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se