Re: Fedora 20 RAID 6 errors on rebuild / check / repair

[Kay Diederichs <kay.diederichs@uni-konstanz.de>]

On 07/24/2014 04:29 AM, George Rapp wrote:
> Hi -
> 
> I have a Fedora 20 media server / MythTV backend utilizing a HighPoint
> RocketRAID 2720SGL controller (Amazon product link:
> http://is.gd/yqo2i1). The server performs fine under normal (minimal)
> read-write operations, but during any high-I/O operations (rebuild
> after mdadm --add, RAID check initiated by "echo check >
> /sys/block/md6/md/sync_action" or "echo repair > ..."), I get sporadic
> errors and poor performance on my RAID 6 array, /dev/md6.
> 
> Wondering if there is anything I can tweak to make my configuration
> more stable. The inability to check or repair this RAID device has me
> nervous.
> 
> The problems seem to start when I see the following error message in
> /var/log/syslog:
> 
>> Jul 22 21:23:37 backend3 kernel: [95876.375990] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
>> Jul 22 21:23:37 backend3 kernel: [95876.376153] ata5.00: failed command: READ DMA
>> Jul 22 21:23:37 backend3 kernel: [95876.376284] ata5.00: cmd c8/00:08:40:11:81/00:00:00:00:00/e3 tag 11 dma 4096 in
>> Jul 22 21:23:37 backend3 kernel: [95876.376284]          res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
>> Jul 22 21:23:37 backend3 kernel: [95876.376750] ata5.00: status: { DRDY }
>> Jul 22 21:23:37 backend3 kernel: [95876.376874] ata5: hard resetting link
>> Jul 22 21:23:37 backend3 kernel: ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
>> Jul 22 21:23:37 backend3 kernel: ata5.00: failed command: READ DMA
>> Jul 22 21:23:37 backend3 kernel: ata5.00: cmd c8/00:08:40:11:81/00:00:00:00:00/e3 tag 11 dma 4096 in
>>          res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
>> Jul 22 21:23:37 backend3 kernel: ata5.00: status: { DRDY }
>> Jul 22 21:23:37 backend3 kernel: ata5: hard resetting link
>> Jul 22 21:23:40 backend3 kernel: [95878.742281] ata5.00: configured for UDMA/133
>> Jul 22 21:23:40 backend3 kernel: [95878.742413] ata5.00: device reported invalid CHS sector 0
>> Jul 22 21:23:40 backend3 kernel: [95878.742542] ata5: EH complete
>> Jul 22 21:23:40 backend3 kernel: ata5.00: configured for UDMA/133
>> Jul 22 21:23:40 backend3 kernel: ata5.00: device reported invalid CHS sector 0
>> Jul 22 21:23:40 backend3 kernel: ata5: EH complete
> 
> 
> I thought the problem might be caused by NCQ being enabled -- previous
> iterations of this error included the string 'ncq', like this:
> 
>> ata7.00: cmd 60/00:00:68:4b:75/03:00:04:00:00/40 tag 0 ncq 393216 in
>>          res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
> 
> 
> so I disabled NCQ by adding "libata.force=noncq" to my kernel boot
> parameters. However, it didn't help, as I still get the "...frozen"
> errors. (I have young children, so any error message that includes the
> word "Frozen" makes me twitchy ... 8^)
> 
> Right now, I'm attempting to rebuild the degraded RAID 6 array after
> swapping out a disk that was getting an increasing number of these
> errors:
> 
>> Device: /dev/sdf [SAT], 35 Currently unreadable (pending) sectors
> 
> 
> I started the rebuild on Monday night in single-user mode via
> 
>> # mdadm --manage /dev/md6 --add /dev/sdf1
> 
> 
> (my other partitions are /dev/sd[bcde]4, but I only created a single
> partition on the new disk, to see if reliability would be better by
> placing my RAID partition on the first partition rather than the last)
> 
> At first, the rebuild was supposed to take 4.3 days. I Googled around
> and found a couple of speed optimization techniques, which I applied:
> 
>> # sysctl -w dev.raid.speed_limit_max=100000
>>
>> # cd /sys/block/md6/md
>> # echo 16384 > stripe_cache_size
> 
> 
> This initially sped up the resync speed to 68-70000K/sec, until I hit
> the first "exception Emask" error like the one I described above --
> now the speed has dropped to 30K/sec, and the rebuild is scheduled to
> last 439 more days! I don't know if I should just mark the new device
> as failed and stop the sync, or let it keep grinding and hope it
> speeds up.
> 
> Any pointers or tips appreciated. I've been running Linux software
> RAID for 4-5 years, but this is the first time I've experienced this
> kind of trouble.
> 
> More data on my system:
> 
> 
> [root@backend3 gwr]# uname -a
> Linux backend3 3.14.4-200.fc20.i686+PAE #1 SMP Tue May 13 14:03:12 UTC
> 2014 i686 i686 i386 GNU/Linux
> 
> [root@backend3 gwr]# mdadm --version
> mdadm - v3.3 - 3rd September 2013
> 
> [root@backend3 log]# mdadm --detail /dev/md6
> /dev/md6:
>         Version : 1.2
>   Creation Time : Sun Apr 24 17:31:27 2011
>      Raid Level : raid6
>      Array Size : 5756723712 (5490.04 GiB 5894.89 GB)
>   Used Dev Size : 1918907904 (1830.01 GiB 1964.96 GB)
>    Raid Devices : 5
>   Total Devices : 5
>     Persistence : Superblock is persistent
> 
>   Intent Bitmap : Internal
> 
>     Update Time : Wed Jul 23 22:15:05 2014
>           State : active, degraded, recovering
>  Active Devices : 4
> Working Devices : 5
>  Failed Devices : 0
>   Spare Devices : 1
> 
>          Layout : left-symmetric
>      Chunk Size : 512K
> 
>  Rebuild Status : 39% complete
> 
>            Name : backend3:md4
>            UUID : 894bc20e:b9479ac9:7bfce54f:0ac12dd9
>          Events : 1659380
> 
>     Number   Major   Minor   RaidDevice State
>        0       8       36        0      active sync   /dev/sdc4
>        1       8       20        1      active sync   /dev/sdb4
>        5       8       68        2      active sync   /dev/sde4
>        4       8       52        3      active sync   /dev/sdd4
>        6       8       81        4      spare rebuilding   /dev/sdf1
> 

George,

this is not necessarily a RAID problem. Can you exclude the possibility
that one or more of the disks have a hardware problem, like the one you
replaced which showed
>> Device: /dev/sdf [SAT], 35 Currently unreadable (pending) sectors
Hardware problems would explain the problems you have.

What does smartctl report about your disks, in particular:
Offline_Uncorrectable
Current_Pending_Sector
Reallocated_Sector_Ct

And, is it always ATA 5.00 that is mentioned in syslog? dmesg and the
"lsdrv" script (google for it) are useful in diagnosing this.

HTH,
Kay