Re: RAID6 data-check took almost 2 hours, clicking sounds, system unresponsive

[NeilBrown <neilb@suse.de>]

On Fri, 8 Apr 2011 02:59:52 -0700 (PDT) Gavin Flower <gavinflower@yahoo.com>
wrote:

> 
> --- On Fri, 8/4/11, NeilBrown <neilb@suse.de> wrote:
> 
> > From: NeilBrown <neilb@suse.de>
> > Subject: Re: RAID6 data-check took almost 2 hours, clicking sounds, system unresponsive
> > To: "Gavin Flower" <gavinflower@yahoo.com>
> > Cc: linux-raid@vger.kernel.org
> > Date: Friday, 8 April, 2011, 21:34
> > On Thu, 7 Apr 2011 18:32:04 -0700
> > (PDT) Gavin Flower <gavinflower@yahoo.com>
> > wrote:
> > 
> > > Hi Neil,
> > > 
> > > My original email may have been eaten: as it did not
> > appear on the list, nor did I get an error message
> > back.  So perhaps there was a problem with the attached
> > files.
> > > 
> > > I will resend the attachments one at a time in
> > separate emails.
> > > 
> > > 
> > > Cheers,
> > > Gavin
> > > 
> > > [begin original]
> > > Hi Neil,
> > > 
> > > Your help (or anybody else's) would be greatly
> > appreciated, yet again
> > 
> > Hi Gavin,
> >  it isn't clear to me what help you want.
> > 
> > Obviously there is some sort of hardware issue - possible a
> > drive, possibly a
> > bus problem - I really don't know.
> > 
> > Apart from that things look normal.
> > 
> > What exactly did you want explained?
> > 
> > NeilBrown
> 
> I guess I was surprised that the RAID system appeared normal and that it did not register any errors.  I was hoping to get an idea as to which drive was problematic.

sdc2 was reporting read error.  md/raid6 computed the data from the other
devices and wrote it back to sdc2.  This appeared to work so md/raid6 assumed
everything was fine again.  It reported this:

Apr  7 08:42:08 saturn kernel: [210414.109880] md/raid:md1: read error corrected (8 sectors at 17195840 on sdc2) 

but didn't fail anything.


> 
> I get the feeling, from your reply, that this is not specifically a RAID problem, that it just happens to affect a RAID array.

No, it was clearly a disk-drive problem.
e.g.
Apr  7 14:42:12 saturn kernel: [231957.756023] ata3.00: failed command: READ FPDMA QUEUED

a READ command sent to a n 'ata' device failed.  i.e. disk error.

> 
> I had thought that the RAID system should have been able to give me better diagnostics, but possibly I am being (inadvertently) unreasonable!

Well.... it did tell you that it got a read error and corrected it.


> 
> Not sure what the significance of this mismatch is, and what I should do about it.
> # cat /sys/block/md2/md/mismatch_cnt 
> 28904 
> # 

I'm not sure if read errors end up counting as mismatches..  They seem to for
raid1.  The raid6 code is more complex and I don't feel like decoding it
right now.

In terms of "what to do about it" - the first thing must be to fix sdc.
Maybe there is a loose cable or a broken cable.  Maybe the device needs to be
replaced.

Once you have resolved that and are fairly sure yours drives are all working,
    echo check > /sys/block/md2/md/sync_action

once that finishes mismatch_cnt should ideally be zero.  If it isn't, try
    echo repair > /sys/block/md2/md/sync_action

but only do that if you are confident that your devices are good.
This will result in the same mismatch_cnt.  However a subsequent 'check'
should then show zero.

NeilBrown



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html