Re: Determining cause of md RAID 'recovery interrupted'

[Roy Sigurd Karlsbakk <roy@karlsbakk.net>]

hi

Could you check for Current_Pending_Sector and Reallocated_Sector_Ct for the drives in the array? You'll find this with smartctl -a /dev/sdX. These should be zero, but a few errors won't sink the ship. Also, check if there is a populated badblocks list on either of the drives. I've written a bit about these here https://wiki.karlsbakk.net/index.php?title=Roy%27s_notes#The_badblock_list. There's also https://raid.wiki.kernel.org/index.php/The_Badblocks_controversy for more info.

Vennlig hilsen

roy
-- 
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.

----- Original Message -----
> From: esqychd2f5@liamekaens.com
> To: "Linux Raid" <linux-raid@vger.kernel.org>
> Sent: Saturday, 16 July, 2022 23:17:25
> Subject: Determining cause of md RAID 'recovery interrupted'

> Hi,
> 
> I'm a long-time md raid user, and a big fan of the project.  I have run into an
> issue that I haven't been able to track down a solution to online.
> 
> I have an md raid array using 12TB Seagate Iron Wolf NAS drives in a RAID6
> configuration.  This array grew from 4 drives to 10 drives over several years,
> and after the restripe to 10 drives it started occasionally dropping drives
> without obvious errors (no read or write issues).
> 
> The server is running Ubuntu 20.04.4 LTS (fully updated) and the drives are
> connected using LSI SAS 9207-8i adapters.
> 
> The dropping of drives has led to the array now being in a degraded state, and I
> can't get it to rebuild.  It fails with a 'recovery interrupted' message. It
> did rebuild successfully a few times, but now fails consistently at the same
> point around 12% done.
> 
> I have confirmed that I can read all data from all of my drives using the
> 'badblocks' tool to read all data from all drives.  No read errors are
> reported.
> 
> The rebuild start up to failure looks like this in the system log:
> [  715.210403] md: md3 stopped.
> [  715.447441] md/raid:md3: device sdd operational as raid disk 1
> [  715.447443] md/raid:md3: device sdp operational as raid disk 9
> [  715.447444] md/raid:md3: device sdc operational as raid disk 7
> [  715.447445] md/raid:md3: device sdb operational as raid disk 6
> [  715.447446] md/raid:md3: device sdm operational as raid disk 5
> [  715.447447] md/raid:md3: device sdn operational as raid disk 4
> [  715.447448] md/raid:md3: device sdq operational as raid disk 3
> [  715.447449] md/raid:md3: device sdo operational as raid disk 2
> [  715.451780] md/raid:md3: raid level 6 active with 8 out of 10 devices,
> algorithm 2
> [  715.451839] md3: detected capacity change from 0 to 96000035258368
> [  715.452035] md: recovery of RAID array md3
> [  715.674492]  md3: p1
> [ 9803.487218] md: md3: recovery interrupted.
> 
> I have the technical data about the drive, but it is very large (181K) so I'll
> post it as a response to this post to minimize clutter.
> There are a few md RAID arrays shown in the logs, the one with the problem is
> 'md3'.
> 
> Initially, I'd like to figure out why the rebuild gets interrupted (later I will
> look into why drives are being dropped).  I would expect an error message
> explaining the interruption, but I haven't been able to find it.  Maybe it is
> in an unexpected system log file?
> 
> One thing I notice is that one of my drives (/dev/sdc) has 'Bad Blocks Present':
>  Bad Block Log : 512 entries available at offset 264 sectors - bad blocks
>  present.
> 
> So, a few questions:
> 
> - Would the 'Bad Blocks Present' for sdc lead to 'recovery interrupted'?
> - More generally, how do I find out what has interrupted the rebuild?
> 
> Thanks in advance for your help!
> 
> Joe