Re: problem with synology external raid

[Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>]

On Fri, 23 Aug 2024 17:54:58 +0200
Miroslav Šulc <miroslav.sulc@fordfrog.com> wrote:

> hi Mariusz, thanks for your reply. please find my comments below.
> 
> Dne 2024-08-23 13:56, Mariusz Tkaczyk napsal:
> > On Thu, 22 Aug 2024 17:02:52 +0200
> > Miroslav Šulc <miroslav.sulc@fordfrog.com> wrote:
> >   
> >> hello,
> >> 
> >> a broken synology raid5 ended up on my desk. the raid was broken for
> >> quite some time, but i got it from the client just a few days back.
> >> 
> >> the raid consisted of 5 disks (no spares, all used):
> >> disk 1 (sdca): according to my understanding, it was removed from the
> >> raid, then re-added, but the synchronization was interrupted, so it
> >> cannot be used to restore the raid  
> > 
> > Hi Miroslav,
> > (If I send something earlier - sorry missclick)
> > 
> >    Array State : AAAAA ('A' == active, '.' == missing, 'R' == 
> > replacing)
> >    Events : 60223
> > 
> > There is no info about interrupted rebuild in the metadata. This drive
> > looks like removed from array, it has old Events number. If it was
> > rebuilt, it finished correctly.  
> 
> there is this line in the metadata from which i supposed the rebuild 
> didn't finish:
> 
> Recovery Offset : 4910216 sectors
> 
> i also tried to recreate the raid from disks 1, 2, 4, and 5, using 
> --assume-clean. the raid was set up, i was able to read lvm from the 
> raid, but when i checked the btrfs filesystem on it or tried to mount 
> it, i faced a serious corruption of the filesystem. btrfs recovery 
> recovered nothing.

disk 1 is outdated so I'm not surprised here. You have better targets
(2-5). See below. "Events" is a counter of metadata updates that happened to the
array. metadata is updated for dirty clean transition or reconfiguration.
Simply bigger difference means that the device was detached earlier so the
data on this removed drive is older and less reliable for filesystem to
recover.

> 
> > The metadata on the drive is frozen on the last state of the device in 
> > array,
> > other members were updated that this device is gone.
> >   
> >> disk 2 (sdcb): is ok and up to date
> >> disk 3 (sdcc): seems to be up to date, still spinning, but there are
> >> many issues with bad sectors  
> > 
> >            Events : 60283
> > 
> >            Layout : left-symmetric
> >        Chunk Size : 64K
> > 
> >      Device Role : Active device 2
> >      Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> > replacing)
> > 
> > It is outdated, Events number is smaller than on "ok" members. Sadly, 
> > probably
> > mdadm will refuse to start this array because you have 2 outdated 
> > drives.  
> 
> i was able to start the array from disks 2-4 in degraded mode. i even 
> tried to add disk 1 to the array and rebuild it, but that failed. my 
> guess is it was because of disk 3.

You don't need to recover this array. Try read data back, you can copy data to
other media if you have any.
Disk 3 looks like damaged so I cannot guarantee it all will be successful.

Recovery process copies the data from existing disks to rebuilt disk, if
original data is not reliable, recovered one won't be reliable too. You don't
need to make it worse.

Just assemble what you can and copy what you can. Degraded state means that you
can still access all data.

Thanks,
Mariusz

> 
> > 
> > On "ok" disks, it is:
> >  Events : 60286
> > Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> > 
> > Second failure is not recorded in metadata and I don't know why events 
> > number is
> > higher. I would expect kernel to stop updating metadata after device 
> > failure.
> > 
> > I will stop here and give a chance more native experienced users to 
> > elaborate.
> > 
> > Looks like raid recreation with --assume-clean is a simplest solution 
> > but it
> > is general advice I can give you (and I propose it too often). You have 
> > copies,
> > you did right first step!
> > 
> > Mariusz
> >   
> >> disk 4 (sdcd): is ok and up to date
> >> disk 5 (sdce): is ok and up to date
> >> 
> >> the raid ran in degraded mode for over two months, client trying to 
> >> make
> >> a copy of the data from it.
> >> 
> >> i made copies of the disk images from disks 1, 2, 4, and 5, at the 
> >> state
> >> shown below. i didn't attempt to make a copy of disk 3 so far. since
> >> then i re-assembled the raid from disk 2-5 so the number of events on
> >> the disk 3 is now a bit higher then on the copies of the disk images.
> >> 
> >> according to my understanding, as the disk 1 never finished the sync, 
> >> i
> >> cannot use it to recover the data, so the only way to recover the data
> >> is to assemble the raid in degraded mode using disk 2-5, if i ever
> >> succeed to make a copy of the disk 3. i'd just like to verify that my
> >> understanding is correct and there is no other way to attempt to 
> >> recover
> >> the data. of course any hints are welcome.
> >> 
> >> here is the info from the partitions of the raid:
> >> 
> >> /dev/sdca5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x2
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >> Recovery Offset : 4910216 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 681c6c33:49df0163:bb4271d4:26c0c76d
> >> 
> >>      Update Time : Tue Jun  4 18:35:54 2024
> >>         Checksum : cf45a6c1 - correct
> >>           Events : 60223
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 0
> >>     Array State : AAAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> /dev/sdcb5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x0
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 0f23d7cd:b93301a9:5289553e:286ab6f0
> >> 
> >>      Update Time : Wed Aug 14 15:09:24 2024
> >>         Checksum : 9c93703e - correct
> >>           Events : 60286
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 1
> >>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> /dev/sdcc5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x0
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 1d1c04b4:24dabd8d:235afb7d:1494b8eb
> >> 
> >>      Update Time : Wed Aug 14 12:42:26 2024
> >>         Checksum : a224ec08 - correct
> >>           Events : 60283
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 2
> >>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> /dev/sdcd5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x0
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 76698d3f:e9c5a397:05ef7553:9fd0af16
> >> 
> >>      Update Time : Wed Aug 14 15:09:24 2024
> >>         Checksum : 38061500 - correct
> >>           Events : 60286
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 4
> >>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> /dev/sdce5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x0
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 9c7077f8:3120195a:1af11955:6bcebd99
> >> 
> >>      Update Time : Wed Aug 14 15:09:24 2024
> >>         Checksum : 38177651 - correct
> >>           Events : 60286
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 3
> >>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> 
> >> 
> >> thank you for your help.
> >> 
> >> miroslav
> >>   
>