problem with synology external raid

linux-raid.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* problem with synology external raid
@ 2024-08-22 15:02 Miroslav Šulc
  2024-08-23 11:56 ` Mariusz Tkaczyk
                   ` (2 more replies)
  0 siblings, 3 replies; 6+ messages in thread
From: Miroslav Šulc @ 2024-08-22 15:02 UTC (permalink / raw)
  To: linux-raid

hello,

a broken synology raid5 ended up on my desk. the raid was broken for 
quite some time, but i got it from the client just a few days back.

the raid consisted of 5 disks (no spares, all used):
disk 1 (sdca): according to my understanding, it was removed from the 
raid, then re-added, but the synchronization was interrupted, so it 
cannot be used to restore the raid
disk 2 (sdcb): is ok and up to date
disk 3 (sdcc): seems to be up to date, still spinning, but there are 
many issues with bad sectors
disk 4 (sdcd): is ok and up to date
disk 5 (sdce): is ok and up to date

the raid ran in degraded mode for over two months, client trying to make 
a copy of the data from it.

i made copies of the disk images from disks 1, 2, 4, and 5, at the state 
shown below. i didn't attempt to make a copy of disk 3 so far. since 
then i re-assembled the raid from disk 2-5 so the number of events on 
the disk 3 is now a bit higher then on the copies of the disk images.

according to my understanding, as the disk 1 never finished the sync, i 
cannot use it to recover the data, so the only way to recover the data 
is to assemble the raid in degraded mode using disk 2-5, if i ever 
succeed to make a copy of the disk 3. i'd just like to verify that my 
understanding is correct and there is no other way to attempt to recover 
the data. of course any hints are welcome.

here is the info from the partitions of the raid:

/dev/sdca5:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x2
      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
            Name : DS_KLIENT:4
   Creation Time : Tue Mar 31 11:40:19 2020
      Raid Level : raid5
    Raid Devices : 5

  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
     Data Offset : 2048 sectors
    Super Offset : 8 sectors
Recovery Offset : 4910216 sectors
    Unused Space : before=1968 sectors, after=0 sectors
           State : clean
     Device UUID : 681c6c33:49df0163:bb4271d4:26c0c76d

     Update Time : Tue Jun  4 18:35:54 2024
        Checksum : cf45a6c1 - correct
          Events : 60223

          Layout : left-symmetric
      Chunk Size : 64K

    Device Role : Active device 0
    Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdcb5:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
            Name : DS_KLIENT:4
   Creation Time : Tue Mar 31 11:40:19 2020
      Raid Level : raid5
    Raid Devices : 5

  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
     Data Offset : 2048 sectors
    Super Offset : 8 sectors
    Unused Space : before=1968 sectors, after=0 sectors
           State : clean
     Device UUID : 0f23d7cd:b93301a9:5289553e:286ab6f0

     Update Time : Wed Aug 14 15:09:24 2024
        Checksum : 9c93703e - correct
          Events : 60286

          Layout : left-symmetric
      Chunk Size : 64K

    Device Role : Active device 1
    Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdcc5:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
            Name : DS_KLIENT:4
   Creation Time : Tue Mar 31 11:40:19 2020
      Raid Level : raid5
    Raid Devices : 5

  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
     Data Offset : 2048 sectors
    Super Offset : 8 sectors
    Unused Space : before=1968 sectors, after=0 sectors
           State : clean
     Device UUID : 1d1c04b4:24dabd8d:235afb7d:1494b8eb

     Update Time : Wed Aug 14 12:42:26 2024
        Checksum : a224ec08 - correct
          Events : 60283

          Layout : left-symmetric
      Chunk Size : 64K

    Device Role : Active device 2
    Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdcd5:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
            Name : DS_KLIENT:4
   Creation Time : Tue Mar 31 11:40:19 2020
      Raid Level : raid5
    Raid Devices : 5

  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
     Data Offset : 2048 sectors
    Super Offset : 8 sectors
    Unused Space : before=1968 sectors, after=0 sectors
           State : clean
     Device UUID : 76698d3f:e9c5a397:05ef7553:9fd0af16

     Update Time : Wed Aug 14 15:09:24 2024
        Checksum : 38061500 - correct
          Events : 60286

          Layout : left-symmetric
      Chunk Size : 64K

    Device Role : Active device 4
    Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdce5:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x0
      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
            Name : DS_KLIENT:4
   Creation Time : Tue Mar 31 11:40:19 2020
      Raid Level : raid5
    Raid Devices : 5

  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
     Data Offset : 2048 sectors
    Super Offset : 8 sectors
    Unused Space : before=1968 sectors, after=0 sectors
           State : clean
     Device UUID : 9c7077f8:3120195a:1af11955:6bcebd99

     Update Time : Wed Aug 14 15:09:24 2024
        Checksum : 38177651 - correct
          Events : 60286

          Layout : left-symmetric
      Chunk Size : 64K

    Device Role : Active device 3
    Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)


thank you for your help.

miroslav

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: problem with synology external raid
  2024-08-22 15:02 problem with synology external raid Miroslav Šulc
@ 2024-08-23 11:56 ` Mariusz Tkaczyk
  2024-08-23 15:54   ` Miroslav Šulc
       [not found] ` <CALtW_ahB+mDUWVhc0Y5KCnkvn+0PyWprCnpD4ufgw2b_6UhFrg@mail.gmail.com>
  2024-08-23 16:27 ` Miroslav Šulc
  2 siblings, 1 reply; 6+ messages in thread
From: Mariusz Tkaczyk @ 2024-08-23 11:56 UTC (permalink / raw)
  To: Miroslav Šulc; +Cc: linux-raid

On Thu, 22 Aug 2024 17:02:52 +0200
Miroslav Šulc <miroslav.sulc@fordfrog.com> wrote:

> hello,
> 
> a broken synology raid5 ended up on my desk. the raid was broken for 
> quite some time, but i got it from the client just a few days back.
> 
> the raid consisted of 5 disks (no spares, all used):
> disk 1 (sdca): according to my understanding, it was removed from the 
> raid, then re-added, but the synchronization was interrupted, so it 
> cannot be used to restore the raid

Hi Miroslav,
(If I send something earlier - sorry missclick)

   Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
   Events : 60223

There is no info about interrupted rebuild in the metadata. This drive
looks like removed from array, it has old Events number. If it was
rebuilt, it finished correctly.

The metadata on the drive is frozen on the last state of the device in array,
other members were updated that this device is gone. 

> disk 2 (sdcb): is ok and up to date
> disk 3 (sdcc): seems to be up to date, still spinning, but there are 
> many issues with bad sectors

           Events : 60283
 
           Layout : left-symmetric
       Chunk Size : 64K
 
     Device Role : Active device 2
     Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)

It is outdated, Events number is smaller than on "ok" members. Sadly, probably
mdadm will refuse to start this array because you have 2 outdated drives.

On "ok" disks, it is:
 Events : 60286
Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)

Second failure is not recorded in metadata and I don't know why events number is
higher. I would expect kernel to stop updating metadata after device failure.

I will stop here and give a chance more native experienced users to elaborate.

Looks like raid recreation with --assume-clean is a simplest solution but it
is general advice I can give you (and I propose it too often). You have copies,
you did right first step!

Mariusz

> disk 4 (sdcd): is ok and up to date
> disk 5 (sdce): is ok and up to date
> 
> the raid ran in degraded mode for over two months, client trying to make 
> a copy of the data from it.
> 
> i made copies of the disk images from disks 1, 2, 4, and 5, at the state 
> shown below. i didn't attempt to make a copy of disk 3 so far. since 
> then i re-assembled the raid from disk 2-5 so the number of events on 
> the disk 3 is now a bit higher then on the copies of the disk images.
> 
> according to my understanding, as the disk 1 never finished the sync, i 
> cannot use it to recover the data, so the only way to recover the data 
> is to assemble the raid in degraded mode using disk 2-5, if i ever 
> succeed to make a copy of the disk 3. i'd just like to verify that my 
> understanding is correct and there is no other way to attempt to recover 
> the data. of course any hints are welcome.
> 
> here is the info from the partitions of the raid:
> 
> /dev/sdca5:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x2
>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>             Name : DS_KLIENT:4
>    Creation Time : Tue Mar 31 11:40:19 2020
>       Raid Level : raid5
>     Raid Devices : 5
> 
>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>      Data Offset : 2048 sectors
>     Super Offset : 8 sectors
> Recovery Offset : 4910216 sectors
>     Unused Space : before=1968 sectors, after=0 sectors
>            State : clean
>      Device UUID : 681c6c33:49df0163:bb4271d4:26c0c76d
> 
>      Update Time : Tue Jun  4 18:35:54 2024
>         Checksum : cf45a6c1 - correct
>           Events : 60223
> 
>           Layout : left-symmetric
>       Chunk Size : 64K
> 
>     Device Role : Active device 0
>     Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdcb5:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x0
>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>             Name : DS_KLIENT:4
>    Creation Time : Tue Mar 31 11:40:19 2020
>       Raid Level : raid5
>     Raid Devices : 5
> 
>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>      Data Offset : 2048 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=1968 sectors, after=0 sectors
>            State : clean
>      Device UUID : 0f23d7cd:b93301a9:5289553e:286ab6f0
> 
>      Update Time : Wed Aug 14 15:09:24 2024
>         Checksum : 9c93703e - correct
>           Events : 60286
> 
>           Layout : left-symmetric
>       Chunk Size : 64K
> 
>     Device Role : Active device 1
>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdcc5:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x0
>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>             Name : DS_KLIENT:4
>    Creation Time : Tue Mar 31 11:40:19 2020
>       Raid Level : raid5
>     Raid Devices : 5
> 
>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>      Data Offset : 2048 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=1968 sectors, after=0 sectors
>            State : clean
>      Device UUID : 1d1c04b4:24dabd8d:235afb7d:1494b8eb
> 
>      Update Time : Wed Aug 14 12:42:26 2024
>         Checksum : a224ec08 - correct
>           Events : 60283
> 
>           Layout : left-symmetric
>       Chunk Size : 64K
> 
>     Device Role : Active device 2
>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdcd5:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x0
>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>             Name : DS_KLIENT:4
>    Creation Time : Tue Mar 31 11:40:19 2020
>       Raid Level : raid5
>     Raid Devices : 5
> 
>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>      Data Offset : 2048 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=1968 sectors, after=0 sectors
>            State : clean
>      Device UUID : 76698d3f:e9c5a397:05ef7553:9fd0af16
> 
>      Update Time : Wed Aug 14 15:09:24 2024
>         Checksum : 38061500 - correct
>           Events : 60286
> 
>           Layout : left-symmetric
>       Chunk Size : 64K
> 
>     Device Role : Active device 4
>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> /dev/sdce5:
>            Magic : a92b4efc
>          Version : 1.2
>      Feature Map : 0x0
>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>             Name : DS_KLIENT:4
>    Creation Time : Tue Mar 31 11:40:19 2020
>       Raid Level : raid5
>     Raid Devices : 5
> 
>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>      Data Offset : 2048 sectors
>     Super Offset : 8 sectors
>     Unused Space : before=1968 sectors, after=0 sectors
>            State : clean
>      Device UUID : 9c7077f8:3120195a:1af11955:6bcebd99
> 
>      Update Time : Wed Aug 14 15:09:24 2024
>         Checksum : 38177651 - correct
>           Events : 60286
> 
>           Layout : left-symmetric
>       Chunk Size : 64K
> 
>     Device Role : Active device 3
>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> 
> thank you for your help.
> 
> miroslav
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: problem with synology external raid
  2024-08-23 11:56 ` Mariusz Tkaczyk
@ 2024-08-23 15:54   ` Miroslav Šulc
  2024-08-27  7:38     ` Mariusz Tkaczyk
  0 siblings, 1 reply; 6+ messages in thread
From: Miroslav Šulc @ 2024-08-23 15:54 UTC (permalink / raw)
  To: linux-raid

hi Mariusz, thanks for your reply. please find my comments below.

Dne 2024-08-23 13:56, Mariusz Tkaczyk napsal:
> On Thu, 22 Aug 2024 17:02:52 +0200
> Miroslav Šulc <miroslav.sulc@fordfrog.com> wrote:
> 
>> hello,
>> 
>> a broken synology raid5 ended up on my desk. the raid was broken for
>> quite some time, but i got it from the client just a few days back.
>> 
>> the raid consisted of 5 disks (no spares, all used):
>> disk 1 (sdca): according to my understanding, it was removed from the
>> raid, then re-added, but the synchronization was interrupted, so it
>> cannot be used to restore the raid
> 
> Hi Miroslav,
> (If I send something earlier - sorry missclick)
> 
>    Array State : AAAAA ('A' == active, '.' == missing, 'R' == 
> replacing)
>    Events : 60223
> 
> There is no info about interrupted rebuild in the metadata. This drive
> looks like removed from array, it has old Events number. If it was
> rebuilt, it finished correctly.

there is this line in the metadata from which i supposed the rebuild 
didn't finish:

Recovery Offset : 4910216 sectors

i also tried to recreate the raid from disks 1, 2, 4, and 5, using 
--assume-clean. the raid was set up, i was able to read lvm from the 
raid, but when i checked the btrfs filesystem on it or tried to mount 
it, i faced a serious corruption of the filesystem. btrfs recovery 
recovered nothing.

> The metadata on the drive is frozen on the last state of the device in 
> array,
> other members were updated that this device is gone.
> 
>> disk 2 (sdcb): is ok and up to date
>> disk 3 (sdcc): seems to be up to date, still spinning, but there are
>> many issues with bad sectors
> 
>            Events : 60283
> 
>            Layout : left-symmetric
>        Chunk Size : 64K
> 
>      Device Role : Active device 2
>      Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> replacing)
> 
> It is outdated, Events number is smaller than on "ok" members. Sadly, 
> probably
> mdadm will refuse to start this array because you have 2 outdated 
> drives.

i was able to start the array from disks 2-4 in degraded mode. i even 
tried to add disk 1 to the array and rebuild it, but that failed. my 
guess is it was because of disk 3.

> 
> On "ok" disks, it is:
>  Events : 60286
> Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> 
> Second failure is not recorded in metadata and I don't know why events 
> number is
> higher. I would expect kernel to stop updating metadata after device 
> failure.
> 
> I will stop here and give a chance more native experienced users to 
> elaborate.
> 
> Looks like raid recreation with --assume-clean is a simplest solution 
> but it
> is general advice I can give you (and I propose it too often). You have 
> copies,
> you did right first step!
> 
> Mariusz
> 
>> disk 4 (sdcd): is ok and up to date
>> disk 5 (sdce): is ok and up to date
>> 
>> the raid ran in degraded mode for over two months, client trying to 
>> make
>> a copy of the data from it.
>> 
>> i made copies of the disk images from disks 1, 2, 4, and 5, at the 
>> state
>> shown below. i didn't attempt to make a copy of disk 3 so far. since
>> then i re-assembled the raid from disk 2-5 so the number of events on
>> the disk 3 is now a bit higher then on the copies of the disk images.
>> 
>> according to my understanding, as the disk 1 never finished the sync, 
>> i
>> cannot use it to recover the data, so the only way to recover the data
>> is to assemble the raid in degraded mode using disk 2-5, if i ever
>> succeed to make a copy of the disk 3. i'd just like to verify that my
>> understanding is correct and there is no other way to attempt to 
>> recover
>> the data. of course any hints are welcome.
>> 
>> here is the info from the partitions of the raid:
>> 
>> /dev/sdca5:
>>            Magic : a92b4efc
>>          Version : 1.2
>>      Feature Map : 0x2
>>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>>             Name : DS_KLIENT:4
>>    Creation Time : Tue Mar 31 11:40:19 2020
>>       Raid Level : raid5
>>     Raid Devices : 5
>> 
>>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>>      Data Offset : 2048 sectors
>>     Super Offset : 8 sectors
>> Recovery Offset : 4910216 sectors
>>     Unused Space : before=1968 sectors, after=0 sectors
>>            State : clean
>>      Device UUID : 681c6c33:49df0163:bb4271d4:26c0c76d
>> 
>>      Update Time : Tue Jun  4 18:35:54 2024
>>         Checksum : cf45a6c1 - correct
>>           Events : 60223
>> 
>>           Layout : left-symmetric
>>       Chunk Size : 64K
>> 
>>     Device Role : Active device 0
>>     Array State : AAAAA ('A' == active, '.' == missing, 'R' == 
>> replacing)
>> /dev/sdcb5:
>>            Magic : a92b4efc
>>          Version : 1.2
>>      Feature Map : 0x0
>>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>>             Name : DS_KLIENT:4
>>    Creation Time : Tue Mar 31 11:40:19 2020
>>       Raid Level : raid5
>>     Raid Devices : 5
>> 
>>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>>      Data Offset : 2048 sectors
>>     Super Offset : 8 sectors
>>     Unused Space : before=1968 sectors, after=0 sectors
>>            State : clean
>>      Device UUID : 0f23d7cd:b93301a9:5289553e:286ab6f0
>> 
>>      Update Time : Wed Aug 14 15:09:24 2024
>>         Checksum : 9c93703e - correct
>>           Events : 60286
>> 
>>           Layout : left-symmetric
>>       Chunk Size : 64K
>> 
>>     Device Role : Active device 1
>>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
>> replacing)
>> /dev/sdcc5:
>>            Magic : a92b4efc
>>          Version : 1.2
>>      Feature Map : 0x0
>>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>>             Name : DS_KLIENT:4
>>    Creation Time : Tue Mar 31 11:40:19 2020
>>       Raid Level : raid5
>>     Raid Devices : 5
>> 
>>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>>      Data Offset : 2048 sectors
>>     Super Offset : 8 sectors
>>     Unused Space : before=1968 sectors, after=0 sectors
>>            State : clean
>>      Device UUID : 1d1c04b4:24dabd8d:235afb7d:1494b8eb
>> 
>>      Update Time : Wed Aug 14 12:42:26 2024
>>         Checksum : a224ec08 - correct
>>           Events : 60283
>> 
>>           Layout : left-symmetric
>>       Chunk Size : 64K
>> 
>>     Device Role : Active device 2
>>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
>> replacing)
>> /dev/sdcd5:
>>            Magic : a92b4efc
>>          Version : 1.2
>>      Feature Map : 0x0
>>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>>             Name : DS_KLIENT:4
>>    Creation Time : Tue Mar 31 11:40:19 2020
>>       Raid Level : raid5
>>     Raid Devices : 5
>> 
>>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>>      Data Offset : 2048 sectors
>>     Super Offset : 8 sectors
>>     Unused Space : before=1968 sectors, after=0 sectors
>>            State : clean
>>      Device UUID : 76698d3f:e9c5a397:05ef7553:9fd0af16
>> 
>>      Update Time : Wed Aug 14 15:09:24 2024
>>         Checksum : 38061500 - correct
>>           Events : 60286
>> 
>>           Layout : left-symmetric
>>       Chunk Size : 64K
>> 
>>     Device Role : Active device 4
>>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
>> replacing)
>> /dev/sdce5:
>>            Magic : a92b4efc
>>          Version : 1.2
>>      Feature Map : 0x0
>>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>>             Name : DS_KLIENT:4
>>    Creation Time : Tue Mar 31 11:40:19 2020
>>       Raid Level : raid5
>>     Raid Devices : 5
>> 
>>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>>      Data Offset : 2048 sectors
>>     Super Offset : 8 sectors
>>     Unused Space : before=1968 sectors, after=0 sectors
>>            State : clean
>>      Device UUID : 9c7077f8:3120195a:1af11955:6bcebd99
>> 
>>      Update Time : Wed Aug 14 15:09:24 2024
>>         Checksum : 38177651 - correct
>>           Events : 60286
>> 
>>           Layout : left-symmetric
>>       Chunk Size : 64K
>> 
>>     Device Role : Active device 3
>>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
>> replacing)
>> 
>> 
>> thank you for your help.
>> 
>> miroslav
>> 

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: problem with synology external raid
  2024-08-23 15:54   ` Miroslav Šulc
@ 2024-08-27  7:38     ` Mariusz Tkaczyk
  0 siblings, 0 replies; 6+ messages in thread
From: Mariusz Tkaczyk @ 2024-08-27  7:38 UTC (permalink / raw)
  To: Miroslav Šulc; +Cc: linux-raid

On Fri, 23 Aug 2024 17:54:58 +0200
Miroslav Šulc <miroslav.sulc@fordfrog.com> wrote:

> hi Mariusz, thanks for your reply. please find my comments below.
> 
> Dne 2024-08-23 13:56, Mariusz Tkaczyk napsal:
> > On Thu, 22 Aug 2024 17:02:52 +0200
> > Miroslav Šulc <miroslav.sulc@fordfrog.com> wrote:
> >   
> >> hello,
> >> 
> >> a broken synology raid5 ended up on my desk. the raid was broken for
> >> quite some time, but i got it from the client just a few days back.
> >> 
> >> the raid consisted of 5 disks (no spares, all used):
> >> disk 1 (sdca): according to my understanding, it was removed from the
> >> raid, then re-added, but the synchronization was interrupted, so it
> >> cannot be used to restore the raid  
> > 
> > Hi Miroslav,
> > (If I send something earlier - sorry missclick)
> > 
> >    Array State : AAAAA ('A' == active, '.' == missing, 'R' == 
> > replacing)
> >    Events : 60223
> > 
> > There is no info about interrupted rebuild in the metadata. This drive
> > looks like removed from array, it has old Events number. If it was
> > rebuilt, it finished correctly.  
> 
> there is this line in the metadata from which i supposed the rebuild 
> didn't finish:
> 
> Recovery Offset : 4910216 sectors
> 
> i also tried to recreate the raid from disks 1, 2, 4, and 5, using 
> --assume-clean. the raid was set up, i was able to read lvm from the 
> raid, but when i checked the btrfs filesystem on it or tried to mount 
> it, i faced a serious corruption of the filesystem. btrfs recovery 
> recovered nothing.

disk 1 is outdated so I'm not surprised here. You have better targets
(2-5). See below. "Events" is a counter of metadata updates that happened to the
array. metadata is updated for dirty clean transition or reconfiguration.
Simply bigger difference means that the device was detached earlier so the
data on this removed drive is older and less reliable for filesystem to
recover.

> 
> > The metadata on the drive is frozen on the last state of the device in 
> > array,
> > other members were updated that this device is gone.
> >   
> >> disk 2 (sdcb): is ok and up to date
> >> disk 3 (sdcc): seems to be up to date, still spinning, but there are
> >> many issues with bad sectors  
> > 
> >            Events : 60283
> > 
> >            Layout : left-symmetric
> >        Chunk Size : 64K
> > 
> >      Device Role : Active device 2
> >      Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> > replacing)
> > 
> > It is outdated, Events number is smaller than on "ok" members. Sadly, 
> > probably
> > mdadm will refuse to start this array because you have 2 outdated 
> > drives.  
> 
> i was able to start the array from disks 2-4 in degraded mode. i even 
> tried to add disk 1 to the array and rebuild it, but that failed. my 
> guess is it was because of disk 3.

You don't need to recover this array. Try read data back, you can copy data to
other media if you have any.
Disk 3 looks like damaged so I cannot guarantee it all will be successful.

Recovery process copies the data from existing disks to rebuilt disk, if
original data is not reliable, recovered one won't be reliable too. You don't
need to make it worse.

Just assemble what you can and copy what you can. Degraded state means that you
can still access all data.

Thanks,
Mariusz

> 
> > 
> > On "ok" disks, it is:
> >  Events : 60286
> > Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
> > 
> > Second failure is not recorded in metadata and I don't know why events 
> > number is
> > higher. I would expect kernel to stop updating metadata after device 
> > failure.
> > 
> > I will stop here and give a chance more native experienced users to 
> > elaborate.
> > 
> > Looks like raid recreation with --assume-clean is a simplest solution 
> > but it
> > is general advice I can give you (and I propose it too often). You have 
> > copies,
> > you did right first step!
> > 
> > Mariusz
> >   
> >> disk 4 (sdcd): is ok and up to date
> >> disk 5 (sdce): is ok and up to date
> >> 
> >> the raid ran in degraded mode for over two months, client trying to 
> >> make
> >> a copy of the data from it.
> >> 
> >> i made copies of the disk images from disks 1, 2, 4, and 5, at the 
> >> state
> >> shown below. i didn't attempt to make a copy of disk 3 so far. since
> >> then i re-assembled the raid from disk 2-5 so the number of events on
> >> the disk 3 is now a bit higher then on the copies of the disk images.
> >> 
> >> according to my understanding, as the disk 1 never finished the sync, 
> >> i
> >> cannot use it to recover the data, so the only way to recover the data
> >> is to assemble the raid in degraded mode using disk 2-5, if i ever
> >> succeed to make a copy of the disk 3. i'd just like to verify that my
> >> understanding is correct and there is no other way to attempt to 
> >> recover
> >> the data. of course any hints are welcome.
> >> 
> >> here is the info from the partitions of the raid:
> >> 
> >> /dev/sdca5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x2
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >> Recovery Offset : 4910216 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 681c6c33:49df0163:bb4271d4:26c0c76d
> >> 
> >>      Update Time : Tue Jun  4 18:35:54 2024
> >>         Checksum : cf45a6c1 - correct
> >>           Events : 60223
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 0
> >>     Array State : AAAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> /dev/sdcb5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x0
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 0f23d7cd:b93301a9:5289553e:286ab6f0
> >> 
> >>      Update Time : Wed Aug 14 15:09:24 2024
> >>         Checksum : 9c93703e - correct
> >>           Events : 60286
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 1
> >>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> /dev/sdcc5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x0
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 1d1c04b4:24dabd8d:235afb7d:1494b8eb
> >> 
> >>      Update Time : Wed Aug 14 12:42:26 2024
> >>         Checksum : a224ec08 - correct
> >>           Events : 60283
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 2
> >>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> /dev/sdcd5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x0
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 76698d3f:e9c5a397:05ef7553:9fd0af16
> >> 
> >>      Update Time : Wed Aug 14 15:09:24 2024
> >>         Checksum : 38061500 - correct
> >>           Events : 60286
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 4
> >>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> /dev/sdce5:
> >>            Magic : a92b4efc
> >>          Version : 1.2
> >>      Feature Map : 0x0
> >>       Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
> >>             Name : DS_KLIENT:4
> >>    Creation Time : Tue Mar 31 11:40:19 2020
> >>       Raid Level : raid5
> >>     Raid Devices : 5
> >> 
> >>   Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
> >>       Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
> >>      Data Offset : 2048 sectors
> >>     Super Offset : 8 sectors
> >>     Unused Space : before=1968 sectors, after=0 sectors
> >>            State : clean
> >>      Device UUID : 9c7077f8:3120195a:1af11955:6bcebd99
> >> 
> >>      Update Time : Wed Aug 14 15:09:24 2024
> >>         Checksum : 38177651 - correct
> >>           Events : 60286
> >> 
> >>           Layout : left-symmetric
> >>       Chunk Size : 64K
> >> 
> >>     Device Role : Active device 3
> >>     Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> >> replacing)
> >> 
> >> 
> >> thank you for your help.
> >> 
> >> miroslav
> >>   
> 


^ permalink raw reply	[flat|nested] 6+ messages in thread

[parent not found: <CALtW_ahB+mDUWVhc0Y5KCnkvn+0PyWprCnpD4ufgw2b_6UhFrg@mail.gmail.com>]

* Re: problem with synology external raid
       [not found] ` <CALtW_ahB+mDUWVhc0Y5KCnkvn+0PyWprCnpD4ufgw2b_6UhFrg@mail.gmail.com>
@ 2024-08-23 15:56   ` Miroslav Šulc
  0 siblings, 0 replies; 6+ messages in thread
From: Miroslav Šulc @ 2024-08-23 15:56 UTC (permalink / raw)
  To: linux-raid

hi Dragan, thank you for your reply.

Dne 2024-08-22 18:24, Dragan Milivojević napsal:
>> according to my understanding, as the disk 1 never finished the sync, 
>> i
>> cannot use it to recover the data, so the only way to recover the data
>> is to assemble the raid in degraded mode using disk 2-5, if i ever
>> succeed to make a copy of the disk 3. i'd just like to verify that my
>> understanding is correct and there is no other way to attempt to 
>> recover
>> the data. of course any hints are welcome.
> 
> You figured it right, the only advice is to use dd_rescue in an attempt 
> to get
> most of the data from the disk #3.

today i finally tried to ddrescue the broken disk, but the reading is so 
slow that it seems to be unrecoverable this way.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: problem with synology external raid
  2024-08-22 15:02 problem with synology external raid Miroslav Šulc
  2024-08-23 11:56 ` Mariusz Tkaczyk
       [not found] ` <CALtW_ahB+mDUWVhc0Y5KCnkvn+0PyWprCnpD4ufgw2b_6UhFrg@mail.gmail.com>
@ 2024-08-23 16:27 ` Miroslav Šulc
  2 siblings, 0 replies; 6+ messages in thread
From: Miroslav Šulc @ 2024-08-23 16:27 UTC (permalink / raw)
  To: linux-raid

currently, i can see these two options for recovering the data:

1) send the disk 3 (the broken one) for repair, copy the data, and 
restore the array using disks 2-5
    this brings more expenses for the client but the chances for 
recovering the data are probably pretty high, if the data from the disk 
3 is recovered - i might need to use --force on the assembly or even 
--assume-clean as the number of events is little bit increased because i 
re-assembled the array from disks 2-6 after i made the backups of the 
other disks
2) investigate deeper whether i can or cannot re-create the array using 
disks 1, 2, 4 and 5
    i am confused about this solution, because of that (imo) interrupted 
rebuild on disk 1 (Recovery Offset : 4910216 sectors). on the other 
hand, i was able to assemble the array, read the lvm logical volume, 
"just" the btrfs filesystem was reported to be seriously corrupted that 
even btrfs recovery recovered nothing. but i didn't try to repair the 
filesystem yet.

any hints?

Dne 2024-08-22 17:02, Miroslav Šulc napsal:
> hello,
> 
> a broken synology raid5 ended up on my desk. the raid was broken for 
> quite some time, but i got it from the client just a few days back.
> 
> the raid consisted of 5 disks (no spares, all used):
> disk 1 (sdca): according to my understanding, it was removed from the 
> raid, then re-added, but the synchronization was interrupted, so it 
> cannot be used to restore the raid
> disk 2 (sdcb): is ok and up to date
> disk 3 (sdcc): seems to be up to date, still spinning, but there are 
> many issues with bad sectors
> disk 4 (sdcd): is ok and up to date
> disk 5 (sdce): is ok and up to date
> 
> the raid ran in degraded mode for over two months, client trying to 
> make a copy of the data from it.
> 
> i made copies of the disk images from disks 1, 2, 4, and 5, at the 
> state shown below. i didn't attempt to make a copy of disk 3 so far. 
> since then i re-assembled the raid from disk 2-5 so the number of 
> events on the disk 3 is now a bit higher then on the copies of the disk 
> images.
> 
> according to my understanding, as the disk 1 never finished the sync, i 
> cannot use it to recover the data, so the only way to recover the data 
> is to assemble the raid in degraded mode using disk 2-5, if i ever 
> succeed to make a copy of the disk 3. i'd just like to verify that my 
> understanding is correct and there is no other way to attempt to 
> recover the data. of course any hints are welcome.
> 
> here is the info from the partitions of the raid:
> 
> /dev/sdca5:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x2
>      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>            Name : DS_KLIENT:4
>   Creation Time : Tue Mar 31 11:40:19 2020
>      Raid Level : raid5
>    Raid Devices : 5
> 
>  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
> Recovery Offset : 4910216 sectors
>    Unused Space : before=1968 sectors, after=0 sectors
>           State : clean
>     Device UUID : 681c6c33:49df0163:bb4271d4:26c0c76d
> 
>     Update Time : Tue Jun  4 18:35:54 2024
>        Checksum : cf45a6c1 - correct
>          Events : 60223
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 0
>    Array State : AAAAA ('A' == active, '.' == missing, 'R' == 
> replacing)
> /dev/sdcb5:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>            Name : DS_KLIENT:4
>   Creation Time : Tue Mar 31 11:40:19 2020
>      Raid Level : raid5
>    Raid Devices : 5
> 
>  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=0 sectors
>           State : clean
>     Device UUID : 0f23d7cd:b93301a9:5289553e:286ab6f0
> 
>     Update Time : Wed Aug 14 15:09:24 2024
>        Checksum : 9c93703e - correct
>          Events : 60286
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 1
>    Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> replacing)
> /dev/sdcc5:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>            Name : DS_KLIENT:4
>   Creation Time : Tue Mar 31 11:40:19 2020
>      Raid Level : raid5
>    Raid Devices : 5
> 
>  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=0 sectors
>           State : clean
>     Device UUID : 1d1c04b4:24dabd8d:235afb7d:1494b8eb
> 
>     Update Time : Wed Aug 14 12:42:26 2024
>        Checksum : a224ec08 - correct
>          Events : 60283
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 2
>    Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> replacing)
> /dev/sdcd5:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>            Name : DS_KLIENT:4
>   Creation Time : Tue Mar 31 11:40:19 2020
>      Raid Level : raid5
>    Raid Devices : 5
> 
>  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=0 sectors
>           State : clean
>     Device UUID : 76698d3f:e9c5a397:05ef7553:9fd0af16
> 
>     Update Time : Wed Aug 14 15:09:24 2024
>        Checksum : 38061500 - correct
>          Events : 60286
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 4
>    Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> replacing)
> /dev/sdce5:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : f697911c:bc85b162:13eaba4e:d1152a4f
>            Name : DS_KLIENT:4
>   Creation Time : Tue Mar 31 11:40:19 2020
>      Raid Level : raid5
>    Raid Devices : 5
> 
>  Avail Dev Size : 15618390912 (7447.43 GiB 7996.62 GB)
>      Array Size : 31236781824 (29789.72 GiB 31986.46 GB)
>     Data Offset : 2048 sectors
>    Super Offset : 8 sectors
>    Unused Space : before=1968 sectors, after=0 sectors
>           State : clean
>     Device UUID : 9c7077f8:3120195a:1af11955:6bcebd99
> 
>     Update Time : Wed Aug 14 15:09:24 2024
>        Checksum : 38177651 - correct
>          Events : 60286
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>    Device Role : Active device 3
>    Array State : .AAAA ('A' == active, '.' == missing, 'R' == 
> replacing)
> 
> 
> thank you for your help.
> 
> miroslav

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2024-08-27  7:39 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-08-22 15:02 problem with synology external raid Miroslav Šulc
2024-08-23 11:56 ` Mariusz Tkaczyk
2024-08-23 15:54   ` Miroslav Šulc
2024-08-27  7:38     ` Mariusz Tkaczyk
     [not found] ` <CALtW_ahB+mDUWVhc0Y5KCnkvn+0PyWprCnpD4ufgw2b_6UhFrg@mail.gmail.com>
2024-08-23 15:56   ` Miroslav Šulc
2024-08-23 16:27 ` Miroslav Šulc

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).