Hi List I posted this to the Open Media Vault (my NAS O/S) list a few days ago without response. A definitive answer would be appreciated. Have been running a 4 disk Raid 6 setup for over two years without any issues, until suddenly on June 27 disks on my OMV NAS became 100% full, including the NFS mounted volumes in the raid set. There'd been major disk activity overnight, but foolishly didn't investigate. Rsnapshot normally backs up two desktop machines onto the raid setup: the next morning found that one of the backup directories was not on raid but suddenly was on the root directory of the OMV machine, and being a total backup (several Gb) this accounted for the 100% reading for the OMV/NAS machine. The logs indicated that mdstat had discovered "dirty degraded array" presumably due to faulty sdb, so had withdrawn that disk, and then couldn't run the raid set (logs below show) Bought a new disk and installed on July 4, and raid rebuilt overnight (see July 5 Rebuild finished below) Since then I've been unable to mount or access any data. Have followed instructions as per Linux Raid Wiki's "Recovering a failed software RAID" & "RAID Recovery" , but still no success. I've attached the results of their suggestions in the attached log file "linux_raid_wiki_logs.txt". Any help appreciated - even if it's just to tell me my RAID sets are hosed! P.S. This line looks ominous? !!! ===============Jun 27 16:52:21 keruru kernel: [ 2.912440] md: md0 stopped. Jun 27 16:52:21 keruru kernel: [ 2.922315] md: bind Jun 27 16:52:21 keruru kernel: [ 2.922508] md: bind Jun 27 16:52:21 keruru kernel: [ 2.922643] md: bind Jun 27 16:52:21 keruru kernel: [ 2.922777] md: bind Jun 27 16:52:21 keruru kernel: [ 2.922808] md: kicking non-fresh sdb from array! Jun 27 16:52:21 keruru kernel: [ 2.922820] md: unbind Jun 27 16:52:21 keruru kernel: [ 2.927107] md: export_rdev(sdb) Jun 27 16:52:21 keruru kernel: [ 2.994973] raid6: sse2x1 588 MB/s Jun 27 16:52:21 keruru kernel: [ 3.062926] raid6: sse2x2 1395 MB/s Jun 27 16:52:21 keruru kernel: [ 3.130841] raid6: sse2x4 2397 MB/s Jun 27 16:52:21 keruru kernel: [ 3.130844] raid6: using algorithm sse2x4 (2397 MB/s) Jun 27 16:52:21 keruru kernel: [ 3.130846] raid6: using ssse3x2 recovery algorithm Jun 27 16:52:21 keruru kernel: [ 3.130866] Switched to clocksource tsc Jun 27 16:52:21 keruru kernel: [ 3.131227] xor: automatically using best checksumming function: Jun 27 16:52:21 keruru kernel: [ 3.170797] avx : 6164.000 MB/sec Jun 27 16:52:21 keruru kernel: [ 3.171121] async_tx: api initialized (async) Jun 27 16:52:21 keruru kernel: [ 3.172809] md: raid6 personality registered for level 6 Jun 27 16:52:21 keruru kernel: [ 3.172812] md: raid5 personality registered for level 5 Jun 27 16:52:21 keruru kernel: [ 3.172815] md: raid4 personality registered for level 4 Jun 27 16:52:21 keruru kernel: [ 3.173218] md/raid:md0: not clean -- starting background reconstruction Jun 27 16:52:21 keruru kernel: [ 3.173236] md/raid:md0: device sdd operational as raid disk 1 Jun 27 16:52:21 keruru kernel: [ 3.173239] md/raid:md0: device sde operational as raid disk 3 Jun 27 16:52:21 keruru kernel: [ 3.173242] md/raid:md0: device sdc operational as raid disk 2 Jun 27 16:52:21 keruru kernel: [ 3.173706] md/raid:md0: allocated 0kB Jun 27 16:52:21 keruru kernel: [ 3.173745] md/raid:md0: cannot start dirty degraded array. Jun 27 16:52:21 keruru kernel: [ 3.173811] RAID conf printout: Jun 27 16:52:21 keruru kernel: [ 3.173814] --- level:6 rd:4 wd:3 Jun 27 16:52:21 keruru kernel: [ 3.173816] disk 1, o:1, dev:sdd Jun 27 16:52:21 keruru kernel: [ 3.173818] disk 2, o:1, dev:sdc Jun 27 16:52:21 keruru kernel: [ 3.173820] disk 3, o:1, dev:sde Jun 27 16:52:21 keruru kernel: [ 3.174025] md/raid:md0: failed to run raid set. Jun 27 16:52:21 keruru kernel: [ 3.174071] md: pers->run() failed ... =============== New disk added - sdb =============== Jul 5 21:06:18 keruru mdadm[2497]: RebuildFinished event detected on md device /dev/md0, component device mismatches found: 1847058224 (on raid level 6) Jul 6 09:45:52 keruru kernel: [ 1195.390879] raid6: sse2x1 249 MB/s Jul 6 09:45:52 keruru kernel: [ 1195.458735] raid6: sse2x2 476 MB/s Jul 6 09:45:52 keruru kernel: [ 1195.526632] raid6: sse2x4 839 MB/s Jul 6 09:45:52 keruru kernel: [ 1195.526638] raid6: using algorithm sse2x4 (839 MB/s) Jul 6 09:45:52 keruru kernel: [ 1195.526644] raid6: using ssse3x2 recovery algorithm Jul 6 09:45:52 keruru kernel: [ 1195.578970] md: raid6 personality registered for level 6 Jul 6 09:45:52 keruru kernel: [ 1195.578980] md: raid5 personality registered for level 5 Jul 6 09:45:52 keruru kernel: [ 1195.578985] md: raid4 personality registered for level 4 Jul 6 09:45:52 keruru kernel: [ 1195.580003] md/raid:md0: device sdb operational as raid disk 0 Jul 6 09:45:52 keruru kernel: [ 1195.580012] md/raid:md0: device sde operational as raid disk 3 Jul 6 09:45:52 keruru kernel: [ 1195.580018] md/raid:md0: device sdd operational as raid disk 2 Jul 6 09:45:52 keruru kernel: [ 1195.580025] md/raid:md0: device sdc operational as raid disk 1 Jul 6 09:45:52 keruru kernel: [ 1195.581091] md/raid:md0: allocated 0kB Jul 6 09:45:52 keruru kernel: [ 1195.581180] md/raid:md0: raid level 6 active with 4 out of 4 devices, algorithm 2 Jul 6 09:52:30 keruru kernel: [ 4.186106] raid6: sse2x1 602 MB/s Jul 6 09:52:30 keruru kernel: [ 4.254006] raid6: sse2x2 906 MB/s Jul 6 09:52:30 keruru kernel: [ 4.186106] raid6: sse2x1 602 MB/s Jul 6 09:52:30 keruru kernel: [ 4.254006] raid6: sse2x2 906 MB/s Jul 6 09:52:30 keruru kernel: [ 4.321957] raid6: sse2x4 1130 MB/s Jul 6 09:52:30 keruru kernel: [ 4.321964] raid6: using algorithm sse2x4 (1130 MB/s) Jul 6 09:52:30 keruru kernel: [ 4.321967] raid6: using ssse3x2 recovery algorithm Jul 6 09:52:30 keruru kernel: [ 4.368478] md: raid6 personality registered for level 6 Jul 6 09:52:30 keruru kernel: [ 4.368486] md: raid5 personality registered for level 5 Jul 6 09:52:30 keruru kernel: [ 4.368490] md: raid4 personality registered for level 4 Jul 6 09:52:30 keruru kernel: [ 4.369179] md/raid:md0: device sdb operational as raid disk 0 Jul 6 09:52:30 keruru kernel: [ 4.369185] md/raid:md0: device sde operational as raid disk 3 Jul 6 09:52:30 keruru kernel: [ 4.369189] md/raid:md0: device sdd operational as raid disk 2 Jul 6 09:52:30 keruru kernel: [ 4.369194] md/raid:md0: device sdc operational as raid disk 1 Jul 6 09:52:30 keruru kernel: [ 4.369974] md/raid:md0: allocated 0kB Jul 6 09:52:30 keruru kernel: [ 4.372062] md/raid:md0: raid level 6 active with 4 out of 4 devices, algorithm 2 Jul 6 12:56:15 keruru kernel: [ 4.442184] raid6: sse2x1 739 MB/s Jul 6 12:56:15 keruru kernel: [ 4.510060] raid6: sse2x2 1480 MB/s Jul 6 12:56:15 keruru kernel: [ 4.577985] raid6: sse2x4 1605 MB/s Jul 6 12:56:15 keruru kernel: [ 4.577993] raid6: using algorithm sse2x4 (1605 MB/s) Jul 6 12:56:15 keruru kernel: [ 4.577997] raid6: using ssse3x2 recovery algorithm Jul 6 12:56:15 keruru kernel: [ 4.622570] md: raid6 personality registered for level 6 Jul 6 12:56:15 keruru kernel: [ 4.622577] md: raid5 personality registered for level 5 Jul 6 12:56:15 keruru kernel: [ 4.622580] md: raid4 personality registered for level 4 Jul 6 12:56:15 keruru kernel: [ 4.623261] md/raid:md0: device sdb operational as raid disk 0 Jul 6 12:56:15 keruru kernel: [ 4.623266] md/raid:md0: device sde operational as raid disk 3 Jul 6 12:56:15 keruru kernel: [ 4.623269] md/raid:md0: device sdd operational as raid disk 2 Jul 6 12:56:15 keruru kernel: [ 4.623273] md/raid:md0: device sdc operational as raid disk 1 Jul 6 12:56:15 keruru kernel: [ 4.624064] md/raid:md0: allocated 0kB Jul 6 12:56:15 keruru kernel: [ 4.624131] md/raid:md0: raid level 6 active with 4 out of 4 devices, algorithm 2 Jul 6 16:54:43 keruru kernel: [14401.858429] md/raid:md0: device sdb operational as raid disk 0 Jul 6 16:54:43 keruru kernel: [14401.858442] md/raid:md0: device sde operational as raid disk 3 Jul 6 16:54:43 keruru kernel: [14401.858449] md/raid:md0: device sdd operational as raid disk 2 Jul 6 16:54:43 keruru kernel: [14401.858455] md/raid:md0: device sdc operational as raid disk 1 Jul 6 16:54:43 keruru kernel: [14401.859915] md/raid:md0: allocated 0kB Jul 6 16:54:43 keruru kernel: [14401.860000] md/raid:md0: raid level 6 active with 4 out of 4 devices, algorithm 2