Re: RAID 6, 6 device array - all devices lost superblock

Peter Sanders
2022-09-02T15:24:01Z

contents of /proc/mdstat

root@superior:/mnt/backup# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
unused devices: <none>
root@superior:/mnt/backup#

Here are the steps I ran (minus some mounting other devices and
looking around for mdadm tracks on the old os disk)

410 DEVICES=$(cat /proc/partitions | parallel --tagstring {5}
--colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t'
echo /dev/{1})
411 apt install parallel
412 DEVICES=$(cat /proc/partitions | parallel --tagstring {5}
--colsep ' +' mdadm -E /dev/{5} |grep $UUID | parallel --colsep '\t'
echo /dev/{1})
413 echo $DEVICES
414 cat /proc/partitions
415 DEVICES=/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg
416 DEVICES="/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg"
417 echo $DEVICES
418 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7
{#}' ::: $DEVICES
419 ls /dev/loop*
420 dc
421 cd /mnt/backup/
422 ls
423 parallel truncate -s300G overlay-{/} ::: $DEVICES
424 ls
425 ls -la
426 df -h
427 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f
--show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup
create {/}' ::: $DEVICES
428 ls /dev/mapper/
429 OVERLAYS=$(parallel echo /dev/mapper/{/} ::: $DEVICES)
430 echo $OVERLAYS
431 dmsetup status
432 mdadm --assemble --force /dev/md1 $OVERLAYS
433 history
434 dmsetup status
435 echo $OVERLAYS
436 mdadm --assemble --force /dev/md0 $OVERLAYS
437 cat /proc/partitions
438 mkdir /mnt/oldroot
<< look for inird mdadm files >>
484 echo $OVERLAYS
485 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb
/dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf
/dev/mapper/sdg
<< cancelled out of 485, review instructions... >>
486 mdadm --create /dev/md0 --level=raid6 -n 6 /dev/mapper/sdb
/dev/mapper/sdc /dev/mapper/sdd /dev/mapper/sde /dev/mapper/sdf
/dev/mapper/sdg
487 fsck -n /dev/md0
488 mdadm --stop /dev/md0
489 echo $DEVICES
490 parallel 'dmsetup remove {/}; rm overlay-{/}' ::: $DEVICES
491 dmsetup status
492 ls
493 rm overlay-*
494 ls
495 parallel losetup -d ::: /dev/loop[0-9]*
496 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b 7
{#}' ::: $DEVICES
497 parallel truncate -s300G overlay-{/} ::: $DEVICES
498 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f
--show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup
create {/}' ::: $DEVICES
499 dmsetup status
500 /sbin/reboot
501 history
502 dmsetup status
503 mount
504 cat /proc/partitions
505 nano /etc/fstab
506 mount /mnt/backup/
507 ls /mnt/backup/
508 rm /mnt/backup/
509 rm /mnt/backup/overlay-sd*
510 emacs setupOverlay &
511 ps auxww | grep emacs
512 kill 65017
513 ls /dev/loo*
514 DEVICES='/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg'
515 echo $DEVICES
516 parallel 'test -e /dev/loop{#} || mknod -m 660 /dev/loop{#} b
7 {#}' ::: $DEVICES
517 ls /dev/loo*
518 parallel truncate -s4000G overlay-{/} ::: $DEVICES
519 ls
520 rm overlay-sd*
521 cd /mnt/bak
522 cd /mnt/backup/
523 ls
524 parallel truncate -s4000G overlay-{/} ::: $DEVICES
525 ls -la
526 blockdev --getsize /dev/sdb
527 man losetup
528 man losetup
529 parallel 'size=$(blockdev --getsize {}); loop=$(losetup -f
--show -- overlay-{/}); echo 0 $size snapshot {} $loop P 8 | dmsetup
create {/}' ::: $DEVICES
530 dmsetup status
531 history | grep mdadm
532 history
533 dmsetup status
534 history | grep dmsetup
535 dmsetup status
536 dmsetup remove sdg
537 dmsetup ls --tree
538 lsof
539 dmsetup ls --tre
540 dmsetup ls --tree
541 lsof | grep -i sdg
542 lsof | grep -i sdf
543 history |grep dmsetup | less
544 dmsetup status
545 history > ~plsander/Documents/raidIssues/joblog

On Wed, Aug 31, 2022 at 4:37 PM John Stoffel <john@stoffel.org> wrote:
>
> >>>>> "Peter" == Peter Sanders <plsander@gmail.com> writes:
>
> > encountering a puzzling situation.
> > dmsetup is failing to return.
>
> I don't think you need to use dmsetup in your case, but can you post
> *all* the commands you ran before you got to this point, and the
> output of
>
> cat /proc/mdstat
>
> as well? Thinking on this some more, you might need to actually also
> add:
>
> --assume-clean
>
> to the 'mdadm create ....' string, since you don't want it to zero the
> array or anything.
>
> Sorry for not remembering this at the time!
>
> So if you can, please just start over from scratch, showing the setup
> of the loop devices, the overlayfs setup, and the building the RAID6
> array, along with the cat /proc/mdstat after you do the initial build.
>
> John
>
> P.S. For those who hated my email citing tool, I pulled it out for
> now. Only citing with > now. :-)
>
> > root@superior:/mnt/backup# dmsetup status
> > sdg: 0 5860533168 snapshot 16/8388608000 16
> > sdf: 0 5860533168 snapshot 16/8388608000 16
> > sde: 0 5860533168 snapshot 16/8388608000 16
> > sdd: 0 5860533168 snapshot 16/8388608000 16
> > sdc: 0 5860533168 snapshot 16/8388608000 16
> > sdb: 0 5860533168 snapshot 16/8388608000 16
>
> > dmsetup remove sdg runs for hours.
> > Canceled it, ran dmsetup ls --tree and find that sdg is not present in the list.
>
> > dmsetup status shows:
> > sdf: 0 5860533168 snapshot 16/8388608000 16
> > sde: 0 5860533168 snapshot 16/8388608000 16
> > sdd: 0 5860533168 snapshot 16/8388608000 16
> > sdc: 0 5860533168 snapshot 16/8388608000 16
> > sdb: 0 5860533168 snapshot 16/8388608000 16
>
> > dmsetup ls --tree
> > root@superior:/mnt/backup# dmsetup ls --tree
> > sdf (253:3)
> > ├─ (7:3)
> > └─ (8:80)
> > sde (253:1)
> > ├─ (7:1)
> > └─ (8:64)
> > sdd (253:2)
> > ├─ (7:2)
> > └─ (8:48)
> > sdc (253:0)
> > ├─ (7:0)
> > └─ (8:32)
> > sdb (253:5)
> > ├─ (7:5)
> > └─ (8:16)
>
> > any suggestions?
>
>
>
> > On Tue, Aug 30, 2022 at 2:03 PM Wols Lists <antlists@youngman.org.uk> wrote:
> >>
> >> On 30/08/2022 14:27, Peter Sanders wrote:
> >> >
> >> > And the victory conditions would be a mountable file system that passes a fsck?
> >>
> >> Yes. Just make sure you delve through the file system a bit and satisfy
> >> yourself it looks good, too ...
> >>
> >> Cheers,
> >> Wol