Re: Suggestions for building new 44TB Raid5 array

[Roman Mamedov <rm@romanrm.net>]

On Fri, 10 Jun 2022 21:51:20 -0700
Marc MERLIN <marc@merlins.org> wrote:

> Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger
> files (100MB to 10GB).

> 2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 >  /sys/block/bcache64/bcache/attach 
>    gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
>    [writethrough] writeback writearound none

Maybe try LVM Cache this time?

> 3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64  /dev/bcache64
> 4) cryptsetup luksOpen /dev/bcache64 dshelf1

What's the threat scenario for LUKS on the array?

A major one for me would be not to be having to RMA a disk with all my data
still on the platters. But with RAID5, a single disk by itself would not
contain easily discernible or usable data. Or if you're protecting against
unauthorized access to the entire array, then never mind.

> 5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1

Personally I have switched from Btrfs on MD to individual disks and MergerFS.

The rationale for no RAID is the simplicity and resilience of the individual
single-disk filesystems, and that anything important or not easily
re-obtainable is backed up anyways; so the protection from single disk
failures is not as important, compared to the introduced complexity and the
chance of losing the entire huge FS (like you had).

-- 
With respect,
Roman