btrfs on top of bcache on top of dmcrypt on top of md raid5

btrfs on top of bcache on top of dmcrypt on top of md raid5

[Marc MERLIN]

I have a 5 drive md array with dmcrypt on top, and btrfs on top of that.
Kernel: 4.4 but the filesystem was created 2 years ago with an older version
of btrfs.
It's littered with files and hardlinks (it's a backup server). Mostly it
gets btrfs receive data, and rsyncs of filesystem trees that are
occasionally hardlinked to keep history (for data that wasn't on btrfs to
start with).
Basically the filesystem works, but it's slow, I can see that my system
feels sluggish when backups are running, cronjobs that are somewhat time
critical also fail to run in time when rsyncs/backups to that filesystem,
are running.

It's time to re-create it, but this time I'm looking at adding bcache in the
middle (backed by an encrypted ssd) to hopefully help with the random I/O
bits that won't be as fast on disk backed raid5.

Are there best practises in doing this?

Are there issues with the default filesystem options in btrfs?

Do I want -m dup considering it's ultimately backed by raid5/hdd and not
ssd? (I would think yes, but I've noticed -m dup gets disabled when bcache
is in the middle, probably because the detection gets foiled).

Do I want to mess with --nodesize or --sectorsize and adjust for ssd write
block size? (with ext4, I use -b 4096 -E stride=128,stripe-width=128 )

Any specific configuration I ought to do with bcache or mdadm chunk sizes?

Does align-payload look ok?
cryptsetup luksFormat --align-payload=8192 -s 256 -c aes-xts-plain

Thanks,
Marc

PS: for reference:
As discussed in the past, there seems to be a general agreement that dmcrypt
on top of mdadm is better than mdadm on top of dmcrypt now that dmcrypt is
multithreaded.
My current array and encryption look like this.

Currently, I have:
gargamel:~# mdadm --detail /dev/md8
/dev/md8:   
        Version : 1.2
  Creation Time : Sat Apr 19 23:03:59 2014
     Raid Level : raid5
     Array Size : 7813523456 (7451.56 GiB 8001.05 GB)
  Used Dev Size : 1953380864 (1862.89 GiB 2000.26 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Feb 11 08:26:45 2016
          State : active 
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 256K

gargamel:~# cryptsetup luksDump /dev/md8
LUKS header information for /dev/md8

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 3072
MK bits:        256

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: btrfs on top of bcache on top of dmcrypt on top of md raid5

[Chris Murphy]

Use all defaults for everything. Anything new by show should do the
right thing including 4096 byte alignment.

gargamel:~# cryptsetup luksDump /dev/md8
[snip]
Payload offset: 3072

This is a bit weird because the default is 4096. But because the LUKS
offset (header + payload + extra unused space) is 2MiB, so it doesn't
affect alignment. There may be unpatched (fixes not backported) in the
tools of current long term supported distros, that can cause
misalignment. Probably top concern would be parted/libparted, which
would start partition 1 at LBA 63, which is not aligned. The upstream
tools for a long time now have set partition 1 to LBA 2048, but these
crusty old unpatched versions just seem to persist like a booger you
can't flick off. It's really annoying - this idea of "stable bugs"
that go on and on for a decade.



Chris Murphy

Re: btrfs on top of bcache on top of dmcrypt on top of md raid5

[Marc MERLIN]

On Sun, Feb 14, 2016 at 01:43:05PM -0700, Chris Murphy wrote:
> Use all defaults for everything. Anything new by show should do the
> right thing including 4096 byte alignment.
> 
> gargamel:~# cryptsetup luksDump /dev/md8
> [snip]
> Payload offset: 3072
> 
> This is a bit weird because the default is 4096. But because the LUKS
> offset (header + payload + extra unused space) is 2MiB, so it doesn't
> affect alignment. There may be unpatched (fixes not backported) in the
> tools of current long term supported distros, that can cause
> misalignment. Probably top concern would be parted/libparted, which
> would start partition 1 at LBA 63, which is not aligned. The upstream
> tools for a long time now have set partition 1 to LBA 2048, but these
> crusty old unpatched versions just seem to persist like a booger you
> can't flick off. It's really annoying - this idea of "stable bugs"
> that go on and on for a decade.

Indeed. Thankfully my partitions now start at 2048 like you say.

The only thing I did wrong last time (when using bcache) is
md5 - dmcrypt - bcache - btrfs
ssd - dmcrypt /

This was stupid, I needed to do this:
md5  - bcache - dmcrypt - btrfs
ssd /

So I think at this point, just to be future proof, I'm going to add bcache 
on top of all block devices I have, before putting dmcrypt on top, even if I
don't have a cache device.
That way I can later add a cache device without problems.

Without doing that, adding bcache later is a full re-install :(

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/