btrfs on software RAID0

btrfs on software RAID0

[john terragon]

Hi.
I'm about to try btrfs on an RAID0 md device (to be precise there will
be dm-crypt in between the md device and btrfs). If I used ext4 I
would set the stride and stripe_width extended options. Is there
anything similar I should be doing with mkfs.btrfs? Or maybe some
mount options beneficial to this kind of setting.

Thanks
John

Re: btrfs on software RAID0

[Marc MERLIN]

On Mon, May 05, 2014 at 10:51:46PM +0200, john terragon wrote:
> Hi.
> I'm about to try btrfs on an RAID0 md device (to be precise there will
> be dm-crypt in between the md device and btrfs). If I used ext4 I
> would set the stride and stripe_width extended options. Is there
> anything similar I should be doing with mkfs.btrfs? Or maybe some
> mount options beneficial to this kind of setting.

This is not directly an answer to your question, so far I haven't used a
special option like this with btrfs on my arrays although my
undertstanding is that it's not as important as with ext4.

That said, please read
http://marc.merlins.org/perso/btrfs/post_2014-04-27_Btrfs-Multi-Device-Dmcrypt.html

1) use align-payload=1024 on cryptsetup instead of something bigger like
8192. This will reduce write amplification (if you're not on an SSD).

2) you don't need md0 in the middle, crypt each device and then use
btrfs built in raid0 which will be faster (and is stable, at least as
far as we know :) ).

Then use /etc/crypttab or a script like this
http://marc.merlins.org/linux/scripts/start-btrfs-dmcrypt
to decrypt all your devices in one swoop and mount btrfs.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs on software RAID0

[ronnie sahlberg]

start-btrfs-dmcrypt :
...
echo "$pwd" |
...

Hmmm. This makes the plaintext password visible in ps output.
It is probably better to pass this in by redirecting a file to stdin.



On Mon, May 5, 2014 at 2:25 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Mon, May 05, 2014 at 10:51:46PM +0200, john terragon wrote:
>> Hi.
>> I'm about to try btrfs on an RAID0 md device (to be precise there will
>> be dm-crypt in between the md device and btrfs). If I used ext4 I
>> would set the stride and stripe_width extended options. Is there
>> anything similar I should be doing with mkfs.btrfs? Or maybe some
>> mount options beneficial to this kind of setting.
>
> This is not directly an answer to your question, so far I haven't used a
> special option like this with btrfs on my arrays although my
> undertstanding is that it's not as important as with ext4.
>
> That said, please read
> http://marc.merlins.org/perso/btrfs/post_2014-04-27_Btrfs-Multi-Device-Dmcrypt.html
>
> 1) use align-payload=1024 on cryptsetup instead of something bigger like
> 8192. This will reduce write amplification (if you're not on an SSD).
>
> 2) you don't need md0 in the middle, crypt each device and then use
> btrfs built in raid0 which will be faster (and is stable, at least as
> far as we know :) ).
>
> Then use /etc/crypttab or a script like this
> http://marc.merlins.org/linux/scripts/start-btrfs-dmcrypt
> to decrypt all your devices in one swoop and mount btrfs.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs on software RAID0

[Marc MERLIN]

On Mon, May 05, 2014 at 02:42:53PM -0700, ronnie sahlberg wrote:
> start-btrfs-dmcrypt :
> ...
> echo "$pwd" |
> ...
> 
> Hmmm. This makes the plaintext password visible in ps output.
> It is probably better to pass this in by redirecting a file to stdin.

echo is built in, it will not show up in ps.

Either way, the password management is a placeholder, the script I
actually use is different and custom to my own use.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs on software RAID0

[john terragon]

On Mon, May 5, 2014 at 11:25 PM, Marc MERLIN <marc@merlins.org> wrote:
> This is not directly an answer to your question, so far I haven't used a
> special option like this with btrfs on my arrays although my
> undertstanding is that it's not as important as with ext4.
>
> That said, please read
> http://marc.merlins.org/perso/btrfs/post_2014-04-27_Btrfs-Multi-Device-Dmcrypt.html
>
> 1) use align-payload=1024 on cryptsetup instead of something bigger like
> 8192. This will reduce write amplification (if you're not on an SSD).
>
> 2) you don't need md0 in the middle, crypt each device and then use
> btrfs built in raid0 which will be faster (and is stable, at least as
> far as we know :) ).
>
> Then use /etc/crypttab or a script like this
> http://marc.merlins.org/linux/scripts/start-btrfs-dmcrypt
> to decrypt all your devices in one swoop and mount btrfs.


I know about btrfs native raid capabilities but to be honest most of
the times I see people having "scary" problems with btrfs is when they
use it with multiple devices. So far my experience with btrfs has been
pretty smooth (always with btrfs on top of a single device) and I
wanted to let that part of btrfs to maybe mature a little bit more.
But maybe I'm wrong, so maybe I'll give both approaches a try.

About unlocking all the dm-crypt device in one swoop, there's this script too

https://github.com/gebi/keyctl_keyscript

which uses the kernel keyring to temporarily store the passphrase.
I was thinking about using it in a dm-crypt->md-raid->btrfs setting to
have one thread for each dm-crypt device, but probably aesni
instructions are fast enough
to not cause the single dm-crypt thread  in a md-raid->dm-crypt->btrfs setting
to become a bottleneck (at least with hdds, with sdds it might be a
different story)

John

Re: btrfs on software RAID0

[john terragon]

just one last doubt:

why do you use --align-payload=1024? (or 8912)
Cryptsetup man says that the default for the payload alignment is 2048
(512-byte sectors). So, it's already aligned by default to 4K-byte
physical sectors (if that was your concern). Am I missing something?

John

On Mon, May 5, 2014 at 11:25 PM, Marc MERLIN <marc@merlins.org> wrote:
> On Mon, May 05, 2014 at 10:51:46PM +0200, john terragon wrote:
>> Hi.
>> I'm about to try btrfs on an RAID0 md device (to be precise there will
>> be dm-crypt in between the md device and btrfs). If I used ext4 I
>> would set the stride and stripe_width extended options. Is there
>> anything similar I should be doing with mkfs.btrfs? Or maybe some
>> mount options beneficial to this kind of setting.
>
> This is not directly an answer to your question, so far I haven't used a
> special option like this with btrfs on my arrays although my
> undertstanding is that it's not as important as with ext4.
>
> That said, please read
> http://marc.merlins.org/perso/btrfs/post_2014-04-27_Btrfs-Multi-Device-Dmcrypt.html
>
> 1) use align-payload=1024 on cryptsetup instead of something bigger like
> 8192. This will reduce write amplification (if you're not on an SSD).
>
> 2) you don't need md0 in the middle, crypt each device and then use
> btrfs built in raid0 which will be faster (and is stable, at least as
> far as we know :) ).
>
> Then use /etc/crypttab or a script like this
> http://marc.merlins.org/linux/scripts/start-btrfs-dmcrypt
> to decrypt all your devices in one swoop and mount btrfs.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> Microsoft is to operating systems ....
>                                       .... what McDonalds is to gourmet cooking
> Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs on software RAID0

[Marc MERLIN]

On Tue, May 06, 2014 at 09:02:46AM +0200, john terragon wrote:
> just one last doubt:
> 
> why do you use --align-payload=1024? (or 8912)
> Cryptsetup man says that the default for the payload alignment is 2048
> (512-byte sectors). So, it's already aligned by default to 4K-byte
> physical sectors (if that was your concern). Am I missing something?

With 4K sectors, I agree that 2048 would be better.

What I was trying to do there is avoid write amplification.

After reading 
http://wiki.drewhess.com/wiki/Creating_an_encrypted_filesystem_on_a_partition
I went with
mdadm --create /dev/md8 --level=5 --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=/boot/bitmap-md8
which I believe required me to use
cryptsetup luksFormat --align-payload=1024 -s 256 -c aes-xts-plain64  /dev/md8
(that was with 5 drives, or 4 drives with data).

Would agree with the math?

If so, for 4K sector sizes, if we have to use align-payload=1024, in
turn I'd have to use --chunk=512. 
Does that sound right?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901