BTRFS, SSD and single metadata

BTRFS, SSD and single metadata

[Swâmi Petaramesh]

Hi,

I created a BTRFS filesytem over LVM over LUKS encryption on an SSD [yes, I 
know...], and I noticed that the FS got created with metadata in "DUP" mode, 
contrary to what "man mkfs.btrfs" says for SSDs -> it would be supposed to be 
"SINGLE"...

Well I don't know if my system didn't identify the SSD because of the LVM+LUKS 
stack (however it mounts well by itself with the "ssd" flag and accepts the 
"discard" option [yes, I know...]), or if the manpage is obsolete or if this 
feature just doesn't work...?

The SSD being a Micron RealSSD C400

For both SSD preservation and data integrity, would it be advisable to change 
metadata to "SINGLE" using a rebalance, or if I'd better just leave things the 
way they are...?

TIA for any insight.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Tout le malheur des hommes vient de ce qu'ils ne vivent pas dans _le_ monde,
mais dans _leur_ monde.
-- Héraclite.

Re: BTRFS, SSD and single metadata

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1204 bytes --]

On 2014-06-16 03:54, Swâmi Petaramesh wrote:
> Hi,
> 
> I created a BTRFS filesytem over LVM over LUKS encryption on an SSD [yes, I 
> know...], and I noticed that the FS got created with metadata in "DUP" mode, 
> contrary to what "man mkfs.btrfs" says for SSDs -> it would be supposed to be 
> "SINGLE"...
> 
> Well I don't know if my system didn't identify the SSD because of the LVM+LUKS 
> stack (however it mounts well by itself with the "ssd" flag and accepts the 
> "discard" option [yes, I know...]), or if the manpage is obsolete or if this 
> feature just doesn't work...?
> 
> The SSD being a Micron RealSSD C400
> 
> For both SSD preservation and data integrity, would it be advisable to change 
> metadata to "SINGLE" using a rebalance, or if I'd better just leave things the 
> way they are...?
> 
> TIA for any insight.
> 
What mkfs.btrfs looks at is
/sys/block/<whatever-device>/queue/rotational, if that is 1 it knows
that the device isn't a SSD.  I believe that LVM passes through whatever
the next lower layer's value is, but dmcrypt (and by extension LUKS)
always force it to a 1 (possibly to prevent programs from using
heuristics for enabling discard)


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: BTRFS, SSD and single metadata

[Swâmi Petaramesh]

Hi Austin, and thanks for your reply.

Le lundi 16 juin 2014, 07:09:55 Austin S Hemmelgarn a écrit :
> 
> What mkfs.btrfs looks at is
> /sys/block/<whatever-device>/queue/rotational, if that is 1 it knows
> that the device isn't a SSD.  I believe that LVM passes through whatever
> the next lower layer's value is, but dmcrypt (and by extension LUKS)
> always force it to a 1 (possibly to prevent programs from using
> heuristics for enabling discard)

In the current "running condition", the system clearly sees this is *not* 
rotational, even thru the LVM/dmcrypt stack :

# mount | grep btrfs
/dev/mapper/VG-LINUX on / type btrfs 
(rw,noatime,seclabel,compress=lzo,ssd,discard,space_cache,autodefrag)

# ll /dev/mapper/VGV-LINUX
lrwxrwxrwx. 1 root root 7 16 juin  09:21 /dev/mapper/VG-LINUX -> ../dm-1

# cat /sys/block/dm-1/queue/rotational 
0

...However, at mkfs.btrfs time, it migth well not have seen it, as I made it 
from a live USB key in which both the lvm.conf and crypttab had not been 
taylored to allow "trim" commands...

However, now that the FS is created, I still wonder whether I should use a 
rebalance to change the metadata from DUP to SINGLE, or if Id' better stay 
with DUP...

Kind regards.


-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: BTRFS, SSD and single metadata

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

On 2014-06-16 07:18, Swâmi Petaramesh wrote:
> Hi Austin, and thanks for your reply.
> 
> Le lundi 16 juin 2014, 07:09:55 Austin S Hemmelgarn a écrit :
>>
>> What mkfs.btrfs looks at is
>> /sys/block/<whatever-device>/queue/rotational, if that is 1 it knows
>> that the device isn't a SSD.  I believe that LVM passes through whatever
>> the next lower layer's value is, but dmcrypt (and by extension LUKS)
>> always force it to a 1 (possibly to prevent programs from using
>> heuristics for enabling discard)
> 
> In the current "running condition", the system clearly sees this is *not* 
> rotational, even thru the LVM/dmcrypt stack :
> 
> # mount | grep btrfs
> /dev/mapper/VG-LINUX on / type btrfs 
> (rw,noatime,seclabel,compress=lzo,ssd,discard,space_cache,autodefrag)
> 
> # ll /dev/mapper/VGV-LINUX
> lrwxrwxrwx. 1 root root 7 16 juin  09:21 /dev/mapper/VG-LINUX -> ../dm-1
> 
> # cat /sys/block/dm-1/queue/rotational 
> 0
> 
> ...However, at mkfs.btrfs time, it migth well not have seen it, as I made it 
> from a live USB key in which both the lvm.conf and crypttab had not been 
> taylored to allow "trim" commands...
> 
> However, now that the FS is created, I still wonder whether I should use a 
> rebalance to change the metadata from DUP to SINGLE, or if Id' better stay 
> with DUP...
> 
> Kind regards.
> 
> 
I'd personally stay with the DUP profile, but then that's just me being
paranoid.  You will almost certainly get better performance using the
SINGLE profile instead of DUP, but this is mostly due to it requiring
fewer blocks to be encrypted by LUKS (Which is almost certainly your
primary bottleneck unless you have some high-end crypto-accelerator card).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2967 bytes --]

Re: BTRFS, SSD and single metadata

[Russell Coker]

On Mon, 16 Jun 2014 07:23:14 Austin S Hemmelgarn wrote:
> I'd personally stay with the DUP profile, but then that's just me being
> paranoid.  You will almost certainly get better performance using the
> SINGLE profile instead of DUP, but this is mostly due to it requiring
> fewer blocks to be encrypted by LUKS (Which is almost certainly your
> primary bottleneck unless you have some high-end crypto-accelerator card).

On my Q8400 workstation running BTRFS over LUKS on an Intel SSD the primary 
bottleneck has always been BTRFS.  The message I wrote earlier today about 
BTRFS fallocate() performance was on this system, I had BTRFS using kernel CPU 
time for periods of 10+ seconds without ANY disk IO - so LUKS wasn't a 
performance issue.

So far I've never seen LUKS be a performance bottleneck.  When running LUKS on 
spinning media the disk seek performance will almost always be the bottleneck.

The worst case for LUKS is transferring large amounts of data such as 
contiguous reads.  In a contiguous read test I'm seeing 120MB/s for LUKS on a 
SSD and 200MB/s for direct access to the same SSD.  That is a reasonable 
difference, but it's not something I've been able to hit with any real-world 
use while BTRFS metadata performance is often an issue.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: BTRFS, SSD and single metadata

[Duncan]

Swâmi Petaramesh posted on Mon, 16 Jun 2014 09:54:01 +0200 as excerpted:

> I created a BTRFS filesytem over LVM over LUKS encryption on an SSD
> [yes, I know...], and I noticed that the FS got created with metadata in
> "DUP" mode, contrary to what "man mkfs.btrfs" says for SSDs -> it would
> be supposed to be "SINGLE"...
> 
> Well I don't know if my system didn't identify the SSD because of the
> LVM+LUKS stack (however it mounts well by itself with the "ssd" flag and
> accepts the "discard" option [yes, I know...]), or if the manpage is
> obsolete or if this feature just doesn't work...?

Does btrfs automatically add the ssd mount option or do you have to add 
it?  If you have to add it, that means btrfs isn't detecting the ssd, 
which would explain why mkfs.btrfs didn't detect it either... as you said 
very likely due to the LVM over LUKS stack.

I believe the detection is actually based on what the kernel reports.  I 
may be mistaken and I'm not running stacked devices ATM in ordered to 
check them, but check /sys/block/<device>/queue/rotational.  On my ssds, 
the value in that file is 0.  On my spinning rust, it's 1.  If that is 
indeed what it's looking at, you can verify that your hardware device is 
actually detected by the kernel as rotational 0, and then check the 
various layers of your stack and see where the rotational 0 (or the 
rotational file itself) gets dropped.

> The SSD being a Micron RealSSD C400
> 
> For both SSD preservation and data integrity, would it be advisable to
> change metadata to "SINGLE" using a rebalance, or if I'd better just
> leave things the way they are...?
> 
> TIA for any insight.

That's a very good question, on which there has been some debate on this 
list.

The first question:  Does your SSD firmware do compression and dedup, or 
not?  IIRC the sandforce (I believe that's the firmware name) firmware 
does compression and dedup, but not all firmware does.  I know a bullet-
point feature of my SSDs (several Corsair Neutron 256 GB, FWIW) is that 
they do NOT do this sort of compression and dedup -- what the system 
tells them to save is what they save, and if you tell it to save a 
hundred copies of the same thing, that's what it does.  (These SSDs are 
targeted at commercial usage and this is billed as a performance 
reliability feature.)

The reason originally given for defaulting to single mode metadata on 
SSDs was that it was due to this possible dedup -- dup-mode metadata 
might actually end up single-copy-only due to the firmware compression 
and dedup in any case.  Between that and their typically smaller size, I 
guess the decision was that single was the best default.

However, it occurs to me that with the LUKS encryption layer, I'm not 
entirely sure if duplication at the btrfs level would end up as the same 
encrypted stream headed to the hardware in any case.  If it would encrypt 
the two copies of the dup-mode metadata as different, then the hardware 
dedup/compression wouldn't work on it anyway.  OTOH, if it encrypts them 
as the same stream headed to hardware, then again, it would matter.

Meanwhile... which is better and should you rebalance to single?  In 
terms of data integrity, dup mode is definitely better, since if there's 
damage to one copy such that it doesn't pass checksum verification, you 
still have the other copy to read from... and to rebuild the damaged copy 
from.

OTOH, single does take less space, and performance should be slightly 
better.  If you're keeping good backups anyway, or if the ssd's firmware 
might be mucking with things leaving you with only a single copy in any 
case, single mode could be a better choice.


FWIW, while most of my partitions are btrfs raid1 here, so the second 
copy is on a different physical device, /boot is an exception.  I have a 
separate /boot on each device, pointed at by the grub loaded on each 
device so I can use the BIOS boot selector to choose which one I boot.  
That lets me keep a working /boot that I boot most of the time on one 
device, and a backup /boot on the other device, in case something goes 
wrong with the first.

So those dedicated /boot partitions are an exception to my normal btrfs 
raid1.  They're both (working and primary backup) 256 MiB mixed-data/
metadata mode as they're so small, which means data and metadata must 
both be the same mode, and they're both set to dup mode.

Which means they effectively only hold (a bit under) 128 MiB worth of 
data, since it's dup mode for both data and metadata, but 128 MiB is fine 
for /boot, as long as I don't let too many kernels build up.

So I'd obviously recommend dup mode, unless you know your ssd's firmware 
is going to dedup it anyway.  But that's just me.  As I said, there has 
been some discussion about it on the list, and some people make other 
choices.  You can of course dig in the list archives if you want to see 
previous threads on the subject, but ultimately, it's upto you. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS, SSD and single metadata

[Swâmi Petaramesh]

Le lundi 16 juin 2014, 12:16:33 Duncan a écrit :
> Does btrfs automatically add the ssd mount option or do you have to add 
> it?  If you have to add it, that means btrfs isn't detecting the ssd, 

First time I mounted the freshly created filesystem, it actually added the 
"ssd" option by itself. Thus indeed, it could see this was a SSD...

> However, it occurs to me that with the LUKS encryption layer, I'm not 
> entirely sure if duplication at the btrfs level would end up as the same 
> encrypted stream headed to the hardware in any case.  If it would encrypt 
> the two copies of the dup-mode metadata as different, then the hardware 
> dedup/compression wouldn't work on it anyway.  OTOH, if it encrypts them 
> as the same stream headed to hardware, then again, it would matter.

That makes an excellent point. 2 copies of the same binary data in different 
filesystem sectors, process thru LUKS, will create entirely different binary 
ciphertext, so for sure both metadata copies will always be different and 
cannot be "deduped" by SSD firmware...

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Les êtres humains, cernés par leurs besoins, s'agitent sans but comme des
lapins pris au piège. Ainsi, moine, détourne-toi de ces besoins et trouve
la liberté.
-- Buddha Shakyamuni