Re: FYIO: A rant about btrfs

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6059 bytes --]

On 2015-09-16 12:51, Vincent Olivier wrote:
> Hi,
>
>
>> On Sep 16, 2015, at 11:20 AM, Austin S Hemmelgarn <ahferroin7@gmail.com> wrote:
>>
>> On 2015-09-16 10:43, M G Berberich wrote:
>>> Hello,
>>>
>>> just for information. I stumbled about a rant about btrfs-performance:
>>>
>>>   http://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp
> I read it too.
>> It is worth noting a few things that were done incorrectly in this testing:
>> 1. _NEVER_ turn off write barriers (nobarrier mount option), doing so subtly breaks the data integrity guarantees of _ALL_ filesystems, but especially so on COW filesystems like BTRFS.  With this off, you will have a much higher chance that a power loss will cause data loss.  It shouldn't be turned off unless you are also turning off write-caching in the hardware or know for certain that no write-reordering is done by the hardware (and almost all modern hardware does write-reordering for performance reasons).
> But can the “nobarrier” mount option affect performances negatively for Btrfs (and not only data integrity)?
Using it improves performance for every filesystem on Linux that 
supports it.  This does not mean that it is _EVER_ a good idea to do so. 
  This mount option is one of the few things on my list of things that I 
will _NEVER_ personally provide support to people for, because it almost 
guarantees that you will lose data if the system dies unexpectedly (even 
if it's for a reason other than power loss).
>> 2. He provides no comparison of any other filesystem with TRIM support turned on (it is very likely that all filesystems will demonstrate such performance drops.  Based on that graph, it looks like the device doesn't support asynchronous trim commands).
> I think he means by the text surrounding the only graph that mentions TRIM that this exact same test on the other filesystems he benchmarked yield much better results.
Possibly, but there are also known issues with TRIM/DISCARD on BTRFS in 
4.0.  And his claim is still baseless unless he actually provides 
reference for it.
>> 3. He's testing it for a workload is a known and documented problem for BTRFS, and claiming that that means that it isn't worth considering as a general usage filesystem.  Most people don't run RDBMS servers on their systems, and as such, such a workload is not worth considering for most people.
> Apparently RDBMS being a problem on Btrfs is neither known nor documented enough (he’s right about the contrast with claiming publicly that Btrfs is indeed production ready).
OK, maybe not documented, but RDBMS falls under 'Large files with highly 
random access patterns and heavy RMW usage', which is a known issue for 
BTRFS, and also applies to VM images.
> My view on this is that having one filesystem to rule them all (all storage technologies, all use cases) is unrealistic. Also the time when you could put your NAS on an old i386 with 3MB of RAM is over. Compression, checksumming, COW, snapshotting, quotas, etc. are all computationally intensive features. In 2015 block storage has become computationally intensive. How about saying non-root Btrfs RAID10 is the best choice for a Samba NAS on rotational-HDDs with no SMR (my use case)? For root and RDBMS, I use ext4 on a M.2 SSD and with a sane initramfs and the most recent stable kernel. I am happy with the performances and delighted with the features Btrfs provides. I think it is much more productive to document and compare the most successful Btrfs deployments rather than trying to find bugs and bottlenecks for use cases that are the development focus of other filesystems. I don’t know, I might not make a lot of sense here but on top of refactoring the Gotchas, I would be happy to
 start a successful deployment story section on the wiki and maybe improve my usage of Btrfs along the way (who else here is using Btrfs in a similar fashion?).
Agreed, there's a reason that XFS was never the default in most Linux 
distributions, and similarly why there are so many filesystem drivers 
available.  Any given filesystem can have a number of arguments made 
against it, for example:
  * ZFS: Ridiculously resource hungry, and doesn't use the normal page-cache
  * XFS: filesystems can't be shrunk, and tends to perform slow under 
light load compared to most other filesystems.
  * NTFS: Poor file layout for many use cases, and clusters all the 
metadata together in one place.
  * ext*: Lacks some useful functionality (reflinks for example), and 
the file layout and aggressive journaling are usually bad for flash.
  * reiserfs: numerous gotchas in usage, and fsck loses it's mind when 
dealing with filestems that have reiserfs images stored in them as 
regular files
>> His points about the degree of performance jitter are valid however, as are the complaints of apparent CPU intensive stalls in the BTRFS code, and I occasionally see both on my own systems.
> Me too. My two cents is that focusing on improving performances for Btrfs-optimal use cases is much more interesting than bringing new features like automatically turning COW off for RDBMS usage or debugging TRIM support.
It depends, BTRFS is still not feature complete with the overall intent 
when it was started (raid56 and qgroups being the two big issues at the 
moment), and attempting to optimize things tends to introduce bugs, 
which we have quite enough of already without people adding more (and 
they still seem to be breeding like rabbits).

That said, my systems (which are usually doing mostly CPU or memory 
bound tasks, and not I/O bound like the aforementioned benchmarks were 
testing) run no slower than they did with ext4 as the main filesystem, 
and in some cases work much faster (even after averaging out the jitter 
in performance).  Based on this, I wouldn't advocate it for most server 
usage (except possibly as the root filesystem), but it does work very 
well for most desktop usage patterns and a number of HPC usage patterns 
as well.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]