Re: Questions on using BtrFS for fileserver

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3025 bytes --]

On 2014-08-19 12:21, M G Berberich wrote:
> Hello,
> 
> we are thinking about using BtrFS on standard hardware for a
> fileserver with about 50T (100T raw) of storage (25×4TByte).
> 
> This is what I understood so far. Is this right?
> 
> · incremental send/receive works.
> 
> · There is no support for hotspares (spare disks that automatically
>   replaces faulty disk).
> 
> · BtrFS with RAID1 is fairly stable.
> 
> · RAID 5/6 spreads all data over all devices, leading to performance
>   problems on large diskarrays, and there is no option to limit the
>   numbers of disk per stripe so far.
> 
> Some questions:
> 
> · There where reports, that bcache with btrfs leads to corruption. Is
>   this still so?
Based on some testing I did last month, bcache with anything has the
potential to cause data corruption.
> 
> · If a disk failes, does BtrFS rebalance automatically? (This would
>   give a a kind o hotspare behavior)
No, but it wouldn't be hard to write a simple monitoring program to do
this from userspace.  IIRC, the big issue is that you need to add a
device in-place of the failed one for the re-balance to work.
> 
> · Besides using bcache, are there any possibilities to boost
>   performance by adding (dedicated) cache-SSDs to a BtrFS?
Like mentioned in one of the other responses, I would suggest looking
into dm-cache.  BTRFS itself does not have any functionality for this,
although there has been talk of implementing device priorities for
reads, which could provide a similar performance boost.
> 
> · Are there any reports/papers/web-pages about BtrFS-systems this size
>   in use? Praises, complains, performance-reviews, whatever…
While it doesn't quite fit the description, I have had very good success
with a very active 2TB BTRFS RAID10 filesystem consisting of BTRFS on
four unpartitioned 1TB SATA III hard drives.  The filesystem gets in
excess of 100GB of data written to it each day (almost all rewrites
however), and is what I use for /home, /var/log, and /var/lib, and I've
had no issues with it that were caused by BTRFS, and in-fact, the very
fact that it uses BTRFS helped me recover data when the storage
controller they are connected to went bad.  On average, I get about 125%
of raw disk performance on writes, and about 110% on reads.

If you are using a very large number of disks, then I would not suggest
that you use BTRFS RAID10, but instead BTRFS RAID1, as RAID10 will try
to stripe things across ALL of the devices in the filesystem, and unless
you have no more than about four times as many disks as storage
controllers (that is, each controller has no more than four disks
attached to it), the overhead outweighs the benefit of striping the data.

Also, just to make sure it's clear, in BTRFS RAID1, each block gets
written EXACTLY twice.  On the plus side though, this means that if you
do set-up a caching mechanism, you may be able to keep most of the array
spun down a majority of the time.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]