Btrfs is amazing! (a lack-of-bug report)

Btrfs is amazing! (a lack-of-bug report)

[Tyler Bletsch]

I just wanted to drop you guys a line to say that I am stunned with how 
excellent btrfs is. I did some testing, and the things that it did were 
amazing. I took a 4-disk RAID 5 and walked it all the way down to a 
one-disk volume and back again, mixed in devices of different sizes in 
different modes, balanced it in every direction, trashed data on drives 
without the OS knowing, and did every other form of torture I could 
think of, all while looping file integrity tests, and it was perfect.

The ease of use and simplicity were great.  I was dreading having to 
administer ZFS in order to get snapshots and other features, but now I 
don't have to. With the exception of enterprisey features like SSD 
intent logs and stuff, it is hands down far better than ZFS.

Thanks for the great work!

Regards,
Tyler Bletsch

Re: Btrfs is amazing! (a lack-of-bug report)

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On 2015-08-16 23:35, Tyler Bletsch wrote:
> I just wanted to drop you guys a line to say that I am stunned with how
> excellent btrfs is. I did some testing, and the things that it did were
> amazing. I took a 4-disk RAID 5 and walked it all the way down to a
> one-disk volume and back again, mixed in devices of different sizes in
> different modes, balanced it in every direction, trashed data on drives
> without the OS knowing, and did every other form of torture I could
> think of, all while looping file integrity tests, and it was perfect.
>
> The ease of use and simplicity were great.  I was dreading having to
> administer ZFS in order to get snapshots and other features, but now I
> don't have to. With the exception of enterprisey features like SSD
> intent logs and stuff, it is hands down far better than ZFS.
>
> Thanks for the great work!
>
It's nice to hear success stories for once.  I would suggest being 
careful of using BTRFS raid5 or raid6 in production, there are probably 
still bugs that haven't yet been discovered.  Secondarily, if you can 
deal with slightly more setup and maintenance overhead, BTRFS works 
_very_ well on top of LVM (it makes online data migration much easier, 
and provides easy ways to do layered RAID setups).



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Btrfs is amazing! (a lack-of-bug report)

[Tyler Bletsch]

Thanks. I will be trying raid5 in production, but "production" in this 
case just means my home file server, with btrfs snapshot+sync for all 
data and appropriate offsite non-btrfs backups for critical data. If it 
hoses up, I'll post a bug report.

Going to try to avoid LVM, since half the appeal of btrfs for me is 
getting away from the multiple duct-taped layers of indirection that I 
you get currently with ext4/MD/LVM setups.

- Tyler

On 8/17/2015 7:48 AM, Austin S Hemmelgarn wrote:
> On 2015-08-16 23:35, Tyler Bletsch wrote:
>> I just wanted to drop you guys a line to say that I am stunned with how
>> excellent btrfs is. I did some testing, and the things that it did were
>> amazing. I took a 4-disk RAID 5 and walked it all the way down to a
>> one-disk volume and back again, mixed in devices of different sizes in
>> different modes, balanced it in every direction, trashed data on drives
>> without the OS knowing, and did every other form of torture I could
>> think of, all while looping file integrity tests, and it was perfect.
>>
>> The ease of use and simplicity were great.  I was dreading having to
>> administer ZFS in order to get snapshots and other features, but now I
>> don't have to. With the exception of enterprisey features like SSD
>> intent logs and stuff, it is hands down far better than ZFS.
>>
>> Thanks for the great work!
>>
> It's nice to hear success stories for once.  I would suggest being 
> careful of using BTRFS raid5 or raid6 in production, there are 
> probably still bugs that haven't yet been discovered. Secondarily, if 
> you can deal with slightly more setup and maintenance overhead, BTRFS 
> works _very_ well on top of LVM (it makes online data migration much 
> easier, and provides easy ways to do layered RAID setups).
>
>

Re: Btrfs is amazing! (a lack-of-bug report)

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

On 2015-08-17 15:18, Tyler Bletsch wrote:
> Thanks. I will be trying raid5 in production, but "production" in this
> case just means my home file server, with btrfs snapshot+sync for all
> data and appropriate offsite non-btrfs backups for critical data. If it
> hoses up, I'll post a bug report.
So far, that's been my use case for btrfs raid6, and (barring one bug I 
found involving an interaction between btrfs and thinly provisioned 
storage that shouldn't be an issue for you if you're not using LVM thin 
pools) I have yet to hit any bugs, although based on the lists, I've 
probably been _very_ lucky in that respect.  If you're willing to take a 
marginal performance hit (about 1-3% in my experience, which is notably 
less than the performance difference between MD-RAID5 and MD-RAID6), and 
have at least four disks, I'd suggest using btrfs's raid6 profile 
instead of raid5, they use exactly the same code, it's just that raid6 
has one more calculation involved and provides better protection against 
data corruption due to the double parity.
>
> Going to try to avoid LVM, since half the appeal of btrfs for me is
> getting away from the multiple duct-taped layers of indirection that I
> you get currently with ext4/MD/LVM setups.
Understandable, my main reasons for having LVM are storing virtual 
machine disk images and the fact that MD/DM raid is still ridiculously 
fast compared to btrfs raid (many of my btrfs raid volumes are 
themselves on top of LVM managed software raid0 or raid1 volumes).



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Btrfs is amazing! (a lack-of-bug report)

[Tyler Bletsch]

Thanks.  I'd consider raid6, but since I'll be backing up to a second 
btrfs raid5 array, I think I have sufficient redundancy, since 
equivalent to raid 5+1 on paper. I'm doing that rather than something 
like raid10 in a single box because I want the redundancy of a second 
physical server so I can failover in the event of a system-level 
component failure.

(And of course, "failover" means "continue being able to watch TV shows 
and stuff")

A question about what you said -- when you say people have hit bugs in 
the raid56 code, which flavor do these bugs tend to be? Are they 
"minding my own business and suddenly it falls over" bugs or "I tried to 
do something weird with btrfs and it screwed up" bugs?

Thanks,
Tyler

On 8/18/2015 7:42 AM, Austin S Hemmelgarn wrote:
> On 2015-08-17 15:18, Tyler Bletsch wrote:
>> Thanks. I will be trying raid5 in production, but "production" in this
>> case just means my home file server, with btrfs snapshot+sync for all
>> data and appropriate offsite non-btrfs backups for critical data. If it
>> hoses up, I'll post a bug report.
> So far, that's been my use case for btrfs raid6, and (barring one bug 
> I found involving an interaction between btrfs and thinly provisioned 
> storage that shouldn't be an issue for you if you're not using LVM 
> thin pools) I have yet to hit any bugs, although based on the lists, 
> I've probably been _very_ lucky in that respect.  If you're willing to 
> take a marginal performance hit (about 1-3% in my experience, which is 
> notably less than the performance difference between MD-RAID5 and 
> MD-RAID6), and have at least four disks, I'd suggest using btrfs's 
> raid6 profile instead of raid5, they use exactly the same code, it's 
> just that raid6 has one more calculation involved and provides better 
> protection against data corruption due to the double parity.
>>
>> Going to try to avoid LVM, since half the appeal of btrfs for me is
>> getting away from the multiple duct-taped layers of indirection that I
>> you get currently with ext4/MD/LVM setups.
> Understandable, my main reasons for having LVM are storing virtual 
> machine disk images and the fact that MD/DM raid is still ridiculously 
> fast compared to btrfs raid (many of my btrfs raid volumes are 
> themselves on top of LVM managed software raid0 or raid1 volumes).
>
>

Re: Btrfs is amazing! (a lack-of-bug report)

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1484 bytes --]

On 2015-08-19 13:24, Tyler Bletsch wrote:
> Thanks.  I'd consider raid6, but since I'll be backing up to a second
> btrfs raid5 array, I think I have sufficient redundancy, since
> equivalent to raid 5+1 on paper. I'm doing that rather than something
> like raid10 in a single box because I want the redundancy of a second
> physical server so I can failover in the event of a system-level
> component failure.
>
> (And of course, "failover" means "continue being able to watch TV shows
> and stuff")
>
> A question about what you said -- when you say people have hit bugs in
> the raid56 code, which flavor do these bugs tend to be? Are they
> "minding my own business and suddenly it falls over" bugs or "I tried to
> do something weird with btrfs and it screwed up" bugs?
More along the lines of 'I tried to do something that works fine with 
the other raid profiles and it kind of messed up the filesystem'.  In 
general, you should be safe as long as you are using at least Linux 4.0 
and the most recent version of btrfs-progs.  It's been a while since I 
saw any raid56 related bugs that caused actual data loss.  If you are 
using this on SSD's though, I would wait, there are known issues with 
DISCARD/TRIM not working correctly on btrfs right now (nothing involving 
data loss, just problems with it not properly trimming free space and 
therefore causing issues with wear-leveling), and it looks like the fix 
won't be in 4.2 as of right now.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Btrfs is amazing! (a lack-of-bug report)

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1715 bytes --]

On 2015-08-20 07:52, Austin S Hemmelgarn wrote:
> On 2015-08-19 13:24, Tyler Bletsch wrote:
>> Thanks.  I'd consider raid6, but since I'll be backing up to a second
>> btrfs raid5 array, I think I have sufficient redundancy, since
>> equivalent to raid 5+1 on paper. I'm doing that rather than something
>> like raid10 in a single box because I want the redundancy of a second
>> physical server so I can failover in the event of a system-level
>> component failure.
>>
>> (And of course, "failover" means "continue being able to watch TV shows
>> and stuff")
>>
>> A question about what you said -- when you say people have hit bugs in
>> the raid56 code, which flavor do these bugs tend to be? Are they
>> "minding my own business and suddenly it falls over" bugs or "I tried to
>> do something weird with btrfs and it screwed up" bugs?
> More along the lines of 'I tried to do something that works fine with
> the other raid profiles and it kind of messed up the filesystem'.  In
> general, you should be safe as long as you are using at least Linux 4.0
> and the most recent version of btrfs-progs.  It's been a while since I
> saw any raid56 related bugs that caused actual data loss.  If you are
> using this on SSD's though, I would wait, there are known issues with
> DISCARD/TRIM not working correctly on btrfs right now (nothing involving
> data loss, just problems with it not properly trimming free space and
> therefore causing issues with wear-leveling), and it looks like the fix
> won't be in 4.2 as of right now.
>
>
On second thought, you might want to wait until 4.3, I just saw this thread:
http://thread.gmane.org/gmane.comp.file-systems.btrfs/47321/focus=47325


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Btrfs is amazing! (a lack-of-bug report)

[Tyler Bletsch]

(Resending to list as plaintext (*correctly* this time))

I see. I'll probably make the backup array a raid10 then.

If/when I do see a disk failure on the raid5, are there any specific 
steps it would be helpful for me to take to capture the state so you 
folks can have a useful bug report?

I plan to run the latest stock kernel from the mainline kernel ppa on 
Ubuntu, with btrfs-progs coming from the git.

- Tyler

On 8/20/2015 8:16 AM, Donald Pearson wrote:
>
> Raid56 works fine until you have a drive with problems which really 
> means it doesn't work because you only use parity to handle the case 
> of a drive with problems.
>
> Maintenance procedures such as scrubs are also a magnitude of order 
> slower than the other raid profiles.
>
> I would use the raid10 profile on at least one of your pools.
>
> On Aug 20, 2015 7:03 AM, "Austin S Hemmelgarn" <ahferroin7@gmail.com 
> <mailto:ahferroin7@gmail.com>> wrote:
>
>     On 2015-08-20 07:52, Austin S Hemmelgarn wrote:
>
>         On 2015-08-19 13:24, Tyler Bletsch wrote:
>
>             Thanks.  I'd consider raid6, but since I'll be backing up
>             to a second
>             btrfs raid5 array, I think I have sufficient redundancy, since
>             equivalent to raid 5+1 on paper. I'm doing that rather
>             than something
>             like raid10 in a single box because I want the redundancy
>             of a second
>             physical server so I can failover in the event of a
>             system-level
>             component failure.
>
>             (And of course, "failover" means "continue being able to
>             watch TV shows
>             and stuff")
>
>             A question about what you said -- when you say people have
>             hit bugs in
>             the raid56 code, which flavor do these bugs tend to be?
>             Are they
>             "minding my own business and suddenly it falls over" bugs
>             or "I tried to
>             do something weird with btrfs and it screwed up" bugs?
>
>         More along the lines of 'I tried to do something that works
>         fine with
>         the other raid profiles and it kind of messed up the
>         filesystem'.  In
>         general, you should be safe as long as you are using at least
>         Linux 4.0
>         and the most recent version of btrfs-progs.  It's been a while
>         since I
>         saw any raid56 related bugs that caused actual data loss. If
>         you are
>         using this on SSD's though, I would wait, there are known
>         issues with
>         DISCARD/TRIM not working correctly on btrfs right now (nothing
>         involving
>         data loss, just problems with it not properly trimming free
>         space and
>         therefore causing issues with wear-leveling), and it looks
>         like the fix
>         won't be in 4.2 as of right now.
>
>
>     On second thought, you might want to wait until 4.3, I just saw
>     this thread:
>     http://thread.gmane.org/gmane.comp.file-systems.btrfs/47321/focus=47325
>