Response to Bcachefs Claims

Response to Bcachefs Claims

[Vincent Olivier]

Hi,

I have been using Btrfs for almost a year now with a 16x4TB RAID10 and its 8x4TB RAID0 backup (using incremental snapshots diffs). I have always tried to stay at the latest stable kernel (currently 4.1.6). But I might be moving to Fedora 22 because Centos 7 has significant incompatibilities with the 4.1.x kernel series.

I have seen the news about Bcachefs aiming to be Btrfs-complete while being extX-stable.

What are the chances Bcachefs beats Btrfs at being the Linux kernel's next "official" file system ? I chose Btrfs over ZFS because it seemed like the only "next-gen" heir to ext4/xfs.

I have been having a few problems with Btrfs myself. I have only one that remains unresolved : I haven't found the best way to mount Btrfs at boot time. "LABEL=" won't work for known reasons (I don't understand however why a mount can't do its own "device scan" transparently). "UUID=" won't work for unknown reasons (haven't got a reply on this, maybe it's the same as "LABEL="). And I will use /dev/* in fstab for stability reasons. Right now I'm mounting the fs manually after a "device scan" and picking up the first device that shows up in the "fi show" run. I can "live" with that but I suppose that things like this contribute to the feeling that Btrfs is actually still experimental contrarily to claims that it is production-ready.

For my own sake and other's I would like to maintain (if nobody is already working on that nor needs any help) a centralized human-readable digest of known issues that would be featured prominently on top of the Btrfs wiki. I would merge the Gotchas page and the various known issues pages (including the various multi-device mount gotchas here and there).

Answers ? Comments ? Help ?

Thanks,

Vincent

Re: Response to Bcachefs Claims

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2962 bytes --]

On 2015-08-25 11:22, Vincent Olivier wrote:
> Hi,
>
> I have been using Btrfs for almost a year now with a 16x4TB RAID10 and its 8x4TB RAID0 backup (using incremental snapshots diffs). I have always tried to stay at the latest stable kernel (currently 4.1.6). But I might be moving to Fedora 22 because Centos 7 has significant incompatibilities with the 4.1.x kernel series.
>
> I have seen the news about Bcachefs aiming to be Btrfs-complete while being extX-stable.
>
> What are the chances Bcachefs beats Btrfs at being the Linux kernel's next "official" file system ? I chose Btrfs over ZFS because it seemed like the only "next-gen" heir to ext4/xfs.
>
> I have been having a few problems with Btrfs myself. I have only one that remains unresolved : I haven't found the best way to mount Btrfs at boot time. "LABEL=" won't work for known reasons (I don't understand however why a mount can't do its own "device scan" transparently). "UUID=" won't work for unknown reasons (haven't got a reply on this, maybe it's the same as "LABEL="). And I will use /dev/* in fstab for stability reasons. Right now I'm mounting the fs manually after a "device scan" and picking up the first device that shows up in the "fi show" run. I can "live" with that but I suppose that things like this contribute to the feeling that Btrfs is actually still experimental contrarily to claims that it is production-ready.
>
> For my own sake and other's I would like to maintain (if nobody is already working on that nor needs any help) a centralized human-readable digest of known issues that would be featured prominently on top of the Btrfs wiki. I would merge the Gotchas page and the various known issues pages (including the various multi-device mount gotchas here and there).
>
> Answers ? Comments ? Help ?
First off, I think this is a wonderful idea.  The list of known issues 
isn't always particularly up to date, and isn't as trivial to find as it 
should be.

Secondly, as far as UUID= not working from the kernel commandline, I 
believe it's for the same reason that LABEL= doesn't work consistently. 
  I would like to point out that LABEL= does work if you use a sane 
initramfs (I boot my desktop using it, although I'm only using an 
initramfs because I have root on a pair of LVM volumes), and I _think_ 
it might also work if you pas device= options in the rootoptions= string.

Thirdly, there is one issue that I haven't seen anyone else mention and 
that is not present on any of the wiki pages at last check:
If you try to use BTRFS on top of LVM thin-provisioning (or 
theoretically any dm-thinp setup), and turn off zeroing of newly 
allocated blocks, this can cause BTRFS to have corruption issues.  I 
think this is actually a dm-thinp issue, not a BTRFS one (turning this 
option off does not clear the discard_zeroes_data flag for the device, 
which I would assume that BTRFS trusts), but is still worth mentioning.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Response to Bcachefs Claims

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

On Tue, 25 Aug 2015 11:22:34 -0400 (EDT)
"Vincent Olivier" <vincent@up4.com> wrote:

> "UUID=" won't work for unknown reasons (haven't got a reply on this, maybe
> it's the same as "LABEL="). And I will use /dev/* in fstab for stability
> reasons. 

Take a look at the contents of /dev/disk/by-id/, specifically ata-*
These symlinks are made from HDD model and serial numbers, so can be relied on
to remain stable, no matter in which order your HDDs appear in the system.
Maybe these will come handy for your fstab, I used them a lot and still do,
especially for swap partitions.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Response to Bcachefs Claims

[Chris Mason]

On Tue, Aug 25, 2015 at 11:22:34AM -0400, Vincent Olivier wrote:
> Hi,
> 
> I have been using Btrfs for almost a year now with a 16x4TB RAID10 and
> its 8x4TB RAID0 backup (using incremental snapshots diffs). I have
> always tried to stay at the latest stable kernel (currently 4.1.6).
> But I might be moving to Fedora 22 because Centos 7 has significant
> incompatibilities with the 4.1.x kernel series.
> 
> I have seen the news about Bcachefs aiming to be Btrfs-complete while
> being extX-stable.
> 
> What are the chances Bcachefs beats Btrfs at being the Linux kernel's
> next "official" file system ? I chose Btrfs over ZFS because it seemed
> like the only "next-gen" heir to ext4/xfs.

Kent is a smart guy, and I'm always glad to see more work in Linux
filesystems.  We all work on the parts we think are important and people
use the best tool they can find for their workloads.  Sometimes that'll
be btrfs, sometimes not.  Btrfs has certainly come a long way and is
continuing to improve pretty dramatically.

> 
> I have been having a few problems with Btrfs myself. I have only one
> that remains unresolved : I haven't found the best way to mount Btrfs
> at boot time. "LABEL=" won't work for known reasons (I don't
> understand however why a mount can't do its own "device scan"
> transparently). "UUID=" won't work for unknown reasons (haven't got a
> reply on this, maybe it's the same as "LABEL="). And I will use /dev/*
> in fstab for stability reasons. Right now I'm mounting the fs manually
> after a "device scan" and picking up the first device that shows up in
> the "fi show" run. I can "live" with that but I suppose that things
> like this contribute to the feeling that Btrfs is actually still
> experimental contrarily to claims that it is production-ready.
> 

LABEL= and UUID= both need a scan on multi device filesystems.  If your
initrd does it, things will work fine.

-chris

Re: Response to Bcachefs Claims

[Chris Murphy]

On Tue, Aug 25, 2015 at 9:22 AM, Vincent Olivier <vincent@up4.com> wrote:

> For my own sake and other's I would like to maintain (if nobody is already working on that nor needs any help) a centralized human-readable digest of known issues that would be featured prominently on top of the Btrfs wiki. I would merge the Gotchas page and the various known issues pages (including the various multi-device mount gotchas here and there).

Some way to organize problems by distribution would be needed to be complete.

For example I don't have the discover & mount by label or uuid problem
you mention on Fedora, since forever. I haven't experienced it on
Fedora 19/20 which is approximately what CentOS 7 is based on. To
figure that out means using one or more boot parameters:
systemd.log_level=debug or rd.udev.debug or rd.debug to find out
what's happening.

The one problem case I still have with the latest versions is multiple
device Btrfs volume UUID doesn't exist when 1 or more devices are
missing.


-- 
Chris Murphy

Re: Response to Bcachefs Claims

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

On 2015-08-25 12:13, Roman Mamedov wrote:
> On Tue, 25 Aug 2015 11:22:34 -0400 (EDT)
> "Vincent Olivier" <vincent@up4.com> wrote:
>
>> "UUID=" won't work for unknown reasons (haven't got a reply on this, maybe
>> it's the same as "LABEL="). And I will use /dev/* in fstab for stability
>> reasons.
>
> Take a look at the contents of /dev/disk/by-id/, specifically ata-*
> These symlinks are made from HDD model and serial numbers, so can be relied on
> to remain stable, no matter in which order your HDDs appear in the system.
> Maybe these will come handy for your fstab, I used them a lot and still do,
> especially for swap partitions.
>
Unless things have changed significantly recently, these are all created 
by udev, and therefore useless unless you are using an initramfs or 
userspace is already up.  They are really useful however if you have to 
boot into a recovery system (which is 90% of where I use them).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Response to Bcachefs Claims

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

On 2015-08-25 12:39, Chris Murphy wrote:
> On Tue, Aug 25, 2015 at 9:22 AM, Vincent Olivier <vincent@up4.com> wrote:
>
>> For my own sake and other's I would like to maintain (if nobody is already working on that nor needs any help) a centralized human-readable digest of known issues that would be featured prominently on top of the Btrfs wiki. I would merge the Gotchas page and the various known issues pages (including the various multi-device mount gotchas here and there).
>
> Some way to organize problems by distribution would be needed to be complete.
>
> For example I don't have the discover & mount by label or uuid problem
> you mention on Fedora, since forever. I haven't experienced it on
> Fedora 19/20 which is approximately what CentOS 7 is based on. To
> figure that out means using one or more boot parameters:
> systemd.log_level=debug or rd.udev.debug or rd.debug to find out
> what's happening.
That would be because the Fedora initrd does a device scan before trying 
to mount root.  Most of the big distro's do this now, but of course you 
need to be using an initrd for it to work.
> The one problem case I still have with the latest versions is multiple
> device Btrfs volume UUID doesn't exist when 1 or more devices are
> missing.
I've seen this myself, but only sometimes.  Based on my own testing, it 
only seems to happen when one of the disks with a system chunk on it is 
missing (for example, in a 3 device raid1 setup, one of the disks won't 
have a system chunk on it, and if disk goes missing, I can still mount 
by UUID fine).



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Response to Bcachefs Claims

[Chris Murphy]

On Tue, Aug 25, 2015 at 11:59 AM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-08-25 12:39, Chris Murphy wrote:

>> The one problem case I still have with the latest versions is multiple
>> device Btrfs volume UUID doesn't exist when 1 or more devices are
>> missing.
>
> I've seen this myself, but only sometimes.  Based on my own testing, it only
> seems to happen when one of the disks with a system chunk on it is missing
> (for example, in a 3 device raid1 setup, one of the disks won't have a
> system chunk on it, and if disk goes missing, I can still mount by UUID
> fine).

I'm virtually certain I've reproduced this with a 2 disk raid1, both
should have system chunks. Separately, it seems to me that every disk
should have a copy of the system chunk, i.e. make it fixed as either
DUP on single or n-copies raid1 when on multiple devices. *shrug*



-- 
Chris Murphy

Re: Response to Bcachefs Claims

[Suman Chakravartula]

On Tue, Aug 25, 2015 at 8:53 AM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-08-25 11:22, Vincent Olivier wrote:
>>
>> Hi,
>>
>> I have been using Btrfs for almost a year now with a 16x4TB RAID10 and its
>> 8x4TB RAID0 backup (using incremental snapshots diffs). I have always tried
>> to stay at the latest stable kernel (currently 4.1.6). But I might be moving
>> to Fedora 22 because Centos 7 has significant incompatibilities with the
>> 4.1.x kernel series.
>>
>> I have seen the news about Bcachefs aiming to be Btrfs-complete while
>> being extX-stable.
>>
>> What are the chances Bcachefs beats Btrfs at being the Linux kernel's next
>> "official" file system ? I chose Btrfs over ZFS because it seemed like the
>> only "next-gen" heir to ext4/xfs.
>>
>> I have been having a few problems with Btrfs myself. I have only one that
>> remains unresolved : I haven't found the best way to mount Btrfs at boot
>> time. "LABEL=" won't work for known reasons (I don't understand however why
>> a mount can't do its own "device scan" transparently). "UUID=" won't work
>> for unknown reasons (haven't got a reply on this, maybe it's the same as
>> "LABEL="). And I will use /dev/* in fstab for stability reasons. Right now
>> I'm mounting the fs manually after a "device scan" and picking up the first
>> device that shows up in the "fi show" run. I can "live" with that but I
>> suppose that things like this contribute to the feeling that Btrfs is
>> actually still experimental contrarily to claims that it is
>> production-ready.
>>
>> For my own sake and other's I would like to maintain (if nobody is already
>> working on that nor needs any help) a centralized human-readable digest of
>> known issues that would be featured prominently on top of the Btrfs wiki. I
>> would merge the Gotchas page and the various known issues pages (including
>> the various multi-device mount gotchas here and there).
>>
>> Answers ? Comments ? Help ?
>
> First off, I think this is a wonderful idea.  The list of known issues isn't
> always particularly up to date, and isn't as trivial to find as it should
> be.
>

I think it's a great idea too and would like to contribute. Can
someone please lead the way?


-- 
Suman Chakravartula
Founder @ Rockstor

http://rockstor.com
http://rockstor.com/blog
http://forum.rockstor.com

Re: Response to Bcachefs Claims

[Duncan]

Vincent Olivier posted on Tue, 25 Aug 2015 11:22:34 -0400 as excerpted:

> Hi,
> 
> I have been using Btrfs for almost a year now with a 16x4TB RAID10 and
> its 8x4TB RAID0 backup (using incremental snapshots diffs). I have
> always tried to stay at the latest stable kernel (currently 4.1.6). But
> I might be moving to Fedora 22 because Centos 7 has significant
> incompatibilities with the 4.1.x kernel series.
> 
> I have seen the news about Bcachefs aiming to be Btrfs-complete while
> being extX-stable.
> 
> What are the chances Bcachefs beats Btrfs at being the Linux kernel's
> next "official" file system ? I chose Btrfs over ZFS because it seemed
> like the only "next-gen" heir to ext4/xfs.

I've seen the news and welcome the btrfs competition, but expect it'll be 
quite some time before they're both as featureful and at least as "not 
fully stable, but getting there" as btrfs is.  Like the btrfs devs 
originally, I think their estimates will turn out to be wildly 
optimistic, altho it /is/ quite likely they'll do it in fewer years than 
btrfs, as there's more data on what actually works and what doesn't work 
well in the Linux context, now, and because they're starting from an 
already working bcache base.

But meanwhile, either features or stability will suffer.  I strongly 
suspect they may go for stability and speed first, with ext4-comparable 
features first in the development queue except that bcache is already 
multi-device by definition, and may indeed beat btrfs to simple-case full 
stability, since btrfs went broad before deep, covering many features 
before stabilizing.  What would be missing?  I'd guess things like 
transparent compression, some of the available on btrfs raid levels, 
perhaps subvolumes/snapshots, and very possibly quotas, which have proven 
devilishly complex and hard to get right on btrfs, so omitting that alone 
for the initial to-stable feature list could easily mean a few man-years 
of development they can simply skip, to be implemented later.

> I have been having a few problems with Btrfs myself. I have only one
> that remains unresolved : I haven't found the best way to mount Btrfs at
> boot time. "LABEL=" won't work for known reasons (I don't understand
> however why a mount can't do its own "device scan" transparently).
> "UUID=" won't work for unknown reasons (haven't got a reply on this,
> maybe it's the same as "LABEL="). And I will use /dev/* in fstab for
> stability reasons. Right now I'm mounting the fs manually after a
> "device scan" and picking up the first device that shows up in the "fi
> show" run. I can "live" with that but I suppose that things like this
> contribute to the feeling that Btrfs is actually still experimental
> contrarily to claims that it is production-ready.

Basically, there's no way to do multi-device btrfs as rootfs without an 
initr*.  In theory, the kernel commandline rootflags=device= should do 
it, but last I checked anyway, some kernel versions ago, it was still 
broken, tho rootflags=degraded, for instance, would work (mounting the 
pair-devices btrfs raid1 filesystem with only the single device pointed 
at with root=), so there was definitely something wrong with device=, 
specifically.  (My theory is that the kernel commandline parser is seeing 
the two = chars and trying to break at the wrong one, thus instead of 
seeing rootflags and parsing the options from it, it's seeing 
rootflags=device, which the kernel doesn't know anything about, so 
ignores.)

Tho people do use device= in fstab or otherwise provided to mount, and it 
works.  It's just the kernel that can't deal with it in rootflags=, for 
whatever reason.

A multi-device btrfs mounted after / (and /usr...) are up, however, 
should work, provided a btrfs device scan has been performed already, so 
the kernel knows where all the individual device components are.  That's 
also how a working initr* mounts a multi-device btrfs root, BTW, doing a 
btrfs device scan before attempting the mount.  Otherwise, the mount will 
fail.

As for mounting using LABEL= (my preferred method), UUID=, PARTUUID=, 
PARTLABEL=, ID=, PATH=... those are all setup by udev, so udev must be 
run before mounts of that nature will work.  Again, shouldn't be a 
problem for a rootfs if the initr* has udev running before the mount 
attempt; indeed, I use root=LABEL= as a kernel commandline option from 
grub all the time and it just works; but if the initr* doesn't run udev 
first, or if it's a direct kernel root mount with no initr*, it's almost 
certainly not going to work, because udev is what creates those /dev/disk/
by-* entries that both the kernel and mount use for mounts using *= as 
the device.

> For my own sake and other's I would like to maintain (if nobody is
> already working on that nor needs any help) a centralized human-readable
> digest of known issues that would be featured prominently on top of the
> Btrfs wiki. I would merge the Gotchas page and the various known issues
> pages (including the various multi-device mount gotchas here and there).
> 
> Answers ? Comments ? Help ?

I'd personally just put a link to the gotchas page up right near the top, 
preferably in a warning box to increase visibility, and put the real work 
into collecting everything to the gotchas page and then keeping it 
current.  But since it's you volunteering to do the work, you can do it 
your way, big list on the front page, if you like. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Response to Bcachefs Claims

[Vincent Olivier]

I'm still parsing through the multi-device advices. Will be back on this when I'm done. And I'll probably switch distro to Archlinux which seems the way to go if one is using cutting-edge kernel features like Btrfs.

As for the work on "Gotchas/Known Issues" on the Btrfs wiki, I also think that the best way is to start with the Gotchas page and put a more prominent link for it on the home page.

I would restructure the Gotchas page in the following ways :

* Add the mention "Known Issues" close to the title;

* Only keep current issues/gotchas (current stable kernel, current userspace utilities release), archive every other on a separate page;

* Group by smallest feature encompassing the issue : multi-device, quotas, subvolumes, compression, conversion from extX, interactions with other things like LVM, MD, encryption, etc.;

* Something that is "new" and not as thoroughly tested as other features should be listed there as well (I think) until there is a consensus (to be defined) on it being reliable enough to be taken out of the list. Or maybe that should go on another page ?

* Provide links to HOWTOs or best practices for the features discussed (multi-device GOTCHAS should link to multi-device HOWTO).

I will be thinking about it more before doing anything and still welcoming ideas.

Thanks !

Vincent

Re: Response to Bcachefs Claims

[David Sterba]

Hi,

thanks for stepping up.

On Wed, Aug 26, 2015 at 03:24:46PM -0400, Vincent Olivier wrote:
> As for the work on "Gotchas/Known Issues" on the Btrfs wiki, I also
> think that the best way is to start with the Gotchas page and put a
> more prominent link for it on the home page.

Agreed.

> I would restructure the Gotchas page in the following ways :
> 
> * Add the mention "Known Issues" close to the title;
> 
> * Only keep current issues/gotchas (current stable kernel, current
> userspace utilities release), archive every other on a separate page;

IMO makes sense to keep the current and the previous, ie reflecting
versions that are likely still in wide use.

> * Group by smallest feature encompassing the issue : multi-device,
> quotas, subvolumes, compression, conversion from extX, interactions
> with other things like LVM, MD, encryption, etc.;
> 
> * Something that is "new" and not as thoroughly tested as other
> features should be listed there as well (I think) until there is a
> consensus (to be defined) on it being reliable enough to be taken out
> of the list. Or maybe that should go on another page ?

I'd keep it on the same page, at least for now. 

> * Provide links to HOWTOs or best practices for the features discussed
> (multi-device GOTCHAS should link to multi-device HOWTO).
> 
> I will be thinking about it more before doing anything and still
> welcoming ideas.

Otherwise all points seem ok to me, feel free to start. Some gotchas may
be covered on another pages (FAQ), but we can figure out the best page
later.