btrfs-convert missing in btrfs-tools v4.15.1

btrfs-convert missing in btrfs-tools v4.15.1

[jkexcel]

[-- Attachment #1: Type: text/plain, Size: 650 bytes --]

I'm an end user trying to use btrfs-convert but when I installed btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the installation was successful, and it shows that v4.15.1-1build1 was installed.


However, when using the command  # brtfs-convert  /dev/sda4 (with the drive unmounted) the resulting error appears "command not found"


I also tried command "btrfs convert" in case this was folded into the main tool, but this also failed.


1. Is btrfs-convert still available?

2. Where can I find it?

3. Has btrfs-convert been replaced? what is it's new name?

4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or older?

[-- Attachment #2: Type: text/html, Size: 1740 bytes --]

Re: btrfs-convert missing in btrfs-tools v4.15.1

[Nicholas D Steeves]

[-- Attachment #1: Type: text/plain, Size: 2081 bytes --]

Resending because I forgot to CC list.

Hi jkexcel,

On 28 July 2018 at 16:50, jkexcel <jkexcel@comcast.net> wrote:
>
> I'm an end user trying to use btrfs-convert but when I installed
> btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the
> installation was successful, and it shows that v4.15.1-1build1 was
> installed.
>
> However, when using the command  # brtfs-convert  /dev/sda4 (with the
> drive unmounted) the resulting error appears "command not found"
> I also tried command "btrfs convert" in case this was folded into the
> main tool, but this also failed.
>
> 1. Is btrfs-convert still available?
>
> 2. Where can I find it?
>
> 3. Has btrfs-convert been replaced? what is it's new name?
>
> 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or
> older?

You can blame me for that.  In Debian several users had reported
release-critical issues in btrfs-convert, so I submitted a patch to
disable it for the forseable future, eg:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489

Also, please consider the official status "As of 4.0 kernels this feature
is not often used or well tested anymore, and there have been some reports
that the conversion doesn't work reliably. Feel free to try it out, but
make sure you have backups" (
https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ).

I'm happy to hear it is still disabled in Ubuntu, where many more
users would be affected.  IIRC OpenSUSE LEAP and SLED 15 reenabled it
(it was previously disabled there), so maybe it needs specific kernel
versions or patches to not trigger RC bugs, and/or very specific
btrfs-progs versions, and/or very specific e2fslibs, and/or specific
combinations?  While I very much look forward to the day when
btrfs-convert can be relied on in Debian, I don't think we're there
yet.  Please take this as an opportunity to test that your backups are
restorable, mkfs.btrfs, and then restore from backup.  P.S. I have no
idea if Ubuntu has additional btrfs support.

Cheers,
Nicholas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: btrfs-convert missing in btrfs-tools v4.15.1

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3103 bytes --]



On 2018年07月29日 05:34, Nicholas D Steeves wrote:
> Resending because I forgot to CC list.
> 
> Hi jkexcel,
> 
> On 28 July 2018 at 16:50, jkexcel <jkexcel@comcast.net> wrote:
>>
>> I'm an end user trying to use btrfs-convert but when I installed
>> btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the
>> installation was successful, and it shows that v4.15.1-1build1 was
>> installed.
>>
>> However, when using the command  # brtfs-convert  /dev/sda4 (with the
>> drive unmounted) the resulting error appears "command not found"
>> I also tried command "btrfs convert" in case this was folded into the
>> main tool, but this also failed.
>>
>> 1. Is btrfs-convert still available?
>>
>> 2. Where can I find it?
>>
>> 3. Has btrfs-convert been replaced? what is it's new name?
>>
>> 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or
>> older?
> 
> You can blame me for that.  In Debian several users had reported
> release-critical issues in btrfs-convert, so I submitted a patch to
> disable it for the forseable future, eg:
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489

Both report looks pretty old (4.7~4.9).

In fact, just in v4.10 we had a big rework for convert.
It should work much better after that.

Furthermore, we have newer (but smaller) fixes to remove a lot of
BUG_ON(), and do graceful exit for ENOSPC case since then.

And the design of btrfs-convert (at least for the latest btrfs-convert)
is to ensure until everything goes well, we won't touch any bytes of the
ext* fs (in fact we open the ext* fs in RO mode).
So it at least won't corrupt the ext* fs.

> 
> Also, please consider the official status "As of 4.0 kernels this feature
> is not often used or well tested anymore, and there have been some reports
> that the conversion doesn't work reliably. Feel free to try it out, but
> make sure you have backups" (
> https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ).

The wiki page looks needs to be updated.

Both btrfs-convert and base btrfs-progs are improving, especially after
v4.10 btrfs-convert goes through a big rework and works well so far, and
even added support for reiserfs under the same framwork.

So IMHO it's at least worth trying.

Thanks,
Qu

> 
> I'm happy to hear it is still disabled in Ubuntu, where many more
> users would be affected.  IIRC OpenSUSE LEAP and SLED 15 reenabled it
> (it was previously disabled there), so maybe it needs specific kernel
> versions or patches to not trigger RC bugs, and/or very specific
> btrfs-progs versions, and/or very specific e2fslibs, and/or specific
> combinations?  While I very much look forward to the day when
> btrfs-convert can be relied on in Debian, I don't think we're there
> yet.  Please take this as an opportunity to test that your backups are
> restorable, mkfs.btrfs, and then restore from backup.  P.S. I have no
> idea if Ubuntu has additional btrfs support.
> 
> Cheers,
> Nicholas
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: btrfs-convert missing in btrfs-tools v4.15.1

[Nicholas D Steeves]

[-- Attachment #1: Type: text/plain, Size: 3194 bytes --]

Hi Qu,

On Sun, Jul 29, 2018 at 07:44:05AM +0800, Qu Wenruo wrote:
> 
> 
> On 2018年07月29日 05:34, Nicholas D Steeves wrote:
> > Resending because I forgot to CC list.
> > 
> > Hi jkexcel,
> > 
> > On 28 July 2018 at 16:50, jkexcel <jkexcel@comcast.net> wrote:
> >>
> >> I'm an end user trying to use btrfs-convert but when I installed
> >> btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the
> >> installation was successful, and it shows that v4.15.1-1build1 was
> >> installed.
> >>
> >> However, when using the command  # brtfs-convert  /dev/sda4 (with the
> >> drive unmounted) the resulting error appears "command not found"
> >> I also tried command "btrfs convert" in case this was folded into the
> >> main tool, but this also failed.
> >>
> >> 1. Is btrfs-convert still available?
> >>
> >> 2. Where can I find it?
> >>
> >> 3. Has btrfs-convert been replaced? what is it's new name?
> >>
> >> 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or
> >> older?
> > 
> > You can blame me for that.  In Debian several users had reported
> > release-critical issues in btrfs-convert, so I submitted a patch to
> > disable it for the forseable future, eg:
> > 
> >   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798
> >   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489
> 
> Both report looks pretty old (4.7~4.9).
> 
> In fact, just in v4.10 we had a big rework for convert.
> It should work much better after that.
> 
> Furthermore, we have newer (but smaller) fixes to remove a lot of
> BUG_ON(), and do graceful exit for ENOSPC case since then.
> 
> And the design of btrfs-convert (at least for the latest btrfs-convert)
> is to ensure until everything goes well, we won't touch any bytes of the
> ext* fs (in fact we open the ext* fs in RO mode).
> So it at least won't corrupt the ext* fs.
> 
> > 
> > Also, please consider the official status "As of 4.0 kernels this feature
> > is not often used or well tested anymore, and there have been some reports
> > that the conversion doesn't work reliably. Feel free to try it out, but
> > make sure you have backups" (
> > https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ).
> 
> The wiki page looks needs to be updated.
> 
> Both btrfs-convert and base btrfs-progs are improving, especially after
> v4.10 btrfs-convert goes through a big rework and works well so far, and
> even added support for reiserfs under the same framwork.
> 
> So IMHO it's at least worth trying.
> 
> Thanks,
> Qu

Thank you for sharing the cut-off where success became more likely :-)
Debian 9 could have had 4.9.1 at the newest, so it wouldn't have had
btrfs-convert.  So it sounds like btrfs-convert could have been
enabled for the experimental suite (which is almost only used by
Debian developers and not users) for 4.10.  Looking at the changelog
it seems we might have had to disable it again before 4.14.1 or 4.16.

I'm happy we're having this conversation now, because the time to give
it another try is probably sometime in the next month :-)  See my long
email to David for the caveats.

Cheers,
Nicholas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: btrfs-convert missing in btrfs-tools v4.15.1

[David Sterba]

On Sat, Jul 28, 2018 at 05:34:49PM -0400, Nicholas D Steeves wrote:
> On 28 July 2018 at 16:50, jkexcel <jkexcel@comcast.net> wrote:
> > I'm an end user trying to use btrfs-convert but when I installed
> > btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the
> > installation was successful, and it shows that v4.15.1-1build1 was
> > installed.
> >
> > However, when using the command  # brtfs-convert  /dev/sda4 (with the
> > drive unmounted) the resulting error appears "command not found"
> > I also tried command "btrfs convert" in case this was folded into the
> > main tool, but this also failed.
> >
> > 1. Is btrfs-convert still available?
> >
> > 2. Where can I find it?
> >
> > 3. Has btrfs-convert been replaced? what is it's new name?
> >
> > 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or
> > older?
> 
> You can blame me for that.  In Debian several users had reported
> release-critical issues in btrfs-convert, so I submitted a patch to
> disable it for the forseable future, eg:
> 
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798
>   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489

The reports are against version 4.7.3 released one year before the time
of the bug reports. The fix for the reported bug happened in 4.10, that
was half a year before the bug.

> Also, please consider the official status "As of 4.0 kernels this feature
> is not often used or well tested anymore, and there have been some reports
> that the conversion doesn't work reliably. Feel free to try it out, but
> make sure you have backups" (
> https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ).

Sorry that this you take it as official status. The wiki is open to
edits and such claims appear there from time to time. I've removed it,
it's been there since 2015 when it possibly was accurate but it's not
anymore.

> I'm happy to hear it is still disabled in Ubuntu, where many more
> users would be affected.  IIRC OpenSUSE LEAP and SLED 15 reenabled it
> (it was previously disabled there), so maybe it needs specific kernel
> versions or patches to not trigger RC bugs, and/or very specific
> btrfs-progs versions, and/or very specific e2fslibs, and/or specific
> combinations?  While I very much look forward to the day when
> btrfs-convert can be relied on in Debian, I don't think we're there
> yet.

So if there's no way to update package to newer version or nobody who
backports fixes, then it's better not to ship the tool. There's nothing
close to the 'specific version of X', besides using more up to date
versions. Alternatively, there could have been a separate package with
the convert tool, with a warning about the known issues.

It's in my interest to ship all tools in distros, but there's also only
that much what the upstream community can do. If you're going to
reconsider the status of btrfs-convert in Debian, please let me know.

Re: btrfs-convert missing in btrfs-tools v4.15.1

[Nicholas D Steeves]

[-- Attachment #1: Type: text/plain, Size: 9000 bytes --]

Hi everyone,

Sorry for the delay replying, I've been busy with other work.  Reply
follows inline.  P.S. sorry about the table in this email that is 99
columns wide.

On Thu, Aug 09, 2018 at 01:50:46PM +0200, David Sterba wrote:
> On Sat, Jul 28, 2018 at 05:34:49PM -0400, Nicholas D Steeves wrote:
> > On 28 July 2018 at 16:50, jkexcel <jkexcel@comcast.net> wrote:
> > > I'm an end user trying to use btrfs-convert but when I installed
> > > btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the
> > > installation was successful, and it shows that v4.15.1-1build1 was
> > > installed.
> > >
> > > However, when using the command  # brtfs-convert  /dev/sda4 (with the
> > > drive unmounted) the resulting error appears "command not found"
> > > I also tried command "btrfs convert" in case this was folded into the
> > > main tool, but this also failed.
> > >
> > > 1. Is btrfs-convert still available?
> > >
> > > 2. Where can I find it?
> > >
> > > 3. Has btrfs-convert been replaced? what is it's new name?
> > >
> > > 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or
> > > older?
> > 
> > You can blame me for that.  In Debian several users had reported
> > release-critical issues in btrfs-convert, so I submitted a patch to
> > disable it for the forseable future, eg:
> > 
> >   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798
> >   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489
> 
> The reports are against version 4.7.3 released one year before the time
> of the bug reports. The fix for the reported bug happened in 4.10, that
> was half a year before the bug.

Debian stable will always have an old version, which will be in use
for two to four years.  Btrfs-progs 4.7.3 will be in use in Debian 9
until at least 2021, possibly longer.  Btw, I strongly believe Debian 9
should have shipped with btrfs-progs 4.9.1, but alas the primary
maintainer didn't upload it on time.

For "buster" (Debian 10), which will probably be released in mid 2019,
the newest possible btrfs-progs version that could be included is
whatever is current at the end of January 2019.  Exceptions are
sometimes granted for an unblock of a newer version.  For example, if
an LTS kernel won't be released until March, and the release,
technical committee, and kernel team decide that's the version we
want, then a preapproved exception will be granted.  At that time an
argument can be made for preapproval of a newer btrfs-progs as well.

That said, I try to keep a backported newer version of btrfs-progs for
the stable Debian release reasonably up-to-date (backports are
available to users on a per-package opt-in basis).  That's the channel
for feature enablement.  Also, my apologies, at the moment this
backport is very much out of date--it stalled while investigating
which packages would be broken by the library reorganisation;
although, from what I can tell that would only be snapper.

> > Also, please consider the official status "As of 4.0 kernels this feature
> > is not often used or well tested anymore, and there have been some reports
> > that the conversion doesn't work reliably. Feel free to try it out, but
> > make sure you have backups" (
> > https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ).
> 
> Sorry that this you take it as official status. The wiki is open to
> edits and such claims appear there from time to time. I've removed it,
> it's been there since 2015 when it possibly was accurate but it's not
> anymore.

Is there a more authoritative and up-to-date location for various
statuses?  It would be nice to have something in btrfs-progs as a
table like this, or exportable in some kind of human-friendly format:

+-------+-----------------------------------------+------------+------------------+---------------+
|Feature|1st mainline version where feature       |LTS kernel 1|LTS kernel 2      |LTS kernel 3   |
|       |appeared                                 |eg: 4.4     |eg: 4.9           |eg: 4.14       |
+-------+-----------------------------------------+------------+------------------+---------------+
|convert|Assume -progs and kernel                 |exp?        |mostly?           |stable?        |
|from   |vessions are strongly                    |            |  amend           |               |
|ext3/4 |associated, for simplicity.              |            |  status with:    |               |
|       |                                         |            |4.9.z:testing     |               |
+-------+-----------------------------------------+------------+------------------+---------------+
|foo    |3.16:danger||exp||mostly||testing||stable|exp         |mostly            |testing        |
+-------+-----------------------------------------+------------+------------------+---------------+

Ideally something that could be trivially exported to the format the
btrfs.wiki.kernel.org wiki uses.  The trick is to make it convenient
to maintain...  Other than CSV, I'm not sure what would work well for
non-emacs users.  When the oldest LTS kernel is considered depreciated
by the btrfs upstream community, "depreciated" is added to the column
header, and when a 4th LTS kernel becomes available the oldest LTS
column is dropped and a new empty column is appended.  Oh, there could
also be a column for current mainline version, but I don't want to
maintain that column, and I suspect no one else would either.

It would also be nice to add various optimisations to the "Feature"
column as they become available.  Ideally it would also be nice to
have an auto-generated bug matrix, but I digress.

If there is not yet a canonical/official table for various btrfs
features' statuses then I'd be happy to start work on one--given that
the official upstream wiki is nonauthoritative...  At the moment 1)
I'd use either a table like the one above, something from org-mode, or
a CSV.  2) I'd fill it in from the wiki, with versions informed by the
bugs I'm aware of  3) I would always err on the side of caution,
because I believe that the most important thing for btrfs right now
are incident-free success stories, and overly optimistic
recommendations will not provide these.  4) I'd submit the patch for
discussion and review.

> > I'm happy to hear it is still disabled in Ubuntu, where many more
> > users would be affected.  IIRC OpenSUSE LEAP and SLED 15 reenabled it
> > (it was previously disabled there), so maybe it needs specific kernel
> > versions or patches to not trigger RC bugs, and/or very specific
> > btrfs-progs versions, and/or very specific e2fslibs, and/or specific
> > combinations?  While I very much look forward to the day when
> > btrfs-convert can be relied on in Debian, I don't think we're there
> > yet.
> 
> So if there's no way to update package to newer version or nobody who
> backports fixes, then it's better not to ship the tool. There's nothing
> close to the 'specific version of X', besides using more up to date
> versions. Alternatively, there could have been a separate package with
> the convert tool, with a warning about the known issues.

Briefly, in Debian, backported fixes (to the stable version) are only
approved for security and release critical bugs, and never for feature
enablement.  Supposing btrfs-convert had been its own package, this
package would have been dropped from testing if a targeted fix
couldn't be found and uploaded within a week or two.  During the
freeze entry of new upstream versions is blocked, and reentry of
dropped packages is also blocked.

Once Debian has KVM-isolated machines for running continuous
integration, the plan is to enable self-tests in CI.  Generally
regressions will also block migration of new versions, so that's yet
another way Debian ends up with old versions--unless the issues are
fixed...but at least this will occur during the general development
phase of the release cycle, where there is a lot more time to fix
issues :-)

> It's in my interest to ship all tools in distros, but there's also only
> that much what the upstream community can do. If you're going to
> reconsider the status of btrfs-convert in Debian, please let me know.

Yes, I'd be happy to advocate for its reinclusion if the answer to
4/5 of the following questions is "yes".  Does SUSE now recommend the
use of btrfs-convert to its enterprise customers?  The following is a
frustrating criteria, but: Can a random desktop user run btrfs-convert
against their ext4 rootfs and expect the operation to succeed?  Is
btrfs-convert now sufficiently trusted that it can be recommended with
the same degree of confidence as a backup, mkfs.btrfs, then restore to
new filesystem approach?  Does the user of a btrfs volume created with
btrfs-convert have an equal or lesser probability of encountering bugs
compared to a one who used mkfs.btrfs?

Sincerely,
Nicholas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: btrfs-convert missing in btrfs-tools v4.15.1

[Duncan]

Nicholas D Steeves posted on Thu, 23 Aug 2018 14:15:18 -0400 as excerpted:

>> It's in my interest to ship all tools in distros, but there's also only
>> that much what the upstream community can do. If you're going to
>> reconsider the status of btrfs-convert in Debian, please let me know.
> 
> Yes, I'd be happy to advocate for its reinclusion if the answer to 4/5
> of the following questions is "yes".  Does SUSE now recommend the use of
> btrfs-convert to its enterprise customers?  The following is a
> frustrating criteria, but: Can a random desktop user run btrfs-convert
> against their ext4 rootfs and expect the operation to succeed?  Is
> btrfs-convert now sufficiently trusted that it can be recommended with
> the same degree of confidence as a backup, mkfs.btrfs, then restore to
> new filesystem approach?  Does the user of a btrfs volume created with
> btrfs-convert have an equal or lesser probability of encountering bugs
> compared to a one who used mkfs.btrfs?

Just a user and list regular here, and gentoo not debian, but for what it 
counts...

I'd personally never consider or recommend a filesystem converter over 
the backup, mkfs-to-new-fs, restore-to-new-fs, method, for three reasons.

1) Regardless of how stable a filesystem converter is and what two 
filesystems the conversion is between, "things" /do/ occasionally happen, 
thus making it irresponsible to use or recommend use of such a converter 
without a suitably current and tested backup, "just in case."

(This is of course a special case of the sysadmin's first rule of 
backups, that the true value of data is defined not by any arbitrary 
claims, but by the number of backups of that data it's considered worth 
the time/trouble/resources to make/have.  If the data value is trivial 
enough, sure, don't bother with the backup, but if it's of /that/ low a 
value, so low it's not worth a backup even when doing something as 
theoretically risky as a filesystem conversion, why is it worth the time 
and trouble to bother converting it in the first place, instead of just 
blowing it away and starting clean?)

2) Once a backup is considered "strongly recommended", as we've just 
established that it should be in 1 regardless of the stability of the 
converter, just using the existing filesystem as that backup and starting 
fresh with a mkfs for the new filesystem and copying things over is 
simply put the easiest, simplest and cleanest method to change 
filesystems.

3) (Pretty much)[1] Regardless of the filesystems in question, a fresh 
mkfs and clean sequential transfer of files from the old-fs/backup to the 
new one is pretty well guaranteed to be better optimized than conversion 
from an existing filesystem of a different type, particularly one that 
has been in normal operation for awhile and thus has operational 
fragmentation of both data and free-space.  That's in addition to being 
less bug-prone, even for a "stable" converter.


Restating: So(1) doing a conversion without a backup is irresponsible, 
(2) the easiest backup and conversion method is directly using the old fs 
as the backup, and copying over to the freshly mkfs-ed new filesystem, 
and (3) a freshly mkfs-ed filesystem and sequential copy of files to it 
from backup, whether that be the old filesystem or not, is going to be 
more efficient and less bug-prone than an in-place conversion.

Given the above, why would /anyone/ /sane/ consider using a converter?  
It simply doesn't make sense, even if the converter were as stable as the 
most stable filesystems we have.


So as a distro btrfs package maintainer, do what you wish in terms of the 
converter, but were it me, I might actually consider replacing it with an 
executable that simply printed out some form of the above argument, with 
a pointer to the sources should they still be interested after having 
read that argument.[2] Then, if people really are determined to 
unnecessarily waste their time to get a less efficient filesystem, 
possibly risking their data in the process of getting it, they can always 
build the converter from sources themselves.

---
[1] I debated omitting the qualifier as I know of no exceptions, but I'm 
not a filesystem expert and while I'm a bit skeptical, I suppose it's 
possible that they might exist.

[2] There's actually btrfs precedent for this in the form of the 
executable built as fsck.btrfs, which does nothing (successfully) but 
possibly print a message referring people to btrfs check, if run in 
interactive mode.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman