Another defrag question

Another defrag question

[Swâmi Petaramesh]

Hi again,

Having numerous snapshots, I prefer to ask rather than take the risk of
exploding my storage space, better safe than sorry ;-)

"man btrfs" states :

« NOTE: defragmenting with kernels up to 2.6.37 will unlink COW-ed
copies of data, don't use it if  you  use  snapshots,  have 
deduplicated  your  data  or  made  copies with cp --reflink. »

I use :
# uname -r
3.5.0-24-generic

...So should I expect that defraging my BTRFS will be smart enough not
to uncow my snapshots ? Is it actually able to defrag both the file and
its snapshots altogether, keeping all this as a single physical copy of
the defragged data ?

TIA.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Another defrag question

[Liu Bo]

On Thu, Feb 21, 2013 at 04:46:14PM +0100, Swâmi Petaramesh wrote:
> Hi again,
> 
> Having numerous snapshots, I prefer to ask rather than take the risk of
> exploding my storage space, better safe than sorry ;-)
> 
> "man btrfs" states :
> 
> « NOTE: defragmenting with kernels up to 2.6.37 will unlink COW-ed
> copies of data, don't use it if  you  use  snapshots,  have 
> deduplicated  your  data  or  made  copies with cp --reflink. »
> 
> I use :
> # uname -r
> 3.5.0-24-generic
> 
> ...So should I expect that defraging my BTRFS will be smart enough not
> to uncow my snapshots ? Is it actually able to defrag both the file and
> its snapshots altogether, keeping all this as a single physical copy of
> the defragged data ?

Well, there is already a patch which addresses your concern and it's
'snapshot-aware defrag' feature and now in v6, it's not merged yet.

thanks,
liubo

> 
> TIA.
> 
> -- 
> Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
> Ne cherchez pas : Je ne suis pas sur Facebook.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Another defrag question

[Swâmi Petaramesh]

Le 21/02/2013 16:50, Liu Bo a écrit :
> Well, there is already a patch which addresses your concern and it's
> 'snapshot-aware defrag' feature and now in v6, it's not merged yet.
> thanks, liubo 

Hi Liu,

So should I understand that, even though the manpage states that the
issue is for kernels <= 2.6.37, I should however not defrag until the
patch you mention is actually merged in a future kernel release ?

TIA.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Another defrag question

[Calvin Walton]

On Thu, 2013-02-21 at 16:46 +0100, Swâmi Petaramesh wrote:
> Hi again,
> 
> Having numerous snapshots, I prefer to ask rather than take the risk of
> exploding my storage space, better safe than sorry ;-)
> 
> "man btrfs" states :
> 
> « NOTE: defragmenting with kernels up to 2.6.37 will unlink COW-ed
> copies of data, don't use it if  you  use  snapshots,  have 
> deduplicated  your  data  or  made  copies with cp --reflink. »
> 
> I use :
> # uname -r
> 3.5.0-24-generic
> 
> ...So should I expect that defraging my BTRFS will be smart enough not
> to uncow my snapshots ? Is it actually able to defrag both the file and
> its snapshots altogether, keeping all this as a single physical copy of
> the defragged data ?

The message in the btrfs man page is a bit out of date - defragmenting
files on btrfs will uncow files on all currently released kernels.

There's a patch available to fix this, but it hasn't been merged yet. It
might show up in 3.9 or 3.10.

You really should upgrade your kernel, however. 3.5.0 is rather old in
btrfs-years! Lots of fixes have gone into newer kernels.


-- 
Calvin Walton <calvin.walton@kepstin.ca>

Re: Another defrag question

[Swâmi Petaramesh]

Le 21/02/2013 16:54, Calvin Walton a écrit :
> You really should upgrade your kernel, however. 3.5.0 is rather old in
> btrfs-years! Lots of fixes have gone into newer kernels.

Hi Calvin,

I expect Ubuntu 13.04 to come with kernel 3.7 in April. Having Ubuntu
kernel upgrades every 6 months (and several machines), I try to stick as
much as I can to my distro's kernel. I'm no developper myself and can
hardly consider that the kernel that came with the latest distro 4
months ago is actually _that_ old... So I'll live with it for 2 more
months - unless I'm pretty sure that upgrading to some more recent
kernel than the one that Ubuntu 13.04 will have would give me huge speed
improvements or features I need much - (and I assume that being able to
defrag, with my snapshots, would be a good reason for me...)

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Another defrag question

[Blair Zajac]

On 02/21/2013 08:01 AM, Swâmi Petaramesh wrote:
> Le 21/02/2013 16:54, Calvin Walton a écrit :
>> You really should upgrade your kernel, however. 3.5.0 is rather old in
>> btrfs-years! Lots of fixes have gone into newer kernels.
>
> Hi Calvin,
>
> I expect Ubuntu 13.04 to come with kernel 3.7 in April.

13.04 has 3.8, they just updated to the final 3.8 release.

Blair

Re: Another defrag question

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

On Thu, Feb 21, 2013 at 05:01:30PM +0100, Swâmi Petaramesh wrote:
> Le 21/02/2013 16:54, Calvin Walton a écrit :
> > You really should upgrade your kernel, however. 3.5.0 is rather old in
> > btrfs-years! Lots of fixes have gone into newer kernels.
> 
> Hi Calvin,
> 
> I expect Ubuntu 13.04 to come with kernel 3.7 in April. Having Ubuntu
> kernel upgrades every 6 months (and several machines), I try to stick as
> much as I can to my distro's kernel. I'm no developper myself and can
> hardly consider that the kernel that came with the latest distro 4
> months ago is actually _that_ old... So I'll live with it for 2 more
> months - unless I'm pretty sure that upgrading to some more recent
> kernel than the one that Ubuntu 13.04 will have would give me huge speed
> improvements or features I need much - (and I assume that being able to

   3.7 is quite a bit faster.

   Plus, if something does go wrong with your FS, and you're running
an older kernel, you'll get limited amounts of sympathy, because quite
a lot of the problems people encounter with older kernels have already
been fixed in newer ones.

   Finally, Ubuntu publish the latest kernels in a PPA[1], so there's
not really much excuse for not keeping up with them.

> defrag, with my snapshots, would be a good reason for me...)

   Hugo.

[1] http://kernel.ubuntu.com/~kernel-ppa/mainline/

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
               --- Nostalgia isn't what it used to be. ---               

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Another defrag question

[Swâmi Petaramesh]

Le 21/02/2013 17:38, Hugo Mills a écrit :
> Plus, if something does go wrong with your FS, and you're running an
> older kernel, you'll get limited amounts of sympathy, because quite a
> lot of the problems people encounter with older kernels have already
> been fixed in newer ones.

The matter, as usual, is that what may be a "new" kernel from a user's
standpoint, may be an "old" kernel from a developer's standpoint.

Should this exclude all standard users from developer's "sympathy", or
reduce it to very limited amounts ?

Users are supposed to use released material, not bleeding-edge
compile-it-yourself devs toys...

> Finally, Ubuntu publish the latest kernels in a PPA[1], so there's not
> really much excuse for not keeping up with them. 

Yep. AFAIK the MAINLINE PPA is a Ubuntu-packaged STANDARD kernel without
any of the specific Ubuntu kernel patches.

That makes it very inappropriate for daily use, and the Ubuntu devs ask
for trying it ONLY when reporting kernel bugs, to check if said bug may
have been fixed upstream in latest kernel, or not.

Last time I tried this standard kernel, on one of my machines, it just
wouldn't boot ; on another one, it couldn't manage power (battery etc)
correctly, and on the machine on which I write this email, it couldn't
see my SD card reader anymore - for this one is managed by a driver
which still didn't make it into the standard kernel.

I use Ubuntu on several -dekstop- machines that benefit and need the
Ubuntu patches on their Ubuntus. I'm not (currently) running a headless
generic server for which my main and only concern would be BTRFS...

So I try to stick to the very old - albeit maximum 6 months old ! -
stock Ubuntu kernel...

Well... Sometimes I upgrade to next Beta :-]

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Another defrag question

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2737 bytes --]

On Thu, Feb 21, 2013 at 06:03:17PM +0100, Swâmi Petaramesh wrote:
> Le 21/02/2013 17:38, Hugo Mills a écrit :
> > Plus, if something does go wrong with your FS, and you're running an
> > older kernel, you'll get limited amounts of sympathy, because quite a
> > lot of the problems people encounter with older kernels have already
> > been fixed in newer ones.
> 
> The matter, as usual, is that what may be a "new" kernel from a user's
> standpoint, may be an "old" kernel from a developer's standpoint.
> 
> Should this exclude all standard users from developer's "sympathy", or
> reduce it to very limited amounts ?

   For an filesystem still under heavy development, and still marked
"experimental", yes, it should. In fact, *particularly* for a
filesystem, where a bug can cause persistent damage which isn't easily
fixable by upgrading to a later kernel (which is the case with most
kernel bugs).

> Users are supposed to use released material, not bleeding-edge
> compile-it-yourself devs toys...

   Correct. But btrfs isn't at that stage yet. It's getting visibly
closer, but it's not quite there. Hence the very strong recommendation
to keep up with the latest code.

   Hugo.

> > Finally, Ubuntu publish the latest kernels in a PPA[1], so there's not
> > really much excuse for not keeping up with them. 
> 
> Yep. AFAIK the MAINLINE PPA is a Ubuntu-packaged STANDARD kernel without
> any of the specific Ubuntu kernel patches.
> 
> That makes it very inappropriate for daily use, and the Ubuntu devs ask
> for trying it ONLY when reporting kernel bugs, to check if said bug may
> have been fixed upstream in latest kernel, or not.
> 
> Last time I tried this standard kernel, on one of my machines, it just
> wouldn't boot ; on another one, it couldn't manage power (battery etc)
> correctly, and on the machine on which I write this email, it couldn't
> see my SD card reader anymore - for this one is managed by a driver
> which still didn't make it into the standard kernel.
> 
> I use Ubuntu on several -dekstop- machines that benefit and need the
> Ubuntu patches on their Ubuntus. I'm not (currently) running a headless
> generic server for which my main and only concern would be BTRFS...
> 
> So I try to stick to the very old - albeit maximum 6 months old ! -
> stock Ubuntu kernel...

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- I get nervous when I see words like 'mayhaps' in a novel, ---    
                because I fear that just round the corner                
                          is lurking 'forsooth'                          

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Another defrag question

[Swâmi Petaramesh]

Le 21/02/2013 18:25, Hugo Mills a écrit :
> Correct. But btrfs isn't at that stage yet. It's getting visibly
> closer, but it's not quite there. Hence the very strong recommendation
> to keep up with the latest code. Hugo. 

The matter is that BTRFS had many early adopters just because it is -
and has been for long now - in the mainline Linux kernel, so supposed
stable and good choice for the future.

To be honest (and not wanting to troll, promised) this is the only
single reason for which I use BTRFS on 5 of my 6 machines at home - just
because I thought that "Just upgrade the distro every 6 months and it
will become better and better over time, no hassle, make my life easy".

OTOH my 6th machine runs native ZFS on Linux, and I have to tell that it
shows orders of magnitude better performance and never gave me a single
problem in several (3 ?) years. Only upgrading the distro is always a
big frightening and problematic. And initial installation was a bit tricky.

Probably a lot of BTRS early adopters choosed it for the same reason why
I did : Included in the standard kernel, easy to install, and *expected*
to improve quickly - yes, I already made the move back and forth twice,
between ext4 and BTRFS, on 2 machines... About one year ago I choosed to
jump "for good" and stay, but performance degrades so quickly and so
much that every couple of days I wonder if I won't rollback to ex4 again
(and again), or shift to ZFS once for all and just forget about it...

Everytime I show my Linux machines to friends and say : “Hey, I got the
most advanced filesystem on earth !” I soon get the answer “Oh boy,
that's the slowest machine boot and FS I've ever seen since I was
reading floppy disks on my 386SX in 1991 ! Can you really live with this ?”

So, for "not quite there" and the return codes "+20" that have been a
minor pain in the arse for a couple years but the line is still in the
code... I can understand developer's PoV, been there, done that, but
still, BTRFS might in the end lose a numer of its early adopters if it
keeps being "not quite there" too long.

Shitfing to ZFS is just a PPA and 2 apt-get install commands away... It
will definitely be easier than start playing with mainline PPA Ubuntu
kernels...

Kind regards.

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E
Ne cherchez pas : Je ne suis pas sur Facebook.

Re: Another defrag question

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 4949 bytes --]

On Thu, Feb 21, 2013 at 06:47:28PM +0100, Swâmi Petaramesh wrote:
> Le 21/02/2013 18:25, Hugo Mills a écrit :
> > Correct. But btrfs isn't at that stage yet. It's getting visibly
> > closer, but it's not quite there. Hence the very strong recommendation
> > to keep up with the latest code. Hugo. 
> 
> The matter is that BTRFS had many early adopters just because it is -
> and has been for long now - in the mainline Linux kernel, so supposed
> stable and good choice for the future.

   I'm afraid that's not actually a valid conclusion. Just because
something makes it into the mainline kernel, it doesn't mean that it's
considered stable -- just that any instabilities aren't going to
affect the rest of the kernel.

> To be honest (and not wanting to troll, promised) this is the only
> single reason for which I use BTRFS on 5 of my 6 machines at home - just
> because I thought that "Just upgrade the distro every 6 months and it
> will become better and better over time, no hassle, make my life easy".

   That approach means you're still somewhere between 1 and 3 kernel
versions out of date, on something which is still undergoing
significant changes on most kernel releases.

> OTOH my 6th machine runs native ZFS on Linux, and I have to tell that it
> shows orders of magnitude better performance and never gave me a single
> problem in several (3 ?) years. Only upgrading the distro is always a
> big frightening and problematic. And initial installation was a bit tricky.

   ZFS has had something like an extra 5 years of development on it,
so it's not surprising that it's more stable.

> Probably a lot of BTRS early adopters choosed it for the same reason why
> I did : Included in the standard kernel, easy to install, and *expected*
> to improve quickly

   But you don't get the improvements if you don't get the kernels...

> - yes, I already made the move back and forth twice,
> between ext4 and BTRFS, on 2 machines... About one year ago I choosed to
> jump "for good" and stay, but performance degrades so quickly and so
> much that every couple of days I wonder if I won't rollback to ex4 again
> (and again), or shift to ZFS once for all and just forget about it...

   Performance was greatly improved in 3.7. You may also find
performance problems with a very full filesystem, or with workloads
which write heavily to one file (databases, and bitcoin's DB in
particular). The latter issue can largely be dealt with by judicious
use of nocow attributes. Plus go for 3.7 or later anyway.

> Everytime I show my Linux machines to friends and say : “Hey, I got the
> most advanced filesystem on earth !” I soon get the answer “Oh boy,
> that's the slowest machine boot and FS I've ever seen since I was
> reading floppy disks on my 386SX in 1991 ! Can you really live with this ?”
> 
> So, for "not quite there" and the return codes "+20" that have been a
> minor pain in the arse for a couple years but the line is still in the
> code...

   There's been about one question (not even complaint) about that
return code every... maybe 3-6 months? It's not high on the list of
priorities of things to fix, compared to some of the other problems
that have shown up, or even the repeated questions (like
mis-interpretation of the output of df, btrfs fi df and btrfs fi
show).

>  I can understand developer's PoV, been there, done that, but
> still, BTRFS might in the end lose a numer of its early adopters if it
> keeps being "not quite there" too long.
> 
> Shitfing to ZFS is just a PPA and 2 apt-get install commands away... It
> will definitely be easier than start playing with mainline PPA Ubuntu
> kernels...

   Well, given that btrfs is still getting at least one significant
fix to a major bug or set of fixes in every release, it would still
seem sensible to keep up with the latest developments. If that's not
what you expected, then I'm sorry if we've failed to get the point
across. We do try to manage expectations on the btrfs website, and
we're quite up-front about the need for keeping up with releases
there.

   If that's more than you want to do, that's OK -- I wouldn't expect
it to be something that everyone would want to do. Many people do keep
up with the kernels through the Ubuntu PPA (or other, similar
repositories for other distributions), and don't have problems. If it
doesn't work out for you, then parting company without rancour or
prejudice may be the best thing. Temporarily, I hope, of course.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- I get nervous when I see words like 'mayhaps' in a novel, ---    
                because I fear that just round the corner                
                          is lurking 'forsooth'                          

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Another defrag question

[Bardur Arantsson]

On 02/21/2013 06:47 PM, Swâmi Petaramesh wrote:
> Le 21/02/2013 18:25, Hugo Mills a écrit :
>> Correct. But btrfs isn't at that stage yet. It's getting visibly
>> closer, but it's not quite there. Hence the very strong recommendation
>> to keep up with the latest code. Hugo. 
> 
> The matter is that BTRFS had many early adopters just because it is -
> and has been for long now - in the mainline Linux kernel, so supposed
> stable and good choice for the future.
> 
> To be honest (and not wanting to troll, promised) this is the only
> single reason for which I use BTRFS on 5 of my 6 machines at home - just
> because I thought that "Just upgrade the distro every 6 months and it
> will become better and better over time, no hassle, make my life easy".
> 

Unfortunately many distros don't make it obvious, but Btrfs is still
hidden behind a giant EXPERIMENTAL label in the kernel configuration.

Regards,

Re: Another defrag question

[David Sterba]

On Thu, Feb 21, 2013 at 09:58:16PM +0100, Bardur Arantsson wrote:
> On 02/21/2013 06:47 PM, Swâmi Petaramesh wrote:
> > Le 21/02/2013 18:25, Hugo Mills a écrit :
> >> Correct. But btrfs isn't at that stage yet. It's getting visibly
> >> closer, but it's not quite there. Hence the very strong recommendation
> >> to keep up with the latest code. Hugo. 
> > 
> > The matter is that BTRFS had many early adopters just because it is -
> > and has been for long now - in the mainline Linux kernel, so supposed
> > stable and good choice for the future.
> > 
> > To be honest (and not wanting to troll, promised) this is the only
> > single reason for which I use BTRFS on 5 of my 6 machines at home - just
> > because I thought that "Just upgrade the distro every 6 months and it
> > will become better and better over time, no hassle, make my life easy".
> > 
> 
> Unfortunately many distros don't make it obvious, but Btrfs is still
> hidden behind a giant EXPERIMENTAL label in the kernel configuration.

Removed in 3.9-rc1 as a part of a broad Kconfig cleanup

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commit;h=38db331b578005d32155bb6f6a80654ef127cff5

 The CONFIG_EXPERIMENTAL config item has not carried much meaning for a
 while now and is almost always enabled by default. As agreed during the
 Linux kernel summit, remove it from any "depends on" lines in Kconfigs.

--- a/fs/btrfs/Kconfig
+++ b/fs/btrfs/Kconfig
@@ -1,6 +1,5 @@
 config BTRFS_FS
-       tristate "Btrfs filesystem (EXPERIMENTAL) Unstable disk format"
-       depends on EXPERIMENTAL
+       tristate "Btrfs filesystem Unstable disk format"
        select LIBCRC32C
        select ZLIB_INFLATE
        select ZLIB_DEFLATE
---

is it still experimental then?

Re: Another defrag question

[Bardur Arantsson]

On 02/21/2013 10:56 PM, David Sterba wrote:
> On Thu, Feb 21, 2013 at 09:58:16PM +0100, Bardur Arantsson wrote:
>> On 02/21/2013 06:47 PM, Swâmi Petaramesh wrote:
>>> Le 21/02/2013 18:25, Hugo Mills a écrit :
>>>> Correct. But btrfs isn't at that stage yet. It's getting visibly
>>>> closer, but it's not quite there. Hence the very strong recommendation
>>>> to keep up with the latest code. Hugo. 
>>>
>>> The matter is that BTRFS had many early adopters just because it is -
>>> and has been for long now - in the mainline Linux kernel, so supposed
>>> stable and good choice for the future.
>>>
>>> To be honest (and not wanting to troll, promised) this is the only
>>> single reason for which I use BTRFS on 5 of my 6 machines at home - just
>>> because I thought that "Just upgrade the distro every 6 months and it
>>> will become better and better over time, no hassle, make my life easy".
>>>
>>
>> Unfortunately many distros don't make it obvious, but Btrfs is still
>> hidden behind a giant EXPERIMENTAL label in the kernel configuration.
> 
> Removed in 3.9-rc1 as a part of a broad Kconfig cleanup
> 
[--snip--]
> 
> is it still experimental then?

Interesting, but IMO having the experimental label taken off is a
necessary, but not sufficient condition for $X to be considered stable :).

Regards,

Re: Another defrag question

[Johannes Hirte]

On Thu, 21 Feb 2013 18:47:28 +0100
Swâmi Petaramesh <swami@petaramesh.org> wrote:

> Le 21/02/2013 18:25, Hugo Mills a écrit :
> > Correct. But btrfs isn't at that stage yet. It's getting visibly
> > closer, but it's not quite there. Hence the very strong
> > recommendation to keep up with the latest code. Hugo. 
> 
> The matter is that BTRFS had many early adopters just because it is -
> and has been for long now - in the mainline Linux kernel, so supposed
> stable and good choice for the future.

And it's marked as EXPERIMENTAL. So if you want to join the game, you
have to accept the rules.

> OTOH my 6th machine runs native ZFS on Linux, and I have to tell that
> it shows orders of magnitude better performance and never gave me a
> single problem in several (3 ?) years. Only upgrading the distro is
> always a big frightening and problematic. And initial installation
> was a bit tricky.

You didn't, many other had. I remember a lot of threads in the
OpenSolaris forum, where the solution for problems was: recreate your
filesystem, replay your backup.

> Everytime I show my Linux machines to friends and say : “Hey, I got
> the most advanced filesystem on earth !” I soon get the answer “Oh
> boy, that's the slowest machine boot and FS I've ever seen since I was
> reading floppy disks on my 386SX in 1991 ! Can you really live with
> this ?”

Did you presented it while (re)creating the inode_cache? Sounds a
little like that.

> So, for "not quite there" and the return codes "+20" that have been a
> minor pain in the arse for a couple years but the line is still in the
> code... I can understand developer's PoV, been there, done that, but
> still, BTRFS might in the end lose a numer of its early adopters if it
> keeps being "not quite there" too long.
> 
> Shitfing to ZFS is just a PPA and 2 apt-get install commands away...
> It will definitely be easier than start playing with mainline PPA
> Ubuntu kernels...

So why do you bother with btrfs, if ZFS fit your needs?


regards,
  Johannes