btrfstune settings

btrfstune settings

[Oliver Freyermuth]

Dear btrfs experts, 

I hope this is the correct place to ask, the wiki and manpages did not help me on these questions. 

BTRFS has gained extended inode refs, skinny metadata and no-holes quite a while ago and these are now the defaults for new mkfs.btrfs. btrfstune can activate those features. 

However, I miss two things: 
- How do I see on an existing FS which of these features are on? 
  btrfstune (it seems) can only "set", but not "get" the feature flags. 
- Is it worthwhile / recommended / safe to activate those on existing FS? 
  Are there any steps (e.g. balancing metadata with -musage=0, I'd guess) needed to make them become active afterwards? 

If there is any documentation on this and I missed it, please RTFM me. 

Cheers and thanks a lot, 
	Oliver

Re: btrfstune settings

[Duncan]

Oliver Freyermuth posted on Sun, 28 Aug 2016 05:42:39 +0200 as excerpted:

> Dear btrfs experts,
> 
> I hope this is the correct place to ask, the wiki and manpages did not
> help me on these questions.
> 
> BTRFS has gained extended inode refs, skinny metadata and no-holes quite
> a while ago and these are now the defaults for new mkfs.btrfs. btrfstune
> can activate those features.
> 
> However, I miss two things:
> - How do I see on an existing FS which of these features are on?
>   btrfstune (it seems) can only "set", but not "get" the feature flags.

Try btrfs-show-super <device>.  The incompat_flags section enumerates the 
flags that are on (at least with a reasonably new btrfs-progs).

> - Is it worthwhile / recommended / safe to activate those on existing
> FS?
>   Are there any steps (e.g. balancing metadata with -musage=0, I'd
>   guess) needed to make them become active afterwards?

If they're intended to be changed on an existing filesystem, btrfstune 
should allow it, otherwise it doesn't.  Node/leaf-size used to default to 
sectorsize (4K on x86/amd64), for instance, while now defaulting to 16K, 
but can't be changed on an existing filesystem, only at mkfs time, so 
that's the only place with the option.

In general it should be safe to activate flags via btrfstune, but since 
they'll generally apply only to newly written data, any benefits on a 
mature filesystem will be limited.  For that reason as well as to get the 
benefit of 16K nodesize which you can't except at creation, I recommend 
backing up and recreating the filesystem from a fresh mkfs.btrfs.

Since btrfs is considered still stabilizing, having a backup is very 
strongly recommended in any case unless the value of the data simply 
isn't worth the hassle, and once you have a backup tested, available and 
ready to use should it be necessary, blowing the existing filesystem away 
and starting with a fresh filesystem created with the options you want 
isn't much more difficult and may well be faster than a full balance 
anyway. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfstune settings

[Kai Krakow]

Am Sun, 28 Aug 2016 08:27:59 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@cox.net>:

> > However, I miss two things:
> > - How do I see on an existing FS which of these features are on?
> >   btrfstune (it seems) can only "set", but not "get" the feature
> > flags.  
> 
> Try btrfs-show-super <device>.  The incompat_flags section enumerates
> the flags that are on (at least with a reasonably new btrfs-progs).

It's also in the sub directories of /sys/fs/btrfs...

-- 
Regards,
Kai

Replies to list-only preferred.

Re: btrfstune settings

[Oliver Freyermuth]

(sorry if my Message-ID header is missing, I am not subscribed to the mailing list, 
so I reply using mail-archive)

> Try btrfs-show-super <device>.  The incompat_flags section enumerates the 
> flags that are on (at least with a reasonably new btrfs-progs).

Thanks a lot for this hint - I totally missed that, since I only looked 
at manpages and features of the "all-in-one" btrfs tool up to now. 
Also thanks for the hint to /sys/fs/btrfs, this is even more readable! 

And thanks a lot for your detailed answer(s) in general! 

> In general it should be safe to activate flags via btrfstune, but since 
> they'll generally apply only to newly written data, any benefits on a 
> mature filesystem will be limited.  For that reason as well as to get the 
> benefit of 16K nodesize which you can't except at creation, I recommend 
> backing up and recreating the filesystem from a fresh mkfs.btrfs.

That would certainly be the best option, however, 
I have two issues with that: 
- Any replay of a backup will do a lot of writes on the SSD, 
  reducing lifetime. 
- I did not yet figure out a working way to get a "complete" backup for a btrfs. 
  btrfs send is too slow for me, and rsync does an incomplete backup, 
  since all FS-special attrs (like the NOCOW attr from 'chattr +C')
  are not copied, even when xattr copy is on. 

My last backup replay was from an rsync backup (these I do regularly) 
and afterwards I did a manual fixing of these special attrs, which is some bit of extra work. 
Also, the machine is offline during backup replay, 
while I could still use it during a simple rebalance. 

The good news is my old FS already has 16K leafsize, it's only missing skinny metadata
(and no-holes, apparently that's not default in mkfs.btrfs yet?). 

This reduces the scope of my question a bit - is it sufficient and worthwhile to "btrfstune -x" 
and then "btrfs balance start -musage=0 /" to convert all metadata, and thus gain some space? 

Or are the gains not worth it and I should just wait until the next time I need to replay a backup for other reasons? 

Cheers and thanks for your reply, 
	Oliver

Re: btrfstune settings

[Kai Krakow]

Am Sun, 28 Aug 2016 18:18:52 +0200
schrieb Oliver Freyermuth <o.freyermuth@googlemail.com>:

> (sorry if my Message-ID header is missing, I am not subscribed to the
> mailing list, so I reply using mail-archive)
> 
> > Try btrfs-show-super <device>.  The incompat_flags section
> > enumerates the flags that are on (at least with a reasonably new
> > btrfs-progs).  
> 
> Thanks a lot for this hint - I totally missed that, since I only
> looked at manpages and features of the "all-in-one" btrfs tool up to
> now. Also thanks for the hint to /sys/fs/btrfs, this is even more
> readable! 
> 
> And thanks a lot for your detailed answer(s) in general! 
> 
> > In general it should be safe to activate flags via btrfstune, but
> > since they'll generally apply only to newly written data, any
> > benefits on a mature filesystem will be limited.  For that reason
> > as well as to get the benefit of 16K nodesize which you can't
> > except at creation, I recommend backing up and recreating the
> > filesystem from a fresh mkfs.btrfs.  
> 
> That would certainly be the best option, however, 
> I have two issues with that: 
> - Any replay of a backup will do a lot of writes on the SSD, 
>   reducing lifetime. 
> - I did not yet figure out a working way to get a "complete" backup
> for a btrfs. btrfs send is too slow for me, and rsync does an
> incomplete backup, since all FS-special attrs (like the NOCOW attr
> from 'chattr +C') are not copied, even when xattr copy is on. 

Try borgbackup, I'm using it very successfully. It is very fast,
supports very impressive deduplication and compression, retention
policies, and remote backups - and it is available as a single binary
version so you can more easily use it for disaster recovery. One
downside: while it seems to restore nocow attributes, it seems to do it
in a silly way (it first creates the file, then sets the attributes,
which of course won't work for nocow). I have not checked that
extensively, only had to restore once yet.

> My last backup replay was from an rsync backup (these I do regularly) 
> and afterwards I did a manual fixing of these special attrs, which is
> some bit of extra work. Also, the machine is offline during backup
> replay, while I could still use it during a simple rebalance. 

It's hard to circumvent this. I had once the idea to rsync backups to a
btrfs device, then during recovery use it as a seeding device. I never
tried it but it could work. Your system could be almost immediately back
online this way but you'll have to do some special extra steps to get
the seeding device back into a state which you can use again for your
backups.

> The good news is my old FS already has 16K leafsize, it's only
> missing skinny metadata (and no-holes, apparently that's not default
> in mkfs.btrfs yet?). 

AFAIR no-holes and skinny-metadata can be enabled on existing
filesystems...

> This reduces the scope of my question a bit - is it sufficient and
> worthwhile to "btrfstune -x" and then "btrfs balance start
> -musage=0 /" to convert all metadata, and thus gain some space? 

Not sure... Especially about no-holes. You could also try to
"defragment" meta data using

$ find /mnt/btrfs-pool -type d -print0 | xargs -0 btrfs fi defrag

> Or are the gains not worth it and I should just wait until the next
> time I need to replay a backup for other reasons? 

It's probably not worth the hassle... Wait until you are forced to
recreate the filesystem or you are having enough spare time to do it.
However, some settings can be changed for existing filesystems and I
would simply ignore the fact existing meta data needs to be converted.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: btrfstune settings

[Oliver Freyermuth]

> Try borgbackup, I'm using it very successfully. It is very fast,
> supports very impressive deduplication and compression, retention
> policies, and remote backups - and it is available as a single binary
> version so you can more easily use it for disaster recovery. One
> downside: while it seems to restore nocow attributes, it seems to do it
> in a silly way (it first creates the file, then sets the attributes,
> which of course won't work for nocow). I have not checked that
> extensively, only had to restore once yet.

Wow - this looks like the holy grail I've been waiting for, not sure how
I have missed that up to now. 
Especially the deduplication across several backupped systems on the backup target
is interesting, I originally planned to do that using duperemove on the backup target
to dedupe across the readonly snapshots. 
I'll certainly give borgbackup a spin as soon as I have time to look into it! 

> It's probably not worth the hassle... Wait until you are forced to
> recreate the filesystem or you are having enough spare time to do it.
> However, some settings can be changed for existing filesystems and I
> would simply ignore the fact existing meta data needs to be converted.

Thanks a lot for this hint and also your other points. I think I'll go that way,
comparing a "similar" system with skinny metadata and one without the space usage for metadata
seems not to be different by a noticeable factor. 
Since my systems are SSD only, to avoid unnecessary writes I'll then just wait until
I really need to replay a backup. 

Thanks a lot and best regards, 
	Oliver

Re: btrfstune settings

[Lionel Bouton]

Hi,

happy borgbackup user here. This is probably off-topic for most but as
many users probably are evaluating send/receive versus other backup
solutions, I'll keep linux-btrfs in the loop.

On 28/08/2016 20:10, Oliver Freyermuth wrote:
>> Try borgbackup, I'm using it very successfully. It is very fast,
>> supports very impressive deduplication and compression, retention
>> policies, and remote backups - and it is available as a single binary
>> version so you can more easily use it for disaster recovery. One
>> downside: while it seems to restore nocow attributes, it seems to do it
>> in a silly way (it first creates the file, then sets the attributes,
>> which of course won't work for nocow). I have not checked that
>> extensively, only had to restore once yet.
> Wow - this looks like the holy grail I've been waiting for, not sure how
> I have missed that up to now. 
> Especially the deduplication across several backupped systems on the backup target
> is interesting, I originally planned to do that using duperemove on the backup target
> to dedupe across the readonly snapshots.

Note that only one backup can happen at a given time on a single repository.
You'll probably have to both schedule the backups to avoid collisions
and use "--lock-wait" with a large enough parameter to avoid backup
failures.

There's another twist : borgbackup maintains a local index of the
repositories' content, when it detects that it is out of sync (if
several systems use the same repository it will) it has to update its
index from remote.
I'm not sure how heavy this update can be (it seems it uses some form of
delta). I have a ~user/.cache/borg of ~2GB for a user with ~9TB of data
to backup in ~8M files but I don't share repositories so I'm not
affected by this).

Best regards,

Lionel

Re: btrfstune settings

[Kai Krakow]

Am Sun, 28 Aug 2016 20:10:38 +0200
schrieb Oliver Freyermuth <o.freyermuth@googlemail.com>:

> > Try borgbackup, I'm using it very successfully. It is very fast,
> > supports very impressive deduplication and compression, retention
> > policies, and remote backups - and it is available as a single
> > binary version so you can more easily use it for disaster recovery.
> > One downside: while it seems to restore nocow attributes, it seems
> > to do it in a silly way (it first creates the file, then sets the
> > attributes, which of course won't work for nocow). I have not
> > checked that extensively, only had to restore once yet.  
> 
> Wow - this looks like the holy grail I've been waiting for, not sure
> how I have missed that up to now. 
> Especially the deduplication across several backupped systems on the
> backup target is interesting, I originally planned to do that using
> duperemove on the backup target to dedupe across the readonly
> snapshots. I'll certainly give borgbackup a spin as soon as I have
> time to look into it! 

Apparently, only one client can access the repo at the same time. So
you have to fiddle around with backup time windows and maybe the option
for lock timeouts and retries via your service manager. I'm using it to
backup a multi server shop system, it will first create an SQL dump,
then start backing up each host one after the next to a local
repository. In the end, I'll rsync it to an offsite location. It
requires some scripting and ssh foo but it works reliable.

At home, I'm using a simple systemd job to wake my system up at night
and put it back to sleep later:

$ systemctl cat internal-backup.{timer,service}
# /etc/systemd/system/internal-backup.timer
[Unit]
Description=Daily Backup Timer

[Timer]
OnCalendar=03:00
WakeSystem=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/internal-backup.service
[Unit]
Description=Daily Backup Service

[Service]
Environment=BORG_REPO=/mnt/private/backup/jupiter.sol.local.borg
Environment=BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
Environment=BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
Type=oneshot
IOSchedulingClass=idle
IOSchedulingPriority=7
CPUSchedulingPolicy=batch
Nice=3
ProtectHome=read-only
ProtectSystem=full
PrivateTmp=yes
ReadWriteDirectories=/root/.cache/borg
ReadOnlyDirectories=/mnt/btrfs-pool
WorkingDirectory=/mnt/btrfs-pool
ExecStart=/usr/bin/systemd-inhibit /usr/bin/borg create -v --stats
--exclude-caches --compression zlib --exclude
'pp:gentoo/usr/portage/distfiles' --exclude 'pp:gentoo/rootfs/var/tmp'
--exclude
'pp:gentoo/usr/portage/packages' ::'system@{now:%%Y%%m%%d-%%H%%M}' .
ExecStartPost=/usr/bin/borg prune -v --prefix 'system@' --keep-daily 7
--keep-weekly 5 --keep-monthly 12 --list --stats --save-space


The result then looks like this:

$ sudo -H borg
info /mnt/private/backup/jupiter.sol.local.borg::system@20160828-0255
Name: system@20160828-0255 Fingerprint:
786c9c8a18d5d04ba2a745d91777cf30345613b78d2afaa74192db871aa220de
Hostname: jupiter.sol.local Username: root
Time (start): Sun, 2016-08-28 02:55:24
Time (end):   Sun, 2016-08-28 03:08:13
Command line: /usr/lib/python-exec/python3.4/borg create -v --stats
--exclude-caches --compression zlib --exclude
pp:gentoo/usr/portage/distfiles --exclude pp:gentoo/rootfs/var/tmp
--exclude pp:gentoo/usr/portage/packages ::system@{now:%Y%m%d-%H%M} .
Number of files: 2356372

                Original size  Compressed size  Deduplicated size
This archive:         1.73 TB          1.49 TB            1.02 GB
All archives:        17.26 TB         14.94 TB            1.78 TB

                Unique chunks     Total chunks
Chunk index:          2858371         28473400


Initial backup creation is much slower than rsync but successive runs
are much faster (around 15-20 minutes instead of 1 hour).

-- 
Regards,
Kai

Replies to list-only preferred.

Re: btrfstune settings

[Kai Krakow]

Am Sun, 28 Aug 2016 18:18:52 +0200
schrieb Oliver Freyermuth <o.freyermuth@googlemail.com>:

> That would certainly be the best option, however, 
> I have two issues with that: 
> - Any replay of a backup will do a lot of writes on the SSD, 
>   reducing lifetime. 

I'm using bcache in front of my harddisks (writeback mode) to get
almost-SSD performance. That way, when I restore, I can simply switch
to writearound mode, restore, then switch back to writeback mode.

But in the long term, writeback mode will be extremely heavy on SSD
wearing when the bcache is small because it will have to evict a lot of
data from the cache. I'm using 500GB bcache for 3x 1TB harddisk. 500GB
is more than my usual working set so bcache almost never has to evict
data from the cache.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: btrfstune settings

[David Sterba]

Hi,

On Sun, Aug 28, 2016 at 05:42:39AM +0200, Oliver Freyermuth wrote:
> I hope this is the correct place to ask, the wiki and manpages did not
> help me on these questions. 
> 
> BTRFS has gained extended inode refs, skinny metadata and no-holes
> quite a while ago and these are now the defaults for new mkfs.btrfs.
> btrfstune can activate those features. 
> 
> However, I miss two things: 
> - How do I see on an existing FS which of these features are on? 
>   btrfstune (it seems) can only "set", but not "get" the feature flags. 
> - Is it worthwhile / recommended / safe to activate those on existing FS? 
>   Are there any steps (e.g. balancing metadata with -musage=0, I'd
>   guess) needed to make them become active afterwards? 
> 
> If there is any documentation on this and I missed it, please RTFM me. 

all your questions should be now answered and documentation cross
referenced among the relevant manual pages (will be synced to wiki at
release time:

https://github.com/kdave/btrfs-progs/blob/devel/Documentation/mkfs.btrfs.asciidoc
https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfstune.asciidoc
https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfs-man5.asciidoc

see sections 'FILESYSTEM FEATURES'.

Re: btrfstune settings

[Oliver Freyermuth]

Hi,

Am 01.09.2016 um 19:14 schrieb David Sterba:
> all your questions should be now answered and documentation cross
> referenced among the relevant manual pages (will be synced to wiki at
> release time:
> 
> https://github.com/kdave/btrfs-progs/blob/devel/Documentation/mkfs.btrfs.asciidoc
> https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfstune.asciidoc
> https://github.com/kdave/btrfs-progs/blob/devel/Documentation/btrfs-man5.asciidoc
> 
> see sections 'FILESYSTEM FEATURES'.
> 
Thanks a lot for this documentation effort! Just to confirm, this indeed
answers all my questions as a user.

I hope it will be helpful to many other btrfs-users in the future.

Best wishes,
	Oliver