Confusion about snapshots containers

Confusion about snapshots containers

[Tim Cuthbertson]

I have recently switched from multiple partitions with multiple
btrfs's to a flat layout. I will try to keep my question concise.

I am confused as to whether a snapshots container should be a normal
directory or a mountable subvolume. I do not understand how it can be
a normal directory while being at the same level as, for example, a
rootfs subvolume. This is with the understanding that the rootfs is
NOT at the btrfs top level.

Which should it be, a normal directory or a mountable subvolume
directly under btrfs top level? If either way can work, what are the
pros and cons of each?

Thank you

Tim Cuthbertson

Re: Confusion about snapshots containers

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1766 bytes --]

On Wed, Mar 29, 2017 at 04:27:30PM -0500, Tim Cuthbertson wrote:
> I have recently switched from multiple partitions with multiple
> btrfs's to a flat layout. I will try to keep my question concise.
> 
> I am confused as to whether a snapshots container should be a normal
> directory or a mountable subvolume. I do not understand how it can be
> a normal directory while being at the same level as, for example, a
> rootfs subvolume. This is with the understanding that the rootfs is
> NOT at the btrfs top level.
> 
> Which should it be, a normal directory or a mountable subvolume
> directly under btrfs top level? If either way can work, what are the
> pros and cons of each?

   The current best practice recommendation is that it should be a
normal directory, not contained within any of the subvolumes that are
being snapshotted. So (using the @-prefix convention to indicate a
subvol), you'd have something like this:

<top-level>
   @root
   @home
   snapshots
      root
         @2017-03-28
         @2017-03-29
      home
         @2017-03-28
         @2017-03-29

   To use this, mount the top level of the FS (-o subvolid=0) on a
known path, such as /media/btrfs/<fslabel>/, and do the subvol
management, and nothing else, under that mount.

(Optionally, you can flatten the dir hierarchy to
/snapshots/@root-2017-03-28, but I prefer the slightly deeper version
above).

   The snapshots container can be either a dir or a subvol, but you
gain almost nothing from it being a subvol, and you lose the ability
to move subvols/snapshots in and out of it cheaply with mv. Hence the
recommendation to use a directory.

   Hugo.

-- 
Hugo Mills             | O tempura! O moresushi!
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Fwd: Confusion about snapshots containers

[Tim Cuthbertson]

---------- Forwarded message ----------
From: Hugo Mills <hugo@carfax.org.uk>
Date: Wed, Mar 29, 2017 at 4:55 PM
Subject: Re: Confusion about snapshots containers
To: Tim Cuthbertson <ratcheer@gmail.com>
Cc: "linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>


On Wed, Mar 29, 2017 at 04:27:30PM -0500, Tim Cuthbertson wrote:
> I have recently switched from multiple partitions with multiple
> btrfs's to a flat layout. I will try to keep my question concise.
>
> I am confused as to whether a snapshots container should be a normal
> directory or a mountable subvolume. I do not understand how it can be
> a normal directory while being at the same level as, for example, a
> rootfs subvolume. This is with the understanding that the rootfs is
> NOT at the btrfs top level.
>
> Which should it be, a normal directory or a mountable subvolume
> directly under btrfs top level? If either way can work, what are the
> pros and cons of each?

   The current best practice recommendation is that it should be a
normal directory, not contained within any of the subvolumes that are
being snapshotted. So (using the @-prefix convention to indicate a
subvol), you'd have something like this:

<top-level>
   @root
   @home
   snapshots
      root
         @2017-03-28
         @2017-03-29
      home
         @2017-03-28
         @2017-03-29

   To use this, mount the top level of the FS (-o subvolid=0) on a
known path, such as /media/btrfs/<fslabel>/, and do the subvol
management, and nothing else, under that mount.

(Optionally, you can flatten the dir hierarchy to
/snapshots/@root-2017-03-28, but I prefer the slightly deeper version
above).

   The snapshots container can be either a dir or a subvol, but you
gain almost nothing from it being a subvol, and you lose the ability
to move subvols/snapshots in and out of it cheaply with mv. Hence the
recommendation to use a directory.

   Hugo.

--
Hugo Mills             | O tempura! O moresushi!
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |


Thank you Hugo. That seems to work.

So, another question...

Do I then leave the top level mounted all the time for snapshots, or
should I create them, send them to external storage, and umount until
next time?

Re: Fwd: Confusion about snapshots containers

[Duncan]

Tim Cuthbertson posted on Wed, 29 Mar 2017 18:20:52 -0500 as excerpted:

> So, another question...
> 
> Do I then leave the top level mounted all the time for snapshots, or
> should I create them, send them to external storage, and umount until
> next time?

Keep in mind that because snapshots contain older versions of whatever 
they're snapshotting, they're a potential security issue, at least if 
some of those older versions are libs or binaries.  Consider the fact 
that you may have had security-updates since the snapshot, thus leaving 
your working copies unaffected by whatever security vulns the updates 
fixed.  If the old versions remain around where normal users have access 
to them, particularly if they're setuid or similar, they have access to 
those old and now known vulns in setuid executables!  (Of course users 
can grab vulnerable versions elsewhere or build them themselves, but they 
can't set them setuid root unless they /are/ root, so finding an existing 
setuid-root executable with known vulns is finding the keys to the 
kingdom.)

So keeping snapshots unmounted and out of the normally accessible 
filesystem tree by default is recommended, at least if you're at all 
concerned about someone untrusted getting access to a normal user account 
and being able to use snapshots with known vulns of setuid executables as 
root-escalation methods.

Another possibility is setting the snapshot subdir 700 perms, so non-
super-users can't normally access anything in it anyway.  Of course 
that's a problem if you want them to have access to snapshots of their 
own stuff for recovery purposes, but it's useful if you can do it.

Good admins will do both of these at once if possible as they know and 
observe the defense-in-depth mantra, knowing all too well how easy a 
single layer of defense yields to fat-fingering or previously unknown 
vulns.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Fwd: Confusion about snapshots containers

[Tim Cuthbertson]

On Wed, Mar 29, 2017 at 10:46 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Tim Cuthbertson posted on Wed, 29 Mar 2017 18:20:52 -0500 as excerpted:
>
>> So, another question...
>>
>> Do I then leave the top level mounted all the time for snapshots, or
>> should I create them, send them to external storage, and umount until
>> next time?
>
> Keep in mind that because snapshots contain older versions of whatever
> they're snapshotting, they're a potential security issue, at least if
> some of those older versions are libs or binaries.  Consider the fact
> that you may have had security-updates since the snapshot, thus leaving
> your working copies unaffected by whatever security vulns the updates
> fixed.  If the old versions remain around where normal users have access
> to them, particularly if they're setuid or similar, they have access to
> those old and now known vulns in setuid executables!  (Of course users
> can grab vulnerable versions elsewhere or build them themselves, but they
> can't set them setuid root unless they /are/ root, so finding an existing
> setuid-root executable with known vulns is finding the keys to the
> kingdom.)
>
> So keeping snapshots unmounted and out of the normally accessible
> filesystem tree by default is recommended, at least if you're at all
> concerned about someone untrusted getting access to a normal user account
> and being able to use snapshots with known vulns of setuid executables as
> root-escalation methods.
>
> Another possibility is setting the snapshot subdir 700 perms, so non-
> super-users can't normally access anything in it anyway.  Of course
> that's a problem if you want them to have access to snapshots of their
> own stuff for recovery purposes, but it's useful if you can do it.
>
> Good admins will do both of these at once if possible as they know and
> observe the defense-in-depth mantra, knowing all too well how easy a
> single layer of defense yields to fat-fingering or previously unknown
> vulns.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thank you, Duncan. I will try to take all that into consideration.
These are really just fairly simple personal home systems, but
security is still important.

Tim Cuthbertson

Re: Fwd: Confusion about snapshots containers

[Austin S. Hemmelgarn]

On 2017-03-30 09:07, Tim Cuthbertson wrote:
> On Wed, Mar 29, 2017 at 10:46 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>> Tim Cuthbertson posted on Wed, 29 Mar 2017 18:20:52 -0500 as excerpted:
>>
>>> So, another question...
>>>
>>> Do I then leave the top level mounted all the time for snapshots, or
>>> should I create them, send them to external storage, and umount until
>>> next time?
>>
>> Keep in mind that because snapshots contain older versions of whatever
>> they're snapshotting, they're a potential security issue, at least if
>> some of those older versions are libs or binaries.  Consider the fact
>> that you may have had security-updates since the snapshot, thus leaving
>> your working copies unaffected by whatever security vulns the updates
>> fixed.  If the old versions remain around where normal users have access
>> to them, particularly if they're setuid or similar, they have access to
>> those old and now known vulns in setuid executables!  (Of course users
>> can grab vulnerable versions elsewhere or build them themselves, but they
>> can't set them setuid root unless they /are/ root, so finding an existing
>> setuid-root executable with known vulns is finding the keys to the
>> kingdom.)
>>
>> So keeping snapshots unmounted and out of the normally accessible
>> filesystem tree by default is recommended, at least if you're at all
>> concerned about someone untrusted getting access to a normal user account
>> and being able to use snapshots with known vulns of setuid executables as
>> root-escalation methods.
>>
>> Another possibility is setting the snapshot subdir 700 perms, so non-
>> super-users can't normally access anything in it anyway.  Of course
>> that's a problem if you want them to have access to snapshots of their
>> own stuff for recovery purposes, but it's useful if you can do it.
>>
>> Good admins will do both of these at once if possible as they know and
>> observe the defense-in-depth mantra, knowing all too well how easy a
>> single layer of defense yields to fat-fingering or previously unknown
>> vulns.
>>
>> --
>> Duncan - List replies preferred.   No HTML msgs.
>> "Every nonfree program has a lord, a master --
>> and if you use the program, he is your master."  Richard Stallman
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> Thank you, Duncan. I will try to take all that into consideration.
> These are really just fairly simple personal home systems, but
> security is still important.
On the note of the old binaries and libraries bit, nodev, noexec, and 
nosuid are all per-mountpoint, not per-volume, so you can mitigate some 
of the rsik by always mounting with those flags.  Despite that, it's 
still a good idea to not have anything more than you need mounted at any 
given time (it's a lot harder to screw up a filesystem which isn't mounted).

Re: Confusion about snapshots containers

[Kai Krakow]

Am Wed, 29 Mar 2017 16:27:30 -0500
schrieb Tim Cuthbertson <ratcheer@gmail.com>:

> I have recently switched from multiple partitions with multiple
> btrfs's to a flat layout. I will try to keep my question concise.
> 
> I am confused as to whether a snapshots container should be a normal
> directory or a mountable subvolume. I do not understand how it can be
> a normal directory while being at the same level as, for example, a
> rootfs subvolume. This is with the understanding that the rootfs is
> NOT at the btrfs top level.
> 
> Which should it be, a normal directory or a mountable subvolume
> directly under btrfs top level? If either way can work, what are the
> pros and cons of each?

I think there is no exact standard you could follow. Many distributions
seems to go for the standard to prepend subvolumes with "@" if they are
meant to be mounted. However, I'm not doing so.

Generally speaking, subvolumes organize your volume into logical
containers which make sense to be snapshotted on their own. Snapshots
won't propagate to sub-subvolumes, it is important to keep that in mind
while designing your idea of a structure.

I'm using it like this:

In subvol=0 I have the following subvolumes:

/* - contains distribution specific file systems
/home - contains home directories
/snapshots - contains snapshots I want to keep
/other
  - misc stuff, i.e. a dump of the subvol structure in a txt
  - a copy of my restore script
  - some other supporting docs for restore
  - this subvolume is kept in sync with my backup volume

This means: If I mount one of the rootfs, my home will not be part of
this mount automatically because that subvolume is out of scope of the
rootfs.

Now I have the following subvolumes below these:

/gentoo/rootfs - rootfs of my main distribution
  Note 1: Everything below (except subvolumes) should be maintained
          by the package manager.
  Note 2: currently I installed no other distributions
  Note 3: I could have called it main-system-rootfs

/gentoo/usr
  - actually not a subvolume but a directory for volumes shareable with
    other distribution instances

/gentoo/usr/portage - portage, shareable by other gentoo instances
/gentoo/usr/src - the gentoo linux kernel sources, shareable

The following are put below /gentoo/rootfs so they not need to be
mounted separately:

/gentoo/rootfs/var/log
  - log volume because I don't want it to be snapshotted
/gentoo/rootfs/var/tmp
  - tmp volume because it makes no sense to be snapshotted
/gentoo/rootfs/var/lib/machines
  - subvolume for keeping nspawn containers
/gentoo/rootfs/var/lib/machines/*
  - different machines cloned from each other
/gentoo/rootfs/usr/local
  - non-package manager stuff

/home/myuser - my user home
/home/myuser/.VirtualBox
  - VirtualBox machines because I want them snapshotted separately

/etc/fstab now only mounts subvolumes outside of the scope
of /gentoo/rootfs:

LABEL=system /home        btrfs compress=lzo,subvol=home,noatime
LABEL=system /usr/portage btrfs noauto,compress=lzo,subvol=gentoo/usr/portage,noatime,x-systemd.automount
LABEL=system /usr/src     btrfs noauto,compress=lzo,subvol=gentoo/usr/src,noatime,x-systemd.automount

Additionally, I mount the subvol=0 for two special purposes:

LABEL=system /mnt/btrfs-pool btrfs noauto,compress=lzo,subvolid=0,x-systemd.automount,noatime

First: For managing all the subvolumes and have an untampered view
(without tmpfs or special purpose mounts) to the volumes.

Second: To take a clean backup of the whole system.

Now, I can give the bootloader subvol=gentoo/rootfs to select which
system to boot (or make it the default subvolume).

Maybe you get the idea and find that idea helpful.

PS: It can make sense to have var/lib/machines outside of the rootfs
scope if you want to share it with other distributions.

-- 
Regards,
Kai

Replies to list-only preferred.