Possibility to have a "transient" snapshot?

Possibility to have a "transient" snapshot?

[James West]

This is just a random idea that popped through my mind while I was 
looking into hardening a filesystem against damage, might be 
impractical, but the idea seems promising, and well suited to a snapshot 
file system.

I'm sure some creative shell scripting could do something like this 
already, but I was more looking for something more bulletproof.

General idea would be to have a transient snapshot (optional quota 
support possibility here) on top of a base snapshot (possibly readonly). 
On system start/restart (whether clean or dirty), the transient snapshot 
would be flushed, and the system would restart the snapshot, basically 
restarting from the base snapshot. If desired, the transient snapshot 
could be promoted to a regular snapshot (say after a software upgrade). 
If desired, a different base snapshot could be selected (although I'm 
sure the file system would have to be restarted to do this)

 From a caching perspective, this could make a noticable performance 
difference, since if you're running in a transient snapshot, the file 
system can be _extremely_ lazy about committing changes to disk.

For the optional quote support I mentioned, on an unattended box, if the 
quota gets exceeded, a system reboot would probably fully correct the 
system. (Presumably a log file got out of control in that situation).

Re: Possibility to have a "transient" snapshot?

[Duncan]

James West posted on Fri, 05 Dec 2014 12:27:39 -0600 as excerpted:

> I'm sure some creative shell scripting could do something like this
> already,

Indeed...

> but I was more looking for something more bulletproof.

See below...

> General idea would be to have a transient snapshot (optional quota
> support possibility here) on top of a base snapshot (possibly readonly).
> On system start/restart (whether clean or dirty), the transient snapshot
> would be flushed, and the system would restart the snapshot, basically
> restarting from the base snapshot.

So to this point we're effectively restoring from a "golden image" at 
boot, wiping the history of the last session away.  I guess a lot of 
internet cafes do this sort of thing, either directly, or (these days) 
running from a VM, with multiple terminals, each logging into their own VM 
image.

> If desired, the transient snapshot
> could be promoted to a regular snapshot (say after a software upgrade).
> If desired, a different base snapshot could be selected (although I'm
> sure the file system would have to be restarted to do this)

So optionally make the transient image permanent, effectively upgrading 
it to the golden image (presumably with a fallback to the previous golden 
image if necessary).

>  From a caching perspective, this could make a noticable performance
> difference, since if you're running in a transient snapshot, the file
> system can be _extremely_ lazy about committing changes to disk.

Indeed.


I'm doing something here with a similar effect for root, except much less 
complicated and it doesn't require btrfs, tho btrfs snapshotting would be 
a useful if complicating variant.

Basically all I did is stick ro in the mount-options for root, so when it 
would normally be remounted rw, it's remounted with all the extra 
operating options, but still ro.

I only switch it to rw when I'm actively doing system maintenance.  Since 
I have all the other options I want loaded and in fstab already, all I 
have to do is remount rw, and it automatically picks up the compression, 
autodefrag, etc, from fstab and the previous mount.

Tho I don't use snapshots, instead preferring a backup of root, done 
manually when I've boot-tested the current config and believe it's stable 
enough, plus that it's time for a new backup.  Since root is only 8 GiB 
in size, backup can and does consist of a simple mkfs on the backup (also 
8 GiB), followed by mounting it and a mount-bind of root somewhere else 
so I can get at anything normally under a mountpoint and a straight copy 
won't accidentally stray into other filesystems, and copy everything 
over.  fstab is already a symlink (effectively) pointing to 
fstab.working, with an fstab.backup already prepared as well, so after 
the copy I switch the fstab symlink pointer on the backup and modify an 
ID file (making it easier to double-check what root is actually mounted).

I then umount the backup and the bind-mount, reboot, and select the grub 
menu entry that sets the kernel commandline root= to the backup instead 
of working copy, and verify that the backup is actually bootable.

Thus I effectively have a working (normal root) and backup (backup root) 
"golden image", with the working golden image mounted writable and 
updated whenever I update or modify system configuration, and the 
fallback image selectable from grub.

Of course I have secondary and tertiary backups as well, tertiary not 
normally attached, just in case, as well.

Gets rid of a lot of headaches, since I don't have to worry about root 
being corrupted in the normal crash-case at all, and if the working root 
/is/ ever unbootable either due to bad update or corruption while mounted 
writable, I can always boot to several levels of backup and have the 
fully working system I'm used to, including access to all manpages, X, 
the internet, etc, just as it was when I did the backup, to use as a 
recovery image. =:^)

/home and /var/log (with others including the packages partition mounted 
only on demand) are of course mounted writable, with their own backups as 
well, but it's nice to have a fully functional and uncorrupted root 
complete with all tools and reference material, that I know will boot 
even if they're corrupted, to use when doing repair if they do get 
corrupted. =:^)

I'm using fully independent filesystems instead of subvolumes and 
snapshots, since it didn't make any sense to me to risk putting all my 
data eggs in a single filesystem basket, and have them all broken at once 
if the bottom falls out, given that subvolumes and snapshots are still on 
the same filesystem, such that if one subvolume/snapshot is corrupted and 
unmountable, there's a good chance they're all gone.  Better to have 
partition and filesystem barriers in place to contain the damage, 
particularly when corruption on the writable /var/log could easily mean 
the read-only root that I'd otherwise be using to repair /var/log, is 
corrupted as well.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Possibility to have a "transient" snapshot?

[Chris Murphy]

On Fri, Dec 5, 2014 at 11:27 AM, James West <james@terminalsystems.com> wrote:

> General idea would be to have a transient snapshot (optional quota support
> possibility here) on top of a base snapshot (possibly readonly). On system
> start/restart (whether clean or dirty), the transient snapshot would be
> flushed, and the system would restart the snapshot, basically restarting
> from the base snapshot.

Sounds similar to this idea:
http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html

About 1/3 of the way down it gets to a proposal to Btrfs as a way to
get to a stateless system, which is basically what you want to be able
to rollback to. A variation on this that might serve the use case
better is seed device. You can either drop the added device that
stores changes to the seed device, or the volume (seed+added device)
can become another seed if you want to make the current state
persistent at next boot.

And still another possibility is overlayfs, which isn't Btrfs specific.



-- 
Chris Murphy

Re: Possibility to have a "transient" snapshot?

[James West]

I was just looking into using overlayfs, and although it has some 
promise, I think it's biggest drawback is the upperdir will have to be 
some sort of storage backed filesystem. From my limited understanding of 
tmpfs, it's not supposed to be the greatest with many large files (and 
my system in particular would be downloading many large movies/videos, 
and doing any kind of os update to test it would involve many changes 
all over the volume, which could be problematic to commit to a golden 
state.)

I could partition the main drive in 2 parts, and dynamically zero-out 
then create the volume in the second partition on each boot, but I'm 
still saving no drive writes, and not really extending the life of the 
hardware (which is one of my premises.)

On 05/12/2014 11:12 PM, Chris Murphy wrote:
> On Fri, Dec 5, 2014 at 11:27 AM, James West <james@terminalsystems.com> wrote:
>
>> General idea would be to have a transient snapshot (optional quota support
>> possibility here) on top of a base snapshot (possibly readonly). On system
>> start/restart (whether clean or dirty), the transient snapshot would be
>> flushed, and the system would restart the snapshot, basically restarting
>> from the base snapshot.
> Sounds similar to this idea:
> http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html
>
> About 1/3 of the way down it gets to a proposal to Btrfs as a way to
> get to a stateless system, which is basically what you want to be able
> to rollback to. A variation on this that might serve the use case
> better is seed device. You can either drop the added device that
> stores changes to the seed device, or the volume (seed+added device)
> can become another seed if you want to make the current state
> persistent at next boot.
>
> And still another possibility is overlayfs, which isn't Btrfs specific.
>
>
>

Re: Possibility to have a "transient" snapshot?

[Robert White]

On 12/10/2014 11:52 AM, James West wrote:
> I was just looking into using overlayfs, and although it has some
> promise, I think it's biggest drawback is the upperdir will have to be
> some sort of storage backed filesystem. From my limited understanding of
> tmpfs, it's not supposed to be the greatest with many large files (and
> my system in particular would be downloading many large movies/videos,
> and doing any kind of os update to test it would involve many changes
> all over the volume, which could be problematic to commit to a golden
> state.)
>
> I could partition the main drive in 2 parts, and dynamically zero-out
> then create the volume in the second partition on each boot, but I'm
> still saving no drive writes, and not really extending the life of the
> hardware (which is one of my premises.)

You are over-thinking the "transient" part way too much. If the 
underlying device is not an SSD then most of your wear is immaterial. 
And if it is an SSD, you wear is still pretty damn immaterial.

The "full weight" snapshots are plenty "transient" if you delete them 
between uses and they don't do any recursive copying so they are almost 
wear-free.

[If you want to wear out a hard disk, park it's heads a lot. (The My 
Book series of WD external enclosures had a _horrible_ default of 
parking the heads after every eight seconds of idle time. Ouch.)]

A normal hard disk's runtime (provided its not a lemon) is shorter than 
its mean write wear time anyway.

So the best thing to do in your case is to customize your initramfs to 
do what you need and then "hide" your stuff from normal use. Consider 
this (untested but) hiper-simple init script... (assumes busybox in the 
initramfs providing mount and a few of the other basic tools and the 
btrfs command).

--- snip ---
#!/bin/bash
mkdir /dev
mount -t devtmpfs none /dev
mkdir /scratch
mount -t btrfs /dev/sda1 /scratch
if [ -d /scratch/active ]; then
   btrfs subvol del /scratch/active
fi
btrfs subvol snap /scratch/__Master /scratch/active
mkdir /root
mount -o subvol=/active /dev/sda1 /root
umount /scratch
rmdir /scratch
umount /dev
busybox switch_root /root /sbin/init "$@"
--- snip ---


Every time you boot it makes a fresh snapshot of the /__Master subvolume 
of /dev/sda1 into /active and mounts that as root then
treats that as the root of the file system.

Estimated human-scale time to run this script is in the 
one-second-or-less range.

None of the files in /__Master are then visible to the running system, 
so they won't be subject to search via find or locate etc.

Problem solved.

When you want to do maintenance you can log into you box and do

mount /dev/sda1 /mnt

at which point /__Master is visible as /mnt/__Master.

You can do your backup snapshots and your maintenance via the /mnt view 
without purturbing your running system.

chroot /mnt/__Master /bin/bash

That gives you the "native view" of your master system in that shell. 
 From that shell all your package tools will work just fine etc.

You can prep new or covariant system is snapshots parallel to /__Master 
and use rename to select the __Master for the next reboot.

Even better, since snapshots of snapshots are not degenerate in any way 
at all, you can create multiple system roots as /Whatever and 
/OtherThing (and so on) and always do your maintenance there. Then 
before any reboot you can snap /mnt/Whatever into /mnt/__Master (using 
the same technique as for /active) and then reboot. On that reboot the 
new /__Master will be the master for the new /active.

All of the snapshot activities are almost instant (except for the 
cleanup of the previous /active if it's full of a lot of changes, but 
that will happen in the background so you don't have to care much about 
that time).


ASIDE: And I keep pointing people at it, but I do a lot of experimental 
boot behaviors while testing hardware and such for my job, and my 
baseline initramfs builder at http://underdog.sourceforge.net is easy to 
customize and plenty stable. It already sucks in the btrfs and command 
and friends, and you can take control of the boot-up to do experimental 
tweaks by adding "bash" to the kernel boot args for an individual boot.