[DRAFT] Snapshot boot design document

[DRAFT] Snapshot boot design document

[Bryn M. Reeves]

We're drafting a proposal for a generic snapshot boot mechanism, to
allow booting into previous snapshots of the system for LVM2 and
BTRFS volumes (as well as Stratis[1] in the future).

The current version of the design is available on GitHub (as a PDF
or LyX source):

  https://github.com/bmr-cymru/snapshot-boot-docs

This documents the current state of support for booting into snapshot
volumes, as well as a brief survey of some of the current competitive
implementations, and proposes new software components to facilitate
easier use of the facility and additional functionality that can be
provided once basic infrastructure is available.

The document is still under active development and new revisions will be
issued as feedback is received and integrated: we'll review the work
as we go and decide on a final implementation.

Regards,
Bryn.

[1] https://stratis-storage.github.io/

Re: [DRAFT] Snapshot boot design document

[Thorsten Kukuk]

Hi,

On Thu, Jun 01, Bryn M. Reeves wrote:

> We're drafting a proposal for a generic snapshot boot mechanism, to
> allow booting into previous snapshots of the system for LVM2 and
> BTRFS volumes (as well as Stratis[1] in the future).

I like that proposal.

Some comments from me:

We use snapshots currently for another use case: transactional updates.
Means, create a snapshot, update the system inside of the snapshot and
if no error occurs, mark this snapshot as the new default root filesystem
for the next reboot.

About Future extensions: I see 
1.4.1 Goal: Simplify management of large numbers of snapshots 

The biggest roblems we see with our customers are already very good
summarized there, that's why I think this is something important
for the tools, the one taking the snapshot and the boot manager.
And should be for that reason not only be a future extension.

Wish from me for a future extension:
Automatic rollback on boot failure. If this happens after the root
filesystem is mounted, it is pretty trivial. But if the problem is
that the harddisk cannot be found, maybe because the kernel update
has problems with the harddisk controller, this needs help of the
boot manager.

2 Linux boot process management

Isn't there "gummiboot" missing? To my knowledge this is needed
for the Rasperry PI 3.

Between, it's openSUSE, lower O, upper U, since the beginning.
Same for SUSE, it's upper U since 18 years or so.

2.3.2 perl-Bootlaoder
Support for anything else beside grub2 is, if not already removed,
outdated and no longer official supported.

5.3.3 Support expiry of snapshots by user specified policy
You should consider something like priorisation for snapshots. Else the
result of a lot of small changes to config files can lead to deletion
of snapshots with the real big changes.
For that reason we had to introduce this for SLES, else a rollback
wouldn't have been possible too fast.

  Thorsten

-- 
Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany
GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg)

Re: [DRAFT] Snapshot boot design document

[Thorsten Kukuk]

On Wed, Jun 07, Thorsten Kukuk wrote:

> 
> Hi,
> 
> On Thu, Jun 01, Bryn M. Reeves wrote:
> 
> > We're drafting a proposal for a generic snapshot boot mechanism, to
> > allow booting into previous snapshots of the system for LVM2 and
> > BTRFS volumes (as well as Stratis[1] in the future).
> 
> I like that proposal.
> 
> Some comments from me:

One more: please don't forget read-only root filesystems with all of
your designs. btrfs snapshots are read-only, so if you boot an old
snapshot, you have a read-only root filesystem.
And if you play with transactional updates by snapshots and rollback,
you will really like read-only root filesystems.

  Thorsten

-- 
Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & CaaSP
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany
GF: Felix Imendoerffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nuernberg)

Re: [DRAFT] Snapshot boot design document

[Bryn M. Reeves]

On Thu, Jun 08, 2017 at 10:17:05AM +0200, Thorsten Kukuk wrote:
> On Wed, Jun 07, Thorsten Kukuk wrote:
> One more: please don't forget read-only root filesystems with all of
> your designs. btrfs snapshots are read-only, so if you boot an old
> snapshot, you have a read-only root filesystem.
> And if you play with transactional updates by snapshots and rollback,
> you will really like read-only root filesystems.

Good point again, thanks. Coming from a device-mapper and LVM2 background
that's my natural priority, but we are very clear that we want to create
a solution that will work for the full range of snapshot techniques now
available on Linux (including potentially ZFS although it's not a high
priority, simply due to how few distributions ship it).

Regards,
Bryn.

Re: [DRAFT] Snapshot boot design document

[Bryn M. Reeves]

Hi Thorsten,

Thanks for looking at the proposal, and taking time to give feedback:
it's very much appreciated.

On Wed, Jun 07, 2017 at 06:23:30PM +0200, Thorsten Kukuk wrote:
> We use snapshots currently for another use case: transactional updates.
> Means, create a snapshot, update the system inside of the snapshot and
> if no error occurs, mark this snapshot as the new default root filesystem
> for the next reboot.

I think this is a sub-case of use case (4) on page 4 of v0.55. I'll
definitely add some text to make this clear though, as there are
different ways of using the snapshot in this case (update in snap,
and then switch or merge, or update in main and rollback).

> About Future extensions: I see 
> 1.4.1 Goal: Simplify management of large numbers of snapshots 
> 
> The biggest roblems we see with our customers are already very good
> summarized there, that's why I think this is something important
> for the tools, the one taking the snapshot and the boot manager.
> And should be for that reason not only be a future extension.

Agreed: I am pushing to include these future plans asap. The fact
they are listed separately is really more a consequence of how
the proposal came about (to address the booting problem), rather
than how we view their relative importance.

> Wish from me for a future extension:
> Automatic rollback on boot failure. If this happens after the root
> filesystem is mounted, it is pretty trivial. But if the problem is
> that the harddisk cannot be found, maybe because the kernel update
> has problems with the harddisk controller, this needs help of the
> boot manager.

This is a good point, and some boot loaders (grub2 e.g.) do have the
ability to record success or failure for a boot attempt, and to
modify behaviour on the next boot. Where possible we should make use
of this type of facility.

> 2 Linux boot process management
> 
> Isn't there "gummiboot" missing? To my knowledge this is needed
> for the Rasperry PI 3.

A very good point: if nothing else, gummiboot/systemd-boot is now
the reference implementation of the BootLoader Specification, which
is really key to the work that we are planning.

I'll add a section on this.
 
> Between, it's openSUSE, lower O, upper U, since the beginning.
> Same for SUSE, it's upper U since 18 years or so.

Ugh. Thank you for the correction - I have been writing these marks
wrongly for many, many years apparently. I'll get these fixed.

> 2.3.2 perl-Bootlaoder
> Support for anything else beside grub2 is, if not already removed,
> outdated and no longer official supported.

Ack, thanks. I'll add a note.

> 5.3.3 Support expiry of snapshots by user specified policy
> You should consider something like priorisation for snapshots. Else the
> result of a lot of small changes to config files can lead to deletion
> of snapshots with the real big changes.
> For that reason we had to introduce this for SLES, else a rollback
> wouldn't have been possible too fast.

Yes, this is something I've thought about, but didn't really have
a clear enough idea of to include so far. I'll think some more about
it - even just a quick summary as you've written might do for now,
so that the need does not get lost.

Thanks again for taking the time to look at this.

Regards,
Bryn.