Possible to wait for snapshot deletion?

Possible to wait for snapshot deletion?

[Kai Krakow]

Hi!

Is it technically possible to wait for a snapshot completely purged from 
disk? I imagine an option like "--wait" for btrfs delete subvolume.

This would fit some purposes I'm planning to implement:

* In a backup scenario have a subprocess which deletes snapshots one by one,
  starting with the oldest. If free space raises above a certain threshold,
  pause the subprocess. If number of kept snapshots falls below a certain
  threshold, exit the subprocess. When the backup job finished, it joins the
  subprocess to wait for a pending subvolume deletion if any, then syncs the
  filesystem and waits some grace time for uncommitted writes, then shuts
  the system down or hibernates it.

* Wait for pending background subvolume deletion before putting the system
  to sleep.

* Get better control of cron jobs working with subvolumes so jobs either do
  not overlap or do not put too much parallel work on the file system.

Scrubbing does allow that. Why don't snapshots do? At least I found no 
options for it.

Background information: If btrfs is shut down improperly, already deleted 
subvolume entries may reappear in their directories. The deletion is not 
gracefully resumed. Since deletion of subvolumes/snapshots can take hours, 
this is a problem for systems that are not guaranteed to be up all the time 
or suffer from disconnected power, especially if multiple snapshots are 
being deleted at once. With an option to wait, a script managing snapshot 
deletion could more gracefully resume its job.

-- 
Replies to list only preferred.

Re: Possible to wait for snapshot deletion?

[Brendan Hide]

On 2014/02/13 09:02 PM, Kai Krakow wrote:
> Hi!
>
> Is it technically possible to wait for a snapshot completely purged from
> disk? I imagine an option like "--wait" for btrfs delete subvolume.
>
> This would fit some purposes I'm planning to implement:
>
> * In a backup scenario
I have a similar use-case for this also involving backups. In my case I 
have a script that uses a btrfs filesystem for the backup store using 
snapshots. At the end of each run, if diskspace usage is below a 
predefined threshold, it will delete old snapshots until the diskspace 
usage is below that threshold again.

Of course, the first time I added the automatic deletion, it deleted far 
more than was necessary due to the fact that the actual freeing of 
diskspace is asynchronous from the command completion. I ended up 
setting a small delay (of about 60 seconds) between each iteration and 
also set it to monitor system load. If load is not low enough after the 
delay then it waits another 60 seconds.

This complicated (frankly broken) workaround would be completely 
unnecessary with a --wait switch.

Alternatively, perhaps a knob where we can see if a subvolume deletion 
is in progress could help.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Possible to wait for snapshot deletion?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1995 bytes --]

On Thu, Feb 13, 2014 at 08:02:43PM +0100, Kai Krakow wrote:
> Hi!
> 
> Is it technically possible to wait for a snapshot completely purged from 
> disk? I imagine an option like "--wait" for btrfs delete subvolume.

   I'm sure there's been some patches for that around here somewhere,
some time ago. I guess they didn't get merged.

   Hugo.

> This would fit some purposes I'm planning to implement:
> 
> * In a backup scenario have a subprocess which deletes snapshots one by one,
>   starting with the oldest. If free space raises above a certain threshold,
>   pause the subprocess. If number of kept snapshots falls below a certain
>   threshold, exit the subprocess. When the backup job finished, it joins the
>   subprocess to wait for a pending subvolume deletion if any, then syncs the
>   filesystem and waits some grace time for uncommitted writes, then shuts
>   the system down or hibernates it.
> 
> * Wait for pending background subvolume deletion before putting the system
>   to sleep.
> 
> * Get better control of cron jobs working with subvolumes so jobs either do
>   not overlap or do not put too much parallel work on the file system.
> 
> Scrubbing does allow that. Why don't snapshots do? At least I found no 
> options for it.
> 
> Background information: If btrfs is shut down improperly, already deleted 
> subvolume entries may reappear in their directories. The deletion is not 
> gracefully resumed. Since deletion of subvolumes/snapshots can take hours, 
> this is a problem for systems that are not guaranteed to be up all the time 
> or suffer from disconnected power, especially if multiple snapshots are 
> being deleted at once. With an option to wait, a script managing snapshot 
> deletion could more gracefully resume its job.
> 

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
     --- Gentlemen!  You can't fight here! This is the War Room! ---     

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: Possible to wait for snapshot deletion?

[Garry T. Williams]

On 2-13-14 20:02:43 Kai Krakow wrote:
> Is it technically possible to wait for a snapshot completely purged
> from disk? I imagine an option like "--wait" for btrfs delete
> subvolume.

This may be what you're looking for:

    http://www.spinics.net/lists/linux-btrfs/msg29833.html 

-- 
Garry T. Williams

Re: Possible to wait for snapshot deletion?

[Holger Hoffstätte]

On Thu, 13 Feb 2014 20:02:43 +0100, Kai Krakow wrote:

> Is it technically possible to wait for a snapshot completely purged from 
> disk? I imagine an option like "--wait" for btrfs delete subvolume.

That would indeed be sweet (see pointer to patch by Garry Williams).

In the meantime have you tried issuing a simple "sync" in your script
after every snapshot deletion? I've observed that this kicks the cleaner
into action.

-h

Re: Possible to wait for snapshot deletion?

[Kai Krakow]

Garry T. Williams <gtwilliams@gmail.com> schrieb:

> On 2-13-14 20:02:43 Kai Krakow wrote:
>> Is it technically possible to wait for a snapshot completely purged
>> from disk? I imagine an option like "--wait" for btrfs delete
>> subvolume.
> 
> This may be what you're looking for:
> 
>     http://www.spinics.net/lists/linux-btrfs/msg29833.html
> 

Looks fantastic. Can I vote to get it into official sources? ;-)

-- 
Replies to list only preferred.

Re: Possible to wait for snapshot deletion?

[Kai Krakow]

Holger Hoffstätte <holger.hoffstaette@googlemail.com> schrieb:

> On Thu, 13 Feb 2014 20:02:43 +0100, Kai Krakow wrote:
> 
>> Is it technically possible to wait for a snapshot completely purged from
>> disk? I imagine an option like "--wait" for btrfs delete subvolume.
> 
> That would indeed be sweet (see pointer to patch by Garry Williams).
> 
> In the meantime have you tried issuing a simple "sync" in your script
> after every snapshot deletion? I've observed that this kicks the cleaner
> into action.

I was under the impression that a sync command just doesn't do the trick. It 
kicks in the cleaner but returns to early. But I din't try that again 
lately.

But I'm actually currently using "btrfs fs sync" to do it because a normal 
"sync" command would affect all my filesystems and I'd like to keep it 
isolated.

I'll post a link to my backup script for reference in another post in this 
thread.

-- 
Replies to list only preferred.

Re: Possible to wait for snapshot deletion?

[Kai Krakow]

Brendan Hide <brendan@swiftspirit.co.za> schrieb:

>> Is it technically possible to wait for a snapshot completely purged from
>> disk? I imagine an option like "--wait" for btrfs delete subvolume.
>>
>> This would fit some purposes I'm planning to implement:
>>
>> * In a backup scenario
>
> I have a similar use-case for this also involving backups. In my case I
> have a script that uses a btrfs filesystem for the backup store using
> snapshots. At the end of each run, if diskspace usage is below a
> predefined threshold, it will delete old snapshots until the diskspace
> usage is below that threshold again.

Yeah, I thought of that approach first, too... But:

> Of course, the first time I added the automatic deletion, it deleted far
> more than was necessary due to the fact that the actual freeing of
> diskspace is asynchronous from the command completion. I ended up
> setting a small delay (of about 60 seconds) between each iteration and
> also set it to monitor system load. If load is not low enough after the
> delay then it waits another 60 seconds.

Due to btrfs' behavior that cannot reliably work as you also figured out. It 
will need quirky work-arounds totally dependent on system load.

> This complicated (frankly broken) workaround would be completely
> unnecessary with a --wait switch.

That's why I had the idea. ;-)

> Alternatively, perhaps a knob where we can see if a subvolume deletion
> is in progress could help.

Like "btrfs scrub status"...

If you're feeling curious, here's what I've implemented yet (snapshot 
deletion still on the todo list):

https://gist.github.com/kakra/5520370

My idea is to fork-off a background process which constantly removes old 
snapshots (within sane bounds yet to be defined) and then exits. Some 
mechanism has to be found to not run into a deadlock situation here. Then, 
at the end issue a bash "wait" to join the processes again and run a final 
sync. However, I don't like to issue explicit syncs from within the 
subprocess as the filesystem is busy with rsync at the same time.

PS: Yes, I know there's btrfs send/receive - but it doesn't seem ready for 
the big show yet because it still has many strange quirks and should not be 
run unattended yet.

-- 
Replies to list only preferred.

Re: Possible to wait for snapshot deletion?

[David Sterba]

On Fri, Feb 14, 2014 at 01:12:58AM +0100, Kai Krakow wrote:
> Garry T. Williams <gtwilliams@gmail.com> schrieb:
> 
> > On 2-13-14 20:02:43 Kai Krakow wrote:
> >> Is it technically possible to wait for a snapshot completely purged
> >> from disk? I imagine an option like "--wait" for btrfs delete
> >> subvolume.
> > 
> > This may be what you're looking for:
> > 
> >     http://www.spinics.net/lists/linux-btrfs/msg29833.html
> > 
> 
> Looks fantastic. Can I vote to get it into official sources? ;-)

This has been in the development integration branch for some time and
is present in the integration branch pending for the next release.

Re: Possible to wait for snapshot deletion?

[David Sterba]

On Thu, Feb 13, 2014 at 08:02:43PM +0100, Kai Krakow wrote:
> Is it technically possible to wait for a snapshot completely purged from 
> disk? I imagine an option like "--wait" for btrfs delete subvolume.

I have the patch WIP, will look at it again.

> This would fit some purposes I'm planning to implement:
> 
> * In a backup scenario have a subprocess which deletes snapshots one by one,
>   starting with the oldest. If free space raises above a certain threshold,
>   pause the subprocess. If number of kept snapshots falls below a certain
>   threshold, exit the subprocess. When the backup job finished, it joins the
>   subprocess to wait for a pending subvolume deletion if any, then syncs the
>   filesystem and waits some grace time for uncommitted writes, then shuts
>   the system down or hibernates it.
> 
> * Wait for pending background subvolume deletion before putting the system
>   to sleep.
> 
> * Get better control of cron jobs working with subvolumes so jobs either do
>   not overlap or do not put too much parallel work on the file system.

These usecases should be covered by the 'wait' command.

> If btrfs is shut down improperly, already deleted 
> subvolume entries may reappear in their directories. The deletion is not 
> gracefully resumed. Since deletion of subvolumes/snapshots can take hours, 
> this is a problem for systems that are not guaranteed to be up all the time 
> or suffer from disconnected power, especially if multiple snapshots are 
> being deleted at once. With an option to wait, a script managing snapshot 
> deletion could more gracefully resume its job.

As mentioned down the thread, next progs release will be able to force a
transaction commit when the subvoulme is deleted.