* Re: OK to take hourly snapshots, then cull older ones?
2013-10-15 4:05 OK to take hourly snapshots, then cull older ones? David Madden
@ 2013-10-15 4:43 ` Marc MERLIN
2013-10-15 4:47 ` Duncan
` (2 subsequent siblings)
3 siblings, 0 replies; 9+ messages in thread
From: Marc MERLIN @ 2013-10-15 4:43 UTC (permalink / raw)
To: David Madden; +Cc: linux-btrfs
On Mon, Oct 14, 2013 at 09:05:33PM -0700, David Madden wrote:
> I'd like to use BTRFS to do something like the old NetApp snapshot
> system: every hour or so, there'd be a snapshot, then the 23 of the
> snapshots during a day would be deleted, leaving just a day snapshot,
> then after a month, 6 of 7 snapshots would be deleted, leaving just a
> week snapshot, and so on.
>
> Is this a reasonable thing to do in a cron job with a BTRFS filesystem?
> Apart from running out of space, are there any resources that might get
> used up? Has anybody done this for a year or two in an active
> filesystem, and encountered success or weirdness?
Way ahead of you there, been doing this for a year, no problems:
gandalfthegreat:~$ cat /etc/cron.d/btrfs_backup
0 * * * * root test -d /mnt/btrfs_pool1 && grep -q rootflags=subvol=root /proc/cmdline && /var/local/scr/btrfs_snaps hourly 3 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
1 0 * * * root test -d /mnt/btrfs_pool1 && grep -q rootflags=subvol=root /proc/cmdline && /var/local/scr/btrfs_snaps daily 4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
2 0 * * 0 root test -d /mnt/btrfs_pool1 && grep -q rootflags=subvol=root /proc/cmdline && /var/local/scr/btrfs_snaps weekly 4 | egrep -v '(Create a snapshot of|Will delete the oldest|Delete subvolume|Making snapshot of )'
/var/local/scr/btrfs_snaps:
---------------------------
#!/bin/bash
: ${BTRFSROOT:=/mnt/btrfs_pool1}
DATE="$(date '+%Y%m%d_%H:%M:%S')"
type=${1:-hourly}
keep=${2:-3}
cd "$BTRFSROOT"
for i in *
do
# Skip snapshot names.
[ "${i/_/-}" != "$i" ] && continue
# Skip duplicate dirs once a year on DST 1h rewind.
test -d "$BTRFSROOT/${i}_${type}_$DATE" && continue
echo "Making snapshot of $type"
/sbin/btrfs subvolume snapshot "$BTRFSROOT"/$i "$BTRFSROOT/${i}_${type}_$DATE"
count="$(ls -d ${i}_${type}_* | wc -l)"
echo "Will delete the oldest $clip snapshots for $type"
clip=$(( $count - $keep ))
if [ $clip -gt 0 ]; then
for sub in $(ls -d ${i}_${type}_* | head -n $clip)
do
#echo "Will delete $sub"
/sbin/btrfs subvolume delete "$sub"
done
fi
done
---------------------------
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: OK to take hourly snapshots, then cull older ones?
2013-10-15 4:05 OK to take hourly snapshots, then cull older ones? David Madden
2013-10-15 4:43 ` Marc MERLIN
@ 2013-10-15 4:47 ` Duncan
2013-10-15 4:53 ` Roman Mamedov
2013-11-03 11:50 ` Matthias G. Eckermann
3 siblings, 0 replies; 9+ messages in thread
From: Duncan @ 2013-10-15 4:47 UTC (permalink / raw)
To: linux-btrfs
David Madden posted on Mon, 14 Oct 2013 21:05:33 -0700 as excerpted:
> I'd like to use BTRFS to do something like the old NetApp snapshot
> system:
> every hour or so, there'd be a snapshot, then the 23 of the snapshots
> during a day would be deleted, leaving just a day snapshot, then after a
> month, 6 of 7 snapshots would be deleted, leaving just a week snapshot,
> and so on.
>
> Is this a reasonable thing to do in a cron job with a BTRFS filesystem?
> Apart from running out of space, are there any resources that might get
> used up? Has anybody done this for a year or two in an active
> filesystem, and encountered success or weirdness?
There's discussion of this idea along with links to existing tools/
scripts for it, on the wiki:
https://btrfs.wiki.kernel.org
In particular, see documentation, guides and usage, use cases,
2. snapshots and subvolumes, 2.2. backups/time-machine. However, that
you didn't already know that was covered indicates that you either
weren't aware of the wiki, or haven't read much on it recently, so
there's likely a lot more information there that you'll find useful if
you spend some time looking around and reading.
(I haven't done a whole lot with snapshotting myself as it doesn't fit my
use case very well, but I knew about it from reading the wiki and had
tagged it in my mind to look up again later should I need the
information, so it was a matter of just a few seconds to find it again
and type the path above so you could find it too. Since I /haven't/ done
much with snapshotting myself, I can't help much in saying which of the
listed tools will be easiest, but that script link points at a list post
with a pre-made script and crontab entries that look like they do just
about exactly what you outline. =:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: OK to take hourly snapshots, then cull older ones?
2013-10-15 4:05 OK to take hourly snapshots, then cull older ones? David Madden
2013-10-15 4:43 ` Marc MERLIN
2013-10-15 4:47 ` Duncan
@ 2013-10-15 4:53 ` Roman Mamedov
2013-10-15 5:05 ` David Madden
2013-11-03 11:50 ` Matthias G. Eckermann
3 siblings, 1 reply; 9+ messages in thread
From: Roman Mamedov @ 2013-10-15 4:53 UTC (permalink / raw)
To: David Madden; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]
On Mon, 14 Oct 2013 21:05:33 -0700
David Madden <dhm@mersenne.com> wrote:
> I'd like to use BTRFS to do something like the old NetApp snapshot
> system: every hour or so, there'd be a snapshot, then the 23 of the
> snapshots during a day would be deleted, leaving just a day snapshot,
> then after a month, 6 of 7 snapshots would be deleted, leaving just a
> week snapshot, and so on.
>
> Is this a reasonable thing to do in a cron job with a BTRFS filesystem?
> Apart from running out of space, are there any resources that might get
> used up? Has anybody done this for a year or two in an active
> filesystem, and encountered success or weirdness?
Sure, that's one of the more awesome uses of btrfs.
But keep in mind that old snapshots on the same FS are not to be used instead
of a proper backup to external media/servers. If a block happened not to
change for a year, and it gets damaged on disk, it will become damaged in all
the snapshots all the way back to a year ago, and you lose that data.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: OK to take hourly snapshots, then cull older ones?
2013-10-15 4:53 ` Roman Mamedov
@ 2013-10-15 5:05 ` David Madden
2013-10-15 5:14 ` Avi Miller
0 siblings, 1 reply; 9+ messages in thread
From: David Madden @ 2013-10-15 5:05 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2351 bytes --]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 14-Oct-2013 21:53 , Roman Mamedov wrote:
> Sure, that's one of the more awesome uses of btrfs.
>
> But keep in mind that old snapshots on the same FS are not to be
> used instead of a proper backup to external media/servers. If a
> block happened not to change for a year, and it gets damaged on
> disk, it will become damaged in all the snapshots all the way back
> to a year ago, and you lose that data.
I haven't looked at the wiki carefully enough to understand this, but
could one reasonably back up the snapshots of one BTRFS filesystem to
an independent BTRFS filesystem, in a more efficient way than just
dump/restore or cpio or something?
It sounds like you could rely on on-the-fly deduping to save space on
the backup filesystem, but you'd still be reading the whole source
snapshot tree and sending it to be deduped/written. Seems like you
could improve things by just pulling the changed blocks from the
snapshot to the previous snapshot, which would (I guess) be the only
blocks actually _in_ the snapshot.
I guess what would really help is a "Best Practices Guide for BTRFS
Snapshotting and Backup." Does anything like that exist?
Thanks,
- --
Mersenne Law LLC · www.mersenne.com · +1-503-679-1671
- - Small Business, Startup and Intellectual Property Law -
1500 SW First Ave. · Suite 1170 · Portland, Oregon 97201
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.20 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJSXM0RAAoJEIQakGgVoKPCbNUP/RppplehqwASZXuIHcVGZBW/
4Q0sKcA19O4VVW8/Byy5Wa1cH3DHf24X4S8IHdZIvdn30EfYNI7q489ldmwQk9h8
To2EdsbpG9VlRWQ6W3zSv1iwqp1i/1W1miB6dkk4hHIEZ64BilaFyKInuCJCdS1T
anQFmFBUMGbMGdMXbIxrLgHhwYr7VSyp4RXxtCspeQHENT1zUiy6tyzEDqoP5vf+
ZDScBXK8G/MFijQTr6BWfCyiuS3OFostCp9dITmwNtLc194Ae/TDF7+ZNo/CzpOy
uP4pEZDPBYraEEcZBeRm5Jefa6VPkr16aoiJKjEH0StHimuHDNX17epgLTqSJx+J
JN0AJoxLxHQa7A8ny3aOzIjibKGP0Aa8KBfl6saRD8DMFtILHLMPCUOJ+l16Pcj/
gfipOZEpiBDHY4EIzYJRq/yYMHmw5SrvZeoPKdIPgDjJ7ovB9fteTwmLVSaiccM0
nd7TTMYiqm6PPnh7FEZWopfMEvx35LSxxI5Q6nuNqgQ33EgtYtM5yJC1cISJH1gf
HwQWmr2yqwZ/xgEQc0jI4LO6ISTx4xsY0NKM5aUr6ayzbmE5kRRfnNkUKpDd8by9
CpuFg898XIO+9Cd9do0MvLjVxg62WU8YxPiC+bcwc5tGKAiVjQaTeTyfYrLDmGaj
sGCmWIXC8dGrrqsaGthw
=ioaG
-----END PGP SIGNATURE-----
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: OK to take hourly snapshots, then cull older ones?
2013-10-15 4:05 OK to take hourly snapshots, then cull older ones? David Madden
` (2 preceding siblings ...)
2013-10-15 4:53 ` Roman Mamedov
@ 2013-11-03 11:50 ` Matthias G. Eckermann
2013-11-05 2:51 ` Marc MERLIN
3 siblings, 1 reply; 9+ messages in thread
From: Matthias G. Eckermann @ 2013-11-03 11:50 UTC (permalink / raw)
To: David Madden, linux-btrfs
Hello David and all,
On Mon, Oct 14, 2013 at 21:05 David Madden wrote:
> I'd like to use BTRFS to do something like the old NetApp
> snapshot system: every hour or so, there'd be a snapshot,
> then the 23 of the snapshots during a day would be
> deleted, leaving just a day snapshot, then after a month,
> 6 of 7 snapshots would be deleted, leaving just a week
> snapshot, and so on.
This is implemented in "Snapper", see:
http://snapper.io/
It's by default delivered with openSUSE and SUSE Linux
Enterprise, binaries are available for "everything else"
as well.
> Is this a reasonable thing to do in a cron job with a
> BTRFS filesystem? Apart from running out of space, are
> there any resources that might get used up? Has anybody
> done this for a year or two in an active filesystem, and
> encountered success or weirdness?
Space is _the_ main issue. If you want automated comparison
of snapshots, you'll also need some CPU time in addition.
Enjoy.
so long -
MgE
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: OK to take hourly snapshots, then cull older ones?
2013-11-03 11:50 ` Matthias G. Eckermann
@ 2013-11-05 2:51 ` Marc MERLIN
2013-11-06 0:08 ` Matthias G. Eckermann
0 siblings, 1 reply; 9+ messages in thread
From: Marc MERLIN @ 2013-11-05 2:51 UTC (permalink / raw)
To: Matthias G. Eckermann; +Cc: David Madden, linux-btrfs
On Sun, Nov 03, 2013 at 12:50:24PM +0100, Matthias G. Eckermann wrote:
> Hello David and all,
>
> On Mon, Oct 14, 2013 at 21:05 David Madden wrote:
>
> > I'd like to use BTRFS to do something like the old NetApp
> > snapshot system: every hour or so, there'd be a snapshot,
> > then the 23 of the snapshots during a day would be
> > deleted, leaving just a day snapshot, then after a month,
> > 6 of 7 snapshots would be deleted, leaving just a week
> > snapshot, and so on.
>
> This is implemented in "Snapper", see:
> http://snapper.io/
> It's by default delivered with openSUSE and SUSE Linux
> Enterprise, binaries are available for "everything else"
> as well.
Just curious, what does it do more than the 20 line shellscript I
posted?
http://marc.merlins.org/linux/scripts/btrfs_snaps
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: OK to take hourly snapshots, then cull older ones?
2013-11-05 2:51 ` Marc MERLIN
@ 2013-11-06 0:08 ` Matthias G. Eckermann
0 siblings, 0 replies; 9+ messages in thread
From: Matthias G. Eckermann @ 2013-11-06 0:08 UTC (permalink / raw)
To: Marc MERLIN, David Madden, linux-btrfs
Hello Marc and all,
On Mon, Nov 04, 2013 at 06:51:11PM -0800 Marc MERLIN wrote:
> On Sun, Nov 03, 2013 at 12:50PM +0100, Matthias G. Eckermann wrote:
> > On Mon, Oct 14, 2013 at 21:05 David Madden wrote:
> >
> > > I'd like to use BTRFS to do something like the old NetApp
> > > snapshot system: every hour or so, there'd be a snapshot,
> > > then the 23 of the snapshots during a day would be
> > > deleted, leaving just a day snapshot, then after a month,
> > > 6 of 7 snapshots would be deleted, leaving just a week
> > > snapshot, and so on.
> >
> > This is implemented in "Snapper", see:
> > http://snapper.io/
> > It's by default delivered with openSUSE and SUSE Linux
> > Enterprise, binaries are available for "everything else"
> > as well.
>
> Just curious, what does it do more than the 20 line shellscript I
> posted?
> http://marc.merlins.org/linux/scripts/btrfs_snaps
You asked ...
Snapper does not only handle snapshotting itself, but a
lot of steps around it, to make it easier for an
administrator to handle snapshotting.
While primarily offering a cmdline utility, Snapper also
has integration into D-BUS, thus other tools can ask for
snapshots on a specific subvolume. To make this secure,
access rights are stored in a per subvolume configuration
among other attributes and rules.
Based on that Snapper offers (rough summary):
- Managing configurations (create, delete, list, ...)
- Managing snapshots (create, delete, list)
- Add and modify metadata of snapshots
- Compare snapshots aka "diff"
- Roll-back snapshots (selective roll-back, per file)
Within the configuration you can add
- rules for creation and removal of snapshots
- access rights
Off-Topic: Snapper also works with DM based snapshots.
Other projects using snapper:
* Samba 4 has a prototype implementation of Windows'
FSRVP server for SMB share shadow-copies ("snapshots")
using Snapper via D-Bus. See:
http://snapper.io/videos.html
* The Systems Management stack of openSUSE / SUSE
Linux Enterprise (ZYpp, YaST) uses Snapper
automatically, if "/" is on btrfs.
<SelfAdulation>
One also can use it for regular snapshots to the $HOME
directory; see my description here:
https://www.suse.com/communities/conversations/menu-du-jour-vivaneau-vert-sur-lit-de-legumes-au-beurre-et-supremes-de-pamplemousse-2/
and here
https://www.suse.com/communities/conversations/sieste-siesta/
</SelfAdulation>
Hope this explains.
so long -
MgE
--
Matthias G. Eckermann, Berlin, Germany
Private : matthias.g.eckermann@t-online.de
Business: mge@suse.com
^ permalink raw reply [flat|nested] 9+ messages in thread