restic backup with btrfs /

restic backup with btrfs /

[Ulli Horlacher]

restic (https://restic.net/) is a great backup tool but has some
limitations or design flaws: one is, it believes that any subvolume is on a
different filesystem. This means: "restic backup --one-file-system /" will
only backup the root subvolume, but no other subvolumes like /home
/var/spool etc... 

One has to add every subvolume to the argument list. Bad if you 
create new subvolumes and forget to update the backup cronjob.
When you later need to restore a file, there will be none...

My idea is now: I do not backup the original /, but do:

mount --bind / /backup/restic
restic backup /backup/restic
umount /backup/restic

Next evolution step: 
I could recursivly mount-bind other filesystems into /backup/restic/
For example:

mount --bind /local /backup/restic/local
mount --bind /data /backup/restic/ldata

That I would have a "all in one backup".

What do you think of this idea?

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<20250628173308.GB847325@tik.uni-stuttgart.de>

Re: restic backup with btrfs /

[Mark Harmstone]

On 28/06/2025 6.33 pm, Ulli Horlacher wrote:
> restic (https://restic.net/) is a great backup tool but has some
> limitations or design flaws: one is, it believes that any subvolume is on a
> different filesystem. This means: "restic backup --one-file-system /" will
> only backup the root subvolume, but no other subvolumes like /home
> /var/spool etc...
> 
> One has to add every subvolume to the argument list. Bad if you
> create new subvolumes and forget to update the backup cronjob.
> When you later need to restore a file, there will be none...

This is a bug in Restic that needs to be fixed, though they may not see it that
way. The issue is that btrfs uses a different device number for each subvolume -
I believe this is because subvols can share inode numbers, and VFS gets confused
otherwise.

The same behaviour can be seen with rsync, which restic might well use
internally.

> My idea is now: I do not backup the original /, but do:
> 
> mount --bind / /backup/restic
> restic backup /backup/restic
> umount /backup/restic
> 
> Next evolution step:
> I could recursivly mount-bind other filesystems into /backup/restic/
> For example:
> 
> mount --bind /local /backup/restic/local
> mount --bind /data /backup/restic/ldata

This is unlikely to work, as bind mounts will also be seen as different
filesystems.

> That I would have a "all in one backup".
> 
> What do you think of this idea?
> 

Re: restic backup with btrfs /

[Christoph Hellwig]

On Mon, Jun 30, 2025 at 11:07:06AM +0100, Mark Harmstone wrote:
> On 28/06/2025 6.33 pm, Ulli Horlacher wrote:
> > restic (https://restic.net/) is a great backup tool but has some
> > limitations or design flaws: one is, it believes that any subvolume is on a
> > different filesystem. This means: "restic backup --one-file-system /" will
> > only backup the root subvolume, but no other subvolumes like /home
> > /var/spool etc...
> > 
> > One has to add every subvolume to the argument list. Bad if you
> > create new subvolumes and forget to update the backup cronjob.
> > When you later need to restore a file, there will be none...
> 
> This is a bug in Restic that needs to be fixed, though they may not see
> it that way.

.. because it isn't.  btrfs is the buggy one here breaking the old
Unix tradition also (somewhat vaguely) encoded in Posix that st_dev must
not changed except at a mount point.

Re: restic backup with btrfs /

[Mark Harmstone]

On 30/06/2025 2.41 pm, Christoph Hellwig wrote:
> On Mon, Jun 30, 2025 at 11:07:06AM +0100, Mark Harmstone wrote:
>> On 28/06/2025 6.33 pm, Ulli Horlacher wrote:
>>> restic (https://restic.net/) is a great backup tool but has some
>>> limitations or design flaws: one is, it believes that any subvolume is on a
>>> different filesystem. This means: "restic backup --one-file-system /" will
>>> only backup the root subvolume, but no other subvolumes like /home
>>> /var/spool etc...
>>>
>>> One has to add every subvolume to the argument list. Bad if you
>>> create new subvolumes and forget to update the backup cronjob.
>>> When you later need to restore a file, there will be none...
>>
>> This is a bug in Restic that needs to be fixed, though they may not see
>> it that way.
> 
> .. because it isn't.  btrfs is the buggy one here breaking the old
> Unix tradition also (somewhat vaguely) encoded in Posix that st_dev must
> not changed except at a mount point.

I know. But if POSIX assumes that every inode has a unique combination of
st_dev and st_ino, and that both are 64 bits, that's incompatible with the
design of btrfs.

Re: restic backup with btrfs /

[Christoph Hellwig]

On Mon, Jun 30, 2025 at 02:53:23PM +0100, Mark Harmstone wrote:
> > .. because it isn't.  btrfs is the buggy one here breaking the old
> > Unix tradition also (somewhat vaguely) encoded in Posix that st_dev must
> > not changed except at a mount point.
> 
> I know. But if POSIX assumes that every inode has a unique combination of
> st_dev and st_ino, and that both are 64 bits, that's incompatible with the
> design of btrfs.

I know, we had a few discussions on that.

Re: restic backup with btrfs /

[Ulli Horlacher]

On Mon 2025-06-30 (11:07), Mark Harmstone wrote:
> On 28/06/2025 6.33 pm, Ulli Horlacher wrote:
> 
> > restic (https://restic.net/) is a great backup tool but has some
> > limitations or design flaws: one is, it believes that any subvolume is on a
> > different filesystem. This means: "restic backup --one-file-system /" will
> > only backup the root subvolume, but no other subvolumes like /home
> > /var/spool etc...
> 
> This is a bug in Restic that needs to be fixed

I have reported this already at the restic bug tracker, but the restic
developpers say, it is a btrfs problem and they will not fix it.


> > My idea is now: I do not backup the original /, but do:
> >
> > mount --bind / /backup/restic
> > restic backup /backup/restic
> > umount /backup/restic
> >
> > Next evolution step:
> > I could recursivly mount-bind other filesystems into /backup/restic/
> > For example:
> >
> > mount --bind /local /backup/restic/local
> > mount --bind /data /backup/restic/ldata
> 
> This is unlikely to work, as bind mounts will also be seen as different
> filesystems.

No problem here, because I will will use the option --one-file-system
See above.

"restic backup /" vs "restic backup /backup/restic" !
The first one includes EVERYTHING: /proc /nfs /mnt/tmp ...


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<82b6ddab-0eef-47b8-8010-9b09fcb70444@harmstone.com>

Re: restic backup with btrfs /

[Ulli Horlacher]

On Mon 2025-06-30 (19:30), Ulli Horlacher wrote:

> > > mount --bind / /backup/restic
> > > restic backup /backup/restic
> > > umount /backup/restic
> > >
> > 
> > This is unlikely to work, as bind mounts will also be seen as different
> > filesystems.
> 
> No problem here, because I will will use the option --one-file-system

TYPO! I will NOT use the option --one-file-system here!


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<20250630173041.GA1048979@tik.uni-stuttgart.de>

Re: restic backup with btrfs /

[Robert Krig]

Why don't you just create a backup script, where you create a list of 
subvolumes dynamically and then back those up, rather than hardcode 
specific subvolumes?

I would use btrfs tools to get a list of subvolumes and their paths, 
create snapshots and mount them in a similar directory tree and point 
restic at the snapshot and backup up from there.
Either that, or if you insist on use --one-file-system, backup up each 
subvolume snapshot individually.

Am 28.06.25 um 19:33 schrieb Ulli Horlacher:
> restic (https://restic.net/) is a great backup tool but has some
> limitations or design flaws: one is, it believes that any subvolume is on a
> different filesystem. This means: "restic backup --one-file-system /" will
> only backup the root subvolume, but no other subvolumes like /home
> /var/spool etc...
>
> One has to add every subvolume to the argument list. Bad if you
> create new subvolumes and forget to update the backup cronjob.
> When you later need to restore a file, there will be none...
>
> My idea is now: I do not backup the original /, but do:
>
> mount --bind / /backup/restic
> restic backup /backup/restic
> umount /backup/restic
>
> Next evolution step:
> I could recursivly mount-bind other filesystems into /backup/restic/
> For example:
>
> mount --bind /local /backup/restic/local
> mount --bind /data /backup/restic/ldata
>
> That I would have a "all in one backup".
>
> What do you think of this idea?
>

Re: restic backup with btrfs /

[Ulli Horlacher]

On Thu 2025-07-03 (10:00), Robert Krig wrote:

> Why don't you just create a backup script, where you create a list of
> subvolumes dynamically and then back those up, rather than hardcode
> specific subvolumes?

Indeed, this is my plan :-)
I have used "harcoded subvolumes" only as a simple example to demonstrate
my idea.


> I would use btrfs tools to get a list of subvolumes and their paths,

With standard btrfs commands this is not easy, but I have already written
btrfs_list which does this job:

root@fex:~# btrfs_list /local/test
ACCESS-MODE SUBVOLUME <- SNAPSHOT-PARENT
rw /local/test
ro /local/test/.snapshot/2025-06-29_1414.test <- /local/test
ro /local/test/.snapshot/2025-06-29_1416.test <- /local/test
ro /local/test/.snapshot/2025-06-29_1418.test <- /local/test
rw /local/test/sv1
rw /local/test/sv1/sv1_1
rw /local/test/sv1/sv1_1/ss <- /local/test/sv1/sv1_1
rw /local/test/sv1/sv1_2
rw /local/test/sv2


> create snapshots and mount them in a similar directory tree and point
> restic at the snapshot and backup up from there.

In this case ".snapshot" is in the restic path, which I do not like.
Therefore I will use mount --bind ...

Meanwhile I have written btrfs_rsnapshot to create and delete snapshots of
nested subvolumes recursivly. Example:

root@fex:~# btrfs_rsnapshot /local
Create a snapshot of '/local' in '/local/.snapshot/_local'
Create a snapshot of '/local/home' in '/local/.snapshot/_local/home'
Create a snapshot of '/local/home/framstag/blubb' in '/local/.snapshot/_local/home/framstag/blubb'
Create a snapshot of '/local/home/smc/test' in '/local/.snapshot/_local/home/smc/test'
Create a snapshot of '/local/test' in '/local/.snapshot/_local/test'
Create a snapshot of '/local/test/sv1' in '/local/.snapshot/_local/test/sv1'
Create a snapshot of '/local/test/sv1/sv1_1' in '/local/.snapshot/_local/test/sv1/sv1_1'
Create a snapshot of '/local/test/sv1/sv1_1/ss' in '/local/.snapshot/_local/test/sv1/sv1_1/ss'
Create a snapshot of '/local/test/sv1/sv1_2' in '/local/.snapshot/_local/test/sv1/sv1_2'
Create a snapshot of '/local/test/sv2' in '/local/.snapshot/_local/test/sv2'
Create a snapshot of '/local/tmp/test' in '/local/.snapshot/_local/tmp/test'

root@fex:~# btrfs_rsnapshot -d /local
Delete subvolume (no-commit): '/local/.snapshot/_local/tmp/test'
Delete subvolume (no-commit): '/local/.snapshot/_local/test/sv2'
Delete subvolume (no-commit): '/local/.snapshot/_local/test/sv1/sv1_2'
Delete subvolume (no-commit): '/local/.snapshot/_local/test/sv1/sv1_1/ss'
Delete subvolume (no-commit): '/local/.snapshot/_local/test/sv1/sv1_1'
Delete subvolume (no-commit): '/local/.snapshot/_local/test/sv1'
Delete subvolume (no-commit): '/local/.snapshot/_local/test'
Delete subvolume (no-commit): '/local/.snapshot/_local/home/smc/test'
Delete subvolume (no-commit): '/local/.snapshot/_local/home/framstag/blubb'
Delete subvolume (no-commit): '/local/.snapshot/_local/home'
Delete subvolume (no-commit): '/local/.snapshot/_local'


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<d882ddfe-b0a2-4b1f-ac28-145652ace126@render-wahnsinn.de>

Re: restic backup with btrfs /

[Ulli Horlacher]

On Sat 2025-06-28 (19:33), Ulli Horlacher wrote:

> My idea is now: I do not backup the original /, but do:
> 
> mount --bind / /backup/restic
> restic backup /backup/restic
> umount /backup/restic
> 
> Next evolution step: 
> I could recursivly mount-bind other filesystems into /backup/restic/
> For example:
> 
> mount --bind /local /backup/restic/local
> mount --bind /data /backup/restic/ldata
> 
> That I would have a "all in one backup".

Meanwhile I haved implemented it and also use of (temporary) snapshots!
Instead of /backup/restic I am using /__ (just another name).

restix-backup does:

- create recursive snapshots of all rw subvolumes
- bind-mount the toplevel snapshots on /__
- run restic backup on /__
- umount everything below /__
- delete all temporary snapshots

Example (I have typed only the first line, all other lines are output): 

root@fex:~# restix-backup -s //

# btrfs_rsnapshot -w /
Create a snapshot of '/' in '/.snapshot/_'
Create a snapshot of '/test' in '/.snapshot/_/test'
Create a snapshot of '/tmp' in '/.snapshot/_/tmp'

# mount --bind /.snapshot/_ /__

# btrfs_rsnapshot -w /local
Create a snapshot of '/local' in '/local/.snapshot/_'
Create a snapshot of '/local/home' in '/local/.snapshot/_/home'
Create a snapshot of '/local/home/framstag/blubb' in '/local/.snapshot/_/home/framstag/blubb'
Create a snapshot of '/local/home/smc/test' in '/local/.snapshot/_/home/smc/test'
Create a snapshot of '/local/test' in '/local/.snapshot/_/test'
Create a snapshot of '/local/test/sv1' in '/local/.snapshot/_/test/sv1'
Create a snapshot of '/local/test/sv1/sv1_1' in '/local/.snapshot/_/test/sv1/sv1_1'
Create a snapshot of '/local/test/sv1/sv1_1/ss' in '/local/.snapshot/_/test/sv1/sv1_1/ss'
Create a snapshot of '/local/test/sv1/sv1_2' in '/local/.snapshot/_/test/sv1/sv1_2'
Create a snapshot of '/local/test/sv2' in '/local/.snapshot/_/test/sv2'
Create a snapshot of '/local/tmp/test' in '/local/.snapshot/_/tmp/test'

# mount --bind /local/.snapshot/_ /__/local

# cd /

# findattr __ > __/.chattr

# restic backup -v -e /tmp -e /backup -e .cache -e .del -e tmp/\* -e \*.tmp -H fex --tag __ __
RESTIC_ENV=/root/.restic/s3tik.env
SG_TENANT=fex
RESTIC_REPOSITORY=s3:s3.tik.uni-stuttgart.de/fex.restic

# /opt/s3tik/restic backup -v -e /tmp -e /backup -e .cache -e .del -e 'tmp/*' -e '*.tmp' -H fex --tag __ __
open repository
repository 13e662d4 opened (version 2, compression level auto)
using parent snapshot 20b48e38
load index files
[0:02] 100.00%  36 / 36 index files loaded
start scan on [__]
start backup on [__]
scan finished in 4.728s: 254436 files, 46.446 GiB

Files:           1 new,   105 changed, 254330 unmodified
Dirs:            2 new, 18860 changed,  2765 unmodified
Data Blobs:     37 new
Tree Blobs:    168 new
Added to the repository: 35.310 MiB (7.926 MiB stored)

processed 254436 files, 46.446 GiB in 0:32
snapshot 1624348b saved

# restic unlock
SG_TENANT=fex
RESTIC_REPOSITORY=s3:s3.tik.uni-stuttgart.de/fex.restic

# /opt/s3tik/restic unlock

# umount /__//local

# btrfs_rsnapshot -d /local
Delete subvolume (no-commit): '/local/.snapshot/_/tmp/test'
Delete subvolume (no-commit): '/local/.snapshot/_/test/sv2'
Delete subvolume (no-commit): '/local/.snapshot/_/test/sv1/sv1_2'
Delete subvolume (no-commit): '/local/.snapshot/_/test/sv1/sv1_1/ss'
Delete subvolume (no-commit): '/local/.snapshot/_/test/sv1/sv1_1'
Delete subvolume (no-commit): '/local/.snapshot/_/test/sv1'
Delete subvolume (no-commit): '/local/.snapshot/_/test'
Delete subvolume (no-commit): '/local/.snapshot/_/home/smc/test'
Delete subvolume (no-commit): '/local/.snapshot/_/home/framstag/blubb'
Delete subvolume (no-commit): '/local/.snapshot/_/home'
Delete subvolume (no-commit): '/local/.snapshot/_'

# umount /__//

# btrfs_rsnapshot -d /
Delete subvolume (no-commit): '/.snapshot/_/tmp'
Delete subvolume (no-commit): '/.snapshot/_/test'
Delete subvolume (no-commit): '/.snapshot/_'

# restic forget --prune --tag __ --host fex --keep-daily 7 --keep-weekly 4
SG_TENANT=fex
RESTIC_REPOSITORY=s3:s3.tik.uni-stuttgart.de/fex.restic

# /opt/s3tik/restic forget --prune --tag __ --host fex --keep-daily 7 --keep-weekly 4
Applying Policy: keep 7 daily, 4 weekly snapshots
keep 2 snapshots:
ID        Time                 Host        Tags        Reasons                 Paths  Size
------------------------------------------------------------------------------------------------
20b48e38  2025-07-07 13:53:14  fex         __          oldest daily snapshot   /__    46.446 GiB
                                                       oldest weekly snapshot
1624348b  2025-07-07 14:05:42  fex         __          daily snapshot          /__    46.446 GiB
                                                       weekly snapshot
------------------------------------------------------------------------------------------------
2 snapshots


# restic check
SG_TENANT=fex
RESTIC_REPOSITORY=s3:s3.tik.uni-stuttgart.de/fex.restic

# /opt/s3tik/restic check
using temporary cache in /tmp/root/restic-check-cache-988330367
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
[0:12] 100.00%  35 / 35 snapshots
no errors were found


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<20250628173308.GB847325@tik.uni-stuttgart.de>