Re: Can the output of FIEMAP on BTRFS be used to check if a file and its reflink copy might have diverged?

[Demi Marie Obenour <demiobenour@gmail.com>]

[-- Attachment #1.1.1: Type: text/plain, Size: 3158 bytes --]

On 9/21/25 20:50, Qu Wenruo wrote:
> 
> 
> 在 2025/9/22 09:37, Demi Marie Obenour 写道:
>> Wyng Backup (https://codeberg.org/tasket/wyng-backup) relies on FIEMAP
>> to determine which parts of a file have not changed since it was last
>> backed up.  Specifically, the output of filefrag -v is passed to sort and
>> then to uniq, and differences between the outputs for the file and
>> the previous version (a reflink copy) determine what gets backed up.
>>
>> Is this safe under BTRFS,
> 
> No. There are several factors affecting this, some are minor some are not:
> 
> - Inlined extents
>    The returned bytenr is unreliable in that case.
>    Although the fiemap flags should indicate that, with 'inline' flag
>    set.
> 
> - Balance
>    Btrfs can balance the data extents, which will result the change of
>    the fiemap.
> 
>    E.g.
>    ## Before balance
>    # md5sum  /mnt/btrfs/foobar
>    27c9068d1b51da575a53ad34c57ca5cc  /mnt/btrfs/foobar
>    # filefrag -v /mnt/btrfs/foobar
>    Filesystem type is: 9123683e
>    File size of /mnt/btrfs/foobar is 65536 (8 blocks of 8192 bytes)
>     ext:     logical_offset:        physical_offset: length:   expected: 
> flags:
>       0:        0..       7:       1664..      1671:      8: 
> last,eof
>    /mnt/btrfs/foobar: 1 extent found
> 
>    ## Do data balance
>    # btrfs balance start -d /mnt/btrfs/
>    Done, had to relocate 1 out of 3 chunks
> 
>    ## After data balannce
>    # filefrag -v /mnt/btrfs/foobar
>    Filesystem type is: 9123683e
>    File size of /mnt/btrfs/foobar is 65536 (8 blocks of 8192 bytes)
>      ext:     logical_offset:        physical_offset: length: 
> expected: flags:
>       0:        0..       7:      36480..     36487:      8: 
> last,eof
>    /mnt/btrfs/foobar: 1 extent found
> 
> 
> - NODATACOW cases.
>    In that case new data is written into the same location, without any
>    extra new data extents. This completely breaks the assumption.
> 
> - Dirty data that is not yet written into the disk
>    In that case fiemap won't show those data but only the ones that are
>    on the disk.
> 
>> or can it result in data loss due to data
>> not being backed up that should be?  In other words, can it result
>> in data being considered unchanged when it really is?
> 
> Dirty data and NODATACOW will result data being considered unchanged 
> using fiemap only.
> 
> And balance will make the unchanged data to be considered changed.
> 
> So overall, fiemap based solution on btrfs is unreliable.

Can one implement this reliably with TREE_SEARCH_V2 or by parsing
the output of `btrfs send`?  Using `btrfs receive` to apply deltas
might work, but the backups must be encrypted with a key the
destination doesn't have, and that means the backups must be
opaque to the remote.  Therefore, I think periodic full backups
with incremental backups on top of them (each containing the hash
of the last) is the best that could be done.  This means that old
backups cannot be garbage-collected without taking a full backup
first.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]