File changing in snapshot

File changing in snapshot

[Oliver O.]

I seem to have observed a file on a (writable) snapshot changing 
although there were no writes occuring on the snapshot itself. This is 
not supposed to happen, right?

Sequence of events:

1. A (writable) snapshot @home-2014-04-16 is taken on a @home subvolume 
mounted at /home.

2. The current subvolume (@home) is used (via /home) for continued write 
access, no writes are supposed to occur on the snapshot.

3. The snapshot @home-2014-04-16 is backed up (using rdiff-backup) to a 
different disk.

4. As the backup is compared to the snapshot @home-2014-04-16, one file 
differs.

Further analysis:

The file in @home-2014-04-16 changed its contents (but not mtime or 
other metadata) after being backup up from that snapshot. 5 bytes differ.

The previous snapshot @home-2014-04-10 contains the file in its original 
form, which is identical to the backup just taken from @home-2014-04-16.

The file change appears identically
- on the last snapshot @home-2014-04-16 and
- on the current subvolume used for writing (@home).

btrfs scrub reports 0 errors.

At the time the snapshot @home-2014-04-16 was taken, the file (an Excel 
file) was probably accessed through Samba, which supposedly uses kernel 
oplocks (if that makes a difference).

Versions:
- Kernel 3.11.0-19-generic (Ubuntu 13.10)
- btrfs-tools 0.19+20130705-1
- Samba 3.6.18

Any ideas?

Re: File changing in snapshot

[Chris Mason]

On 04/17/2014 11:39 AM, Oliver O. wrote:
> I seem to have observed a file on a (writable) snapshot changing
> although there were no writes occuring on the snapshot itself. This is
> not supposed to happen, right?
>
> Sequence of events:
>
> 1. A (writable) snapshot @home-2014-04-16 is taken on a @home subvolume
> mounted at /home.
>
> 2. The current subvolume (@home) is used (via /home) for continued write
> access, no writes are supposed to occur on the snapshot.
>
> 3. The snapshot @home-2014-04-16 is backed up (using rdiff-backup) to a
> different disk.
>
> 4. As the backup is compared to the snapshot @home-2014-04-16, one file
> differs.
>
> Further analysis:
>
> The file in @home-2014-04-16 changed its contents (but not mtime or
> other metadata) after being backup up from that snapshot. 5 bytes differ.
>
> The previous snapshot @home-2014-04-10 contains the file in its original
> form, which is identical to the backup just taken from @home-2014-04-16.
>
> The file change appears identically
> - on the last snapshot @home-2014-04-16 and
> - on the current subvolume used for writing (@home).
>
> btrfs scrub reports 0 errors.
>
> At the time the snapshot @home-2014-04-16 was taken, the file (an Excel
> file) was probably accessed through Samba, which supposedly uses kernel
> oplocks (if that makes a difference).
>
> Versions:
> - Kernel 3.11.0-19-generic (Ubuntu 13.10)
> - btrfs-tools 0.19+20130705-1
> - Samba 3.6.18

Was this a nodatacow file?

-chris

Re: File changing in snapshot

[Oliver O.]

Am 17.04.2014 17:56, schrieb Chris Mason:
> On 04/17/2014 11:39 AM, Oliver O. wrote:
>> I seem to have observed a file on a (writable) snapshot changing
>> although there were no writes occuring on the snapshot itself. This is
>> not supposed to happen, right?
>
> Was this a nodatacow file?
>
> -chris

No. Mount options:

/dev/sda2 on /home type btrfs (rw,noatime,nodiratime,subvol=@home)

Re: File changing in snapshot

[Oliver O.]

Am 17.04.2014 18:11, schrieb Oliver O.:
>
> Am 17.04.2014 17:56, schrieb Chris Mason:
>> On 04/17/2014 11:39 AM, Oliver O. wrote:
>>> I seem to have observed a file on a (writable) snapshot changing
>>> although there were no writes occuring on the snapshot itself. This is
>>> not supposed to happen, right?
>>
>> Was this a nodatacow file?
>>
>> -chris
>
> No. Mount options:
>
> /dev/sda2 on /home type btrfs (rw,noatime,nodiratime,subvol=@home)
>

Some additional observations:
- Subvolume used for writing: @home
- Snapshot used for backup: @home-2014-04-16
- Preceding snapshot: @home-2014-04-10

# lsattr -v @home-2014-04-10/path/to/MyFile.xls 
@home-2014-04-16/path/to/MyFile.xls @home/path/to/MyFile.xls
36066 ---------------- @home-2014-04-10/path/to/MyFile.xls
36066 ---------------- @home-2014-04-16/path/to/MyFile.xls
36066 ---------------- @home/path/to/MyFile.xls

# btrfs subvolume find-new @home-2014-04-10 99999999
transid marker was 180026

# btrfs subvolume find-new @home-2014-04-16 99999999
transid marker was 194551

# btrfs subvolume find-new @home 99999999
transid marker was 197735

# btrfs subvolume find-new @home 36066 | grep "MyFile.xls"
inode 24426 file offset 0 len 20480 disk start 9108635648 offset 0 gen 
193090 flags NONE path/to/MyFile.xls
inode 24426 file offset 20480 len 36864 disk start 25510342656 offset 
20480 gen 36067 flags NONE path/to/MyFile.xls
inode 24426 file offset 57344 len 4096 disk start 9108582400 offset 0 
gen 193090 flags NONE path/to/MyFile.xls


Conclusions:

The sequence of events seems to be:

1. The file was changed with generation 193090.
2. The snapshot for backup (@home-2014-04-16) was taken (generation 194551).
3. As the backup was reading from the snapshot, it was seeing stale data 
(MyFile still at generation 36067).
4. As the backup comparison was reading from the snapshot, is was seeing 
more recent data, causing a discrepancy with the stale data just backed up.

It seems that the snapshot needed some time to become stable for 
reading. From a file system user's point of view, I'd expect a snapshot 
to be atomic and stable at any time. Maybe some kind of caching problem 
here?

Maybe it's time to file a bug report, but first: suggestions and ideas 
appreciated.

Solved, false alarm - Re: File changing in snapshot

[Oliver O.]

Am 17.04.2014 23:29, schrieb Oliver O.:
> Conclusions:
>
> The sequence of events seems to be:
>
> 1. The file was changed with generation 193090.
> 2. The snapshot for backup (@home-2014-04-16) was taken (generation 
> 194551).
> 3. As the backup was reading from the snapshot, it was seeing stale 
> data (MyFile still at generation 36067).

This conclusion was wrong:

Rdiff-backup never really considered the modified Excel file for backup, 
as its file modification time did not indicate that the file was newer 
than the last backup. So rdiff-backup always took the file copy already 
available in the previous backup.

What breaks the incremental backup mechanism is Excel modifying a file, 
then changing its modification timestamp back to its original value (see 
comment 10 at https://bugzilla.samba.org/show_bug.cgi?id=1601#c10).

> 4. As the backup comparison was reading from the snapshot, is was 
> seeing more recent data, causing a discrepancy with the stale data 
> just backed up.

The discrepancy between the file's modified contents were discovered 
because the comparison was content-based (MD5 checksums) and compared 
every file regardless of modification dates.

>
> It seems that the snapshot needed some time to become stable for 
> reading. From a file system user's point of view, I'd expect a 
> snapshot to be atomic and stable at any time. Maybe some kind of 
> caching problem here?
>
> Maybe it's time to file a bug report, but first: suggestions and ideas 
> appreciated.
>
Result: False alarm - this is definitely not a Btrfs problem. In this 
case snapshot behavior was as expected: atomic and stable.

Sorry for any irritation.

Re: Solved, false alarm - Re: File changing in snapshot

[Chris Mason]

On 04/18/2014 11:50 AM, Oliver O. wrote:
> Am 17.04.2014 23:29, schrieb Oliver O.:

> Result: False alarm - this is definitely not a Btrfs problem. In this
> case snapshot behavior was as expected: atomic and stable.
>
> Sorry for any irritation.
>

But, mystery solved and documented for later generations ;)  Thanks for 
nailing it down and posting the result.

-chris