Problems with incremental btrfs send (out of order instructions?)

Problems with incremental btrfs send (out of order instructions?)

[Mathijs Kwik]

Hi all,

For some time, I've successfully deployed btrfs send/receive as a
viable backup solution.
It's fast and flexible and nicely scriptable =)
However, every once in a while, trouble strikes on the receiving end
with a message like:

ERROR: rename nixpkgs/pkgs/applications/version-management/subversion-1.2.x/default.nix
-> o2979-81788-0 failed. No such file or directory

Judging from looking at the dump-stream, I think sometimes a directory
gets moved (subversion-1.2.x in the example) to some temporary name
(or it gets deleted), but a little later a file within that directory
is being changed.
This seems to happen most often with filesystem-heavy applications
like a git tree or a webbrowser's cache dir.

I've tried many things to recover from such a situation, like removing
the offending directory and retrying from a new snapshot, but nothing
seems to work except for a full (non-incremental) send which is
expensive.

After some narrowing down, I managed to capture a problematic
sendstream. To reproduce:
- I created a separate subvolume (so there are no privacy-related
issues with sharing the dump)
- put a moderately large git tree on it (I used
https://github.com/NixOS/nixpkgs)
- perform an initial btrfs send of the subvolume
- jump around git history (check out very early revision, check out HEAD again)
- perform an incremental (with parent) btrfs send to a file
- try to receive this file on another machine
- about 80% chance this triggers it, if not, jump/branch some more with git

The problematic dump is here:
https://bluescreen303.nl/btrfs-send-error.dump.xz
The error provided above goes with this dump.

The dump was on kernel 3.9.x but it still happens on 3.10.4
btrfs-progs 650e656a8b9c1fbe4ec5cd8c48ae285b8abd3b69
I noticed there have been some commits since that revision but nothing
mentions send/receive.

Can someone have a look at my dumpfile and confirm this is indeed a
problem with btrfs send as opposed to something I am doing wrong?

Thanks!
Mathijs

Re: Problems with incremental btrfs send (out of order instructions?)

[Stefan Behrens]

On Wed, 31 Jul 2013 23:33:59 +0200, Mathijs Kwik wrote:
> Hi all,
> 
> For some time, I've successfully deployed btrfs send/receive as a
> viable backup solution.
> It's fast and flexible and nicely scriptable =)
> However, every once in a while, trouble strikes on the receiving end
> with a message like:
> 
> ERROR: rename nixpkgs/pkgs/applications/version-management/subversion-1.2.x/default.nix
> -> o2979-81788-0 failed. No such file or directory

I've also seen such an errors. It's an error in the send code.

[...]
> Can someone have a look at my dumpfile and confirm this is indeed a
> problem with btrfs send as opposed to something I am doing wrong?

According to your dumpfile, the file nixpkgs/pkgs/applications/version-management/subversion-1.2.x/default.nix needs to be there in the parent subvolume. Apparently it isn't. Since btrfs send enforces that sent snapshots are read-only, and btrfs receive sets received subvolumes to read-only immediately after the reception, it can't be your fault. It has to be the fault of the btrfs send logic.

If you enter a bug on bugzilla.kernel.org (refer to
<https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_do_I_report_bugs_and_issues.3F>),
you increase the probability that somebody fixes the issue some day :)

Re: Problems with incremental btrfs send (out of order instructions?)

[Xavier Bassery]

On 2013-08-01 10:49, Stefan Behrens wrote:
> On Wed, 31 Jul 2013 23:33:59 +0200, Mathijs Kwik wrote:
>> Hi all,
>>
>> For some time, I've successfully deployed btrfs send/receive as a
>> viable backup solution.
>> It's fast and flexible and nicely scriptable =)
>> However, every once in a while, trouble strikes on the receiving end
>> with a message like:
>>
>> ERROR: rename 
>> nixpkgs/pkgs/applications/version-management/subversion-1.2.x/default.nix
>> -> o2979-81788-0 failed. No such file or directory
>
> I've also seen such an errors. It's an error in the send code.
>
[...]
> If you enter a bug on bugzilla.kernel.org (refer to
> 
> <https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_do_I_report_bugs_and_issues.3F>),
> you increase the probability that somebody fixes the issue some day 
> :)


This bug has been reported in BZ #60673 [1] and was fixed in Josef's 
btrfs-next commit
titled "Btrfs: check our parent dir when doing a compare send" [2].
This fix hasn't been merged in the stable kernel yet.

Thanks to all.


[1] https://bugzilla.kernel.org/show_bug.cgi?id=60673
[2] 
http://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/fs/btrfs/send.c?id=441f6d6f7115a927f3e13f7acc53184a6081dbe0

Re: Problems with incremental btrfs send (out of order instructions?)

[Mathijs Kwik]

Xavier Bassery <xavier@bartica.org> writes:

> On 2013-08-01 10:49, Stefan Behrens wrote:
>> On Wed, 31 Jul 2013 23:33:59 +0200, Mathijs Kwik wrote:
>>> Hi all,
>>>
>>> For some time, I've successfully deployed btrfs send/receive as a
>>> viable backup solution.
>>> It's fast and flexible and nicely scriptable =)
>>> However, every once in a while, trouble strikes on the receiving end
>>> with a message like:
>>>
>>> ERROR: rename
>>> nixpkgs/pkgs/applications/version-management/subversion-1.2.x/default.nix
>>> -> o2979-81788-0 failed. No such file or directory
>>
>> I've also seen such an errors. It's an error in the send code.
>>
> [...]
>> If you enter a bug on bugzilla.kernel.org (refer to
>>
>> <https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_do_I_report_bugs_and_issues.3F>),
>> you increase the probability that somebody fixes the issue some day
>> :)
>
>
> This bug has been reported in BZ #60673 [1] and was fixed in Josef's
> btrfs-next commit
> titled "Btrfs: check our parent dir when doing a compare send" [2].
> This fix hasn't been merged in the stable kernel yet.
>
> Thanks to all.

I have been using the patch for 10 days now, and it's a huge
improvement, but we're not out of the woods yet it seems :(

https://bugzilla.kernel.org/show_bug.cgi?id=60673

>
>
> [1] https://bugzilla.kernel.org/show_bug.cgi?id=60673
> [2]
> http://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/fs/btrfs/send.c?id=441f6d6f7115a927f3e13f7acc53184a6081dbe0