[resend] btrfs-send -c fails: reproduction case

[resend] btrfs-send -c fails: reproduction case

[Zachary Vance]

Apologies for the repeat send for anyone that gets this twice; I was
informed there was a problem with my mail setup and many ISPs would
block the first message. I have also simplified the reproduction case.

---

Reproduction case after running into the same problem as Paride
Legovini: http://article.gmane.org/gmane.comp.file-systems.btrfs/48706/match=send

To recap, the basic problem is that you can get into a state where
"btrfs-send -c" fails for a subvolume, with "ERROR: parent
determination failed for 9622". It appears the problem happens when
the 'parent_uuid' does not exist.

I'm pretty sure this isn't the intended behavior, and it's certainly
preventing me from copying my backups right now.

burn% sudo btrfs subvol create a
Create subvolume './a'
burn% sudo btrfs subvol snapshot a -r a-r
Create a readonly snapshot of 'a' in './a-r'
burn% sudo btrfs subvol create b1
Create subvolume './b1'
burn% sudo btrfs subvol snapshot b1 -r b2
Create a readonly snapshot of 'b1' in './b2'
burn% sudo btrfs subvol del b1
Delete subvolume (no-commit): '/data/b1'

burn% sudo btrfs send -c b2 a-r >/dev/null
At subvol a-r
ERROR: parent determination failed for 9637

/data/a-r
        Name:                   a-r
        UUID:                   819415e9-ea57-4a4d-9745-78a65f19b614
        Parent UUID:            e63b985a-3463-be4c-8ad5-900cd147ef7d
        Received UUID:          -
        Creation time:          2016-02-20 10:19:59 -0800
        Subvolume ID:           9638
        Generation:             18524
        Gen at creation:        18524
        Parent ID:              5
        Top level ID:           5
        Flags:                  readonly
        Snapshot(s):
burn% sudo btrfs subvolume show b2
/data/b2
        Name:                   b2
        UUID:                   e1d0314d-9c2a-5240-9f4c-1716b7e87741
        Parent UUID:            00920504-c78b-ee42-af9f-05f4600321bb
        Received UUID:          -
        Creation time:          2016-02-20 10:19:01 -0800
        Subvolume ID:           9637
        Generation:             18522
        Gen at creation:        18522
        Parent ID:              5
        Top level ID:           5
        Flags:                  readonly
        Snapshot(s):

[root@burn ~]# btrfs --version
btrfs-progs v4.4.1

[root@burn ~]# uname -a
Linux burn 4.5.0-1-ARCH #1 SMP PREEMPT Tue Mar 15 09:41:03 CET 2016
x86_64 GNU/Linux

---
Zachary Vance

Re: [resend] btrfs-send -c fails: reproduction case

[Chris Murphy]

On Sun, Apr 17, 2016 at 3:55 PM, Zachary Vance <vanceza@gmail.com> wrote:

> To recap, the basic problem is that you can get into a state where
> "btrfs-send -c" fails for a subvolume, with "ERROR: parent
> determination failed for 9622". It appears the problem happens when
> the 'parent_uuid' does not exist.
>
> I'm pretty sure this isn't the intended behavior, and it's certainly
> preventing me from copying my backups right now.

Duncan replied to the original post.
http://www.spinics.net/lists/linux-btrfs/msg54275.html



-- 
Chris Murphy

Re: [resend] btrfs-send -c fails: reproduction case

[Zachary Vance]

+linux-btrfs and with new policy

On 04/16/2016 08:37 PM, Duncan wrote:
> Zachary Vance posted on Sat, 16 Apr 2016 13:08:17 -0700 as excerpted:
>
>> Reproduction case after running into the same problem as Paride
>> Legovini:
>> http://article.gmane.org/gmane.comp.file-systems.btrfs/48706/match=send
>
> [As a btrfs user and list-regular with a personal use-case that doesn't
> include send/receive, I generally skip posts on that topic, leaving them
> for those with more direct experience. Except... the above thread didn't
> seem to get /any/ replies except from others with the same problem, and
> while you just posted yours a few hours ago, yours hasn't either so far.
> But while I don't follow send/receive /that/ closely and could be wrong,
> I did happen to read this one and think I know the problem, so...]

So... I may need to re-send the original post, basically, Chris Murphy
privately pointed a problem with my mail hosting which causes it to be
classified as spam by gmail. (Mailing lists can't validly format mail
as me with my restrictive SPF policy).

> OK, now you delete the parent subvolumes and the subvolume that contained
> them, leaving:
>
> /root/ (pwd and there originally)
> |
> + a/ (/root/a, ro snap of (now deleted) /root/m/a, no parent left)
> |
> + b/ (/root/b, ro snap of (now deleted) /root/m/b, no parent left)
>
>> [root@burn ~]# sudo btrfs send -c a b >/dev/null
>> At subvol b ERROR: parent determination failed for 9622
>
> AFAIK, that's entirely expected, because there /is/ no parent now to
> incrementally send against -- you deleted the parents and can no longer
> incrementally send against them.

I'm not incrementally sending against the parent, as I understand -c.
Failure here was surprising.

Also, I'd like to flag that there are two notions of "parent" -- a
parent ID and a parent_uuid, and it would be great if that error
message specified which more clearly since both are involved in the
reproduction case.

The exact same steps, keeping 'm' as a folder rather than a subvolume,
also fails, which clarifies what's happening. The message is related
to parent_uuid.

> Instead, at this point you have to do a full (non-incremental) send,
> since there's nothing to match against as the parent for an incremental
> send.
>
>
> However, what I believe you /wanted/ to do is something like this,
> assuming monthly snapshots to make it easy:
> [...]

Every time I start asking about snapshots people tell me how I could
set up my process differently which is annoying because:
1) They assume I'm doing a single, incremental backup process, which is wrong
2) I already have years/terabytes pf data I need to transfer, so I
can't change what I did in the past
3) They inevitably recommend their favorite method, which doesn't work
for me, because I have different requirements

I realize you're also trying to provide basic explanation of -p and -c
here but it's EVERY time I ask about snapshots...

Anyway, I'm doing a one-time transfer of a large number of snapshots
which are currently stuck on a .img file. Over the years they've been
re-organized repeatedly and I've snapshotted many from writable to
read-only (most of the cases of the parent being deleted, I think). My
goal is to get them into a new filesystem, but I can't do a full send
in every case, because then it would balloon up to a petabyte or so.

> Does that make more sense now, or were you actually already doing it that
> way and still getting the errors, and just didn't say so, or did I just
> confuse you even further (hopefully not!)?

I was already making sure all -c references were both present and
unmodified, I think the confusion is mostly around whether the parent
required to use -c, and whether it's an implicit reference volume in
particular. If it's required, it's impossible to preserve
de-duplication after deleting the original parent which would be
really bad.

Re: [resend] btrfs-send -c fails: reproduction case

[Henk Slager]

>>> Reproduction case after running into the same problem as Paride
>>> Legovini:
>>> http://article.gmane.org/gmane.comp.file-systems.btrfs/48706/match=send

Your case is not the same as in this thread from Paride IMO. The error
message is the same, but that doesn't mean the call tree leading to it
is the same. My first impression from your 1st and also your 2nd (in
resend thread) example was that btrfs' error message is correct and
usage of -c is wrong.

More important is that the problem that Paride describes, is solved
for kernel/tools v4.5.0/v4.5.1

[...]
> Every time I start asking about snapshots people tell me how I could
> set up my process differently which is annoying because:
> 1) They assume I'm doing a single, incremental backup process, which is wrong
> 2) I already have years/terabytes pf data I need to transfer, so I
> can't change what I did in the past
> 3) They inevitably recommend their favorite method, which doesn't work
> for me, because I have different requirements

You would have to write a clear set of requirements, as an extension
to what btrfs send|receive currently can do.
>From the examples it is simply not clear what you want.

[...]
> Anyway, I'm doing a one-time transfer of a large number of snapshots
> which are currently stuck on a .img file. Over the years they've been

So, this is yet another problem; What is the background here?

> re-organized repeatedly and I've snapshotted many from writable to
> read-only (most of the cases of the parent being deleted, I think). My
> goal is to get them into a new filesystem, but I can't do a full send
> in every case, because then it would balloon up to a petabyte or so.

This gives some hint what the situation is. How many ro snapshots is
it? How much data in total as uncompressed diskblocks?

>> Does that make more sense now, or were you actually already doing it that
>> way and still getting the errors, and just didn't say so, or did I just
>> confuse you even further (hopefully not!)?
>
> I was already making sure all -c references were both present and
> unmodified, I think the confusion is mostly around whether the parent
> required to use -c, and whether it's an implicit reference volume in
> particular. If it's required, it's impossible to preserve
> de-duplication after deleting the original parent which would be
> really bad.

Can you give examples for your real data how bad it is? And how you
expect or want it to be ?

Re: [resend] btrfs-send -c fails: reproduction case

[Duncan]

Zachary Vance posted on Sun, 17 Apr 2016 18:46:13 -0700 as excerpted:

> I was already making sure all -c references were both present and
> unmodified, I think the confusion is mostly around whether the parent
> required to use -c, and whether it's an implicit reference volume in
> particular. If it's required, it's impossible to preserve de-duplication
> after deleting the original parent which would be really bad.

AFAIK, send -c does require a parent, but if there's clones and no parent 
given, it will try to pick its own from the list of clones given.

Which explains the error complaining about being unable to find a parent, 
when given only clone sources that (based on your reproducer) had nothing 
in common with the snapshots listed as clones.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman