btrfs-find-root duration?

btrfs-find-root duration?

[Markus Binsteiner]

It seems I've accidentally deleted all files in my home directory,
which sits in its own btrfs partition (lvm on luks). Now I'm trying to
find the roots to be able to use btrfs restore later on.

btrfs-find-root seems to be taking ages though. I've run it like so:

btrfs-find-root /dev/mapper/think--big-home  -o 5 > roots.txt

After 16 hours, there is still no output, but it's still running
utilizing 100% of one core. Is there any way to gauge how much longer
it'll take? Should there have been output already while it's running?

When I run it without redirecting stdout, I get:

$ btrfs-find-root /dev/mapper/think--big-home  -o 5

Superblock doesn't contain generation info for root 5
Superblock doesn't contain the level info for root 5

When I omit the '-o 5', it says:

$ btrfs-find-root /dev/mapper/think--big-home

Superblock thinks the generation is 593818
Superblock thinks the level is 0

Is the latter the way to run it? Did that initially, but that  didn't
return any results in a reasonable timeframe either.

The filesystem was created with Debian Jessie, but I'm using Ubuntu (
btrfs-progs v4.7.3 ) to try to restore the files at the moment.

Thanks!

Re: btrfs-find-root duration?

[Xin Zhou]

Hi Markus, 

Some file system automatically generates snapshot, and create a hidden folder for recovery,
if the user accidently deletes some files.

It seems btrfs also has a autosnap feature,
so if this option has been enabled before deletion,
or the volume has been mannually generated snapshots, then probably it might be able to perform fast recover.

Regards,
Xin 

Sent: Saturday, December 10, 2016 at 4:12 PM
From: "Markus Binsteiner" <makkus@gmail.com>
To: linux-btrfs@vger.kernel.org
Subject: btrfs-find-root duration?
It seems I've accidentally deleted all files in my home directory,
which sits in its own btrfs partition (lvm on luks). Now I'm trying to
find the roots to be able to use btrfs restore later on.

btrfs-find-root seems to be taking ages though. I've run it like so:

btrfs-find-root /dev/mapper/think--big-home -o 5 > roots.txt

After 16 hours, there is still no output, but it's still running
utilizing 100% of one core. Is there any way to gauge how much longer
it'll take? Should there have been output already while it's running?

When I run it without redirecting stdout, I get:

$ btrfs-find-root /dev/mapper/think--big-home -o 5

Superblock doesn't contain generation info for root 5
Superblock doesn't contain the level info for root 5

When I omit the '-o 5', it says:

$ btrfs-find-root /dev/mapper/think--big-home

Superblock thinks the generation is 593818
Superblock thinks the level is 0

Is the latter the way to run it? Did that initially, but that didn't
return any results in a reasonable timeframe either.

The filesystem was created with Debian Jessie, but I'm using Ubuntu (
btrfs-progs v4.7.3 ) to try to restore the files at the moment.

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: btrfs-find-root duration?

[Markus Binsteiner]

Hi Xin,

thanks. I did not enable autosnap, and I'm pretty sure Debian didn't
do it for me either, as I would have seen the subvolumes created by it
at some stage. Good to know about this feature though, will definitely
use it next time around.

Cheers,
Markus

On Sun, Dec 11, 2016 at 2:17 PM, Xin Zhou <xin.zhou@gmx.com> wrote:
> Hi Markus,
>
> Some file system automatically generates snapshot, and create a hidden
> folder for recovery,
> if the user accidently deletes some files.
>
> It seems btrfs also has a autosnap feature,
> so if this option has been enabled before deletion,
> or the volume has been mannually generated snapshots, then probably it might
> be able to perform fast recover.
>
> Regards,
> Xin
>
> Sent: Saturday, December 10, 2016 at 4:12 PM
> From: "Markus Binsteiner" <makkus@gmail.com>
> To: linux-btrfs@vger.kernel.org
> Subject: btrfs-find-root duration?
> It seems I've accidentally deleted all files in my home directory,
> which sits in its own btrfs partition (lvm on luks). Now I'm trying to
> find the roots to be able to use btrfs restore later on.
>
> btrfs-find-root seems to be taking ages though. I've run it like so:
>
> btrfs-find-root /dev/mapper/think--big-home -o 5 > roots.txt
>
> After 16 hours, there is still no output, but it's still running
> utilizing 100% of one core. Is there any way to gauge how much longer
> it'll take? Should there have been output already while it's running?
>
> When I run it without redirecting stdout, I get:
>
> $ btrfs-find-root /dev/mapper/think--big-home -o 5
>
> Superblock doesn't contain generation info for root 5
> Superblock doesn't contain the level info for root 5
>
> When I omit the '-o 5', it says:
>
> $ btrfs-find-root /dev/mapper/think--big-home
>
> Superblock thinks the generation is 593818
> Superblock thinks the level is 0
>
> Is the latter the way to run it? Did that initially, but that didn't
> return any results in a reasonable timeframe either.
>
> The filesystem was created with Debian Jessie, but I'm using Ubuntu (
> btrfs-progs v4.7.3 ) to try to restore the files at the moment.
>
> Thanks!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: btrfs-find-root duration?

[Austin S. Hemmelgarn]

On 2016-12-10 20:42, Markus Binsteiner wrote:
> Hi Xin,
>
> thanks. I did not enable autosnap, and I'm pretty sure Debian didn't
> do it for me either, as I would have seen the subvolumes created by it
> at some stage. Good to know about this feature though, will definitely
> use it next time around.
BTRFS itself has no such feature.  What Xin is probably thinking of is a 
piece of third-party software called 'snapper' that gets installed by 
default on at least SUSE and Ubuntu when using BTRFS, but not on Debian 
because they make no assumptions about individual use-case (and there 
are _lots_ of people (myself included) who absolutely do not want 
automatic snapshotting eating up space on our filesystems behind our 
back).  If you do choose to use snapper, make sure to update the 
settings to fit your needs, as the defaults (at least, the defaults 
upstream and on SUSE) are absolutely brain-dead and will eat your 
filesystem (or at least completely kill it's performance) in a couple of 
months on average.
>
> Cheers,
> Markus
>
> On Sun, Dec 11, 2016 at 2:17 PM, Xin Zhou <xin.zhou@gmx.com> wrote:
>> Hi Markus,
>>
>> Some file system automatically generates snapshot, and create a hidden
>> folder for recovery,
>> if the user accidently deletes some files.
>>
>> It seems btrfs also has a autosnap feature,
>> so if this option has been enabled before deletion,
>> or the volume has been mannually generated snapshots, then probably it might
>> be able to perform fast recover.
>>
>> Regards,
>> Xin
>>
>> Sent: Saturday, December 10, 2016 at 4:12 PM
>> From: "Markus Binsteiner" <makkus@gmail.com>
>> To: linux-btrfs@vger.kernel.org
>> Subject: btrfs-find-root duration?
>> It seems I've accidentally deleted all files in my home directory,
>> which sits in its own btrfs partition (lvm on luks). Now I'm trying to
>> find the roots to be able to use btrfs restore later on.
>>
>> btrfs-find-root seems to be taking ages though. I've run it like so:
>>
>> btrfs-find-root /dev/mapper/think--big-home -o 5 > roots.txt
>>
>> After 16 hours, there is still no output, but it's still running
>> utilizing 100% of one core. Is there any way to gauge how much longer
>> it'll take? Should there have been output already while it's running?
>>
>> When I run it without redirecting stdout, I get:
>>
>> $ btrfs-find-root /dev/mapper/think--big-home -o 5
>>
>> Superblock doesn't contain generation info for root 5
>> Superblock doesn't contain the level info for root 5
>>
>> When I omit the '-o 5', it says:
>>
>> $ btrfs-find-root /dev/mapper/think--big-home
>>
>> Superblock thinks the generation is 593818
>> Superblock thinks the level is 0
>>
>> Is the latter the way to run it? Did that initially, but that didn't
>> return any results in a reasonable timeframe either.
>>
>> The filesystem was created with Debian Jessie, but I'm using Ubuntu (
>> btrfs-progs v4.7.3 ) to try to restore the files at the moment.
>>
>> Thanks!
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: btrfs-find-root duration?

[Chris Murphy]

On Sat, Dec 10, 2016 at 5:12 PM, Markus Binsteiner <makkus@gmail.com> wrote:
> It seems I've accidentally deleted all files in my home directory,
> which sits in its own btrfs partition (lvm on luks). Now I'm trying to
> find the roots to be able to use btrfs restore later on.
>
> btrfs-find-root seems to be taking ages though. I've run it like so:
>
> btrfs-find-root /dev/mapper/think--big-home  -o 5 > roots.txt

Uhh, just do btrfs-find-root by itself to get everything it can find.
And then work backwards from the most recent generation using btrfs
restore -t using each root bytenr from btrfs-find-root. The more
recent the generation, the better your luck that it hasn't been
overwritten yet; but too recent and your data may not exist in that
root. It really depends how fast you umounted the volume after
deleting everything.



-- 
Chris Murphy

Re: btrfs-find-root duration?

[Markus Binsteiner]

I recon it took me about 5 minutes to realise what I'd done, then I
unmounted the volume. I don't think I wrote anything inbetween, but
there were a few applications open at that time, so there might have
been some i/o.

When you say 'by itself', you mean without the '-o 5'?

I've tried that initially, but it run for a few hours with no output
beside the initial 'Superblock...'.  I realized I forgot to redirect
the stdout which I thought that would be a good idea so I restarted it
with '-o 5' (found some advice said that's the thing to do if it was
the root subvolume that was deleted).

Anyway, restarted it without '-o 5', let's see whether it makes any
difference. Is there any indication on how long it should take? Just
roughly, hours, days? I've got about 150GB of data on that partition I
think. Also, is there supposed to be incremental output, or will it be
one big wall of text once it's finished?

As I said, when I tried before there was no output at all for hours,
which seemed a bit strange.

Thanks for your help!

On Sun, Dec 11, 2016 at 6:47 PM, Chris Murphy <lists@colorremedies.com> wrote:
> On Sat, Dec 10, 2016 at 5:12 PM, Markus Binsteiner <makkus@gmail.com> wrote:
>> It seems I've accidentally deleted all files in my home directory,
>> which sits in its own btrfs partition (lvm on luks). Now I'm trying to
>> find the roots to be able to use btrfs restore later on.
>>
>> btrfs-find-root seems to be taking ages though. I've run it like so:
>>
>> btrfs-find-root /dev/mapper/think--big-home  -o 5 > roots.txt
>
> Uhh, just do btrfs-find-root by itself to get everything it can find.
> And then work backwards from the most recent generation using btrfs
> restore -t using each root bytenr from btrfs-find-root. The more
> recent the generation, the better your luck that it hasn't been
> overwritten yet; but too recent and your data may not exist in that
> root. It really depends how fast you umounted the volume after
> deleting everything.
>
>
>
> --
> Chris Murphy

Re: btrfs-find-root duration?

[Chris Murphy]

Yes. Command and device only.

>
> I've tried that initially, but it run for a few hours with no output
> beside the initial 'Superblock...'.

OK when I do it on a file system with just 14GiB of metadata it's
maybe 15 seconds. So a few minutes sounds sorta suspicious to me but,
*shrug* I don't have a file system the same size to try it on, maybe
it's a memory intensive task and once the system gets low on RAM while
traversing the file system it slows done a ton.



-- 
Chris Murphy

Re: btrfs-find-root duration?

[Markus Binsteiner]

> OK when I do it on a file system with just 14GiB of metadata it's
> maybe 15 seconds. So a few minutes sounds sorta suspicious to me but,
> *shrug* I don't have a file system the same size to try it on, maybe
> it's a memory intensive task and once the system gets low on RAM while
> traversing the file system it slows done a ton.

Ok, thanks, looks like there is some other issue then as well. The
process doesn't take up any memory at all, just 100% of one core.

Maybe I'll try to use it with an older version of btrfs-progs, from
Debian Jessie. Don't think it'll make any difference, but I don't know
what else to try. At this point I'm more curious than anything else.
I've got backups for most of my stuff, just a few rogue scripts I'd
have to re-write. Still, would be nice to get those back.

Re: btrfs-find-root duration?

[Chris Murphy]

On Sun, Dec 11, 2016 at 4:30 PM, Markus Binsteiner <makkus@gmail.com> wrote:
>> OK when I do it on a file system with just 14GiB of metadata it's
>> maybe 15 seconds. So a few minutes sounds sorta suspicious to me but,
>> *shrug* I don't have a file system the same size to try it on, maybe
>> it's a memory intensive task and once the system gets low on RAM while
>> traversing the file system it slows done a ton.
>
> Ok, thanks, looks like there is some other issue then as well. The
> process doesn't take up any memory at all, just 100% of one core.
>
> Maybe I'll try to use it with an older version of btrfs-progs, from
> Debian Jessie. Don't think it'll make any difference, but I don't know
> what else to try. At this point I'm more curious than anything else.
> I've got backups for most of my stuff, just a few rogue scripts I'd
> have to re-write. Still, would be nice to get those back.

You might try 'btrfs check' without repairing, using a recent version
of btrfs-progs and see if it finds anything unusual.

Although, are there many snapshots? That would cause the rentention of roots.


-- 
Chris Murphy

Re: btrfs-find-root duration?

[Markus Binsteiner]

> You might try 'btrfs check' without repairing, using a recent version
> of btrfs-progs and see if it finds anything unusual.

Not quite sure what that output means, but btrfs check returns instantly:

$ sudo btrfs check /dev/mapper/think--big-home
Checking filesystem on /dev/mapper/think--big-home
UUID: 7f1ce0ed-5986-43ae-b0dd-727eee19fd08
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 655360 bytes used err is 0
total csum bytes: 0
total tree bytes: 131072
total fs tree bytes: 32768
total extent tree bytes: 16384
btree space waste bytes: 123528
file data blocks allocated: 524288
 referenced 524288

When I do the same thing on the root-OS partition that is on the same
disk, it takes a bit longer to complete, and I'm getting:

$ sudo btrfs check /dev/mapper/think--big-jessie
Checking filesystem on /dev/mapper/think--big-jessie
UUID: d82e2746-4164-41fa-b528-5a5838d818c6
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 29295181824 bytes used err is 0
total csum bytes: 27623652
total tree bytes: 973848576
total fs tree bytes: 874446848
total extent tree bytes: 62554112
btree space waste bytes: 154131992
file data blocks allocated: 29202391040
 referenced 30898016256

So I reckon the tree(s) in my home partition is just empty? Why would
btrfs-find-tree take so long to complete though? Also, I've tried to
run btrfs-find-tree on the root-OS partition, but that also didn't
complete within the 10 minutes I tried (so far).

> Although, are there many snapshots? That would cause the rentention of roots.

I can't remember exactly, but it's very possible I didn't use any
snapshots on this machine at all.

Re: btrfs-find-root duration?

[Chris Murphy]

[-- Attachment #1: Type: text/plain, Size: 2513 bytes --]

On Sun, Dec 11, 2016 at 5:12 PM, Markus Binsteiner <makkus@gmail.com> wrote:
>> You might try 'btrfs check' without repairing, using a recent version
>> of btrfs-progs and see if it finds anything unusual.
>
> Not quite sure what that output means, but btrfs check returns instantly:
>
> $ sudo btrfs check /dev/mapper/think--big-home
> Checking filesystem on /dev/mapper/think--big-home
> UUID: 7f1ce0ed-5986-43ae-b0dd-727eee19fd08
> checking extents
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> found 655360 bytes used err is 0
> total csum bytes: 0
> total tree bytes: 131072
> total fs tree bytes: 32768
> total extent tree bytes: 16384
> btree space waste bytes: 123528
> file data blocks allocated: 524288
>  referenced 524288

This is what I'd expect if the volume has only had a mkfs done and
then mounted and umounted. No files. What do you get for

btrfs ins dump-s -fa /dev/mapper/think--big-home

>
> When I do the same thing on the root-OS partition that is on the same
> disk, it takes a bit longer to complete, and I'm getting:
>
> $ sudo btrfs check /dev/mapper/think--big-jessie
> Checking filesystem on /dev/mapper/think--big-jessie
> UUID: d82e2746-4164-41fa-b528-5a5838d818c6
> checking extents
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> found 29295181824 bytes used err is 0
> total csum bytes: 27623652
> total tree bytes: 973848576
> total fs tree bytes: 874446848
> total extent tree bytes: 62554112
> btree space waste bytes: 154131992
> file data blocks allocated: 29202391040
>  referenced 30898016256
>
> So I reckon the tree(s) in my home partition is just empty? Why would
> btrfs-find-tree take so long to complete though?

Looks like a bug. I'm not sure when this regression began, but I see
it with btrfs-progs-4.8.5-1.fc26.x86_64 on a new file system, with no
files and also one volume that has a single file. But it's not hanging
on older file systems.

# btrfs-find-root /dev/mapper/vg-test2
Superblock thinks the generation is 8
Superblock thinks the level is 0

Indefinite hang.



>Also, I've tried to
> run btrfs-find-tree on the root-OS partition, but that also didn't
> complete within the 10 minutes I tried (so far).

Yeah same here, but unlike your case it completes fast for older file
systems with a decent amount of data on it. I'm not sure what the
pattern is here that results in the hang. Unfortunately strace is not
revealing I think - attached anyway.



-- 
Chris Murphy

[-- Attachment #2: strace_btrfsfindroot.txt --]
[-- Type: text/plain, Size: 6972 bytes --]

root@f25h ~]# strace btrfs-find-root /dev/mapper/vg-test2
execve("/sbin/btrfs-find-root", ["btrfs-find-root", "/dev/mapper/vg-test2"], [/* 31 vars */]) = 0
brk(NULL)                               = 0x5571d01f7000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f389a106000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=99038, ...}) = 0
mmap(NULL, 99038, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f389a0ed000
close(3)                                = 0
open("/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\24\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19528, ...}) = 0
mmap(NULL, 2113552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3899cde000
mprotect(0x7f3899ce2000, 2093056, PROT_NONE) = 0
mmap(0x7f3899ee1000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f3899ee1000
mmap(0x7f3899ee2000, 16, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3899ee2000
close(3)                                = 0
open("/lib64/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\206\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=275520, ...}) = 0
mmap(NULL, 2368448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3899a9b000
mprotect(0x7f3899ad8000, 2097152, PROT_NONE) = 0
mmap(0x7f3899cd8000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3d000) = 0x7f3899cd8000
mmap(0x7f3899cdd000, 960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3899cdd000
close(3)                                = 0
open("/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0` \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=89520, ...}) = 0
mmap(NULL, 2183272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3899885000
mprotect(0x7f389989a000, 2093056, PROT_NONE) = 0
mmap(0x7f3899a99000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7f3899a99000
close(3)                                = 0
open("/lib64/liblzo2.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@%\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=142312, ...}) = 0
mmap(NULL, 2236424, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3899662000
mprotect(0x7f3899684000, 2093056, PROT_NONE) = 0
mmap(0x7f3899883000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x21000) = 0x7f3899883000
mmap(0x7f3899884000, 8, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3899884000
close(3)                                = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240`\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=151112, ...}) = 0
mmap(NULL, 2217000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3899444000
mprotect(0x7f389945c000, 2097152, PROT_NONE) = 0
mmap(0x7f389965c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f389965c000
mmap(0x7f389965e000, 13352, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f389965e000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \5\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2115832, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f389a0eb000
mmap(NULL, 3955040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f389907e000
mprotect(0x7f389923b000, 2093056, PROT_NONE) = 0
mmap(0x7f389943a000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bc000) = 0x7f389943a000
mmap(0x7f3899440000, 14688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3899440000
close(3)                                = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f389a0e8000
arch_prctl(ARCH_SET_FS, 0x7f389a0e88c0) = 0
mprotect(0x7f389943a000, 16384, PROT_READ) = 0
mprotect(0x7f389965c000, 4096, PROT_READ) = 0
mprotect(0x7f3899883000, 4096, PROT_READ) = 0
mprotect(0x7f3899a99000, 4096, PROT_READ) = 0
mprotect(0x7f3899ee1000, 4096, PROT_READ) = 0
mprotect(0x7f3899cd8000, 16384, PROT_READ) = 0
mprotect(0x5571cfab4000, 8192, PROT_READ) = 0
mprotect(0x7f389a108000, 4096, PROT_READ) = 0
munmap(0x7f389a0ed000, 99038)           = 0
set_tid_address(0x7f389a0e8b90)         = 3952
set_robust_list(0x7f389a0e8ba0, 24)     = 0
rt_sigaction(SIGRTMIN, {0x7f3899449b40, [], SA_RESTORER|SA_SIGINFO, 0x7f38994555c0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7f3899449bd0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f38994555c0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
stat("/dev/mapper/vg-test2", {st_mode=S_IFBLK|0660, st_rdev=makedev(253, 7), ...}) = 0
open("/dev/mapper/vg-test2", O_RDONLY)  = 3
fadvise64(3, 0, 0, POSIX_FADV_DONTNEED) = 0
brk(NULL)                               = 0x5571d01f7000
brk(0x5571d0218000)                     = 0x5571d0218000
brk(NULL)                               = 0x5571d0218000
lseek(3, 0, SEEK_END)                   = 268435456000
lseek(3, 0, SEEK_SET)                   = 0
pread64(3, "\231d\4\244\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 65536) = 4096
open("/dev/mapper/vg-test2", O_RDONLY)  = 4
fadvise64(4, 0, 0, POSIX_FADV_DONTNEED) = 0
pread64(3, "\231d\4\244\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 65536) = 4096
pread64(4, "s+\260t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 147456) = 16384
pread64(4, "\335X/*\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 21151744) = 16384
pread64(4, "'\353m\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 21168128) = 16384
pread64(4, ">%\2348\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 21135360) = 16384
pread64(4, "\240{\227\366\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 21184512) = 16384
pread64(4, "O\224,/\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 16384, 21004288) = 16384
close(3)                                = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
write(1, "Superblock thinks the generation"..., 39Superblock thinks the generation is 10
) = 39
write(1, "Superblock thinks the level is 0"..., 33Superblock thinks the level is 0
) = 33
^C--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
strace: Process 3952 detached
[root@f25h ~]# 

Re: btrfs-find-root duration?

[Markus Binsteiner]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

> This is what I'd expect if the volume has only had a mkfs done and
> then mounted and umounted. No files. What do you get for
>
> btrfs ins dump-s -fa /dev/mapper/think--big-home

(attached)

Also tried btrfs check -b /dev/mapper/think--big-home, but that errored:

# btrfs  check -b /dev/mapper/think--big-home
volumes.c:1588: btrfs_chunk_readonly: Assertion `!ce` failed.
btrfs(+0x50940)[0x56351dc8f940]
btrfs(+0x50967)[0x56351dc8f967]
btrfs(btrfs_chunk_readonly+0x5a)[0x56351dc91c93]
btrfs(btrfs_read_block_groups+0x1dc)[0x56351dc8749b]
btrfs(btrfs_setup_all_roots+0x33b)[0x56351dc82adc]
btrfs(+0x43ed8)[0x56351dc82ed8]
btrfs(open_ctree_fs_info+0xd5)[0x56351dc82ffd]
btrfs(cmd_check+0x457)[0x56351dc6c332]
btrfs(main+0x139)[0x56351dc4efab]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7efee8d6d3f1]
btrfs(_start+0x2a)[0x56351dc4efea]

So, you reckon going back to an older version of btrfs-progs might be
worth a shot then?

[-- Attachment #2: btrfs_ins_dump.txt --]
[-- Type: text/plain, Size: 6725 bytes --]

superblock: bytenr=65536, device=/dev/mapper/think--big-home
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0xa64b343b [match]
bytenr			65536
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			7f1ce0ed-5986-43ae-b0dd-727eee19fd08
label			
generation		593818
root			138821632
sys_array_size		226
chunk_root_generation	593818
root_level		0
chunk_root		21020672
chunk_root_level	0
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		254007050240
bytes_used		655360
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		1
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x69
			( MIXED_BACKREF |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF )
cache_generation	593818
uuid_tree_generation	593818
dev_item.uuid		59b96438-be86-4397-a842-1e1c6efd7479
dev_item.fsid		7f1ce0ed-5986-43ae-b0dd-727eee19fd08 [match]
dev_item.type		0
dev_item.total_bytes	254007050240
dev_item.bytes_used	182066348032
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0)
		chunk length 4194304 owner 2 stripe_len 65536
		type SYSTEM num_stripes 1
			stripe 0 devid 1 offset 0
			dev uuid: 59b96438-be86-4397-a842-1e1c6efd7479
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
		chunk length 8388608 owner 2 stripe_len 65536
		type SYSTEM|DUP num_stripes 2
			stripe 0 devid 1 offset 20971520
			dev uuid: 59b96438-be86-4397-a842-1e1c6efd7479
			stripe 1 devid 1 offset 29360128
			dev uuid: 59b96438-be86-4397-a842-1e1c6efd7479
backup_roots[4]:
	backup 0:
		backup_tree_root:	29360128	gen: 593817	level: 1
		backup_chunk_root:	20971520	gen: 591139	level: 1
		backup_extent_root:	29376512	gen: 593817	level: 1
		backup_fs_root:		132562944	gen: 593815	level: 0
		backup_dev_root:	29409280	gen: 593817	level: 0
		backup_csum_root:	29491200	gen: 593817	level: 0
		backup_total_bytes:	254007050240
		backup_bytes_used:	43532288
		backup_num_devices:	1

	backup 1:
		backup_tree_root:	138821632	gen: 593818	level: 0
		backup_chunk_root:	21020672	gen: 593818	level: 0
		backup_extent_root:	138805248	gen: 593818	level: 0
		backup_fs_root:		132562944	gen: 593815	level: 0
		backup_dev_root:	132841472	gen: 593818	level: 0
		backup_csum_root:	138870784	gen: 593818	level: 0
		backup_total_bytes:	254007050240
		backup_bytes_used:	655360
		backup_num_devices:	1

	backup 2:
		backup_tree_root:	138788864	gen: 593815	level: 1
		backup_chunk_root:	20971520	gen: 591139	level: 1
		backup_extent_root:	132841472	gen: 593815	level: 1
		backup_fs_root:		132562944	gen: 593815	level: 0
		backup_dev_root:	1018118144	gen: 592983	level: 0
		backup_csum_root:	134725632	gen: 593815	level: 0
		backup_total_bytes:	254007050240
		backup_bytes_used:	43532288
		backup_num_devices:	1

	backup 3:
		backup_tree_root:	138870784	gen: 593816	level: 1
		backup_chunk_root:	20971520	gen: 591139	level: 1
		backup_extent_root:	138887168	gen: 593816	level: 1
		backup_fs_root:		132562944	gen: 593815	level: 0
		backup_dev_root:	1018118144	gen: 592983	level: 0
		backup_csum_root:	138952704	gen: 593816	level: 0
		backup_total_bytes:	254007050240
		backup_bytes_used:	43532288
		backup_num_devices:	1


superblock: bytenr=67108864, device=/dev/mapper/think--big-home
---------------------------------------------------------
csum_type		0 (crc32c)
csum_size		4
csum			0x062a1cf5 [match]
bytenr			67108864
flags			0x1
			( WRITTEN )
magic			_BHRfS_M [match]
fsid			7f1ce0ed-5986-43ae-b0dd-727eee19fd08
label			
generation		593818
root			138821632
sys_array_size		226
chunk_root_generation	593818
root_level		0
chunk_root		21020672
chunk_root_level	0
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		254007050240
bytes_used		655360
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		1
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x69
			( MIXED_BACKREF |
			  COMPRESS_LZO |
			  BIG_METADATA |
			  EXTENDED_IREF )
cache_generation	593818
uuid_tree_generation	593818
dev_item.uuid		59b96438-be86-4397-a842-1e1c6efd7479
dev_item.fsid		7f1ce0ed-5986-43ae-b0dd-727eee19fd08 [match]
dev_item.type		0
dev_item.total_bytes	254007050240
dev_item.bytes_used	182066348032
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		1
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0
sys_chunk_array[2048]:
	item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0)
		chunk length 4194304 owner 2 stripe_len 65536
		type SYSTEM num_stripes 1
			stripe 0 devid 1 offset 0
			dev uuid: 59b96438-be86-4397-a842-1e1c6efd7479
	item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
		chunk length 8388608 owner 2 stripe_len 65536
		type SYSTEM|DUP num_stripes 2
			stripe 0 devid 1 offset 20971520
			dev uuid: 59b96438-be86-4397-a842-1e1c6efd7479
			stripe 1 devid 1 offset 29360128
			dev uuid: 59b96438-be86-4397-a842-1e1c6efd7479
backup_roots[4]:
	backup 0:
		backup_tree_root:	29360128	gen: 593817	level: 1
		backup_chunk_root:	20971520	gen: 591139	level: 1
		backup_extent_root:	29376512	gen: 593817	level: 1
		backup_fs_root:		132562944	gen: 593815	level: 0
		backup_dev_root:	29409280	gen: 593817	level: 0
		backup_csum_root:	29491200	gen: 593817	level: 0
		backup_total_bytes:	254007050240
		backup_bytes_used:	43532288
		backup_num_devices:	1

	backup 1:
		backup_tree_root:	138821632	gen: 593818	level: 0
		backup_chunk_root:	21020672	gen: 593818	level: 0
		backup_extent_root:	138805248	gen: 593818	level: 0
		backup_fs_root:		132562944	gen: 593815	level: 0
		backup_dev_root:	132841472	gen: 593818	level: 0
		backup_csum_root:	138870784	gen: 593818	level: 0
		backup_total_bytes:	254007050240
		backup_bytes_used:	655360
		backup_num_devices:	1

	backup 2:
		backup_tree_root:	138788864	gen: 593815	level: 1
		backup_chunk_root:	20971520	gen: 591139	level: 1
		backup_extent_root:	132841472	gen: 593815	level: 1
		backup_fs_root:		132562944	gen: 593815	level: 0
		backup_dev_root:	1018118144	gen: 592983	level: 0
		backup_csum_root:	134725632	gen: 593815	level: 0
		backup_total_bytes:	254007050240
		backup_bytes_used:	43532288
		backup_num_devices:	1

	backup 3:
		backup_tree_root:	138870784	gen: 593816	level: 1
		backup_chunk_root:	20971520	gen: 591139	level: 1
		backup_extent_root:	138887168	gen: 593816	level: 1
		backup_fs_root:		132562944	gen: 593815	level: 0
		backup_dev_root:	1018118144	gen: 592983	level: 0
		backup_csum_root:	138952704	gen: 593816	level: 0
		backup_total_bytes:	254007050240
		backup_bytes_used:	43532288
		backup_num_devices:	1

Re: btrfs-find-root duration?

[Chris Murphy]

I don't know, maybe. This is not a new file system, clearly, it has
half million+ generations.

backup_roots[4]:
    backup 0:
        backup_tree_root:    29360128    gen: 593817    level: 1
        backup_chunk_root:    20971520    gen: 591139    level: 1
        backup_extent_root:    29376512    gen: 593817    level: 1
        backup_fs_root:        132562944    gen: 593815    level: 0
        backup_dev_root:    29409280    gen: 593817    level: 0
        backup_csum_root:    29491200    gen: 593817    level: 0
        backup_total_bytes:    254007050240
        backup_bytes_used:    43532288
        backup_num_devices:    1


    backup 1:
        backup_tree_root:    138821632    gen: 593818    level: 0
        backup_chunk_root:    21020672    gen: 593818    level: 0
        backup_extent_root:    138805248    gen: 593818    level: 0
        backup_fs_root:        132562944    gen: 593815    level: 0
        backup_dev_root:    132841472    gen: 593818    level: 0
        backup_csum_root:    138870784    gen: 593818    level: 0
        backup_total_bytes:    254007050240
        backup_bytes_used:    655360
        backup_num_devices:    1


The old chunk root and tree root might come in handy, from backup0,
which is the generation prior to the one currently set in the super
blocks. The bytes_used vs dev_item.bytes_used is also interesting,
huge difference. So it looks like this was umounted very soon after
you realized what you did.

What do you get for

btrfs restore -l <dev>

Maybe just try this, which is the oldest generation we know about
right now, which is found in backup root 2.

sudo btrfs restore -D -v -t 138788864 /dev/mapper/brick1 .

That's a dry run so the path to save stuff to doesn't matter, so I'm
just using . for that. But what you'll get is a file listing. If
there's nothing listed, then it's a dead end. But if it's listing
everything you want, it's a win. Just point to a proper destination
and remove -D. However, another possibility is that it's a partial -
i.e. it might be a generation in between the cleaner running, so some
stuff is restored but not all of it.

Yeah, maybe there's a way to get an older generation and root with an
older copy of btrfs-progs. I think the current one isn't working very
well...on yet another file system I have that is very new (week) but
has hours of writes on it,

[chris@f25s ~]$ sudo btrfs-find-root /dev/mapper/brick1
Superblock thinks the generation is 5727
Superblock thinks the level is 1
Found tree root at 426847567872 gen 5727 level 1


That's it. So no hang, but only one tree root which is just bogus.
Even the superblock has more information than this. I'm used to seeing
dozens or more.


Chris Murphy

Re: btrfs-find-root duration?

[Chris Murphy]

Another idea is btrfs-find-root -a. This is slow for me, it took about
a minute for less than 1GiB of metadata. But I've got over 50
candidate tree roots and generations.

But still you can try the tree root for the oldest generation in your
full superblock listing, like I described. If that restore dry run is
an empty listing, or partial, then try the btrfs-find-root -a option
to get more candidate generations. For the most part each lower
generation number is 30 seconds older. So if you can think about when
you umounted the file system in relation to when the delete happened,
you can get closer to the most recent generation that still has your
data.


Chris Murphy

Re: btrfs-find-root duration?

[Markus Binsteiner]

On Mon, Dec 12, 2016 at 5:12 PM, Chris Murphy <lists@colorremedies.com> wrote:
> Another idea is btrfs-find-root -a. This is slow for me, it took about
> a minute for less than 1GiB of metadata. But I've got over 50
> candidate tree roots and generations.

Same behaviour with the newer versioned btrfs-find-root, just hangs.
The older one doesn't seem to have that option.

Re: btrfs-find-root duration?

[Markus Binsteiner]

Ok, some news. I chrooted into the old OS-root (Jessie), and low and
behold, old-version btrfs-find-root seemed to work:

# btrfs-find-root /dev/mapper/think--big-home
Super think's the tree root is at 138821632, chunk root 21020672
Well block 4194304 seems great, but generation doesn't match, have=2,
want=593818 level 0
Well block seems great, but generation doesn't match, have=3,
want=593818 level 0
Well block 29360128 seems great, but generation doesn't match,
have=593817, want=593818 level 1
Well block 29425664 seems great, but generation doesn't match,
have=593817, want=593818 level 0
Well block 29507584 seems great, but generation doesn't match,
have=593817, want=593818 level 0
Well block 29523968 seems great, but generation doesn't match,
have=593817, want=593818 level 0
Found tree root at 138821632 gen 593818 level 0

That's it though. Shouldn't there be many more lines for a filesytem that old?

I've tried btrfs restore for a few of those (both using old and new
btrfs restore versions), doesn't crash for the older generations:

# btrfs-find-root /dev/mapper/think--big-home
Super think's the tree root is at 138821632, chunk root 21020672
Well block 4194304 seems great, but generation doesn't match, have=2,
want=593818 level 0
Well block 4243456 seems great, but generation doesn't match, have=3,
want=593818 level 0
Well block 29360128 seems great, but generation doesn't match,
have=593817, want=593818 level 1
Well block 29425664 seems great, but generation doesn't match,
have=593817, want=593818 level 0
Well block 29507584 seems great, but generation doesn't match,
have=593817, want=593818 level 0
Well block 29523968 seems great, but generation doesn't match,
have=593817, want=593818 level 0
Found tree root at 138821632 gen 593818 level 0
root@think:/root# btrfs restore /dev/mapper/think--big-home /tmp -D -v
-i -t 4243456
parent transid verify failed on 4243456 wanted 593818 found 3
parent transid verify failed on 4243456 wanted 593818 found 3
Ignoring transid failure
This is a dry-run, no files are going to be restored
Reached the end of the tree searching the directory

But does crashes for 29360128 and 29425664:

# btrfs restore /dev/mapper/think--big-home /tmp -D -v -i -t 29360128
parent transid verify failed on 29360128 wanted 593818 found 593817
parent transid verify failed on 29360128 wanted 593818 found 593817
parent transid verify failed on 29360128 wanted 593818 found 593817
parent transid verify failed on 29360128 wanted 593818 found 593817
Ignoring transid failure
volumes.c:1554: btrfs_chunk_readonly: Assertion `!ce` failed.
btrfs[0x435f1e]
btrfs[0x435f42]
btrfs[0x4384f9]
btrfs[0x42f44a]
btrfs[0x42ab64]
btrfs[0x42aec5]
btrfs[0x42af74]
btrfs[0x41e7d9]
btrfs[0x40b46a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f618e103b45]
btrfs[0x40b497]

and fails for 29507584 and 29523968:

# btrfs restore /dev/mapper/think--big-home /tmp -D -v -i -t 29507584
parent transid verify failed on 29507584 wanted 593818 found 593817
parent transid verify failed on 29507584 wanted 593818 found 593817
parent transid verify failed on 29507584 wanted 593818 found 593817
parent transid verify failed on 29507584 wanted 593818 found 593817
Ignoring transid failure
Couldn't setup extent tree
Couldn't setup device tree
Could not open root, trying backup super
parent transid verify failed on 29507584 wanted 593818 found 593817
parent transid verify failed on 29507584 wanted 593818 found 593817
parent transid verify failed on 29507584 wanted 593818 found 593817
parent transid verify failed on 29507584 wanted 593818 found 593817
Ignoring transid failure
Couldn't setup extent tree
Couldn't setup device tree
Could not open root, trying backup super
No valid Btrfs found on /dev/mapper/think--big-home
Could not open root, trying backup super


> The old chunk root and tree root might come in handy, from backup0,
> which is the generation prior to the one currently set in the super
> blocks. The bytes_used vs dev_item.bytes_used is also interesting,
> huge difference. So it looks like this was umounted very soon after
> you realized what you did.

Yes, Did that as soon as I realized something was amiss. To be honest,
I'm not even 100% sure anymore what exactly I did, I just assumed I
deleted my home directory because I was deleting a directory a minute
or so before everything fell to pieces and my home directory was gone.
Also, it wouldn't be totally unlike me to have done that :-)


> What do you get for
>
> btrfs restore -l <dev>
>
> Maybe just try this, which is the oldest generation we know about
> right now, which is found in backup root 2.
>
> sudo btrfs restore -D -v -t 138788864 /dev/mapper/brick1 .

# btrfs restore -D -v -t 138788864 /dev/mapper/think--big-home  .
checksum verify failed on 138788864 found 2E842E3E wanted 9D4DED61
checksum verify failed on 138788864 found 2E842E3E wanted 9D4DED61
checksum verify failed on 138788864 found 62A944D5 wanted 0C5CE041
checksum verify failed on 138788864 found 62A944D5 wanted 0C5CE041
bytenr mismatch, want=138788864, have=3463101449461224633
Couldn't read tree root
Could not open root, trying backup super
checksum verify failed on 138788864 found 2E842E3E wanted 9D4DED61
checksum verify failed on 138788864 found 2E842E3E wanted 9D4DED61
checksum verify failed on 138788864 found 62A944D5 wanted 0C5CE041
checksum verify failed on 138788864 found 62A944D5 wanted 0C5CE041
bytenr mismatch, want=138788864, have=3463101449461224633
Couldn't read tree root
Could not open root, trying backup super
ERROR: superblock bytenr 274877906944 is larger than device size 254007050240
Could not open root, trying backup super

Cheers,
Markus