BTRFS fsck tool

BTRFS fsck tool

[Alexey A Nikitin]

Hi, everybody

I have BTRFS RAID0 setup with two disks. After some incident where I
had to force shutdown machine this array won't mount anymore, even
after "btrfs device scan". Unfortunately, I don't have backups since I
can't afford another 4TB of storage. I've read that fsck tool for
BTRFS that is expected to be released soon should allow recovery from
this situation, but I can't find info as to where I may find the
up-to-date release announcements so that I can download it when it's
out. Can somebody tell me how I can keep track of fsck development
status?

Best,

Alexey

--
This message was created with 100% recycled electrons

Re: BTRFS fsck tool

[cwillu]

On Fri, Mar 4, 2011 at 5:59 PM, Alexey A Nikitin <moonwalker@syrius.us> wrote:
> I have BTRFS RAID0 setup with two disks. After some incident where I
> had to force shutdown machine this array won't mount anymore, even
> after "btrfs device scan". Unfortunately, I don't have backups since I
> can't afford another 4TB of storage. I've read that fsck tool for
> BTRFS that is expected to be released soon should allow recovery from
> this situation, but I can't find info as to where I may find the
> up-to-date release announcements so that I can download it when it's
> out. Can somebody tell me how I can keep track of fsck development
> status?

I'd expect an announcement on this mailing list.

Depending on why it doesn't mount, there's a btrfs-select-super
command in btrfs-progs which may get you up and running again.

Re: BTRFS fsck tool

[Alexey A Nikitin]

I pulled the btrfs-select-super source from get branch 'next',
compiled it, but when I run it using

./btrfs-select-super -s 1 /dev/sdb1

all I get is

btrfs-select-super: disk-io.c:739: open_ctree_fd: Assertion
`!(!tree_root->node)' failed

Any ideas what might be wrong and how I can remedy it? I'm using
Debian Wheezy x64 with it's own btrfs-tools, could that be a problem,
should I completely replace stock btrfs-tools with the one from git
for it to work properly?

PS: I subscribed to this list now, so there is no more need to cc me.

Best,

Alexey

--
This message was created with 100% recycled electrons



2011/3/4 cwillu <cwillu@cwillu.com>:
> On Fri, Mar 4, 2011 at 5:59 PM, Alexey A Nikitin <moonwalker@syrius.us> wrote:
>> I have BTRFS RAID0 setup with two disks. After some incident where I
>> had to force shutdown machine this array won't mount anymore, even
>> after "btrfs device scan". Unfortunately, I don't have backups since I
>> can't afford another 4TB of storage. I've read that fsck tool for
>> BTRFS that is expected to be released soon should allow recovery from
>> this situation, but I can't find info as to where I may find the
>> up-to-date release announcements so that I can download it when it's
>> out. Can somebody tell me how I can keep track of fsck development
>> status?
>
> I'd expect an announcement on this mailing list.
>
> Depending on why it doesn't mount, there's a btrfs-select-super
> command in btrfs-progs which may get you up and running again.
>

Re: BTRFS fsck tool

[cwillu]

On Fri, Mar 4, 2011 at 7:20 PM, Alexey A Nikitin <moonwalker@syrius.us> wrote:
> I pulled the btrfs-select-super source from get branch 'next',
> compiled it, but when I run it using
>
> ./btrfs-select-super -s 1 /dev/sdb1
>
> all I get is
>
> btrfs-select-super: disk-io.c:739: open_ctree_fd: Assertion
> `!(!tree_root->node)' failed
>
> Any ideas what might be wrong and how I can remedy it? I'm using
> Debian Wheezy x64 with it's own btrfs-tools, could that be a problem,
> should I completely replace stock btrfs-tools with the one from git
> for it to work properly?

Make sure you run it as root, and try both -s 1 and -s 2.

> PS: I subscribed to this list now, so there is no more need to cc me.

Reply to all does it automatically.

Re: BTRFS fsck tool

[Alexey A Nikitin]

2011/3/4 cwillu <cwillu@cwillu.com>:
> On Fri, Mar 4, 2011 at 7:20 PM, Alexey A Nikitin <moonwalker@syrius.us> wrote:
>> all I get is
>>
>> btrfs-select-super: disk-io.c:739: open_ctree_fd: Assertion
>> `!(!tree_root->node)' failed
>
> Make sure you run it as root, and try both -s 1 and -s 2.
>

Done that - commands were ran from root shell and I tried both -s 1
and -s 2. It gives same error with both parameters for both
btrfs-select-super and btrfsck.

>> PS: I subscribed to this list now, so there is no more need to cc me.
>
> Reply to all does it automatically.
>

Indeed it does, I didn't know that.


Best,

Alexey

--
This message was created with 100% recycled electrons

Re: BTRFS fsck tool

[Helmut Hullen]

Hallo, Alexey,

Du meintest am 04.03.11:

> I have BTRFS RAID0 setup with two disks. After some incident where I
> had to force shutdown machine this array won't mount anymore, even
> after "btrfs device scan". Unfortunately, I don't have backups since
> I can't afford another 4TB of storage.

Same result (with another history) here. I've asked 2 data rescue  
companies (well known), they can't yet recover btrfs.

I had to learn that btrfs is still in experimental status.

Viele Gruesse!
Helmut

Re: BTRFS fsck tool

[Alexey A Nikitin]

2011/3/5 Helmut Hullen <Hullen@t-online.de>:
> I had to learn that btrfs is still in experimental status.

Oh, I was perfectly aware of it's experimental status when I was
transferring to btrfs all my home systems, that's why I'm not even
thinking about moving to btrfs my servers. I moved to btrfs because
all other fs had unacceptable performance on my old MLC SSDs and it
was the only non-destructive method of building RAID out of HDDs with
ext4 partitions.

Best,

Alexey

--
This message was created with 100% recycled electrons

Re: BTRFS fsck tool

[Spelic]

On 03/05/2011 12:59 AM, Alexey A Nikitin wrote:
> Hi, everybody
>
> I have BTRFS RAID0 setup with two disks. After some incident where I
> had to force shutdown machine this array won't mount anymore


Have you "reset" the machine or cut the power?
I don't really know btrfs, but in btrfs wiki there is written:

"Note that Btrfs does not yet have a fsck tool that can fix errors.
While Btrfs is stable on a stable machine, it is currently possible to
corrupt a filesystem irrecoverably if your machine crashes or loses
power on disks that don't handle flush requests correctly. This will be
fixed when the fsck tool is ready."

is this your scenario? If you leave the power and stop submitting writes
to the disks (like with a reset) the filesystem should be preserved, if
that sentence is true.
Did you cut the power while there was some process writing to it? If the
kernel was panicked it is probably safe to cut the power (because the
last write happened long time ago and disks had time to put that to the
platters)

Also what disks brand/model do you have? The sentence speaks about disks
which don't honour flush+FUA requests, I think. Enterprise disks should
honour that, consumer disks might not.

I am interesed in what happened because I am evaluating using btrfs for
very large backups (still losing those wouldn't be totally nice)

Thank you

Re: BTRFS fsck tool

[Alexey A Nikitin]

2011/3/7 Spelic <spelic@shiftmail.org>
>
> On 03/05/2011 12:59 AM, Alexey A Nikitin wrote:
> > Hi, everybody
> >
> > I have BTRFS RAID0 setup with two disks. After some incident where I
> > had to force shutdown machine this array won't mount anymore
>
>
> Have you "reset" the machine or cut the power?

4s power button.

> is this your scenario? If you leave the power and stop submitting writes
> to the disks (like with a reset) the filesystem should be preserved, if
> that sentence is true.
> Did you cut the power while there was some process writing to it? If the
> kernel was panicked it is probably safe to cut the power (because the
> last write happened long time ago and disks had time to put that to the
> platters)

Disks were idle. Kernel didn't panic, machine lost network and that
old laptop has weird glitch that if you start it with closed lid
screen will never come back on later on unless you restart it. Since I
have no external monitor and at the time magic SysRq was disabled I
had no other option but to force shutdown it and then start it up
again. All of the btrfs partitions on internal drive are perfectly
fine.

> Also what disks brand/model do you have? The sentence speaks about disks
> which don't honour flush+FUA requests, I think. Enterprise disks should
> honour that, consumer disks might not.

Disks in question are two plain consumer grade Samsung 2TB SATA disks
in external USB 2.0/eSATA enclosures with independent power supplies,
connected through USB 2.0 due to lack of eSATA support in that old
laptop that works as my home server.

> I am interesed in what happened because I am evaluating using btrfs for
> very large backups (still losing those wouldn't be totally nice)

If it is just a backup I wouldn't bother with performance and features
if I were you and would go instead with most tested and reliable
system that still provides the bare minimum of absolutely must
features. While the snapshots feature is a nice thing to have, IMHO
it's not exactly for stash-away backups. I'd go instead with ext3 or
XFS, depending on the size of files and overall size of backups. The
only reason why I went experimenting with btrfs RAID0 on my USB setup
is because I'm a reckless experimenter when it doesn't involve
production systems.


Best,

Alexey

--
This message was created with 100% recycled electrons

Re: BTRFS fsck tool

[Peter Stuge]

Hi,

Alexey A Nikitin wrote:
> I went experimenting with btrfs RAID0 on my USB setup .. because
> I'm a reckless experimenter when it doesn't involve production
> systems.

I encountered the same broken root node issue. (see thread resize ate
my root node) and I'd like to understand it better. Hopefully there
will come something out of this. I'm surprised that my resize broke
the filesystem for mounting, when it looked good right after the
resize.

When I looked through the archive and found your thread I started
thinking, and I am not absolutely sure if there was some disk
activity when I cut power to my system, but I believe there was not.
(I ran reboot, but soft poweroff doesn't work with recent kernels. At
the Rebooting system message I switched hard power off.)


//Peter

Re: BTRFS fsck tool

[Clemens Eisserer]

Well, the missing file-system checker is the main reason I don't use
btrfs in production environments.
The other issue are servere performance problems in corner cases (e.g.
when deleting 15GB data takes 100% cpu and hours)

- Clemens

2011/3/8 Peter Stuge <peter@stuge.se>:
> Hi,
>
> Alexey A Nikitin wrote:
>> I went experimenting with btrfs RAID0 on my USB setup .. because
>> I'm a reckless experimenter when it doesn't involve production
>> systems.
>
> I encountered the same broken root node issue. (see thread resize ate
> my root node) and I'd like to understand it better. Hopefully there
> will come something out of this. I'm surprised that my resize broke
> the filesystem for mounting, when it looked good right after the
> resize.
>
> When I looked through the archive and found your thread I started
> thinking, and I am not absolutely sure if there was some disk
> activity when I cut power to my system, but I believe there was not.
> (I ran reboot, but soft poweroff doesn't work with recent kernels. At
> the Rebooting system message I switched hard power off.)
>
>
> //Peter
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =C2=A0http://vger.kernel.org/majordomo-info.ht=
ml
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS fsck tool

[Chris Mason]

Excerpts from Peter Stuge's message of 2011-03-08 01:52:55 -0500:
> Hi,
> 
> Alexey A Nikitin wrote:
> > I went experimenting with btrfs RAID0 on my USB setup .. because
> > I'm a reckless experimenter when it doesn't involve production
> > systems.
> 
> I encountered the same broken root node issue. (see thread resize ate
> my root node) and I'd like to understand it better. Hopefully there
> will come something out of this. I'm surprised that my resize broke
> the filesystem for mounting, when it looked good right after the
> resize.
> 
> When I looked through the archive and found your thread I started
> thinking, and I am not absolutely sure if there was some disk
> activity when I cut power to my system, but I believe there was not.
> (I ran reboot, but soft poweroff doesn't work with recent kernels. At
> the Rebooting system message I switched hard power off.)

Cutting the power isn't problem unless you're using something
where cache flushes are not supported.

Which kernel were you on?  Was btrfs directly accessing the disks or
were things like LVM in use?

Recent kernels (.37 and higher) have improved support for barriers in
LVM and friends, but btrfs directly using the disks should have been
safe for a long time.

I think with the resize something else went wrong.

-chris

Re: BTRFS fsck tool

[Peter Stuge]

Chris Mason wrote:
> Cutting the power isn't problem unless you're using something
> where cache flushes are not supported.

Nod. I've had very abrupt system outage before, without problems.


> Which kernel were you on?

2.6.38-rc6 + wireless-testing.git


> Was btrfs directly accessing the disks or were things like LVM in use?

Directly.


> I think with the resize something else went wrong.

I don't know where to look exactly. Any hint on which data structures
to focus on are welcome.


//Peter

Re: BTRFS fsck tool

[Chris Mason]

Excerpts from Peter Stuge's message of 2011-03-10 08:29:37 -0500:
> Chris Mason wrote:
> > Cutting the power isn't problem unless you're using something
> > where cache flushes are not supported.
> 
> Nod. I've had very abrupt system outage before, without problems.
> 
> > Which kernel were you on?
> 
> 2.6.38-rc6 + wireless-testing.git
> 
> > Was btrfs directly accessing the disks or were things like LVM in use?
> 
> Directly.
> 
> > I think with the resize something else went wrong.
> 
> I don't know where to look exactly. Any hint on which data structures
> to focus on are welcome.

I moved my questions back to the resize thread ;)

-chris

Re: BTRFS fsck tool

[Alexey A Nikitin]

2011/3/10 Chris Mason <chris.mason@oracle.com>
>
> Which kernel were you on? =C2=A0Was btrfs directly accessing the disk=
s or
> were things like LVM in use?
> Recent kernels (.37 and higher) have improved support for barriers in
> LVM and friends, but btrfs directly using the disks should have been
> safe for a long time.

Now that's funny. I'm using  2.6.32-5-amd64 from Debian Wheezy and
while btrfs on top of LVM works perfectly stable I have now trouble
with FS that is directly on partition. Does it make difference
stability-wise if partition table on disk is GPT rather than
old-school MS-DOS? Because all my disks, LVM and non-LVM, are set with
GPT.

Best,

Alexey

--
This message was created with 100% recycled electrons
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS fsck tool

[Chris Mason]

Excerpts from Alexey A Nikitin's message of 2011-03-10 12:30:54 -0500:
> 2011/3/10 Chris Mason <chris.mason@oracle.com>
> >
> > Which kernel were you on? =C2=A0Was btrfs directly accessing the di=
sks or
> > were things like LVM in use?
> > Recent kernels (.37 and higher) have improved support for barriers =
in
> > LVM and friends, but btrfs directly using the disks should have bee=
n
> > safe for a long time.
>=20
> Now that's funny. I'm using  2.6.32-5-amd64 from Debian Wheezy and
> while btrfs on top of LVM works perfectly stable I have now trouble
> with FS that is directly on partition. Does it make difference
> stability-wise if partition table on disk is GPT rather than
> old-school MS-DOS? Because all my disks, LVM and non-LVM, are set wit=
h
> GPT.

LVM is very reliable, the only time you run into trouble is if you have
a drive with writeback cache enabled (99.99% of sata drives) and LVM
isn't passing the barriers through.  GPT vs MS-DOS isn't crucial, its
just important that if you have a writeback cache,  you either get the
barriers down to the drive or you turn off the writeback cache.

-chris
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS fsck tool

[Spelic]

On 03/10/2011 02:02 PM, Chris Mason wrote:
> Cutting the power isn't problem unless you're using something
> where cache flushes are not supported.

Some disks lie about cache flush having completed.

But why doesn't the option for mounting with an earlier superblock work?
Could you improve that area?

I think that, except on "near to enospc" situations, the new blocks
being allocated for FS operation should have been free for a while, at
least one hour (*). In this way new filesystem operations would not
overwrite old superblocks and old data structures and these would remain
readable to get a consistent "old version" of the filesystem.
So in case of abrupt poweroff and wrong flush mechanism, the user could
still mount with an, e.g., 10-minutes older superblock and get a
workable 10-minutes older version of the filesystem.
Or am I missing something?

(*) 1 hour would render highly unlikely that data stays for that long
uncommitted in the drive's writeback cache. The drive flushes all data
out whenever it's idle...

Thank you

Re: BTRFS fsck tool

[Ric Wheeler]

On 03/12/2011 05:49 PM, Spelic wrote:
> On 03/10/2011 02:02 PM, Chris Mason wrote:
>> Cutting the power isn't problem unless you're using something
>> where cache flushes are not supported.
> Some disks lie about cache flush having completed.

This is really not true for modern enterprise class drives. You might have more 
issues with USB thumbdrives and other really low end parts.

Ric

> But why doesn't the option for mounting with an earlier superblock work?
> Could you improve that area?
>
> I think that, except on "near to enospc" situations, the new blocks
> being allocated for FS operation should have been free for a while, at
> least one hour (*). In this way new filesystem operations would not
> overwrite old superblocks and old data structures and these would remain
> readable to get a consistent "old version" of the filesystem.
> So in case of abrupt poweroff and wrong flush mechanism, the user could
> still mount with an, e.g., 10-minutes older superblock and get a
> workable 10-minutes older version of the filesystem.
> Or am I missing something?
>
> (*) 1 hour would render highly unlikely that data stays for that long
> uncommitted in the drive's writeback cache. The drive flushes all data
> out whenever it's idle...
>
> Thank you
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS fsck tool

[Hubert Kario]

On Sunday, March 13, 2011 00:53:00 Ric Wheeler wrote:
> On 03/12/2011 05:49 PM, Spelic wrote:
> > On 03/10/2011 02:02 PM, Chris Mason wrote:
> >> Cutting the power isn't problem unless you're using something
> >> where cache flushes are not supported.
> >=20
> > Some disks lie about cache flush having completed.
>=20
> This is really not true for modern enterprise class drives. You might=
 have
> more issues with USB thumbdrives and other really low end parts.

btrfs is supposed to be an ext3/4 replacement - it _will_ be used with =
low end=20
parts (commodity SATA HDDs)

Regards
--=20
Hubert Kario
QBS - Quality Business Software
02-656 Warszawa, ul. Ksawer=C3=B3w 30/85
tel. +48 (22) 646-61-51, 646-74-24
www.qbs.com.pl
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: BTRFS fsck tool

[Peter Stuge]

Hubert Kario wrote:
> btrfs is supposed to be an ext3/4 replacement

Maybe not just yet. :)


//Peter