btrfs suddenly lost all om my huge free space

btrfs suddenly lost all om my huge free space

[Tommy Pettersson]

Hi,

(I'm not subscribed to the list, so please CC me.)

I have a btrfs with raid1 on two identical unpartitioned disks.
Today I noticed that df (normal df) said I am 77 % full. This
was a chock, because since forever it has been around 12 %.


# btrfs fi show
Label: 'green'  uuid: dd83031c-2447-4736-a8f6-9bd9cdeea879
        Total devices 2 FS bytes used 212.88GB
        devid    2 size 1.82TB used 356.04GB path /dev/sdb
        devid    1 size 1.82TB used 356.06GB path /dev/sda

# btrfs fi df /
Data, RAID1: total=276.00GB, used=209.02GB
Data: total=8.00M, used=0.00
System, RAID1: total=40.00MB, used=64.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=80.00GB, used=3.88GB
Metadata: total=8.00MB, used=0.00

# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          3.7T  426G   134G  77% /


The thing that has drastically changed is Avail in the output
from df.

I tried a btrfs balance, which self-aborted after some hours
with No space left on device. I deleted two snapshots, so I got
some free space and could use the system again.

The balance, although it didn't finish, seems to have reduced
the used space, but it also reduced the "available" space:


# btrfs fi show
Label: 'green'  uuid: dd83031c-2447-4736-a8f6-9bd9cdeea879
        Total devices 2 FS bytes used 212.88GB
        devid    2 size 1.82TB used 356.04GB path /dev/sdb
        devid    1 size 1.82TB used 215.01GB path /dev/sda

# btrfs fi df /
Data, RAID1: total=210.00GB, used=197.97GB
System, RAID1: total=8.00MB, used=44.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=5.00GB, used=3.41GB

# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          3.7T  403G   25G  95% /


I made an unqualified guess that the space cache was corrupted,
and tried to mount with option clear_cache and nospace_cache.
Both of them caused btrfs to scan my disks for a couple of
minutes at boot, but the amount of available space did not
improve.

What can I do to help locate the cause of this problem?


Regards,
Tommy

Re: btrfs suddenly lost all om my huge free space

[Goffredo Baroncelli]

Hi,

did you used the latest kernel version ?
The other thing that you could try is a scrub looking for a defective 
page.. but I don't think so....

BR
G.Baroncelli



On 2012-10-14 02:19, Tommy Pettersson wrote:
> Hi,
>
> (I'm not subscribed to the list, so please CC me.)
>
> I have a btrfs with raid1 on two identical unpartitioned disks.
> Today I noticed that df (normal df) said I am 77 % full. This
> was a chock, because since forever it has been around 12 %.
>
>
> # btrfs fi show
> Label: 'green'  uuid: dd83031c-2447-4736-a8f6-9bd9cdeea879
>          Total devices 2 FS bytes used 212.88GB
>          devid    2 size 1.82TB used 356.04GB path /dev/sdb
>          devid    1 size 1.82TB used 356.06GB path /dev/sda
>
> # btrfs fi df /
> Data, RAID1: total=276.00GB, used=209.02GB
> Data: total=8.00M, used=0.00
> System, RAID1: total=40.00MB, used=64.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=80.00GB, used=3.88GB
> Metadata: total=8.00MB, used=0.00
>
> # df -h
> Filesystem      Size  Used Avail Use% Mounted on
> rootfs          3.7T  426G   134G  77% /
>
>
> The thing that has drastically changed is Avail in the output
> from df.
>
> I tried a btrfs balance, which self-aborted after some hours
> with No space left on device. I deleted two snapshots, so I got
> some free space and could use the system again.
>
> The balance, although it didn't finish, seems to have reduced
> the used space, but it also reduced the "available" space:
>
>
> # btrfs fi show
> Label: 'green'  uuid: dd83031c-2447-4736-a8f6-9bd9cdeea879
>          Total devices 2 FS bytes used 212.88GB
>          devid    2 size 1.82TB used 356.04GB path /dev/sdb
>          devid    1 size 1.82TB used 215.01GB path /dev/sda
>
> # btrfs fi df /
> Data, RAID1: total=210.00GB, used=197.97GB
> System, RAID1: total=8.00MB, used=44.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=5.00GB, used=3.41GB
>
> # df -h
> Filesystem      Size  Used Avail Use% Mounted on
> rootfs          3.7T  403G   25G  95% /
>
>
> I made an unqualified guess that the space cache was corrupted,
> and tried to mount with option clear_cache and nospace_cache.
> Both of them caused btrfs to scan my disks for a couple of
> minutes at boot, but the amount of available space did not
> improve.
>
> What can I do to help locate the cause of this problem?
>
>
> Regards,
> Tommy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: btrfs suddenly lost all om my huge free space

[Tommy Pettersson]

The problem has been resolved, but I think it will be impossible
to figure out what went wrong. The root cause was I accidentally
messed up my initrd so that btrfs was mounted without prior dev
scan (which I think didn't work with earlier kernels, but now
(3.4.9-gentoo) it "worked" in a very bad way it seems), and
possibly that I also mounted subvolid=0 (containing the subvol I
previously mounted as / ) with conflicting mount options for
space_cache.

But after I had realized and fixed that, it was too late. Both
Scrub and Balance, and reading from the filesystem, behaved
strange. The output of df jumped between 95 % and 12 %, while I
got many lines about wrong checksums, unexpected tree parent
generation something, and free space inode generation (0) did
not match free space cache. It sometimes said it corrected
things, but it didn't seem to help, and at random points I would
get a kernel panic.

# uname -a
Linux fruit64 3.4.9-gentoo #2 SMP PREEMPT Sat Sep 1 17:34:38 CEST 2012 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ AuthenticAMD GNU/Linux

# btrfs --version
Btrfs Btrfs v0.19

It would have been nice to debug this mess so that btrfs could
handled it in the future, and not do all the strange things with
the free space and cause kernel panics, but I had to get my
system back up.

The good news is that even this torture of my bits didn't
actually kill them. I eventually cleared the btrfs master record
on one of the disks, mounted in degraded mode, added it back,
waited seven hours for balance to finish, and now my filesystem
is consistent again, and everything is back to normal. So no
need to restore from my daily backup yet. :-)


Regards,
Tommy


On Sun, Oct 14, 2012 at 06:11:58PM +0200, Goffredo Baroncelli wrote:
> Hi,
> 
> did you used the latest kernel version ?
> The other thing that you could try is a scrub looking for a defective 
> page.. but I don't think so....
> 
> BR
> G.Baroncelli
> 
> 
> 
> On 2012-10-14 02:19, Tommy Pettersson wrote:
> > Hi,
> >
> > (I'm not subscribed to the list, so please CC me.)
> >
> > I have a btrfs with raid1 on two identical unpartitioned disks.
> > Today I noticed that df (normal df) said I am 77 % full. This
> > was a chock, because since forever it has been around 12 %.
> >
> >
> > # btrfs fi show
> > Label: 'green'  uuid: dd83031c-2447-4736-a8f6-9bd9cdeea879
> >          Total devices 2 FS bytes used 212.88GB
> >          devid    2 size 1.82TB used 356.04GB path /dev/sdb
> >          devid    1 size 1.82TB used 356.06GB path /dev/sda
> >
> > # btrfs fi df /
> > Data, RAID1: total=276.00GB, used=209.02GB
> > Data: total=8.00M, used=0.00
> > System, RAID1: total=40.00MB, used=64.00KB
> > System: total=4.00MB, used=0.00
> > Metadata, RAID1: total=80.00GB, used=3.88GB
> > Metadata: total=8.00MB, used=0.00
> >
> > # df -h
> > Filesystem      Size  Used Avail Use% Mounted on
> > rootfs          3.7T  426G   134G  77% /
> >
> >
> > The thing that has drastically changed is Avail in the output
> > from df.
> >
> > I tried a btrfs balance, which self-aborted after some hours
> > with No space left on device. I deleted two snapshots, so I got
> > some free space and could use the system again.
> >
> > The balance, although it didn't finish, seems to have reduced
> > the used space, but it also reduced the "available" space:
> >
> >
> > # btrfs fi show
> > Label: 'green'  uuid: dd83031c-2447-4736-a8f6-9bd9cdeea879
> >          Total devices 2 FS bytes used 212.88GB
> >          devid    2 size 1.82TB used 356.04GB path /dev/sdb
> >          devid    1 size 1.82TB used 215.01GB path /dev/sda
> >
> > # btrfs fi df /
> > Data, RAID1: total=210.00GB, used=197.97GB
> > System, RAID1: total=8.00MB, used=44.00KB
> > System: total=4.00MB, used=0.00
> > Metadata, RAID1: total=5.00GB, used=3.41GB
> >
> > # df -h
> > Filesystem      Size  Used Avail Use% Mounted on
> > rootfs          3.7T  403G   25G  95% /
> >
> >
> > I made an unqualified guess that the space cache was corrupted,
> > and tried to mount with option clear_cache and nospace_cache.
> > Both of them caused btrfs to scan my disks for a couple of
> > minutes at boot, but the amount of available space did not
> > improve.
> >
> > What can I do to help locate the cause of this problem?
> >
> >
> > Regards,
> > Tommy
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 

Re: btrfs suddenly lost all om my huge free space

[Goffredo Baroncelli]

On 2012-10-14 20:35, Tommy Pettersson wrote:
> The problem has been resolved, but I think it will be impossible
> to figure out what went wrong. The root cause was I accidentally
> messed up my initrd so that btrfs was mounted without prior dev
> scan (which I think didn't work with earlier kernels, but now
> (3.4.9-gentoo) it "worked" in a very bad way it seems), and
> possibly that I also mounted subvolid=0 (containing the subvol I
> previously mounted as / ) with conflicting mount options for
> space_cache.

This is a very strange behaviour; I am not aware of any bug which could 
justify this.
>
> But after I had realized and fixed that, it was too late. Both
> Scrub and Balance, and reading from the filesystem, behaved
> strange. The output of df jumped between 95 % and 12 %, while I
> got many lines about wrong checksums, unexpected tree parent
> generation something, and free space inode generation (0) did
> not match free space cache. It sometimes said it corrected
> things, but it didn't seem to help, and at random points I would
> get a kernel panic.
>
> # uname -a
> Linux fruit64 3.4.9-gentoo #2 SMP PREEMPT Sat Sep 1 17:34:38 CEST 2012 x86_64 AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ AuthenticAMD GNU/Linux

The 3.4.9 is a quite old kernel. I am guessing if a recent kernel would 
still behave as you described.

>
> # btrfs --version
> Btrfs Btrfs v0.19
>
> It would have been nice to debug this mess so that btrfs could
> handled it in the future, and not do all the strange things with
> the free space and cause kernel panics, but I had to get my
> system back up.
>
> The good news is that even this torture of my bits didn't
> actually kill them. I eventually cleared the btrfs master record
> on one of the disks, mounted in degraded mode, added it back,
> waited seven hours for balance to finish, and now my filesystem
> is consistent again, and everything is back to normal. So no
> need to restore from my daily backup yet. :-)

Good !

>
>
> Regards,
> Tommy
>
>
> On Sun, Oct 14, 2012 at 06:11:58PM +0200, Goffredo Baroncelli wrote:
>> Hi,
>>
>> did you used the latest kernel version ?
>> The other thing that you could try is a scrub looking for a defective
>> page.. but I don't think so....
>>
>> BR
>> G.Baroncelli
>>
>>
>>
>> On 2012-10-14 02:19, Tommy Pettersson wrote:
>>> Hi,
>>>
>>> (I'm not subscribed to the list, so please CC me.)
>>>
>>> I have a btrfs with raid1 on two identical unpartitioned disks.
>>> Today I noticed that df (normal df) said I am 77 % full. This
>>> was a chock, because since forever it has been around 12 %.
>>>
>>>
>>> # btrfs fi show
>>> Label: 'green'  uuid: dd83031c-2447-4736-a8f6-9bd9cdeea879
>>>           Total devices 2 FS bytes used 212.88GB
>>>           devid    2 size 1.82TB used 356.04GB path /dev/sdb
>>>           devid    1 size 1.82TB used 356.06GB path /dev/sda
>>>
>>> # btrfs fi df /
>>> Data, RAID1: total=276.00GB, used=209.02GB
>>> Data: total=8.00M, used=0.00
>>> System, RAID1: total=40.00MB, used=64.00KB
>>> System: total=4.00MB, used=0.00
>>> Metadata, RAID1: total=80.00GB, used=3.88GB
>>> Metadata: total=8.00MB, used=0.00
>>>
>>> # df -h
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> rootfs          3.7T  426G   134G  77% /
>>>
>>>
>>> The thing that has drastically changed is Avail in the output
>>> from df.
>>>
>>> I tried a btrfs balance, which self-aborted after some hours
>>> with No space left on device. I deleted two snapshots, so I got
>>> some free space and could use the system again.
>>>
>>> The balance, although it didn't finish, seems to have reduced
>>> the used space, but it also reduced the "available" space:
>>>
>>>
>>> # btrfs fi show
>>> Label: 'green'  uuid: dd83031c-2447-4736-a8f6-9bd9cdeea879
>>>           Total devices 2 FS bytes used 212.88GB
>>>           devid    2 size 1.82TB used 356.04GB path /dev/sdb
>>>           devid    1 size 1.82TB used 215.01GB path /dev/sda
>>>
>>> # btrfs fi df /
>>> Data, RAID1: total=210.00GB, used=197.97GB
>>> System, RAID1: total=8.00MB, used=44.00KB
>>> System: total=4.00MB, used=0.00
>>> Metadata, RAID1: total=5.00GB, used=3.41GB
>>>
>>> # df -h
>>> Filesystem      Size  Used Avail Use% Mounted on
>>> rootfs          3.7T  403G   25G  95% /
>>>
>>>
>>> I made an unqualified guess that the space cache was corrupted,
>>> and tried to mount with option clear_cache and nospace_cache.
>>> Both of them caused btrfs to scan my disks for a couple of
>>> minutes at boot, but the amount of available space did not
>>> improve.
>>>
>>> What can I do to help locate the cause of this problem?
>>>
>>>
>>> Regards,
>>> Tommy
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>