corrupt filesystem after snapshots on 3.2

corrupt filesystem after snapshots on 3.2

[Russell Coker]

[-- Attachment #1: Type: Text/Plain, Size: 603 bytes --]

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695886

The above bug concerns a RAID-1 filesystem that got corrupt when it ran out of 
disk space due to too many snapshots while running a 3.2.x kernel.  Running it 
with a 3.7.1-1~experimental.1 kernel from Debian still doesn't work (I've 
attached dmesg output after a du on the filesystem caused a kernel panic).  
This particular test run was done with the filesystem mounted -o recovery.

Also btrfsck doesn't seem to help things.

Any suggestions?

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

[-- Attachment #2: dmesg.gz --]
[-- Type: application/x-gzip, Size: 17765 bytes --]

Re: corrupt filesystem after snapshots on 3.2

[Chris Murphy]

On Dec 28, 2012, at 4:31 PM, Russell Coker <russell+btrfs@coker.com.au> wrote:

> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695886
> 
> The above bug concerns a RAID-1 filesystem that got corrupt when it ran out of 
> disk space due to too many snapshots while running a 3.2.x kernel.  Running it 
> with a 3.7.1-1~experimental.1 kernel from Debian still doesn't work (I've 
> attached dmesg output after a du on the filesystem caused a kernel panic).  
> This particular test run was done with the filesystem mounted -o recovery.
> 
> Also btrfsck doesn't seem to help things.
> 
> Any suggestions?

Add another disk, or disk partition, to the Btrfs volume. That'll give it more space and hopefully you can back out at that point.


Chris Murphy

Re: corrupt filesystem after snapshots on 3.2

[Russell Coker]

On Sat, 29 Dec 2012, Chris Murphy <lists@colorremedies.com> wrote:
> Add another disk, or disk partition, to the Btrfs volume. That'll give it
> more space and hopefully you can back out at that point.

# df -h /mnt/tmp
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/rjca-crypt   12G   12G  637M  95% /mnt/tmp

I didn't expect that to work because the filesystem already reported free 
space.  I removed the snapshots and had problems during/after the removal so 
it seemed that there was space.

# btrfs filesystem show --all-devices
Label: none  uuid: 0f0b48e9-d2f7-4280-b417-4d1f5a933975
        Total devices 3 FS bytes used 5.63GB
        devid    1 size 6.00GB used 6.00GB path /dev/dm-16
        devid    2 size 6.00GB used 6.00GB path /dev/dm-17
        devid    3 size 6.00GB used 19.00MB path /dev/dm-18

However I've added another device and now "du -h" completes without any kernel 
panic (a major improvement).  Now I'm seeing the above, apparently 19MB used 
on the new dm-18 device seems to be making all the difference.

# btrfs subvol list /mnt/tmp
ERROR: Failed to lookup path for root 0 - No such file or directory

Strangely I was getting the above error while the btrfs scrub was in progress, 
but it went away when the scrub finished.  That seems like a bug to me.

Now the filesystem is now OK and I've copied all the data off it.  The data 
appears to be all intact (much of it could be verified against another system).

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: corrupt filesystem after snapshots on 3.2

[Chris Murphy]

On Dec 28, 2012, at 8:14 PM, Russell Coker <russell@coker.com.au> wrote:

> On Sat, 29 Dec 2012, Chris Murphy <lists@colorremedies.com> wrote:
>> Add another disk, or disk partition, to the Btrfs volume. That'll give it
>> more space and hopefully you can back out at that point.
> 
> # df -h /mnt/tmp
> Filesystem              Size  Used Avail Use% Mounted on
> /dev/mapper/rjca-crypt   12G   12G  637M  95% /mnt/tmp
> 
> I didn't expect that to work because the filesystem already reported free 
> space.  I removed the snapshots and had problems during/after the removal so 
> it seemed that there was space.
> 
> # btrfs filesystem show --all-devices
> Label: none  uuid: 0f0b48e9-d2f7-4280-b417-4d1f5a933975
>        Total devices 3 FS bytes used 5.63GB
>        devid    1 size 6.00GB used 6.00GB path /dev/dm-16
>        devid    2 size 6.00GB used 6.00GB path /dev/dm-17
>        devid    3 size 6.00GB used 19.00MB path /dev/dm-18
> 
> However I've added another device and now "du -h" completes without any kernel 
> panic (a major improvement).  Now I'm seeing the above, apparently 19MB used 
> on the new dm-18 device seems to be making all the difference.
> 
> # btrfs subvol list /mnt/tmp
> ERROR: Failed to lookup path for root 0 - No such file or directory
> 
> Strangely I was getting the above error while the btrfs scrub was in progress, 
> but it went away when the scrub finished.  That seems like a bug to me.
> 
> Now the filesystem is now OK and I've copied all the data off it.  The data 
> appears to be all intact (much of it could be verified against another system).


Do you have a 'btrfs fi df /mnt' from before adding another device? And also after. I'm not sure what the pattern is but it seems Btrfs still can get wedged into a space corner with what seems like some space left in (conventional) df, but in fact it's insufficient somehow. For now I think it needs to be given a wide berth, except if you're intentionally trying to poke it with a stick to see what happens. (I like poking things. People too.)


Chris Murphy