btrfsck out of memory for big fs

btrfsck out of memory for big fs

[litaibaichina]

Hi Guys,

I had a big btrfs on a md device,  it be mounted , but after a while it will become ready only:
# btrfs fi df /data/
Data, single: total=24.46TiB, used=24.46TiB
System, DUP: total=8.00MiB, used=2.59MiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=81.00GiB, used=79.71GiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=264.28MiB

# dmesg -T
[Thu Aug 16 18:16:31 2018] BTRFS error (device md127): parent transid verify failed on 26603622694912 wanted 185320 found 207817
[Thu Aug 16 18:16:31 2018] BTRFS error (device md127): parent transid verify failed on 26603622694912 wanted 185320 found 207817
[Thu Aug 16 18:16:31 2018] BTRFS warning (device md127): Skipping commit of aborted transaction.
[Thu Aug 16 18:16:31 2018] BTRFS: error (device md127) in cleanup_transaction:1864: errno=-5 IO failure
[Thu Aug 16 18:16:31 2018] BTRFS info (device md127): forced readonly
[Thu Aug 16 18:16:31 2018] BTRFS info (device md127): delayed_refs has NO entry

I want to use btrfsck to check it,  but it will OOM :
# btrfsck /dev/md127
Checking filesystem on /dev/md127
UUID: 6b87a52f-9a5f-4d03-b345-9d954c2ce259
checking extents
Killed

I am trying to use lowmem mode,  but as I tried before, it may OOM too,    any ideas ?

Thanks.

Re: btrfsck out of memory for big fs

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 1898 bytes --]



On 2018/8/17 上午10:44, litaibaichina@gmail.com wrote:
> Hi Guys,
> 
> I had a big btrfs on a md device,  it be mounted , but after a while it will become ready only:
> # btrfs fi df /data/
> Data, single: total=24.46TiB, used=24.46TiB
> System, DUP: total=8.00MiB, used=2.59MiB
> System, single: total=4.00MiB, used=0.00B
> Metadata, DUP: total=81.00GiB, used=79.71GiB
> Metadata, single: total=8.00MiB, used=0.00B
> GlobalReserve, single: total=512.00MiB, used=264.28MiB
> 
> # dmesg -T
> [Thu Aug 16 18:16:31 2018] BTRFS error (device md127): parent transid verify failed on 26603622694912 wanted 185320 found 207817
> [Thu Aug 16 18:16:31 2018] BTRFS error (device md127): parent transid verify failed on 26603622694912 wanted 185320 found 207817

Transaction id mismatch, normally this means some of the fs is already
corrupted before.

And considering the transid gap, the corruption happened quite a long
time ago.

> [Thu Aug 16 18:16:31 2018] BTRFS warning (device md127): Skipping commit of aborted transaction.
> [Thu Aug 16 18:16:31 2018] BTRFS: error (device md127) in cleanup_transaction:1864: errno=-5 IO failure
> [Thu Aug 16 18:16:31 2018] BTRFS info (device md127): forced readonly
> [Thu Aug 16 18:16:31 2018] BTRFS info (device md127): delayed_refs has NO entry
> 
> I want to use btrfsck to check it,  but it will OOM :
> # btrfsck /dev/md127
> Checking filesystem on /dev/md127
> UUID: 6b87a52f-9a5f-4d03-b345-9d954c2ce259
> checking extents
> Killed
> 
> I am trying to use lowmem mode,  but as I tried before, it may OOM too,    any ideas ?

This depends on which version you tried before.

Lowmem mode used to do partial lowmem and partial normal mode check, and
normal mode caused the OOM.

Latest lowmem mode should not cause OOM, but please ger ready for the
super long run time.

Thanks,
Qu
> 
> Thanks.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: btrfsck out of memory for big fs

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2693 bytes --]



On 2018/8/17 下午1:26, litaibaichina@gmail.com wrote:
> thanks Qu.
> I am runing 4.12.
> /# btrfs --version
> btrfs-progs v4.12
> 
> Do you think btrfsck can fix the fs issue ?

Nope, transid is pretty tricky especially for super old corruption.

>  Or we better backup data and re-do the fs ? 

I'd recommend to backup data asap.

Thanks,
Qu

> 
> ______________________________________________________________________________________________________________________________________________________________________
> From: Qu Wenruo
> Date: 2018-08-17 12:34
> To: litaibaichina@gmail.com; linux-btrfs@vger.kernel.org
> Subject: Re: btrfsck out of memory for big fs
>  
>  
> On 2018/8/17 上午10:44, litaibaichina@gmail.com wrote:
>> Hi Guys,
>>
>> I had a big btrfs on a md device,  it be mounted , but after a while it will become ready only:
>> # btrfs fi df /data/
>> Data, single: total=24.46TiB, used=24.46TiB
>> System, DUP: total=8.00MiB, used=2.59MiB
>> System, single: total=4.00MiB, used=0.00B
>> Metadata, DUP: total=81.00GiB, used=79.71GiB
>> Metadata, single: total=8.00MiB, used=0.00B
>> GlobalReserve, single: total=512.00MiB, used=264.28MiB
>>
>> # dmesg -T
>> [Thu Aug 16 18:16:31 2018] BTRFS error (device md127): parent transid verify failed on 26603622694912 wanted 185320 found 207817
>> [Thu Aug 16 18:16:31 2018] BTRFS error (device md127): parent transid verify failed on 26603622694912 wanted 185320 found 207817
>  
> Transaction id mismatch, normally this means some of the fs is already
> corrupted before.
>  
> And considering the transid gap, the corruption happened quite a long
> time ago.
>  
>> [Thu Aug 16 18:16:31 2018] BTRFS warning (device md127): Skipping commit of aborted transaction.
>> [Thu Aug 16 18:16:31 2018] BTRFS: error (device md127) in cleanup_transaction:1864: errno=-5 IO failure
>> [Thu Aug 16 18:16:31 2018] BTRFS info (device md127): forced readonly
>> [Thu Aug 16 18:16:31 2018] BTRFS info (device md127): delayed_refs has NO entry
>>
>> I want to use btrfsck to check it,  but it will OOM :
>> # btrfsck /dev/md127
>> Checking filesystem on /dev/md127
>> UUID: 6b87a52f-9a5f-4d03-b345-9d954c2ce259
>> checking extents
>> Killed
>>
>> I am trying to use lowmem mode,  but as I tried before, it may OOM too,    any ideas ?
>  
> This depends on which version you tried before.
>  
> Lowmem mode used to do partial lowmem and partial normal mode check, and
> normal mode caused the OOM.
>  
> Latest lowmem mode should not cause OOM, but please ger ready for the
> super long run time.
>  
> Thanks,
> Qu
>>
>> Thanks.
>>
>  
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]