Meta data full on a Readynas

Meta data full on a Readynas

[Mike Aubury]

Hi,
Wondering if anyone can help me.
I've got a readynas which uses btrfs - and the meta data is full.

root@ReadyNAS:~# uname -a
Linux ReadyNAS 3.0.101.RN_ARM.4 #1 Mon Jun 29 19:02:35 PDT 2015 armv7l GNU/Linux
root@ReadyNAS:~#   btrfs --version
Btrfs v3.17.3
root@ReadyNAS:~#   btrfs fi show
Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
        Total devices 3 FS bytes used 5.88TiB
        devid    1 size 2.71TiB used 2.71TiB path /dev/md127
        devid    2 size 1.82TiB used 1.82TiB path /dev/md126
        devid    3 size 1.82TiB used 1.82TiB path /dev/md125

Btrfs v3.17.3
root@ReadyNAS:~#   btrfs fi df /data
Data, single: total=6.33TiB, used=5.87TiB
System, RAID1: total=32.00MiB, used=768.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, RAID1: total=9.00GiB, used=8.98GiB
Metadata, DUP: total=3.00GiB, used=2.12GiB


At some point (I dont know if its btrfs, or some Readynas process) -
the filesystem goes readyonly.
This seems to scupper most attempts at fixing things.

I've tried all sorts,I've tried deleting off files - but - when the
machine reboots they re-appear, various different "btrfs balance" type
commands - they seem to exit because of lack of free space - eg

# btrfs  balance start /data -dlimit=3 -v
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x20): balancing, limit=3
ERROR: error during balancing '/data' - No space left on device
There may be more info in syslog - try dmesg | tail
# dmesg | tail
BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
BTRFS info (device md125): forced readonly
BTRFS debug (device md125): run_one_delayed_ref returned -28
BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
No space left
BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
BTRFS debug (device md125): run_one_delayed_ref returned -28
BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
No space left
BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
BTRFS debug (device md125): run_one_delayed_ref returned -28
BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
No space left



Nothing seems to fix it, every time I reboot - I get the same 8.98GB
used out of 9GB

I tried the readynas forum - the advice seemed to be wipe the device
and start again...


Any suggestion before I just wipe the /data and start again ?

Re: Meta data full on a Readynas

[Chris Murphy]

On Thu, Aug 27, 2015 at 4:02 AM, Mike Aubury <mike@aubit.com> wrote:
> Hi,
> Wondering if anyone can help me.
> I've got a readynas which uses btrfs - and the meta data is full.
>
> root@ReadyNAS:~# uname -a
> Linux ReadyNAS 3.0.101.RN_ARM.4 #1 Mon Jun 29 19:02:35 PDT 2015 armv7l GNU/Linux
> root@ReadyNAS:~#   btrfs --version
> Btrfs v3.17.3
> root@ReadyNAS:~#   btrfs fi show
> Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
>         Total devices 3 FS bytes used 5.88TiB
>         devid    1 size 2.71TiB used 2.71TiB path /dev/md127
>         devid    2 size 1.82TiB used 1.82TiB path /dev/md126
>         devid    3 size 1.82TiB used 1.82TiB path /dev/md125
>
> Btrfs v3.17.3
> root@ReadyNAS:~#   btrfs fi df /data
> Data, single: total=6.33TiB, used=5.87TiB
> System, RAID1: total=32.00MiB, used=768.00KiB
> System, single: total=4.00MiB, used=0.00B
> Metadata, RAID1: total=9.00GiB, used=8.98GiB
> Metadata, DUP: total=3.00GiB, used=2.12GiB
>
>
> At some point (I dont know if its btrfs, or some Readynas process) -
> the filesystem goes readyonly.
> This seems to scupper most attempts at fixing things.
>
> I've tried all sorts,I've tried deleting off files - but - when the
> machine reboots they re-appear, various different "btrfs balance" type
> commands - they seem to exit because of lack of free space - eg
>
> # btrfs  balance start /data -dlimit=3 -v
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x20): balancing, limit=3
> ERROR: error during balancing '/data' - No space left on device
> There may be more info in syslog - try dmesg | tail
> # dmesg | tail
> BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
> BTRFS info (device md125): forced readonly
> BTRFS debug (device md125): run_one_delayed_ref returned -28
> BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
> No space left
> BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
> BTRFS debug (device md125): run_one_delayed_ref returned -28
> BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
> No space left
> BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
> BTRFS debug (device md125): run_one_delayed_ref returned -28
> BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
> No space left
>
>
>
> Nothing seems to fix it, every time I reboot - I get the same 8.98GB
> used out of 9GB
>
> I tried the readynas forum - the advice seemed to be wipe the device
> and start again...
>
>
> Any suggestion before I just wipe the /data and start again ?

First umount then remount, because it needs to be rw first. If you
can't get it to mount rw, then it's stuck.

Next, connect a USB stick that's already erased (wipefs -a on each
partition and whole block device), then add the whole block device.
Ideally it's small, around 2-8GB, you can partition it if it's a large
USB stick.

btrfs dev add /dev/<usbstick> /data
btrfs balance start /data
btrfs dev delete /dev/<usbstick> /data

You could use filters, such as -dusage=0 or -dusage=15 to make it go a
bit faster. Realize that some fs metadata will end up on the USB
stick, so don't remove it until the dev delete completes and fi show
lists that it's not part of the volume anymore.

-- 
Chris Murphy

Re: Meta data full on a Readynas

[Mike Aubury]

Thanks for that.
I found something similar online - using a loop back device and a file
on a usb stick.
I created a 2GB file, and mounted that tried the balance - and once
completed - rebooted.
It seems to have fried the whole thing - won't mount at all now - with
or without the usb device in place.

Currently running a "recover" to try to pull off what I can..




On 27 August 2015 at 17:23, Chris Murphy <lists@colorremedies.com> wrote:
> On Thu, Aug 27, 2015 at 4:02 AM, Mike Aubury <mike@aubit.com> wrote:
>> Hi,
>> Wondering if anyone can help me.
>> I've got a readynas which uses btrfs - and the meta data is full.
>>
>> root@ReadyNAS:~# uname -a
>> Linux ReadyNAS 3.0.101.RN_ARM.4 #1 Mon Jun 29 19:02:35 PDT 2015 armv7l GNU/Linux
>> root@ReadyNAS:~#   btrfs --version
>> Btrfs v3.17.3
>> root@ReadyNAS:~#   btrfs fi show
>> Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
>>         Total devices 3 FS bytes used 5.88TiB
>>         devid    1 size 2.71TiB used 2.71TiB path /dev/md127
>>         devid    2 size 1.82TiB used 1.82TiB path /dev/md126
>>         devid    3 size 1.82TiB used 1.82TiB path /dev/md125
>>
>> Btrfs v3.17.3
>> root@ReadyNAS:~#   btrfs fi df /data
>> Data, single: total=6.33TiB, used=5.87TiB
>> System, RAID1: total=32.00MiB, used=768.00KiB
>> System, single: total=4.00MiB, used=0.00B
>> Metadata, RAID1: total=9.00GiB, used=8.98GiB
>> Metadata, DUP: total=3.00GiB, used=2.12GiB
>>
>>
>> At some point (I dont know if its btrfs, or some Readynas process) -
>> the filesystem goes readyonly.
>> This seems to scupper most attempts at fixing things.
>>
>> I've tried all sorts,I've tried deleting off files - but - when the
>> machine reboots they re-appear, various different "btrfs balance" type
>> commands - they seem to exit because of lack of free space - eg
>>
>> # btrfs  balance start /data -dlimit=3 -v
>> Dumping filters: flags 0x1, state 0x0, force is off
>>   DATA (flags 0x20): balancing, limit=3
>> ERROR: error during balancing '/data' - No space left on device
>> There may be more info in syslog - try dmesg | tail
>> # dmesg | tail
>> BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
>> BTRFS info (device md125): forced readonly
>> BTRFS debug (device md125): run_one_delayed_ref returned -28
>> BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
>> No space left
>> BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
>> BTRFS debug (device md125): run_one_delayed_ref returned -28
>> BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
>> No space left
>> BTRFS error (device md125) in __btrfs_free_extent:5648: errno=-28 No space left
>> BTRFS debug (device md125): run_one_delayed_ref returned -28
>> BTRFS error (device md125) in btrfs_run_delayed_refs:2688: errno=-28
>> No space left
>>
>>
>>
>> Nothing seems to fix it, every time I reboot - I get the same 8.98GB
>> used out of 9GB
>>
>> I tried the readynas forum - the advice seemed to be wipe the device
>> and start again...
>>
>>
>> Any suggestion before I just wipe the /data and start again ?
>
> First umount then remount, because it needs to be rw first. If you
> can't get it to mount rw, then it's stuck.
>
> Next, connect a USB stick that's already erased (wipefs -a on each
> partition and whole block device), then add the whole block device.
> Ideally it's small, around 2-8GB, you can partition it if it's a large
> USB stick.
>
> btrfs dev add /dev/<usbstick> /data
> btrfs balance start /data
> btrfs dev delete /dev/<usbstick> /data
>
> You could use filters, such as -dusage=0 or -dusage=15 to make it go a
> bit faster. Realize that some fs metadata will end up on the USB
> stick, so don't remove it until the dev delete completes and fi show
> lists that it's not part of the volume anymore.
>
> --
> Chris Murphy

Re: Meta data full on a Readynas

[Chris Murphy]

On Thu, Aug 27, 2015 at 10:35 AM, Mike Aubury <mike@aubit.com> wrote:
> Thanks for that.
> I found something similar online - using a loop back device and a file
> on a usb stick.
> I created a 2GB file, and mounted that tried the balance - and once
> completed - rebooted.

OK but did you delete the loop device before rebooting? If not, then
that's why. You need to reactivate that loop device from the file to
mount the fs normally. It's not good enough to just balance. You have
to use btrfs dev del on the extra device so the fs moves everything
off that soon to be removed device.

>>> root@ReadyNAS:~#   btrfs fi df /data
>>> Data, single: total=6.33TiB, used=5.87TiB
>>> System, RAID1: total=32.00MiB, used=768.00KiB
>>> System, single: total=4.00MiB, used=0.00B
>>> Metadata, RAID1: total=9.00GiB, used=8.98GiB
>>> Metadata, DUP: total=3.00GiB, used=2.12GiB

I just noticed this, and it's confusing. All data is single copy. Some
of the metadata is DUP on *one* device rather than RAID1. So it's
entirely possible that some of the DUP metadata, without conversion,
ends up on the loop device and if that loop device isn't first deleted
before the reboot, then you have a broken file system because that
necessary fs metadata isn't available anywhere. The metadata DUP is
kinda dangerous in multiple device cases.



-- 
Chris Murphy

Re: Meta data full on a Readynas

[Chris Murphy]

On Thu, Aug 27, 2015 at 11:18 AM, Mike Aubury <mike@aubit.com> wrote:
> Offlist - thanks for the help so far..

No, I'm changing it back to onlist. It's not appropriate to take this offlist.

This whole conversation is relevant to others who end up in this same
situation. And the ReadyNAS folks should have better support for this,
you paid them money for this product, and it was their choice to use
Btrfs. They should support you. But in the meantime the Btrfs list is
the right place for the discussion.


>
>
> The loop device is still there
>
> # losetup  -a
> /dev/loop0: [0861]:198 (/media/USB_HDD_1/t1/tmpbtrfs)
>
> But doesnt seem to be being picked up :
>
> # btrfs fi show
> warning devid 4 not found already
> Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
>         Total devices 4 FS bytes used 6.09TiB
>         devid    1 size 2.71TiB used 2.71TiB path /dev/md127
>         devid    2 size 1.82TiB used 1.82TiB path /dev/md126
>         devid    3 size 1.82TiB used 1.82TiB path /dev/md125
>         *** Some devices missing


Try

# btrfs dev scan


> Trying a mount in that state gives :
>
> # mount /data
> mount: wrong fs type, bad option, bad superblock on /dev/md126,
>        missing codepage or helper program, or other error
>        In some cases useful info is found in syslog - try
>        dmesg | tail  or so
>
> dmesg gives :
> btrfs: failed to read chunk tree on md125
> btrfs: open_ctree failed

If dev scan doesn't work you could try to mount with -o ro,degraded
but I do not recommend it until you fist explore all options to make
btrfs aware of this loop back device.

In the future I think it's better to use the USB stick directly rather
than loopback files which just adds more complexity to an already
fragile situation.


> FWIW - When I used the usb file - it never had any "usage" reported in
> the "fi show" - used was always 0GB, I think the thing is just totally
> messed up, and all i've done is mess it up more!

There is at least some metadata on it, or it wouldn't be get fussy
that the device is missing. So you need to get the loop back device
visible to Btrfs and then it will almost certainly mount at least ro
at which point you can get data off of it first; and then you can deal
with the btrfs dev delete part.





-- 
Chris Murphy

Re: Meta data full on a Readynas

[Mike Aubury]

#  btrfs dev scan
Scanning for Btrfs filesystems

(back to # prompt)


# mount -o ro,degraded /data


Seems to work, /data is mounted

# btrfs fi show
Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
        Total devices 4 FS bytes used 6.09TiB
        devid    1 size 2.71TiB used 2.71TiB path /dev/md127
        devid    2 size 1.82TiB used 1.82TiB path /dev/md126
        devid    3 size 1.82TiB used 1.82TiB path /dev/md125
        *** Some devices missing


Where next ?


On 27 August 2015 at 18:27, Chris Murphy <lists@colorremedies.com> wrote:
> On Thu, Aug 27, 2015 at 11:18 AM, Mike Aubury <mike@aubit.com> wrote:
>> Offlist - thanks for the help so far..
>
> No, I'm changing it back to onlist. It's not appropriate to take this offlist.
>
> This whole conversation is relevant to others who end up in this same
> situation. And the ReadyNAS folks should have better support for this,
> you paid them money for this product, and it was their choice to use
> Btrfs. They should support you. But in the meantime the Btrfs list is
> the right place for the discussion.
>
>
>>
>>
>> The loop device is still there
>>
>> # losetup  -a
>> /dev/loop0: [0861]:198 (/media/USB_HDD_1/t1/tmpbtrfs)
>>
>> But doesnt seem to be being picked up :
>>
>> # btrfs fi show
>> warning devid 4 not found already
>> Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
>>         Total devices 4 FS bytes used 6.09TiB
>>         devid    1 size 2.71TiB used 2.71TiB path /dev/md127
>>         devid    2 size 1.82TiB used 1.82TiB path /dev/md126
>>         devid    3 size 1.82TiB used 1.82TiB path /dev/md125
>>         *** Some devices missing
>
>
> Try
>
> # btrfs dev scan
>
>
>> Trying a mount in that state gives :
>>
>> # mount /data
>> mount: wrong fs type, bad option, bad superblock on /dev/md126,
>>        missing codepage or helper program, or other error
>>        In some cases useful info is found in syslog - try
>>        dmesg | tail  or so
>>
>> dmesg gives :
>> btrfs: failed to read chunk tree on md125
>> btrfs: open_ctree failed
>
> If dev scan doesn't work you could try to mount with -o ro,degraded
> but I do not recommend it until you fist explore all options to make
> btrfs aware of this loop back device.
>
> In the future I think it's better to use the USB stick directly rather
> than loopback files which just adds more complexity to an already
> fragile situation.
>
>
>> FWIW - When I used the usb file - it never had any "usage" reported in
>> the "fi show" - used was always 0GB, I think the thing is just totally
>> messed up, and all i've done is mess it up more!
>
> There is at least some metadata on it, or it wouldn't be get fussy
> that the device is missing. So you need to get the loop back device
> visible to Btrfs and then it will almost certainly mount at least ro
> at which point you can get data off of it first; and then you can deal
> with the btrfs dev delete part.
>
>
>
>
>
> --
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Meta data full on a Readynas

[Chris Murphy]

On Thu, Aug 27, 2015 at 11:33 AM, Mike Aubury <mike@aubit.com> wrote:
> #  btrfs dev scan
> Scanning for Btrfs filesystems
>
> (back to # prompt)
>
>
> # mount -o ro,degraded /data
>
>
> Seems to work, /data is mounted
>
> # btrfs fi show
> Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
>         Total devices 4 FS bytes used 6.09TiB
>         devid    1 size 2.71TiB used 2.71TiB path /dev/md127
>         devid    2 size 1.82TiB used 1.82TiB path /dev/md126
>         devid    3 size 1.82TiB used 1.82TiB path /dev/md125
>         *** Some devices missing
>
>
> Where next ?
>
>
> On 27 August 2015 at 18:27, Chris Murphy <lists@colorremedies.com> wrote:
>> On Thu, Aug 27, 2015 at 11:18 AM, Mike Aubury <mike@aubit.com> wrote:
>>> Offlist - thanks for the help so far..
>>
>> No, I'm changing it back to onlist. It's not appropriate to take this offlist.
>>
>> This whole conversation is relevant to others who end up in this same
>> situation. And the ReadyNAS folks should have better support for this,
>> you paid them money for this product, and it was their choice to use
>> Btrfs. They should support you. But in the meantime the Btrfs list is
>> the right place for the discussion.
>>
>>
>>>
>>>
>>> The loop device is still there
>>>
>>> # losetup  -a
>>> /dev/loop0: [0861]:198 (/media/USB_HDD_1/t1/tmpbtrfs)
>>>
>>> But doesnt seem to be being picked up :
>>>
>>> # btrfs fi show
>>> warning devid 4 not found already
>>> Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
>>>         Total devices 4 FS bytes used 6.09TiB
>>>         devid    1 size 2.71TiB used 2.71TiB path /dev/md127
>>>         devid    2 size 1.82TiB used 1.82TiB path /dev/md126
>>>         devid    3 size 1.82TiB used 1.82TiB path /dev/md125
>>>         *** Some devices missing
>>
>>
>> Try
>>
>> # btrfs dev scan
>>
>>
>>> Trying a mount in that state gives :
>>>
>>> # mount /data
>>> mount: wrong fs type, bad option, bad superblock on /dev/md126,
>>>        missing codepage or helper program, or other error
>>>        In some cases useful info is found in syslog - try
>>>        dmesg | tail  or so
>>>
>>> dmesg gives :
>>> btrfs: failed to read chunk tree on md125
>>> btrfs: open_ctree failed
>>
>> If dev scan doesn't work you could try to mount with -o ro,degraded
>> but I do not recommend it until you fist explore all options to make
>> btrfs aware of this loop back device.
>>
>> In the future I think it's better to use the USB stick directly rather
>> than loopback files which just adds more complexity to an already
>> fragile situation.
>>
>>
>>> FWIW - When I used the usb file - it never had any "usage" reported in
>>> the "fi show" - used was always 0GB, I think the thing is just totally
>>> messed up, and all i've done is mess it up more!
>>
>> There is at least some metadata on it, or it wouldn't be get fussy
>> that the device is missing. So you need to get the loop back device
>> visible to Btrfs and then it will almost certainly mount at least ro
>> at which point you can get data off of it first; and then you can deal
>> with the btrfs dev delete part.
>>
>>
>>
>>
>>
>> --
>> Chris Murphy
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Chris Murphy

Re: Meta data full on a Readynas

[Chris Murphy]

On Thu, Aug 27, 2015 at 11:41 AM, Chris Murphy <lists@colorremedies.com> wrote:
> On Thu, Aug 27, 2015 at 11:33 AM, Mike Aubury <mike@aubit.com> wrote:
>> #  btrfs dev scan
>> Scanning for Btrfs filesystems
>>
>> (back to # prompt)
>>
>>
>> # mount -o ro,degraded /data
>>
>>
>> Seems to work, /data is mounted
>>
>> # btrfs fi show
>> Label: '0e36d290:data'  uuid: e75a9856-d9e5-4d02-92e6-a217056c69b7
>>         Total devices 4 FS bytes used 6.09TiB
>>         devid    1 size 2.71TiB used 2.71TiB path /dev/md127
>>         devid    2 size 1.82TiB used 1.82TiB path /dev/md126
>>         devid    3 size 1.82TiB used 1.82TiB path /dev/md125
>>         *** Some devices missing
>>
>>
>> Where next ?

Well now at least you can suck the data off the volume...

If you can't figure out why the loop back mount device continues to be
missing, you might as well just take advantage of the fact you've got
it ro mounted and get everything off the volume in preparation to wipe
it and start over.



-- 
Chris Murphy