ERROR: scrubbing <dev> failed for device id 1: ret=-1, errno=28 (No space left on device)

ERROR: scrubbing <dev> failed for device id 1: ret=-1, errno=28 (No space left on device)

[Christoph Biedl]

Hello,

checking all my btr file systems after seeing a lot of trouble on a
particular installation, I came across this:

# btrfs scrub start -B -c3 /mnt/schroot/
ERROR: scrubbing /mnt/schroot/ failed for device id 1: ret=-1, errno=28 (No space left on device)
scrub canceled for 89a02165-5975-46c7-8565-1247874531a2
        scrub started at Fri Dec 18 08:20:27 2015 and was aborted after 00:00:01
        total bytes scrubbed: 7.70MiB with 0 errors

btrfsck shows a *lot* of the "bad extent [458797056, 458801152), type
mismatch with chunk" messages that have been discussed here but
otherwise exits zero.

df shows 75% usage (4.4 of 6.5 Gibyte used)).

# btrfs fi df /mnt/schroot/
Data, single: total=5.17GiB, used=3.75GiB
System, DUP: total=8.00MiB, used=4.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=409.56MiB, used=143.44MiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=48.00MiB, used=0.00B

This is btrfs-progs v4.3 running on kernel 4.3.3

Any idea how to proceed?

    Christoph

Re: ERROR: scrubbing <dev> failed for device id 1: ret=-1, errno=28 (No space left on device)

[Henk Slager]

Hello  Christoph,

you need to do some balancing, as I think that the free space is too
fragmented and the system fails to allocate extra metadata space
needed for scrub doing writes into metadata.

# btrfs balance start -dusage=<number>
with number somewhere between 5 and 50; first start with lower number
and repeat and increase until the 5.17GiB gets closer down to the
3.75GiB

Same for metadata:
# btrfs balance start -musage=<number>

or maybe scrub already succeeds by just the data balancing. You can do
it also at once:
# btrfs balance start -musage=<number> -dusage=<number>

The issue is that for for a filesystem of this size (6.5GiB), it would
have been better to have mixed data/metadata allocation. See help of
mkfs.btrfs. So maybe you want to recreate the fs like that, otherwise
you might easily hit this problem again if you don't monitor free
space / balance regularly.

/Henk

On Fri, Dec 18, 2015 at 8:28 AM, Christoph Biedl
<linux-kernel.bfrz@manchmal.in-ulm.de> wrote:
> Hello,
>
> checking all my btr file systems after seeing a lot of trouble on a
> particular installation, I came across this:
>
> # btrfs scrub start -B -c3 /mnt/schroot/
> ERROR: scrubbing /mnt/schroot/ failed for device id 1: ret=-1, errno=28 (No space left on device)
> scrub canceled for 89a02165-5975-46c7-8565-1247874531a2
>         scrub started at Fri Dec 18 08:20:27 2015 and was aborted after 00:00:01
>         total bytes scrubbed: 7.70MiB with 0 errors
>
> btrfsck shows a *lot* of the "bad extent [458797056, 458801152), type
> mismatch with chunk" messages that have been discussed here but
> otherwise exits zero.
>
> df shows 75% usage (4.4 of 6.5 Gibyte used)).
>
> # btrfs fi df /mnt/schroot/
> Data, single: total=5.17GiB, used=3.75GiB
> System, DUP: total=8.00MiB, used=4.00KiB
> System, single: total=4.00MiB, used=0.00B
> Metadata, DUP: total=409.56MiB, used=143.44MiB
> Metadata, single: total=8.00MiB, used=0.00B
> GlobalReserve, single: total=48.00MiB, used=0.00B
>
> This is btrfs-progs v4.3 running on kernel 4.3.3
>
> Any idea how to proceed?
>
>     Christoph
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ERROR: scrubbing <dev> failed for device id 1: ret=-1, errno=28 (No space left on device)

[Christoph Biedl]

Henk Slager wrote...

> you need to do some balancing, as I think that the free space is too
> fragmented and the system fails to allocate extra metadata space
> needed for scrub doing writes into metadata.
> 
> # btrfs balance start -dusage=<number>
> with number somewhere between 5 and 50; first start with lower number
> and repeat and increase until the 5.17GiB gets closer down to the
> 3.75GiB

Thanks, that one worked with number 100. Still some ENOSPC errors but
appeareantly good enough to make scrubbing work.

> The issue is that for for a filesystem of this size (6.5GiB), it would
> have been better to have mixed data/metadata allocation. See help of
> mkfs.btrfs. So maybe you want to recreate the fs like that, otherwise
> you might easily hit this problem again if you don't monitor free
> space / balance regularly.

Will keep that in mind. Just out of curiousity: Initially, the file
system size was just 4 Gibyte. Is it possible this made things worse?


Oh, and by the way:

--- a/Documentation/mkfs.btrfs.asciidoc
+++ b/Documentation/mkfs.btrfs.asciidoc
@@ -77,7 +77,7 @@ where the space is reserved for the other block group type, is not available for
 allocation and can lead to ENOSPC state.
 +
 The recommended size for the mixed mode is for filesystems less than 1GiB. The
-soft recommendation is to use it for filesystems smaller than 5GiB. Thie mixed
+soft recommendation is to use it for filesystems smaller than 5GiB. The mixed
 mode may lead to degraded performance on larger filesystems, but is otherwise
 usable, even on multiple devices.
 +


    Christoph

Re: ERROR: scrubbing <dev> failed for device id 1: ret=-1, errno=28 (No space left on device)

[Henk Slager]

On Fri, Dec 18, 2015 at 9:38 PM, Christoph Biedl
<linux-kernel.bfrz@manchmal.in-ulm.de> wrote:
> Henk Slager wrote...
>
>> you need to do some balancing, as I think that the free space is too
>> fragmented and the system fails to allocate extra metadata space
>> needed for scrub doing writes into metadata.
>>
>> # btrfs balance start -dusage=<number>
>> with number somewhere between 5 and 50; first start with lower number
>> and repeat and increase until the 5.17GiB gets closer down to the
>> 3.75GiB
>
> Thanks, that one worked with number 100. Still some ENOSPC errors but
> appeareantly good enough to make scrubbing work.
>
>> The issue is that for for a filesystem of this size (6.5GiB), it would
>> have been better to have mixed data/metadata allocation. See help of
>> mkfs.btrfs. So maybe you want to recreate the fs like that, otherwise
>> you might easily hit this problem again if you don't monitor free
>> space / balance regularly.
>
> Will keep that in mind. Just out of curiousity: Initially, the file
> system size was just 4 Gibyte. Is it possible this made things worse?
That should not matter.

> Oh, and by the way:
>
> --- a/Documentation/mkfs.btrfs.asciidoc
> +++ b/Documentation/mkfs.btrfs.asciidoc
> @@ -77,7 +77,7 @@ where the space is reserved for the other block group type, is not available for
>  allocation and can lead to ENOSPC state.
>  +
>  The recommended size for the mixed mode is for filesystems less than 1GiB. The
> -soft recommendation is to use it for filesystems smaller than 5GiB. Thie mixed
> +soft recommendation is to use it for filesystems smaller than 5GiB. The mixed
>  mode may lead to degraded performance on larger filesystems, but is otherwise
>  usable, even on multiple devices.
>  +
>
>
>     Christoph
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html