ENOSPC with 270GiB free

ENOSPC with 270GiB free

[Goswin von Brederlow]

Hi,

I'm getting a ENOSPC error from btrfs despite there still being plenty
of space left:

% df -m /mnt/nas3
Filesystem         1M-blocks     Used Available Use% Mounted on
/dev/mapper/nas3-a  19077220 18805132    270773  99% /mnt/nas3

% btrfs fi show
Label: none  uuid: 4b18f84e-2499-41ca-81ff-fe1783c11491
        Total devices 1 FS bytes used 17.91TiB
        devid    1 size 18.19TiB used 17.94TiB path /dev/mapper/nas3-a

Btrfs v3.12

% btrfs fi df
Data, single: total=17.89TiB, used=17.88TiB
System, DUP: total=32.00MiB, used=1.92MiB
Metadata, DUP: total=25.50GiB, used=24.89GiB

As you can see there are still 270GiB free and plenty of block groups
free on the device too.

So why isn't btrfs allocating a new block group to store more data?

MfG
	Goswin

Re: ENOSPC with 270GiB free

[Duncan]

Goswin von Brederlow posted on Sun, 16 Feb 2014 14:58:08 +0100 as
excerpted:

> As you can see there are still 270GiB free and plenty of block groups
> free on the device too.
> 
> So why isn't btrfs allocating a new block group to store more data?

I saw this on a much (much) smaller filesystem a few weeks ago, when I 
redid my /boot.  In my case it was under a gig total, so mixed-mode, but 
copying files over in a particular order errored some of them out with 
ENOSPC.  But the way I was copying (using mc) left the ones that hadn't 
copied selected, and I tried a copy of them again, and/or used mc's 
directory-diff to find the missing files and copy them over again.  After 
about three times, they all copied.

So some combination of size and metadata wasn't triggering a new block 
allocation, but coming in a different order, it triggered fine.  Again, 
this was mixed-mode, so data/metadata blocks mixed, and it didn't matter 
which ran out first since they were combined.

I wonder if you're running into something similar.  Can you try doing the 
copy in a different order, or is it one big file?

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: ENOSPC with 270GiB free

[Goswin von Brederlow]

On Sun, Feb 16, 2014 at 06:18:38PM +0000, Duncan wrote:
> Goswin von Brederlow posted on Sun, 16 Feb 2014 14:58:08 +0100 as
> excerpted:
> 
> > As you can see there are still 270GiB free and plenty of block groups
> > free on the device too.
> > 
> > So why isn't btrfs allocating a new block group to store more data?
> 
> I saw this on a much (much) smaller filesystem a few weeks ago, when I 
> redid my /boot.  In my case it was under a gig total, so mixed-mode, but 
> copying files over in a particular order errored some of them out with 
> ENOSPC.  But the way I was copying (using mc) left the ones that hadn't 
> copied selected, and I tried a copy of them again, and/or used mc's 
> directory-diff to find the missing files and copy them over again.  After 
> about three times, they all copied.
> 
> So some combination of size and metadata wasn't triggering a new block 
> allocation, but coming in a different order, it triggered fine.  Again, 
> this was mixed-mode, so data/metadata blocks mixed, and it didn't matter 
> which ran out first since they were combined.
> 
> I wonder if you're running into something similar.  Can you try doing the 
> copy in a different order, or is it one big file?
> 

I'm using rsync and towards the last few GB before it gives ENOSPC the
filesystem gets realy slow and eats more and more cpu time. I'm
copying multi gigabyte files and the second last file managed 2MB/s
and the failed one came down to ~100K/s at the end. So trying a lot of
different files isn't realy feasable, timewise.

MfG
	Goswin

Re: ENOSPC with 270GiB free

[Dan van der Ster]

Did you already try this?? [1]:

   btrfs fi balance start -dusage=5 /mnt/nas3

Cheers, dan

[1] https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space

On Sun, Feb 16, 2014 at 2:58 PM, Goswin von Brederlow <goswin-v-b@web.de> wrote:
> Hi,
>
> I'm getting a ENOSPC error from btrfs despite there still being plenty
> of space left:
>
> % df -m /mnt/nas3
> Filesystem         1M-blocks     Used Available Use% Mounted on
> /dev/mapper/nas3-a  19077220 18805132    270773  99% /mnt/nas3
>
> % btrfs fi show
> Label: none  uuid: 4b18f84e-2499-41ca-81ff-fe1783c11491
>         Total devices 1 FS bytes used 17.91TiB
>         devid    1 size 18.19TiB used 17.94TiB path /dev/mapper/nas3-a
>
> Btrfs v3.12
>
> % btrfs fi df
> Data, single: total=17.89TiB, used=17.88TiB
> System, DUP: total=32.00MiB, used=1.92MiB
> Metadata, DUP: total=25.50GiB, used=24.89GiB
>
> As you can see there are still 270GiB free and plenty of block groups
> free on the device too.
>
> So why isn't btrfs allocating a new block group to store more data?
>
> MfG
>         Goswin
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ENOSPC with 270GiB free

[Goswin von Brederlow]

On Mon, Feb 17, 2014 at 08:42:23AM +0100, Dan van der Ster wrote:
> Did you already try this?? [1]:
> 
>    btrfs fi balance start -dusage=5 /mnt/nas3
> 
> Cheers, dan
> 
> [1] https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
> 
> On Sun, Feb 16, 2014 at 2:58 PM, Goswin von Brederlow <goswin-v-b@web.de> wrote:
> > Hi,
> >
> > I'm getting a ENOSPC error from btrfs despite there still being plenty
> > of space left:
> >
> > % df -m /mnt/nas3
> > Filesystem         1M-blocks     Used Available Use% Mounted on
> > /dev/mapper/nas3-a  19077220 18805132    270773  99% /mnt/nas3
> >
> > % btrfs fi show
> > Label: none  uuid: 4b18f84e-2499-41ca-81ff-fe1783c11491
> >         Total devices 1 FS bytes used 17.91TiB
> >         devid    1 size 18.19TiB used 17.94TiB path /dev/mapper/nas3-a
> >
> > Btrfs v3.12
> >
> > % btrfs fi df
> > Data, single: total=17.89TiB, used=17.88TiB
> > System, DUP: total=32.00MiB, used=1.92MiB
> > Metadata, DUP: total=25.50GiB, used=24.89GiB
> >
> > As you can see there are still 270GiB free and plenty of block groups
> > free on the device too.
> >
> > So why isn't btrfs allocating a new block group to store more data?
> >
> > MfG
> >         Goswin

I did and that isn't the problem. Balancing only frees up partially
used block groups so they can be reused. But the problem is that the
remaining free block groups are not getting used.

MfG
	Goswin

Re: ENOSPC with 270GiB free

[Josef Bacik]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/16/2014 08:58 AM, Goswin von Brederlow wrote:
> Hi,
> 
> I'm getting a ENOSPC error from btrfs despite there still being
> plenty of space left:
> 
> % df -m /mnt/nas3 Filesystem         1M-blocks     Used Available
> Use% Mounted on /dev/mapper/nas3-a  19077220 18805132    270773
> 99% /mnt/nas3
> 
> % btrfs fi show Label: none  uuid:
> 4b18f84e-2499-41ca-81ff-fe1783c11491 Total devices 1 FS bytes used
> 17.91TiB devid    1 size 18.19TiB used 17.94TiB path
> /dev/mapper/nas3-a
> 
> Btrfs v3.12
> 
> % btrfs fi df Data, single: total=17.89TiB, used=17.88TiB System,
> DUP: total=32.00MiB, used=1.92MiB Metadata, DUP: total=25.50GiB,
> used=24.89GiB
> 
> As you can see there are still 270GiB free and plenty of block
> groups free on the device too.
> 
> So why isn't btrfs allocating a new block group to store more
> data?
> 
> 

What kernel?  Can you give btrfs-next a try?  Mount with -o
enospc_debug and when you get enospc send the dmesg.  Thanks,

Josef
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTA2bsAAoJEANb+wAKly3BLB4P/0W5RCk9YZQ+WJcDs8/g82Be
G5oWx+0N2eYKIttCWxrvXlx2mpVP5mWimzYqs+V7SvaVgbm3KlwuID9JQpuOQ1lH
ihnxr48kdwNa8J159oEU7oVRJ84D1b0D21MZzIKU9YoNoDHTQQsELe4aojniKMIY
4RFh/8qWjMm2BGKW4jZwKzIHoF9rt+kU7Lhfm9XOi0OCut8FlFSjSwsiaN8NFJAf
DFtkgfZTRJUxiq3dXn7m6yzspS3L5atAfd862UbFDrA7EcYIZnc+zw9/hUxX8JwP
N3espmJQ03DRn8LLoOZtBMZ+NUnRgoOd33k1z9jeB7RnqN165vSQaYibUw6Zk+Vh
qADBAy+f8aHtBF8yNIuwqEXH9D5fXlfgYUnWW/YBiw6+dbguxb4CKIzkdh67paah
zvpLrXkZd95taqXFkaj0PZydwSVkSIbrPuNllRku02Bz8XxPjE0rlW+wCr2uHI2y
Ne2jpGdCClJCfLTZC8SsHaLeYktvEVBnQ6ZOriU+Rl1hzvi+ieG7leRHIa1K7VED
669zuZ4QfYvdU7lnXzTuC36NmmaSC4wrLaYTPAytU4OLB5dIby9nyF+25t60hq5n
pb2R5tn7heH9928aVktFMTuAWUBTI7B1O1ZGB/cTcYKduYXOD9Y5ljEalA7HhUpb
PwuiepAP1uBkiBPQDYH1
=tCLT
-----END PGP SIGNATURE-----

Re: ENOSPC with 270GiB free

[Goswin von Brederlow]

On Tue, Feb 18, 2014 at 08:58:10AM -0500, Josef Bacik wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 02/16/2014 08:58 AM, Goswin von Brederlow wrote:
> > Hi,
> > 
> > I'm getting a ENOSPC error from btrfs despite there still being
> > plenty of space left:
> > 
> > % df -m /mnt/nas3 Filesystem         1M-blocks     Used Available
> > Use% Mounted on /dev/mapper/nas3-a  19077220 18805132    270773
> > 99% /mnt/nas3
> > 
> > % btrfs fi show Label: none  uuid:
> > 4b18f84e-2499-41ca-81ff-fe1783c11491 Total devices 1 FS bytes used
> > 17.91TiB devid    1 size 18.19TiB used 17.94TiB path
> > /dev/mapper/nas3-a
> > 
> > Btrfs v3.12
> > 
> > % btrfs fi df Data, single: total=17.89TiB, used=17.88TiB System,
> > DUP: total=32.00MiB, used=1.92MiB Metadata, DUP: total=25.50GiB,
> > used=24.89GiB
> > 
> > As you can see there are still 270GiB free and plenty of block
> > groups free on the device too.
> > 
> > So why isn't btrfs allocating a new block group to store more
> > data?
> > 
> > 
> 
> What kernel?  Can you give btrfs-next a try?  Mount with -o
> enospc_debug and when you get enospc send the dmesg.  Thanks,
> 
> Josef

Standard Debian linux kernel:

Linux nas3 3.12-1-amd64 #1 SMP Debian 3.12.9-1 (2014-02-01) x86_64 GNU/Linux

Compiling new kernel and rebooting will take some time. But it looks
like I can remount with enospc_debug. I will try if that outputs
anything usefull first. So far I got:

[258988.006643] btrfs: disk space caching is enabled

MfG
	Goswin