* Re: Weird space issues (with more than plenty left; over 100GiB free on 465.63GiB RAID partition)
2015-11-24 2:31 Weird space issues (with more than plenty left; over 100GiB free on 465.63GiB RAID partition) Steven Hoff
@ 2015-11-24 5:24 ` Duncan
0 siblings, 0 replies; 2+ messages in thread
From: Duncan @ 2015-11-24 5:24 UTC (permalink / raw)
To: linux-btrfs
Steven Hoff posted on Mon, 23 Nov 2015 19:31:43 -0700 as excerpted:
> BTRFS Community,
>
> I seem to be having a bit of an issue. A comment on
> https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22
No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space
> suggested reporting this to the mailing list.
>
> Issue: BTRFS claims to be running out of space
>
> Setup:
> /home: 2 500GB hard drives in MDADM RAID1 with BTRFS being the primary
> filesystem
>
> Output of various commands:
>
> $ sudo btrfs fi df
> /home Data, single: total=334.00GiB, used=333.84GiB
> System, DUP: total=32.00MiB, used=80.00KiB
> Metadata, DUP: total=1.50GiB, used=987.17MiB
> unknown, single: total=336.00MiB, used=0.00
>
> $ sudo btrfs fi show /home
> Label: none uuid: (private information removed)
> Total devices 1 FS bytes used 334.80GiB
> devid 1 size 465.63GiB used 337.06GiB path /dev/md0p1
> Btrfs v3.12
>
> $ df /home
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/md0p1 488252416 352418008 134988424 73% /home
>
>
> I would like to note that "total" in the following line:
> Data, single: total=334.00GiB, used=333.84GiB gets smaller whenever data
> is deleted or moved from /home. I'm not sure if that is by design, or if
> it is a bug. Either way, every once in a while applications will start
> closing unexpectedly complaining that there is no space left.
Critical information not provided: kernel version. However, if it's
anything as old as the 3.12 btrfs-progs userspace that's being reported,
you're running seriously ancient versions, and an upgrade to something
semi-recent is *strongly* recommended.
Normally, the kernel version is most critical, and running either a
current release series (4.3), or an LTS (long term support) version no
further back than the penultimate (next to latest, 3.18 as 4.1 is the
latest LTS, but the currently in development 4.4 has already been
announced as another LTS so people on 3.18 should be strongly considering
an upgrade to 4.1 by now).
As for btrfs userspace, while it's generally not so critical, 3.12 is
getting a bit ridiculously old now, and a good rule of thumb is to run at
least the version matching that of your kernel series, since a userspace
release of a particular version comes shortly after the kernel release,
and is being coded at the same time, so they're designed with each other
in mind. Tho running a newer userspace shouldn't be a problem, either.
Following that rule of thumb, if you're following the kernel rule as
well, you'll be running at least a 3.18 series kernel, so should be
running at least a 3.18 series userspace. That will ensure at least that
it doesn't get a ridiculously old as your current 3.12 userspace.
But you do seem to be running at least a newer kernel than your
userspace, as otherwise you'd not be getting that unknown line in btrfs
fi df. FWIW, newer userspace correctly reports that as global reserve.
And while it doesn't yet report it as such, the global reserve (again,
reported as unknown on your ancient userspace) actually comes from the
metadata number, so can be added to it. There's actually some very
recent on-list discussion on this topic, suggesting either making the
global reserve a sub-entry under metadata, or eliminating it entirely,
simply adding the global reserve size to metadata's used figure.
If you do that, the metadata line becomes 1.5 gig size, 1.3 gig used. As
metadata chunks are nominally a quarter gig but your metadata is dup
mode, so writing will be to two of them at once, you're into your last
already allocated set of metadata chunks, and similarly, data usage is
into its last data chunk (nominally 1 GiB in size).
So that means before you do much of anything, either data or metadata
will run out on its existing allocation, and new chunks should be
allocated.
Now according to btrfs fi show, you have plenty of unallocated space
(device line, 337 gig allocated, of 465 gig size, so over a hundred gig
of unallocated free space), so allocating fresh chunks shouldn't be a
problem at all.
But evidently in your case, at least some of the time, those allocations
are failing. Certainly, in the past, that was known to happen with some
versions.
But that would be kernel code, and you didn't post what kernel version
you're running.
Assuming based on your positively ancient btrfs userspace and the above
hints that your kernel version, while newer, isn't exactly current... the
first thing to do would be to get a current 4.3 series kernel, at least
for testing, and see if the problem goes away. If so and you you want to
return to an older version and you weren't already on one of them, try
the latest release of either the 3.18 or 4.1 kernel series, and see if
the bug is still gone or is there again.
Based on the results of the above, we'll know whether it's a current
kernel bug, or corrected in current but still in the LTS kernels, or only
in whatever non-current non-LTS version you are currently running.
If the bug isn't in current or either of the last two LTS kernel series,
then the solution is simple, run a still supported kernel. If it's in
the LTS series but not in current, then we know there's a bugfix in
current that needs to be backported to LTS-stable. If it's in current,
then it's an active bug, and assuming the devs aren't already working on
it, they're likely to ask you to run with the enospc_debug mount option
and to post the resulting kernel logs from when the problem triggers.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 2+ messages in thread