* Re: Transaction aborted error -28 clone_finish_inode_update
2018-10-04 16:44 Transaction aborted error -28 clone_finish_inode_update David Goodwin
@ 2018-10-05 10:18 ` Duncan
0 siblings, 0 replies; 2+ messages in thread
From: Duncan @ 2018-10-05 10:18 UTC (permalink / raw)
To: linux-btrfs
David Goodwin posted on Thu, 04 Oct 2018 17:44:46 +0100 as excerpted:
> While trying to run/use bedup ( https://github.com/g2p/bedup ) .... I
> hit this :
>
>
> [Thu Oct 4 15:34:51 2018] ------------[ cut here ]------------
> [Thu Oct 4 15:34:51 2018] BTRFS: Transaction aborted (error -28)
> [Thu Oct 4 15:34:51 2018] WARNING: CPU: 0 PID: 28832 at
> fs/btrfs/ioctl.c:3671 clone_finish_inode_update+0xf3/0x140
> [Thu Oct 4 15:34:51 2018] CPU: 0 PID: 28832 Comm: bedup Not tainted
> 4.18.10-psi-dg1 #1
[snipping a bunch of stuff that I as a non-dev list regular can't do much
with anyway]
> [Thu Oct 4 15:34:51 2018] BTRFS: error (device xvdg) in
> clone_finish_inode_update:3671: errno=-28 No space left
> [Thu Oct 4 15:34:51 2018] BTRFS info (device xvdg): forced readonly
> % btrfs fi us /filesystem/
> Overall:
> Device size: 7.12TiB
> Device allocated: 6.80TiB
> Device unallocated: 330.93GiB
> Device missing: 0.00B
> Used: 6.51TiB
> Free (estimated): 629.87GiB (min: 629.87GiB)
> Data ratio: 1.00
> Metadata ratio: 1.00
> Global reserve: 512.00MiB (used: 0.00B)
>
> Data+Metadata,single: Size:6.80TiB, Used:6.51TiB
> /dev/xvdf 1.69TiB
> /dev/xvdg 3.12TiB
> /dev/xvdi 1.99TiB
>
> System,single: Size:32.00MiB, Used:780.00KiB
> /dev/xvdf 32.00MiB
>
> Unallocated:
> /dev/xvdf 320.97GiB
> /dev/xvdg 949.00MiB
> /dev/xvdi 9.03GiB
>
>
> I kind of think there is sufficient free space..... at least globally
> within the filesystem.
>
> Does it require balancing to redistribute the unallocated space better?
> Or is something misbehaving?
The latter, but unfortunately there's not much you can do about it at
this point but wait for fixes, unless you want to spit up that huge
filesystem into several smaller ones.
In general, btrfs has at least four kinds of "space" that it can run out
of, tho in your case it appears you're running mixed-mode so data and
metadata space are combined into one.
* Unallocated space: This is space that remains entirely unallocated in
the filesystem. It matters most when the balance between data and
metadata space gets off.
This isn't a problem for you as in single mode space can be allocated
from any device and you have one with hundreds of gigs unallocated. It
also tends to be less of a problem on mixed-bg mode, which you're
running, as there's no distinction in mixed-mode between data and
metadata.
* Data chunk space:
* Metadata chunk space:
Because you're running mixed-bg mode, there's no distinction between
these two, but for normal mode, running out of one or the other while all
the free space is allocated to chunks of the other type, can be a problem.
* Global reserve: Taken from metadata, the global reserve is space the
system won't normally use, that it tries to keep clear in ordered to be
able to finish transactions once they're started, as btrfs' copy-on-write
semantics means even deleting stuff requires a bit of additional space
temporarily.
This seems to actually be where the problem is, because currently,
certain btrfs operations such as reflinking/cloning/snapshotting (that
is, just what you were doing) don't really calculate the needed space
correctly and use arbitrary figures, which can be *wildly* off, while
conversely a bare half-gig of global-reserve for a huge 7+ TiB filesystem
seems rather proportionally small. (Consider that my small pair-device
btrfs raid1 root filesystem, 8-GiB/device, 16 GiB total, has a 16 MiB
reserve, proportionally, your 7+ TB filesystem would have 7+ GiB reserve,
but it only has a half GiB.)
So relatively small btrfs' don't tend to run into the problem, because
they have proportionally larger reserves to begin with. Plus they
probably don't have proportionally as many snapshots/reflinks/etc,
either, so the problem simply doesn't trigger for them.
Now I'm not a dev and my own use-case doesn't include either snapshotting
or deduping, so I haven't paid that much attention to the specifics, but
I have seen some recent patches on-list that based on the explanations
should go some way toward fixing this problem by using more realistic
figures for global-reserve calculations. At this point those patches
would be for 4.20 (which might be 5.0), or possibly 4.21, but the devs
are indeed working on the problem and it should get better within a
couple kernel cycles.
Alternatively perhaps the global reserve size could be bumped up on such
large filesystems, but let's see if the more realistic operations-reserve
calculations can fix things, first, as arguably that shouldn't be
necessary once the calculations aren't so arbitrarily wild.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 2+ messages in thread