Re: why am I getting "No space left on device" here?

Duncan
2014-01-15T19:06:18Z

Tomasz Chmielewski posted on Wed, 15 Jan 2014 11:55:43 +0100 as excerpted:

> I'm no longer able to write to this btrfs filesystem:
>
> # df -h /home
> Filesystem Size Used Avail Use% Mounted on
> /dev/sdb4 5.2T 3.6T 1.6T 71% /home

FWIW, standard df doesn't really know how to work with btrfs' advanced
layout yet, so its output is, let's say "less than ideal", on btrfs,
particularly on the various btrfs multi-device configurations.

btrfs fi show and btrfs fi df, combined, form the usable replacement on
btrfs. Fortunately you listed their output as well...

> # btrfs fi show /home Label: crawler-btrfs uuid:
> 60f1759c-45f6-4484-9f60-66a4e9bbf2b6
> Total devices 2 FS bytes used 1.80TiB
> devid 3 size 2.56TiB used 1.80TiB path /dev/sdb4
> devid 4 size 2.56TiB used 1.80TiB path /dev/sda4
>
> Btrfs v3.12

Looks pretty reasonable and well balanced, as a raid1 should be. =:^)

Only 1.80 TiB of 2.56 TiB on each device allocated, so there's plenty
of room left to allocate additional chunks as needed. =:^)

> # btrfs filesystem df /home
> Data, RAID1: total=1.75TiB, used=1.75TiB
> System, RAID1: total=32.00MiB, used=268.00KiB
> Metadata, RAID1: total=53.00GiB, used=51.71GiB

Data chunks are full, 1.75 TiB of 1.75 TiB, so it'll need to allocate a
new data chunk pretty quickly when you start copying. (raid1 mode, so
it'd allocate chunks in pairs on the two devices). FWIW, data chunks are
1 GiB each.

Metadata chunks, 51.71 GiB used of 53.00 GiB. 1.25+ GiB free. Metadata
chunks are a quarter GiB (256 MiB) each, so that's several chunks worth,
free.

> However:
>
> # dd if=/dev/urandom of=bigfile
> dd: writing to `bigfile': No space left on device
> 186+0 records in
> 185+0 records out
> 94720 bytes (95 kB) copied, 0.0144045 s, 6.6 MB/s
>
>
> I don't understand why - can anyone explain?

Well, there's two levels of explanation here, but unfortunately they
don't fully cover it. Still, here's what's available:

At the first level, as hinted above in the df comments, btrfs' space
calculation is MUCH more complex than that of a normal filesystem.

First, unlike a normal filesystem, btrfs data and metadata are treated
separately, and they're very unlikely to run out together, so one or the
other will be out while the other has room left. Then there's the fact
that metadata is dup by default, while data is single, so metadata by
default takes up twice the space it normally would. (Plus of course
btrfs has checksums and even small partial-block file-tail data in its
metadata, in addition to it all being dupped, so there's a lot MORE
metadata to deal with on btrfs, than on a normal filesystem.)

In ordered to deal with that, btrfs sets up the empty filesystem as a big
reserve pool of potential chunks that can be allocated to data or
metadata as needed, so there's the whole already allocated vs. still
unallocated and free to allocate thing, as displayed by btrfs fi show,
that other filesystems don't normally deal with. Meanwhile, btrfs fi df
displays, separately for each of data, metadata and system chunks, how
much of the already allocated space is actually used. You can see my
comments on your output above.

Then there's the whole multi-device thing and the various raid modes that
btrfs has, that simply don't apply to normal filesystems. Both data/
metadata as raid1 with two devices is actually rather simple, since one
copy goes to each device. Actually, that's even simpler than the default
single-device case, since a single device defaults to dup metadata,
single data, which is harder to figure out than a two-device raid1's
simple one copy to each device rule. But a two-device raid1 is the
simple case!

Then there's the fact that eventually, the plan is to allow different
subvolumes to be configurable with different raid levels, so it could
well be that you'd have raid1, raid10, raid6, raid5, and single, all on
the same filesystem!

No *WONDER* df doesn't know how to report all this! Actually, they're
already working on making df better for the simple all-one-type cases at
least, but I doubt it'll ever be "good" at reporting for btrfs in the
complex cases, since it's simply too simple a tool for that job.

This is actually covered on the btrfs wiki in the FAQ as well, altho I
think I covered it more thoroughly above. But they'll give you some
hints for dealing with the problem as well, and I'd definitely recommend
spending some time reading the wiki in any case, since there's certainly
more there that you're likely to find very useful as an admin running
btrfs on your systems.

FAQ (space-related, see 1.3 and 1.4, and 4.4 thru 4.10)
https://btrfs.wiki.kernel.org/index.php/FAQ

General btrfs wiki link (bookmark it! =:^)
https://btrfs.wiki.kernel.org

The space-related FAQ entries should cover the theory, and give you some
hints for fixing the problem as well, but there appears to be more going
on in your case, as you have _PLENTY_ of unallocated space remaining so
allocating more shouldn't be a problem. And I had a similar issue
recently as well -- plenty of space left (tho in my case it was on a
small mixed-mode filesystem).

That's the second level which I alluded to, where the FAQ and the answers
above don't really cover things.

In my case, I was copying over a bunch of files at once. Actually, I had
just done a fresh mkfs.btrfs on the /boot on one of my two ssds (with the
other one still bootable in case something went wrong while I was setting
up the new /boot, of course), and was trying to install grub2's modules
and config files to it once again. As I said, that's a small (sub 1-gig
so mixed-mode instead of separate data/metadata) filesystem, and the
files in question were pretty small, too

But what I found here, was that while some files copied just fine, others
failed. HOWEVER, I was using mc (aka midnight commander), and I used its
directory diff feature to figure out what had copied and what hadn't,
which left all the uncopied files in the source selected, so I could try
copying them once again.

And the weird thing was, while the original copy errored out due with a
no-space error, when I tried again to copy the files that hadn't copied,
more of them copied over without error! By doing this a couple times, I
was able to get everything copied over.

What happened was this. When the error occurred, while I had unallocated
space left as shown in btrfs fi show, btrfs fi df showed nearly full
usage. (Again, with a sub-gig filesystem, btrfs uses mixed-mode by
default, so data/metadata combine, so it was just the single mixed-type
chunk that was about full, not separate data/metadata. And a balance as
suggested in the FAQ... didn't help, and I mount with
compress=lzo,autodefrag already, and it was a fresh filesystem, so...)

But by trying the file copies again for just the files that had been
missed the first time, the order was different, and something, somehow,
triggered a new chunk allocation, that for whatever reason, btrfs had
failed to allocate when it should have, the first time around.

Which brings us back to your case. While I was dealing with a small sub-
gig filesystem and thus mixed-mode, you're dealing with a large
filesystem and separate data/metadata chunks.

But just as my already allocated mixed-mode chunks were just about full
and I needed another one allocated to complete the job, so your data
chunks are full or very close, according to btrfs fi df, and you need a
new one allocated (and if the file is greater than a gig in size, likely
more than one) to finish the job.

And in both our cases, there's plenty of unallocated space in the pool,
but for whatever reason, btrfs isn't allocating that new chunk when it
should! Why, I can't say, but as I mentioned, I was able to work around
the problem here by trying the remaining files in a different order, and
at some point, btrfs figured out it needed that new chunk allocated, and
everything went fine after that.

So... why btrfs is failing to allocate a new chunk when it needs to I
can't say, but I *CAN* say you're not the only one to have run into the
problem recently; I did too.

And just as I did, here, with a bit of monkeying around, you can
/probably/ get btrfs to allocate that new data chunk and get on with
things. But the trouble is, since I don't know what the exact problem is
or what exactly I did to persuade btrfs to do that new chunk allocation,
I can't tell you exactly what to do to get it to happen, all I can do is
suggest you try copying smaller files or files of different sizes around
a bit, hoping to trigger that allocation.

Once that chunk allocation happens, you should be good for at least a
gig, since that's the data-chunk size, but if your file is over a gig in
size, you may run into the problem again. In that case... Well, you
could try copying several gigs of smaller files, then once it allocates
what you need, delete them, leaving the data chunks allocated but with
enough unused space to copy the original multi-gig file over.

But there's certainly some sort of chunk allocation bug involved here,
since there was for me and is for you certainly unallocated space
available to allocate new chunks, and from the btrfs fi df, we can see
that the existing chunks are full and a new chunk SHOULD be allocated,
but isn't being allocated, thus the bug. It'll probably be fixed in
time, but meanwhile, try monkeying around a bit with other file sizes to
hopefully work around the issue.

HTH =:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman