* Re: No space left on device (again)
2014-02-25 20:27 ` Marcus Sundman
@ 2014-02-25 20:30 ` Josef Bacik
2014-02-26 10:38 ` Sander
2014-02-27 0:16 ` Marcus Sundman
2014-02-25 20:30 ` cwillu
2014-02-25 20:40 ` Hugo Mills
2 siblings, 2 replies; 12+ messages in thread
From: Josef Bacik @ 2014-02-25 20:30 UTC (permalink / raw)
To: Marcus Sundman, linux-btrfs; +Cc: Hugo Mills, Jim Salter
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/25/2014 03:27 PM, Marcus Sundman wrote:
> On 25.02.2014 22:19, Hugo Mills wrote:
>> On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote:
>>> 370GB of 410GB used isn't really "fine", it's over 90% usage.
>>>
>>> That said, I'd be interested to know why btrfs fi show
>>> /dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G
>>> used...
>> This is an FAQ...
>>
>> btrfs fi show tells you how much is allocated out of the
>> available pool on each disk. btrfs fi df then shows how much of
>> that allocated space (in each category) is used.
>
> What is the difference between the "used 371.11GB" and the "used
> 412.54GB" displayed by "btrfs fi show"?
>
>> The problem here is also in the FAQ: the metadata is close to
>> full -- typically something like 500-750 MiB of headroom is
>> needed in metadata. The FS can't allocate more metadata because
>> it's allocated everything already (total=used in btrfs fi show),
>> so the solution is to do a filtered balance:
>>
>> btrfs balance start -dusage=5 /mountpoint
>
> Of course that was the first thing I tried, and it didn't help *at*
> *all*:
>
The -dusage=<number> is a means to an end, so if that doesn't work try
a larger number, up to 100. Really once you pass 50 and it's not
working then it's time to just do a balance. The next thing is to use
compression (too late for this option really) or add another disk.
Thanks,
Josef
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTDP1mAAoJEANb+wAKly3BQooQAMjRfSc0fJcrVFhNqAXvuxKO
f9ALYcENk7mVEMiyPg9v/7oiHW+FSQuJxaJ+yPlLTvepFmh3VE00t4PHsTCheAld
KdIoP9KzvCrmPSH5YxBg9Tmk60lonYnGfCsggK3kgRP8JIHUcBRKzlFKAc3p8vzY
UEEYLS0AYXvBAPdY3bUwhB5txMDvNbetew8ak5nUgZaC1J4rWe7LL6ccv845hhf5
OHcKLu3MjR6ejW9qDK9RC2ryjCY+vDCUMO4VPpma3OZ6OHA3g7niaXYqBEvht7O0
yNeGQA54O1goIFo9knPG4XE7WurlF69n+knQA7dsqjux9bIXHVClCCwsrUpXJIY3
98KigtPVz737j1dNqaeQdn6xl6kwRpj2lRhWuPBZ1AnJNWr1cLK2jmWOuFmyrdi8
vTNJJvUlpwGnnUHdXu8WjMCSNgXX9pIXDZIC/f22wjZFyY23yfSYOTCXTEOlpH8b
b4R7VnZ6QfxGgJnqhHPWBE7FRaekI2MLfT6aYVL2sNvVPZoMo0wzjnxshwp2+8dK
C0j2sNsoRdKPlGECYuh2eG40JQG2JKnZr2bMSwDI40GvDcQmyG7gD+jouOtei/vv
KUDDfhNGnv0SdqmAk/UC4Cl1dDD9Y4Z0kAuAd7iTJhAcLszteeGqiDvdvtup5o7i
HZcFmZX6EPBZXKrIcnzJ
=+Oe0
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: No space left on device (again)
2014-02-25 20:30 ` Josef Bacik
@ 2014-02-26 10:38 ` Sander
2014-02-27 0:16 ` Marcus Sundman
1 sibling, 0 replies; 12+ messages in thread
From: Sander @ 2014-02-26 10:38 UTC (permalink / raw)
To: Josef Bacik; +Cc: Marcus Sundman, linux-btrfs, Hugo Mills, Jim Salter
Josef Bacik wrote (ao):
> The -dusage=<number> is a means to an end, so if that doesn't work try
> a larger number, up to 100. Really once you pass 50 and it's not
> working then it's time to just do a balance. The next thing is to use
> compression (too late for this option really) or add another disk.
btrfs filesystem defragment -vrc /home/
would apply compression to all files, no?
Sander
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: No space left on device (again)
2014-02-25 20:30 ` Josef Bacik
2014-02-26 10:38 ` Sander
@ 2014-02-27 0:16 ` Marcus Sundman
2014-02-27 1:17 ` Josef Bacik
2014-02-27 7:48 ` Hugo Mills
1 sibling, 2 replies; 12+ messages in thread
From: Marcus Sundman @ 2014-02-27 0:16 UTC (permalink / raw)
To: linux-btrfs; +Cc: Josef Bacik, Jim Salter, Hugo Mills
On 25.02.2014 22:30, Josef Bacik wrote:
> On 02/25/2014 03:27 PM, Marcus Sundman wrote:
>> On 25.02.2014 22:19, Hugo Mills wrote:
>>> On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote:
>>>> 370GB of 410GB used isn't really "fine", it's over 90% usage.
>>>>
>>>> That said, I'd be interested to know why btrfs fi show
>>>> /dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G
>>>> used...
>>> This is an FAQ...
>>>
>>> btrfs fi show tells you how much is allocated out of the
>>> available pool on each disk. btrfs fi df then shows how much of
>>> that allocated space (in each category) is used.
>> What is the difference between the "used 371.11GB" and the "used
>> 412.54GB" displayed by "btrfs fi show"?
>>
>>> The problem here is also in the FAQ: the metadata is close to
>>> full -- typically something like 500-750 MiB of headroom is
>>> needed in metadata. The FS can't allocate more metadata because
>>> it's allocated everything already (total=used in btrfs fi show),
>>> so the solution is to do a filtered balance:
>>>
>>> btrfs balance start -dusage=5 /mountpoint
>> Of course that was the first thing I tried, and it didn't help *at*
>> *all*:
>>
> The -dusage=<number> is a means to an end, so if that doesn't work try
> a larger number, up to 100. Really once you pass 50 and it's not
> working then it's time to just do a balance. The next thing is to use
> compression (too late for this option really) or add another disk.
So it relocates some chunks. What will that do? Does it mean I can now
use the remaining 45 GB? Or will it run out of "disk space" again after
using a gig or two?
If it's the allocated metadata space that is the problem then how can I
pre-allocate more of it so it won't run out of it?
- Marcus
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: No space left on device (again)
2014-02-27 0:16 ` Marcus Sundman
@ 2014-02-27 1:17 ` Josef Bacik
2014-02-27 7:48 ` Hugo Mills
1 sibling, 0 replies; 12+ messages in thread
From: Josef Bacik @ 2014-02-27 1:17 UTC (permalink / raw)
To: Marcus Sundman, linux-btrfs; +Cc: Jim Salter, Hugo Mills
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/26/2014 07:16 PM, Marcus Sundman wrote:
> On 25.02.2014 22:30, Josef Bacik wrote:
>> On 02/25/2014 03:27 PM, Marcus Sundman wrote:
>>> On 25.02.2014 22:19, Hugo Mills wrote:
>>>> On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote:
>>>>> 370GB of 410GB used isn't really "fine", it's over 90%
>>>>> usage.
>>>>>
>>>>> That said, I'd be interested to know why btrfs fi show
>>>>> /dev/sda3 shows 412.54G used, but btrfs fi df /home shows
>>>>> 379G used...
>>>> This is an FAQ...
>>>>
>>>> btrfs fi show tells you how much is allocated out of the
>>>> available pool on each disk. btrfs fi df then shows how much
>>>> of that allocated space (in each category) is used.
>>> What is the difference between the "used 371.11GB" and the
>>> "used 412.54GB" displayed by "btrfs fi show"?
>>>
>>>> The problem here is also in the FAQ: the metadata is close
>>>> to full -- typically something like 500-750 MiB of headroom
>>>> is needed in metadata. The FS can't allocate more metadata
>>>> because it's allocated everything already (total=used in
>>>> btrfs fi show), so the solution is to do a filtered balance:
>>>>
>>>> btrfs balance start -dusage=5 /mountpoint
>>> Of course that was the first thing I tried, and it didn't help
>>> *at* *all*:
>>>
>> The -dusage=<number> is a means to an end, so if that doesn't
>> work try a larger number, up to 100. Really once you pass 50 and
>> it's not working then it's time to just do a balance. The next
>> thing is to use compression (too late for this option really) or
>> add another disk.
>
> So it relocates some chunks. What will that do? Does it mean I can
> now use the remaining 45 GB? Or will it run out of "disk space"
> again after using a gig or two? If it's the allocated metadata
> space that is the problem then how can I pre-allocate more of it so
> it won't run out of it?
>
>
It will allocate more as it sees fit, there is a metadata_ratio=N
option which lets you force it to allocate a metadata chunk for every
N data chunk allocations, but that shouldn't be needed in this case.
What does btrfs fi df and btrfs show look like now? Thanks,
Josef
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTDpIMAAoJEANb+wAKly3BfKUQAK/WzvL/SENlUhruUoL7ADid
tM6ivq2BZOwmlw+CzOIzjrwv9gwRe+UAfjKjGstueLoVY+lHNYaCC5OI94OOWDPo
SyLJdJhcA3FYYBlyz7EC6gG4fNFyJg6egshoyMmWJuny5ivfnltdQhLLz2IQIHek
Ud9ElN+SMvCeQmGZxdKg5yc2oIgBdc5xGfegtfuCqFkdhu+BZTbcXFsOD4Pjnsg0
Mtw8H5YBeXcFBV34I8F6l+O3AGjDl8jF/cFuCEbRTJQFkvpKhoHMw+O2AWykDQqc
xw3H1YghGeY1fqN2geyYSYVGVOGxWeO2kju7Itom8Ph5AinMPMf8lpe97nChp2hr
D3T1QwhkmMD9T1O02hvF9C8E46q2iyjOrPxPU8z3LsRTKaNXNzJ+u1P2ac7kIQSk
x2stB9u0Qluut+5twzLQuefzoCNf/2RtAlL2cPXyq6ikLHBSNkebqbGBD2IpTsa7
5TsXNHWbIc0maDdXrJ0BYJ7obEaU/2nBCkSA3DaGBupTtC3vwSWAoAEw8/JtVeQr
ARXSIPblXZqwrGyJyvtNmC8zjDGAD93H/xr0+oCOzHqPIr7NvTT9vZKfENmblsfm
vx8wmgbSBzWp9W9YpUO5X0ZjLestPzOziIesCrMJ2yKBUEUtafiFdbT5D7e9dn8x
7Z2MZcYyk5H7QF65+IWt
=8iSO
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: No space left on device (again)
2014-02-27 0:16 ` Marcus Sundman
2014-02-27 1:17 ` Josef Bacik
@ 2014-02-27 7:48 ` Hugo Mills
1 sibling, 0 replies; 12+ messages in thread
From: Hugo Mills @ 2014-02-27 7:48 UTC (permalink / raw)
To: Marcus Sundman; +Cc: linux-btrfs, Josef Bacik, Jim Salter
[-- Attachment #1: Type: text/plain, Size: 3055 bytes --]
On Thu, Feb 27, 2014 at 02:16:07AM +0200, Marcus Sundman wrote:
> On 25.02.2014 22:30, Josef Bacik wrote:
> >On 02/25/2014 03:27 PM, Marcus Sundman wrote:
> >>On 25.02.2014 22:19, Hugo Mills wrote:
> >>>On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote:
> >>>>370GB of 410GB used isn't really "fine", it's over 90% usage.
> >>>>
> >>>>That said, I'd be interested to know why btrfs fi show
> >>>>/dev/sda3 shows 412.54G used, but btrfs fi df /home shows 379G
> >>>>used...
> >>>This is an FAQ...
> >>>
> >>>btrfs fi show tells you how much is allocated out of the
> >>>available pool on each disk. btrfs fi df then shows how much of
> >>>that allocated space (in each category) is used.
> >>What is the difference between the "used 371.11GB" and the "used
> >>412.54GB" displayed by "btrfs fi show"?
> >>
> >>>The problem here is also in the FAQ: the metadata is close to
> >>>full -- typically something like 500-750 MiB of headroom is
> >>>needed in metadata. The FS can't allocate more metadata because
> >>>it's allocated everything already (total=used in btrfs fi show),
> >>>so the solution is to do a filtered balance:
> >>>
> >>>btrfs balance start -dusage=5 /mountpoint
> >>Of course that was the first thing I tried, and it didn't help *at*
> >>*all*:
> >>
> >The -dusage=<number> is a means to an end, so if that doesn't work try
> >a larger number, up to 100. Really once you pass 50 and it's not
> >working then it's time to just do a balance. The next thing is to use
> >compression (too late for this option really) or add another disk.
>
> So it relocates some chunks. What will that do? Does it mean I can
> now use the remaining 45 GB? Or will it run out of "disk space"
> again after using a gig or two?
What it's done is moved some of the data around. One effect of this
process is that when a balance operation balances a chunk, it
deallocates that space. Therefore you should now see that the "used"
value for the device in btrfs fi show is lower than the "total" value.
Also, the "total" value for data in btrfs fi df should now be lower
than it was before.
This means that the next time the FS needs some metadata space
(probably real soon now, because it was running out of it earlier), it
now has uncommitted space to allocate to metadata.
> If it's the allocated metadata space that is the problem then how
> can I pre-allocate more of it so it won't run out of it?
It's possible that when filling up the remaining 45 GiB of space
the metadata will need to expand again, and you will need to run a
balance to do the same job again. Unless you have lots and lots of
small files or small fragments (or lots more snapshots than you've
been making to date), I'd say it's probably unlikely, though.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Well, sir, the floor is yours. But remember, the ---
roof is ours!
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: No space left on device (again)
2014-02-25 20:27 ` Marcus Sundman
2014-02-25 20:30 ` Josef Bacik
@ 2014-02-25 20:30 ` cwillu
2014-02-25 20:40 ` Hugo Mills
2 siblings, 0 replies; 12+ messages in thread
From: cwillu @ 2014-02-25 20:30 UTC (permalink / raw)
To: Marcus Sundman; +Cc: linux-btrfs, Hugo Mills, Jim Salter
Try btrfs filesystem balance start -dusage=15 /home, and gradually
increase it until you see it relocate at least one chunk.
On Tue, Feb 25, 2014 at 2:27 PM, Marcus Sundman <sundman@iki.fi> wrote:
> On 25.02.2014 22:19, Hugo Mills wrote:
>>
>> On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote:
>>>
>>> 370GB of 410GB used isn't really "fine", it's over 90% usage.
>>>
>>> That said, I'd be interested to know why btrfs fi show /dev/sda3
>>> shows 412.54G used, but btrfs fi df /home shows 379G used...
>>
>> This is an FAQ...
>>
>> btrfs fi show tells you how much is allocated out of the available
>> pool on each disk. btrfs fi df then shows how much of that allocated
>> space (in each category) is used.
>
>
> What is the difference between the "used 371.11GB" and the "used 412.54GB"
> displayed by "btrfs fi show"?
>
>
>> The problem here is also in the FAQ: the metadata is close to full
>> -- typically something like 500-750 MiB of headroom is needed in
>> metadata. The FS can't allocate more metadata because it's allocated
>> everything already (total=used in btrfs fi show), so the solution is
>> to do a filtered balance:
>>
>> btrfs balance start -dusage=5 /mountpoint
>
>
> Of course that was the first thing I tried, and it didn't help *at* *all*:
>
>> # btrfs filesystem balance start -dusage=5 /home
>> Done, had to relocate 0 out of 415 chunks
>> #
>
>
> ... and it really didn't free anything.
>
>
>>> On 02/25/2014 11:49 AM, Marcus Sundman wrote:
>>>>
>>>> Hi
>>>>
>>>> I get "No space left on device" and it is unclear why:
>>>>
>>>>> # df -h|grep sda3
>>>>> /dev/sda3 413G 368G 45G 90% /home
>>>>> # btrfs filesystem show /dev/sda3
>>>>> Label: 'home' uuid: 46279061-51f4-40c2-afd0-61d6faab7f60
>>>>> Total devices 1 FS bytes used 371.11GB
>>>>> devid 1 size 412.54GB used 412.54GB path /dev/sda3
>>>>>
>>>>> Btrfs v0.20-rc1
>>>>> # btrfs filesystem df /home
>>>>> Data: total=410.52GB, used=369.61GB
>>>>> System: total=4.00MB, used=64.00KB
>>>>> Metadata: total=2.01GB, used=1.50GB
>>>>> #
>>>>
>>>> So, 'data' and 'metadata' seem to be fine(?), but 'system' is a
>>>> bit low. Is that it? If so, can I do something about it? Or should
>>>> I look somewhere else?
>>>>
>>>> I really wish I could get a warning before running out of disk
>>>> space, instead of everything breaking suddenly when there seems to
>>>> be lots and lots of space left.
>>>>
>>>> - Marcus
>>>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: No space left on device (again)
2014-02-25 20:27 ` Marcus Sundman
2014-02-25 20:30 ` Josef Bacik
2014-02-25 20:30 ` cwillu
@ 2014-02-25 20:40 ` Hugo Mills
2 siblings, 0 replies; 12+ messages in thread
From: Hugo Mills @ 2014-02-25 20:40 UTC (permalink / raw)
To: Marcus Sundman; +Cc: linux-btrfs, Jim Salter
[-- Attachment #1: Type: text/plain, Size: 3284 bytes --]
On Tue, Feb 25, 2014 at 10:27:58PM +0200, Marcus Sundman wrote:
> On 25.02.2014 22:19, Hugo Mills wrote:
> >On Tue, Feb 25, 2014 at 01:05:51PM -0500, Jim Salter wrote:
> >>370GB of 410GB used isn't really "fine", it's over 90% usage.
> >>
> >>That said, I'd be interested to know why btrfs fi show /dev/sda3
> >>shows 412.54G used, but btrfs fi df /home shows 379G used...
> > This is an FAQ...
> >
> > btrfs fi show tells you how much is allocated out of the available
> >pool on each disk. btrfs fi df then shows how much of that allocated
> >space (in each category) is used.
>
> What is the difference between the "used 371.11GB" and the "used
> 412.54GB" displayed by "btrfs fi show"?
The first and smaller value is (in this case) the sum of all the
"used" values in the output of btrfs fi df. The second is the total
amount of raw disk space that has been allocated. (Yes, this is a bit
of a mess -- there's been discussions recently about improving the
output).
> > The problem here is also in the FAQ: the metadata is close to full
> >-- typically something like 500-750 MiB of headroom is needed in
> >metadata. The FS can't allocate more metadata because it's allocated
> >everything already (total=used in btrfs fi show), so the solution is
> >to do a filtered balance:
> >
> >btrfs balance start -dusage=5 /mountpoint
>
> Of course that was the first thing I tried, and it didn't help *at* *all*:
>
> ># btrfs filesystem balance start -dusage=5 /home
> >Done, had to relocate 0 out of 415 chunks
> >#
>
> ... and it really didn't free anything.
OK, so crank up the number to 10 and try again. If that still moves
nothing, then increase again until it frees up something. With an FS
at 90% full or so, particularly if it's seen heavy use, the chances
are reasonable that all of the chunks are over 5% full, so putting the
value up will eventually get to the point of moving at least one
chunk, which should be all that's needed at this point.
Hugo.
> >>On 02/25/2014 11:49 AM, Marcus Sundman wrote:
> >>>Hi
> >>>
> >>>I get "No space left on device" and it is unclear why:
> >>>
> >>>># df -h|grep sda3
> >>>>/dev/sda3 413G 368G 45G 90% /home
> >>>># btrfs filesystem show /dev/sda3
> >>>>Label: 'home' uuid: 46279061-51f4-40c2-afd0-61d6faab7f60
> >>>> Total devices 1 FS bytes used 371.11GB
> >>>> devid 1 size 412.54GB used 412.54GB path /dev/sda3
> >>>>
> >>>>Btrfs v0.20-rc1
> >>>># btrfs filesystem df /home
> >>>>Data: total=410.52GB, used=369.61GB
> >>>>System: total=4.00MB, used=64.00KB
> >>>>Metadata: total=2.01GB, used=1.50GB
> >>>>#
> >>>So, 'data' and 'metadata' seem to be fine(?), but 'system' is a
> >>>bit low. Is that it? If so, can I do something about it? Or should
> >>>I look somewhere else?
> >>>
> >>>I really wish I could get a warning before running out of disk
> >>>space, instead of everything breaking suddenly when there seems to
> >>>be lots and lots of space left.
> >>>
> >>>- Marcus
> >>>
>
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Jazz is the sort of music where no-one plays anything the ---
same way once.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread