* Issue: errno:28 (No space left on device)
@ 2016-08-09 9:06 Thomas
2016-08-09 16:50 ` Chris Murphy
0 siblings, 1 reply; 9+ messages in thread
From: Thomas @ 2016-08-09 9:06 UTC (permalink / raw)
To: linux-btrfs
Hello!
I'm facing a severe issue with Debian installation using BTRFS:
errno:28 (No space left on device)
After a few minutes of system usage I get this error message for any action, means there's really no disk space left.
However, the output of btrfs fi usage / reports that ~10% of disk space should be free.
I have 2 questions:
1. Why is the OS reporting an error "Issue: errno:28 (No space left on device)" whereas "btrfs fi usage" reports free space?
2. How can I solve this issue w/o the possibility of extending the partition?
THX
---------------------------------------------------------------------------------------
thomas@pc8-nb:~$ uname -a
Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1 (2016-06-04) x86_64 GNU/Linux
thomas@pc8-nb:~$ sudo btrfs --version
btrfs-progs v4.5.2
thomas@pc8-nb:~$ sudo btrfs fi usage /
Overall:
Device size: 20.53GiB
Device allocated: 20.53GiB
Device unallocated: 0.00B
Device missing: 0.00B
Used: 18.25GiB
Free (estimated): 1.96GiB (min: 1.96GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 160.00MiB (used: 400.00KiB)
Data,single: Size:19.76GiB, Used:17.81GiB
/dev/sda8 19.76GiB
Metadata,single: Size:776.00MiB, Used:455.73MiB
/dev/sda8 776.00MiB
System,single: Size:4.00MiB, Used:16.00KiB
/dev/sda8 4.00MiB
Unallocated:
/dev/sda8 0.00B
thomas@pc8-nb:~$ sudo btrfs fi show
Label: 'siduction' uuid: dbb9433e-a84f-4e84-9200-49350b31a3f6
Total devices 1 FS bytes used 18.25GiB
devid 1 size 20.53GiB used 20.53GiB path /dev/sda8
thomas@pc8-nb:~$ sudo btrfs su li /
ID 257 gen 5498245 top level 5 path @
ID 258 gen 5498245 top level 5 path @home
ID 259 gen 5488356 top level 5 path @tmp
ID 260 gen 2029641 top level 5 path @var_lib_lxc
ID 261 gen 5480763 top level 5 path @var_log
ID 262 gen 5467204 top level 5 path @var_spool
ID 263 gen 5467205 top level 5 path @var_tmp
ID 290 gen 5464299 top level 257 path .snapshots
ID 291 gen 2026169 top level 258 path @home/.snapshots
ID 293 gen 2029641 top level 260 path @var_lib_lxc/.snapshots
ID 355 gen 163972 top level 290 path .snapshots/17/snapshot
ID 356 gen 167205 top level 290 path .snapshots/18/snapshot
ID 357 gen 5383634 top level 5 path @var_cache_apt_archives
ID 358 gen 5464163 top level 290 path .snapshots/19/snapshot
thomas@pc8-nb:~$ sudo journalctl | grep space
[...]
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: [Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: [Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: [Errno 28] No space left on device
[...]
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://ftp.fau.de/debian/dists/stable/main/Contents-i386 Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/contrib/binary-i386/Packages.gz Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/contrib/i18n/Translation-en Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-amd64 Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-i386 Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-amd64/Packages.gz Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-i386/Packages.gz Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/non-free/i18n/Translation-en Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://mirror.unitedcolo.de/debian/dists/stable/non-free/Contents-amd64 Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://ftp.fau.de/debian/dists/stable/non-free/Contents-i386 Write error - write (28: No space left on device)
[...]
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Issue: errno:28 (No space left on device)
2016-08-09 9:06 Issue: errno:28 (No space left on device) Thomas
@ 2016-08-09 16:50 ` Chris Murphy
0 siblings, 0 replies; 9+ messages in thread
From: Chris Murphy @ 2016-08-09 16:50 UTC (permalink / raw)
To: Thomas; +Cc: Btrfs BTRFS
On Tue, Aug 9, 2016 at 3:06 AM, Thomas <c.monty@web.de> wrote:
> thomas@pc8-nb:~$ uname -a
> Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1
> (2016-06-04) x86_64 GNU/Linux
Try 4.5.7 or 4.7 and see if it fixes the problem, there's a related
patch in both of those that's not yet in 4.6 series. The patch mainly
permits using existing chunks for balance to consolidate extents,
rather than requiring new chunks. So you might also have to do a
filtered balance. I would probably start with -dusage=25 and see where
that gets you.
> Device size: 20.53GiB
> Device allocated: 20.53GiB
>From Btrfs perspective the volume is "fully allocated" and as such
cannot create new data or metadata chunks. Whatever workload you're
doing at the time of enospc may be filling up data or metadata unused
space in existing chunks, triggering need for newly allocated chunk,
which can't happen because there's no space left for new chunks.
> Data,single: Size:19.76GiB, Used:17.81GiB
> /dev/sda8 19.76GiB
>
> Metadata,single: Size:776.00MiB, Used:455.73MiB
> /dev/sda8 776.00MiB
I'll guess there's some metadata centric workload, maybe with small
files that are being stored inline, that's filling up the metadata
chunks, leading to enospc.
If changing kernel versions doesn't work, then you'll need to add more
space somehow. The file system is ~90% full anyway.
--
Chris Murphy
^ permalink raw reply [flat|nested] 9+ messages in thread
* Issue: errno:28 (No space left on device)
@ 2016-08-09 9:09 Thomas
0 siblings, 0 replies; 9+ messages in thread
From: Thomas @ 2016-08-09 9:09 UTC (permalink / raw)
To: linux-btrfs
Hello!
I'm facing a severe issue with Debian installation using BTRFS:
errno:28 (No space left on device)
After a few minutes of system usage I get this error message for any
action, means there's really no disk space left.
However, the output of btrfs fi usage / reports that ~10% of disk space
should be free.
I have 2 questions:
1. Why is the OS reporting an error "Issue: errno:28 (No space left on
device)" whereas "btrfs fi usage" reports free space?
2. How can I solve this issue w/o the possibility of extending the
partition?
THX
---------------------------------------------------------------------------------------
thomas@pc8-nb:~$ uname -a
Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1
(2016-06-04) x86_64 GNU/Linux
thomas@pc8-nb:~$ sudo btrfs --version
btrfs-progs v4.5.2
thomas@pc8-nb:~$ sudo btrfs fi usage /
Overall:
Device size: 20.53GiB
Device allocated: 20.53GiB
Device unallocated: 0.00B
Device missing: 0.00B
Used: 18.25GiB
Free (estimated): 1.96GiB (min: 1.96GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 160.00MiB (used: 400.00KiB)
Data,single: Size:19.76GiB, Used:17.81GiB
/dev/sda8 19.76GiB
Metadata,single: Size:776.00MiB, Used:455.73MiB
/dev/sda8 776.00MiB
System,single: Size:4.00MiB, Used:16.00KiB
/dev/sda8 4.00MiB
Unallocated:
/dev/sda8 0.00B
thomas@pc8-nb:~$ sudo btrfs fi show
Label: 'siduction' uuid: dbb9433e-a84f-4e84-9200-49350b31a3f6
Total devices 1 FS bytes used 18.25GiB
devid 1 size 20.53GiB used 20.53GiB path /dev/sda8
thomas@pc8-nb:~$ sudo btrfs su li /
ID 257 gen 5498245 top level 5 path @
ID 258 gen 5498245 top level 5 path @home
ID 259 gen 5488356 top level 5 path @tmp
ID 260 gen 2029641 top level 5 path @var_lib_lxc
ID 261 gen 5480763 top level 5 path @var_log
ID 262 gen 5467204 top level 5 path @var_spool
ID 263 gen 5467205 top level 5 path @var_tmp
ID 290 gen 5464299 top level 257 path .snapshots
ID 291 gen 2026169 top level 258 path @home/.snapshots
ID 293 gen 2029641 top level 260 path @var_lib_lxc/.snapshots
ID 355 gen 163972 top level 290 path .snapshots/17/snapshot
ID 356 gen 167205 top level 290 path .snapshots/18/snapshot
ID 357 gen 5383634 top level 5 path @var_cache_apt_archives
ID 358 gen 5464163 top level 290 path .snapshots/19/snapshot
thomas@pc8-nb:~$ sudo journalctl | grep space
[...]
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
[Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
[Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
[Errno 28] No space left on device
[...]
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://ftp.fau.de/debian/dists/stable/main/Contents-i386 Write error -
write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/contrib/binary-i386/Packages.gz
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/contrib/i18n/Translation-en
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-amd64
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-i386
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-amd64/Packages.gz
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-i386/Packages.gz
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/non-free/i18n/Translation-en
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://mirror.unitedcolo.de/debian/dists/stable/non-free/Contents-amd64
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://ftp.fau.de/debian/dists/stable/non-free/Contents-i386 Write
error - write (28: No space left on device)
[...]
^ permalink raw reply [flat|nested] 9+ messages in thread* Issue: errno:28 (No space left on device)
@ 2016-08-09 11:50 Thomas
2016-08-09 12:21 ` Tomasz Torcz
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Thomas @ 2016-08-09 11:50 UTC (permalink / raw)
To: linux-btrfs
Hello!
I'm facing a severe issue with Debian installation using BTRFS:
errno:28 (No space left on device)
After a few minutes of system usage I get this error message for any
action, means there's really no disk space left.
However, the output of btrfs fi usage / reports that ~10% of disk space
should be free.
I have 2 questions:
1. Why is the OS reporting an error "Issue: errno:28 (No space left on
device)" whereas "btrfs fi usage" reports free space?
2. How can I solve this issue w/o the possibility of extending the
partition?
THX
---------------------------------------------------------------------------------------
thomas@pc8-nb:~$ uname -a
Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1
(2016-06-04) x86_64 GNU/Linux
thomas@pc8-nb:~$ sudo btrfs --version
btrfs-progs v4.5.2
thomas@pc8-nb:~$ sudo btrfs fi usage /
Overall:
Device size: 20.53GiB
Device allocated: 20.53GiB
Device unallocated: 0.00B
Device missing: 0.00B
Used: 18.25GiB
Free (estimated): 1.96GiB (min: 1.96GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 160.00MiB (used: 400.00KiB)
Data,single: Size:19.76GiB, Used:17.81GiB
/dev/sda8 19.76GiB
Metadata,single: Size:776.00MiB, Used:455.73MiB
/dev/sda8 776.00MiB
System,single: Size:4.00MiB, Used:16.00KiB
/dev/sda8 4.00MiB
Unallocated:
/dev/sda8 0.00B
thomas@pc8-nb:~$ sudo btrfs fi show
Label: 'siduction' uuid: dbb9433e-a84f-4e84-9200-49350b31a3f6
Total devices 1 FS bytes used 18.25GiB
devid 1 size 20.53GiB used 20.53GiB path /dev/sda8
thomas@pc8-nb:~$ sudo btrfs su li /
ID 257 gen 5498245 top level 5 path @
ID 258 gen 5498245 top level 5 path @home
ID 259 gen 5488356 top level 5 path @tmp
ID 260 gen 2029641 top level 5 path @var_lib_lxc
ID 261 gen 5480763 top level 5 path @var_log
ID 262 gen 5467204 top level 5 path @var_spool
ID 263 gen 5467205 top level 5 path @var_tmp
ID 290 gen 5464299 top level 257 path .snapshots
ID 291 gen 2026169 top level 258 path @home/.snapshots
ID 293 gen 2029641 top level 260 path @var_lib_lxc/.snapshots
ID 355 gen 163972 top level 290 path .snapshots/17/snapshot
ID 356 gen 167205 top level 290 path .snapshots/18/snapshot
ID 357 gen 5383634 top level 5 path @var_cache_apt_archives
ID 358 gen 5464163 top level 290 path .snapshots/19/snapshot
thomas@pc8-nb:~$ sudo journalctl | grep space
[...]
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
[Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
[Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
[Errno 28] No space left on device
[...]
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://ftp.fau.de/debian/dists/stable/main/Contents-i386 Write error -
write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/contrib/binary-i386/Packages.gz
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/contrib/i18n/Translation-en
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-amd64
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-i386
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-amd64/Packages.gz
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-i386/Packages.gz
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://debian.mirror.lrz.de/debian/dists/stable/non-free/i18n/Translation-en
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://mirror.unitedcolo.de/debian/dists/stable/non-free/Contents-amd64
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
http://ftp.fau.de/debian/dists/stable/non-free/Contents-i386 Write
error - write (28: No space left on device)
[...]
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Issue: errno:28 (No space left on device)
2016-08-09 11:50 Thomas
@ 2016-08-09 12:21 ` Tomasz Torcz
2016-08-09 12:56 ` Austin S. Hemmelgarn
2016-08-09 13:47 ` Josef Bacik
2 siblings, 0 replies; 9+ messages in thread
From: Tomasz Torcz @ 2016-08-09 12:21 UTC (permalink / raw)
To: linux-btrfs
On Tue, Aug 09, 2016 at 01:50:04PM +0200, Thomas wrote:
> Hello!
>
> I'm facing a severe issue with Debian installation using BTRFS:
> errno:28 (No space left on device)
Sending this message 3x won't make it more important.
--
Tomasz Torcz ,,(...) today's high-end is tomorrow's embedded processor.''
xmpp: zdzichubg@chrome.pl -- Mitchell Blank on LKML
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Issue: errno:28 (No space left on device)
2016-08-09 11:50 Thomas
2016-08-09 12:21 ` Tomasz Torcz
@ 2016-08-09 12:56 ` Austin S. Hemmelgarn
2016-08-09 13:39 ` Noah Massey
2016-08-09 13:47 ` Josef Bacik
2 siblings, 1 reply; 9+ messages in thread
From: Austin S. Hemmelgarn @ 2016-08-09 12:56 UTC (permalink / raw)
To: Thomas, linux-btrfs
On 2016-08-09 07:50, Thomas wrote:
> Hello!
First things first:
Mailing lists are asynchronous. You will almost _never_ get an
immediate response, and will quite often not get a response for a few
hours at least. Sending a message more than once when you don't get a
response does not make it more likely you get a response quickly, and in
fact is a good way to seriously annoy people and make them less willing
to help you.
>
> I'm facing a severe issue with Debian installation using BTRFS:
> errno:28 (No space left on device)
>
> After a few minutes of system usage I get this error message for any
> action, means there's really no disk space left.
>
> However, the output of btrfs fi usage / reports that ~10% of disk space
> should be free.
>
> I have 2 questions:
> 1. Why is the OS reporting an error "Issue: errno:28 (No space left on
> device)" whereas "btrfs fi usage" reports free space?
BTRFS allocates disk space in two stages. In the first stage, it
allocates large chunks that then get used only for metadata or data.
Within those chunks, it then allocates blocks for the given type of data
on-demand. In your case, the filesystem has no room to allocate new
chunks, but has some free space in existing chunks. This shouldn't
result in allocation failures for new files, but for currently unknown
reasons it sometimes does, and this is a known issue that we have been
unable to completely fix so far (we've technically 'fixed' this about 5
or 6 times already, but there appear to be lots of odd corner cases that
haven't been found yet).
> 2. How can I solve this issue w/o the possibility of extending the
> partition?
Right now, you have three options:
1. Extend the partition, run a full balance, and then shrink the
partition again.
2. Temporarily add another device (make sure it's at least 4GB), run a
full balance, and then remove the device.
3. Recreate the filesystem from scratch and restore from a backup.
Regardless of which option you choose, you either need to free up some
disk space, or get a bigger storage device, as running within a few GB
of a full filesystem is a known trigger for this issue. Depending on
how frequently you snapshot the system, just deleting some old snapshots
may help. Making sure to clean out the package cache after running
updates will probably help too.
>
>
> THX
>
> ---------------------------------------------------------------------------------------
>
>
> thomas@pc8-nb:~$ uname -a
> Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1
> (2016-06-04) x86_64 GNU/Linux
>
>
> thomas@pc8-nb:~$ sudo btrfs --version
> btrfs-progs v4.5.2
>
>
> thomas@pc8-nb:~$ sudo btrfs fi usage /
> Overall:
> Device size: 20.53GiB
> Device allocated: 20.53GiB
> Device unallocated: 0.00B
> Device missing: 0.00B
> Used: 18.25GiB
> Free (estimated): 1.96GiB (min: 1.96GiB)
> Data ratio: 1.00
> Metadata ratio: 1.00
> Global reserve: 160.00MiB (used: 400.00KiB)
>
> Data,single: Size:19.76GiB, Used:17.81GiB
> /dev/sda8 19.76GiB
>
> Metadata,single: Size:776.00MiB, Used:455.73MiB
> /dev/sda8 776.00MiB
>
> System,single: Size:4.00MiB, Used:16.00KiB
> /dev/sda8 4.00MiB
>
> Unallocated:
> /dev/sda8 0.00B
>
>
> thomas@pc8-nb:~$ sudo btrfs fi show
> Label: 'siduction' uuid: dbb9433e-a84f-4e84-9200-49350b31a3f6
> Total devices 1 FS bytes used 18.25GiB
> devid 1 size 20.53GiB used 20.53GiB path /dev/sda8
>
>
> thomas@pc8-nb:~$ sudo btrfs su li /
> ID 257 gen 5498245 top level 5 path @
> ID 258 gen 5498245 top level 5 path @home
> ID 259 gen 5488356 top level 5 path @tmp
> ID 260 gen 2029641 top level 5 path @var_lib_lxc
> ID 261 gen 5480763 top level 5 path @var_log
> ID 262 gen 5467204 top level 5 path @var_spool
> ID 263 gen 5467205 top level 5 path @var_tmp
> ID 290 gen 5464299 top level 257 path .snapshots
> ID 291 gen 2026169 top level 258 path @home/.snapshots
> ID 293 gen 2029641 top level 260 path @var_lib_lxc/.snapshots
> ID 355 gen 163972 top level 290 path .snapshots/17/snapshot
> ID 356 gen 167205 top level 290 path .snapshots/18/snapshot
> ID 357 gen 5383634 top level 5 path @var_cache_apt_archives
> ID 358 gen 5464163 top level 290 path .snapshots/19/snapshot
>
>
> thomas@pc8-nb:~$ sudo journalctl | grep space
> [...]
> Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
> [Errno 28] No space left on device
> Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
> [Errno 28] No space left on device
> Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
> [Errno 28] No space left on device
> [...]
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://ftp.fau.de/debian/dists/stable/main/Contents-i386 Write error -
> write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/contrib/binary-i386/Packages.gz
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/contrib/i18n/Translation-en
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-amd64
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-i386
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-amd64/Packages.gz
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-i386/Packages.gz
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/non-free/i18n/Translation-en
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://mirror.unitedcolo.de/debian/dists/stable/non-free/Contents-amd64
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://ftp.fau.de/debian/dists/stable/non-free/Contents-i386 Write
> error - write (28: No space left on device)
> [...]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Issue: errno:28 (No space left on device)
2016-08-09 12:56 ` Austin S. Hemmelgarn
@ 2016-08-09 13:39 ` Noah Massey
2016-08-09 13:56 ` Roman Mamedov
0 siblings, 1 reply; 9+ messages in thread
From: Noah Massey @ 2016-08-09 13:39 UTC (permalink / raw)
To: Austin S. Hemmelgarn; +Cc: Thomas, linux-btrfs
On Tue, Aug 9, 2016 at 8:56 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2016-08-09 07:50, Thomas wrote:
>>
>> Hello!
>
> First things first:
> Mailing lists are asynchronous. You will almost _never_ get an immediate
> response, and will quite often not get a response for a few hours at least.
> Sending a message more than once when you don't get a response does not make
> it more likely you get a response quickly, and in fact is a good way to
> seriously annoy people and make them less willing to help you.
>>
>>
>> I'm facing a severe issue with Debian installation using BTRFS:
>> errno:28 (No space left on device)
>>
>> After a few minutes of system usage I get this error message for any
>> action, means there's really no disk space left.
>>
>> However, the output of btrfs fi usage / reports that ~10% of disk space
>> should be free.
>>
>> I have 2 questions:
>> 1. Why is the OS reporting an error "Issue: errno:28 (No space left on
>> device)" whereas "btrfs fi usage" reports free space?
>
> BTRFS allocates disk space in two stages. In the first stage, it allocates
> large chunks that then get used only for metadata or data. Within those
> chunks, it then allocates blocks for the given type of data on-demand. In
> your case, the filesystem has no room to allocate new chunks, but has some
> free space in existing chunks. This shouldn't result in allocation failures
> for new files, but for currently unknown reasons it sometimes does, and this
> is a known issue that we have been unable to completely fix so far (we've
> technically 'fixed' this about 5 or 6 times already, but there appear to be
> lots of odd corner cases that haven't been found yet).
>>
>> 2. How can I solve this issue w/o the possibility of extending the
>> partition?
>
> Right now, you have three options:
> 1. Extend the partition, run a full balance, and then shrink the partition
> again.
> 2. Temporarily add another device (make sure it's at least 4GB), run a full
> balance, and then remove the device.
> 3. Recreate the filesystem from scratch and restore from a backup.
>
A quick note recreating the filesystem:
Since your data and metadata profiles are both 'single', you may want
to consider creating the filesystem with the --mixed option, which
allows / forces BTRFS to use a single collection of data blocks for
both data and metadata.
This can help with ENOMEM errors when all space is allocated, but the
allocation does not match the currently required type.
~ Noah
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Issue: errno:28 (No space left on device)
2016-08-09 13:39 ` Noah Massey
@ 2016-08-09 13:56 ` Roman Mamedov
0 siblings, 0 replies; 9+ messages in thread
From: Roman Mamedov @ 2016-08-09 13:56 UTC (permalink / raw)
To: Noah Massey; +Cc: Austin S. Hemmelgarn, Thomas, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 896 bytes --]
On Tue, 9 Aug 2016 09:39:45 -0400
Noah Massey <noah.massey@gmail.com> wrote:
> Since your data and metadata profiles are both 'single', you may want
> to consider creating the filesystem with the --mixed option, which
> allows / forces BTRFS to use a single collection of data blocks for
> both data and metadata.
That also seems to be detrimental to performance, I had servers with mixed
profile FSes which went painfully slow after a couple of years (daily snapshot
creation and deletion, ~30-60 snapshots kept at all times), migrated them all
now to the regular split profile. Fresh off mkfs they are fast of course, so
not entirely certain "mixed" was the culprit, will see.
One more problem with mixed is that you can't use the leaf and nodesize of 16k,
which are accepted to be faster than the older default (and mixed's only
option) of 4k.
--
With respect,
Roman
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Issue: errno:28 (No space left on device)
2016-08-09 11:50 Thomas
2016-08-09 12:21 ` Tomasz Torcz
2016-08-09 12:56 ` Austin S. Hemmelgarn
@ 2016-08-09 13:47 ` Josef Bacik
2 siblings, 0 replies; 9+ messages in thread
From: Josef Bacik @ 2016-08-09 13:47 UTC (permalink / raw)
To: Thomas, linux-btrfs
On 08/09/2016 07:50 AM, Thomas wrote:
> Hello!
>
> I'm facing a severe issue with Debian installation using BTRFS:
> errno:28 (No space left on device)
>
> After a few minutes of system usage I get this error message for any action,
> means there's really no disk space left.
>
> However, the output of btrfs fi usage / reports that ~10% of disk space should
> be free.
>
> I have 2 questions:
> 1. Why is the OS reporting an error "Issue: errno:28 (No space left on device)"
> whereas "btrfs fi usage" reports free space?
> 2. How can I solve this issue w/o the possibility of extending the partition?
>
Please try linus's latest tree, it has some enospc accounting fixes that have
recently gone in. Thanks,
Josef
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-08-09 16:50 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-08-09 9:06 Issue: errno:28 (No space left on device) Thomas
2016-08-09 16:50 ` Chris Murphy
-- strict thread matches above, loose matches on Subject: below --
2016-08-09 9:09 Thomas
2016-08-09 11:50 Thomas
2016-08-09 12:21 ` Tomasz Torcz
2016-08-09 12:56 ` Austin S. Hemmelgarn
2016-08-09 13:39 ` Noah Massey
2016-08-09 13:56 ` Roman Mamedov
2016-08-09 13:47 ` Josef Bacik
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).