From: Tomasz Chmielewski <tch@virtall.com>
To: linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: 4.4.0 - no space left with >1.7 TB free space left
Date: Mon, 08 Feb 2016 18:22:34 +0900 [thread overview]
Message-ID: <0c19f800da3013f22f2ab0d77d5a9efb@admin.virtall.com> (raw)
Linux 4.4.0 - btrfs is mainly used to host lots of test containers,
often snapshots, and at times, there is heavy IO in many of them for
extended periods of time. btrfs is on HDDs.
Every few days I'm getting "no space left" in a container running mongo
3.2.1 database. Interestingly, haven't seen this issue in containers
with MySQL. All databases have chattr +C set on their directories.
Why would it fail, if there is so much space left?
2016-02-07T06:06:14.648+0000 E STORAGE [thread1] WiredTiger (28)
[1454825174:633585][9105:0x7f2b7e33e700],
file:collection-33-7895599108848542105.wt, WT_SESSION.checkpoint:
collection-33-7895599108848542105.wt write error: failed to write 4096
bytes at offset 20480: No space left on device
2016-02-07T06:06:14.648+0000 E STORAGE [thread1] WiredTiger (28)
[1454825174:648740][9105:0x7f2b7e33e700], checkpoint-server: checkpoint
server error: No space left on device
2016-02-07T06:06:14.648+0000 E STORAGE [thread1] WiredTiger (-31804)
[1454825174:648766][9105:0x7f2b7e33e700], checkpoint-server: the process
must exit and restart: WT_PANIC: WiredTiger library panic
2016-02-07T06:06:14.648+0000 I - [thread1] Fatal Assertion 28558
2016-02-07T06:06:14.648+0000 I - [thread1]
***aborting after fassert() failure
2016-02-07T06:06:14.694+0000 I - [WTJournalFlusher] Fatal
Assertion 28559
2016-02-07T06:06:14.694+0000 I - [WTJournalFlusher]
***aborting after fassert() failure
2016-02-07T06:06:15.203+0000 F - [WTJournalFlusher] Got signal: 6
(Aborted).
# df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/sda4 2.7T 1.1T 1.7T 39% /srv
# btrfs fi df /srv
Data, RAID1: total=1.25TiB, used=1014.01GiB
System, RAID1: total=32.00MiB, used=240.00KiB
Metadata, RAID1: total=15.00GiB, used=13.13GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /srv
Label: 'btrfs' uuid: 105b2e0c-8af2-45ee-b4c8-14ff0a3ca899
Total devices 2 FS bytes used 1.00TiB
devid 1 size 2.63TiB used 1.26TiB path /dev/sda4
devid 2 size 2.63TiB used 1.26TiB path /dev/sdb4
btrfs-progs v4.0.1
Tomasz Chmielewski
http://wpkg.org
next reply other threads:[~2016-02-08 9:28 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-08 9:22 Tomasz Chmielewski [this message]
2016-02-08 11:24 ` 4.4.0 - no space left with >1.7 TB free space left Roman Mamedov
2016-02-08 12:15 ` Tomasz Chmielewski
2016-02-08 12:17 ` Roman Mamedov
2016-04-08 11:36 ` Tomasz Chmielewski
2016-04-08 11:53 ` Roman Mamedov
2016-04-08 13:20 ` Tomasz Chmielewski
2016-04-08 19:53 ` Duncan
2016-05-12 6:03 ` Tomasz Chmielewski
2016-05-12 6:07 ` Tomasz Chmielewski
2016-09-15 8:02 ` Roman Mamedov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0c19f800da3013f22f2ab0d77d5a9efb@admin.virtall.com \
--to=tch@virtall.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).