From: Stan Hoeppner <stan@hardwarefreak.com>
To: Bernhard Schmidt <berni@birkenwald.de>
Cc: xfs@oss.sgi.com
Subject: Re: Premature "No Space left on device" on XFS
Date: Thu, 06 Oct 2011 19:22:08 -0500 [thread overview]
Message-ID: <4E8E4630.8030108@hardwarefreak.com> (raw)
In-Reply-To: <4E8E079B.4040103@birkenwald.de>
On 10/6/2011 2:55 PM, Bernhard Schmidt wrote:
> Hi,
>
> this is an XFS-related summary of a problem report I sent to the postfix
> mailinglist a few minutes ago after a bulkmail test system blew up
> during a stress test.
>
> We have a few MTAs running SLES11.1 amd64 (2.6.32.45-0.3-default), 10 GB
> XFS Spooldirectory with default blocksize (4k). It was bombarded with
> mails faster than it could send them on, which eventually led to almost
> 2 million files of ~1.5kB in one directory. Suddenly, this started to
> happen
>
> lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # touch a
> touch: cannot touch `a': No space left on device
> lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # df .
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sdb 10475520 7471160 3004360 72%
> /var/spool/postfix-bulk
> lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # df -i .
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/sdb 10485760 1742528 8743232 17% /var/spool/postfix-bulk
>
> So we could not create any file in the spool directory anymore despite
> df claiming to have both free blocks and inodes. This led to a pretty
> spectacular lockup of the mail processing chain.
>
> My theory is that XFS is using a full 4k block for each 1.5kB file,
> which accounts to some loss. But still, 10GB / 4kB makes 2.5 mio files,
> which have surely not been reached here. Is there that high overhead?
> Why is neither df-metric reporting this problem? Is there any way to get
> reasonable readings out of df in this case? The system would have
> stopped accepting mail from outside if the freespace would have sunk
> below 2GB, so out-of-space happened way to early for it.
Dig deeper so you can get past theory and find facts. Do you see any
errors in dmseg?
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2011-10-07 0:22 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-10-06 19:55 Premature "No Space left on device" on XFS Bernhard Schmidt
2011-10-07 0:22 ` Stan Hoeppner [this message]
2011-10-07 0:47 ` Bernhard Schmidt
2011-10-07 1:37 ` Dave Chinner
2011-10-07 8:40 ` Gim Leong Chin
2011-10-07 23:20 ` Dave Chinner
2011-10-07 11:40 ` Michael Monnerie
2011-10-07 23:17 ` Dave Chinner
2011-10-07 13:49 ` Bernhard Schmidt
2011-10-07 23:14 ` Dave Chinner
2011-10-08 12:29 ` Bernhard Schmidt
2011-10-08 13:18 ` Christoph Hellwig
2011-10-08 22:34 ` Dave Chinner
2011-10-09 14:46 ` Christoph Hellwig
2011-10-08 22:30 ` Dave Chinner
2011-10-07 13:58 ` Bryan J Smith
2011-10-07 23:31 ` Dave Chinner
2011-10-08 6:30 ` Bryan J Smith
2011-10-08 13:16 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E8E4630.8030108@hardwarefreak.com \
--to=stan@hardwarefreak.com \
--cc=berni@birkenwald.de \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox