From: Stan Hoeppner <stan@hardwarefreak.com>
To: Bernhard Schmidt <berni@birkenwald.de>
Cc: xfs@oss.sgi.com
Subject: Re: Premature "No Space left on device" on XFS
Date: Thu, 06 Oct 2011 19:22:08 -0500 [thread overview]
Message-ID: <4E8E4630.8030108@hardwarefreak.com> (raw)
In-Reply-To: <4E8E079B.4040103@birkenwald.de>
On 10/6/2011 2:55 PM, Bernhard Schmidt wrote:
> Hi,
>
> this is an XFS-related summary of a problem report I sent to the postfix
> mailinglist a few minutes ago after a bulkmail test system blew up
> during a stress test.
>
> We have a few MTAs running SLES11.1 amd64 (2.6.32.45-0.3-default), 10 GB
> XFS Spooldirectory with default blocksize (4k). It was bombarded with
> mails faster than it could send them on, which eventually led to almost
> 2 million files of ~1.5kB in one directory. Suddenly, this started to
> happen
>
> lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # touch a
> touch: cannot touch `a': No space left on device
> lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # df .
> Filesystem 1K-blocks Used Available Use% Mounted on
> /dev/sdb 10475520 7471160 3004360 72%
> /var/spool/postfix-bulk
> lxmhs45:/var/spool/postfix-bulk/postfix-bulkinhss # df -i .
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/sdb 10485760 1742528 8743232 17% /var/spool/postfix-bulk
>
> So we could not create any file in the spool directory anymore despite
> df claiming to have both free blocks and inodes. This led to a pretty
> spectacular lockup of the mail processing chain.
>
> My theory is that XFS is using a full 4k block for each 1.5kB file,
> which accounts to some loss. But still, 10GB / 4kB makes 2.5 mio files,
> which have surely not been reached here. Is there that high overhead?
> Why is neither df-metric reporting this problem? Is there any way to get
> reasonable readings out of df in this case? The system would have
> stopped accepting mail from outside if the freespace would have sunk
> below 2GB, so out-of-space happened way to early for it.
Dig deeper so you can get past theory and find facts. Do you see any
errors in dmseg?
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2011-10-07 0:22 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-10-06 19:55 Premature "No Space left on device" on XFS Bernhard Schmidt
2011-10-07 0:22 ` Stan Hoeppner [this message]
2011-10-07 0:47 ` Bernhard Schmidt
2011-10-07 1:37 ` Dave Chinner
2011-10-07 8:40 ` Gim Leong Chin
2011-10-07 23:20 ` Dave Chinner
2011-10-07 11:40 ` Michael Monnerie
2011-10-07 23:17 ` Dave Chinner
2011-10-07 13:49 ` Bernhard Schmidt
2011-10-07 23:14 ` Dave Chinner
2011-10-08 12:29 ` Bernhard Schmidt
2011-10-08 13:18 ` Christoph Hellwig
2011-10-08 22:34 ` Dave Chinner
2011-10-09 14:46 ` Christoph Hellwig
2011-10-08 22:30 ` Dave Chinner
2011-10-07 13:58 ` Bryan J Smith
2011-10-07 23:31 ` Dave Chinner
2011-10-08 6:30 ` Bryan J Smith
2011-10-08 13:16 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4E8E4630.8030108@hardwarefreak.com \
--to=stan@hardwarefreak.com \
--cc=berni@birkenwald.de \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.