From: Brian Foster <bfoster@redhat.com>
To: Dave Chinner <david@fromorbit.com>
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 06/13] xfs: xfs_sync_data is redundant.
Date: Mon, 01 Oct 2012 20:44:01 -0400 [thread overview]
Message-ID: <506A38D1.6090204@redhat.com> (raw)
In-Reply-To: <20121002001021.GJ23520@dastard>
On 10/01/2012 08:10 PM, Dave Chinner wrote:
> Hi, Brian.
>
> On Mon, Oct 01, 2012 at 04:14:40PM -0400, Brian Foster wrote:
>> Warning: This message has had one or more attachments removed
>> Warning: (273.out.bad).
>> Warning: Please read the "boprocket-Attachment-Warning.txt" attachment(s) for more information.
>
> Which says:
>
>> At Mon Oct 1 20:14:58 2012 the virus scanner said:
>> MailScanner: Attempt to hide real filename extension (273.out.bad)
>
> Looks like your mailer did something wrong with the attachment....
>
Ugh, sorry. The output file was filled with messages like so:
_porter 31 not complete
_porter 79 not complete
_porter 149 not complete
_porter 74 not complete
_porter 161 not complete
_porter 54 not complete
_porter 98 not complete
_porter 99 not complete
...
>
>> Heads up... I was doing some testing against my eofblocks set rebased
>> against this patchset and I'm reproducing a new 273 failure. The failure
>> bisects down to this patch.
>>
>> With the bisection, I'm running xfs top of tree plus the following patch:
>>
>> xfs: only update the last_sync_lsn when a transaction completes
>>
>> ... and patches 1-6 of this set on top of that. i.e.:
>>
>> xfs: xfs_sync_data is redundant.
>> xfs: Bring some sanity to log unmounting
>> xfs: sync work is now only periodic log work
>> xfs: don't run the sync work if the filesystem is read-only
>> xfs: rationalise xfs_mount_wq users
>> xfs: xfs_syncd_stop must die
>> xfs: only update the last_sync_lsn when a transaction completes
>> xfs: Make inode32 a remountable option
>>
>> This is on a 16p (according to /proc/cpuinfo) x86-64 system with 32GB
>> RAM. The test and scratch volumes are both 500GB lvm volumes on top of a
>> hardware raid.
>> I haven't looked into this at all yet but I wanted to
>> drop it on the list for now. The 273 output is attached.
>
> I bet you had writes fail with ENOSPC - 201 * 426
> = 85626 files of 8k each, that gives 685MB. When the test is
> running, I see upwards of 1.5GB of space consumed, which then slowly
> drops again as data files are closed and data is written.
>
> Some of that space is specualtive preallocation (4k per file, I
> think), but also a significant amount of it is metadata reservation
> for delayed allocation (4 blocks per file, IIRC). If I've only got
> 2GB RAM on my machine, then writeback starts at 200MB written, and
> so well before the fs runs out of space the metadata reservations
> are being released.
>
> I just upped the VM to 8GB RAM, and immediately I see the test
> starting to fail. And this is in 273.full:
>
> cp: cannot create regular file `/mnt/scratch/sub_198/origin/file_141': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_198/origin/file_142': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_198/origin/file_1cp: cannot create regular filcp: cannot create regular file `/mnt/scratch/sub_198/origin/file_147': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_198/origin/file_1cp: cannot create regular filcp: writing `/mnt/scratch/sub_198/origin/file_149': No space left cp: writing `/mnt/scratch/sub_156/origin/file_275': No space left on device
> cp: failed to extencp: writing `/mnt/scratch/sub_198/origin/file_150': No space left cp: writing `/mnt/scratch/sub_156/origin/file_276': No space left on device
> cp: failed to extencp: cannot create regular file `/mnt/scratch/sub_124/origin/file_3cp: cannot create regular filcp: writing `/mnt/scratch/sub_124/origin/file_378': No space left cp: writing `/mnt/scratch/sub_173/origin/file_250': No space left on device
> cp: failed to extencp: writing `/mnt/scratch/sub_124/origin/file_379': No space left cp: cannot create regular file `/mnt/scratch/sub_173/origin/file_2cp: cannot create regular file `/mnt/scratch/sub_134/origin/file_337': No space left on device
> cp: cannot create regular filcp: cannot create regular filcp: writing `/mnt/scratch/sub_159/origin/file_307': No space left on device
> cp: failed to extend `/mnt/scratch/sub_159/origin/file_307': No space left on device
> cp: writing `/mnt/scratch/sub_159/origin/file_308': No space left on device
> cp: failed to extend `/mnt/scratch/sub_159/origin/file_308': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_309': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_310': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_311': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_312': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_313': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_314': No space left on device
> .....
>
Yep, I see the same thing...
> So, turning off speculative preallocation via the allocsize mount
> option doesn't fix the problem. IOWs, the problem is too much active
> metadata reservation. If we are caching 685MB, that's less than the
> writeback thresholds of a large memory machine, so the metadata
> reservations won't be trimmed at all until ENOSPC actually occurs
> and writeback is then started.
>
> The problem is that writeback_inodes_sb_if_idle() does not block if
> there is already writeback in progress, so the callers just keep
> hitting ENOSPC rather than being throttled waiting for delalloc
> conversion.
>
> The patch below should fix this - it changes xfs_flush_inodes() to
> us sync_inodes_sb(), which will issue IO and block waiting for it to
> complete, just like xfs_flush_inodes() used to. Indeed, it passes
> again on my VM with 8GB RAM....
>
I gave this a quick couple runs against 273 and it passes (on top of the
entire die-xfssyncd-die patchset). I'll kick off another full run on
this box overnight. Thanks!
Brian
> Cheers,
>
> Dave.
>
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2012-10-02 0:44 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-09-28 4:44 [PATCH V3 00/13] xfs: remove the xfssyncd mess Dave Chinner
2012-09-28 4:44 ` [PATCH 01/13] xfs: xfs_syncd_stop must die Dave Chinner
2012-09-28 4:44 ` [PATCH 02/13] xfs: rationalise xfs_mount_wq users Dave Chinner
2012-09-28 12:27 ` Christoph Hellwig
2012-09-28 4:44 ` [PATCH 03/13] xfs: don't run the sync work if the filesystem is read-only Dave Chinner
2012-09-28 12:27 ` Christoph Hellwig
2012-09-28 4:44 ` [PATCH 04/13] xfs: sync work is now only periodic log work Dave Chinner
2012-09-28 12:31 ` Christoph Hellwig
2012-09-28 4:44 ` [PATCH 05/13] xfs: Bring some sanity to log unmounting Dave Chinner
2012-09-28 4:44 ` [PATCH 06/13] xfs: xfs_sync_data is redundant Dave Chinner
2012-09-28 12:34 ` Christoph Hellwig
2012-10-01 17:44 ` Ben Myers
2012-10-02 0:14 ` Dave Chinner
2012-10-02 19:39 ` Ben Myers
2012-09-28 21:44 ` Mark Tinguely
2012-10-01 20:14 ` Brian Foster
2012-10-01 21:31 ` Mark Tinguely
2012-10-02 0:10 ` Dave Chinner
2012-10-02 0:44 ` Brian Foster [this message]
2012-10-02 13:01 ` Brian Foster
2012-10-02 20:51 ` Dave Chinner
2012-10-02 21:13 ` Brian Foster
2012-10-04 0:05 ` Ben Myers
2012-10-04 1:07 ` Dave Chinner
2012-10-02 13:22 ` Christoph Hellwig
2012-10-02 20:24 ` Dave Chinner
2012-10-02 20:25 ` Ben Myers
2012-09-28 4:44 ` [PATCH 07/13] xfs: syncd workqueue is no more Dave Chinner
2012-09-28 12:35 ` Christoph Hellwig
2012-09-28 18:17 ` Mark Tinguely
2012-10-01 17:54 ` Ben Myers
2012-09-28 4:44 ` [PATCH 08/13] xfs: xfs_sync_fsdata is redundant Dave Chinner
2012-09-28 4:44 ` [PATCH 09/13] xfs: move xfs_quiesce_attr() into xfs_super.c Dave Chinner
2012-09-28 4:44 ` [PATCH 10/13] xfs: xfs_quiesce_attr() should quiesce the log like unmount Dave Chinner
2012-09-28 4:44 ` [PATCH 11/13] xfs: rename xfs_sync.[ch] to xfs_icache.[ch] Dave Chinner
2012-09-28 4:44 ` [PATCH 12/13] xfs: move inode locking functions to xfs_inode.c Dave Chinner
2012-09-28 4:44 ` [PATCH 13/13] xfs: remove xfs_iget.c Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=506A38D1.6090204@redhat.com \
--to=bfoster@redhat.com \
--cc=david@fromorbit.com \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.