public inbox for linux-xfs@vger.kernel.org
 help / color / mirror / Atom feed
From: Brian Foster <bfoster@redhat.com>
To: Dave Chinner <david@fromorbit.com>
Cc: xfs@oss.sgi.com
Subject: Re: [PATCH 06/13] xfs: xfs_sync_data is redundant.
Date: Mon, 01 Oct 2012 20:44:01 -0400	[thread overview]
Message-ID: <506A38D1.6090204@redhat.com> (raw)
In-Reply-To: <20121002001021.GJ23520@dastard>

On 10/01/2012 08:10 PM, Dave Chinner wrote:
> Hi, Brian.
> 
> On Mon, Oct 01, 2012 at 04:14:40PM -0400, Brian Foster wrote:
>> Warning: This message has had one or more attachments removed
>> Warning: (273.out.bad).
>> Warning: Please read the "boprocket-Attachment-Warning.txt" attachment(s) for more information.
> 
> Which says:
> 
>> At Mon Oct  1 20:14:58 2012 the virus scanner said:                
>>    MailScanner: Attempt to hide real filename extension (273.out.bad)
> 
> Looks like your mailer did something wrong with the attachment....
> 

Ugh, sorry. The output file was filled with messages like so:

_porter 31 not complete
_porter 79 not complete
_porter 149 not complete
_porter 74 not complete
_porter 161 not complete
_porter 54 not complete
_porter 98 not complete
_porter 99 not complete
...

> 
>> Heads up... I was doing some testing against my eofblocks set rebased
>> against this patchset and I'm reproducing a new 273 failure. The failure
>> bisects down to this patch.
>>
>> With the bisection, I'm running xfs top of tree plus the following patch:
>>
>> xfs: only update the last_sync_lsn when a transaction completes
>>
>> ... and patches 1-6 of this set on top of that. i.e.:
>>
>> xfs: xfs_sync_data is redundant.
>> xfs: Bring some sanity to log unmounting
>> xfs: sync work is now only periodic log work
>> xfs: don't run the sync work if the filesystem is read-only
>> xfs: rationalise xfs_mount_wq users
>> xfs: xfs_syncd_stop must die
>> xfs: only update the last_sync_lsn when a transaction completes
>> xfs: Make inode32 a remountable option
>>
>> This is on a 16p (according to /proc/cpuinfo) x86-64 system with 32GB
>> RAM. The test and scratch volumes are both 500GB lvm volumes on top of a
>> hardware raid.
>> I haven't looked into this at all yet but I wanted to
>> drop it on the list for now. The 273 output is attached.
> 
> I bet you had writes fail with ENOSPC - 201 * 426
> = 85626 files of 8k each, that gives 685MB. When the test is
> running, I see upwards of 1.5GB of space consumed, which then slowly
> drops again as data files are closed and data is written.
> 
> Some of that space is specualtive preallocation (4k per file, I
> think), but also a significant amount of it is metadata reservation
> for delayed allocation (4 blocks per file, IIRC). If I've only got
> 2GB RAM on my machine, then writeback starts at 200MB written, and
> so well before the fs runs out of space the metadata reservations
> are being released.
> 
> I just upped the VM to 8GB RAM, and immediately I see the test
> starting to fail. And this is in 273.full:
> 
> cp: cannot create regular file `/mnt/scratch/sub_198/origin/file_141': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_198/origin/file_142': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_198/origin/file_1cp: cannot create regular filcp: cannot create regular file `/mnt/scratch/sub_198/origin/file_147': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_198/origin/file_1cp: cannot create regular filcp: writing `/mnt/scratch/sub_198/origin/file_149': No space left cp: writing `/mnt/scratch/sub_156/origin/file_275': No space left on device
> cp: failed to extencp: writing `/mnt/scratch/sub_198/origin/file_150': No space left cp: writing `/mnt/scratch/sub_156/origin/file_276': No space left on device
> cp: failed to extencp: cannot create regular file `/mnt/scratch/sub_124/origin/file_3cp: cannot create regular filcp: writing `/mnt/scratch/sub_124/origin/file_378': No space left cp: writing `/mnt/scratch/sub_173/origin/file_250': No space left on device
> cp: failed to extencp: writing `/mnt/scratch/sub_124/origin/file_379': No space left cp: cannot create regular file `/mnt/scratch/sub_173/origin/file_2cp: cannot create regular file `/mnt/scratch/sub_134/origin/file_337': No space left on device
> cp: cannot create regular filcp: cannot create regular filcp: writing `/mnt/scratch/sub_159/origin/file_307': No space left on device
> cp: failed to extend `/mnt/scratch/sub_159/origin/file_307': No space left on device
> cp: writing `/mnt/scratch/sub_159/origin/file_308': No space left on device
> cp: failed to extend `/mnt/scratch/sub_159/origin/file_308': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_309': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_310': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_311': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_312': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_313': No space left on device
> cp: cannot create regular file `/mnt/scratch/sub_159/origin/file_314': No space left on device
> .....
> 

Yep, I see the same thing...

> So, turning off speculative preallocation via the allocsize mount
> option doesn't fix the problem. IOWs, the problem is too much active
> metadata reservation.  If we are caching 685MB, that's less than the
> writeback thresholds of a large memory machine, so the metadata
> reservations won't be trimmed at all until ENOSPC actually occurs
> and writeback is then started.
> 
> The problem is that writeback_inodes_sb_if_idle() does not block if
> there is already writeback in progress, so the callers just keep
> hitting ENOSPC rather than being throttled waiting for delalloc
> conversion.
> 
> The patch below should fix this - it changes xfs_flush_inodes() to
> us sync_inodes_sb(), which will issue IO and block waiting for it to
> complete, just like xfs_flush_inodes() used to. Indeed, it passes
> again on my VM with 8GB RAM....
> 

I gave this a quick couple runs against 273 and it passes (on top of the
entire die-xfssyncd-die patchset). I'll kick off another full run on
this box overnight. Thanks!

Brian

> Cheers,
> 
> Dave.
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

  reply	other threads:[~2012-10-02  0:44 UTC|newest]

Thread overview: 37+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-09-28  4:44 [PATCH V3 00/13] xfs: remove the xfssyncd mess Dave Chinner
2012-09-28  4:44 ` [PATCH 01/13] xfs: xfs_syncd_stop must die Dave Chinner
2012-09-28  4:44 ` [PATCH 02/13] xfs: rationalise xfs_mount_wq users Dave Chinner
2012-09-28 12:27   ` Christoph Hellwig
2012-09-28  4:44 ` [PATCH 03/13] xfs: don't run the sync work if the filesystem is read-only Dave Chinner
2012-09-28 12:27   ` Christoph Hellwig
2012-09-28  4:44 ` [PATCH 04/13] xfs: sync work is now only periodic log work Dave Chinner
2012-09-28 12:31   ` Christoph Hellwig
2012-09-28  4:44 ` [PATCH 05/13] xfs: Bring some sanity to log unmounting Dave Chinner
2012-09-28  4:44 ` [PATCH 06/13] xfs: xfs_sync_data is redundant Dave Chinner
2012-09-28 12:34   ` Christoph Hellwig
2012-10-01 17:44     ` Ben Myers
2012-10-02  0:14       ` Dave Chinner
2012-10-02 19:39         ` Ben Myers
2012-09-28 21:44   ` Mark Tinguely
2012-10-01 20:14   ` Brian Foster
2012-10-01 21:31     ` Mark Tinguely
2012-10-02  0:10     ` Dave Chinner
2012-10-02  0:44       ` Brian Foster [this message]
2012-10-02 13:01         ` Brian Foster
2012-10-02 20:51           ` Dave Chinner
2012-10-02 21:13             ` Brian Foster
2012-10-04  0:05             ` Ben Myers
2012-10-04  1:07               ` Dave Chinner
2012-10-02 13:22       ` Christoph Hellwig
2012-10-02 20:24         ` Dave Chinner
2012-10-02 20:25     ` Ben Myers
2012-09-28  4:44 ` [PATCH 07/13] xfs: syncd workqueue is no more Dave Chinner
2012-09-28 12:35   ` Christoph Hellwig
2012-09-28 18:17   ` Mark Tinguely
2012-10-01 17:54     ` Ben Myers
2012-09-28  4:44 ` [PATCH 08/13] xfs: xfs_sync_fsdata is redundant Dave Chinner
2012-09-28  4:44 ` [PATCH 09/13] xfs: move xfs_quiesce_attr() into xfs_super.c Dave Chinner
2012-09-28  4:44 ` [PATCH 10/13] xfs: xfs_quiesce_attr() should quiesce the log like unmount Dave Chinner
2012-09-28  4:44 ` [PATCH 11/13] xfs: rename xfs_sync.[ch] to xfs_icache.[ch] Dave Chinner
2012-09-28  4:44 ` [PATCH 12/13] xfs: move inode locking functions to xfs_inode.c Dave Chinner
2012-09-28  4:44 ` [PATCH 13/13] xfs: remove xfs_iget.c Dave Chinner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=506A38D1.6090204@redhat.com \
    --to=bfoster@redhat.com \
    --cc=david@fromorbit.com \
    --cc=xfs@oss.sgi.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox