[BUG, bisect] hang when copying large file to disc

[BUG, bisect] hang when copying large file to disc

[Jeremiah Mahler]

If a large file (> 1 GB) is transferred from a USB stick or a mmc
to an local disc (using ext4), it will proceed normally until it
has transferred approximately 600 MB.  Then the speed will drop off
to zero and the terminal performing the operation will usually
hang.  And sometimes other terminals will hang as well.  A reboot
is required to get the system working again.

  mount /dev/sdb1 /mnt
  pv /mnt/large_file > large_file_out
  (... ~600 MB ... hang)

I have performed a bisect and found that commit ef39794651347 introduced
the bug.  This commit is present in the latest -next 20150130.

  From ef397946513470381c0e9ed5b355cd1c9855a364 Mon Sep 17 00:00:00 2001
  From: Theodore Ts'o <tytso@mit.edu>
  Date: Sat, 24 Jan 2015 20:51:05 -0500
  Subject: [PATCH] vfs: add support for a lazytime mount option
  
  Add a new mount option which enables a new "lazytime" mode.  This mode
  causes atime, mtime, and ctime updates to only be made to the
  in-memory version of the inode.  The on-disk times will only get
  updated when (a) if the inode needs to be updated for some non-time
  related change, (b) if userspace calls fsync(), syncfs() or sync(), or
  (c) just before an undeleted inode is evicted from memory.
  
  This is OK according to POSIX because there are no guarantees after a
  crash unless userspace explicitly requests via a fsync(2) call.
  
  For workloads which feature a large number of random write to a
  preallocated file, the lazytime mount option significantly reduces
  writes to the inode table.  The repeated 4k writes to a single block
  will result in undesirable stress on flash devices and SMR disk
  drives.  Even on conventional HDD's, the repeated writes to the inode
  table block will trigger Adjacent Track Interference (ATI) remediation
  latencies, which very negatively impact long tail latencies --- which
  is a very big deal for web serving tiers (for example).
  
  Google-Bug-Id: 18297052
  
  Signed-off-by: Theodore Ts'o <tytso@mit.edu>
  ---
   fs/ext4/inode.c                  |  6 ++++
   fs/fs-writeback.c                | 62 +++++++++++++++++++++++++++++++++-------
   fs/gfs2/file.c                   |  4 +--
   fs/inode.c                       | 56 +++++++++++++++++++++++++-----------
   fs/jfs/file.c                    |  2 +-
   fs/libfs.c                       |  2 +-
   fs/proc_namespace.c              |  1 +
   fs/sync.c                        |  8 ++++++
   include/linux/backing-dev.h      |  1 +
   include/linux/fs.h               |  5 ++++
   include/trace/events/writeback.h | 60 +++++++++++++++++++++++++++++++++++++-
   include/uapi/linux/fs.h          |  4 ++-
   mm/backing-dev.c                 | 10 +++++--
   13 files changed, 186 insertions(+), 35 deletions(-)

If I can do anything else to help, let me know.

-- 
- Jeremiah Mahler

Re: [BUG, bisect] hang when copying large file to disc

[Theodore Ts'o]

On Fri, Jan 30, 2015 at 07:40:51PM -0800, Jeremiah Mahler wrote:
> 
> If a large file (> 1 GB) is transferred from a USB stick or a mmc
> to an local disc (using ext4), it will proceed normally until it
> has transferred approximately 600 MB.  Then the speed will drop off
> to zero and the terminal performing the operation will usually
> hang.  And sometimes other terminals will hang as well.  A reboot
> is required to get the system working again.
> 
>   mount /dev/sdb1 /mnt
>   pv /mnt/large_file > large_file_out
>   (... ~600 MB ... hang)
> 
> I have performed a bisect and found that commit ef39794651347 introduced
> the bug.  This commit is present in the latest -next 20150130.

Thanks for the bug report.  I was able to reproduce the problem, and
found the problem.  Unfortunately I accidentally introduce this in my
most recent patchset.  I'll fix it up, but here is the patch until I
get the a new version pushed out.

diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 82c2984..0046861 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -1192,9 +1192,9 @@ void __mark_inode_dirty(struct inode *inode, int flags)
 
 		trace_writeback_dirty_inode(inode, flags);
 	}
-	dirtytime = flags & I_DIRTY_TIME;
 	if (flags & I_DIRTY_INODE)
 		flags &= ~I_DIRTY_TIME;
+	dirtytime = flags & I_DIRTY_TIME;
 
 	/*
 	 * Paired with smp_mb() in __writeback_single_inode() for the

	   	       		   			      	  - Ted

Re: [BUG, bisect] hang when copying large file to disc

[Jeremiah Mahler]

Ted,

On Sat, Jan 31, 2015 at 09:02:30PM -0500, Theodore Ts'o wrote:
> On Fri, Jan 30, 2015 at 07:40:51PM -0800, Jeremiah Mahler wrote:
> > 
[...]
> 
> Thanks for the bug report.  I was able to reproduce the problem, and
> found the problem.  Unfortunately I accidentally introduce this in my
> most recent patchset.  I'll fix it up, but here is the patch until I
> get the a new version pushed out.
> 
> diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
> index 82c2984..0046861 100644
> --- a/fs/fs-writeback.c
> +++ b/fs/fs-writeback.c
> @@ -1192,9 +1192,9 @@ void __mark_inode_dirty(struct inode *inode, int flags)
>  
>  		trace_writeback_dirty_inode(inode, flags);
>  	}
> -	dirtytime = flags & I_DIRTY_TIME;
>  	if (flags & I_DIRTY_INODE)
>  		flags &= ~I_DIRTY_TIME;
> +	dirtytime = flags & I_DIRTY_TIME;
>  
>  	/*
>  	 * Paired with smp_mb() in __writeback_single_inode() for the
> 
> 	   	       		   			      	  - Ted

That solved the problem.  Thanks for the quick fix.

-- 
- Jeremiah Mahler