Re: [PATCH] ext4: fix ext4_evict_inode() racing against workqueue processing code

[Zheng Liu <gnehzuil.liu@gmail.com>]

On Tue, Mar 26, 2013 at 09:34:03PM +0100, Jan Kara wrote:
> On Tue 26-03-13 13:52:51, Zheng Liu wrote:
> > Sorry for the late reply.
> > 
> > On Wed, Mar 20, 2013 at 10:45:23AM -0400, Theodore Ts'o wrote:
> > > On Wed, Mar 20, 2013 at 09:14:42AM -0500, Eric Sandeen wrote:
> > > > 
> > > > As an aside, is there any reason to have "dioread_nolock" as an option
> > > > at this point?  If it works now, would you ever *not* want it?
> > > > 
> > > > (granted it doesn't work with some journaling options etc, but that
> > > > behavior could be automatic, w/o the need for special mount options).
> > > 
> > > The primary restriction is that diread_nolock doesn't work when fs
> > > block size != page size.  If your proposal is that we automatically
> > > enable diread_nolock when we can use it safely, that's definitely
> > > something to consider for the next merge window.
> > 
> > Yes, I also think we can automatically enable dioread_nolock because it
> > brings us some benefits.
>   But isn't there also some overhead due to buffered writes having to go
> through uninit->init conversion?

Hi Jan,

Here is my test result.  I use fio to do some performance tests.  The
test environment is a Dell desktop with a Intel(R) Core(TM)2 Duo CPU
E8400  @ 3.00GHz, 4G memory, 1 Intel 320 SSD.  I test three cases that
are sync dio read/write, aio dio read/write, and sync buffered io.
The result is as below.  Due to too many number we post some key data
here.

>From the result we can see that after dioread_nolock enables the max
latency can be reduced dramatically for sync dio and sync bio, and other
value doesn't be affected too much.  If I miss some important test case,
please let me know.

Thanks,
                                                - Zheng

The kernel version is 3.9.0-rc4+.

= sync dio =
== fio config file ==
  [global]
  ioengine=psync
  direct=1
  bs=4k
  thread
  group_reporting
  directory=/mnt/sda1/
  filename=testfile
  filesize=10g
  size=10g
  runtime=120
  iodepth=16
  fallocate=0
  
  [reader]
  rw=randread
  numjobs=8
  
  [writer]
  rw=randwrite
  numjobs=2

== result ==
=== w/o dioread_nolock ===
  read : io=5142.1MB, bw=43885KB/s, iops=10971 , runt=120002msec
    clat (usec): min=123 , max=679911 , avg=727.21, stdev=3986.35
     lat (usec): min=123 , max=679911 , avg=727.40, stdev=3986.35
  write: io=542548KB, bw=4521.2KB/s, iops=1130 , runt=120003msec
    clat (usec): min=136 , max=2743.3K, avg=1766.78, stdev=14844.10
     lat (usec): min=137 , max=2743.3K, avg=1767.14, stdev=14844.10

=== w/ dioread_nolock ===
  read : io=5072.2MB, bw=43282KB/s, iops=10820 , runt=120002msec
    clat (usec): min=158 , max=291847 , avg=737.38, stdev=3944.14
     lat (usec): min=158 , max=291848 , avg=737.56, stdev=3944.14
  write: io=589752KB, bw=4914.6KB/s, iops=1228 , runt=120001msec
    clat (usec): min=138 , max=2048.3K, avg=1625.15, stdev=11344.36
     lat (usec): min=139 , max=2048.3K, avg=1625.50, stdev=11344.35

= aio dio =
== fio config file ==
  [global]
  ioengine=libaio
  direct=1
  bs=4k
  thread
  group_reporting
  directory=/mnt/sda1/
  filename=testfile
  filesize=10g
  size=10g
  runtime=120
  iodepth=64
  fallocate=0
  
  [reader]
  rw=randread
  numjobs=8
  
  [writer]
  rw=randwrite
  numjobs=2

== result ==
=== w/o dioread_nolock ===
  read : io=4865.4MB, bw=41510KB/s, iops=10377 , runt=120014msec
    slat (usec): min=3 , max=320273 , avg=700.79, stdev=6070.93
    clat (usec): min=232 , max=958452 , avg=48627.21, stdev=53296.33
     lat (usec): min=248 , max=958471 , avg=49328.36, stdev=53905.27
  write: io=684100KB, bw=5700.3KB/s, iops=1425 , runt=120013msec
    slat (usec): min=5 , max=349019 , avg=1339.75, stdev=9538.63
    clat (usec): min=140 , max=1060.1K, avg=88459.40, stdev=95903.63
     lat (usec): min=211 , max=1060.1K, avg=89799.48, stdev=97025.95

=== w/ dioread_nolock ===
  read : io=4869.1MB, bw=41544KB/s, iops=10385 , runt=120037msec
    slat (usec): min=4 , max=322951 , avg=700.33, stdev=5987.29
    clat (usec): min=229 , max=703241 , avg=48584.27, stdev=52570.02
     lat (usec): min=248 , max=703247 , avg=49284.94, stdev=53185.97
  write: io=698856KB, bw=5821.1KB/s, iops=1455 , runt=120038msec
    slat (usec): min=7 , max=367042 , avg=1297.87, stdev=9316.21
    clat (usec): min=178 , max=968434 , avg=86625.80, stdev=92500.21
     lat (usec): min=284 , max=1020.8K, avg=87924.01, stdev=93554.86

= sync buffered io =
== fio config file ==
  [global]
  ioengine=psync
  direct=0
  bs=4k
  thread
  group_reporting
  directory=/mnt/sda1/
  filename=testfile
  filesize=10g
  size=10g
  runtime=120
  iodepth=16
  fallocate=0
  
  [reader]
  rw=randread
  numjobs=8
  
  [writer]
  rw=randwrite
  numjobs=2
  
== result ==
=== w/o dioread_nolock ===
  read : io=7577.6MB, bw=64661KB/s, iops=16165 , runt=120001msec
    clat (usec): min=1 , max=3107.8K, avg=493.15, stdev=5368.87
     lat (usec): min=1 , max=3107.8K, avg=493.31, stdev=5368.87
  write: io=764256KB, bw=6350.2KB/s, iops=1587 , runt=120353msec
    clat (usec): min=2 , max=42443K, avg=1257.41, stdev=138832.28
     lat (usec): min=2 , max=42443K, avg=1257.57, stdev=138832.29

=== w/ dioread_nolock ===
  read : io=7613.1MB, bw=64972KB/s, iops=16243 , runt=120001msec
    clat (usec): min=1 , max=1126.4K, avg=490.79, stdev=3742.64
     lat (usec): min=1 , max=1126.4K, avg=490.95, stdev=3742.64
  write: io=763428KB, bw=6351.1KB/s, iops=1587 , runt=120189msec
    clat (usec): min=2 , max=45783K, avg=1258.23, stdev=163258.23
     lat (usec): min=2 , max=45783K, avg=1258.37, stdev=163258.23