kernelnewbies.kernelnewbies.org archive mirror
 help / color / mirror / Atom feed
* which local FS supports concurrent direct IO write?
@ 2012-01-13 21:41 Zheng Da
  2012-01-14 11:45 ` Raghavendra D Prabhu
  0 siblings, 1 reply; 5+ messages in thread
From: Zheng Da @ 2012-01-13 21:41 UTC (permalink / raw)
  To: kernelnewbies

Hello,

I'm looking for a FS in Linux that supports concurrent direct IO write.
ext4 supports concurrent direct IO read if we mount it with dioread_nolock,
but doesn't support concurrent writes. XFS doesn't support concurrent
direct IO at all. It locks the inode exclusive if it's direct IO. I tried
btrfs, and it seems it doesn't support concurrent direct IO either though I
haven't looked into its code.
Is there a local FS that support concurrent direct IO write? It seems NFS
supports it (
http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/),
but I'm looking for local FS.

Thanks,
Da
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120113/50888ba3/attachment.html 

^ permalink raw reply	[flat|nested] 5+ messages in thread
* which local FS supports concurrent direct IO write?
  2012-01-13 21:41 which local FS supports concurrent direct IO write? Zheng Da
@ 2012-01-14 11:45 ` Raghavendra D Prabhu
  2012-01-15 20:17   ` Zheng Da
  0 siblings, 1 reply; 5+ messages in thread
From: Raghavendra D Prabhu @ 2012-01-14 11:45 UTC (permalink / raw)
  To: kernelnewbies

Hi Zheng,

* On Fri, Jan 13, 2012 at 04:41:16PM -0500, Zheng Da <zhengda1936@gmail.com> wrote:
>Hello,
>
>I'm looking for a FS in Linux that supports concurrent direct IO write.
>ext4 supports concurrent direct IO read if we mount it with dioread_nolock,
>but doesn't support concurrent writes. XFS doesn't support concurrent
>direct IO at all. It locks the inode exclusive if it's direct IO. I tried
>btrfs, and it seems it doesn't support concurrent direct IO either though I
>haven't looked into its code.
>Is there a local FS that support concurrent direct IO write? It seems NFS
>supports it (
>http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/),
>but I'm looking for local FS.
>
>Thanks,
>Da

>_______________________________________________
>Kernelnewbies mailing list
>Kernelnewbies at kernelnewbies.org
>http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

XFS locks inode exclusive only if it is an unaligned Direct 
IO, which is apparently done to prevent race conditions -- 
refer to this  http://oss.sgi.com/archives/xfs/2011-01/msg00157.html 
Also the behavior of Ext4 under dioread_nolock is supported 
by XFS by default and in a much better way. Also Ext4 is the only 
one which uses DIO_LOCKING while doing direct io.



Regards,
-- 
Raghavendra Prabhu
GPG Id : 0xD72BE977
Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
www: wnohang.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120114/da6fb250/attachment.bin 

^ permalink raw reply	[flat|nested] 5+ messages in thread
* which local FS supports concurrent direct IO write?
  2012-01-14 11:45 ` Raghavendra D Prabhu
@ 2012-01-15 20:17   ` Zheng Da
  2012-01-15 21:22     ` Raghavendra D Prabhu
  0 siblings, 1 reply; 5+ messages in thread
From: Zheng Da @ 2012-01-15 20:17 UTC (permalink / raw)
  To: kernelnewbies

Thanks. I was reading the code of kernel 3.0. XFS starts to support
concurrent direct IO since kernel 3.1.5.
But concurrent direct IO write still doesn't work well in kernel 3.2. I
wrote a test program that accesses a 4G file randomly (read and write), and
I ran it with 8 threads and the machine has 8 cores. It turns out that only
1 core is running. I'm pretty sure xfs_rw_ilock is locked
with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write.

lockstat shows me that there is a lot of wait time in ip->i_lock. It seems
the lock is locked exclusively.
               &(&ip->i_lock)->mr_lock-W:         31568          36170
      0.24       20048.25     7589157.99         130154        3146848
      0.00         217.70     1238310.72
               &(&ip->i_lock)->mr_lock-R:         11251          11886
      0.24       20043.01     2895595.18          46671         526309
      0.00          63.80      264097.96
               -------------------------
                 &(&ip->i_lock)->mr_lock          36170
 [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]
                 &(&ip->i_lock)->mr_lock          11886
 [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]
               -------------------------
                 &(&ip->i_lock)->mr_lock          38555
 [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]
                 &(&ip->i_lock)->mr_lock           9501
 [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]

Then I used systemtap to instrument xfs_ilock and find there are at least 3
functions that lock ip->i_lock exclusively during write.
Is there any popular FS that supports concurrent direct IO well?

Thanks,
Da

On Sat, Jan 14, 2012 at 6:45 AM, Raghavendra D Prabhu <
raghu.prabhu13@gmail.com> wrote:

> Hi Zheng,
>
>
> * On Fri, Jan 13, 2012 at 04:41:16PM -0500, Zheng Da <
> zhengda1936 at gmail.com> wrote:
>
>> Hello,
>>
>> I'm looking for a FS in Linux that supports concurrent direct IO write.
>> ext4 supports concurrent direct IO read if we mount it with
>> dioread_nolock,
>> but doesn't support concurrent writes. XFS doesn't support concurrent
>> direct IO at all. It locks the inode exclusive if it's direct IO. I tried
>> btrfs, and it seems it doesn't support concurrent direct IO either though
>> I
>> haven't looked into its code.
>> Is there a local FS that support concurrent direct IO write? It seems NFS
>> supports it (
>> http://kevinclosson.wordpress.**com/2011/08/12/file-systems-**
>> for-a-database-choose-one-**that-couples-direct-io-and-**
>> concurrent-io-whats-this-have-**to-do-with-nfs-harken-back-5-**
>> 2-years-to-find-out/<http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/>
>> ),
>> but I'm looking for local FS.
>>
>> Thanks,
>> Da
>>
>
>  ______________________________**_________________
>> Kernelnewbies mailing list
>> Kernelnewbies at kernelnewbies.**org <Kernelnewbies@kernelnewbies.org>
>> http://lists.kernelnewbies.**org/mailman/listinfo/**kernelnewbies<http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies>
>>
>
> XFS locks inode exclusive only if it is an unaligned Direct IO, which is
> apparently done to prevent race conditions -- refer to this
> http://oss.sgi.com/archives/**xfs/2011-01/msg00157.html<http://oss.sgi.com/archives/xfs/2011-01/msg00157.html>Also the behavior of Ext4 under dioread_nolock is supported by XFS by
> default and in a much better way. Also Ext4 is the only one which uses
> DIO_LOCKING while doing direct io.
>
>
>
> Regards,
> --
> Raghavendra Prabhu
> GPG Id : 0xD72BE977
> Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
> www: wnohang.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120115/00c1d48c/attachment.html 

^ permalink raw reply	[flat|nested] 5+ messages in thread
* which local FS supports concurrent direct IO write?
  2012-01-15 20:17   ` Zheng Da
@ 2012-01-15 21:22     ` Raghavendra D Prabhu
  2012-01-15 21:48       ` Zheng Da
  0 siblings, 1 reply; 5+ messages in thread
From: Raghavendra D Prabhu @ 2012-01-15 21:22 UTC (permalink / raw)
  To: kernelnewbies

Hi Zheng,

Interesting analysis. 

* On Sun, Jan 15, 2012 at 03:17:12PM -0500, Zheng Da <zhengda1936@gmail.com> wrote:
>Thanks. I was reading the code of kernel 3.0. XFS starts to support
>concurrent direct IO since kernel 3.1.5.
>But concurrent direct IO write still doesn't work well in kernel 3.2. 
 From what I have heard it has supported it from sometime back. I 
think you may need to ask in xfs general ML about this.


>I wrote a test program that accesses a 4G file randomly (read and write), and
>I ran it with 8 threads and the machine has 8 cores. It turns out that only
>1 core is running. I'm pretty sure xfs_rw_ilock is locked
>with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write.
>
>lockstat shows me that there is a lot of wait time in ip->i_lock. It seems
>the lock is locked exclusively.
>               &(&ip->i_lock)->mr_lock-W:         31568          36170
>      0.24       20048.25     7589157.99         130154        3146848
>      0.00         217.70     1238310.72
>               &(&ip->i_lock)->mr_lock-R:         11251          11886
>      0.24       20043.01     2895595.18          46671         526309
>      0.00          63.80      264097.96
>               -------------------------
>                 &(&ip->i_lock)->mr_lock          36170
> [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]
>                 &(&ip->i_lock)->mr_lock          11886
> [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]
>               -------------------------
>                 &(&ip->i_lock)->mr_lock          38555
> [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]
>                 &(&ip->i_lock)->mr_lock           9501
> [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]
>
>Then I used systemtap to instrument xfs_ilock and find there are at least 3
>functions that lock ip->i_lock exclusively during write.

 From what I saw in xfs_file_dio_aio_write code, it uses EXCL only 
if there is unaligned IO   or there are cached pages to be 
invalidated after shared lock is obtained *but* it demotes that 
lock to SHARED just before generic_file_direct_write. 

>Is there any popular FS that supports concurrent direct IO well?
>
>Thanks,
>Da
>
>On Sat, Jan 14, 2012 at 6:45 AM, Raghavendra D Prabhu <
>raghu.prabhu13 at gmail.com> wrote:
>
>> Hi Zheng,


>> * On Fri, Jan 13, 2012 at 04:41:16PM -0500, Zheng Da <
>> zhengda1936 at gmail.com> wrote:

>>> Hello,

>>> I'm looking for a FS in Linux that supports concurrent direct IO write.
>>> ext4 supports concurrent direct IO read if we mount it with
>>> dioread_nolock,
>>> but doesn't support concurrent writes. XFS doesn't support concurrent
>>> direct IO at all. It locks the inode exclusive if it's direct IO. I tried
>>> btrfs, and it seems it doesn't support concurrent direct IO either though
>>> I
>>> haven't looked into its code.
>>> Is there a local FS that support concurrent direct IO write? It seems NFS
>>> supports it (
>>> http://kevinclosson.wordpress.**com/2011/08/12/file-systems-**
>>> for-a-database-choose-one-**that-couples-direct-io-and-**
>>> concurrent-io-whats-this-have-**to-do-with-nfs-harken-back-5-**
>>> 2-years-to-find-out/<http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/>
>>> ),
>>> but I'm looking for local FS.

>>> Thanks,
>>> Da


>>  ______________________________**_________________
>>> Kernelnewbies mailing list
>>> Kernelnewbies at kernelnewbies.**org <Kernelnewbies@kernelnewbies.org>
>>> http://lists.kernelnewbies.**org/mailman/listinfo/**kernelnewbies<http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies>


>> XFS locks inode exclusive only if it is an unaligned Direct IO, which is
>> apparently done to prevent race conditions -- refer to this
>> http://oss.sgi.com/archives/**xfs/2011-01/msg00157.html<http://oss.sgi.com/archives/xfs/2011-01/msg00157.html>Also the behavior of Ext4 under dioread_nolock is supported by XFS by
>> default and in a much better way. Also Ext4 is the only one which uses
>> DIO_LOCKING while doing direct io.



>> Regards,
>> --
>> Raghavendra Prabhu
>> GPG Id : 0xD72BE977
>> Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
>> www: wnohang.net




Regards,
-- 
Raghavendra Prabhu
GPG Id : 0xD72BE977
Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
www: wnohang.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120116/80d92106/attachment.bin 

^ permalink raw reply	[flat|nested] 5+ messages in thread
* which local FS supports concurrent direct IO write?
  2012-01-15 21:22     ` Raghavendra D Prabhu
@ 2012-01-15 21:48       ` Zheng Da
  0 siblings, 0 replies; 5+ messages in thread
From: Zheng Da @ 2012-01-15 21:48 UTC (permalink / raw)
  To: kernelnewbies

On Sun, Jan 15, 2012 at 4:22 PM, Raghavendra D Prabhu <
raghu.prabhu13@gmail.com> wrote:

> Hi Zheng,
>
> Interesting analysis.
> * On Sun, Jan 15, 2012 at 03:17:12PM -0500, Zheng Da <
> zhengda1936 at gmail.com> wrote:
>
>> Thanks. I was reading the code of kernel 3.0. XFS starts to support
>> concurrent direct IO since kernel 3.1.5.
>> But concurrent direct IO write still doesn't work well in kernel 3.2.
>>
> From what I have heard it has supported it from sometime back. I think you
> may need to ask in xfs general ML about this.

I didn't know this ML. I'll ask them for help.

>
>  I wrote a test program that accesses a 4G file randomly (read and write),
>> and
>> I ran it with 8 threads and the machine has 8 cores. It turns out that
>> only
>> 1 core is running. I'm pretty sure xfs_rw_ilock is locked
>> with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write.
>>
>> lockstat shows me that there is a lot of wait time in ip->i_lock. It seems
>> the lock is locked exclusively.
>>              &(&ip->i_lock)->mr_lock-W:         31568          36170
>>     0.24       20048.25     7589157.99         130154        3146848
>>     0.00         217.70     1238310.72
>>              &(&ip->i_lock)->mr_lock-R:         11251          11886
>>     0.24       20043.01     2895595.18          46671         526309
>>     0.00          63.80      264097.96
>>              -------------------------
>>                &(&ip->i_lock)->mr_lock          36170
>> [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]
>>                &(&ip->i_lock)->mr_lock          11886
>> [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]
>>              -------------------------
>>                &(&ip->i_lock)->mr_lock          38555
>> [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]
>>                &(&ip->i_lock)->mr_lock           9501
>> [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]
>>
>> Then I used systemtap to instrument xfs_ilock and find there are at least
>> 3
>> functions that lock ip->i_lock exclusively during write.
>>
>
> From what I saw in xfs_file_dio_aio_write code, it uses EXCL only if there
> is unaligned IO   or there are cached pages to be invalidated after shared
> lock is obtained *but* it demotes that lock to SHARED just before
> generic_file_direct_write.

 Actually, there are two locks for an inode, i_lock and i_iolock. systemtap
shows me that i_iolock is already locked to SHARED, but i_lock is locked
exclusively somewhere else. Even though I don't think I have found the
right spot that hurts concurrency so badly.

Thanks,
Da
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120115/8c2ff3e1/attachment.html 

^ permalink raw reply	[flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-01-15 21:48 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-13 21:41 which local FS supports concurrent direct IO write? Zheng Da
2012-01-14 11:45 ` Raghavendra D Prabhu
2012-01-15 20:17   ` Zheng Da
2012-01-15 21:22     ` Raghavendra D Prabhu
2012-01-15 21:48       ` Zheng Da

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).