* which local FS supports concurrent direct IO write? @ 2012-01-13 21:41 Zheng Da 2012-01-14 11:45 ` Raghavendra D Prabhu 0 siblings, 1 reply; 5+ messages in thread From: Zheng Da @ 2012-01-13 21:41 UTC (permalink / raw) To: kernelnewbies Hello, I'm looking for a FS in Linux that supports concurrent direct IO write. ext4 supports concurrent direct IO read if we mount it with dioread_nolock, but doesn't support concurrent writes. XFS doesn't support concurrent direct IO at all. It locks the inode exclusive if it's direct IO. I tried btrfs, and it seems it doesn't support concurrent direct IO either though I haven't looked into its code. Is there a local FS that support concurrent direct IO write? It seems NFS supports it ( http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/), but I'm looking for local FS. Thanks, Da -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120113/50888ba3/attachment.html ^ permalink raw reply [flat|nested] 5+ messages in thread
* which local FS supports concurrent direct IO write? 2012-01-13 21:41 which local FS supports concurrent direct IO write? Zheng Da @ 2012-01-14 11:45 ` Raghavendra D Prabhu 2012-01-15 20:17 ` Zheng Da 0 siblings, 1 reply; 5+ messages in thread From: Raghavendra D Prabhu @ 2012-01-14 11:45 UTC (permalink / raw) To: kernelnewbies Hi Zheng, * On Fri, Jan 13, 2012 at 04:41:16PM -0500, Zheng Da <zhengda1936@gmail.com> wrote: >Hello, > >I'm looking for a FS in Linux that supports concurrent direct IO write. >ext4 supports concurrent direct IO read if we mount it with dioread_nolock, >but doesn't support concurrent writes. XFS doesn't support concurrent >direct IO at all. It locks the inode exclusive if it's direct IO. I tried >btrfs, and it seems it doesn't support concurrent direct IO either though I >haven't looked into its code. >Is there a local FS that support concurrent direct IO write? It seems NFS >supports it ( >http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/), >but I'm looking for local FS. > >Thanks, >Da >_______________________________________________ >Kernelnewbies mailing list >Kernelnewbies at kernelnewbies.org >http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies XFS locks inode exclusive only if it is an unaligned Direct IO, which is apparently done to prevent race conditions -- refer to this http://oss.sgi.com/archives/xfs/2011-01/msg00157.html Also the behavior of Ext4 under dioread_nolock is supported by XFS by default and in a much better way. Also Ext4 is the only one which uses DIO_LOCKING while doing direct io. Regards, -- Raghavendra Prabhu GPG Id : 0xD72BE977 Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977 www: wnohang.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120114/da6fb250/attachment.bin ^ permalink raw reply [flat|nested] 5+ messages in thread
* which local FS supports concurrent direct IO write? 2012-01-14 11:45 ` Raghavendra D Prabhu @ 2012-01-15 20:17 ` Zheng Da 2012-01-15 21:22 ` Raghavendra D Prabhu 0 siblings, 1 reply; 5+ messages in thread From: Zheng Da @ 2012-01-15 20:17 UTC (permalink / raw) To: kernelnewbies Thanks. I was reading the code of kernel 3.0. XFS starts to support concurrent direct IO since kernel 3.1.5. But concurrent direct IO write still doesn't work well in kernel 3.2. I wrote a test program that accesses a 4G file randomly (read and write), and I ran it with 8 threads and the machine has 8 cores. It turns out that only 1 core is running. I'm pretty sure xfs_rw_ilock is locked with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write. lockstat shows me that there is a lot of wait time in ip->i_lock. It seems the lock is locked exclusively. &(&ip->i_lock)->mr_lock-W: 31568 36170 0.24 20048.25 7589157.99 130154 3146848 0.00 217.70 1238310.72 &(&ip->i_lock)->mr_lock-R: 11251 11886 0.24 20043.01 2895595.18 46671 526309 0.00 63.80 264097.96 ------------------------- &(&ip->i_lock)->mr_lock 36170 [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs] &(&ip->i_lock)->mr_lock 11886 [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs] ------------------------- &(&ip->i_lock)->mr_lock 38555 [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs] &(&ip->i_lock)->mr_lock 9501 [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs] Then I used systemtap to instrument xfs_ilock and find there are at least 3 functions that lock ip->i_lock exclusively during write. Is there any popular FS that supports concurrent direct IO well? Thanks, Da On Sat, Jan 14, 2012 at 6:45 AM, Raghavendra D Prabhu < raghu.prabhu13@gmail.com> wrote: > Hi Zheng, > > > * On Fri, Jan 13, 2012 at 04:41:16PM -0500, Zheng Da < > zhengda1936 at gmail.com> wrote: > >> Hello, >> >> I'm looking for a FS in Linux that supports concurrent direct IO write. >> ext4 supports concurrent direct IO read if we mount it with >> dioread_nolock, >> but doesn't support concurrent writes. XFS doesn't support concurrent >> direct IO at all. It locks the inode exclusive if it's direct IO. I tried >> btrfs, and it seems it doesn't support concurrent direct IO either though >> I >> haven't looked into its code. >> Is there a local FS that support concurrent direct IO write? It seems NFS >> supports it ( >> http://kevinclosson.wordpress.**com/2011/08/12/file-systems-** >> for-a-database-choose-one-**that-couples-direct-io-and-** >> concurrent-io-whats-this-have-**to-do-with-nfs-harken-back-5-** >> 2-years-to-find-out/<http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/> >> ), >> but I'm looking for local FS. >> >> Thanks, >> Da >> > > ______________________________**_________________ >> Kernelnewbies mailing list >> Kernelnewbies at kernelnewbies.**org <Kernelnewbies@kernelnewbies.org> >> http://lists.kernelnewbies.**org/mailman/listinfo/**kernelnewbies<http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies> >> > > XFS locks inode exclusive only if it is an unaligned Direct IO, which is > apparently done to prevent race conditions -- refer to this > http://oss.sgi.com/archives/**xfs/2011-01/msg00157.html<http://oss.sgi.com/archives/xfs/2011-01/msg00157.html>Also the behavior of Ext4 under dioread_nolock is supported by XFS by > default and in a much better way. Also Ext4 is the only one which uses > DIO_LOCKING while doing direct io. > > > > Regards, > -- > Raghavendra Prabhu > GPG Id : 0xD72BE977 > Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977 > www: wnohang.net > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120115/00c1d48c/attachment.html ^ permalink raw reply [flat|nested] 5+ messages in thread
* which local FS supports concurrent direct IO write? 2012-01-15 20:17 ` Zheng Da @ 2012-01-15 21:22 ` Raghavendra D Prabhu 2012-01-15 21:48 ` Zheng Da 0 siblings, 1 reply; 5+ messages in thread From: Raghavendra D Prabhu @ 2012-01-15 21:22 UTC (permalink / raw) To: kernelnewbies Hi Zheng, Interesting analysis. * On Sun, Jan 15, 2012 at 03:17:12PM -0500, Zheng Da <zhengda1936@gmail.com> wrote: >Thanks. I was reading the code of kernel 3.0. XFS starts to support >concurrent direct IO since kernel 3.1.5. >But concurrent direct IO write still doesn't work well in kernel 3.2. From what I have heard it has supported it from sometime back. I think you may need to ask in xfs general ML about this. >I wrote a test program that accesses a 4G file randomly (read and write), and >I ran it with 8 threads and the machine has 8 cores. It turns out that only >1 core is running. I'm pretty sure xfs_rw_ilock is locked >with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write. > >lockstat shows me that there is a lot of wait time in ip->i_lock. It seems >the lock is locked exclusively. > &(&ip->i_lock)->mr_lock-W: 31568 36170 > 0.24 20048.25 7589157.99 130154 3146848 > 0.00 217.70 1238310.72 > &(&ip->i_lock)->mr_lock-R: 11251 11886 > 0.24 20043.01 2895595.18 46671 526309 > 0.00 63.80 264097.96 > ------------------------- > &(&ip->i_lock)->mr_lock 36170 > [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs] > &(&ip->i_lock)->mr_lock 11886 > [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs] > ------------------------- > &(&ip->i_lock)->mr_lock 38555 > [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs] > &(&ip->i_lock)->mr_lock 9501 > [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs] > >Then I used systemtap to instrument xfs_ilock and find there are at least 3 >functions that lock ip->i_lock exclusively during write. From what I saw in xfs_file_dio_aio_write code, it uses EXCL only if there is unaligned IO or there are cached pages to be invalidated after shared lock is obtained *but* it demotes that lock to SHARED just before generic_file_direct_write. >Is there any popular FS that supports concurrent direct IO well? > >Thanks, >Da > >On Sat, Jan 14, 2012 at 6:45 AM, Raghavendra D Prabhu < >raghu.prabhu13 at gmail.com> wrote: > >> Hi Zheng, >> * On Fri, Jan 13, 2012 at 04:41:16PM -0500, Zheng Da < >> zhengda1936 at gmail.com> wrote: >>> Hello, >>> I'm looking for a FS in Linux that supports concurrent direct IO write. >>> ext4 supports concurrent direct IO read if we mount it with >>> dioread_nolock, >>> but doesn't support concurrent writes. XFS doesn't support concurrent >>> direct IO at all. It locks the inode exclusive if it's direct IO. I tried >>> btrfs, and it seems it doesn't support concurrent direct IO either though >>> I >>> haven't looked into its code. >>> Is there a local FS that support concurrent direct IO write? It seems NFS >>> supports it ( >>> http://kevinclosson.wordpress.**com/2011/08/12/file-systems-** >>> for-a-database-choose-one-**that-couples-direct-io-and-** >>> concurrent-io-whats-this-have-**to-do-with-nfs-harken-back-5-** >>> 2-years-to-find-out/<http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/> >>> ), >>> but I'm looking for local FS. >>> Thanks, >>> Da >> ______________________________**_________________ >>> Kernelnewbies mailing list >>> Kernelnewbies at kernelnewbies.**org <Kernelnewbies@kernelnewbies.org> >>> http://lists.kernelnewbies.**org/mailman/listinfo/**kernelnewbies<http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies> >> XFS locks inode exclusive only if it is an unaligned Direct IO, which is >> apparently done to prevent race conditions -- refer to this >> http://oss.sgi.com/archives/**xfs/2011-01/msg00157.html<http://oss.sgi.com/archives/xfs/2011-01/msg00157.html>Also the behavior of Ext4 under dioread_nolock is supported by XFS by >> default and in a much better way. Also Ext4 is the only one which uses >> DIO_LOCKING while doing direct io. >> Regards, >> -- >> Raghavendra Prabhu >> GPG Id : 0xD72BE977 >> Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977 >> www: wnohang.net Regards, -- Raghavendra Prabhu GPG Id : 0xD72BE977 Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977 www: wnohang.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120116/80d92106/attachment.bin ^ permalink raw reply [flat|nested] 5+ messages in thread
* which local FS supports concurrent direct IO write? 2012-01-15 21:22 ` Raghavendra D Prabhu @ 2012-01-15 21:48 ` Zheng Da 0 siblings, 0 replies; 5+ messages in thread From: Zheng Da @ 2012-01-15 21:48 UTC (permalink / raw) To: kernelnewbies On Sun, Jan 15, 2012 at 4:22 PM, Raghavendra D Prabhu < raghu.prabhu13@gmail.com> wrote: > Hi Zheng, > > Interesting analysis. > * On Sun, Jan 15, 2012 at 03:17:12PM -0500, Zheng Da < > zhengda1936 at gmail.com> wrote: > >> Thanks. I was reading the code of kernel 3.0. XFS starts to support >> concurrent direct IO since kernel 3.1.5. >> But concurrent direct IO write still doesn't work well in kernel 3.2. >> > From what I have heard it has supported it from sometime back. I think you > may need to ask in xfs general ML about this. I didn't know this ML. I'll ask them for help. > > I wrote a test program that accesses a 4G file randomly (read and write), >> and >> I ran it with 8 threads and the machine has 8 cores. It turns out that >> only >> 1 core is running. I'm pretty sure xfs_rw_ilock is locked >> with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write. >> >> lockstat shows me that there is a lot of wait time in ip->i_lock. It seems >> the lock is locked exclusively. >> &(&ip->i_lock)->mr_lock-W: 31568 36170 >> 0.24 20048.25 7589157.99 130154 3146848 >> 0.00 217.70 1238310.72 >> &(&ip->i_lock)->mr_lock-R: 11251 11886 >> 0.24 20043.01 2895595.18 46671 526309 >> 0.00 63.80 264097.96 >> ------------------------- >> &(&ip->i_lock)->mr_lock 36170 >> [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs] >> &(&ip->i_lock)->mr_lock 11886 >> [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs] >> ------------------------- >> &(&ip->i_lock)->mr_lock 38555 >> [<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs] >> &(&ip->i_lock)->mr_lock 9501 >> [<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs] >> >> Then I used systemtap to instrument xfs_ilock and find there are at least >> 3 >> functions that lock ip->i_lock exclusively during write. >> > > From what I saw in xfs_file_dio_aio_write code, it uses EXCL only if there > is unaligned IO or there are cached pages to be invalidated after shared > lock is obtained *but* it demotes that lock to SHARED just before > generic_file_direct_write. Actually, there are two locks for an inode, i_lock and i_iolock. systemtap shows me that i_iolock is already locked to SHARED, but i_lock is locked exclusively somewhere else. Even though I don't think I have found the right spot that hurts concurrency so badly. Thanks, Da -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120115/8c2ff3e1/attachment.html ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2012-01-15 21:48 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-01-13 21:41 which local FS supports concurrent direct IO write? Zheng Da 2012-01-14 11:45 ` Raghavendra D Prabhu 2012-01-15 20:17 ` Zheng Da 2012-01-15 21:22 ` Raghavendra D Prabhu 2012-01-15 21:48 ` Zheng Da
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).