From mboxrd@z Thu Jan 1 00:00:00 1970 From: raghu.prabhu13@gmail.com (Raghavendra D Prabhu) Date: Mon, 16 Jan 2012 02:52:23 +0530 Subject: which local FS supports concurrent direct IO write? In-Reply-To: References: <20120114114520.GB4192@Xye> Message-ID: <20120115212223.GA59459@Xye> To: kernelnewbies@lists.kernelnewbies.org List-Id: kernelnewbies.lists.kernelnewbies.org Hi Zheng, Interesting analysis. * On Sun, Jan 15, 2012 at 03:17:12PM -0500, Zheng Da wrote: >Thanks. I was reading the code of kernel 3.0. XFS starts to support >concurrent direct IO since kernel 3.1.5. >But concurrent direct IO write still doesn't work well in kernel 3.2. From what I have heard it has supported it from sometime back. I think you may need to ask in xfs general ML about this. >I wrote a test program that accesses a 4G file randomly (read and write), and >I ran it with 8 threads and the machine has 8 cores. It turns out that only >1 core is running. I'm pretty sure xfs_rw_ilock is locked >with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write. > >lockstat shows me that there is a lot of wait time in ip->i_lock. It seems >the lock is locked exclusively. > &(&ip->i_lock)->mr_lock-W: 31568 36170 > 0.24 20048.25 7589157.99 130154 3146848 > 0.00 217.70 1238310.72 > &(&ip->i_lock)->mr_lock-R: 11251 11886 > 0.24 20043.01 2895595.18 46671 526309 > 0.00 63.80 264097.96 > ------------------------- > &(&ip->i_lock)->mr_lock 36170 > [] xfs_ilock+0xb2/0x110 [xfs] > &(&ip->i_lock)->mr_lock 11886 > [] xfs_ilock+0xea/0x110 [xfs] > ------------------------- > &(&ip->i_lock)->mr_lock 38555 > [] xfs_ilock+0xb2/0x110 [xfs] > &(&ip->i_lock)->mr_lock 9501 > [] xfs_ilock+0xea/0x110 [xfs] > >Then I used systemtap to instrument xfs_ilock and find there are at least 3 >functions that lock ip->i_lock exclusively during write. From what I saw in xfs_file_dio_aio_write code, it uses EXCL only if there is unaligned IO or there are cached pages to be invalidated after shared lock is obtained *but* it demotes that lock to SHARED just before generic_file_direct_write. >Is there any popular FS that supports concurrent direct IO well? > >Thanks, >Da > >On Sat, Jan 14, 2012 at 6:45 AM, Raghavendra D Prabhu < >raghu.prabhu13 at gmail.com> wrote: > >> Hi Zheng, >> * On Fri, Jan 13, 2012 at 04:41:16PM -0500, Zheng Da < >> zhengda1936 at gmail.com> wrote: >>> Hello, >>> I'm looking for a FS in Linux that supports concurrent direct IO write. >>> ext4 supports concurrent direct IO read if we mount it with >>> dioread_nolock, >>> but doesn't support concurrent writes. XFS doesn't support concurrent >>> direct IO at all. It locks the inode exclusive if it's direct IO. I tried >>> btrfs, and it seems it doesn't support concurrent direct IO either though >>> I >>> haven't looked into its code. >>> Is there a local FS that support concurrent direct IO write? It seems NFS >>> supports it ( >>> http://kevinclosson.wordpress.**com/2011/08/12/file-systems-** >>> for-a-database-choose-one-**that-couples-direct-io-and-** >>> concurrent-io-whats-this-have-**to-do-with-nfs-harken-back-5-** >>> 2-years-to-find-out/ >>> ), >>> but I'm looking for local FS. >>> Thanks, >>> Da >> ______________________________**_________________ >>> Kernelnewbies mailing list >>> Kernelnewbies at kernelnewbies.**org >>> http://lists.kernelnewbies.**org/mailman/listinfo/**kernelnewbies >> XFS locks inode exclusive only if it is an unaligned Direct IO, which is >> apparently done to prevent race conditions -- refer to this >> http://oss.sgi.com/archives/**xfs/2011-01/msg00157.htmlAlso the behavior of Ext4 under dioread_nolock is supported by XFS by >> default and in a much better way. Also Ext4 is the only one which uses >> DIO_LOCKING while doing direct io. >> Regards, >> -- >> Raghavendra Prabhu >> GPG Id : 0xD72BE977 >> Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977 >> www: wnohang.net Regards, -- Raghavendra Prabhu GPG Id : 0xD72BE977 Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977 www: wnohang.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 490 bytes Desc: not available Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120116/80d92106/attachment.bin