From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Wim Colgate" Subject: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 6 Aug 2007 09:02:08 -0700 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0319697139==" To: Return-path: Received: from sc8-sf-mx2-b.sourceforge.net ([10.3.1.92] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II53h-0001l5-Eh for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 09:03:54 -0700 Received: from webmailrdm.xensource.com ([66.228.214.202]) by mail.sourceforge.net with esmtp (Exim 4.44) id 1II53l-0005Jr-B7 for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 09:03:57 -0700 List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net This is a multi-part message in MIME format. --===============0319697139== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C7D843.64A63736" This is a multi-part message in MIME format. ------_=_NextPart_001_01C7D843.64A63736 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, =20 My last question was a bit too nebulous, and didn't really expect an answer. Learning my lesson, I have a specific question. =20 If I have a soft mount, and open a file with O_DIRECT and O_SYNC, should I ever expect a callback (nfs_writeback_done) with a successful task->tk_status (i.e >=3D 0) with the committed state (resp->verf->committed) set to NFS_UNSTABLE?=20 =20 A secondary question: if the above is expected, does this occur because someone is caching the write and is there a mechanism to disable this effect? =20 Regards, =20 Wim ------_=_NextPart_001_01C7D843.64A63736 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi,

My last question was a bit too nebulous, and = didn’t really expect an answer. Learning my lesson, I have a specific = question.

If I have a soft mount, and open a file with O_DIRECT = and O_SYNC, should I ever expect a callback (nfs_writeback_done) with a = successful task->tk_status (i.e >=3D 0) with the committed state = (resp->verf->committed) set to NFS_UNSTABLE?

A secondary question: if the above is expected, does = this occur because someone is caching the write and is there a mechanism to = disable this effect?

Regards,

Wim

------_=_NextPart_001_01C7D843.64A63736-- --===============0319697139== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ --===============0319697139== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs --===============0319697139==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chuck Lever Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 12:37:48 -0400 Message-ID: <46B74E5C.4060400@oracle.com> References: Reply-To: chuck.lever@oracle.com Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------000608090401060506010306" Cc: nfs@lists.sourceforge.net To: Wim Colgate Return-path: Received: from sc8-sf-mx1-b.sourceforge.net ([10.3.1.91] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II5bo-0006NO-RO for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 09:39:08 -0700 Received: from rgminet01.oracle.com ([148.87.113.118]) by mail.sourceforge.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.44) id 1II5br-00087w-M9 for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 09:39:13 -0700 In-Reply-To: List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net This is a multi-part message in MIME format. --------------000608090401060506010306 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Wim Colgate wrote: > If I have a soft mount, and open a file with O_DIRECT and O_SYNC, should > I ever expect a callback (nfs_writeback_done) with a successful > task->tk_status (i.e >= 0) with the committed state > (resp->verf->committed) set to NFS_UNSTABLE? Yes, this can happen if the server decides to return NFS_UNSTABLE. Rare, but possible. > A secondary question: if the above is expected, does this occur because > someone is caching the write and is there a mechanism to disable this > effect? Servers can return NFS_UNSTABLE to any WRITE request, so I can't think of a way this might be disabled. --------------000608090401060506010306 Content-Type: text/x-vcard; charset=utf-8; name="chuck.lever.vcf" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="chuck.lever.vcf" begin:vcard fn:Chuck Lever n:Lever;Chuck org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff tel;work:+1 248 614 5091 x-mozilla-html:FALSE url:http://oss.oracle.com/~cel version:2.1 end:vcard --------------000608090401060506010306 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ --------------000608090401060506010306 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs --------------000608090401060506010306-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chuck Lever Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 13:10:22 -0400 Message-ID: <46B755FE.1010909@oracle.com> References: <46B74E5C.4060400@oracle.com> Reply-To: chuck.lever@oracle.com Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------010202090605040201030303" Cc: nfs@lists.sourceforge.net To: Wim Colgate Return-path: Received: from sc8-sf-mx2-b.sourceforge.net ([10.3.1.92] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II67S-0001DG-Pm for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 10:11:50 -0700 Received: from rgminet01.oracle.com ([148.87.113.118]) by mail.sourceforge.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.44) id 1II67V-0002WT-Hp for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 10:11:54 -0700 In-Reply-To: <46B74E5C.4060400@oracle.com> List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net This is a multi-part message in MIME format. --------------010202090605040201030303 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Chuck Lever wrote: > Wim Colgate wrote: >> If I have a soft mount, and open a file with O_DIRECT and O_SYNC, >> should I ever expect a callback (nfs_writeback_done) with a successful >> task->tk_status (i.e >= 0) with the committed state >> (resp->verf->committed) set to NFS_UNSTABLE? > > Yes, this can happen if the server decides to return NFS_UNSTABLE. Rare, > but possible. Let me be more clear about this. O_DIRECT and O_SYNC determine client behavior only. However, they don't necessarily force NFS_FILE_SYNC writes all the time. For example, if an application issues a direct write request that is much larger than the current wsize, the Linux NFS client will send the write using NFS_UNSTABLE requests followed by a COMMIT. That makes it easier for the server to schedule disk writes more efficiently. >> A secondary question: if the above is expected, does this occur >> because someone is caching the write and is there a mechanism to >> disable this effect? > > Servers can return NFS_UNSTABLE to any WRITE request, so I can't think > of a way this might be disabled. Even though an NFS client requests an NFS_FILE_SYNC write, the server still has the choice of returning something less, even NFS_UNSTABLE. In general that's a rare occurrence, but is something I've seen in practice. --------------010202090605040201030303 Content-Type: text/x-vcard; charset=utf-8; name="chuck.lever.vcf" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="chuck.lever.vcf" begin:vcard fn:Chuck Lever n:Lever;Chuck org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff tel;work:+1 248 614 5091 x-mozilla-html:FALSE url:http://oss.oracle.com/~cel version:2.1 end:vcard --------------010202090605040201030303 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ --------------010202090605040201030303 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs --------------010202090605040201030303-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Peter Staubach Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 13:33:02 -0400 Message-ID: <46B75B4E.8060000@redhat.com> References: <46B74E5C.4060400@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: nfs@lists.sourceforge.net, Wim Colgate To: chuck.lever@oracle.com Return-path: Received: from sc8-sf-mx1-b.sourceforge.net ([10.3.1.91] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II6S1-0003NK-Os for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 10:33:05 -0700 Received: from mx1.redhat.com ([66.187.233.31]) by mail.sourceforge.net with esmtp (Exim 4.44) id 1II6S5-00063e-Jc for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 10:33:09 -0700 In-Reply-To: <46B74E5C.4060400@oracle.com> List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net Chuck Lever wrote: > Wim Colgate wrote: >> If I have a soft mount, and open a file with O_DIRECT and O_SYNC, >> should I ever expect a callback (nfs_writeback_done) with a >> successful task->tk_status (i.e >= 0) with the committed state >> (resp->verf->committed) set to NFS_UNSTABLE? > > Yes, this can happen if the server decides to return NFS_UNSTABLE. > Rare, but possible. > >> A secondary question: if the above is expected, does this occur >> because someone is caching the write and is there a mechanism to >> disable this effect? > > Servers can return NFS_UNSTABLE to any WRITE request, so I can't think > of a way this might be disabled. Actually, it would be a protocol error for a server to return a commitment level less than was requested by the client. The server can return a greater commitment level, but not less than. ps ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Wim Colgate" Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 6 Aug 2007 10:40:49 -0700 Message-ID: References: <46B75B4E.8060000@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: nfs@lists.sourceforge.net To: "Peter Staubach" , Return-path: Received: from sc8-sf-mx2-b.sourceforge.net ([10.3.1.92] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II6bL-0004O7-Jc for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 10:42:44 -0700 Received: from webmailrdm.xensource.com ([66.228.214.202]) by mail.sourceforge.net with esmtp (Exim 4.44) id 1II6bP-0007fP-Jk for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 10:42:47 -0700 In-Reply-To: <46B75B4E.8060000@redhat.com> List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net Interesting information. Specifically I am trying to inject errors by manually (but politely) bringing the NFS server down then up, then down (rinse and repeat ...) while doing IO from a linux client. As mentioned the open file is O_DIRECT and O_SYNC -- which I thought should mean either the data hits the server's storage or I should get an error; and I'm more than happy to deal with an IO error. I'm confident the writes are less than wsize (4096 bytes to be precise). Is there a 100% guaranteed method to get the behavior I thought O_DIRECT and O_SYNC was providing? Thanks, Wim -----Original Message----- From: Peter Staubach [mailto:staubach@redhat.com] Sent: Monday, August 06, 2007 10:33 AM To: chuck.lever@oracle.com Cc: Wim Colgate; nfs@lists.sourceforge.net Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync. Chuck Lever wrote: > Wim Colgate wrote: >> If I have a soft mount, and open a file with O_DIRECT and O_SYNC, >> should I ever expect a callback (nfs_writeback_done) with a >> successful task->tk_status (i.e >= 0) with the committed state >> (resp->verf->committed) set to NFS_UNSTABLE? > > Yes, this can happen if the server decides to return NFS_UNSTABLE. > Rare, but possible. > >> A secondary question: if the above is expected, does this occur >> because someone is caching the write and is there a mechanism to >> disable this effect? > > Servers can return NFS_UNSTABLE to any WRITE request, so I can't think > of a way this might be disabled. Actually, it would be a protocol error for a server to return a commitment level less than was requested by the client. The server can return a greater commitment level, but not less than. ps ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs From mboxrd@z Thu Jan 1 00:00:00 1970 From: Trond Myklebust Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 14:58:30 -0400 Message-ID: <1186426710.6616.57.camel@localhost> References: <46B74E5C.4060400@oracle.com> <46B755FE.1010909@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: nfs@lists.sourceforge.net, Wim Colgate To: chuck.lever@oracle.com Return-path: Received: from sc8-sf-mx2-b.sourceforge.net ([10.3.1.92] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II7mo-0003Sw-PB for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 11:58:39 -0700 Received: from pat.uio.no ([129.240.10.15] ident=[U2FsdGVkX19KYs08YBljbzjsuZxZb6u+ILhtc57IC2o=]) by mail.sourceforge.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.44) id 1II7mq-00054X-BC for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 11:58:43 -0700 In-Reply-To: <46B755FE.1010909@oracle.com> List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net On Mon, 2007-08-06 at 13:10 -0400, Chuck Lever wrote: > Even though an NFS client requests an NFS_FILE_SYNC write, the server > still has the choice of returning something less, even NFS_UNSTABLE. In > general that's a rare occurrence, but is something I've seen in practice. As Peter said, a server that return anything other than FILE_SYNC to a FILE_SYNC write request would be in clear violation of the description of WRITE semantics on page 51 of RFC1813: committed The server should return an indication of the level of commitment of the data and metadata via committed. If the server committed all data and metadata to stable storage, committed should be set to FILE_SYNC. If the level of commitment was at least as strong as DATA_SYNC, then committed should be set to DATA_SYNC. Otherwise, committed must be returned as UNSTABLE. If stable was FILE_SYNC, then committed must also be FILE_SYNC: anything else constitutes a protocol violation. If stable was DATA_SYNC, then committed may be FILE_SYNC or DATA_SYNC: anything else constitutes a protocol violation. If stable was UNSTABLE, then committed may be either FILE_SYNC, DATA_SYNC, or UNSTABLE. I see no reason why we should care about supporting such a server. Trond ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chuck Lever Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 15:13:15 -0400 Message-ID: <46B772CB.1080906@oracle.com> References: <46B74E5C.4060400@oracle.com> <46B755FE.1010909@oracle.com> <1186426710.6616.57.camel@localhost> Reply-To: chuck.lever@oracle.com Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------040505030806000806040406" Cc: nfs@lists.sourceforge.net, Wim Colgate To: Trond Myklebust Return-path: Received: from sc8-sf-mx2-b.sourceforge.net ([10.3.1.92] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II825-0005Ej-Jz for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:14:25 -0700 Received: from agminet01.oracle.com ([141.146.126.228]) by mail.sourceforge.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.44) id 1II828-0002s0-BB for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:14:29 -0700 In-Reply-To: <1186426710.6616.57.camel@localhost> List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net This is a multi-part message in MIME format. --------------040505030806000806040406 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Trond Myklebust wrote: > On Mon, 2007-08-06 at 13:10 -0400, Chuck Lever wrote: >> Even though an NFS client requests an NFS_FILE_SYNC write, the server >> still has the choice of returning something less, even NFS_UNSTABLE. In >> general that's a rare occurrence, but is something I've seen in practice. > > As Peter said, a server that return anything other than FILE_SYNC to a > FILE_SYNC write request would be in clear violation of the description > of WRITE semantics on page 51 of RFC1813: > > committed > The server should return an indication of the level of > commitment of the data and metadata via committed. If > the server committed all data and metadata to stable > storage, committed should be set to FILE_SYNC. If the > level of commitment was at least as strong as > DATA_SYNC, then committed should be set to DATA_SYNC. > Otherwise, committed must be returned as UNSTABLE. If > stable was FILE_SYNC, then committed must also be > FILE_SYNC: anything else constitutes a protocol > violation. If stable was DATA_SYNC, then committed may > be FILE_SYNC or DATA_SYNC: anything else constitutes a > protocol violation. If stable was UNSTABLE, then > committed may be either FILE_SYNC, DATA_SYNC, or > UNSTABLE. > > I see no reason why we should care about supporting such a server. I said nothing about whether the server should or should not return such a value. I just said that it is a possibility, and that I have observed the behavior in the field. The client, if it is a good implementation, needs to check for this possibility and throw an error in that case. --------------040505030806000806040406 Content-Type: text/x-vcard; charset=utf-8; name="chuck.lever.vcf" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="chuck.lever.vcf" begin:vcard fn:Chuck Lever n:Lever;Chuck org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff tel;work:+1 248 614 5091 x-mozilla-html:FALSE url:http://oss.oracle.com/~cel version:2.1 end:vcard --------------040505030806000806040406 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ --------------040505030806000806040406 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs --------------040505030806000806040406-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chuck Lever Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 15:16:28 -0400 Message-ID: <46B7738C.4020503@oracle.com> References: Reply-To: chuck.lever@oracle.com Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------040909080606070901000405" Cc: nfs@lists.sourceforge.net To: Wim Colgate Return-path: Received: from sc8-sf-mx1-b.sourceforge.net ([10.3.1.91] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II85t-0005h5-C7 for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:18:23 -0700 Received: from agminet01.oracle.com ([141.146.126.228]) by mail.sourceforge.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.44) id 1II85v-0005or-8X for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:18:23 -0700 In-Reply-To: List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net This is a multi-part message in MIME format. --------------040909080606070901000405 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Wim Colgate wrote: > Specifically I am trying to inject errors by manually (but politely) > bringing the NFS server down then up, then down (rinse and repeat ...) > while doing IO from a linux client. As mentioned the open file is > O_DIRECT and O_SYNC -- which I thought should mean either the data hits > the server's storage or I should get an error; and I'm more than happy > to deal with an IO error. > > I'm confident the writes are less than wsize (4096 bytes to be precise). > > > Is there a 100% guaranteed method to get the behavior I thought O_DIRECT > and O_SYNC was providing? What behavior did you expect O_DIRECT + O_SYNC to provide? O_DIRECT means "don't cache data" and O_SYNC means "make sure the data is flushed to the server's disk before each write() system call returns." Technically, you don't need NFS_FILE_SYNC writes to do either of those. Which kernel are you testing? The client's use of NFS_FILE_SYNC writes changed over time. > -----Original Message----- > From: Peter Staubach [mailto:staubach@redhat.com] > Sent: Monday, August 06, 2007 10:33 AM > To: chuck.lever@oracle.com > Cc: Wim Colgate; nfs@lists.sourceforge.net > Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync. > > Chuck Lever wrote: >> Wim Colgate wrote: >>> If I have a soft mount, and open a file with O_DIRECT and O_SYNC, >>> should I ever expect a callback (nfs_writeback_done) with a >>> successful task->tk_status (i.e >= 0) with the committed state >>> (resp->verf->committed) set to NFS_UNSTABLE? >> Yes, this can happen if the server decides to return NFS_UNSTABLE. >> Rare, but possible. >> >>> A secondary question: if the above is expected, does this occur >>> because someone is caching the write and is there a mechanism to >>> disable this effect? >> Servers can return NFS_UNSTABLE to any WRITE request, so I can't think > >> of a way this might be disabled. > > Actually, it would be a protocol error for a server to return > a commitment level less than was requested by the client. The > server can return a greater commitment level, but not less than. > > ps --------------040909080606070901000405 Content-Type: text/x-vcard; charset=utf-8; name="chuck.lever.vcf" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="chuck.lever.vcf" begin:vcard fn:Chuck Lever n:Lever;Chuck org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff tel;work:+1 248 614 5091 x-mozilla-html:FALSE url:http://oss.oracle.com/~cel version:2.1 end:vcard --------------040909080606070901000405 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ --------------040909080606070901000405 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs --------------040909080606070901000405-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Trond Myklebust Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 15:19:42 -0400 Message-ID: <1186427982.6616.67.camel@localhost> References: <46B74E5C.4060400@oracle.com> <46B755FE.1010909@oracle.com> <1186426710.6616.57.camel@localhost> <46B772CB.1080906@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: nfs@lists.sourceforge.net, Wim Colgate To: chuck.lever@oracle.com Return-path: Received: from sc8-sf-mx1-b.sourceforge.net ([10.3.1.91] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II87M-0005sU-FJ for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:19:54 -0700 Received: from pat.uio.no ([129.240.10.15] ident=[U2FsdGVkX19CAZ0/z4tDjitsHqBI3DcLozpZM3okiHA=]) by mail.sourceforge.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.44) id 1II87M-0006kj-IZ for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:19:56 -0700 In-Reply-To: <46B772CB.1080906@oracle.com> List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net On Mon, 2007-08-06 at 15:13 -0400, Chuck Lever wrote: > Trond Myklebust wrote: > > On Mon, 2007-08-06 at 13:10 -0400, Chuck Lever wrote: > >> Even though an NFS client requests an NFS_FILE_SYNC write, the server > >> still has the choice of returning something less, even NFS_UNSTABLE. In > >> general that's a rare occurrence, but is something I've seen in practice. > > > > As Peter said, a server that return anything other than FILE_SYNC to a > > FILE_SYNC write request would be in clear violation of the description > > of WRITE semantics on page 51 of RFC1813: > > > > committed > > The server should return an indication of the level of > > commitment of the data and metadata via committed. If > > the server committed all data and metadata to stable > > storage, committed should be set to FILE_SYNC. If the > > level of commitment was at least as strong as > > DATA_SYNC, then committed should be set to DATA_SYNC. > > Otherwise, committed must be returned as UNSTABLE. If > > stable was FILE_SYNC, then committed must also be > > FILE_SYNC: anything else constitutes a protocol > > violation. If stable was DATA_SYNC, then committed may > > be FILE_SYNC or DATA_SYNC: anything else constitutes a > > protocol violation. If stable was UNSTABLE, then > > committed may be either FILE_SYNC, DATA_SYNC, or > > UNSTABLE. > > > > I see no reason why we should care about supporting such a server. > > I said nothing about whether the server should or should not return such > a value. I just said that it is a possibility, and that I have observed > the behavior in the field. > > The client, if it is a good implementation, needs to check for this > possibility and throw an error in that case. We have never supported servers that blatantly violate the protocol, and I see no reason to be burdening the client with a whole load of checks for server protocol violations either. If you want a tool for testing servers, then use something like pynfs. Trond ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chuck Lever Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 15:42:15 -0400 Message-ID: <46B77997.6000804@oracle.com> References: Reply-To: chuck.lever@oracle.com Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------060600040906000608000201" Cc: nfs@lists.sourceforge.net To: Wim Colgate Return-path: Received: from sc8-sf-mx2-b.sourceforge.net ([10.3.1.92] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II8Tg-0008KZ-Ei for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:42:57 -0700 Received: from agminet01.oracle.com ([141.146.126.228]) by mail.sourceforge.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.44) id 1II8Tj-0004A4-4Q for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:43:00 -0700 In-Reply-To: List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net This is a multi-part message in MIME format. --------------060600040906000608000201 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Wim Colgate wrote: > The linux kernel I was using is 2.6.18-8. > > To be fair, I was not trying to force NFS_FILE_SYNC; to make a long > story short, I started with O_DIRECT (please don't cache data). I moved > to add O_SYNC (don't return until my data is written safely). And when I > couldn't explain why I was missing some data (discrepancy between client > and server), I started investigating what was happening under the hood. In fact O_DIRECT also guarantees that the data is on the server's disk before the write() call returns. In some older versions of the client, O_SYNC forced the direct I/O engine to use NFS_FILE_SYNC writes for everything. I don't think that logic is there any more. But what you describe above is a bug. A network dump would be the next step to understand the true interaction between the client and the server during a server reboot. There were some bugs in the client's direct I/O engine where server reboot recovery might result in data loss. Trond fixed a couple of bugs in this area around 2.6.19 or 20. It would be interesting if you tested a later kernel, just for behavioral comparison. > -----Original Message----- > From: Chuck Lever [mailto:chuck.lever@oracle.com] > Sent: Monday, August 06, 2007 12:16 PM > To: Wim Colgate > Cc: nfs@lists.sourceforge.net > Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync. > > Wim Colgate wrote: >> Specifically I am trying to inject errors by manually (but politely) >> bringing the NFS server down then up, then down (rinse and repeat ...) >> while doing IO from a linux client. As mentioned the open file is >> O_DIRECT and O_SYNC -- which I thought should mean either the data > hits >> the server's storage or I should get an error; and I'm more than happy >> to deal with an IO error. >> >> I'm confident the writes are less than wsize (4096 bytes to be > precise). >> >> Is there a 100% guaranteed method to get the behavior I thought > O_DIRECT >> and O_SYNC was providing? > > What behavior did you expect O_DIRECT + O_SYNC to provide? O_DIRECT > means "don't cache data" and O_SYNC means "make sure the data is flushed > > to the server's disk before each write() system call returns." > Technically, you don't need NFS_FILE_SYNC writes to do either of those. > > Which kernel are you testing? The client's use of NFS_FILE_SYNC writes > changed over time. > >> -----Original Message----- >> From: Peter Staubach [mailto:staubach@redhat.com] >> Sent: Monday, August 06, 2007 10:33 AM >> To: chuck.lever@oracle.com >> Cc: Wim Colgate; nfs@lists.sourceforge.net >> Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync. >> >> Chuck Lever wrote: >>> Wim Colgate wrote: >>>> If I have a soft mount, and open a file with O_DIRECT and O_SYNC, >>>> should I ever expect a callback (nfs_writeback_done) with a >>>> successful task->tk_status (i.e >= 0) with the committed state >>>> (resp->verf->committed) set to NFS_UNSTABLE? >>> Yes, this can happen if the server decides to return NFS_UNSTABLE. >>> Rare, but possible. >>> >>>> A secondary question: if the above is expected, does this occur >>>> because someone is caching the write and is there a mechanism to >>>> disable this effect? >>> Servers can return NFS_UNSTABLE to any WRITE request, so I can't > think >>> of a way this might be disabled. >> Actually, it would be a protocol error for a server to return >> a commitment level less than was requested by the client. The >> server can return a greater commitment level, but not less than. >> >> ps --------------060600040906000608000201 Content-Type: text/x-vcard; charset=utf-8; name="chuck.lever.vcf" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="chuck.lever.vcf" begin:vcard fn:Chuck Lever n:Lever;Chuck org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff tel;work:+1 248 614 5091 x-mozilla-html:FALSE url:http://oss.oracle.com/~cel version:2.1 end:vcard --------------060600040906000608000201 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ --------------060600040906000608000201 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs --------------060600040906000608000201-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chuck Lever Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 06 Aug 2007 15:35:04 -0400 Message-ID: <46B777E8.8080401@oracle.com> References: <46B74E5C.4060400@oracle.com> <46B755FE.1010909@oracle.com> <1186426710.6616.57.camel@localhost> <46B772CB.1080906@oracle.com> <1186427982.6616.67.camel@localhost> Reply-To: chuck.lever@oracle.com Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------060108080706050904070707" Cc: nfs@lists.sourceforge.net, Wim Colgate To: Trond Myklebust Return-path: Received: from sc8-sf-mx1-b.sourceforge.net ([10.3.1.91] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II9Co-0004nA-LB for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 13:29:34 -0700 Received: from agminet02.oracle.com ([141.146.126.229]) by mail.sourceforge.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.44) id 1II9Cr-0005xq-8x for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 13:29:38 -0700 Received: from agminet01.oracle.com (agminet01.oracle.com [141.146.126.228]) by agminet02.oracle.com (Switch-3.2.4/Switch-3.1.7) with ESMTP id l76KT1ga019324 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 6 Aug 2007 15:29:01 -0500 In-Reply-To: <1186427982.6616.67.camel@localhost> List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net This is a multi-part message in MIME format. --------------060108080706050904070707 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Trond Myklebust wrote: >>> I see no reason why we should care about supporting such a server. >> I said nothing about whether the server should or should not return such >> a value. I just said that it is a possibility, and that I have observed >> the behavior in the field. >> >> The client, if it is a good implementation, needs to check for this >> possibility and throw an error in that case. > > We have never supported servers that blatantly violate the protocol, and > I see no reason to be burdening the client with a whole load of checks > for server protocol violations either. Take a look at nfs_writeback_done(). You'll see a specific check for a "known bug in Tru64 < 5.0" that is exactly the check to see if the server is violating this part of the protocol. Tru64 is not the only server where this can happen. And in fact, the behavior *is* actually "supported" by the client -- I believe Linux will attempt to post a COMMIT if the server returns NFS_UNSTABLE to *any* write the client has done. > If you want a tool for testing servers, then use something like pynfs. No argument there. Wim, you can precisely determine the request stream a server sees with a test client such as pynfs. However, it appeared that you were looking at whole system client-server interaction during server instability. --------------060108080706050904070707 Content-Type: text/x-vcard; charset=utf-8; name="chuck.lever.vcf" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="chuck.lever.vcf" begin:vcard fn:Chuck Lever n:Lever;Chuck org:Oracle Corporation;Corporate Architecture: Linux Projects Group adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA title:Principal Member of Staff tel;work:+1 248 614 5091 x-mozilla-html:FALSE url:http://oss.oracle.com/~cel version:2.1 end:vcard --------------060108080706050904070707 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ --------------060108080706050904070707 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs --------------060108080706050904070707-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Wim Colgate" Subject: Re: NFS_UNSTABLE vs. FILE and DATA sync. Date: Mon, 6 Aug 2007 12:33:27 -0700 Message-ID: References: <46B7738C.4020503@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: nfs@lists.sourceforge.net To: Return-path: Received: from sc8-sf-mx2-b.sourceforge.net ([10.3.1.92] helo=mail.sourceforge.net) by sc8-sf-list2-new.sourceforge.net with esmtp (Exim 4.43) id 1II8LR-0007V3-Aj for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:34:25 -0700 Received: from webmailrdm.xensource.com ([66.228.214.202]) by mail.sourceforge.net with esmtp (Exim 4.44) id 1II8LV-0001Yb-7Z for nfs@lists.sourceforge.net; Mon, 06 Aug 2007 12:34:29 -0700 In-Reply-To: <46B7738C.4020503@oracle.com> List-Id: "Discussion of NFS under Linux development, interoperability, and testing." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: nfs-bounces@lists.sourceforge.net Errors-To: nfs-bounces@lists.sourceforge.net Hi Chuck, The linux kernel I was using is 2.6.18-8. To be fair, I was not trying to force NFS_FILE_SYNC; to make a long story short, I started with O_DIRECT (please don't cache data). I moved to add O_SYNC (don't return until my data is written safely). And when I couldn't explain why I was missing some data (discrepancy between client and server), I started investigating what was happening under the hood. I didn't really want to start a controversy -- I just wanted to understand what was happening. My understanding of NFS is fairly pedestrian in that I merely get the big picture. Which is why I posted here. Thanks, Wim -----Original Message----- From: Chuck Lever [mailto:chuck.lever@oracle.com] Sent: Monday, August 06, 2007 12:16 PM To: Wim Colgate Cc: nfs@lists.sourceforge.net Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync. Wim Colgate wrote: > Specifically I am trying to inject errors by manually (but politely) > bringing the NFS server down then up, then down (rinse and repeat ...) > while doing IO from a linux client. As mentioned the open file is > O_DIRECT and O_SYNC -- which I thought should mean either the data hits > the server's storage or I should get an error; and I'm more than happy > to deal with an IO error. > > I'm confident the writes are less than wsize (4096 bytes to be precise). > > > Is there a 100% guaranteed method to get the behavior I thought O_DIRECT > and O_SYNC was providing? What behavior did you expect O_DIRECT + O_SYNC to provide? O_DIRECT means "don't cache data" and O_SYNC means "make sure the data is flushed to the server's disk before each write() system call returns." Technically, you don't need NFS_FILE_SYNC writes to do either of those. Which kernel are you testing? The client's use of NFS_FILE_SYNC writes changed over time. > -----Original Message----- > From: Peter Staubach [mailto:staubach@redhat.com] > Sent: Monday, August 06, 2007 10:33 AM > To: chuck.lever@oracle.com > Cc: Wim Colgate; nfs@lists.sourceforge.net > Subject: Re: [NFS] NFS_UNSTABLE vs. FILE and DATA sync. > > Chuck Lever wrote: >> Wim Colgate wrote: >>> If I have a soft mount, and open a file with O_DIRECT and O_SYNC, >>> should I ever expect a callback (nfs_writeback_done) with a >>> successful task->tk_status (i.e >= 0) with the committed state >>> (resp->verf->committed) set to NFS_UNSTABLE? >> Yes, this can happen if the server decides to return NFS_UNSTABLE. >> Rare, but possible. >> >>> A secondary question: if the above is expected, does this occur >>> because someone is caching the write and is there a mechanism to >>> disable this effect? >> Servers can return NFS_UNSTABLE to any WRITE request, so I can't think > >> of a way this might be disabled. > > Actually, it would be a protocol error for a server to return > a commitment level less than was requested by the client. The > server can return a greater commitment level, but not less than. > > ps ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs