Apparent race condition w/ NFSv3 mktemp+write+rename

Apparent race condition w/ NFSv3 mktemp+write+rename

[Charles Duffy]

Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) 
NFSv3 client and server, I've experienced the following sporadic issue:

function atomic_write() {
     local filename="$1"
     local tempfile
     tempfile="$(mktemp "${filename}.XXXXXX")"
     cat >"${tempfile}" && mv "${tempfile}" "${filename}"
}

$ atomic_write /mnt/foobar <<<"baz"
mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or 
directory

Are my assumptions regarding available semantics fair and reasonable?

Re: Apparent race condition w/ NFSv3 mktemp+write+rename

[Jeff Layton]

On Tue, 24 Nov 2009 13:26:44 -0600
Charles Duffy <charles-7s4ginpmXnFeoWH0uzbU5w@public.gmane.org> wrote:

> Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) 
> NFSv3 client and server, I've experienced the following sporadic issue:
> 
> function atomic_write() {
>      local filename="$1"
>      local tempfile
>      tempfile="$(mktemp "${filename}.XXXXXX")"
>      cat >"${tempfile}" && mv "${tempfile}" "${filename}"
> }
> 
> $ atomic_write /mnt/foobar <<<"baz"
> mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or 
> directory
> 
> Are my assumptions regarding available semantics fair and reasonable?
> 

Known problem with O_EXCL creates. Fixed in RHEL fairly recently. You
probably want a newer kernel.

-- 
Jeff Layton <jlayton@redhat.com>

Re: Apparent race condition w/ NFSv3 mktemp+write+rename

[Peter Staubach]

Charles Duffy wrote:
> Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5)
> NFSv3 client and server, I've experienced the following sporadic issue:
> 
> function atomic_write() {
>     local filename="$1"
>     local tempfile
>     tempfile="$(mktemp "${filename}.XXXXXX")"
>     cat >"${tempfile}" && mv "${tempfile}" "${filename}"
> }
> 
> $ atomic_write /mnt/foobar <<<"baz"
> mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or
> directory
> 
> Are my assumptions regarding available semantics fair and reasonable?
> 

At the end, does /mnt/foobar exist and contain the string, "baz"?

What mount options are being used on the client?

It sounds to me as if the duplicate request cache on the server
is not working, perhaps because entries are being recycled too
quickly, and does not catch a retransmitted RENAME operation.

	Thanx...

		ps

Re: Apparent race condition w/ NFSv3 mktemp+write+rename

[Jeff Layton]

On Tue, 24 Nov 2009 15:03:01 -0500
Jeff Layton <jlayton@redhat.com> wrote:

> On Tue, 24 Nov 2009 13:26:44 -0600
> Charles Duffy <charles@dyfis.net> wrote:
> 
> > Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) 
> > NFSv3 client and server, I've experienced the following sporadic issue:
> > 
> > function atomic_write() {
> >      local filename="$1"
> >      local tempfile
> >      tempfile="$(mktemp "${filename}.XXXXXX")"
> >      cat >"${tempfile}" && mv "${tempfile}" "${filename}"
> > }
> > 
> > $ atomic_write /mnt/foobar <<<"baz"
> > mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or 
> > directory
> > 
> > Are my assumptions regarding available semantics fair and reasonable?
> > 
> 
> Known problem with O_EXCL creates. Fixed in RHEL fairly recently. You
> probably want a newer kernel.
> 

My mistake...I didn't read the version closely enough. I was thinking
of another issue. I'm not sure if a newer kernel will help you.

-- 
Jeff Layton <jlayton@redhat.com>

Re: Apparent race condition w/ NFSv3 mktemp+write+rename

[Charles Duffy]

Peter Staubach wrote:
> At the end, does /mnt/foobar exist and contain the string, "baz"?
>   
I'll follow up with that information with that after the issue is next 
reproduced. (The mv operation is replacing a prior instance of 
/mnt/foobar, so content will need to be inspected to determine whether 
it was successful).

> What mount options are being used on the client?
>   
rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=<redacted>

Re: Apparent race condition w/ NFSv3 mktemp+write+rename

[Peter Staubach]

Charles Duffy wrote:
> Peter Staubach wrote:
>> At the end, does /mnt/foobar exist and contain the string, "baz"?
>>   
> I'll follow up with that information with that after the issue is next
> reproduced. (The mv operation is replacing a prior instance of
> /mnt/foobar, so content will need to be inspected to determine whether
> it was successful).
> 
>> What mount options are being used on the client?
>>   
> rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=<redacted>
> 

Hmmm.  This appears to be the defaults.

Does this take a very long time to run when it fails like this?

	Thanx...

		ps

Re: Apparent race condition w/ NFSv3 mktemp+write+rename

[Charles Duffy]

Peter Staubach wrote:
> Does this take a very long time to run when it fails like this?
>   
Total runtime for script invocations where the failure occurred has in 
all cases (two today -- one at 7:59AM, the other at 12:08PM) been less 
than three seconds.

Re: Apparent race condition w/ NFSv3 mktemp+write+rename

[Peter Staubach]

Charles Duffy wrote:
> Peter Staubach wrote:
>> Does this take a very long time to run when it fails like this?
>>   
> Total runtime for script invocations where the failure occurred has in
> all cases (two today -- one at 7:59AM, the other at 12:08PM) been less
> than three seconds.

Interesting.  And non-deterministic, I'll bet?

	Thanx...

		ps