file open for write fails under stress

file open for write fails under stress

[Omry Yadan]

I have an application that open many files for writing very fast, in a
way - it can almost be looked as a file system stress test.


my application is single threaded, and I noticed the following problem
when running a single instance of it:


at some point, after opening (and closing) many files for writing, the
application fails to open a file for writing.

if I sleep for 1 second and try again, it works.


my NFS server runs Debian etch 32 bit, with

nfs-kernel-server and nfs-common (1:1.0.10-6+etch.1).


client is also etch 32.

kernel on both is :2.6.18-6-686 #1 SMP Tue Jun 17 21:31:27 UTC 2008 i686


both machines are quad core.



I tried to mount with tcp and udp, and tried both sync and async export
point.


is this a known issue?

any known workaround that will not require messing up the application
with retry logic?


    Omry.

Re: file open for write fails under stress

[Trond Myklebust]

On Tue, 2008-10-14 at 00:51 +0200, Omry Yadan wrote:
> I have an application that open many files for writing very fast, in a
> way - it can almost be looked as a file system stress test.
> 
> 
> my application is single threaded, and I noticed the following problem
> when running a single instance of it:
> 
> 
> at some point, after opening (and closing) many files for writing, the
> application fails to open a file for writing.
> 
> if I sleep for 1 second and try again, it works.

What version of nfs, filesystem on the server, export options, mount
options,...?

Cheers
  Trond

Re: file open for write fails under stress

[Omry Yadan]

Trond Myklebust wrote:

> On Tue, 2008-10-14 at 00:51 +0200, Omry Yadan wrote:
>   
>> I have an application that open many files for writing very fast, in a
>> way - it can almost be looked as a file system stress test.
>>
>>
>> my application is single threaded, and I noticed the following problem
>> when running a single instance of it:
>>
>>
>> at some point, after opening (and closing) many files for writing, the
>> application fails to open a file for writing.
>>
>> if I sleep for 1 second and try again, it works.
>>     
>
> What version of nfs, filesystem on the server, export options, mount
> options,...?
>
>   

/etc/exports:
/var/vizi 38.99.126.160/28(rw,sync,no_subtree_check)
server file system : ext3 on a raid5 (900gb, 3x500gb hd).
server nfs version: 1.0.10

client mount:
nas1:/var/vizi on /var/vizi type nfs (rw,tcp,addr=38.99.126.163)

Re: file open for write fails under stress

[Trond Myklebust]

On Tue, 2008-10-14 at 15:49 +0200, Omry Yadan wrote:
> Trond Myklebust wrote:
> 
> > On Tue, 2008-10-14 at 00:51 +0200, Omry Yadan wrote:
> >   
> >> I have an application that open many files for writing very fast, in a
> >> way - it can almost be looked as a file system stress test.
> >>
> >>
> >> my application is single threaded, and I noticed the following problem
> >> when running a single instance of it:
> >>
> >>
> >> at some point, after opening (and closing) many files for writing, the
> >> application fails to open a file for writing.
> >>
> >> if I sleep for 1 second and try again, it works.
> >>     
> >
> > What version of nfs, filesystem on the server, export options, mount
> > options,...?
> >
> >   
> 
> /etc/exports:
> /var/vizi 38.99.126.160/28(rw,sync,no_subtree_check)
> server file system : ext3 on a raid5 (900gb, 3x500gb hd).
> server nfs version: 1.0.10

> client mount:
> nas1:/var/vizi on /var/vizi type nfs (rw,tcp,addr=38.99.126.163)

So I assume it is NFS version 3, then?

What error does the application get when the open() fails?

Cheers
  Trond

Re: file open for write fails under stress

[Omry Yadan]

>> /etc/exports:
>> /var/vizi 38.99.126.160/28(rw,sync,no_subtree_check)
>> server file system : ext3 on a raid5 (900gb, 3x500gb hd).
>> server nfs version: 1.0.10
>>     
>
>   
>> client mount:
>> nas1:/var/vizi on /var/vizi type nfs (rw,tcp,addr=38.99.126.163)
>>     
>
> So I assume it is NFS version 3, then?
>
> What error does the application get when the open() fails?
>   
according to nfsstat, yes - it's NFS v3.
after open fails (I use fstream binary_file(name,ios::out|ios::binary) )
the error message returned from strerror_r(errno, buf, 1000) is
something like "Input Output error".

Re: file open for write fails under stress

[J. Bruce Fields]

On Tue, Oct 14, 2008 at 12:51:50AM +0200, Omry Yadan wrote:
> I have an application that open many files for writing very fast, in a
> way - it can almost be looked as a file system stress test.
> 
> 
> my application is single threaded, and I noticed the following problem
> when running a single instance of it:
> 
> 
> at some point, after opening (and closing) many files for writing, the
> application fails to open a file for writing.

Do you know what error exactly the open is failing with?

--b.

> if I sleep for 1 second and try again, it works.
> 
> 
> my NFS server runs Debian etch 32 bit, with
> 
> nfs-kernel-server and nfs-common (1:1.0.10-6+etch.1).
> 
> 
> client is also etch 32.
> 
> kernel on both is :2.6.18-6-686 #1 SMP Tue Jun 17 21:31:27 UTC 2008 i686
> 
> 
> both machines are quad core.
> 
> 
> 
> I tried to mount with tcp and udp, and tried both sync and async export
> point.
> 
> 
> is this a known issue?
> 
> any known workaround that will not require messing up the application
> with retry logic?
> 
> 
>     Omry.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: file open for write fails under stress

[Omry Yadan]

J. Bruce Fields wrote:

> On Tue, Oct 14, 2008 at 12:51:50AM +0200, Omry Yadan wrote:
>   
>> I have an application that open many files for writing very fast, in a
>> way - it can almost be looked as a file system stress test.
>>
>>
>> my application is single threaded, and I noticed the following problem
>> when running a single instance of it:
>>
>>
>> at some point, after opening (and closing) many files for writing, the
>> application fails to open a file for writing.
>>     
>
> Do you know what error exactly the open is failing with?
>   
As I wrote in a separate message to the list:
the error string is "Input Output error".
I can get the exact errno if you like, but it's probably a generic IO
error code.