issues with project quota over nfs

issues with project quota over nfs

[brian wince]

[-- Attachment #1.1: Type: text/plain, Size: 774 bytes --]

All,
We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
We are using project quota's on directories within this partition and exporting this.
When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.

Is this a known issue and if so is there anything that can be done to resolve this.

We do not see this issue if on the locally mounted partition. This only happens over NFS.

Please let me know if you need more information or if this should be addressed on a different list.

TIA,

Brian


      

[-- Attachment #1.2: Type: text/html, Size: 1143 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: issues with project quota over nfs

[Greg Banks]

brian wince wrote:
> All,
> We have a Quad-core server running gentoo 2.6.27-r7 that is used as an
> nfs server for a xfs partition.
> We are using project quota's on directories within this partition and
> exporting this.
> When a client tries to untar in the dir with project quotas set and
> reaches the quota limit the load on the NFS server goes up
> dramatically and stays like this for a few minutes after the tar has
> gone through trying to write all of it files and failing due to quota
> limit exceeded.
>  
> Is this a known issue and if so is there anything that can be done to
> resolve this.

You probably want to update the client kernels to have this commit:

http://git.kernel.org/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Flinux-2.6.git&a=commit&h=7b159fc18d417980f57aef64cab3417ee6af70f8
>  
> We do not see this issue if on the locally mounted partition. This
> only happens over NFS.
>  
> Please let me know if you need more information or if this should be
> addressed on a different list.

You should ask NFS questions on the NFS list, linux-nfs@vger.kernel.org.

-- 
Greg Banks, P.Engineer, SGI Australian Software Group.
the brightly coloured sporks of revolution.
I don't speak for SGI.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

issues with project quota over nfs

[Brian Wince]

All,
We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
We are using project quota's on directories within this partition and exporting this.
When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.
 
Is this a known issue and if so is there anything that can be done to resolve this.
 
We do not see this issue if on the locally mounted partition. This only happens over NFS.
 
Please let me know if you need more information or if this should be addressed on a different list.
 
TIA,

Brian

Re: issues with project quota over nfs

[J. Bruce Fields]

On Fri, Mar 13, 2009 at 09:59:26AM -0700, Brian Wince wrote:
> All,
> We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
> We are using project quota's on directories within this partition and exporting this.
> When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.
>  
> Is this a known issue and if so is there anything that can be done to resolve this.
>  
> We do not see this issue if on the locally mounted partition. This only happens over NFS.
>  
> Please let me know if you need more information or if this should be addressed on a different list.

It doesn't sounds familiar to me.  Might also be worth cc:'ing xfs
people.

--b.

Re: issues with project quota over nfs

[Greg Banks]

J. Bruce Fields wrote:
> On Fri, Mar 13, 2009 at 09:59:26AM -0700, Brian Wince wrote:
>   
>> All,
>> We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
>> We are using project quota's on directories within this partition and exporting this.
>> When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.
>>  
>> Is this a known issue and if so is there anything that can be done to resolve this.
>>  
>> We do not see this issue if on the locally mounted partition. This only happens over NFS.
>>  
>> Please let me know if you need more information or if this should be addressed on a different list.
>>     
>
> It doesn't sounds familiar to me.  Might also be worth cc:'ing xfs
> people.
>
>
>   
Oh dear.  Brian asked it there frst, and I redirected him here.  In the
absence of network traces, it smelt to me like the old client problem
where errors like ENOSPC or EDQUOT during writeback were only returned
to the app on close() and not the next write(), thus causing clients
which exceed  quota during long streaming writes pointlessly to send a
lot of data to the server all of which is dropped on the server.

-- 
Greg Banks, P.Engineer, SGI Australian Software Group.
the brightly coloured sporks of revolution.
I don't speak for SGI.

Re: issues with project quota over nfs

[Trond Myklebust]

On Mon, 2009-03-16 at 14:03 +1100, Greg Banks wrote:
> J. Bruce Fields wrote:
> > On Fri, Mar 13, 2009 at 09:59:26AM -0700, Brian Wince wrote:
> >   
> >> All,
> >> We have a Quad-core server running gentoo 2.6.27-r7 that is used as an nfs server for a xfs partition.
> >> We are using project quota's on directories within this partition and exporting this.
> >> When a client tries to untar in the dir with project quotas set and reaches the quota limit the load on the NFS server goes up dramatically and stays like this for a few minutes after the tar has gone through trying to write all of it files and failing due to quota limit exceeded.
> >>  
> >> Is this a known issue and if so is there anything that can be done to resolve this.
> >>  
> >> We do not see this issue if on the locally mounted partition. This only happens over NFS.
> >>  
> >> Please let me know if you need more information or if this should be addressed on a different list.
> >>     
> >
> > It doesn't sounds familiar to me.  Might also be worth cc:'ing xfs
> > people.
> >
> >
> >   
> Oh dear.  Brian asked it there frst, and I redirected him here.  In the
> absence of network traces, it smelt to me like the old client problem
> where errors like ENOSPC or EDQUOT during writeback were only returned
> to the app on close() and not the next write(), thus causing clients
> which exceed  quota during long streaming writes pointlessly to send a
> lot of data to the server all of which is dropped on the server.
> 

The patch "NFS: Fall back to synchronous writes when a background write
errors..."
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git&a=commitdiff&h=7b159fc18d417980f57aef64cab3417ee6af70f8
was merged prior to 2.6.27.

That helps catch the problem earlier on the client, but IIRC XFS had
(has?) some kind of advanced error recovery routines that exacerbated
the problem on the server side by taking forever to figure out that this
was an irrecoverable error.

Cheers
  Trond