umount fails with device busy

umount fails with device busy

[z00t]

Hi guys,

I get the device busy error when I try to umount an NFS file system althoug=
h there is no process with an open file on it (at least fuser -m /mnt/point=
 and lsof +f -- /mnt/point don't find anything).

To be more precise this only happens, when I run the umount right after sto=
pping an application which has its executables and work directory on this f=
ile system. Running umount again some seconds later works fine. =


I do not have any submounts under this mount point. There are no packet tra=
nsmit or receive errors nor any NFS/RPC errors only a very small number of =
RPC retransmitts. So it seems that I have a stable environment. Client and =
server is SLES9 with 2.6.5-7.202.7-smp/X86_64. =


I enabled NFS debug output and the only noticeable thing is, that some NFS =
operations are still ongoing after the application processes have already t=
erminated:

  Aug 23 08:25:23 marvin kernel: NFS reply getattr
  Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 ct=3D1 i=
nfo=3D0x6)
  Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation complete
  Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
  Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
  Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
  Aug 23 08:25:23 marvin kernel: NFS call  getattr
  Aug 23 08:25:23 marvin kernel: NFS reply getattr
  Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=3D1 info=
=3D0x6)
  Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
  Aug 23 08:25:25 marvin kernel: NFS call  fsstat
  Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
  Aug 23 08:25:35 marvin kernel: NFS call  fsstat
  Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
  Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
  Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
  Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
  Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
  Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS: dentr=
y_delete(log/mux_marvin.trc, 0)
  Aug 23 08:25:38 marvin kernel:  OK
  Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS: dentr=
y_delete(log/pw_marvin_1.trc, 0)
  Aug 23 08:25:38 marvin kernel:  OK
  Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS: dentr=
y_delete(log/pw_marvin_0.trc, 0)
  Aug 23 08:25:38 marvin kernel:  OK
  Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS: dentry=
_delete(net.trc, 0)
  Aug 23 08:25:38 marvin kernel:  OK
  Aug 23 08:25:45 marvin kernel: NFS call  fsstat
  Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
  Aug 23 08:25:55 marvin kernel: NFS call  fsstat

When I run umount during these ops I get an device busy. Any help would be =
very appreciated!


-- =

Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! =

Ideal f=FCr Modem und ISDN: http://www.gmx.net/de/go/smartsurfer

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 3462 bytes --]

z00t@gmx.net wrote:
> Hi guys,
> 
> I get the device busy error when I try to umount an NFS file system although there is no process with an open file on it (at least fuser -m /mnt/point and lsof +f -- /mnt/point don't find anything).
> 
> To be more precise this only happens, when I run the umount right after stopping an application which has its executables and work directory on this file system. Running umount again some seconds later works fine. 
> 
> I do not have any submounts under this mount point. There are no packet transmit or receive errors nor any NFS/RPC errors only a very small number of RPC retransmitts. So it seems that I have a stable environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64. 
> 
> I enabled NFS debug output and the only noticeable thing is, that some NFS operations are still ongoing after the application processes have already terminated:
> 
>   Aug 23 08:25:23 marvin kernel: NFS reply getattr
>   Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 ct=1 info=0x6)
>   Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation complete
>   Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
>   Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
>   Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
>   Aug 23 08:25:23 marvin kernel: NFS call  getattr
>   Aug 23 08:25:23 marvin kernel: NFS reply getattr
>   Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1 info=0x6)
>   Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
>   Aug 23 08:25:25 marvin kernel: NFS call  fsstat
>   Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
>   Aug 23 08:25:35 marvin kernel: NFS call  fsstat
>   Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
>   Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
>   Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
>   Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
>   Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
>   Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
>   Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS: dentry_delete(log/mux_marvin.trc, 0)
>   Aug 23 08:25:38 marvin kernel:  OK
>   Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
>   Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS: dentry_delete(log/pw_marvin_1.trc, 0)
>   Aug 23 08:25:38 marvin kernel:  OK
>   Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
>   Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS: dentry_delete(log/pw_marvin_0.trc, 0)
>   Aug 23 08:25:38 marvin kernel:  OK
>   Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
>   Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS: dentry_delete(net.trc, 0)
>   Aug 23 08:25:38 marvin kernel:  OK
>   Aug 23 08:25:45 marvin kernel: NFS call  fsstat
>   Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
>   Aug 23 08:25:55 marvin kernel: NFS call  fsstat
> 
> When I run umount during these ops I get an device busy. Any help would be very appreciated!

This is normal and expected behavior.  One problem may be that your 
server is slow, and thus there are RPCs left outstanding for a bit on 
your client after your application exits.  The COMMIT calls from that 
trace suggest that there is dirty data the client is writing back to the 
server.

[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 290 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 315 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Zoot]

On Thu, Aug 23, 2007 at 04:03:17PM -0400, Chuck Lever wrote:
> z00t@gmx.net wrote:
> >Hi guys,
> >
> >I get the device busy error when I try to umount an NFS file system 
> >although there is no process with an open file on it (at least fuser -m 
> >/mnt/point and lsof +f -- /mnt/point don't find anything).
> >
> >To be more precise this only happens, when I run the umount right after 
> >stopping an application which has its executables and work directory on 
> >this file system. Running umount again some seconds later works fine. 
> >I do not have any submounts under this mount point. There are no packet 
> >transmit or receive errors nor any NFS/RPC errors only a very small number 
> >of RPC retransmitts. So it seems that I have a stable environment. Client 
> >and server is SLES9 with 2.6.5-7.202.7-smp/X86_64. 
> >I enabled NFS debug output and the only noticeable thing is, that some NFS 
> >operations are still ongoing after the application processes have already 
> >terminated:
> >
> >  Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >  Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 ct=1 
> >  info=0x6)
> >  Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation complete
> >  Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
> >  Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
> >  Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
> >  Aug 23 08:25:23 marvin kernel: NFS call  getattr
> >  Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >  Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1 
> >  info=0x6)
> >  Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
> >  Aug 23 08:25:25 marvin kernel: NFS call  fsstat
> >  Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
> >  Aug 23 08:25:35 marvin kernel: NFS call  fsstat
> >  Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
> >  Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
> >  Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
> >  Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
> >  Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
> >  Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
> >  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS: 
> >  dentry_delete(log/mux_marvin.trc, 0)
> >  Aug 23 08:25:38 marvin kernel:  OK
> >  Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
> >  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS: 
> >  dentry_delete(log/pw_marvin_1.trc, 0)
> >  Aug 23 08:25:38 marvin kernel:  OK
> >  Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
> >  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS: 
> >  dentry_delete(log/pw_marvin_0.trc, 0)
> >  Aug 23 08:25:38 marvin kernel:  OK
> >  Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
> >  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS: 
> >  dentry_delete(net.trc, 0)
> >  Aug 23 08:25:38 marvin kernel:  OK
> >  Aug 23 08:25:45 marvin kernel: NFS call  fsstat
> >  Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
> >  Aug 23 08:25:55 marvin kernel: NFS call  fsstat
> >
> >When I run umount during these ops I get an device busy. Any help would be 
> >very appreciated!
> 
> This is normal and expected behavior.  One problem may be that your 
> server is slow, and thus there are RPCs left outstanding for a bit on 
> your client after your application exits.  The COMMIT calls from that 
> trace suggest that there is dirty data the client is writing back to the 
> server.
But I had expected that umount blocks until the commit calls are finished as it is with local file systems?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

Zoot wrote:
>> This is normal and expected behavior.  One problem may be that your 
>> server is slow, and thus there are RPCs left outstanding for a bit on 
>> your client after your application exits.  The COMMIT calls from that 
>> trace suggest that there is dirty data the client is writing back to the 
>> server.
> But I had expected that umount blocks until the commit calls are
 > finished as it is with local file systems?

Sorry, it doesn't.  It has no way of knowing how long those RPCs will take.

There is a "lazy unmount" facility that removes the NFS share from your 
client's name space and handles the outstanding RPCs in the background. 
  I'm not sure if it's working, but you could try "umount -l".

[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 290 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 315 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Zoot]

On Thu, Aug 23, 2007 at 04:32:50PM -0400, Chuck Lever wrote:
> Zoot wrote:
> >>This is normal and expected behavior.  One problem may be that your 
> >>server is slow, and thus there are RPCs left outstanding for a bit on 
> >>your client after your application exits.  The COMMIT calls from that 
> >>trace suggest that there is dirty data the client is writing back to the 
> >>server.
> >But I had expected that umount blocks until the commit calls are
> > finished as it is with local file systems?
> 
> Sorry, it doesn't.  It has no way of knowing how long those RPCs will take.
> 
> There is a "lazy unmount" facility that removes the NFS share from your 
> client's name space and handles the outstanding RPCs in the background. 
>  I'm not sure if it's working, but you could try "umount -l".
What's with sync() would it wait for the nfs commits? 

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Peter Staubach]

Chuck Lever wrote:
> z00t@gmx.net wrote:
>> Hi guys,
>>
>> I get the device busy error when I try to umount an NFS file system 
>> although there is no process with an open file on it (at least fuser 
>> -m /mnt/point and lsof +f -- /mnt/point don't find anything).
>>
>> To be more precise this only happens, when I run the umount right 
>> after stopping an application which has its executables and work 
>> directory on this file system. Running umount again some seconds 
>> later works fine.
>> I do not have any submounts under this mount point. There are no 
>> packet transmit or receive errors nor any NFS/RPC errors only a very 
>> small number of RPC retransmitts. So it seems that I have a stable 
>> environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
>> I enabled NFS debug output and the only noticeable thing is, that 
>> some NFS operations are still ongoing after the application processes 
>> have already terminated:
>>
>>   Aug 23 08:25:23 marvin kernel: NFS reply getattr
>>   Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 
>> ct=1 info=0x6)
>>   Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation 
>> complete
>>   Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
>>   Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
>>   Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
>>   Aug 23 08:25:23 marvin kernel: NFS call  getattr
>>   Aug 23 08:25:23 marvin kernel: NFS reply getattr
>>   Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1 
>> info=0x6)
>>   Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
>>   Aug 23 08:25:25 marvin kernel: NFS call  fsstat
>>   Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
>>   Aug 23 08:25:35 marvin kernel: NFS call  fsstat
>>   Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
>>   Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
>>   Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
>>   Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
>>   Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
>>   Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
>>   Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS: 
>> dentry_delete(log/mux_marvin.trc, 0)
>>   Aug 23 08:25:38 marvin kernel:  OK
>>   Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
>>   Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS: 
>> dentry_delete(log/pw_marvin_1.trc, 0)
>>   Aug 23 08:25:38 marvin kernel:  OK
>>   Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
>>   Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS: 
>> dentry_delete(log/pw_marvin_0.trc, 0)
>>   Aug 23 08:25:38 marvin kernel:  OK
>>   Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
>>   Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS: 
>> dentry_delete(net.trc, 0)
>>   Aug 23 08:25:38 marvin kernel:  OK
>>   Aug 23 08:25:45 marvin kernel: NFS call  fsstat
>>   Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
>>   Aug 23 08:25:55 marvin kernel: NFS call  fsstat
>>
>> When I run umount during these ops I get an device busy. Any help 
>> would be very appreciated!
>
> This is normal and expected behavior.  One problem may be that your 
> server is slow, and thus there are RPCs left outstanding for a bit on 
> your client after your application exits.  The COMMIT calls from that 
> trace suggest that there is dirty data the client is writing back to 
> the server. 

It seems to me that this should not be the expected behavior unless
the file system is mounted "nocto".  Is it?

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

>>> When I run umount during these ops I get an device busy. Any help 
>>> would be very appreciated!
>>
>> This is normal and expected behavior.  One problem may be that your 
>> server is slow, and thus there are RPCs left outstanding for a bit on 
>> your client after your application exits.  The COMMIT calls from that 
>> trace suggest that there is dirty data the client is writing back to 
>> the server. 
> 
> It seems to me that this should not be the expected behavior unless
> the file system is mounted "nocto".  Is it?

I'm a little puzzled myself about what dirty data there might be left 
after the application quits.  However, I'll be there is an mmap() 
lurking somewhere in the background...

Data that was dirtied via a mapped file is not subject to the writeback 
part of close-to-open.

[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 290 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 315 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Peter Staubach]

Chuck Lever wrote:
>>>> When I run umount during these ops I get an device busy. Any help 
>>>> would be very appreciated!
>>>
>>> This is normal and expected behavior.  One problem may be that your 
>>> server is slow, and thus there are RPCs left outstanding for a bit 
>>> on your client after your application exits.  The COMMIT calls from 
>>> that trace suggest that there is dirty data the client is writing 
>>> back to the server. 
>>
>> It seems to me that this should not be the expected behavior unless
>> the file system is mounted "nocto".  Is it?
>
> I'm a little puzzled myself about what dirty data there might be left 
> after the application quits.  However, I'll be there is an mmap() 
> lurking somewhere in the background...
>
> Data that was dirtied via a mapped file is not subject to the 
> writeback part of close-to-open.

I don't think that I agree with this last statement.  Although
it can not be implemented using the normal close system call,
something should trigger the flush of any dirty pages and wait
for them to complete.

I view close-to-open as a semantic and not just as a syntax.
I view it as "check on first reference" and "flush when last
reference is released".

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]



Peter Staubach wrote:
> Chuck Lever wrote:
>>>>> When I run umount during these ops I get an device busy. Any help 
>>>>> would be very appreciated!
>>>>
>>>> This is normal and expected behavior.  One problem may be that your 
>>>> server is slow, and thus there are RPCs left outstanding for a bit 
>>>> on your client after your application exits.  The COMMIT calls from 
>>>> that trace suggest that there is dirty data the client is writing 
>>>> back to the server. 
>>>
>>> It seems to me that this should not be the expected behavior unless
>>> the file system is mounted "nocto".  Is it?
>>
>> I'm a little puzzled myself about what dirty data there might be left 
>> after the application quits.  However, I'll be there is an mmap() 
>> lurking somewhere in the background...
>>
>> Data that was dirtied via a mapped file is not subject to the 
>> writeback part of close-to-open.
> 
> I don't think that I agree with this last statement.  Although
> it can not be implemented using the normal close system call,
> something should trigger the flush of any dirty pages and wait
> for them to complete.

This is the way Linux implements dirty mmaps on NFS because that's the 
way Linus wants it.  I didn't mean to imply that's the way it *should* 
work, merely the way it *does* work in Linux.

[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 290 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
url:http://oss.oracle.com/~cel
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 315 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Peter Staubach]

Chuck Lever wrote:
>
>
> Peter Staubach wrote:
>> Chuck Lever wrote:
>>>>>> When I run umount during these ops I get an device busy. Any help 
>>>>>> would be very appreciated!
>>>>>
>>>>> This is normal and expected behavior.  One problem may be that 
>>>>> your server is slow, and thus there are RPCs left outstanding for 
>>>>> a bit on your client after your application exits.  The COMMIT 
>>>>> calls from that trace suggest that there is dirty data the client 
>>>>> is writing back to the server. 
>>>>
>>>> It seems to me that this should not be the expected behavior unless
>>>> the file system is mounted "nocto".  Is it?
>>>
>>> I'm a little puzzled myself about what dirty data there might be 
>>> left after the application quits.  However, I'll be there is an 
>>> mmap() lurking somewhere in the background...
>>>
>>> Data that was dirtied via a mapped file is not subject to the 
>>> writeback part of close-to-open.
>>
>> I don't think that I agree with this last statement.  Although
>> it can not be implemented using the normal close system call,
>> something should trigger the flush of any dirty pages and wait
>> for them to complete.
>
> This is the way Linux implements dirty mmaps on NFS because that's the 
> way Linus wants it.  I didn't mean to imply that's the way it *should* 
> work, merely the way it *does* work in Linux.

Yes, I recognize that.  That's the way that it worked in Solaris
too, until I decided that it should work differently and implemented
it.

That said, I don't know how hard it would be to sell the new
semantic to Linus.  This may tie in with some other work that
I am doing, so perhaps I will give it a shot.

    Thanx...

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Zoot]

On Thu, Aug 23, 2007 at 04:35:28PM -0400, Chuck Lever wrote:
> >>>When I run umount during these ops I get an device busy. Any help 
> >>>would be very appreciated!
> >>
> >>This is normal and expected behavior.  One problem may be that your 
> >>server is slow, and thus there are RPCs left outstanding for a bit on 
> >>your client after your application exits.  The COMMIT calls from that 
> >>trace suggest that there is dirty data the client is writing back to 
> >>the server. 
> >
> >It seems to me that this should not be the expected behavior unless
> >the file system is mounted "nocto".  Is it?
> 
> I'm a little puzzled myself about what dirty data there might be left 
> after the application quits.  However, I'll be there is an mmap() 
> lurking somewhere in the background...
> 
> Data that was dirtied via a mapped file is not subject to the writeback 
> part of close-to-open.
At least according to the NFS debug output are the commits only related
to .trc files which are used by the app to store it's debug output. Not
used for mmap.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Zoot]

On Thu, Aug 23, 2007 at 04:29:46PM -0400, Peter Staubach wrote:
> Chuck Lever wrote:
> >z00t@gmx.net wrote:
> >>Hi guys,
> >>
> >>I get the device busy error when I try to umount an NFS file system 
> >>although there is no process with an open file on it (at least fuser 
> >>-m /mnt/point and lsof +f -- /mnt/point don't find anything).
> >>
> >>To be more precise this only happens, when I run the umount right 
> >>after stopping an application which has its executables and work 
> >>directory on this file system. Running umount again some seconds 
> >>later works fine.
> >>I do not have any submounts under this mount point. There are no 
> >>packet transmit or receive errors nor any NFS/RPC errors only a very 
> >>small number of RPC retransmitts. So it seems that I have a stable 
> >>environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
> >>I enabled NFS debug output and the only noticeable thing is, that 
> >>some NFS operations are still ongoing after the application processes 
> >>have already terminated:
> >>
> >>  Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >>  Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 
> >>ct=1 info=0x6)
> >>  Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation 
> >>complete
> >>  Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
> >>  Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
> >>  Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
> >>  Aug 23 08:25:23 marvin kernel: NFS call  getattr
> >>  Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >>  Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1 
> >>info=0x6)
> >>  Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
> >>  Aug 23 08:25:25 marvin kernel: NFS call  fsstat
> >>  Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
> >>  Aug 23 08:25:35 marvin kernel: NFS call  fsstat
> >>  Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
> >>  Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
> >>  Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
> >>  Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
> >>  Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
> >>  Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
> >>  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS: 
> >>dentry_delete(log/mux_marvin.trc, 0)
> >>  Aug 23 08:25:38 marvin kernel:  OK
> >>  Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
> >>  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS: 
> >>dentry_delete(log/pw_marvin_1.trc, 0)
> >>  Aug 23 08:25:38 marvin kernel:  OK
> >>  Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
> >>  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS: 
> >>dentry_delete(log/pw_marvin_0.trc, 0)
> >>  Aug 23 08:25:38 marvin kernel:  OK
> >>  Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
> >>  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS: 
> >>dentry_delete(net.trc, 0)
> >>  Aug 23 08:25:38 marvin kernel:  OK
> >>  Aug 23 08:25:45 marvin kernel: NFS call  fsstat
> >>  Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
> >>  Aug 23 08:25:55 marvin kernel: NFS call  fsstat
> >>
> >>When I run umount during these ops I get an device busy. Any help 
> >>would be very appreciated!
> >
> >This is normal and expected behavior.  One problem may be that your 
> >server is slow, and thus there are RPCs left outstanding for a bit on 
> >your client after your application exits.  The COMMIT calls from that 
> >trace suggest that there is dirty data the client is writing back to 
> >the server. 
> 
> It seems to me that this should not be the expected behavior unless
> the file system is mounted "nocto".  Is it?
> 
>    Thanx...
> 
>       ps
No, mount options are: nfsvers=3,soft,tcp,rsize=32768,wsize=32768

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Peter Staubach]

Zoot wrote:
> On Thu, Aug 23, 2007 at 04:29:46PM -0400, Peter Staubach wrote:
>   
>> Chuck Lever wrote:
>>     
>>> z00t@gmx.net wrote:
>>>       
>>>> Hi guys,
>>>>
>>>> I get the device busy error when I try to umount an NFS file system 
>>>> although there is no process with an open file on it (at least fuser 
>>>> -m /mnt/point and lsof +f -- /mnt/point don't find anything).
>>>>
>>>> To be more precise this only happens, when I run the umount right 
>>>> after stopping an application which has its executables and work 
>>>> directory on this file system. Running umount again some seconds 
>>>> later works fine.
>>>> I do not have any submounts under this mount point. There are no 
>>>> packet transmit or receive errors nor any NFS/RPC errors only a very 
>>>> small number of RPC retransmitts. So it seems that I have a stable 
>>>> environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
>>>> I enabled NFS debug output and the only noticeable thing is, that 
>>>> some NFS operations are still ongoing after the application processes 
>>>> have already terminated:
>>>>
>>>>  Aug 23 08:25:23 marvin kernel: NFS reply getattr
>>>>  Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 
>>>> ct=1 info=0x6)
>>>>  Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation 
>>>> complete
>>>>  Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
>>>>  Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
>>>>  Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
>>>>  Aug 23 08:25:23 marvin kernel: NFS call  getattr
>>>>  Aug 23 08:25:23 marvin kernel: NFS reply getattr
>>>>  Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1 
>>>> info=0x6)
>>>>  Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
>>>>  Aug 23 08:25:25 marvin kernel: NFS call  fsstat
>>>>  Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
>>>>  Aug 23 08:25:35 marvin kernel: NFS call  fsstat
>>>>  Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
>>>>  Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
>>>>  Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
>>>>  Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
>>>>  Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
>>>>  Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
>>>>  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS: 
>>>> dentry_delete(log/mux_marvin.trc, 0)
>>>>  Aug 23 08:25:38 marvin kernel:  OK
>>>>  Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
>>>>  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS: 
>>>> dentry_delete(log/pw_marvin_1.trc, 0)
>>>>  Aug 23 08:25:38 marvin kernel:  OK
>>>>  Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
>>>>  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS: 
>>>> dentry_delete(log/pw_marvin_0.trc, 0)
>>>>  Aug 23 08:25:38 marvin kernel:  OK
>>>>  Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
>>>>  Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS: 
>>>> dentry_delete(net.trc, 0)
>>>>  Aug 23 08:25:38 marvin kernel:  OK
>>>>  Aug 23 08:25:45 marvin kernel: NFS call  fsstat
>>>>  Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
>>>>  Aug 23 08:25:55 marvin kernel: NFS call  fsstat
>>>>
>>>> When I run umount during these ops I get an device busy. Any help 
>>>> would be very appreciated!
>>>>         
>>> This is normal and expected behavior.  One problem may be that your 
>>> server is slow, and thus there are RPCs left outstanding for a bit on 
>>> your client after your application exits.  The COMMIT calls from that 
>>> trace suggest that there is dirty data the client is writing back to 
>>> the server. 
>>>       
>> It seems to me that this should not be the expected behavior unless
>> the file system is mounted "nocto".  Is it?
>>
>>    Thanx...
>>
>>       ps
>>     
> No, mount options are: nfsvers=3,soft,tcp,rsize=32768,wsize=32768
>   

"soft"?

Dangerous.

       ps

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: umount fails with device busy

[Zoot]

On Thu, Aug 23, 2007 at 04:53:58PM -0400, Peter Staubach wrote:
> Zoot wrote:
> >On Thu, Aug 23, 2007 at 04:29:46PM -0400, Peter Staubach wrote:
> >  
> >>Chuck Lever wrote:
> >>    
> >>>z00t@gmx.net wrote:
> >>>      
> >>>>Hi guys,
> >>>>
> >>>>I get the device busy error when I try to umount an NFS file system 
> >>>>although there is no process with an open file on it (at least fuser 
> >>>>-m /mnt/point and lsof +f -- /mnt/point don't find anything).
> >>>>
> >>>>To be more precise this only happens, when I run the umount right 
> >>>>after stopping an application which has its executables and work 
> >>>>directory on this file system. Running umount again some seconds 
> >>>>later works fine.
> >>>>I do not have any submounts under this mount point. There are no 
> >>>>packet transmit or receive errors nor any NFS/RPC errors only a very 
> >>>>small number of RPC retransmitts. So it seems that I have a stable 
> >>>>environment. Client and server is SLES9 with 2.6.5-7.202.7-smp/X86_64.
> >>>>I enabled NFS debug output and the only noticeable thing is, that 
> >>>>some NFS operations are still ongoing after the application processes 
> >>>>have already terminated:
> >>>>
> >>>> Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >>>> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/149296 
> >>>>ct=1 info=0x6)
> >>>> Aug 23 08:25:23 marvin kernel: NFS: (0:12/149296) revalidation 
> >>>>complete
> >>>> Aug 23 08:25:23 marvin kernel: nfs: flush(0:12/149296)
> >>>> Aug 23 08:25:23 marvin kernel: NFS: dentry_delete(bin/stop, 0)
> >>>> Aug 23 08:25:23 marvin kernel: NFS: revalidating (0:12/2)
> >>>> Aug 23 08:25:23 marvin kernel: NFS call  getattr
> >>>> Aug 23 08:25:23 marvin kernel: NFS reply getattr
> >>>> Aug 23 08:25:23 marvin kernel: NFS: nfs_update_inode(0:12/2 ct=1 
> >>>>info=0x6)
> >>>> Aug 23 08:25:23 marvin kernel: NFS: (0:12/2) revalidation complete
> >>>> Aug 23 08:25:25 marvin kernel: NFS call  fsstat
> >>>> Aug 23 08:25:25 marvin kernel: NFS reply statfs: 0
> >>>> Aug 23 08:25:35 marvin kernel: NFS call  fsstat
> >>>> Aug 23 08:25:35 marvin kernel: NFS reply statfs: 0
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6368 initiated commit call
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6369 initiated commit call
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6370 initiated commit call
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6371 initiated commit call
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6368 nfs_commit_done (status 0)
> >>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/121805 3207@0)NFS: 
> >>>>dentry_delete(log/mux_marvin.trc, 0)
> >>>> Aug 23 08:25:38 marvin kernel:  OK
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6369 nfs_commit_done (status 0)
> >>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149304 2238@0)NFS: 
> >>>>dentry_delete(log/pw_marvin_1.trc, 0)
> >>>> Aug 23 08:25:38 marvin kernel:  OK
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6370 nfs_commit_done (status 0)
> >>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149297 2238@0)NFS: 
> >>>>dentry_delete(log/pw_marvin_0.trc, 0)
> >>>> Aug 23 08:25:38 marvin kernel:  OK
> >>>> Aug 23 08:25:38 marvin kernel: NFS: 6371 nfs_commit_done (status 0)
> >>>> Aug 23 08:25:38 marvin kernel: NFS: commit (0:12/149295 965@0)NFS: 
> >>>>dentry_delete(net.trc, 0)
> >>>> Aug 23 08:25:38 marvin kernel:  OK
> >>>> Aug 23 08:25:45 marvin kernel: NFS call  fsstat
> >>>> Aug 23 08:25:45 marvin kernel: NFS reply statfs: 0
> >>>> Aug 23 08:25:55 marvin kernel: NFS call  fsstat
> >>>>
> >>>>When I run umount during these ops I get an device busy. Any help 
> >>>>would be very appreciated!
> >>>>        
> >>>This is normal and expected behavior.  One problem may be that your 
> >>>server is slow, and thus there are RPCs left outstanding for a bit on 
> >>>your client after your application exits.  The COMMIT calls from that 
> >>>trace suggest that there is dirty data the client is writing back to 
> >>>the server. 
> >>>      
> >>It seems to me that this should not be the expected behavior unless
> >>the file system is mounted "nocto".  Is it?
> >>
> >>   Thanx...
> >>
> >>      ps
> >>    
> >No, mount options are: nfsvers=3,soft,tcp,rsize=32768,wsize=32768
> >  
> 
> "soft"?
> 
> Dangerous.
> 
>       ps
Yes, I know I know. But it's the same with "hard".

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs