"VFS: Busy inodes after unmount of ceph" kernel-message after umount

"VFS: Busy inodes after unmount of ceph" kernel-message after umount

[Thomas Mueller]

hi

since running an new snapshot test* that creates some(many?) snapshots I 
get this kernel message on umount:

VFS: Busy inodes after unmount of ceph. Self-destruct in 5 seconds.  Have 
a nice day...

this is with kernel 2.6.34 and ceph-client-standalone

- Thomas


* see script here:
http://github.com/vinzent/ceph-testsuite/blob/master/tests/snaptest-2

Re: "VFS: Busy inodes after unmount of ceph" kernel-message after umount

[Sage Weil]

On Sat, 5 Jun 2010, Thomas Mueller wrote:
> hi
> 
> since running an new snapshot test* that creates some(many?) snapshots I 
> get this kernel message on umount:
> 
> VFS: Busy inodes after unmount of ceph. Self-destruct in 5 seconds.  Have 
> a nice day...
> 
> this is with kernel 2.6.34 and ceph-client-standalone

This one is new.  Can you say a bit more about what workload induced the 
crash?  Or better yet, are you able to reproduce it?

thanks-
sage

Re: "VFS: Busy inodes after unmount of ceph" kernel-message after umount

[Thomas Mueller]

Am Sun, 06 Jun 2010 21:10:28 -0700 schrieb Sage Weil:

> On Sat, 5 Jun 2010, Thomas Mueller wrote:
>> hi
>> 
>> since running an new snapshot test* that creates some(many?) snapshots
>> I get this kernel message on umount:
>> 
>> VFS: Busy inodes after unmount of ceph. Self-destruct in 5 seconds. 
>> Have a nice day...
>> 
>> this is with kernel 2.6.34 and ceph-client-standalone
> 
> This one is new.  Can you say a bit more about what workload induced the
> crash?  Or better yet, are you able to reproduce it?
> 

yes, i get the message every time i run the test and do the umount. the 
system is debian squeeze with 4GB of RAM and 2.6.34 kernel from 
experimental repo. cannot test it on centos5 right now. 


what the script does:
- mkdir snaptest-2; cd snaptest-2
- create 100 dirs with 1 textfile in it
- mkdir .snap/test-1
- create 100 dirs with 1 textfile in it
- for every subdir create a snapshot (mkdir $i/.snap/snaptest) and then 
create 10 textfiles
- mkdir .snap/test-2
- mkdir copysnap1; cp -R .snap/test-1 copysnap1/
- mkdir .snap/test-3
- rm -Rf ./*
- umount

(http://github.com/vinzent/ceph-testsuite/blob/master/tests/snaptest-2)

- Thomas

Re: "VFS: Busy inodes after unmount of ceph" kernel-message after umount

[Sage Weil]

> > On Sat, 5 Jun 2010, Thomas Mueller wrote:
> >> hi
> >> 
> >> since running an new snapshot test* that creates some(many?) snapshots
> >> I get this kernel message on umount:
> >> 
> >> VFS: Busy inodes after unmount of ceph. Self-destruct in 5 seconds. 
> >> Have a nice day...
> >> 
> >> this is with kernel 2.6.34 and ceph-client-standalone

This turned out to be an MDS bug (not replying to snap flush messages, 
leaving the inodes pinned).  The server side is fixed in the unstable 
and testing branches.  The client should still be fixed to make some 
noise about such problems.

Thanks for that test script... being able to reliably reproduce the 
problem made fixing it so much easier!

sage

Re: "VFS: Busy inodes after unmount of ceph" kernel-message after umount

[Thomas Mueller]

Am Mon, 07 Jun 2010 22:21:33 -0700 schrieb Sage Weil:

>> > On Sat, 5 Jun 2010, Thomas Mueller wrote:
>> >> hi
>> >> 
>> >> since running an new snapshot test* that creates some(many?)
>> >> snapshots I get this kernel message on umount:
>> >> 
>> >> VFS: Busy inodes after unmount of ceph. Self-destruct in 5 seconds.
>> >> Have a nice day...
>> >> 
>> >> this is with kernel 2.6.34 and ceph-client-standalone
> 
> This turned out to be an MDS bug (not replying to snap flush messages,
> leaving the inodes pinned).  The server side is fixed in the unstable
> and testing branches.  The client should still be fixed to make some
> noise about such problems.
> 
> Thanks for that test script... being able to reliably reproduce the
> problem made fixing it so much easier!
> 
> sage

thanks, verified that the message is gone.

I noticed that it takes now longer to execute the script:
test time now: 1300s
test time before: 800-900s


- Thomas