Possible deadlock in vfs layer, namei.c

Possible deadlock in vfs layer, namei.c

[Joshua Hudson]

I've been hunting down various deadlocks caused by hard links to directories.
I found one that can happen *without* such things.

Consider this:

17 3 drwxr-xr-x root sys 4096 dir
18 2 drwxr-xr-x root sys 4096 dir/subdir
19 1 -rwxr-xr-x root sys 1733 dir/subdir/file

process 1 does: rename("dir/subdir/file", "dir/file")
process 2 does: rmdir("dir/subdir")

from namei.c (function: lock_rename), rename takes:
1. s_vfs_rename_sem,
2. dir/subdir: p1->d_inode->i_sem
3. dir: p2->d_inode->i_sem

rmdir takes (multiple functions):
1. dir  (sys_rename)
2. dir/subdir (vfs_rename)

Suppose this happends:
[proc 1] s_vfs_rename_sem
[proc 1] dir/subdir
*task switch*
[proc 2] dir
[proc 2] dir/subdir ->deadlock

preempt_disable() won't help here. Could also be triggered on a SMP
machine w/o preempt.
Am studying situation, doesn't look promising.

Re: Possible deadlock in vfs layer, namei.c

[Al Viro]

On Wed, Mar 01, 2006 at 06:46:42PM -0800, Joshua Hudson wrote:
> I've been hunting down various deadlocks caused by hard links to directories.
> I found one that can happen *without* such things.
 
> process 1 does: rename("dir/subdir/file", "dir/file")
> process 2 does: rmdir("dir/subdir")
> 
> from namei.c (function: lock_rename), rename takes:
> 1. s_vfs_rename_sem,
> 2. dir/subdir: p1->d_inode->i_sem
> 3. dir: p2->d_inode->i_sem

No, it doesn't.  Wrong order - it will take dir before dir/subdir.
RTFM - Documentation/filesystems/directory-locking is there for
purpose.

Fwd: Possible deadlock in vfs layer, namei.c

[Joshua Hudson]

On 3/1/06, Al Viro <viro@ftp.linux.org.uk> wrote:
> On Wed, Mar 01, 2006 at 06:46:42PM -0800, Joshua Hudson wrote:
> > I've been hunting down various deadlocks caused by hard links to directories.
> > I found one that can happen *without* such things.
>
> > process 1 does: rename("dir/subdir/file", "dir/file")
> > process 2 does: rmdir("dir/subdir")
> >
> > from namei.c (function: lock_rename), rename takes:
> > 1. s_vfs_rename_sem,
> > 2. dir/subdir: p1->d_inode->i_sem
> > 3. dir: p2->d_inode->i_sem
>
> No, it doesn't.  Wrong order - it will take dir before dir/subdir.
> RTFM - Documentation/filesystems/directory-locking is there for
> purpose.
>
Well, bloody stupid of me, didn't notice the slight differences of the
locking in the for loops.