Correct way to handle an error condition in inode lookup

[Priya Kamala <priya.kamala@perpetual-data.com>]

Hi all,

I am working on a linux filesystem filter driver used for archiving. The 
filter is layered on top of xfs. We are currently trying to implement a 
feature to incrementally recover a filesystem, in other words repopulate 
a directory on-the-fly. During initial recovery only the root directory 
is populated. Other directories get populated when they are accessed for 
the first time. To rebuild a directory the filter has to temporarily 
unlock the directory inode's i_mutex. This does mean that other requests 
can come in at the same time resulting in the following condition.

I launch two threads simultaneously. Thread 1 lists and deletes the 
contents of a folder and finally the folder itself, while Thread 2 
renames the first file in the same folder.

Thread 1                                            Thread 2
------------                                            ------------

readdir
release i_mutex to rebuild dir
acquire lock A to rebuild dir
rebuild offline dir starts
                                                             lookup file.0
                                                             release 
i_mutex to rebuild dir
                                                             acquire 
lock A to rebuild dir
rebuild dir completes
acquire i_mutex
mark directory online
release lock A
                                                             no need to 
rebuild as directory has been populated by Thread 1
revalidate file.0
unlink file.0
                                                             acquire i_mutex
                                                             release lock A
                                                             lookup 
returns valid dentry with inode created during Thread 1 rebuild
                                                             rename 
file.0 (Crashes with "Internal error xfs_trans_cancel at line 1925 of 
file fs/xfs/xfs_trans.c")
evict_inode
destroy_inode


Right now I'm using d_unlinked to check if a valid d_entry has been 
unlinked. I've tried using d_delete if the file has been unlinked, but 
this will fail if there is a third thread trying to stat the same file 
and lookup will return a valid d_entry causing rename to crash. What is 
the correct way to detect that a file has been unlinked but the inode 
hasn't been destroyed yet in the lookup function and how should this be 
handled? I am working with kernel version 3.2.52. Let me know if you 
need additional information.

If this is not the correct forum for this question, please point me in 
the right direction.

Thanks,
Priya