[PATCH v2 00/19] Prepare to lift lookup out of exclusive lock for directory ops

[NeilBrown <neilb@ownmail.net>]

This patch set progresses my effort to improve concurrency of
directory operations and specifically to allow concurrent updates
in a given directory.

It is a selection of patches from the 53-patch set I posted in March
which got relatively little response.  Maybe a shorter set will be more
approachable.

This set:
 - prepares the VFS in various ways
 - make use of these preparations in ovl and NFS (the most challenging
   filesystems for lookup as they do the most interesting things)
 - make use in efivars and shmem which for different reasons need a small 
   change that seemed worth including here.

The goal that these patch work towards is moving lookup out of i_rwsem
on the directory - except for the actual ->lookup call.  This is itself
a step towards allowing broad concurrency of operations in a given
directory.

There are two particular requirements before lookup can move outside the lock:
1/ d_drop() mustn't be used before an operation completes: the dentry being present
   in the dcache becomes part of the locking protocol.  This in turn requires
   d_splice_alias() to work with hashed negative dentries.
2/ d_alloc_parallel() mustn't be called while i_rw_sem is held, as this would
   result in a lock inversion.  So d_alloc_noblock and others are introduced
   to handle the various cases.
   In a few cases we need to drop and re-take i_rw_sem inside ->lookup.
   As lookup might be called with a shared or exclusive lock this requires
   a new LOOKUP_SHARED flag which is ugly but can be removed after the
   lookup is moved out of the lock (then ->lookup will only ever be called
   with a shared lock).

The full set of patches including these 19 and the rest to complete the
lifting of lookup out of the exclusive lock can be found at
   github/neilbrown/linux in branch pdirops

Significant changes since last time are:
 - use wait_var_event for d_alloc_parallel() rather than effectively
   duplicating that infrastructure - as suggested by Christop
 - changes to ovl_readdir handling as discussed with Amir.

Thanks,
NeilBrown


 [PATCH v2 01/19] VFS: fix various typos in documentation for
 [PATCH v2 02/19] VFS: enhance d_splice_alias() to handle in-lookup
 [PATCH v2 03/19] VFS: allow d_alloc_name() to be used with ->d_hash
 [PATCH v2 04/19] VFS: use wait_var_event for waiting in
 [PATCH v2 05/19] VFS: introduce d_alloc_noblock()
 [PATCH v2 06/19] VFS: add d_duplicate()
 [PATCH v2 07/19] VFS: Add LOOKUP_SHARED flag.
 [PATCH v2 08/19] VFS/xfs/ntfs: drop parent lock across
 [PATCH v2 09/19] ovl: stop using lookup_one() in iterate_shared()
 [PATCH v2 10/19] VFS/ovl: add d_alloc_noblock_return()
 [PATCH v2 11/19] efivarfs: use d_alloc_name()
 [PATCH v2 12/19] shmem: use d_duplicate()
 [PATCH v2 13/19] nfs: remove d_drop()/d_alloc_parallel() from
 [PATCH v2 14/19] nfs: use d_splice_alias() in nfs_link()
 [PATCH v2 15/19] nfs: don't d_drop() before d_splice_alias()
 [PATCH v2 16/19] nfs: don't d_drop() before d_splice_alias() in
 [PATCH v2 17/19] nfs: Use d_alloc_noblock() in nfs_prime_dcache()
 [PATCH v2 18/19] nfs: use d_alloc_noblock() in silly-rename
 [PATCH v2 19/19] nfs: use d_duplicate()