[LSF/MM TOPIC] Parallelize file operation (like creation, unlink) under large shared directory

[LSF/MM TOPIC] Parallelize file operation (like creation, unlink) under large shared directory

[wangdi]

Single directory performance is a critical in some use cases. For 
example the multiple application threads might create hundreds of 
thousands of files in a single directory simultaneously within a short 
window of time.

Currently, both filename lookup and file system modifying operations 
(such as create and unlink) are protected with a single lock for the 
entire directory. It might be useful to remove this lock, so multiple 
application threads can access the directory simultaneously.

Thanks
WangDi

Re: [LSF/MM TOPIC] Parallelize file operation (like creation, unlink) under large shared directory

[Boaz Harrosh]

On 01/21/2012 10:54 PM, wangdi wrote:
> Single directory performance is a critical in some use cases. For 
> example the multiple application threads might create hundreds of 
> thousands of files in a single directory simultaneously within a short 
> window of time.
> 

read NFSD here ;-)

> Currently, both filename lookup and file system modifying operations 
> (such as create and unlink) are protected with a single lock for the 
> entire directory. It might be useful to remove this lock, so multiple 
> application threads can access the directory simultaneously.
> 

I agree about create, unlink, and so on. But don't we have some lockless
look up in place since a few Kernels ago?

But yes the topic is very interesting, though I'd suspect its hard to
implement.

> Thanks
> WangDi
> 

Thanks
Boaz

Re: [LSF/MM TOPIC] Parallelize file operation (like creation, unlink) under large shared directory

[wangdi]

On 01/22/2012 05:55 AM, Boaz Harrosh wrote:
> On 01/21/2012 10:54 PM, wangdi wrote:
>> Single directory performance is a critical in some use cases. For
>> example the multiple application threads might create hundreds of
>> thousands of files in a single directory simultaneously within a short
>> window of time.
>>
> read NFSD here ;-)
>
>> Currently, both filename lookup and file system modifying operations
>> (such as create and unlink) are protected with a single lock for the
>> entire directory. It might be useful to remove this lock, so multiple
>> application threads can access the directory simultaneously.
>>
> I agree about create, unlink, and so on. But don't we have some lockless
> look up in place since a few Kernels ago?

  The lockless lookup is only for lookups in the dcache and not in the backing filesystem, which is serialized by i_mutex in the VFS(real_lookup).


> But yes the topic is very interesting, though I'd suspect its hard to
> implement.

We actually already implemented this for ext4, and we saw a lot performance improvement(at least 30% improvements for open/create in a single directory)for lustre stack,
but we want to make this improvement accessible through the VFS. Probably XFS and Btrfs could also benefit from this.

Thanks
WangDi


>> Thanks
>> WangDi
>>
> Thanks
> Boaz

Re: [LSF/MM TOPIC] Parallelize file operation (like creation, unlink) under large shared directory

[Al Viro]

On Sun, Jan 22, 2012 at 12:31:31PM -0800, wangdi wrote:

> We actually already implemented this for ext4, and we saw a lot performance improvement(at least 30% improvements for open/create in a single directory)for lustre stack,
> but we want to make this improvement accessible through the VFS. Probably XFS and Btrfs could also benefit from this.

You do realize that i_mutex locking is relied upon for protection of a lot
of stuff besides the obvious (i.e. on-disk directory contents)?

I'm not saying that it's hopeless, but it's highly non-trivial; the things
like rmdir/mount races, access to ->d_parent/->d_name in a lot of code,
etc. need to be taken care of and it is a _lot_ of code review to deal
with - just to verify the correctness of such changes.

Re: [LSF/MM TOPIC] Parallelize file operation (like creation, unlink) under large shared directory

[wangdi]

On 01/22/2012 12:39 PM, Al Viro wrote:
> On Sun, Jan 22, 2012 at 12:31:31PM -0800, wangdi wrote:
>
>> We actually already implemented this for ext4, and we saw a lot performance improvement(at least 30% improvements for open/create in a single directory)for lustre stack,
>> but we want to make this improvement accessible through the VFS. Probably XFS and Btrfs could also benefit from this.
> You do realize that i_mutex locking is relied upon for protection of a lot
> of stuff besides the obvious (i.e. on-disk directory contents)?
> I'm not saying that it's hopeless, but it's highly non-trivial; the things
> like rmdir/mount races, access to ->d_parent/->d_name in a lot of code,
> etc. need to be taken care of and it is a _lot_ of code review to deal
> with - just to verify the correctness of such changes.

Yes, I agree it is non-trivial change here. What I want to say is that 
i_mutex lock might be too big in some cases, and it just serializes 
everything. So it might be useful  if we could refine this lock a bit.  
For example we can define this lock with several modes, (read, write, 
current read, current write, exclusive etc), and different code can get 
the lock with different mode as required, which might bring us some 
concurrency.

Thanks
WangDi

Re: [LSF/MM TOPIC] Parallelize file operation (like creation, unlink) under large shared directory

[Dave Chinner]

On Sun, Jan 22, 2012 at 01:17:18PM -0800, wangdi wrote:
> On 01/22/2012 12:39 PM, Al Viro wrote:
> >On Sun, Jan 22, 2012 at 12:31:31PM -0800, wangdi wrote:
> >
> >>We actually already implemented this for ext4, and we saw a lot performance improvement(at least 30% improvements for open/create in a single directory)for lustre stack,
> >>but we want to make this improvement accessible through the VFS. Probably XFS and Btrfs could also benefit from this.
> >You do realize that i_mutex locking is relied upon for protection of a lot
> >of stuff besides the obvious (i.e. on-disk directory contents)?
> >I'm not saying that it's hopeless, but it's highly non-trivial; the things
> >like rmdir/mount races, access to ->d_parent/->d_name in a lot of code,
> >etc. need to be taken care of and it is a _lot_ of code review to deal
> >with - just to verify the correctness of such changes.
> 
> Yes, I agree it is non-trivial change here. What I want to say is
> that i_mutex lock might be too big in some cases, and it just
> serializes everything. So it might be useful  if we could refine
> this lock a bit.  For example we can define this lock with several
> modes, (read, write, current read, current write, exclusive etc),
> and different code can get the lock with different mode as required,
> which might bring us some concurrency.

Irix used a read/write lock for the VFS level locking. XFS still has
that same locking for it's directory operations. Lookup takes it in
read mode, and now XFS has lockless inode cache lookups, too. Hence
if we can etch the correct Cthulu Summoning Patterns^W^W^W locking
primitives in the VFS, XFS can definitely take advantage of it...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com