Re: [patch 1/2] fs: cleanup files_lock

[tim <tim.c.chen@linux.intel.com>]

Nick Piggin wrote:
> I would like to start sending bits of vfs scalability improvements to
> be reviewed and hopefully merged.
>
> I will start with files_lock. Last time this one came up, it was
> criticised because some hoped to get rid of files list, and because
> it didn't have enough justification of the scalability improvement.
>
> For the first criticism, it isn't any more difficult to rip out if
> we are ever able to remove files list. For the second, I have gathered
> some statistics and written better justification. Andi I believe is
> finding kbuild is becoming limited by files lock on larger systems.


I did some testing doing multi-threaded kernel compile 
(skipping link stage) on a 4 socket Nehalem-EX class 
machines with 8 cores per socket.
In the past we have found 
heavy contention on the files_lock. Nick's patch to 
reorganize files_lock is beneficial.

With  64 threads compile, the distribution of user, 
system, idle, IO wait time improved and the
details are as follow:

                us      sys     idle    IO wait (in %)  
2.6.34-rc2      51.25   28.25   17.25   3.25      
+nick's patch   53.75   18.5    19      8.75    

We spend 10% less cpu on system time contending for files_lock.
Contention of files_lock is gone from our profiling data.

The throughput with Nick's patch is a bit better 
(24.9 times faster than single threaded compile vs 24.5 times without
patch).  

Tim