Re: [patch 2/2] fix file counting

[Eric Dumazet <dada1@cosmosbay.com>]

Dipankar Sarma a écrit :
> On Fri, Jan 27, 2006 at 03:28:57PM -0800, Andrew Morton wrote:
>> "Paul E. McKenney" <paulmck@us.ibm.com> wrote:
>>>>> I am using a patch that seems sligthly better : It removes the filp_count_lock 
>>>>> as yours but introduces a percpu variable, and a lazy nr_files . (Its value 
>>>>> can be off with a delta of +/- 16*num_possible_cpus()
>>>> Yes, I think that is better.
>>> I agree that Eric's approach likely improves performance on large systems
>>> due to decreased cache thrashing.  However, the real problem is getting
>>> both good throughput and good latency in RCU callback processing, given
>>> Lee Revell's latency testing results.  Once we get that in hand, then
>>> we should consider Eric's approach.
>> Dipankar's patch risks worsening large-SMP scalability, doesn't it? 
>> Putting an atomic op into the file_free path?
> 
> Here are some numbers from a 32-way (HT) P4 xeon box with kernbench -
> 

Hi Dipankar, thank you very much for doing these tests.

To have an idea of the number of files that are allocated/freed during a 
kernel build, I added 4 fields in struct files_stat.

4th field is the central atomic_t nr_files.
5th field is the counter of allocated files.
6th field is the counter of freed files.
7th field is the counter of changes done on central atomic_t.

(Test machine is a dual Xeon HT, 2 GHz)
# make clean
# cat /proc/sys/fs/file-nr
119     0       206071  119     39169   39022   1131
[root@localhost linux-2.6.16-rc1-mm4]# time make -j4 bzImage
798.49user 82.36system 4:56.56elapsed 297%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (62major+7223263minor)pagefaults 0swaps

# cat /proc/sys/fs/file-nr
153     0       206071  153     755767  755620  5119

So thats (755767-39169)/(4*60+56) = 2420 files opened per second.

Number of changes on central atomic_t was :
(5119-1131)/(4*60+56) = 13 per second.


13 atomic ops per second (instead of 2420*2 per second if I had one atomic_t 
as in your patch), this is certainly not something we can notice in a typical 
SMP machine... maybe on a big NUMA machine it could be different ?

Eric