Re: [RFC][PATCH 00/13] percpu rwsem -v2

[Daniel Wagner <daniel.wagner@bmw-carit.de>]

On 06/22/2015 09:05 PM, Peter Zijlstra wrote:
> On Mon, Jun 22, 2015 at 08:11:14PM +0200, Daniel Wagner wrote:
>> On 06/22/2015 02:16 PM, Peter Zijlstra wrote:
>>> Also, since Linus thinks lglocks is a failed locking primitive (which I whole
>>> heartedly agree with, its preempt-disable latencies are an abomination), it
>>> also converts the global part of fs/locks's usage of lglock over to a
>>> percpu-rwsem and uses a per-cpu spinlock for the local part. This both provides
>>> another (4th) percpu-rwsem users and removes an lglock user.
>>
>> I did a quick lockperf run with these patches on a 4 socket E5-4610 machine.
>> These microbenches execercise the fs' locks a bit.
>>
>> I suspect I got the wrong tree. The patches did not apply cleanly. The resulting
>> kernel boots fine and doesn't explode... so far...
> 
> Its against tip/master, although I expect the locking/core bits that
> were sent to Linus earlier today to be the biggest missing piece.
> 
> All I really did was build a kernel with lockdep enabled and boot +
> build a kernel to see it didn't go belly up.
> 
>> The results aren't looking too bad. Though building a kernel with 'make -j200'
>> was extreme slow. I'll look into it tomorrow.

So this turns out to be false alarm. I had icecream installed/actived
and that interfered with gcc. Stupid me.

The machine has 0.5TB memory and doesn't seem to be really concerned about
'make -j200'

make clean && time make -j200

mainline 4.1.0
2nd run
	real    1m7.595s
	user    28m43.125s
	sys     3m48.189s


tip v4.1-2756-ge3d06bd
2nd run
	real    1m6.871s
	user    28m50.803s
	sys     3m50.223s
3rd run
	real    1m6.974s
	user    28m52.093s
	sys     3m50.259s


tip v4.1-2769-g6ce2591 (percpu-rwsem)
2nd run
	real    1m7.847s
	user    29m0.439s
	sys     3m51.181s
3rd run
	real    1m7.113s
	user    29m3.127s
	sys     3m51.516s



Compared to 'make -j64' on tip v4.1-2756-ge3d06bd
2nd run
	real    1m7.605s
	user    28m3.121s
	sys     3m52.541s

>> https://git.samba.org/jlayton/linux.git/?p=jlayton/lockperf.git;a=summary
> 
> Sweet, I wasn't aware these existed. I'll go have a play.
> 
>> posix01
>>                              mean   variance      sigma        max        min
>>                     4.1.0   121.9020 27882.5260   166.9806   603.5509     0.0063
>>              percpu-rwsem   185.3981 38474.3836   196.1489   580.6532     0.0073
>>
>>
>> posix02
>>                              mean   variance      sigma        max        min
>>                     4.1.0    12.7461     3.1802     1.7833    15.5411     8.1018
>>              percpu-rwsem    16.2341     4.3038     2.0746    19.3271    11.1751
>>
> 
> These two seem to hurt, lemme go look at what they do.

Now here the same tests with tip and tip+percpu-rwsem. The patches
applied cleanly :)

I put all the raw data here[1] in case someone is interested. Some of the
test behave a bit strange, running extremely fast compared to the other runs.
That is probably the result of me trying to reduce the run time to the min.


flock01
                             mean   variance      sigma        max        min
                    4.1.0    11.7075   816.3341    28.5716   125.6552     0.0021
       4.1.0+percpu-rwsem    11.4614   760.1345    27.5705   132.5030     0.0026
                      tip     6.8390   329.3037    18.1467    81.0373     0.0021
         tip+percpu-rwsem    10.0870   546.7435    23.3825   106.2396     0.0026


flock02
                             mean   variance      sigma        max        min
                    4.1.0     7.0197     1.1812     1.0868    10.6188     5.1706
       4.1.0+percpu-rwsem     9.3194     1.3443     1.1594    11.5902     6.6138
                      tip     7.1057     1.6719     1.2930    11.2362     5.1434
         tip+percpu-rwsem     9.0357     1.9874     1.4097    14.0254     6.4380


lease01
                             mean   variance      sigma        max        min
                    4.1.0    41.8361    23.8462     4.8833    51.3493    28.5859
       4.1.0+percpu-rwsem    40.2738    20.8323     4.5642    49.6037    28.0704
                      tip    30.2617    13.0900     3.6180    36.6398    20.2085
         tip+percpu-rwsem    31.2730    17.9787     4.2401    37.8981    19.2944


lease02
                             mean   variance      sigma        max        min
                    4.1.0    71.2159    12.7763     3.5744    77.8432    58.0390
       4.1.0+percpu-rwsem    71.4312    14.7688     3.8430    76.5036    57.8615
                      tip    20.2019     5.2042     2.2813    23.1071    13.4647
         tip+percpu-rwsem    20.8305     6.6631     2.5813    23.8034    11.2815


posix01
                             mean   variance      sigma        max        min
                    4.1.0   121.9020 27882.5260   166.9806   603.5509     0.0063
       4.1.0+percpu-rwsem   185.3981 38474.3836   196.1489   580.6532     0.0073
                      tip   129.2736 23752.7122   154.1191   474.0604     0.0063
         tip+percpu-rwsem   142.6474 24732.1571   157.2646   468.7478     0.0072


posix02
                             mean   variance      sigma        max        min
                    4.1.0    12.7461     3.1802     1.7833    15.5411     8.1018
       4.1.0+percpu-rwsem    16.2341     4.3038     2.0746    19.3271    11.1751
                      tip    13.2810     5.3958     2.3229    20.1243     8.9361
         tip+percpu-rwsem    15.6802     4.7514     2.1798    21.5704     9.4074


posix03
                             mean   variance      sigma        max        min
                    4.1.0     0.9121     0.0000     0.0000     0.9121     0.9121
       4.1.0+percpu-rwsem     0.9379     0.0000     0.0000     0.9379     0.9379
                      tip     0.8647     0.0009     0.0297     0.9274     0.7995
         tip+percpu-rwsem     0.8147     0.0003     0.0161     0.8530     0.7824


posix04
                             mean   variance      sigma        max        min
                    4.1.0     0.0703     0.0044     0.0664     0.6764     0.0437
       4.1.0+percpu-rwsem     0.0675     0.0007     0.0267     0.3236     0.0491
                      tip     0.0618     0.0027     0.0521     0.5642     0.0453
         tip+percpu-rwsem     0.0658     0.0003     0.0175     0.1793     0.0493


cheers,
daniel

[1] http://monom.org/percpu-rwsem/