Re: VMA lookup with RCU

[Vaidyanathan Srinivasan <svaidy@linux.vnet.ibm.com>]

[-- Attachment #1: Type: text/plain, Size: 3271 bytes --]



>> Apparently our IBM friends on this thread have a workload where mmap_sem
>> does hurt, and I suspect its a massively threaded Java app on a somewhat
>> larger box (8-16 cpus), which does a bit of faulting around.
>>
>> But I'll let them tell about it :-)
>>
> 
> Nick,
> 
> We used the latest glibc (with the private futexes fix) and the latest
> kernel. We see improvements in scalability, but at 12-16 CPU's, we see
> a slowdown. Vaidy has been using ebizzy for testing mmap_sem
> scalability.
> 

Hi Peter and Nick,

We have been doing some tests with ebizzy 0.2 workload.
Here are some of the test results...

ebizzy-futex.png plots the performance impact of private futex while
ebizzy-rcu-vma.png plots the performance of Peter's RCU VMA look patch
against base kernel with and without private futex.

We can observe in both the plots that private futex improved scaling
significantly from 4 CPUs to 8 CPUs but we still have scaling issues beyond
12 CPUs.

Peter's RCU based b+tree vma lookup approach gives marginal performance
improvement till 4 to 8 CPUs but does not help beyond that.

Perhaps the scaling problem area shifts beyond 8-12 cpus and it is not just
the mmap_sem and vma lookup.

The significant oprofile output for various configurations are listed below:

12 CPUs 2.6.23-rc6 No private futex:

samples  %        symbol name
6908330  23.7520  __down_read
4990895  17.1595  __up_read
2165162   7.4442  find_vma
2069868   7.1166  futex_wait
2063447   7.0945  futex_wake
1557829   5.3561  drop_futex_key_refs
741268    2.5486  task_rq_lock
638947    2.1968  schedule
600493    2.0646  system_call
515924    1.7738  copy_user_generic_unrolled
399672    1.3741  mwait_idle

12 CPUs 2.6.23-rc6 with private futex:

samples  %        symbol name
2095531  15.5092  task_rq_lock
1094271   8.0988  schedule
1068093   7.9050  futex_wake
516250    3.8208  futex_wait
469220    3.4727  mwait_idle
468979    3.4710  system_call
443208    3.2802  idle_cpu
438301    3.2439  update_curr
397231    2.9399  try_to_wake_up
364424    2.6971  apic_timer_interrupt
362633    2.6839  scheduler_tick

8 CPUs 2.6.23-rc9 + RCU VMA + private futex:

samples  %        symbol name
386111   10.4460  mwait_idle
367289    9.9368  __btree_search
286286    7.7453  apic_timer_interrupt
272863    7.3821  find_busiest_group
230268    6.2298  scheduler_tick
224902    6.0846  copy_user_generic_unrolled
188991    5.1130  memset
123692    3.3464  __exit_idle
91930     2.4871  hrtimer_run_queues
87796     2.3753  task_rq_lock
85968     2.3258  run_rebalance_domains

12 CPUs 2.6.23-rc9 + RCU VMA + private futex:

samples  %        symbol name
14505427 18.6891  task_rq_lock
10323001 13.3004  futex_wake
6980246   8.9935  schedule
4768884   6.1443  futex_wait
2981997   3.8421  idle_cpu
2939439   3.7872  system_call
2655022   3.4208  update_curr
2433540   3.1354  try_to_wake_up
1786021   2.3011  check_preempt_curr_fair
1692711   2.1809  syscall_trace_enter
1486381   1.9151  thread_return

All the above test results has the impact of oprofile included.  Running
oprofile also may significantly increase mmap_sem contention.

I Will run the tests again without oprofile to understand the impact of
oprofile itself.

Please let me know your comments and suggestions.

--Vaidy

[-- Attachment #2: ebizzy-futex.png --]
[-- Type: image/png, Size: 4324 bytes --]

[-- Attachment #3: ebizzy-rcu-vma.png --]
[-- Type: image/png, Size: 4917 bytes --]