From: Nadia Derbey <Nadia.Derbey@bull.net>
To: Mike Galbraith <efault@gmx.de>
Cc: Manfred Spraul <manfred@colorfullife.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
paulmck@linux.vnet.ibm.com,
Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Pavel Emelianov <xemul@openvz.org>
Subject: Re: Scalability requirements for sysv ipc (+namespaces broken with SEM_UNDO)
Date: Fri, 04 Apr 2008 16:59:27 +0200 [thread overview]
Message-ID: <47F6424F.9010405@bull.net> (raw)
In-Reply-To: <1206897488.4593.11.camel@marge.simson.net>
[-- Attachment #1: Type: text/plain, Size: 1940 bytes --]
Mike Galbraith wrote:
> On Sun, 2008-03-30 at 16:12 +0200, Manfred Spraul wrote:
>
>>Mike Galbraith wrote:
>>
>>>On Sat, 2008-03-22 at 20:08 +0100, Manfred Spraul wrote:
>>>
>>>
>>>
>>>>just the normal performance of 2.6.25-rc3 is abyssimal, 55 to 60% slower
>>>>than 2.6.18.8:
>>>>
>>>
>>>After manually reverting 3e148c79938aa39035669c1cfa3ff60722134535,
>>>2.6.25.git scaled linearly
>>
>>We can't just revert that patch: with IDR, a global lock is mandatory :-(
>>We must either revert the whole idea of using IDR or live with the
>>reduced scalability.
>
>
> Yeah, I looked at the problem, but didn't know what the heck to do about
> it, so just grabbed my axe to verify/quantify.
>
>
>>Actually, there are further bugs: the undo structures are not
>>namespace-aware, thus semop with SEM_UNDO, unshare, create new array
>>with same id, but more semaphores, another semop with SEM_UNDO will
>>corrupt kernel memory :-(
>>I'll try to clean up the bugs first, then I'll look at the scalability
>>again.
>
>
> Great!
>
> -Mike
>
>
>
>
I could get better results with the following solution:
wrote an RCU-based idr api (layers allocation is managed similarly to
the radix-tree one)
Using it in the ipc code makes me get rid of the read lock taken in
ipc_lock() (the one introduced in 3e148c79938aa39035669c1cfa3ff60722134535).
You'll find the results in attachment (kernel is 2.6.25-rc3-mm1).
output.25_rc3_mm1.ref.8 --> pmsg output for the 2.6.25-rc3-mm1
plot.25_rc3_mm1.ref.8 --> previous file results for use by gnuplot
output.25_rc3_mm1.ridr.8 --> pmsg output for the 2.6.25-rc3-mm1
+ rcu-based idrs
plot.25_rc3_mm1.ridr.8 --> previous file results for use by gnuplot
I think I should be able to send a patch next week. It is presently an
uggly code: I copied idr.c and idr.h into ridr.c and ridr.h to go fast,
so didn't do any code factorization.
Regards
Nadia
[-- Attachment #2: output.25_rc3_mm1.ref.8 --]
[-- Type: text/x-troff-man, Size: 5846 bytes --]
pmsg [nr queues] [timeout]
Using 1 queues/cpus (2 threads) for 5 seconds.
thread 0: sysvmsg 0 type 1 bound to 0001h
thread 1: sysvmsg 0 type 0 bound to 0001h
Result matrix:
Thread 0: 488650 1: 488650
Total: 977300
pmsg [nr queues] [timeout]
Using 2 queues/cpus (4 threads) for 5 seconds.
thread 0: sysvmsg 32768 type 1 bound to 0001h
thread 1: sysvmsg 65537 type 1 bound to 0002h
thread 3: sysvmsg 65537 type 0 bound to 0002h
thread 2: sysvmsg 32768 type 0 bound to 0001h
Result matrix:
Thread 0: 223991 2: 223991
Thread 1: 225588 3: 225588
Total: 899158
pmsg [nr queues] [timeout]
Using 3 queues/cpus (6 threads) for 5 seconds.
thread 0: sysvmsg 98304 type 1 bound to 0001h
thread 1: sysvmsg 131073 type 1 bound to 0002h
thread 2: sysvmsg 163842 type 1 bound to 0004h
thread 5: sysvmsg 163842 type 0 bound to 0004h
thread 4: sysvmsg 131073 type 0 bound to 0002h
thread 3: sysvmsg 98304 type 0 bound to 0001h
Result matrix:
Thread 0: 183407 3: 183407
Thread 1: 184030 4: 184030
Thread 2: 357875 5: 357876
Total: 1450625
pmsg [nr queues] [timeout]
Using 4 queues/cpus (8 threads) for 5 seconds.
thread 0: sysvmsg 196608 type 1 bound to 0001h
thread 1: sysvmsg 229377 type 1 bound to 0002h
thread 2: sysvmsg 262146 type 1 bound to 0004h
thread 3: sysvmsg 294915 type 1 bound to 0008h
thread 5: sysvmsg 229377 type 0 bound to 0002h
thread 6: sysvmsg 262146 type 0 bound to 0004h
thread 7: sysvmsg 294915 type 0 bound to 0008h
thread 4: sysvmsg 196608 type 0 bound to 0001h
Result matrix:
Thread 0: 166911 4: 166912
Thread 1: 159281 5: 159281
Thread 2: 166024 6: 166024
Thread 3: 167440 7: 167440
Total: 1319313
pmsg [nr queues] [timeout]
Using 5 queues/cpus (10 threads) for 5 seconds.
thread 0: sysvmsg 327680 type 1 bound to 0001h
thread 2: sysvmsg 393218 type 1 bound to 0004h
thread 3: sysvmsg 425987 type 1 bound to 0008h
thread 4: sysvmsg 458756 type 1 bound to 0010h
thread 9: sysvmsg 458756 type 0 bound to 0010h
thread 6: sysvmsg 360449 type 0 bound to 0002h
thread 8: sysvmsg 425987 type 0 bound to 0008h
thread 7: sysvmsg 393218 type 0 bound to 0004h
thread 1: sysvmsg 360449 type 1 bound to 0002h
thread 5: sysvmsg 327680 type 0 bound to 0001h
Result matrix:
Thread 0: 39740 5: 39740
Thread 1: 40399 6: 40399
Thread 2: 40326 7: 40327
Thread 3: 39290 8: 39290
Thread 4: 68684 9: 68685
Total: 456880
pmsg [nr queues] [timeout]
Using 6 queues/cpus (12 threads) for 5 seconds.
thread 0: sysvmsg 491520 type 1 bound to 0001h
thread 1: sysvmsg 524289 type 1 bound to 0002h
thread 2: sysvmsg 557058 type 1 bound to 0004h
thread 3: sysvmsg 589827 type 1 bound to 0008h
thread 4: sysvmsg 622596 type 1 bound to 0010h
thread 5: sysvmsg 655365 type 1 bound to 0020h
thread 6: sysvmsg 491520 type 0 bound to 0001h
thread 11: sysvmsg 655365 type 0 bound to 0020h
thread 10: sysvmsg 622596 type 0 bound to 0010h
thread 8: sysvmsg 557058 type 0 bound to 0004h
thread 9: sysvmsg 589827 type 0 bound to 0008h
thread 7: sysvmsg 524289 type 0 bound to 0002h
Result matrix:
Thread 0: 27901 6: 27901
Thread 1: 28554 7: 28555
Thread 2: 28471 8: 28472
Thread 3: 28015 9: 28016
Thread 4: 28213 10: 28213
Thread 5: 28396 11: 28396
Total: 339103
pmsg [nr queues] [timeout]
Using 7 queues/cpus (14 threads) for 5 seconds.
thread 0: sysvmsg 688128 type 1 bound to 0001h
thread 1: sysvmsg 720897 type 1 bound to 0002h
thread 2: sysvmsg 753666 type 1 bound to 0004h
thread 3: sysvmsg 786435 type 1 bound to 0008h
thread 4: sysvmsg 819204 type 1 bound to 0010h
thread 5: sysvmsg 851973 type 1 bound to 0020h
thread 6: sysvmsg 884742 type 1 bound to 0040h
thread 13: sysvmsg 884742 type 0 bound to 0040h
thread 7: sysvmsg 688128 type 0 bound to 0001h
thread 11: sysvmsg 819204 type 0 bound to 0010h
thread 12: sysvmsg 851973 type 0 bound to 0020h
thread 8: sysvmsg 720897 type 0 bound to 0002h
thread 10: sysvmsg 786435 type 0 bound to 0008h
thread 9: sysvmsg 753666 type 0 bound to 0004h
Result matrix:
Thread 0: 12201 7: 12201
Thread 1: 12451 8: 12452
Thread 2: 12345 9: 12345
Thread 3: 12277 10: 12278
Thread 4: 12259 11: 12259
Thread 5: 12364 12: 12365
Thread 6: 24666 13: 24666
Total: 197129
pmsg [nr queues] [timeout]
Using 8 queues/cpus (16 threads) for 5 seconds.
thread 0: sysvmsg 917504 type 1 bound to 0001h
thread 1: sysvmsg 950273 type 1 bound to 0002h
thread 2: sysvmsg 983042 type 1 bound to 0004h
thread 3: sysvmsg 1015811 type 1 bound to 0008h
thread 4: sysvmsg 1048580 type 1 bound to 0010h
thread 5: sysvmsg 1081349 type 1 bound to 0020h
thread 6: sysvmsg 1114118 type 1 bound to 0040h
thread 7: sysvmsg 1146887 type 1 bound to 0080h
thread 15: sysvmsg 1146887 type 0 bound to 0080h
thread 8: sysvmsg 917504 type 0 bound to 0001h
thread 14: sysvmsg 1114118 type 0 bound to 0040h
thread 13: sysvmsg 1081349 type 0 bound to 0020h
thread 12: sysvmsg 1048580 type 0 bound to 0010h
thread 11: sysvmsg 1015811 type 0 bound to 0008h
thread 10: sysvmsg 983042 type 0 bound to 0004h
thread 9: sysvmsg 950273 type 0 bound to 0002h
Result matrix:
Thread 0: 11082 8: 11083
Thread 1: 11461 9: 11461
Thread 2: 11430 10: 11431
Thread 3: 11184 11: 11185
Thread 4: 11373 12: 11374
Thread 5: 11290 13: 11291
Thread 6: 11265 14: 11266
Thread 7: 11324 15: 11325
Total: 180825
[-- Attachment #3: plot.25_rc3_mm1.ref.8 --]
[-- Type: text/x-troff-man, Size: 74 bytes --]
1 977300
2 899158
3 1450625
4 1319313
5 456880
6 339103
7 197129
8 180825
[-- Attachment #4: output.25_rc3_mm1.ridr.8 --]
[-- Type: text/x-troff-man, Size: 5851 bytes --]
pmsg [nr queues] [timeout]
Using 1 queues/cpus (2 threads) for 5 seconds.
thread 0: sysvmsg 0 type 1 bound to 0001h
thread 1: sysvmsg 0 type 0 bound to 0001h
Result matrix:
Thread 0: 549365 1: 549365
Total: 1098730
pmsg [nr queues] [timeout]
Using 2 queues/cpus (4 threads) for 5 seconds.
thread 0: sysvmsg 32768 type 1 bound to 0001h
thread 1: sysvmsg 65537 type 1 bound to 0002h
thread 3: sysvmsg 65537 type 0 bound to 0002h
thread 2: sysvmsg 32768 type 0 bound to 0001h
Result matrix:
Thread 0: 245002 2: 245003
Thread 1: 246618 3: 246619
Total: 983242
pmsg [nr queues] [timeout]
Using 3 queues/cpus (6 threads) for 5 seconds.
thread 0: sysvmsg 98304 type 1 bound to 0001h
thread 1: sysvmsg 131073 type 1 bound to 0002h
thread 2: sysvmsg 163842 type 1 bound to 0004h
thread 5: sysvmsg 163842 type 0 bound to 0004h
thread 4: sysvmsg 131073 type 0 bound to 0002h
thread 3: sysvmsg 98304 type 0 bound to 0001h
Result matrix:
Thread 0: 231585 3: 231586
Thread 1: 233256 4: 233256
Thread 2: 509630 5: 509631
Total: 1948944
pmsg [nr queues] [timeout]
Using 4 queues/cpus (8 threads) for 5 seconds.
thread 0: sysvmsg 196608 type 1 bound to 0001h
thread 1: sysvmsg 229377 type 1 bound to 0002h
thread 2: sysvmsg 262146 type 1 bound to 0004h
thread 3: sysvmsg 294915 type 1 bound to 0008h
thread 5: sysvmsg 229377 type 0 bound to 0002h
thread 6: sysvmsg 262146 type 0 bound to 0004h
thread 7: sysvmsg 294915 type 0 bound to 0008h
thread 4: sysvmsg 196608 type 0 bound to 0001h
Result matrix:
Thread 0: 233392 4: 233392
Thread 1: 234485 5: 234486
Thread 2: 235604 6: 235604
Thread 3: 235683 7: 235683
Total: 1878329
pmsg [nr queues] [timeout]
Using 5 queues/cpus (10 threads) for 5 seconds.
thread 0: sysvmsg 327680 type 1 bound to 0001h
thread 2: sysvmsg 393218 type 1 bound to 0004h
thread 3: sysvmsg 425987 type 1 bound to 0008h
thread 4: sysvmsg 458756 type 1 bound to 0010h
thread 1: sysvmsg 360449 type 1 bound to 0002h
thread 9: sysvmsg 458756 type 0 bound to 0010h
thread 6: sysvmsg 360449 type 0 bound to 0002h
thread 7: sysvmsg 393218 type 0 bound to 0004h
thread 8: sysvmsg 425987 type 0 bound to 0008h
thread 5: sysvmsg 327680 type 0 bound to 0001h
Result matrix:
Thread 0: 216094 5: 216095
Thread 1: 227109 6: 227110
Thread 2: 222042 7: 222042
Thread 3: 222708 8: 222708
Thread 4: 467186 9: 467187
Total: 2710281
pmsg [nr queues] [timeout]
Using 6 queues/cpus (12 threads) for 5 seconds.
thread 0: sysvmsg 491520 type 1 bound to 0001h
thread 1: sysvmsg 524289 type 1 bound to 0002h
thread 2: sysvmsg 557058 type 1 bound to 0004h
thread 3: sysvmsg 589827 type 1 bound to 0008h
thread 4: sysvmsg 622596 type 1 bound to 0010h
thread 5: sysvmsg 655365 type 1 bound to 0020h
thread 6: sysvmsg 491520 type 0 bound to 0001h
thread 11: sysvmsg 655365 type 0 bound to 0020h
thread 8: sysvmsg 557058 type 0 bound to 0004h
thread 10: sysvmsg 622596 type 0 bound to 0010h
thread 9: sysvmsg 589827 type 0 bound to 0008h
thread 7: sysvmsg 524289 type 0 bound to 0002h
Result matrix:
Thread 0: 224027 6: 224028
Thread 1: 225394 7: 225394
Thread 2: 223545 8: 223545
Thread 3: 223599 9: 223599
Thread 4: 224632 10: 224633
Thread 5: 224511 11: 224512
Total: 2691419
pmsg [nr queues] [timeout]
Using 7 queues/cpus (14 threads) for 5 seconds.
thread 0: sysvmsg 688128 type 1 bound to 0001h
thread 1: sysvmsg 720897 type 1 bound to 0002h
thread 2: sysvmsg 753666 type 1 bound to 0004h
thread 3: sysvmsg 786435 type 1 bound to 0008h
thread 4: sysvmsg 819204 type 1 bound to 0010h
thread 5: sysvmsg 851973 type 1 bound to 0020h
thread 6: sysvmsg 884742 type 1 bound to 0040h
thread 13: sysvmsg 884742 type 0 bound to 0040h
thread 8: sysvmsg 720897 type 0 bound to 0002h
thread 9: sysvmsg 753666 type 0 bound to 0004h
thread 10: sysvmsg 786435 type 0 bound to 0008h
thread 11: sysvmsg 819204 type 0 bound to 0010h
thread 7: sysvmsg 688128 type 0 bound to 0001h
thread 12: sysvmsg 851973 type 0 bound to 0020h
Result matrix:
Thread 0: 188264 7: 188264
Thread 1: 190677 8: 190677
Thread 2: 188850 9: 188851
Thread 3: 188925 10: 188926
Thread 4: 190333 11: 190334
Thread 5: 189235 12: 189235
Thread 6: 386862 13: 386863
Total: 3046296
pmsg [nr queues] [timeout]
Using 8 queues/cpus (16 threads) for 5 seconds.
thread 0: sysvmsg 917504 type 1 bound to 0001h
thread 1: sysvmsg 950273 type 1 bound to 0002h
thread 2: sysvmsg 983042 type 1 bound to 0004h
thread 3: sysvmsg 1015811 type 1 bound to 0008h
thread 4: sysvmsg 1048580 type 1 bound to 0010h
thread 5: sysvmsg 1081349 type 1 bound to 0020h
thread 6: sysvmsg 1114118 type 1 bound to 0040h
thread 7: sysvmsg 1146887 type 1 bound to 0080h
thread 8: sysvmsg 917504 type 0 bound to 0001h
thread 10: sysvmsg 983042 type 0 bound to 0004h
thread 11: sysvmsg 1015811 type 0 bound to 0008h
thread 12: sysvmsg 1048580 type 0 bound to 0010h
thread 13: sysvmsg 1081349 type 0 bound to 0020h
thread 9: sysvmsg 950273 type 0 bound to 0002h
thread 15: sysvmsg 1146887 type 0 bound to 0080h
thread 14: sysvmsg 1114118 type 0 bound to 0040h
Result matrix:
Thread 0: 187613 8: 187614
Thread 1: 190488 9: 190489
Thread 2: 190112 10: 190113
Thread 3: 190374 11: 190375
Thread 4: 190658 12: 190658
Thread 5: 190508 13: 190508
Thread 6: 189222 14: 189223
Thread 7: 190272 15: 190272
Total: 3038499
[-- Attachment #5: plot.25_rc3_mm1.ridr.8 --]
[-- Type: text/x-troff-man, Size: 79 bytes --]
1 1098730
2 983242
3 1948944
4 1878329
5 2710281
6 2691419
7 3046296
8 3038499
next prev parent reply other threads:[~2008-04-04 15:00 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-03-21 9:41 Scalability requirements for sysv ipc (was: ipc: store ipcs into IDRs) Manfred Spraul
2008-03-21 12:45 ` Nadia Derbey
2008-03-21 13:33 ` Scalability requirements for sysv ipc Manfred Spraul
2008-03-21 14:13 ` Paul E. McKenney
2008-03-21 16:08 ` Manfred Spraul
2008-03-22 5:43 ` Mike Galbraith
2008-03-22 10:10 ` Manfred Spraul
2008-03-22 11:53 ` Mike Galbraith
2008-03-22 14:22 ` Manfred Spraul
2008-03-22 19:08 ` Manfred Spraul
2008-03-25 15:50 ` Mike Galbraith
2008-03-25 16:13 ` Peter Zijlstra
2008-03-25 18:31 ` Mike Galbraith
2008-03-26 6:18 ` Mike Galbraith
2008-03-30 14:12 ` Scalability requirements for sysv ipc (+namespaces broken with SEM_UNDO) Manfred Spraul
2008-03-30 15:21 ` David Newall
2008-03-30 17:18 ` Mike Galbraith
2008-04-04 14:59 ` Nadia Derbey [this message]
2008-04-04 15:03 ` Nadia Derbey
2008-03-22 19:35 ` Scalability requirements for sysv ipc Mike Galbraith
2008-03-23 6:38 ` Manfred Spraul
2008-03-23 7:15 ` Mike Galbraith
2008-03-23 7:08 ` Mike Galbraith
2008-03-23 7:20 ` Mike Galbraith
2008-03-27 22:29 ` Bill Davidsen
2008-03-28 9:49 ` Manfred Spraul
2008-03-25 16:00 ` Nadia Derbey
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=47F6424F.9010405@bull.net \
--to=nadia.derbey@bull.net \
--cc=akpm@linux-foundation.org \
--cc=efault@gmx.de \
--cc=linux-kernel@vger.kernel.org \
--cc=manfred@colorfullife.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=xemul@openvz.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.