Re: [RFC -mm 0/6] sysv ipc: scale msgmnb with the number of cpus

Nick Piggin
2008-06-06T08:23:49Z

On Friday 06 June 2008 16:09, Solofo.Ramangalahy@bull.net wrote:
> The size in bytes of a SysV IPC message queue, msgmnb, is too small
> for large machines, but we don't want to bloat small machines

What's your evidence for this? Can you provide before / after
performance numbers?

Also, when scaling things like this, it is probably more usual
to use a log scale rather than linear, so that's a thought.

>
> Several methods are used already to modify (mainly increase) msgmnb:
> . distribution specific patch
> . system wide sysctl.conf
> . application specific tuning via /proc/sys/kernel/msgmnb
>
> Integrating this series would:
> . reflect hardware and software evolutions and diversity,
> . reduce configuration/tuning for the applications.
>
> Here is the timeline of the evolution of MSG* #defines:
> Year 1994 1999 1999 2008
> Version 1.0 2.3.27 2.3.30 2.6.24
> #define MSGMNI 128 128 16 16
> #define MSGMAX 4056 8192 8192 8192
> #define MSGMNB 16384 16384 16384 16384
>
> This patch series scales msgmnb, with respect to the number of
> cpus/cores for larger machines. For uniprocessor machines the value
> does not increase.
>
> This series is similar to (and depends on) the series which scales
> msgmni, the number of IPC message queue identifiers, to the amount of
> low memory.
> While Nadia's previous series scaled msgmni along the memory axis,
> hence the message pool (msgmni x msgmnb), this series uses a second
> axis: the number of online CPUs.
> As well as covering the (cpu,memory) space of machines size, this
> reflects the parallelism allowed by lockless send/receive for
> in-flight messages in queues (msgmnb / msgmax messages).
>
> The initial scaling is done at initialization of the ipc namespace.
> Furthermore, the value becomes dynamic with respect to cpu hotplug.
>
> The msgmni and msgmnb values become dependent, as the value of msgmni
> is computed with respect to the value of msgmnb.
>
> The series is as follows:
> . patch 1 introduces the scaling function
> . patch 2 deals with cpu hotplug
> . patch 3 allows user space to disable the scaling mechanism
> . patch 4 allows user space to reenable the scaling mechanism
> . patch 5 finer grain disabling/reenabling scaling mechanism
> (disconnect msgmnb and msgmni)
> . patch 6 adds documentation
>
> ---
>
> The series applies to 2.6.26-rc2-mm1 + patch suppressing KERN_INFO
> messages as discussed at:
> http://article.gmane.org/gmane.linux.kernel/686229
> "[PATCH 1/1] Only output msgmni value at boot time"
> (in mmotm: ipc-only-output-msgmni-value-at-boot-time.patch)
>
> The plan would be to have this ready for the 2.6.27 merge window if
> there are no objections.
>
> Documentation/sysctl/kernel.txt | 27 ++++++++++++++++++++++
> include/linux/ipc_namespace.h | 4 ++-
> include/linux/msg.h | 5 ++++
> ipc/ipc_sysctl.c | 48
> ++++++++++++++++++++++++++++++---------- ipc/ipcns_notifier.c |
> 23 +++++++------------
> ipc/msg.c | 25 +++++++++++++++++---
> ipc/util.c | 28 +++++++++++++++++++++++
> ipc/util.h | 1
> 8 files changed, 131 insertions(+), 30 deletions(-)