Re: [RFC][PATCH 1/6] Storing ipcs into IDRs

[Nadia Derbey <Nadia.Derbey@bull.net>]

Nadia Derbey wrote:
> Andi Kleen wrote:
> 
>> Nadia.Derbey@bull.net writes:
>>
>>
>>> This patch introduces ipcs storage into IDRs. The main changes are:
>>>  . This ipc_ids structure is changed: the entries array is changed 
>>> into a
>>>    root idr structure.
>>>  . The grow_ary() routine is removed: it is not needed anymore when 
>>> adding
>>>    an ipc structure, since we are now using the IDR facility.
>>>  . The ipc_rmid() routine interface is changed:
>>>       . there is no need for this routine to return the pointer 
>>> passed in as
>>>         argument: it is now declared as a void
>>>       . since the id is now part of the kern_ipc_perm structure, no 
>>> need to
>>>         have it as an argument to the routine
>>>
>>
>>
>> Thanks for doing this work. It was long overdue.
>>
>> Do you have any data how this changes memory consumption with many 
>> objects?
>> -Andi
>>
> 
> Andi,
> 
> Here are the results I got when creating 32768 (IPCMNI) msg queues with 
> the patched kernel:
> 
> http://akt.sourceforge.net/results/2.6.23-rc2-idr/msg11/output.new
> 
> It's the output from msg11.c. This script does what follows:
>     . gets sysinfo(2) results
>     . captures /proc/meminfo
>     . captures /proc/slabinfo
>     . creates XX msg queues (XX given as parameter)
>     . captures /proc/meminfo
>     . captures /proc/slabinfo
>     . gets sysinfo results
>     . outputs all the results
>     . removes the created ipcs
> 
> sysinfo results: a BEFORE and an AFTER column are output where necessary.
> BEFORE means "before creating the objects"
> AFTER means "after the objects have been created"
> 
> meminfo results: the BEFORE and AFTER files are pasted
> 
> slabinfo results: only the differences between the BEFORE and the AFTER 
> are output.
> 
> 
> Here are also the sizes for the ref and the patched kernel:
> 
> size linux-2.6.23-rc2.ref/vmlinux linux-2.6.23-rc2/vmlinux
>    text    data     bss     dec     hex filename
> 4432697  496450  602112 5531259  54667b linux-2.6.23-rc2.ref/vmlinux
> 4430747  496450  602112 5529309  545edd linux-2.6.23-rc2/vmlinux
> 
> 
> 
> The http://akt.sourceforge.net/results/2.6.23-rc2-idr/msg11 directory is 
> structured as follows:
> 
> msg11.c: the script I used to generate the results
> output.ref: the output from msg11 with the ref kernel
> output.new: the output from msg11 with the patched kernel
> size: the output from the size command
> ref: directory with the results files for the ref kernel
> new: directory with the results files for the patched kernel
> In these 2 directories:
> *_mem_*before: /proc/meminfo before creating the msg queues
> *_mem_*after: /proc/meminfo after creating the msg queues
> *_slab_*before: /proc/slabinfo before creating the msg queues
> *_slab_*after: /proc/slabinfo after creating the msg queues
> 

Andi,

Here is an annalysis of the results I sent you yesterday (I guess you 
don't have enoguh time to look at everything):

ref code:

1) since /proc/sys/kernel/msgmni has been set to 32768, vmalloc(0x20014) 
is called to allocate the entries[] array (see grow_ary() --> 
ipc_rcu_alloc()).

==>

Before msg queues allocation:  VmallocUsed = 2860 kB
After msg queues allocation:   VmallocUsed = 2996 KB

Once allocated this array is never freed.

Unfortunately, in the result I sent you yesterday, you can't see the 
evolution since it was not the 1st time I was running the test, so the 
vmalloc() has not been called.


2) Since 32768 msg queues have been created and a msg_queue structure <
PAGE_SIZE, kmalloc(0x6C) is called 32768 times

==> size-128 in slabinfo: (I slighty simplified the output to make it 
fit in the mail):

                               objs          slabs
          # name   <active>  <num> <size> : <active> <num>
before : size-128     1070   1320    128 :       44    44
after  : size-128    33840  33840    128 :     1128  1128


patched code:
1) since 32768 msg queues are created, idr_pre_get() is called 32768 times
==> idr_layer_cache in slabinfo:

                                         objs          slabs
          # name           <active>  <num> <size> : <active> <num>
before : idr_layer_cache       112    116    136 :        4     4
after  : idr_layer_cache      1189   1189    136 :       41    41

2) This point remains unchanged compared to the ref code.



Regards,
Nadia