Re: Possible Bug in VM accounting (Committed_AS) on x86_64 architecture ?

Re: Possible Bug in VM accounting (Committed_AS) on x86_64 architecture ?

[Kunal Trivedi]

Hi,
I am running into a problem and due to limited understanding unable to
solve it...
Problem:
-------------
On 64 bit machines (running linux 2.6.xx), Committed_AS grows over the
period of time. Within 3-4 weeks system reaches a stage where further
malloc() returns -ENOMEM. Test shows that just running simple program
(which malloc, touch, free memory) cause this increase in Committed_AS
number.  (I am using vm-overcommit... Below detailed information)

System/Kernel Spec:
------------------------------
-----
CentOS kernel, 2.6.9-34.EL-x86_64_SMP
Physical: 8G
Swap: 2G
Machine type: AMD
Nothing un-usual in .config. Pretty much using standard options. (If
needed I will send .config)
Here is /proc/meminfo numbers:

MemTotal:      8169736 kB
MemFree:       1256676 kB
Buffers:        123496 kB
Cached:        5009620 kB
SwapCached:          0 kB
Active:        5297288 kB
Inactive:      1003628 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8169736 kB
LowFree:       1256676 kB
SwapTotal:     2096472 kB
SwapFree:      2096472 kB
Dirty:           10036 kB
Writeback:           0 kB
Mapped:        5814464 kB
Slab:           571352 kB
Committed_AS:  7125024 kB
PageTables:      12916 kB
VmallocTotal: 536870911 kB
VmallocUsed:    268300 kB
VmallocChunk: 536601679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

OverCommit Options:
---------------------------------
vm.overcommit_ratio = 100
vm.overcommit_memory = 2

Hence my virtual limit is 10G (8*100/100 + 2)

Detailed Description:
-------------------------------
I have noticed that 64 bit machine with overcommit policy (as above)
starts giving problem within 3-4 weeks. To prove that I've written
small program.
 It allocates memory of different sizes (not that it matters much due
to caching of diffeent malloc. I am using standard ptmalloc). Sizes
are 16B, 32B, 64B, 256B, 1024B, 57B, 127B and so on... . Then it
touches that memory (memset) and then free it. These operations are
being performed in while(1) loop.

Observations:
----------------------
Committed_AS: number gorws 5M per hour. I made sure that nothing else
is running on the system during that time...

Is there any obvious problem reported for vm accounting on 64 bit
architecture ? Or this is expected ? Or vm-overcommit is only meant
for 32 bit architecture ?

Please advice..

Thanks in advance.
-- 

-Kunal

Re: Possible Bug in VM accounting (Committed_AS) on x86_64 architecture ?

[Alan]

> I have noticed that 64 bit machine with overcommit policy (as above)
> starts giving problem within 3-4 weeks. To prove that I've written
> small program.

The older RHEL kernels had some cases that didn't quite account exactly
but current ones ought to be right - for Centos I'd expect similar but
ask there not here as it is a very old and branched away kernel.

>  It allocates memory of different sizes (not that it matters much due
> to caching of diffeent malloc. I am using standard ptmalloc). Sizes
> are 16B, 32B, 64B, 256B, 1024B, 57B, 127B and so on... . Then it
> touches that memory (memset) and then free it. These operations are
> being performed in while(1) loop.

I would expect that, it's fragmentation. The real test is whether the
values go back properly when you kill the program.

Re: Possible Bug in VM accounting (Committed_AS) on x86_64 architecture ?

[Kunal Trivedi]

Hi Alan,
Sorry for late followup on this.

I did found the problem. It was 32 bit binary running on 64 bit arch.
Actually main kernel had fixed this problem in 2.6.14
(http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=2fd4ef85e0db9ed75c98e13953257a967ea55e03)
But apparently CentOS has not ported it yet.

Thanks for your reply

-Kunal