In kvm, a single process cannot allocate more than 3.6GB ram. Why?

In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Tanel Kokk]

Does anybody have a glue why process are not able to allocate more than
3.6GB memory ?

What should i do to allow process allocate more than 3.6GB memory? Is it
possible at all?


I have hostserver:
linux kernel: 2.6.30.9 (arch amd64)
RAM: 64GB
CPUs: 2xQuadCore AMD Opteron 2378

Then I have several kvm guests on it, like:
linux kernel: 2.6.30.4 (arch amd64)
RAM: 6GB
CPUs: 2xQEMU Virtual CPU version 0.9.1 (vendor: AuthenticAMD)
cpu flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx lm pni hypervisor


VM use virtio for net and block drivers.


Tanel

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Thomas Mueller]

Am Fri, 11 Dec 2009 11:03:27 +0200 schrieb Tanel Kokk:

> Does anybody have a glue why process are not able to allocate more than
> 3.6GB memory ?
> 
> What should i do to allow process allocate more than 3.6GB memory? Is it
> possible at all?
> 

just some thoughts:

* is qemu/kvm 64bit?
* check "ulimit -a" on console if a process is limited on memory usage 
(config file /etc/security/limits.conf)

- Thomas

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Tanel Kokk]

On 12/11/09 11:55, Thomas Mueller wrote:
> 
> 
> 
> Am Fri, 11 Dec 2009 11:03:27 +0200 schrieb Tanel Kokk:
> 
>> Does anybody have a glue why process are not able to allocate more than
>> 3.6GB memory ?
>>
>> What should i do to allow process allocate more than 3.6GB memory? Is it
>> possible at all?
>>
> 
> just some thoughts:
> 
> * is qemu/kvm 64bit?
# file /usr/bin/kvm
/usr/bin/kvm: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.8, stripped

> * check "ulimit -a" on console if a process is limited on memory usage 
> (config file /etc/security/limits.conf)

# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 53248
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 99999
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) unlimited
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


Tanel

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Avi Kivity]

On 12/11/2009 11:03 AM, Tanel Kokk wrote:
> Does anybody have a glue why process are not able to allocate more than
> 3.6GB memory ?
>
> What should i do to allow process allocate more than 3.6GB memory? Is it
> possible at all?
>    

A guest or host process?  And what happens when you try?

Please describe what you are doing step by step with more details.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Tanel Kokk]

On 12/12/09 10:55, Avi Kivity wrote:
> On 12/11/2009 11:03 AM, Tanel Kokk wrote:
>> Does anybody have a glue why process are not able to allocate more than
>> 3.6GB memory ?
>>
>> What should i do to allow process allocate more than 3.6GB memory? Is it
>> possible at all?
>>    
> 
> A guest or host process?  And what happens when you try?

guest process

We discovered that if postgres process (on guest) used to use only 3.6GB
memory regardless of guest had 8GB memory in common.

Then I made simple python script, which just eat memory:
------------------------------------
#!/usr/bin/python
import time, random

i = 0
while (i < 100000000):
    a[i] = random.random()
    i = i + 1

time.sleep(600)
------------------------------------
And I get python MemoryError, when script has devoured about 3.6GB ram


Tanel

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Avi Kivity]

On 12/12/2009 11:23 AM, Tanel Kokk wrote:
> On 12/12/09 10:55, Avi Kivity wrote:
>    
>> On 12/11/2009 11:03 AM, Tanel Kokk wrote:
>>      
>>> Does anybody have a glue why process are not able to allocate more than
>>> 3.6GB memory ?
>>>
>>> What should i do to allow process allocate more than 3.6GB memory? Is it
>>> possible at all?
>>>
>>>        
>> A guest or host process?  And what happens when you try?
>>      
> guest process
>
> We discovered that if postgres process (on guest) used to use only 3.6GB
> memory regardless of guest had 8GB memory in common.
>
>    


That's probably a postgres configuration error (or you're using 32-bit 
postgres).


> Then I made simple python script, which just eat memory:
> ------------------------------------
> #!/usr/bin/python
> import time, random
>
> i = 0
> while (i<  100000000):
>      a[i] = random.random()
>      i = i + 1
>
> time.sleep(600)
> ------------------------------------
> And I get python MemoryError, when script has devoured about 3.6GB ram
>    


Doesn't mean much - Python may allocate a lot more memory when extending 
the array.

What does 'cat /proc/meminfo' in the guest say?

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Tanel Kokk]

On 12/12/09 11:35, Avi Kivity wrote:
> On 12/12/2009 11:23 AM, Tanel Kokk wrote:
>> On 12/12/09 10:55, Avi Kivity wrote:
>>   
>>> On 12/11/2009 11:03 AM, Tanel Kokk wrote:
>>>     
>>>> Does anybody have a glue why process are not able to allocate more than
>>>> 3.6GB memory ?
>>>>
>>>> What should i do to allow process allocate more than 3.6GB memory?
>>>> Is it
>>>> possible at all?
>>>>
>>>>        
>>> A guest or host process?  And what happens when you try?
>>>      
>> guest process
>>
>> We discovered that if postgres process (on guest) used to use only 3.6GB
>> memory regardless of guest had 8GB memory in common.
>>
>>    
> 
> 
> That's probably a postgres configuration error (or you're using 32-bit
> postgres).

postgres is 64bit:
root@lu2-kvm-db1 ~]# file /usr/lib/postgresql/8.3/bin/postgres
/usr/lib/postgresql/8.3/bin/postgres: ELF 64-bit LSB executable, x86-64,
version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux
2.6.8, not stripped

> 
> 
>> Then I made simple python script, which just eat memory:
>> ------------------------------------
>> #!/usr/bin/python
>> import time, random
>>
>> i = 0
>> while (i<  100000000):
>>      a[i] = random.random()
>>      i = i + 1
>>
>> time.sleep(600)
>> ------------------------------------
>> And I get python MemoryError, when script has devoured about 3.6GB ram
>>    
> 
> 
> Doesn't mean much - Python may allocate a lot more memory when extending
> the array.
> 
> What does 'cat /proc/meminfo' in the guest say?
> 

[root@lu2-kvm-db1 ~]# cat /proc/meminfo
MemTotal:       10267308 kB
MemFree:          228944 kB
Buffers:           19680 kB
Cached:          6069524 kB
SwapCached:            0 kB
Active:          8175200 kB
Inactive:        1374884 kB
Active(anon):    3238044 kB
Inactive(anon):   544104 kB
Active(file):    4937156 kB
Inactive(file):   830780 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:             15968 kB
Writeback:             0 kB
AnonPages:       3461108 kB
Mapped:           327816 kB
Slab:             171452 kB
SReclaimable:     121756 kB
SUnreclaim:        49696 kB
PageTables:       244268 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8727208 kB
Committed_AS:    6056168 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       38180 kB
VmallocChunk:   34359645255 kB
DirectMap4k:        4032 kB
DirectMap2M:    10481664 kB

Tanel

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Avi Kivity]

On 12/12/2009 12:11 PM, Tanel Kokk wrote:
>
> [root@lu2-kvm-db1 ~]# cat /proc/meminfo
> MemTotal:       10267308 kB
> MemFree:          228944 kB
> Buffers:           19680 kB
> Cached:          6069524 kB
> SwapCached:            0 kB
> Active:          8175200 kB
> Inactive:        1374884 kB
> Active(anon):    3238044 kB
> Inactive(anon):   544104 kB
> Active(file):    4937156 kB
> Inactive(file):   830780 kB
> Unevictable:           0 kB
> Mlocked:               0 kB
> SwapTotal:             0 kB
> SwapFree:              0 kB
> Dirty:             15968 kB
> Writeback:             0 kB
> AnonPages:       3461108 kB
> Mapped:           327816 kB
> Slab:             171452 kB
> SReclaimable:     121756 kB
> SUnreclaim:        49696 kB
> PageTables:       244268 kB
> NFS_Unstable:          0 kB
> Bounce:                0 kB
> WritebackTmp:          0 kB
> CommitLimit:     8727208 kB
> Committed_AS:    6056168 kB
> VmallocTotal:   34359738367 kB
> VmallocUsed:       38180 kB
> VmallocChunk:   34359645255 kB
> DirectMap4k:        4032 kB
> DirectMap2M:    10481664 kB
>    

What guest kernel is this?  What's the value of the guest's 
/proc/sys/vm/overcommit_memory?

-- 
error compiling committee.c: too many arguments to function

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Tanel Kokk]

On 12/13/09 12:04, Avi Kivity wrote:
> On 12/12/2009 12:11 PM, Tanel Kokk wrote:
>>
>> [root@lu2-kvm-db1 ~]# cat /proc/meminfo
>> MemTotal:       10267308 kB
>> MemFree:          228944 kB
>> Buffers:           19680 kB
>> Cached:          6069524 kB
>> SwapCached:            0 kB
>> Active:          8175200 kB
>> Inactive:        1374884 kB
>> Active(anon):    3238044 kB
>> Inactive(anon):   544104 kB
>> Active(file):    4937156 kB
>> Inactive(file):   830780 kB
>> Unevictable:           0 kB
>> Mlocked:               0 kB
>> SwapTotal:             0 kB
>> SwapFree:              0 kB
>> Dirty:             15968 kB
>> Writeback:             0 kB
>> AnonPages:       3461108 kB
>> Mapped:           327816 kB
>> Slab:             171452 kB
>> SReclaimable:     121756 kB
>> SUnreclaim:        49696 kB
>> PageTables:       244268 kB
>> NFS_Unstable:          0 kB
>> Bounce:                0 kB
>> WritebackTmp:          0 kB
>> CommitLimit:     8727208 kB
>> Committed_AS:    6056168 kB
>> VmallocTotal:   34359738367 kB
>> VmallocUsed:       38180 kB
>> VmallocChunk:   34359645255 kB
>> DirectMap4k:        4032 kB
>> DirectMap2M:    10481664 kB
>>    
> 
> What guest kernel is this?  What's the value of the guest's
> /proc/sys/vm/overcommit_memory?
> 

[root@lu2-kvm-db1 ~]# uname -a
Linux lu2-kvm-db1 2.6.30.7 #1 SMP Mon Sep 21 17:39:41 UTC 2009 x86_64
GNU/Linux

[root@lu2-kvm-db1 ~]# cat /proc/sys/vm/overcommit_memory
2


Tanel

Re: In kvm, a single process cannot allocate more than 3.6GB ram. Why?

[Avi Kivity]

On 12/13/2009 01:30 PM, Tanel Kokk wrote:
>
>> What guest kernel is this?  What's the value of the guest's
>> /proc/sys/vm/overcommit_memory?
>>
>>      
> [root@lu2-kvm-db1 ~]# uname -a
> Linux lu2-kvm-db1 2.6.30.7 #1 SMP Mon Sep 21 17:39:41 UTC 2009 x86_64
> GNU/Linux
>
> [root@lu2-kvm-db1 ~]# cat /proc/sys/vm/overcommit_memory
> 2
>
>
>    

Ah.  Set it to zero or add some swap.

-- 
error compiling committee.c: too many arguments to function