From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([208.118.235.92]:44480)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <gaowanlong@cn.fujitsu.com>) id 1Ucuqs-0006b8-Pw
	for qemu-devel@nongnu.org; Thu, 16 May 2013 05:51:59 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <gaowanlong@cn.fujitsu.com>) id 1Ucuqr-0000At-Gb
	for qemu-devel@nongnu.org; Thu, 16 May 2013 05:51:58 -0400
Received: from [222.73.24.84] (port=15632 helo=song.cn.fujitsu.com)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <gaowanlong@cn.fujitsu.com>) id 1Ucuqr-00009E-2r
	for qemu-devel@nongnu.org; Thu, 16 May 2013 05:51:57 -0400
Message-ID: <5194ABFD.8040200@cn.fujitsu.com>
Date: Thu, 16 May 2013 17:50:53 +0800
From: Wanlong Gao <gaowanlong@cn.fujitsu.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=ISO-8859-1
Subject: [Qemu-devel] QEMU NUMA and memory allocation problem
Reply-To: gaowanlong@cn.fujitsu.com
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: qemu-devel <qemu-devel@nongnu.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>, ehabkost@redhat.com

Hi,

We just met a problem of QEMU memory allocation.
Here is the description:

On my host, I have two nodes,
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 3021 MB
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2881 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 



I created a guest using the following XML:

...
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <cpu>
    <topology sockets='2' cores='1' threads='1'/>
    <numa>
      <cell cpus='0' memory='524288'/>
      <cell cpus='1' memory='524288'/>
    </numa>
  </cpu>
...

As you can see, I assigned 1G memory to this guest, pined vcpu0 to the host CPU 2,
it's in host node0, pined vcpu1 to the host CPU 3 that is in host node1.
The guest also has two nodes, each node contains 512M memory.

Now, I started the guest, then printed the host numa state :
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2647 MB  <=== freecell of node0
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2746 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 

Then I tried to allocate memory from guest node0 using the following code:
> #include <memory.h>
> #include <numa.h>
> 
> #define MEM (1024*1024*300)
> 
> int main(void)
> {
> 	char *p = numa_alloc_onnode(MEM, 0);
> 	memset(p, 0, MEM);
> 	sleep(1000);
> 	numa_free(p, MEM);
> 	return 0;
> }

And printed the host numa state, it shows that this 300M memory is allocated from host node0,

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2345 MB	<===== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 


Then, I tried the same method to allocate 300M memory from guest node1, and printed the host
numa state:

# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2
node 0 size: 4010 MB
node 0 free: 2059 MB	<=== reduced ~300M
node 1 cpus: 1 3
node 1 size: 4030 MB
node 1 free: 2767 MB	<=== no change
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 


To see that this 300M memory is allocated from host node0 again, but not host node1 as
I expected.

We think that QEMU can't handled this numa memory allocation well, and it will cause the
cross node memory access performance regression.

Any thoughts? Or, am I missing something?


Thanks,
Wanlong Gao