can't create a vNUMA enabled PV guest

can't create a vNUMA enabled PV guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3385 bytes --]

Hey,

I know Wei is away, so I'll try to find the time to look at this myself,
but I figured I'll let know about it, in case someone has obvious (or
not :-D) ideas.

I think I'm facing a bug that prevents creating PV guests with a vNUMA
topology. I'm pretty sure I tested this before, while reviewing Wei's
patches, so it must be something introduced between then and now (yes,
we need a vNUMA OSSTest test case... I'll see about putting one
together).

So, here we are. With this as base config:

  name        = 'test'
  # Kernel, params and imags
  kernel      = '/root/3.19.0+/vmlinuz-3.19.0+'
  ramdisk     = '/root/3.19.0+/initrd.img-3.19.0+'
  # CPUs and Memory and related
  vcpus       = '4'
  memory      = '1024'
  vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20"  ],
            [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10"  ] ]
  # Disks
  root        = '/dev/xvda1 ro'
  disk        = [
                    'phy:/dev/vms/test-pv-disk,xvda1,w',
                ]
  # Networking
  dhcp        = 'dhcp'
  vif         = [ 'mac=00:16:3E:FA:A7:9B,bridge=xenbr0' ]



If I build a HVM guest, everything works:

(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 129874):
(XEN)     Node 0: 113466
(XEN)     Node 1: 16408
(XEN) Domain 14 (total: 262251):
(XEN)     Node 0: 131029
(XEN)     Node 1: 131222
(XEN)      2 vnodes, 4 vcpus, guest physical layout:
(XEN)          0: pnode   0, vcpus 0-1 
(XEN)            0000000000000000 - 000000001f800000
(XEN)          1: pnode   1, vcpus 2-3
(XEN)            000000001f800000 - 000000003f800000

root@test:~# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1
node 0 size: 411 MB
node 0 free: 311 MB
node 1 cpus: 2 3
node 1 size: 442 MB
node 1 free: 406 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 


If I build a PV guest, it breaks:

root@Zhaman:~# xl create -c /etc/xen/test.cfg
Parsing config from /etc/xen/test.cfg
xc: error: panic: xc_dom_x86.c:940: arch_setup_meminit: failed to allocate 0x20000 pages (v=1, p=1)
: Internal error
xc: error: panic: xc_dom_boot.c:155: xc_dom_boot_mem_init: can't allocate low memory for domain: Out of memory
libxl: error: libxl_dom.c:731:libxl__build_pv: xc_dom_boot_mem_init failed: Device or resource busy
libxl: error: libxl_create.c:1174:domcreate_rebuild_done: cannot (re-)build domain: -3
libxl: error: libxl.c:1586:libxl__destroy_domid: non-existant domain 15
libxl: error: libxl.c:1544:domain_destroy_callback: unable to destroy guest with domid 15
libxl: error: libxl.c:1471:domain_destroy_cb: destruction of domain 15 failed

(XEN) d0v1 Over-allocation for domain 15: 262656 > 262400
(XEN) memory.c:155:d0v1 Could not allocate order=9 extent: id=15 memflags=210 (0 of 512)
(XEN) d0v1 Over-allocation for domain 15: 262401 > 262400
(XEN) memory.c:155:d0v1 Could not allocate order=0 extent: id=15 memflags=210 (256 of 131072)


As said, I'll be looking into it in the next days. If, in the meanwhile,
someone has any ideas, that would be much appreciated. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: can't create a vNUMA enabled PV guest

[Elena Ufimtseva]

On Wed, Jul 1, 2015 at 10:42 AM, Dario Faggioli
<dario.faggioli@citrix.com> wrote:
> Hey,
>
> I know Wei is away, so I'll try to find the time to look at this myself,
> but I figured I'll let know about it, in case someone has obvious (or
> not :-D) ideas.
>
> I think I'm facing a bug that prevents creating PV guests with a vNUMA
> topology. I'm pretty sure I tested this before, while reviewing Wei's
> patches, so it must be something introduced between then and now (yes,
> we need a vNUMA OSSTest test case... I'll see about putting one
> together).
>
> So, here we are. With this as base config:
>
>   name        = 'test'
>   # Kernel, params and imags
>   kernel      = '/root/3.19.0+/vmlinuz-3.19.0+'
>   ramdisk     = '/root/3.19.0+/initrd.img-3.19.0+'
>   # CPUs and Memory and related
>   vcpus       = '4'
>   memory      = '1024'
>   vnuma = [ [ "pnode=0","size=512","vcpus=0-1","vdistances=10,20"  ],
>             [ "pnode=1","size=512","vcpus=2-3","vdistances=20,10"  ] ]
>   # Disks
>   root        = '/dev/xvda1 ro'
>   disk        = [
>                     'phy:/dev/vms/test-pv-disk,xvda1,w',
>                 ]
>   # Networking
>   dhcp        = 'dhcp'
>   vif         = [ 'mac=00:16:3E:FA:A7:9B,bridge=xenbr0' ]
>
>
>
> If I build a HVM guest, everything works:
>
> (XEN) Memory location of each domain:
> (XEN) Domain 0 (total: 129874):
> (XEN)     Node 0: 113466
> (XEN)     Node 1: 16408
> (XEN) Domain 14 (total: 262251):
> (XEN)     Node 0: 131029
> (XEN)     Node 1: 131222
> (XEN)      2 vnodes, 4 vcpus, guest physical layout:
> (XEN)          0: pnode   0, vcpus 0-1
> (XEN)            0000000000000000 - 000000001f800000
> (XEN)          1: pnode   1, vcpus 2-3
> (XEN)            000000001f800000 - 000000003f800000
>
> root@test:~# numactl --hardware
> available: 2 nodes (0-1)
> node 0 cpus: 0 1
> node 0 size: 411 MB
> node 0 free: 311 MB
> node 1 cpus: 2 3
> node 1 size: 442 MB
> node 1 free: 406 MB
> node distances:
> node   0   1
>   0:  10  20
>   1:  20  10
>
>
> If I build a PV guest, it breaks:
>
> root@Zhaman:~# xl create -c /etc/xen/test.cfg
> Parsing config from /etc/xen/test.cfg
> xc: error: panic: xc_dom_x86.c:940: arch_setup_meminit: failed to allocate 0x20000 pages (v=1, p=1)
> : Internal error
> xc: error: panic: xc_dom_boot.c:155: xc_dom_boot_mem_init: can't allocate low memory for domain: Out of memory
> libxl: error: libxl_dom.c:731:libxl__build_pv: xc_dom_boot_mem_init failed: Device or resource busy
> libxl: error: libxl_create.c:1174:domcreate_rebuild_done: cannot (re-)build domain: -3
> libxl: error: libxl.c:1586:libxl__destroy_domid: non-existant domain 15
> libxl: error: libxl.c:1544:domain_destroy_callback: unable to destroy guest with domid 15
> libxl: error: libxl.c:1471:domain_destroy_cb: destruction of domain 15 failed
>
> (XEN) d0v1 Over-allocation for domain 15: 262656 > 262400
> (XEN) memory.c:155:d0v1 Could not allocate order=9 extent: id=15 memflags=210 (0 of 512)
> (XEN) d0v1 Over-allocation for domain 15: 262401 > 262400
> (XEN) memory.c:155:d0v1 Could not allocate order=0 extent: id=15 memflags=210 (256 of 131072)
>
>
> As said, I'll be looking into it in the next days. If, in the meanwhile,
> someone has any ideas, that would be much appreciated. :-)

Hi Dario!

The kernel you are running maybe missing vNUMA patch.
Konrad asked me if the patch was upstream. well, It is not, I think I
abandoned it :).
I will address latest comments and other changes in the v6 review and post it.

Elena


>
> Regards,
> Dario
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>



-- 
Elena

Re: can't create a vNUMA enabled PV guest

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2499 bytes --]

On Thu, 2015-07-02 at 16:25 -0400, Elena Ufimtseva wrote:
> On Wed, Jul 1, 2015 at 10:42 AM, Dario Faggioli

> > If I build a PV guest, it breaks:
> >
> > root@Zhaman:~# xl create -c /etc/xen/test.cfg
> > Parsing config from /etc/xen/test.cfg
> > xc: error: panic: xc_dom_x86.c:940: arch_setup_meminit: failed to allocate 0x20000 pages (v=1, p=1)
> > : Internal error
> > xc: error: panic: xc_dom_boot.c:155: xc_dom_boot_mem_init: can't allocate low memory for domain: Out of memory
> > libxl: error: libxl_dom.c:731:libxl__build_pv: xc_dom_boot_mem_init failed: Device or resource busy
> > libxl: error: libxl_create.c:1174:domcreate_rebuild_done: cannot (re-)build domain: -3
> > libxl: error: libxl.c:1586:libxl__destroy_domid: non-existant domain 15
> > libxl: error: libxl.c:1544:domain_destroy_callback: unable to destroy guest with domid 15
> > libxl: error: libxl.c:1471:domain_destroy_cb: destruction of domain 15 failed
> >
> > (XEN) d0v1 Over-allocation for domain 15: 262656 > 262400
> > (XEN) memory.c:155:d0v1 Could not allocate order=9 extent: id=15 memflags=210 (0 of 512)
> > (XEN) d0v1 Over-allocation for domain 15: 262401 > 262400
> > (XEN) memory.c:155:d0v1 Could not allocate order=0 extent: id=15 memflags=210 (256 of 131072)
> >
> >
> > As said, I'll be looking into it in the next days. If, in the meanwhile,
> > someone has any ideas, that would be much appreciated. :-)
> 
> Hi Dario!
> 
Hi! :-)

> The kernel you are running maybe missing vNUMA patch.
>
Mmm... Interesting. I was indeed assuming that everything was upstream
already!

That being said, if it were a DomU kernel issue, I expected the kernel
itself to complain, rather than libxl and Xen. In fact, from the logs
above, I think this is all happening during domain creation, when the
kernel does not yet have a say in it.

As I said, I'll try to have a look, but I'm not sure if I'll be able to
by today...

> Konrad asked me if the patch was upstream. well, It is not, I think I
> abandoned it :).
> I will address latest comments and other changes in the v6 review and post it.
> 
Yeah, well, if we really are still missing it, the sooner we get it in,
the better! :-)

Thanks and Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel