[PATCH 0/6] x86: break up post-boot non-order-zero allocations

[PATCH 0/6] x86: break up post-boot non-order-zero allocations

[Jan Beulich]

While tmem is most affected by this, due to fragmentation it is generally
a bad idea to require runtime allocations of more than a single page in
size.

1: remove direct cpumask_t members from struct vcpu and struct domain
2: x86: split struct vcpu
3: x86: move pv-only members of struct vcpu to struct pv_vcpu
4: x86: split struct domain
5: x86: introduce alloc_vcpu_guest_context()
6: passthrough: use domain pirq as index of struct hvm_irq_dpci's hvm_timer array

With this, structure sizes are below page size, and no longer depend
significantly on NR_CPUS. This series, however, doesn't eliminate
all non-order-zero allocations that happen post boot (i.e. mostly
during domain creation). Items that are known to need addressing
are
- nr_irqs-sized allocation of ->arch.irq_pirq[] in
  xen/arch/x86/domain.c:arch_domain_create()
- ->nr_pirqs-sized allocations in
  xen/drivers/passthrough/io.c:pt_irq_create_bind_vtd()
- ->nr_pirqs-sized allocation of ->arch.pirq_irq[] in
  xen/arch/x86/domain.c:arch_domain_create()
- ->nr_pirqs-sized allocation of ->pirq_to_evtchn[] in
  xen/common/domain.c:domain_create()

Signed-off-by: Jan Beulich <jbeulich@novell.com>

Re: [PATCH 0/6] x86: break up post-boot non-order-zero allocations

[Keir Fraser]

On 05/04/2011 09:19, "Jan Beulich" <JBeulich@novell.com> wrote:

> With this, structure sizes are below page size, and no longer depend
> significantly on NR_CPUS. This series, however, doesn't eliminate
> all non-order-zero allocations that happen post boot (i.e. mostly
> during domain creation). Items that are known to need addressing
> are
> - nr_irqs-sized allocation of ->arch.irq_pirq[] in
>   xen/arch/x86/domain.c:arch_domain_create()
> - ->nr_pirqs-sized allocations in
>   xen/drivers/passthrough/io.c:pt_irq_create_bind_vtd()
> - ->nr_pirqs-sized allocation of ->arch.pirq_irq[] in
>   xen/arch/x86/domain.c:arch_domain_create()
> - ->nr_pirqs-sized allocation of ->pirq_to_evtchn[] in
>   xen/common/domain.c:domain_create()

I wonder whether some of these irq-indexed and pirq-indexed values could be
agregated together into two separate structs, and then keep them in
irq-indexed and a pirq-indexed radix trees? It might actually be more space
efficient if nr_irqs or nr_pirqs is large, quite apart from the benefit of
getting rid of the multi-page allocations.

 -- Keir

RE: [PATCH 0/6] x86: break up post-boot non-order-zero allocations

[Dan Magenheimer]

Thanks very much Jan for making forward progress on this!

A couple things:

IIRC, PCI-passthrough is another user of multi-page allocations.

At some point, does it make sense to eliminate the multi-page
allocation functionality, at least through the "normal"
page allocation routines, and add a:

/* documentation about why not to use this */
alloc_contiguous_pages_DEPRECATED(order)

call so that it is very explicit when future page allocation
users "regress" by adding a multi-page allocation request?

Thanks again!
Dan

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@novell.com]
> Sent: Tuesday, April 05, 2011 2:19 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] [PATCH 0/6] x86: break up post-boot non-order-zero
> allocations
> 
> While tmem is most affected by this, due to fragmentation it is
> generally
> a bad idea to require runtime allocations of more than a single page in
> size.
> 
> 1: remove direct cpumask_t members from struct vcpu and struct domain
> 2: x86: split struct vcpu
> 3: x86: move pv-only members of struct vcpu to struct pv_vcpu
> 4: x86: split struct domain
> 5: x86: introduce alloc_vcpu_guest_context()
> 6: passthrough: use domain pirq as index of struct hvm_irq_dpci's
> hvm_timer array
> 
> With this, structure sizes are below page size, and no longer depend
> significantly on NR_CPUS. This series, however, doesn't eliminate
> all non-order-zero allocations that happen post boot (i.e. mostly
> during domain creation). Items that are known to need addressing
> are
> - nr_irqs-sized allocation of ->arch.irq_pirq[] in
>   xen/arch/x86/domain.c:arch_domain_create()
> - ->nr_pirqs-sized allocations in
>   xen/drivers/passthrough/io.c:pt_irq_create_bind_vtd()
> - ->nr_pirqs-sized allocation of ->arch.pirq_irq[] in
>   xen/arch/x86/domain.c:arch_domain_create()
> - ->nr_pirqs-sized allocation of ->pirq_to_evtchn[] in
>   xen/common/domain.c:domain_create()
> 
> Signed-off-by: Jan Beulich <jbeulich@novell.com>
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: [PATCH 0/6] x86: break up post-boot non-order-zero allocations

[Jan Beulich]

>>> On 06.04.11 at 04:50, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> IIRC, PCI-passthrough is another user of multi-page allocations.

As noted in the description, with the intended solution being to
switch from arrays to radix trees. Or do you know of other than
the mentioned cases?

> At some point, does it make sense to eliminate the multi-page
> allocation functionality, at least through the "normal"
> page allocation routines, and add a:
> 
> /* documentation about why not to use this */
> alloc_contiguous_pages_DEPRECATED(order)
> 
> call so that it is very explicit when future page allocation
> users "regress" by adding a multi-page allocation request?

I wouldn't go that far. Instead I was considering adding a (perhaps
one-time) warning so that things wouldn't start outright failing, but
remaining code paths get pointed out. But that ought to happen
only once all known allocations have been dealt with.

Jan