Re: [PATCH 5/5] AMD IOMMU: widen NUMA nodes to be allocated from

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 06/03/2015 07:50, Jan Beulich wrote:
>>>> On 05.03.15 at 18:30, <andrew.cooper3@citrix.com> wrote:
>> On 26/02/15 13:56, Jan Beulich wrote:
>>> --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
>>> +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h
>>> @@ -158,12 +158,12 @@ static inline unsigned long region_to_pa
>>>      return (PAGE_ALIGN(addr + size) - (addr & PAGE_MASK)) >> PAGE_SHIFT;
>>>  }
>>>  
>>> -static inline struct page_info* alloc_amd_iommu_pgtable(void)
>>> +static inline struct page_info *alloc_amd_iommu_pgtable(struct domain *d)
>>>  {
>>>      struct page_info *pg;
>>>      void *vaddr;
>>>  
>>> -    pg = alloc_domheap_page(NULL, 0);
>>> +    pg = alloc_domheap_page(d, MEMF_no_owner);
>> Same comment as with the VT-d side of things.  This should be based on
>> the proximity information of the IOMMU, not of the owning domain.
> I think I buy this argument on the VT-d side (under the assumption
> that there's going to be at least one IOMMU per node), but I'm not
> sure here: The most modern AMD box I have has just a single
> IOMMU for 4 nodes it reports.

It is not possible for an IOMMU to cover multiple NUMA nodes worth of
IO, because of the position it has to sit relative to the IO root ports
and QPI/HT links.

In AMD systems, the IOMMUs lives in the northbridges, meaning one per
numa node (as it is the northbridges which contain the hypertransport links)

The BIOS/firmware will only report IOMMUs from northbridges which have
IO connected to their IO hypertransport link (most systems in the wild
have all IO hanging off one or two Numa nodes).  On the other hand, I
have an AMD system with 8 IOMMUs in use.

In Intel systems, there is one IOMMU for each socket (to cover the
on-chip root ports and GPU if applicable) and one IOMMU in the IOH/PCH
(depending on generation) to cover the legacy IO.


In all cases, the IOMMUs are local to a single NUMA node, and would
benefit from having the control pages and pagetables allocated in local RAM.

~Andrew