[PATCH, RFC 0/7] PCI multi-segment support

[PATCH, RFC 0/7] PCI multi-segment support

[Jan Beulich]

In order for Xen to be able to boot on systems with multiple PCI segments
(also called domains), a number of changes are necessary to the
hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as
in most code paths and definitions there were not even provisions for
passing a segment number.

The hypercall interface changes may need some discussion before
applying the patches, in particular

- whether the way PHYSDEVOP_map_pirq gets re-used is acceptable,
  or whether alternatively we should define a replacement one sub-
  hypercall
- whether PHYSDEVOP_manage_pci_* should be deprecated
- whether the bit assignments for the four uses of machine_bdf in
  the domctl interface can be re-defined

Additionally, in the AMD IOMMU code there are two places where I
was unable to identify how the segment value ought to be retrieved.
Since I'm unaware of multi-segment AMD-based systems, imo this
should not be a reason to not commit the changes proposed.

1: introduce notion of PCI segments
2: add new physdevop-s
3: adjust domctl interface
4: VT-d specific adjustments
5: AMD-IOMMU specific adjustments
6: Pass-through adjustments
7: config space accessor adjustments

Signed-off-by: Jan Beulich <jbeulich@novell.com>

Re: [PATCH, RFC 0/7] PCI multi-segment support

[Jan Beulich]

>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote:
> In order for Xen to be able to boot on systems with multiple PCI segments
> (also called domains), a number of changes are necessary to the
> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as
> in most code paths and definitions there were not even provisions for
> passing a segment number.
> 
> The hypercall interface changes may need some discussion before
> applying the patches, in particular
> 
> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable,
>   or whether alternatively we should define a replacement one sub-
>   hypercall
> - whether PHYSDEVOP_manage_pci_* should be deprecated
> - whether the bit assignments for the four uses of machine_bdf in
>   the domctl interface can be re-defined

No comment from either of you on the proposed changes?

Jan

> Additionally, in the AMD IOMMU code there are two places where I
> was unable to identify how the segment value ought to be retrieved.
> Since I'm unaware of multi-segment AMD-based systems, imo this
> should not be a reason to not commit the changes proposed.
> 
> 1: introduce notion of PCI segments
> 2: add new physdevop-s
> 3: adjust domctl interface
> 4: VT-d specific adjustments
> 5: AMD-IOMMU specific adjustments
> 6: Pass-through adjustments
> 7: config space accessor adjustments
> 
> Signed-off-by: Jan Beulich <jbeulich@novell.com>

Re: [PATCH, RFC 0/7] PCI multi-segment support

[Keir Fraser]

On 05/09/2011 14:18, "Jan Beulich" <JBeulich@suse.com> wrote:

>>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote:
>> In order for Xen to be able to boot on systems with multiple PCI segments
>> (also called domains), a number of changes are necessary to the
>> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as
>> in most code paths and definitions there were not even provisions for
>> passing a segment number.
>> 
>> The hypercall interface changes may need some discussion before
>> applying the patches, in particular
>> 
>> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable,
>>   or whether alternatively we should define a replacement one sub-
>>   hypercall
>> - whether PHYSDEVOP_manage_pci_* should be deprecated
>> - whether the bit assignments for the four uses of machine_bdf in
>>   the domctl interface can be re-defined
> 
> No comment from either of you on the proposed changes?

I'm personally fine with folding segment into the bus field. Otherwise we
just end up with more compat cruft.

I don't have an opinion on the PHYSDEVOP_manage_pci_* hypercalls. In fact I
don't know much about them at all.

I've always considered the domctl interface subject to change, but you don't
seem to redefine anything that already exists? You just give meaning to bits
24-31 of an existing 32-bit parameter?

 -- Keir

> Jan
> 
>> Additionally, in the AMD IOMMU code there are two places where I
>> was unable to identify how the segment value ought to be retrieved.
>> Since I'm unaware of multi-segment AMD-based systems, imo this
>> should not be a reason to not commit the changes proposed.
>> 
>> 1: introduce notion of PCI segments
>> 2: add new physdevop-s
>> 3: adjust domctl interface
>> 4: VT-d specific adjustments
>> 5: AMD-IOMMU specific adjustments
>> 6: Pass-through adjustments
>> 7: config space accessor adjustments
>> 
>> Signed-off-by: Jan Beulich <jbeulich@novell.com>
> 

Re: [PATCH, RFC 0/7] PCI multi-segment support

[Jan Beulich]

>>> On 05.09.11 at 15:33, Keir Fraser <keir.xen@gmail.com> wrote:
> On 05/09/2011 14:18, "Jan Beulich" <JBeulich@suse.com> wrote:
> 
>>>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote:
>>> In order for Xen to be able to boot on systems with multiple PCI segments
>>> (also called domains), a number of changes are necessary to the
>>> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as
>>> in most code paths and definitions there were not even provisions for
>>> passing a segment number.
>>> 
>>> The hypercall interface changes may need some discussion before
>>> applying the patches, in particular
>>> 
>>> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable,
>>>   or whether alternatively we should define a replacement one sub-
>>>   hypercall
>>> - whether PHYSDEVOP_manage_pci_* should be deprecated
>>> - whether the bit assignments for the four uses of machine_bdf in
>>>   the domctl interface can be re-defined
>> 
>> No comment from either of you on the proposed changes?
> 
> I'm personally fine with folding segment into the bus field. Otherwise we
> just end up with more compat cruft.
> 
> I don't have an opinion on the PHYSDEVOP_manage_pci_* hypercalls. In fact I
> don't know much about them at all.
> 
> I've always considered the domctl interface subject to change, but you don't
> seem to redefine anything that already exists? You just give meaning to bits
> 24-31 of an existing 32-bit parameter?

I'm trying to avoid incompatible changes when possible (due to
out-of-tree consumers like libvirt, and due to the hacks required to
use domctl interfaces from the kernel). Now here we need 16 bits, but
have two sets of 8 (at bottom and top), hence I'd favor doing an
incompatible change here (moving the bdf bits down to 0...15, and
using 16...31 for the segment), perhaps renaming the field to
machine_sbdf (to force compile-time noticing of the change at least
for those that actually use our headers). But as the odd bit assignment
could have other (hidden) reasons I coded things first to not do any
re-assignments.

Jan

Re: [PATCH, RFC 0/7] PCI multi-segment support

[Keir Fraser]

On 05/09/2011 14:49, "Jan Beulich" <JBeulich@suse.com> wrote:

>>>> On 05.09.11 at 15:33, Keir Fraser <keir.xen@gmail.com> wrote:
>> On 05/09/2011 14:18, "Jan Beulich" <JBeulich@suse.com> wrote:
>> 
>>>>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote:
>>>> In order for Xen to be able to boot on systems with multiple PCI segments
>>>> (also called domains), a number of changes are necessary to the
>>>> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as
>>>> in most code paths and definitions there were not even provisions for
>>>> passing a segment number.
>>>> 
>>>> The hypercall interface changes may need some discussion before
>>>> applying the patches, in particular
>>>> 
>>>> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable,
>>>>   or whether alternatively we should define a replacement one sub-
>>>>   hypercall
>>>> - whether PHYSDEVOP_manage_pci_* should be deprecated
>>>> - whether the bit assignments for the four uses of machine_bdf in
>>>>   the domctl interface can be re-defined
>>> 
>>> No comment from either of you on the proposed changes?
>> 
>> I'm personally fine with folding segment into the bus field. Otherwise we
>> just end up with more compat cruft.
>> 
>> I don't have an opinion on the PHYSDEVOP_manage_pci_* hypercalls. In fact I
>> don't know much about them at all.
>> 
>> I've always considered the domctl interface subject to change, but you don't
>> seem to redefine anything that already exists? You just give meaning to bits
>> 24-31 of an existing 32-bit parameter?
> 
> I'm trying to avoid incompatible changes when possible (due to
> out-of-tree consumers like libvirt,

I think the intention is to maintain API compatibility for libxenlight, and
have out-of-tree tool stacks/librariues build on top of that.

I think there are libvirt bindings to libxenlight now, for example?

My conclusion would be you can do the cleaner change to domctl. Interested
in Ian Jackson's view however.

 -- Keir

> and due to the hacks required to
> use domctl interfaces from the kernel). Now here we need 16 bits, but
> have two sets of 8 (at bottom and top), hence I'd favor doing an
> incompatible change here (moving the bdf bits down to 0...15, and
> using 16...31 for the segment), perhaps renaming the field to
> machine_sbdf (to force compile-time noticing of the change at least
> for those that actually use our headers). But as the odd bit assignment
> could have other (hidden) reasons I coded things first to not do any
> re-assignments.
> 
> Jan
> 

Re: [PATCH, RFC 0/7] PCI multi-segment support

[Jan Beulich]

>>> On 05.09.11 at 16:05, Keir Fraser <keir@xen.org> wrote:
> On 05/09/2011 14:49, "Jan Beulich" <JBeulich@suse.com> wrote:
> 
>>>>> On 05.09.11 at 15:33, Keir Fraser <keir.xen@gmail.com> wrote:
>>> On 05/09/2011 14:18, "Jan Beulich" <JBeulich@suse.com> wrote:
>>> 
>>>>>>> On 25.08.11 at 16:54, "Jan Beulich" <JBeulich@novell.com> wrote:
>>>>> In order for Xen to be able to boot on systems with multiple PCI segments
>>>>> (also called domains), a number of changes are necessary to the
>>>>> hypervisor, the hypercall interface, the tools, and the Dom0 kernel, as
>>>>> in most code paths and definitions there were not even provisions for
>>>>> passing a segment number.
>>>>> 
>>>>> The hypercall interface changes may need some discussion before
>>>>> applying the patches, in particular
>>>>> 
>>>>> - whether the way PHYSDEVOP_map_pirq gets re-used is acceptable,
>>>>>   or whether alternatively we should define a replacement one sub-
>>>>>   hypercall
>>>>> - whether PHYSDEVOP_manage_pci_* should be deprecated
>>>>> - whether the bit assignments for the four uses of machine_bdf in
>>>>>   the domctl interface can be re-defined
>>>> 
>>>> No comment from either of you on the proposed changes?
>>> 
>>> I'm personally fine with folding segment into the bus field. Otherwise we
>>> just end up with more compat cruft.
>>> 
>>> I don't have an opinion on the PHYSDEVOP_manage_pci_* hypercalls. In fact I
>>> don't know much about them at all.
>>> 
>>> I've always considered the domctl interface subject to change, but you don't
>>> seem to redefine anything that already exists? You just give meaning to bits
>>> 24-31 of an existing 32-bit parameter?
>> 
>> I'm trying to avoid incompatible changes when possible (due to
>> out-of-tree consumers like libvirt,
> 
> I think the intention is to maintain API compatibility for libxenlight, and
> have out-of-tree tool stacks/librariues build on top of that.
> 
> I think there are libvirt bindings to libxenlight now, for example?
> 
> My conclusion would be you can do the cleaner change to domctl. Interested
> in Ian Jackson's view however.

Ian?

>> and due to the hacks required to
>> use domctl interfaces from the kernel). Now here we need 16 bits, but
>> have two sets of 8 (at bottom and top), hence I'd favor doing an
>> incompatible change here (moving the bdf bits down to 0...15, and
>> using 16...31 for the segment), perhaps renaming the field to
>> machine_sbdf (to force compile-time noticing of the change at least
>> for those that actually use our headers). But as the odd bit assignment
>> could have other (hidden) reasons I coded things first to not do any
>> re-assignments.
>> 
>> Jan
>> 

Re: [PATCH, RFC 0/7] PCI multi-segment support

[Ian Jackson]

Keir Fraser writes ("Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment support"):
> I think the intention is to maintain API compatibility for libxenlight, and
> have out-of-tree tool stacks/librariues build on top of that.

Yes, but we're not there yet.

> I think there are libvirt bindings to libxenlight now, for example?

There are but I'm not sure of their release status.

> My conclusion would be you can do the cleaner change to domctl. Interested
> in Ian Jackson's view however.

We need to preserve compatibility with other versions of linux and
qemu.  Do these programs need to know about these changes too ?

Ian.

Re: [PATCH, RFC 0/7] PCI multi-segment support

[Jan Beulich]

>>> On 20.09.11 at 20:02, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Keir Fraser writes ("Re: [Xen-devel] [PATCH, RFC 0/7] PCI multi-segment 
> support"):
>> My conclusion would be you can do the cleaner change to domctl. Interested
>> in Ian Jackson's view however.
> 
> We need to preserve compatibility with other versions of linux and
> qemu.  Do these programs need to know about these changes too ?

I didn't find any references to those domctl operations there (the
kernel certainly shouldn't have any, despite there being a couple of
things that currently can't be done without the use of domctls in
the kernel), and qemu also shouldn't have anything to do with the
operations changed here.

Jan