Xen 4.2 and PCI hotplug.

Xen 4.2 and PCI hotplug.

[Konrad Rzeszutek Wilk]

Hey,

I was wondering if the ACPI PCI hotplug (so inserting a new PCIe card
in a server that supports said functionality) is something that Intel
has been testing or using?

Re: Xen 4.2 and PCI hotplug.

[Zhang, Xiantao]

Are you playing with Xen ?  so far,  Xen doesn't support PCIe device hot-plug feature yet.  But if you want to hot-plug a PCIe device to a Xen guest, it should supported well.  
Xiantao 

> -----Original Message-----
> From: Konrad Rzeszutek [mailto:ketuzsezr@gmail.com] On Behalf Of Konrad
> Rzeszutek Wilk
> Sent: Wednesday, December 19, 2012 6:46 AM
> To: Zhang, Xiantao; xen-devel@lists.xensource.com
> Subject: Xen 4.2 and PCI hotplug.
> 
> Hey,
> 
> I was wondering if the ACPI PCI hotplug (so inserting a new PCIe card in a
> server that supports said functionality) is something that Intel has been
> testing or using?

Re: Xen 4.2 and PCI hotplug.

[Jan Beulich]

>>> On 19.12.12 at 09:13, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
> Are you playing with Xen ?  so far,  Xen doesn't support PCIe device hot-plug 
> feature yet.

When saying Xen, I assume you mean the pv-ops kernel instead? So
far I was under the impression that this worked even with the very
old 2.6.18 tree (as much or as little as hotplug there worked in the
native case). And given that there are no special requirements on
the hypervisor to make this work, it's not even obvious to me what
would be missing in the pv-ops kernel to make it work.

All that is of course with me not having any practical experience
with hotplug, due to the lack of capable hardware...

Jan

Re: Xen 4.2 and PCI hotplug.

[Zhang, Xiantao]

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Wednesday, December 19, 2012 4:48 PM
> To: Zhang, Xiantao
> Cc: Konrad Rzeszutek Wilk; xen-devel
> Subject: Re: [Xen-devel] Xen 4.2 and PCI hotplug.
> 
> >>> On 19.12.12 at 09:13, "Zhang, Xiantao" <xiantao.zhang@intel.com>
> wrote:
> > Are you playing with Xen ?  so far,  Xen doesn't support PCIe device
> > hot-plug feature yet.
> 
> When saying Xen, I assume you mean the pv-ops kernel instead? So far I was
> under the impression that this worked even with the very old 2.6.18 tree (as
> much or as little as hotplug there worked in the native case). And given that
> there are no special requirements on the hypervisor to make this work, it's
> not even obvious to me what would be missing in the pv-ops kernel to make
> it work. 
Oh, my fault!  Perhaps we don't need to do anything for pv-ops kernel to support device hot-plug if native system has it supported.  Actually, we didn't do such testings before, since it is a native feature, not a Xen-specific one. 
Xiantao

Re: Xen 4.2 and PCI hotplug.

[Andrew Cooper]

On 19/12/2012 09:14, Zhang, Xiantao wrote:
>> -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: Wednesday, December 19, 2012 4:48 PM
>> To: Zhang, Xiantao
>> Cc: Konrad Rzeszutek Wilk; xen-devel
>> Subject: Re: [Xen-devel] Xen 4.2 and PCI hotplug.
>>
>>>>> On 19.12.12 at 09:13, "Zhang, Xiantao" <xiantao.zhang@intel.com>
>> wrote:
>>> Are you playing with Xen ?  so far,  Xen doesn't support PCIe device
>>> hot-plug feature yet.
>> When saying Xen, I assume you mean the pv-ops kernel instead? So far I was
>> under the impression that this worked even with the very old 2.6.18 tree (as
>> much or as little as hotplug there worked in the native case). And given that
>> there are no special requirements on the hypervisor to make this work, it's
>> not even obvious to me what would be missing in the pv-ops kernel to make
>> it work. 
> Oh, my fault!  Perhaps we don't need to do anything for pv-ops kernel to support device hot-plug if native system has it supported.  Actually, we didn't do such testings before, since it is a native feature, not a Xen-specific one. 
> Xiantao

My current understanding is that on boot, Xen scans the PCI bus, then
dom0 rescans it later.  If a hotplug event gets serviced by dom0, does
there not need to be some hypercall informing Xen that a new device has
appeared?  I expect PCIPassthrough would not work correctly on a
hotplugged device which Xen is unaware of.

(But if I have got the wrong end of the stick, or this mechanism already
exist, please ignore me.  It just strikes me as a little xen-specific,
even if the bulk of it is native)

~Andrew

>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Xen 4.2 and PCI hotplug.

[Jan Beulich]

>>> On 19.12.12 at 15:36, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> On 19/12/2012 09:14, Zhang, Xiantao wrote:
>>> -----Original Message-----
>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>> Sent: Wednesday, December 19, 2012 4:48 PM
>>> To: Zhang, Xiantao
>>> Cc: Konrad Rzeszutek Wilk; xen-devel
>>> Subject: Re: [Xen-devel] Xen 4.2 and PCI hotplug.
>>>
>>>>>> On 19.12.12 at 09:13, "Zhang, Xiantao" <xiantao.zhang@intel.com>
>>> wrote:
>>>> Are you playing with Xen ?  so far,  Xen doesn't support PCIe device
>>>> hot-plug feature yet.
>>> When saying Xen, I assume you mean the pv-ops kernel instead? So far I was
>>> under the impression that this worked even with the very old 2.6.18 tree (as
>>> much or as little as hotplug there worked in the native case). And given 
> that
>>> there are no special requirements on the hypervisor to make this work, it's
>>> not even obvious to me what would be missing in the pv-ops kernel to make
>>> it work. 
>> Oh, my fault!  Perhaps we don't need to do anything for pv-ops kernel to 
> support device hot-plug if native system has it supported.  Actually, we 
> didn't do such testings before, since it is a native feature, not a 
> Xen-specific one. 
>> Xiantao
> 
> My current understanding is that on boot, Xen scans the PCI bus, then
> dom0 rescans it later.  If a hotplug event gets serviced by dom0, does
> there not need to be some hypercall informing Xen that a new device has
> appeared?  I expect PCIPassthrough would not work correctly on a
> hotplugged device which Xen is unaware of.

Sure - such a hypercall exists and is - from all I can tell - being made
not only during the boot time bus scan, but also during hotplug
processing. See drivers/xen/pci.c.

Jan

Re: Xen 4.2 and PCI hotplug.

[Konrad Rzeszutek Wilk]

On Wed, Dec 19, 2012 at 03:08:31PM +0000, Jan Beulich wrote:
> >>> On 19.12.12 at 15:36, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> > On 19/12/2012 09:14, Zhang, Xiantao wrote:
> >>> -----Original Message-----
> >>> From: Jan Beulich [mailto:JBeulich@suse.com]
> >>> Sent: Wednesday, December 19, 2012 4:48 PM
> >>> To: Zhang, Xiantao
> >>> Cc: Konrad Rzeszutek Wilk; xen-devel
> >>> Subject: Re: [Xen-devel] Xen 4.2 and PCI hotplug.
> >>>
> >>>>>> On 19.12.12 at 09:13, "Zhang, Xiantao" <xiantao.zhang@intel.com>
> >>> wrote:
> >>>> Are you playing with Xen ?  so far,  Xen doesn't support PCIe device
> >>>> hot-plug feature yet.
> >>> When saying Xen, I assume you mean the pv-ops kernel instead? So far I was
> >>> under the impression that this worked even with the very old 2.6.18 tree (as
> >>> much or as little as hotplug there worked in the native case). And given 
> > that
> >>> there are no special requirements on the hypervisor to make this work, it's
> >>> not even obvious to me what would be missing in the pv-ops kernel to make
> >>> it work. 
> >> Oh, my fault!  Perhaps we don't need to do anything for pv-ops kernel to 
> > support device hot-plug if native system has it supported.  Actually, we 
> > didn't do such testings before, since it is a native feature, not a 
> > Xen-specific one. 
> >> Xiantao
> > 
> > My current understanding is that on boot, Xen scans the PCI bus, then
> > dom0 rescans it later.  If a hotplug event gets serviced by dom0, does
> > there not need to be some hypercall informing Xen that a new device has
> > appeared?  I expect PCIPassthrough would not work correctly on a
> > hotplugged device which Xen is unaware of.
> 
> Sure - such a hypercall exists and is - from all I can tell - being made
> not only during the boot time bus scan, but also during hotplug
> processing. See drivers/xen/pci.c.

Right. What I am not sure about - and this is why I am asking Intel - whether
that notifier gets called _after_ dom0 has interogated the new PCI device.

And if so, would Xen (which gets trapped on the 0xcf8) would not let dom0
actually interogate the new BDF b/c it hasn't been told about this BDF
existence.

This is all speculation - and I might be very well completly wrong about
all of this - hence this email-thread.

Testing would of course tell for sure, but I am not sure if I even have
such hardware.
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: Xen 4.2 and PCI hotplug.

[Konrad Rzeszutek Wilk]

On Wed, Dec 19, 2012 at 08:47:58AM +0000, Jan Beulich wrote:
> >>> On 19.12.12 at 09:13, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
> > Are you playing with Xen ?  so far,  Xen doesn't support PCIe device hot-plug 
> > feature yet.
> 
> When saying Xen, I assume you mean the pv-ops kernel instead? So

I am looking at the PCI AER which can do native hotplug. And from
the Linux side it ought to work right -- but there is some scanning
of said new PCI device "before" the notifications are sent (which
we latch on and do the hypercall on). At least that is from a cursory
read of the code.

I presume that "before" the hypercall is sent Xen would just
ignore the 0xcf8 reads. But after the hypercall it would populate
appropiately whatever it needs. But it would never get to that point
b/c Linux couldn't read the device.

I might be of course completly off-base - hence asking if
Intel has done any sort of testing on this?

> far I was under the impression that this worked even with the very
> old 2.6.18 tree (as much or as little as hotplug there worked in the
> native case). And given that there are no special requirements on
> the hypervisor to make this work, it's not even obvious to me what
> would be missing in the pv-ops kernel to make it work.
> 
> All that is of course with me not having any practical experience
> with hotplug, due to the lack of capable hardware...
> 
> Jan
> 
>