Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Simon Horman]

On Wed, Apr 22, 2009 at 09:11:58AM +1000, Simon Horman wrote:
> On Tue, Apr 21, 2009 at 05:16:14PM +0800, Cui, Dexuan wrote:
> > Simon Horman wrote:
> > > Hi,
> > > 
> > > in preparation for 3.4.0 I have made a number of updates to the VT-d
> > > Howto. I have tried to retain the original information while
> > > reorganising things a bit. I have also added information on recent
> > > work I have done in this area. 
> > > 
> > > Apologies in advance for any errors, accidental deletions, etc...
> > > 
> > > http://wiki.xensource.com/xenwiki/VTdHowTo
> > 
> > Hi Simon,
> > There is a bug: we can't pci-detach a statically-assigned (I mean: use the pci string in hvm config file) devices for hvm guest.
> > Accordong my limited experience, after a device is assigned to hvm guest, ioemu does write the correct vslots value into xenstore, but looks xend can't get the correct value somwhow.
> > Could you help to have a look? Thanks!
> 
> Sure, I can look into that. Do you have a version that you are worried
> about? It did seem to work the last time that I checked.

Hi Dexuan,

I am seeing this bug too. I'll try and get a fix ASAP.

-- 
Simon Horman
  VA Linux Systems Japan K.K. Satellite Lab in Sydney, Australia
  H: www.vergenet.net/~horms/            W: www.valinux.co.jp/en

Re: Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Akio Takebe]

Hi, Simon

Simon Horman wrote:
> On Wed, Apr 22, 2009 at 09:11:58AM +1000, Simon Horman wrote:
>> On Tue, Apr 21, 2009 at 05:16:14PM +0800, Cui, Dexuan wrote:
>>> Simon Horman wrote:
>>>> Hi,
>>>>
>>>> in preparation for 3.4.0 I have made a number of updates to the VT-d
>>>> Howto. I have tried to retain the original information while
>>>> reorganising things a bit. I have also added information on recent
>>>> work I have done in this area. 
>>>>
>>>> Apologies in advance for any errors, accidental deletions, etc...
>>>>
>>>> http://wiki.xensource.com/xenwiki/VTdHowTo
>>> Hi Simon,
>>> There is a bug: we can't pci-detach a statically-assigned (I mean: use the pci string in hvm config file) devices for hvm guest.
>>> Accordong my limited experience, after a device is assigned to hvm guest, ioemu does write the correct vslots value into xenstore, but looks xend can't get the correct value somwhow.
>>> Could you help to have a look? Thanks!
>> Sure, I can look into that. Do you have a version that you are worried
>> about? It did seem to work the last time that I checked.
> 
> Hi Dexuan,
> 
> I am seeing this bug too. I'll try and get a fix ASAP.
> 

Does AUTO_PHP_SLOT also support hotplug?

@tools/python/xen/xend/XendDomainInfo.py

     def hvm_destroyPCIDevice(self, vslot):
         log.debug("hvm_destroyPCIDevice called %s", vslot)

         if not self.info.is_hvm():
             raise VmError("hvm_destroyPCIDevice called on non-HVM guest")
... snip ...
         if vslot == AUTO_PHP_SLOT:
             raise VmError("Device @ vslot 0x%x doesn't support hotplug." % (vslot))

Best Regards,

Akio Takebe

Re: Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Simon Horman]

On Wed, Apr 22, 2009 at 11:18:53AM +0900, Akio Takebe wrote:
> Hi, Simon
>
> Simon Horman wrote:
>> On Wed, Apr 22, 2009 at 09:11:58AM +1000, Simon Horman wrote:
>>> On Tue, Apr 21, 2009 at 05:16:14PM +0800, Cui, Dexuan wrote:
>>>> Simon Horman wrote:
>>>>> Hi,
>>>>>
>>>>> in preparation for 3.4.0 I have made a number of updates to the VT-d
>>>>> Howto. I have tried to retain the original information while
>>>>> reorganising things a bit. I have also added information on recent
>>>>> work I have done in this area. 
>>>>>
>>>>> Apologies in advance for any errors, accidental deletions, etc...
>>>>>
>>>>> http://wiki.xensource.com/xenwiki/VTdHowTo
>>>> Hi Simon,
>>>> There is a bug: we can't pci-detach a statically-assigned (I mean: use the pci string in hvm config file) devices for hvm guest.
>>>> Accordong my limited experience, after a device is assigned to hvm guest, ioemu does write the correct vslots value into xenstore, but looks xend can't get the correct value somwhow.
>>>> Could you help to have a look? Thanks!
>>> Sure, I can look into that. Do you have a version that you are worried
>>> about? It did seem to work the last time that I checked.
>>
>> Hi Dexuan,
>>
>> I am seeing this bug too. I'll try and get a fix ASAP.
>>
>
> Does AUTO_PHP_SLOT also support hotplug?
>
> @tools/python/xen/xend/XendDomainInfo.py
>
>     def hvm_destroyPCIDevice(self, vslot):
>         log.debug("hvm_destroyPCIDevice called %s", vslot)
>
>         if not self.info.is_hvm():
>             raise VmError("hvm_destroyPCIDevice called on non-HVM guest")
> ... snip ...
>         if vslot == AUTO_PHP_SLOT:
>             raise VmError("Device @ vslot 0x%x doesn't support hotplug." % (vslot))
>

Hi Takebe-san,

I agree with your analysis, the problem seems to be that
AUTO_PHP_SLOT doesn't support hot-plug.

-- 
Simon Horman
  VA Linux Systems Japan K.K. Satellite Lab in Sydney, Australia
  H: www.vergenet.net/~horms/            W: www.valinux.co.jp/en

Re: Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Simon Horman]

On Wed, Apr 22, 2009 at 11:44:50AM +1000, Simon Horman wrote:
> On Wed, Apr 22, 2009 at 09:11:58AM +1000, Simon Horman wrote:
> > On Tue, Apr 21, 2009 at 05:16:14PM +0800, Cui, Dexuan wrote:
> > > Simon Horman wrote:
> > > > Hi,
> > > > 
> > > > in preparation for 3.4.0 I have made a number of updates to the VT-d
> > > > Howto. I have tried to retain the original information while
> > > > reorganising things a bit. I have also added information on recent
> > > > work I have done in this area. 
> > > > 
> > > > Apologies in advance for any errors, accidental deletions, etc...
> > > > 
> > > > http://wiki.xensource.com/xenwiki/VTdHowTo
> > > 
> > > Hi Simon,
> > > There is a bug: we can't pci-detach a statically-assigned (I mean: use the pci string in hvm config file) devices for hvm guest.
> > > Accordong my limited experience, after a device is assigned to hvm guest, ioemu does write the correct vslots value into xenstore, but looks xend can't get the correct value somwhow.
> > > Could you help to have a look? Thanks!
> > 
> > Sure, I can look into that. Do you have a version that you are worried
> > about? It did seem to work the last time that I checked.
> 
> Hi Dexuan,
> 
> I am seeing this bug too. I'll try and get a fix ASAP.

Hi,

I've poked a little further into this and the problem is partly
a missunderstanding on my part.

In the case where a function is specified in the hvm.conf in
the form BUS:DEV:FUNC (e.g. 00:1d.0), then there is a check inside
xm that prevents the function from subsequently being hot-unplugged.
The check is that the vslot number is set to auto-select. And presumably
the idea was that statically assigned devices couldn't be hot-unplugged,
which to be honest was my initial assumption of how things work.

If, however, the function in the hvm.conf is of the form BUS:DEV:FUNC@VSLOT
(e.g. 00:1d.00@1e) which I added recently, then the check is bypassed,
because vslot is set to 1e not auto. This is how I was doing my tests. I
assumed this was the behaviour for all cases, and documented it as such.

This leaves us with 3 choices:

1. Leave the inconsistency, which seems like a bad idea.
2. Add a check to make sure both forms can't be hot-unplugged
   This is more work than;
3. Remove the check and allow statically assigned devies to be removed.
   This seems like a good option, and if there is consensus I can
   fix this up into a propper patch.

diff -r 6ba4e34d21d3 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py	Tue Apr 21 16:31:26 2009 +0100
+++ b/tools/python/xen/xend/XendDomainInfo.py	Wed Apr 22 12:42:58 2009 +1000
@@ -1092,9 +1092,6 @@
         if devnum >= pci_len:
             raise VmError("Device @ vslot 0x%x doesn't exist." % (vslot))
 
-        if vslot == AUTO_PHP_SLOT:
-            raise VmError("Device @ vslot 0x%x doesn't support hotplug." % (vslot))
-
         # Check the co-assignment.
         # To pci-detach a device D from domN, we should ensure: for each DD in the
         # list of D's co-assignment devices, DD is not assigned (to domN).

Re: Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Zhai, Edwin]

Simon Horman wrote:
> Hi,
>
> I've poked a little further into this and the problem is partly
> a missunderstanding on my part.
>
> In the case where a function is specified in the hvm.conf in
> the form BUS:DEV:FUNC (e.g. 00:1d.0), then there is a check inside
> xm that prevents the function from subsequently being hot-unplugged.
> The check is that the vslot number is set to auto-select. And presumably
> the idea was that statically assigned devices couldn't be hot-unplugged,
> which to be honest was my initial assumption of how things work.
>   

All assigned device should not has a "auto-select" vslot after being 
assigned to guest, so I'm curious which lead this "auto-select".
Static assigned device also support hotplug, because qemu will select 
then return the vslot to xend.



> If, however, the function in the hvm.conf is of the form BUS:DEV:FUNC@VSLOT
> (e.g. 00:1d.00@1e) which I added recently, then the check is bypassed,
> because vslot is set to 1e not auto. This is how I was doing my tests. I
> assumed this was the behaviour for all cases, and documented it as such.
>
> This leaves us with 3 choices:
>
> 1. Leave the inconsistency, which seems like a bad idea.
> 2. Add a check to make sure both forms can't be hot-unplugged
>    This is more work than;
> 3. Remove the check and allow statically assigned devies to be removed.
>    This seems like a good option, and if there is consensus I can
>    fix this up into a propper patch.
>
> diff -r 6ba4e34d21d3 tools/python/xen/xend/XendDomainInfo.py
> --- a/tools/python/xen/xend/XendDomainInfo.py	Tue Apr 21 16:31:26 2009 +0100
> +++ b/tools/python/xen/xend/XendDomainInfo.py	Wed Apr 22 12:42:58 2009 +1000
> @@ -1092,9 +1092,6 @@
>          if devnum >= pci_len:
>              raise VmError("Device @ vslot 0x%x doesn't exist." % (vslot))
>  
> -        if vslot == AUTO_PHP_SLOT:
> -            raise VmError("Device @ vslot 0x%x doesn't support hotplug." % (vslot))
> -
>          # Check the co-assignment.
>          # To pci-detach a device D from domN, we should ensure: for each DD in the
>          # list of D's co-assignment devices, DD is not assigned (to domN).
>
>   

-- 
best rgds,
edwin

Re: Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Simon Horman]

On Wed, Apr 22, 2009 at 11:37:30AM +0800, Zhai, Edwin wrote:
>
>
> Simon Horman wrote:
>> Hi,
>>
>> I've poked a little further into this and the problem is partly
>> a missunderstanding on my part.
>>
>> In the case where a function is specified in the hvm.conf in
>> the form BUS:DEV:FUNC (e.g. 00:1d.0), then there is a check inside
>> xm that prevents the function from subsequently being hot-unplugged.
>> The check is that the vslot number is set to auto-select. And presumably
>> the idea was that statically assigned devices couldn't be hot-unplugged,
>> which to be honest was my initial assumption of how things work.
>>   
>
> All assigned device should not has a "auto-select" vslot after being  
> assigned to guest, so I'm curious which lead this "auto-select".
> Static assigned device also support hotplug, because qemu will select  
> then return the vslot to xend.

That is what I thought too, I'll investigate further.

>> If, however, the function in the hvm.conf is of the form BUS:DEV:FUNC@VSLOT
>> (e.g. 00:1d.00@1e) which I added recently, then the check is bypassed,
>> because vslot is set to 1e not auto. This is how I was doing my tests. I
>> assumed this was the behaviour for all cases, and documented it as such.
>>
>> This leaves us with 3 choices:
>>
>> 1. Leave the inconsistency, which seems like a bad idea.
>> 2. Add a check to make sure both forms can't be hot-unplugged
>>    This is more work than;
>> 3. Remove the check and allow statically assigned devies to be removed.
>>    This seems like a good option, and if there is consensus I can
>>    fix this up into a propper patch.
>>
>> diff -r 6ba4e34d21d3 tools/python/xen/xend/XendDomainInfo.py
>> --- a/tools/python/xen/xend/XendDomainInfo.py	Tue Apr 21 16:31:26 2009 +0100
>> +++ b/tools/python/xen/xend/XendDomainInfo.py	Wed Apr 22 12:42:58 2009 +1000
>> @@ -1092,9 +1092,6 @@
>>          if devnum >= pci_len:
>>              raise VmError("Device @ vslot 0x%x doesn't exist." % (vslot))
>>  -        if vslot == AUTO_PHP_SLOT:
>> -            raise VmError("Device @ vslot 0x%x doesn't support hotplug." % (vslot))
>> -
>>          # Check the co-assignment.
>>          # To pci-detach a device D from domN, we should ensure: for each DD in the
>>          # list of D's co-assignment devices, DD is not assigned (to domN).
>>
>>   
>
> -- 
> best rgds,
> edwin
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Simon Horman
  VA Linux Systems Japan K.K. Satellite Lab in Sydney, Australia
  H: www.vergenet.net/~horms/            W: www.valinux.co.jp/en

Re: Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Zhai, Edwin]

Simon,

In previous solution, the "vslots" entry in xenstore is only used for 
qemu => xend communication, and not present until qemu create it.
But with support of vslot in config file, this xenstore entry are 
created by xend at the beginning...  Now xend will get value from itself 
(AUTO_PHP_SLOT when not specified) without waiting for qemu:(

Temp solution is let xend to check if the return value is AUTO_PHP_SLOT, 
but xend still can't know qemu's response. We need make sure,  if can 
not support specified vslots in config, qemu should notify xend. Then 
xend should know if this is qemu's response or the specified vslots 
requirement.

Simon Horman wrote:
> On Wed, Apr 22, 2009 at 11:37:30AM +0800, Zhai, Edwin wrote:
>   
>> Simon Horman wrote:
>>     
>>> Hi,
>>>
>>> I've poked a little further into this and the problem is partly
>>> a missunderstanding on my part.
>>>
>>> In the case where a function is specified in the hvm.conf in
>>> the form BUS:DEV:FUNC (e.g. 00:1d.0), then there is a check inside
>>> xm that prevents the function from subsequently being hot-unplugged.
>>> The check is that the vslot number is set to auto-select. And presumably
>>> the idea was that statically assigned devices couldn't be hot-unplugged,
>>> which to be honest was my initial assumption of how things work.
>>>   
>>>       
>> All assigned device should not has a "auto-select" vslot after being  
>> assigned to guest, so I'm curious which lead this "auto-select".
>> Static assigned device also support hotplug, because qemu will select  
>> then return the vslot to xend.
>>     
>
> That is what I thought too, I'll investigate further.
>
>   
>>> If, however, the function in the hvm.conf is of the form BUS:DEV:FUNC@VSLOT
>>> (e.g. 00:1d.00@1e) which I added recently, then the check is bypassed,
>>> because vslot is set to 1e not auto. This is how I was doing my tests. I
>>> assumed this was the behaviour for all cases, and documented it as such.
>>>
>>> This leaves us with 3 choices:
>>>
>>> 1. Leave the inconsistency, which seems like a bad idea.
>>> 2. Add a check to make sure both forms can't be hot-unplugged
>>>    This is more work than;
>>> 3. Remove the check and allow statically assigned devies to be removed.
>>>    This seems like a good option, and if there is consensus I can
>>>    fix this up into a propper patch.
>>>
>>> diff -r 6ba4e34d21d3 tools/python/xen/xend/XendDomainInfo.py
>>> --- a/tools/python/xen/xend/XendDomainInfo.py	Tue Apr 21 16:31:26 2009 +0100
>>> +++ b/tools/python/xen/xend/XendDomainInfo.py	Wed Apr 22 12:42:58 2009 +1000
>>> @@ -1092,9 +1092,6 @@
>>>          if devnum >= pci_len:
>>>              raise VmError("Device @ vslot 0x%x doesn't exist." % (vslot))
>>>  -        if vslot == AUTO_PHP_SLOT:
>>> -            raise VmError("Device @ vslot 0x%x doesn't support hotplug." % (vslot))
>>> -
>>>          # Check the co-assignment.
>>>          # To pci-detach a device D from domN, we should ensure: for each DD in the
>>>          # list of D's co-assignment devices, DD is not assigned (to domN).
>>>
>>>   
>>>       
>> -- 
>> best rgds,
>> edwin
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>     
>
>   

-- 
best rgds,
edwin

Re: Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Simon Horman]

On Wed, Apr 22, 2009 at 04:28:10PM +0800, Zhai, Edwin wrote:
> Simon,
>
> In previous solution, the "vslots" entry in xenstore is only used for  
> qemu => xend communication, and not present until qemu create it.
> But with support of vslot in config file, this xenstore entry are  
> created by xend at the beginning...  Now xend will get value from itself  
> (AUTO_PHP_SLOT when not specified) without waiting for qemu:(
>
> Temp solution is let xend to check if the return value is AUTO_PHP_SLOT,  
> but xend still can't know qemu's response. We need make sure,  if can  
> not support specified vslots in config, qemu should notify xend. Then  
> xend should know if this is qemu's response or the specified vslots  
> requirement.

After some investigation it turns out that the problem was not with
the use of vslots - which is still only used for qemu=>xend communication.
The problem was with renaming vslt to vslots because in the dictionary
that describes a PCI device, these entries were different.

I have posted a patch to resolve this problem to xen-devel as
"xm: Fix hot-unplug of statically-assigned devices".

http://lists.xensource.com/archives/html/xen-devel/2009-04/msg00897.html

-- 
Simon Horman
  VA Linux Systems Japan K.K. Satellite Lab in Sydney, Australia
  H: www.vergenet.net/~horms/            W: www.valinux.co.jp/en

Re: Bug in pci-detach (Was: Re: Re: VT-d HowTo Update)

[Simon Horman]

On Wed, Apr 22, 2009 at 12:50:00PM +1000, Simon Horman wrote:
> Hi,
> 
> I've poked a little further into this and the problem is partly
> a missunderstanding on my part.
> 
> In the case where a function is specified in the hvm.conf in
> the form BUS:DEV:FUNC (e.g. 00:1d.0), then there is a check inside
> xm that prevents the function from subsequently being hot-unplugged.
> The check is that the vslot number is set to auto-select. And presumably
> the idea was that statically assigned devices couldn't be hot-unplugged,
> which to be honest was my initial assumption of how things work.
> 
> If, however, the function in the hvm.conf is of the form BUS:DEV:FUNC@VSLOT
> (e.g. 00:1d.00@1e) which I added recently, then the check is bypassed,
> because vslot is set to 1e not auto. This is how I was doing my tests. I
> assumed this was the behaviour for all cases, and documented it as such.
> 
> This leaves us with 3 choices:
> 
> 1. Leave the inconsistency, which seems like a bad idea.
> 2. Add a check to make sure both forms can't be hot-unplugged
>    This is more work than;
> 3. Remove the check and allow statically assigned devies to be removed.
>    This seems like a good option, and if there is consensus I can
>    fix this up into a propper patch.
> 
> diff -r 6ba4e34d21d3 tools/python/xen/xend/XendDomainInfo.py
> --- a/tools/python/xen/xend/XendDomainInfo.py	Tue Apr 21 16:31:26 2009 +0100
> +++ b/tools/python/xen/xend/XendDomainInfo.py	Wed Apr 22 12:42:58 2009 +1000
> @@ -1092,9 +1092,6 @@
>          if devnum >= pci_len:
>              raise VmError("Device @ vslot 0x%x doesn't exist." % (vslot))
>  
> -        if vslot == AUTO_PHP_SLOT:
> -            raise VmError("Device @ vslot 0x%x doesn't support hotplug." % (vslot))
> -
>          # Check the co-assignment.
>          # To pci-detach a device D from domN, we should ensure: for each DD in the
>          # list of D's co-assignment devices, DD is not assigned (to domN).

Hi,

I now believe the analysis above to be incorrect and the patch provided
not to be apropriate for 3.4.

I have posted a new fix to xen-devel as "xm: Fix hot-unplug of
statically-assigned devices".

http://lists.xensource.com/archives/html/xen-devel/2009-04/msg00897.html

-- 
Simon Horman
  VA Linux Systems Japan K.K. Satellite Lab in Sydney, Australia
  H: www.vergenet.net/~horms/            W: www.valinux.co.jp/en