Bug when disabling/enabling a PT device with MSI enabled

Bug when disabling/enabling a PT device with MSI enabled

[Tom Rotenberg]

Hi,

I am using the xen-3,4, and i am doing pass-through for a device with
MSI enabled (the device of-course supports MSI) to a domU with Windows
XP. When i disable and then re-enable the device using Windows device
manager, everything looks like it's working fine, but the device seems
to malfunction.
After digging a little bit, i saw that the problem is, because the bit
of the interrupt-status (bit 3) in the status register (offset 0x6 in
the PCI config space) was turned on somehow -  this proibably caused
the MSI to malfunction (because the IntX assertion was enabled - thus
disabling the MSI) - and this lead to the problem i experienced.

Any ideas on how to solve this issue?

Tom

Re: Bug when disabling/enabling a PT device with MSI enabled

[Konrad Rzeszutek Wilk]

On Tue, Nov 24, 2009 at 08:42:49PM +0200, Tom Rotenberg wrote:
> Hi,
> 
> I am using the xen-3,4, and i am doing pass-through for a device with
> MSI enabled (the device of-course supports MSI) to a domU with Windows
> XP. When i disable and then re-enable the device using Windows device
> manager, everything looks like it's working fine, but the device seems
> to malfunction.
> After digging a little bit, i saw that the problem is, because the bit
> of the interrupt-status (bit 3) in the status register (offset 0x6 in
> the PCI config space) was turned on somehow -  this proibably caused
> the MSI to malfunction (because the IntX assertion was enabled - thus
> disabling the MSI) - and this lead to the problem i experienced.
> 
> Any ideas on how to solve this issue?

Looks like QEMU is not doing its job. It ought to be tracking
when an MSI devices transforms to a INTx PCI device (and vice-versa) and
make the appropiate Xen calls. Look in ioemu-dir/hw/pass-through.c

Re: Bug when disabling/enabling a PT device with MSI enabled

[Tom Rotenberg]

Can u please supply more details? how can i fix this issue?
I think that when the hardware reset occured, the MSI was disabled,
and that needs to be re-enabled again... no?

On Tue, Nov 24, 2009 at 9:48 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Tue, Nov 24, 2009 at 08:42:49PM +0200, Tom Rotenberg wrote:
>> Hi,
>>
>> I am using the xen-3,4, and i am doing pass-through for a device with
>> MSI enabled (the device of-course supports MSI) to a domU with Windows
>> XP. When i disable and then re-enable the device using Windows device
>> manager, everything looks like it's working fine, but the device seems
>> to malfunction.
>> After digging a little bit, i saw that the problem is, because the bit
>> of the interrupt-status (bit 3) in the status register (offset 0x6 in
>> the PCI config space) was turned on somehow -  this proibably caused
>> the MSI to malfunction (because the IntX assertion was enabled - thus
>> disabling the MSI) - and this lead to the problem i experienced.
>>
>> Any ideas on how to solve this issue?
>
> Looks like QEMU is not doing its job. It ought to be tracking
> when an MSI devices transforms to a INTx PCI device (and vice-versa) and
> make the appropiate Xen calls. Look in ioemu-dir/hw/pass-through.c
>

Re: Bug when disabling/enabling a PT device with MSI enabled

[Konrad Rzeszutek Wilk]

On Tue, Nov 24, 2009 at 10:27:38PM +0200, Tom Rotenberg wrote:
> Can u please supply more details? how can i fix this issue?

.. snip ..
> > make the appropiate Xen calls. Look in ioemu-dir/hw/pass-through.c

Start digging in the code.

RE: Bug when disabling/enabling a PT device with MSI enabled

[Kamala Narasimhan]

Couple of questions - 

1) What kind of device are you referring to below?  Display by any chance?
2) Which tree is your dom0 kernel based on?

There is a chance you are missing some msi patches in your dom0 kernel.  You might want to do a quick search for msi patches in http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and see if you are missing anything obvious.  I recently encountered a similar issue in a slightly different situation, a subset of the problem was caused because of missing msi patches.

Kamala

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
> Sent: Tuesday, November 24, 2009 1:43 PM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] Bug when disabling/enabling a PT device with MSI
> enabled
> 
> Hi,
> 
> I am using the xen-3,4, and i am doing pass-through for a device with
> MSI enabled (the device of-course supports MSI) to a domU with Windows
> XP. When i disable and then re-enable the device using Windows device
> manager, everything looks like it's working fine, but the device seems
> to malfunction.
> After digging a little bit, i saw that the problem is, because the bit
> of the interrupt-status (bit 3) in the status register (offset 0x6 in
> the PCI config space) was turned on somehow -  this proibably caused
> the MSI to malfunction (because the IntX assertion was enabled - thus
> disabling the MSI) - and this lead to the problem i experienced.
> 
> Any ideas on how to solve this issue?
> 
> Tom
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Bug when disabling/enabling a PT device with MSI enabled

[Tom Rotenberg]

Hi,

1) It's not display, i am using a wlan device (which Windows tends to
disable and re-enable in certain cases)
2) I am using a 2.6.24 xen kernel

I will do the check with the 2.6.18 xen kernel, and see if it's
responsible for this bug.

On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
<Kamala.Narasimhan@citrix.com> wrote:
>
> Couple of questions -
>
> 1) What kind of device are you referring to below?  Display by any chance?
> 2) Which tree is your dom0 kernel based on?
>
> There is a chance you are missing some msi patches in your dom0 kernel.  You might want to do a quick search for msi patches in http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and see if you are missing anything obvious.  I recently encountered a similar issue in a slightly different situation, a subset of the problem was caused because of missing msi patches.
>
> Kamala
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>> Sent: Tuesday, November 24, 2009 1:43 PM
>> To: xen-devel@lists.xensource.com
>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with MSI
>> enabled
>>
>> Hi,
>>
>> I am using the xen-3,4, and i am doing pass-through for a device with
>> MSI enabled (the device of-course supports MSI) to a domU with Windows
>> XP. When i disable and then re-enable the device using Windows device
>> manager, everything looks like it's working fine, but the device seems
>> to malfunction.
>> After digging a little bit, i saw that the problem is, because the bit
>> of the interrupt-status (bit 3) in the status register (offset 0x6 in
>> the PCI config space) was turned on somehow -  this proibably caused
>> the MSI to malfunction (because the IntX assertion was enabled - thus
>> disabling the MSI) - and this lead to the problem i experienced.
>>
>> Any ideas on how to solve this issue?
>>
>> Tom
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

Re: Bug when disabling/enabling a PT device with MSI enabled

[Tom Rotenberg]

Well, just tested it with the 2.6.18 latest kernel, and the bug still exists.
Also, i think that iit's more related to how qemu handles the
"disable/enable" of domU for PT devices with MSI... but i don't know
where to start from? it looks like when re-enabling a PT device with
MSI, we need qemu to re-initialize the device according to it's
original MSI state (and not to what it was resetted before).

Does anyone has any clue about it?

On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
<Kamala.Narasimhan@citrix.com> wrote:
>
> Couple of questions -
>
> 1) What kind of device are you referring to below?  Display by any chance?
> 2) Which tree is your dom0 kernel based on?
>
> There is a chance you are missing some msi patches in your dom0 kernel.  You might want to do a quick search for msi patches in http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and see if you are missing anything obvious.  I recently encountered a similar issue in a slightly different situation, a subset of the problem was caused because of missing msi patches.
>
> Kamala
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>> Sent: Tuesday, November 24, 2009 1:43 PM
>> To: xen-devel@lists.xensource.com
>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with MSI
>> enabled
>>
>> Hi,
>>
>> I am using the xen-3,4, and i am doing pass-through for a device with
>> MSI enabled (the device of-course supports MSI) to a domU with Windows
>> XP. When i disable and then re-enable the device using Windows device
>> manager, everything looks like it's working fine, but the device seems
>> to malfunction.
>> After digging a little bit, i saw that the problem is, because the bit
>> of the interrupt-status (bit 3) in the status register (offset 0x6 in
>> the PCI config space) was turned on somehow -  this proibably caused
>> the MSI to malfunction (because the IntX assertion was enabled - thus
>> disabling the MSI) - and this lead to the problem i experienced.
>>
>> Any ideas on how to solve this issue?
>>
>> Tom
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

RE: Bug when disabling/enabling a PT device with MSI enabled

[Jiang, Yunhong]

This should be about qemu's MSI support. The code is in qemu/hw/pt-msi.c for all MSI emulation. 
Windows XP does not support MSI if I remember correctly. SO this issue should be about MSI translation, i.e. guest is using IOAPIC while host is using MSI. Xen hypervisor will translate the MSI to guest's IOAPIC. 

--jyh
________________________________________
From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Tom Rotenberg [tom.rotenberg@gmail.com]
Sent: Wednesday, November 25, 2009 1:27 AM
To: Kamala Narasimhan
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled

Well, just tested it with the 2.6.18 latest kernel, and the bug still exists.
Also, i think that iit's more related to how qemu handles the
"disable/enable" of domU for PT devices with MSI... but i don't know
where to start from? it looks like when re-enabling a PT device with
MSI, we need qemu to re-initialize the device according to it's
original MSI state (and not to what it was resetted before).

Does anyone has any clue about it?

On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
<Kamala.Narasimhan@citrix.com> wrote:
>
> Couple of questions -
>
> 1) What kind of device are you referring to below?  Display by any chance?
> 2) Which tree is your dom0 kernel based on?
>
> There is a chance you are missing some msi patches in your dom0 kernel.  You might want to do a quick search for msi patches in http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and see if you are missing anything obvious.  I recently encountered a similar issue in a slightly different situation, a subset of the problem was caused because of missing msi patches.
>
> Kamala
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>> Sent: Tuesday, November 24, 2009 1:43 PM
>> To: xen-devel@lists.xensource.com
>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with MSI
>> enabled
>>
>> Hi,
>>
>> I am using the xen-3,4, and i am doing pass-through for a device with
>> MSI enabled (the device of-course supports MSI) to a domU with Windows
>> XP. When i disable and then re-enable the device using Windows device
>> manager, everything looks like it's working fine, but the device seems
>> to malfunction.
>> After digging a little bit, i saw that the problem is, because the bit
>> of the interrupt-status (bit 3) in the status register (offset 0x6 in
>> the PCI config space) was turned on somehow -  this proibably caused
>> the MSI to malfunction (because the IntX assertion was enabled - thus
>> disabling the MSI) - and this lead to the problem i experienced.
>>
>> Any ideas on how to solve this issue?
>>
>> Tom
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Bug when disabling/enabling a PT device with MSI enabled

[Jiang, Yunhong]

Sorry send out the mail too quickly.
I realized it is MSI translation when I finished first section but forgot remove it, so please ignore  first section.
You only need care about MSI translation, the code is in hw/pass-through.c.  You can check qemu's changelog at a8d1bd66.

--jyh
________________________________________
From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Jiang, Yunhong [yunhong.jiang@intel.com]
Sent: Wednesday, November 25, 2009 2:10 AM
To: Tom Rotenberg; Kamala Narasimhan
Cc: xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled

This should be about qemu's MSI support. The code is in qemu/hw/pt-msi.c for all MSI emulation.
Windows XP does not support MSI if I remember correctly. SO this issue should be about MSI translation, i.e. guest is using IOAPIC while host is using MSI. Xen hypervisor will translate the MSI to guest's IOAPIC.

--jyh
________________________________________
From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Tom Rotenberg [tom.rotenberg@gmail.com]
Sent: Wednesday, November 25, 2009 1:27 AM
To: Kamala Narasimhan
Cc: xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled

Well, just tested it with the 2.6.18 latest kernel, and the bug still exists.
Also, i think that iit's more related to how qemu handles the
"disable/enable" of domU for PT devices with MSI... but i don't know
where to start from? it looks like when re-enabling a PT device with
MSI, we need qemu to re-initialize the device according to it's
original MSI state (and not to what it was resetted before).

Does anyone has any clue about it?

On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
<Kamala.Narasimhan@citrix.com> wrote:
>
> Couple of questions -
>
> 1) What kind of device are you referring to below?  Display by any chance?
> 2) Which tree is your dom0 kernel based on?
>
> There is a chance you are missing some msi patches in your dom0 kernel.  You might want to do a quick search for msi patches in http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and see if you are missing anything obvious.  I recently encountered a similar issue in a slightly different situation, a subset of the problem was caused because of missing msi patches.
>
> Kamala
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>> Sent: Tuesday, November 24, 2009 1:43 PM
>> To: xen-devel@lists.xensource.com
>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with MSI
>> enabled
>>
>> Hi,
>>
>> I am using the xen-3,4, and i am doing pass-through for a device with
>> MSI enabled (the device of-course supports MSI) to a domU with Windows
>> XP. When i disable and then re-enable the device using Windows device
>> manager, everything looks like it's working fine, but the device seems
>> to malfunction.
>> After digging a little bit, i saw that the problem is, because the bit
>> of the interrupt-status (bit 3) in the status register (offset 0x6 in
>> the PCI config space) was turned on somehow -  this proibably caused
>> the MSI to malfunction (because the IntX assertion was enabled - thus
>> disabling the MSI) - and this lead to the problem i experienced.
>>
>> Any ideas on how to solve this issue?
>>
>> Tom
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Bug when disabling/enabling a PT device with MSI enabled

[Tom Rotenberg]

Well, to try and see if the problem is msi related or not, i
registered the device with the option of 'msi_translate=0' to force
the use of intx instead of MSI, and i still got the problem... so i
guess it's not related to the MSI.

It looks like the IntX bit (bit 3 of status word) is runed on somehow
outside qemu, and that's the problem... could that be?

On Wed, Nov 25, 2009 at 11:17 AM, Jiang, Yunhong
<yunhong.jiang@intel.com> wrote:
> Sorry send out the mail too quickly.
> I realized it is MSI translation when I finished first section but forgot remove it, so please ignore  first section.
> You only need care about MSI translation, the code is in hw/pass-through.c.  You can check qemu's changelog at a8d1bd66.
>
> --jyh
> ________________________________________
> From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Jiang, Yunhong [yunhong.jiang@intel.com]
> Sent: Wednesday, November 25, 2009 2:10 AM
> To: Tom Rotenberg; Kamala Narasimhan
> Cc: xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled
>
> This should be about qemu's MSI support. The code is in qemu/hw/pt-msi.c for all MSI emulation.
> Windows XP does not support MSI if I remember correctly. SO this issue should be about MSI translation, i.e. guest is using IOAPIC while host is using MSI. Xen hypervisor will translate the MSI to guest's IOAPIC.
>
> --jyh
> ________________________________________
> From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Tom Rotenberg [tom.rotenberg@gmail.com]
> Sent: Wednesday, November 25, 2009 1:27 AM
> To: Kamala Narasimhan
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled
>
> Well, just tested it with the 2.6.18 latest kernel, and the bug still exists.
> Also, i think that iit's more related to how qemu handles the
> "disable/enable" of domU for PT devices with MSI... but i don't know
> where to start from? it looks like when re-enabling a PT device with
> MSI, we need qemu to re-initialize the device according to it's
> original MSI state (and not to what it was resetted before).
>
> Does anyone has any clue about it?
>
> On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
> <Kamala.Narasimhan@citrix.com> wrote:
>>
>> Couple of questions -
>>
>> 1) What kind of device are you referring to below?  Display by any chance?
>> 2) Which tree is your dom0 kernel based on?
>>
>> There is a chance you are missing some msi patches in your dom0 kernel.  You might want to do a quick search for msi patches in http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and see if you are missing anything obvious.  I recently encountered a similar issue in a slightly different situation, a subset of the problem was caused because of missing msi patches.
>>
>> Kamala
>>
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>>> Sent: Tuesday, November 24, 2009 1:43 PM
>>> To: xen-devel@lists.xensource.com
>>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with MSI
>>> enabled
>>>
>>> Hi,
>>>
>>> I am using the xen-3,4, and i am doing pass-through for a device with
>>> MSI enabled (the device of-course supports MSI) to a domU with Windows
>>> XP. When i disable and then re-enable the device using Windows device
>>> manager, everything looks like it's working fine, but the device seems
>>> to malfunction.
>>> After digging a little bit, i saw that the problem is, because the bit
>>> of the interrupt-status (bit 3) in the status register (offset 0x6 in
>>> the PCI config space) was turned on somehow -  this proibably caused
>>> the MSI to malfunction (because the IntX assertion was enabled - thus
>>> disabling the MSI) - and this lead to the problem i experienced.
>>>
>>> Any ideas on how to solve this issue?
>>>
>>> Tom
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: Bug when disabling/enabling a PT device with MSI enabled

[Jiang, Yunhong]

If the msi_translate=0, why IntX bit is turned on is wrong? I assume guest should use intX in such situation.

--jyh
________________________________________
From: Tom Rotenberg [tom.rotenberg@gmail.com]
Sent: Wednesday, November 25, 2009 2:29 AM
To: Jiang, Yunhong
Cc: Kamala Narasimhan; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled

Well, to try and see if the problem is msi related or not, i
registered the device with the option of 'msi_translate=0' to force
the use of intx instead of MSI, and i still got the problem... so i
guess it's not related to the MSI.

It looks like the IntX bit (bit 3 of status word) is runed on somehow
outside qemu, and that's the problem... could that be?

On Wed, Nov 25, 2009 at 11:17 AM, Jiang, Yunhong
<yunhong.jiang@intel.com> wrote:
> Sorry send out the mail too quickly.
> I realized it is MSI translation when I finished first section but forgot remove it, so please ignore  first section.
> You only need care about MSI translation, the code is in hw/pass-through.c.  You can check qemu's changelog at a8d1bd66.
>
> --jyh
> ________________________________________
> From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Jiang, Yunhong [yunhong.jiang@intel.com]
> Sent: Wednesday, November 25, 2009 2:10 AM
> To: Tom Rotenberg; Kamala Narasimhan
> Cc: xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled
>
> This should be about qemu's MSI support. The code is in qemu/hw/pt-msi.c for all MSI emulation.
> Windows XP does not support MSI if I remember correctly. SO this issue should be about MSI translation, i.e. guest is using IOAPIC while host is using MSI. Xen hypervisor will translate the MSI to guest's IOAPIC.
>
> --jyh
> ________________________________________
> From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Tom Rotenberg [tom.rotenberg@gmail.com]
> Sent: Wednesday, November 25, 2009 1:27 AM
> To: Kamala Narasimhan
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled
>
> Well, just tested it with the 2.6.18 latest kernel, and the bug still exists.
> Also, i think that iit's more related to how qemu handles the
> "disable/enable" of domU for PT devices with MSI... but i don't know
> where to start from? it looks like when re-enabling a PT device with
> MSI, we need qemu to re-initialize the device according to it's
> original MSI state (and not to what it was resetted before).
>
> Does anyone has any clue about it?
>
> On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
> <Kamala.Narasimhan@citrix.com> wrote:
>>
>> Couple of questions -
>>
>> 1) What kind of device are you referring to below?  Display by any chance?
>> 2) Which tree is your dom0 kernel based on?
>>
>> There is a chance you are missing some msi patches in your dom0 kernel.  You might want to do a quick search for msi patches in http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and see if you are missing anything obvious.  I recently encountered a similar issue in a slightly different situation, a subset of the problem was caused because of missing msi patches.
>>
>> Kamala
>>
>>> -----Original Message-----
>>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>>> Sent: Tuesday, November 24, 2009 1:43 PM
>>> To: xen-devel@lists.xensource.com
>>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with MSI
>>> enabled
>>>
>>> Hi,
>>>
>>> I am using the xen-3,4, and i am doing pass-through for a device with
>>> MSI enabled (the device of-course supports MSI) to a domU with Windows
>>> XP. When i disable and then re-enable the device using Windows device
>>> manager, everything looks like it's working fine, but the device seems
>>> to malfunction.
>>> After digging a little bit, i saw that the problem is, because the bit
>>> of the interrupt-status (bit 3) in the status register (offset 0x6 in
>>> the PCI config space) was turned on somehow -  this proibably caused
>>> the MSI to malfunction (because the IntX assertion was enabled - thus
>>> disabling the MSI) - and this lead to the problem i experienced.
>>>
>>> Any ideas on how to solve this issue?
>>>
>>> Tom
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Bug when disabling/enabling a PT device with MSI enabled

[Tom Rotenberg]

Well, you are correct... however, the situation is that the bgefore
the disable the intx bit was turned off, and only after disable and
then re-enable it was turned on, and that's the strange part of it...

On Wed, Nov 25, 2009 at 11:38 AM, Jiang, Yunhong
<yunhong.jiang@intel.com> wrote:
> If the msi_translate=0, why IntX bit is turned on is wrong? I assume guest should use intX in such situation.
>
> --jyh
> ________________________________________
> From: Tom Rotenberg [tom.rotenberg@gmail.com]
> Sent: Wednesday, November 25, 2009 2:29 AM
> To: Jiang, Yunhong
> Cc: Kamala Narasimhan; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled
>
> Well, to try and see if the problem is msi related or not, i
> registered the device with the option of 'msi_translate=0' to force
> the use of intx instead of MSI, and i still got the problem... so i
> guess it's not related to the MSI.
>
> It looks like the IntX bit (bit 3 of status word) is runed on somehow
> outside qemu, and that's the problem... could that be?
>
> On Wed, Nov 25, 2009 at 11:17 AM, Jiang, Yunhong
> <yunhong.jiang@intel.com> wrote:
>> Sorry send out the mail too quickly.
>> I realized it is MSI translation when I finished first section but forgot remove it, so please ignore  first section.
>> You only need care about MSI translation, the code is in hw/pass-through.c.  You can check qemu's changelog at a8d1bd66.
>>
>> --jyh
>> ________________________________________
>> From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Jiang, Yunhong [yunhong.jiang@intel.com]
>> Sent: Wednesday, November 25, 2009 2:10 AM
>> To: Tom Rotenberg; Kamala Narasimhan
>> Cc: xen-devel@lists.xensource.com
>> Subject: RE: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled
>>
>> This should be about qemu's MSI support. The code is in qemu/hw/pt-msi.c for all MSI emulation.
>> Windows XP does not support MSI if I remember correctly. SO this issue should be about MSI translation, i.e. guest is using IOAPIC while host is using MSI. Xen hypervisor will translate the MSI to guest's IOAPIC.
>>
>> --jyh
>> ________________________________________
>> From: xen-devel-bounces@lists.xensource.com [xen-devel-bounces@lists.xensource.com] On Behalf Of Tom Rotenberg [tom.rotenberg@gmail.com]
>> Sent: Wednesday, November 25, 2009 1:27 AM
>> To: Kamala Narasimhan
>> Cc: xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device with MSI       enabled
>>
>> Well, just tested it with the 2.6.18 latest kernel, and the bug still exists.
>> Also, i think that iit's more related to how qemu handles the
>> "disable/enable" of domU for PT devices with MSI... but i don't know
>> where to start from? it looks like when re-enabling a PT device with
>> MSI, we need qemu to re-initialize the device according to it's
>> original MSI state (and not to what it was resetted before).
>>
>> Does anyone has any clue about it?
>>
>> On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
>> <Kamala.Narasimhan@citrix.com> wrote:
>>>
>>> Couple of questions -
>>>
>>> 1) What kind of device are you referring to below?  Display by any chance?
>>> 2) Which tree is your dom0 kernel based on?
>>>
>>> There is a chance you are missing some msi patches in your dom0 kernel.  You might want to do a quick search for msi patches in http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and see if you are missing anything obvious.  I recently encountered a similar issue in a slightly different situation, a subset of the problem was caused because of missing msi patches.
>>>
>>> Kamala
>>>
>>>> -----Original Message-----
>>>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>>>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>>>> Sent: Tuesday, November 24, 2009 1:43 PM
>>>> To: xen-devel@lists.xensource.com
>>>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with MSI
>>>> enabled
>>>>
>>>> Hi,
>>>>
>>>> I am using the xen-3,4, and i am doing pass-through for a device with
>>>> MSI enabled (the device of-course supports MSI) to a domU with Windows
>>>> XP. When i disable and then re-enable the device using Windows device
>>>> manager, everything looks like it's working fine, but the device seems
>>>> to malfunction.
>>>> After digging a little bit, i saw that the problem is, because the bit
>>>> of the interrupt-status (bit 3) in the status register (offset 0x6 in
>>>> the PCI config space) was turned on somehow -  this proibably caused
>>>> the MSI to malfunction (because the IntX assertion was enabled - thus
>>>> disabling the MSI) - and this lead to the problem i experienced.
>>>>
>>>> Any ideas on how to solve this issue?
>>>>
>>>> Tom
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>

RE: Bug when disabling/enabling a PT device with MSI enabled

[Jiang, Yunhong]

Tom Rotenberg wrote:
> Well, you are correct... however, the situation is that the bgefore
> the disable the intx bit was turned off, and only after disable and
> then re-enable it was turned on, and that's the strange part of it...

You mean even if the msi_translate == 0? That's strange. If this happens only under msi_translate, it is reasonable and maybe something wrong. So can you give me a confirmation?

--jyh

> 
> On Wed, Nov 25, 2009 at 11:38 AM, Jiang, Yunhong
> <yunhong.jiang@intel.com> wrote:
>> If the msi_translate=0, why IntX bit is turned on is wrong?
> I assume guest should use intX in such situation.
>> 
>> --jyh
>> ________________________________________
>> From: Tom Rotenberg [tom.rotenberg@gmail.com]
>> Sent: Wednesday, November 25, 2009 2:29 AM
>> To: Jiang, Yunhong
>> Cc: Kamala Narasimhan; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device
>> with MSI       enabled 
>> 
>> Well, to try and see if the problem is msi related or not, i
>> registered the device with the option of 'msi_translate=0' to force
>> the use of intx instead of MSI, and i still got the problem... so i
>> guess it's not related to the MSI.
>> 
>> It looks like the IntX bit (bit 3 of status word) is runed on somehow
>> outside qemu, and that's the problem... could that be?
>> 
>> On Wed, Nov 25, 2009 at 11:17 AM, Jiang, Yunhong
>> <yunhong.jiang@intel.com> wrote:
>>> Sorry send out the mail too quickly.
>>> I realized it is MSI translation when I finished first
> section but forgot remove it, so please ignore  first section.
>>> You only need care about MSI translation, the code is in
> hw/pass-through.c.  You can check qemu's changelog at a8d1bd66.
>>> 
>>> --jyh
>>> ________________________________________
>>> From: xen-devel-bounces@lists.xensource.com
> [xen-devel-bounces@lists.xensource.com] On Behalf Of Jiang,
> Yunhong [yunhong.jiang@intel.com]
>>> Sent: Wednesday, November 25, 2009 2:10 AM
>>> To: Tom Rotenberg; Kamala Narasimhan
>>> Cc: xen-devel@lists.xensource.com
>>> Subject: RE: [Xen-devel] Bug when disabling/enabling a PT device
>>> with MSI       enabled 
>>> 
>>> This should be about qemu's MSI support. The code is in
> qemu/hw/pt-msi.c for all MSI emulation.
>>> Windows XP does not support MSI if I remember correctly. SO
> this issue should be about MSI translation, i.e. guest is
> using IOAPIC while host is using MSI. Xen hypervisor will
> translate the MSI to guest's IOAPIC.
>>> 
>>> --jyh
>>> ________________________________________
>>> From: xen-devel-bounces@lists.xensource.com
> [xen-devel-bounces@lists.xensource.com] On Behalf Of Tom
> Rotenberg [tom.rotenberg@gmail.com]
>>> Sent: Wednesday, November 25, 2009 1:27 AM
>>> To: Kamala Narasimhan
>>> Cc: xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device
>>> with MSI       enabled 
>>> 
>>> Well, just tested it with the 2.6.18 latest kernel, and the bug
>>> still exists. Also, i think that iit's more related to how qemu
>>> handles the "disable/enable" of domU for PT devices with MSI... but
>>> i don't know where to start from? it looks like when re-enabling a
>>> PT device with MSI, we need qemu to re-initialize the device
>>> according to it's original MSI state (and not to what it was
>>> resetted before). 
>>> 
>>> Does anyone has any clue about it?
>>> 
>>> On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
>>> <Kamala.Narasimhan@citrix.com> wrote:
>>>> 
>>>> Couple of questions -
>>>> 
>>>> 1) What kind of device are you referring to below? Display by any
>>>> chance? 2) Which tree is your dom0 kernel based on?
>>>> 
>>>> There is a chance you are missing some msi patches in your
> dom0 kernel.  You might want to do a quick search for msi
> patches in
> http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and
> see if you are missing anything obvious.  I recently
> encountered a similar issue in a slightly different situation,
> a subset of the problem was caused because of missing msi patches.
>>>> 
>>>> Kamala
>>>> 
>>>>> -----Original Message-----
>>>>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>>>>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>>>>> Sent: Tuesday, November 24, 2009 1:43 PM
>>>>> To: xen-devel@lists.xensource.com
>>>>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with
>>>>> MSI enabled 
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I am using the xen-3,4, and i am doing pass-through for a device
>>>>> with MSI enabled (the device of-course supports MSI) to a domU
>>>>> with Windows XP. When i disable and then re-enable the device
>>>>> using Windows device manager, everything looks like it's working
>>>>> fine, but the device seems to malfunction. After digging a little
>>>>> bit, i saw that the problem is, because the bit of the
>>>>> interrupt-status (bit 3) in the status register (offset 0x6 in
>>>>> the PCI config space) was turned on somehow -  this proibably
>>>>> caused the MSI to malfunction (because the IntX assertion was
>>>>> enabled - thus disabling the MSI) - and this lead to the problem
>>>>> i experienced.  
>>>>> 
>>>>> Any ideas on how to solve this issue?
>>>>> 
>>>>> Tom
>>>>> 
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xensource.com
>>>>> http://lists.xensource.com/xen-devel
>>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel

Re: Bug when disabling/enabling a PT device with MSI enabled

[Tom Rotenberg]

It happens also when msi_translte = 0 , and i think that it's not
related to the MSI at all... i also checked other devices which use
MSI (like the audio device), and they worked fine.
You can take a look at the new mail i sent, which incorporates more
details about this problem.

On Wed, Nov 25, 2009 at 4:47 PM, Jiang, Yunhong <yunhong.jiang@intel.com> wrote:
> Tom Rotenberg wrote:
>> Well, you are correct... however, the situation is that the bgefore
>> the disable the intx bit was turned off, and only after disable and
>> then re-enable it was turned on, and that's the strange part of it...
>
> You mean even if the msi_translate == 0? That's strange. If this happens only under msi_translate, it is reasonable and maybe something wrong. So can you give me a confirmation?
>
> --jyh
>
>>
>> On Wed, Nov 25, 2009 at 11:38 AM, Jiang, Yunhong
>> <yunhong.jiang@intel.com> wrote:
>>> If the msi_translate=0, why IntX bit is turned on is wrong?
>> I assume guest should use intX in such situation.
>>>
>>> --jyh
>>> ________________________________________
>>> From: Tom Rotenberg [tom.rotenberg@gmail.com]
>>> Sent: Wednesday, November 25, 2009 2:29 AM
>>> To: Jiang, Yunhong
>>> Cc: Kamala Narasimhan; xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device
>>> with MSI       enabled
>>>
>>> Well, to try and see if the problem is msi related or not, i
>>> registered the device with the option of 'msi_translate=0' to force
>>> the use of intx instead of MSI, and i still got the problem... so i
>>> guess it's not related to the MSI.
>>>
>>> It looks like the IntX bit (bit 3 of status word) is runed on somehow
>>> outside qemu, and that's the problem... could that be?
>>>
>>> On Wed, Nov 25, 2009 at 11:17 AM, Jiang, Yunhong
>>> <yunhong.jiang@intel.com> wrote:
>>>> Sorry send out the mail too quickly.
>>>> I realized it is MSI translation when I finished first
>> section but forgot remove it, so please ignore  first section.
>>>> You only need care about MSI translation, the code is in
>> hw/pass-through.c.  You can check qemu's changelog at a8d1bd66.
>>>>
>>>> --jyh
>>>> ________________________________________
>>>> From: xen-devel-bounces@lists.xensource.com
>> [xen-devel-bounces@lists.xensource.com] On Behalf Of Jiang,
>> Yunhong [yunhong.jiang@intel.com]
>>>> Sent: Wednesday, November 25, 2009 2:10 AM
>>>> To: Tom Rotenberg; Kamala Narasimhan
>>>> Cc: xen-devel@lists.xensource.com
>>>> Subject: RE: [Xen-devel] Bug when disabling/enabling a PT device
>>>> with MSI       enabled
>>>>
>>>> This should be about qemu's MSI support. The code is in
>> qemu/hw/pt-msi.c for all MSI emulation.
>>>> Windows XP does not support MSI if I remember correctly. SO
>> this issue should be about MSI translation, i.e. guest is
>> using IOAPIC while host is using MSI. Xen hypervisor will
>> translate the MSI to guest's IOAPIC.
>>>>
>>>> --jyh
>>>> ________________________________________
>>>> From: xen-devel-bounces@lists.xensource.com
>> [xen-devel-bounces@lists.xensource.com] On Behalf Of Tom
>> Rotenberg [tom.rotenberg@gmail.com]
>>>> Sent: Wednesday, November 25, 2009 1:27 AM
>>>> To: Kamala Narasimhan
>>>> Cc: xen-devel@lists.xensource.com
>>>> Subject: Re: [Xen-devel] Bug when disabling/enabling a PT device
>>>> with MSI       enabled
>>>>
>>>> Well, just tested it with the 2.6.18 latest kernel, and the bug
>>>> still exists. Also, i think that iit's more related to how qemu
>>>> handles the "disable/enable" of domU for PT devices with MSI... but
>>>> i don't know where to start from? it looks like when re-enabling a
>>>> PT device with MSI, we need qemu to re-initialize the device
>>>> according to it's original MSI state (and not to what it was
>>>> resetted before).
>>>>
>>>> Does anyone has any clue about it?
>>>>
>>>> On Wed, Nov 25, 2009 at 4:25 AM, Kamala Narasimhan
>>>> <Kamala.Narasimhan@citrix.com> wrote:
>>>>>
>>>>> Couple of questions -
>>>>>
>>>>> 1) What kind of device are you referring to below? Display by any
>>>>> chance? 2) Which tree is your dom0 kernel based on?
>>>>>
>>>>> There is a chance you are missing some msi patches in your
>> dom0 kernel.  You might want to do a quick search for msi
>> patches in
>> http://xenbits.xen.org/staging/linux-2.6.18-xen.hg?rev=msi and
>> see if you are missing anything obvious.  I recently
>> encountered a similar issue in a slightly different situation,
>> a subset of the problem was caused because of missing msi patches.
>>>>>
>>>>> Kamala
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>>>>>> bounces@lists.xensource.com] On Behalf Of Tom Rotenberg
>>>>>> Sent: Tuesday, November 24, 2009 1:43 PM
>>>>>> To: xen-devel@lists.xensource.com
>>>>>> Subject: [Xen-devel] Bug when disabling/enabling a PT device with
>>>>>> MSI enabled
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am using the xen-3,4, and i am doing pass-through for a device
>>>>>> with MSI enabled (the device of-course supports MSI) to a domU
>>>>>> with Windows XP. When i disable and then re-enable the device
>>>>>> using Windows device manager, everything looks like it's working
>>>>>> fine, but the device seems to malfunction. After digging a little
>>>>>> bit, i saw that the problem is, because the bit of the
>>>>>> interrupt-status (bit 3) in the status register (offset 0x6 in
>>>>>> the PCI config space) was turned on somehow -  this proibably
>>>>>> caused the MSI to malfunction (because the IntX assertion was
>>>>>> enabled - thus disabling the MSI) - and this lead to the problem
>>>>>> i experienced.
>>>>>>
>>>>>> Any ideas on how to solve this issue?
>>>>>>
>>>>>> Tom
>>>>>>
>>>>>> _______________________________________________
>>>>>> Xen-devel mailing list
>>>>>> Xen-devel@lists.xensource.com
>>>>>> http://lists.xensource.com/xen-devel
>>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel