[RFC] iommu/vt-d: Use the SIRTP when enabling remapping

[RFC] iommu/vt-d: Use the SIRTP when enabling remapping

[Nathan Zimmer]

The previous change (iommu/vt-d: Don't store SIRTP request) to this area
caused a crash in our simulator. In particular is seems that by the time a
UART interrupt is sent through the system, we don't see interrupt remapping
to be enabled.  So the interrupt does not get translated to a logical
interrupt and crashes.

OR'ing the SIRTP request to make sure it is seen but hopefully not sticky.
This seems like a clean fix, at least on our simulator; if you don't agree,
our simulator guy will take a closer look at our iommu model.

Found testing on our simulator, not real hardware.

Signed-off-by: Nathan Zimmer <nzimmer@sgi.com>
Signed-off-by: Gary Kroening <gfk@sgi.com>
Cc: Jan Kiszka <jan.kiszka@siemens.com>
Cc: Joerg Roedel <jroedel@suse.de>

---
 drivers/iommu/intel_irq_remapping.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/iommu/intel_irq_remapping.c b/drivers/iommu/intel_irq_remapping.c
index a872874..f586e41 100644
--- a/drivers/iommu/intel_irq_remapping.c
+++ b/drivers/iommu/intel_irq_remapping.c
@@ -455,7 +455,7 @@ static void iommu_set_irq_remapping(struct intel_iommu *iommu, int mode)
 	/* Enable interrupt-remapping */
 	iommu->gcmd |= DMA_GCMD_IRE;
 	iommu->gcmd &= ~DMA_GCMD_CFI;  /* Block compatibility-format MSIs */
-	writel(iommu->gcmd, iommu->reg + DMAR_GCMD_REG);
+	writel(iommu->gcmd | DMA_GCMD_SIRTP, iommu->reg + DMAR_GCMD_REG);
 
 	IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
 		      readl, (sts & DMA_GSTS_IRES), sts);
-- 
1.8.2.1

Re: [RFC] iommu/vt-d: Use the SIRTP when enabling remapping

[Jan Kiszka]

On 2014-09-09 23:11, Nathan Zimmer wrote:
> The previous change (iommu/vt-d: Don't store SIRTP request) to this area
> caused a crash in our simulator. In particular is seems that by the time a
> UART interrupt is sent through the system, we don't see interrupt remapping
> to be enabled.  So the interrupt does not get translated to a logical
> interrupt and crashes.
> 
> OR'ing the SIRTP request to make sure it is seen but hopefully not sticky.
> This seems like a clean fix, at least on our simulator; if you don't agree,
> our simulator guy will take a closer look at our iommu model.

Check the VT-d spec (6.7, Set Interrupt Remapping Table Pointer
Operation): "Software must always follow the interrupt-remapping table
pointer set operation with a global invalidate of the IEC to ensure
hardware references the new structures *before* enabling interrupt
remapping." There is also (command register description): "If multiple
control fields in this register need to be modified, software must
serialize the modifications through multiple writes to this register."
This, in combination with the command registers description of bits 24
and 25 strongly suggests that your model is broken.

> 
> Found testing on our simulator, not real hardware.

Funnily, the original issue was found by the QEMU model of VT-d that
used to take the last cited sentence very strictly (I asked to remove it
due to the preexisting Linux versions).

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] iommu/vt-d: Use the SIRTP when enabling remapping

[Jan Kiszka]

On 2014-09-09 23:11, Nathan Zimmer wrote:
> The previous change (iommu/vt-d: Don't store SIRTP request) to this area
> caused a crash in our simulator. In particular is seems that by the time a
> UART interrupt is sent through the system, we don't see interrupt remapping
> to be enabled.  So the interrupt does not get translated to a logical
> interrupt and crashes.
> 
> OR'ing the SIRTP request to make sure it is seen but hopefully not sticky.
> This seems like a clean fix, at least on our simulator; if you don't agree,
> our simulator guy will take a closer look at our iommu model.

Check the VT-d spec (6.7, Set Interrupt Remapping Table Pointer
Operation): "Software must always follow the interrupt-remapping table
pointer set operation with a global invalidate of the IEC to ensure
hardware references the new structures *before* enabling interrupt
remapping." There is also (command register description): "If multiple
control fields in this register need to be modified, software must
serialize the modifications through multiple writes to this register."
This, in combination with the command registers description of bits 24
and 25 strongly suggests that your model is broken.

> 
> Found testing on our simulator, not real hardware.

Funnily, the original issue was found by the QEMU model of VT-d that
used to take the last cited sentence very strictly (I asked to remove it
due to the preexisting Linux versions).

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC] iommu/vt-d: Use the SIRTP when enabling remapping

[Gary Kroening]

On 9/10/2014 1:56 AM, Jan Kiszka wrote:
> On 2014-09-09 23:11, Nathan Zimmer wrote:
>> The previous change (iommu/vt-d: Don't store SIRTP request) to this area
>> caused a crash in our simulator. In particular is seems that by the time a
>> UART interrupt is sent through the system, we don't see interrupt remapping
>> to be enabled.  So the interrupt does not get translated to a logical
>> interrupt and crashes.
>>
>> OR'ing the SIRTP request to make sure it is seen but hopefully not sticky.
>> This seems like a clean fix, at least on our simulator; if you don't agree,
>> our simulator guy will take a closer look at our iommu model.
>
> Check the VT-d spec (6.7, Set Interrupt Remapping Table Pointer
> Operation): "Software must always follow the interrupt-remapping table
> pointer set operation with a global invalidate of the IEC to ensure
> hardware references the new structures *before* enabling interrupt
> remapping." There is also (command register description): "If multiple
> control fields in this register need to be modified, software must
> serialize the modifications through multiple writes to this register."
> This, in combination with the command registers description of bits 24
> and 25 strongly suggests that your model is broken.

Thanks, Jan. I'll read up on it. Still working out some kinks with
the relatively new IOMMU/interrupt-remapping logic which we added over
the last 18 months or so. Sorry for any distraction, and thanks for the
help/info.
Gary

>
>>
>> Found testing on our simulator, not real hardware.
>
> Funnily, the original issue was found by the QEMU model of VT-d that
> used to take the last cited sentence very strictly (I asked to remove it
> due to the preexisting Linux versions).
>
> Jan
>

Re: [RFC] iommu/vt-d: Use the SIRTP when enabling remapping

[Gary Kroening]

I'm not really a kernel developer and don't know the proper etiquette for
this, but just wanted to say thanks to Jan for the guidance. With your help,
I was able to track our problem down to a much more fundamental problem of
incorrect bit position of the fields in our VT-d GLBCMD register, attributable
to our now-manual process of generating register information from Intel
socket documentation. (We used to used Intel's SoftSDV to determine register
fields, but that's no longer available and we have to use fallible human
eyes! Thousands of registers in each new socket...)

Gary

On 09/10/2014 02:58 AM, Gary Kroening wrote:
> On 9/10/2014 1:56 AM, Jan Kiszka wrote:
>> On 2014-09-09 23:11, Nathan Zimmer wrote:
>>> The previous change (iommu/vt-d: Don't store SIRTP request) to this area
>>> caused a crash in our simulator. In particular is seems that by the time a
>>> UART interrupt is sent through the system, we don't see interrupt remapping
>>> to be enabled.  So the interrupt does not get translated to a logical
>>> interrupt and crashes.
>>>
>>> OR'ing the SIRTP request to make sure it is seen but hopefully not sticky.
>>> This seems like a clean fix, at least on our simulator; if you don't agree,
>>> our simulator guy will take a closer look at our iommu model.
>>
>> Check the VT-d spec (6.7, Set Interrupt Remapping Table Pointer
>> Operation): "Software must always follow the interrupt-remapping table
>> pointer set operation with a global invalidate of the IEC to ensure
>> hardware references the new structures *before* enabling interrupt
>> remapping." There is also (command register description): "If multiple
>> control fields in this register need to be modified, software must
>> serialize the modifications through multiple writes to this register."
>> This, in combination with the command registers description of bits 24
>> and 25 strongly suggests that your model is broken.
>
> Thanks, Jan. I'll read up on it. Still working out some kinks with
> the relatively new IOMMU/interrupt-remapping logic which we added over
> the last 18 months or so. Sorry for any distraction, and thanks for the
> help/info.
> Gary
>
>>
>>>
>>> Found testing on our simulator, not real hardware.
>>
>> Funnily, the original issue was found by the QEMU model of VT-d that
>> used to take the last cited sentence very strictly (I asked to remove it
>> due to the preexisting Linux versions).
>>
>> Jan
>>

Re: [RFC] iommu/vt-d: Use the SIRTP when enabling remapping

[Gary Kroening]

I'm not really a kernel developer and don't know the proper etiquette for
this, but just wanted to say thanks to Jan for the guidance. With your help,
I was able to track our problem down to a much more fundamental problem of
incorrect bit position of the fields in our VT-d GLBCMD register, attributable
to our now-manual process of generating register information from Intel
socket documentation. (We used to used Intel's SoftSDV to determine register
fields, but that's no longer available and we have to use fallible human
eyes! Thousands of registers in each new socket...)

Gary

On 09/10/2014 02:58 AM, Gary Kroening wrote:
> On 9/10/2014 1:56 AM, Jan Kiszka wrote:
>> On 2014-09-09 23:11, Nathan Zimmer wrote:
>>> The previous change (iommu/vt-d: Don't store SIRTP request) to this area
>>> caused a crash in our simulator. In particular is seems that by the time a
>>> UART interrupt is sent through the system, we don't see interrupt remapping
>>> to be enabled.  So the interrupt does not get translated to a logical
>>> interrupt and crashes.
>>>
>>> OR'ing the SIRTP request to make sure it is seen but hopefully not sticky.
>>> This seems like a clean fix, at least on our simulator; if you don't agree,
>>> our simulator guy will take a closer look at our iommu model.
>>
>> Check the VT-d spec (6.7, Set Interrupt Remapping Table Pointer
>> Operation): "Software must always follow the interrupt-remapping table
>> pointer set operation with a global invalidate of the IEC to ensure
>> hardware references the new structures *before* enabling interrupt
>> remapping." There is also (command register description): "If multiple
>> control fields in this register need to be modified, software must
>> serialize the modifications through multiple writes to this register."
>> This, in combination with the command registers description of bits 24
>> and 25 strongly suggests that your model is broken.
>
> Thanks, Jan. I'll read up on it. Still working out some kinks with
> the relatively new IOMMU/interrupt-remapping logic which we added over
> the last 18 months or so. Sorry for any distraction, and thanks for the
> help/info.
> Gary
>
>>
>>>
>>> Found testing on our simulator, not real hardware.
>>
>> Funnily, the original issue was found by the QEMU model of VT-d that
>> used to take the last cited sentence very strictly (I asked to remove it
>> due to the preexisting Linux versions).
>>
>> Jan
>>