[PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Li, Zhen-Hua]

If intel_iommu is enabled, when kdump kernel boots, the old root entry
 should be cleared, otherwise it may cause DMAR error.

To make it works for more enviroments, this patch does not use 
is_kdump_kernel() to check it,  but reads the register to check whether 
hardware is using old root entry.

Signed-off-by: Li, Zhen-Hua <zhen-hual@hp.com>
---
 drivers/iommu/dmar.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/drivers/iommu/dmar.c b/drivers/iommu/dmar.c
index 60ab474..7b4fa90 100644
--- a/drivers/iommu/dmar.c
+++ b/drivers/iommu/dmar.c
@@ -66,6 +66,8 @@ static int dmar_dev_scope_status = 1;
 static int alloc_iommu(struct dmar_drhd_unit *drhd);
 static void free_iommu(struct intel_iommu *iommu);
 
+static int iommu_check_root_entry(struct intel_iommu *iommu);
+
 static void __init dmar_register_drhd_unit(struct dmar_drhd_unit *drhd)
 {
 	/*
@@ -987,6 +989,8 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)
 						       intel_iommu_groups,
 						       iommu->name);
 
+	iommu_check_root_entry(iommu);
+
 	return 0;
 
  err_unmap:
@@ -1666,5 +1670,32 @@ static int __init dmar_free_unused_resources(void)
 	return 0;
 }
 
+static int iommu_check_root_entry(struct intel_iommu *iommu)
+{
+	u64 re_pa;
+
+	re_pa = dmar_readq(iommu->reg + DMAR_RTADDR_REG);
+
+	/* This only works for hardware error and kdump kernel */
+	if (unlikely(re_pa != 0)) {
+		u32 sts;
+		unsigned long flag;
+
+		raw_spin_lock_irqsave(&iommu->register_lock, flag);
+		dmar_writeq(iommu->reg + DMAR_RTADDR_REG, 0);
+
+		writel(iommu->gcmd | DMA_GCMD_SRTP, iommu->reg + DMAR_GCMD_REG);
+
+		/* Make sure hardware complete it */
+		IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
+			      readl, (sts & DMA_GSTS_RTPS), sts);
+
+		raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
+
+	}
+
+	return 0;
+}
+
 late_initcall(dmar_free_unused_resources);
 IOMMU_INIT_POST(detect_intel_iommu);
-- 
2.0.0-rc0

RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Li, Zhen-Hua]

There is a bug when Linux running on an HP large system: 
	when kdump kernel runs, the hardware is still using the old 
root entry. This causes error message when iommu not finished initialization.

-----Original Message-----
From: Li, Zhen-Hua 
Sent: Monday, August 18, 2014 4:59 PM
To: David Woodhouse; Joerg Roedel; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
Cc: Li, Zhen-Hua
Subject: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

If intel_iommu is enabled, when kdump kernel boots, the old root entry
 should be cleared, otherwise it may cause DMAR error.

To make it works for more enviroments, this patch does not use 
is_kdump_kernel() to check it,  but reads the register to check whether 
hardware is using old root entry.

Signed-off-by: Li, Zhen-Hua <zhen-hual@hp.com>
---
 drivers/iommu/dmar.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/drivers/iommu/dmar.c b/drivers/iommu/dmar.c
index 60ab474..7b4fa90 100644
--- a/drivers/iommu/dmar.c
+++ b/drivers/iommu/dmar.c
@@ -66,6 +66,8 @@ static int dmar_dev_scope_status = 1;
 static int alloc_iommu(struct dmar_drhd_unit *drhd);
 static void free_iommu(struct intel_iommu *iommu);
 
+static int iommu_check_root_entry(struct intel_iommu *iommu);
+
 static void __init dmar_register_drhd_unit(struct dmar_drhd_unit *drhd)
 {
 	/*
@@ -987,6 +989,8 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)
 						       intel_iommu_groups,
 						       iommu->name);
 
+	iommu_check_root_entry(iommu);
+
 	return 0;
 
  err_unmap:
@@ -1666,5 +1670,32 @@ static int __init dmar_free_unused_resources(void)
 	return 0;
 }
 
+static int iommu_check_root_entry(struct intel_iommu *iommu)
+{
+	u64 re_pa;
+
+	re_pa = dmar_readq(iommu->reg + DMAR_RTADDR_REG);
+
+	/* This only works for hardware error and kdump kernel */
+	if (unlikely(re_pa != 0)) {
+		u32 sts;
+		unsigned long flag;
+
+		raw_spin_lock_irqsave(&iommu->register_lock, flag);
+		dmar_writeq(iommu->reg + DMAR_RTADDR_REG, 0);
+
+		writel(iommu->gcmd | DMA_GCMD_SRTP, iommu->reg + DMAR_GCMD_REG);
+
+		/* Make sure hardware complete it */
+		IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
+			      readl, (sts & DMA_GSTS_RTPS), sts);
+
+		raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
+
+	}
+
+	return 0;
+}
+
 late_initcall(dmar_free_unused_resources);
 IOMMU_INIT_POST(detect_intel_iommu);
-- 
2.0.0-rc0

Re: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Joerg Roedel]

On Mon, Aug 18, 2014 at 11:01:56PM +0000, Li, Zhen-Hua wrote:
> There is a bug when Linux running on an HP large system: 
> 	when kdump kernel runs, the hardware is still using the old 
> root entry. This causes error message when iommu not finished initialization.

What error message are you seeing? When the kdump kernel boots the iommu
should be still enabled from the old kernel with the old root-entry. So
any in-flight DMA initiated from the old kernel can still pass and there
should be no error messages.

When you clear the root-entry that in-flight DMA might go to another
random location in system memory or just fail, no?


	Joerg

RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Li, Zhen-Hua]

: [fault reason 01] Present bit in root entry is clear
It appears when iommu initializing in the kdump kernel.

-----Original Message-----
From: Joerg Roedel [mailto:joro@8bytes.org] 
Sent: Tuesday, August 19, 2014 7:23 AM
To: Li, Zhen-Hua
Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
Subject: Re: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

On Mon, Aug 18, 2014 at 11:01:56PM +0000, Li, Zhen-Hua wrote:
> There is a bug when Linux running on an HP large system: 
> 	when kdump kernel runs, the hardware is still using the old 
> root entry. This causes error message when iommu not finished initialization.

What error message are you seeing? When the kdump kernel boots the iommu
should be still enabled from the old kernel with the old root-entry. So
any in-flight DMA initiated from the old kernel can still pass and there
should be no error messages.

When you clear the root-entry that in-flight DMA might go to another
random location in system memory or just fail, no?


	Joerg

RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Li, Zhen-Hua]

When the dump kernel boots, it will initialize iommu again, and the root entry will be allocted 
in another memory region.

That means, no matter kernel clears the old root entry table or not,  the dump kernel will use 
another memory region when iommu initializing.

-----Original Message-----
From: Li, Zhen-Hua 
Sent: Tuesday, August 19, 2014 7:27 AM
To: 'Joerg Roedel'
Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
Subject: RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

: [fault reason 01] Present bit in root entry is clear
It appears when iommu initializing in the kdump kernel.

-----Original Message-----
From: Joerg Roedel [mailto:joro@8bytes.org] 
Sent: Tuesday, August 19, 2014 7:23 AM
To: Li, Zhen-Hua
Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
Subject: Re: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

On Mon, Aug 18, 2014 at 11:01:56PM +0000, Li, Zhen-Hua wrote:
> There is a bug when Linux running on an HP large system: 
> 	when kdump kernel runs, the hardware is still using the old 
> root entry. This causes error message when iommu not finished initialization.

What error message are you seeing? When the kdump kernel boots the iommu
should be still enabled from the old kernel with the old root-entry. So
any in-flight DMA initiated from the old kernel can still pass and there
should be no error messages.

When you clear the root-entry that in-flight DMA might go to another
random location in system memory or just fail, no?


	Joerg

RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Li, Zhen-Hua]

My debugging result is this: 

1. Clear the old root entry table, dump kernel will choose another
 memory region for root entry.
2. Do NOT clear the old root entry, when dump kernel initializing 
the iommu data structure, it  will allocate memory for root entry, 
this is different from the old address. 

If not clear old entry , the error message appears before dump kernel
finishes the iommu init works, and also appears in other places(before 
device inits). 

If I clear the old root entry, the error message disappears before iommu
init work finish, but still appears in other places. 

-----Original Message-----
From: Li, Zhen-Hua 
Sent: Tuesday, August 19, 2014 7:48 AM
To: Li, Zhen-Hua; Joerg Roedel
Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
Subject: RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

When the dump kernel boots, it will initialize iommu again, and the root entry will be allocted 
in another memory region.

That means, no matter kernel clears the old root entry table or not,  the dump kernel will use 
another memory region when iommu initializing.

-----Original Message-----
From: Li, Zhen-Hua 
Sent: Tuesday, August 19, 2014 7:27 AM
To: 'Joerg Roedel'
Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
Subject: RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

: [fault reason 01] Present bit in root entry is clear
It appears when iommu initializing in the kdump kernel.

-----Original Message-----
From: Joerg Roedel [mailto:joro@8bytes.org] 
Sent: Tuesday, August 19, 2014 7:23 AM
To: Li, Zhen-Hua
Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
Subject: Re: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

On Mon, Aug 18, 2014 at 11:01:56PM +0000, Li, Zhen-Hua wrote:
> There is a bug when Linux running on an HP large system: 
> 	when kdump kernel runs, the hardware is still using the old 
> root entry. This causes error message when iommu not finished initialization.

What error message are you seeing? When the kdump kernel boots the iommu
should be still enabled from the old kernel with the old root-entry. So
any in-flight DMA initiated from the old kernel can still pass and there
should be no error messages.

When you clear the root-entry that in-flight DMA might go to another
random location in system memory or just fail, no?


	Joerg

Re: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Li, ZhenHua]

I found there are more data need to be cleared for the dump kernel.
So please ignore this patch, I will send out another one.

Thanks
Zhenhua

On 08/19/2014 07:59 AM, Li, Zhen-Hua wrote:
> My debugging result is this:
>
> 1. Clear the old root entry table, dump kernel will choose another
>   memory region for root entry.
> 2. Do NOT clear the old root entry, when dump kernel initializing
> the iommu data structure, it  will allocate memory for root entry,
> this is different from the old address.
>
> If not clear old entry , the error message appears before dump kernel
> finishes the iommu init works, and also appears in other places(before
> device inits).
>
> If I clear the old root entry, the error message disappears before iommu
> init work finish, but still appears in other places.
>
> -----Original Message-----
> From: Li, Zhen-Hua
> Sent: Tuesday, August 19, 2014 7:48 AM
> To: Li, Zhen-Hua; Joerg Roedel
> Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
> Subject: RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel
>
> When the dump kernel boots, it will initialize iommu again, and the root entry will be allocted
> in another memory region.
>
> That means, no matter kernel clears the old root entry table or not,  the dump kernel will use
> another memory region when iommu initializing.
>
> -----Original Message-----
> From: Li, Zhen-Hua
> Sent: Tuesday, August 19, 2014 7:27 AM
> To: 'Joerg Roedel'
> Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
> Subject: RE: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel
>
> : [fault reason 01] Present bit in root entry is clear
> It appears when iommu initializing in the kdump kernel.
>
> -----Original Message-----
> From: Joerg Roedel [mailto:joro@8bytes.org]
> Sent: Tuesday, August 19, 2014 7:23 AM
> To: Li, Zhen-Hua
> Cc: David Woodhouse; iommu@lists.linux-foundation.org; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel
>
> On Mon, Aug 18, 2014 at 11:01:56PM +0000, Li, Zhen-Hua wrote:
>> There is a bug when Linux running on an HP large system:
>> 	when kdump kernel runs, the hardware is still using the old
>> root entry. This causes error message when iommu not finished initialization.
> What error message are you seeing? When the kdump kernel boots the iommu
> should be still enabled from the old kernel with the old root-entry. So
> any in-flight DMA initiated from the old kernel can still pass and there
> should be no error messages.
>
> When you clear the root-entry that in-flight DMA might go to another
> random location in system memory or just fail, no?
>
>
> 	Joerg
>

Re: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Joerg Roedel]

On Mon, Aug 18, 2014 at 11:27:01PM +0000, Li, Zhen-Hua wrote:
> : [fault reason 01] Present bit in root entry is clear
> It appears when iommu initializing in the kdump kernel.

Hmm, do you have an explanation how this can happen? From how I read the
code, the kdump kernel disables translation on the IOMMU, then sets a
new root entry, and then re-enabled translation. To me it looks like
there is no point in time where translation is enabled and the
root-entry is clear (present-bit==0).

But obviously I am missing something if you see the message above.

Btw, have you looked into this patch-set posted earlier this year:

	https://lkml.org/lkml/2014/4/24/836

It approaches the same problem-space, but also cares about in-flight
DMA.


	Joerg

Re: [PATCH 1/1] iommu/vt-d : clear old root entry for dump kernel

[Li, ZhenHua]

On my system, error message:
[    4.322008] dmar: DRHD: handling fault status reg 2
[    4.327484] dmar: DMAR:[DMA Read] Request device [21:00.0] fault addr 
fff66000
[    4.327484] DMAR:[fault reason 01] Present bit in root entry is clear

fault happens on device : 21:00.0

To describe this problem clearly, we mark two time points:
A: Just before dump kernel init the DMAR/IOMMU structures.
B: Just after DMAR/IOMMU finished initialization.
C: Just before  21:00.0

When the first kernel crashed, then the dump kernel will boot.
Then there comes a DMA request on  21:00.0 when durting A to C. And this 
will cause this error.

If I clear the root entry table when iommu structure is allocted, this 
error will not happen during A to B, but still can be seen during B and 
C.  I guess the cause is the un-expected DMA request.

I will apply Bill Sumner's patch to see whether it will cause the errors 
disappear.


-- Zhenhua

On 08/19/2014 07:02 PM, Joerg Roedel wrote:
> On Mon, Aug 18, 2014 at 11:27:01PM +0000, Li, Zhen-Hua wrote:
>> : [fault reason 01] Present bit in root entry is clear
>> It appears when iommu initializing in the kdump kernel.
>
> Hmm, do you have an explanation how this can happen? From how I read the
> code, the kdump kernel disables translation on the IOMMU, then sets a
> new root entry, and then re-enabled translation. To me it looks like
> there is no point in time where translation is enabled and the
> root-entry is clear (present-bit==0).
>
> But obviously I am missing something if you see the message above.
>
> Btw, have you looked into this patch-set posted earlier this year:
>
> 	https://lkml.org/lkml/2014/4/24/836
>
> It approaches the same problem-space, but also cares about in-flight
> DMA.
>
>
> 	Joerg
>