[PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

Hi,

Some IOMMUv2 capable devices do not work correctly when SME is
active, because their DMA mask does not include the encryption bit, so
that they can not DMA to encrypted memory directly.

The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2
capable devices into an identity mapped domain. Fix that by not
forcing an identity mapped domain on devices when SME is active and
forbid using their IOMMUv2 functionality.

Please review.

Thanks,

	Joerg

Joerg Roedel (2):
  iommu/amd: Do not force direct mapping when SME is active
  iommu/amd: Do not use IOMMUv2 functionality when SME is active

 drivers/iommu/amd/iommu.c    | 7 ++++++-
 drivers/iommu/amd/iommu_v2.c | 7 +++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

-- 
2.28.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

Hi,

Some IOMMUv2 capable devices do not work correctly when SME is
active, because their DMA mask does not include the encryption bit, so
that they can not DMA to encrypted memory directly.

The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2
capable devices into an identity mapped domain. Fix that by not
forcing an identity mapped domain on devices when SME is active and
forbid using their IOMMUv2 functionality.

Please review.

Thanks,

	Joerg

Joerg Roedel (2):
  iommu/amd: Do not force direct mapping when SME is active
  iommu/amd: Do not use IOMMUv2 functionality when SME is active

 drivers/iommu/amd/iommu.c    | 7 ++++++-
 drivers/iommu/amd/iommu_v2.c | 7 +++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

-- 
2.28.0

[PATCH 1/2] iommu/amd: Do not force direct mapping when SME is active

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

Do not force devices supporting IOMMUv2 to be direct mapped when memory
encryption is active. This might cause them to be unusable because their
DMA mask does not include the encryption bit.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/amd/iommu.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index ba9f3dbc5b94..77e4268e41cf 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2659,7 +2659,12 @@ static int amd_iommu_def_domain_type(struct device *dev)
 	if (!dev_data)
 		return 0;
 
-	if (dev_data->iommu_v2)
+	/*
+	 * Do not identity map IOMMUv2 capable devices when memory encryption is
+	 * active, because some of those devices (AMD GPUs) don't have the
+	 * encryption bit in their DMA-mask and require remapping.
+	 */
+	if (!mem_encrypt_active() && dev_data->iommu_v2)
 		return IOMMU_DOMAIN_IDENTITY;
 
 	return 0;
-- 
2.28.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH 1/2] iommu/amd: Do not force direct mapping when SME is active

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

Do not force devices supporting IOMMUv2 to be direct mapped when memory
encryption is active. This might cause them to be unusable because their
DMA mask does not include the encryption bit.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/amd/iommu.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index ba9f3dbc5b94..77e4268e41cf 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -2659,7 +2659,12 @@ static int amd_iommu_def_domain_type(struct device *dev)
 	if (!dev_data)
 		return 0;
 
-	if (dev_data->iommu_v2)
+	/*
+	 * Do not identity map IOMMUv2 capable devices when memory encryption is
+	 * active, because some of those devices (AMD GPUs) don't have the
+	 * encryption bit in their DMA-mask and require remapping.
+	 */
+	if (!mem_encrypt_active() && dev_data->iommu_v2)
 		return IOMMU_DOMAIN_IDENTITY;
 
 	return 0;
-- 
2.28.0

RE: [PATCH 1/2] iommu/amd: Do not force direct mapping when SME is active

[Deucher, Alexander]

[AMD Public Use]

+ Felix, Christian

> -----Original Message-----
> From: Joerg Roedel <joro@8bytes.org>
> Sent: Monday, August 24, 2020 6:54 AM
> To: iommu@lists.linux-foundation.org
> Cc: Joerg Roedel <joro@8bytes.org>; jroedel@suse.de; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Deucher, Alexander
> <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org
> Subject: [PATCH 1/2] iommu/amd: Do not force direct mapping when SME is
> active
> 
> From: Joerg Roedel <jroedel@suse.de>
> 
> Do not force devices supporting IOMMUv2 to be direct mapped when
> memory encryption is active. This might cause them to be unusable because
> their DMA mask does not include the encryption bit.
> 
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
>  drivers/iommu/amd/iommu.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
> index ba9f3dbc5b94..77e4268e41cf 100644
> --- a/drivers/iommu/amd/iommu.c
> +++ b/drivers/iommu/amd/iommu.c
> @@ -2659,7 +2659,12 @@ static int amd_iommu_def_domain_type(struct
> device *dev)
>  	if (!dev_data)
>  		return 0;
> 
> -	if (dev_data->iommu_v2)
> +	/*
> +	 * Do not identity map IOMMUv2 capable devices when memory
> encryption is
> +	 * active, because some of those devices (AMD GPUs) don't have the
> +	 * encryption bit in their DMA-mask and require remapping.
> +	 */

I think on the integrated GPUs in APUs I'd prefer to have the identity mapping over SME, but I guess this is fine because you have to explicitly enable SME and if you do that you know what you are getting into.

Alex

> +	if (!mem_encrypt_active() && dev_data->iommu_v2)
>  		return IOMMU_DOMAIN_IDENTITY;
> 
>  	return 0;
> --
> 2.28.0
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 1/2] iommu/amd: Do not force direct mapping when SME is active

[Deucher, Alexander]

[AMD Public Use]

+ Felix, Christian

> -----Original Message-----
> From: Joerg Roedel <joro@8bytes.org>
> Sent: Monday, August 24, 2020 6:54 AM
> To: iommu@lists.linux-foundation.org
> Cc: Joerg Roedel <joro@8bytes.org>; jroedel@suse.de; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Deucher, Alexander
> <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org
> Subject: [PATCH 1/2] iommu/amd: Do not force direct mapping when SME is
> active
> 
> From: Joerg Roedel <jroedel@suse.de>
> 
> Do not force devices supporting IOMMUv2 to be direct mapped when
> memory encryption is active. This might cause them to be unusable because
> their DMA mask does not include the encryption bit.
> 
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
>  drivers/iommu/amd/iommu.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
> index ba9f3dbc5b94..77e4268e41cf 100644
> --- a/drivers/iommu/amd/iommu.c
> +++ b/drivers/iommu/amd/iommu.c
> @@ -2659,7 +2659,12 @@ static int amd_iommu_def_domain_type(struct
> device *dev)
>  	if (!dev_data)
>  		return 0;
> 
> -	if (dev_data->iommu_v2)
> +	/*
> +	 * Do not identity map IOMMUv2 capable devices when memory
> encryption is
> +	 * active, because some of those devices (AMD GPUs) don't have the
> +	 * encryption bit in their DMA-mask and require remapping.
> +	 */

I think on the integrated GPUs in APUs I'd prefer to have the identity mapping over SME, but I guess this is fine because you have to explicitly enable SME and if you do that you know what you are getting into.

Alex

> +	if (!mem_encrypt_active() && dev_data->iommu_v2)
>  		return IOMMU_DOMAIN_IDENTITY;
> 
>  	return 0;
> --
> 2.28.0

[PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when SME is active

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

When memory encryption is active the device is likely not in a direct
mapped domain. Forbid using IOMMUv2 functionality for now until finer
grained checks for this have been implemented.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/amd/iommu_v2.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c
index c259108ab6dd..0d175aed1d92 100644
--- a/drivers/iommu/amd/iommu_v2.c
+++ b/drivers/iommu/amd/iommu_v2.c
@@ -737,6 +737,13 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
 
 	might_sleep();
 
+	/*
+	 * When memory encryption is active the device is likely not in a
+	 * direct-mapped domain. Forbid using IOMMUv2 functionality for now.
+	 */
+	if (mem_encrypt_active())
+		return -ENODEV;
+
 	if (!amd_iommu_v2_supported())
 		return -ENODEV;
 
-- 
2.28.0

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

[PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when SME is active

[Joerg Roedel]

From: Joerg Roedel <jroedel@suse.de>

When memory encryption is active the device is likely not in a direct
mapped domain. Forbid using IOMMUv2 functionality for now until finer
grained checks for this have been implemented.

Signed-off-by: Joerg Roedel <jroedel@suse.de>
---
 drivers/iommu/amd/iommu_v2.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c
index c259108ab6dd..0d175aed1d92 100644
--- a/drivers/iommu/amd/iommu_v2.c
+++ b/drivers/iommu/amd/iommu_v2.c
@@ -737,6 +737,13 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids)
 
 	might_sleep();
 
+	/*
+	 * When memory encryption is active the device is likely not in a
+	 * direct-mapped domain. Forbid using IOMMUv2 functionality for now.
+	 */
+	if (mem_encrypt_active())
+		return -ENODEV;
+
 	if (!amd_iommu_v2_supported())
 		return -ENODEV;
 
-- 
2.28.0

RE: [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when SME is active

[Deucher, Alexander]

[AMD Public Use]

+ Felix, Christian

> -----Original Message-----
> From: Joerg Roedel <joro@8bytes.org>
> Sent: Monday, August 24, 2020 6:54 AM
> To: iommu@lists.linux-foundation.org
> Cc: Joerg Roedel <joro@8bytes.org>; jroedel@suse.de; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Deucher, Alexander
> <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org
> Subject: [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when
> SME is active
> 
> From: Joerg Roedel <jroedel@suse.de>
> 
> When memory encryption is active the device is likely not in a direct mapped
> domain. Forbid using IOMMUv2 functionality for now until finer grained
> checks for this have been implemented.
> 
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
>  drivers/iommu/amd/iommu_v2.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/iommu/amd/iommu_v2.c
> b/drivers/iommu/amd/iommu_v2.c index c259108ab6dd..0d175aed1d92
> 100644
> --- a/drivers/iommu/amd/iommu_v2.c
> +++ b/drivers/iommu/amd/iommu_v2.c
> @@ -737,6 +737,13 @@ int amd_iommu_init_device(struct pci_dev *pdev,
> int pasids)
> 
>  	might_sleep();
> 
> +	/*
> +	 * When memory encryption is active the device is likely not in a
> +	 * direct-mapped domain. Forbid using IOMMUv2 functionality for
> now.
> +	 */
> +	if (mem_encrypt_active())
> +		return -ENODEV;
> +
>  	if (!amd_iommu_v2_supported())
>  		return -ENODEV;
> 
> --
> 2.28.0
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when SME is active

[Deucher, Alexander]

[AMD Public Use]

+ Felix, Christian

> -----Original Message-----
> From: Joerg Roedel <joro@8bytes.org>
> Sent: Monday, August 24, 2020 6:54 AM
> To: iommu@lists.linux-foundation.org
> Cc: Joerg Roedel <joro@8bytes.org>; jroedel@suse.de; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; Deucher, Alexander
> <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org
> Subject: [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when
> SME is active
> 
> From: Joerg Roedel <jroedel@suse.de>
> 
> When memory encryption is active the device is likely not in a direct mapped
> domain. Forbid using IOMMUv2 functionality for now until finer grained
> checks for this have been implemented.
> 
> Signed-off-by: Joerg Roedel <jroedel@suse.de>
> ---
>  drivers/iommu/amd/iommu_v2.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/drivers/iommu/amd/iommu_v2.c
> b/drivers/iommu/amd/iommu_v2.c index c259108ab6dd..0d175aed1d92
> 100644
> --- a/drivers/iommu/amd/iommu_v2.c
> +++ b/drivers/iommu/amd/iommu_v2.c
> @@ -737,6 +737,13 @@ int amd_iommu_init_device(struct pci_dev *pdev,
> int pasids)
> 
>  	might_sleep();
> 
> +	/*
> +	 * When memory encryption is active the device is likely not in a
> +	 * direct-mapped domain. Forbid using IOMMUv2 functionality for
> now.
> +	 */
> +	if (mem_encrypt_active())
> +		return -ENODEV;
> +
>  	if (!amd_iommu_v2_supported())
>  		return -ENODEV;
> 
> --
> 2.28.0

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[-- Attachment #1.1: Type: text/plain, Size: 1434 bytes --]

[AMD Official Use Only - Internal Distribution Only]

+ Felix
________________________________
From: Joerg Roedel <joro@8bytes.org>
Sent: Monday, August 24, 2020 6:54 AM
To: iommu@lists.linux-foundation.org <iommu@lists.linux-foundation.org>
Cc: Joerg Roedel <joro@8bytes.org>; jroedel@suse.de <jroedel@suse.de>; Lendacky, Thomas <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org <linux-kernel@vger.kernel.org>
Subject: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

From: Joerg Roedel <jroedel@suse.de>

Hi,

Some IOMMUv2 capable devices do not work correctly when SME is
active, because their DMA mask does not include the encryption bit, so
that they can not DMA to encrypted memory directly.

The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2
capable devices into an identity mapped domain. Fix that by not
forcing an identity mapped domain on devices when SME is active and
forbid using their IOMMUv2 functionality.

Please review.

Thanks,

        Joerg

Joerg Roedel (2):
  iommu/amd: Do not force direct mapping when SME is active
  iommu/amd: Do not use IOMMUv2 functionality when SME is active

 drivers/iommu/amd/iommu.c    | 7 ++++++-
 drivers/iommu/amd/iommu_v2.c | 7 +++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

--
2.28.0


[-- Attachment #1.2: Type: text/html, Size: 2510 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Felix Kuehling]

[+Ray]


Thanks for the heads up. Currently KFD won't work on APUs when IOMMUv2
is disabled. But Ray is working on fallbacks that will allow KFD to work
on APUs even without IOMMUv2, similar to our dGPUs. Along with changes
in ROCm user mode, those fallbacks are necessary for making ROCm on APUs
generally useful.


How common is SME on typical PCs or laptops that would use AMD APUs?


Alex, do you know if anyone has tested amdgpu on an APU with SME
enabled? Is this considered something we support?


Thanks,
  Felix


Am 2020-08-26 um 10:14 a.m. schrieb Deucher, Alexander:
>
> [AMD Official Use Only - Internal Distribution Only]
>
>
> + Felix
> ------------------------------------------------------------------------
> *From:* Joerg Roedel <joro@8bytes.org>
> *Sent:* Monday, August 24, 2020 6:54 AM
> *To:* iommu@lists.linux-foundation.org <iommu@lists.linux-foundation.org>
> *Cc:* Joerg Roedel <joro@8bytes.org>; jroedel@suse.de
> <jroedel@suse.de>; Lendacky, Thomas <Thomas.Lendacky@amd.com>;
> Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; Deucher,
> Alexander <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org
> <linux-kernel@vger.kernel.org>
> *Subject:* [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active
>  
> From: Joerg Roedel <jroedel@suse.de>
>
> Hi,
>
> Some IOMMUv2 capable devices do not work correctly when SME is
> active, because their DMA mask does not include the encryption bit, so
> that they can not DMA to encrypted memory directly.
>
> The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2
> capable devices into an identity mapped domain. Fix that by not
> forcing an identity mapped domain on devices when SME is active and
> forbid using their IOMMUv2 functionality.
>
> Please review.
>
> Thanks,
>
>         Joerg
>
> Joerg Roedel (2):
>   iommu/amd: Do not force direct mapping when SME is active
>   iommu/amd: Do not use IOMMUv2 functionality when SME is active
>
>  drivers/iommu/amd/iommu.c    | 7 ++++++-
>  drivers/iommu/amd/iommu_v2.c | 7 +++++++
>  2 files changed, 13 insertions(+), 1 deletion(-)
>
> -- 
> 2.28.0
>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Felix Kuehling]

[+Ray]


Thanks for the heads up. Currently KFD won't work on APUs when IOMMUv2
is disabled. But Ray is working on fallbacks that will allow KFD to work
on APUs even without IOMMUv2, similar to our dGPUs. Along with changes
in ROCm user mode, those fallbacks are necessary for making ROCm on APUs
generally useful.


How common is SME on typical PCs or laptops that would use AMD APUs?


Alex, do you know if anyone has tested amdgpu on an APU with SME
enabled? Is this considered something we support?


Thanks,
  Felix


Am 2020-08-26 um 10:14 a.m. schrieb Deucher, Alexander:
>
> [AMD Official Use Only - Internal Distribution Only]
>
>
> + Felix
> ------------------------------------------------------------------------
> *From:* Joerg Roedel <joro@8bytes.org>
> *Sent:* Monday, August 24, 2020 6:54 AM
> *To:* iommu@lists.linux-foundation.org <iommu@lists.linux-foundation.org>
> *Cc:* Joerg Roedel <joro@8bytes.org>; jroedel@suse.de
> <jroedel@suse.de>; Lendacky, Thomas <Thomas.Lendacky@amd.com>;
> Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; Deucher,
> Alexander <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org
> <linux-kernel@vger.kernel.org>
> *Subject:* [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active
>  
> From: Joerg Roedel <jroedel@suse.de>
>
> Hi,
>
> Some IOMMUv2 capable devices do not work correctly when SME is
> active, because their DMA mask does not include the encryption bit, so
> that they can not DMA to encrypted memory directly.
>
> The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2
> capable devices into an identity mapped domain. Fix that by not
> forcing an identity mapped domain on devices when SME is active and
> forbid using their IOMMUv2 functionality.
>
> Please review.
>
> Thanks,
>
>         Joerg
>
> Joerg Roedel (2):
>   iommu/amd: Do not force direct mapping when SME is active
>   iommu/amd: Do not use IOMMUv2 functionality when SME is active
>
>  drivers/iommu/amd/iommu.c    | 7 ++++++-
>  drivers/iommu/amd/iommu_v2.c | 7 +++++++
>  2 files changed, 13 insertions(+), 1 deletion(-)
>
> -- 
> 2.28.0
>

RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[AMD Public Use]

 + Christian

> -----Original Message-----
> From: Kuehling, Felix <Felix.Kuehling@amd.com>
> Sent: Wednesday, August 26, 2020 11:22 AM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>; Joerg Roedel
> <joro@8bytes.org>; iommu@lists.linux-foundation.org; Huang, Ray
> <Ray.Huang@amd.com>
> Cc: jroedel@suse.de; Lendacky, Thomas <Thomas.Lendacky@amd.com>;
> Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> active
> 
> [+Ray]
> 
> 
> Thanks for the heads up. Currently KFD won't work on APUs when IOMMUv2
> is disabled. But Ray is working on fallbacks that will allow KFD to work on
> APUs even without IOMMUv2, similar to our dGPUs. Along with changes in
> ROCm user mode, those fallbacks are necessary for making ROCm on APUs
> generally useful.
> 
> 
> How common is SME on typical PCs or laptops that would use AMD APUs?

I think the hw supports it, but it as far as I know it's not formally productized on client parts.

> 
> 
> Alex, do you know if anyone has tested amdgpu on an APU with SME
> enabled? Is this considered something we support?

It's not something we've tested.  I'm not even sure the GPU portion of APUs will work properly without an identity mapping.  SME should work properly with dGPUs however, so this is a proper fix for them.  We don't use the IOMMUv2 path on dGPUs at all.

Alex

> 
> 
> Thanks,
>   Felix
> 
> 
> Am 2020-08-26 um 10:14 a.m. schrieb Deucher, Alexander:
> >
> > [AMD Official Use Only - Internal Distribution Only]
> >
> >
> > + Felix
> > ----------------------------------------------------------------------
> > --
> > *From:* Joerg Roedel <joro@8bytes.org>
> > *Sent:* Monday, August 24, 2020 6:54 AM
> > *To:* iommu@lists.linux-foundation.org
> > <iommu@lists.linux-foundation.org>
> > *Cc:* Joerg Roedel <joro@8bytes.org>; jroedel@suse.de
> > <jroedel@suse.de>; Lendacky, Thomas <Thomas.Lendacky@amd.com>;
> > Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; Deucher,
> > Alexander <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org
> > <linux-kernel@vger.kernel.org>
> > *Subject:* [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> > active
> >
> > From: Joerg Roedel <jroedel@suse.de>
> >
> > Hi,
> >
> > Some IOMMUv2 capable devices do not work correctly when SME is active,
> > because their DMA mask does not include the encryption bit, so that
> > they can not DMA to encrypted memory directly.
> >
> > The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2
> > capable devices into an identity mapped domain. Fix that by not
> > forcing an identity mapped domain on devices when SME is active and
> > forbid using their IOMMUv2 functionality.
> >
> > Please review.
> >
> > Thanks,
> >
> >         Joerg
> >
> > Joerg Roedel (2):
> >   iommu/amd: Do not force direct mapping when SME is active
> >   iommu/amd: Do not use IOMMUv2 functionality when SME is active
> >
> >  drivers/iommu/amd/iommu.c    | 7 ++++++-
> >  drivers/iommu/amd/iommu_v2.c | 7 +++++++
> >  2 files changed, 13 insertions(+), 1 deletion(-)
> >
> > --
> > 2.28.0
> >
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[AMD Public Use]

 + Christian

> -----Original Message-----
> From: Kuehling, Felix <Felix.Kuehling@amd.com>
> Sent: Wednesday, August 26, 2020 11:22 AM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>; Joerg Roedel
> <joro@8bytes.org>; iommu@lists.linux-foundation.org; Huang, Ray
> <Ray.Huang@amd.com>
> Cc: jroedel@suse.de; Lendacky, Thomas <Thomas.Lendacky@amd.com>;
> Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> active
> 
> [+Ray]
> 
> 
> Thanks for the heads up. Currently KFD won't work on APUs when IOMMUv2
> is disabled. But Ray is working on fallbacks that will allow KFD to work on
> APUs even without IOMMUv2, similar to our dGPUs. Along with changes in
> ROCm user mode, those fallbacks are necessary for making ROCm on APUs
> generally useful.
> 
> 
> How common is SME on typical PCs or laptops that would use AMD APUs?

I think the hw supports it, but it as far as I know it's not formally productized on client parts.

> 
> 
> Alex, do you know if anyone has tested amdgpu on an APU with SME
> enabled? Is this considered something we support?

It's not something we've tested.  I'm not even sure the GPU portion of APUs will work properly without an identity mapping.  SME should work properly with dGPUs however, so this is a proper fix for them.  We don't use the IOMMUv2 path on dGPUs at all.

Alex

> 
> 
> Thanks,
>   Felix
> 
> 
> Am 2020-08-26 um 10:14 a.m. schrieb Deucher, Alexander:
> >
> > [AMD Official Use Only - Internal Distribution Only]
> >
> >
> > + Felix
> > ----------------------------------------------------------------------
> > --
> > *From:* Joerg Roedel <joro@8bytes.org>
> > *Sent:* Monday, August 24, 2020 6:54 AM
> > *To:* iommu@lists.linux-foundation.org
> > <iommu@lists.linux-foundation.org>
> > *Cc:* Joerg Roedel <joro@8bytes.org>; jroedel@suse.de
> > <jroedel@suse.de>; Lendacky, Thomas <Thomas.Lendacky@amd.com>;
> > Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; Deucher,
> > Alexander <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org
> > <linux-kernel@vger.kernel.org>
> > *Subject:* [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> > active
> >
> > From: Joerg Roedel <jroedel@suse.de>
> >
> > Hi,
> >
> > Some IOMMUv2 capable devices do not work correctly when SME is active,
> > because their DMA mask does not include the encryption bit, so that
> > they can not DMA to encrypted memory directly.
> >
> > The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2
> > capable devices into an identity mapped domain. Fix that by not
> > forcing an identity mapped domain on devices when SME is active and
> > forbid using their IOMMUv2 functionality.
> >
> > Please review.
> >
> > Thanks,
> >
> >         Joerg
> >
> > Joerg Roedel (2):
> >   iommu/amd: Do not force direct mapping when SME is active
> >   iommu/amd: Do not use IOMMUv2 functionality when SME is active
> >
> >  drivers/iommu/amd/iommu.c    | 7 ++++++-
> >  drivers/iommu/amd/iommu_v2.c | 7 +++++++
> >  2 files changed, 13 insertions(+), 1 deletion(-)
> >
> > --
> > 2.28.0
> >

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[jroedel]

On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote:
> > Alex, do you know if anyone has tested amdgpu on an APU with SME
> > enabled? Is this considered something we support?
> 
> It's not something we've tested.  I'm not even sure the GPU portion of
> APUs will work properly without an identity mapping.  SME should work
> properly with dGPUs however, so this is a proper fix for them.  We
> don't use the IOMMUv2 path on dGPUs at all.

Is it possible to make the IOMMUv2 paths optional on iGPUs as well when
SME is active (or better, when the GPU is not identity mapped)?

Regards,

	Joerg
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[jroedel]

On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote:
> > Alex, do you know if anyone has tested amdgpu on an APU with SME
> > enabled? Is this considered something we support?
> 
> It's not something we've tested.  I'm not even sure the GPU portion of
> APUs will work properly without an identity mapping.  SME should work
> properly with dGPUs however, so this is a proper fix for them.  We
> don't use the IOMMUv2 path on dGPUs at all.

Is it possible to make the IOMMUv2 paths optional on iGPUs as well when
SME is active (or better, when the GPU is not identity mapped)?

Regards,

	Joerg

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Felix Kuehling]

Am 2020-08-28 um 9:46 a.m. schrieb jroedel@suse.de:
> On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote:
>>> Alex, do you know if anyone has tested amdgpu on an APU with SME
>>> enabled? Is this considered something we support?
>> It's not something we've tested.  I'm not even sure the GPU portion of
>> APUs will work properly without an identity mapping.  SME should work
>> properly with dGPUs however, so this is a proper fix for them.  We
>> don't use the IOMMUv2 path on dGPUs at all.
> Is it possible to make the IOMMUv2 paths optional on iGPUs as well when
> SME is active (or better, when the GPU is not identity mapped)?

Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not
needed for graphics. And we're making it optional for KFD as well.

The question Alex and I raised here is more general. We may have some
assumptions in the amdgpu driver that are broken when the framebuffer is
not identity mapped. This would break the iGPU in a more general sense,
regardless of KFD and IOMMUv2. In that case, we don't really need to
worry about breaking KFD because we have a much bigger problem.

Regards,
  Felix


>
> Regards,
>
> 	Joerg
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Felix Kuehling]

Am 2020-08-28 um 9:46 a.m. schrieb jroedel@suse.de:
> On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote:
>>> Alex, do you know if anyone has tested amdgpu on an APU with SME
>>> enabled? Is this considered something we support?
>> It's not something we've tested.  I'm not even sure the GPU portion of
>> APUs will work properly without an identity mapping.  SME should work
>> properly with dGPUs however, so this is a proper fix for them.  We
>> don't use the IOMMUv2 path on dGPUs at all.
> Is it possible to make the IOMMUv2 paths optional on iGPUs as well when
> SME is active (or better, when the GPU is not identity mapped)?

Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not
needed for graphics. And we're making it optional for KFD as well.

The question Alex and I raised here is more general. We may have some
assumptions in the amdgpu driver that are broken when the framebuffer is
not identity mapped. This would break the iGPU in a more general sense,
regardless of KFD and IOMMUv2. In that case, we don't really need to
worry about breaking KFD because we have a much bigger problem.

Regards,
  Felix


>
> Regards,
>
> 	Joerg

RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[AMD Public Use]

> -----Original Message-----
> From: Kuehling, Felix <Felix.Kuehling@amd.com>
> Sent: Friday, August 28, 2020 9:55 AM
> To: jroedel@suse.de; Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: Joerg Roedel <joro@8bytes.org>; iommu@lists.linux-foundation.org;
> Huang, Ray <Ray.Huang@amd.com>; Koenig, Christian
> <Christian.Koenig@amd.com>; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> active
> 
> Am 2020-08-28 um 9:46 a.m. schrieb jroedel@suse.de:
> > On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote:
> >>> Alex, do you know if anyone has tested amdgpu on an APU with SME
> >>> enabled? Is this considered something we support?
> >> It's not something we've tested.  I'm not even sure the GPU portion
> >> of APUs will work properly without an identity mapping.  SME should
> >> work properly with dGPUs however, so this is a proper fix for them.
> >> We don't use the IOMMUv2 path on dGPUs at all.
> > Is it possible to make the IOMMUv2 paths optional on iGPUs as well
> > when SME is active (or better, when the GPU is not identity mapped)?
> 
> Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not needed
> for graphics. And we're making it optional for KFD as well.
> 
> The question Alex and I raised here is more general. We may have some
> assumptions in the amdgpu driver that are broken when the framebuffer is
> not identity mapped. This would break the iGPU in a more general sense,
> regardless of KFD and IOMMUv2. In that case, we don't really need to worry
> about breaking KFD because we have a much bigger problem.

There are hw bugs on Raven and probably Carrizo/Stoney where they need 1:1 mapping to avoid bugs in some corner cases with the displays.  Other GPUs should be fine.  The VIDs is 0x1002 and the DIDs are 0x15dd and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876, 0x9877 and 0x98e4 for carrizo and stoney.  As long as we preserve the 1:1 mapping for those asics, we should be fine.

Alex

> 
> Regards,
>   Felix
> 
> 
> >
> > Regards,
> >
> > 	Joerg
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[AMD Public Use]

> -----Original Message-----
> From: Kuehling, Felix <Felix.Kuehling@amd.com>
> Sent: Friday, August 28, 2020 9:55 AM
> To: jroedel@suse.de; Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: Joerg Roedel <joro@8bytes.org>; iommu@lists.linux-foundation.org;
> Huang, Ray <Ray.Huang@amd.com>; Koenig, Christian
> <Christian.Koenig@amd.com>; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> active
> 
> Am 2020-08-28 um 9:46 a.m. schrieb jroedel@suse.de:
> > On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote:
> >>> Alex, do you know if anyone has tested amdgpu on an APU with SME
> >>> enabled? Is this considered something we support?
> >> It's not something we've tested.  I'm not even sure the GPU portion
> >> of APUs will work properly without an identity mapping.  SME should
> >> work properly with dGPUs however, so this is a proper fix for them.
> >> We don't use the IOMMUv2 path on dGPUs at all.
> > Is it possible to make the IOMMUv2 paths optional on iGPUs as well
> > when SME is active (or better, when the GPU is not identity mapped)?
> 
> Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not needed
> for graphics. And we're making it optional for KFD as well.
> 
> The question Alex and I raised here is more general. We may have some
> assumptions in the amdgpu driver that are broken when the framebuffer is
> not identity mapped. This would break the iGPU in a more general sense,
> regardless of KFD and IOMMUv2. In that case, we don't really need to worry
> about breaking KFD because we have a much bigger problem.

There are hw bugs on Raven and probably Carrizo/Stoney where they need 1:1 mapping to avoid bugs in some corner cases with the displays.  Other GPUs should be fine.  The VIDs is 0x1002 and the DIDs are 0x15dd and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876, 0x9877 and 0x98e4 for carrizo and stoney.  As long as we preserve the 1:1 mapping for those asics, we should be fine.

Alex

> 
> Regards,
>   Felix
> 
> 
> >
> > Regards,
> >
> > 	Joerg

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[jroedel]

On Fri, Aug 28, 2020 at 03:11:32PM +0000, Deucher, Alexander wrote:
> There are hw bugs on Raven and probably Carrizo/Stoney where they need
> 1:1 mapping to avoid bugs in some corner cases with the displays.
> Other GPUs should be fine.  The VIDs is 0x1002 and the DIDs are 0x15dd
> and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876,
> 0x9877 and 0x98e4 for carrizo and stoney.  As long as we
> preserve the 1:1 mapping for those asics, we should be fine.

Okay, Stoney at least has no Zen-based CPU, so no support for memory
encryption anyway. How about Raven, is it paired with a Zen CPU?

Regards,

	Joerg
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[jroedel]

On Fri, Aug 28, 2020 at 03:11:32PM +0000, Deucher, Alexander wrote:
> There are hw bugs on Raven and probably Carrizo/Stoney where they need
> 1:1 mapping to avoid bugs in some corner cases with the displays.
> Other GPUs should be fine.  The VIDs is 0x1002 and the DIDs are 0x15dd
> and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876,
> 0x9877 and 0x98e4 for carrizo and stoney.  As long as we
> preserve the 1:1 mapping for those asics, we should be fine.

Okay, Stoney at least has no Zen-based CPU, so no support for memory
encryption anyway. How about Raven, is it paired with a Zen CPU?

Regards,

	Joerg

RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[AMD Public Use]

> -----Original Message-----
> From: jroedel@suse.de <jroedel@suse.de>
> Sent: Friday, August 28, 2020 11:30 AM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Joerg Roedel
> <joro@8bytes.org>; iommu@lists.linux-foundation.org; Huang, Ray
> <Ray.Huang@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>;
> Lendacky, Thomas <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> active
> 
> On Fri, Aug 28, 2020 at 03:11:32PM +0000, Deucher, Alexander wrote:
> > There are hw bugs on Raven and probably Carrizo/Stoney where they need
> > 1:1 mapping to avoid bugs in some corner cases with the displays.
> > Other GPUs should be fine.  The VIDs is 0x1002 and the DIDs are 0x15dd
> > and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876,
> > 0x9877 and 0x98e4 for carrizo and stoney.  As long as we preserve the
> > 1:1 mapping for those asics, we should be fine.
> 
> Okay, Stoney at least has no Zen-based CPU, so no support for memory
> encryption anyway. How about Raven, is it paired with a Zen CPU?

Ah, right,  So CZ and ST are not an issue.  Raven is paired with Zen based CPUs.

Thanks,

Alex
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[AMD Public Use]

> -----Original Message-----
> From: jroedel@suse.de <jroedel@suse.de>
> Sent: Friday, August 28, 2020 11:30 AM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Joerg Roedel
> <joro@8bytes.org>; iommu@lists.linux-foundation.org; Huang, Ray
> <Ray.Huang@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>;
> Lendacky, Thomas <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> active
> 
> On Fri, Aug 28, 2020 at 03:11:32PM +0000, Deucher, Alexander wrote:
> > There are hw bugs on Raven and probably Carrizo/Stoney where they need
> > 1:1 mapping to avoid bugs in some corner cases with the displays.
> > Other GPUs should be fine.  The VIDs is 0x1002 and the DIDs are 0x15dd
> > and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876,
> > 0x9877 and 0x98e4 for carrizo and stoney.  As long as we preserve the
> > 1:1 mapping for those asics, we should be fine.
> 
> Okay, Stoney at least has no Zen-based CPU, so no support for memory
> encryption anyway. How about Raven, is it paired with a Zen CPU?

Ah, right,  So CZ and ST are not an issue.  Raven is paired with Zen based CPUs.

Thanks,

Alex

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Joerg Roedel]

On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote:
> Ah, right,  So CZ and ST are not an issue.  Raven is paired with Zen based CPUs.

Okay, so for the Raven case, can you add code to the amdgpu driver which
makes it fail to initialize on Raven when SME is active? There is a
global checking function for that, so that shouldn't be hard to do.

Regards,

	Joerg
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Joerg Roedel]

On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote:
> Ah, right,  So CZ and ST are not an issue.  Raven is paired with Zen based CPUs.

Okay, so for the Raven case, can you add code to the amdgpu driver which
makes it fail to initialize on Raven when SME is active? There is a
global checking function for that, so that shouldn't be hard to do.

Regards,

	Joerg

RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

[AMD Official Use Only - Internal Distribution Only]

> -----Original Message-----
> From: Joerg Roedel <joro@8bytes.org>
> Sent: Friday, September 4, 2020 6:06 AM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: jroedel@suse.de; Kuehling, Felix <Felix.Kuehling@amd.com>;
> iommu@lists.linux-foundation.org; Huang, Ray <Ray.Huang@amd.com>;
> Koenig, Christian <Christian.Koenig@amd.com>; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> active
>
> On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote:
> > Ah, right,  So CZ and ST are not an issue.  Raven is paired with Zen based
> CPUs.
>
> Okay, so for the Raven case, can you add code to the amdgpu driver which
> makes it fail to initialize on Raven when SME is active? There is a global
> checking function for that, so that shouldn't be hard to do.
>

Sure.  How about the attached patch?

Alex


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-drm-amdgpu-Fail-to-load-on-RAVEN-if-SME-is-active.patch --]
[-- Type: text/x-patch; name="0001-drm-amdgpu-Fail-to-load-on-RAVEN-if-SME-is-active.patch", Size: 1438 bytes --]

From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001
From: Alex Deucher <alexander.deucher@amd.com>
Date: Sun, 6 Sep 2020 12:05:12 -0400
Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active

Due to hardware bugs, scatter/gather display on raven requires
a 1:1 IOMMU mapping, however, SME (System Memory Encryption)
requires an indirect IOMMU mapping because the encryption bit
is beyond the DMA mask of the chip.  As such, the two are
incompatible.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 12e16445df7c..d87d37c25329 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
 		return -ENODEV;
 	}
 
+	/* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping,
+	 * however, SME requires an indirect IOMMU mapping because the encryption
+	 * bit is beyond the DMA mask of the chip.
+	 */
+	if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) {
+		dev_info(&pdev->dev,
+			 "SME is not compatible with RAVEN\n");
+		return -ENOTSUPP;
+	}
+
 #ifdef CONFIG_DRM_AMDGPU_SI
 	if (!amdgpu_si_support) {
 		switch (flags & AMD_ASIC_MASK) {
-- 
2.25.4


[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Deucher, Alexander]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

[AMD Official Use Only - Internal Distribution Only]

> -----Original Message-----
> From: Joerg Roedel <joro@8bytes.org>
> Sent: Friday, September 4, 2020 6:06 AM
> To: Deucher, Alexander <Alexander.Deucher@amd.com>
> Cc: jroedel@suse.de; Kuehling, Felix <Felix.Kuehling@amd.com>;
> iommu@lists.linux-foundation.org; Huang, Ray <Ray.Huang@amd.com>;
> Koenig, Christian <Christian.Koenig@amd.com>; Lendacky, Thomas
> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org
> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
> active
>
> On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote:
> > Ah, right,  So CZ and ST are not an issue.  Raven is paired with Zen based
> CPUs.
>
> Okay, so for the Raven case, can you add code to the amdgpu driver which
> makes it fail to initialize on Raven when SME is active? There is a global
> checking function for that, so that shouldn't be hard to do.
>

Sure.  How about the attached patch?

Alex


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-drm-amdgpu-Fail-to-load-on-RAVEN-if-SME-is-active.patch --]
[-- Type: text/x-patch; name="0001-drm-amdgpu-Fail-to-load-on-RAVEN-if-SME-is-active.patch", Size: 1438 bytes --]

From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001
From: Alex Deucher <alexander.deucher@amd.com>
Date: Sun, 6 Sep 2020 12:05:12 -0400
Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active

Due to hardware bugs, scatter/gather display on raven requires
a 1:1 IOMMU mapping, however, SME (System Memory Encryption)
requires an indirect IOMMU mapping because the encryption bit
is beyond the DMA mask of the chip.  As such, the two are
incompatible.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 12e16445df7c..d87d37c25329 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
 		return -ENODEV;
 	}
 
+	/* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping,
+	 * however, SME requires an indirect IOMMU mapping because the encryption
+	 * bit is beyond the DMA mask of the chip.
+	 */
+	if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) {
+		dev_info(&pdev->dev,
+			 "SME is not compatible with RAVEN\n");
+		return -ENOTSUPP;
+	}
+
 #ifdef CONFIG_DRM_AMDGPU_SI
 	if (!amdgpu_si_support) {
 		switch (flags & AMD_ASIC_MASK) {
-- 
2.25.4

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Joerg Roedel]

On Sun, Sep 06, 2020 at 04:08:58PM +0000, Deucher, Alexander wrote:
> From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001
> From: Alex Deucher <alexander.deucher@amd.com>
> Date: Sun, 6 Sep 2020 12:05:12 -0400
> Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active
> 
> Due to hardware bugs, scatter/gather display on raven requires
> a 1:1 IOMMU mapping, however, SME (System Memory Encryption)
> requires an indirect IOMMU mapping because the encryption bit
> is beyond the DMA mask of the chip.  As such, the two are
> incompatible.
> 
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 12e16445df7c..d87d37c25329 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
>  		return -ENODEV;
>  	}
>  
> +	/* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping,
> +	 * however, SME requires an indirect IOMMU mapping because the encryption
> +	 * bit is beyond the DMA mask of the chip.
> +	 */
> +	if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) {
> +		dev_info(&pdev->dev,
> +			 "SME is not compatible with RAVEN\n");
> +		return -ENOTSUPP;
> +	}
> +
>  #ifdef CONFIG_DRM_AMDGPU_SI
>  	if (!amdgpu_si_support) {
>  		switch (flags & AMD_ASIC_MASK) {
> -- 
> 2.25.4
>

Looks good to me, thanks.

Acked-by: Joerg Roedel <jroedel@suse.de>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Joerg Roedel]

On Sun, Sep 06, 2020 at 04:08:58PM +0000, Deucher, Alexander wrote:
> From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001
> From: Alex Deucher <alexander.deucher@amd.com>
> Date: Sun, 6 Sep 2020 12:05:12 -0400
> Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active
> 
> Due to hardware bugs, scatter/gather display on raven requires
> a 1:1 IOMMU mapping, however, SME (System Memory Encryption)
> requires an indirect IOMMU mapping because the encryption bit
> is beyond the DMA mask of the chip.  As such, the two are
> incompatible.
> 
> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 12e16445df7c..d87d37c25329 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
>  		return -ENODEV;
>  	}
>  
> +	/* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping,
> +	 * however, SME requires an indirect IOMMU mapping because the encryption
> +	 * bit is beyond the DMA mask of the chip.
> +	 */
> +	if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) {
> +		dev_info(&pdev->dev,
> +			 "SME is not compatible with RAVEN\n");
> +		return -ENOTSUPP;
> +	}
> +
>  #ifdef CONFIG_DRM_AMDGPU_SI
>  	if (!amdgpu_si_support) {
>  		switch (flags & AMD_ASIC_MASK) {
> -- 
> 2.25.4
>

Looks good to me, thanks.

Acked-by: Joerg Roedel <jroedel@suse.de>

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Christian König]

Am 07.09.20 um 12:44 schrieb Joerg Roedel:
> On Sun, Sep 06, 2020 at 04:08:58PM +0000, Deucher, Alexander wrote:
>>  From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001
>> From: Alex Deucher <alexander.deucher@amd.com>
>> Date: Sun, 6 Sep 2020 12:05:12 -0400
>> Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active
>>
>> Due to hardware bugs, scatter/gather display on raven requires
>> a 1:1 IOMMU mapping, however, SME (System Memory Encryption)
>> requires an indirect IOMMU mapping because the encryption bit
>> is beyond the DMA mask of the chip.  As such, the two are
>> incompatible.
>>
>> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> index 12e16445df7c..d87d37c25329 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> @@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
>>   		return -ENODEV;
>>   	}
>>   
>> +	/* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping,
>> +	 * however, SME requires an indirect IOMMU mapping because the encryption
>> +	 * bit is beyond the DMA mask of the chip.
>> +	 */
>> +	if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) {
>> +		dev_info(&pdev->dev,
>> +			 "SME is not compatible with RAVEN\n");
>> +		return -ENOTSUPP;
>> +	}
>> +
>>   #ifdef CONFIG_DRM_AMDGPU_SI
>>   	if (!amdgpu_si_support) {
>>   		switch (flags & AMD_ASIC_MASK) {
>> -- 
>> 2.25.4
>>
> Looks good to me, thanks.
>
> Acked-by: Joerg Roedel <jroedel@suse.de>

This is really unfortunate, but I don't see any other solution either.

Reviewed-by: Christian König <christian.koenig@amd.com>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Christian König]

Am 07.09.20 um 12:44 schrieb Joerg Roedel:
> On Sun, Sep 06, 2020 at 04:08:58PM +0000, Deucher, Alexander wrote:
>>  From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001
>> From: Alex Deucher <alexander.deucher@amd.com>
>> Date: Sun, 6 Sep 2020 12:05:12 -0400
>> Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active
>>
>> Due to hardware bugs, scatter/gather display on raven requires
>> a 1:1 IOMMU mapping, however, SME (System Memory Encryption)
>> requires an indirect IOMMU mapping because the encryption bit
>> is beyond the DMA mask of the chip.  As such, the two are
>> incompatible.
>>
>> Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> index 12e16445df7c..d87d37c25329 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> @@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev,
>>   		return -ENODEV;
>>   	}
>>   
>> +	/* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping,
>> +	 * however, SME requires an indirect IOMMU mapping because the encryption
>> +	 * bit is beyond the DMA mask of the chip.
>> +	 */
>> +	if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) {
>> +		dev_info(&pdev->dev,
>> +			 "SME is not compatible with RAVEN\n");
>> +		return -ENOTSUPP;
>> +	}
>> +
>>   #ifdef CONFIG_DRM_AMDGPU_SI
>>   	if (!amdgpu_si_support) {
>>   		switch (flags & AMD_ASIC_MASK) {
>> -- 
>> 2.25.4
>>
> Looks good to me, thanks.
>
> Acked-by: Joerg Roedel <jroedel@suse.de>

This is really unfortunate, but I don't see any other solution either.

Reviewed-by: Christian König <christian.koenig@amd.com>

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Felix Kuehling]

Am 2020-09-06 um 12:08 p.m. schrieb Deucher, Alexander:
> [AMD Official Use Only - Internal Distribution Only]
>
>> -----Original Message-----
>> From: Joerg Roedel <joro@8bytes.org>
>> Sent: Friday, September 4, 2020 6:06 AM
>> To: Deucher, Alexander <Alexander.Deucher@amd.com>
>> Cc: jroedel@suse.de; Kuehling, Felix <Felix.Kuehling@amd.com>;
>> iommu@lists.linux-foundation.org; Huang, Ray <Ray.Huang@amd.com>;
>> Koenig, Christian <Christian.Koenig@amd.com>; Lendacky, Thomas
>> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
>> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
>> active
>>
>> On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote:
>>> Ah, right,  So CZ and ST are not an issue.  Raven is paired with Zen based
>> CPUs.
>>
>> Okay, so for the Raven case, can you add code to the amdgpu driver which
>> makes it fail to initialize on Raven when SME is active? There is a global
>> checking function for that, so that shouldn't be hard to do.
>>
> Sure.  How about the attached patch?

The patch is

Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>

Thanks,
  Felix


>
> Alex
>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[Felix Kuehling]

Am 2020-09-06 um 12:08 p.m. schrieb Deucher, Alexander:
> [AMD Official Use Only - Internal Distribution Only]
>
>> -----Original Message-----
>> From: Joerg Roedel <joro@8bytes.org>
>> Sent: Friday, September 4, 2020 6:06 AM
>> To: Deucher, Alexander <Alexander.Deucher@amd.com>
>> Cc: jroedel@suse.de; Kuehling, Felix <Felix.Kuehling@amd.com>;
>> iommu@lists.linux-foundation.org; Huang, Ray <Ray.Huang@amd.com>;
>> Koenig, Christian <Christian.Koenig@amd.com>; Lendacky, Thomas
>> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee
>> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org
>> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is
>> active
>>
>> On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote:
>>> Ah, right,  So CZ and ST are not an issue.  Raven is paired with Zen based
>> CPUs.
>>
>> Okay, so for the Raven case, can you add code to the amdgpu driver which
>> makes it fail to initialize on Raven when SME is active? There is a global
>> checking function for that, so that shouldn't be hard to do.
>>
> Sure.  How about the attached patch?

The patch is

Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>

Thanks,
  Felix


>
> Alex
>

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[jroedel]

Hi Felix,

On Fri, Aug 28, 2020 at 09:54:59AM -0400, Felix Kuehling wrote:
> Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not
> needed for graphics. And we're making it optional for KFD as well.

Okay, KFD should fail gracefully because it can't initialize the
device's iommuv2 functionality.


Regards,

	Joerg
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active

[jroedel]

Hi Felix,

On Fri, Aug 28, 2020 at 09:54:59AM -0400, Felix Kuehling wrote:
> Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not
> needed for graphics. And we're making it optional for KFD as well.

Okay, KFD should fail gracefully because it can't initialize the
device's iommuv2 functionality.


Regards,

	Joerg