[PATCH v2] vfio iommu: Add dma available capability

[PATCH v2] vfio iommu: Add dma available capability

[Matthew Rosato]

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added
a limit to the number of concurrent DMA requests for a vfio container.
However, lazy unmapping in s390 can in fact cause quite a large number of
outstanding DMA requests to build up prior to being purged, potentially 
the entire guest DMA space.  This results in unexpected 'VFIO_MAP_DMA
failed: No space left on device' conditions seen in QEMU.

This patch proposes to provide the remaining number of allowable DMA 
requests via the VFIO_IOMMU_GET_INFO ioctl as a new capability.  A 
subsequent patchset to QEMU would collect this information and use it in 
s390 PCI support to tap the guest on the shoulder before overrunning the 
vfio limit.

Changes from v1:
- Report dma_avail instead of the limit, which might not have been accurate
  anyway
- Text/naming changes throughout due to the above

Matthew Rosato (1):
  vfio iommu: Add dma available capability

 drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
 include/uapi/linux/vfio.h       | 16 ++++++++++++++++
 2 files changed, 33 insertions(+)

-- 
1.8.3.1

[PATCH v2] vfio iommu: Add dma available capability

[Matthew Rosato]

Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container")
added the ability to limit the number of memory backed DMA mappings.
However on s390x, when lazy mapping is in use, we use a very large
number of concurrent mappings.  Let's provide the current allowable
number of DMA mappings to userspace via the IOMMU info chain so that
userspace can take appropriate mitigation.

Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
---
 drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
 include/uapi/linux/vfio.h       | 16 ++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 5fbf0c1..15e21db 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -2609,6 +2609,20 @@ static int vfio_iommu_migration_build_caps(struct vfio_iommu *iommu,
 	return vfio_info_add_capability(caps, &cap_mig.header, sizeof(cap_mig));
 }
 
+static int vfio_iommu_dma_avail_build_caps(struct vfio_iommu *iommu,
+					   struct vfio_info_cap *caps)
+{
+	struct vfio_iommu_type1_info_dma_avail cap_dma_avail;
+
+	cap_dma_avail.header.id = VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL;
+	cap_dma_avail.header.version = 1;
+
+	cap_dma_avail.avail = iommu->dma_avail;
+
+	return vfio_info_add_capability(caps, &cap_dma_avail.header,
+					sizeof(cap_dma_avail));
+}
+
 static int vfio_iommu_type1_get_info(struct vfio_iommu *iommu,
 				     unsigned long arg)
 {
@@ -2642,6 +2656,9 @@ static int vfio_iommu_type1_get_info(struct vfio_iommu *iommu,
 	ret = vfio_iommu_migration_build_caps(iommu, &caps);
 
 	if (!ret)
+		ret = vfio_iommu_dma_avail_build_caps(iommu, &caps);
+
+	if (!ret)
 		ret = vfio_iommu_iova_build_caps(iommu, &caps);
 
 	mutex_unlock(&iommu->lock);
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 9204705..a8cc4a5 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
 	__u64	max_dirty_bitmap_size;		/* in bytes */
 };
 
+/*
+ * The DMA available capability allows to report the current number of
+ * simultaneously outstanding DMA mappings that are allowed.
+ *
+ * The structures below define version 1 of this capability.
+ *
+ * max: specifies the maximum number of outstanding DMA mappings allowed.
+ */
+#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
+
+struct vfio_iommu_type1_info_dma_avail {
+	struct	vfio_info_cap_header header;
+	__u32	avail;
+};
+
+
 #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
 
 /**
-- 
1.8.3.1

Re: [PATCH v2] vfio iommu: Add dma available capability

[Cornelia Huck]

On Mon, 14 Sep 2020 18:25:31 -0400
Matthew Rosato <mjrosato@linux.ibm.com> wrote:

> Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container")
> added the ability to limit the number of memory backed DMA mappings.
> However on s390x, when lazy mapping is in use, we use a very large
> number of concurrent mappings.  Let's provide the current allowable
> number of DMA mappings to userspace via the IOMMU info chain so that
> userspace can take appropriate mitigation.
> 
> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
> ---
>  drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
>  include/uapi/linux/vfio.h       | 16 ++++++++++++++++
>  2 files changed, 33 insertions(+)

(...)

> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> index 9204705..a8cc4a5 100644
> --- a/include/uapi/linux/vfio.h
> +++ b/include/uapi/linux/vfio.h
> @@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
>  	__u64	max_dirty_bitmap_size;		/* in bytes */
>  };
>  
> +/*
> + * The DMA available capability allows to report the current number of
> + * simultaneously outstanding DMA mappings that are allowed.
> + *
> + * The structures below define version 1 of this capability.

"The structure below defines..." ?

> + *
> + * max: specifies the maximum number of outstanding DMA mappings allowed.

I think you forgot to tweak that one:

"avail: specifies the current number of outstanding DMA mappings allowed."

?

> + */
> +#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
> +
> +struct vfio_iommu_type1_info_dma_avail {
> +	struct	vfio_info_cap_header header;
> +	__u32	avail;
> +};
> +
> +
>  #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>  
>  /**

Re: [PATCH v2] vfio iommu: Add dma available capability

[Matthew Rosato]

On 9/15/20 5:44 AM, Cornelia Huck wrote:
> On Mon, 14 Sep 2020 18:25:31 -0400
> Matthew Rosato <mjrosato@linux.ibm.com> wrote:
> 
>> Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container")
>> added the ability to limit the number of memory backed DMA mappings.
>> However on s390x, when lazy mapping is in use, we use a very large
>> number of concurrent mappings.  Let's provide the current allowable
>> number of DMA mappings to userspace via the IOMMU info chain so that
>> userspace can take appropriate mitigation.
>>
>> Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
>> ---
>>   drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
>>   include/uapi/linux/vfio.h       | 16 ++++++++++++++++
>>   2 files changed, 33 insertions(+)
> 
> (...)
> 
>> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>> index 9204705..a8cc4a5 100644
>> --- a/include/uapi/linux/vfio.h
>> +++ b/include/uapi/linux/vfio.h
>> @@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
>>   	__u64	max_dirty_bitmap_size;		/* in bytes */
>>   };
>>   
>> +/*
>> + * The DMA available capability allows to report the current number of
>> + * simultaneously outstanding DMA mappings that are allowed.
>> + *
>> + * The structures below define version 1 of this capability.
> 
> "The structure below defines..." ?
> 

OK

>> + *
>> + * max: specifies the maximum number of outstanding DMA mappings allowed.
> 
> I think you forgot to tweak that one:
> 
> "avail: specifies the current number of outstanding DMA mappings allowed."
> 
> ?

Yep, that's a leftover from v1 :(

> 
>> + */
>> +#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
>> +
>> +struct vfio_iommu_type1_info_dma_avail {
>> +	struct	vfio_info_cap_header header;
>> +	__u32	avail;
>> +};
>> +
>> +
>>   #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>>   
>>   /**
> 

Re: [PATCH v2] vfio iommu: Add dma available capability

[Matthew Rosato]

On 9/14/20 6:25 PM, Matthew Rosato wrote:
> Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added
> a limit to the number of concurrent DMA requests for a vfio container.
> However, lazy unmapping in s390 can in fact cause quite a large number of
> outstanding DMA requests to build up prior to being purged, potentially
> the entire guest DMA space.  This results in unexpected 'VFIO_MAP_DMA
> failed: No space left on device' conditions seen in QEMU.
> 
> This patch proposes to provide the remaining number of allowable DMA
> requests via the VFIO_IOMMU_GET_INFO ioctl as a new capability.  A
> subsequent patchset to QEMU would collect this information and use it in
> s390 PCI support to tap the guest on the shoulder before overrunning the
> vfio limit.

Link to associated v2 QEMU patchset:
https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg05067.html

> 
> Changes from v1:
> - Report dma_avail instead of the limit, which might not have been accurate
>    anyway
> - Text/naming changes throughout due to the above
> 
> Matthew Rosato (1):
>    vfio iommu: Add dma available capability
> 
>   drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
>   include/uapi/linux/vfio.h       | 16 ++++++++++++++++
>   2 files changed, 33 insertions(+)
>