[PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[huaqian.li]

From: Li Hua Qian <huaqian.li@siemens.com>

This patchset introduces a change to make the IO_TLB_SEGSIZE parameter
configurable via a new kernel configuration option (CONFIG_SWIOTLB_SEGSIZE).

In certain applications, the default value of IO_TLB_SEGSIZE (128) may
not be sufficient for memory allocation, leading to runtime errors. By
making this parameter configurable, users can adjust the segment size to
better suit their specific use cases, improving flexibility and system
stability.

Li Hua Qian (1):
  swiotlb: Make IO_TLB_SEGSIZE configurable

 include/linux/swiotlb.h | 2 +-
 kernel/dma/Kconfig      | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

-- 
2.34.1

[PATCH 1/1] swiotlb: Make IO_TLB_SEGSIZE configurable

[huaqian.li]

From: Li Hua Qian <huaqian.li@siemens.com>

In some applications, the default value of 128 is not sufficient for
memory allocation and can cause runtime errors. This change makes
IO_TLB_SEGSIZE configurable, allowing it to be increased if needed.

Signed-off-by: Li Hua Qian <huaqian.li@siemens.com>
---
 include/linux/swiotlb.h | 2 +-
 kernel/dma/Kconfig      | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 3dae0f592063..145c71f8329d 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -23,7 +23,7 @@ struct scatterlist;
  * must be a power of 2.  What is the appropriate value ?
  * The complexity of {map,unmap}_single is linearly dependent on this value.
  */
-#define IO_TLB_SEGSIZE	128
+#define IO_TLB_SEGSIZE CONFIG_SWIOTLB_SEGSIZE
 
 /*
  * log of the size of each IO TLB slab.  The number of slabs is command line
diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig
index 31cfdb6b4bc3..38bfa84b96b6 100644
--- a/kernel/dma/Kconfig
+++ b/kernel/dma/Kconfig
@@ -102,6 +102,13 @@ config SWIOTLB_DYNAMIC
 
 	  If unsure, say N.
 
+config SWIOTLB_SEGSIZE
+	int "SWIOTLB segment size"
+	default 128
+	help
+	  Set the maximum allowable number of contiguous slabs to map.
+	  Must be a power of 2.
+
 config DMA_BOUNCE_UNALIGNED_KMALLOC
 	bool
 	depends on SWIOTLB
-- 
2.34.1

Re: [PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[Marek Szyprowski]

On 22.04.2025 08:37, huaqian.li@siemens.com wrote:
> From: Li Hua Qian <huaqian.li@siemens.com>
>
> This patchset introduces a change to make the IO_TLB_SEGSIZE parameter
> configurable via a new kernel configuration option (CONFIG_SWIOTLB_SEGSIZE).
>
> In certain applications, the default value of IO_TLB_SEGSIZE (128) may
> not be sufficient for memory allocation, leading to runtime errors. By
> making this parameter configurable, users can adjust the segment size to
> better suit their specific use cases, improving flexibility and system
> stability.

Could You elaborate a bit more what are those certain applications that 
require increasing IO_TLB_SEGSIZE? I'm not against it, but such change 
should be well justified and described, while the above cover-letter 
doesn't provide anything more than is written in the patch description.


> Li Hua Qian (1):
>    swiotlb: Make IO_TLB_SEGSIZE configurable
>
>   include/linux/swiotlb.h | 2 +-
>   kernel/dma/Kconfig      | 7 +++++++
>   2 files changed, 8 insertions(+), 1 deletion(-)
>
Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[Li, Hua Qian]

On Tue, 2025-04-22 at 15:36 +0200, Marek Szyprowski wrote:
> On 22.04.2025 08:37, huaqian.li@siemens.com wrote:
> > From: Li Hua Qian <huaqian.li@siemens.com>
> > 
> > This patchset introduces a change to make the IO_TLB_SEGSIZE
> > parameter
> > configurable via a new kernel configuration option
> > (CONFIG_SWIOTLB_SEGSIZE).
> > 
> > In certain applications, the default value of IO_TLB_SEGSIZE (128)
> > may
> > not be sufficient for memory allocation, leading to runtime errors.
> > By
> > making this parameter configurable, users can adjust the segment
> > size to
> > better suit their specific use cases, improving flexibility and
> > system
> > stability.
> 
> Could You elaborate a bit more what are those certain applications
> that 
> require increasing IO_TLB_SEGSIZE? I'm not against it, but such
> change 
> should be well justified and described, while the above cover-letter 
> doesn't provide anything more than is written in the patch
> description.
Thank you for your feedback, Marek.

To provide more context, one specific application that requires
increasing IO_TLB_SEGSIZE is the Hailo 8 PCIe AI card. This card uses
dma_alloc_coherent to allocate descriptor lists, as seen in the Hailo
driver implementation here: 
https://github.com/hailo-ai/hailort-drivers/blob/7161f9ee5918029bd4497f590003c2f87ec32507/linux/vdma/memory.c#L322
The maximum size (nslots) for these allocations can reach 160, which
exceeds the current default value of IO_TLB_SEGSIZE (128).

Since IO_TLB_SEGSIZE is defined as a constant in the kernel:

`#define IO_TLB_SEGSIZE 128`


this limitation causes swiotlb_search_pool_area, 
https://github.com/torvalds/linux/blame/v6.15-rc2/kernel/dma/swiotlb.c#L1085,
(or swiotlb_do_find_slots in older kernels) to fail when attempting to
allocate contiguous physical memory (CMA). This results in runtime
errors and prevents the Hailo 8 card from functioning correctly in
certain configurations.

By making IO_TLB_SEGSIZE configurable via a kernel configuration option
(CONFIG_SWIOTLB_SEGSIZE), users can adjust the segment size to
accommodate such use cases. This change improves flexibility and
ensures that systems can be tailored to meet the requirements of
specific hardware, such as the Hailo 8 PCIe AI card, without requiring
kernel source modifications.

I hope this example clarifies the need for this change. Please let me
know if further details or additional examples are required.

Best Regards,
Li Hua Qian
> 
> 
> > Li Hua Qian (1):
> >    swiotlb: Make IO_TLB_SEGSIZE configurable
> > 
> >   include/linux/swiotlb.h | 2 +-
> >   kernel/dma/Kconfig      | 7 +++++++
> >   2 files changed, 8 insertions(+), 1 deletion(-)
> > 
> Best regards

-- 
Hua Qian Li
Siemens AG
www.siemens.com

Re: [PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[Robin Murphy]

On 24/04/2025 6:12 am, Li, Hua Qian wrote:
> On Tue, 2025-04-22 at 15:36 +0200, Marek Szyprowski wrote:
>> On 22.04.2025 08:37, huaqian.li@siemens.com wrote:
>>> From: Li Hua Qian <huaqian.li@siemens.com>
>>>
>>> This patchset introduces a change to make the IO_TLB_SEGSIZE
>>> parameter
>>> configurable via a new kernel configuration option
>>> (CONFIG_SWIOTLB_SEGSIZE).
>>>
>>> In certain applications, the default value of IO_TLB_SEGSIZE (128)
>>> may
>>> not be sufficient for memory allocation, leading to runtime errors.
>>> By
>>> making this parameter configurable, users can adjust the segment
>>> size to
>>> better suit their specific use cases, improving flexibility and
>>> system
>>> stability.
>>
>> Could You elaborate a bit more what are those certain applications
>> that
>> require increasing IO_TLB_SEGSIZE? I'm not against it, but such
>> change
>> should be well justified and described, while the above cover-letter
>> doesn't provide anything more than is written in the patch
>> description.
> Thank you for your feedback, Marek.
> 
> To provide more context, one specific application that requires
> increasing IO_TLB_SEGSIZE is the Hailo 8 PCIe AI card. This card uses
> dma_alloc_coherent to allocate descriptor lists, as seen in the Hailo
> driver implementation here:
> https://github.com/hailo-ai/hailort-drivers/blob/7161f9ee5918029bd4497f590003c2f87ec32507/linux/vdma/memory.c#L322
> The maximum size (nslots) for these allocations can reach 160, which
> exceeds the current default value of IO_TLB_SEGSIZE (128).
> 
> Since IO_TLB_SEGSIZE is defined as a constant in the kernel:
> 
> `#define IO_TLB_SEGSIZE 128`
> 
> 
> this limitation causes swiotlb_search_pool_area,
> https://github.com/torvalds/linux/blame/v6.15-rc2/kernel/dma/swiotlb.c#L1085,
> (or swiotlb_do_find_slots in older kernels) to fail when attempting to
> allocate contiguous physical memory (CMA). This results in runtime
> errors and prevents the Hailo 8 card from functioning correctly in
> certain configurations.

Hmm, dma_alloc_coherent() should really not be trying to allocate from 
SWIOTLB in the first place - how is that happening?

If you're using restricted DMA for a device which wants significant 
coherent allocations, then it wants to have it's own shared-dma-pool for 
those *as well* as the restricted-dma-pool for bouncing streaming DMA.

Thanks,
Robin.

> By making IO_TLB_SEGSIZE configurable via a kernel configuration option
> (CONFIG_SWIOTLB_SEGSIZE), users can adjust the segment size to
> accommodate such use cases. This change improves flexibility and
> ensures that systems can be tailored to meet the requirements of
> specific hardware, such as the Hailo 8 PCIe AI card, without requiring
> kernel source modifications.
> 
> I hope this example clarifies the need for this change. Please let me
> know if further details or additional examples are required.
> 
> Best Regards,
> Li Hua Qian
>>
>>
>>> Li Hua Qian (1):
>>>     swiotlb: Make IO_TLB_SEGSIZE configurable
>>>
>>>    include/linux/swiotlb.h | 2 +-
>>>    kernel/dma/Kconfig      | 7 +++++++
>>>    2 files changed, 8 insertions(+), 1 deletion(-)
>>>
>> Best regards
> 

Re: [PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[Li, Hua Qian]

On Thu, 2025-04-24 at 13:58 +0100, Robin Murphy wrote:
> On 24/04/2025 6:12 am, Li, Hua Qian wrote:
> > On Tue, 2025-04-22 at 15:36 +0200, Marek Szyprowski wrote:
> > > On 22.04.2025 08:37, huaqian.li@siemens.com wrote:
> > > > From: Li Hua Qian <huaqian.li@siemens.com>
> > > >
> > > > This patchset introduces a change to make the IO_TLB_SEGSIZE
> > > > parameter
> > > > configurable via a new kernel configuration option
> > > > (CONFIG_SWIOTLB_SEGSIZE).
> > > >
> > > > In certain applications, the default value of IO_TLB_SEGSIZE
> > > > (128)
> > > > may
> > > > not be sufficient for memory allocation, leading to runtime
> > > > errors.
> > > > By
> > > > making this parameter configurable, users can adjust the
> > > > segment
> > > > size to
> > > > better suit their specific use cases, improving flexibility and
> > > > system
> > > > stability.
> > >
> > > Could You elaborate a bit more what are those certain
> > > applications
> > > that
> > > require increasing IO_TLB_SEGSIZE? I'm not against it, but such
> > > change
> > > should be well justified and described, while the above cover-
> > > letter
> > > doesn't provide anything more than is written in the patch
> > > description.
> > Thank you for your feedback, Marek.
> >
> > To provide more context, one specific application that requires
> > increasing IO_TLB_SEGSIZE is the Hailo 8 PCIe AI card. This card
> > uses
> > dma_alloc_coherent to allocate descriptor lists, as seen in the
> > Hailo
> > driver implementation here:
> > https://github.com/hailo-ai/hailort-drivers/blob/7161f9ee5918029bd4497f590003c2f87ec32507/linux/vdma/memory.c#L322
> > The maximum size (nslots) for these allocations can reach 160,
> > which
> > exceeds the current default value of IO_TLB_SEGSIZE (128).
> >
> > Since IO_TLB_SEGSIZE is defined as a constant in the kernel:
> >
> > `#define IO_TLB_SEGSIZE 128`
> >
> >
> > this limitation causes swiotlb_search_pool_area,
> > https://github.com/torvalds/linux/blame/v6.15-rc2/kernel/dma/swiotlb.c#L1085
> > ,
> > (or swiotlb_do_find_slots in older kernels) to fail when attempting
> > to
> > allocate contiguous physical memory (CMA). This results in runtime
> > errors and prevents the Hailo 8 card from functioning correctly in
> > certain configurations.
>
> Hmm, dma_alloc_coherent() should really not be trying to allocate
> from
> SWIOTLB in the first place - how is that happening?
>
> If you're using restricted DMA for a device which wants significant
> coherent allocations, then it wants to have it's own shared-dma-pool
> for
> those *as well* as the restricted-dma-pool for bouncing streaming
> DMA.
>
> Thanks,
> Robin.

Hi Robin,

Regarding the specific Hailo Card case, the issue arises due
to the capabilities of certain SoCs or CPUs. For example, many
K3 SoCs lack an IOMMU, which is typically used to isolate the
system against DMA-based attacks of external PCI devices.

Taking the TI AM65 as an example, it doesn't have an IOMMU, but
instead includes a Peripheral Virtualization Unit (PVU). The
PVU provides functionality similar to an IOMMU and is used to
isolate PCI devices from the Linux host, and the SWIOTLB is
used to manp all DMA buffers from a static memory carve-out.

You can find more details and background information here:
https://lore.kernel.org/all/20250422061406.112539-1-huaqian.li@siemens.com/

>
> > By making IO_TLB_SEGSIZE configurable via a kernel configuration
> > option
> > (CONFIG_SWIOTLB_SEGSIZE), users can adjust the segment size to
> > accommodate such use cases. This change improves flexibility and
> > ensures that systems can be tailored to meet the requirements of
> > specific hardware, such as the Hailo 8 PCIe AI card, without
> > requiring
> > kernel source modifications.
> >
> > I hope this example clarifies the need for this change. Please let
> > me
> > know if further details or additional examples are required.
> >
> > Best Regards,
> > Li Hua Qian
> > >
> > >
> > > > Li Hua Qian (1):
> > > >     swiotlb: Make IO_TLB_SEGSIZE configurable
> > > >
> > > >    include/linux/swiotlb.h | 2 +-
> > > >    kernel/dma/Kconfig      | 7 +++++++
> > > >    2 files changed, 8 insertions(+), 1 deletion(-)
> > > >
> > > Best regards
> >

--
Hua Qian Li
Siemens AG
http://www.siemens.com/

Re: [PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[Robin Murphy]

On 2025-04-25 6:32 am, Li, Hua Qian wrote:
> On Thu, 2025-04-24 at 13:58 +0100, Robin Murphy wrote:
>> On 24/04/2025 6:12 am, Li, Hua Qian wrote:
>>> On Tue, 2025-04-22 at 15:36 +0200, Marek Szyprowski wrote:
>>>> On 22.04.2025 08:37, huaqian.li@siemens.com wrote:
>>>>> From: Li Hua Qian <huaqian.li@siemens.com>
>>>>>
>>>>> This patchset introduces a change to make the IO_TLB_SEGSIZE
>>>>> parameter
>>>>> configurable via a new kernel configuration option
>>>>> (CONFIG_SWIOTLB_SEGSIZE).
>>>>>
>>>>> In certain applications, the default value of IO_TLB_SEGSIZE
>>>>> (128)
>>>>> may
>>>>> not be sufficient for memory allocation, leading to runtime
>>>>> errors.
>>>>> By
>>>>> making this parameter configurable, users can adjust the
>>>>> segment
>>>>> size to
>>>>> better suit their specific use cases, improving flexibility and
>>>>> system
>>>>> stability.
>>>>
>>>> Could You elaborate a bit more what are those certain
>>>> applications
>>>> that
>>>> require increasing IO_TLB_SEGSIZE? I'm not against it, but such
>>>> change
>>>> should be well justified and described, while the above cover-
>>>> letter
>>>> doesn't provide anything more than is written in the patch
>>>> description.
>>> Thank you for your feedback, Marek.
>>>
>>> To provide more context, one specific application that requires
>>> increasing IO_TLB_SEGSIZE is the Hailo 8 PCIe AI card. This card
>>> uses
>>> dma_alloc_coherent to allocate descriptor lists, as seen in the
>>> Hailo
>>> driver implementation here:
>>> https://github.com/hailo-ai/hailort-drivers/blob/7161f9ee5918029bd4497f590003c2f87ec32507/linux/vdma/memory.c#L322
>>> The maximum size (nslots) for these allocations can reach 160,
>>> which
>>> exceeds the current default value of IO_TLB_SEGSIZE (128).
>>>
>>> Since IO_TLB_SEGSIZE is defined as a constant in the kernel:
>>>
>>> `#define IO_TLB_SEGSIZE 128`
>>>
>>>
>>> this limitation causes swiotlb_search_pool_area,
>>> https://github.com/torvalds/linux/blame/v6.15-rc2/kernel/dma/swiotlb.c#L1085
>>> ,
>>> (or swiotlb_do_find_slots in older kernels) to fail when attempting
>>> to
>>> allocate contiguous physical memory (CMA). This results in runtime
>>> errors and prevents the Hailo 8 card from functioning correctly in
>>> certain configurations.
>>
>> Hmm, dma_alloc_coherent() should really not be trying to allocate
>> from
>> SWIOTLB in the first place - how is that happening?
>>
>> If you're using restricted DMA for a device which wants significant
>> coherent allocations, then it wants to have it's own shared-dma-pool
>> for
>> those *as well* as the restricted-dma-pool for bouncing streaming
>> DMA.
>>
>> Thanks,
>> Robin.
> 
> Hi Robin,
> 
> Regarding the specific Hailo Card case, the issue arises due
> to the capabilities of certain SoCs or CPUs. For example, many
> K3 SoCs lack an IOMMU, which is typically used to isolate the
> system against DMA-based attacks of external PCI devices.
> 
> Taking the TI AM65 as an example, it doesn't have an IOMMU, but
> instead includes a Peripheral Virtualization Unit (PVU). The
> PVU provides functionality similar to an IOMMU and is used to
> isolate PCI devices from the Linux host, and the SWIOTLB is
> used to manp all DMA buffers from a static memory carve-out.

And as I said, if you want to support general coherent allocations then 
you should use part of that carveout for a regular coherent DMA pool. 
The restricted pool is only intended for streaming DMA - swiotlb_alloc() 
is only meant as a convenience fallback for the kind of devices which 
mostly do streaming DMA but make one or two small coherent allocations 
from a suitable context. It does not work for *all* valid usage of 
dma_alloc_attrs(), and if you want to do this for arbitrary PCI devices 
then you almost certainly *do* need to be able to support drivers which 
make allocations in atomic context.

Thanks,
Robin.

> You can find more details and background information here:
> https://lore.kernel.org/all/20250422061406.112539-1-huaqian.li@siemens.com/
> 
>>
>>> By making IO_TLB_SEGSIZE configurable via a kernel configuration
>>> option
>>> (CONFIG_SWIOTLB_SEGSIZE), users can adjust the segment size to
>>> accommodate such use cases. This change improves flexibility and
>>> ensures that systems can be tailored to meet the requirements of
>>> specific hardware, such as the Hailo 8 PCIe AI card, without
>>> requiring
>>> kernel source modifications.
>>>
>>> I hope this example clarifies the need for this change. Please let
>>> me
>>> know if further details or additional examples are required.
>>>
>>> Best Regards,
>>> Li Hua Qian
>>>>
>>>>
>>>>> Li Hua Qian (1):
>>>>>      swiotlb: Make IO_TLB_SEGSIZE configurable
>>>>>
>>>>>     include/linux/swiotlb.h | 2 +-
>>>>>     kernel/dma/Kconfig      | 7 +++++++
>>>>>     2 files changed, 8 insertions(+), 1 deletion(-)
>>>>>
>>>> Best regards
>>>
> 
> --
> Hua Qian Li
> Siemens AG
> http://www.siemens.com/

Re: [PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[Li, Hua Qian]

On Fri, 2025-04-25 at 11:42 +0100, Robin Murphy wrote:
> On 2025-04-25 6:32 am, Li, Hua Qian wrote:
> > On Thu, 2025-04-24 at 13:58 +0100, Robin Murphy wrote:
> > > On 24/04/2025 6:12 am, Li, Hua Qian wrote:
> > > > On Tue, 2025-04-22 at 15:36 +0200, Marek Szyprowski wrote:
> > > > > On 22.04.2025 08:37, huaqian.li@siemens.com wrote:
> > > > > > From: Li Hua Qian <huaqian.li@siemens.com>
> > > > > >
> > > > > > This patchset introduces a change to make the
> > > > > > IO_TLB_SEGSIZE
> > > > > > parameter
> > > > > > configurable via a new kernel configuration option
> > > > > > (CONFIG_SWIOTLB_SEGSIZE).
> > > > > >
> > > > > > In certain applications, the default value of
> > > > > > IO_TLB_SEGSIZE
> > > > > > (128)
> > > > > > may
> > > > > > not be sufficient for memory allocation, leading to runtime
> > > > > > errors.
> > > > > > By
> > > > > > making this parameter configurable, users can adjust the
> > > > > > segment
> > > > > > size to
> > > > > > better suit their specific use cases, improving flexibility
> > > > > > and
> > > > > > system
> > > > > > stability.
> > > > >
> > > > > Could You elaborate a bit more what are those certain
> > > > > applications
> > > > > that
> > > > > require increasing IO_TLB_SEGSIZE? I'm not against it, but
> > > > > such
> > > > > change
> > > > > should be well justified and described, while the above
> > > > > cover-
> > > > > letter
> > > > > doesn't provide anything more than is written in the patch
> > > > > description.
> > > > Thank you for your feedback, Marek.
> > > >
> > > > To provide more context, one specific application that requires
> > > > increasing IO_TLB_SEGSIZE is the Hailo 8 PCIe AI card. This
> > > > card
> > > > uses
> > > > dma_alloc_coherent to allocate descriptor lists, as seen in the
> > > > Hailo
> > > > driver implementation here:
> > > > https://github.com/hailo-ai/hailort-drivers/blob/7161f9ee5918029bd4497f590003c2f87ec32507/linux/vdma/memory.c#L322
> > > > The maximum size (nslots) for these allocations can reach 160,
> > > > which
> > > > exceeds the current default value of IO_TLB_SEGSIZE (128).
> > > >
> > > > Since IO_TLB_SEGSIZE is defined as a constant in the kernel:
> > > >
> > > > `#define IO_TLB_SEGSIZE 128`
> > > >
> > > >
> > > > this limitation causes swiotlb_search_pool_area,
> > > > https://github.com/torvalds/linux/blame/v6.15-rc2/kernel/dma/swiotlb.c#L1085
> > > > ,
> > > > (or swiotlb_do_find_slots in older kernels) to fail when
> > > > attempting
> > > > to
> > > > allocate contiguous physical memory (CMA). This results in
> > > > runtime
> > > > errors and prevents the Hailo 8 card from functioning correctly
> > > > in
> > > > certain configurations.
> > >
> > > Hmm, dma_alloc_coherent() should really not be trying to allocate
> > > from
> > > SWIOTLB in the first place - how is that happening?
> > >
> > > If you're using restricted DMA for a device which wants
> > > significant
> > > coherent allocations, then it wants to have it's own shared-dma-
> > > pool
> > > for
> > > those *as well* as the restricted-dma-pool for bouncing streaming
> > > DMA.
> > >
> > > Thanks,
> > > Robin.
> >
> > Hi Robin,
> >
> > Regarding the specific Hailo Card case, the issue arises due
> > to the capabilities of certain SoCs or CPUs. For example, many
> > K3 SoCs lack an IOMMU, which is typically used to isolate the
> > system against DMA-based attacks of external PCI devices.
> >
> > Taking the TI AM65 as an example, it doesn't have an IOMMU, but
> > instead includes a Peripheral Virtualization Unit (PVU). The
> > PVU provides functionality similar to an IOMMU and is used to
> > isolate PCI devices from the Linux host, and the SWIOTLB is
> > used to manp all DMA buffers from a static memory carve-out.
>
> And as I said, if you want to support general coherent allocations
> then
> you should use part of that carveout for a regular coherent DMA pool.
> The restricted pool is only intended for streaming DMA -
> swiotlb_alloc()
> is only meant as a convenience fallback for the kind of devices which
> mostly do streaming DMA but make one or two small coherent
> allocations
> from a suitable context. It does not work for *all* valid usage of
> dma_alloc_attrs(), and if you want to do this for arbitrary PCI
> devices
> then you almost certainly *do* need to be able to support drivers
> which
> make allocations in atomic context.
>
> Thanks,
> Robin.
Thank you for your detailed explanation, Robin. I understand your
points and agree with your suggestions. Based on this, I will conclude
this upstreaming patchset here. Thank you again for your guiidance and
support.

Best Regards,
Li Hua Qian
>
> > You can find more details and background information here:
> > https://lore.kernel.org/all/20250422061406.112539-1-huaqian.li@siemens.com/
> >
> > >
> > > > By making IO_TLB_SEGSIZE configurable via a kernel
> > > > configuration
> > > > option
> > > > (CONFIG_SWIOTLB_SEGSIZE), users can adjust the segment size to
> > > > accommodate such use cases. This change improves flexibility
> > > > and
> > > > ensures that systems can be tailored to meet the requirements
> > > > of
> > > > specific hardware, such as the Hailo 8 PCIe AI card, without
> > > > requiring
> > > > kernel source modifications.
> > > >
> > > > I hope this example clarifies the need for this change. Please
> > > > let
> > > > me
> > > > know if further details or additional examples are required.
> > > >
> > > > Best Regards,
> > > > Li Hua Qian
> > > > >
> > > > >
> > > > > > Li Hua Qian (1):
> > > > > >      swiotlb: Make IO_TLB_SEGSIZE configurable
> > > > > >
> > > > > >     include/linux/swiotlb.h | 2 +-
> > > > > >     kernel/dma/Kconfig      | 7 +++++++
> > > > > >     2 files changed, 8 insertions(+), 1 deletion(-)
> > > > > >
> > > > > Best regards
> > > >
> >
> > --
> > Hua Qian Li
> > Siemens AG
> > http://www.siemens.com/
>

--
Hua Qian Li
Siemens AG
http://www.siemens.com/

Re: [PATCH 0/1] swiotlb: Make IO_TLB_SEGSIZE Configurable

[Konrad Rzeszutek Wilk]

On Fri, Apr 25, 2025 at 11:42:24AM +0100, Robin Murphy wrote:
> On 2025-04-25 6:32 am, Li, Hua Qian wrote:
> > On Thu, 2025-04-24 at 13:58 +0100, Robin Murphy wrote:
> > > On 24/04/2025 6:12 am, Li, Hua Qian wrote:
> > > > On Tue, 2025-04-22 at 15:36 +0200, Marek Szyprowski wrote:
> > > > > On 22.04.2025 08:37, huaqian.li@siemens.com wrote:
> > > > > > From: Li Hua Qian <huaqian.li@siemens.com>
> > > > > > 
> > > > > > This patchset introduces a change to make the IO_TLB_SEGSIZE
> > > > > > parameter
> > > > > > configurable via a new kernel configuration option
> > > > > > (CONFIG_SWIOTLB_SEGSIZE).
> > > > > > 
> > > > > > In certain applications, the default value of IO_TLB_SEGSIZE
> > > > > > (128)
> > > > > > may
> > > > > > not be sufficient for memory allocation, leading to runtime
> > > > > > errors.
> > > > > > By
> > > > > > making this parameter configurable, users can adjust the
> > > > > > segment
> > > > > > size to
> > > > > > better suit their specific use cases, improving flexibility and
> > > > > > system
> > > > > > stability.
> > > > > 
> > > > > Could You elaborate a bit more what are those certain
> > > > > applications
> > > > > that
> > > > > require increasing IO_TLB_SEGSIZE? I'm not against it, but such
> > > > > change
> > > > > should be well justified and described, while the above cover-
> > > > > letter
> > > > > doesn't provide anything more than is written in the patch
> > > > > description.
> > > > Thank you for your feedback, Marek.
> > > > 
> > > > To provide more context, one specific application that requires
> > > > increasing IO_TLB_SEGSIZE is the Hailo 8 PCIe AI card. This card
> > > > uses
> > > > dma_alloc_coherent to allocate descriptor lists, as seen in the
> > > > Hailo
> > > > driver implementation here:
> > > > https://github.com/hailo-ai/hailort-drivers/blob/7161f9ee5918029bd4497f590003c2f87ec32507/linux/vdma/memory.c#L322
> > > > The maximum size (nslots) for these allocations can reach 160,
> > > > which
> > > > exceeds the current default value of IO_TLB_SEGSIZE (128).
> > > > 
> > > > Since IO_TLB_SEGSIZE is defined as a constant in the kernel:
> > > > 
> > > > `#define IO_TLB_SEGSIZE 128`
> > > > 
> > > > 
> > > > this limitation causes swiotlb_search_pool_area,
> > > > https://github.com/torvalds/linux/blame/v6.15-rc2/kernel/dma/swiotlb.c#L1085
> > > > ,
> > > > (or swiotlb_do_find_slots in older kernels) to fail when attempting
> > > > to
> > > > allocate contiguous physical memory (CMA). This results in runtime
> > > > errors and prevents the Hailo 8 card from functioning correctly in
> > > > certain configurations.
> > > 
> > > Hmm, dma_alloc_coherent() should really not be trying to allocate
> > > from
> > > SWIOTLB in the first place - how is that happening?
> > > 
> > > If you're using restricted DMA for a device which wants significant
> > > coherent allocations, then it wants to have it's own shared-dma-pool
> > > for
> > > those *as well* as the restricted-dma-pool for bouncing streaming
> > > DMA.
> > > 
> > > Thanks,
> > > Robin.
> > 
> > Hi Robin,
> > 
> > Regarding the specific Hailo Card case, the issue arises due
> > to the capabilities of certain SoCs or CPUs. For example, many
> > K3 SoCs lack an IOMMU, which is typically used to isolate the
> > system against DMA-based attacks of external PCI devices.
> > 
> > Taking the TI AM65 as an example, it doesn't have an IOMMU, but
> > instead includes a Peripheral Virtualization Unit (PVU). The
> > PVU provides functionality similar to an IOMMU and is used to
> > isolate PCI devices from the Linux host, and the SWIOTLB is
> > used to manp all DMA buffers from a static memory carve-out.
> 
> And as I said, if you want to support general coherent allocations then you
> should use part of that carveout for a regular coherent DMA pool. The
> restricted pool is only intended for streaming DMA - swiotlb_alloc() is only
> meant as a convenience fallback for the kind of devices which mostly do
> streaming DMA but make one or two small coherent allocations from a suitable
> context. It does not work for *all* valid usage of dma_alloc_attrs(), and if
> you want to do this for arbitrary PCI devices then you almost certainly *do*
> need to be able to support drivers which make allocations in atomic context.

If they utilize dma_alloc_coherent and setup at boot-time those buffers
then that does exactly the same thing as the coherent DMA pool. Albeit
less flexible, but nonethless the same thing.

That seems like a valid use-case, no?