Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Hiroshi Doyu]

Hi Inki/Marek,

On Tue, 16 Oct 2012 02:50:16 +0200
Inki Dae <inki.dae@samsung.com> wrote:

> 2012/10/15 Marek Szyprowski <m.szyprowski@samsung.com>:
> > Hello,
> >
> > Some devices, which have IOMMU, for some use cases might require to
> > allocate a buffers for DMA which is contiguous in physical memory. Such
> > use cases appears for example in DRM subsystem when one wants to improve
> > performance or use secure buffer protection.
> >
> > I would like to ask if adding a new attribute, as proposed in this RFC
> > is a good idea? I feel that it might be an attribute just for a single
> > driver, but I would like to know your opinion. Should we look for other
> > solution?
> >
> 
> In addition, currently we have worked dma-mapping-based iommu support
> for exynos drm driver with this patch set so this patch set has been
> tested with iommu enabled exynos drm driver and worked fine. actually,
> this feature is needed for secure mode such as TrustZone. in case of
> Exynos SoC, memory region for secure mode should be physically
> contiguous and also maybe OMAP but now dma-mapping framework doesn't
> guarantee physically continuous memory allocation so this patch set
> would make it possible.

Agree that the contigous memory allocation is necessary for us too.

In addition to those contiguous/discontiguous page allocation, is
there any way to _import_ anonymous pages allocated by a process to be
used in dma-mapping API later?

I'm considering the following scenario, an user process allocates a
buffer by malloc() in advance, and then it asks some driver to convert
that buffer into IOMMU'able/DMA'able ones later. In this case, pages
are discouguous and even they may not be yet allocated at
malloc()/mmap().

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Russell King - ARM Linux]

On Tue, Oct 16, 2012 at 09:04:34AM +0300, Hiroshi Doyu wrote:
> In addition to those contiguous/discontiguous page allocation, is
> there any way to _import_ anonymous pages allocated by a process to be
> used in dma-mapping API later?
> 
> I'm considering the following scenario, an user process allocates a
> buffer by malloc() in advance, and then it asks some driver to convert
> that buffer into IOMMU'able/DMA'able ones later. In this case, pages
> are discouguous and even they may not be yet allocated at
> malloc()/mmap().

That situation is covered.  It's the streaming API you're wanting for that.
dma_map_sg() - but you may need additional cache handling via
flush_dcache_page() to ensure that your code is safe for all CPU cache
architectures.

Remember that pages allocated into userspace will be cacheable, so a cache
flush is required before they can be DMA'd.  Hence the streaming API.

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Catalin Marinas]

On 16 October 2012 09:59, Russell King - ARM Linux
<linux-lFZ/pmaqli7XmaaqVzeoHQ@public.gmane.org> wrote:
> On Tue, Oct 16, 2012 at 09:04:34AM +0300, Hiroshi Doyu wrote:
>> In addition to those contiguous/discontiguous page allocation, is
>> there any way to _import_ anonymous pages allocated by a process to be
>> used in dma-mapping API later?
>>
>> I'm considering the following scenario, an user process allocates a
>> buffer by malloc() in advance, and then it asks some driver to convert
>> that buffer into IOMMU'able/DMA'able ones later. In this case, pages
>> are discouguous and even they may not be yet allocated at
>> malloc()/mmap().
>
> That situation is covered.  It's the streaming API you're wanting for that.
> dma_map_sg() - but you may need additional cache handling via
> flush_dcache_page() to ensure that your code is safe for all CPU cache
> architectures.

For user-allocated pages you first need get_user_pages() to make sure
they are in memory (and will stay there). This function also calls
flush_dcache_page(). Then you can build the sg list for dma_map_sg().

-- 
Catalin

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Hiroshi Doyu]

Hi Russell,

Russell King - ARM Linux <linux@arm.linux.org.uk> wrote @ Tue, 16 Oct 2012 10:59:28 +0200:

> On Tue, Oct 16, 2012 at 09:04:34AM +0300, Hiroshi Doyu wrote:
> > In addition to those contiguous/discontiguous page allocation, is
> > there any way to _import_ anonymous pages allocated by a process to be
> > used in dma-mapping API later?
> > 
> > I'm considering the following scenario, an user process allocates a
> > buffer by malloc() in advance, and then it asks some driver to convert
> > that buffer into IOMMU'able/DMA'able ones later. In this case, pages
> > are discouguous and even they may not be yet allocated at
> > malloc()/mmap().
> 
> That situation is covered.  It's the streaming API you're wanting for that.
> dma_map_sg() - but you may need additional cache handling via
> flush_dcache_page() to ensure that your code is safe for all CPU cache
> architectures.
>
> Remember that pages allocated into userspace will be cacheable, so a cache
> flush is required before they can be DMA'd.  Hence the streaming
> API.

Is the syscall "cacheflush()" supposed to be the knob for that?

Or is there any other ones to have more precise control, "clean",
"invalidate" and "flush", from userland in generic way?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Russell King - ARM Linux]

On Tue, Oct 16, 2012 at 12:27:55PM +0200, Hiroshi Doyu wrote:
> Hi Russell,
> 
> Russell King - ARM Linux <linux@arm.linux.org.uk> wrote @ Tue, 16 Oct 2012 10:59:28 +0200:
> 
> > On Tue, Oct 16, 2012 at 09:04:34AM +0300, Hiroshi Doyu wrote:
> > > In addition to those contiguous/discontiguous page allocation, is
> > > there any way to _import_ anonymous pages allocated by a process to be
> > > used in dma-mapping API later?
> > > 
> > > I'm considering the following scenario, an user process allocates a
> > > buffer by malloc() in advance, and then it asks some driver to convert
> > > that buffer into IOMMU'able/DMA'able ones later. In this case, pages
> > > are discouguous and even they may not be yet allocated at
> > > malloc()/mmap().
> > 
> > That situation is covered.  It's the streaming API you're wanting for that.
> > dma_map_sg() - but you may need additional cache handling via
> > flush_dcache_page() to ensure that your code is safe for all CPU cache
> > architectures.
> >
> > Remember that pages allocated into userspace will be cacheable, so a cache
> > flush is required before they can be DMA'd.  Hence the streaming
> > API.
> 
> Is the syscall "cacheflush()" supposed to be the knob for that?
> 
> Or is there any other ones to have more precise control, "clean",
> "invalidate" and "flush", from userland in generic way?

No other syscalls are required - this sequence will do everything you
need to perform DMA on pages mapped into userspace:

	get_user_pages()
	convert array of struct page * to scatterlist
	dma_map_sg()
	perform DMA
	dma_unmap_sg()
	for each page in sg()
		page_cache_release(page);

If you get the list of pages some other way (eg, via shmem_read_mapping_page_gfp)
then additional maintanence may be required (though that may be a bug in
shmem - remember this stuff hasn't been well tested on ARM before.)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Inki Dae]

Hi Hiroshi,

2012/10/16 Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>:
> Hi Inki/Marek,
>
> On Tue, 16 Oct 2012 02:50:16 +0200
> Inki Dae <inki.dae-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org> wrote:
>
>> 2012/10/15 Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>:
>> > Hello,
>> >
>> > Some devices, which have IOMMU, for some use cases might require to
>> > allocate a buffers for DMA which is contiguous in physical memory. Such
>> > use cases appears for example in DRM subsystem when one wants to improve
>> > performance or use secure buffer protection.
>> >
>> > I would like to ask if adding a new attribute, as proposed in this RFC
>> > is a good idea? I feel that it might be an attribute just for a single
>> > driver, but I would like to know your opinion. Should we look for other
>> > solution?
>> >
>>
>> In addition, currently we have worked dma-mapping-based iommu support
>> for exynos drm driver with this patch set so this patch set has been
>> tested with iommu enabled exynos drm driver and worked fine. actually,
>> this feature is needed for secure mode such as TrustZone. in case of
>> Exynos SoC, memory region for secure mode should be physically
>> contiguous and also maybe OMAP but now dma-mapping framework doesn't
>> guarantee physically continuous memory allocation so this patch set
>> would make it possible.
>
> Agree that the contigous memory allocation is necessary for us too.
>
> In addition to those contiguous/discontiguous page allocation, is
> there any way to _import_ anonymous pages allocated by a process to be
> used in dma-mapping API later?
>
> I'm considering the following scenario, an user process allocates a
> buffer by malloc() in advance, and then it asks some driver to convert
> that buffer into IOMMU'able/DMA'able ones later. In this case, pages
> are discouguous and even they may not be yet allocated at
> malloc()/mmap().
>

I'm not sure I understand what you mean but we had already tried this
way and for this, you can refer to below link,
               http://www.mail-archive.com/dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org/msg22555.html

but this way had been pointed out by drm guys because the pages could
be used through gem object after that pages had been freed by free()
anyway their pointing was reasonable and I'm trying another way, this
is the way that the pages to user space has same life time with dma
operation. in other word, if dma completed access to that pages then
also that pages will be freed. actually drm-based via driver of
mainline kernel is using same way


> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo-Bw31MaZKKs0EbZ0PF+XxCw@public.gmane.org  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org"> email-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org </a>

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Russell King - ARM Linux]

On Tue, Oct 16, 2012 at 07:12:49PM +0900, Inki Dae wrote:
> Hi Hiroshi,
> 
> I'm not sure I understand what you mean but we had already tried this
> way and for this, you can refer to below link,
>                http://www.mail-archive.com/dri-devel@lists.freedesktop.org/msg22555.html
> 
> but this way had been pointed out by drm guys because the pages could
> be used through gem object after that pages had been freed by free()
> anyway their pointing was reasonable and I'm trying another way, this
> is the way that the pages to user space has same life time with dma
> operation. in other word, if dma completed access to that pages then
> also that pages will be freed. actually drm-based via driver of
> mainline kernel is using same way

I don't know about Hiroshi, but the above "sentence" - and I mean the 7
line sentence - is very difficult to understand and wears readers out.

If your GPU hardware has a MMU, then the problem of dealing with userspace
pages is very easy.  Do it the same way that the i915 driver and the rest
of DRM does.  Use shmem backed memory.

I'm doing that for the Dove DRM driver and it works a real treat, and as
the pages are backed by page cache pages, you can use all the normal
page refcounting on them to prevent them being freed until your DMA has
completed.  All my X pixmaps are shmem backed drm objects, except for
the scanout buffers which are dumb drm objects (because they must be
contiguous.)

In fact, get_user_pages() will take the reference for you before you pass
them over to dma_map_sg().  On completion of DMA, you just need to use
dma_unmap_sg() and release each page.

If you don't want to use get_user_pages() (which other drivers don't) then
you need to following the i915 example and get each page out of shmem
individually.

(My situation on the Dove hardware is a little different, because the
kernel DRM driver isn't involved with the GPU - it merely provides the
memory for pixmaps.  The GPU software stack, being a chunk of closed
source userspace library with open source kernel driver, means that
things are more complicated; the kernel side GPU driver uses
get_user_pages() to pin them prior to building the GPU's MMU table.)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Inki Dae]

Hi Russell,

2012/10/16 Russell King - ARM Linux <linux-lFZ/pmaqli7XmaaqVzeoHQ@public.gmane.org>:
> On Tue, Oct 16, 2012 at 07:12:49PM +0900, Inki Dae wrote:
>> Hi Hiroshi,
>>
>> I'm not sure I understand what you mean but we had already tried this
>> way and for this, you can refer to below link,
>>                http://www.mail-archive.com/dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org/msg22555.html
>>
>> but this way had been pointed out by drm guys because the pages could
>> be used through gem object after that pages had been freed by free()
>> anyway their pointing was reasonable and I'm trying another way, this
>> is the way that the pages to user space has same life time with dma
>> operation. in other word, if dma completed access to that pages then
>> also that pages will be freed. actually drm-based via driver of
>> mainline kernel is using same way
>
> I don't know about Hiroshi, but the above "sentence" - and I mean the 7
> line sentence - is very difficult to understand and wears readers out.
>

Sorry for this. Please see below comments.

> If your GPU hardware has a MMU, then the problem of dealing with userspace
> pages is very easy.  Do it the same way that the i915 driver and the rest
> of DRM does.  Use shmem backed memory.
>
> I'm doing that for the Dove DRM driver and it works a real treat, and as
> the pages are backed by page cache pages, you can use all the normal
> page refcounting on them to prevent them being freed until your DMA has
> completed.  All my X pixmaps are shmem backed drm objects, except for
> the scanout buffers which are dumb drm objects (because they must be
> contiguous.)
>
> In fact, get_user_pages() will take the reference for you before you pass
> them over to dma_map_sg().  On completion of DMA, you just need to use
> dma_unmap_sg() and release each page.
>

It's exactly same as ours. Besides, I know get_user_pages() takes 2
reference counts if the user process has never accessed user region
allocated by malloc(). Then, if the user calls free(), the page
reference count becomes 1 and becomes 0 with put_page() call. And the
reverse holds as well. This means how the pages backed are used by dma
and freed. dma_map_sg() just does cache operation properly and maps
these pages with iommu table. There may be my missing point.

Thanks,
Inki Dae

> If you don't want to use get_user_pages() (which other drivers don't) then
> you need to following the i915 example and get each page out of shmem
> individually.
>
> (My situation on the Dove hardware is a little different, because the
> kernel DRM driver isn't involved with the GPU - it merely provides the
> memory for pixmaps.  The GPU software stack, being a chunk of closed
> source userspace library with open source kernel driver, means that
> things are more complicated; the kernel side GPU driver uses
> get_user_pages() to pin them prior to building the GPU's MMU table.)
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo-Bw31MaZKKs0EbZ0PF+XxCw@public.gmane.org  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org"> email-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org </a>

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Hiroshi Doyu]

Hi Inki,

Inki Dae <inki.dae@samsung.com> wrote @ Tue, 16 Oct 2012 12:12:49 +0200:

> Hi Hiroshi,
> 
> 2012/10/16 Hiroshi Doyu <hdoyu@nvidia.com>:
> > Hi Inki/Marek,
> >
> > On Tue, 16 Oct 2012 02:50:16 +0200
> > Inki Dae <inki.dae@samsung.com> wrote:
> >
> >> 2012/10/15 Marek Szyprowski <m.szyprowski@samsung.com>:
> >> > Hello,
> >> >
> >> > Some devices, which have IOMMU, for some use cases might require to
> >> > allocate a buffers for DMA which is contiguous in physical memory. Such
> >> > use cases appears for example in DRM subsystem when one wants to improve
> >> > performance or use secure buffer protection.
> >> >
> >> > I would like to ask if adding a new attribute, as proposed in this RFC
> >> > is a good idea? I feel that it might be an attribute just for a single
> >> > driver, but I would like to know your opinion. Should we look for other
> >> > solution?
> >> >
> >>
> >> In addition, currently we have worked dma-mapping-based iommu support
> >> for exynos drm driver with this patch set so this patch set has been
> >> tested with iommu enabled exynos drm driver and worked fine. actually,
> >> this feature is needed for secure mode such as TrustZone. in case of
> >> Exynos SoC, memory region for secure mode should be physically
> >> contiguous and also maybe OMAP but now dma-mapping framework doesn't
> >> guarantee physically continuous memory allocation so this patch set
> >> would make it possible.
> >
> > Agree that the contigous memory allocation is necessary for us too.
> >
> > In addition to those contiguous/discontiguous page allocation, is
> > there any way to _import_ anonymous pages allocated by a process to be
> > used in dma-mapping API later?
> >
> > I'm considering the following scenario, an user process allocates a
> > buffer by malloc() in advance, and then it asks some driver to convert
> > that buffer into IOMMU'able/DMA'able ones later. In this case, pages
> > are discouguous and even they may not be yet allocated at
> > malloc()/mmap().
> >
> 
> I'm not sure I understand what you mean but we had already tried this
> way and for this, you can refer to below link,
>                http://www.mail-archive.com/dri-devel@lists.freedesktop.org/msg22555.html

The above patch doesn't seem to have so much platform/SoC specific
code but rather it could common over other SoC as well. Is there any
plan to make it more generic, which can be used by other DRM drivers?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Linaro-mm-sig] [RFC 0/2] DMA-mapping & IOMMU - physically contiguous allocations

[Inki Dae]

Hi Hiroshi,



2012. 10. 16. 오후 11:13 Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org> 작성:

> Hi Inki,
> 
> Inki Dae <inki.dae-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org> wrote @ Tue, 16 Oct 2012 12:12:49 +0200:
> 
>> Hi Hiroshi,
>> 
>> 2012/10/16 Hiroshi Doyu <hdoyu-DDmLM1+adcrQT0dZR+AlfA@public.gmane.org>:
>>> Hi Inki/Marek,
>>> 
>>> On Tue, 16 Oct 2012 02:50:16 +0200
>>> Inki Dae <inki.dae-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org> wrote:
>>> 
>>>> 2012/10/15 Marek Szyprowski <m.szyprowski-Sze3O3UU22JBDgjK7y7TUQ@public.gmane.org>:
>>>>> Hello,
>>>>> 
>>>>> Some devices, which have IOMMU, for some use cases might require to
>>>>> allocate a buffers for DMA which is contiguous in physical memory. Such
>>>>> use cases appears for example in DRM subsystem when one wants to improve
>>>>> performance or use secure buffer protection.
>>>>> 
>>>>> I would like to ask if adding a new attribute, as proposed in this RFC
>>>>> is a good idea? I feel that it might be an attribute just for a single
>>>>> driver, but I would like to know your opinion. Should we look for other
>>>>> solution?
>>>>> 
>>>> 
>>>> In addition, currently we have worked dma-mapping-based iommu support
>>>> for exynos drm driver with this patch set so this patch set has been
>>>> tested with iommu enabled exynos drm driver and worked fine. actually,
>>>> this feature is needed for secure mode such as TrustZone. in case of
>>>> Exynos SoC, memory region for secure mode should be physically
>>>> contiguous and also maybe OMAP but now dma-mapping framework doesn't
>>>> guarantee physically continuous memory allocation so this patch set
>>>> would make it possible.
>>> 
>>> Agree that the contigous memory allocation is necessary for us too.
>>> 
>>> In addition to those contiguous/discontiguous page allocation, is
>>> there any way to _import_ anonymous pages allocated by a process to be
>>> used in dma-mapping API later?
>>> 
>>> I'm considering the following scenario, an user process allocates a
>>> buffer by malloc() in advance, and then it asks some driver to convert
>>> that buffer into IOMMU'able/DMA'able ones later. In this case, pages
>>> are discouguous and even they may not be yet allocated at
>>> malloc()/mmap().
>>> 
>> 
>> I'm not sure I understand what you mean but we had already tried this
>> way and for this, you can refer to below link,
>>               http://www.mail-archive.com/dri-devel-PD4FTy7X32lNgt0PjOBp9w@public.gmane.orgp.org/msg22555.html
> 
> The above patch doesn't seem to have so much platform/SoC specific
> code but rather it could common over other SoC as well. Is there any
> plan to make it more generic, which can be used by other DRM drivers?
> 

Right, the above patch has no any platform/SoC specific code but doesn't use dma-mapping API . Anyway we should refrain from using such thing because gem object could still be used and shared with other processes even if user process freed user region allocated by malloc()

And our new patch in progress would resolve this issue and this way is similar to drm-based via driver of mainline kernel. And this patch isn't considered for common use and is specific to platform/SoC so much. The pages backed can be used only by 2d gpu's dma.

Thanks,
Inki Dae

> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo-Bw31MaZKKs0EbZ0PF+XxCw@public.gmane.org  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org"> email-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org </a>