Mapping huge user buffers for DMA

Mapping huge user buffers for DMA

[Stephen Williams]

I have a PPC405GPr system with an image processing device, that
is creating potentially huge amounts of data. In one setup I
have a 256Meg system, and I'm trying to map a 192Meg destination
buffer using map_user_kiovec and an array of kiobufs.

I'm finding, however, that I'm getting an Oops in map_user_kiovec
when it tries this, and I'm wondering where I need to look for
any limits I might be overrunning.

Also, I've been considering skipping kiobufs all together and
instead using code like this (lifted from map_user_kiobuf)

	/* Try to fault in all of the necessary pages */
	down_read(&mm->mmap_sem);
	/* rw==READ means read from disk, write into memory area */
	err = get_user_pages(current, mm, va, pgcount,
			(rw==READ), 0, iobuf->maplist, NULL);
	up_read(&mm->mmap_sem);

to get the user pages directly. This is really what I want, and
I do not need the other functionality of kiobufs. Is the
get_user_pages function kosher for use by drivers? Is there
a limit to what get_user_pages may map?


-- 
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."

Re: Mapping huge user buffers for DMA

[Roger Larsson]

On Tuesday 30 August 2005 19.27, Stephen Williams wrote:
> I have a PPC405GPr system with an image processing device, that
> is creating potentially huge amounts of data. In one setup I
> have a 256Meg system, and I'm trying to map a 192Meg destination
> buffer using map_user_kiovec and an array of kiobufs.

Never tried to map that amount at once. Our code looks, or looked, like this.
Found some code...
/usr/src/linuxppc_2_4_devel/arch/cris/drivers/examples/kiobuftest.c 

>
> I'm finding, however, that I'm getting an Oops in map_user_kiovec
> when it tries this, and I'm wondering where I need to look for
> any limits I might be overrunning.

What does the Oops say??
Hmm... How much space will the vector itself require?

>
> Also, I've been considering skipping kiobufs all together and
> instead using code like this (lifted from map_user_kiobuf)
>
> 	/* Try to fault in all of the necessary pages */
> 	down_read(&mm->mmap_sem);
> 	/* rw==READ means read from disk, write into memory area */
> 	err = get_user_pages(current, mm, va, pgcount,
> 			(rw==READ), 0, iobuf->maplist, NULL);
> 	up_read(&mm->mmap_sem);
>
> to get the user pages directly. This is really what I want, and
> I do not need the other functionality of kiobufs. Is the
> get_user_pages function kosher for use by drivers? Is there
> a limit to what get_user_pages may map?

Isn't this the way it is done in 2.6 anyway?

/RogerL

Re: Mapping huge user buffers for DMA

[Stephen Williams]

Roger Larsson wrote:
> On Tuesday 30 August 2005 19.27, Stephen Williams wrote:
> 
>>I have a PPC405GPr system with an image processing device, that
>>is creating potentially huge amounts of data. In one setup I
>>have a 256Meg system, and I'm trying to map a 192Meg destination
>>buffer using map_user_kiovec and an array of kiobufs.
> 
> 
> Never tried to map that amount at once. Our code looks, or looked, like this.
> Found some code...
> /usr/src/linuxppc_2_4_devel/arch/cris/drivers/examples/kiobuftest.c 

That's similar to where I'm at now, and I'm finding it inadequate.

>>I'm finding, however, that I'm getting an Oops in map_user_kiovec
>>when it tries this, and I'm wondering where I need to look for
>>any limits I might be overrunning.
> 
> 
> What does the Oops say??
> Hmm... How much space will the vector itself require?

If placed into a single kiobuf, it would need 49,152 page pointers,
which would be 192K bytes for the maplist. Since the kernel uses
kmalloc to allocate the maplist, that won't work. Thus, I've
created an array of kiobufs to hold a smaller chunk of the pages
for the user buffer.

The Oops is a sig11 within the map_user_kiovec. (?!)

I also see errors from the kernel being unable to allocate
pages, so it seems like something is not handling an OOM
situation very well.

> 
>>Also, I've been considering skipping kiobufs all together and
>>instead using code like this (lifted from map_user_kiobuf)
>>
>>	/* Try to fault in all of the necessary pages */
>>	down_read(&mm->mmap_sem);
>>	/* rw==READ means read from disk, write into memory area */
>>	err = get_user_pages(current, mm, va, pgcount,
>>			(rw==READ), 0, iobuf->maplist, NULL);
>>	up_read(&mm->mmap_sem);
>>
>>to get the user pages directly. This is really what I want, and
>>I do not need the other functionality of kiobufs. Is the
>>get_user_pages function kosher for use by drivers? Is there
>>a limit to what get_user_pages may map?
> 
> 
> Isn't this the way it is done in 2.6 anyway?

Well that's how the map_user_kiovec in 2.4 does it. I'm just
wondering if I were to do it myself using get_user_pages, would
I be peeking around the curtain at interfaces not meant to be
public. I'm thinking I need to ues get_user_pages myself to
avoid the extra overhead of kiobufs, and I can use vmalloc'ed
memory to hold those page pointers for later unmap so I don't
run up against the kmalloc limit.

-- 
Steve Williams                "The woods are lovely, dark and deep.
steve at icarus.com           But I have promises to keep,
http://www.icarus.com         and lines to code before I sleep,
http://www.picturel.com       And lines to code before I sleep."

Re: Mapping huge user buffers for DMA

[Clemens Koller]

Hello, Stephen!

Please have a look at the whole thread around my posts at:
http://lkml.org/lkml/2005/8/4/201

I am working on an mpc8540 (e500) ppc and we are pushing
data from the local bus directly to the userspace
mlock()ed memory with the dma engine, using scatter/gather
(chaining and page/chunk-defragmentation/compression) transfers.

Currently, I send the application's user virtual and mlock()ed
address to the kernel driver through an ioctl and do an iopa()
at the user virtual addresses (pages) to get to the physical
addresses (pages). The system works stable and we can fill up
the whole memory (192MByte) with currently 144MByte/s
(still improvements possible).

If you are interested in our design, feel free to contact
me directly.

Greets,

Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19


Stephen Williams wrote:
> 
> I have a PPC405GPr system with an image processing device, that
> is creating potentially huge amounts of data. In one setup I
> have a 256Meg system, and I'm trying to map a 192Meg destination
> buffer using map_user_kiovec and an array of kiobufs.
> 
> I'm finding, however, that I'm getting an Oops in map_user_kiovec
> when it tries this, and I'm wondering where I need to look for
> any limits I might be overrunning.
> 
> Also, I've been considering skipping kiobufs all together and
> instead using code like this (lifted from map_user_kiobuf)
> 
>     /* Try to fault in all of the necessary pages */
>     down_read(&mm->mmap_sem);
>     /* rw==READ means read from disk, write into memory area */
>     err = get_user_pages(current, mm, va, pgcount,
>             (rw==READ), 0, iobuf->maplist, NULL);
>     up_read(&mm->mmap_sem);
> 
> to get the user pages directly. This is really what I want, and
> I do not need the other functionality of kiobufs. Is the
> get_user_pages function kosher for use by drivers? Is there
> a limit to what get_user_pages may map?
> 
>