Why is mmap()ed reserved memory so slow?

Why is mmap()ed reserved memory so slow?

[David Chau]

Hi all,

I'm working on a driver for the Broadcom 1250, and I am using reserved 
memory for some data buffers. The board comes with 256 MB of RAM, so I 
boot Linux with "mem=253M" to reserve some RAM at the top of memory, and 
then mmap() /dev/mem starting at 253 MB.

The problem is that accessing this memory is ridiculously slow. A simple 
benchmark revealed that it takes about 200 cycles to read a 64-bit 
number. If I mmap() /dev/zero instead, a read takes under 3 cycles.

For those of you who knows how the Linux VM works, could you tell me why 
the memory access is so slow? It look like it might be invoking the 
page-fault handler on every read. How can I make memory access faster?

Thanks,
David

Re: Why is mmap()ed reserved memory so slow?

[Dan Malek]

On Jul 15, 2005, at 6:21 PM, David Chau wrote:

> For those of you who knows how the Linux VM works, could you tell me 
> why the memory access is so slow? It look like it might be invoking 
> the page-fault handler on every read. How can I make memory access 
> faster?

How about a little more info, like what kernel are you using and what 
are
the parameters you are sending to mmap()?

One of the things that happens here is /dev/mem is thinking this memory 
is
not real memory (because you said the system has only 253M of real
memory), so it treats it like IO space.  This causes changes to the 
attributes
of the pages, most notably the CCA type for cache or pipeline behavior,
which isn't what you want in this case.

The better way to approach this is to place an mmap() function in the
associated driver that works in conjunction with the application to gain
shared access as you expect.  This also closes a hole where an errant
application could write into unexpected places through /dev/mem.

Thanks.

	-- Dan

Re: Why is mmap()ed reserved memory so slow?

[Ralf Baechle]

On Fri, Jul 15, 2005 at 06:21:44PM -0400, David Chau wrote:

> I'm working on a driver for the Broadcom 1250, and I am using reserved 
> memory for some data buffers. The board comes with 256 MB of RAM, so I 
> boot Linux with "mem=253M" to reserve some RAM at the top of memory, and 
> then mmap() /dev/mem starting at 253 MB.
> 
> The problem is that accessing this memory is ridiculously slow. A simple 
> benchmark revealed that it takes about 200 cycles to read a 64-bit 
> number.

mmap will create uncached mappings for anything above the highest RAM
address.

> If I mmap() /dev/zero instead, a read takes under 3 cycles.

Because you have a cache hits.  No RAM is that fast.

Above 200 cycles really is how horribly slow RAM is compared to a moderatly
clocked system.

> For those of you who knows how the Linux VM works, could you tell me why 
> the memory access is so slow? It look like it might be invoking the 
> page-fault handler on every read. How can I make memory access faster?

  Ralf

Re: Why is mmap()ed reserved memory so slow?

[David Chau]

Dan Malek wrote:

> How about a little more info, like what kernel are you using and what are
> the parameters you are sending to mmap()?

Linux (none) 2.4.31 #412 SMP Fri Jul 15 16:26:05 EDT 2005 mips unknown
(unmodified kernel from linux-mips.org).
It's running on the SB1 on a Broadcom 1250 board.

I mmap() with:
int mem_fd = open("/dev/mem", O_RDWR);
void* mem_base =
    mmap(NULL, DRIVER_MEM_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED,
         mem_fd, DRIVER_MEM_PHYS_BASE);
Where driver_mem_phys_base = 253M, and driver_mem_size=1M.

> The better way to approach this is to place an mmap() function in the
> associated driver that works in conjunction with the application to gain
> shared access as you expect.  This also closes a hole where an errant
> application could write into unexpected places through /dev/mem.


Could you point me to an example of this so I can figure out how to do it?

Thanks,
David