Why is mmap()ed reserved memory so slow?

[David Chau <dchau@mazunetworks.com>]

Hi all,

I'm working on a driver for the Broadcom 1250, and I am using reserved 
memory for some data buffers. The board comes with 256 MB of RAM, so I 
boot Linux with "mem=253M" to reserve some RAM at the top of memory, and 
then mmap() /dev/mem starting at 253 MB.

The problem is that accessing this memory is ridiculously slow. A simple 
benchmark revealed that it takes about 200 cycles to read a 64-bit 
number. If I mmap() /dev/zero instead, a read takes under 3 cycles.

For those of you who knows how the Linux VM works, could you tell me why 
the memory access is so slow? It look like it might be invoking the 
page-fault handler on every read. How can I make memory access faster?

Thanks,
David