mmap() of /dev/kmem issue

mmap() of /dev/kmem issue

[Mario Smarduch]

I have problem of viewing memory contents via mmap()
of /dev/kmem. After mmap()'ing /dev/kmem for reading
it appears that data accessed in the kernel lives in
different cache lines from data accessed in the user space.
The lseek()/read() from kernel reflects latest updates,
but same reads from user space mmap fetches old data, and
vice versa when updating from user. When I flush the
caches (cat of a big file to another file), then user mapped
value reflects the latest changes but then again remains
frozen. I'm having trouble understanding how can this
happen since the caches are physically indexed.
Any ideas what the problem could be?

- Mario.

Re: mmap() of /dev/kmem issue

[Bjorn Helgaas]

On Tuesday 04 May 2004 10:30 am, Mario Smarduch wrote:
> I have problem of viewing memory contents via mmap()
> of /dev/kmem.
> ... 
> Any ideas what the problem could be?

Can you post a test case?  What kernel are you running?

Re: mmap() of /dev/kmem issue

[Grant Grundler]

On Tue, May 04, 2004 at 11:30:21AM -0500, Mario Smarduch wrote:
> I have problem of viewing memory contents via mmap() of /dev/kmem.

1) which regions/addresses are you accessing?
2) what values do you see/should see?
3) test case?

grant

> After mmap()'ing /dev/kmem for reading
> it appears that data accessed in the kernel lives in
> different cache lines from data accessed in the user space.
> The lseek()/read() from kernel reflects latest updates,
> but same reads from user space mmap fetches old data, and
> vice versa when updating from user. When I flush the
> caches (cat of a big file to another file), then user mapped
> value reflects the latest changes but then again remains
> frozen. I'm having trouble understanding how can this
> happen since the caches are physically indexed.
> Any ideas what the problem could be?
> 
> - Mario.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

Bjorn Helgaas wrote:

> On Tuesday 04 May 2004 10:30 am, Mario Smarduch wrote:
> > I have problem of viewing memory contents via mmap()
> > of /dev/kmem.
> > ...
> > Any ideas what the problem could be?
>
> Can you post a test case?  What kernel are you running?

To run the test first verify the PAGE_MASK matches. Secondly
compile tstmmapkmem.c, and run runtval.sh it expects an upto date
/boot/System.map file. It seems like the region doesn't appear to
be mapped cachable.

runtval.sh
-------

#!/bin/ksh

jiffies=0x`grep "D jiffies" /boot/System.map | awk '{ print $1 }'`
./tstmmapkmem $jiffies

tstmmapkmem.c - cc -o tstmmapkmem tstmmapkmem.c
----------
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <libgen.h>
#include <sys/time.h>
#include <stdlib.h>
#include <sys/mman.h>

#define IDENBASE 0xe000000000000000
#define DEVICE   "/dev/kmem"
#define PAGE_MASK (0x4000 - 1) /* page is 16k */

#define wmb() __asm__ __volatile__ ("": : :"memory")

extern int errno;
main(int argc, char *argv[])
{
   int fd;
   ulong mapaddr;
   volatile long *jifptr, jiffies;

   jiffies = strtoul(argv[1], 0, 16);
   printf("jiffies = 0x%lx+AFw-n", jiffies);
   printf("pid = %d+AFw-n", getpid());

   if((fd = open(DEVICE, O_RDWR)) < 0)
        printf("open failed - errno = %d+AFw-n", errno), exit(errno);

   jiffies -= IDENBASE;
   mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
PROT_WRITE|PROT_R
EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK);
   if(mapaddr = -1UL) {
      printf("mmap failed errno = %d+AFw-n", errno);
      exit(-1);
   }
   printf("mapaddr = 0x%lx+AFw-n", mapaddr);
   jiffies = mapaddr +- (jiffies & PAGE_MASK);
   jifptr = (long *) jiffies;
   printf("jifptr = 0x%lx+AFw-n", jifptr);
pragain:
   printf("jiffies: %ld+AFw-n",  *jifptr);

   printf("Type return to read again (and flush cashe in meantime):+AFw-n");

   getchar();
   goto pragain;
}

- Mario.

Re: mmap() of /dev/kmem issue

[Grant Grundler]

On Tue, May 04, 2004 at 01:12:21PM -0500, Mario Smarduch wrote:
> jiffies=0x`grep "D jiffies" /boot/System.map | awk '{ print $1 }'`
> ./tstmmapkmem $jiffies

Maybe the process is being prempted?
Ie maybe the problem is the contents are "stale" by the
time they get displayed on the console/terminal.
jiffies gets updated 1000/second so on occasion I would
expect this to happen.
Maybe pick something that doesn't get updated so frequently?

grant

> #define DEVICE   "/dev/kmem"
> #define PAGE_MASK (0x4000 - 1) /* page is 16k */
> 
> #define wmb() __asm__ __volatile__ ("": : :"memory")
> 
> extern int errno;
> main(int argc, char *argv[])
> {
>    int fd;
>    ulong mapaddr;
>    volatile long *jifptr, jiffies;
> 
>    jiffies = strtoul(argv[1], 0, 16);
>    printf("jiffies = 0x%lx\n", jiffies);
>    printf("pid = %d\n", getpid());
> 
>    if((fd = open(DEVICE, O_RDWR)) < 0)
>         printf("open failed - errno = %d\n", errno), exit(errno);
> 
>    jiffies -= IDENBASE;
>    mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
> PROT_WRITE|PROT_R
> EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & ~PAGE_MASK);
>    if(mapaddr = -1UL) {
>       printf("mmap failed errno = %d\n", errno);
>       exit(-1);
>    }
>    printf("mapaddr = 0x%lx\n", mapaddr);
>    jiffies = mapaddr + (jiffies & PAGE_MASK);
>    jifptr = (long *) jiffies;
>    printf("jifptr = 0x%lx\n", jifptr);
> pragain:
>    printf("jiffies: %ld\n",  *jifptr);
> 
>    printf("Type return to read again (and flush cashe in meantime):\n");
> 
>    getchar();
>    goto pragain;
> }
> 
> - Mario.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

Grant Grundler wrote:

> On Tue, May 04, 2004 at 01:12:21PM -0500, Mario Smarduch wrote:
> > jiffies=0x`grep "D jiffies" /boot/System.map | awk '{ print $1 }'`
> > ./tstmmapkmem $jiffies
>
> Maybe the process is being prempted?
> Ie maybe the problem is the contents are "stale" by the
> time they get displayed on the console/terminal.
> jiffies gets updated 1000/second so on occasion I would
> expect this to happen.
> Maybe pick something that doesn't get updated so frequently?

I can't see how that could have a bearing if both kernel/user
are hitting the cache you should see the same values everytime.

- Mario

>
>
> grant
>
> > #define DEVICE   "/dev/kmem"
> > #define PAGE_MASK (0x4000 - 1) /* page is 16k */
> >
> > #define wmb() __asm__ __volatile__ ("": : :"memory")
> >
> > extern int errno;
> > main(int argc, char *argv[])
> > {
> >    int fd;
> >    ulong mapaddr;
> >    volatile long *jifptr, jiffies;
> >
> >    jiffies = strtoul(argv[1], 0, 16);
> >    printf("jiffies = 0x%lx+AFw-n", jiffies);
> >    printf("pid = %d+AFw-n", getpid());
> >
> >    if((fd = open(DEVICE, O_RDWR)) < 0)
> >         printf("open failed - errno = %d+AFw-n", errno), exit(errno);
> >
> >    jiffies -= IDENBASE;
> >    mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
> > PROT_WRITE|PROT_R
> > EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK);
> >    if(mapaddr = -1UL) {
> >       printf("mmap failed errno = %d+AFw-n", errno);
> >       exit(-1);
> >    }
> >    printf("mapaddr = 0x%lx+AFw-n", mapaddr);
> >    jiffies = mapaddr  (jiffies & PAGE_MASK);
> >    jifptr = (long *) jiffies;
> >    printf("jifptr = 0x%lx+AFw-n", jifptr);
> > pragain:
> >    printf("jiffies: %ld+AFw-n",  *jifptr);
> >
> >    printf("Type return to read again (and flush cashe in meantime):+AFw-n");
> >
> >    getchar();
> >    goto pragain;
> > }
> >
> > - Mario.
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

It appears that from mmap_mem() noncached_address()
succeeds on PageReserved for ranges in: KERNEL_START-
KERNEL_END, I tried a temporary workaround by making
an exception to addresses in that range and things work (that
is for this program only) but what sort of problems would such a
change introduce?

if(PageReserved(page) && !((va >= KERNEL_START) && (va < KERNEL_END)) )
            return (1);

I would think allot of people would like to mmap() /dev/kmem
to take a quick look at kernel values, atleast that's what I've
seen on other Unix variants.

- mario.




Grant Grundler wrote:

> On Tue, May 04, 2004 at 01:12:21PM -0500, Mario Smarduch wrote:
> > jiffies=0x`grep "D jiffies" /boot/System.map | awk '{ print $1 }'`
> > ./tstmmapkmem $jiffies
>
> Maybe the process is being prempted?
> Ie maybe the problem is the contents are "stale" by the
> time they get displayed on the console/terminal.
> jiffies gets updated 1000/second so on occasion I would
> expect this to happen.
> Maybe pick something that doesn't get updated so frequently?
>
> grant
>
> > #define DEVICE   "/dev/kmem"
> > #define PAGE_MASK (0x4000 - 1) /* page is 16k */
> >
> > #define wmb() __asm__ __volatile__ ("": : :"memory")
> >
> > extern int errno;
> > main(int argc, char *argv[])
> > {
> >    int fd;
> >    ulong mapaddr;
> >    volatile long *jifptr, jiffies;
> >
> >    jiffies = strtoul(argv[1], 0, 16);
> >    printf("jiffies = 0x%lx+AFw-n", jiffies);
> >    printf("pid = %d+AFw-n", getpid());
> >
> >    if((fd = open(DEVICE, O_RDWR)) < 0)
> >         printf("open failed - errno = %d+AFw-n", errno), exit(errno);
> >
> >    jiffies -= IDENBASE;
> >    mapaddr=(ulong) mmap((void *)0x2000000080000000, 0x8000000,
> > PROT_WRITE|PROT_R
> > EAD|PROT_EXEC, MAP_SHARED | MAP_FIXED, fd, jiffies & +AH4-PAGE_MASK);
> >    if(mapaddr = -1UL) {
> >       printf("mmap failed errno = %d+AFw-n", errno);
> >       exit(-1);
> >    }
> >    printf("mapaddr = 0x%lx+AFw-n", mapaddr);
> >    jiffies = mapaddr  (jiffies & PAGE_MASK);
> >    jifptr = (long *) jiffies;
> >    printf("jifptr = 0x%lx+AFw-n", jifptr);
> > pragain:
> >    printf("jiffies: %ld+AFw-n",  *jifptr);
> >
> >    printf("Type return to read again (and flush cashe in meantime):+AFw-n");
> >
> >    getchar();
> >    goto pragain;
> > }
> >
> > - Mario.
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mmap() of /dev/kmem issue

[Grant Grundler]

On Tue, May 04, 2004 at 05:27:41PM -0500, Mario Smarduch wrote:
> I would think allot of people would like to mmap() /dev/kmem
> to take a quick look at kernel values, atleast that's what I've
> seen on other Unix variants.

yes - and that's a religious issue. I know people would
like like to see /dev/kmem completely removed.

Many wise kernel engineers have advised me to add /proc
or similar "architected" interfaces. That way a program
will merely get an error instead of crashing the box
when the kernel changes it's data structures or how said
structures get used.

grant

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

Grant Grundler wrote:

> On Tue, May 04, 2004 at 05:27:41PM -0500, Mario Smarduch wrote:
> > I would think allot of people would like to mmap() /dev/kmem
> > to take a quick look at kernel values, atleast that's what I've
> > seen on other Unix variants.
>
> yes - and that's a religious issue. I know people would
> like like to see /dev/kmem completely removed.
>
> Many wise kernel engineers have advised me to add /proc
> or similar "architected" interfaces. That way a program
> will merely get an error instead of crashing the box
> when the kernel changes it's data structures or how said
> structures get used.
>
> grant

That's understandable. But there are cases where
kernel values just need to be read timely. As things
are right now you can still corrupt the kernel but
are unable to do anything useful because of attribute
aliasing to that range. Perhaps a rejection of mapping
such regions for writes should be imposed, but I'm
not sure how mapping it as uncachable helps? To
me this behaviour seems buggy.

- Mario.

RE: mmap() of /dev/kmem issue

[Luck, Tony]

>That's understandable. But there are cases where
>kernel values just need to be read timely. As things
>are right now you can still corrupt the kernel but
>are unable to do anything useful because of attribute
>aliasing to that range. Perhaps a rejection of mapping
>such regions for writes should be imposed, but I'm
>not sure how mapping it as uncachable helps? To
>me this behaviour seems buggy.

Mapping kernel memory as uncacheable into a user process
seems like a recipe for disaster (well at least for an
MCA when the processor catches you using mixed attributes).

-Tony

Re: mmap() of /dev/kmem issue

[Grant Grundler]

On Wed, May 05, 2004 at 09:57:41AM -0500, Mario Smarduch wrote:
> But there are cases where
> kernel values just need to be read timely.

Watching kernel data from userspace is an impossible task.
Unless one has a "real time" scheduler and *halts* all kernel
activity, the kernel values visible to user space will lag
what the kernel does. Something like KGDB can do that.
It's not something I expect to work correctly.
Ergo define an interface (eg gettimeofday() or /proc/sys/kernel/*)
to get "sane" values and move on.

> As things
> are right now you can still corrupt the kernel but
> are unable to do anything useful because of attribute
> aliasing to that range. Perhaps a rejection of mapping
> such regions for writes should be imposed,

Maybe. I don't understand the VM well enough.

> but I'm
> not sure how mapping it as uncachable helps?
> To me this behaviour seems buggy.

I'm still inclined to think the approach is wrong.
Maybe you have a better test case that doesn't involve user
interaction and shows why the data has to come from /dev/kmem?

grant

Re: mmap() of /dev/kmem issue

[Matthew Wilcox]

On Wed, May 05, 2004 at 08:08:21AM -0700, Luck, Tony wrote:
> Mapping kernel memory as uncacheable into a user process
> seems like a recipe for disaster (well at least for an
> MCA when the processor catches you using mixed attributes).

... which is why we don't do that.

See drivers/char/mem.c::mmap_mem().  We call uncached_access() which does:

#elif defined(CONFIG_IA64)
        /*
         * On ia64, we ignore O_SYNC because we cannot tolerate memory attribute 
aliases.
         */
        return !(efi_mem_attributes(addr) & EFI_MEMORY_WB);

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

"Luck, Tony" wrote:

> >That's understandable. But there are cases where
> >kernel values just need to be read timely. As things
> >are right now you can still corrupt the kernel but
> >are unable to do anything useful because of attribute
> >aliasing to that range. Perhaps a rejection of mapping
> >such regions for writes should be imposed, but I'm
> >not sure how mapping it as uncachable helps? To
> >me this behaviour seems buggy.
>
> Mapping kernel memory as uncacheable into a user process
> seems like a recipe for disaster (well at least for an
> MCA when the processor catches you using mixed attributes).
>
> -Tony

The background on the issue is that I'm trying to mmap()
some data in KERNEL_START-KERNEL_END range
to read some data, without having to go through a system
call. This results in uncacheble mapping in
user space, which can results in what you stated above.
Its something I did routinly on other Unix variants.
And I think that restriction is not there on Linux IA32
boxes. The reason its mapped uncached its because
that range is marked reserved. I'm guessing that reserved
memory map ranges are uncachable (just guessing here),
but in this case the range in question is accessed cacheble
by the kernel.

- Mario.

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

Matthew Wilcox wrote:

> On Wed, May 05, 2004 at 08:08:21AM -0700, Luck, Tony wrote:
> > Mapping kernel memory as uncacheable into a user process
> > seems like a recipe for disaster (well at least for an
> > MCA when the processor catches you using mixed attributes).
>
> ... which is why we don't do that.
>
> See drivers/char/mem.c::mmap_mem().  We call uncached_access() which does:
>
> #elif defined(CONFIG_IA64)
>         /*
>          * On ia64, we ignore O_SYNC because we cannot tolerate memory attribute
> aliases.
>          */
>         return !(efi_mem_attributes(addr) & EFI_MEMORY_WB);

That's great but what release is that in?

- mario.

>
>
> --
> "Next the statesmen will invent cheap lies, putting the blame upon
> the nation that is attacked, and every man will be glad of those
> conscience-soothing falsities, and will diligently study them, and refuse
> to examine any refutations of them; and thus he will by and by convince
> himself that the war is just, and will thank God for the better sleep
> he enjoys after this process of grotesque self-deception." -- Mark Twain
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mmap() of /dev/kmem issue

[Matthew Wilcox]

On Wed, May 05, 2004 at 10:53:10AM -0500, Mario Smarduch wrote:
> That's great but what release is that in?

This is in 2.6.6-rc3.  I don't know when it was added.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: mmap() of /dev/kmem issue

[Chris Wedgwood]

On Wed, May 05, 2004 at 05:07:15PM +0100, Matthew Wilcox wrote:

> This is in 2.6.6-rc3.  I don't know when it was added.

Almost a year ago:

       http://linux.bkbits.net:8080/linux-2.5/cset%401.1046.264.19?nav=index.html|src/.|src/drivers|src/drivers/char|related/drivers/char/mem.c

(there are some relevant changes since then too obviously)


   --cw

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

Grant Grundler wrote:

> On Wed, May 05, 2004 at 09:57:41AM -0500, Mario Smarduch wrote:
> > But there are cases where
> > kernel values just need to be read timely.
>
> Watching kernel data from userspace is an impossible task.
> Unless one has a "real time" scheduler and *halts* all kernel
> activity, the kernel values visible to user space will lag
> what the kernel does. Something like KGDB can do that.
> It's not something I expect to work correctly.

I've worked with protocol applications (Frame Relay/LAPD)
that used /dev/kmem to synchronize across kernel allocated
buffers to bypass standard system call overhead.  The tasks ran
at highest priority and were locked. I'm not advacating that
kind of use but its been done.

>
> Ergo define an interface (eg gettimeofday() or /proc/sys/kernel/*)
> to get "sane" values and move on.

That's kind of what this is intended for.

- Mario.

>
>
> > As things
> > are right now you can still corrupt the kernel but
> > are unable to do anything useful because of attribute
> > aliasing to that range. Perhaps a rejection of mapping
> > such regions for writes should be imposed,
>
> Maybe. I don't understand the VM well enough.
>
> > but I'm
> > not sure how mapping it as uncachable helps?
> > To me this behaviour seems buggy.
>
> I'm still inclined to think the approach is wrong.
> Maybe you have a better test case that doesn't involve user
> interaction and shows why the data has to come from /dev/kmem?
>
> grant

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

Matthew Wilcox wrote:

> On Wed, May 05, 2004 at 10:53:10AM -0500, Mario Smarduch wrote:
> > That's great but what release is that in?
>
> This is in 2.6.6-rc3.  I don't know when it was added.
>
> --
> "Next the statesmen will invent cheap lies, putting the blame upon
> the nation that is attacked, and every man will be glad of those
> conscience-soothing falsities, and will diligently study them, and refuse
> to examine any refutations of them; and thus he will by and by convince
> himself that the war is just, and will thank God for the better sleep
> he enjoys after this process of grotesque self-deception." -- Mark Twain

Is the patch applicable to 2.4 variants?

RE: mmap() of /dev/kmem issue

[Luck, Tony]

>On Wed, May 05, 2004 at 10:53:10AM -0500, Mario Smarduch wrote:
>> That's great but what release is that in?
>
>This is in 2.6.6-rc3.  I don't know when it was added.

N.B. your test program will need some changes to work under
2.6 as the kernel is linked into virtual addresses in region 5
instead of region 7.

So "IDENBASE" and "jiffies - IDENBASE" would need to be changed.

In fact there might be another whole layer of difficulty writing
your test program, as figuring the physical address of the kernel
object that you want to look at is a lot harder as the bootloader
may have picked any physical address to load the kernel (though
in practice at the moment it only does this on SGI sn2 machines).

-Tony

Re: mmap() of /dev/kmem issue

[Mario Smarduch]

"Luck, Tony" wrote:

> >On Wed, May 05, 2004 at 10:53:10AM -0500, Mario Smarduch wrote:
> >> That's great but what release is that in?
> >
> >This is in 2.6.6-rc3.  I don't know when it was added.
>
> N.B. your test program will need some changes to work under
> 2.6 as the kernel is linked into virtual addresses in region 5
> instead of region 7.
>
> So "IDENBASE" and "jiffies - IDENBASE" would need to be changed.
>
> In fact there might be another whole layer of difficulty writing
> your test program, as figuring the physical address of the kernel
> object that you want to look at is a lot harder as the bootloader
> may have picked any physical address to load the kernel (though
> in practice at the moment it only does this on SGI sn2 machines).
>
> -Tony

Tony thanks for the pointers. I actually need to
have this work on 2.4 though.