Re: forcibly unmap pages in driver?

Re: forcibly unmap pages in driver?

[Dan Maas]

Just an update to my situation... I've implemented my idea of clearing the
associated PTE's when I need to free the DMA buffer, then re-filling them in
nopage(). This seems to work fine; if the user process tries anything fishy,
it gets a SIGBUS instead of accessing the old mapping.

I encountered two difficulties with the implementation:

1) zap_page_range(), flush_cache_range(), and flush_tlb_range() are not
exported to drivers. I basically copied the guts of zap_page_range() into my
driver, which seems to work OK on x86, but I know it will have trouble with
architectures that require special treatment of PTE manipulation...

2) the state of mm->mmap_sem is unknown when my file_operations->release()
function is called. If release() is called when the last FD closes, then
mm->mmap_sem is not taken. But if release() is called from do_munmap, then
mmap_sem has already been taken. So, it is risky to mess with vma's inside
of release()...

Regards,
Dan

> >> Later, the program calls the ioctl() again to set a smaller
> >> buffer size, or closes the file descriptor. At this point
> >> I'd like to shrink the buffer or free it completely. But I
> >> can't assume that the program will be nice and munmap() the
> >> region for me
>
> > Look at drivers/char/drm, for example.  At mmap time they allocate a
> > vm_ops to the address space.  With that you catch changes to the vma
> > structure initiated by a user mmap, munmap, etc.  You could also
> > dynamically map the pages in using the nopage method (optional).

forcibly unmap pages in driver?

[Dan Maas]

I am writing a device driver that, like many others, exposes a shared memory
region to user-space via mmap(). The region is allocated with vmalloc(), the
pages are marked reserved, and the user-space mapping is implemented with
remap_page_range().

In my driver, I may have to free the underlying vmalloc() region while the
user-space program is still running. I need to remove the user-space
mapping -- otherwise the user process would still have access to the
now-freed pages. I need an inverse of remap_page_range().

Is zap_page_range() the function I am looking for? Unfortunately it's not
exported to modules =(. As a quick fix, I was thinking I could just remap
all of the user pages to point to a zeroed page or something...

Another question- in the mm.c sources, I see that many of the memory-mapping
functions are surrounded by calls to flush_cache_range() and
flush_tlb_range(). But I don't see these calls in many drivers. Is it
necessary to make them when my driver maps or unmaps the shared memory
region?

Regards,
Dan

Re: forcibly unmap pages in driver?

[Pete Wyckoff]

dmaas@dcine.com said:
> I am writing a device driver that, like many others, exposes a shared memory
> region to user-space via mmap(). The region is allocated with vmalloc(), the
> pages are marked reserved, and the user-space mapping is implemented with
> remap_page_range().
> 
> In my driver, I may have to free the underlying vmalloc() region while the
> user-space program is still running. I need to remove the user-space
> mapping -- otherwise the user process would still have access to the
> now-freed pages. I need an inverse of remap_page_range().
> 
> Is zap_page_range() the function I am looking for? Unfortunately it's not
> exported to modules =(. As a quick fix, I was thinking I could just remap
> all of the user pages to point to a zeroed page or something...
> 
> Another question- in the mm.c sources, I see that many of the memory-mapping
> functions are surrounded by calls to flush_cache_range() and
> flush_tlb_range(). But I don't see these calls in many drivers. Is it
> necessary to make them when my driver maps or unmaps the shared memory
> region?

That seems a bit perverse.  How will the poor userspace program know
not to access the pages you have yanked away from it?  If you plan
to kill it, better to do that directly.  If you plan to signal it
that the mapping is gone, it can just call munmap() itself.

However, do_munmap() will call zap_page_range() for you and take care of
cache and TLB flushing if you're going to do this in the kernel.

Your driver mmap function is called by do_mmap_pgoff() which takes
care of those issues, and there is no (*munmap) in file_operations---
perhaps you are the first driver writer to want to unmap in the kernel.

		-- Pete

Re: forcibly unmap pages in driver?

[Dan Maas]

> That seems a bit perverse.  How will the poor userspace program know
> not to access the pages you have yanked away from it?  If you plan
> to kill it, better to do that directly.  If you plan to signal it
> that the mapping is gone, it can just call munmap() itself.

Thanks Pete. I will explain situation I am envisioning; perhaps there is a
better way to handle this --

My driver uses a variable-size DMA buffer that it shares with user-space; I
provide an ioctl() to choose the buffer size and allocate the buffer. Say
the user program chooses a large buffer size, and mmap()s the entire buffer.
Later, the program calls the ioctl() again to set a smaller buffer size, or
closes the file descriptor. At this point I'd like to shrink the buffer or
free it completely. But I can't assume that the program will be nice and
munmap() the region for me - it might still have the large buffer mapped.
What should I do here?

An easy solution would to allocate the largest possible buffer as my driver
is loaded, even if not all of it will be exposed to user-space. I don't
really like this choice because the buffer needs to be pinned in memory, and
the largest useful buffer size is very big (several tens of MB). Maybe I
should disallow more than one buffer allocation per open() of the device...
But the memory mapping will stay around even after close(), correct? I'd
hate to have to keep the buffer around until my driver module is unloaded.

> However, do_munmap() will call zap_page_range() for you and take care of
> cache and TLB flushing if you're going to do this in the kernel.

I'm not sure if I could use do_munmap() -- how will I know if the user
program has called munmap() already, and then mmap()ed something else in the
same place? Then I'd be killing the wrong mapping...

Regards,
Dan

Re: forcibly unmap pages in driver?

[Pete Wyckoff]

dmaas@dcine.com said:
> My driver uses a variable-size DMA buffer that it shares with user-space; I
> provide an ioctl() to choose the buffer size and allocate the buffer. Say
> the user program chooses a large buffer size, and mmap()s the entire buffer.
> Later, the program calls the ioctl() again to set a smaller buffer size, or
> closes the file descriptor. At this point I'd like to shrink the buffer or
> free it completely. But I can't assume that the program will be nice and
> munmap() the region for me - it might still have the large buffer mapped.
> What should I do here?
> 
> An easy solution would to allocate the largest possible buffer as my driver
> is loaded, even if not all of it will be exposed to user-space. I don't
> really like this choice because the buffer needs to be pinned in memory, and
> the largest useful buffer size is very big (several tens of MB). Maybe I
> should disallow more than one buffer allocation per open() of the device...
> But the memory mapping will stay around even after close(), correct? I'd
> hate to have to keep the buffer around until my driver module is unloaded.

I see.  Your explanation makes sense, and close won't affect the mmap
(unless you explicitly trigger it in the driver, which you should).
Other drivers deal with this; you may want to go grepping for do_munmap
and see how they handle it.

[ > pw@osc.edu said: ]
> > However, do_munmap() will call zap_page_range() for you and take care of
> > cache and TLB flushing if you're going to do this in the kernel.
> 
> I'm not sure if I could use do_munmap() -- how will I know if the user
> program has called munmap() already, and then mmap()ed something else in the
> same place? Then I'd be killing the wrong mapping...

Look at drivers/char/drm, for example.  At mmap time they allocate a
vm_ops to the address space.  With that you catch changes to the vma
structure initiated by a user mmap, munmap, etc.  You could also
dynamically map the pages in using the nopage method (optional).

The less elegant way of doing this is to put in your userspace API some
conditions which say:  if you do the following:

    open(fd);
    ioctl(fd, give_me_the_buf);
    munmap(some_or_all_of_the_buf);
    buf2 = mmap(...);
    close(fd);  /* or ioctl(fd, shrink_the_buf); */
    buf2[27] = 3;

you will be killed with a sigbus.  I.e. don't manually munmap the
region.

		-- Pete

Re: forcibly unmap pages in driver?

[Dan Maas]

>> Later, the program calls the ioctl() again to set a smaller
>> buffer size, or closes the file descriptor. At this point
>> I'd like to shrink the buffer or free it completely. But I
>> can't assume that the program will be nice and munmap() the
>> region for me

> Look at drivers/char/drm, for example.  At mmap time they allocate a
> vm_ops to the address space.  With that you catch changes to the vma
> structure initiated by a user mmap, munmap, etc.  You could also
> dynamically map the pages in using the nopage method (optional).

OK I think I have a solution... Whenever I need to re-allocate or free the
DMA buffer, I could set all of the user's corresponding page table entries
to deny all access. Then I'd get a page fault on the next access to the
buffer, and inside nopage() I could update the user's mapping or send a
SIGBUS as appropriate (hmm, just like restoring a file mapping that was
thrown away)... So I just have to figure out how to find the user's page
table entries that are pointing to the DMA buffer.

Regards,
Dan

Re: forcibly unmap pages in driver?

[Martin Diehl]

On Tue, 5 Jun 2001, Pete Wyckoff wrote:

> dmaas@dcine.com said:
> > user-space program is still running. I need to remove the user-space
> > mapping -- otherwise the user process would still have access to the
> > now-freed pages. I need an inverse of remap_page_range().
> 
> That seems a bit perverse.  How will the poor userspace program know
> not to access the pages you have yanked away from it?  If you plan
> to kill it, better to do that directly.  If you plan to signal it
> that the mapping is gone, it can just call munmap() itself.

I can very well imagine situations where you want to unmap a buffer 
beneath the userspace program to make it dying when not following the
rules - see below.

> However, do_munmap() will call zap_page_range() for you and take care of
> cache and TLB flushing if you're going to do this in the kernel.
> 
> Your driver mmap function is called by do_mmap_pgoff() which takes
> care of those issues, and there is no (*munmap) in file_operations---
> perhaps you are the first driver writer to want to unmap in the kernel.

Well, I don't know whether he is the first one, but if so, I'm probably
the other one thinking about (temporarily) unmapping mmap(2)ed memory
directly on behalf of the kernel. The idea is as follows:

Let's assume the mmap'ed buffer should be either accessible by userland
or (mutually exclusive) dedicated to I/O or DMA. So you need some
synchronisation like userland calling ioctl(2) to release the buffer and
blocks on select(2) or read(2) to get buffer access re-granted after DMA
has finished. Let's further assume you want to make absolutely sure the
userland app would really _never_, under no circumstance, access the
buffer at the wrong moment where it would read corrupted data, i.e. after
the ioctl() but before the file gets ready for nonblocking read again.

Well, you could validate all the paths in the app that might access the
mmap'ed buffer at some point and make sure this is either impossible or
will be detected somehow. Furthermore, for any future change you have to
do some re-validation... --- or, you simply make the mapping inaccessible
inside the critical window so the application would immediately die if it
would violate the rule. Note that it is not sufficient to require the app
to do the unmap itself after calling ioctl() because this would not solve
the problem - so you pay two syscalls for nothing.

I believe this is a good reason to do the unmap in the drivers'
ioctl() and remap again after DMA has finished. Basically, one could also
temporarily switch to PROT_NONE to get the same behaviour, but I
personally prefer unmapping due to more flexibility.

Although not tested so far, I think it should be possible by simply
calling do_munmap(), which is exported by mm/mmap.c. AFAICS the only thing
to take care of is the correct handling of mm->mmap_sem. Question remains
however, whether there is any reason not to do so, wrt. future changes
for example.

Regards
Martin