get_user_pages() and shared memory question

get_user_pages() and shared memory question

[Timur Tabi]

Hi,

Is it possible for a page of memory that's been "grabbed" with get_user_pages() to ever be 
allocated to another process?  I'm assuming the answer is no, but I have a specific case I 
want to ask about.

Let's say an application allocates some shared memory, and then calls into a driver which 
calls get_user_pages().  The driver exits without releasing the pages, so they now have a 
reference count on them.  Then the application deallocates the shared memory.  At this 
point, the virtual addresses disappear, and no process owns them, but the pages still have 
a reference count.

Another process now tries to allocate a shared memory buffer.  Is there any way that this 
new buffer can contain those pages that were grabbed with get_user_pages() (i.e. that 
already have a reference count)?

Until 2.6.7, there was a bug in the VM where a page that was grabbed with get_user_pages() 
could be swapped out.  Those of you familar with the OpenIB work know what I'm talking 
about.  Would that bug affect anything I'm talking about?

-- 
Timur Tabi
Staff Software Engineer
timur.tabi@ammasso.com

One thing a Southern boy will never say is,
"I don't think duct tape will fix it."
      -- Ed Smylie, NASA engineer for Apollo 13

Re: get_user_pages() and shared memory question

[Roland Dreier]

    Timur> Hi, Is it possible for a page of memory that's been
    Timur> "grabbed" with get_user_pages() to ever be allocated to
    Timur> another process?  I'm assuming the answer is no, but I have
    Timur> a specific case I want to ask about.

Not to the best of my knowledge, although you should probably read the
code to convince yourself.

    Timur> Until 2.6.7, there was a bug in the VM where a page that
    Timur> was grabbed with get_user_pages() could be swapped out.
    Timur> Those of you familar with the OpenIB work know what I'm
    Timur> talking about.  Would that bug affect anything I'm talking
    Timur> about?

This isn't what the bug caused.  What could happen was that the
swapper could unmap a page from a process's virtual memory map before
it noticed that the page had an elevated reference count.  The page
wouldn't get swapped out, but when the process caused a page fault to
bring the virtual address back, it would get a different piece of
physical memory.

 - R.

Re: get_user_pages() and shared memory question

[Hugh Dickins]

On Tue, 21 Jun 2005, Timur Tabi wrote:
> 
> Is it possible for a page of memory that's been "grabbed" with
> get_user_pages() to ever be allocated to another process?  I'm assuming the
> answer is no, but I have a specific case I want to ask about.
> 
> Let's say an application allocates some shared memory, and then calls into a
> driver which calls get_user_pages().  The driver exits without releasing the
> pages, so they now have a reference count on them.  Then the application
> deallocates the shared memory.  At this point, the virtual addresses
> disappear, and no process owns them, but the pages still have a reference
> count.
> 
> Another process now tries to allocate a shared memory buffer.  Is there any
> way that this new buffer can contain those pages that were grabbed with
> get_user_pages() (i.e. that already have a reference count)?

It depends on what you mean by allocate and deallocate.  If the second
process is attaching the same shared memory segment as the first process
had attached, then yes, its buffer will contain those very pages which
the driver erroneously failed to release.

> Until 2.6.7, there was a bug in the VM where a page that was grabbed with
> get_user_pages() could be swapped out.  Those of you familar with the OpenIB
> work know what I'm talking about.  Would that bug affect anything I'm talking
> about?

No.  That was a bug peculiar to anonymous memory,
whereas shared memory is treated like file cache.

Hugh

Re: get_user_pages() and shared memory question

[Timur Tabi]

Hugh Dickins wrote:

> It depends on what you mean by allocate and deallocate.  If the second
> process is attaching the same shared memory segment as the first process
> had attached, then yes, its buffer will contain those very pages which
> the driver erroneously failed to release.

No, I'm talking about when the first process completely destroys the shared memory segment 
so that it no longer exists.  No processes are attached to it, and any attempt to attach 
to it results in an error, because it doesn't exist.

In this case, when a process creates a new memory segment, I just want to know whether the 
pages with a non-zero refcount (because of the get_user_pages() call) can ever be used in 
a new shared memory segment.

I'm assuming the answer is no, because that would defeat the purpose of refcount (right?). 
  I've been looking at the code and reading books on the VM, but I get lost easily.  It 
appears that the function which allocates a page is shmem_alloc_page(), which calls 
alloc_page() to do the actual work.  If that's correct, is it possible for alloc_page() to 
return a page that has been previously "claimed" by get_user_pages()?  I'm looking at 
__alloc_pages(), and I don't see any calls to page_count(), so I guess there's some other 
mechanism (either in get_user_pages() or in the way the VM works) that prevents this 
possibility.  However, I'm getting dangerously close to my limit of understanding the 
Linux VM.

Thanks for replying to my message.  I really appreciate the help in understanding the 
Linux VM.

-- 
Timur Tabi
Staff Software Engineer
timur.tabi@ammasso.com

One thing a Southern boy will never say is,
"I don't think duct tape will fix it."
      -- Ed Smylie, NASA engineer for Apollo 13

Re: get_user_pages() and shared memory question

[Hugh Dickins]

On Tue, 21 Jun 2005, Timur Tabi wrote:
> 
> In this case, when a process creates a new memory segment, I just want to know
> whether the pages with a non-zero refcount (because of the get_user_pages()
> call) can ever be used in a new shared memory segment.
> 
> I'm assuming the answer is no, because that would defeat the purpose of
> refcount (right?).

Exactly right.

Hugh

Re: get_user_pages() and shared memory question

[Brice Goglin]

Le 21.06.2005 17:10, Timur Tabi a écrit :
> Let's say an application allocates some shared memory, and then calls
> into a driver which calls get_user_pages().  The driver exits without
> releasing the pages, so they now have a reference count on them.

Preventing the driver from doing this would probably be the
right solution here... If the driver called get_user_pages,
it is its responsibility to release the pages.

Brice

Re: get_user_pages() and shared memory question

[Timur Tabi]

Brice Goglin wrote:

> Preventing the driver from doing this would probably be the
> right solution here... If the driver called get_user_pages,
> it is its responsibility to release the pages.

The driver does release the pages, but only when asked to do so.  If the process dies, 
then the driver automatically cleans up, but otherwise how is the driver to know that the 
memory is no longer needed?

Perhaps you mean that the driver should release the pages before it exits.  Unfortunately, 
that defeats the purpose of calling get_user_pages() in the first place.  The driver needs 
to pin the application's buffers so that the subsequent DMA operations work.  This driver 
supports an RDMA adapter that transfer network data directly to the application's buffers.

You're probably now thinking, "Well, why doesn't the driver just allocate the buffers on 
behalf of the app?"  There are two reasons why we can't do that.  One, the app may need 
have gigabytes of memory for the RDMA operations.  Two, the APIs we need to support allow 
the app to allocate memory any way it sees fit.

-- 
Timur Tabi
Staff Software Engineer
timur.tabi@ammasso.com

One thing a Southern boy will never say is,
"I don't think duct tape will fix it."
      -- Ed Smylie, NASA engineer for Apollo 13