put_page on transparent huge page leaks?

put_page on transparent huge page leaks?

[Jay Cornwall]

Hi,

I'm tracking a possible memory leak in iommu/amd. The driver uses this 
logic to fault a page in response to a PRI from a device:

npages = get_user_pages(fault->state->task, fault->state->mm,
                         fault->address, 1, write, 0, &page, NULL);

if (npages == 1)
         put_page(page);
else
         ...

This works correctly when get_user_pages returns a 4KB page. When 
transparent huge pages are enabled any 2MB page returned by this call 
appears to leak on process exit. The non-cached memory usage stays 
elevated by the set of faulted 2MB pages. This behavior is not observed 
when the exception handler demand faults 2MB pages.

I notice there is a difference in reference count between the 4KB/2MB 
paths.

get_user_pages (4KB): page_count()=3, page_mapcount()=1
put_page       (4KB): page_count()=2, page_mapcount()=1

get_user_pages (2MB): page_count()=3, page_mapcount()=1
put_page       (2MB): page_count()=3, page_mapcount()=0

I'm concerned that the driver appears to be holding a reference count 
after put_page(). Am I interpreting this observation correctly?

Re: put_page on transparent huge page leaks?

[Kirill A. Shutemov]

On Fri, Feb 21, 2014 at 11:23:39AM -0600, Jay Cornwall wrote:
> Hi,
> 
> I'm tracking a possible memory leak in iommu/amd. The driver uses this logic
> to fault a page in response to a PRI from a device:
> 
> npages = get_user_pages(fault->state->task, fault->state->mm,
>                         fault->address, 1, write, 0, &page, NULL);
> 
> if (npages == 1)
>         put_page(page);
> else
>         ...
> 
> This works correctly when get_user_pages returns a 4KB page. When
> transparent huge pages are enabled any 2MB page returned by this call
> appears to leak on process exit. The non-cached memory usage stays elevated
> by the set of faulted 2MB pages. This behavior is not observed when the
> exception handler demand faults 2MB pages.
> 
> I notice there is a difference in reference count between the 4KB/2MB paths.
> 
> get_user_pages (4KB): page_count()=3, page_mapcount()=1
> put_page       (4KB): page_count()=2, page_mapcount()=1
> 
> get_user_pages (2MB): page_count()=3, page_mapcount()=1
> put_page       (2MB): page_count()=3, page_mapcount()=0
> 
> I'm concerned that the driver appears to be holding a reference count after
> put_page(). Am I interpreting this observation correctly?

Could you show output of dump_page() on 2M pages for both points?

My guess is that your page is PageTail(). Refcounting for tail pages is
different: on get_page() we increase *->_mapcount* of tail and increase
->_count of relevant head page. ->_count of tail pages should always be
zero, but it's 3 in your case which is odd.

BTW, I don't see where you take mmap_sem in drivers/iommu/amd_iommu_v2.c,
which is required for gup. Do I miss something?

-- 
 Kirill A. Shutemov

Re: put_page on transparent huge page leaks?

[Jay Cornwall]

On 2014-02-21 20:31, Kirill A. Shutemov wrote:
> On Fri, Feb 21, 2014 at 11:23:39AM -0600, Jay Cornwall wrote:
>> I'm tracking a possible memory leak in iommu/amd. The driver uses this 
>> logic
>> to fault a page in response to a PRI from a device:
>> 
>> npages = get_user_pages(fault->state->task, fault->state->mm,
>>                         fault->address, 1, write, 0, &page, NULL);
>> 
>> if (npages == 1)
>>         put_page(page);
>> else
>>         ...
>> 
>> This works correctly when get_user_pages returns a 4KB page. When
>> transparent huge pages are enabled any 2MB page returned by this call
>> appears to leak on process exit. The non-cached memory usage stays 
>> elevated
>> by the set of faulted 2MB pages. This behavior is not observed when 
>> the
>> exception handler demand faults 2MB pages.
> 
> Could you show output of dump_page() on 2M pages for both points?

get_user_pages():
   page:ffffea000ffa00c0 count:0 mapcount:1 mapping:          (null) 
index:0x0
   page flags: 0x2fffe0000008000(tail)
   // page_count(page)=3 (head page)
put_page():
   page:ffffea000ffa00c0 count:0 mapcount:0 mapping:          (null) 
index:0x0
   page flags: 0x2fffe0000008000(tail)
   // page_count(page)=3 (head page)

Repeat on the same page.

get_user_pages():
   page:ffffea000ffa00c0 count:0 mapcount:1 mapping:          (null) 
index:0x0
   page flags: 0x2fffe0000008000(tail)
   // page_count(page)=4 (head page)
put_page():
   page:ffffea000ffa00c0 count:0 mapcount:0 mapping:          (null) 
index:0x0
   page flags: 0x2fffe0000008000(tail)
   // page_count(page)=4 (head page)

The head page appears to be leaking a reference. There is *no leak* if 
the driver faults the head page directly.

> My guess is that your page is PageTail(). Refcounting for tail pages is
> different: on get_page() we increase *->_mapcount* of tail and increase
> ->_count of relevant head page. ->_count of tail pages should always be
> zero, but it's 3 in your case which is odd.

That's correct, this is a tail page. page_count() references the head 
page:

   static inline int page_count(struct page *page)
   {
           return atomic_read(&compound_head(page)->_count);
   }

> BTW, I don't see where you take mmap_sem in 
> drivers/iommu/amd_iommu_v2.c,
> which is required for gup. Do I miss something?

You're right. I have a patch applied on my local branch to fix this.

Re: put_page on transparent huge page leaks?

[Jay Cornwall]

On 2014-02-22 11:44, Jay Cornwall wrote:
> On 2014-02-21 20:31, Kirill A. Shutemov wrote:
>> On Fri, Feb 21, 2014 at 11:23:39AM -0600, Jay Cornwall wrote:
>>> I'm tracking a possible memory leak in iommu/amd. The driver uses 
>>> this logic
>>> to fault a page in response to a PRI from a device:
> 
> The head page appears to be leaking a reference. There is *no leak* if
> the driver faults the head page directly.

My apologies, this was fixed somewhere in the patch series: "mm: tail 
page refcounting optimization for slab and hugetlbfs" (Jan 22nd).

Our merge was slightly older than I'd thought.