Re: [PATCH 10 of 20] ipath - support for userspace apps using core driver

["Bryan O'Sullivan" <bos@pathscale.com>]

On Thu, 2006-03-16 at 20:10 +0000, Hugh Dickins wrote:

> You wisely remarked earlier that you'd not yet checked for memory leaks:
> that is of course the complementary, less obvious, error to the troubles
> you've been having so far: and I wish you luck when you come to check,
> hoping that I haven't merely misled you from one side to the other!

Well, as it turns out, we're not out of the woods.

The memory that we allocate with dma_alloc_coherent is indeed being
handed back to us as compound pages, but something horrible is happening
after that.

Our nopage handler does an unconditional get_page on every page that it
gets handed, but nothing seems to ever call put_page on those pages.  At
least, I infer this to be the case, because the page counts look all
wrong when I free them.  The kernel thinks I leak 4.5MB of memory every
time I run "hello, world" (~1300 allocations of 4K).

If you recall, we have three different areas of memory that userspace
can mmap.  Two of these (rcvhdrq and pioavailregs) are allocated at
driver init time, and freed when the driver exits.  The third (egrbufs)
is allocated when the process opens the char device, and closed when the
char device's release method is called.

I've instrumented my calls to dma_alloc_coherent and dma_free_coherent,
and they are getting called when they should be.  However, on each
successive run of "hello, world", the page counts on the rcvhdrq and
pioavailregs pages (the ones allocated at driver start) increase,
without ever decreasing.  And for the egrbufs pages, the page counts go
up to 8 (I'm allocating 32K at a time, so get_page is called 8 times per
compound page), but I never get handed the same struct page on two
successive runs, so I conclude that the struct pages are leaking.

So my belief is that I am freeing the memory behind the struct pages,
but the struct pages themselves stay stuck with a permanently elevated
count.  This is obviously not a winning strategy.

On 2.6.15, this all "works", but I have the poor kernel thinking it
loses 4.5MB of memory on every run.  On 2.6.16-rc6, I get a BUG.  I'll
have to do a bit of work to reproduce it, so I can't paste it here yet.

So my quandary is thus: what am I doing wrong?  It seems that my calls
to get_page are more or less correct, but should I be doing a put_page
somewhere, too, such as when the char dev's release method is called,
prior to calling dma_free_coherent?

	<b