pagevec_lookup_tag() and pagevec_release() substitutes and an OOM

pagevec_lookup_tag() and pagevec_release() substitutes and an OOM

[Martin Jambor]

Hi all,

I am implementing aops->writepages of a filesystem and for various
reasons I cannot use mpage_writepages. On the other hand, I would like
to do be able to get a vector of pages and process it page by page (in
my way) just like mpage_writepages gets pages.

However, pagevec functions are not exported (I don't understand why)
and so I cannot use them directly by my module but rather I had to do
a bit of cutting and pasting from 2.6.14 kernel source :-(

So my approach was to use a plain array of pointers to pages and
instead of calling

unsigned pagevec_lookup_tag(struct pagevec *pvec, struct address_space *mapping,
		pgoff_t *index, int tag, unsigned nr_pages)

I use my own function which is basically copied find_get_pages_tag()
(which also isn't exported):

static unsigned my_find_get_pages_tag(struct address_space *mapping,
				   pgoff_t *index, int tag, unsigned int nr_pages,
                                   struct page **pages)
{
	unsigned int i;
	unsigned int ret;

	read_lock_irq(&mapping->tree_lock);
	ret = radix_tree_gang_lookup_tag(&mapping->page_tree,
				(void **)pages, *index, nr_pages, tag);
	for (i = 0; i < ret; i++)
		page_cache_get(pages[i]);
	if (ret)
		*index = pages[ret - 1]->index + 1;
	read_unlock_irq(&mapping->tree_lock);
	return ret;
}

instead of pagevec_release() I use a much simpler function:

static void release_pagearray(struct page **pages, unsigned int nr_pages)
{
	unsigned i;
	
	for (i = 0; i < nr_pages; i++)
		page_cache_release(pages[i]);
}

Everything seems to be working well except that I get an OOM. The free
and vmstat utilities show that "cache" is the only thing that is
getting unreasonably bigger. There are many reasons why I believe the
page cache pages are still somehow pinned down by my code and cannot
be freed when I am done with them. I must admit that I find it quite
difficult to uderstand the pagevec source and if someone knew what I
am doing wrong, I would be very grateful.

(Yes, I have carefully checked several times very carefully that I
set&clear writeback as indicated in Documentation/filesystmes/Locking
and how it is done by mpage_writepages.)

Thank you very much in advance for any comment.

Martin

Re: pagevec_lookup_tag() and pagevec_release() substitutes and an OOM

[Martin Jambor]

Please disregard my previous post, I have found the bug right at the
spot where I've been starring for the last two days. (I knew this
would happen, I am sorry but I guess if I didn't send the mail, it
would take me even longer to discover what was going on).

The trouble was I was not removing pages from LRU and mm didn't like that.

However, why are not pagevec functions exported for modules? They seem
very useful and I hate to cut and paste stuff.

Thanks very much anyway,

Martin

On 2/8/06, Martin Jambor <jambormartin@gmail.com> wrote:
> Hi all,
>
> I am implementing aops->writepages of a filesystem and for various
> reasons I cannot use mpage_writepages. On the other hand, I would like
> to do be able to get a vector of pages and process it page by page (in
> my way) just like mpage_writepages gets pages...

Re: pagevec_lookup_tag() and pagevec_release() substitutes and an OOM

[Andrew Morton]

Martin Jambor <jambormartin@gmail.com> wrote:
>
> However, why are not pagevec functions exported for modules?
>

Nothing in-kernel uses them.   If/when your fs is merged, we can export them.

If your fs is GPL and reasonably serious/useful then perhaps we could
export them now, although there are no guarantees that we won't change them
or remove them or whatever in the future.

Re: pagevec_lookup_tag() and pagevec_release() substitutes and an OOM

[Dave Kleikamp]

On Tue, 2006-02-07 at 23:48 -0800, Andrew Morton wrote:
> Martin Jambor <jambormartin@gmail.com> wrote:
> >
> > However, why are not pagevec functions exported for modules?
> >
> 
> Nothing in-kernel uses them.   If/when your fs is merged, we can export them.

As of 2.6.16-rc1, they are exported.  cifs now uses them.

> If your fs is GPL and reasonably serious/useful then perhaps we could
> export them now, although there are no guarantees that we won't change them
> or remove them or whatever in the future.

Thanks,
Shaggy
-- 
David Kleikamp
IBM Linux Technology Center