OOM problem caused by fs

OOM problem caused by fs

[Seth Huang]

Hello everyone,

Our group is developing a new file system for linux and we got stuck
with out-of-memory problem.

When creating large files in our fs, the system will run out of
memory(I mean the kernel starts to dump memory usage repeatedly and
the oom killer begins to kill processes) as long as the amount of data
exceeds the capacity of free memory, even if the kernel is flushing
out dirty pages.

If i'm right, when available memory is low, the writes will be blocked
in page cache allocation until some dirty pages are cleaned. I've
checked pdflush, it works fine in our system, which means dirty pages
can be flushed out and cleaned in time. However, it still crashes the
system. I've no idea how could this happen.

Has anyone experienced the same thing? Any advices will be appreciated.

Thanks,
Seth

Re: OOM problem caused by fs

[Boaz Harrosh]

On 06/07/2010 11:34 AM, Seth Huang wrote:
> Hello everyone,
> 
> Our group is developing a new file system for linux and we got stuck
> with out-of-memory problem.
> 
> When creating large files in our fs, the system will run out of
> memory(I mean the kernel starts to dump memory usage repeatedly and
> the oom killer begins to kill processes) as long as the amount of data
> exceeds the capacity of free memory, even if the kernel is flushing
> out dirty pages.
> 
> If i'm right, when available memory is low, the writes will be blocked
> in page cache allocation until some dirty pages are cleaned. I've
> checked pdflush, it works fine in our system, which means dirty pages
> can be flushed out and cleaned in time. However, it still crashes the
> system. I've no idea how could this happen.
> 
> Has anyone experienced the same thing? Any advices will be appreciated.
> 
> Thanks,
> Seth

Code!? a git tree URL? ...?

Boaz

Re: OOM problem caused by fs

[Nick Piggin]

On Mon, Jun 07, 2010 at 04:34:16PM +0800, Seth Huang wrote:
> Hello everyone,
> 
> Our group is developing a new file system for linux and we got stuck
> with out-of-memory problem.
> 
> When creating large files in our fs, the system will run out of
> memory(I mean the kernel starts to dump memory usage repeatedly and
> the oom killer begins to kill processes) as long as the amount of data
> exceeds the capacity of free memory, even if the kernel is flushing
> out dirty pages.
> 
> If i'm right, when available memory is low, the writes will be blocked
> in page cache allocation until some dirty pages are cleaned. I've
> checked pdflush, it works fine in our system, which means dirty pages
> can be flushed out and cleaned in time. However, it still crashes the
> system. I've no idea how could this happen.
> 
> Has anyone experienced the same thing? Any advices will be appreciated.

Do you have a pointer to your source?

Are you using set_page_dirty to dirty the pages?

Are you sure you don't have a refcount leak?

Re: OOM problem caused by fs

[Seth Huang]

> Are you using set_page_dirty to dirty the pages?

Since part of our fs resembles nfs, we use
__set_page_dirty_nobuffers() for
address_space_operations->set_page_dirty, the same way nfs works.

> Are you sure you don't have a refcount leak?

One of my partners have already checked the page flags and reference
count. They seem to be all right.
Maybe we should check them again.

Re: OOM problem caused by fs

[Tao Ma]

Hi Seth,

On 06/08/2010 01:34 AM, Seth Huang wrote:
>> Are you using set_page_dirty to dirty the pages?
>
> Since part of our fs resembles nfs, we use
> __set_page_dirty_nobuffers() for
> address_space_operations->set_page_dirty, the same way nfs works.
>
>> Are you sure you don't have a refcount leak?
>
> One of my partners have already checked the page flags and reference
> count. They seem to be all right.
> Maybe we should check them again.
Which kernel version do you use?
Have you tried to change vm.lowmem_reserve_ratio? Check
Documentation/sysctl/vm.txt for how to use it.

Regards,
Tao

Re: OOM problem caused by fs

[Seth Huang]

On Wed, Jun 9, 2010 at 12:01 PM, Tao Ma <tao.ma@oracle.com> wrote:
> Hi Seth,
>
> Which kernel version do you use?
> Have you tried to change vm.lowmem_reserve_ratio? Check
> Documentation/sysctl/vm.txt for how to use it.

After re-check the reference count of cache pages, we do found leaks
as suggested by Nick.

Thank you all the same, Tao!

Seth