OOM problem caused by fs

[Seth Huang <seth.hg@gmail.com>]

Hello everyone,

Our group is developing a new file system for linux and we got stuck
with out-of-memory problem.

When creating large files in our fs, the system will run out of
memory(I mean the kernel starts to dump memory usage repeatedly and
the oom killer begins to kill processes) as long as the amount of data
exceeds the capacity of free memory, even if the kernel is flushing
out dirty pages.

If i'm right, when available memory is low, the writes will be blocked
in page cache allocation until some dirty pages are cleaned. I've
checked pdflush, it works fine in our system, which means dirty pages
can be flushed out and cleaned in time. However, it still crashes the
system. I've no idea how could this happen.

Has anyone experienced the same thing? Any advices will be appreciated.

Thanks,
Seth