Possible memory allocation deadlock - kernel 3.2.1

Possible memory allocation deadlock - kernel 3.2.1

[pkamala]

Hi all,

I am testing the kernel 3.2.1 (32-bit) XFS filesystem on a 1 TB LVM logical
volume. The volume currently has 3653816 files, 3655726 used inodes and 782G
free space. When the volume is mounted, quotacheck starts up and after about
5 minutes the oom-killer is triggered and starts killing processes. From the
message log, it looks like the system is running out of Normal memory. After
a while I see the following messages after which I power cycle the system:

XFS: possible memory allocation deadlock in kmem_zone_alloc (mode: 0x2d0)
XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:
0x250)

The kernel is configured for 2G/2G memory split. I tried setting
vm.overcommit_memory to 2 and vm.overcommit_ratio first to 80 and then 50
and that did not help.

The server has an Intel Quad core Xeon CPU X3450 @ 2.67GHz with 4 GB RAM.
The swap partition is 8GB in size.

Is this a known issue and if so, is a patch available? Is there a kernel
configuration parameter that can be adjusted to fix this?

Do let me know if you need additional information about the test setup or
kernel configuration.

Thanks for your help.

Priya


-- 
View this message in context: http://old.nabble.com/Possible-memory-allocation-deadlock---kernel-3.2.1-tp34049469p34049469.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Possible memory allocation deadlock - kernel 3.2.1

[Ben Myers]

Hi Priya,

On Thu, Jun 21, 2012 at 08:51:38AM -0700, pkamala wrote:
> I am testing the kernel 3.2.1 (32-bit) XFS filesystem on a 1 TB LVM logical
> volume. The volume currently has 3653816 files, 3655726 used inodes and 782G
> free space. When the volume is mounted, quotacheck starts up and after about
> 5 minutes the oom-killer is triggered and starts killing processes. From the
> message log, it looks like the system is running out of Normal memory. After
> a while I see the following messages after which I power cycle the system:
> 
> XFS: possible memory allocation deadlock in kmem_zone_alloc (mode: 0x2d0)
> XFS: possible memory allocation deadlock in xfs_buf_allocate_memory (mode:
> 0x250)

That sounds familiar.  Take a look at /proc/slabinfo and see if the
xfs_inode cache is using up all of your memory.  If so, here is a fix
for this from Dave that works by allowing inode reclaim to work during
mount:

commit 8a00ebe4cfc90eda9ecb575ba97e22021cd8cf70
Author: Dave Chinner <david@fromorbit.com>
Date:   Fri Apr 13 12:10:44 2012 +0000

    xfs: Ensure inode reclaim can run during quotacheck

There is a related fix that you will want:

commit 1307bbd2af67283131728637e9489002adb26f10
Author: Ben Myers <bpm@sgi.com>
Date:   Tue May 15 14:26:55 2012 -0500

    xfs: protect xfs_sync_worker with s_umount semaphore

The above made the 3.5 merge window.  The latter commit is functional
but we're not very happy with using the s_umount semaphore here, so it
is being replaced with something that is a bit cleaner very soon.   I
will propose them for the 3.2-stable branch.

Regards,
Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Possible memory allocation deadlock - kernel 3.2.1

[pkamala]

Hi Ben,

Thanks for the quick response. Yes, this does look like the problem I'm
seeing. slabtop shows xfs_inode using 1683004K Cache Size just before I lose
connection to the server. I'll check out the patches.

Regards,
Priya
-- 
View this message in context: http://old.nabble.com/Possible-memory-allocation-deadlock---kernel-3.2.1-tp34049469p34049926.html
Sent from the Xfs - General mailing list archive at Nabble.com.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs