XFS crashes with shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to delete nr

XFS crashes with shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to delete nr

[Stefan Priebe - Profihost AG]

Hi guys,

we're seeing a really bad behaviour on one of our machines running 
vanilla 2.6.32.40 kernel.

It freezes from time to time or processes starts to hang. At the same 
time the following message appears in the kernel log:

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-274207938304

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549059303168

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549500554116

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549360922112

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549407219078

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549480231300

Any ideas? An xfs_repair says everything is fine.

-- 
Regards,
   Stefan Priebe

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

XFS crashes with shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to delete nr

[Stefan Priebe - Profihost AG]

Hi,

please HELP ME! We're seeing a really bad behaviour on one of our 
machines running vanilla 2.6.32.40 kernel.

It freezes from time to time or processes starts to hang. At the same 
time the following message appears in the kernel log:

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-274207938304

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549059303168

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549500554116

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549360922112

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549407219078

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549480231300

Any ideas? An xfs_repair says everything is fine.

-- 
Regards,
   Stefan Priebe

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

XFS crashes with shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to delete nr

[Stefan Priebe - Profihost AG]

Hi guys,

we're seeing a really bad behaviour on one of our machines running 
vanilla 2.6.32.40 kernel.

It freezes from time to time or processes starts to hang. At the same 
time the following message appears in the kernel log:

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-274207938304

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549059303168

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549500554116

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549360922112

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549407219078

shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to 
delete nr=-549480231300

Any ideas? An xfs_repair says everything is fine.

-- 
Regards,
   Stefan Priebe

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS crashes with shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to delete nr

[Dave Chinner]

On Wed, Jun 01, 2011 at 12:54:55PM +0200, Stefan Priebe - Profihost AG wrote:
> Hi guys,
> 
> we're seeing a really bad behaviour on one of our machines running
> vanilla 2.6.32.40 kernel.
> 
> It freezes from time to time or processes starts to hang. At the
> same time the following message appears in the kernel log:

Perhaps 2.6.32.40 needs this patch:

commit 081003fff467ea0e727f66d5d435b4f473a789b3
Author: Johannes Weiner <hannes@cmpxchg.org>
Date:   Fri Oct 1 07:43:54 2010 +0000

    xfs: properly account for reclaimed inodes

    When marking an inode reclaimable, a per-AG counter is increased, the
    inode is tagged reclaimable in its per-AG tree, and, when this is the
    first reclaimable inode in the AG, the AG entry in the per-mount tree
    is also tagged.

    When an inode is finally reclaimed, however, it is only deleted from
    the per-AG tree.  Neither the counter is decreased, nor is the parent
    tree's AG entry untagged properly.

    Since the tags in the per-mount tree are not cleared, the inode
    shrinker iterates over all AGs that have had reclaimable inodes at one
    point in time.

    The counters on the other hand signal an increasing amount of slab
    objects to reclaim.  Since "70e60ce xfs: convert inode shrinker to
    per-filesystem context" this is not a real issue anymore because the
    shrinker bails out after one iteration.

    But the problem was observable on a machine running v2.6.34, where the
    reclaimable work increased and each process going into direct reclaim
    eventually got stuck on the xfs inode shrinking path, trying to scan
    several million objects.

    Fix this by properly unwinding the reclaimable-state tracking of an
    inode when it is reclaimed.

    Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
    Cc: stable@kernel.org
    Reviewed-by: Dave Chinner <dchinner@redhat.com>
    Signed-off-by: Alex Elder <aelder@sgi.com>

> 
> shrink_slab: xfs_reclaim_inode_shrink+0x0/0x10d negative objects to
> delete nr=-274207938304

That's an error messge that was introduced in 2.6.34, and the above
patch was introduced in 2.6.36. Obvious a bug has been backported to
2.6.32, but was the fix? It was clearly marked for stable kernels,
but I have no I have no idea if the stable kernel folks pushed it
back to .32. I really don't have the time to track what fixes were
or were not backported to what kernels because there are too many
"long term stable" kernels in existance now.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs