Re: [xfs] Calltrace in 2.6.27 kernel]

Re: [xfs] Calltrace in 2.6.27 kernel]

[aluno3]

Hi

I ran addr2line for both vmlinux and for first call trace (2.6.27)
add2line returned:

addr2line -e ./vmlinux c029553d
fs/xfs/xfs_log.c:3528

in source:

        /*
         * If this happens during log recovery, don't worry about
         * locking; the log isn't open for business yet.
         */
        if (!log ||
            log->l_flags & XLOG_ACTIVE_RECOVERY) {
                mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
                XFS_BUF_DONE(mp->m_sb_bp);                            
//3528 line
                return 0;
        }



and for second call trace (2.6.27.5 + patch for XFS from 2.6.28rc)
add2line returned:

addr2line -e ./vmlinux c029571d
fs/xfs/xfs_log.c:3561


in source:

         /*
         * If this happens during log recovery, don't worry about
         * locking; the log isn't open for business yet.
         */
        if (!log ||
            log->l_flags & XLOG_ACTIVE_RECOVERY) {
                mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
                XFS_BUF_DONE(mp->m_sb_bp);                        //3561
line
                return 0;
        }


>
> On Thu, Nov 13, 2008 at 03:22:58PM +0100, aluno3@poczta.onet.pl wrote:
>   
>> Hi
>>
>> I tested kernel 2.6.27 with stress test using fsstress,dd,LVM and
>> snapshots. After a few hours and overflow of snapshot I got call trace:
>>
>> device-mapper: snapshots: Invalidating snapshot: Unable to allocate
>> exception.
>>     
>
> This is a message from device mapper telling your that it got an EIO
> in pending_complete()
>
>   
>> I/O error in filesystem ("dm-49") meta-data dev dm-49 block
>> 0x1d4c3b0       ("xlog_recover_do..(read#2)")
>>  error 5 buf count 8192
>> XFS: log mount/recovery failed: error 5
>> XFS: log mount failed
>>     
>
> And now XFS complains that it got this error passed up, so far so good.
>
>   
>> BUG: unable to handle kernel NULL pointer dereference at 0000002c
>> IP: [<c029553d>] xfs_log_force_umount+0x3d/0x170
>> *pdpt = 00000000345cf001 *pde = 0000000000000000
>>     
>
> But ut should not crash.  Can you run 
>
> 	addr2line -e /path/to/your/kernel xfs_log_force_umount+0x3d/0x170
>
> (you'll need a kernel with debuginfo for that)
>
>
>   

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [xfs] Calltrace in 2.6.27 kernel]

[Christoph Hellwig]

On Fri, Nov 14, 2008 at 09:49:47AM +0100, aluno3@poczta.onet.pl wrote:
> Hi
> 
> I ran addr2line for both vmlinux and for first call trace (2.6.27)
> add2line returned:
> 
> addr2line -e ./vmlinux c029553d
> fs/xfs/xfs_log.c:3528
> 
> in source:
> 
>         /*
>          * If this happens during log recovery, don't worry about
>          * locking; the log isn't open for business yet.
>          */
>         if (!log ||
>             log->l_flags & XLOG_ACTIVE_RECOVERY) {
>                 mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
>                 XFS_BUF_DONE(mp->m_sb_bp);                            
> //3528 line

Thanks a lot.  I think I figured out what happens:

 - we fail the buffer read in xlog_recover_do_inode_trans, and thus
   abort the mount.
 - but before that log recovery has started delayed writeback of inode
   buffers
 - during the mount error handling we call xfs_freesb, which NULLs
   mp->m_sb_bp after the inode has been unmounted
 - but the device close tries to flush all delayed buffers after that,
   and the inode buffer triggers the filesystem shutdown case because
   it also had an I/O error.
 - Now we try to mark the superblock buffer done, but it's not there
   anymore.

The easy fix is to just check for a NULL mp->m_sb_bp before marking it
done.  Note that this will only fix the XFS oops, but not the underling
DM issue, for which I'd ask on the device mapper list.


Index: xfs-2.6/fs/xfs/xfs_log.c
===================================================================
--- xfs-2.6.orig/fs/xfs/xfs_log.c	2008-11-14 17:28:55.000000000 +0100
+++ xfs-2.6/fs/xfs/xfs_log.c	2008-11-14 17:29:20.000000000 +0100
@@ -3525,7 +3525,8 @@ xfs_log_force_umount(
 	if (!log ||
 	    log->l_flags & XLOG_ACTIVE_RECOVERY) {
 		mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
-		XFS_BUF_DONE(mp->m_sb_bp);
+		if (mp->m_sb_bp)
+			XFS_BUF_DONE(mp->m_sb_bp);
 		return 0;
 	}
 
@@ -3546,7 +3547,8 @@ xfs_log_force_umount(
 	spin_lock(&log->l_icloglock);
 	spin_lock(&log->l_grant_lock);
 	mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
-	XFS_BUF_DONE(mp->m_sb_bp);
+	if (mp->m_sb_bp)
+		XFS_BUF_DONE(mp->m_sb_bp);
 	/*
 	 * This flag is sort of redundant because of the mount flag, but
 	 * it's good to maintain the separation between the log and the rest

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [xfs] Calltrace in 2.6.27 kernel]

[aluno3]

Hi

I tested this patch through a few days and I did not get call trace from
kernel.Will this patch be included to official kernel? Thanks and best


Christoph Hellwig wrote:
> On Fri, Nov 14, 2008 at 09:49:47AM +0100, aluno3@poczta.onet.pl wrote:
>   
>> Hi
>>
>> I ran addr2line for both vmlinux and for first call trace (2.6.27)
>> add2line returned:
>>
>> addr2line -e ./vmlinux c029553d
>> fs/xfs/xfs_log.c:3528
>>
>> in source:
>>
>>         /*
>>          * If this happens during log recovery, don't worry about
>>          * locking; the log isn't open for business yet.
>>          */
>>         if (!log ||
>>             log->l_flags & XLOG_ACTIVE_RECOVERY) {
>>                 mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
>>                 XFS_BUF_DONE(mp->m_sb_bp);                            
>> //3528 line
>>     
>
> Thanks a lot.  I think I figured out what happens:
>
>  - we fail the buffer read in xlog_recover_do_inode_trans, and thus
>    abort the mount.
>  - but before that log recovery has started delayed writeback of inode
>    buffers
>  - during the mount error handling we call xfs_freesb, which NULLs
>    mp->m_sb_bp after the inode has been unmounted
>  - but the device close tries to flush all delayed buffers after that,
>    and the inode buffer triggers the filesystem shutdown case because
>    it also had an I/O error.
>  - Now we try to mark the superblock buffer done, but it's not there
>    anymore.
>
> The easy fix is to just check for a NULL mp->m_sb_bp before marking it
> done.  Note that this will only fix the XFS oops, but not the underling
> DM issue, for which I'd ask on the device mapper list.
>
>
> Index: xfs-2.6/fs/xfs/xfs_log.c
> ===================================================================
> --- xfs-2.6.orig/fs/xfs/xfs_log.c	2008-11-14 17:28:55.000000000 +0100
> +++ xfs-2.6/fs/xfs/xfs_log.c	2008-11-14 17:29:20.000000000 +0100
> @@ -3525,7 +3525,8 @@ xfs_log_force_umount(
>  	if (!log ||
>  	    log->l_flags & XLOG_ACTIVE_RECOVERY) {
>  		mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
> -		XFS_BUF_DONE(mp->m_sb_bp);
> +		if (mp->m_sb_bp)
> +			XFS_BUF_DONE(mp->m_sb_bp);
>  		return 0;
>  	}
>  
> @@ -3546,7 +3547,8 @@ xfs_log_force_umount(
>  	spin_lock(&log->l_icloglock);
>  	spin_lock(&log->l_grant_lock);
>  	mp->m_flags |= XFS_MOUNT_FS_SHUTDOWN;
> -	XFS_BUF_DONE(mp->m_sb_bp);
> +	if (mp->m_sb_bp)
> +		XFS_BUF_DONE(mp->m_sb_bp);
>  	/*
>  	 * This flag is sort of redundant because of the mount flag, but
>  	 * it's good to maintain the separation between the log and the rest
>
>   

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs