ext3 umount hangs

ext3 umount hangs

[Steven Pratt]

Has anyone else seen hangs trying to umount ext3 volumes?  I am seeing 
this repeatedly after running tiobench on an ext3 volume.  This was only 
showing up in the mm tree, but as of 2.5.72-bk2 I am now seeing it in 
the mainline.

Steve

Re: ext3 umount hangs

[Andrew Morton]

Steven Pratt <slpratt@austin.ibm.com> wrote:
>
> Has anyone else seen hangs trying to umount ext3 volumes?

Nope.

>  I am seeing 
> this repeatedly after running tiobench on an ext3 volume.  This was only 
> showing up in the mm tree, but as of 2.5.72-bk2 I am now seeing it in 
> the mainline.

It would have been nice to have heard about it before this...

Could you please debug it a bit?  sysrq-T, etc?

Does the system hang, or does the umount hang?

Re: ext3 umount hangs

[Steven Pratt]

Andrew Morton wrote:

>Steven Pratt <slpratt@austin.ibm.com> wrote:
>  
>
>>Has anyone else seen hangs trying to umount ext3 volumes?
>>    
>>
>
>Nope.
>
>> I am seeing 
>>this repeatedly after running tiobench on an ext3 volume.  This was only 
>>showing up in the mm tree, but as of 2.5.72-bk2 I am now seeing it in 
>>the mainline.
>>    
>>
>
>It would have been nice to have heard about it before this...
>  
>
Sorry, lots of thing on my plate.

>Could you please debug it a bit?  sysrq-T, etc?
>
Yes, I will try to get this as soon as possible.

>Does the system hang, or does the umount hang?
>
Just the umount hangs, but it is severe enough to not be able to 
shutdown the system.  Have to power off.

Steve

>  
>

Re: ext3 umount hangs

[Steven Pratt]

Andrew Morton wrote:

>Steven Pratt <slpratt@austin.ibm.com> wrote:
>  
>
>>Has anyone else seen hangs trying to umount ext3 volumes?
>>    
>>
>Could you please debug it a bit?  sysrq-T, etc?
>
Here is the trace of the hung process:

umount        D 00000001 290213268 18747  18746                     (NOTLB)
Call Trace:
 [<c01a1ae8>] journal_kill_thread+0xa8/0xe0
 [<c011c780>] default_wake_function+0x0/0x30
 [<c016d8c5>] destroy_inode+0x35/0x60
 [<c01a2b80>] journal_destroy+0x10/0x1c0
 [<c019a97e>] ext3_xattr_put_super+0x1e/0x30
 [<c0196209>] ext3_put_super+0x29/0x1a0
 [<c015b64b>] generic_shutdown_super+0x16b/0x180
 [<c015c03d>] kill_block_super+0x1d/0x50
 [<c015b3fd>] deactivate_super+0x5d/0x90
 [<c0170fef>] sys_umount+0x3f/0xa0
 [<c0171067>] sys_oldumount+0x17/0x20
 [<c010afaf>] syscall_call+0x7/0xb


Here is a link to the console log with full sysrq trace:

http://www-124.ibm.com/developerworks/opensource/linuxperf/logs/ext3umount

Steve

Re: ext3 umount hangs

[Andrew Morton]

Steven Pratt <slpratt@austin.ibm.com> wrote:
>
> Here is the trace of the hung process:
> 
>  umount        D 00000001 290213268 18747  18746                     (NOTLB)
>  Call Trace:
>   [<c01a1ae8>] journal_kill_thread+0xa8/0xe0

whoops.  I bet you're seeing this when using some script which does the
unmount.

Might this help?


diff -puN fs/jbd/journal.c~kjournald-shutdown-fix fs/jbd/journal.c
--- 25/fs/jbd/journal.c~kjournald-shutdown-fix	2003-06-19 12:58:43.000000000 -0700
+++ 25-akpm/fs/jbd/journal.c	2003-06-19 12:58:43.000000000 -0700
@@ -161,7 +161,7 @@ loop:
 		del_timer_sync(journal->j_commit_timer);
 		journal_commit_transaction(journal);
 		spin_lock(&journal->j_state_lock);
-		goto loop;
+		goto end_loop;
 	}
 
 	wake_up(&journal->j_wait_done_commit);
@@ -210,7 +210,7 @@ loop:
 		journal->j_commit_request = transaction->t_tid;
 		jbd_debug(1, "woke because of timeout\n");
 	}
-
+end_loop:
 	if (!(journal->j_flags & JFS_UNMOUNT))
 		goto loop;
 
@@ -230,12 +230,16 @@ static void journal_start_thread(journal
 
 static void journal_kill_thread(journal_t *journal)
 {
+	spin_lock(&journal->j_state_lock);
 	journal->j_flags |= JFS_UNMOUNT;
 
 	while (journal->j_task) {
 		wake_up(&journal->j_wait_commit);
+		spin_unlock(&journal->j_state_lock);
 		wait_event(journal->j_wait_done_commit, journal->j_task == 0);
+		spin_lock(&journal->j_state_lock);
 	}
+	spin_unlock(&journal->j_state_lock);
 }
 
 /*

_

Re: ext3 umount hangs

[Steven Pratt]

Andrew Morton wrote:

>Steven Pratt <slpratt@austin.ibm.com> wrote:
>  
>
>>Here is the trace of the hung process:
>>
>> umount        D 00000001 290213268 18747  18746                     (NOTLB)
>> Call Trace:
>>  [<c01a1ae8>] journal_kill_thread+0xa8/0xe0
>>    
>>
>
>whoops.  I bet you're seeing this when using some script which does the
>unmount.
>  
>
Yes, I was.  :-)

>Might this help?
>  
>
Will try this tonight.  Should have an answer tomorrow.

Thanks for the quick response.
Steve

Re: ext3 umount hangs

[Steven Pratt]

Andrew Morton wrote:

>Steven Pratt <slpratt@austin.ibm.com> wrote:
>  
>
>>Here is the trace of the hung process:
>>
>> umount        D 00000001 290213268 18747  18746                     (NOTLB)
>> Call Trace:
>>  [<c01a1ae8>] journal_kill_thread+0xa8/0xe0
>>    
>>
>
>whoops.  I bet you're seeing this when using some script which does the
>unmount.
>
>Might this help?
>  
>
Yes, this fixed the problem. Thanks.

Steve