[Ocfs2-devel] ocfs2 cannot continue when JBD2 has aborted the journal,

[Ocfs2-devel] ocfs2 cannot continue when JBD2 has aborted the journal,

[Zhangguanghui]

Hi all,

A tiny race about JBD2 has aborted to jbd2_journal_flush,

because of unstable storage link and I/O stress.

while JBD2 state is aborted, have been -EIO error,

may cause all cluster nodes hung. so I thinks

JBD2 has aborted the journal, ocfs2 cannot continue and trigger ocfs2_abort.

Thanks, Any ideas about this patch?


description:

ocfs2_commit_thread
  ocfs2_commit_cache
    jbd2_journal_flush


--- journal.c 2015-12-17 11:36:39.140542941 +0800
+++ journal.c.diff 2015-12-17 11:39:21.308542922 +0800
@@ -328,6 +328,9 @@
if (status < 0) {
up_write(&journal->j_trans_barrier);
mlog_errno(status);
+ if (is_journal_aborted(journal)) {
+ ocfs2_abort(osb->sb, "Detect aborted journal,while committing cache.");
+ }
goto finally;
}
________________________________
zhangguanghui
-------------------------------------------------------------------------------------------------------------------------------------
????????????????????????????????????????
????????????????????????????????????????
????????????????????????????????????????
???
This e-mail and its attachments contain confidential information from H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20151217/29877c0d/attachment.html 

[Ocfs2-devel] ocfs2 cannot continue when JBD2 has aborted the journal,

[Joseph Qi]

Hi Guanghui,
Could you please describe the problem you encountered more specifically?
I don't think this change is in a fair way.

On 2015/12/17 13:33, Zhangguanghui wrote:
> Hi all,
> 
> A tiny race about JBD2 has aborted to jbd2_journal_flush, 
> 
> because of unstable storage link and I/O stress.
> 
> while JBD2 state is aborted, have been -EIO error,
> 
> may cause all cluster nodes hung. so I thinks 
> 
> JBD2 has aborted the journal, ocfs2 cannot continue and trigger ocfs2_abort. 
> 
> Thanks, Any ideas about this patch?
> 
> 
> description:
> 
> ocfs2_commit_thread
>   ocfs2_commit_cache
>     jbd2_journal_flush
> 
> 
> --- journal.c 2015-12-17 11:36:39.140542941 +0800
> +++ journal.c.diff 2015-12-17 11:39:21.308542922 +0800
> @@ -328,6 +328,9 @@
> if (status < 0) {
> up_write(&journal->j_trans_barrier);
> mlog_errno(status);
> + if (is_journal_aborted(journal)) {
> + ocfs2_abort(osb->sb, "Detect aborted journal,while committing cache.");
> + }
> goto finally;
> }
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
---
> zhangguanghui
> -------------------------------------------------------------------------------------------------------------------------------------
> ????????????????????????????????????????
> ????????????????????????????????????????
> ????????????????????????????????????????
> ???
> This e-mail and its attachments contain confidential information from H3C, which is
> intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
> 
> 
> _______________________________________________
> Ocfs2-devel mailing list
> Ocfs2-devel at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
> 

[Ocfs2-devel] ocfs2 cannot continue when JBD2 has aborted the journal,

[Zhangguanghui]

Hi  Joseph<mailto:joseph.qi@huawei.com>

The following locking order can cause a deadlock.
 Node  A                                                                                                           Node B                                                              Node C
     Super lock  EX
        ocfs2_commit_thread
             ocfs2_commit_cache
              jbd2_journal_flush  while  journal is aborted , have been -EIO error.
     do not wake_up(&osb->dc_event)
     do not  downconvert EX->NL

while Node B required EX lock or PR lock, may cause nodes hung.
So reset Node A,  Node B and Node C will be normal.
Thanks a lot
________________________________
zhangguanghui

From: Joseph Qi<mailto:joseph.qi@huawei.com>
Date: 2015-12-18 09:05
To: zhangguanghui 10102 (CCPL)<mailto:zhang.guanghui@h3c.com>
CC: ocfs2-devel at oss.oracle.com<mailto:ocfs2-devel@oss.oracle.com>
Subject: Re: [Ocfs2-devel] ocfs2 cannot continue when JBD2 has aborted the journal,

Hi Guanghui,
Could you please describe the problem you encountered more specifically?
I don't think this change is in a fair way.

On 2015/12/17 13:33, Zhangguanghui wrote:
> Hi all,
>
> A tiny race about JBD2 has aborted to jbd2_journal_flush,
>
> because of unstable storage link and I/O stress.
>
> while JBD2 state is aborted, have been -EIO error,
>
> may cause all cluster nodes hung. so I thinks
>
> JBD2 has aborted the journal, ocfs2 cannot continue and trigger ocfs2_abort.
>
> Thanks, Any ideas about this patch?
>
>
> description:
>
> ocfs2_commit_thread
>   ocfs2_commit_cache
>     jbd2_journal_flush
>
>
> --- journal.c 2015-12-17 11:36:39.140542941 +0800
> +++ journal.c.diff 2015-12-17 11:39:21.308542922 +0800
> @@ -328,6 +328,9 @@
> if (status < 0) {
> up_write(&journal->j_trans_barrier);
> mlog_errno(status);
> + if (is_journal_aborted(journal)) {
> + ocfs2_abort(osb->sb, "Detect aborted journal,while committing cache.");
> + }
> goto finally;
> }
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
---
> zhangguanghui
> -------------------------------------------------------------------------------------------------------------------------------------
> ????????????????????????????????????????
> ????????????????????????????????????????
> ????????????????????????????????????????
> ???
> This e-mail and its attachments contain confidential information from H3C, which is
> intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
>
>
> _______________________________________________
> Ocfs2-devel mailing list
> Ocfs2-devel at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20151218/daea7981/attachment-0001.html 

[Ocfs2-devel] ocfs2 cannot continue when JBD2 has aborted the journal,

[Ryan Ding]

Hi Guanghui,

I think I encounter an problem just like you. But it's not race case.

Every time ocfs2_commit_threadreceive an errorfrom jbd2_journal_flush 
(which may cause by disk io error), it will continue to try commit 
journal. But in this case, journal should run into abort state, so retry 
commit is useless. And even worse, the lock resource hold by this node 
can not be release, so entire cluster hung.

I have write a patch about this, and my solution is just like yours, 
will send it in another email.

Thanks,
Ryan

On 12/17/2015 01:33 PM, Zhangguanghui wrote:
> Hi all,
> A tiny race aboutJBD2has aborted to jbd2_journal_flush,
> because of unstable storagelink and I/O stress.
> whileJBD2state is aborted, have been -EIO error,
> may cause all cluster nodes hung. so I thinks
> JBD2 has aborted the journal, ocfs2 cannot continue andtriggerocfs2_abort.
> Thanks,Any ideas about this patch?
> description:
> ocfs2_commit_thread
>    ocfs2_commit_cache
>      jbd2_journal_flush
> --- journal.c 2015-12-17 11:36:39.140542941 +0800
> +++ journal.c.diff 2015-12-17 11:39:21.308542922 +0800
> @@ -328,6 +328,9 @@
> if (status < 0) {
> up_write(&journal->j_trans_barrier);
> mlog_errno(status);
> + if (is_journal_aborted(journal)) {
> + ocfs2_abort(osb->sb, "Detect aborted journal,while committing cache.");
> + }
> goto finally;
> }
> ------------------------------------------------------------------------
> zhangguanghui
> -------------------------------------------------------------------------------------------------------------------------------------
> ??????????????????????????,?????????????
> ?????????????????????(??????????????????
> ???)?????????????????,??????????????????
> ??!
> This e-mail and its attachments contain confidential information from 
> H3C, which is
> intended only for the person or entity whose address is listed above. 
> Any use of the
> information contained herein in any way (including, but not limited 
> to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the 
> intended
> recipient(s) is prohibited. If you receive this e-mail in error, 
> please notify the sender
> by phone or email immediately and delete it!
>
>
> _______________________________________________
> Ocfs2-devel mailing list
> Ocfs2-devel at oss.oracle.com
> https://oss.oracle.com/mailman/listinfo/ocfs2-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20151218/4252dcb3/attachment.html