From: Junxiao Bi <junxiao.bi@oracle.com>
To: ocfs2-devel@oss.oracle.com
Subject: [Ocfs2-devel] [ocfs2-test] all nodes hung when run multiple reflink test for v4.3
Date: Tue, 1 Dec 2015 10:55:19 +0800 [thread overview]
Message-ID: <565D0C17.8060308@oracle.com> (raw)
In-Reply-To: <565D7BCB020000F90002076A@relay2.provo.novell.com>
On 12/01/2015 10:51 AM, Gang He wrote:
> Hello Junxiao,
>
> Could you share which Linux distribution your test cases was ran on? the kernel looks very new.
I am using Oracle Linux 6. The kernel is built by me.
Thanks,
Junxiao.
>
>
> Thanks
> Gang
>
>
>
>
>>>>
>> Hi,
>>
>> When run a full ocfs2-test to kernel v4.3, all nodes hung at
>> multiple-reflink test. Does anybody ever saw this? If anybody is
>> interested in it, please let me know, i have vmcores for them.
>>
>> Node 1:
>> ======================
>> [79321.329122] INFO: task multi_reflink_t:24205 blocked for more than
>> 120 seconds.
>> [79321.335057] Tainted: G OE 4.3.0 #3
>> [79321.345968] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [79321.357346] multi_reflink_t D ffff88007f416980 0 24205 24199
>> 0x00000080
>> [79321.363623] ffff88003ffdb868 0000000000000086 ffffffff81a25500
>> ffff88007c6bab00
>> [79321.371393] ffff88006a2bcc6c ffff88007af5b370 ffff88007af5b2c0
>> ffff880071885a80
>> [79321.376586] ffff88003ffdb848 ffffffffa057a89d ffff88007af5b2c0
>> 0000000000000000
>> [79321.380613] Call Trace:
>> [79321.381942] [<ffffffffa057a89d>] ? dlm_kick_thread+0x7d/0xa0 [ocfs2_dlm]
>> [79321.385431] [<ffffffff816a6d1e>] schedule+0x3e/0x80
>> [79321.388026] [<ffffffff816a9778>] schedule_timeout+0x1c8/0x220
>> [79321.391093] [<ffffffffa058ceda>] ? dlmlock+0x9a/0x8b0 [ocfs2_dlm]
>> [79321.394258] [<ffffffff810c5f41>] ?
>> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>> [79321.398390] [<ffffffff816a7cce>] wait_for_completion+0xde/0x110
>> [79321.401481] [<ffffffff810a81b0>] ? try_to_wake_up+0x240/0x240
>> [79321.405127] [<ffffffffa066f65d>] __ocfs2_cluster_lock+0x20d/0x720
>> [ocfs2]
>> [79321.409042] [<ffffffff8112e8f7>] ? delayacct_end+0x67/0x80
>> [79321.412046] [<ffffffff810c5f41>] ?
>> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>> [79321.416241] [<ffffffffa0674841>]
>> ocfs2_inode_lock_full_nested+0x181/0x400 [ocfs2]
>> [79321.420159] [<ffffffffa068b24f>] ?
>> ocfs2_mv_orphaned_inode_to_new+0xbf/0x7c0 [ocfs2]
>> [79321.424161] [<ffffffffa068b24f>]
>> ocfs2_mv_orphaned_inode_to_new+0xbf/0x7c0 [ocfs2]
>> [79321.428021] [<ffffffffa0674153>] ? ocfs2_rw_unlock+0x123/0x160 [ocfs2]
>> [79321.431353] [<ffffffffa069aef2>] ocfs2_reflink+0x1b2/0x480 [ocfs2]
>> [79321.434551] [<ffffffffa069b305>] ocfs2_vfs_reflink+0x145/0x1e0 [ocfs2]
>> [79321.437988] [<ffffffffa069b4f3>] ocfs2_reflink_ioctl+0x153/0x1b0 [ocfs2]
>> [79321.441549] [<ffffffff810c5f41>] ?
>> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>> [79321.445733] [<ffffffffa06817a8>] ocfs2_ioctl+0x1f8/0x400 [ocfs2]
>> [79321.449014] [<ffffffff812066d9>] ? do_filp_open+0x99/0xe0
>> [79321.451962] [<ffffffff81212d62>] ? __fd_install+0x32/0xf0
>> [79321.454811] [<ffffffff81209083>] do_vfs_ioctl+0x73/0x380
>> [79321.457601] [<ffffffff81003596>] ? do_audit_syscall_entry+0x66/0x70
>> [79321.460838] [<ffffffff81209422>] SyS_ioctl+0x92/0xa0
>> [79321.463489] [<ffffffff816aa6ee>] entry_SYSCALL_64_fastpath+0x12/0x71
>> [79321.466827] INFO: task multi_reflink_t:24206 blocked for more than
>> 120 seconds.
>> [79321.470759] Tainted: G OE 4.3.0 #3
>> [79321.473627] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [79321.478804] multi_reflink_t D ffff88007f456980 0 24206 24199
>> 0x00000080
>> [79321.483813] ffff88007afe3bf8 0000000000000086 ffff88007c502b00
>> ffff880037b02b00
>> [79321.488384] ffff88007afe3ca8 ffffffff81204b00 ffff88007afe3d68
>> 000000007afe3c18
>> [79321.493049] 0000000000000000 0000000000000000 ffff880037b02b00
>> ffff8800174c3042
>> [79321.497531] Call Trace:
>> [79321.498830] [<ffffffff81204b00>] ? filename_parentat+0x100/0x170
>> [79321.501924] [<ffffffff816a6d1e>] schedule+0x3e/0x80
>> [79321.504452] [<ffffffff816a6f4e>] schedule_preempt_disabled+0xe/0x10
>> [79321.507794] [<ffffffff816a889c>] __mutex_lock_slowpath+0x8c/0x100
>> [79321.511024] [<ffffffff816a8933>] mutex_lock+0x23/0x40
>> [79321.513752] [<ffffffff81204bed>] filename_create+0x7d/0x150
>> [79321.516747] [<ffffffff81204d44>] user_path_create+0x34/0x50
>> [79321.519687] [<ffffffffa069b476>] ocfs2_reflink_ioctl+0xd6/0x1b0 [ocfs2]
>> [79321.523162] [<ffffffff810c5f41>] ?
>> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>> [79321.527211] [<ffffffffa06817a8>] ocfs2_ioctl+0x1f8/0x400 [ocfs2]
>> [79321.530377] [<ffffffff812066d9>] ? do_filp_open+0x99/0xe0
>> [79321.533197] [<ffffffff81212d62>] ? __fd_install+0x32/0xf0
>> [79321.536018] [<ffffffff81209083>] do_vfs_ioctl+0x73/0x380
>> [79321.538928] [<ffffffff81003596>] ? do_audit_syscall_entry+0x66/0x70
>> [79321.542737] [<ffffffff81209422>] SyS_ioctl+0x92/0xa0
>> [79321.545395] [<ffffffff816aa6ee>] entry_SYSCALL_64_fastpath+0x12/0x71
>>
>> =====================================================
>>
>> Node 2:
>> =====================================================
>> [79682.381129] INFO: task multi_reflink_t:11279 blocked for more than
>> 120 seconds.
>> [79682.387929] Tainted: G OE 4.3.0 #1
>> [79682.393352] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [79682.400733] multi_reflink_t D ffff88003fc16980 0 11279 11278
>> 0x00000080
>> [79682.408206] ffff88003ba475f8 0000000000000086 ffffffff81a25500
>> ffff88003c8d4080
>> [79682.417174] ffff88003ba475c8 ffffffff8117992f ffffea00004b9cc0
>> ffff88003fc16980
>> [79682.422624] 7fffffffffffffff 0000000000000000 0000000000000001
>> ffffea00004b9cc0
>> [79682.427041] Call Trace:
>> [79682.428426] [<ffffffff8117992f>] ? find_get_entry+0x2f/0xc0
>> [79682.433074] [<ffffffff816a68fe>] schedule+0x3e/0x80
>> [79682.437402] [<ffffffff816a9358>] schedule_timeout+0x1c8/0x220
>> [79682.444401] [<ffffffffa067eee4>] ?
>> ocfs2_inode_cache_unlock+0x14/0x20 [ocfs2]
>> [79682.450495] [<ffffffffa06bb1e9>] ?
>> ocfs2_metadata_cache_unlock+0x19/0x30 [ocfs2]
>> [79682.455303] [<ffffffffa06bb399>] ? ocfs2_buffer_cached+0x99/0x170
>> [ocfs2]
>> [79682.459708] [<ffffffffa067eee4>] ?
>> ocfs2_inode_cache_unlock+0x14/0x20 [ocfs2]
>> [79682.464037] [<ffffffffa06bb1e9>] ?
>> ocfs2_metadata_cache_unlock+0x19/0x30 [ocfs2]
>> [79682.468109] [<ffffffff810c5f41>] ?
>> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>> [79682.472386] [<ffffffff816a78ae>] wait_for_completion+0xde/0x110
>> [79682.475510] [<ffffffff810a81b0>] ? try_to_wake_up+0x240/0x240
>> [79682.478685] [<ffffffffa066f65d>] __ocfs2_cluster_lock+0x20d/0x720
>> [ocfs2]
>> [79682.482821] [<ffffffff810c5f41>] ?
>> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>> [79682.487010] [<ffffffffa0674841>]
>> ocfs2_inode_lock_full_nested+0x181/0x400 [ocfs2]
>> [79682.490913] [<ffffffffa06d0db3>] ? ocfs2_iop_get_acl+0x53/0x113 [ocfs2]
>> [79682.494435] [<ffffffff81210cd2>] ? igrab+0x42/0x70
>> [79682.496977] [<ffffffffa06d0db3>] ocfs2_iop_get_acl+0x53/0x113 [ocfs2]
>> [79682.500353] [<ffffffff81254583>] get_acl+0x53/0x70
>> [79682.502912] [<ffffffff81254923>] posix_acl_create+0x73/0x130
>> [79682.505869] [<ffffffffa068f0bf>] ocfs2_mknod+0x7cf/0x1140 [ocfs2]
>> [79682.509043] [<ffffffffa068fba2>] ocfs2_create+0x62/0x110 [ocfs2]
>> [79682.512160] [<ffffffff8120be25>] ? __d_alloc+0x65/0x190
>> [79682.514878] [<ffffffff81201b3e>] ? __inode_permission+0x4e/0xd0
>> [79682.517933] [<ffffffff81202cf5>] vfs_create+0xd5/0x100
>> [79682.520641] [<ffffffff812009ed>] ? lookup_real+0x1d/0x60
>> [79682.523421] [<ffffffff81203a03>] lookup_open+0x173/0x1a0
>> [79682.526202] [<ffffffff810c59c6>] ? percpu_down_read+0x16/0x70
>> [79682.529199] [<ffffffff81205fea>] do_last+0x31a/0x830
>> [79682.531813] [<ffffffff81201b3e>] ? __inode_permission+0x4e/0xd0
>> [79682.534926] [<ffffffff81201bd8>] ? inode_permission+0x18/0x50
>> [79682.538376] [<ffffffff812046b0>] ? link_path_walk+0x290/0x550
>> [79682.541724] [<ffffffff8120657c>] path_openat+0x7c/0x140
>> [79682.544699] [<ffffffff812066c5>] do_filp_open+0x85/0xe0
>> [79682.547536] [<ffffffff8120190f>] ? getname_flags+0x7f/0x1f0
>> [79682.550459] [<ffffffff811f613a>] do_sys_open+0x11a/0x220
>> [79682.553238] [<ffffffff8100374b>] ?
>> syscall_trace_enter_phase1+0x15b/0x170
>> [79682.556745] [<ffffffff811f627e>] SyS_open+0x1e/0x20
>> [79682.559317] [<ffffffff816aa2ae>] entry_SYSCALL_64_fastpath+0x12/0x71
>>
>> ============================================
>>
>> Node 3:
>> ============================================
>>
>> [79682.135120] INFO: task multi_reflink_t:11263 blocked for more than
>> 120 seconds.
>> [79682.141115] Tainted: G OE 4.3.0 #1
>> [79682.147279] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [79682.164553] multi_reflink_t D ffff88003ec16980 0 11263 11262
>> 0x00000080
>> [79682.180223] ffff88003c2db698 0000000000000082 ffffffff81a25500
>> ffff88003b8fc080
>> [79682.190947] ffff88003c2db668 ffffffff810c0a73 ffffffffa025de04
>> ffff88003c2f92c0
>> [79682.199118] ffff88003b9d3f00 ffff88003c2db6b0 ffff88003c2f9370
>> 0000000000000000
>> [79682.207337] Call Trace:
>> [79682.209986] [<ffffffff810c0a73>] ? __wake_up+0x53/0x70
>> [79682.215598] [<ffffffffa025de04>] ?
>> o2net_send_message_vec+0x154/0x900 [ocfs2_nodemanager]
>> [79682.224433] [<ffffffff816a68fe>] schedule+0x3e/0x80
>> [79682.229756] [<ffffffff816a9358>] schedule_timeout+0x1c8/0x220
>> [79682.236005] [<ffffffffa058ceda>] ? dlmlock+0x9a/0x8b0 [ocfs2_dlm]
>> [79682.242615] [<ffffffff810a677a>] ? finish_task_switch+0x7a/0x200
>> [79682.249146] [<ffffffff810c5f41>] ?
>> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>> [79682.257676] [<ffffffffa025de04>] ?
>> o2net_send_message_vec+0x154/0x900 [ocfs2_nodemanager]
>> [79682.266506] [<ffffffff816a78ae>] wait_for_completion+0xde/0x110
>> [79682.272914] [<ffffffff810a81b0>] ? try_to_wake_up+0x240/0x240
>> [79682.279173] [<ffffffffa066f65d>] __ocfs2_cluster_lock+0x20d/0x720
>> [ocfs2]
>> [79682.286557] [<ffffffff816a68fe>] ? schedule+0x3e/0x80
>> [79682.292054] [<ffffffff810c5f41>] ?
>> __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
>> [79682.300603] [<ffffffffa0674841>]
>> ocfs2_inode_lock_full_nested+0x181/0x400 [ocfs2]
>> [79682.308835] [<ffffffffa068eafe>] ? ocfs2_mknod+0x20e/0x1140 [ocfs2]
>> [79682.315667] [<ffffffffa068eafe>] ocfs2_mknod+0x20e/0x1140 [ocfs2]
>> [79682.322302] [<ffffffffa068fba2>] ocfs2_create+0x62/0x110 [ocfs2]
>> [79682.328885] [<ffffffff8120be25>] ? __d_alloc+0x65/0x190
>> [79682.334580] [<ffffffff81201b3e>] ? __inode_permission+0x4e/0xd0
>> [79682.340972] [<ffffffff81202cf5>] vfs_create+0xd5/0x100
>> [79682.346595] [<ffffffff812009ed>] ? lookup_real+0x1d/0x60
>> [79682.352381] [<ffffffff81203a03>] lookup_open+0x173/0x1a0
>> [79682.358183] [<ffffffff810c59c6>] ? percpu_down_read+0x16/0x70
>> [79682.364433] [<ffffffff81205fea>] do_last+0x31a/0x830
>> [79682.369837] [<ffffffff81201b3e>] ? __inode_permission+0x4e/0xd0
>> [79682.376324] [<ffffffff81201bd8>] ? inode_permission+0x18/0x50
>> [79682.382610] [<ffffffff812046b0>] ? link_path_walk+0x290/0x550
>> [79682.388855] [<ffffffff8120657c>] path_openat+0x7c/0x140
>> [79682.394539] [<ffffffff812066c5>] do_filp_open+0x85/0xe0
>> [79682.400226] [<ffffffff8120190f>] ? getname_flags+0x7f/0x1f0
>> [79682.406273] [<ffffffff811f613a>] do_sys_open+0x11a/0x220
>> [79682.412060] [<ffffffff8100374b>] ?
>> syscall_trace_enter_phase1+0x15b/0x170
>> [79682.419401] [<ffffffff811f627e>] SyS_open+0x1e/0x20
>> [79682.424729] [<ffffffff816aa2ae>] entry_SYSCALL_64_fastpath+0x12/0x71
>>
>> =======================================================
>>
>> Thanks,
>> Junxiao.
>>
>> _______________________________________________
>> Ocfs2-devel mailing list
>> Ocfs2-devel at oss.oracle.com
>> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
prev parent reply other threads:[~2015-12-01 2:55 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-01 2:03 [Ocfs2-devel] [ocfs2-test] all nodes hung when run multiple reflink test for v4.3 Junxiao Bi
2015-12-01 2:51 ` Gang He
2015-12-01 2:55 ` Junxiao Bi [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=565D0C17.8060308@oracle.com \
--to=junxiao.bi@oracle.com \
--cc=ocfs2-devel@oss.oracle.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.