From: Dan Mick <dan.mick@inktank.com>
To: Mandell Degerness <mandell@pistoncloud.com>
Cc: ceph-devel@vger.kernel.org
Subject: Re: Possible deadlock condition
Date: Mon, 18 Jun 2012 16:34:09 -0700 [thread overview]
Message-ID: <4FDFBAF1.9090109@inktank.com> (raw)
In-Reply-To: <CA+jddaM0muy1un2pEyRh6h8wnZTJbqUjPmKa95WjQS69BHQQRA@mail.gmail.com>
I don't know enough to know if there's a connection, but I do note this
prior thread that sounds kinda similar:
http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/6574
On 06/18/2012 04:08 PM, Mandell Degerness wrote:
> None of the OSDs seem to be more than 82% full. I didn't think we were
> running quite that close to the margin, but it is still far from
> actually full.
>
>
> On Mon, Jun 18, 2012 at 3:57 PM, Dan Mick<dan.mick@inktank.com> wrote:
>> Does the xfs on the OSD have plenty of free space left, or could this be an
>> allocation deadlock?
>>
>>
>> On 06/18/2012 03:17 PM, Mandell Degerness wrote:
>>>
>>> Here is, perhaps, a more useful traceback from a different run of
>>> tests that we just ran into:
>>>
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.680815] INFO: task
>>> flush-254:0:29582 blocked for more than 120 seconds.
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.681040] "echo 0>
>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.681458] flush-254:0
>>> D ffff880bd9ca2fc0 0 29582 2 0x00000000
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.681740]
>>> ffff88006e51d160 0000000000000046 0000000000000002 ffff88061b362040
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.682173]
>>> ffff88006e51d160 00000000000120c0 00000000000120c0 00000000000120c0
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.682659]
>>> ffff88006e51dfd8 00000000000120c0 00000000000120c0 ffff88006e51dfd8
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.683088] Call Trace:
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.683302]
>>> [<ffffffff81520132>] schedule+0x5a/0x5c
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.683514]
>>> [<ffffffff815203e7>] schedule_timeout+0x36/0xe3
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.683784]
>>> [<ffffffff8101e0b2>] ? physflat_send_IPI_mask+0xe/0x10
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.683999]
>>> [<ffffffff8101a237>] ? native_smp_send_reschedule+0x46/0x48
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.684219]
>>> [<ffffffff811e0071>] ? list_move_tail+0x27/0x2c
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.684432]
>>> [<ffffffff81520d13>] __down_common+0x90/0xd4
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.684708]
>>> [<ffffffff811e1120>] ? _xfs_buf_find+0x17f/0x210
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.684925]
>>> [<ffffffff81520dca>] __down+0x1d/0x1f
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.685139]
>>> [<ffffffff8105db4e>] down+0x2d/0x3d
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.685350]
>>> [<ffffffff811e0f68>] xfs_buf_lock+0x76/0xaf
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.685565]
>>> [<ffffffff811e1120>] _xfs_buf_find+0x17f/0x210
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.685836]
>>> [<ffffffff811e13b6>] xfs_buf_get+0x2a/0x177
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.686052]
>>> [<ffffffff811e19f6>] xfs_buf_read+0x1f/0xca
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.686270]
>>> [<ffffffff8122a0b7>] xfs_trans_read_buf+0x205/0x308
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.686490]
>>> [<ffffffff81205e01>] xfs_btree_read_buf_block.clone.22+0x4f/0xa7
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.687015]
>>> [<ffffffff8122a3ee>] ? xfs_trans_log_buf+0xb2/0xc1
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.687232]
>>> [<ffffffff81205edd>] xfs_btree_lookup_get_block+0x84/0xac
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.687449]
>>> [<ffffffff81208e83>] xfs_btree_lookup+0x12b/0x3dc
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.687721]
>>> [<ffffffff811f6bb2>] ? xfs_alloc_vextent+0x447/0x469
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.687939]
>>> [<ffffffff811fd171>] xfs_bmbt_lookup_eq+0x1f/0x21
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.688156]
>>> [<ffffffff811ffa88>] xfs_bmap_add_extent_delay_real+0x5b5/0xfec
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.688378]
>>> [<ffffffff810f155b>] ? kmem_cache_alloc+0x87/0xf3
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.688650]
>>> [<ffffffff81204c40>] ? xfs_bmbt_init_cursor+0x3f/0x107
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.688867]
>>> [<ffffffff81201160>] xfs_bmapi_allocate+0x1f6/0x23a
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.689084]
>>> [<ffffffff812185bd>] ? xfs_iext_bno_to_irec+0x95/0xb9
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.689301]
>>> [<ffffffff81203414>] xfs_bmapi_write+0x32d/0x5a2
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.689519]
>>> [<ffffffff811e99e4>] xfs_iomap_write_allocate+0x1a5/0x29f
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.689797]
>>> [<ffffffff811df12a>] xfs_map_blocks+0x13e/0x1dd
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.690016]
>>> [<ffffffff811dfbff>] xfs_vm_writepage+0x24e/0x410
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.690233]
>>> [<ffffffff810bde1e>] __writepage+0x17/0x30
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.690446]
>>> [<ffffffff810be6ed>] write_cache_pages+0x276/0x3c8
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.690693]
>>> [<ffffffff810bde07>] ? set_page_dirty+0x60/0x60
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.690908]
>>> [<ffffffff810be884>] generic_writepages+0x45/0x5c
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.691123]
>>> [<ffffffff811defcb>] xfs_vm_writepages+0x4d/0x54
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.691337]
>>> [<ffffffff810bf832>] do_writepages+0x21/0x2a
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.691552]
>>> [<ffffffff811218f5>] writeback_single_inode+0x12a/0x2cc
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.691800]
>>> [<ffffffff81121d92>] writeback_sb_inodes+0x174/0x215
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.692016]
>>> [<ffffffff81122185>] __writeback_inodes_wb+0x78/0xb9
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.692231]
>>> [<ffffffff811224b5>] wb_writeback+0x136/0x22a
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.692444]
>>> [<ffffffff810becd1>] ? determine_dirtyable_memory+0x1d/0x26
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.692692]
>>> [<ffffffff81122d1e>] wb_do_writeback+0x19c/0x1b7
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.692907]
>>> [<ffffffff81122dc5>] bdi_writeback_thread+0x8c/0x20f
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.693122]
>>> [<ffffffff81122d39>] ? wb_do_writeback+0x1b7/0x1b7
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.693336]
>>> [<ffffffff81122d39>] ? wb_do_writeback+0x1b7/0x1b7
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.693553]
>>> [<ffffffff8105911d>] kthread+0x82/0x8a
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.693803]
>>> [<ffffffff81523c34>] kernel_thread_helper+0x4/0x10
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.694018]
>>> [<ffffffff8105909b>] ? kthread_worker_fn+0x13b/0x13b
>>> Jun 18 17:58:51 node-172-29-0-15 kernel: [242522.694232]
>>> [<ffffffff81523c30>] ? gs_change+0xb/0xb
>>>
>>>
>>> On Mon, Jun 18, 2012 at 11:37 AM, Mandell Degerness
>>> <mandell@pistoncloud.com> wrote:
>>>>
>>>> We've been seeing random issues of apparent deadlocks. We are running
>>>> ceph 0.47 on kernel 3.2.18. OSDs are running on XFS file system.
>>>> mysqld (which ran into the particular problems in the attached kernel
>>>> log) is running on an RBD with XFS (mounted on a system which includes
>>>> OSDs). We have sync_fs, and gcc ver 4.5.3-r2. The mysqld process in
>>>> both instances returned an error to the calling process.
>>>>
>>>> Regards,
>>>> Mandell Degerness
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-06-18 23:34 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CA+jddaPMp5R0adi2sLVUWeFytDzfOjAeryXL+jPjGAk8kKqafg@mail.gmail.com>
2012-06-18 22:17 ` Possible deadlock condition Mandell Degerness
2012-06-18 22:57 ` Dan Mick
2012-06-18 23:08 ` Mandell Degerness
2012-06-18 23:34 ` Dan Mick [this message]
2012-06-20 22:34 ` Mandell Degerness
2012-06-20 22:42 ` Sage Weil
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FDFBAF1.9090109@inktank.com \
--to=dan.mick@inktank.com \
--cc=ceph-devel@vger.kernel.org \
--cc=mandell@pistoncloud.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox