From: Tao Ma <tao.ma@oracle.com>
To: Jeff Moyer <jmoyer@redhat.com>
Cc: axboe@kernel.dk, linux-kernel@vger.kernel.org,
"ocfs2-devel@oss.oracle.com" <ocfs2-devel@oss.oracle.com>,
linux-ext4@vger.kernel.org, vgoyal@redhat.com
Subject: Re: [PATCH 0/3 v5][RFC] ext3/4: enhance fsync performance when using CFQ
Date: Wed, 30 Jun 2010 08:31:16 +0800 [thread overview]
Message-ID: <4C2A9054.20500@oracle.com> (raw)
In-Reply-To: <x49wrthlwfk.fsf@segfault.boston.devel.redhat.com>
Hi Jeff,
On 06/29/2010 10:56 PM, Jeff Moyer wrote:
> Tao Ma<tao.ma@oracle.com> writes:
>
>> Hi Jeff,
>>
>> On 06/27/2010 09:48 PM, Jeff Moyer wrote:
>>> Tao Ma<tao.ma@oracle.com> writes:
>>>> I am sorry to say that the patch make jbd2 locked up when I tested
>>>> fs_mark using ocfs2.
>>>> I have attached the log from my netconsole server. After I reverted
>>>> the patch [3/3], the box works again.
>>>
>>> I can't reproduce this, unfortunately. Also, when building with the
>>> .config you sent me, the disassembly doesn't line up with the stack
>>> trace you posted.
>>>
>>> I'm not sure why yielding the queue would cause a deadlock. The only
>>> explanation I can come up with is that I/O is not being issued. I'm
>>> assuming that no other I/O will be completed to the file system in
>>> question. Is that right? Could you send along the output from sysrq-t?
>> yes, I just mounted it and begin the test, so there should be no
>> outstanding I/O. So do you need me to setup another disk for test?
>> I have attached the sysrq output in sysrq.log. please check.
>
> Well, if it doesn't take long to reproduce, then it might be helpful to
> see a blktrace of the run. However, it might also just be worth waiting
> for the next version of the patch to see if that fixes your issue.
>
>> btw, I also met with a NULL pointer deference in cfq_yield. I have
>> attached the null.log also. This seems to be related to the previous
>> deadlock and happens when I try to remount the same volume after
>> reboot and ocfs2 try to do some recovery.
>
> Pid: 4130, comm: ocfs2_wq Not tainted 2.6.35-rc3+ #5 0MM599/OptiPlex 745
> RIP: 0010:[<ffffffff82161537>]
> [<ffffffff82161537>] cfq_yield+0x5f/0x135
> RSP: 0018:ffff880123061c60 EFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff88012c2b5ea8 RCX: ffff88012c3a30d0
>
> ffffffff82161528: e8 69 eb ff ff callq ffffffff82160096<cfq_cic_lookup>
> ffffffff8216152d: 49 89 c6 mov %rax,%r14
> ffffffff82161530: 48 8b 85 00 06 00 00 mov 0x600(%rbp),%rax
> ffffffff82161537: f0 48 ff 00 lock incq (%rax)
>
> I'm pretty sure that's a NULL pointer deref of the tsk->iocontext that
> was passed into the yield function. I've since fixed that, so your
> recovery code should be safe in the newest version (which I've not yet
> posted).
ok, so could you please cc me when the new patches are out? It would be
easier for me to track it. Thanks.
Regards,
Tao
WARNING: multiple messages have this Message-ID (diff)
From: Tao Ma <tao.ma@oracle.com>
To: Jeff Moyer <jmoyer@redhat.com>
Cc: axboe@kernel.dk, linux-kernel@vger.kernel.org,
"ocfs2-devel@oss.oracle.com" <ocfs2-devel@oss.oracle.com>,
linux-ext4@vger.kernel.org, vgoyal@redhat.com
Subject: [Ocfs2-devel] [PATCH 0/3 v5][RFC] ext3/4: enhance fsync performance when using CFQ
Date: Wed, 30 Jun 2010 08:31:16 +0800 [thread overview]
Message-ID: <4C2A9054.20500@oracle.com> (raw)
In-Reply-To: <x49wrthlwfk.fsf@segfault.boston.devel.redhat.com>
Hi Jeff,
On 06/29/2010 10:56 PM, Jeff Moyer wrote:
> Tao Ma<tao.ma@oracle.com> writes:
>
>> Hi Jeff,
>>
>> On 06/27/2010 09:48 PM, Jeff Moyer wrote:
>>> Tao Ma<tao.ma@oracle.com> writes:
>>>> I am sorry to say that the patch make jbd2 locked up when I tested
>>>> fs_mark using ocfs2.
>>>> I have attached the log from my netconsole server. After I reverted
>>>> the patch [3/3], the box works again.
>>>
>>> I can't reproduce this, unfortunately. Also, when building with the
>>> .config you sent me, the disassembly doesn't line up with the stack
>>> trace you posted.
>>>
>>> I'm not sure why yielding the queue would cause a deadlock. The only
>>> explanation I can come up with is that I/O is not being issued. I'm
>>> assuming that no other I/O will be completed to the file system in
>>> question. Is that right? Could you send along the output from sysrq-t?
>> yes, I just mounted it and begin the test, so there should be no
>> outstanding I/O. So do you need me to setup another disk for test?
>> I have attached the sysrq output in sysrq.log. please check.
>
> Well, if it doesn't take long to reproduce, then it might be helpful to
> see a blktrace of the run. However, it might also just be worth waiting
> for the next version of the patch to see if that fixes your issue.
>
>> btw, I also met with a NULL pointer deference in cfq_yield. I have
>> attached the null.log also. This seems to be related to the previous
>> deadlock and happens when I try to remount the same volume after
>> reboot and ocfs2 try to do some recovery.
>
> Pid: 4130, comm: ocfs2_wq Not tainted 2.6.35-rc3+ #5 0MM599/OptiPlex 745
> RIP: 0010:[<ffffffff82161537>]
> [<ffffffff82161537>] cfq_yield+0x5f/0x135
> RSP: 0018:ffff880123061c60 EFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff88012c2b5ea8 RCX: ffff88012c3a30d0
>
> ffffffff82161528: e8 69 eb ff ff callq ffffffff82160096<cfq_cic_lookup>
> ffffffff8216152d: 49 89 c6 mov %rax,%r14
> ffffffff82161530: 48 8b 85 00 06 00 00 mov 0x600(%rbp),%rax
> ffffffff82161537: f0 48 ff 00 lock incq (%rax)
>
> I'm pretty sure that's a NULL pointer deref of the tsk->iocontext that
> was passed into the yield function. I've since fixed that, so your
> recovery code should be safe in the newest version (which I've not yet
> posted).
ok, so could you please cc me when the new patches are out? It would be
easier for me to track it. Thanks.
Regards,
Tao
WARNING: multiple messages have this Message-ID (diff)
From: Tao Ma <tao.ma@oracle.com>
To: Jeff Moyer <jmoyer@redhat.com>
Cc: axboe@kernel.dk, vgoyal@redhat.com, linux-kernel@vger.kernel.org,
linux-ext4@vger.kernel.org, Joel Becker <joel.becker@oracle.com>,
Sunil Mushran <sunil.mushran@oracle.com>,
"ocfs2-devel@oss.oracle.com" <ocfs2-devel@oss.oracle.com>
Subject: Re: [PATCH 0/3 v5][RFC] ext3/4: enhance fsync performance when using CFQ
Date: Wed, 30 Jun 2010 08:31:16 +0800 [thread overview]
Message-ID: <4C2A9054.20500@oracle.com> (raw)
In-Reply-To: <x49wrthlwfk.fsf@segfault.boston.devel.redhat.com>
Hi Jeff,
On 06/29/2010 10:56 PM, Jeff Moyer wrote:
> Tao Ma<tao.ma@oracle.com> writes:
>
>> Hi Jeff,
>>
>> On 06/27/2010 09:48 PM, Jeff Moyer wrote:
>>> Tao Ma<tao.ma@oracle.com> writes:
>>>> I am sorry to say that the patch make jbd2 locked up when I tested
>>>> fs_mark using ocfs2.
>>>> I have attached the log from my netconsole server. After I reverted
>>>> the patch [3/3], the box works again.
>>>
>>> I can't reproduce this, unfortunately. Also, when building with the
>>> .config you sent me, the disassembly doesn't line up with the stack
>>> trace you posted.
>>>
>>> I'm not sure why yielding the queue would cause a deadlock. The only
>>> explanation I can come up with is that I/O is not being issued. I'm
>>> assuming that no other I/O will be completed to the file system in
>>> question. Is that right? Could you send along the output from sysrq-t?
>> yes, I just mounted it and begin the test, so there should be no
>> outstanding I/O. So do you need me to setup another disk for test?
>> I have attached the sysrq output in sysrq.log. please check.
>
> Well, if it doesn't take long to reproduce, then it might be helpful to
> see a blktrace of the run. However, it might also just be worth waiting
> for the next version of the patch to see if that fixes your issue.
>
>> btw, I also met with a NULL pointer deference in cfq_yield. I have
>> attached the null.log also. This seems to be related to the previous
>> deadlock and happens when I try to remount the same volume after
>> reboot and ocfs2 try to do some recovery.
>
> Pid: 4130, comm: ocfs2_wq Not tainted 2.6.35-rc3+ #5 0MM599/OptiPlex 745
> RIP: 0010:[<ffffffff82161537>]
> [<ffffffff82161537>] cfq_yield+0x5f/0x135
> RSP: 0018:ffff880123061c60 EFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff88012c2b5ea8 RCX: ffff88012c3a30d0
>
> ffffffff82161528: e8 69 eb ff ff callq ffffffff82160096<cfq_cic_lookup>
> ffffffff8216152d: 49 89 c6 mov %rax,%r14
> ffffffff82161530: 48 8b 85 00 06 00 00 mov 0x600(%rbp),%rax
> ffffffff82161537: f0 48 ff 00 lock incq (%rax)
>
> I'm pretty sure that's a NULL pointer deref of the tsk->iocontext that
> was passed into the yield function. I've since fixed that, so your
> recovery code should be safe in the newest version (which I've not yet
> posted).
ok, so could you please cc me when the new patches are out? It would be
easier for me to track it. Thanks.
Regards,
Tao
next prev parent reply other threads:[~2010-06-30 0:31 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-06-22 21:34 [PATCH 0/3 v5][RFC] ext3/4: enhance fsync performance when using CFQ Jeff Moyer
2010-06-22 21:35 ` [PATCH 1/3] block: Implement a blk_yield function to voluntarily give up the I/O scheduler Jeff Moyer
2010-06-23 5:04 ` Andrew Morton
2010-06-23 14:50 ` Jeff Moyer
2010-06-24 0:46 ` Vivek Goyal
2010-06-25 16:51 ` Jeff Moyer
2010-06-25 18:55 ` Jens Axboe
2010-06-25 19:57 ` Jeff Moyer
2010-06-25 20:02 ` Vivek Goyal
2010-06-22 21:35 ` [PATCH 2/3] jbd: yield the device queue when waiting for commits Jeff Moyer
2010-06-22 21:35 ` [PATCH 3/3] jbd2: yield the device queue when waiting for journal commits Jeff Moyer
2010-06-22 22:13 ` [PATCH 0/3 v5][RFC] ext3/4: enhance fsync performance when using CFQ Joel Becker
2010-06-23 9:20 ` Christoph Hellwig
2010-06-23 13:03 ` Jeff Moyer
2010-06-23 9:30 ` Tao Ma
2010-06-23 13:06 ` Jeff Moyer
2010-06-24 5:54 ` Tao Ma
2010-06-24 5:54 ` Tao Ma
2010-06-24 5:54 ` [Ocfs2-devel] " Tao Ma
2010-06-24 14:56 ` Jeff Moyer
2010-06-24 14:56 ` [Ocfs2-devel] " Jeff Moyer
2010-06-27 13:48 ` Jeff Moyer
2010-06-27 13:48 ` [Ocfs2-devel] " Jeff Moyer
2010-06-28 6:41 ` Tao Ma
2010-06-28 6:41 ` Tao Ma
2010-06-28 6:41 ` [Ocfs2-devel] " Tao Ma
2010-06-28 13:58 ` Jeff Moyer
2010-06-28 13:58 ` [Ocfs2-devel] " Jeff Moyer
2010-06-28 23:16 ` Tao Ma
2010-06-28 23:16 ` Tao Ma
2010-06-28 23:16 ` [Ocfs2-devel] " Tao Ma
2010-06-29 14:56 ` Jeff Moyer
2010-06-29 14:56 ` [Ocfs2-devel] " Jeff Moyer
2010-06-30 0:31 ` Tao Ma [this message]
2010-06-30 0:31 ` Tao Ma
2010-06-30 0:31 ` [Ocfs2-devel] " Tao Ma
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4C2A9054.20500@oracle.com \
--to=tao.ma@oracle.com \
--cc=axboe@kernel.dk \
--cc=jmoyer@redhat.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=ocfs2-devel@oss.oracle.com \
--cc=vgoyal@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.