From: Steven Pratt <slpratt@austin.ibm.com>
To: Chris Mason <chris.mason@oracle.com>,
Eric Whitney <eric.whitney@hp.com>,
linux-btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: Updated performance results
Date: Wed, 16 Sep 2009 12:57:22 -0500 [thread overview]
Message-ID: <4AB12702.30601@austin.ibm.com> (raw)
In-Reply-To: <4AB1010F.70001@austin.ibm.com>
Steven Pratt wrote:
> Chris Mason wrote:
>> On Mon, Sep 14, 2009 at 04:41:48PM -0500, Steven Pratt wrote:
>>
>>> Only bit of bad news is I did get one error that crashed the system
>>> on single threaded nocow run. So that data point is missing.
>>> Output below:
>>>
>>
>> I hope I've got this fixed. If you pull from the master branch of
>> btrfs-unstable there are fixes for async thread races. The single
>> patch I sent before is included, but not enough.
>>
> Glad you said that. Keeps me from sending the email that said the
> patch didn't help :-)
>
> Steve
Well, still getting oopses even with new code.
Lots of:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] BUG: soft lockup - CPU#10
stuck for 61s! [btrfs-endio-1:30250]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Pid: 30250, comm:
btrfs-endio-1 Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RIP:
0010:[<ffffffff81153920>] [<ffffffff81153920>] crc32c+0x20/0x26
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RSP:
0018:ffff88013a857cc8 EFLAGS: 00000217
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RAX: 0000000000000040 RBX:
ffff88013a857cc8 RCX: ffff88013d8022c0
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RDX: 0000000000000010 RSI:
ffff88001d349ff0 RDI: 0000000041703e71
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] RBP: ffffffff8100c4ee R08:
0000000000000000 R09: 0000000000000000
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] R10: ffff88013a857d30 R11:
0000000000000002 R12: ffff88013a857d10
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] R13: 0000000000000002 R14:
ffff88013a857cb0 R15: ffffffff8100c38e
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] FS:
0000000000000000(0000) GS:ffff880028159000(0000) knlGS:0000000000000000
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] CS: 0010 DS: 0018 ES:
0018 CR0: 000000008005003b
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] CR2: 0000000000000043 CR3:
00000001368f7000 CR4: 00000000000006e0
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] Call Trace:
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffff8115397e>] ?
chksum_update+0x10/0x18
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffff81150084>] ?
crypto_shash_update+0x1a/0x1c
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffff81175c34>] ?
crc32c+0x4c/0x60
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffffa0391d0f>] ?
get_state_private+0x38/0x6f [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffffa0376688>] ?
btrfs_csum_data+0xd/0xf [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffffa037fefc>] ?
btrfs_readpage_end_io_hook+0x158/0x27b [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffffa0392a46>] ?
end_bio_extent_readpage+0xb8/0x1c0 [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.942754] [<ffffffff810e5733>] ?
bio_endio+0x26/0x28
Sep 16 11:07:27 btrfs1 kernel: [ 1862.947656] [<ffffffffa037666e>] ?
end_workqueue_fn+0x111/0x11e [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.947823] [<ffffffffa039a490>] ?
worker_loop+0x12a/0x3ea [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.947823] [<ffffffffa039a366>] ?
worker_loop+0x0/0x3ea [btrfs]
Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800] [<ffffffff810544e4>] ?
kthread+0x8f/0x97
Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800] [<ffffffff8100ca1a>] ?
child_rip+0xa/0x20
Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800] [<ffffffff81054455>] ?
kthread+0x0/0x97
Sep 16 11:07:27 btrfs1 kernel: [ 1862.948800] [<ffffffff8100ca10>] ?
child_rip+0x0/0x20
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Pid: 31421, comm:
btrfs-endio-wri Not tainted 2.6.31-autokern1 #1 IBM x3950-[88726RU]-
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RIP:
0010:[<ffffffffa036afb3>] [<ffffffffa036afb3>]
alloc_reserved_file_extent+0x8d/0x1c3 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RSP:
0018:ffff8800aa555af0 EFLAGS: 00010282
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RAX: 00000000ffffffef RBX:
ffff88013b55e000 RCX: 0000000000000002
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RDX: 0000000000000001 RSI:
0000000000000000 RDI: ffff88012f20a9a0
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] RBP: ffff8800aa555b60 R08:
ffff8800aa555888 R09: ffff8800aa555880
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] R10: ffff880077937400 R11:
00000000fffffffa R12: 000000000000001d
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] R13: ffff880077937400 R14:
0000000000000000 R15: 0000000000000000
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] FS:
0000000000000000(0000) GS:ffff88002804b000(0000) knlGS:0000000000000000
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] CS: 0010 DS: 0018 ES:
0018 CR0: 000000008005003b
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] CR2: 00000000007c0000 CR3:
000000013e038000 CR4: 00000000000006f0
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Process btrfs-endio-wri
(pid: 31421, threadinfo ffff8800aa554000, task ffff8801395447a0)
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Stack:
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] ffff880077937400
0000000000000a7c 0000000000000005 0000000000000000
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] <0> ffff880101d0c800
ffff8801140bbd20 000000b2aa555b60 ffffffffa036a190
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] <0> 000000350000091d
ffff8801090fdd40 ffff88013a4e9d40 0000000000000001
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] Call Trace:
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa036a190>] ?
update_reserved_extents+0xa7/0xbe [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa036f430>]
run_one_delayed_ref+0x382/0x42f [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffff8100c4ee>] ?
apic_timer_interrupt+0xe/0x20
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa03700b1>]
run_clustered_refs+0x237/0x2b4 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa03a5665>] ?
btrfs_find_ref_cluster+0xdc/0x115 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa03701da>]
btrfs_run_delayed_refs+0xac/0x195 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa0379a76>]
__btrfs_end_transaction+0x59/0xfe [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa0379b36>]
btrfs_end_transaction+0xb/0xd [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa037f29b>]
btrfs_finish_ordered_io+0x23c/0x265 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa037f2d9>]
btrfs_writepage_end_io_hook+0x15/0x17 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa0392901>]
end_bio_extent_writepage+0xa5/0x132 [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffff810e5733>]
bio_endio+0x26/0x28
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa037666e>]
end_workqueue_fn+0x111/0x11e [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa039a490>]
worker_loop+0x12a/0x3ea [btrfs]
Sep 16 11:54:47 btrfs1 kernel: [ 4703.082621] [<ffffffffa039a366>] ?
worker_loop+0x0/0x3ea [btrfs]
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] [<ffffffff810544e4>]
kthread+0x8f/0x97
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] [<ffffffff8100ca1a>]
child_rip+0xa/0x20
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] [<ffffffff81054455>] ?
kthread+0x0/0x97
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] [<ffffffff8100ca10>] ?
child_rip+0x0/0x20
Sep 16 11:54:48 btrfs1 kernel: [ 4703.082621] Code: 08 4c 8d 45 d4 41 8d
44 24 18 48 8b 73 20 48 8b 4d 18 41 b9 01 00 00 00 48 8b 7d b8 4c 89 ea
89 45 d4 e8 93 e3 ff ff 85 c0 74 04 <0f> 0b eb fe 49 63 75 40 4d 8b 65
00 49 83 cf 01 4c 89 e7 48 6b
Happened on 2 machines.
Steve
>
>> -chris
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2009-09-16 17:57 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-07-23 18:35 Updated performance results Steven Pratt
2009-07-23 21:00 ` Chris Mason
2009-07-23 22:04 ` Steven Pratt
2009-07-24 13:24 ` Chris Mason
2009-07-24 14:00 ` Chris Mason
2009-07-24 15:05 ` Steven Pratt
2009-07-28 20:12 ` Steven Pratt
2009-07-28 20:23 ` Chris Mason
2009-07-28 21:10 ` Steven Pratt
2009-08-05 20:35 ` Chris Mason
2009-08-07 7:30 ` debian developer
2009-08-07 13:56 ` Steven Pratt
2009-08-07 13:56 ` Steven Pratt
2009-08-07 23:12 ` Chris Mason
2009-08-31 17:49 ` Steven Pratt
2009-09-11 19:29 ` Chris Mason
2009-09-11 21:35 ` Steven Pratt
2009-09-14 13:51 ` Chris Mason
2009-09-14 17:20 ` Jens Axboe
2009-09-14 21:41 ` Steven Pratt
2009-09-14 23:13 ` Chris Mason
2009-09-16 0:52 ` Chris Mason
2009-09-16 15:15 ` Steven Pratt
2009-09-16 17:57 ` Steven Pratt [this message]
2009-09-16 18:07 ` Chris Mason
2009-09-16 18:15 ` Steven Pratt
2009-09-16 18:17 ` Chris Mason
2009-09-16 18:16 ` Steven Pratt
2009-09-16 18:20 ` Chris Mason
2009-09-16 18:37 ` Steven Pratt
2009-09-17 18:32 ` Eric Whitney
2009-09-17 18:39 ` Steven Pratt
2009-09-17 18:52 ` Chris Mason
2009-09-17 20:17 ` Chris Mason
2009-09-17 20:43 ` Chris Mason
2009-09-17 22:04 ` Steven Pratt
2009-09-18 20:14 ` Chris Mason
2009-09-23 15:24 ` Steven Pratt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4AB12702.30601@austin.ibm.com \
--to=slpratt@austin.ibm.com \
--cc=chris.mason@oracle.com \
--cc=eric.whitney@hp.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox