From: Marc MERLIN <marc@merlins.org>
To: Josef Bacik <jbacik@fb.com>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: btrfs on 3.14rc5 stuck on "btrfs_tree_read_lock sync"
Date: Mon, 7 Apr 2014 11:51:21 -0700 [thread overview]
Message-ID: <20140407185121.GI10222@merlins.org> (raw)
In-Reply-To: <5342CE0C.3070707@fb.com>
On Mon, Apr 07, 2014 at 12:10:52PM -0400, Josef Bacik wrote:
> On 04/07/2014 12:05 PM, Marc MERLIN wrote:
> >I was debugging my why backup failed to run, and eventually found it was
> >stuck on sync:
> >14080 18:18 btrfs_tree_read_lock sync
> >
> >This was hung for hours on this lock.
> >
> >Strangely, it looks like taking my sysrq-w hung the machine pretty hard for
> >close to 30sec, but this seems to have unhung sync and in the end btrfs send
> >completed after that.
> >
> >Sysqrq-w is here:
> >https://urldefense.proofpoint.com/v1/url?u=http://marc.merlins.org/tmp/sysrq-btrfs-sync-hang.txt&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=cKCbChRKsMpTX8ybrSkonQ%3D%3D%0A&m=IHXWC1Chbc0jEiUWu1v4Va9NOphtjPbjYp6yVMdUmXM%3D%0A&s=bd787a3422e9ff0972d2d09de7d424f56589aadc9d6db33e19fc44886dce604f
>
> Try Chris's integration branch in a few hours and see if that fixes
> it. Thanks,
Mmmh, so I rebooted that server with 3.14.0 (no rc), and it was
deadlocked a long time during boot (about 10mn) before it unlocked
itself and finished booting.
This is a bit vexing, I don't yet know which of my 3 btrfs filesystems
is causing this, and how to fix it.
After boot, it seems ok enough.
You're recommending that I try btrfs-next on a 3.15 pre kernel, correct?
If so would it be likely to fix my filesystem and let me go back to a
stable 3.14? (I'm a bit warry about running some unstable 3.15 on it :).
Is there a chance balance or some file system cleaning will fix this?
For now, during boot, I get:
INFO: task btrfs-transacti:3633 blocked for more than 120 seconds.
Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
btrfs-transacti D ffff88020d762680 0 3633 2 0x00000000
ffff88020c6c7dc0 0000000000000046 ffff88020c6c7fd8 ffff88020d762150
00000000000141c0 ffff88020d762150 ffff88020e11be90 ffff8802106271e8
0000000000000000 ffff880210627000 ffff8800c5c82740 ffff88020c6c7dd0
Call Trace:
[<ffffffff8160c331>] schedule+0x73/0x75
[<ffffffff8122a5f9>] wait_current_trans.isra.15+0x98/0xf4
[<ffffffff810850c9>] ? finish_wait+0x65/0x65
[<ffffffff8122b812>] start_transaction+0x202/0x4f2
[<ffffffff8122bb9e>] btrfs_attach_transaction+0x17/0x19
[<ffffffff812277a8>] transaction_kthread+0xd6/0x1ab
[<ffffffff812276d2>] ? btrfs_cleanup_transaction+0x43f/0x43f
[<ffffffff8106bc56>] kthread+0xae/0xb6
[<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
[<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
[<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
INFO: task btrfs-transacti:3633 blocked for more than 120 seconds.
Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
btrfs-transacti D ffff88020d762680 0 3633 2 0x00000000
ffff88020c6c7dc0 0000000000000046 ffff88020c6c7fd8 ffff88020d762150
00000000000141c0 ffff88020d762150 ffff88020e11be90 ffff8802106271e8
0000000000000000 ffff880210627000 ffff8800c5c82740 ffff88020c6c7dd0
Call Trace:
[<ffffffff8160c331>] schedule+0x73/0x75
[<ffffffff8122a5f9>] wait_current_trans.isra.15+0x98/0xf4
[<ffffffff810850c9>] ? finish_wait+0x65/0x65
[<ffffffff8122b812>] start_transaction+0x202/0x4f2
[<ffffffff8122bb9e>] btrfs_attach_transaction+0x17/0x19
[<ffffffff812277a8>] transaction_kthread+0xd6/0x1ab
[<ffffffff812276d2>] ? btrfs_cleanup_transaction+0x43f/0x43f
[<ffffffff8106bc56>] kthread+0xae/0xb6
[<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
[<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
[<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
INFO: task btrfs-transacti:3633 blocked for more than 120 seconds.
Not tainted 3.14.0-rc5-amd64-i915-preempt-20140216c #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
btrfs-transacti D ffff88020d762680 0 3633 2 0x00000000
ffff88020c6c7dc0 0000000000000046 ffff88020c6c7fd8 ffff88020d762150
00000000000141c0 ffff88020d762150 ffff88020e11be90 ffff8802106271e8
0000000000000000 ffff880210627000 ffff8800c5c82740 ffff88020c6c7dd0
Call Trace:
[<ffffffff8160c331>] schedule+0x73/0x75
[<ffffffff8122a5f9>] wait_current_trans.isra.15+0x98/0xf4
[<ffffffff810850c9>] ? finish_wait+0x65/0x65
[<ffffffff8122b812>] start_transaction+0x202/0x4f2
[<ffffffff8122bb9e>] btrfs_attach_transaction+0x17/0x19
[<ffffffff812277a8>] transaction_kthread+0xd6/0x1ab
[<ffffffff812276d2>] ? btrfs_cleanup_transaction+0x43f/0x43f
[<ffffffff8106bc56>] kthread+0xae/0xb6
[<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
[<ffffffff816153fc>] ret_from_fork+0x7c/0xb0
[<ffffffff8106bba8>] ? __kthread_parkme+0x61/0x61
Eventually the boot finishes, but it hangs way too long.
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
next prev parent reply other threads:[~2014-04-07 18:51 UTC|newest]
Thread overview: 124+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-07 16:05 btrfs on 3.14rc5 stuck on "btrfs_tree_read_lock sync" Marc MERLIN
2014-04-07 16:10 ` Josef Bacik
2014-04-07 18:51 ` Marc MERLIN [this message]
2014-04-07 19:32 ` Chris Mason
2014-04-07 20:00 ` Marc MERLIN
2014-04-09 17:38 ` Marc MERLIN
2014-03-25 1:49 ` How to debug very very slow file delete? Marc MERLIN
2014-03-25 12:13 ` How to debug very very slow file delete? (btrfs on md-raid5) Martin
2014-03-25 13:57 ` Xavier Nicollet
2014-03-25 16:41 ` Marc MERLIN
2014-04-10 17:07 ` How to debug very very slow file delete? (btrfs on md-raid5 with many files, 70GB metadata) Marc MERLIN
2014-04-11 14:15 ` How to debug very very slow file delete? (btrfs on md-raid5) Chris Samuel
2014-04-11 17:23 ` Marc MERLIN
2014-04-11 18:00 ` Duncan
2014-04-11 19:15 ` Roman Mamedov
2014-04-12 20:25 ` very slow btrfs filesystem: any data needed before I wipe it? Marc MERLIN
2014-04-13 4:02 ` Duncan
2014-04-14 1:43 ` Marc MERLIN
2014-04-14 10:28 ` Duncan
2014-04-16 22:35 ` Marc MERLIN
2014-04-13 14:57 ` Marc MERLIN
2014-04-13 16:59 ` what does your btrfsck look like? Marc MERLIN
2014-04-14 2:15 ` How to debug very very slow file delete? Liu Bo
2014-04-14 2:21 ` Liu Bo
2014-06-09 23:40 ` btrfs balance crash BUG ON fs/btrfs/relocation.c:1062 or RIP build_backref_tree+0x9fc/0xcc4 Marc MERLIN
2014-06-10 0:32 ` Russell Coker
2014-06-10 4:58 ` Marc MERLIN
2014-06-14 16:21 ` Marc MERLIN
2014-06-17 18:29 ` Josef Bacik
2014-06-17 18:55 ` Marc MERLIN
2014-06-18 15:26 ` Josef Bacik
2014-06-18 20:21 ` Marc MERLIN
2014-06-19 16:12 ` Josef Bacik
2014-06-19 22:25 ` Marc MERLIN
2014-06-19 22:50 ` Josef Bacik
2014-06-20 0:53 ` Marc MERLIN
2014-06-20 15:40 ` Josef Bacik
2014-06-25 19:40 ` Marc MERLIN
2014-06-25 21:05 ` Josef Bacik
2015-05-05 21:02 ` 3.19.6: __btrfs_free_extent:5987: errno=-2 No such entry, did btrfs check --repair break it? Marc MERLIN
2015-05-06 11:04 ` Duncan
2015-05-06 17:25 ` Chris Murphy
2015-05-07 3:15 ` Duncan
2015-05-06 17:49 ` Marc MERLIN
-- strict thread matches above, loose matches on Subject: below --
2014-09-03 17:42 kernel BUG at fs/btrfs/extent-tree.c:7727! with 3.17-rc3 Tomasz Chmielewski
2014-09-03 12:04 ` kernel BUG at fs/btrfs/relocation.c:1065 in 3.14.16 to 3.17-rc3 Olivier Bonvalet
2014-09-29 14:13 ` Liu Bo
[not found] ` <20140824000720.GN3875@merlins.org>
[not found] ` <20140926214821.GX13219@merlins.org>
[not found] ` <20150502141102.GB1809@merlins.org>
[not found] ` <20150501210013.GH13624@merlins.org>
2015-04-29 23:21 ` 3.19.3, btrfs send/receive error: failed to clone extents Marc MERLIN
2015-05-02 16:30 ` 3.19.3: check tree block failed + WARNING: device 0 not present on scrub Marc MERLIN
2015-05-02 16:50 ` Christian Dysthe
2015-05-02 17:05 ` Marc MERLIN
2015-05-02 17:20 ` Christian Dysthe
2015-05-02 17:29 ` Marc MERLIN
2015-05-02 18:56 ` Christian Dysthe
2015-05-05 6:32 ` Marc MERLIN
2015-05-05 19:56 ` 3.19.6: __btrfs_free_extent:5987: errno=-2 No such entry Marc MERLIN
2014-09-08 18:04 ` kernel BUG at fs/btrfs/extent-tree.c:7727! with 3.17-rc3 Tomasz Chmielewski
2014-10-04 1:19 ` Tomasz Chmielewski
2014-04-02 8:29 [PATCH 00/27] Replace the old man page with asciidoc and man page for each btrfs subcommand Qu Wenruo
2014-04-02 8:29 ` [PATCH 01/27] btrfs-progs: Introduce asciidoc based man page and btrfs man page Qu Wenruo
2014-04-02 8:29 ` [PATCH 02/27] btrfs-progs: Convert man page for btrfs-subvolume Qu Wenruo
2014-04-02 8:29 ` [PATCH 03/27] btrfs-progs: Convert man page for filesystem subcommand Qu Wenruo
2014-04-02 8:29 ` [PATCH 04/27] btrfs-progs: Convert man page for btrfs-balance Qu Wenruo
2014-04-02 8:29 ` [PATCH 05/27] btrfs-progs: Convert man page for btrfs-device subcommand Qu Wenruo
2014-04-02 8:29 ` [PATCH 06/27] btrfs-progs: Convert man page for btrfs-scrub Qu Wenruo
2014-04-02 8:29 ` [PATCH 07/27] btrfs-progs: Convert man page for btrfs-check Qu Wenruo
2014-04-02 8:29 ` [PATCH 08/27] btrfs-progs: Convert man page for btrfs-rescue Qu Wenruo
2014-04-02 8:29 ` [PATCH 09/27] btrfs-progs: Convert man page for btrfs-inspect-internal Qu Wenruo
2014-04-02 8:29 ` [PATCH 10/27] btrfs-progs: Convert man page for btrfs-send Qu Wenruo
2014-04-02 8:29 ` [PATCH 11/27] btrfs-progs: Convert man page for btrfs-receive Qu Wenruo
2014-04-02 8:29 ` [PATCH 12/27] btrfs-progs: Convert man page for btrfs-quota Qu Wenruo
2014-04-02 8:29 ` [PATCH 13/27] btrfs-progs: Convert and enhance the man page of btrfs-qgroup Qu Wenruo
2014-04-02 8:29 ` [PATCH 14/27] btrfs-progs: Convert man page for btrfs-replace Qu Wenruo
2014-04-04 20:29 ` Marc MERLIN
2014-04-08 1:20 ` Qu Wenruo
2014-04-02 8:29 ` [PATCH 15/27] btrfs-progs: Convert man page for btrfs-dedup Qu Wenruo
2014-04-02 8:29 ` [PATCH 16/27] btrfs-progs: Convert man page for btrfsck Qu Wenruo
2014-04-02 8:29 ` [PATCH 17/27] btrfs-progs: Convert man page for btrfs-convert Qu Wenruo
2014-04-02 8:29 ` [PATCH 18/27] btrfs-progs: Convert man page for btrfs-debug-tree Qu Wenruo
2014-04-02 8:29 ` [PATCH 19/27] btrfs-progs: Convert man page for btrfs-find-root Qu Wenruo
2014-04-02 8:29 ` [PATCH 20/27] btrfs-progs: Convert man page for btrfs-image Qu Wenruo
2014-04-02 8:29 ` [PATCH 21/27] btrfs-progs: Convert man page for btrfs-map-logical Qu Wenruo
2014-04-02 8:29 ` [PATCH 22/27] btrfs-progs: Convert man page for btrfs-show-super Qu Wenruo
2014-04-02 8:29 ` [PATCH 23/27] btrfs-progs: Convert man page for btrfstune Qu Wenruo
2014-04-02 8:29 ` [PATCH 24/27] btrfs-progs: Convert man page for btrfs-zero-log Qu Wenruo
2014-04-04 18:46 ` Marc MERLIN
2014-04-05 22:00 ` cwillu
2014-04-05 22:02 ` Marc MERLIN
2014-04-05 22:03 ` Hugo Mills
2014-04-05 22:21 ` Marc MERLIN
2014-04-05 22:05 ` Marc MERLIN
2014-04-05 22:02 ` Hugo Mills
2014-04-08 1:42 ` Qu Wenruo
2014-04-11 5:54 ` Marc MERLIN
2014-04-02 8:29 ` [PATCH 25/27] btrfs-progs: Convert man page for fsck.btrfs Qu Wenruo
2014-04-02 8:29 ` [PATCH 26/27] btrfs-progs: Convert man page for mkfs.btrfs Qu Wenruo
2014-04-02 8:29 ` [PATCH 27/27] btrfs-progs: Switch to the new asciidoc Documentation Qu Wenruo
2014-04-02 13:24 ` [PATCH 00/27] Replace the old man page with asciidoc and man page for each btrfs subcommand Chris Mason
2014-04-02 14:47 ` Marc MERLIN
2014-04-03 20:33 ` Zach Brown
2014-04-02 17:29 ` David Sterba
2014-04-16 17:12 ` David Sterba
2014-04-16 17:16 ` [PATCH] btrfs-progs: doc: link btrfsck to btrfs-check David Sterba
2014-04-17 0:47 ` Qu Wenruo
2014-04-18 14:48 ` David Sterba
2014-04-30 12:14 ` WorMzy Tykashi
2014-05-05 14:57 ` David Sterba
2014-05-08 1:40 ` Qu Wenruo
2014-05-12 14:09 ` David Sterba
2014-06-03 9:38 ` WorMzy Tykashi
2014-06-03 12:19 ` David Sterba
2014-05-17 17:43 ` [PATCH 00/27] Replace the old man page with asciidoc and man page for each btrfs subcommand Hugo Mills
2014-05-17 18:22 ` Hugo Mills
2014-05-18 7:04 ` Qu Wenruo
2014-05-18 12:05 ` Hugo Mills
2014-05-18 16:02 ` Brendan Hide
2014-05-19 0:35 ` Qu Wenruo
2014-05-18 6:51 ` Qu Wenruo
2014-05-18 10:10 ` Hugo Mills
2014-05-19 13:02 ` Chris Mason
2014-05-19 14:01 ` David Sterba
2014-05-19 14:33 ` David Sterba
2014-05-20 0:34 ` Qu Wenruo
2014-05-20 11:08 ` David Sterba
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140407185121.GI10222@merlins.org \
--to=marc@merlins.org \
--cc=jbacik@fb.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).