From: Simon Kirby <sim@hostway.ca>
To: "Yan, Zheng " <yanzheng@21cn.com>
Cc: Dave Cundiff <syshackmin@gmail.com>, linux-btrfs@vger.kernel.org
Subject: Re: Intermittent no space errors
Date: Tue, 3 Aug 2010 17:24:15 -0700 [thread overview]
Message-ID: <20100804002415.GA23720@hostway.ca> (raw)
In-Reply-To: <AANLkTikP_q7zhA2J69Rai964StGtMqCrhf2hGKPA9+J7@mail.gmail.com>
On Wed, Jul 28, 2010 at 08:31:10AM +0800, Yan, Zheng wrote:
> On Wed, Jul 28, 2010 at 4:30 AM, Dave Cundiff <syshackmin@gmail.com> wrote:
> > Hello,
> >
> > I installed the git repo kernel and added some debug to the ENOSPC
> > returns. Unfortunately its still failing. If it helps any its bombing
> > out in btrfs_check_data_free_space() in extent-tree.c. Returning on
> > the ENOSPC at line 2959.
> >
> > Unfortunately that is the extent of my ability to debug a filesystem. :P
>
> This is really helpful, thank you very much.
We're seeing this too, since upgrading from 2.6.33.2 + merged old git btrfs
unstable HEAD to plain 2.6.35.
[sroot@backup01:.../.rmagic]# rm *
rm: cannot remove `WEEKLY_bar3d.png': No space left on device
rm: cannot remove `WEEKLY.html': No space left on device
rm: cannot remove `YEARLY_bar3d.png': No space left on device
rm: cannot remove `YEARLY.html': No space left on device
[sroot@backup01:.../.rmagic]# l
total 25
drwxr-xr-x 1 1037300 1037300 108 2010-08-03 18:19 ./
drwxr-xr-x 1 1037300 1037300 28 2009-11-07 05:57 ../
-rw-r--r-- 1 1037300 1037300 5720 2010-04-23 13:39 WEEKLY_bar3d.png
-rw-r--r-- 1 1037300 1037300 11882 2010-04-23 13:39 WEEKLY.html
-rw-r--r-- 1 1037300 1037300 2998 2010-04-23 13:39 YEARLY_bar3d.png
-rw-r--r-- 1 1037300 1037300 3016 2010-04-23 13:39 YEARLY.html
[sroot@backup01:.../.rmagic]# rm *
rm: cannot remove `WEEKLY_bar3d.png': No space left on device
rm: cannot remove `YEARLY.html': No space left on device
[sroot@backup01:.../.rmagic]# rm *
rm: cannot remove `WEEKLY_bar3d.png': No space left on device
rm: cannot remove `YEARLY.html': No space left on device
[sroot@backup01:.../.rmagic]# rm *
[sroot@backup01:.../.rmagic]#
[sroot@backup01:/root]# btrfs filesystem df /backup/bu001/vol04
Data: total=2.55TB, used=2.20TB
Metadata: total=230.13GB, used=183.83GB
System: total=12.00MB, used=548.00KB
[sroot@backup01:/root]# df -P /backup/bu001/vol04
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/bu001-vol04 3221225472 2742785400 478440072 86% /backup/bu001/vol04
[sroot@backup01:/root]# l /dev/mapper/bu001-vol04
brw-rw---- 1 root disk 252, 10 2010-08-03 16:02 /dev/mapper/bu001-vol04
[sroot@backup01:/root]# btrfs filesystem show /dev/dm-10
failed to read /dev/sr0
Label: none uuid: 0c55f5f4-b618-4ec2-9dbc-e3e70a901e1a
Total devices 1 FS bytes used 2.37TB
devid 1 size 3.00TB used 3.00TB path /dev/dm-10
Btrfs Btrfs v0.19
We're also seeing things like this in dmesg, which appear to be coming
from btrfs-cleaner cleaning some old snapshot:
Aug 3 18:40:45 backup01 kernel: ------------[ cut here ]------------
Aug 3 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180()
Aug 3 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950
Aug 3 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2
Aug 3 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tainted: G W 2.6.35-hw #1
Aug 3 18:40:45 backup01 kernel: Call Trace:
Aug 3 18:40:45 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block_rsv_check+0x151/0x180
Aug 3 18:40:45 backup01 kernel: [<ffffffff8104b270>] warn_slowpath_common+0x80/0xd0
Aug 3 18:40:45 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath_null+0x15/0x20
Aug 3 18:40:45 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_rsv_check+0x151/0x180
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d5ea1>] btrfs_should_end_transaction+0x61/0x90
Aug 3 18:40:45 backup01 kernel: [<ffffffff812c842d>] btrfs_drop_snapshot+0x21d/0x5f0
Aug 3 18:40:45 backup01 kernel: [<ffffffff81662d72>] ? schedule+0x3f2/0x750
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d463a>] btrfs_clean_old_snapshots+0x12a/0x160
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d0f00>] cleaner_kthread+0x160/0x190
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d0da0>] ? cleaner_kthread+0x0/0x190
Aug 3 18:40:45 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/0xb0
Aug 3 18:40:45 backup01 kernel: [<ffffffff8100aca4>] kernel_thread_helper+0x4/0x10
Aug 3 18:40:45 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0/0xb0
Aug 3 18:40:45 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thread_helper+0x0/0x10
Aug 3 18:40:45 backup01 kernel: ---[ end trace cffc4418e2c1f45d ]---
Aug 3 18:40:45 backup01 kernel: block_rsv size 16194207744 reserved 4497289216 freed 0 78598144
Aug 3 18:40:45 backup01 kernel: ------------[ cut here ]------------
Aug 3 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180()
Aug 3 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950
Aug 3 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2
Aug 3 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tainted: G W 2.6.35-hw #1
Aug 3 18:40:45 backup01 kernel: Call Trace:
Aug 3 18:40:45 backup01 kernel: [<ffffffff812f2eb0>] ? map_extent_buffer+0xb0/0xc0
Aug 3 18:40:45 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block_rsv_check+0x151/0x180
Aug 3 18:40:45 backup01 kernel: [<ffffffff8104b270>] warn_slowpath_common+0x80/0xd0
Aug 3 18:40:45 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath_null+0x15/0x20
Aug 3 18:40:45 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_rsv_check+0x151/0x180
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d56fa>] __btrfs_end_transaction+0x19a/0x220
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d578e>] btrfs_end_transaction_throttle+0xe/0x10
Aug 3 18:40:45 backup01 kernel: [<ffffffff812c84f1>] btrfs_drop_snapshot+0x2e1/0x5f0
Aug 3 18:40:45 backup01 kernel: [<ffffffff81662d72>] ? schedule+0x3f2/0x750
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d463a>] btrfs_clean_old_snapshots+0x12a/0x160
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d0f00>] cleaner_kthread+0x160/0x190
Aug 3 18:40:45 backup01 kernel: [<ffffffff812d0da0>] ? cleaner_kthread+0x0/0x190
Aug 3 18:40:45 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/0xb0
Aug 3 18:40:45 backup01 kernel: [<ffffffff8100aca4>] kernel_thread_helper+0x4/0x10
Aug 3 18:40:45 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0/0xb0
Aug 3 18:40:45 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thread_helper+0x0/0x10
Aug 3 18:40:45 backup01 kernel: ---[ end trace cffc4418e2c1f45e ]---
Aug 3 18:40:45 backup01 kernel: block_rsv size 16194207744 reserved 4497281024 freed 8192 78598144
Aug 3 18:44:44 backup01 kernel: ------------[ cut here ]------------
Aug 3 18:44:44 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180()
Aug 3 18:44:44 backup01 kernel: Hardware name: PowerEdge 1950
Aug 3 18:44:44 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2
Aug 3 18:44:44 backup01 kernel: Pid: 7526, comm: btrfs-transacti Tainted: G W 2.6.35-hw #1
Aug 3 18:44:44 backup01 kernel: Call Trace:
Aug 3 18:44:44 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block_rsv_check+0x151/0x180
Aug 3 18:44:44 backup01 kernel: [<ffffffff8104b270>] warn_slowpath_common+0x80/0xd0
Aug 3 18:44:44 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath_null+0x15/0x20
Aug 3 18:44:44 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_rsv_check+0x151/0x180
Aug 3 18:44:44 backup01 kernel: [<ffffffff812d56fa>] __btrfs_end_transaction+0x19a/0x220
Aug 3 18:44:44 backup01 kernel: [<ffffffff812d579b>] btrfs_end_transaction+0xb/0x10
Aug 3 18:44:44 backup01 kernel: [<ffffffff812d5d63>] btrfs_commit_transaction+0x5c3/0x6a0
Aug 3 18:44:44 backup01 kernel: [<ffffffff810683e0>] ? autoremove_wake_function+0x0/0x40
Aug 3 18:44:44 backup01 kernel: [<ffffffff812d0d90>] transaction_kthread+0x250/0x260
Aug 3 18:44:44 backup01 kernel: [<ffffffff812d0b40>] ? transaction_kthread+0x0/0x260
Aug 3 18:44:44 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/0xb0
Aug 3 18:44:44 backup01 kernel: [<ffffffff8100aca4>] kernel_thread_helper+0x4/0x10
Aug 3 18:44:44 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0/0xb0
Aug 3 18:44:44 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thread_helper+0x0/0x10
Aug 3 18:44:44 backup01 kernel: ---[ end trace cffc4418e2c1f45f ]---
Aug 3 18:44:44 backup01 kernel: block_rsv size 16194207744 reserved 4522696704 freed 53190656 0
I rebuilt with the #if 0 changed to #if 1 on extent-tree.c:2947:
#if 1 /* I hope we never need this code again, just in case */
ha! :) "rm" is succeeding everywhere so far, and so this path hasn't
been hit yet. Perhaps it has to fight with the btrfs-cleaner, or
something. Will post a follow-up later.
Simon-
next prev parent reply other threads:[~2010-08-04 0:24 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-07-26 21:09 Intermittent no space errors Dave Cundiff
2010-07-27 13:19 ` Yan, Zheng
2010-07-27 20:30 ` Dave Cundiff
2010-07-28 0:31 ` Yan, Zheng
2010-08-04 0:24 ` Simon Kirby [this message]
2010-08-04 11:21 ` Yan, Zheng
2010-08-09 23:25 ` Simon Kirby
2010-07-29 8:10 ` Justin Ossevoort
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100804002415.GA23720@hostway.ca \
--to=sim@hostway.ca \
--cc=linux-btrfs@vger.kernel.org \
--cc=syshackmin@gmail.com \
--cc=yanzheng@21cn.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).