* Intermittent no space errors @ 2010-07-26 21:09 Dave Cundiff 2010-07-27 13:19 ` Yan, Zheng 0 siblings, 1 reply; 8+ messages in thread From: Dave Cundiff @ 2010-07-26 21:09 UTC (permalink / raw) To: linux-btrfs Hello, On 2.6.35-rc5 I'm seeing some weird behavior under heavy IO loads. I have a backup process that fires up several rsync processes. These mirror several dozen servers to individual sub-volumes. Everyday I snapshot each sub-volume and rsync over it. The problem I'm seeing is my rsync processes are failing randomly with "No space left on device". This is a 6 Terabyte volume with plenty of free space. Mount options: /dev/sdb on /backups type btrfs (rw,max_inline=0,compress) [root@rsync1 ~]# btrfs filesystem df /backups/ Data: total=1.88TB, used=1.88TB Metadata: total=43.38GB, used=32.06GB System: total=12.00MB, used=260.00KB [root@rsync1 ~]# df /dev/sdb Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdb 5781249024 2087273084 3693975940 37% /backups They don't all fail at once. Normally I have 4-5 running at a time and 1 or 2 will drop out with a no space error. The rest continue on. I've noticed it will generally occur on ones that are in the middle of transferring a very large file. If I lighten the load to one rsync at a time it appears to happen less frequently. Any known issues I should be aware of? Thanks, -- Dave Cundiff System Administrator A2Hosting, Inc http://www.a2hosting.com ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Intermittent no space errors 2010-07-26 21:09 Intermittent no space errors Dave Cundiff @ 2010-07-27 13:19 ` Yan, Zheng 2010-07-27 20:30 ` Dave Cundiff 0 siblings, 1 reply; 8+ messages in thread From: Yan, Zheng @ 2010-07-27 13:19 UTC (permalink / raw) To: Dave Cundiff; +Cc: linux-btrfs On Tue, Jul 27, 2010 at 5:09 AM, Dave Cundiff <syshackmin@gmail.com> wr= ote: > Hello, > > On 2.6.35-rc5 I'm seeing some weird behavior under heavy IO loads. I > have a backup process that fires up several rsync processes. These > mirror several dozen servers to individual sub-volumes. Everyday I > snapshot each sub-volume and rsync over it. > > The problem I'm seeing is my rsync processes are failing randomly wit= h > "No space left on device". This is a 6 Terabyte volume with plenty of > free space. > > Mount options: > /dev/sdb on /backups type btrfs (rw,max_inline=3D0,compress) > > [root@rsync1 ~]# btrfs filesystem df /backups/ > Data: total=3D1.88TB, used=3D1.88TB > Metadata: total=3D43.38GB, used=3D32.06GB > System: total=3D12.00MB, used=3D260.00KB > > [root@rsync1 ~]# df /dev/sdb > Filesystem =A0 =A0 =A0 =A0 =A0 1K-blocks =A0 =A0 =A0Used Available Us= e% Mounted on > /dev/sdb =A0 =A0 =A0 =A0 =A0 =A0 5781249024 2087273084 3693975940 =A0= 37% /backups > > They don't all fail at once. Normally I have 4-5 running at a time an= d > 1 or 2 will drop out with a no space error. The rest continue on. I'v= e > noticed it will generally occur on ones that are in the middle of > transferring a very large file. If I lighten the load to one rsync at > a time it appears to happen less frequently. > > Any known issues I should be aware of? > Thank you for reporting this. I will dig in. Yan, Zheng -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Intermittent no space errors 2010-07-27 13:19 ` Yan, Zheng @ 2010-07-27 20:30 ` Dave Cundiff 2010-07-28 0:31 ` Yan, Zheng 2010-07-29 8:10 ` Justin Ossevoort 0 siblings, 2 replies; 8+ messages in thread From: Dave Cundiff @ 2010-07-27 20:30 UTC (permalink / raw) To: linux-btrfs Hello, I installed the git repo kernel and added some debug to the ENOSPC returns. Unfortunately its still failing. If it helps any its bombing out in btrfs_check_data_free_space() in extent-tree.c. Returning on the ENOSPC at line 2959. Unfortunately that is the extent of my ability to debug a filesystem. := P Thanks, On Tue, Jul 27, 2010 at 9:19 AM, Yan, Zheng <yanzheng@21cn.com> wrote: > On Tue, Jul 27, 2010 at 5:09 AM, Dave Cundiff <syshackmin@gmail.com> = wrote: >> Hello, >> >> On 2.6.35-rc5 I'm seeing some weird behavior under heavy IO loads. I >> have a backup process that fires up several rsync processes. These >> mirror several dozen servers to individual sub-volumes. Everyday I >> snapshot each sub-volume and rsync over it. >> >> The problem I'm seeing is my rsync processes are failing randomly wi= th >> "No space left on device". This is a 6 Terabyte volume with plenty o= f >> free space. >> >> Mount options: >> /dev/sdb on /backups type btrfs (rw,max_inline=3D0,compress) >> >> [root@rsync1 ~]# btrfs filesystem df /backups/ >> Data: total=3D1.88TB, used=3D1.88TB >> Metadata: total=3D43.38GB, used=3D32.06GB >> System: total=3D12.00MB, used=3D260.00KB >> >> [root@rsync1 ~]# df /dev/sdb >> Filesystem =A0 =A0 =A0 =A0 =A0 1K-blocks =A0 =A0 =A0Used Available U= se% Mounted on >> /dev/sdb =A0 =A0 =A0 =A0 =A0 =A0 5781249024 2087273084 3693975940 =A0= 37% /backups >> >> They don't all fail at once. Normally I have 4-5 running at a time a= nd >> 1 or 2 will drop out with a no space error. The rest continue on. I'= ve >> noticed it will generally occur on ones that are in the middle of >> transferring a very large file. If I lighten the load to one rsync a= t >> a time it appears to happen less frequently. >> >> Any known issues I should be aware of? >> > > Thank you for reporting this. I will dig in. > > Yan, Zheng > --=20 Dave Cundiff System Administrator A2Hosting, Inc http://www.a2hosting.com -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Intermittent no space errors 2010-07-27 20:30 ` Dave Cundiff @ 2010-07-28 0:31 ` Yan, Zheng 2010-08-04 0:24 ` Simon Kirby 2010-07-29 8:10 ` Justin Ossevoort 1 sibling, 1 reply; 8+ messages in thread From: Yan, Zheng @ 2010-07-28 0:31 UTC (permalink / raw) To: Dave Cundiff; +Cc: linux-btrfs On Wed, Jul 28, 2010 at 4:30 AM, Dave Cundiff <syshackmin@gmail.com> wrote: > Hello, > > I installed the git repo kernel and added some debug to the ENOSPC > returns. Unfortunately its still failing. If it helps any its bombing > out in btrfs_check_data_free_space() in extent-tree.c. Returning on > the ENOSPC at line 2959. > > Unfortunately that is the extent of my ability to debug a filesystem. :P > This is really helpful, thank you very much. Yan, Zheng ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Intermittent no space errors 2010-07-28 0:31 ` Yan, Zheng @ 2010-08-04 0:24 ` Simon Kirby 2010-08-04 11:21 ` Yan, Zheng 0 siblings, 1 reply; 8+ messages in thread From: Simon Kirby @ 2010-08-04 0:24 UTC (permalink / raw) To: Yan, Zheng ; +Cc: Dave Cundiff, linux-btrfs On Wed, Jul 28, 2010 at 08:31:10AM +0800, Yan, Zheng wrote: > On Wed, Jul 28, 2010 at 4:30 AM, Dave Cundiff <syshackmin@gmail.com> wrote: > > Hello, > > > > I installed the git repo kernel and added some debug to the ENOSPC > > returns. Unfortunately its still failing. If it helps any its bombing > > out in btrfs_check_data_free_space() in extent-tree.c. Returning on > > the ENOSPC at line 2959. > > > > Unfortunately that is the extent of my ability to debug a filesystem. :P > > This is really helpful, thank you very much. We're seeing this too, since upgrading from 2.6.33.2 + merged old git btrfs unstable HEAD to plain 2.6.35. [sroot@backup01:.../.rmagic]# rm * rm: cannot remove `WEEKLY_bar3d.png': No space left on device rm: cannot remove `WEEKLY.html': No space left on device rm: cannot remove `YEARLY_bar3d.png': No space left on device rm: cannot remove `YEARLY.html': No space left on device [sroot@backup01:.../.rmagic]# l total 25 drwxr-xr-x 1 1037300 1037300 108 2010-08-03 18:19 ./ drwxr-xr-x 1 1037300 1037300 28 2009-11-07 05:57 ../ -rw-r--r-- 1 1037300 1037300 5720 2010-04-23 13:39 WEEKLY_bar3d.png -rw-r--r-- 1 1037300 1037300 11882 2010-04-23 13:39 WEEKLY.html -rw-r--r-- 1 1037300 1037300 2998 2010-04-23 13:39 YEARLY_bar3d.png -rw-r--r-- 1 1037300 1037300 3016 2010-04-23 13:39 YEARLY.html [sroot@backup01:.../.rmagic]# rm * rm: cannot remove `WEEKLY_bar3d.png': No space left on device rm: cannot remove `YEARLY.html': No space left on device [sroot@backup01:.../.rmagic]# rm * rm: cannot remove `WEEKLY_bar3d.png': No space left on device rm: cannot remove `YEARLY.html': No space left on device [sroot@backup01:.../.rmagic]# rm * [sroot@backup01:.../.rmagic]# [sroot@backup01:/root]# btrfs filesystem df /backup/bu001/vol04 Data: total=2.55TB, used=2.20TB Metadata: total=230.13GB, used=183.83GB System: total=12.00MB, used=548.00KB [sroot@backup01:/root]# df -P /backup/bu001/vol04 Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/bu001-vol04 3221225472 2742785400 478440072 86% /backup/bu001/vol04 [sroot@backup01:/root]# l /dev/mapper/bu001-vol04 brw-rw---- 1 root disk 252, 10 2010-08-03 16:02 /dev/mapper/bu001-vol04 [sroot@backup01:/root]# btrfs filesystem show /dev/dm-10 failed to read /dev/sr0 Label: none uuid: 0c55f5f4-b618-4ec2-9dbc-e3e70a901e1a Total devices 1 FS bytes used 2.37TB devid 1 size 3.00TB used 3.00TB path /dev/dm-10 Btrfs Btrfs v0.19 We're also seeing things like this in dmesg, which appear to be coming from btrfs-cleaner cleaning some old snapshot: Aug 3 18:40:45 backup01 kernel: ------------[ cut here ]------------ Aug 3 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180() Aug 3 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950 Aug 3 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 Aug 3 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tainted: G W 2.6.35-hw #1 Aug 3 18:40:45 backup01 kernel: Call Trace: Aug 3 18:40:45 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block_rsv_check+0x151/0x180 Aug 3 18:40:45 backup01 kernel: [<ffffffff8104b270>] warn_slowpath_common+0x80/0xd0 Aug 3 18:40:45 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath_null+0x15/0x20 Aug 3 18:40:45 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_rsv_check+0x151/0x180 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d5ea1>] btrfs_should_end_transaction+0x61/0x90 Aug 3 18:40:45 backup01 kernel: [<ffffffff812c842d>] btrfs_drop_snapshot+0x21d/0x5f0 Aug 3 18:40:45 backup01 kernel: [<ffffffff81662d72>] ? schedule+0x3f2/0x750 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d463a>] btrfs_clean_old_snapshots+0x12a/0x160 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d0f00>] cleaner_kthread+0x160/0x190 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d0da0>] ? cleaner_kthread+0x0/0x190 Aug 3 18:40:45 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/0xb0 Aug 3 18:40:45 backup01 kernel: [<ffffffff8100aca4>] kernel_thread_helper+0x4/0x10 Aug 3 18:40:45 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0/0xb0 Aug 3 18:40:45 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thread_helper+0x0/0x10 Aug 3 18:40:45 backup01 kernel: ---[ end trace cffc4418e2c1f45d ]--- Aug 3 18:40:45 backup01 kernel: block_rsv size 16194207744 reserved 4497289216 freed 0 78598144 Aug 3 18:40:45 backup01 kernel: ------------[ cut here ]------------ Aug 3 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180() Aug 3 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950 Aug 3 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 Aug 3 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tainted: G W 2.6.35-hw #1 Aug 3 18:40:45 backup01 kernel: Call Trace: Aug 3 18:40:45 backup01 kernel: [<ffffffff812f2eb0>] ? map_extent_buffer+0xb0/0xc0 Aug 3 18:40:45 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block_rsv_check+0x151/0x180 Aug 3 18:40:45 backup01 kernel: [<ffffffff8104b270>] warn_slowpath_common+0x80/0xd0 Aug 3 18:40:45 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath_null+0x15/0x20 Aug 3 18:40:45 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_rsv_check+0x151/0x180 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d56fa>] __btrfs_end_transaction+0x19a/0x220 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d578e>] btrfs_end_transaction_throttle+0xe/0x10 Aug 3 18:40:45 backup01 kernel: [<ffffffff812c84f1>] btrfs_drop_snapshot+0x2e1/0x5f0 Aug 3 18:40:45 backup01 kernel: [<ffffffff81662d72>] ? schedule+0x3f2/0x750 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d463a>] btrfs_clean_old_snapshots+0x12a/0x160 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d0f00>] cleaner_kthread+0x160/0x190 Aug 3 18:40:45 backup01 kernel: [<ffffffff812d0da0>] ? cleaner_kthread+0x0/0x190 Aug 3 18:40:45 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/0xb0 Aug 3 18:40:45 backup01 kernel: [<ffffffff8100aca4>] kernel_thread_helper+0x4/0x10 Aug 3 18:40:45 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0/0xb0 Aug 3 18:40:45 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thread_helper+0x0/0x10 Aug 3 18:40:45 backup01 kernel: ---[ end trace cffc4418e2c1f45e ]--- Aug 3 18:40:45 backup01 kernel: block_rsv size 16194207744 reserved 4497281024 freed 8192 78598144 Aug 3 18:44:44 backup01 kernel: ------------[ cut here ]------------ Aug 3 18:44:44 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180() Aug 3 18:44:44 backup01 kernel: Hardware name: PowerEdge 1950 Aug 3 18:44:44 backup01 kernel: Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 Aug 3 18:44:44 backup01 kernel: Pid: 7526, comm: btrfs-transacti Tainted: G W 2.6.35-hw #1 Aug 3 18:44:44 backup01 kernel: Call Trace: Aug 3 18:44:44 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block_rsv_check+0x151/0x180 Aug 3 18:44:44 backup01 kernel: [<ffffffff8104b270>] warn_slowpath_common+0x80/0xd0 Aug 3 18:44:44 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath_null+0x15/0x20 Aug 3 18:44:44 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_rsv_check+0x151/0x180 Aug 3 18:44:44 backup01 kernel: [<ffffffff812d56fa>] __btrfs_end_transaction+0x19a/0x220 Aug 3 18:44:44 backup01 kernel: [<ffffffff812d579b>] btrfs_end_transaction+0xb/0x10 Aug 3 18:44:44 backup01 kernel: [<ffffffff812d5d63>] btrfs_commit_transaction+0x5c3/0x6a0 Aug 3 18:44:44 backup01 kernel: [<ffffffff810683e0>] ? autoremove_wake_function+0x0/0x40 Aug 3 18:44:44 backup01 kernel: [<ffffffff812d0d90>] transaction_kthread+0x250/0x260 Aug 3 18:44:44 backup01 kernel: [<ffffffff812d0b40>] ? transaction_kthread+0x0/0x260 Aug 3 18:44:44 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/0xb0 Aug 3 18:44:44 backup01 kernel: [<ffffffff8100aca4>] kernel_thread_helper+0x4/0x10 Aug 3 18:44:44 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0/0xb0 Aug 3 18:44:44 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thread_helper+0x0/0x10 Aug 3 18:44:44 backup01 kernel: ---[ end trace cffc4418e2c1f45f ]--- Aug 3 18:44:44 backup01 kernel: block_rsv size 16194207744 reserved 4522696704 freed 53190656 0 I rebuilt with the #if 0 changed to #if 1 on extent-tree.c:2947: #if 1 /* I hope we never need this code again, just in case */ ha! :) "rm" is succeeding everywhere so far, and so this path hasn't been hit yet. Perhaps it has to fight with the btrfs-cleaner, or something. Will post a follow-up later. Simon- ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Intermittent no space errors 2010-08-04 0:24 ` Simon Kirby @ 2010-08-04 11:21 ` Yan, Zheng 2010-08-09 23:25 ` Simon Kirby 0 siblings, 1 reply; 8+ messages in thread From: Yan, Zheng @ 2010-08-04 11:21 UTC (permalink / raw) To: Simon Kirby; +Cc: Dave Cundiff, linux-btrfs On Wed, Aug 4, 2010 at 8:24 AM, Simon Kirby <sim@hostway.ca> wrote: > On Wed, Jul 28, 2010 at 08:31:10AM +0800, Yan, Zheng =A0wrote: > >> On Wed, Jul 28, 2010 at 4:30 AM, Dave Cundiff <syshackmin@gmail.com>= wrote: >> > Hello, >> > >> > I installed the git repo kernel and added some debug to the ENOSPC >> > returns. Unfortunately its still failing. If it helps any its bomb= ing >> > out in btrfs_check_data_free_space() in extent-tree.c. Returning o= n >> > the ENOSPC at line 2959. >> > >> > Unfortunately that is the extent of my ability to debug a filesyst= em. :P >> >> This is really helpful, thank you very much. > > We're seeing this too, since upgrading from 2.6.33.2 + merged old git= btrfs > unstable HEAD to plain 2.6.35. > > [sroot@backup01:.../.rmagic]# rm * > rm: cannot remove `WEEKLY_bar3d.png': No space left on device > rm: cannot remove `WEEKLY.html': No space left on device > rm: cannot remove `YEARLY_bar3d.png': No space left on device > rm: cannot remove `YEARLY.html': No space left on device > [sroot@backup01:.../.rmagic]# l > total 25 > drwxr-xr-x 1 1037300 1037300 =A0 108 2010-08-03 18:19 ./ > drwxr-xr-x 1 1037300 1037300 =A0 =A028 2009-11-07 05:57 ../ > -rw-r--r-- 1 1037300 1037300 =A05720 2010-04-23 13:39 WEEKLY_bar3d.pn= g > -rw-r--r-- 1 1037300 1037300 11882 2010-04-23 13:39 WEEKLY.html > -rw-r--r-- 1 1037300 1037300 =A02998 2010-04-23 13:39 YEARLY_bar3d.pn= g > -rw-r--r-- 1 1037300 1037300 =A03016 2010-04-23 13:39 YEARLY.html > [sroot@backup01:.../.rmagic]# rm * > rm: cannot remove `WEEKLY_bar3d.png': No space left on device > rm: cannot remove `YEARLY.html': No space left on device > [sroot@backup01:.../.rmagic]# rm * > rm: cannot remove `WEEKLY_bar3d.png': No space left on device > rm: cannot remove `YEARLY.html': No space left on device > [sroot@backup01:.../.rmagic]# rm * > [sroot@backup01:.../.rmagic]# > > [sroot@backup01:/root]# btrfs filesystem df /backup/bu001/vol04 > Data: total=3D2.55TB, used=3D2.20TB > Metadata: total=3D230.13GB, used=3D183.83GB > System: total=3D12.00MB, used=3D548.00KB > [sroot@backup01:/root]# df -P /backup/bu001/vol04 > Filesystem =A0 =A0 =A0 =A0 1024-blocks =A0 =A0 =A0Used Available Capa= city Mounted on > /dev/mapper/bu001-vol04 3221225472 2742785400 478440072 =A0 =A0 =A086= % /backup/bu001/vol04 > [sroot@backup01:/root]# l /dev/mapper/bu001-vol04 > brw-rw---- 1 root disk 252, 10 2010-08-03 16:02 /dev/mapper/bu001-vol= 04 > [sroot@backup01:/root]# btrfs filesystem show /dev/dm-10 > failed to read /dev/sr0 > Label: none =A0uuid: 0c55f5f4-b618-4ec2-9dbc-e3e70a901e1a > =A0 =A0 =A0 =A0Total devices 1 FS bytes used 2.37TB > =A0 =A0 =A0 =A0devid =A0 =A01 size 3.00TB used 3.00TB path /dev/dm-10 > > Btrfs Btrfs v0.19 > > We're also seeing things like this in dmesg, which appear to be comin= g > from btrfs-cleaner cleaning some old snapshot: > > Aug =A03 18:40:45 backup01 kernel: ------------[ cut here ]----------= -- > Aug =A03 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c= :3441 btrfs_block_rsv_check+0x151/0x180() > Aug =A03 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950 > Aug =A03 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ip= mi_si ipmi_msghandler aoe bnx2 > Aug =A03 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tai= nted: G =A0 =A0 =A0 =A0W =A0 2.6.35-hw #1 > Aug =A03 18:40:45 backup01 kernel: Call Trace: > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block= _rsv_check+0x151/0x180 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff8104b270>] warn_slowpath= _common+0x80/0xd0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath= _null+0x15/0x20 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_r= sv_check+0x151/0x180 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d5ea1>] btrfs_should_= end_transaction+0x61/0x90 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812c842d>] btrfs_drop_sn= apshot+0x21d/0x5f0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff81662d72>] ? schedule+0x= 3f2/0x750 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d463a>] btrfs_clean_o= ld_snapshots+0x12a/0x160 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d0f00>] cleaner_kthre= ad+0x160/0x190 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d0da0>] ? cleaner_kth= read+0x0/0x190 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/= 0xb0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff8100aca4>] kernel_thread= _helper+0x4/0x10 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0= /0xb0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thre= ad_helper+0x0/0x10 > Aug =A03 18:40:45 backup01 kernel: ---[ end trace cffc4418e2c1f45d ]-= -- > Aug =A03 18:40:45 backup01 kernel: block_rsv size 16194207744 reserve= d 4497289216 freed 0 78598144 > Aug =A03 18:40:45 backup01 kernel: ------------[ cut here ]----------= -- > Aug =A03 18:40:45 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c= :3441 btrfs_block_rsv_check+0x151/0x180() > Aug =A03 18:40:45 backup01 kernel: Hardware name: PowerEdge 1950 > Aug =A03 18:40:45 backup01 kernel: Modules linked in: ipmi_devintf ip= mi_si ipmi_msghandler aoe bnx2 > Aug =A03 18:40:45 backup01 kernel: Pid: 7525, comm: btrfs-cleaner Tai= nted: G =A0 =A0 =A0 =A0W =A0 2.6.35-hw #1 > Aug =A03 18:40:45 backup01 kernel: Call Trace: > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812f2eb0>] ? map_extent_= buffer+0xb0/0xc0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block= _rsv_check+0x151/0x180 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff8104b270>] warn_slowpath= _common+0x80/0xd0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath= _null+0x15/0x20 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_r= sv_check+0x151/0x180 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d56fa>] __btrfs_end_t= ransaction+0x19a/0x220 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d578e>] btrfs_end_tra= nsaction_throttle+0xe/0x10 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812c84f1>] btrfs_drop_sn= apshot+0x2e1/0x5f0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff81662d72>] ? schedule+0x= 3f2/0x750 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d463a>] btrfs_clean_o= ld_snapshots+0x12a/0x160 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d0f00>] cleaner_kthre= ad+0x160/0x190 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff812d0da0>] ? cleaner_kth= read+0x0/0x190 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/= 0xb0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff8100aca4>] kernel_thread= _helper+0x4/0x10 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0= /0xb0 > Aug =A03 18:40:45 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thre= ad_helper+0x0/0x10 > Aug =A03 18:40:45 backup01 kernel: ---[ end trace cffc4418e2c1f45e ]-= -- > Aug =A03 18:40:45 backup01 kernel: block_rsv size 16194207744 reserve= d 4497281024 freed 8192 78598144 > Aug =A03 18:44:44 backup01 kernel: ------------[ cut here ]----------= -- > Aug =A03 18:44:44 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c= :3441 btrfs_block_rsv_check+0x151/0x180() > Aug =A03 18:44:44 backup01 kernel: Hardware name: PowerEdge 1950 > Aug =A03 18:44:44 backup01 kernel: Modules linked in: ipmi_devintf ip= mi_si ipmi_msghandler aoe bnx2 > Aug =A03 18:44:44 backup01 kernel: Pid: 7526, comm: btrfs-transacti T= ainted: G =A0 =A0 =A0 =A0W =A0 2.6.35-hw #1 > Aug =A03 18:44:44 backup01 kernel: Call Trace: > Aug =A03 18:44:44 backup01 kernel: [<ffffffff812c56a1>] ? btrfs_block= _rsv_check+0x151/0x180 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff8104b270>] warn_slowpath= _common+0x80/0xd0 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff8104b2d5>] warn_slowpath= _null+0x15/0x20 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff812c56a1>] btrfs_block_r= sv_check+0x151/0x180 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff812d56fa>] __btrfs_end_t= ransaction+0x19a/0x220 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff812d579b>] btrfs_end_tra= nsaction+0xb/0x10 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff812d5d63>] btrfs_commit_= transaction+0x5c3/0x6a0 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff810683e0>] ? autoremove_= wake_function+0x0/0x40 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff812d0d90>] transaction_k= thread+0x250/0x260 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff812d0b40>] ? transaction= _kthread+0x0/0x260 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff81067ea6>] kthread+0x96/= 0xb0 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff8100aca4>] kernel_thread= _helper+0x4/0x10 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff81067e10>] ? kthread+0x0= /0xb0 > Aug =A03 18:44:44 backup01 kernel: [<ffffffff8100aca0>] ? kernel_thre= ad_helper+0x0/0x10 > Aug =A03 18:44:44 backup01 kernel: ---[ end trace cffc4418e2c1f45f ]-= -- > Aug =A03 18:44:44 backup01 kernel: block_rsv size 16194207744 reserve= d 4522696704 freed 53190656 0 > These warning is because btrfs in 2.6.35 reserves more metadata space for internal use than older kernel. Your FS is too full, btrfs can't reserve enough metadata space. > I rebuilt with the #if 0 changed to #if 1 on extent-tree.c:2947: > > #if 1 /* I hope we never need this code again, just in case */ > > ha! :) =A0"rm" is succeeding everywhere so far, and so this path hasn= 't > been hit yet. =A0Perhaps it has to fight with the btrfs-cleaner, or > something. =A0Will post a follow-up later. yes, it has to fight with the btrfs-cleaner. Yan, Zheng -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Intermittent no space errors 2010-08-04 11:21 ` Yan, Zheng @ 2010-08-09 23:25 ` Simon Kirby 0 siblings, 0 replies; 8+ messages in thread From: Simon Kirby @ 2010-08-09 23:25 UTC (permalink / raw) To: Yan, Zheng; +Cc: Dave Cundiff, linux-btrfs On Wed, Aug 04, 2010 at 07:21:00PM +0800, Yan, Zheng wrote: > > We're seeing this too, since upgrading from 2.6.33.2 + merged old git btrfs > > unstable HEAD to plain 2.6.35. > > > > [sroot@backup01:.../.rmagic]# rm * > > rm: cannot remove `WEEKLY_bar3d.png': No space left on device > > rm: cannot remove `WEEKLY.html': No space left on device > > rm: cannot remove `YEARLY_bar3d.png': No space left on device > > rm: cannot remove `YEARLY.html': No space left on device >... > > Aug ?3 18:44:44 backup01 kernel: ------------[ cut here ]------------ > > Aug ?3 18:44:44 backup01 kernel: WARNING: at fs/btrfs/extent-tree.c:3441 btrfs_block_rsv_check+0x151/0x180() >... > > These warning is because btrfs in 2.6.35 reserves more metadata space > for internal use > than older kernel. Your FS is too full, btrfs can't reserve enough > metadata space. Hello! Is it possible that 2.6.33.2 btrfs has mucked up the on-disk stuff in a way that causes 2.6.35 to be unhappy? The file system in question was reported to be 85% full, according to "df". In the meantime, we've been having some other problems on 2.6.35; for example, rsync has been trying to append a block to a file for the past 5 days. The file system is reported as 45% full: [sroot@backup01:/root]# df -Pt btrfs /backup/bu000/vol05/ Filesystem 1024-blocks Used Available Capacity Mounted on /dev/mapper/bu000-vol05 3221225472 1429529324 1791696148 45% /backup/bu000/vol05 [sroot@backup01:/root]# btrfs files df /backup/bu000/vol05 Data: total=1.57TB, used=1.31TB Metadata: total=15.51GB, used=10.48GB System: total=12.00MB, used=192.00KB At some point today, the kernel also spat this out: BUG: soft lockup - CPU#3 stuck for 61s! [rsync:21903] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 CPU 3 Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler aoe bnx2 Pid: 21903, comm: rsync Tainted: G W 2.6.35-hw #2 0NK937/PowerEdge 1950 RIP: 0010:[<ffffffff81101a2d>] [<ffffffff81101a2d>] iput+0x5d/0x70 RSP: 0018:ffff8802c14abb48 EFLAGS: 00000246 RAX: 0000000000000000 RBX: ffff8802c14abb58 RCX: 0000000000000003 RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff88007c075980 RBP: ffffffff8100a84e R08: 0000000000000001 R09: 8000000000000000 R10: 0000000000000002 R11: 0000000000000000 R12: ffffffffffffff66 R13: ffffffff81af04e0 R14: 0000000000000000 R15: 7fffffffffffffff FS: 00007fd13bbb06e0(0000) GS:ffff880001cc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000002f5a108 CR3: 00000001eb94a000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process rsync (pid: 21903, threadinfo ffff8802c14aa000, task ffff880080b04b00) Stack: ffff88007c075888 ffff88007c0757b0 ffff8802c14abb98 ffffffff812d7439 <0> ffffffff81664cde 0000000000000001 0000000004d80000 0000000000004000 <0> ffff88042a708178 ffff88042a708000 ffff8802c14abc08 ffffffff812c599c Call Trace: [<ffffffff812d7439>] ? btrfs_start_one_delalloc_inode+0x129/0x160 [<ffffffff81664cde>] ? _raw_spin_lock+0xe/0x20 [<ffffffff812c599c>] ? shrink_delalloc+0x8c/0x130 [<ffffffff812c5f39>] ? btrfs_delalloc_reserve_metadata+0x189/0x190 [<ffffffff8110180e>] ? file_update_time+0x11e/0x180 [<ffffffff812c5f83>] ? btrfs_delalloc_reserve_space+0x43/0x60 [<ffffffff812e2a98>] ? btrfs_file_aio_write+0x508/0x970 [<ffffffff8100a84e>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff810ec1b1>] ? do_sync_write+0xd1/0x120 [<ffffffff810fc767>] ? poll_select_copy_remaining+0xf7/0x140 [<ffffffff810ecd2b>] ? vfs_write+0xcb/0x1a0 [<ffffffff810ecef0>] ? sys_write+0x50/0x90 [<ffffffff81009f02>] ? system_call_fastpath+0x16/0x1b Code: 00 01 00 00 48 c7 c2 a0 2c 10 81 48 8b 40 30 48 85 c0 74 12 48 8b 50 20 48 c7 c0 a0 2c 10 81 48 85 d2 48 0 Call Trace: [<ffffffff812d7439>] ? btrfs_start_one_delalloc_inode+0x129/0x160 [<ffffffff81664cde>] ? _raw_spin_lock+0xe/0x20 [<ffffffff812c599c>] ? shrink_delalloc+0x8c/0x130 [<ffffffff812c5f39>] ? btrfs_delalloc_reserve_metadata+0x189/0x190 [<ffffffff8110180e>] ? file_update_time+0x11e/0x180 [<ffffffff812c5f83>] ? btrfs_delalloc_reserve_space+0x43/0x60 [<ffffffff812e2a98>] ? btrfs_file_aio_write+0x508/0x970 [<ffffffff8100a84e>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff810ec1b1>] ? do_sync_write+0xd1/0x120 [<ffffffff810fc767>] ? poll_select_copy_remaining+0xf7/0x140 [<ffffffff810ecd2b>] ? vfs_write+0xcb/0x1a0 [<ffffffff810ecef0>] ? sys_write+0x50/0x90 [<ffffffff81009f02>] ? system_call_fastpath+0x16/0x1b [sroot@backup01:/root]# ls -l /proc/21903/fd/1 lrwx------ 1 root root 64 2010-08-09 18:21 /proc/21903/fd/1 -> /backup/bu000/vol05/vg005_web11_backup/2010-08-04-17-00/64/54/.../customer file.mov.aYX4Js [sroot@backup01:/root]# ls -lL /proc/21903/fd/1 -rw------- 1 root root 977797120 2010-08-04 20:39 /proc/21903/fd/1 [sroot@backup01:/root]# ps auxw|grep rsync root 21903 73.2 0.0 12912 192 ? R Aug04 5177:08 rsync -aHq --numeric-ids --exclude-from=/etc/backups/backup.exclude --delete --delete-excluded /storage/vg005/web11/64/54/ /backup/bu000/vol05/vg005_web11_backup/2010-08-04-17-00/64/54 In other words, the rsync has made no progress for 5 days (or at least the mtime hasn't changed since then). "perf top" still shows output like this, showing that btrfs is trying to btrfs_find_space_cluster all of the time: samples pcnt function DSO _______ _____ ______________________________ _________________ 2127.00 11.9% find_next_bit [kernel] 1914.00 10.7% find_next_zero_bit [kernel] 1580.00 8.8% schedule [kernel] 1340.00 7.5% btrfs_find_space_cluster [kernel] 1238.00 6.9% _raw_spin_lock_irqsave [kernel] 1017.00 5.7% _raw_spin_lock [kernel] 662.00 3.7% sched_clock_local [kernel] 615.00 3.4% native_read_tsc [kernel] 590.00 3.3% _raw_spin_lock_irq [kernel] 468.00 2.6% _raw_spin_unlock_irqrestore [kernel] 405.00 2.3% schedule_timeout [kernel] 338.00 1.9% native_sched_clock [kernel] 329.00 1.8% update_curr [kernel] 323.00 1.8% lock_timer_base [kernel] 297.00 1.7% btrfs_start_one_delalloc_inode [kernel] 285.00 1.6% pick_next_task_fair [kernel] 267.00 1.5% try_to_del_timer_sync [kernel] 248.00 1.4% sched_clock_cpu [kernel] 245.00 1.4% deflate_fast [kernel] So, is it possible that the older btrfs code left things in a way that wouldn't have happened if we had started with 3.6.35 to begin with? In the case of the 45% full file system, it seems it should be able to allocate more space without issue. Simon- ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Intermittent no space errors 2010-07-27 20:30 ` Dave Cundiff 2010-07-28 0:31 ` Yan, Zheng @ 2010-07-29 8:10 ` Justin Ossevoort 1 sibling, 0 replies; 8+ messages in thread From: Justin Ossevoort @ 2010-07-29 8:10 UTC (permalink / raw) To: linux-btrfs Hello, Pherhaps it would be a good idea to add a tracepoint before each return ENOSPC? It shouldn't matter too much since a reasonable assumption would be that filesystems aren't running out of space most of the time. And you can use 'perf' for more insight in these cases without recompiling the kernel. Regards, justin.... On 27/07/10 22:30, Dave Cundiff wrote: > Hello, > > I installed the git repo kernel and added some debug to the ENOSPC > returns. Unfortunately its still failing. If it helps any its bombing > out in btrfs_check_data_free_space() in extent-tree.c. Returning on > the ENOSPC at line 2959. > > Unfortunately that is the extent of my ability to debug a filesystem. :P > > Thanks, > > On Tue, Jul 27, 2010 at 9:19 AM, Yan, Zheng <yanzheng@21cn.com> wrote: > >> On Tue, Jul 27, 2010 at 5:09 AM, Dave Cundiff <syshackmin@gmail.com> wrote: >> >>> Hello, >>> >>> On 2.6.35-rc5 I'm seeing some weird behavior under heavy IO loads. I >>> have a backup process that fires up several rsync processes. These >>> mirror several dozen servers to individual sub-volumes. Everyday I >>> snapshot each sub-volume and rsync over it. >>> >>> The problem I'm seeing is my rsync processes are failing randomly with >>> "No space left on device". This is a 6 Terabyte volume with plenty of >>> free space. >>> >>> Mount options: >>> /dev/sdb on /backups type btrfs (rw,max_inline=0,compress) >>> >>> [root@rsync1 ~]# btrfs filesystem df /backups/ >>> Data: total=1.88TB, used=1.88TB >>> Metadata: total=43.38GB, used=32.06GB >>> System: total=12.00MB, used=260.00KB >>> >>> [root@rsync1 ~]# df /dev/sdb >>> Filesystem 1K-blocks Used Available Use% Mounted on >>> /dev/sdb 5781249024 2087273084 3693975940 37% /backups >>> >>> They don't all fail at once. Normally I have 4-5 running at a time and >>> 1 or 2 will drop out with a no space error. The rest continue on. I've >>> noticed it will generally occur on ones that are in the middle of >>> transferring a very large file. If I lighten the load to one rsync at >>> a time it appears to happen less frequently. >>> >>> Any known issues I should be aware of? >>> >>> >> Thank you for reporting this. I will dig in. >> >> Yan, Zheng >> >> > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2010-08-09 23:25 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-07-26 21:09 Intermittent no space errors Dave Cundiff 2010-07-27 13:19 ` Yan, Zheng 2010-07-27 20:30 ` Dave Cundiff 2010-07-28 0:31 ` Yan, Zheng 2010-08-04 0:24 ` Simon Kirby 2010-08-04 11:21 ` Yan, Zheng 2010-08-09 23:25 ` Simon Kirby 2010-07-29 8:10 ` Justin Ossevoort
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).