* btrfs full, but not full, can't rebalance
@ 2015-07-03 5:32 Rich Rauenzahn
2015-07-03 5:45 ` Donald Pearson
0 siblings, 1 reply; 10+ messages in thread
From: Rich Rauenzahn @ 2015-07-03 5:32 UTC (permalink / raw)
To: linux-btrfs
Running on CentOS7 ... / got full, I removed the files, but it still
thinks it is full. I've tried following the FAQ, even adding a
loopback device during the rebalance.
# btrfs fi show /
Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
Total devices 2 FS bytes used 24.27GiB
devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3
devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3
# btrfs fi df /
Data, RAID1: total=107.02GiB, used=22.12GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=4.05GiB, used=2.15GiB
GlobalReserve, single: total=512.00MiB, used=0.00
# btrfs balance start -v -dusage=1 /
Dumping filters: flags 0x1, state 0x0, force is off
DATA (flags 0x2): balancing, usage=1
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
# btrfs balance start -m /
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
What can I do?
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: btrfs full, but not full, can't rebalance 2015-07-03 5:32 btrfs full, but not full, can't rebalance Rich Rauenzahn @ 2015-07-03 5:45 ` Donald Pearson 2015-07-03 5:57 ` Rich Rauenzahn 0 siblings, 1 reply; 10+ messages in thread From: Donald Pearson @ 2015-07-03 5:45 UTC (permalink / raw) To: Rich Rauenzahn; +Cc: Btrfs BTRFS Have you seen this article? I think the interesting part for you is the "balance cannot run because the filesystem is full" heading. http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: > Running on CentOS7 ... / got full, I removed the files, but it still > thinks it is full. I've tried following the FAQ, even adding a > loopback device during the rebalance. > > # btrfs fi show / > Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a > Total devices 2 FS bytes used 24.27GiB > devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3 > devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3 > > # btrfs fi df / > Data, RAID1: total=107.02GiB, used=22.12GiB > System, RAID1: total=32.00MiB, used=16.00KiB > Metadata, RAID1: total=4.05GiB, used=2.15GiB > GlobalReserve, single: total=512.00MiB, used=0.00 > > # btrfs balance start -v -dusage=1 / > Dumping filters: flags 0x1, state 0x0, force is off > DATA (flags 0x2): balancing, usage=1 > ERROR: error during balancing '/' - No space left on device > There may be more info in syslog - try dmesg | tail > > # btrfs balance start -m / > ERROR: error during balancing '/' - No space left on device > There may be more info in syslog - try dmesg | tail > > What can I do? > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: btrfs full, but not full, can't rebalance 2015-07-03 5:45 ` Donald Pearson @ 2015-07-03 5:57 ` Rich Rauenzahn 2015-07-03 6:01 ` Donald Pearson 2015-07-03 7:56 ` Rich Rauenzahn 0 siblings, 2 replies; 10+ messages in thread From: Rich Rauenzahn @ 2015-07-03 5:57 UTC (permalink / raw) To: Donald Pearson; +Cc: Btrfs BTRFS Yes, I tried that -- and adding the loopback device. # btrfs device add /dev/loop1 / Performing full device TRIM (5.00GiB) ... # btrfs fi show / Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a Total devices 3 FS bytes used 17.13GiB devid 1 size 111.11GiB used 111.10GiB path /dev/sdf3 devid 2 size 111.11GiB used 111.10GiB path /dev/sdg3 devid 3 size 5.00GiB used 0.00 path /dev/loop1 Btrfs v3.16.2 # btrfs balance start -m / ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail # btrfs balance start -v -dusage=1 / Dumping filters: flags 0x1, state 0x0, force is off DATA (flags 0x2): balancing, usage=1 ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson <donaldwhpearson@gmail.com> wrote: > Have you seen this article? > > I think the interesting part for you is the "balance cannot run > because the filesystem is full" heading. > > http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html > > On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: >> Running on CentOS7 ... / got full, I removed the files, but it still >> thinks it is full. I've tried following the FAQ, even adding a >> loopback device during the rebalance. >> >> # btrfs fi show / >> Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a >> Total devices 2 FS bytes used 24.27GiB >> devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3 >> devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3 >> >> # btrfs fi df / >> Data, RAID1: total=107.02GiB, used=22.12GiB >> System, RAID1: total=32.00MiB, used=16.00KiB >> Metadata, RAID1: total=4.05GiB, used=2.15GiB >> GlobalReserve, single: total=512.00MiB, used=0.00 >> >> # btrfs balance start -v -dusage=1 / >> Dumping filters: flags 0x1, state 0x0, force is off >> DATA (flags 0x2): balancing, usage=1 >> ERROR: error during balancing '/' - No space left on device >> There may be more info in syslog - try dmesg | tail >> >> # btrfs balance start -m / >> ERROR: error during balancing '/' - No space left on device >> There may be more info in syslog - try dmesg | tail >> >> What can I do? >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: btrfs full, but not full, can't rebalance 2015-07-03 5:57 ` Rich Rauenzahn @ 2015-07-03 6:01 ` Donald Pearson 2015-07-03 6:03 ` Rich Rauenzahn 2015-07-03 7:56 ` Rich Rauenzahn 1 sibling, 1 reply; 10+ messages in thread From: Donald Pearson @ 2015-07-03 6:01 UTC (permalink / raw) To: Rich Rauenzahn; +Cc: Btrfs BTRFS Because this is raid1 I believe you need another for that to work. On Fri, Jul 3, 2015 at 12:57 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: > Yes, I tried that -- and adding the loopback device. > > # btrfs device add /dev/loop1 / > Performing full device TRIM (5.00GiB) ... > > # btrfs fi show / > Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a > Total devices 3 FS bytes used 17.13GiB > devid 1 size 111.11GiB used 111.10GiB path /dev/sdf3 > devid 2 size 111.11GiB used 111.10GiB path /dev/sdg3 > devid 3 size 5.00GiB used 0.00 path /dev/loop1 > > Btrfs v3.16.2 > > # btrfs balance start -m / > ERROR: error during balancing '/' - No space left on device > There may be more info in syslog - try dmesg | tail > > # btrfs balance start -v -dusage=1 / > Dumping filters: flags 0x1, state 0x0, force is off > DATA (flags 0x2): balancing, usage=1 > ERROR: error during balancing '/' - No space left on device > There may be more info in syslog - try dmesg | tail > > > > On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson > <donaldwhpearson@gmail.com> wrote: >> Have you seen this article? >> >> I think the interesting part for you is the "balance cannot run >> because the filesystem is full" heading. >> >> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html >> >> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: >>> Running on CentOS7 ... / got full, I removed the files, but it still >>> thinks it is full. I've tried following the FAQ, even adding a >>> loopback device during the rebalance. >>> >>> # btrfs fi show / >>> Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a >>> Total devices 2 FS bytes used 24.27GiB >>> devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3 >>> devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3 >>> >>> # btrfs fi df / >>> Data, RAID1: total=107.02GiB, used=22.12GiB >>> System, RAID1: total=32.00MiB, used=16.00KiB >>> Metadata, RAID1: total=4.05GiB, used=2.15GiB >>> GlobalReserve, single: total=512.00MiB, used=0.00 >>> >>> # btrfs balance start -v -dusage=1 / >>> Dumping filters: flags 0x1, state 0x0, force is off >>> DATA (flags 0x2): balancing, usage=1 >>> ERROR: error during balancing '/' - No space left on device >>> There may be more info in syslog - try dmesg | tail >>> >>> # btrfs balance start -m / >>> ERROR: error during balancing '/' - No space left on device >>> There may be more info in syslog - try dmesg | tail >>> >>> What can I do? >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: btrfs full, but not full, can't rebalance 2015-07-03 6:01 ` Donald Pearson @ 2015-07-03 6:03 ` Rich Rauenzahn 2015-07-03 6:05 ` Donald Pearson 0 siblings, 1 reply; 10+ messages in thread From: Rich Rauenzahn @ 2015-07-03 6:03 UTC (permalink / raw) To: Donald Pearson; +Cc: Btrfs BTRFS Yes -- I just figured that out as well! Now why did it suddenly fill up? (I still get the failure rebalancing ...) # btrfs fi show / Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a Total devices 4 FS bytes used 17.12GiB devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3 devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3 devid 3 size 5.00GiB used 5.00GiB path /dev/loop1 devid 4 size 5.00GiB used 5.00GiB path /dev/loop2 Btrfs v3.16.2 On Thu, Jul 2, 2015 at 11:01 PM, Donald Pearson <donaldwhpearson@gmail.com> wrote: > Because this is raid1 I believe you need another for that to work. > > On Fri, Jul 3, 2015 at 12:57 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: >> Yes, I tried that -- and adding the loopback device. >> >> # btrfs device add /dev/loop1 / >> Performing full device TRIM (5.00GiB) ... >> >> # btrfs fi show / >> Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a >> Total devices 3 FS bytes used 17.13GiB >> devid 1 size 111.11GiB used 111.10GiB path /dev/sdf3 >> devid 2 size 111.11GiB used 111.10GiB path /dev/sdg3 >> devid 3 size 5.00GiB used 0.00 path /dev/loop1 >> >> Btrfs v3.16.2 >> >> # btrfs balance start -m / >> ERROR: error during balancing '/' - No space left on device >> There may be more info in syslog - try dmesg | tail >> >> # btrfs balance start -v -dusage=1 / >> Dumping filters: flags 0x1, state 0x0, force is off >> DATA (flags 0x2): balancing, usage=1 >> ERROR: error during balancing '/' - No space left on device >> There may be more info in syslog - try dmesg | tail >> >> >> >> On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson >> <donaldwhpearson@gmail.com> wrote: >>> Have you seen this article? >>> >>> I think the interesting part for you is the "balance cannot run >>> because the filesystem is full" heading. >>> >>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html >>> >>> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: >>>> Running on CentOS7 ... / got full, I removed the files, but it still >>>> thinks it is full. I've tried following the FAQ, even adding a >>>> loopback device during the rebalance. >>>> >>>> # btrfs fi show / >>>> Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a >>>> Total devices 2 FS bytes used 24.27GiB >>>> devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3 >>>> devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3 >>>> >>>> # btrfs fi df / >>>> Data, RAID1: total=107.02GiB, used=22.12GiB >>>> System, RAID1: total=32.00MiB, used=16.00KiB >>>> Metadata, RAID1: total=4.05GiB, used=2.15GiB >>>> GlobalReserve, single: total=512.00MiB, used=0.00 >>>> >>>> # btrfs balance start -v -dusage=1 / >>>> Dumping filters: flags 0x1, state 0x0, force is off >>>> DATA (flags 0x2): balancing, usage=1 >>>> ERROR: error during balancing '/' - No space left on device >>>> There may be more info in syslog - try dmesg | tail >>>> >>>> # btrfs balance start -m / >>>> ERROR: error during balancing '/' - No space left on device >>>> There may be more info in syslog - try dmesg | tail >>>> >>>> What can I do? >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: btrfs full, but not full, can't rebalance 2015-07-03 6:03 ` Rich Rauenzahn @ 2015-07-03 6:05 ` Donald Pearson 0 siblings, 0 replies; 10+ messages in thread From: Donald Pearson @ 2015-07-03 6:05 UTC (permalink / raw) To: Rich Rauenzahn; +Cc: Btrfs BTRFS what does the fi df , or btrfs fi usage show now On Fri, Jul 3, 2015 at 1:03 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: > Yes -- I just figured that out as well! > > Now why did it suddenly fill up? (I still get the failure rebalancing ...) > > # btrfs fi show / > Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a > Total devices 4 FS bytes used 17.12GiB > devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3 > devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3 > devid 3 size 5.00GiB used 5.00GiB path /dev/loop1 > devid 4 size 5.00GiB used 5.00GiB path /dev/loop2 > > Btrfs v3.16.2 > > > On Thu, Jul 2, 2015 at 11:01 PM, Donald Pearson > <donaldwhpearson@gmail.com> wrote: >> Because this is raid1 I believe you need another for that to work. >> >> On Fri, Jul 3, 2015 at 12:57 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: >>> Yes, I tried that -- and adding the loopback device. >>> >>> # btrfs device add /dev/loop1 / >>> Performing full device TRIM (5.00GiB) ... >>> >>> # btrfs fi show / >>> Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a >>> Total devices 3 FS bytes used 17.13GiB >>> devid 1 size 111.11GiB used 111.10GiB path /dev/sdf3 >>> devid 2 size 111.11GiB used 111.10GiB path /dev/sdg3 >>> devid 3 size 5.00GiB used 0.00 path /dev/loop1 >>> >>> Btrfs v3.16.2 >>> >>> # btrfs balance start -m / >>> ERROR: error during balancing '/' - No space left on device >>> There may be more info in syslog - try dmesg | tail >>> >>> # btrfs balance start -v -dusage=1 / >>> Dumping filters: flags 0x1, state 0x0, force is off >>> DATA (flags 0x2): balancing, usage=1 >>> ERROR: error during balancing '/' - No space left on device >>> There may be more info in syslog - try dmesg | tail >>> >>> >>> >>> On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson >>> <donaldwhpearson@gmail.com> wrote: >>>> Have you seen this article? >>>> >>>> I think the interesting part for you is the "balance cannot run >>>> because the filesystem is full" heading. >>>> >>>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html >>>> >>>> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: >>>>> Running on CentOS7 ... / got full, I removed the files, but it still >>>>> thinks it is full. I've tried following the FAQ, even adding a >>>>> loopback device during the rebalance. >>>>> >>>>> # btrfs fi show / >>>>> Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a >>>>> Total devices 2 FS bytes used 24.27GiB >>>>> devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3 >>>>> devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3 >>>>> >>>>> # btrfs fi df / >>>>> Data, RAID1: total=107.02GiB, used=22.12GiB >>>>> System, RAID1: total=32.00MiB, used=16.00KiB >>>>> Metadata, RAID1: total=4.05GiB, used=2.15GiB >>>>> GlobalReserve, single: total=512.00MiB, used=0.00 >>>>> >>>>> # btrfs balance start -v -dusage=1 / >>>>> Dumping filters: flags 0x1, state 0x0, force is off >>>>> DATA (flags 0x2): balancing, usage=1 >>>>> ERROR: error during balancing '/' - No space left on device >>>>> There may be more info in syslog - try dmesg | tail >>>>> >>>>> # btrfs balance start -m / >>>>> ERROR: error during balancing '/' - No space left on device >>>>> There may be more info in syslog - try dmesg | tail >>>>> >>>>> What can I do? >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: btrfs full, but not full, can't rebalance 2015-07-03 5:57 ` Rich Rauenzahn 2015-07-03 6:01 ` Donald Pearson @ 2015-07-03 7:56 ` Rich Rauenzahn 2015-07-03 17:28 ` Rich Rauenzahn 1 sibling, 1 reply; 10+ messages in thread From: Rich Rauenzahn @ 2015-07-03 7:56 UTC (permalink / raw) To: Donald Pearson; +Cc: Btrfs BTRFS Donald and I went offline so I will summarize where I am now: (1) Since I'm RAID1, I had to add two loopbacks. That makes sense! (2) The two 5GB loopbacks filled up almost instantly and did me no good. (3) I created two more 100 GB loopbacks (hint: use /usr/bin/truncate not /usr/bin/dd to create the empty files!) I then was able to make some progress with balancing with -dusage=1. I went to -dusage=2 and it failed with enospc again. A little while later -dusage=2 worked. I believe I made it up to -dusage=5 and then higher values failed. At this point I saw that my 'btrfs fi show' showed that I had some headroom, so I dropped all of the loopbacks from my volume. But I found that I still could not "touch /tmp/foo". I then just tried running btrfs balance start. I noticed in dmesg that although btrfs balance was exiting with an error about space, it was making progress on some extents. btrfs fi show also reflected that. So I began a loop... while ! btrfs balance start /; do btrfs fi show /; btrfs fi df / ; done It is still running, but after a few iterations I am now able to touch a file /tmp/foo successfully and the fi show seems to still be coming down to a proper value, although occasionally it bumps up the wrong direction: ]# while ! btrfs balance start /; do btrfs fi show /; btrfs fi df / ; done ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a Total devices 2 FS bytes used 17.25GiB devid 1 size 111.11GiB used 84.03GiB path /dev/sdf3 devid 2 size 111.11GiB used 84.03GiB path /dev/sdg3 Btrfs v3.16.2 Data, RAID1: total=41.00GiB, used=16.54GiB System, RAID1: total=32.00MiB, used=16.00KiB Metadata, RAID1: total=43.00GiB, used=724.38MiB GlobalReserve, single: total=256.00MiB, used=336.00KiB ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a Total devices 2 FS bytes used 17.25GiB devid 1 size 111.11GiB used 85.03GiB path /dev/sdf3 devid 2 size 111.11GiB used 85.03GiB path /dev/sdg3 Btrfs v3.16.2 Data, RAID1: total=40.00GiB, used=16.54GiB System, RAID1: total=32.00MiB, used=16.00KiB Metadata, RAID1: total=45.00GiB, used=724.30MiB GlobalReserve, single: total=256.00MiB, used=256.00KiB ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a Total devices 2 FS bytes used 17.25GiB devid 1 size 111.11GiB used 86.03GiB path /dev/sdf3 devid 2 size 111.11GiB used 86.03GiB path /dev/sdg3 Btrfs v3.16.2 Data, RAID1: total=39.00GiB, used=16.54GiB System, RAID1: total=32.00MiB, used=16.00KiB Metadata, RAID1: total=47.00GiB, used=723.78MiB GlobalReserve, single: total=256.00MiB, used=272.00KiB Many thanks to Donald for getting me to this point! On Thu, Jul 2, 2015 at 10:57 PM, Rich Rauenzahn <rrauenza@gmail.com> wrote: > Yes, I tried that -- and adding the loopback device. > > # btrfs device add /dev/loop1 / > Performing full device TRIM (5.00GiB) ... > > # btrfs fi show / > Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a > Total devices 3 FS bytes used 17.13GiB > devid 1 size 111.11GiB used 111.10GiB path /dev/sdf3 > devid 2 size 111.11GiB used 111.10GiB path /dev/sdg3 > devid 3 size 5.00GiB used 0.00 path /dev/loop1 > > Btrfs v3.16.2 > > # btrfs balance start -m / > ERROR: error during balancing '/' - No space left on device > There may be more info in syslog - try dmesg | tail > > # btrfs balance start -v -dusage=1 / > Dumping filters: flags 0x1, state 0x0, force is off > DATA (flags 0x2): balancing, usage=1 > ERROR: error during balancing '/' - No space left on device > There may be more info in syslog - try dmesg | tail > > > > On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson > <donaldwhpearson@gmail.com> wrote: >> Have you seen this article? >> >> I think the interesting part for you is the "balance cannot run >> because the filesystem is full" heading. >> >> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html >> >> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote: >>> Running on CentOS7 ... / got full, I removed the files, but it still >>> thinks it is full. I've tried following the FAQ, even adding a >>> loopback device during the rebalance. >>> >>> # btrfs fi show / >>> Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a >>> Total devices 2 FS bytes used 24.27GiB >>> devid 1 size 111.11GiB used 111.05GiB path /dev/sdf3 >>> devid 2 size 111.11GiB used 111.05GiB path /dev/sdg3 >>> >>> # btrfs fi df / >>> Data, RAID1: total=107.02GiB, used=22.12GiB >>> System, RAID1: total=32.00MiB, used=16.00KiB >>> Metadata, RAID1: total=4.05GiB, used=2.15GiB >>> GlobalReserve, single: total=512.00MiB, used=0.00 >>> >>> # btrfs balance start -v -dusage=1 / >>> Dumping filters: flags 0x1, state 0x0, force is off >>> DATA (flags 0x2): balancing, usage=1 >>> ERROR: error during balancing '/' - No space left on device >>> There may be more info in syslog - try dmesg | tail >>> >>> # btrfs balance start -m / >>> ERROR: error during balancing '/' - No space left on device >>> There may be more info in syslog - try dmesg | tail >>> >>> What can I do? >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: btrfs full, but not full, can't rebalance 2015-07-03 7:56 ` Rich Rauenzahn @ 2015-07-03 17:28 ` Rich Rauenzahn 2015-07-03 18:04 ` Chris Murphy 0 siblings, 1 reply; 10+ messages in thread From: Rich Rauenzahn @ 2015-07-03 17:28 UTC (permalink / raw) To: Donald Pearson; +Cc: Btrfs BTRFS I'm still seeing periodic issues here. The filesystem will go "full" occasionally even though there appears to be plenty of space. Running another rebalance seems to fix it ... but I don't see why the system thinks it needs to be rebalanced. # while ! btrfs balance start /; do btrfs fi show /; btrfs fi df / ; done ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a Total devices 2 FS bytes used 17.06GiB devid 1 size 111.11GiB used 22.03GiB path /dev/sdf3 devid 2 size 111.11GiB used 22.03GiB path /dev/sdg3 Btrfs v3.16.2 Data, RAID1: total=18.00GiB, used=16.36GiB System, RAID1: total=32.00MiB, used=16.00KiB Metadata, RAID1: total=4.00GiB, used=711.34MiB GlobalReserve, single: total=240.00MiB, used=128.00KiB ERROR: error during balancing '/' - No space left on device There may be more info in syslog - try dmesg | tail Label: 'centos7' uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a Total devices 2 FS bytes used 17.06GiB devid 1 size 111.11GiB used 25.06GiB path /dev/sdf3 devid 2 size 111.11GiB used 25.06GiB path /dev/sdg3 Btrfs v3.16.2 Data, RAID1: total=17.00GiB, used=16.36GiB System, RAID1: total=64.00MiB, used=16.00KiB Metadata, RAID1: total=8.00GiB, used=711.67MiB GlobalReserve, single: total=240.00MiB, used=144.00KiB [...] For now, this is going into a cron job ... Rich ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: btrfs full, but not full, can't rebalance 2015-07-03 17:28 ` Rich Rauenzahn @ 2015-07-03 18:04 ` Chris Murphy 2015-07-12 6:10 ` Rich Rauenzahn 0 siblings, 1 reply; 10+ messages in thread From: Chris Murphy @ 2015-07-03 18:04 UTC (permalink / raw) To: Btrfs BTRFS What kernel version are you using? I suggest reading the whole thread but in particular this note about CentOS 7 and Btrfs. http://www.spinics.net/lists/linux-btrfs/msg40850.html So kernel 3.16 isn't necessarily too old, but I know there's been a lot of on-going balance related work that's not in that kernel and thus won't be in the CentOS 7.1 kernel either. And then there's also some possibility the older btrfs-progs contributes to the problem, even though I think most of the balance code is actually in the kernel. While the CentOS 7.1 kernel is going to be more stable/predictable overall (not with respect to Btrfs but everything the kernel does): a.) Btrfs is a tech preview b.) To unwind the problems you've having with this volume right now, I'd really suggest a much newer kernel. > Total devices 2 FS bytes used 17.06GiB > devid 1 size 111.11GiB used 22.03GiB path /dev/sdf3 > devid 2 size 111.11GiB used 22.03GiB path /dev/sdg3 There's tons of unallocated space for new data or metadata chunks. > Data, RAID1: total=18.00GiB, used=16.36GiB > System, RAID1: total=32.00MiB, used=16.00KiB > Metadata, RAID1: total=4.00GiB, used=711.34MiB > GlobalReserve, single: total=240.00MiB, used=128.00KiB > ERROR: error during balancing '/' - No space left on device And there's tons of space in already created data and metadata chunks. So something is wrong, and I really wouldn't persist on trying to make this work with an older kernel and progs. So I suggest putting the volume on a system with a 4.1 kernel and 4.1 progs, and running a btrfs check without --repair and report back with the full output. Once you get things fixed up, it seems going back to the CentOS 7.1 kernel is fine. But periodically you might have to revert to a newer kernel for some maintenance, at least until the fix for the problem makes it into a future CentOS kernel. Chris Murphy ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: btrfs full, but not full, can't rebalance 2015-07-03 18:04 ` Chris Murphy @ 2015-07-12 6:10 ` Rich Rauenzahn 0 siblings, 0 replies; 10+ messages in thread From: Rich Rauenzahn @ 2015-07-12 6:10 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS Just a final note -- I'm finally back in person with the CentOS 7 server and so booted it to the latest kernel-ml from elrepo. It is a 4.1 kernel. But while still remote with the older 3.10 kernel, I also tried doing a 'mount -oremount,clear_cache /' I can't swear it helped, but things did seem to get better afterwards... so it may be worth a try for anyone reading this thread with a similar problem later. (Unless someone can definitively say that clear_cache doesn't work with -oremount.) Rich ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-07-12 6:10 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-07-03 5:32 btrfs full, but not full, can't rebalance Rich Rauenzahn 2015-07-03 5:45 ` Donald Pearson 2015-07-03 5:57 ` Rich Rauenzahn 2015-07-03 6:01 ` Donald Pearson 2015-07-03 6:03 ` Rich Rauenzahn 2015-07-03 6:05 ` Donald Pearson 2015-07-03 7:56 ` Rich Rauenzahn 2015-07-03 17:28 ` Rich Rauenzahn 2015-07-03 18:04 ` Chris Murphy 2015-07-12 6:10 ` Rich Rauenzahn
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).