* Out of space error even though there's 100 GB unused?
@ 2016-07-06 9:55 Stanislaw Kaminski
[not found] ` <CAGGqMYQeAN3HR7s-Om864pfV2n_qfZN6Wyyamj=EwKmm4o5NgA@mail.gmail.com>
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Stanislaw Kaminski @ 2016-07-06 9:55 UTC (permalink / raw)
To: linux-btrfs
Hi,
I am fighting with this since at least Monday - see
https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left
Here's the data:
# uname -a
Linux archb3 4.6.3-2-ARCH #1 PREEMPT Wed Jun 29 07:15:33 MDT 2016
armv5tel GNU/Linux
# btrfs --version
btrfs-progs v4.6
# btrfs fi show
Label: 'home' uuid: 1c7e35e8-f013-4f65-9d19-eaa168ac088b
Total devices 1 FS bytes used 1.71TiB
devid 1 size 1.81TiB used 1.71TiB path /dev/sda4
# btrfs fi df /home
Data, single: total=1.71TiB, used=1.71TiB
System, DUP: total=32.00MiB, used=224.00KiB
Metadata, DUP: total=4.00GiB, used=2.07GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs f usage -T /home
Overall:
Device size: 1.81TiB
Device allocated: 1.71TiB
Device unallocated: 97.89GiB
Device missing: 0.00B
Used: 1.71TiB
Free (estimated): 98.22GiB (min: 49.27GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data Metadata System
Id Path single DUP DUP Unallocated
-- --------- ------- -------- --------- -----------
1 /dev/sda4 1.71TiB 8.00GiB 64.00MiB 97.89GiB
-- --------- ------- -------- --------- -----------
Total 1.71TiB 4.00GiB 32.00MiB 97.89GiB
Used 1.71TiB 2.07GiB 224.00KiB
# btrfs fi du -s /home
Total Exclusive Set shared Filename
1.60TiB 1.60TiB 0.00B /home
# btrfs f resize 1:+1G /home/
Resize '/home/' of '1:+1G'
ERROR: unable to resize '/home/': no enough free space
This all is after closely following:
https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
So, already did full volume rebalance, defrag, rebooted multiple times
- still, "Error: out of disk space".
To sum up:
- my files sum to 1.6 TiB
- disk usage is shown to be 1.71 TiB
- volume size is 1.81 TiB
- btrfs util shows I have ~98 GiB free space on the volume
- I am getting "out of space" message
Bonus:
- I removed 50 GB of data from the drive and I still get "out of
space" message after writing ~1 GB.
Help would be very appreciated.
Cheers,
Stan
^ permalink raw reply [flat|nested] 10+ messages in thread[parent not found: <CAGGqMYQeAN3HR7s-Om864pfV2n_qfZN6Wyyamj=EwKmm4o5NgA@mail.gmail.com>]
* Re: Out of space error even though there's 100 GB unused? [not found] ` <CAGGqMYQeAN3HR7s-Om864pfV2n_qfZN6Wyyamj=EwKmm4o5NgA@mail.gmail.com> @ 2016-07-06 10:30 ` Stanislaw Kaminski 0 siblings, 0 replies; 10+ messages in thread From: Stanislaw Kaminski @ 2016-07-06 10:30 UTC (permalink / raw) To: Alexander Fougner; +Cc: linux-btrfs Hi Alex, Thank for having a look. "You're trying to resize a fs that is probably already fully using the block device it's on. I don't see anything incorrect happening here, but I might be missing something." This was just to show that I can't do this, I know that it is already utilizing the entire block device. "The unallocated space will be allocated if you start writing files to it." That's what I would expect, unfortunately it's kind of hard to write files to it, as I get "Out of space" error. Tongue-in-cheek, if you know how to ignore the issue and start writing files, it would solve my issue. Bottom line: if the disk is really full, then none of the tools shows that. If it is not (and I suspect it's not - as I mentioned, I just removed 50 GB of data from it), then why am I getting "out of space"? As for block device size: # fdisk -l Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 6070B645-D738-4730-BEF7-989210EF1DD7 Device Start End Sectors Size Type /dev/sda1 2048 133119 131072 64M Linux filesystem /dev/sda2 133120 2230271 2097152 1G Linux swap /dev/sda3 2230272 19007487 16777216 8G Linux filesystem /dev/sda4 19007488 3907029134 3888021647 1.8T Linux home Cheers, Stan 2016-07-06 12:10 GMT+02:00 Alexander Fougner <fougner89@gmail.com>: > > Den 6 juli 2016 12:03 em skrev "Stanislaw Kaminski" > <stasheck.fora@gmail.com>: >> > >> Hi, >> I am fighting with this since at least Monday - see >> >> https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left >> >> Here's the data: >> # uname -a >> Linux archb3 4.6.3-2-ARCH #1 PREEMPT Wed Jun 29 07:15:33 MDT 2016 >> armv5tel GNU/Linux >> >> # btrfs --version >> btrfs-progs v4.6 >> >> # btrfs fi show >> Label: 'home' uuid: 1c7e35e8-f013-4f65-9d19-eaa168ac088b >> Total devices 1 FS bytes used 1.71TiB >> devid 1 size 1.81TiB used 1.71TiB path /dev/sda4 >> >> # btrfs fi df /home >> Data, single: total=1.71TiB, used=1.71TiB >> System, DUP: total=32.00MiB, used=224.00KiB >> Metadata, DUP: total=4.00GiB, used=2.07GiB >> GlobalReserve, single: total=512.00MiB, used=0.00B >> >> # btrfs f usage -T /home >> Overall: >> Device size: 1.81TiB >> Device allocated: 1.71TiB >> Device unallocated: 97.89GiB >> Device missing: 0.00B >> Used: 1.71TiB >> Free (estimated): 98.22GiB (min: 49.27GiB) >> Data ratio: 1.00 >> Metadata ratio: 2.00 >> Global reserve: 512.00MiB (used: 0.00B) >> >> Data Metadata System >> Id Path single DUP DUP Unallocated >> -- --------- ------- -------- --------- ----------- >> 1 /dev/sda4 1.71TiB 8.00GiB 64.00MiB 97.89GiB >> -- --------- ------- -------- --------- ----------- >> Total 1.71TiB 4.00GiB 32.00MiB 97.89GiB >> Used 1.71TiB 2.07GiB 224.00KiB >> >> # btrfs fi du -s /home >> Total Exclusive Set shared Filename >> 1.60TiB 1.60TiB 0.00B /home >> >> # btrfs f resize 1:+1G /home/ >> Resize '/home/' of '1:+1G' >> ERROR: unable to resize '/home/': no enough free space >> > > You're trying to resize a fs that is probably already fully using the block > device it's on. I don't see anything incorrect happening here, but I might > be missing something. > > The used space amounting to 1.6TiB is not as reliable as the btrfs fi df > tool. > The unallocated space will be allocated if you start writing files to it. > What size is the parent block device? > >> This all is after closely following: >> >> https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21 >> >> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html >> >> So, already did full volume rebalance, defrag, rebooted multiple times >> - still, "Error: out of disk space". >> >> To sum up: >> - my files sum to 1.6 TiB >> - disk usage is shown to be 1.71 TiB >> - volume size is 1.81 TiB >> - btrfs util shows I have ~98 GiB free space on the volume >> - I am getting "out of space" message >> >> Bonus: >> - I removed 50 GB of data from the drive and I still get "out of >> space" message after writing ~1 GB. >> >> Help would be very appreciated. >> >> Cheers, >> Stan >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Out of space error even though there's 100 GB unused? 2016-07-06 9:55 Out of space error even though there's 100 GB unused? Stanislaw Kaminski [not found] ` <CAGGqMYQeAN3HR7s-Om864pfV2n_qfZN6Wyyamj=EwKmm4o5NgA@mail.gmail.com> @ 2016-07-06 10:34 ` Hugo Mills 2016-07-06 10:43 ` Stanislaw Kaminski 2016-07-06 17:42 ` Chris Murphy 2 siblings, 1 reply; 10+ messages in thread From: Hugo Mills @ 2016-07-06 10:34 UTC (permalink / raw) To: Stanislaw Kaminski; +Cc: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 3216 bytes --] On Wed, Jul 06, 2016 at 11:55:42AM +0200, Stanislaw Kaminski wrote: > Hi, > I am fighting with this since at least Monday - see > https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left > > Here's the data: > # uname -a > Linux archb3 4.6.3-2-ARCH #1 PREEMPT Wed Jun 29 07:15:33 MDT 2016 > armv5tel GNU/Linux > > # btrfs --version > btrfs-progs v4.6 > > # btrfs fi show > Label: 'home' uuid: 1c7e35e8-f013-4f65-9d19-eaa168ac088b > Total devices 1 FS bytes used 1.71TiB > devid 1 size 1.81TiB used 1.71TiB path /dev/sda4 In this state, you should definitely not be seeing out of space errors. This is, therefore, a bug you're seeing. I've not been following things as closely as I'd like of late, but I think there was a bug recently involving the free space cache. It might be worth unmounting the FS and mounting again with the nospace_cache option, just to see if that helps. Hugo. > # btrfs fi df /home > Data, single: total=1.71TiB, used=1.71TiB > System, DUP: total=32.00MiB, used=224.00KiB > Metadata, DUP: total=4.00GiB, used=2.07GiB > GlobalReserve, single: total=512.00MiB, used=0.00B > > # btrfs f usage -T /home > Overall: > Device size: 1.81TiB > Device allocated: 1.71TiB > Device unallocated: 97.89GiB > Device missing: 0.00B > Used: 1.71TiB > Free (estimated): 98.22GiB (min: 49.27GiB) > Data ratio: 1.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 0.00B) > > Data Metadata System > Id Path single DUP DUP Unallocated > -- --------- ------- -------- --------- ----------- > 1 /dev/sda4 1.71TiB 8.00GiB 64.00MiB 97.89GiB > -- --------- ------- -------- --------- ----------- > Total 1.71TiB 4.00GiB 32.00MiB 97.89GiB > Used 1.71TiB 2.07GiB 224.00KiB > > # btrfs fi du -s /home > Total Exclusive Set shared Filename > 1.60TiB 1.60TiB 0.00B /home > > # btrfs f resize 1:+1G /home/ > Resize '/home/' of '1:+1G' > ERROR: unable to resize '/home/': no enough free space > > This all is after closely following: > https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21 > http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html > > So, already did full volume rebalance, defrag, rebooted multiple times > - still, "Error: out of disk space". > > To sum up: > - my files sum to 1.6 TiB > - disk usage is shown to be 1.71 TiB > - volume size is 1.81 TiB > - btrfs util shows I have ~98 GiB free space on the volume > - I am getting "out of space" message > > Bonus: > - I removed 50 GB of data from the drive and I still get "out of > space" message after writing ~1 GB. > > Help would be very appreciated. > > Cheers, > Stan -- Hugo Mills | You can play with your friends' privates, but you hugo@... carfax.org.uk | can't play with your friends' childrens' privates. http://carfax.org.uk/ | PGP: E2AB1DE4 | C++ coding rule [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Out of space error even though there's 100 GB unused? 2016-07-06 10:34 ` Hugo Mills @ 2016-07-06 10:43 ` Stanislaw Kaminski 0 siblings, 0 replies; 10+ messages in thread From: Stanislaw Kaminski @ 2016-07-06 10:43 UTC (permalink / raw) To: Hugo Mills, Stanislaw Kaminski, linux-btrfs Hi Hugo, I agree that it seems to be a bug, and I'll be glad to help nail that down - if only because I have no other drive to move the data to :-) As for your suggestion - no change: [root@archb3 stan]# mount | grep home /dev/sda4 on /home type btrfs (rw,relatime,nospace_cache,clear_cache,subvolid=5,subvol=/) [root@archb3 stan]# touch test touch: cannot touch 'test': No space left on device Cheers, Stan 2016-07-06 12:34 GMT+02:00 Hugo Mills <hugo@carfax.org.uk>: > On Wed, Jul 06, 2016 at 11:55:42AM +0200, Stanislaw Kaminski wrote: >> Hi, >> I am fighting with this since at least Monday - see >> https://superuser.com/questions/1096658/btrfs-out-of-space-even-though-there-should-be-10-left >> >> Here's the data: >> # uname -a >> Linux archb3 4.6.3-2-ARCH #1 PREEMPT Wed Jun 29 07:15:33 MDT 2016 >> armv5tel GNU/Linux >> >> # btrfs --version >> btrfs-progs v4.6 >> >> # btrfs fi show >> Label: 'home' uuid: 1c7e35e8-f013-4f65-9d19-eaa168ac088b >> Total devices 1 FS bytes used 1.71TiB >> devid 1 size 1.81TiB used 1.71TiB path /dev/sda4 > > In this state, you should definitely not be seeing out of space > errors. This is, therefore, a bug you're seeing. > > I've not been following things as closely as I'd like of late, but > I think there was a bug recently involving the free space cache. It > might be worth unmounting the FS and mounting again with the > nospace_cache option, just to see if that helps. > > Hugo. > >> # btrfs fi df /home >> Data, single: total=1.71TiB, used=1.71TiB >> System, DUP: total=32.00MiB, used=224.00KiB >> Metadata, DUP: total=4.00GiB, used=2.07GiB >> GlobalReserve, single: total=512.00MiB, used=0.00B >> >> # btrfs f usage -T /home >> Overall: >> Device size: 1.81TiB >> Device allocated: 1.71TiB >> Device unallocated: 97.89GiB >> Device missing: 0.00B >> Used: 1.71TiB >> Free (estimated): 98.22GiB (min: 49.27GiB) >> Data ratio: 1.00 >> Metadata ratio: 2.00 >> Global reserve: 512.00MiB (used: 0.00B) >> >> Data Metadata System >> Id Path single DUP DUP Unallocated >> -- --------- ------- -------- --------- ----------- >> 1 /dev/sda4 1.71TiB 8.00GiB 64.00MiB 97.89GiB >> -- --------- ------- -------- --------- ----------- >> Total 1.71TiB 4.00GiB 32.00MiB 97.89GiB >> Used 1.71TiB 2.07GiB 224.00KiB >> >> # btrfs fi du -s /home >> Total Exclusive Set shared Filename >> 1.60TiB 1.60TiB 0.00B /home >> >> # btrfs f resize 1:+1G /home/ >> Resize '/home/' of '1:+1G' >> ERROR: unable to resize '/home/': no enough free space >> >> This all is after closely following: >> https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_I_ran_out_of_disk_space.21 >> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html >> >> So, already did full volume rebalance, defrag, rebooted multiple times >> - still, "Error: out of disk space". >> >> To sum up: >> - my files sum to 1.6 TiB >> - disk usage is shown to be 1.71 TiB >> - volume size is 1.81 TiB >> - btrfs util shows I have ~98 GiB free space on the volume >> - I am getting "out of space" message >> >> Bonus: >> - I removed 50 GB of data from the drive and I still get "out of >> space" message after writing ~1 GB. >> >> Help would be very appreciated. >> >> Cheers, >> Stan > > -- > Hugo Mills | You can play with your friends' privates, but you > hugo@... carfax.org.uk | can't play with your friends' childrens' privates. > http://carfax.org.uk/ | > PGP: E2AB1DE4 | C++ coding rule ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Out of space error even though there's 100 GB unused? 2016-07-06 9:55 Out of space error even though there's 100 GB unused? Stanislaw Kaminski [not found] ` <CAGGqMYQeAN3HR7s-Om864pfV2n_qfZN6Wyyamj=EwKmm4o5NgA@mail.gmail.com> 2016-07-06 10:34 ` Hugo Mills @ 2016-07-06 17:42 ` Chris Murphy 2016-07-07 10:18 ` Stanislaw Kaminski 2 siblings, 1 reply; 10+ messages in thread From: Chris Murphy @ 2016-07-06 17:42 UTC (permalink / raw) To: Stanislaw Kaminski; +Cc: Btrfs BTRFS On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski <stasheck.fora@gmail.com> wrote: > Device unallocated: 97.89GiB There should be no problem creating any type of block group from this much space. It's a bug. I would try regression testing. Kernel 4.5.7 has some changes that may or may not relate to this (they should only relate when there is no unallocated space left) so you could try 4.5.6 and 4.5.7. And also 4.4.14. But also the kernel messages are important. There is this obscure enospc with error -28, so either with or without enospc_debug mount option is useful to try in 4.6.3 (I think it's less useful in older kernels). But do try nospace_cache first. If that works, you could then mount with clear_cache one time and see if that provides an enduring fix. It can take some time to rebuild the cache after clear_cache is used. -- Chris Murphy ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Out of space error even though there's 100 GB unused? 2016-07-06 17:42 ` Chris Murphy @ 2016-07-07 10:18 ` Stanislaw Kaminski 2016-07-07 10:28 ` Stanislaw Kaminski 0 siblings, 1 reply; 10+ messages in thread From: Stanislaw Kaminski @ 2016-07-07 10:18 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS Hi all, I downgraded to 4.4.1-1 - all fine, 4.5.5.-1 - also fine, then got back to 4.6.3-2 - and it's still fine. Apparently running under different kernel somehow fixed the glitch (as far as I can test...). That leaves me with the other question: before issues, I 1.6 TiB was used, now all the tools report 1.7 TiB issued (except for btrfs fs du /home, this reports 1.6 TiB). How is that possible? Cheers, Stan 2016-07-06 19:42 GMT+02:00 Chris Murphy <lists@colorremedies.com>: > On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski > <stasheck.fora@gmail.com> wrote: > >> Device unallocated: 97.89GiB > > There should be no problem creating any type of block group from this > much space. It's a bug. > > I would try regression testing. Kernel 4.5.7 has some changes that may > or may not relate to this (they should only relate when there is no > unallocated space left) so you could try 4.5.6 and 4.5.7. And also > 4.4.14. > > But also the kernel messages are important. There is this obscure > enospc with error -28, so either with or without enospc_debug mount > option is useful to try in 4.6.3 (I think it's less useful in older > kernels). > > But do try nospace_cache first. If that works, you could then mount > with clear_cache one time and see if that provides an enduring fix. It > can take some time to rebuild the cache after clear_cache is used. > > > > -- > Chris Murphy ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Out of space error even though there's 100 GB unused? 2016-07-07 10:18 ` Stanislaw Kaminski @ 2016-07-07 10:28 ` Stanislaw Kaminski 2016-07-07 15:17 ` Stanislaw Kaminski 0 siblings, 1 reply; 10+ messages in thread From: Stanislaw Kaminski @ 2016-07-07 10:28 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS Too early report, the issue is back. Back to testing.... 2016-07-07 12:18 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>: > Hi all, > I downgraded to 4.4.1-1 - all fine, 4.5.5.-1 - also fine, then got > back to 4.6.3-2 - and it's still fine. Apparently running under > different kernel somehow fixed the glitch (as far as I can test...). > > That leaves me with the other question: before issues, I 1.6 TiB was > used, now all the tools report 1.7 TiB issued (except for btrfs fs du > /home, this reports 1.6 TiB). How is that possible? > > Cheers, > Stan > > 2016-07-06 19:42 GMT+02:00 Chris Murphy <lists@colorremedies.com>: >> On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski >> <stasheck.fora@gmail.com> wrote: >> >>> Device unallocated: 97.89GiB >> >> There should be no problem creating any type of block group from this >> much space. It's a bug. >> >> I would try regression testing. Kernel 4.5.7 has some changes that may >> or may not relate to this (they should only relate when there is no >> unallocated space left) so you could try 4.5.6 and 4.5.7. And also >> 4.4.14. >> >> But also the kernel messages are important. There is this obscure >> enospc with error -28, so either with or without enospc_debug mount >> option is useful to try in 4.6.3 (I think it's less useful in older >> kernels). >> >> But do try nospace_cache first. If that works, you could then mount >> with clear_cache one time and see if that provides an enduring fix. It >> can take some time to rebuild the cache after clear_cache is used. >> >> >> >> -- >> Chris Murphy ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Out of space error even though there's 100 GB unused? 2016-07-07 10:28 ` Stanislaw Kaminski @ 2016-07-07 15:17 ` Stanislaw Kaminski 2016-07-07 16:36 ` Henk Slager 0 siblings, 1 reply; 10+ messages in thread From: Stanislaw Kaminski @ 2016-07-07 15:17 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS Hi Chris, Alex, Hugo, Running now: Linux archb3 4.6.2-1-ARCH #1 PREEMPT Mon Jun 13 02:11:34 MDT 2016 armv5tel GNU/Linux Seems to be working fine. I started a defrag, and it seems I'm getting my space back: $ sudo btrfs fi usage /home Overall: Device size: 1.81TiB Device allocated: 1.73TiB Device unallocated: 80.89GiB Device missing: 0.00B Used: 1.65TiB Free (estimated): 159.63GiB (min: 119.19GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 240.00KiB) Data,single: Size:1.72TiB, Used:1.65TiB /dev/sda4 1.72TiB Metadata,DUP: Size:3.50GiB, Used:2.16GiB /dev/sda4 7.00GiB System,DUP: Size:32.00MiB, Used:224.00KiB /dev/sda4 64.00MiB Unallocated: /dev/sda4 80.89GiB I deleted some unfinished torrent, ~10 GB in size, but as you can see, "Free space" has grown by 60 GB (re-checked now and it's 1 GB more now - so definitely caused by defrag). What has changed between 4.6.2 and 4.6.3? Cheers, Stan 2016-07-07 12:28 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>: > Too early report, the issue is back. Back to testing.... > > 2016-07-07 12:18 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>: >> Hi all, >> I downgraded to 4.4.1-1 - all fine, 4.5.5.-1 - also fine, then got >> back to 4.6.3-2 - and it's still fine. Apparently running under >> different kernel somehow fixed the glitch (as far as I can test...). >> >> That leaves me with the other question: before issues, I 1.6 TiB was >> used, now all the tools report 1.7 TiB issued (except for btrfs fs du >> /home, this reports 1.6 TiB). How is that possible? >> >> Cheers, >> Stan >> >> 2016-07-06 19:42 GMT+02:00 Chris Murphy <lists@colorremedies.com>: >>> On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski >>> <stasheck.fora@gmail.com> wrote: >>> >>>> Device unallocated: 97.89GiB >>> >>> There should be no problem creating any type of block group from this >>> much space. It's a bug. >>> >>> I would try regression testing. Kernel 4.5.7 has some changes that may >>> or may not relate to this (they should only relate when there is no >>> unallocated space left) so you could try 4.5.6 and 4.5.7. And also >>> 4.4.14. >>> >>> But also the kernel messages are important. There is this obscure >>> enospc with error -28, so either with or without enospc_debug mount >>> option is useful to try in 4.6.3 (I think it's less useful in older >>> kernels). >>> >>> But do try nospace_cache first. If that works, you could then mount >>> with clear_cache one time and see if that provides an enduring fix. It >>> can take some time to rebuild the cache after clear_cache is used. >>> >>> >>> >>> -- >>> Chris Murphy ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Out of space error even though there's 100 GB unused? 2016-07-07 15:17 ` Stanislaw Kaminski @ 2016-07-07 16:36 ` Henk Slager [not found] ` <CALLEBqFetJRAnwc0sYG-1seAGFBQu9-2nstH8hg-a1drdVdW_g@mail.gmail.com> 0 siblings, 1 reply; 10+ messages in thread From: Henk Slager @ 2016-07-07 16:36 UTC (permalink / raw) To: Stanislaw Kaminski; +Cc: Chris Murphy, Btrfs BTRFS On Thu, Jul 7, 2016 at 5:17 PM, Stanislaw Kaminski <stasheck.fora@gmail.com> wrote: > Hi Chris, Alex, Hugo, > > Running now: Linux archb3 4.6.2-1-ARCH #1 PREEMPT Mon Jun 13 02:11:34 > MDT 2016 armv5tel GNU/Linux > > Seems to be working fine. I started a defrag, and it seems I'm getting > my space back: > $ sudo btrfs fi usage /home > Overall: > Device size: 1.81TiB > Device allocated: 1.73TiB > Device unallocated: 80.89GiB > Device missing: 0.00B > Used: 1.65TiB > Free (estimated): 159.63GiB (min: 119.19GiB) > Data ratio: 1.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 240.00KiB) > > Data,single: Size:1.72TiB, Used:1.65TiB > /dev/sda4 1.72TiB > > Metadata,DUP: Size:3.50GiB, Used:2.16GiB > /dev/sda4 7.00GiB > > System,DUP: Size:32.00MiB, Used:224.00KiB > /dev/sda4 64.00MiB > > Unallocated: > /dev/sda4 80.89GiB > > I deleted some unfinished torrent, ~10 GB in size, but as you can see, > "Free space" has grown by 60 GB (re-checked now and it's 1 GB more now > - so definitely caused by defrag). > > What has changed between 4.6.2 and 4.6.3? https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/diff/?id=v4.6.3&id2=v4.6.2&dt=2 I see no change for Btrfs > Cheers, > Stan > > 2016-07-07 12:28 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>: >> Too early report, the issue is back. Back to testing.... >> >> 2016-07-07 12:18 GMT+02:00 Stanislaw Kaminski <stasheck.fora@gmail.com>: >>> Hi all, >>> I downgraded to 4.4.1-1 - all fine, 4.5.5.-1 - also fine, then got >>> back to 4.6.3-2 - and it's still fine. Apparently running under >>> different kernel somehow fixed the glitch (as far as I can test...). >>> >>> That leaves me with the other question: before issues, I 1.6 TiB was >>> used, now all the tools report 1.7 TiB issued (except for btrfs fs du >>> /home, this reports 1.6 TiB). How is that possible? >>> >>> Cheers, >>> Stan >>> >>> 2016-07-06 19:42 GMT+02:00 Chris Murphy <lists@colorremedies.com>: >>>> On Wed, Jul 6, 2016 at 3:55 AM, Stanislaw Kaminski >>>> <stasheck.fora@gmail.com> wrote: >>>> >>>>> Device unallocated: 97.89GiB >>>> >>>> There should be no problem creating any type of block group from this >>>> much space. It's a bug. >>>> >>>> I would try regression testing. Kernel 4.5.7 has some changes that may >>>> or may not relate to this (they should only relate when there is no >>>> unallocated space left) so you could try 4.5.6 and 4.5.7. And also >>>> 4.4.14. >>>> >>>> But also the kernel messages are important. There is this obscure >>>> enospc with error -28, so either with or without enospc_debug mount >>>> option is useful to try in 4.6.3 (I think it's less useful in older >>>> kernels). >>>> >>>> But do try nospace_cache first. If that works, you could then mount >>>> with clear_cache one time and see if that provides an enduring fix. It >>>> can take some time to rebuild the cache after clear_cache is used. >>>> >>>> >>>> >>>> -- >>>> Chris Murphy > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
[parent not found: <CALLEBqFetJRAnwc0sYG-1seAGFBQu9-2nstH8hg-a1drdVdW_g@mail.gmail.com>]
* Re: Out of space error even though there's 100 GB unused? [not found] ` <CALLEBqFetJRAnwc0sYG-1seAGFBQu9-2nstH8hg-a1drdVdW_g@mail.gmail.com> @ 2016-07-08 14:43 ` Henk Slager 0 siblings, 0 replies; 10+ messages in thread From: Henk Slager @ 2016-07-08 14:43 UTC (permalink / raw) To: Stanislaw Kaminski; +Cc: Btrfs BTRFS On Fri, Jul 8, 2016 at 9:22 AM, Stanislaw Kaminski <stasheck.fora@gmail.com> wrote: > Huh. > > I left defrag running overnight, and now I'm back to my >200 GiB free > space. Also, I got no "out of space" messages in Transmission, and it > successfully downloaded few GBs. > > But in dmesg I have 209 traces, see attached. > > Does that say anything to you? I get some vague clue about the problem, but no-one seems to know exactly the root cause(s). The 4.6.2 code where the warning comes from is this: ... /* * Called if we need to clear a data reservation for this inode * Normally in a error case. * * This one will *NOT* use accurate qgroup reserved space API, just for case * which we can't sleep and is sure it won't affect qgroup reserved space. * Like clear_bit_hook(). */ void btrfs_free_reserved_data_space_noquota(struct inode *inode, u64 start, u64 len) { struct btrfs_root *root = BTRFS_I(inode)->root; struct btrfs_space_info *data_sinfo; /* Make sure the range is aligned to sectorsize */ len = round_up(start + len, root->sectorsize) - round_down(start, root->sectorsize); start = round_down(start, root->sectorsize); data_sinfo = root->fs_info->data_sinfo; spin_lock(&data_sinfo->lock); if (WARN_ON(data_sinfo->bytes_may_use < len)) data_sinfo->bytes_may_use = 0; else data_sinfo->bytes_may_use -= len; trace_btrfs_space_reservation(root->fs_info, "space_info", data_sinfo->flags, len, 0); spin_unlock(&data_sinfo->lock); } ... I think the system is still 'recovering' from getting stuck earlier. What exactly that is, I don't know. You would probably need to enable more debugging facilities in order to figure out from which file or inode the problem comes from. I don't know if you can compile a 4.7 kernel for this Kirkwood SoC, but that would be one way forward. (BTW, is it a 88F6281 or a 88F6282 ?) As far as I remember, Josef Basik has posted some patches that could benefit this case, I am not sure if they made it in 4.7, but that is what I think I would try. Otherwise, it are workarounds: - you could have a look at the cpuload during defrag and normal operation and see how it relates to the rate of issuing this warning - add mount option noatime - as it looks like this this fs is (also) a torrent-client target, you can put the torrents in a directoru ot subvol with noCoW flag set or mount the whole fs with nodatacow - again clean cache and then mount with space_cache=v2. Then new mounts will use v2 then automatically. Only thing I can say it that helped me getting out of kernel crash situation with 4.6.2. 4.7.0-rc5 did not crash on the same fs, so I got it working again ( de-allocations and cleanups, the fs is almost exclusively a btrfs-receive target) - connect the fs to a multi-core x86_64 system running same kernel version for some time and see if you can reproduce the same type of WARN_ONs ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2016-07-08 14:43 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-07-06 9:55 Out of space error even though there's 100 GB unused? Stanislaw Kaminski
[not found] ` <CAGGqMYQeAN3HR7s-Om864pfV2n_qfZN6Wyyamj=EwKmm4o5NgA@mail.gmail.com>
2016-07-06 10:30 ` Stanislaw Kaminski
2016-07-06 10:34 ` Hugo Mills
2016-07-06 10:43 ` Stanislaw Kaminski
2016-07-06 17:42 ` Chris Murphy
2016-07-07 10:18 ` Stanislaw Kaminski
2016-07-07 10:28 ` Stanislaw Kaminski
2016-07-07 15:17 ` Stanislaw Kaminski
2016-07-07 16:36 ` Henk Slager
[not found] ` <CALLEBqFetJRAnwc0sYG-1seAGFBQu9-2nstH8hg-a1drdVdW_g@mail.gmail.com>
2016-07-08 14:43 ` Henk Slager
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).