* strange "No space left on device"
@ 2015-11-07 23:22 Christoph Anton Mitterer
2015-11-07 23:30 ` Hugo Mills
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Christoph Anton Mitterer @ 2015-11-07 23:22 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 8486 bytes --]
Hey.
I just repeatedly did the following twice on a ~8GB USB stick, under
Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).
First, created some GPT on the stick:
Number Start (sector) End (sector) Size Code Name
1 2048 1048575 511.0 MiB EF02 BIOS boot partition
2 1048576 3145727 1024.0 MiB 8300 Linux filesystem
3 3145728 4194303 512.0 MiB 8300 Linux filesystem
4 4194304 6291455 1024.0 MiB 8300 Linux filesystem
5 6291456 15687644 4.5 GiB 8300 Linux filesystem
Then filesystems:
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Label: boot-main
UUID: 150ee9fb-c650-4b5b-8f64-606e589e429a
Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 59.19MiB
System: DUP 12.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb2
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Label: boot-data
UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772
Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 59.19MiB
System: DUP 12.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb3
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive
/dev/sdb4
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Label: boot-archive
UUID: a063cf3b-0322-4ec7-a8c1-2dabecad9f57
Node size: 16384
Sector size: 4096
Filesystem size: 1.00GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 59.19MiB
System: DUP 12.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 1.00GiB /dev/sdb4
root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5
btrfs-progs v4.2.2
See http://btrfs.wiki.kernel.org for more information.
Label: boot-rescue
UUID: 104b7bc3-3b8c-4a08-b0a6-9172ed664e68
Node size: 16384
Sector size: 4096
Filesystem size: 3.98GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 211.75MiB
System: DUP 12.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 3.98GiB /dev/sdb5
No errors in the kernel log.
Then I got usage info:
root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/
Overall:
Device size: 1.00GiB
Device allocated: 126.38MiB
Device unallocated: 897.62MiB
Device missing: 0.00B
Used: 256.00KiB
Free (estimated): 905.62MiB (min: 456.81MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 16.00MiB (used: 0.00B)
Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb3 8.00MiB
Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb3 102.38MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb3 16.00MiB
Unallocated:
/dev/sdb3 897.62MiB
root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/
Overall:
Device size: 1.00GiB
Device allocated: 126.38MiB
Device unallocated: 897.62MiB
Device missing: 0.00B
Used: 256.00KiB
Free (estimated): 905.62MiB (min: 456.81MiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 16.00MiB (used: 0.00B)
Data,single: Size:8.00MiB, Used:0.00B
/dev/sdb2 8.00MiB
Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdb2 102.38MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb2 16.00MiB
Unallocated:
/dev/sdb2 897.62MiB
root@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/
Overall:
Device size: 3.98GiB
Device allocated: 431.50MiB
Device unallocated: 3.56GiB
Device missing: 0.00B
Used: 320.00KiB
Free (estimated): 3.57GiB (min: 1.79GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 16.00MiB (used: 0.00B)
Data,single: Size:8.00MiB, Used:64.00KiB
/dev/sdb5 8.00MiB
Metadata,DUP: Size:203.75MiB, Used:112.00KiB
/dev/sdb5 407.50MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdb5 16.00MiB
Unallocated:
/dev/sdb5 3.56GiB
Still all fine.
Now I write to one of the filesystems (main at first, that still works)
but then when I cp -a from another btrfs to data:
cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
cp: error writing ‘data/SHA512-sums’: No space left on device
cp: failed to extend ‘data/SHA512-sums’: No space left on device
(these files are just a few bytes large)
On the target fs, they're there but all 0 bytes large.
(btw: I've aliased cp to cp --reflink=auto
When I now repeat the usage info:
root@heisenberg:/data/SSS/boot# btrfs filesystem usage data
Overall:
Device size: 1.00GiB
Device allocated: 118.38MiB
Device unallocated: 905.62MiB
Device missing: 0.00B
Used: 256.00KiB
Free (estimated): 8.00EiB (min: 8.00EiB)
Data ratio: -nan
Metadata ratio: 2.00
Global reserve: 16.00MiB (used: 0.00B)
Metadata,DUP: Size:51.19MiB, Used:112.00KiB
/dev/sdc3 102.38MiB
System,DUP: Size:8.00MiB, Used:16.00KiB
/dev/sdc3 16.00MiB
Unallocated:
/dev/sdc3 905.62MiB
It get's bogus... at least the Free space shows 8 EiB (if wishes were
horses)...
Still no error in kernel logs:
3401.934514] BTRFS: device label boot-rescue devid 1 transid 8
/dev/sdc5
[ 3401.985858] BTRFS: device label boot-main devid 1 transid 8
/dev/sdc2
[ 3401.988601] BTRFS: device label boot-data devid 1 transid 9
/dev/sdc3
[ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9
/dev/sdc4
[ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled
[ 3403.721098] BTRFS: has skinny extents
[ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled
[ 3413.964042] BTRFS: has skinny extents
[ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled
[ 3522.902314] BTRFS: has skinny extents
[ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled
[ 3525.055659] BTRFS: has skinny extents
btrfs check doesn't show errors either.
After a balance, i can copy again, even though the usage still shows 8
EiB for a while...
Anyway how can it happen, that on a fresh btrfs with no files at all a
balance is necessary?
Or is there some deeper corruption going on here?
Thanks,
Chris.
[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: strange "No space left on device"
2015-11-07 23:22 strange "No space left on device" Christoph Anton Mitterer
@ 2015-11-07 23:30 ` Hugo Mills
2015-11-08 0:38 ` Christoph Anton Mitterer
2015-11-08 18:18 ` Henk Slager
2015-11-09 11:12 ` Filipe Manana
2 siblings, 1 reply; 8+ messages in thread
From: Hugo Mills @ 2015-11-07 23:30 UTC (permalink / raw)
To: Christoph Anton Mitterer; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 9612 bytes --]
On Sun, Nov 08, 2015 at 12:22:42AM +0100, Christoph Anton Mitterer wrote:
> Hey.
>
> I just repeatedly did the following twice on a ~8GB USB stick, under
> Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).
>
> First, created some GPT on the stick:
> Number Start (sector) End (sector) Size Code Name
> 1 2048 1048575 511.0 MiB EF02 BIOS boot partition
> 2 1048576 3145727 1024.0 MiB 8300 Linux filesystem
> 3 3145728 4194303 512.0 MiB 8300 Linux filesystem
> 4 4194304 6291455 1024.0 MiB 8300 Linux filesystem
> 5 6291456 15687644 4.5 GiB 8300 Linux filesystem
These are all really small.
I would suggest running mkfs with --mixed for all of these
filesystems and trying again.
Hugo.
> Then filesystems:
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-main
> UUID: 150ee9fb-c650-4b5b-8f64-606e589e429a
> Node size: 16384
> Sector size: 4096
> Filesystem size: 1.00GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 59.19MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 1.00GiB /dev/sdb2
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-data
> UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772
> Node size: 16384
> Sector size: 4096
> Filesystem size: 1.00GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 59.19MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 1.00GiB /dev/sdb3
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive
> /dev/sdb4
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-archive
> UUID: a063cf3b-0322-4ec7-a8c1-2dabecad9f57
> Node size: 16384
> Sector size: 4096
> Filesystem size: 1.00GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 59.19MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 1.00GiB /dev/sdb4
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-rescue
> UUID: 104b7bc3-3b8c-4a08-b0a6-9172ed664e68
> Node size: 16384
> Sector size: 4096
> Filesystem size: 3.98GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 211.75MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 3.98GiB /dev/sdb5
>
>
>
> No errors in the kernel log.
>
>
>
> Then I got usage info:
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/
> Overall:
> Device size: 1.00GiB
> Device allocated: 126.38MiB
> Device unallocated: 897.62MiB
> Device missing: 0.00B
> Used: 256.00KiB
> Free (estimated): 905.62MiB (min: 456.81MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Data,single: Size:8.00MiB, Used:0.00B
> /dev/sdb3 8.00MiB
>
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
> /dev/sdb3 102.38MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
> /dev/sdb3 16.00MiB
>
> Unallocated:
> /dev/sdb3 897.62MiB
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/
> Overall:
> Device size: 1.00GiB
> Device allocated: 126.38MiB
> Device unallocated: 897.62MiB
> Device missing: 0.00B
> Used: 256.00KiB
> Free (estimated): 905.62MiB (min: 456.81MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Data,single: Size:8.00MiB, Used:0.00B
> /dev/sdb2 8.00MiB
>
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
> /dev/sdb2 102.38MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
> /dev/sdb2 16.00MiB
>
> Unallocated:
> /dev/sdb2 897.62MiB
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/
> Overall:
> Device size: 3.98GiB
> Device allocated: 431.50MiB
> Device unallocated: 3.56GiB
> Device missing: 0.00B
> Used: 320.00KiB
> Free (estimated): 3.57GiB (min: 1.79GiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Data,single: Size:8.00MiB, Used:64.00KiB
> /dev/sdb5 8.00MiB
>
> Metadata,DUP: Size:203.75MiB, Used:112.00KiB
> /dev/sdb5 407.50MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
> /dev/sdb5 16.00MiB
>
> Unallocated:
> /dev/sdb5 3.56GiB
>
>
>
> Still all fine.
>
> Now I write to one of the filesystems (main at first, that still works)
> but then when I cp -a from another btrfs to data:
> cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
> cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
> cp: error writing ‘data/SHA512-sums’: No space left on device
> cp: failed to extend ‘data/SHA512-sums’: No space left on device
>
> (these files are just a few bytes large)
> On the target fs, they're there but all 0 bytes large.
>
> (btw: I've aliased cp to cp --reflink=auto
>
>
> When I now repeat the usage info:
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage data
> Overall:
> Device size: 1.00GiB
> Device allocated: 118.38MiB
> Device unallocated: 905.62MiB
> Device missing: 0.00B
> Used: 256.00KiB
> Free (estimated): 8.00EiB (min: 8.00EiB)
> Data ratio: -nan
> Metadata ratio: 2.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
> /dev/sdc3 102.38MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
> /dev/sdc3 16.00MiB
>
> Unallocated:
> /dev/sdc3 905.62MiB
>
> It get's bogus... at least the Free space shows 8 EiB (if wishes were
> horses)...
>
> Still no error in kernel logs:
> 3401.934514] BTRFS: device label boot-rescue devid 1 transid 8
> /dev/sdc5
> [ 3401.985858] BTRFS: device label boot-main devid 1 transid 8
> /dev/sdc2
> [ 3401.988601] BTRFS: device label boot-data devid 1 transid 9
> /dev/sdc3
> [ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9
> /dev/sdc4
> [ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled
> [ 3403.721098] BTRFS: has skinny extents
> [ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled
> [ 3413.964042] BTRFS: has skinny extents
> [ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled
> [ 3522.902314] BTRFS: has skinny extents
> [ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled
> [ 3525.055659] BTRFS: has skinny extents
>
>
> btrfs check doesn't show errors either.
>
>
> After a balance, i can copy again, even though the usage still shows 8
> EiB for a while...
>
>
> Anyway how can it happen, that on a fresh btrfs with no files at all a
> balance is necessary?
> Or is there some deeper corruption going on here?
>
>
> Thanks,
> Chris.
--
Hugo Mills | Reintarnation: Coming back from the dead as a
hugo@... carfax.org.uk | hillbilly
http://carfax.org.uk/ |
PGP: E2AB1DE4 |
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: strange "No space left on device"
2015-11-07 23:30 ` Hugo Mills
@ 2015-11-08 0:38 ` Christoph Anton Mitterer
0 siblings, 0 replies; 8+ messages in thread
From: Christoph Anton Mitterer @ 2015-11-08 0:38 UTC (permalink / raw)
To: Hugo Mills; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 845 bytes --]
On Sat, 2015-11-07 at 23:30 +0000, Hugo Mills wrote:
> These are all really small.
Well enough for booting =)
> I would suggest running mkfs with --mixed for all of these
> filesystems and trying again.
I thought btrfs would do that automatically:
https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_small
"Using mixed block groups is recommended for filesystems of 1GiB or
smaller and mkfs.btrfs will force mixed block groups automatically in
that case."
Anyway I even if that's the reason, than I think something's quite
wrong here... a) it doesn't happen always (I just create 10 times a fs
in the same partition, and no problem) ... b) in those cases where I
could produce the issue, it went away after a balance... so perhaps
they just just balance the fs right after mkfs.... o.O
Cheers,
Chris.
[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: strange "No space left on device"
2015-11-07 23:22 strange "No space left on device" Christoph Anton Mitterer
2015-11-07 23:30 ` Hugo Mills
@ 2015-11-08 18:18 ` Henk Slager
2015-11-08 20:39 ` Duncan
2015-11-09 11:12 ` Filipe Manana
2 siblings, 1 reply; 8+ messages in thread
From: Henk Slager @ 2015-11-08 18:18 UTC (permalink / raw)
To: Christoph Anton Mitterer; +Cc: linux-btrfs
[...]
> 3 3145728 4194303 512.0 MiB 8300 Linux filesystem
[...]
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-data
> UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772
> Node size: 16384
> Sector size: 4096
> Filesystem size: 1.00GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 59.19MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 1.00GiB /dev/sdb3
Somehow your mkfs.btrfs (or some other part of the system) assumed the
wrong size for partition 3. To me, that sounds like a possible cause
of the errors and 'EiB' reported size.
For RAID tests in VMs I have also been using 4-8GB disk sizes (instead
of TB sized like the real hardware) and I encountered many 'No space
left on device' message (with older kernels).
A quick test with kernel 4.3.0 now:
# dd if=/dev/zero of=sd.bin bs=1M count=512
# losetup -f sd.bin
# mkfs.btrfs --nodiscard --label boot-data /dev/loop0
SMALL VOLUME: forcing mixed metadata/data groups
btrfs-progs v4.2.3+20151102
See http://btrfs.wiki.kernel.org for more information.
Label: boot-data
UUID: f152781b-b96b-4969-96b8-eede3e3dabc2
Node size: 4096
Sector size: 4096
Filesystem size: 512.00MiB
Block group profiles:
Data+Metadata: single 8.00MiB
System: single 4.00MiB
SSD detected: no
Incompat features: mixed-bg, extref, skinny-metadata
Number of devices: 1
Devices:
ID SIZE PATH
1 512.00MiB /dev/loop0
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: strange "No space left on device"
2015-11-08 18:18 ` Henk Slager
@ 2015-11-08 20:39 ` Duncan
2015-11-08 22:10 ` Christoph Anton Mitterer
0 siblings, 1 reply; 8+ messages in thread
From: Duncan @ 2015-11-08 20:39 UTC (permalink / raw)
To: linux-btrfs
Henk Slager posted on Sun, 08 Nov 2015 19:18:19 +0100 as excerpted:
> [...]
>> 3 3145728 4194303 512.0 MiB 8300 Linux filesystem
> [...]
>> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
>> btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more
>> information.
>>
>> Label: boot-data
>> UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772
>> Node size: 16384
>> Sector size: 4096
>> Filesystem size: 1.00GiB
>> Block group profiles:
>> Data: single 8.00MiB
>> Metadata: DUP 59.19MiB
>> System: DUP 12.00MiB
>> SSD detected: no
>> Incompat features: extref, skinny-metadata
>> Number of devices: 1 Devices:
>> ID SIZE PATH
>> 1 1.00GiB /dev/sdb3
>
> Somehow your mkfs.btrfs (or some other part of the system) assumed the
> wrong size for partition 3. To me, that sounds like a possible cause of
> the errors and 'EiB' reported size.
Wow, yes! Good catch, Henk! =:^) Hugo obviously didn't catch it, and I
wouldn't have either, as the bad size detection behavior is so
unexpected, it just wouldn't occur to me to look!
Something's clearly wrong, and as it could clearly either overwrite the
next partition(s) if the partitioning detection is at fault and doesn't
stop it, or throw all sorts of monkey wrenches into btrfs behavior if it
does, this is rather alarming behavior indeed! Clearly we need devs to
take a look and figure out what bugged out, and how to fix it.
That potentially explains why you got separate data/metadata instead of
mixed mode, and why it worked most of the time but failed occasionally,
since presumably when it worked the size detection worked as well.
(Apparently, btrfs-progs-4.3 does away with the default to mixed-mode at
1 GiB or under, tho it is still recommended. I'm not exactly sure of
why, tho I think it had to do with being able to use sub-GiB btrfs for
testing without having to worry about mixed mode. Still, one would
think, since it's recommended, that they'd simply provide an option to
make it split mode, instead of changing the default to something that's
not recommended. So from 4.3 userspace you'd have to specifically set
mixed, but you're on 4.2.2 userspace, so that shouldn't be the case yet.
And in any case that doesn't explain the very badly detected size and an
attempt to create a 1 GiB btrfs on a 1/2 GiB partition, which is
obviously the real problem, regardless of whatever layer it occurred in.)
Meanwhile, btrfs fi usage doesn't understand mixed-mode yet, and returns
8 EiB for some values if it is used, which is originally what I thought
was happening. But the problem is elsewhere, as it wasn't mixed mode,
but rather, a very /very/ screwed up size detection.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: strange "No space left on device"
2015-11-08 20:39 ` Duncan
@ 2015-11-08 22:10 ` Christoph Anton Mitterer
2015-11-09 1:51 ` Duncan
0 siblings, 1 reply; 8+ messages in thread
From: Christoph Anton Mitterer @ 2015-11-08 22:10 UTC (permalink / raw)
To: Duncan, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1435 bytes --]
On Sun, 2015-11-08 at 20:39 +0000, Duncan wrote:
> Wow, yes! Good catch, Henk! =:^) Hugo obviously didn't catch it,
> and I
> wouldn't have either, as the bad size detection behavior is so
> unexpected, it just wouldn't occur to me to look!
Hmm... all that *may* be more likely an error of myself when copying
and pasting the terminal output together:
I did actually change the 3rd partition to use 1GiB in later tries at
the expense of the 5th one shrinking, so the part table would have
looked like this in these later tries:
512M
1G
1G
1G
4G
Which would again fit the output of the various mkfs.btrfs.
Sorry if that was the case, apologies for any confusion.
The problem seemed to went away when explicitly using --mixed.
> (Apparently, btrfs-progs-4.3 does away with the default to mixed-
> mode at 1 GiB or under, tho it is still recommended.
Well I still had 4.2 ...
> I'm not exactly sure of
> why, tho I think it had to do with being able to use sub-GiB btrfs
> for
> testing without having to worry about mixed mode.
Kinda strange... shouldn't it work out of the box for users and not
developers?
To be honest, no one should need to read through the wiki, just to be
able to create a small sized fs.
And even if no mixed D/M block group allocation was used... it
shouldn't just fail out-of-the-box with a few byte large files on a 1
GB fs.
Cheers,
Chris.
[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5313 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: strange "No space left on device"
2015-11-08 22:10 ` Christoph Anton Mitterer
@ 2015-11-09 1:51 ` Duncan
0 siblings, 0 replies; 8+ messages in thread
From: Duncan @ 2015-11-09 1:51 UTC (permalink / raw)
To: linux-btrfs
Christoph Anton Mitterer posted on Sun, 08 Nov 2015 23:10:57 +0100 as
excerpted:
> On Sun, 2015-11-08 at 20:39 +0000, Duncan wrote:
>> Wow, yes! Good catch, Henk! =:^) Hugo obviously didn't catch it,
>> and I wouldn't have either, as the bad size detection behavior is so
>> unexpected, it just wouldn't occur to me to look!
> Hmm... all that *may* be more likely an error of myself when copying and
> pasting the terminal output together:
>
> I did actually change the 3rd partition to use 1GiB in later tries at
> the expense of the 5th one shrinking, so the part table would have
> looked like this in these later tries:
> 512M 1G 1G 1G 4G
Oops! =:^(
> Which would again fit the output of the various mkfs.btrfs.
>
> Sorry if that was the case, apologies for any confusion.
>
>
> The problem seemed to went away when explicitly using --mixed.
Good. Whatever the cause of it not doing mixed by default (perhaps it's
an off-by-one error and filesystems of exactly 1 GiB weren't defaulting
to mixed as expected), the fact that mixed was the default on small
filesystems would mean separate data/metadata wouldn't have gotten the
same full testing coverage, so maybe there's a bug... In which case the
4.3 change discussed below would certainly tend to trigger it more often,
hopefully getting it fixed. =:^)
>> (Apparently, btrfs-progs-4.3 does away with the default to mixed- mode
>> at 1 GiB or under, tho it is still recommended.
> Well I still had 4.2 ...
That's why I had that as a parenthetical. It didn't apply to your
current case, but could to others, and I've learned to write with both
the lurkers and the googlers who will happen on my post later, in mind.
>> I'm not exactly sure of why, tho I think it had to do with being able
>> to use sub-GiB btrfs for testing without having to worry about mixed
>> mode.
> Kinda strange... shouldn't it work out of the box for users and not
> developers?
> To be honest, no one should need to read through the wiki, just to be
> able to create a small sized fs.
That's why, tho I didn't press it, I asked for clarification when I saw
it mentioned on the btrfs-progs 4.3-rc1 announcement, verifying that
mixed was still recommended. Chris Murphy (another non-btrfs-dev list
regular, I think he works or volunteers for one of the distros)
questioned it as well, on the same announcement.
But not being able to personally show-them-the-code, as long as there's
at least an option to do it the way I want (which there is, the already
mentioned --mixed option), I've learned not to press it.[1]
> And even if no mixed D/M block group allocation was used... it shouldn't
> just fail out-of-the-box with a few byte large files on a 1 GB fs.
You are absolutely correct. There's a bug somewhere. Earlier I thought
it was the size detection bug Henk spotted, since even if there's some
other bug here it'd pale in comparision, but since that bug turned out to
trace to your copy/paste, that isn't it, and we still have the much more
ordinary level bug to deal with.
But as I said above, with progs 4.3 defaulting to unmixed even on small
filesystems now, if indeed it is tied to unmixed on a small filesystem,
that bug's very likely to get MUCH more common, and thus should be fixed
relatively quickly, I'd guess by the time 4.4 comes out, given the
history of other bugs that turn out to be quite common.
---
[1] Tho for my own systems I sometimes patch in the behavior as I think
it should be, as my patching of existing code skills are dramatically
better than my new-coding skills, and I run gentoo so already build
everything from sources. I finally got tired of the kernel not
defaulting to noatime, and hacked up a patch to do it, so I could remove
noatime from all my fstab entries, for instance. But I already use a
mkfs.btrfs helper script that adds options I prefer, so in this
particular case, simply adding --mixed under appropriate conditions in
the script is a far easier personal solution.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: strange "No space left on device"
2015-11-07 23:22 strange "No space left on device" Christoph Anton Mitterer
2015-11-07 23:30 ` Hugo Mills
2015-11-08 18:18 ` Henk Slager
@ 2015-11-09 11:12 ` Filipe Manana
2 siblings, 0 replies; 8+ messages in thread
From: Filipe Manana @ 2015-11-09 11:12 UTC (permalink / raw)
To: Christoph Anton Mitterer; +Cc: linux-btrfs@vger.kernel.org
On Sat, Nov 7, 2015 at 11:22 PM, Christoph Anton Mitterer
<calestyo@scientia.net> wrote:
> Hey.
>
> I just repeatedly did the following twice on a ~8GB USB stick, under
> Debian sid (ergo kernel 4.2.0-1-amd64, btrfsprogs 4.2.2-1).
>
> First, created some GPT on the stick:
> Number Start (sector) End (sector) Size Code Name
> 1 2048 1048575 511.0 MiB EF02 BIOS boot partition
> 2 1048576 3145727 1024.0 MiB 8300 Linux filesystem
> 3 3145728 4194303 512.0 MiB 8300 Linux filesystem
> 4 4194304 6291455 1024.0 MiB 8300 Linux filesystem
> 5 6291456 15687644 4.5 GiB 8300 Linux filesystem
>
>
> Then filesystems:
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-main /dev/sdb2
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-main
> UUID: 150ee9fb-c650-4b5b-8f64-606e589e429a
> Node size: 16384
> Sector size: 4096
> Filesystem size: 1.00GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 59.19MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 1.00GiB /dev/sdb2
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-data /dev/sdb3
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-data
> UUID: b1c1fc77-c965-4f0c-a2b5-44a301fd8772
> Node size: 16384
> Sector size: 4096
> Filesystem size: 1.00GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 59.19MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 1.00GiB /dev/sdb3
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-archive
> /dev/sdb4
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-archive
> UUID: a063cf3b-0322-4ec7-a8c1-2dabecad9f57
> Node size: 16384
> Sector size: 4096
> Filesystem size: 1.00GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 59.19MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 1.00GiB /dev/sdb4
>
> root@heisenberg:~# mkfs.btrfs --nodiscard --label boot-rescue /dev/sdb5
> btrfs-progs v4.2.2
> See http://btrfs.wiki.kernel.org for more information.
>
> Label: boot-rescue
> UUID: 104b7bc3-3b8c-4a08-b0a6-9172ed664e68
> Node size: 16384
> Sector size: 4096
> Filesystem size: 3.98GiB
> Block group profiles:
> Data: single 8.00MiB
> Metadata: DUP 211.75MiB
> System: DUP 12.00MiB
> SSD detected: no
> Incompat features: extref, skinny-metadata
> Number of devices: 1
> Devices:
> ID SIZE PATH
> 1 3.98GiB /dev/sdb5
>
>
>
> No errors in the kernel log.
>
>
>
> Then I got usage info:
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage data/
> Overall:
> Device size: 1.00GiB
> Device allocated: 126.38MiB
> Device unallocated: 897.62MiB
> Device missing: 0.00B
> Used: 256.00KiB
> Free (estimated): 905.62MiB (min: 456.81MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Data,single: Size:8.00MiB, Used:0.00B
> /dev/sdb3 8.00MiB
>
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
> /dev/sdb3 102.38MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
> /dev/sdb3 16.00MiB
>
> Unallocated:
> /dev/sdb3 897.62MiB
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage main/
> Overall:
> Device size: 1.00GiB
> Device allocated: 126.38MiB
> Device unallocated: 897.62MiB
> Device missing: 0.00B
> Used: 256.00KiB
> Free (estimated): 905.62MiB (min: 456.81MiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Data,single: Size:8.00MiB, Used:0.00B
> /dev/sdb2 8.00MiB
>
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
> /dev/sdb2 102.38MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
> /dev/sdb2 16.00MiB
>
> Unallocated:
> /dev/sdb2 897.62MiB
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage rescue/
> Overall:
> Device size: 3.98GiB
> Device allocated: 431.50MiB
> Device unallocated: 3.56GiB
> Device missing: 0.00B
> Used: 320.00KiB
> Free (estimated): 3.57GiB (min: 1.79GiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Data,single: Size:8.00MiB, Used:64.00KiB
> /dev/sdb5 8.00MiB
>
> Metadata,DUP: Size:203.75MiB, Used:112.00KiB
> /dev/sdb5 407.50MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
> /dev/sdb5 16.00MiB
>
> Unallocated:
> /dev/sdb5 3.56GiB
>
>
>
> Still all fine.
>
> Now I write to one of the filesystems (main at first, that still works)
> but then when I cp -a from another btrfs to data:
> cp: error writing ‘data/SHA512-sums.OLD’: No space left on device
> cp: failed to extend ‘data/SHA512-sums.OLD’: No space left on device
> cp: error writing ‘data/SHA512-sums’: No space left on device
> cp: failed to extend ‘data/SHA512-sums’: No space left on device
>
> (these files are just a few bytes large)
> On the target fs, they're there but all 0 bytes large.
>
> (btw: I've aliased cp to cp --reflink=auto
>
>
> When I now repeat the usage info:
> root@heisenberg:/data/SSS/boot# btrfs filesystem usage data
> Overall:
> Device size: 1.00GiB
> Device allocated: 118.38MiB
> Device unallocated: 905.62MiB
> Device missing: 0.00B
> Used: 256.00KiB
> Free (estimated): 8.00EiB (min: 8.00EiB)
> Data ratio: -nan
> Metadata ratio: 2.00
> Global reserve: 16.00MiB (used: 0.00B)
>
> Metadata,DUP: Size:51.19MiB, Used:112.00KiB
> /dev/sdc3 102.38MiB
>
> System,DUP: Size:8.00MiB, Used:16.00KiB
> /dev/sdc3 16.00MiB
>
> Unallocated:
> /dev/sdc3 905.62MiB
>
> It get's bogus... at least the Free space shows 8 EiB (if wishes were
> horses)...
You're likely hitting a bug introduced in kernel 4.2, and it got fixed
in 4.3 by: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=6af3e3adcac595a683bb55299a907d7d1ad61ab3
Because there are no data block groups you get the -nan data ratio
there and the 8EiB, which is a problem in the tools that don't (or
didn't, maybe they were fixed in the meanwhile) expect to find a
filesystem without data block groups. Now what matters the most is the
kernel bug, which is what leads to the ENOSPC errors when attempting
to write data (even a 1 byte file).
Just try a 4.3 kernel, or a 4.2 kernel + that patch and confirm it
fixes the issue.
thanks
>
> Still no error in kernel logs:
> 3401.934514] BTRFS: device label boot-rescue devid 1 transid 8
> /dev/sdc5
> [ 3401.985858] BTRFS: device label boot-main devid 1 transid 8
> /dev/sdc2
> [ 3401.988601] BTRFS: device label boot-data devid 1 transid 9
> /dev/sdc3
> [ 3401.997370] BTRFS: device label boot-archive devid 1 transid 9
> /dev/sdc4
> [ 3403.721091] BTRFS info (device sdc3): disk space caching is enabled
> [ 3403.721098] BTRFS: has skinny extents
> [ 3413.964033] BTRFS info (device sdb3): disk space caching is enabled
> [ 3413.964042] BTRFS: has skinny extents
> [ 3522.902309] BTRFS info (device sdb3): disk space caching is enabled
> [ 3522.902314] BTRFS: has skinny extents
> [ 3525.055653] BTRFS info (device sdc3): disk space caching is enabled
> [ 3525.055659] BTRFS: has skinny extents
>
>
> btrfs check doesn't show errors either.
>
>
> After a balance, i can copy again, even though the usage still shows 8
> EiB for a while...
>
>
> Anyway how can it happen, that on a fresh btrfs with no files at all a
> balance is necessary?
> Or is there some deeper corruption going on here?
>
>
> Thanks,
> Chris.
--
Filipe David Manana,
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2015-11-09 11:12 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-07 23:22 strange "No space left on device" Christoph Anton Mitterer
2015-11-07 23:30 ` Hugo Mills
2015-11-08 0:38 ` Christoph Anton Mitterer
2015-11-08 18:18 ` Henk Slager
2015-11-08 20:39 ` Duncan
2015-11-08 22:10 ` Christoph Anton Mitterer
2015-11-09 1:51 ` Duncan
2015-11-09 11:12 ` Filipe Manana
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox