Help! - "btrfs device delete missing" running out of space

Help! - "btrfs device delete missing" running out of space

[Piotr Pawłow]

Hello,

I replaced a failed 500G disk in btrfs raid1 with 2 smaller ones - 200 
and 250GB. I started "btrfs device delete missing" command, which is 
running since friday, and it still seems far from finished.

It seems to be doing something very strange: used space on the largest 
drive is going down, while smaller drives are filling up to the brim.

I'm afraid it will soon run out of space and I don't know what to do.

The current situation is:

Label: none  uuid: cff1e711-97fe-4c1c-b8a3-6184010b5027
	Total devices 4 FS bytes used 376.66GB
	devid    4 size 172.81GB used 167.00GB path /dev/dm-16
	devid    3 size 232.88GB used 227.00GB path /dev/dm-4
	devid    2 size 452.26GB used 327.01GB path /dev/dm-0
	*** Some devices missing

According to btrfs FAQ and http://carfax.org.uk/btrfs-usage, there should 
be around 404GB of usable space in this configuration. Assuming even 
distribution, used space on each device should be accordingly: 160, 216, 
and 405. The current situation is FAR from that.

I'd be grateful for tips what do to and how to get out of this situation.

Regards

Re: Help! - "btrfs device delete missing" running out of space

[Piotr Pawłow]

Hello,
> distribution, used space on each device should be accordingly: 160, 
> 216, and 405.

The last number should be 376, I copied the wrong one. Anyway, I deleted 
as much data as possible, which probably won't help in the end, but at 
the moment it's still going. Meanwhile, I made a script to replicate this 
problem:

http://pastebin.com/W2c2pJYp

On kernel 3.12.6 the output is:

-------

WARNING! - Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using

adding device /dev/loop1 id 2
fs created label (null) on /dev/loop0
	nodesize 4096 leafsize 4096 sectorsize 4096 size 40.00GB
Btrfs v0.20-rc1
17000+0 records in
17000+0 records out
17825792000 bytes (18 GB) copied, 533,571 s, 33,4 MB/s
Label: none  uuid: f8a01060-94c2-4665-b5ff-f134f9b6ad9b
	Total devices 2 FS bytes used 16.63GB
	devid    2 size 20.00GB used 18.01GB path /dev/loop1
	devid    1 size 20.00GB used 18.03GB path /dev/loop0

Btrfs v0.20-rc1
ERROR: error removing the device 'missing' - No space left on device
Label: none  uuid: f8a01060-94c2-4665-b5ff-f134f9b6ad9b
	Total devices 4 FS bytes used 16.62GB
	devid    4 size 10.00GB used 9.03GB path /dev/loop3
	devid    3 size 10.00GB used 9.25GB path /dev/loop2
	devid    1 size 20.00GB used 12.31GB path /dev/loop0
	*** Some devices missing

Btrfs v0.20-rc1

-------

The "delete missing" logic is pretty much broken, at least in this case. 
Instead of just replicating the data to other drives, it moves some of 
the data which fills up smaller drives and it fails with "No space left 
on device" error.

Regards

Re: Help! - "btrfs device delete missing" running out of space

[Calvin Walton]

On Sun, 2014-01-05 at 19:00 +0000, Piotr Pawłow wrote:
> Hello,
> > distribution, used space on each device should be accordingly: 160, 
> > 216, and 405.
> 
> The last number should be 376, I copied the wrong one. Anyway, I deleted 
> as much data as possible, which probably won't help in the end, but at 
> the moment it's still going. Meanwhile, I made a script to replicate this 
> problem:
> 
> http://pastebin.com/W2c2pJYp

I'm not sure what the solution is, but the issue seems to be that btrfs
is laying out the RAID1 like this:

Mirror A   Mirror B
+------+   +------+
| 250g |   | 300g |
+------+   |      |
|      |   +------+
| 500g |   ////////
|      |   ////////
+------+   ////////

Rather than the hoped-for:

Mirror A   Mirror B
+------+   +------+
| 250g |   |      |
+------+   | 500g |
| 300g |   |      |
|      |   +------+
+------+   ////////

As far as I know, there's no way to tune this at the moment.

-- 
Calvin Walton <calvin.walton@kepstin.ca>

Re: Help! - "btrfs device delete missing" running out of space

[Piotr Pawłow]

Hello,
> I'm not sure what the solution is, but the issue seems to be that
> btrfs is laying out the RAID1 like this:
[snip]

Yeah, it kinda ended up like this. I think the problem stems from the
fact, that restoring redundancy works by relocating block groups, which
rewrites all chunks instead of restoring only the ones that are missing.
And it uses the usual allocation strategy, which fails in this case. It
treats unreplicated data as allocated, when in fact all of that data
will be, sooner or later, freed in the process.

If I succeed deleting the missing device, then running "btrfs balance"
should fix the problem.

Regards