Re: Wrong device?

[Duncan <1i5t5.duncan@cox.net>]

linux-btrfs posted on Mon, 25 Sep 2017 19:11:01 +0300 as excerpted:

Three points first off:

1) Please use a name other than "linux-btrfs".  That can remain the email 
address, but a name to go with it would be nice.  (You can see how my 
name and email address differ, for instance.)

2) Please don't reply to existing threads with a totally different 
topic.  Proper reply-threading clients will see it's a reply to the other 
thread (it's in the headers) and display it as such, even if you change 
the subject line.  If it's a different topic, start a new thread, don't 
reply to an existing thread.

3) I'm not a dev, just another btrfs user and list regular.  So I won't 
attempt to address the real technical or complicated stuff, but I can try 
to reply to the easy stuff at least, freeing the devs for addressing the 
more complicated stuff if they see a bug they can fix or might already be 
working on.

As to the problem...

> I have 15 x 6 TB disks in md-raid (Because btrfs's raid6 was marked as
> not-for-real-use when I first time installed this machine)
> 
> Now I have got both problem twice.
> 
> At this last time I have 233 subvolums, and millions of files (all
> together)

Just a reminder prompted by seeing those numbers tho I'd guess you 
already have this covered...

Sysadmin's backups rule #1:  The true value of your data is defined not 
by any words /claiming/ value nor by what you use it for, as the machine 
doesn't care about that, but rather by the number of backups of it you 
consider it worth having.  No backups means you are defining the data to 
be of less value than the time/trouble/resources to make the backup, so 
loss is never a big deal, because either you either have a backup if you 
considered it important to you, or you already saved what you defined as 
more valuable to you than that data, the time, trouble and resources 
you'd have otherwise put into the backup.

(Similarly with the currency of those backups, only there it's the value 
of the data difference between your last backup and the working copy.  
Once the data in that difference is of more value than the time/trouble/
resources to update the backup, it'll be updated.  Otherwise, the data in 
that delta is obviously not valuable enough to be worth that trouble, and 
thus not valuable enough to be terribly worried about if lost.)

> Then filesystem went to read only with this dmesg:
> 
> [Sat Sep 23 07:25:28 2017] ------------[ cut here ]------------
> [Sat Sep 23 07:25:28 2017] WARNING: CPU: 5 PID: 5431 at
> /build/linux-hwe-edge-CrMNv8/linux-hwe-edge-4.10.0/fs/btrfs/extent-
tree.c:6947
> __btrfs_free_extent.isra.61+0x2cb/0xeb0 [btrfs]
> [Sat Sep 23 07:25:28 2017] BTRFS: Transaction aborted (error -28)

> 4.10.0-26-generic #30~16.04.1-Ubuntu 

Note that this is the mainline btrfs development list, with btrfs still 
stabilizing, not yet fully stable and mature, so this list tends to be 
quite forward focused.  We recommend and best support the latest two 
mainline kernels in two tracks, current and LTS.  The current kernel is 
4.13, so on the current track, 4.13 and 4.12 are recommended and best 
supported.  On the LTS track, the coming 4.14 is scheduled to be an LTS, 
with 4.9 and 4.4 the two previous LTSs before that.

So the 4.9 LTS kernel series is supported, with 4.4 currently supported 
but on its way out and 4.14 on the way in.  And current track 4.13 and 
4.12 are supported, with 4.12 on the way out.

4.10 isn't an LTS kernel, and it's old enough it's already several 
kernels out of current support track.  So upgrading to current 4.13 or 
downgrading to 4.9 LTS series would be recommended.

Meanwhile, your distro is in a better position to support their kernels 
of /whatever/ version since they know what patches they've applied and 
what btrfs fixes they've backported... or not.

Of course we'll still try to help with 4.10, and it's not /too/ dated, 
but you can expect that you might get "does it still happen with a 
current kernel" type questions.

> [Sat Sep 23 07:25:28 2017] BTRFS: error (device sdb) in
> __btrfs_free_extent:6947: errno=-28 No space left [Sat Sep 23 07:25:28
> 2017] BTRFS: error (device sdb) in btrfs_drop_snapshot:9193: errno=-28
> No space left [Sat Sep 23 07:25:28 2017] BTRFS info (device sdb): forced
> readonly
> 
> 
> After a long googling (about more complex situations) I suddenly noticed
> "device sdb" WTF???  Filesystem is mounted from /dev/md3 (sdb is part of
> that mdraid) so btrfs should not even know anything about that /dev/sdb.
> 
> So maybe the error is more simple that I thought.  It tries to allocate
> more metadata from physical drive (not from /dev/md3 as it was supposed
> to do), so that "no space left on device" could be the REAL error...
> 
> But why it is doing so?  Does it help if I add some extra virtualization
> layer, like LVM?

Keep in mind that btrfs, unlike most other filesystems, can be multi-
device.  As such, it needs a way to track which devices are part of each 
filesystem, and it uses the filesystem UUID for that purpose.

Meanwhile, btrfs device scan, which is auto-run by udev after a device 
appears, is what lets the kernel know about all those btrfs containing 
devices and the UUIDs associated with them.

That's why btrfs is listing one of the md components as part of the 
filesystem -- it obviously has the same btrfs UUID as the md device that 
you actually created the filesystem on.

That can cause btrfs to write to the wrong device in some instances, tho 
obviously it doesn't do it all the time or things wouldn't work as well 
as they do.

It's thus recommended that when you're using btrfs on device layering 
such as mdraid or LVM, that you ensure that the lower layer devices 
aren't exposed to btrfs so it doesn't get confused.  I believe LVM can be 
configured to hide the lower layer devices in at least some instances, 
but I'm not sure about mdraid, altho this is the first time I've seen 
that particular issue with it (tho it may simply be because I've not been 
watching for it, Chris Murphy or Hugo are likely to have more information 
about that as they're more active with user support than I am, and are 
more technically skilled too, I believe).

> # btrfs --version btrfs-progs v4.4

As with the kernel version, likewise with the btrfs-progs userspace, tho 
in operation it's less important than the kernel, because for most normal 
operating commands, userspace simply calls the real code in the kernel 
anyway.  But once things begin to go wrong, the userspace version becomes 
more important, because it's the userspace code that handles btrfs check, 
btrfs restore, etc.

So while userspace 4.4 is fine for normal operations, you might want to 
be sure you have a current 4.12 or so available for recovery if needed, 
since it'll have the latest fixes and thus should give you the best 
chance at recovery, if it /is/ needed.

> # btrfs fi usage /data2
> Overall:
>      Device size:                  70.95TiB
>      Device allocated:              4.95TiB
>      Device unallocated:           66.01TiB
>      Device missing:                  0.00B
>      Used:                          4.94TiB
>      Free     (estimated):         66.01TiB      (min: 33.01TiB)
>      Data ratio:                       1.00
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 0.00B)
> 
> Data,single: Size:4.77TiB, Used:4.76TiB
>     /dev/md3        4.77TiB
> 
> Metadata,DUP: Size:92.00GiB, Used:90.79GiB
>     /dev/md3      184.00GiB
> 
> System,DUP: Size:32.00MiB, Used:592.00KiB
>     /dev/md3       64.00MiB
> 
> Unallocated:
>     /dev/md3       66.01TiB

This usage looks healthy.  No problems here. =:^)

That's the easy to see and address stuff, anyway.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman