Device only missing if unmounted

Device only missing if unmounted

[Florian Uekermann]

I am using a few disks in Raid1 mode. This is the output of various
commands in unmounted state.

root@oot:/# uname -a
Linux oot 3.18.0-rc5+ #1 SMP Sun Nov 23 18:01:56 CET 2014 x86_64 GNU/Linux
root@oot:/# btrfs --version
Btrfs v3.17
root@oot:/# btrfs fi show
Label: none uuid: be2b3499-7452-4b91-b664-4ec4d7ff62b9
Total devices 3 FS bytes used 149.68GiB
devid 2 size 465.76GiB used 151.03GiB path /dev/sdb
devid 3 size 465.76GiB used 151.03GiB path /dev/sda
devid 4 size 92.26GiB used 0.00B path /dev/sdc4

Btrfs v3.17
root@oot:/# btrfs fi df /srv/storage/
Data, RAID1: total=150.00GiB, used=149.40GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=1.00GiB, used=287.42MiB
GlobalReserve, single: total=96.00MiB, used=0.00B

I stripped a few lines from the output of btrfs fi show, which
describe other filesystems.
If I unmount the devices in question, I get the following
output (note that missing devices are reported now):

root@oot:/# btrfs fi show
Label: none uuid: be2b3499-7452-4b91-b664-4ec4d7ff62b9
Total devices 4 FS bytes used 149.68GiB
devid 2 size 465.76GiB used 151.03GiB path /dev/sdb
devid 3 size 465.76GiB used 151.03GiB path /dev/sda
devid 4 size 92.26GiB used 0.00B path /dev/sdc4
*** Some devices missing

Btrfs v3.17

I got into this state by removing a device from the array formatting it
and then adding it again (some time ago, I can't remember the exact
sequence of commands)

The discrepancy in output between mounted and unmounted state is not
a problem in itself for me, but this issue seems to interrupt systemds
boot sequence, which only continues successfully if I mount the array manually.
The array is not degraded, I just removed a drive (via btrfs delete I think).

If I try to remove the "missing devices" with the btrfs tool the following
happens:

btrfs dev del missing /srv/storage
ERROR: error removing the device 'missing' - no missing devices found to remove

btrfs balance start /srv/storage/ does not solve the issue.

Re: Device only missing if unmounted

[Anand Jain]

On 09/12/2014 19:15, Florian Uekermann wrote:
> I am using a few disks in Raid1 mode. This is the output of various
> commands in unmounted state.
>
> root@oot:/# uname -a
> Linux oot 3.18.0-rc5+ #1 SMP Sun Nov 23 18:01:56 CET 2014 x86_64 GNU/Linux
> root@oot:/# btrfs --version
> Btrfs v3.17
> root@oot:/# btrfs fi show
> Label: none uuid: be2b3499-7452-4b91-b664-4ec4d7ff62b9
> Total devices 3 FS bytes used 149.68GiB
> devid 2 size 465.76GiB used 151.03GiB path /dev/sdb
> devid 3 size 465.76GiB used 151.03GiB path /dev/sda
> devid 4 size 92.26GiB used 0.00B path /dev/sdc4
>
> Btrfs v3.17
> root@oot:/# btrfs fi df /srv/storage/
> Data, RAID1: total=150.00GiB, used=149.40GiB
> System, RAID1: total=32.00MiB, used=48.00KiB
> Metadata, RAID1: total=1.00GiB, used=287.42MiB
> GlobalReserve, single: total=96.00MiB, used=0.00B
>
> I stripped a few lines from the output of btrfs fi show, which
> describe other filesystems.
> If I unmount the devices in question, I get the following
> output (note that missing devices are reported now):
>
> root@oot:/# btrfs fi show
> Label: none uuid: be2b3499-7452-4b91-b664-4ec4d7ff62b9
> Total devices 4 FS bytes used 149.68GiB
> devid 2 size 465.76GiB used 151.03GiB path /dev/sdb
> devid 3 size 465.76GiB used 151.03GiB path /dev/sda
> devid 4 size 92.26GiB used 0.00B path /dev/sdc4
> *** Some devices missing
>
> Btrfs v3.17
>
> I got into this state by removing a device from the array formatting it
> and then adding it again (some time ago, I can't remember the exact
> sequence of commands)


  above you mention both btrfs fi show outputs in unmounted only.

  It depends on the disk that is read first, you could read super block
  using btrfs-show-super and check if num_device.

> The discrepancy in output between mounted and unmounted state is not
> a problem in itself for me, but this issue seems to interrupt systemds
> boot sequence, which only continues successfully if I mount the array manually.
> The array is not degraded, I just removed a drive (via btrfs delete I think).

It may fail if you are mounting more than one subvol during boot and if 
you don't have this patch.

  commit 0f23ae74f589304bf33233f85737f4fd368549eb
  Author: Chris Mason <clm@fb.com>
  Date:   Thu Sep 18 07:49:05 2014 -0700

    Revert "Btrfs: device_list_add() should not update list when mounted"

      This reverts commit b96de000bc8bc9688b3a2abea4332bd57648a49f.


> If I try to remove the "missing devices" with the btrfs tool the following
> happens:
>
> btrfs dev del missing /srv/storage
> ERROR: error removing the device 'missing' - no missing devices found to remove

  that means you don't see missing when mounted. if you don't see
  missing when mounted then no need to run delete missing. ?


> btrfs balance start /srv/storage/ does not solve the issue.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Device only missing if unmounted

[Florian Uekermann]

Dear Anand,
thank you for your help.

> On December 10, 2014 at 3:46 AM Anand Jain <anand.jain@oracle.com> wrote:
> It depends on the disk that is read first, you could read super block
> using btrfs-show-super and check if num_device.

I checked this for all three devices and num_devices is 4 for all of them.
The full output is below.

> It may fail if you are mounting more than one subvol during boot and if
> you don't have this patch.
>
> commit 0f23ae74f589304bf33233f85737f4fd368549eb
> Author: Chris Mason <clm@fb.com>
> Date: Thu Sep 18 07:49:05 2014 -0700
>
> Revert "Btrfs: device_list_add() should not update list when mounted"
>
> This reverts commit b96de000bc8bc9688b3a2abea4332bd57648a49f.

I am running 3.18-rc5+ from linus tree. If I am not mistaken,
 that commit is his tree in since 3.17-rc6.

So I guess the question is: Why does the superblock say
num_devices=4 (and how do I fix it)?
Or am I misunderstanding the problem?

Best regards,
Florian

root@oot:/home/shared# btrfs-show-super /dev/sdb
superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
csum			0x55a3eb1d [match]
bytenr			65536
flags			0x1
magic			_BHRfS_M [match]
fsid			be2b3499-7452-4b91-b664-4ec4d7ff62b9
label			
generation		3003
root			441395970048
sys_array_size		129
chunk_root_generation	3001
root_level		1
chunk_root		442469761024
chunk_root_level	1
log_root		0
log_root_transid	0
log_root_level		0
total_bytes		109927469in the first place8752
bytes_used		160703954944
sectorsize		4096
nodesize		16384
leafsize		16384
stripesize		4096
root_dir		6
num_devices		4
compat_flags		0x0
compat_ro_flags		0x0
incompat_flags		0x61
			( MIXED_BACKREF |
			  BIG_METADATA |
			  EXTENDED_IREF )
csum_type		0
csum_size		4
cache_generation	3003
uuid_tree_generation	3003
dev_item.uuid		5b8c8d20-c330-48a2-814c-c6986b787d52
dev_item.fsid		be2b3499-7452-4b91-b664-4ec4d7ff62b9 [match]
dev_item.type		0
dev_item.total_bytes	500107862016
dev_item.bytes_used	162168569856
dev_item.io_align	4096
dev_item.io_width	4096
dev_item.sector_size	4096
dev_item.devid		2
dev_item.dev_group	0
dev_item.seek_speed	0
dev_item.bandwidth	0
dev_item.generation	0

Re: Device only missing if unmounted

[Anand Jain]

> I checked this for all three devices and num_devices is 4 for all of them.
> The full output is below.

  so now if you mount and run 'btrfs fi show <mnt>' that should tell
  device missing, which then 'btrfs device missing <mnt>' should be
  successful theoretically.

>> It may fail if you are mounting more than one subvol during boot and if
>> you don't have this patch.
>>
>> commit 0f23ae74f589304bf33233f85737f4fd368549eb
>> Author: Chris Mason <clm@fb.com>
>> Date: Thu Sep 18 07:49:05 2014 -0700
>>
>> Revert "Btrfs: device_list_add() should not update list when mounted"
>>
>> This reverts commit b96de000bc8bc9688b3a2abea4332bd57648a49f.
>
> I am running 3.18-rc5+ from linus tree. If I am not mistaken,
>   that commit is his tree in since 3.17-rc6.

  Ah yes. sorry i just missed that point.

Re: Device only missing if unmounted

[Anand Jain]

Very strange, user space reads num_device as 4 but kernel
reads it as 3 (total_device), thus a device missing when
mounted.
Not sure what kind of hardware at your end, Can cache be
issue ? could you drop cache and check for me.
  echo 3 > /proc/sys/vm/drop_cache

Thanks, Anand

On 21/12/2014 01:32, Florian Uekermann wrote:
>
>  > > I checked this for all three devices and num_devices is 4 for all
> of them.
>  > > The full output is below.
>  >
>  > so now if you mount and run 'btrfs fi show <mnt>' that should tell
>  > device missing, which then 'btrfs device missing <mnt>' should be
>  > successful theoretically.
>
> No it doesn't. It says:
> root@oot:/home/shared# btrfs device delete missing /srv/storage/
> ERROR: error removing the device 'missing' - no missing devices found to
> remove
> and btrfs fi show /srv/storage doesn't report a missing device either
> (after mount).
> This behavior is the reason why I wrote here in the first place. It
> seems like a bug to me.
> Could the btrfs command line utility be the problem here (I am using the
> version the upcoming debian 8 ships)?
> Best regards,
> Florian

Re: Device only missing if unmounted

[Florian Uekermann]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 1390 bytes --]

I wasn't sure when to drop the caches, so I did it before and after mounting. But no change the output of btrfs-show-super always looks the same and btrfs show fi only says "Total devices 4" if the fs is not mounted. After mounting it says 3 and after unmounting it says 4 again.
About the hardware: I am using a normal amd64 consumer Pc with a set of different normal sata disks. The machine is rebooted daily (in case that matters for caches).

To uneducated me, it looks a bit like the btrfs utility is screwing up here when figuring out how many devices should be there. I don't know how closely this is tied to the kernel and how much of the logic is actually part of the kernel here and how much is userspace code. Do you think it makes sense to use some newer upstream version of the btrfs utility?
The debian package containing the btrfs command has 3.17-1.1 as version string.

Best regards,
Florian

On Dec 24, 2014 4:45 PM, Anand Jain <anand.jain@oracle.com> wrote:
>
>
> Very strange, user space reads num_device as 4 but kernel 
> reads it as 3 (total_device), thus a device missing when 
> mounted. 
> Not sure what kind of hardware at your end, Can cache be 
> issue ? could you drop cache and check for me. 
>   echo 3 > /proc/sys/vm/drop_cache 
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm