Curious problem: btrfs device stats & unpriviliged access

Curious problem: btrfs device stats & unpriviliged access

[Holger Hoffstätte]

(moving the discussion here from GH [1])

Apparently there is something weird going on with the device stats
ioctls. I cannot get them to work as regular user, while they work
for David. A friend confirms the same issue on his system - no access
as non-root.

So I made a new empty fs, mounted it, built btrfs-progs-4.17.1 with
debug symbols and stepped into search_chunk_tree_for_fs_info().
Everything is fine, all args are correct, right until:

(gdb) s
1614	ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, &search_args);
(gdb) s
1615	if (ret < 0)
(gdb) p ret
$4 = -1
(gdb) p search_args
$5 = {key = {tree_id = 3, min_objectid = 1, max_objectid = 1, min_offset = 1,
max_offset = 18446744073709551615, min_transid = 0, max_transid = 18446744073709551615,
min_type = 216, max_type = 216, nr_items = 30, unused = 0, unused1 = 0, unused2 = 0,
unused3 = 0, unused4 = 0}, buf = '\000' <repeats 3991 times>}

Looking at the kernel side of things in fs/btrfs/ioctl.c I see both
BTRFS_IOC_TREE_SEARCH[_V2} unconditionally require CAP_SYS_ADMIN.

So why can Dave get his dev stats as unprivileged user?
Does this work for anybody else? And why? :)

cheers
Holger

[1] https://github.com/prometheus/node_exporter/issues/1100#issuecomment-427823190

Re: Curious problem: btrfs device stats & unpriviliged access

[Hans van Kranenburg]

On 10/08/2018 04:27 PM, Holger Hoffstätte wrote:
> (moving the discussion here from GH [1])
> 
> Apparently there is something weird going on with the device stats
> ioctls. I cannot get them to work as regular user, while they work
> for David. A friend confirms the same issue on his system - no access
> as non-root.
> 
> So I made a new empty fs, mounted it, built btrfs-progs-4.17.1 with
> debug symbols and stepped into search_chunk_tree_for_fs_info().
> Everything is fine, all args are correct, right until:
> 
> (gdb) s
> 1614    ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, &search_args);
> (gdb) s
> 1615    if (ret < 0)
> (gdb) p ret
> $4 = -1
> (gdb) p search_args
> $5 = {key = {tree_id = 3, min_objectid = 1, max_objectid = 1, min_offset
> = 1,
> max_offset = 18446744073709551615, min_transid = 0, max_transid =
> 18446744073709551615,
> min_type = 216, max_type = 216, nr_items = 30, unused = 0, unused1 = 0,
> unused2 = 0,
> unused3 = 0, unused4 = 0}, buf = '\000' <repeats 3991 times>}
> 
> Looking at the kernel side of things in fs/btrfs/ioctl.c I see both
> BTRFS_IOC_TREE_SEARCH[_V2} unconditionally require CAP_SYS_ADMIN.

That's the tree search ioctl, for reading arbitrary metadata.

The device stats ioctl is IOC_GET_DEV_STATS...

I can do the device stats ioctl as normal user:

import btrfs
fs = btrfs.FileSystem('/')
btrfs.utils.pretty_print(fs.dev_stats(1))

<btrfs.ioctl.DevStats>
devid: 1
nr_items: 5
flags: 0
write_errs: 0
read_errs: 0
flush_errs: 0
generation_errs: 0
corruption_errs: 0

> So why can Dave get his dev stats as unprivileged user?
> Does this work for anybody else? And why? :)
> 
> cheers
> Holger
> 
> [1]
> https://github.com/prometheus/node_exporter/issues/1100#issuecomment-427823190
> 


-- 
Hans van Kranenburg

Re: Curious problem: btrfs device stats & unpriviliged access

[Hans van Kranenburg]

On 10/08/2018 04:40 PM, Hans van Kranenburg wrote:
> On 10/08/2018 04:27 PM, Holger Hoffstätte wrote:
>> (moving the discussion here from GH [1])
>>
>> Apparently there is something weird going on with the device stats
>> ioctls. I cannot get them to work as regular user, while they work
>> for David. A friend confirms the same issue on his system - no access
>> as non-root.
>>
>> So I made a new empty fs, mounted it, built btrfs-progs-4.17.1 with
>> debug symbols and stepped into search_chunk_tree_for_fs_info().
>> Everything is fine, all args are correct, right until:
>>
>> (gdb) s
>> 1614    ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, &search_args);
>> (gdb) s
>> 1615    if (ret < 0)
>> (gdb) p ret
>> $4 = -1
>> (gdb) p search_args
>> $5 = {key = {tree_id = 3, min_objectid = 1, max_objectid = 1, min_offset
>> = 1,
>> max_offset = 18446744073709551615, min_transid = 0, max_transid =
>> 18446744073709551615,
>> min_type = 216, max_type = 216, nr_items = 30, unused = 0, unused1 = 0,
>> unused2 = 0,
>> unused3 = 0, unused4 = 0}, buf = '\000' <repeats 3991 times>}
>>
>> Looking at the kernel side of things in fs/btrfs/ioctl.c I see both
>> BTRFS_IOC_TREE_SEARCH[_V2} unconditionally require CAP_SYS_ADMIN.
> 
> That's the tree search ioctl, for reading arbitrary metadata.
> 
> The device stats ioctl is IOC_GET_DEV_STATS...
> 
> I can do the device stats ioctl as normal user:
> 
> import btrfs
> fs = btrfs.FileSystem('/')
> btrfs.utils.pretty_print(fs.dev_stats(1))
> 
> <btrfs.ioctl.DevStats>
> devid: 1
> nr_items: 5
> flags: 0
> write_errs: 0
> read_errs: 0
> flush_errs: 0
> generation_errs: 0
> corruption_errs: 0

By the way, I can also do BTRFS_IOC_FS_INFO, BTRFS_IOC_DEV_INFO and
BTRFS_IOC_SPACE_INFO as normal user.

However, while fs_info tells me that there are num_devices devices,
there's no place where you can actually get which devids these are, and
you need to provide them one by one to dev_info and dev_stats... :

btrfs.utils.pretty_print(fs.fs_info())

<btrfs.ioctl.FsInfo>
max_id: 1
num_devices: 1
nodesize: 4096
sectorsize: 4096
clone_alignment: 4096
fsid: 91077ca5-6559-4a90-9d03-912d3a33412e

btrfs.utils.pretty_print(fs.dev_info(1))
<btrfs.ioctl.DevInfo>
devid: 1
bytes_used: 60699967488
total_bytes: 107374182400
path: /dev/xvda
uuid: 7e998baa-b533-4476-9132-d7d748d28044


btrfs.utils.pretty_print(fs.space_info())
-
  <btrfs.ioctl.SpaceInfo>
  flags: Data, single
  total_bytes: 54.00GiB
  used_bytes: 53.27GiB
-
  <btrfs.ioctl.SpaceInfo>
  flags: System, single
  total_bytes: 32.00MiB
  used_bytes: 12.00KiB
-
  <btrfs.ioctl.SpaceInfo>
  flags: Metadata, single
  total_bytes: 2.50GiB
  used_bytes: 1.30GiB
-
  <btrfs.ioctl.SpaceInfo>
  flags: GlobalReserve, single
  total_bytes: 181.02MiB
  used_bytes: 0.00B


> 
>> So why can Dave get his dev stats as unprivileged user?
>> Does this work for anybody else? And why? :)
>>
>> cheers
>> Holger
>>
>> [1]
>> https://github.com/prometheus/node_exporter/issues/1100#issuecomment-427823190
>>
> 
> 


-- 
Hans van Kranenburg

Re: Curious problem: btrfs device stats & unpriviliged access

[Holger Hoffstätte]

On 10/08/18 16:40, Hans van Kranenburg wrote:
>> Looking at the kernel side of things in fs/btrfs/ioctl.c I see both
>> BTRFS_IOC_TREE_SEARCH[_V2} unconditionally require CAP_SYS_ADMIN.
> 
> That's the tree search ioctl, for reading arbitrary metadata.
> 
> The device stats ioctl is IOC_GET_DEV_STATS...

Yeah..OK, that is clear and gave me the hint to ask: why is it
calling this in the first place? And as it turns out [1] is where
it seems to go wrong, as is_block_device() returns 0 (as it should),
fi_args.num_devices is never set (remains 0) and it proceeds to call
everything below, eventually calling the BTRFS_IOC_FS_INFO ioctl in
#1712. And that works fine:

1711 if (fi_args->num_devices != 1) {
(gdb) s
1712	ret = ioctl(fd, BTRFS_IOC_FS_INFO, fi_args);
(gdb) s
1713	if (ret < 0) {
(gdb) p ret
$28 = 0
(gdb) p *fi_args
$30 = {
   max_id = 1,
   num_devices = 1,
   fsid = "z%:\371\315\033A\203\267.\200\255;FH\221",
   nodesize = 16384,
   sectorsize = 4096,
   clone_alignment = 4096,
   reserved32 = 0,
   reserved = {0 <repeats 122 times>}
}

It's only when it goes into search_chunk_tree_for_fs_info()
where things fail due to CAP_SYS_ADMIN.

And all this explains the actual bug: when I call btrfs device stats
not on the mountpoint (as I've been trying all this time) but rather
on the device, it works properly right away as regular user:

(gdb) set args device stats /dev/loop0
(gdb) r
Breakpoint 1, get_fs_info (path=path@entry=0x7fffffffe527 "/dev/loop0", fi_args=fi_args@entry=0x7fffffffd400,
     di_ret=di_ret@entry=0x7fffffffd3f0) at utils.c:1652
1652	{
(gdb) c
Continuing.
[/dev/loop0].write_io_errs    0
[/dev/loop0].read_io_errs     0
[/dev/loop0].flush_io_errs    0
[/dev/loop0].corruption_errs  0
[/dev/loop0].generation_errs  0
[Inferior 1 (process 2805) exited normally]

So this is simply a discrepancy in handling a device vs. the device(s)
for a mountpoint.

> I can do the device stats ioctl as normal user:
> 
> import btrfs
> fs = btrfs.FileSystem('/')
> btrfs.utils.pretty_print(fs.dev_stats(1))
> 
> <btrfs.ioctl.DevStats>
> devid: 1
> nr_items: 5
> flags: 0
> write_errs: 0
> read_errs: 0
> flush_errs: 0
> generation_errs: 0
> corruption_errs: 0

That works for me too, and that's actually the important part. \o/
Glad we talked about it. :}

-h

[1] https://github.com/kdave/btrfs-progs/blob/7faaca0d9f78f7162ae603231f693dd8e1af2a41/utils.c#L1666

Re: Curious problem: btrfs device stats & unpriviliged access

[Hans van Kranenburg]

On 10/08/2018 05:29 PM, Holger Hoffstätte wrote:
> On 10/08/18 16:40, Hans van Kranenburg wrote:
>>> Looking at the kernel side of things in fs/btrfs/ioctl.c I see both
>>> BTRFS_IOC_TREE_SEARCH[_V2} unconditionally require CAP_SYS_ADMIN.
>>
>> That's the tree search ioctl, for reading arbitrary metadata.
>>
>> The device stats ioctl is IOC_GET_DEV_STATS...
> 
> Yeah..OK, that is clear and gave me the hint to ask: why is it
> calling this in the first place? And as it turns out [1] is where
> it seems to go wrong, as is_block_device() returns 0 (as it should),
> fi_args.num_devices is never set (remains 0) and it proceeds to call
> everything below, eventually calling the BTRFS_IOC_FS_INFO ioctl in
> #1712. And that works fine:
> 
> 1711 if (fi_args->num_devices != 1) {
> (gdb) s
> 1712    ret = ioctl(fd, BTRFS_IOC_FS_INFO, fi_args);
> (gdb) s
> 1713    if (ret < 0) {
> (gdb) p ret
> $28 = 0
> (gdb) p *fi_args
> $30 = {
>   max_id = 1,
>   num_devices = 1,
>   fsid = "z%:\371\315\033A\203\267.\200\255;FH\221",
>   nodesize = 16384,
>   sectorsize = 4096,
>   clone_alignment = 4096,
>   reserved32 = 0,
>   reserved = {0 <repeats 122 times>}
> }
> 
> It's only when it goes into search_chunk_tree_for_fs_info()
> where things fail due to CAP_SYS_ADMIN.
> 
> And all this explains the actual bug: when I call btrfs device stats
> not on the mountpoint (as I've been trying all this time) but rather
> on the device, it works properly right away as regular user:
> 
> (gdb) set args device stats /dev/loop0
> (gdb) r
> Breakpoint 1, get_fs_info (path=path@entry=0x7fffffffe527 "/dev/loop0",
> fi_args=fi_args@entry=0x7fffffffd400,
>     di_ret=di_ret@entry=0x7fffffffd3f0) at utils.c:1652
> 1652    {
> (gdb) c
> Continuing.
> [/dev/loop0].write_io_errs    0
> [/dev/loop0].read_io_errs     0
> [/dev/loop0].flush_io_errs    0
> [/dev/loop0].corruption_errs  0
> [/dev/loop0].generation_errs  0
> [Inferior 1 (process 2805) exited normally]
> 
> So this is simply a discrepancy in handling a device vs. the device(s)
> for a mountpoint.

Apparently based on what you point it at, it does a different thing.

If you point it at a block device, it will try opening the block device
to find out which devid it has (from the superblock), find out where it
is mounted and then only ask for stats for this one device.

-# btrfs dev stats /dev/xvdc
[/dev/xvdc].write_io_errs    0
[/dev/xvdc].read_io_errs     0
[/dev/xvdc].flush_io_errs    0
[/dev/xvdc].corruption_errs  0
[/dev/xvdc].generation_errs  0

If you point it at a mounted filesystem, it lists stats for all devices.
Since there's no way to get a list of devids, it directly searches the
chunk tree directly for dev_item objects.

-# btrfs dev stats /mountpoint
[/dev/xvdb].write_io_errs    0
[/dev/xvdb].read_io_errs     0
[/dev/xvdb].flush_io_errs    0
[/dev/xvdb].corruption_errs  0
[/dev/xvdb].generation_errs  0
[/dev/xvdc].write_io_errs    0
[/dev/xvdc].read_io_errs     0
[/dev/xvdc].flush_io_errs    0
[/dev/xvdc].corruption_errs  0
[/dev/xvdc].generation_errs  0
[/dev/xvdd].write_io_errs    0
[/dev/xvdd].read_io_errs     0
[/dev/xvdd].flush_io_errs    0
[/dev/xvdd].corruption_errs  0
[/dev/xvdd].generation_errs  0

I do the same thing in the nagios plugin:

https://github.com/knorrie/python-btrfs/blob/master/examples/nagios/plugins/check_btrfs#L131

fs.devices() also looks for dev_items in the chunk tree:

https://github.com/knorrie/python-btrfs/blob/master/btrfs/ctree.py#L481

So, BOOM! you need root.

Or just start a 0, ignore errors and start trying all devids until you
found num_devices amount of them that work, yolo.

>> I can do the device stats ioctl as normal user:
>>
>> import btrfs
>> fs = btrfs.FileSystem('/')
>> btrfs.utils.pretty_print(fs.dev_stats(1))
>>
>> <btrfs.ioctl.DevStats>
>> devid: 1
>> nr_items: 5
>> flags: 0
>> write_errs: 0
>> read_errs: 0
>> flush_errs: 0
>> generation_errs: 0
>> corruption_errs: 0
> 
> That works for me too, and that's actually the important part. \o/
> Glad we talked about it. :}
> 
> -h
> 
> [1]
> https://github.com/kdave/btrfs-progs/blob/7faaca0d9f78f7162ae603231f693dd8e1af2a41/utils.c#L1666
> 
> 


-- 
Hans van Kranenburg

Re: Curious problem: btrfs device stats & unpriviliged access

[Holger Hoffstätte]

On 10/08/18 17:46, Hans van Kranenburg wrote:
<snip>
> fs.devices() also looks for dev_items in the chunk tree:
> 
> https://github.com/knorrie/python-btrfs/blob/master/btrfs/ctree.py#L481
> 
> So, BOOM! you need root.
> 
> Or just start a 0, ignore errors and start trying all devids until you
> found num_devices amount of them that work, yolo.

Since I need to walk /sys/fs/btrfs/ anyway I *think* I can just look
at the entries in /sys/fs/btrfs/<fsid>/devices/ and query them all
directly.

The skeleton btrfs_exporter already responds to http requests and
returns dummy metrics, using the official python client library.
I've found a nice little python sysfs scraper and now only need to
figure out how best to map the btrfs info in sysfs to useful metrics.
The privileged access issue would only have come into play much later,
but it seems I can avoid it after all, which is great.
I'm also (re-)learning python in the process, so that's the actual
thing slowing me down..

-h

Re: Curious problem: btrfs device stats & unpriviliged access

[Hans van Kranenburg]

On 10/08/2018 06:37 PM, Holger Hoffstätte wrote:
> On 10/08/18 17:46, Hans van Kranenburg wrote:
> <snip>
>> fs.devices() also looks for dev_items in the chunk tree:
>>
>> https://github.com/knorrie/python-btrfs/blob/master/btrfs/ctree.py#L481
>>
>> So, BOOM! you need root.
>>
>> Or just start a 0, ignore errors and start trying all devids until you
>> found num_devices amount of them that work, yolo.
> 
> Since I need to walk /sys/fs/btrfs/ anyway I *think* I can just look
> at the entries in /sys/fs/btrfs/<fsid>/devices/ and query them all
> directly.

But, you still need root for that right? The progs code does a RO open
directly on the block device.

-$ btrfs dev stats /dev/xvdb
ERROR: cannot open /dev/xvdb: Permission denied
ERROR: '/dev/xvdb' is not a mounted btrfs device

stat("/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 0), ...}) = 0
stat("/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 0), ...}) = 0
open("/dev/loop0", O_RDONLY)            = -1 EACCES (Permission denied)

But:

-# btrfs dev stats /dev/xvdb
[/dev/xvdb].write_io_errs    0
[/dev/xvdb].read_io_errs     0
[/dev/xvdb].flush_io_errs    0
[/dev/xvdb].corruption_errs  0
[/dev/xvdb].generation_errs  0

> The skeleton btrfs_exporter already responds to http requests and
> returns dummy metrics, using the official python client library.
> I've found a nice little python sysfs scraper and now only need to
> figure out how best to map the btrfs info in sysfs to useful metrics.
> The privileged access issue would only have come into play much later,
> but it seems I can avoid it after all, which is great.
> I'm also (re-)learning python in the process, so that's the actual
> thing slowing me down..

:)

-- 
Hans van Kranenburg

Re: Curious problem: btrfs device stats & unpriviliged access

[Holger Hoffstätte]

It's been a while.. :D

On 10/8/18 10:02 PM, Hans van Kranenburg wrote:
> On 10/08/2018 06:37 PM, Holger Hoffstätte wrote:
>> On 10/08/18 17:46, Hans van Kranenburg wrote:
>> <snip>
>>> fs.devices() also looks for dev_items in the chunk tree:
>>>
>>> https://github.com/knorrie/python-btrfs/blob/master/btrfs/ctree.py#L481
>>>
>>> So, BOOM! you need root.
>>>
>>> Or just start a 0, ignore errors and start trying all devids until you
>>> found num_devices amount of them that work, yolo.
>>
>> Since I need to walk /sys/fs/btrfs/ anyway I *think* I can just look
>> at the entries in /sys/fs/btrfs/<fsid>/devices/ and query them all
>> directly.
> 
> But, you still need root for that right? The progs code does a RO open
> directly on the block device.
> 
> -$ btrfs dev stats /dev/xvdb
> ERROR: cannot open /dev/xvdb: Permission denied
> ERROR: '/dev/xvdb' is not a mounted btrfs device
> 
> stat("/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 0), ...}) = 0
> stat("/dev/loop0", {st_mode=S_IFBLK|0660, st_rdev=makedev(7, 0), ...}) = 0
> open("/dev/loop0", O_RDONLY)            = -1 EACCES (Permission denied)
> 
> But:
> 
> -# btrfs dev stats /dev/xvdb
> [/dev/xvdb].write_io_errs    0
> [/dev/xvdb].read_io_errs     0
> [/dev/xvdb].flush_io_errs    0
> [/dev/xvdb].corruption_errs  0
> [/dev/xvdb].generation_errs  0

As it turns out you don't need full root, you need permissions to read the device.
Which permissions are those, you ask?

holger>ll /dev/loop0
brw-rw---- 1 root disk 7, 0 Oct 23 02:10 /dev/loop0

Indeed directly reading the device started working when I added myself to the
"disk" group (I was in wheel before, but that wasn't sufficient - good!). \o/
Never go full root. Adding the daemon to a group (e.g. during installation)
is IMHO acceptable.

This particular rabbit hole goes deep since it involves systemd, udev,
session management and eventually alcohol; see:
https://wiki.archlinux.org/index.php/Users_and_groups#Group_list and
https://enotty.pipebreaker.pl/2012/05/23/linux-automatic-user-acl-management/
for things you never really wanted to know. In my case I can get away with
the disk group menbership since I'm on Gentoo with OpenRC.

I still have to figure out the devid(s) for the ioctl, but the above
mentioned Yolo Method™ of iterating after finding the number of devices
in sysfs (under <fs-uuid>/devices/) might just be good enough; we'll see.

-h