RFE: 'btrfs' tools machine readable output

RFE: 'btrfs' tools machine readable output

[Richard W.M. Jones]

I don't have time to implement this right now, so I'm just posting
this as a suggestion/request ...

It would be really helpful if the btrfs tools had a machine-readable
output.

Libguestfs parses btrfs tools output in a number of places, eg:
https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
This is a massive PITA because each time a new release of btrfs-progs
comes along it changes the output slightly, and we end up having
to add all sorts of hacks.

With machine-readable output, there'd be a flag which would
change the output.  eg:

$ btrfs filesystem show
Label: 'ROOT'  uuid: af471cfc-421a-4d51-8811-ce969f76828a
        Total devices 1 FS bytes used 5.29MiB
        devid    1 size 767.97MiB used 92.00MiB path /dev/sda2

would become:

$ btrfs --json filesystem show
{
  "devices": {
     "Label": "ROOT",
     "uuid": "af471cfc-421a-4d51-8811-ce969f76828a",
     /// etc
  }
}

By this example I don't mean that JSON has to be the format -- in fact
it's a terrible format with all sorts of problems -- any format which
is parseable with C libraries would do for us.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Re: RFE: 'btrfs' tools machine readable output

[Martin Steigerwald]

Hello Richard,

On Montag, 16. Mai 2016 13:14:56 CEST Richard W.M. Jones wrote:
> I don't have time to implement this right now, so I'm just posting
> this as a suggestion/request ...
> 
> It would be really helpful if the btrfs tools had a machine-readable
> output.
> 
> Libguestfs parses btrfs tools output in a number of places, eg:
> https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
> This is a massive PITA because each time a new release of btrfs-progs
> comes along it changes the output slightly, and we end up having
> to add all sorts of hacks.
> 
> With machine-readable output, there'd be a flag which would
> change the output.  eg:

I wonder whether parsing a text based output is really the most elegant method 
here.

How about a libbtrfs so that other tools can benefit from btrfs tools 
functionality? This was also desktop environments wishing to make use of 
snapshot functionality or advanced disk usage reporting for example can easily 
make use of it without calling external commands.

Of course it would likely me more effort than to implement structured output.

Thanks,
-- 
Martin

Re: RFE: 'btrfs' tools machine readable output

[Richard W.M. Jones]

On Mon, May 16, 2016 at 02:21:07PM +0200, Martin Steigerwald wrote:
> Hello Richard,
> 
> On Montag, 16. Mai 2016 13:14:56 CEST Richard W.M. Jones wrote:
> > I don't have time to implement this right now, so I'm just posting
> > this as a suggestion/request ...
> > 
> > It would be really helpful if the btrfs tools had a machine-readable
> > output.
> > 
> > Libguestfs parses btrfs tools output in a number of places, eg:
> > https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
> > This is a massive PITA because each time a new release of btrfs-progs
> > comes along it changes the output slightly, and we end up having
> > to add all sorts of hacks.
> > 
> > With machine-readable output, there'd be a flag which would
> > change the output.  eg:
> 
> I wonder whether parsing a text based output is really the most elegant method 
> here.
> 
> How about a libbtrfs so that other tools can benefit from btrfs tools 
> functionality? This was also desktop environments wishing to make use of 
> snapshot functionality or advanced disk usage reporting for example can easily 
> make use of it without calling external commands.
> 
> Of course it would likely me more effort than to implement structured output.

Yes, a similar situation happened with qemu actually.  We've been
talking for over half a decade about putting the qemu block layer into
a library, and it's still not happened.  But we got `qemu-img info' to
have JSON output in a few weeks, and that is now how we query the
properties of VM disk images:

$ qemu-img info --output=json ./builder/fedora.qcow2
{
    "virtual-size": 1073741824,
    "filename": "./builder/fedora.qcow2",
    "cluster-size": 65536,
    "format": "qcow2",
    "actual-size": 7933952,
    "format-specific": {
        "type": "qcow2",
        "data": {
            "compat": "1.1",
            "lazy-refcounts": false,
            "refcount-bits": 16,
            "corrupt": false
        }
    },
    "dirty-flag": false
}

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Re: RFE: 'btrfs' tools machine readable output

[Pino Toscano]

[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]

On Monday 16 May 2016 14:21:07 Martin Steigerwald wrote:
> Hello Richard,
> 
> On Montag, 16. Mai 2016 13:14:56 CEST Richard W.M. Jones wrote:
> > I don't have time to implement this right now, so I'm just posting
> > this as a suggestion/request ...
> > 
> > It would be really helpful if the btrfs tools had a machine-readable
> > output.
> > 
> > Libguestfs parses btrfs tools output in a number of places, eg:
> > https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
> > This is a massive PITA because each time a new release of btrfs-progs
> > comes along it changes the output slightly, and we end up having
> > to add all sorts of hacks.
> > 
> > With machine-readable output, there'd be a flag which would
> > change the output.  eg:
> 
> I wonder whether parsing a text based output is really the most elegant method 
> here.
> 
> How about a libbtrfs so that other tools can benefit from btrfs tools 
> functionality?

btrfs-progs is GPL v2 only, and that may be an issue for consumers
(the libguestfs daemon, the component interacting with devices and
filesystems inside the libguestfs appliance, is GPL v2+ for example).

> Of course it would likely me more effort than to implement structured output.

There's this of course, which is not entirely negligible.

-- 
Pino Toscano

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: RFE: 'btrfs' tools machine readable output

[Austin S. Hemmelgarn]

On 2016-05-16 08:14, Richard W.M. Jones wrote:
> I don't have time to implement this right now, so I'm just posting
> this as a suggestion/request ...
>
> It would be really helpful if the btrfs tools had a machine-readable
> output.
>
> Libguestfs parses btrfs tools output in a number of places, eg:
> https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
> This is a massive PITA because each time a new release of btrfs-progs
> comes along it changes the output slightly, and we end up having
> to add all sorts of hacks.
>
> With machine-readable output, there'd be a flag which would
> change the output.  eg:
>
> $ btrfs filesystem show
> Label: 'ROOT'  uuid: af471cfc-421a-4d51-8811-ce969f76828a
>         Total devices 1 FS bytes used 5.29MiB
>         devid    1 size 767.97MiB used 92.00MiB path /dev/sda2
>
> would become:
>
> $ btrfs --json filesystem show
> {
>   "devices": {
>      "Label": "ROOT",
>      "uuid": "af471cfc-421a-4d51-8811-ce969f76828a",
>      /// etc
>   }
> }
>
> By this example I don't mean that JSON has to be the format -- in fact
> it's a terrible format with all sorts of problems -- any format which
> is parseable with C libraries would do for us.
I would love to see something like this myself, as it would make 
integration with monitoring tools so much easier.  I'd vote for YAML as 
the output format though, as it's easily human readable as well as 
machine parseable while still being space efficient.  Output from your 
example above might look something like this:
---
- filesystems
   - label: 'ROOT'
     uuid: af471cfc-421a-4d51-8811-ce969f76828a
     devices: 1
       - devid: 1
         size: 767.97 MB
         used 92.00 MB
         path: /dev/sda2
     used: 5.29 MB
...
Although of course the numbers would probably just be in bytes instead 
of 'human' values.

Re: RFE: 'btrfs' tools machine readable output

[Nicholas D Steeves]

On 16 May 2016 at 08:39, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote:
> On 2016-05-16 08:14, Richard W.M. Jones wrote:
>>
>> It would be really helpful if the btrfs tools had a machine-readable
>> output.
>> With machine-readable output, there'd be a flag which would
>> change the output.  eg:
>> $ btrfs --json filesystem show
>> {
>>   "devices": {
>>      "Label": "ROOT",
>>      "uuid": "af471cfc-421a-4d51-8811-ce969f76828a",
>>      /// etc
>>   }
>> }
> I would love to see something like this myself, as it would make integration
> with monitoring tools so much easier.  I'd vote for YAML as the output format
> though, as it's easily human readable as well as machine parseable while
> still being space efficient.
> ---
> - filesystems
>   - label: 'ROOT'
>     uuid: af471cfc-421a-4d51-8811-ce969f76828a
>     devices: 1
>       - devid: 1
>         size: 767.97 MB
>         used 92.00 MB
>         path: /dev/sda2
>     used: 5.29 MB
> ...

Rather than adding language-specific output/extensions to btrfs-progs,
what does everyone think of an interface which allows arbitrary
ordering similar to the date command, with arbitrary delimiter
support?

eg: date +%a\ %d\ \%B\ %H:%m

with something like: %D (device), %L (label), %V (volume), %v
(subvolume), %U (volume UUID), %u (subvol UUID), et al.

'avoids formatting the output to any particular convention, and the
order of what is queried can be in any order...but I think the [unit
to operate on] [named unit] bit would necessarily need to come first.
We already have that in btrfs [sub/fi/dev] [name] :-)

Or should it be a separate command...  Something like "btrfs-machine
[unit to operate on] [named unit] [action eg: print] [cryptic line
with user's choice of delimiter and formatting]".  And then there
might also be the option to "btrfs-machine [unit to operate on] [named
unit] return [scrub status, balance status, etc.]" which returns a
single integer value along the lines of idle|in
progress|success|failure.  And at some point in the future
btrfs-machine [unit to operate on] [named unit] progress [operation]
which functions like "fsck -C [fd]".

Now that, I imagine, would be something libvirt users would like,
whether for web interface, GUI, or wrapper script for for a slick NAS
web interfaces.

For now, maybe start with a proof of concept with only the "print"
action implemented?  And eventually merge this into the main btrfs
command something like this: "btrfs -m unit name action args" or
"btrfs --machine unit name action args"

On 16 May 2016 at 08:21, Martin Steigerwald <martin@lichtvoll.de> wrote:
> How about a libbtrfs so that other tools can benefit from btrfs tools
> functionality?
Does much does /usr/lib/x86_64-linux-gnu/libbtrfs.so expose?

What is the path of minimum duplication of work, and minimum long-term
maintenance?  If my assumptions are correct, this is a 2000-level
programming course textbook challenge on writing a program using a
library to query some values...and that might be something I could
manage :-)

Cheers,
Nicholas

Re: RFE: 'btrfs' tools machine readable output

[Austin S. Hemmelgarn]

On 2016-05-27 15:47, Nicholas D Steeves wrote:
> On 16 May 2016 at 08:39, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote:
>> On 2016-05-16 08:14, Richard W.M. Jones wrote:
>>>
>>> It would be really helpful if the btrfs tools had a machine-readable
>>> output.
>>> With machine-readable output, there'd be a flag which would
>>> change the output.  eg:
>>> $ btrfs --json filesystem show
>>> {
>>>   "devices": {
>>>      "Label": "ROOT",
>>>      "uuid": "af471cfc-421a-4d51-8811-ce969f76828a",
>>>      /// etc
>>>   }
>>> }
>> I would love to see something like this myself, as it would make integration
>> with monitoring tools so much easier.  I'd vote for YAML as the output format
>> though, as it's easily human readable as well as machine parseable while
>> still being space efficient.
>> ---
>> - filesystems
>>   - label: 'ROOT'
>>     uuid: af471cfc-421a-4d51-8811-ce969f76828a
>>     devices: 1
>>       - devid: 1
>>         size: 767.97 MB
>>         used 92.00 MB
>>         path: /dev/sda2
>>     used: 5.29 MB
>> ...
>
> Rather than adding language-specific output/extensions to btrfs-progs,
> what does everyone think of an interface which allows arbitrary
> ordering similar to the date command, with arbitrary delimiter
> support?
>
> eg: date +%a\ %d\ \%B\ %H:%m
>
> with something like: %D (device), %L (label), %V (volume), %v
> (subvolume), %U (volume UUID), %u (subvol UUID), et al.
>
> 'avoids formatting the output to any particular convention, and the
> order of what is queried can be in any order...but I think the [unit
> to operate on] [named unit] bit would necessarily need to come first.
> We already have that in btrfs [sub/fi/dev] [name] :-)
>
> Or should it be a separate command...  Something like "btrfs-machine
> [unit to operate on] [named unit] [action eg: print] [cryptic line
> with user's choice of delimiter and formatting]".  And then there
> might also be the option to "btrfs-machine [unit to operate on] [named
> unit] return [scrub status, balance status, etc.]" which returns a
> single integer value along the lines of idle|in
> progress|success|failure.  And at some point in the future
> btrfs-machine [unit to operate on] [named unit] progress [operation]
> which functions like "fsck -C [fd]".
>
> Now that, I imagine, would be something libvirt users would like,
> whether for web interface, GUI, or wrapper script for for a slick NAS
> web interfaces.
>
> For now, maybe start with a proof of concept with only the "print"
> action implemented?  And eventually merge this into the main btrfs
> command something like this: "btrfs -m unit name action args" or
> "btrfs --machine unit name action args"
I'm not sure how practical this would be, the format language would have 
to be pretty complex, and would need to be easily extensible at the same 
time.  We have a functionally infinite upper bound on what can be 
reported by a number of commands (fi df for example is bounded only by 
the number of supported profiles), and trying to sanely handle that gets 
complicated.

Also, once we have _some_ kind of machine parseable output, it becomes 
much easier to use other tools to pull out the exact info you want.
>
> On 16 May 2016 at 08:21, Martin Steigerwald <martin@lichtvoll.de> wrote:
>> How about a libbtrfs so that other tools can benefit from btrfs tools
>> functionality?
> Does much does /usr/lib/x86_64-linux-gnu/libbtrfs.so expose?
>
> What is the path of minimum duplication of work, and minimum long-term
> maintenance?  If my assumptions are correct, this is a 2000-level
> programming course textbook challenge on writing a program using a
> library to query some values...and that might be something I could
> manage :-)
Based on the fact that our CLI interface is really the only API people 
are using, I don't think there's almost anything exposed at a high-level 
in libbtrfs.

Re: RFE: 'btrfs' tools machine readable output

[David Sterba]

On Mon, May 16, 2016 at 01:14:56PM +0100, Richard W.M. Jones wrote:
> I don't have time to implement this right now, so I'm just posting
> this as a suggestion/request ...

Neither do have I, but agree with the idea and the proposed way. Here
are my notes
https://github.com/kdave/drafts/blob/master/btrfs/progs-output-format.txt

> It would be really helpful if the btrfs tools had a machine-readable
> output.
> 
> Libguestfs parses btrfs tools output in a number of places, eg:
> https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
> This is a massive PITA because each time a new release of btrfs-progs
> comes along it changes the output slightly, and we end up having
> to add all sorts of hacks.

The error messages haven't been touched for a long time and they get
fixed or rephrased incrementally. I'm sorry you have to deal with the
breakage but please understand that we don't have resources to do it at
once or to implement complete machine readable output. At the moment the
messages are aimed for a human consumer.

Wiring the various formats into the codebase would be pretty intrusive,
but doable. We can do some commands first as a prototype, the 'fi show'
seems like a good candidate.

> With machine-readable output, there'd be a flag which would
> change the output.  eg:
> 
> $ btrfs filesystem show
> Label: 'ROOT'  uuid: af471cfc-421a-4d51-8811-ce969f76828a
>         Total devices 1 FS bytes used 5.29MiB
>         devid    1 size 767.97MiB used 92.00MiB path /dev/sda2
> 
> would become:
> 
> $ btrfs --json filesystem show
> {
>   "devices": {
>      "Label": "ROOT",
>      "uuid": "af471cfc-421a-4d51-8811-ce969f76828a",
>      /// etc
>   }
> }
> 
> By this example I don't mean that JSON has to be the format -- in fact
> it's a terrible format with all sorts of problems -- any format which
> is parseable with C libraries would do for us.

My ideas was to allow multiple formats, with slight change to the
syntax:

$ btrfs --format=json ...
$ btrfs --format=plain ...
$ btrfs --format=yaml ...

Re: RFE: 'btrfs' tools machine readable output

[Austin S. Hemmelgarn]

On 2016-05-17 05:33, David Sterba wrote:
> On Mon, May 16, 2016 at 01:14:56PM +0100, Richard W.M. Jones wrote:
>> I don't have time to implement this right now, so I'm just posting
>> this as a suggestion/request ...
>
> Neither do have I, but agree with the idea and the proposed way. Here
> are my notes
> https://github.com/kdave/drafts/blob/master/btrfs/progs-output-format.txt
>
>> It would be really helpful if the btrfs tools had a machine-readable
>> output.
>>
>> Libguestfs parses btrfs tools output in a number of places, eg:
>> https://github.com/libguestfs/libguestfs/blob/master/daemon/btrfs.c
>> This is a massive PITA because each time a new release of btrfs-progs
>> comes along it changes the output slightly, and we end up having
>> to add all sorts of hacks.
>
> The error messages haven't been touched for a long time and they get
> fixed or rephrased incrementally. I'm sorry you have to deal with the
> breakage but please understand that we don't have resources to do it at
> once or to implement complete machine readable output. At the moment the
> messages are aimed for a human consumer.
>
> Wiring the various formats into the codebase would be pretty intrusive,
> but doable. We can do some commands first as a prototype, the 'fi show'
> seems like a good candidate.
General list off the top of my head that would be good candidates:
* fi show
* fi df
* fi usage
* scrub status
* subvolume list
* subvolume show
Those cover about 90% of the regular maintenance scenario's that I can 
think of which need more data than just whether or not something succeeded.
>
>> With machine-readable output, there'd be a flag which would
>> change the output.  eg:
>>
>> $ btrfs filesystem show
>> Label: 'ROOT'  uuid: af471cfc-421a-4d51-8811-ce969f76828a
>>         Total devices 1 FS bytes used 5.29MiB
>>         devid    1 size 767.97MiB used 92.00MiB path /dev/sda2
>>
>> would become:
>>
>> $ btrfs --json filesystem show
>> {
>>   "devices": {
>>      "Label": "ROOT",
>>      "uuid": "af471cfc-421a-4d51-8811-ce969f76828a",
>>      /// etc
>>   }
>> }
>>
>> By this example I don't mean that JSON has to be the format -- in fact
>> it's a terrible format with all sorts of problems -- any format which
>> is parseable with C libraries would do for us.
>
> My ideas was to allow multiple formats, with slight change to the
> syntax:
>
> $ btrfs --format=json ...
> $ btrfs --format=plain ...
> $ btrfs --format=yaml ...
I like this idea, with one specific caveat, it would be _really_ nice to 
be able to control at compile time what formats are supported. 
Somebody's eventually going to want XML support, and I'd rather not have 
to depend on expat or something similar in a tool that I regularly build 
statically (I'm willing to tolerate YAML because that's what I'd use, 
but I would probably compile out the JSON support too).

Re: RFE: 'btrfs' tools machine readable output

[David Sterba]

On Tue, May 17, 2016 at 07:14:12AM -0400, Austin S. Hemmelgarn wrote:
> >> By this example I don't mean that JSON has to be the format -- in fact
> >> it's a terrible format with all sorts of problems -- any format which
> >> is parseable with C libraries would do for us.
> >
> > My ideas was to allow multiple formats, with slight change to the
> > syntax:
> >
> > $ btrfs --format=json ...
> > $ btrfs --format=plain ...
> > $ btrfs --format=yaml ...
> I like this idea, with one specific caveat, it would be _really_ nice to 
> be able to control at compile time what formats are supported. 
> Somebody's eventually going to want XML support, and I'd rather not have 
> to depend on expat or something similar in a tool that I regularly build 
> statically (I'm willing to tolerate YAML because that's what I'd use, 
> but I would probably compile out the JSON support too).

I hope we can avoid any 3rd party library dependencies, we're just
generating the output not parsing it. Either copy a version of the
libraries or copy the quoting code, the rest can be simply implemented as
a printf with enhanced format stings.

Re: RFE: 'btrfs' tools machine readable output

[Austin S. Hemmelgarn]

On 2016-05-17 08:23, David Sterba wrote:
> On Tue, May 17, 2016 at 07:14:12AM -0400, Austin S. Hemmelgarn wrote:
>>>> By this example I don't mean that JSON has to be the format -- in fact
>>>> it's a terrible format with all sorts of problems -- any format which
>>>> is parseable with C libraries would do for us.
>>>
>>> My ideas was to allow multiple formats, with slight change to the
>>> syntax:
>>>
>>> $ btrfs --format=json ...
>>> $ btrfs --format=plain ...
>>> $ btrfs --format=yaml ...
>> I like this idea, with one specific caveat, it would be _really_ nice to
>> be able to control at compile time what formats are supported.
>> Somebody's eventually going to want XML support, and I'd rather not have
>> to depend on expat or something similar in a tool that I regularly build
>> statically (I'm willing to tolerate YAML because that's what I'd use,
>> but I would probably compile out the JSON support too).
>
> I hope we can avoid any 3rd party library dependencies, we're just
> generating the output not parsing it. Either copy a version of the
> libraries or copy the quoting code, the rest can be simply implemented as
> a printf with enhanced format stings.
>
As much as I hate the possibility of having 3rd party library 
dependencies in btrfs-progs, I'm a lot more nervous about having the 
equivalent stuff open-coded or having a local copy of the library, as 
both have significant potential for bugs.  A significant majority of 
Linux systems have Python, and therefore have at least some XML library 
(I forget which one Python prefers to use).  Many also have at least one 
JSON library too.  If we make such dependencies optional, and also only 
load the library if we need it at runtime (so we would only load the 
JSON library if we are asked to print JSON output), then we could still 
avoid the overhead for most people.

The other possibility would be to just pick one output format, open code 
that, and be done with it.  It's not hard to write s simple script to 
convert between YAML, JSON, and other similar formats.  If we do go this 
way, I'd still suggest YAML as it's more human friendly, a lot easier to 
parse using scripts due to the line oriented nature of it, and is also 
probably easier to code without any external dependencies.

Re: RFE: 'btrfs' tools machine readable output

[Richard W.M. Jones]

As I said I don't have any particular favourite format.  However I'll
just point you to the code that qemu uses (basically open-coding) to
write json:

http://git.qemu.org/?p=qemu.git;a=blob;f=qobject/qjson.c;h=ef160d2119eb18aa917b71b40e37d54880b1ac7f;hb=HEAD#l83

Hopefully this is useful as a data point.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Re: RFE: 'btrfs' tools machine readable output

[David Sterba]

On Tue, May 17, 2016 at 02:32:14PM +0100, Richard W.M. Jones wrote:
> As I said I don't have any particular favourite format.  However I'll
> just point you to the code that qemu uses (basically open-coding) to
> write json:
> 
> http://git.qemu.org/?p=qemu.git;a=blob;f=qobject/qjson.c;h=ef160d2119eb18aa917b71b40e37d54880b1ac7f;hb=HEAD#l83
> 
> Hopefully this is useful as a data point.

It is, thanks, this amount of code to do json export matches my
expectations of a local version of the sources.