Hello world, python-btrfs

Hello world, python-btrfs

[Hans van Kranenburg]

Hi!

After playing around a bit for a few months with a bunch of
proof-of-concept level scripts to be able to debug my btrfs file
systems, the inevitable happened:

https://github.com/knorrie/python-btrfs/

Currently, the primary goal of this module is to be able to inspect the
internals of an existing filesystem for educational purposes.

In the last week I started sorting out the puzzle pieces I already
assembled, and put most of them together into a proper python library to
query a btrfs filesystem.

The python module acts as a wrapper around the low level kernel calls
and btrfs data structures, presenting them as python objects with
interesting attributes and references to other objects.

Using these helpers, it should be fairly easy to reimplement the same
functionality as for example btrfs fi df, btrfs fi show and btrfs
inspect-internal provide.

For example:

import btrfs
fs = btrfs.FileSystem('/')
for space in fs.space_info():
     print("{0}, {1}: total={2}, used={3}".format(
           btrfs.utils.block_group_type_str(space.flags),
           btrfs.utils.block_group_profile_str(space.flags),
           btrfs.utils.pretty_size(space.total_bytes),
           btrfs.utils.pretty_size(space.used_bytes)))

results in:

Data, single: total=15.94GiB, used=6.30GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=768.00MiB, used=158.72MiB
GlobalReserve, single: total=64.00MiB, used=0.00B

The btrfs directory in the source is the python module which is to be
imported as btrfs.

The examples directory contains example programs that use the library.
For all functionality that's added in the lib, there must be an example
program that uses it.

Currently, the code can look at devices, chunks, block groups with their
usage and lists of data extents. I usually write quite elaborate git
commit messages, so when looking through my commits or doing a git blame
on code, you should (now and later) find a nice description about the
hows whys etc of the code.

Why?!

* Because it's fun!
* Because I get sick of using horrible ducttape regex solutions to
programmatically parse human readable output of external tools to
implement monitoring tools~~!11one.
* Because I might to be able to help other people that also want to
learn the internals of btrfs.
* Much more...

How?

* It's a python library, so the code will have a decent level of
pythonicity. The ioctl search returns a generator, the search key is a
nice object you can just increment etc...
* This is a work in progress. I can only add support for parts of btrfs
that I understand myself first.

Moo!

Hans van Kranenburg

-- 
Hans van Kranenburg - System / Network Engineer
T +31 (0)10 2760434 | hans.van.kranenburg@mendix.com | www.mendix.com

Re: Hello world, python-btrfs

[David Sterba]

On Sat, Jun 18, 2016 at 12:01:53AM +0200, Hans van Kranenburg wrote:
> proof-of-concept level scripts to be able to debug my btrfs file
> systems, the inevitable happened:
> 
> https://github.com/knorrie/python-btrfs/
> 
> Currently, the primary goal of this module is to be able to inspect the
> internals of an existing filesystem for educational purposes.

Looks great, I like. Python is better language for prototyping various
things, also more convenient to write one-shot scripts to do specific
rescue tasks.

> The python module acts as a wrapper around the low level kernel calls
> and btrfs data structures, presenting them as python objects with
> interesting attributes and references to other objects.

I guess it could be extended to read the filesystem objects directly
from the image, so it's not only built around the search ioctl.

> Why?!
> 
> * Because it's fun!

Best reason ever!

> * Because I get sick of using horrible ducttape regex solutions to
> programmatically parse human readable output of external tools to
> implement monitoring tools~~!11one.
> * Because I might to be able to help other people that also want to
> learn the internals of btrfs.
> * Much more...
> 
> How?
> 
> * It's a python library, so the code will have a decent level of
> pythonicity. The ioctl search returns a generator, the search key is a
> nice object you can just increment etc...
> * This is a work in progress. I can only add support for parts of btrfs
> that I understand myself first.

My question is whether it sounds like a good idea to host the python
bindings inside btrfs-progs or not. I can imagine creating scripts to
craft images for testing or verifying results of fsck, so it would be
more convenient to keep them in one place (but using a git submodule
instead would not be a big deal).

I'm mostly interested in the object representation of the filesystem
objects and basic API around that, so if you have other plans with
python-btrfs, we can at least share some of the code. My python-fu is
not that strong so I'd rely on you or others to maintain that if you're
interested.

Re: Hello world, python-btrfs

[Hans van Kranenburg]

Hi,

On 06/23/2016 04:26 PM, David Sterba wrote:
> On Sat, Jun 18, 2016 at 12:01:53AM +0200, Hans van Kranenburg wrote:
>> proof-of-concept level scripts to be able to debug my btrfs file
>> systems, the inevitable happened:
>>
>> https://github.com/knorrie/python-btrfs/
>>
>> Currently, the primary goal of this module is to be able to inspect the
>> internals of an existing filesystem for educational purposes.
>
> Looks great, I like.

Thanks, much appreciated!

> Python is better language for prototyping various
> things, also more convenient to write one-shot scripts to do specific
> rescue tasks.
>
>> The python module acts as a wrapper around the low level kernel calls
>> and btrfs data structures, presenting them as python objects with
>> interesting attributes and references to other objects.
>
> I guess it could be extended to read the filesystem objects directly
> from the image, so it's not only built around the search ioctl.
>
>> Why?!
>>
>> * Because it's fun!
>
> Best reason ever!

Moo!

>> * Because I get sick of using horrible ducttape regex solutions to
>> programmatically parse human readable output of external tools to
>> implement monitoring tools~~!11one.

This is actually the main reason I started it. Or was it being able to 
see where balance got the usage information from...

>> * Because I might to be able to help other people that also want to
>> learn the internals of btrfs.
>> * Much more...
>>
>> How?
>>
>> * It's a python library, so the code will have a decent level of
>> pythonicity. The ioctl search returns a generator, the search key is a
>> nice object you can just increment etc...
>> * This is a work in progress. I can only add support for parts of btrfs
>> that I understand myself first.
>
> My question is whether it sounds like a good idea to host the python
> bindings inside btrfs-progs or not.

Whoa, hold your horses!

> I can imagine creating scripts to
> craft images for testing or verifying results of fsck, so it would be
> more convenient to keep them in one place (but using a git submodule
> instead would not be a big deal).

python-btrfs is just a library. You can just clone the git somewhere, 
start hacking on it and put it in your python path, or pip install or 
apt-get install python-btrfs etc...

Programs that use it can be anywhere. What I have in examples/ now 
should probably end up either in howto style documentation or in proper 
tests for the lib (which requires keeping known-state filesystem images 
to test on etc, exploding into a new level of whoa).

> I'm mostly interested in the object representation of the filesystem
> objects and basic API around that, so if you have other plans with
> python-btrfs, we can at least share some of the code. My python-fu is
> not that strong so I'd rely on you or others to maintain that if you're
> interested.

Ok, I added a little TODO:
https://github.com/knorrie/python-btrfs/blob/master/TODO.md

There's a few things I want to do first at least, which are completing 
to support the full set of data structures using the kernel API, 
finishing to move existing code that I have lying around to use the new 
lib and a bit of extra info to help interested sysadmin/programmers to 
start using the library to peek around in their own filesystem a bit to 
see what's going on.

I haven't looked at how the btrfs-progs treat disks directly, but would 
certainly be interested in learning about that.

 From what you say, I feel that you're thinking about not only looking 
around in filesystems, but having ways to manipulate them?

Can you provide a more concrete example of "craft images for testing" or 
related ideas about how this project could help during regular btrfs 
development?

-- 
Hans van Kranenburg - System / Network Engineer
T +31 (0)10 2760434 | hans.van.kranenburg@mendix.com | www.mendix.com

Re: Hello world, python-btrfs

[David Sterba]

On Fri, Jun 24, 2016 at 01:29:22AM +0200, Hans van Kranenburg wrote:
> > I can imagine creating scripts to
> > craft images for testing or verifying results of fsck, so it would be
> > more convenient to keep them in one place (but using a git submodule
> > instead would not be a big deal).
> 
> python-btrfs is just a library. You can just clone the git somewhere, 
> start hacking on it and put it in your python path, or pip install or 
> apt-get install python-btrfs etc...

So it sounds more appropriate to keep them separate.

> Programs that use it can be anywhere. What I have in examples/ now 
> should probably end up either in howto style documentation or in proper 
> tests for the lib (which requires keeping known-state filesystem images 
> to test on etc, exploding into a new level of whoa).
> 
> > I'm mostly interested in the object representation of the filesystem
> > objects and basic API around that, so if you have other plans with
> > python-btrfs, we can at least share some of the code. My python-fu is
> > not that strong so I'd rely on you or others to maintain that if you're
> > interested.
> 
> Ok, I added a little TODO:
> https://github.com/knorrie/python-btrfs/blob/master/TODO.md
> 
> There's a few things I want to do first at least, which are completing 
> to support the full set of data structures using the kernel API, 
> finishing to move existing code that I have lying around to use the new 
> lib and a bit of extra info to help interested sysadmin/programmers to 
> start using the library to peek around in their own filesystem a bit to 
> see what's going on.

So this will be mostly ioctl based.

> I haven't looked at how the btrfs-progs treat disks directly, but would 
> certainly be interested in learning about that.

The core work lies in the b-tree item manipulation, free block
allocation and maybe more.

>  From what you say, I feel that you're thinking about not only looking 
> around in filesystems, but having ways to manipulate them?

Yes. Initially it can be only in-place modification, like rewrite a
specific member of an item and recalculate the block checksum. We can do that
with btrfs-corrupt-block but the command line interface is not that great.

> Can you provide a more concrete example of "craft images for testing" or 
> related ideas about how this project could help during regular btrfs 
> development?

With a good support of python interface to btrfs structures, we can eg.
lookup specific extent items, swap some pointers and see how fsck
survives.  Writing that as a C code is possible, but IMO more tedious
compared to python.

Re: Hello world, python-btrfs

[Juan Orti Alcaine]

2016-06-18 0:01 GMT+02:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>:
> Hi!
>
> After playing around a bit for a few months with a bunch of
> proof-of-concept level scripts to be able to debug my btrfs file
> systems, the inevitable happened:
>
> https://github.com/knorrie/python-btrfs/
>

Hi, I want just to inform you that I've submitted your module to Fedora.

Thanks for your work.