Btrfs Heatmap - v4 ... colors!!

Btrfs Heatmap - v4 ... colors!!

[Hans van Kranenburg]

Hi,

I just tagged v4 of the Btrfs Heatmap utility, which visualizes the
usage of your btrfs filesystem:

https://github.com/knorrie/btrfs-heatmap

It needs at least python-btrfs v5 to run, because of the fixed backref
handling.

The main changes are:
 * Colors! colors! colors!!!
 * Colors!
 * Improved documentation, thanks to user feedback. The first README
page does focus on quick satisfaction, while there are two extra
subpages about the extent-level details pictures and a scripting HOWTO
which explains how to do everything that the command line options cannot do.
 * The external 2k8 loc png library dependency is dropped, in favour of
a 34 line single purpose png writer implementation directly in the
script itself. This makes it even easier to get the thing going quickly,
and it makes the thing go more quickly as well.

And now, no more text to read. Go and create a bunch of beautiful
pictures and timelapses of your filesystem!

Have fun!

P.S. Here's where it all started... :D
http://logs.tvrrug.org.uk/logs/%23btrfs/2015-12-09.html#2015-12-09T20:54:04

-- 
Hans van Kranenburg

Re: Btrfs Heatmap - v4 ... colors!!

[Hans van Kranenburg]

[-- Attachment #1: Type: text/plain, Size: 3212 bytes --]

Hi Tim,

On 01/18/2017 01:38 AM, Tim Walberg wrote:
> I really like the new release. It's definitely nice to have the added colors.

Yay.

> Just one major request I would have - bring the linear option back. The
> hilbert output is cool for a single device, and is ok for a two-device file
> system, but it's not so easy to interpret especially when you have an odd
> number of devices in your file system - I have a 3-device RAID-1 config.

Ok, so because the curve continues its path and it may switch to a new
device somewhere, you don't see clearly where the new device starts
(there are also no extra helping lines / device borders).

> An
> alternative would be the ability to create separate maps for each device in
> a file system, either by selecting a devid via command line, or by creating
> individual images arranged in an array or something. That was one thing I
> could do with the original version that had a separate script for dumping
> the block groups and another for creating the PNG - I could filter the block
> group list by devid to do individual images.

Have a look at the scripting howto, linked from the main README
somewhere at the bottom of the page:

https://github.com/knorrie/btrfs-heatmap/blob/master/doc/scripting.md

One of the examples, 2.3 [..] separate image per device [..]  is an
example of how to do this. It involves a little bit of scripting, but
gives total flexibility of whatever you want to do.

fs = btrfs.FileSystem('/foo')
for device in fs.devices():
    grid = heatmap.walk_dev_extents(fs, [device])
    grid.write_png('device_%s.png' % device.devid)

The first half of the page explains the params you can give to
heatmap.walk_dev_extents, like also order and size etc.

Stitching the resulting images together in whatever form is up to you,
using your favourite image processing tools afterwards.

> Thanks for all the work that's gone into this so far!

Thanks!

Check out (wip) branch linear:

https://github.com/knorrie/btrfs-heatmap/commits/linear

There's a commit which puts linear and snake mode in, besides the
default hilbert (with --curve linear etc, like there was before). It's
just taking another route walking the grid. :)

The biggest problem with linear / snake is that it destroys the idea of
locality (long lines instead of little blocks of data together), and
when changing orders, it makes the same real point move around instead
of staying at the same spot.

See https://www.youtube.com/watch?v=DuiryHHTrjU&t=5m18s (->7m8s) for a
visual explanation.

Compare attached:
 - hilbert-order-8-size-9.png (bytes_per_pixel 41943040)
 - linear-order-8-size-9.png (bytes_per_pixel 41943040)

To get a bit of a usable result, I need to lower the resolution of the
image drastically, resulting in...
 - linear-order-6-size-9.png (bytes_per_pixel 671088640)
... with 640MiB crammed into a pixel instead of only 40MiB

The result is that usage gets blurred out too much, because block group
boundaries are too different from pixel boundaries.

But, being able to play around with it is fun and serves a purpose, so
I'll put this back in, and write a nice subpage of the README with some
pictures of this kind to explain.

o/
-- 
Hans van Kranenburg

[-- Attachment #2: hilbert-order-8-size-9.png --]
[-- Type: image/png, Size: 21666 bytes --]

[-- Attachment #3: linear-order-8-size-9.png --]
[-- Type: image/png, Size: 11647 bytes --]

[-- Attachment #4: linear-order-6-size-9.png --]
[-- Type: image/png, Size: 9297 bytes --]

Re: Btrfs Heatmap - v4 ... colors!!

[Timofey Titovets]

Thank you for your great work:
JFYI Packaged in AUR:
https://aur.archlinux.org/packages/python-btrfs-heatmap/
-- 
Have a nice day,
Timofey.

Re: Btrfs Heatmap - v4 ... colors!!

[Hans van Kranenburg]

On 02/03/2017 12:25 PM, Timofey Titovets wrote:
> Thank you for your great work:
> JFYI Packaged in AUR:
> https://aur.archlinux.org/packages/python-btrfs-heatmap/

Hey, thanks.

Just wondering... what is that btrfs.py you refer to in...
https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-btrfs ?

It's a package, not a single file, so maybe that compile command doesn't
do anything? I'm not familiar with arch, so correct me if I'm wrong.

.
├── btrfs
│   ├── crc32c.py
│   ├── ctree.py
│   ├── __init__.py
│   ├── ioctl.py
│   ├── utils.py

-- 
Hans van Kranenburg

Re: Btrfs Heatmap - v4 ... colors!!

[Timofey Titovets]

2017-02-03 15:57 GMT+03:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>:
> On 02/03/2017 12:25 PM, Timofey Titovets wrote:
>> Thank you for your great work:
>> JFYI Packaged in AUR:
>> https://aur.archlinux.org/packages/python-btrfs-heatmap/
>
> Hey, thanks.
>
> Just wondering... what is that btrfs.py you refer to in...
> https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-btrfs ?
>
> It's a package, not a single file, so maybe that compile command doesn't
> do anything? I'm not familiar with arch, so correct me if I'm wrong.
>
> .
> ├── btrfs
> │   ├── crc32c.py
> │   ├── ctree.py
> │   ├── __init__.py
> │   ├── ioctl.py
> │   ├── utils.py
>
> --
> Hans van Kranenburg

You right, compile command are useless, so i did remove it, thanks.

-- 
Have a nice day,
Timofey.

Re: Btrfs Heatmap - v4 ... colors!!

[Hans van Kranenburg]

On 02/03/2017 03:18 PM, Timofey Titovets wrote:
> 2017-02-03 15:57 GMT+03:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>:
>> On 02/03/2017 12:25 PM, Timofey Titovets wrote:
>>> Thank you for your great work:
>>> JFYI Packaged in AUR:
>>> https://aur.archlinux.org/packages/python-btrfs-heatmap/
>>
>> Hey, thanks.
>>
>> Just wondering... what is that btrfs.py you refer to in...
>> https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-btrfs ?
>>
>> It's a package, not a single file, so maybe that compile command doesn't
>> do anything? I'm not familiar with arch, so correct me if I'm wrong.
>>
>> .
>> ├── btrfs
>> │   ├── crc32c.py
>> │   ├── ctree.py
>> │   ├── __init__.py
>> │   ├── ioctl.py
>> │   ├── utils.py
>>
> 
> You right, compile command are useless, so i did remove it, thanks.

-$ python2 -m compileall btrfs
Listing btrfs ...
Compiling btrfs/__init__.py ...
Compiling btrfs/crc32c.py ...
Compiling btrfs/ctree.py ...
Compiling btrfs/ioctl.py ...
Compiling btrfs/utils.py ...

.
├── btrfs
│   ├── crc32c.py
│   ├── crc32c.pyc
│   ├── ctree.py
│   ├── ctree.pyc
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── ioctl.py
│   ├── ioctl.pyc
│   ├── utils.py
│   └── utils.pyc


-$ python3 -m compileall btrfs
Listing 'btrfs'...
Compiling 'btrfs/__init__.py'...
Compiling 'btrfs/crc32c.py'...
Compiling 'btrfs/ctree.py'...
Compiling 'btrfs/ioctl.py'...
Compiling 'btrfs/utils.py'...

.
├── btrfs
│   ├── crc32c.py
│   ├── ctree.py
│   ├── __init__.py
│   ├── ioctl.py
│   ├── __pycache__
│   │   ├── crc32c.cpython-35.pyc
│   │   ├── ctree.cpython-35.pyc
│   │   ├── __init__.cpython-35.pyc
│   │   ├── ioctl.cpython-35.pyc
│   │   └── utils.cpython-35.pyc
│   └── utils.py


-- 
Hans van Kranenburg

Re: Btrfs Heatmap - v4 ... colors!!

[Timofey Titovets]

2017-02-03 17:27 GMT+03:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>:
> On 02/03/2017 03:18 PM, Timofey Titovets wrote:
>> 2017-02-03 15:57 GMT+03:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>:
>>> On 02/03/2017 12:25 PM, Timofey Titovets wrote:
>>>> Thank you for your great work:
>>>> JFYI Packaged in AUR:
>>>> https://aur.archlinux.org/packages/python-btrfs-heatmap/
>>>
>>> Hey, thanks.
>>>
>>> Just wondering... what is that btrfs.py you refer to in...
>>> https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-btrfs ?
>>>
>>> It's a package, not a single file, so maybe that compile command doesn't
>>> do anything? I'm not familiar with arch, so correct me if I'm wrong.
>>>
>>> .
>>> ├── btrfs
>>> │   ├── crc32c.py
>>> │   ├── ctree.py
>>> │   ├── __init__.py
>>> │   ├── ioctl.py
>>> │   ├── utils.py
>>>
>>
>> You right, compile command are useless, so i did remove it, thanks.
>
> -$ python2 -m compileall btrfs
> Listing btrfs ...
> Compiling btrfs/__init__.py ...
> Compiling btrfs/crc32c.py ...
> Compiling btrfs/ctree.py ...
> Compiling btrfs/ioctl.py ...
> Compiling btrfs/utils.py ...
>
> .
> ├── btrfs
> │   ├── crc32c.py
> │   ├── crc32c.pyc
> │   ├── ctree.py
> │   ├── ctree.pyc
> │   ├── __init__.py
> │   ├── __init__.pyc
> │   ├── ioctl.py
> │   ├── ioctl.pyc
> │   ├── utils.py
> │   └── utils.pyc
>
>
> -$ python3 -m compileall btrfs
> Listing 'btrfs'...
> Compiling 'btrfs/__init__.py'...
> Compiling 'btrfs/crc32c.py'...
> Compiling 'btrfs/ctree.py'...
> Compiling 'btrfs/ioctl.py'...
> Compiling 'btrfs/utils.py'...
>
> .
> ├── btrfs
> │   ├── crc32c.py
> │   ├── ctree.py
> │   ├── __init__.py
> │   ├── ioctl.py
> │   ├── __pycache__
> │   │   ├── crc32c.cpython-35.pyc
> │   │   ├── ctree.cpython-35.pyc
> │   │   ├── __init__.cpython-35.pyc
> │   │   ├── ioctl.cpython-35.pyc
> │   │   └── utils.cpython-35.pyc
> │   └── utils.py
>
>
> --
> Hans van Kranenburg

➜  python-btrfs git:(master) ✗ tar -vxf python-btrfs-5-1-any.pkg.tar.xz
.PKGINFO
.BUILDINFO
.MTREE
usr/
usr/lib/
usr/lib/python3.6/
usr/lib/python3.6/site-packages/
usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/
usr/lib/python3.6/site-packages/btrfs/
usr/lib/python3.6/site-packages/btrfs/__pycache__/
usr/lib/python3.6/site-packages/btrfs/utils.py
usr/lib/python3.6/site-packages/btrfs/ioctl.py
usr/lib/python3.6/site-packages/btrfs/ctree.py
usr/lib/python3.6/site-packages/btrfs/crc32c.py
usr/lib/python3.6/site-packages/btrfs/__init__.py
usr/lib/python3.6/site-packages/btrfs/__pycache__/utils.cpython-36.pyc
usr/lib/python3.6/site-packages/btrfs/__pycache__/ioctl.cpython-36.pyc
usr/lib/python3.6/site-packages/btrfs/__pycache__/ctree.cpython-36.pyc
usr/lib/python3.6/site-packages/btrfs/__pycache__/crc32c.cpython-36.pyc
usr/lib/python3.6/site-packages/btrfs/__pycache__/__init__.cpython-36.pyc
usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/installed-files.txt
usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/PKG-INFO
usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/dependency_links.txt
usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/top_level.txt
usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/SOURCES.txt

I think it's building automaticaly by:
PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir"
--ignore-installed --no-deps btrfs
-- 
Have a nice day,
Timofey.

Re: Btrfs Heatmap - v4 ... colors!!

[Hans van Kranenburg]

On 02/03/2017 03:36 PM, Timofey Titovets wrote:
> 
> ➜  python-btrfs git:(master) ✗ tar -vxf python-btrfs-5-1-any.pkg.tar.xz
> .PKGINFO
> .BUILDINFO
> .MTREE
> usr/
> usr/lib/
> usr/lib/python3.6/
> usr/lib/python3.6/site-packages/
> usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/
> usr/lib/python3.6/site-packages/btrfs/
> usr/lib/python3.6/site-packages/btrfs/__pycache__/
> usr/lib/python3.6/site-packages/btrfs/utils.py
> usr/lib/python3.6/site-packages/btrfs/ioctl.py
> usr/lib/python3.6/site-packages/btrfs/ctree.py
> usr/lib/python3.6/site-packages/btrfs/crc32c.py
> usr/lib/python3.6/site-packages/btrfs/__init__.py
> usr/lib/python3.6/site-packages/btrfs/__pycache__/utils.cpython-36.pyc
> usr/lib/python3.6/site-packages/btrfs/__pycache__/ioctl.cpython-36.pyc
> usr/lib/python3.6/site-packages/btrfs/__pycache__/ctree.cpython-36.pyc
> usr/lib/python3.6/site-packages/btrfs/__pycache__/crc32c.cpython-36.pyc
> usr/lib/python3.6/site-packages/btrfs/__pycache__/__init__.cpython-36.pyc
> usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/installed-files.txt
> usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/PKG-INFO
> usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/dependency_links.txt
> usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/top_level.txt
> usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/SOURCES.txt
> 
> I think it's building automaticaly by:
> PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir"
> --ignore-installed --no-deps btrfs
> 

Even better :-)

Thanks,

-- 
Hans van Kranenburg