* Btrfs Heatmap - v4 ... colors!!
@ 2017-01-14 19:53 Hans van Kranenburg
[not found] ` <20170118003838.GA4136@comcast.net>
0 siblings, 1 reply; 8+ messages in thread
From: Hans van Kranenburg @ 2017-01-14 19:53 UTC (permalink / raw)
To: linux-btrfs
Hi,
I just tagged v4 of the Btrfs Heatmap utility, which visualizes the
usage of your btrfs filesystem:
https://github.com/knorrie/btrfs-heatmap
It needs at least python-btrfs v5 to run, because of the fixed backref
handling.
The main changes are:
* Colors! colors! colors!!!
* Colors!
* Improved documentation, thanks to user feedback. The first README
page does focus on quick satisfaction, while there are two extra
subpages about the extent-level details pictures and a scripting HOWTO
which explains how to do everything that the command line options cannot do.
* The external 2k8 loc png library dependency is dropped, in favour of
a 34 line single purpose png writer implementation directly in the
script itself. This makes it even easier to get the thing going quickly,
and it makes the thing go more quickly as well.
And now, no more text to read. Go and create a bunch of beautiful
pictures and timelapses of your filesystem!
Have fun!
P.S. Here's where it all started... :D
http://logs.tvrrug.org.uk/logs/%23btrfs/2015-12-09.html#2015-12-09T20:54:04
--
Hans van Kranenburg
^ permalink raw reply [flat|nested] 8+ messages in thread[parent not found: <20170118003838.GA4136@comcast.net>]
* Re: Btrfs Heatmap - v4 ... colors!! [not found] ` <20170118003838.GA4136@comcast.net> @ 2017-01-19 0:08 ` Hans van Kranenburg 2017-02-03 11:25 ` Timofey Titovets 0 siblings, 1 reply; 8+ messages in thread From: Hans van Kranenburg @ 2017-01-19 0:08 UTC (permalink / raw) To: Tim Walberg; +Cc: linux-btrfs [-- Attachment #1: Type: text/plain, Size: 3212 bytes --] Hi Tim, On 01/18/2017 01:38 AM, Tim Walberg wrote: > I really like the new release. It's definitely nice to have the added colors. Yay. > Just one major request I would have - bring the linear option back. The > hilbert output is cool for a single device, and is ok for a two-device file > system, but it's not so easy to interpret especially when you have an odd > number of devices in your file system - I have a 3-device RAID-1 config. Ok, so because the curve continues its path and it may switch to a new device somewhere, you don't see clearly where the new device starts (there are also no extra helping lines / device borders). > An > alternative would be the ability to create separate maps for each device in > a file system, either by selecting a devid via command line, or by creating > individual images arranged in an array or something. That was one thing I > could do with the original version that had a separate script for dumping > the block groups and another for creating the PNG - I could filter the block > group list by devid to do individual images. Have a look at the scripting howto, linked from the main README somewhere at the bottom of the page: https://github.com/knorrie/btrfs-heatmap/blob/master/doc/scripting.md One of the examples, 2.3 [..] separate image per device [..] is an example of how to do this. It involves a little bit of scripting, but gives total flexibility of whatever you want to do. fs = btrfs.FileSystem('/foo') for device in fs.devices(): grid = heatmap.walk_dev_extents(fs, [device]) grid.write_png('device_%s.png' % device.devid) The first half of the page explains the params you can give to heatmap.walk_dev_extents, like also order and size etc. Stitching the resulting images together in whatever form is up to you, using your favourite image processing tools afterwards. > Thanks for all the work that's gone into this so far! Thanks! Check out (wip) branch linear: https://github.com/knorrie/btrfs-heatmap/commits/linear There's a commit which puts linear and snake mode in, besides the default hilbert (with --curve linear etc, like there was before). It's just taking another route walking the grid. :) The biggest problem with linear / snake is that it destroys the idea of locality (long lines instead of little blocks of data together), and when changing orders, it makes the same real point move around instead of staying at the same spot. See https://www.youtube.com/watch?v=DuiryHHTrjU&t=5m18s (->7m8s) for a visual explanation. Compare attached: - hilbert-order-8-size-9.png (bytes_per_pixel 41943040) - linear-order-8-size-9.png (bytes_per_pixel 41943040) To get a bit of a usable result, I need to lower the resolution of the image drastically, resulting in... - linear-order-6-size-9.png (bytes_per_pixel 671088640) ... with 640MiB crammed into a pixel instead of only 40MiB The result is that usage gets blurred out too much, because block group boundaries are too different from pixel boundaries. But, being able to play around with it is fun and serves a purpose, so I'll put this back in, and write a nice subpage of the README with some pictures of this kind to explain. o/ -- Hans van Kranenburg [-- Attachment #2: hilbert-order-8-size-9.png --] [-- Type: image/png, Size: 21666 bytes --] [-- Attachment #3: linear-order-8-size-9.png --] [-- Type: image/png, Size: 11647 bytes --] [-- Attachment #4: linear-order-6-size-9.png --] [-- Type: image/png, Size: 9297 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Btrfs Heatmap - v4 ... colors!! 2017-01-19 0:08 ` Hans van Kranenburg @ 2017-02-03 11:25 ` Timofey Titovets 2017-02-03 12:57 ` Hans van Kranenburg 0 siblings, 1 reply; 8+ messages in thread From: Timofey Titovets @ 2017-02-03 11:25 UTC (permalink / raw) To: Hans van Kranenburg; +Cc: Tim Walberg, linux-btrfs Thank you for your great work: JFYI Packaged in AUR: https://aur.archlinux.org/packages/python-btrfs-heatmap/ -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Btrfs Heatmap - v4 ... colors!! 2017-02-03 11:25 ` Timofey Titovets @ 2017-02-03 12:57 ` Hans van Kranenburg 2017-02-03 14:18 ` Timofey Titovets 0 siblings, 1 reply; 8+ messages in thread From: Hans van Kranenburg @ 2017-02-03 12:57 UTC (permalink / raw) To: Timofey Titovets; +Cc: Tim Walberg, linux-btrfs On 02/03/2017 12:25 PM, Timofey Titovets wrote: > Thank you for your great work: > JFYI Packaged in AUR: > https://aur.archlinux.org/packages/python-btrfs-heatmap/ Hey, thanks. Just wondering... what is that btrfs.py you refer to in... https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-btrfs ? It's a package, not a single file, so maybe that compile command doesn't do anything? I'm not familiar with arch, so correct me if I'm wrong. . ├── btrfs │ ├── crc32c.py │ ├── ctree.py │ ├── __init__.py │ ├── ioctl.py │ ├── utils.py -- Hans van Kranenburg ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Btrfs Heatmap - v4 ... colors!! 2017-02-03 12:57 ` Hans van Kranenburg @ 2017-02-03 14:18 ` Timofey Titovets 2017-02-03 14:27 ` Hans van Kranenburg 0 siblings, 1 reply; 8+ messages in thread From: Timofey Titovets @ 2017-02-03 14:18 UTC (permalink / raw) To: Hans van Kranenburg; +Cc: Tim Walberg, linux-btrfs 2017-02-03 15:57 GMT+03:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>: > On 02/03/2017 12:25 PM, Timofey Titovets wrote: >> Thank you for your great work: >> JFYI Packaged in AUR: >> https://aur.archlinux.org/packages/python-btrfs-heatmap/ > > Hey, thanks. > > Just wondering... what is that btrfs.py you refer to in... > https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-btrfs ? > > It's a package, not a single file, so maybe that compile command doesn't > do anything? I'm not familiar with arch, so correct me if I'm wrong. > > . > ├── btrfs > │ ├── crc32c.py > │ ├── ctree.py > │ ├── __init__.py > │ ├── ioctl.py > │ ├── utils.py > > -- > Hans van Kranenburg You right, compile command are useless, so i did remove it, thanks. -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Btrfs Heatmap - v4 ... colors!! 2017-02-03 14:18 ` Timofey Titovets @ 2017-02-03 14:27 ` Hans van Kranenburg 2017-02-03 14:36 ` Timofey Titovets 0 siblings, 1 reply; 8+ messages in thread From: Hans van Kranenburg @ 2017-02-03 14:27 UTC (permalink / raw) To: Timofey Titovets; +Cc: Tim Walberg, linux-btrfs On 02/03/2017 03:18 PM, Timofey Titovets wrote: > 2017-02-03 15:57 GMT+03:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>: >> On 02/03/2017 12:25 PM, Timofey Titovets wrote: >>> Thank you for your great work: >>> JFYI Packaged in AUR: >>> https://aur.archlinux.org/packages/python-btrfs-heatmap/ >> >> Hey, thanks. >> >> Just wondering... what is that btrfs.py you refer to in... >> https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-btrfs ? >> >> It's a package, not a single file, so maybe that compile command doesn't >> do anything? I'm not familiar with arch, so correct me if I'm wrong. >> >> . >> ├── btrfs >> │ ├── crc32c.py >> │ ├── ctree.py >> │ ├── __init__.py >> │ ├── ioctl.py >> │ ├── utils.py >> > > You right, compile command are useless, so i did remove it, thanks. -$ python2 -m compileall btrfs Listing btrfs ... Compiling btrfs/__init__.py ... Compiling btrfs/crc32c.py ... Compiling btrfs/ctree.py ... Compiling btrfs/ioctl.py ... Compiling btrfs/utils.py ... . ├── btrfs │ ├── crc32c.py │ ├── crc32c.pyc │ ├── ctree.py │ ├── ctree.pyc │ ├── __init__.py │ ├── __init__.pyc │ ├── ioctl.py │ ├── ioctl.pyc │ ├── utils.py │ └── utils.pyc -$ python3 -m compileall btrfs Listing 'btrfs'... Compiling 'btrfs/__init__.py'... Compiling 'btrfs/crc32c.py'... Compiling 'btrfs/ctree.py'... Compiling 'btrfs/ioctl.py'... Compiling 'btrfs/utils.py'... . ├── btrfs │ ├── crc32c.py │ ├── ctree.py │ ├── __init__.py │ ├── ioctl.py │ ├── __pycache__ │ │ ├── crc32c.cpython-35.pyc │ │ ├── ctree.cpython-35.pyc │ │ ├── __init__.cpython-35.pyc │ │ ├── ioctl.cpython-35.pyc │ │ └── utils.cpython-35.pyc │ └── utils.py -- Hans van Kranenburg ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Btrfs Heatmap - v4 ... colors!! 2017-02-03 14:27 ` Hans van Kranenburg @ 2017-02-03 14:36 ` Timofey Titovets 2017-02-03 14:46 ` Hans van Kranenburg 0 siblings, 1 reply; 8+ messages in thread From: Timofey Titovets @ 2017-02-03 14:36 UTC (permalink / raw) To: Hans van Kranenburg; +Cc: Tim Walberg, linux-btrfs 2017-02-03 17:27 GMT+03:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>: > On 02/03/2017 03:18 PM, Timofey Titovets wrote: >> 2017-02-03 15:57 GMT+03:00 Hans van Kranenburg <hans.van.kranenburg@mendix.com>: >>> On 02/03/2017 12:25 PM, Timofey Titovets wrote: >>>> Thank you for your great work: >>>> JFYI Packaged in AUR: >>>> https://aur.archlinux.org/packages/python-btrfs-heatmap/ >>> >>> Hey, thanks. >>> >>> Just wondering... what is that btrfs.py you refer to in... >>> https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-btrfs ? >>> >>> It's a package, not a single file, so maybe that compile command doesn't >>> do anything? I'm not familiar with arch, so correct me if I'm wrong. >>> >>> . >>> ├── btrfs >>> │ ├── crc32c.py >>> │ ├── ctree.py >>> │ ├── __init__.py >>> │ ├── ioctl.py >>> │ ├── utils.py >>> >> >> You right, compile command are useless, so i did remove it, thanks. > > -$ python2 -m compileall btrfs > Listing btrfs ... > Compiling btrfs/__init__.py ... > Compiling btrfs/crc32c.py ... > Compiling btrfs/ctree.py ... > Compiling btrfs/ioctl.py ... > Compiling btrfs/utils.py ... > > . > ├── btrfs > │ ├── crc32c.py > │ ├── crc32c.pyc > │ ├── ctree.py > │ ├── ctree.pyc > │ ├── __init__.py > │ ├── __init__.pyc > │ ├── ioctl.py > │ ├── ioctl.pyc > │ ├── utils.py > │ └── utils.pyc > > > -$ python3 -m compileall btrfs > Listing 'btrfs'... > Compiling 'btrfs/__init__.py'... > Compiling 'btrfs/crc32c.py'... > Compiling 'btrfs/ctree.py'... > Compiling 'btrfs/ioctl.py'... > Compiling 'btrfs/utils.py'... > > . > ├── btrfs > │ ├── crc32c.py > │ ├── ctree.py > │ ├── __init__.py > │ ├── ioctl.py > │ ├── __pycache__ > │ │ ├── crc32c.cpython-35.pyc > │ │ ├── ctree.cpython-35.pyc > │ │ ├── __init__.cpython-35.pyc > │ │ ├── ioctl.cpython-35.pyc > │ │ └── utils.cpython-35.pyc > │ └── utils.py > > > -- > Hans van Kranenburg ➜ python-btrfs git:(master) ✗ tar -vxf python-btrfs-5-1-any.pkg.tar.xz .PKGINFO .BUILDINFO .MTREE usr/ usr/lib/ usr/lib/python3.6/ usr/lib/python3.6/site-packages/ usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/ usr/lib/python3.6/site-packages/btrfs/ usr/lib/python3.6/site-packages/btrfs/__pycache__/ usr/lib/python3.6/site-packages/btrfs/utils.py usr/lib/python3.6/site-packages/btrfs/ioctl.py usr/lib/python3.6/site-packages/btrfs/ctree.py usr/lib/python3.6/site-packages/btrfs/crc32c.py usr/lib/python3.6/site-packages/btrfs/__init__.py usr/lib/python3.6/site-packages/btrfs/__pycache__/utils.cpython-36.pyc usr/lib/python3.6/site-packages/btrfs/__pycache__/ioctl.cpython-36.pyc usr/lib/python3.6/site-packages/btrfs/__pycache__/ctree.cpython-36.pyc usr/lib/python3.6/site-packages/btrfs/__pycache__/crc32c.cpython-36.pyc usr/lib/python3.6/site-packages/btrfs/__pycache__/__init__.cpython-36.pyc usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/installed-files.txt usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/PKG-INFO usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/dependency_links.txt usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/top_level.txt usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/SOURCES.txt I think it's building automaticaly by: PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir" --ignore-installed --no-deps btrfs -- Have a nice day, Timofey. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Btrfs Heatmap - v4 ... colors!! 2017-02-03 14:36 ` Timofey Titovets @ 2017-02-03 14:46 ` Hans van Kranenburg 0 siblings, 0 replies; 8+ messages in thread From: Hans van Kranenburg @ 2017-02-03 14:46 UTC (permalink / raw) To: Timofey Titovets; +Cc: Tim Walberg, linux-btrfs On 02/03/2017 03:36 PM, Timofey Titovets wrote: > > ➜ python-btrfs git:(master) ✗ tar -vxf python-btrfs-5-1-any.pkg.tar.xz > .PKGINFO > .BUILDINFO > .MTREE > usr/ > usr/lib/ > usr/lib/python3.6/ > usr/lib/python3.6/site-packages/ > usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/ > usr/lib/python3.6/site-packages/btrfs/ > usr/lib/python3.6/site-packages/btrfs/__pycache__/ > usr/lib/python3.6/site-packages/btrfs/utils.py > usr/lib/python3.6/site-packages/btrfs/ioctl.py > usr/lib/python3.6/site-packages/btrfs/ctree.py > usr/lib/python3.6/site-packages/btrfs/crc32c.py > usr/lib/python3.6/site-packages/btrfs/__init__.py > usr/lib/python3.6/site-packages/btrfs/__pycache__/utils.cpython-36.pyc > usr/lib/python3.6/site-packages/btrfs/__pycache__/ioctl.cpython-36.pyc > usr/lib/python3.6/site-packages/btrfs/__pycache__/ctree.cpython-36.pyc > usr/lib/python3.6/site-packages/btrfs/__pycache__/crc32c.cpython-36.pyc > usr/lib/python3.6/site-packages/btrfs/__pycache__/__init__.cpython-36.pyc > usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/installed-files.txt > usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/PKG-INFO > usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/dependency_links.txt > usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/top_level.txt > usr/lib/python3.6/site-packages/btrfs-5-py3.6.egg-info/SOURCES.txt > > I think it's building automaticaly by: > PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir" > --ignore-installed --no-deps btrfs > Even better :-) Thanks, -- Hans van Kranenburg ^ permalink raw reply [flat|nested] 8+ messages in thread
![[-- Attachment #2: hilbert-order-8-size-9.png --] [-- Type: image/png, Size: 21666 bytes --]](../../c8f2ebd7-9073-c2e7-5966-5207895c59b7@mendix.com/2-hilbert-order-8-size-9.png){kind=link}
![[-- Attachment #3: linear-order-8-size-9.png --] [-- Type: image/png, Size: 11647 bytes --]](../../c8f2ebd7-9073-c2e7-5966-5207895c59b7@mendix.com/3-linear-order-8-size-9.png){kind=link}
![[-- Attachment #4: linear-order-6-size-9.png --] [-- Type: image/png, Size: 9297 bytes --]](../../c8f2ebd7-9073-c2e7-5966-5207895c59b7@mendix.com/4-linear-order-6-size-9.png){kind=link}
end of thread, other threads:[~2017-02-03 14:46 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-01-14 19:53 Btrfs Heatmap - v4 ... colors!! Hans van Kranenburg
[not found] ` <20170118003838.GA4136@comcast.net>
2017-01-19 0:08 ` Hans van Kranenburg
2017-02-03 11:25 ` Timofey Titovets
2017-02-03 12:57 ` Hans van Kranenburg
2017-02-03 14:18 ` Timofey Titovets
2017-02-03 14:27 ` Hans van Kranenburg
2017-02-03 14:36 ` Timofey Titovets
2017-02-03 14:46 ` Hans van Kranenburg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).