exploit preload daemon to defragment most needed files

exploit preload daemon to defragment most needed files

[Kai Krakow]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

Hey list,

Although this is about the preload daemon, my intended audience for this 
matter is the btrfs community. So I'm posting this here.

I've created a small script here[1] to read the preload daemon state file 
and use this to run the btrfs defragmenter/compressor on these files. The 
idea was to have some proof-of-concept to maybe integrate defragmentation 
capabilities directly into preload.

While I'd like to read some feedback from the community, maybe collect some 
more ideas, I already discovered one drawback which is a show-stopper: btrfs 
won't defragment files currently in use (probably memory-mapped files only 
but these are a lot while a desktop env is running). I didn't look into it 
any closer yet. And I didn't run benchmarks. But my first impressions are 
that preloading is a bit faster and less io-intensive.

Regards,
Kai

[1]: https://gist.github.com/kakra/5520200

[-- Attachment #2: defrag-preloaded.rb --]
[-- Type: application/x-ruby, Size: 600 bytes --]

Re: exploit preload daemon to defragment most needed files

[Tomasz Torcz]

On Sun, May 05, 2013 at 11:42:24AM +0200, Kai Krakow wrote:
> Hey list,
> 
> Although this is about the preload daemon, my intended audience for this 
> matter is the btrfs community. So I'm posting this here.
> 
> I've created a small script here[1] to read the preload daemon state file 
> and use this to run the btrfs defragmenter/compressor on these files. The 
> idea was to have some proof-of-concept to maybe integrate defragmentation 
> capabilities directly into preload.

  In similar spirit, systemd's readhead defrag files on btrfs. Here's the commit:
http://cgit.freedesktop.org/systemd/systemd/commit/?id=746f89063666be8c4d538271c646585ac4fc4353


-- 
Tomasz Torcz               "Never underestimate the bandwidth of a station
xmpp: zdzichubg@chrome.pl    wagon filled with backup tapes." -- Jim Gray

Re: exploit preload daemon to defragment most needed files

[Kai Krakow]

Tomasz Torcz <tomek@pipebreaker.pl> schrieb:

>> Although this is about the preload daemon, my intended audience for this
>> matter is the btrfs community. So I'm posting this here.
>> 
>> I've created a small script here[1] to read the preload daemon state file
>> and use this to run the btrfs defragmenter/compressor on these files. The
>> idea was to have some proof-of-concept to maybe integrate defragmentation
>> capabilities directly into preload.
> 
>   In similar spirit, systemd's readhead defrag files on btrfs. Here's the
>   commit:

Yes! :-) Actually that is where I had my idea from. But preload takes on 
(boot finished, preload user's enviroment) where readahead stops (readahead 
files needed by daemons, until about 10s after last daemon loaded) working 
with my files.

I think it is a great idea doing this kind of stuff in preloaders and 
"readaheaders".

If I understood it right, systemd readahead does not only just defrag the 
files, it relocates them into the order needed during boot. Actually, that 
seems pretty effective, as my system (no SSD) finishes booting all daemons 
within less than 10s (on mraid1/draid0 3 device btrfs, plain old harddisks, 
no SSD). Just lightdm/kdm then take ages to load. Still need to figure out 
why but that's another story.

Regards,
Kai