Re: Contributing to NILFS

[Vyacheslav Dubeyko <slava-yeENwD64cLxBDgjK7y7TUQ@public.gmane.org>]

Hi Andreas,

On Tue, 2012-12-11 at 14:54 +0100, Andreas Rohner wrote:
> Hi Vyacheslav,
> 
> Thanks for your response.
> 
> > > 2. Is there some fundamental difficulty that makes it hard to implement
> > > for a log-structured fs?
> > 
> > I think that the most fundamental possible issue can be a possible
> > performance degradation. But first of all, from my point of view, it
> > needs to discuss what the online defrag is and how it is possible to
> > implement it. What do you mean personally by online defrag? And how do
> > you imagine online defrag mechanism for NILFS2 in particular? When you
> > describe your understanding then it will be possible to discuss about
> > difficulties, I think. :-)
> 
> One way would be to just write out heavily fragmented files sequentially
> and atomically switch to the new blocks. But as you suggested this
> simple approach would probably result in performance degradation,
> because it would eat up free segments and the segments of the old blocks
> would contain more unusable free space, that has to be cleaned first.
> This could result in an undesirable situation where most of the segments
> are 60% full and for every clean segment the cleaner has to read in 4
> half full segments. I think the difficult part is to find a suitable
> heuristic to decide if it is beneficial to defragment a file or not. My
> aim would be to produce as many clean or nearly clean segments as
> possible in the process. I would try to implement and test different
> heuristics and algorithms with differently aged file systems and compare
> the results.
> 

I think that this task hides many difficult questions. How does it
define what files fragmented or not? How does it measure the
fragmentation degree? What fragmentation degree should be a basis for
defragmentation activity? When does it need to detect fragmentation and
how to keep this knowledge? How does it make defragmentation without
performance degradation?

As I understand, when we are talking about defragmentation then we
expect a performance enhancement as a result. But defragmenter activity
can be a background reason of performance degradation. Not every
workload or I/O pattern can be a reason of significant fragmentation.

Also, it is a very important to choose a point of defragmentation. I
mean that it is possible to try to prevent fragmentation or to correct
fragmentation after flushing on the volume. It is possible to have a
some hybrid technique, I think. An I/O pattern or file type can be a
basis for such decision, I think.

As I understand, F2FS [1] has some defragmenting approaches. I think
that it needs to discuss more deeply about technique of detecting
fragmented files and fragmentation degree. But maybe hot data tracking
patch [2,3] will be a basis for such discussion.

I think that it can be a useful some materials about NILFS2. I began a
design document for NILFS2 [4] but unfortunately it is not ended yet. It
was published a review of NILFS2 [5] not so recently.

It exists some defragmentation-related papers but I haven't
comprehensive list. I can mention about "The Effects of Filesystem
Fragmentation" [6]. Maybe it can be useful "A Five-Year Study of
File-System Metadata" [7] and "A File Is Not a File: Understanding the
I/O Behavior of Apple Desktop Applications" [8] papers.

So, I feel necessity to think more deeply about online defragment task
and about what you said. But, anyway, it is a beginning of
discussion. :-) 

[1] http://lwn.net/Articles/518988/
[2] http://lwn.net/Articles/525425/
[3] http://lwn.net/Articles/400029/
[4] http://dubeyko.com/development/FileSystems/NILFS/nilfs2-design.pdf
[5] http://lwn.net/Articles/522507/
[6] http://www.google.ru/url?sa=t&rct=j&q=the%20effects%20of%20filesystem%20fragmentation&source=web&cd=2&ved=0CD0QFjAB&url=http%3A%2F%2Fwww.kernel.org%2Fdoc%2Fols%2F2006%2Fols2006v1-pages-193-208.pdf&ei=6CnIUJeHHYqB4gS6l4GoCQ&usg=AFQjCNFLhxtq89VLzE_fLuX7CDDpk_1Krw&bvm=bv.1355272958,d.bGE&cad=rjt
[7] http://www.google.ru/url?sa=t&rct=j&q=a%20five-year%20study%20of%20file-system%20metadata&source=web&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Fresearch.microsoft.com%2Fpubs%2F72896%2Ffast07-final.pdf&ei=syvIULepFoWE4ASDmYDIBA&usg=AFQjCNE5mFDPqgEvGYe32RkNyVa2oxVxkw&bvm=bv.1355272958,d.bGE&cad=rjt
[8] http://www.google.ru/url?sa=t&rct=j&q=a%20file%20is%20not%20a%20file%3A%20understanding%20the%20i%2Fo%20behavior%20of%20apple%20desktop%20applications&source=web&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Fresearch.cs.wisc.edu%2Fwind%2FPublications%2Fibench-1c-sosp11.pdf&ei=NSzIUP3zCKqK4ASU5oCACw&usg=AFQjCNEFr-bw1Ke382_rQBYGQwI88MPkKg&bvm=bv.1355272958,d.bGE&cad=rjt

With the best regards,
Vyacheslav Dubeyko.


--
To unsubscribe from this list: send the line "unsubscribe linux-nilfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html