Re: ext3 throughput woes on certain (possibly heavily fragmented) files

[Hans Reiser <reiser@namesys.com>]

Nikita Danilov wrote:

>Stephen C. Tweedie writes:
> > Hi,
> > 
> > On Tue, Sep 03, 2002 at 02:24:19AM -0700, Aaron Lehmann wrote:
> > 
> > > [aaronl@vitelus:~]$ time cat mail/debian-legal > /dev/null
> > > cat mail/debian-legal > /dev/null  0.00s user 0.02s system 0% cpu 5.565 total
> > > [aaronl@vitelus:~]$ ls -l mail/debian-legal
> > > -rw-------    1 aaronl   mail      7893525 Sep  3 00:42 mail/debian-legal
> > > [aaronl@vitelus:~]$ time cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null
> > > cat /usr/src/linux-2.4.18.tar.bz2 > /dev/null  0.00s user 0.10s system 16% cpu 0.616 total
> > > [aaronl@vitelus:~]$ ls -l /usr/src/linux-2.4.18.tar.bz2 
> > > -rw-r--r--    1 aaronl   aaronl   24161675 Apr 14 11:53
> > > 
> > > Both files were AFAIK not in any cache, and they are on the same
> > > partition.
> > > 
> > > My current uninformed theory is that this is caused by fragmentation,
> > > since the linux tarball was downloaded all at once but the mailbox I'm
> > > comparing it to has 1695 messages, each of which having been appended
> > > seperately to the file. All of my mailboxes exhibit similarly awful
> > > performance.
> > 
> > Yep, both ext2 and ext3 can get badly fragmented by files which are
> > closed, reopened and appended to frequently like that.
> > 
> > > Do any other filesystems handle this type of thing more gracefully?
> > 
> > There are some ideas from recent FFS changes.  One thing they now do
> > is to defragment things automatically as a file grows by effectively
> > deleting and then reallocating the last 16 blocks of the file.
> > Fragmentation will still occur, but less so, if we do that.
> > 
>
>Another possible solution is to try to "defer" allocation. For example,
>in reiser4 (and XFS, I believe) extents are allocated on the transaction
>commit and as a result, if file was created by several writes, it will
>still be allocated as one extent.
>
> > 
> > Cheers,
> >  Stephen
>
>Nikita.
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>
The FFS approach has an advantage for the case where the file grows too 
slowly for allocation to be delayed.

I think I prefer that we implement a repacker for reiser4 though, as 
that, combined with delayed allocation, will be a balanced and thorough 
solution.

Hans