Re: [ext3][kernels >= 2.6.20.7 at least] KDE going comatose when FS is under heavy write load (massive starvation)

[Alex Tomas <alex@clusterfs.com>]

Andrew Morton wrote:
>> OTOH, the faster we go through data sync part of commit, the better. given
>> that lots of inodes can be dirty with no data to sync, it's going to take
>> long in some cases. it's especially bad because commit doesn't scale to many
>> CPUs.
> 
> eh?

I mean that number inodes to scan can be order of magnitude larger than number
of inodes needing sync for this given transaction. commit thread has to scan them
all (quite amount of CPU, i guess) and we can't use >1 CPU to speed the scan up.

>> also, why would we need to flush *everything* every 5s? just because ext3 does
>> this? sounds strange. if somebody really need this we could add this possibility
>> to regular writeback path (making it tunable). but I'd rather prefer to have
>> a separate (fast, lightweight, scalable) mechanism to support data=ordered.
>>
> 
> Yeah, that would make sense, perhaps.
> 
> Or just speed the existing stuff up.  iirc the main problem in there is unrelated
> to data writeback.  There are situations where the running transaction has to block
> behind metadata writeout which the committing transaction is performing.  I
> reluctantly put that in years ago to get us out of a tight spot and it
> never got optimised.

AFAIU, existing writeback is built around dirty inodes list and dirty bit in
per-inode radix tree. in order to avoid scanning too much (see before) we'd
need a separate list and probably one more bit in radix tree. plus some code
to allow writeback to use new list/tag.

as for the main problem ... I'd very appreciate any details. probably it was
about several blocks in page when one block is allocated in transaction 1 and
next block is being allocated in transaction 2?

thanks, Alex