Re: 2.4.17 SMP hangs ..

[Andrew Morton <akpm@digeo.com>]

Theodore Ts'o wrote:
> 
> On Wed, Nov 20, 2002 at 10:03:50PM -0800, Andrew Morton wrote:
> > Manish Lachwani wrote:
> > >
> > > I am seeing system hangs with 2.4.17 SMP kernel when doing mke2fs accros 12
> > > drives in parallel. However, the hangs only occur when the I/O rate from
> > > vmstat is high:
> > >
> >
> > Quite possibly it has not hung.  You just need to wait half an
> > hour or so.
> >
> > The algorithm isn't very good.
> 
> [Catching up lkml mail after the IETF meeting....]
> 
> Try setting the environment variable "MKE2FS_SYNC" to a value such as
> 10.  This will cause mke2fs to force a sync after writing out every 10
> block groups worth of inode tables.

That will fix it.

> If this fixes the problem, then it means that the kernel isn't
> handling write throttling correctly, and the system is thrashing
> itself to death.

Nah, it's __block_fsync().  That function has to write buffers against
a particular device.  So it searches the global buffer LRU for 32 buffers
against the nominated device, drops the lock, writes the buffers, then
searches again.

So the search complexity is O(n*n/32).  Which means that when you have a
lot of dirty buffers against different devices on the queue the CPU cost
simply explodes.

One workaround is to use sync rather than fsync - because sync uses NODEV
and doesn't have to search past buffers from uninteresting devices. 
Another is to do what you've done.

Probably, just syncing the buffers at i_dirty_data_buffers would suffice.
That would fix it.