Re: Kernel 2.6.8.1: swap storm of death - nr_requests > 1024 on swap partition

Andrew Morton
2004-08-28T21:43:03Z

(Added linux-mm)

Karl Vogel <karl.vogel@pandora.be> wrote:
>
> Andrew Morton wrote:
> > Karl Vogel <karl.vogel@seagha.com> wrote:
> >
> >>Further testing shows that all the schedulers exhibit this exact same
> >> problem when run with a nr_requests size of 8192 on the drive hosting
> >> the swap partition.
> >>
> >> I tried noop, deadline, as and CFQ with:
> >>
> >> echo 8192 >/sys/block/hda/queue/nr_requests
> >
> >
> > That allows up to 2GB of memory to be under writeout at the same time. The
> > VM cannot touch any of that memory.
>
> Well I used that value as it is the default for CFQ.. and it was with
> CFQ that I had the problems. The patch Jens offered to track down the
> problem, commented out this 'q->nr_requests = 8192' in CFQ and it
> helped. Therefor I tried the other schedulers with this value to see if
> it made a difference.
>
> So if I understand you correctly, CFQ shouldn't be using 8192 on 512Mb
> systems?!

Yup. It's asking for trouble to allow that much memory to be unreclaimably
pinned.

Of course, you could have the same problem with just 128 requests per
queue, and lots of queues. I solved all these problems in the dirty memory
writeback paths. But I forgot about swapout!

> With overcommit_memory set to 1, the program can be run again after the
> OOM kill.. but the OOM killing remains.
>
> With overcommit_memory set to 0 a second run fails. I 'think' it's
> because somehow SwapCache is 500Kb after the OOM, so in effect my system
> doesn't have 1Gb to spare anymore. Doing swapoff/swapon frees this and
> then I can do the calloc(1Gb) again.
>
> Another way to free the SwapCached is to generate lots of I/O doing 'dd
> if=/dev/hda of=/dev/null' ... after a while SwapCached is < 1Mb again.
>

urgh. It sounds like the overcommit logic forgot to account swapcache as
reclaimable. It's been a ton of trouble, that code.