I/O very slow under 2.4 (device reading)

I/O very slow under 2.4 (device reading)

[Lehmann]

It might be vm related, it might be not, but I get very funny effects when
running:

   buffer -S1m -s128k -m32m </dev/hde >/dev/null

(buffer is just a fast read/write buffer reading stdin to stdout, i use it
to check wether all sectors of a disk are readable, it's similar to dd,
which creates the same effects).

The problem is that under 2.4.4, 2.4.5 and 2.4.8pre4, the machine first
reads at about 30mb/s. then, after a few minutes, it stops for small
amounts of time, and, even worse, the system becomes totally sluggish,
even unusable for parts of seconds or even a second (mouse doesn't move,
programs need ages to starte etc...)

Then, after quite some time this behaviour stops, and the command reads
only very very slowly:

   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 1  0  0      0   3168 327504  51964   0   0   256     0  123   704   0  46  54
 2  0  0      0   3168 327504  51964   0   0   256     0  333  1167   0  51  49
 1  0  0      0   3168 327504  51964   0   0   256     6  141   740   0  50  50
 2  0  0      0   3172 327500  51964   0   0   256     0  110   661   0  50  50
 1  0  0      0   3128 327548  51960   0   0  8512    44  470  1595   0  45  55
 1  0  0      0   3128 327544  51960   0   0   256     2  394  2071   3  51  46
 1  0  0      0   3124 327544  51960   0   0   320     0  164   798   0  50  50
 1  0  0      0   3124 327540  51964   0   0   256     8  121   743   0  50  50
 1  0  0      0   3124 327540  51964   0   0  1280     0  176   826   0  50  50
 1  0  1      0   3120 327544  51964   0   0   320    64  201   858   0  50  50

as you can see, it reads about 256k/s only, but requires 100% cpu (it's a
dual cpu system and idle == 50 means one cpu is tied up):

   CPU1 states:  0.0% user, 100.0% system,  0.0% nice,  0.0% idle

Tied up in the kernel, btw. the rest of the system works and is fast.
hdparm reports nothing spectacular:

   /dev/hde:
   multcount    = 16 (on)
   I/O support  =  1 (32-bit)
   unmaskirq    =  1 (on)
   using_dma    =  1 (on)
   keepsettings =  1 (on)
   nowerr       =  0 (off)
   readonly     =  0 (off)
   readahead    =  8 (on)
   geometry     = 53614/16/63, sectors = 120103200, start = 0

and indeed the disk is still fast:

   cerebro:~# hdparm -tT /dev/hde

   /dev/hde:
    Timing buffer-cache reads:   128 MB in  0.76 seconds =168.42 MB/sec
    Timing buffered disk reads:  64 MB in  1.78 seconds = 35.96 MB/sec

this is, btw, WHILE the above buffer command was still running.

any ideas how to proceed with this problem?

-- 
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |

Re: I/O very slow under 2.4 (device reading)

[Linus Torvalds]

In article <20010808193158.A4055@cerebro.laendle>,
 <pcg@goof.com ( Marc) (A.) (Lehmann )> wrote:
>It might be vm related, it might be not, but I get very funny effects when
>running:
>
>   buffer -S1m -s128k -m32m </dev/hde >/dev/null
>
>(buffer is just a fast read/write buffer reading stdin to stdout, i use it
>to check wether all sectors of a disk are readable, it's similar to dd,
>which creates the same effects).

I bet it's the same thing that made writes slow - the buffer allocator
and the VM disagree about how/when to allocate memory. It's fixed by the patch
that's floating around in the "VM suckage" thread along with the tweak
to fix "zone_free_plenty()".

I'll make a real pre-patch (2.4.8-pre7) with the full changeset, can you
test that out? I did an equivalent "dd" that you see problems with, and
on my current kernel as long as there isn't other IO activity it stays
at a nice stable 21MB/s which is all my disk can deliver. 

[ Damn, maybe I should get one of those nice big 7200 rpm IBM drives ]

		Linus

Re: I/O very slow under 2.4 (device reading)

[Lehmann]

On Wed, Aug 08, 2001 at 11:12:45AM -0700, Linus Torvalds <torvalds@transmeta.com> wrote:
> and the VM disagree about how/when to allocate memory. It's fixed by the patch
> I'll make a real pre-patch (2.4.8-pre7) with the full changeset, can you

after 10 gigs of reading:

   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  1  0      0   6828 372696  36340   0   0 48292     6  726  2188   0  25  75
 0  1  0      0   6880 370536  39352   0   0 33792     0  726  3725   1  37  62
 1  0  1      0   6880 370160  36340   0   0 25652    48  411  3374   1  82  17
 0  1  0      0   6628 370780  36288   0   0 44788     0  690  2071   1  31  68
 1  0  1      0   6832 372972  36264   0   0 41988     0  674  2933   0  42  57

the system still feels a bit sluggishly, but otherwise NO SIGN of that
problem (no io slowdown, no short freezes). just the usual "disks are
in-use" contentions.

cool ,)

> at a nice stable 21MB/s which is all my disk can deliver. 

you _should_ know that raw speed doesn't mean too much.

> [ Damn, maybe I should get one of those nice big 7200 rpm IBM drives ]

maybe noisier. and for some reason (I swear for ibm drives usually), they
keep getting uncorrectable media errors on all of my machines.. ;) still
they are the best ones available ;)

-- 
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       pcg@goof.com      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |