[REPORT] cfs-v4 vs sd-0.44

[Willy Tarreau <w@1wt.eu>]

Hi Ingo, Hi Con,

I promised to perform some tests on your code. I'm short in time right now,
but I observed behaviours that should be commented on.

1) machine : dual athlon 1533 MHz, 1G RAM, kernel 2.6.21-rc7 + either scheduler
   Test:  ./ocbench -R 250000 -S 750000 -x 8 -y 8
   ocbench: http://linux.1wt.eu/sched/

2) SD-0.44

   Feels good, but becomes jerky at moderately high loads. I've started
   64 ocbench with a 250 ms busy loop and 750 ms sleep time. The system
   always responds correctly but under X, mouse jumps quite a bit and
   typing in xterm or even text console feels slightly jerky. The CPU is
   not completely used, and the load varies a lot (see below). However,
   the load is shared equally between all 64 ocbench, and they do not
   deviate even after 4000 iterations. X uses less than 1% CPU during
   those tests.

   Here's the vmstat output :

willy@pcw:~$ vmstat 1
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0  0  0      0 919856   6648  57788    0    0    22     2    4   148 31 49 20
 0  0  0      0 919856   6648  57788    0    0     0     0    2   285 32 50 19
28  0  0      0 919836   6648  57788    0    0     0     0    0   331 24 40 36
64  0  0      0 919836   6648  57788    0    0     0     0    1   618 23 40 37
65  0  0      0 919836   6648  57788    0    0     0     0    0   571 21 36 43
35  0  0      0 919836   6648  57788    0    0     0     0    3   382 32 50 18
 2  0  0      0 919836   6648  57788    0    0     0     0    0   308 37 61  2
 8  0  0      0 919836   6648  57788    0    0     0     0    1   533 36 65  0
32  0  0      0 919768   6648  57788    0    0     0     0   93   706 33 62  5
62  0  0      0 919712   6648  57788    0    0     0     0   65   617 32 54 13
63  0  0      0 919712   6648  57788    0    0     0     0    1   569 28 48 23
40  0  0      0 919712   6648  57788    0    0     0     0    0   427 26 50 24
 4  0  0      0 919712   6648  57788    0    0     0     0    1   382 29 48 23
 4  0  0      0 919712   6648  57788    0    0     0     0    0   383 34 65  0
14  0  0      0 919712   6648  57788    0    0     0     0    1   769 39 61  0
40  0  0      0 919712   6648  57788    0    0     0     0    0   384 37 52 11
54  0  0      0 919712   6648  57788    0    0     0     0    1   715 31 60  8
58  0  2      0 919712   6648  57788    0    0     0     0    1   611 34 65  0
41  0  0      0 919712   6648  57788    0    0     0     0   19   395 28 45 27
 0  0  0      0 919712   6648  57788    0    0     0     0   31   421 23 32 45
 0  0  0      0 919712   6648  57788    0    0     0     0   31   328 34 44 22
29  0  0      0 919712   6648  57788    0    0     0     0   34   369 32 43 25
65  0  0      0 919712   6648  57788    0    0     0     0   31   410 24 35 40
47  0  1      0 919712   6648  57788    0    0     0     0   42   538 25 39 35

3) CFS-v4

  Feels even better, mouse movements are very smooth even under high load.
  I noticed that X gets reniced to -19 with this scheduler. I've not looked
  at the code yet but this looked suspicious to me. I've reniced it to 0 and
  it did not change any behaviour. Still very good. The 64 ocbench share
  equal CPU time and show exact same progress after 2000 iterations. The CPU
  load is more smoothly spread according to vmstat, and there's no idle (see
  below). BUT I now think it was wrong to let new processes start with no
  timeslice at all, because it can take tens of seconds to start a new process
  when only 64 ocbench are there. Simply starting "killall ocbench" takes about
  10 seconds. On a smaller machine (VIA C3-533), it took me more than one
  minute to do "su -", even from console, so that's not X. BTW, X uses less
  than 1% CPU during those tests.

willy@pcw:~$ vmstat 1
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
12  0  2      0 922120   6532  57540    0    0   299    29   31   386 17 27 57
12  0  2      0 922096   6532  57556    0    0     0     0    1   776 37 63  0
14  0  2      0 922096   6532  57556    0    0     0     0    1   782 35 65  0
13  0  1      0 922096   6532  57556    0    0     0     0    0   782 38 62  0
14  0  1      0 922096   6532  57556    0    0     0     0    1   782 36 64  0
13  0  1      0 922096   6532  57556    0    0     0     0    2   785 38 62  0
13  0  1      0 922096   6532  57556    0    0     0     0    1   774 35 65  0
14  0  1      0 922096   6532  57556    0    0     0     0    0   784 36 64  0
13  0  1      0 922096   6532  57556    0    0     0     0    1   767 37 63  0
13  0  1      0 922096   6532  57556    0    0     0     0    1   785 41 59  0
14  0  1      0 922096   6532  57556    0    0     0     0    0   779 38 62  0
19  0  1      0 922096   6532  57556    0    0     0     0    1   816 38 62  0
22  0  1      0 922096   6532  57556    0    0     0     0    0   817 35 65  0
19  0  1      0 922096   6532  57556    0    0     0     0    1   817 39 61  0
21  0  1      0 922096   6532  57556    0    0     0     0    0   849 36 64  0
20  0  0      0 922096   6532  57556    0    0     0     0    1   793 36 64  0
21  0  0      0 922096   6532  57556    0    0     0     0    0   815 37 63  0
19  0  0      0 922096   6532  57556    0    0     0     0    1   824 35 65  0
21  0  0      0 922096   6532  57556    0    0     0     0    0   817 35 65  0
26  0  0      0 922096   6532  57556    0    0     0     0    1   824 38 62  0
26  0  0      0 922096   6532  57556    0    0     0     0    1   817 35 65  0
26  0  0      0 922096   6532  57556    0    0     0     0    0   811 37 63  0
26  0  0      0 922096   6532  57556    0    0     0     0    1   804 34 66  0
16  0  0      0 922096   6532  57556    0    0     0     0   39   850 35 65  0
18  0  0      0 922096   6532  57556    0    0     0     0    1   801 39 61  0


4) first impressions

I think that CFS is based on a more promising concept but is less mature
and is dangerous right now with certain workloads. SD shows some strange
behaviours like not using all CPU available and a little jerkyness, but is
more robust and may be the less risky solution for a first step towards
a better scheduler in mainline, but it may also probably be the last O(1)
scheduler, which may be replaced sometime later when CFS (or any other one)
shows at the same time the smoothness of CFS and the robustness of SD.

I'm sorry not to spend more time on them right now, I hope that other people
will do.

Regards,
Willy