Re: [2.4.17/18pre] VM and swap - it's really unusable

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Dieter Nützel]

Is it possible to decide, now what should go into 2.4.18 (maybe -pre3) -aa or 
-rmap?
Andrew Morten`s read-latency.patch is a clear winner for me, too.
What about 00_nanosleep-5 and bootmem?
The O(1) scheduler?
Maybe preemption? It is disengageable so nobody should be harmed but we get 
the chance for wider testing.

Any comments?

Thanks,
	Dieter

-- 
Dieter Nützel
Graduate Student, Computer Science

University of Hamburg
Department of Computer Science
@home: Dieter.Nuetzel@hamburg.de

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Luigi Genoni]

On Tue, 8 Jan 2002, Dieter [iso-8859-15] Nützel wrote (passim):

> Is it possible to decide, now what should go into 2.4.18 (maybe -pre3) -aa or
> -rmap?
[...]
> Maybe preemption? It is disengageable so nobody should be harmed but we get
> the chance for wider testing.
>
> Any comments?
preemption?? this is eventually 2.5 stuff, and should not be integrated
into 2.4 stable tree. Of course a backport is possible, when/if it will be
quite well tested and well working on 2.5

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Tue, Jan 08, 2002 at 11:55:59AM +0100, Luigi Genoni wrote:
> 
> 
> On Tue, 8 Jan 2002, Dieter [iso-8859-15] Nützel wrote (passim):
> 
> > Is it possible to decide, now what should go into 2.4.18 (maybe -pre3) -aa or
> > -rmap?
> [...]
> > Maybe preemption? It is disengageable so nobody should be harmed but we get
> > the chance for wider testing.
> >
> > Any comments?
> preemption?? this is eventually 2.5 stuff, and should not be integrated

indeed ("eventually" in the italian sense btw, obvious to me, but not
for l-k).

I'm not against preemption (I can see the benefits about the mean
latency for real time DSP) but the claims about preemption making the
kernel faster doesn't make sense to me. more frequent scheduling,
overhead of branches in the locks (you've to conditional_schedule after
the last preemption lock is released and the cachelines for the per-cpu
preemption locks) and the other preemption stuff can only make the
kernel slower.  Furthmore for multimedia playback any sane kernel out
there with lowlatency fixes applied will work as well as a preemption
kernel that pays for all the preemption overhead.

About the other claim that as the kernel becomes more granular
performance will increase with preemption in kernel, that's obviously
wrong as well, it's clearly the other way around. Maybe it was meant
"latency will decrease further", that's right, but also performance will
decrease if something.

So yes, mean latency will decrease with preemptive kernel, but your CPU
is definitely paying something for it.

> into 2.4 stable tree. Of course a backport is possible, when/if it will be
> quite well tested and well working on 2.5
> 
> 
> 
> 


Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Anton Blanchard]

 
> So yes, mean latency will decrease with preemptive kernel, but your CPU
> is definitely paying something for it.

And Andrew Morton's work suggests he can do a much better job of
reducing latency than -preempt.

Anton

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 8, 2002 02:33 pm, Anton Blanchard wrote:
> Andrea Arcangeli [apparently] wrote:
> > So yes, mean latency will decrease with preemptive kernel, but your CPU
> > is definitely paying something for it.
> 
> And Andrew Morton's work suggests he can do a much better job of
> reducing latency than -preempt.

That's not a particularly clueful comment, Anton.  Obviously, any 
latency-busting hacks that Andrew does could also be patched into a
-preempt kernel.

What a preemptible kernel can do that a non-preemptible kernel can't is: 
reschedule exactly as often as necessary, instead of having lots of extra 
schedule points inserted all over the place, firing when *they* think the 
time is right, which may well be earlier than necessary.

The preemptible approach is much less of a maintainance headache, since 
people don't have to be constantly doing audits to see if something changed, 
and going in to fiddle with scheduling points.

Finally, with preemption, rescheduling can be forced with essentially zero 
latency in response to an arbitrary interrupt such as IO completion, whereas 
the non-preemptive kernel will have to 'coast to a stop'.  In other words, 
the non-preemptive kernel will have little lags between successive IOs, 
whereas the preemptive kernel can submit the next IO immediately.  So there 
are bound to be loads where the preemptive kernel turns in better latency 
*and throughput* than the scheduling point hack.

Mind you, I'm not devaluing Andrew's work, it's good and valuable.  However 
it's good to be aware of why that approach can't equal the latency-busting 
performance of the preemptive approach.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Tue, Jan 08, 2002 at 04:00:11PM +0100, Daniel Phillips wrote:
> On January 8, 2002 02:33 pm, Anton Blanchard wrote:
> > Andrea Arcangeli [apparently] wrote:
> > > So yes, mean latency will decrease with preemptive kernel, but your CPU
> > > is definitely paying something for it.
> > 
> > And Andrew Morton's work suggests he can do a much better job of
> > reducing latency than -preempt.
> 
> That's not a particularly clueful comment, Anton.  Obviously, any 
> latency-busting hacks that Andrew does could also be patched into a
> -preempt kernel.
> 
> What a preemptible kernel can do that a non-preemptible kernel can't is: 
> reschedule exactly as often as necessary, instead of having lots of extra 
> schedule points inserted all over the place, firing when *they* think the 
> time is right, which may well be earlier than necessary.

"extra schedule points all over the place", that's the -preempt kernel
not the lowlatency kernel! (on yeah, you don't see them in the source
but ask your CPU if it sees them)

> The preemptible approach is much less of a maintainance headache, since 
> people don't have to be constantly doing audits to see if something changed, 
> and going in to fiddle with scheduling points.

this yes, it requires less maintainance, but still you should keep in
mind the details about the spinlocks, things like the checks the VM does
in shrink_cache are needed also with preemptive kernel.

> Finally, with preemption, rescheduling can be forced with essentially zero 
> latency in response to an arbitrary interrupt such as IO completion, whereas 
> the non-preemptive kernel will have to 'coast to a stop'.  In other words, 
> the non-preemptive kernel will have little lags between successive IOs, 
> whereas the preemptive kernel can submit the next IO immediately.  So there 
> are bound to be loads where the preemptive kernel turns in better latency 
> *and throughput* than the scheduling point hack.

The I/O pipeline is big enough that a few msec before or later in a
submit_bh shouldn't make a difference, the batch logic in the
ll_rw_block layer also try to reduce the reschedule, and last but not
the least if the task is I/O bound preemptive kernel or not won't make
any difference in the submit_bh latency because no task is eating cpu
and latency will be the one of pure schedule call.

> Mind you, I'm not devaluing Andrew's work, it's good and valuable.  However 
> it's good to be aware of why that approach can't equal the latency-busting 
> performance of the preemptive approach.

I also don't want to devaluate the preemptive kernel approch (the mean
latency it can reach is lower than the one of the lowlat kernel, however
I personally care only about worst case latency and this is why I don't
feel the need of -preempt), but I just wanted to make clear that the
idea that is floating around that preemptive kernel is all goodness is
very far from reality, you get very low mean latency but at a price.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 8, 2002 04:29 pm, Andrea Arcangeli wrote:
> > The preemptible approach is much less of a maintainance headache, since 
> > people don't have to be constantly doing audits to see if something changed, 
> > and going in to fiddle with scheduling points.
> 
> this yes, it requires less maintainance, but still you should keep in
> mind the details about the spinlocks, things like the checks the VM does
> in shrink_cache are needed also with preemptive kernel.

Yes of course, the spinlock regions still have to be analyzed and both
patches have to be maintained for that.  Long duration spinlocks are bad
by any measure, and have to be dealt with anyway.

> > Finally, with preemption, rescheduling can be forced with essentially zero 
> > latency in response to an arbitrary interrupt such as IO completion, whereas 
> > the non-preemptive kernel will have to 'coast to a stop'.  In other words, 
> > the non-preemptive kernel will have little lags between successive IOs, 
> > whereas the preemptive kernel can submit the next IO immediately.  So there 
> > are bound to be loads where the preemptive kernel turns in better latency 
> > *and throughput* than the scheduling point hack.
> 
> The I/O pipeline is big enough that a few msec before or later in a
> submit_bh shouldn't make a difference, the batch logic in the
> ll_rw_block layer also try to reduce the reschedule, and last but not
> the least if the task is I/O bound preemptive kernel or not won't make
> any difference in the submit_bh latency because no task is eating cpu
> and latency will be the one of pure schedule call.

That's not correct.  For one thing, you don't know that no task is eating
CPU, or that nobody is hogging the kernel.  Look at the above, and consider
the part about the little lags between IOs.

> > Mind you, I'm not devaluing Andrew's work, it's good and valuable.  However 
> > it's good to be aware of why that approach can't equal the latency-busting 
> > performance of the preemptive approach.
> 
> I also don't want to devaluate the preemptive kernel approch (the mean
> latency it can reach is lower than the one of the lowlat kernel, however
> I personally care only about worst case latency and this is why I don't
> feel the need of -preempt),

This is exactly the case that -preempt handles well.  On the other hand,
trying to show that scheduling hacks satisfy any given latency bound is
equivalent to solving the halting problem.

I thought you had done some real time work?

> but I just wanted to make clear that the
> idea that is floating around that preemptive kernel is all goodness is
> very far from reality, you get very low mean latency but at a price.

A price lots of people are willing to pay.

By the way, have you measured the cost of -preempt in practice?

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Tue, Jan 08, 2002 at 04:54:58PM +0100, Daniel Phillips wrote:
> On January 8, 2002 04:29 pm, Andrea Arcangeli wrote:
> > > The preemptible approach is much less of a maintainance headache, since 
> > > people don't have to be constantly doing audits to see if something changed, 
> > > and going in to fiddle with scheduling points.
> > 
> > this yes, it requires less maintainance, but still you should keep in
> > mind the details about the spinlocks, things like the checks the VM does
> > in shrink_cache are needed also with preemptive kernel.
> 
> Yes of course, the spinlock regions still have to be analyzed and both
> patches have to be maintained for that.  Long duration spinlocks are bad
> by any measure, and have to be dealt with anyway.
> 
> > > Finally, with preemption, rescheduling can be forced with essentially zero 
> > > latency in response to an arbitrary interrupt such as IO completion, whereas 
> > > the non-preemptive kernel will have to 'coast to a stop'.  In other words, 
> > > the non-preemptive kernel will have little lags between successive IOs, 
> > > whereas the preemptive kernel can submit the next IO immediately.  So there 
> > > are bound to be loads where the preemptive kernel turns in better latency 
> > > *and throughput* than the scheduling point hack.
> > 
> > The I/O pipeline is big enough that a few msec before or later in a
> > submit_bh shouldn't make a difference, the batch logic in the
> > ll_rw_block layer also try to reduce the reschedule, and last but not
> > the least if the task is I/O bound preemptive kernel or not won't make
> > any difference in the submit_bh latency because no task is eating cpu
> > and latency will be the one of pure schedule call.
> 
> That's not correct.  For one thing, you don't know that no task is eating
> CPU, or that nobody is hogging the kernel.  Look at the above, and consider
> the part about the little lags between IOs.

We agree. Actually "if the task is I/O bound", I meant "if nobody is
hogging CPU", I exactly wanted to make the example of no task hogging
CPU in general. 

> > > Mind you, I'm not devaluing Andrew's work, it's good and valuable.  However 
> > > it's good to be aware of why that approach can't equal the latency-busting 
> > > performance of the preemptive approach.
> > 
> > I also don't want to devaluate the preemptive kernel approch (the mean
> > latency it can reach is lower than the one of the lowlat kernel, however
> > I personally care only about worst case latency and this is why I don't
> > feel the need of -preempt),
> 
> This is exactly the case that -preempt handles well.  On the other hand,
> trying to show that scheduling hacks satisfy any given latency bound is
> equivalent to solving the halting problem.
> 
> I thought you had done some real time work?
> 
> > but I just wanted to make clear that the
> > idea that is floating around that preemptive kernel is all goodness is
> > very far from reality, you get very low mean latency but at a price.
> 
> A price lots of people are willing to pay.

I'm not convinced that all those people knows exactly what they're
buying then 8).

> 
> By the way, have you measured the cost of -preempt in practice?

dropping the lock in spin_unlock made a difference in the numbers, so
the overhead must be definitely visible, by simply loading the system
with threaded kernel computation (like webserving etc..etc..).

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Luigi Genoni]

On Tue, 8 Jan 2002, Daniel Phillips wrote:

> On January 8, 2002 04:29 pm, Andrea Arcangeli wrote:
> > > The preemptible approach is much less of a maintainance headache, since
> > > people don't have to be constantly doing audits to see if something changed,
> > > and going in to fiddle with scheduling points.
> >
> > this yes, it requires less maintainance, but still you should keep in
> > mind the details about the spinlocks, things like the checks the VM does
> > in shrink_cache are needed also with preemptive kernel.
>
> Yes of course, the spinlock regions still have to be analyzed and both
> patches have to be maintained for that.  Long duration spinlocks are bad
> by any measure, and have to be dealt with anyway.
>
> > > Finally, with preemption, rescheduling can be forced with essentially zero
> > > latency in response to an arbitrary interrupt such as IO completion, whereas
> > > the non-preemptive kernel will have to 'coast to a stop'.  In other words,
> > > the non-preemptive kernel will have little lags between successive IOs,
> > > whereas the preemptive kernel can submit the next IO immediately.  So there
> > > are bound to be loads where the preemptive kernel turns in better latency
> > > *and throughput* than the scheduling point hack.
> >
> > The I/O pipeline is big enough that a few msec before or later in a
> > submit_bh shouldn't make a difference, the batch logic in the
> > ll_rw_block layer also try to reduce the reschedule, and last but not
> > the least if the task is I/O bound preemptive kernel or not won't make
> > any difference in the submit_bh latency because no task is eating cpu
> > and latency will be the one of pure schedule call.
>
> That's not correct.  For one thing, you don't know that no task is eating
> CPU, or that nobody is hogging the kernel.  Look at the above, and consider
> the part about the little lags between IOs.
>
> > > Mind you, I'm not devaluing Andrew's work, it's good and valuable.  However
> > > it's good to be aware of why that approach can't equal the latency-busting
> > > performance of the preemptive approach.
> >
> > I also don't want to devaluate the preemptive kernel approch (the mean
> > latency it can reach is lower than the one of the lowlat kernel, however
> > I personally care only about worst case latency and this is why I don't
> > feel the need of -preempt),
>
> This is exactly the case that -preempt handles well.  On the other hand,
> trying to show that scheduling hacks satisfy any given latency bound is
> equivalent to solving the halting problem.
>
> I thought you had done some real time work?
>
> > but I just wanted to make clear that the
> > idea that is floating around that preemptive kernel is all goodness is
> > very far from reality, you get very low mean latency but at a price.
>
> A price lots of people are willing to pay
Probably sometimes they are not making a good business. In the reality
preempt is good in many scenarios, as I said, and I agree that for
desktops, and dedicated servers where just one application runs, and
probably the CPU is idle the most of the time, indeed users have a speed
feeling. Please consider that on eavilly loaded servers, with 40 and more
users, some are running gcc, others g77, others g++ compilations, someone
runs pine or mutt or kmail, and netscape, and mozilla, and emacs (someone
form xterm kde or gnome), and and
and... You can have also 4/8 CPU butthey are not infinite ;) (but I talk
mainly thinking of dualAthlon systems).
there is a lot of memory and disk I/O.
This is not a strange scenary on the interactive servers used at SNS.
Here preempt has a too high price
>
> By the way, have you measured the cost of -preempt in practice?
>
Yes, I did a lot of tests, and with current preempt patch definitelly
I was seeing a too big performance loss.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ken Brownfield]

On Wed, Jan 09, 2002 at 12:02:48AM +0100, Luigi Genoni wrote:
| Probably sometimes they are not making a good business. In the reality
| preempt is good in many scenarios, as I said, and I agree that for
| desktops, and dedicated servers where just one application runs, and
| probably the CPU is idle the most of the time, indeed users have a speed
| feeling. Please consider that on eavilly loaded servers, with 40 and more
| users, some are running gcc, others g77, others g++ compilations, someone
| runs pine or mutt or kmail, and netscape, and mozilla, and emacs (someone
| form xterm kde or gnome), and and
| and... You can have also 4/8 CPU butthey are not infinite ;) (but I talk
| mainly thinking of dualAthlon systems).
| there is a lot of memory and disk I/O.
| This is not a strange scenary on the interactive servers used at SNS.
| Here preempt has a too high price

MacOS 9 is the OS for you.

Essentially what the low-latency patches are is cooperative
multitasking.  Which has less overhead in some cases than preemptive as
long as everyone is equally nice and calls WaitNextEvent() within the
right inner loops.  In the absence of preemptive, Andrew's patch is the
next best thing.  But Bad Things happen without preemptive.  Just try
using Mac OS 9. ;)

Preemptive gives better interactivity under load, which is the whole
point of multitasking (think about it).  If you don't want the overhead
(which also exists without preemptive) run #processes == #processors.

Whether or not preemptive is applied, having a large number of processes
active is a performance hit from context switches, cache thrashing, etc.
Preemptive punishes (and rewards) everyone equally, thus better latency.

I'm really surprised that people are still actually arguing _against_
preemptive multitasking in this day and age.  This is a no-brainer in
the long run, where current corner cases aren't holding us back.

At least IMVHO.
-- 
Ken.
brownfld@irridia.com

| > By the way, have you measured the cost of -preempt in practice?
| >
| Yes, I did a lot of tests, and with current preempt patch definitelly
| I was seeing a too big performance loss.
| 
| 
| -
| To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
| the body of a message to majordomo@vger.kernel.org
| More majordomo info at  http://vger.kernel.org/majordomo-info.html
| Please read the FAQ at  http://www.tux.org/lkml/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 18:32, Ken Brownfield wrote:
> On Wed, Jan 09, 2002 at 12:02:48AM +0100, Luigi Genoni wrote:
> | Probably sometimes they are not making a good business. In the reality
> | preempt is good in many scenarios, as I said, and I agree that for
> | desktops, and dedicated servers where just one application runs, and
> | probably the CPU is idle the most of the time, indeed users have a speed
> | feeling. Please consider that on eavilly loaded servers, with 40 and more
> | users, some are running gcc, others g77, others g++ compilations, someone
> | runs pine or mutt or kmail, and netscape, and mozilla, and emacs (someone
> | form xterm kde or gnome), and and
> | and... You can have also 4/8 CPU butthey are not infinite ;) (but I talk
> | mainly thinking of dualAthlon systems).
> | there is a lot of memory and disk I/O.
> | This is not a strange scenary on the interactive servers used at SNS.
> | Here preempt has a too high price
> 
> MacOS 9 is the OS for you.
> 
> Essentially what the low-latency patches are is cooperative
> multitasking.  Which has less overhead in some cases than preemptive as
> long as everyone is equally nice and calls WaitNextEvent() within the
> right inner loops.  In the absence of preemptive, Andrew's patch is the
> next best thing.  But Bad Things happen without preemptive.  Just try
> using Mac OS 9. ;)
> 
> Preemptive gives better interactivity under load, which is the whole
> point of multitasking (think about it).  If you don't want the overhead
> (which also exists without preemptive) run #processes == #processors.
> 
> Whether or not preemptive is applied, having a large number of processes
> active is a performance hit from context switches, cache thrashing, etc.
> Preemptive punishes (and rewards) everyone equally, thus better latency.
> 
> I'm really surprised that people are still actually arguing _against_
> preemptive multitasking in this day and age.  This is a no-brainer in
> the long run, where current corner cases aren't holding us back.

Amen.

Of course, the counter argument is that we, as kernel programmers, can
design everything to behave kindly cooperatively.  I reply "now that the
kernel is SMP safe it is trivial to become preemptible" but some still
don't take the patch.  I'll keep trucking along.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Luigi Genoni]

On Tue, 8 Jan 2002, Ken Brownfield wrote:

> On Wed, Jan 09, 2002 at 12:02:48AM +0100, Luigi Genoni wrote:
> | Probably sometimes they are not making a good business. In the reality
> | preempt is good in many scenarios, as I said, and I agree that for
> | desktops, and dedicated servers where just one application runs, and
> | probably the CPU is idle the most of the time, indeed users have a speed
> | feeling. Please consider that on eavilly loaded servers, with 40 and more
> | users, some are running gcc, others g77, others g++ compilations, someone
> | runs pine or mutt or kmail, and netscape, and mozilla, and emacs (someone
> | form xterm kde or gnome), and and
> | and... You can have also 4/8 CPU butthey are not infinite ;) (but I talk
> | mainly thinking of dualAthlon systems).
> | there is a lot of memory and disk I/O.
> | This is not a strange scenary on the interactive servers used at SNS.
> | Here preempt has a too high price
>
> MacOS 9 is the OS for you.
>
> Essentially what the low-latency patches are is cooperative
> multitasking.  Which has less overhead in some cases than preemptive as
> long as everyone is equally nice and calls WaitNextEvent() within the
> right inner loops.  In the absence of preemptive, Andrew's patch is the
> next best thing.  But Bad Things happen without preemptive.  Just try
> using Mac OS 9 :)
Not exaclty what I was thinking about.
I listened some horror story from MAC sysadmin at SNS
>
> Preemptive gives better interactivity under load, which is the whole
> point of multitasking (think about it).  If you don't want the overhead
> (which also exists without preemptive) run #processes == #processors.
>
> Whether or not preemptive is applied, having a large number of processes
> active is a performance hit from context switches, cache thrashing, etc.
> Preemptive punishes (and rewards) everyone equally, thus better latency.
you are supposing that I want them to be punished equally. But there are
cases when that is not what you want ;). Thing if one users runs a
montecarlo code for test in the server I was describing. This job could
run, let's say, a couple of hour, and also under nice 20 it can suck a
lot.
>
> I'm really surprised that people are still actually arguing _against_
> preemptive multitasking in this day and age.  This is a no-brainer in
> the long run, where current corner cases aren't holding us back.
>
> At least IMVHO.
What I am talking about is some test I did some week ago. The initial post
of this thread, I think, was very clear about that. On the long run, with
a very well tested implementation. Actually it is not a good idea to
insert preempt nside of the 2.4 stable tree,
because there is a lot of work to do
to get a very WELL TESTED implementation.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Preemptive gives better interactivity under load, which is the whole
> point of multitasking (think about it).  If you don't want the overhead
> (which also exists without preemptive) run #processes == #processors.

That is generally not true. Pe-emption is used in user space to prevent
applications doing very stupid things. Pre-emption in a trusted environment
can often be most efficient if done by the programs themselves.

Userspace is not a trusted environment

> I'm really surprised that people are still actually arguing _against_
> preemptive multitasking in this day and age.  This is a no-brainer in
> the long run, where current corner cases aren't holding us back.

Andrew's patches give you 1mS worst case latency for normal situations, that
is below human perception, and below scheduling granularity. In other words
without the efficiency loss and the debugging problems you can place the
far enough latency below other effects that it isnt worth attacking any more.

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[John Alvord]

On Wed, 9 Jan 2002 00:10:38 +0000 (GMT), Alan Cox
<alan@lxorguk.ukuu.org.uk> wrote:

>> Preemptive gives better interactivity under load, which is the whole
>> point of multitasking (think about it).  If you don't want the overhead
>> (which also exists without preemptive) run #processes == #processors.
>
>That is generally not true. Pe-emption is used in user space to prevent
>applications doing very stupid things. Pre-emption in a trusted environment
>can often be most efficient if done by the programs themselves.
>
>Userspace is not a trusted environment
The best part about planned preemption points is that there is minimal
state to save when an interruption occurs.

>
>> I'm really surprised that people are still actually arguing _against_
>> preemptive multitasking in this day and age.  This is a no-brainer in
>> the long run, where current corner cases aren't holding us back.
>
>Andrew's patches give you 1mS worst case latency for normal situations, that
>is below human perception, and below scheduling granularity. In other words
>without the efficiency loss and the debugging problems you can place the
>far enough latency below other effects that it isnt worth attacking any more.

Incidently human visual perception runs around 200 milliseconds
minimum and hearing/touch perception around 100 milliseconds if the
signal has to go through the brain. Of course we extend our
perceptions with tools/programs etc.

john

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 19:29, John Alvord wrote:

> The best part about planned preemption points is that there is minimal
> state to save when an interruption occurs.

Actually, both preempt-kernel and low-latency do about the same amount
of work re saving state.

With preempt-kernel, when a task is preempted in-kernel we AND a flag
value into the preempt count.  That is all we need to keep track of
things.

With low-latency, the task state is set to TASK_RUNNING (which is a
precautionary measure).  So it is about the same, although low-latency
(and lock-break) often also have to do various setup with the locks and
all.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Kent Borg]

On Tue, Jan 08, 2002 at 04:29:59PM -0800, John Alvord wrote:
> Incidently human visual perception runs around 200 milliseconds
> minimum and hearing/touch perception around 100 milliseconds if the
> signal has to go through the brain. Of course we extend our
> perceptions with tools/programs etc.

Cool!  That means movies don't need to run faster than 5
frames/second.  Maybe 10 frames/second for plenty of overkill.  No
need to look at keyboard and mice any more frequently either, what a
relief.  (Any why do silly gamers want to go so much higher?)

Sarcasm mode "off" now...just because some experiments show it takes
humans a long time to push the correct button when you show them a
picture of a banana doesn't mean there is no reason to have a user
interface do anything any faster.  (I can come up with plenty of
examples if you would like.)


OK, now that I have pissed off a big hunk of the folks on the list,
let me bring up a different question: 

How does all this fit into doing a tick-less kernel?

There is something appealing about doing stuff only when there is
stuff to do, like: respond to input, handle some device that becomes
ready, or let another process run for a while.  Didn't IBM do some
nice work on this for Linux?  (*Was* it nice work?)  I was under the
impression that the current kernel isn't that far from being tickless.

A tickless kernel would be wonderful for battery powered devices that
could literally shut off when there be nothing to do, and it seems it
would (trivially?) help performance on high end power hogs too.

Why do we have regular HZ ticks?  (Other than I think I remember Linus
saying that he likes them.)  


Thanks,

-kb, the Kent who knows more about user interfaces than he does
preemption.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Wed, 9 Jan 2002, Kent Borg wrote:

> How does all this fit into doing a tick-less kernel?
> 
> There is something appealing about doing stuff only when there is
> stuff to do, like: respond to input, handle some device that becomes
> ready, or let another process run for a while.  Didn't IBM do some
> nice work on this for Linux?  (*Was* it nice work?)  I was under the
> impression that the current kernel isn't that far from being tickless.
> 
> A tickless kernel would be wonderful for battery powered devices that
> could literally shut off when there be nothing to do, and it seems it
> would (trivially?) help performance on high end power hogs too.
> 
> Why do we have regular HZ ticks?  (Other than I think I remember Linus
> saying that he likes them.)  

Feel free to quantify the savings over the current setup with max power
saving enabled in the kernel. I just don't see how "wonderful" it would
be, given that an idle system currently uses very little battery if you
setup the options to save power.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Feel free to quantify the savings over the current setup with max power
> saving enabled in the kernel. I just don't see how "wonderful" it would
> be, given that an idle system currently uses very little battery if you
> setup the options to save power.

IBM have a tickless kernel patch set for the S/390. Here its not battery at
stake but VM overhead sending timer interrupts to hundreds of otherwise idle
virtual machines

Re: [2.4.17/18pre] VM and swap - it's really unusable

[george anzinger]

Bill Davidsen wrote:
> 
> On Wed, 9 Jan 2002, Kent Borg wrote:
> 
> > How does all this fit into doing a tick-less kernel?
> >
> > There is something appealing about doing stuff only when there is
> > stuff to do, like: respond to input, handle some device that becomes
> > ready, or let another process run for a while.  Didn't IBM do some
> > nice work on this for Linux?  (*Was* it nice work?)  I was under the
> > impression that the current kernel isn't that far from being tickless.
> >
> > A tickless kernel would be wonderful for battery powered devices that
> > could literally shut off when there be nothing to do, and it seems it
> > would (trivially?) help performance on high end power hogs too.
> >
> > Why do we have regular HZ ticks?  (Other than I think I remember Linus
> > saying that he likes them.)
>
I put a patch on sourceforge as part of the high-res-timers
investigation the implemented a tick less kernel with instrumentation. 
It turns out to be overload prone, mostly do to the need to start and
stop a "slice" timer on each schedule() call.  I, for one, think this
issue is dead and rightly so.  The patch is still there for those who
want to try it.  See signature for URL.

 
> Feel free to quantify the savings over the current setup with max power
> saving enabled in the kernel. I just don't see how "wonderful" it would
> be, given that an idle system currently uses very little battery if you
> setup the options to save power.
> 
> --
> bill davidsen <davidsen@tmr.com>
>   CTO, TMR Associates, Inc
> Doing interesting things with little computers since 1979.
> 

-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Jussi Laako]

John Alvord wrote:
> 
> Incidently human visual perception runs around 200 milliseconds
> minimum and hearing/touch perception around 100 milliseconds if the
> signal has to go through the brain. Of course we extend our
> perceptions with tools/programs etc.

Have you ever tried to play piano with 100 ms latency from pressing the key
to sound? I can tell you that it's pretty difficult...

 - Jussi Laako

-- 
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Alan Cox wrote:
> 
> Andrew's patches give you 1mS worst case latency for normal situations, that
> is below human perception, and below scheduling granularity.

The full ll patch is pretty gruesome though.

The high-end audio synth guys claim that two milliseconds is getting
to be too much.  They are generating real-time audio and they do
have more than one round-trip through the software.  It adds up.

Linux is being used in so many different applications now.  You are,
I think, one of the stronger recognisers of the fact that we do not
only use Linux to squirt out html and to provide shell prompts to snotty
students.  Good scheduling responsiveness is a valuable feature.

I haven't seen any figures for embedded XP, but it is said that
if you bend over backwards you can get 10 milliseconds out of NT4,
and 4-5 out of the fabled BeOS.  This is one area where we can
fairly easily be very much the best.  It's low-hanging fruit.

Internal preemptability is, in my opinion, the best way to deliver
this.

I accept your point about it making debugging harder - I would
suggest that the preempt code be altered so that it can be disabled
at runtime, rather than via a rebuild.  I suspect this can be
done at zero cost by setting init_task's preempt count to 1000000
via a kernel boot option.  And at almost-zero cost via a sysctl.

I would further suggest that support be added to the kernel to
allow general users to both detect and find the source of latency
problems.  That's actually pretty easy - realfeel running at
2 kHz only consumes 2-3% of the CPU.  It can just be left ticking
over in the background.

With preemptability merged in 2.5 we can then work to fix the
long-held locks.  Most of them are simple.  Some of them are
very much not.  I'll gladly help with that.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Wed, 2002-01-09 at 00:08, Andrew Morton wrote:
> [snip]
> With preemptability merged in 2.5 we can then work to fix the
> long-held locks.  Most of them are simple.  Some of them are
> very much not.  I'll gladly help with that.

Amen.  Thank you, Andrew.

Let's work together and make a better kernel.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Helge Hafting]

Andrew Morton wrote:
[...]
> I haven't seen any figures for embedded XP, but it is said that
> if you bend over backwards you can get 10 milliseconds out of NT4,
> and 4-5 out of the fabled BeOS.  This is one area where we can
> fairly easily be very much the best.  It's low-hanging fruit.
> 
> Internal preemptability is, in my opinion, the best way to deliver
> this.
> 
> I accept your point about it making debugging harder - I would
> suggest that the preempt code be altered so that it can be disabled
> at runtime, rather than via a rebuild.  I suspect this can be
> done at zero cost by setting init_task's preempt count to 1000000
> via a kernel boot option.  And at almost-zero cost via a sysctl.

And with some bad luck, the bug goes away when you
do this.  The bug of the missing lock...

Helge Hafting

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> The high-end audio synth guys claim that two milliseconds is getting
> to be too much.  They are generating real-time audio and they do
> have more than one round-trip through the software.  It adds up.

Most of the stuff I've seen from high end audio people consists of
overthreaded, chains of code written without any consideration for the
actual cost of execution. There are exceptions - including
people dynamically compiling filters to get ideal cache and latency 
behaviour, but not enough.

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rob Landley]

On Wednesday 09 January 2002 12:00 pm, Alan Cox wrote:
> > The high-end audio synth guys claim that two milliseconds is getting
> > to be too much.  They are generating real-time audio and they do
> > have more than one round-trip through the software.  It adds up.
>
> Most of the stuff I've seen from high end audio people consists of
> overthreaded, chains of code written without any consideration for the
> actual cost of execution. There are exceptions - including
> people dynamically compiling filters to get ideal cache and latency
> behaviour, but not enough.
>
> Alan

News flash: people are writing sub-optimal apps in user space.

Do you want an operating system capable of running real-world code written by 
people who know more about their specific problem domain (audio) than about 
optimal coding in general, or do you want an operating system intended to 
only run well-behaved applications designed and implemented by experts?

Rob

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Rob Landley wrote:
> 
> On Wednesday 09 January 2002 12:00 pm, Alan Cox wrote:
> > > The high-end audio synth guys claim that two milliseconds is getting
> > > to be too much.  They are generating real-time audio and they do
> > > have more than one round-trip through the software.  It adds up.
> >
> > Most of the stuff I've seen from high end audio people consists of
> > overthreaded, chains of code written without any consideration for the
> > actual cost of execution. There are exceptions - including
> > people dynamically compiling filters to get ideal cache and latency
> > behaviour, but not enough.
> >
> > Alan
> 
> News flash: people are writing sub-optimal apps in user space.

Not only in user=space :)

> Do you want an operating system capable of running real-world code written by
> people who know more about their specific problem domain (audio) than about
> optimal coding in general, or do you want an operating system intended to
> only run well-behaved applications designed and implemented by experts?

The people with whom I dealt (Benno Senoner, Dave Philips, Paul
Barton-Davis) with are deeply clueful about this stuff.  

I'll quote from an email Paul set me a year ago.  I don't think
he'll mind.  This is, of course, a quite specialised application
area:


....

There are two kinds of situations where its needed:

  1) real-time effects ("FX") processing
  2) real-time synthesis influenced by external controllers

In (1), we have an incoming audio signal that is to be processed in
some way (echo/flange/equalization/etc. etc.) and then delivered back
to the output audio stream. If the delay between the input and output
is more than a few msecs, there are several possible
consequences:

        * if the original source was acoustic (non-electronic), and
             the processed material is played back over monitors
             close to the acoustic source, you will get interesting
             filtering effects as the two signals interfere with each
             other. 
             
        * the musician will get confused by material in the 
             processed stream arriving "late"

        * the result may be useless.

In (2), a musician is using, for example, a keyboard that sends MIDI
"note on/note off" messages to the computer which are intended to
cause the synthesis engine to start/stop generating certain sounds. If
the delay between the musician pressing the key and hearing the sound
exceeds about 5msec, the system will feel difficult to play; worse, if
there is actual jitter in the +/- 5msec range, it will feel impossible
to play.

....

Without LL, Linux cannot reasonably be used for professional audio
work that involves real time FX or real time synthesis. The default
kernel has worst-case latencies noticeably worse than Windows, and
most people are reluctant to use that system already, not just because
of instability but latencies also. Its not a matter of it being "a bit
of a problem" - the 100msec worst case latencies visible in the
standard kernel make it totally implausible that you would ever deploy
Linux in a situation where RT FX/synthesis were going to happen.

By contrast, if we get LL in place, then we can potentially use Linux
in "black box"/"embedded" systems designed specifically for audio
users; all the flexibility of Linux, but if they choose to ignore most
of that, they'll still have a black rack-mounted box capable of doing
everything (more mostly everything) currently done by dedicated
hardware. As general purpose CPU performance continues to increase,
this becomes more and more overwhelmingly obvious as the way forward
for audio processing.

....

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Timothy Covell]

On Wednesday 09 January 2002 13:57, Andrew Morton wrote:
[snip]
>
> Without LL, Linux cannot reasonably be used for professional audio
> work that involves real time FX or real time synthesis. The default
> kernel has worst-case latencies noticeably worse than Windows, and
> most people are reluctant to use that system already, not just because
> of instability but latencies also. Its not a matter of it being "a bit
> of a problem" - the 100msec worst case latencies visible in the
> standard kernel make it totally implausible that you would ever deploy
> Linux in a situation where RT FX/synthesis were going to happen.
>
> By contrast, if we get LL in place, then we can potentially use Linux
> in "black box"/"embedded" systems designed specifically for audio
> users; all the flexibility of Linux, but if they choose to ignore most
> of that, they'll still have a black rack-mounted box capable of doing
> everything (more mostly everything) currently done by dedicated
> hardware. As general purpose CPU performance continues to increase,
> this becomes more and more overwhelmingly obvious as the way forward
> for audio processing.
>

I keep on seeing these "Blackbox" apps like Tivo using Linux but the
fact remains the average folks cannot get any reasonable kind of A/V
performance and support under Linux.    That's what we need.

Needing to save money and get some fast cash (I'm unemployed), 
yesterday, I swapped out my dual P-III motherboard in my BeOS box
for a Via C-III (700MHz) based system.    And I got my first real hiccups
while using the OS when I was playing MP3s and _launching_ the TV
program _full screen_(640x480 on 640x480 virtual desktop window).

Obviously, when this stuff is done right, more CPU power can only help,
but it still has to be done right.  As I am sure that you know, BeOS claims
average latency of 250 microseconds.


-- 
timothy.covell@ashavan.org.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Do you want an operating system capable of running real-world code written by 
> people who know more about their specific problem domain (audio) than about 
> optimal coding in general, or do you want an operating system intended to 
> only run well-behaved applications designed and implemented by experts?

I want an OS were a reasonably cluefully written audio program works. That
to me means aiming at the 1mS latency mark. Which doesn't seem to be needing
pre-empt. Beyond a typical 1mS latency you have hardware fun to worry about,
and the BIOS SMM code eating you.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rob Landley]

On Wednesday 09 January 2002 09:25 pm, Alan Cox wrote:
> > Do you want an operating system capable of running real-world code
> > written by people who know more about their specific problem domain
> > (audio) than about optimal coding in general, or do you want an operating
> > system intended to only run well-behaved applications designed and
> > implemented by experts?
>
> I want an OS were a reasonably cluefully written audio program works. That
> to me means aiming at the 1mS latency mark. Which doesn't seem to be
> needing pre-empt. Beyond a typical 1mS latency you have hardware fun to
> worry about, and the BIOS SMM code eating you.

I don't know what BIOS SMM code is, or what you mean by "hardware fun".  But 
the worst audio dropouts I have are "cp file.wav /dev/audio" when I forgot to 
kill cron and updatedb started up.  (This is considerably WORSE than mp3 
playing.)  I take it "cp" is badly written? :)

And a sound card with only 1mS of buffer in it is definitely not useable on 
windoze, the minimum buffer in the cheapest $12 PCI sound card I've seen is 
about 1/4 second (250ms).  (Is this what you mean by "hardware fun"?)  Even 
if the app was taking half that, it's still a > 100ms big gap where the OS 
leaves it hanging before you get a dropout.  (Okay, some of that's watermark 
policy, not sending more data to the card until half the buffer is 
exhausted...)  What sound output device DOESN'T have this much cache?  (You 
mentioned USB speakers in your diary at one point, which seemed to be like 
those old "paralell port cable plus a few resistors equals sound output" 
hacks...)

Now VIDEO is a slightly more interesting problem.  (Or synchronizing audio 
and video by sending really tiny chunks of audio.)  There's no hardware 
buffer there to cover our latency sins.  Then again, dropping frames is 
considered normal in the video world, isn't it? :)

Rob

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Chris Friesen]

Rob Landley wrote:

> And a sound card with only 1mS of buffer in it is definitely not useable on
> windoze, the minimum buffer in the cheapest $12 PCI sound card I've seen is
> about 1/4 second (250ms).  (Is this what you mean by "hardware fun"?)  Even
> if the app was taking half that, it's still a > 100ms big gap where the OS
> leaves it hanging before you get a dropout.  (Okay, some of that's watermark
> policy, not sending more data to the card until half the buffer is
> exhausted...)  What sound output device DOESN'T have this much cache?

Imagine taking an input, doing dsp-type calculations on it, and sending it back
as output.  Now...imagine doing it in realtime with the output being fed back to
a monitor speaker.  Think about what would happen if the output of the monitor
speaker is 1/4 second behind the input at the mike.  Now do you see the
problem?  A few ms of delay might be okay.  A few hundred ms definately is not.

> Now VIDEO is a slightly more interesting problem.  (Or synchronizing audio
> and video by sending really tiny chunks of audio.)  There's no hardware
> buffer there to cover our latency sins.  Then again, dropping frames is
> considered normal in the video world, isn't it? :)

If I'm trying to watch a DVD on my computer, and assuming my CPU is powerful
enough to decode in realtime, then I want the DVD player to take
priority--dropping frames just because I'm starting up netscape is not
acceptable.

Chris

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[David Weinehall]

On Thu, Jan 10, 2002 at 01:34:17PM -0500, Chris Friesen wrote:
> Rob Landley wrote:
> 
> > And a sound card with only 1mS of buffer in it is definitely not useable on
> > windoze, the minimum buffer in the cheapest $12 PCI sound card I've seen is
> > about 1/4 second (250ms).  (Is this what you mean by "hardware fun"?)  Even
> > if the app was taking half that, it's still a > 100ms big gap where the OS
> > leaves it hanging before you get a dropout.  (Okay, some of that's watermark
> > policy, not sending more data to the card until half the buffer is
> > exhausted...)  What sound output device DOESN'T have this much cache?
> 
> Imagine taking an input, doing dsp-type calculations on it, and sending it back
> as output.  Now...imagine doing it in realtime with the output being fed back to
> a monitor speaker.  Think about what would happen if the output of the monitor
> speaker is 1/4 second behind the input at the mike.  Now do you see the
> problem?  A few ms of delay might be okay.  A few hundred ms definately is not.
> 
> > Now VIDEO is a slightly more interesting problem.  (Or synchronizing audio
> > and video by sending really tiny chunks of audio.)  There's no hardware
> > buffer there to cover our latency sins.  Then again, dropping frames is
> > considered normal in the video world, isn't it? :)
> 
> If I'm trying to watch a DVD on my computer, and assuming my CPU is powerful
> enough to decode in realtime, then I want the DVD player to take
> priority--dropping frames just because I'm starting up netscape is not
> acceptable.

Ummm, and you couldn't consider refraining from firing up Netscape
while watching the DVD, could you?!

I get your point, but the example was poorly chosen, imho.


Regards: David Weinehall
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Chris Friesen]

David Weinehall wrote:
> 
> On Thu, Jan 10, 2002 at 01:34:17PM -0500, Chris Friesen wrote:

> > If I'm trying to watch a DVD on my computer, and assuming my CPU is powerful
> > enough to decode in realtime, then I want the DVD player to take
> > priority--dropping frames just because I'm starting up netscape is not
> > acceptable.
> 
> Ummm, and you couldn't consider refraining from firing up Netscape
> while watching the DVD, could you?!
> 
> I get your point, but the example was poorly chosen, imho.

I chose netscape because it is probably the largest single app that I have on my
machine.  Other possibilities would be running a kernel compile, a recursive
search for specific file content through the entire filesystem, or anything else
that is likely to cause problems. It might even be someone else in the house
logged into it and running stuff over the network.


-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Jussi Laako]

Chris Friesen wrote:
> 
> machine.  Other possibilities would be running a kernel compile, a 
> recursive search for specific file content through the entire filesystem, 
> or anything else that is likely to cause problems. It might even be 
> someone else in the house logged into it and running stuff over the 
> network.

It's not enjoyable late night DVD watching when updatedb fires up at middle
of the movie. Nor when you are trying to record some audio to the disk.
Vanilla kernel really chokes up on those situations. Lowlatency patches help
a lot on this.


 - Jussi Laako

-- 
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bernd Eckenfels]

In article <3C3DDEA9.E8FAB8DC@nortelnetworks.com> you wrote:
> Imagine taking an input, doing dsp-type calculations on it, and sending it back
> as output.  Now...imagine doing it in realtime with the output being fed back to
> a monitor speaker.  Think about what would happen if the output of the monitor
> speaker is 1/4 second behind the input at the mike.  Now do you see the
> problem?  A few ms of delay might be okay.

What kind of signal run time do you normally have in digital sound processing
equipment? AFAIK one can expect a feew frames with of delay (n x 13ms).

Just dont feed back the processed signal to the singers monitor box.

> If I'm trying to watch a DVD on my computer, and assuming my CPU is powerful
> enough to decode in realtime, then I want the DVD player to take
> priority--dropping frames just because I'm starting up netscape is not
> acceptable.

You do not start up netscape while you do realtime av processing in
professional environemnt.

Well, an easy fix is to have the LL patch and do not use swap. Then you only
need reliable/predictable hardware (which is not so easy to get for PC).

Greetings
Bernd

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> I don't know what BIOS SMM code is, or what you mean by "hardware fun".  But 
> the worst audio dropouts I have are "cp file.wav /dev/audio" when I forgot to 
> kill cron and updatedb started up.  (This is considerably WORSE than mp3 
> playing.)  I take it "cp" is badly written? :)

Those are ones that Andrew's patch should fix nicely. You might need a
decent VM as well though.

The fun below 1mS comes from

	1.	APM bios calls where the bios decides to take >1mS to have
		a chat with your batteries
	2.	Video cards pulling borderline legal PCI tricks to get
		better benchmarketing by stalling the entire bus

> And a sound card with only 1mS of buffer in it is definitely not useable on 
> windoze, the minimum buffer in the cheapest $12 PCI sound card I've seen is 
> about 1/4 second (250ms).  (Is this what you mean by "hardware fun"?)  Even 

For video conferencing and for real world audio mixing you can't use
that 250ms. Not even for games. If your audio is 150mS late in quake you
will notice it, really notice it. And the buffers on the audio card are
btw generally in RAM not the fifo on the chip, so they dont help when the
PCI bus loads up

> exhausted...)  What sound output device DOESN'T have this much cache?  (You 
> mentioned USB speakers in your diary at one point, which seemed to be like 
> those old "paralell port cable plus a few resistors equals sound output" 
> hacks...)

Umm no USB audio is rather good. USB sends isosynchronous, time guaranteed
sample streams down the USB bus, to the speakers where the A to D is clear
of the machine proper.

> Now VIDEO is a slightly more interesting problem.  (Or synchronizing audio 
> and video by sending really tiny chunks of audio.)  There's no hardware 
> buffer there to cover our latency sins.  Then again, dropping frames is 
> considered normal in the video world, isn't it? :)

You'll see those too. Pure playback is ok because you have to buffer
equally rather than reliably hit deadlines

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Nigel Gamble]

On Thu, 10 Jan 2002, Alan Cox wrote:
> The fun below 1mS comes from
>
> 	1.	APM bios calls where the bios decides to take >1mS to have
> 		a chat with your batteries
> 	2.	Video cards pulling borderline legal PCI tricks to get
> 		better benchmarketing by stalling the entire bus

Don't forget the embedded space, where the hardware vendor can ensure
that their hardware is well-behaved.  Even on a PC, it is possible for
someone who cares about realtime to spec a reasonable system.

On good hardware, we can easily do much better than 1ms latency with a
preemptible kernel and a spinlock cleanup.  I don't think the
limitations of some PC hardware should limit our goals for Linux.

Nigel Gamble                                    nigel@nrg.org
Mountain View, CA, USA.                         http://www.nrg.org/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Nigel Gamble wrote:
> 
> On Thu, 10 Jan 2002, Alan Cox wrote:
> > The fun below 1mS comes from
> >
> >       1.      APM bios calls where the bios decides to take >1mS to have
> >               a chat with your batteries
> >       2.      Video cards pulling borderline legal PCI tricks to get
> >               better benchmarketing by stalling the entire bus
> 
> Don't forget the embedded space, where the hardware vendor can ensure
> that their hardware is well-behaved.  Even on a PC, it is possible for
> someone who cares about realtime to spec a reasonable system.
> 
> On good hardware, we can easily do much better than 1ms latency with a
> preemptible kernel and a spinlock cleanup.  I don't think the
> limitations of some PC hardware should limit our goals for Linux.
> 

On 700MHz x86 running Cerberus we can do 50 microseconds average
and 1300 microseconds worst-case today.  

Below 1000 uSec, the required changes get exponentially larger
and more complex.  I doubt that it's sane to try to go below
a millisecond on a desktop-class machine with desktop-class
workload, disk, memory and swap capacities.

On a more constrained system, which is what I expect you're
referring to, 250 microseconds should be achievable.  Whether
or not that is achieved via preemptability is pretty irrelevant.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> On good hardware, we can easily do much better than 1ms latency with a
> preemptible kernel and a spinlock cleanup.  I don't think the
> limitations of some PC hardware should limit our goals for Linux.

Its more than a spinlock cleanup at that point. To do anything useful you have
to tackle both priority inversion and some kind of at least semi-formal 
validation of the code itself. At the point it comes down to validating the
code I'd much rather validate rtlinux than the entire kernel

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Fri, 2002-01-11 at 07:37, Alan Cox wrote:

> Its more than a spinlock cleanup at that point. To do anything useful you have
> to tackle both priority inversion and some kind of at least semi-formal 
> validation of the code itself. At the point it comes down to validating the
> code I'd much rather validate rtlinux than the entire kernel

The preemptible kernel plus the spinlock cleanup could really take us
far.  Having locked at a lot of the long-held locks in the kernel, I am
confident at least reasonable progress could be made.

Beyond that, yah, we need a better locking construct.  Priority
inversion could be solved with a priority-inheriting mutex, which we can
tackle if and when we want to go that route.  Not now.

I want to lay the groundwork for a better kernel.  The preempt-kernel
patch gives real-world improvements, it provides a smoother user desktop
experience -- just look at the positive feedback.  Most importantly,
however, it provides a framework for superior response with our standard
kernel in its standard programming model. 

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Fri, Jan 11, 2002 at 03:33:22PM -0500, Robert Love wrote:
> On Fri, 2002-01-11 at 07:37, Alan Cox wrote:
> The preemptible kernel plus the spinlock cleanup could really take us
> far.  Having locked at a lot of the long-held locks in the kernel, I am
> confident at least reasonable progress could be made.
> 
> Beyond that, yah, we need a better locking construct.  Priority
> inversion could be solved with a priority-inheriting mutex, which we can
> tackle if and when we want to go that route.  Not now.

Backing the car up to the edge of the cliff really gives us
good results. Beyond that, we could jump off the cliff
if we want to go that route.
Preempt leads to inheritance and inheritance leads to disaster.


All the numbers I've seen show Morton's low latency just works better. Are
there other numbers I should look at.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rob Landley]

On Friday 11 January 2002 09:50 pm, yodaiken@fsmlabs.com wrote:
> On Fri, Jan 11, 2002 at 03:33:22PM -0500, Robert Love wrote:
> > On Fri, 2002-01-11 at 07:37, Alan Cox wrote:
> > The preemptible kernel plus the spinlock cleanup could really take us
> > far.  Having locked at a lot of the long-held locks in the kernel, I am
> > confident at least reasonable progress could be made.
> >
> > Beyond that, yah, we need a better locking construct.  Priority
> > inversion could be solved with a priority-inheriting mutex, which we can
> > tackle if and when we want to go that route.  Not now.
>
> Backing the car up to the edge of the cliff really gives us
> good results. Beyond that, we could jump off the cliff
> if we want to go that route.
> Preempt leads to inheritance and inheritance leads to disaster.

I preempt leads to disaster than Linux can't do SMP.  Are you saying that's 
the case?

The preempt patch is really "SMP on UP".  If pre-empt shows up a problem, 
then it's a problem SMP users will see too.  If we can't take advantage of 
the existing SMP locking infrastructure to improve latency and interactive 
feel on UP machines, than SMP for linux DOES NOT WORK.

> All the numbers I've seen show Morton's low latency just works better. Are
> there other numbers I should look at.

This approach is basically a collection of heuristics.  The kernel has been 
profiled and everywhere a latency spike was found, a band-aid was put on it 
(an explicit scheduling point).  This doesn't say there aren't other latency 
spikes, just that with the collection of hardware and software being 
benchmarked, the latency spikes that were found have each had a band-aid 
individually applied to them.

This isn't a BAD thing.  If the benchmarks used to find latency spikes are at 
all like real-world use, then it helps real-world applications.  But of 
COURSE the benchmarks are going to look good, since tuning the kernel to 
those benchmarks is the way the patch was developed!

The majority of the original low latency scheduling point work is handled 
automatically by the SMP on UP kernel.  You don't NEED to insert scheduling 
points anywhere you aren't inside a spinlock.  So the SMP on UP patch makes 
most of the explicit scheduling point patch go away, accomplishing the same 
thing in a less intrusive manner.  (Yes, it makes all kernels act like SMP 
kernels for debugging purposes.  But you can turn it off for debugging if you 
want to, that's just another toggle in the magic sysreq menu.  And this isn't 
entirely a bad thing: applying the enormous UP userbase to the remaining SMP 
bugs is bound to squeeze out one or two more obscure ones, but those bugs DO 
exist already on SMP.)

However, what's left of the explicit scheduling work is still very useful.  
When you ARE inside a spinlock, you can't just schedule, you have to save 
state, drop the lock(s), schedule, re-acquire the locks, and reload your 
state in case somebody else diddled with the structures you were using.  This 
is a lot harder than just scheduling, but breaking up long-held locks like 
this helps SMP scalability, AND helps latency in the SMP-on-UP case.

So the best approach is a combination of the two patches.  SMP-on-UP for 
everything outside of spinlocks, and then manually yielding locks that cause 
problems.  Both Robert Love and Andrew Morton have come out in favor of each 
other's patches on lkml just in the past few days.  The patches work together 
quite well, and each wants to see the other's patch applied.

Rob

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Fri, Jan 11, 2002 at 03:22:08PM -0500, Rob Landley wrote:
> I preempt leads to disaster than Linux can't do SMP.  Are you saying that's 
> the case?
> 
> The preempt patch is really "SMP on UP".  If pre-empt shows up a problem, 

People keep repeating this, it must feel reassuring.

	/* in kernel mode: does it need a lock? */
	m = next_free_page_from_per_cpu_cache();

To start, preemptive means that the optimizations for SMP that reduce
locking by per cpu localization do not work. 
So, as I understand it:
	the preempt patch is really "crappy SMP on UP" 
may be correct. But what you wrote is not.
Did I miss something? That's not a rhetorical question - I recall
being wrong before so go ahead and explain what's wrong with my logic.
        

> then it's a problem SMP users will see too.  If we can't take advantage of 
> the existing SMP locking infrastructure to improve latency and interactive 
> feel on UP machines, than SMP for linux DOES NOT WORK.
> 
> > All the numbers I've seen show Morton's low latency just works better. Are
> > there other numbers I should look at.
> 
> This approach is basically a collection of heuristics.  The kernel has been 

One patch makes the numbers look good (sort of)

One patch does not but "improves feel" and breaks a exceptionally useful 
rule: per cpu data in kernel that is not touched by interrupt code does not
need to be locked.

The basic assumption of the preempt trick is that locking for SMP is 
based on the same principles as locking for preemption and that's completely
false.  

I believe that the preempt path leads inexorably to
mutex-with-stupid-priority-trick and that would be very unfortunate indeed.
It's unavoidable because sooner or later someone will find that preempt +
SCHED_FIFO leads to 
		niced app 1 in K mode gets Sem A
		SCHED_FIFO app prempts and blocks on  Sem A
		whoops! app 2 in K more preempts niced app 1

	Hey my DVD player has stalled, lets add sem_with_revolting_priority_trick!
	Why the hell is UP Windows XP3 blowing away my Linux box on DVD playing while
	Linux now runs with the grace and speed of IRIX? 
	And has anyone fixed all those mysterious hangs caused by the interesting 
	interaction of hundreds of preempted semaphores?


		



> profiled and everywhere a latency spike was found, a band-aid was put on it 
> (an explicit scheduling point).  This doesn't say there aren't other latency 
> spikes, just that with the collection of hardware and software being 
> benchmarked, the latency spikes that were found have each had a band-aid 
> individually applied to them.
> 
> This isn't a BAD thing.  If the benchmarks used to find latency spikes are at 
> all like real-world use, then it helps real-world applications.  But of 
> COURSE the benchmarks are going to look good, since tuning the kernel to 
> those benchmarks is the way the patch was developed!
> 
> The majority of the original low latency scheduling point work is handled 
> automatically by the SMP on UP kernel.  You don't NEED to insert scheduling 
> points anywhere you aren't inside a spinlock.  So the SMP on UP patch makes 
> most of the explicit scheduling point patch go away, accomplishing the same 
> thing in a less intrusive manner.  (Yes, it makes all kernels act like SMP 
> kernels for debugging purposes.  But you can turn it off for debugging if you 
> want to, that's just another toggle in the magic sysreq menu.  And this isn't 
> entirely a bad thing: applying the enormous UP userbase to the remaining SMP 
> bugs is bound to squeeze out one or two more obscure ones, but those bugs DO 
> exist already on SMP.)

This is the logic of _every_ variant of "lets put X windows in the kernel
and let the kernel hackers fix it" .  Konquerer crashed when I used it 
yesterday. Let's put it in the kernel too and apply that enormous UP
userbase to the remaining bugs.

> 
> However, what's left of the explicit scheduling work is still very useful.  
> When you ARE inside a spinlock, you can't just schedule, you have to save 
> state, drop the lock(s), schedule, re-acquire the locks, and reload your 
> state in case somebody else diddled with the structures you were using.  This 
> is a lot harder than just scheduling, but breaking up long-held locks like 
> this helps SMP scalability, AND helps latency in the SMP-on-UP case.
> 
> So the best approach is a combination of the two patches.  SMP-on-UP for 
> everything outside of spinlocks, and then manually yielding locks that cause 
> problems.  Both Robert Love and Andrew Morton have come out in favor of each 
> other's patches on lkml just in the past few days.  The patches work together 
> quite well, and each wants to see the other's patch applied.


> 
> Rob

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> I believe that the preempt path leads inexorably to
> mutex-with-stupid-priority-trick and that would be very unfortunate indeed.
> It's unavoidable because sooner or later someone will find that preempt +
> SCHED_FIFO leads to
>                 niced app 1 in K mode gets Sem A
>                 SCHED_FIFO app prempts and blocks on  Sem A
>                 whoops! app 2 in K more preempts niced app 1

Please explain what's different without the preempt patch.

>         Hey my DVD player has stalled, lets add sem_with_revolting_priority_trick!
>         Why the hell is UP Windows XP3 blowing away my Linux box on DVD playing while
>         Linux now runs with the grace and speed of IRIX?

Because the IRIX implementation sucks, every implementation has to suck?
Somehow I have the suspicion you're trying to discourage everyone from
even trying, because if he'd succeeded you'd loose a big chunk of
potential RTLinux customers.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sat, Jan 12, 2002 at 12:53:06PM +0100, Roman Zippel wrote:
> Hi,
> 
> yodaiken@fsmlabs.com wrote:
> 
> > I believe that the preempt path leads inexorably to
> > mutex-with-stupid-priority-trick and that would be very unfortunate indeed.
> > It's unavoidable because sooner or later someone will find that preempt +
> > SCHED_FIFO leads to
> >                 niced app 1 in K mode gets Sem A
> >                 SCHED_FIFO app prempts and blocks on  Sem A
> >                 whoops! app 2 in K more preempts niced app 1
> 
> Please explain what's different without the preempt patch.

See that "preempt" in line 2 . Linux does not
preempt kernel mode processes otherwise. The beauty of the
non-preemptive kernel is that "in K mode every process makes progress"
and even the "niced app" will complete its use of SemA and 
release it in one run. If you have a reasonably fair scheduler you
can make very useful analysis with Linux now of the form

	Under 50 active proceses in the system means that in every
	2 second interval every process
	will get at least 10ms of time to run.

That's a very valuable property and it goes away in a preemptive kernel 
to get you something vague.


> 
> >         Hey my DVD player has stalled, lets add sem_with_revolting_priority_trick!
> >         Why the hell is UP Windows XP3 blowing away my Linux box on DVD playing while
> >         Linux now runs with the grace and speed of IRIX?
> 
> Because the IRIX implementation sucks, every implementation has to suck?
> Somehow I have the suspicion you're trying to discourage everyone from
> even trying, because if he'd succeeded you'd loose a big chunk of
> potential RTLinux customers.

So your argument is that I'm advocating Andrew Morton's patch which 
reduces latencies more than the preempt patch because I have a
financial interest in not reducing latencies? Subtle.

In any case, motive has no bearing on a technical argument. 
Your motive could be to make the 68K look better by reducing 
performance on other processors for all I know. 

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> > > SCHED_FIFO leads to
> > >                 niced app 1 in K mode gets Sem A
> > >                 SCHED_FIFO app prempts and blocks on  Sem A
> > >                 whoops! app 2 in K more preempts niced app 1
> >
> > Please explain what's different without the preempt patch.
> 
> See that "preempt" in line 2 . Linux does not
> preempt kernel mode processes otherwise. The beauty of the
> non-preemptive kernel is that "in K mode every process makes progress"
> and even the "niced app" will complete its use of SemA and
> release it in one run.

The point of using semaphores is that one can sleep while holding them,
whether this is forced by preemption or voluntary makes no difference.

> If you have a reasonably fair scheduler you
> can make very useful analysis with Linux now of the form
>
>         Under 50 active proceses in the system means that in every
>         2 second interval every process
>         will get at least 10ms of time to run.
> 
> That's a very valuable property and it goes away in a preemptive kernel
> to get you something vague.

How is that changed? AFAIK inserting more schedule points does not
change the behaviour of the scheduler. The niced app will still get its
time.

> So your argument is that I'm advocating Andrew Morton's patch which
> reduces latencies more than the preempt patch because I have a
> financial interest in not reducing latencies? Subtle.

Andrew's patch requires constant audition and Andrew can't audit all
drivers for possible problems. That doesn't mean Andrew's work is
wasted, since it identifies problems, which preempting can't solve, but
it will always be a hunt for the worst cases, where preempting goes for
the general case.

> In any case, motive has no bearing on a technical argument.
> Your motive could be to make the 68K look better by reducing
> performance on other processors for all I know.

I am more than busy to keep it running (together with a few others, who
are left) and more important I make no money of it.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sat, Jan 12, 2002 at 02:25:03PM +0100, Roman Zippel wrote:
> Hi,
> 
> yodaiken@fsmlabs.com wrote:
> 
> > > > SCHED_FIFO leads to
> > > >                 niced app 1 in K mode gets Sem A
> > > >                 SCHED_FIFO app prempts and blocks on  Sem A
> > > >                 whoops! app 2 in K more preempts niced app 1
> > >
> > > Please explain what's different without the preempt patch.
> > 
> > See that "preempt" in line 2 . Linux does not
> > preempt kernel mode processes otherwise. The beauty of the
> > non-preemptive kernel is that "in K mode every process makes progress"
> > and even the "niced app" will complete its use of SemA and
> > release it in one run.
> 
> The point of using semaphores is that one can sleep while holding them,
> whether this is forced by preemption or voluntary makes no difference.

No. The point of using semaphores is that one can sleep while
_waiting_ for the resource. Sleeping while holding semaphores is
a different kettle of lampreys entirely.
And it makes a very big difference
A:
	get sem on memory pool
		do something horrible to pool
	release sem on memory pool

In a preemptive kernel this can cause a deadlock. In a non
preemptive it cannot. You are correct in that 
B:
	get sem on memory pool
		do potentially blocking operations
	release sem
is also dangerous - but I don't think that helps your case.
To fix B, we can enforce a coding rule - one of the reasons why
we have all those atomic ops in the kernel is to be able to 
avoid this problem.
To fix A in a preemptive kernel we need to start messing about with
priorities and that's a major error.
"The current kernel has too many places where processes
can sleep while holding semaphores so we should always have the 
potential of blocking with held semaphores" is, to me, a backwards
argument.

> > If you have a reasonably fair scheduler you
> > can make very useful analysis with Linux now of the form
> >
> >         Under 50 active proceses in the system means that in every
> >         2 second interval every process
> >         will get at least 10ms of time to run.
> > 
> > That's a very valuable property and it goes away in a preemptive kernel
> > to get you something vague.
> 
> How is that changed? AFAIK inserting more schedule points does not
> change the behaviour of the scheduler. The niced app will still get its
> time.

How many times can an app be preempted? In a non preempt kernel
is can be preempted during user mode at timer frequency and no more
and it cannot be preempted during kernel mode. So
	while(1){
		read mpeg data
		process
		write bitmap
		}

Assuming Andrew does not get too ambitious about read/write granularity, once this
process is scheduled on a non-preempt system it will always make progress. The non
preempt kernel says, "your kernel request will complete - if we have resources".
A preempt kernel says: "well, if nobody more important activates you get some time"
Now you do the analysis based on the computation of "goodness" to show that there is
a bound on preemption count during an execution of this process. I don't want to 
have to think that hard. 
Let's suppose the Gnome desktop constantly creates and 
destroys new fresh i/o bound tasks to do something. So with the old fashioned non
preempt (ignoring Andrew) we get
			wait no more than 1 second
			I'm scheduled and start a read 
			wait no more than one second
			I'm scheduled and in user mode for at least 10milliseconds
			wait no more than 1 second
			I'm scheduled and do my write
			...
with preempt we get
			wait no more than 1 second
			I'm scheduled and start a read 
				I'm preempted
				read not done
				come back for 2 microseconds
				preempted again
				haven't issued the damn read request yet 
				ok a miracle happens, I finish the read request
				go to usermode and an interrupt happens
						well it would be stupid to have a goodness
						function in a preempt kernel that lets a low
						priority task finish its time slice so preempt
				...

> 
> > So your argument is that I'm advocating Andrew Morton's patch which
> > reduces latencies more than the preempt patch because I have a
> > financial interest in not reducing latencies? Subtle.
> 
> Andrew's patch requires constant audition and Andrew can't audit all
> drivers for possible problems. That doesn't mean Andrew's work is
> wasted, since it identifies problems, which preempting can't solve, but
> it will always be a hunt for the worst cases, where preempting goes for
> the general case.

the preempt requires constant auditing too - and more complex auditing.
After all, a missed audit in Andrew will simply increase worst case timing.
A missed audit in preempt will hang the system.

> 
> > In any case, motive has no bearing on a technical argument.
> > Your motive could be to make the 68K look better by reducing
> > performance on other processors for all I know.
> 
> I am more than busy to keep it running (together with a few others, who
> are left) and more important I make no money of it.

Come on! First of all, you are causing me a great deal of pain by making
me struggle not to make some bad joke about the economics of Linux companies.
More important, not making money has nothing to do with purity of motivation -
don't you read this list?
And how do I know that you haven't got a stockpile of 68K boards that may
be worth big money once it's known that 68K linux is at the top of the heap?
Much less plausible money making schemes have been tried.

Seriously: for our business, a Linux kernel that can reliably run at millisecond
level latencies is only good. If you could get a Linux kernel to run at 
latencies of 100 microseconds worst case on a 486, I'd be a little more
worried  but even then ...
On a 800Mhz Athlon, RTLinux scheduling jitter is 17microseconds worst case right now.


-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> No. The point of using semaphores is that one can sleep while
> _waiting_ for the resource.
> [...]
> In a preemptive kernel this can cause a deadlock. In a non
> preemptive it cannot. You are correct in that
> B:
>         get sem on memory pool
>                 do potentially blocking operations
>         release sem
> is also dangerous - but I don't think that helps your case.
> To fix B, we can enforce a coding rule - one of the reasons why
> we have all those atomic ops in the kernel is to be able to
> avoid this problem.

Sorry I can't follow you. First, one can sleep while waiting for the
semaphore _and_ while holding it. Second we use atomic ops (e.g. for
resource management) exactly because there are not protected by any
semaphore/spinlock.

> Let's suppose the Gnome desktop constantly creates and
> destroys new fresh i/o bound tasks to do something. So with the old fashioned non
> preempt (ignoring Andrew) we get
> [...]

There is no priority problem! If there is a more important task to run,
the less important one simply has to wait, but it will still get its
time. Your deadlock situation does not exists. The average time a
process has to wait for a lower priority process might be increased, but
the worst case behaviour is still the same.
The problem that does exist is the coarse time slice accounting, which
is easier to exploit with the preempt kernel, but it's not a new
problem. On the other hand it's a solvable problem, which requires no
priority inversion.

> > Andrew's patch requires constant audition and Andrew can't audit all
> > drivers for possible problems. That doesn't mean Andrew's work is
> > wasted, since it identifies problems, which preempting can't solve, but
> > it will always be a hunt for the worst cases, where preempting goes for
> > the general case.
> 
> the preempt requires constant auditing too - and more complex auditing.
> After all, a missed audit in Andrew will simply increase worst case timing.
> A missed audit in preempt will hang the system.

As long as the scheduler isn't changed, this isn't true and as I said
there are latency problems which preempting can't solve, but it will
automatically take care of the rest.

> Come on! First of all, you are causing me a great deal of pain by making
> me struggle not to make some bad joke about the economics of Linux companies.

Feel free, I'm not a big believer in the economics of software companies
in general, anyway.

> More important, not making money has nothing to do with purity of motivation -
> don't you read this list?

Everyone has its motivation and I do respect that, but I'm getting
suspicious as soon as money is involved. If people disagree, they can
still get along nicely and do their thing independently. But if they
have to make a living by getting a share of a cake, it usually only
works as long as there is enough cake, otherwise it can get nasty very
quickly (and usually there is never enough cake).

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sat, Jan 12, 2002 at 06:48:28PM +0100, Roman Zippel wrote:
> Hi,
> 
> yodaiken@fsmlabs.com wrote:
> 
> > No. The point of using semaphores is that one can sleep while
> > _waiting_ for the resource.
> > [...]
> > In a preemptive kernel this can cause a deadlock. In a non
> > preemptive it cannot. You are correct in that
> > B:
> >         get sem on memory pool
> >                 do potentially blocking operations
> >         release sem
> > is also dangerous - but I don't think that helps your case.
> > To fix B, we can enforce a coding rule - one of the reasons why
> > we have all those atomic ops in the kernel is to be able to
> > avoid this problem.
> 
> Sorry I can't follow you. First, one can sleep while waiting for the

We're having a write only discussion - time to stop.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> We're having a write only discussion - time to stop.

Sorry, but I'm still waiting for the proof that preempting deadlocks the
system.
If n running processes have together m time slices, after m ticks every
process will have run it's full share of the time, no matter how often
you schedule. (I assume here a correct time accounting, which is
currently not the case, but that's a different (and not new) problem.)
So even the low priority process will have the same time as before to do
it's job, it will be delayed, but it will not be delayed forever, so I'm
failing to see how preempting Linux should deadlock.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> So even the low priority process will have the same time as before to do
> it's job, it will be delayed, but it will not be delayed forever, so I'm
> failing to see how preempting Linux should deadlock.

First task scheduled takes a resource that a second task needs. 150 other
threads schedule via pre-emption, the one that it should share the resource
with cannot run but the rest do. Repeat. It doesn't deadlock but it goes
massively unfair

Re: [2.4.17/18pre] VM and swap - it's really unusable

[george anzinger]

yodaiken@fsmlabs.com wrote:
> 
> On Sat, Jan 12, 2002 at 02:25:03PM +0100, Roman Zippel wrote:
> > Hi,
> >
> > yodaiken@fsmlabs.com wrote:
> >
> > > > > SCHED_FIFO leads to
> > > > >                 niced app 1 in K mode gets Sem A
> > > > >                 SCHED_FIFO app prempts and blocks on  Sem A
> > > > >                 whoops! app 2 in K more preempts niced app 1
> > > >
> > > > Please explain what's different without the preempt patch.
> > >
> > > See that "preempt" in line 2 . Linux does not
> > > preempt kernel mode processes otherwise. The beauty of the
> > > non-preemptive kernel is that "in K mode every process makes progress"
> > > and even the "niced app" will complete its use of SemA and
> > > release it in one run.
> >
> > The point of using semaphores is that one can sleep while holding them,
> > whether this is forced by preemption or voluntary makes no difference.
> 
> No. The point of using semaphores is that one can sleep while
> _waiting_ for the resource. Sleeping while holding semaphores is
> a different kettle of lampreys entirely.
> And it makes a very big difference
> A:
>         get sem on memory pool
>                 do something horrible to pool
>         release sem on memory pool
> 
> In a preemptive kernel this can cause a deadlock. In a non
> preemptive it cannot. You are correct in that
> B:
>         get sem on memory pool
>                 do potentially blocking operations
>         release sem
> is also dangerous - but I don't think that helps your case.
> To fix B, we can enforce a coding rule - one of the reasons why
> we have all those atomic ops in the kernel is to be able to
> avoid this problem.
> To fix A in a preemptive kernel we need to start messing about with
> priorities and that's a major error.
> "The current kernel has too many places where processes
> can sleep while holding semaphores so we should always have the
> potential of blocking with held semaphores" is, to me, a backwards
> argument.
> 
> > > If you have a reasonably fair scheduler you
> > > can make very useful analysis with Linux now of the form
> > >
> > >         Under 50 active proceses in the system means that in every
> > >         2 second interval every process
> > >         will get at least 10ms of time to run.
> > >
> > > That's a very valuable property and it goes away in a preemptive kernel
> > > to get you something vague.
> >
> > How is that changed? AFAIK inserting more schedule points does not
> > change the behaviour of the scheduler. The niced app will still get its
> > time.
> 
> How many times can an app be preempted? In a non preempt kernel
> is can be preempted during user mode at timer frequency and no more

Uh, it can be and is preempted in user mode by ANY interrupt, be it
keyboard, serial, lan, disc, etc.  The kernel looks for need_resched at
the end of ALL interrupts, not just the timer interrupt.

> and it cannot be preempted during kernel mode. So
>         while(1){
>                 read mpeg data
>                 process
>                 write bitmap
>                 }
> 
> Assuming Andrew does not get too ambitious about read/write granularity, once this
> process is scheduled on a non-preempt system it will always make progress. The non
> preempt kernel says, "your kernel request will complete - if we have resources".
> A preempt kernel says: "well, if nobody more important activates you get some time"
> Now you do the analysis based on the computation of "goodness" to show that there is
> a bound on preemption count during an execution of this process. I don't want to
> have to think that hard.
> Let's suppose the Gnome desktop constantly creates and
> destroys new fresh i/o bound tasks to do something. So with the old fashioned non
> preempt (ignoring Andrew) we get
>                         wait no more than 1 second
>                         I'm scheduled and start a read
>                         wait no more than one second
>                         I'm scheduled and in user mode for at least 10milliseconds
>                         wait no more than 1 second
>                         I'm scheduled and do my write
>                         ...
> with preempt we get
>                         wait no more than 1 second
>                         I'm scheduled and start a read
>                                 I'm preempted
>                                 read not done
>                                 come back for 2 microseconds
>                                 preempted again
>                                 haven't issued the damn read request yet
>                                 ok a miracle happens, I finish the read request
>                                 go to usermode and an interrupt happens
>                                                 well it would be stupid to have a goodness
>                                                 function in a preempt kernel that lets a low
>                                                 priority task finish its time slice so preempt
>                                 ...
> 
> >
> > > So your argument is that I'm advocating Andrew Morton's patch which
> > > reduces latencies more than the preempt patch because I have a
> > > financial interest in not reducing latencies? Subtle.
> >
> > Andrew's patch requires constant audition and Andrew can't audit all
> > drivers for possible problems. That doesn't mean Andrew's work is
> > wasted, since it identifies problems, which preempting can't solve, but
> > it will always be a hunt for the worst cases, where preempting goes for
> > the general case.
> 
> the preempt requires constant auditing too - and more complex auditing.
> After all, a missed audit in Andrew will simply increase worst case timing.
> A missed audit in preempt will hang the system.
> 
> >
> > > In any case, motive has no bearing on a technical argument.
> > > Your motive could be to make the 68K look better by reducing
> > > performance on other processors for all I know.
> >
> > I am more than busy to keep it running (together with a few others, who
> > are left) and more important I make no money of it.
> 
> Come on! First of all, you are causing me a great deal of pain by making
> me struggle not to make some bad joke about the economics of Linux companies.
> More important, not making money has nothing to do with purity of motivation -
> don't you read this list?
> And how do I know that you haven't got a stockpile of 68K boards that may
> be worth big money once it's known that 68K linux is at the top of the heap?
> Much less plausible money making schemes have been tried.
> 
> Seriously: for our business, a Linux kernel that can reliably run at millisecond
> level latencies is only good. If you could get a Linux kernel to run at
> latencies of 100 microseconds worst case on a 486, I'd be a little more
> worried  but even then ...
> On a 800Mhz Athlon, RTLinux scheduling jitter is 17microseconds worst case right now.
> 
> --
> ---------------------------------------------------------
> Victor Yodaiken
> Finite State Machine Labs: The RTLinux Company.
>  www.fsmlabs.com  www.rtlinux.com
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 03:21:14PM -0800, george anzinger wrote:
> > > How is that changed? AFAIK inserting more schedule points does not
> > > change the behaviour of the scheduler. The niced app will still get its
> > > time.
> > 
> > How many times can an app be preempted? In a non preempt kernel
> > is can be preempted during user mode at timer frequency and no more
> 
> Uh, it can be and is preempted in user mode by ANY interrupt, be it
> keyboard, serial, lan, disc, etc.  The kernel looks for need_resched at
> the end of ALL interrupts, not just the timer interrupt.


Ouch.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Helge Hafting]

yodaiken@fsmlabs.com wrote:
> 
> On Mon, Jan 14, 2002 at 03:21:14PM -0800, george anzinger wrote:
> > > > How is that changed? AFAIK inserting more schedule points does not
> > > > change the behaviour of the scheduler. The niced app will still get its
> > > > time.
> > >
> > > How many times can an app be preempted? In a non preempt kernel
> > > is can be preempted during user mode at timer frequency and no more
> >
> > Uh, it can be and is preempted in user mode by ANY interrupt, be it
> > keyboard, serial, lan, disc, etc.  The kernel looks for need_resched at
> > the end of ALL interrupts, not just the timer interrupt.
> 
> Ouch.

Ouch?  It is supposed to be that way.  Consider:

A high-priority task issues a disk read - and blocks.  Some
lower-priority process gets the cpu.  But then the disk io finishes
way before the low-priority process used up its timeslice.
The kernel gets an interrupt from the disk controller because
of that.  Perhaps the block device issues some more requests,
then time comes to return to user space.  The higher priority task
is now ready to run because its IO completed.  So of course
it is preferred over that low-priority thing.  In other words,
the low-priority task got preempted, this time by a disk
interrupt.

The same thing happens whan high-priority tasks waits for
other kinds of io, such as network, serial, and so on.
I am sure you wouldn't want it any other way.  Not
using the opportunity to switch task immediately after an io
completion interrupt would kill latency completely.

Helge Hafting

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Roman Zippel wrote:
> 
> Andrew's patch requires constant audition and Andrew can't audit all
> drivers for possible problems. That doesn't mean Andrew's work is
> wasted, since it identifies problems, which preempting can't solve, but
> it will always be a hunt for the worst cases, where preempting goes for
> the general case.

Guys,

I've heard this so many times, and it just ain't so.   The overwhelming
majority of problem areas are inside locks.  All the complexity and 
maintainability difficulties to which you refer exist in the preempt
patch as well.    There just is no difference.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> >         Hey my DVD player has stalled, lets add sem_with_revolting_priority_trick!
> >         Why the hell is UP Windows XP3 blowing away my Linux box on DVD playing while
> >         Linux now runs with the grace and speed of IRIX?
> 
> Because the IRIX implementation sucks, every implementation has to suck?
> Somehow I have the suspicion you're trying to discourage everyone from
> even trying, because if he'd succeeded you'd loose a big chunk of
> potential RTLinux customers.

Victor has had the same message for years, as have others like Larry McVoy
(in fact if Larry and Victor agree on something its unusual enough to
 remember). So I can vouch for the fact Victor hasn't changed his tune from
before rtlinux was ever any real commercial toy. I think you owe him an
apology.

Now rtlinux and low latency in the main kernel are two different things. One
gives you effectively a small embedded system to program for which talks
to Linux. From that you draw extremely reliable behaviour and very bounded
delay times. Its small enough you can validate it too

RtLinux isn't going to help you one bit when it comes to smooth movie playback 
because the DVD playback is dependant on the Linux file system layers and a
whole pile of other code. Low-latency does this quite nicely, and it takes
you to the point where hardware becomes the biggest latency cause for the
general case. Pre-empt doesn't buy you anything more. You can spend a
millisecond locked in an I/O instruction to an irritating device.

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

Alan Cox wrote:

> > Because the IRIX implementation sucks, every implementation has to suck?
> > Somehow I have the suspicion you're trying to discourage everyone from
> > even trying, because if he'd succeeded you'd loose a big chunk of
> > potential RTLinux customers.
> 
> Victor has had the same message for years, as have others like Larry McVoy
> (in fact if Larry and Victor agree on something its unusual enough to
>  remember). So I can vouch for the fact Victor hasn't changed his tune from
> before rtlinux was ever any real commercial toy. I think you owe him an
> apology.

Did I really say something that bad? I would be actually surprised, if
Victor wouldn't act in the best interest of his company. The other
possibility is that Victor must have had such a terrible experience with
IRIX, so that he thinks any attempts to add better soft realtime or even
hard realtime capabilities (not just as addon) must be doomed to fail.

> RtLinux isn't going to help you one bit when it comes to smooth movie playback
> because the DVD playback is dependant on the Linux file system layers and a
> whole pile of other code. Low-latency does this quite nicely, and it takes
> you to the point where hardware becomes the biggest latency cause for the
> general case. Pre-empt doesn't buy you anything more. You can spend a
> millisecond locked in an I/O instruction to an irritating device.

Preemption doesn't solve of course every problem. It's mainly useful to
get an event as fast as possible from kernel to user space. This can be
the mouse click or the buffer your process is waiting for. Latencies can
quickly sum up here to be sensible.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sat, Jan 12, 2002 at 09:42:26PM +0100, Roman Zippel wrote:
> Hi,
> 
> Alan Cox wrote:
> 
> > > Because the IRIX implementation sucks, every implementation has to suck?
> > > Somehow I have the suspicion you're trying to discourage everyone from
> > > even trying, because if he'd succeeded you'd loose a big chunk of
> > > potential RTLinux customers.
> > 
> > Victor has had the same message for years, as have others like Larry McVoy
> > (in fact if Larry and Victor agree on something its unusual enough to
> >  remember). So I can vouch for the fact Victor hasn't changed his tune from
> > before rtlinux was ever any real commercial toy. I think you owe him an
> > apology.
> 
> Did I really say something that bad? I would be actually surprised, if
> Victor wouldn't act in the best interest of his company. The other
> possibility is that Victor must have had such a terrible experience with
> IRIX, so that he thinks any attempts to add better soft realtime or even
> hard realtime capabilities (not just as addon) must be doomed to fail.

Well, how about a third possibility - that I see a problem you have not
seen and that you should try to argue on technical terms instead of psychoanlyzing
me or looking for financial motives?


-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> Well, how about a third possibility - that I see a problem you have not
> seen and that you should try to argue on technical terms

I just don't see any problem that is really new. Alan's example is one
of more extreme ones, but the only effect is that an operation can be
delayed far more than usual, but not indefinitely.
If you think preemption can cause a deadlock, maybe you could give me a
hint, which of the conditions for a deadlock is changed by preemption?

> instead of psychoanlyzing
> me or looking for financial motives?

If I had known, how easily people are offended by implying they could
act out of financial interest, I hadn't made that comment. Sorry, but
I'm just annoyed, how you attack any attempt to add realtime
capabilities to the kernel, mostly with the argument that it sucks under
IRIX. I people want to try it, let them. I prefer to see patches and if
they should really suck, I would be first one to say so.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sun, Jan 13, 2002 at 04:33:44AM +0100, Roman Zippel wrote:
> Hi,
> 
> yodaiken@fsmlabs.com wrote:
> 
> > Well, how about a third possibility - that I see a problem you have not
> > seen and that you should try to argue on technical terms
> 
> I just don't see any problem that is really new. Alan's example is one
> of more extreme ones, but the only effect is that an operation can be
> delayed far more than usual, but not indefinitely.
> If you think preemption can cause a deadlock, maybe you could give me a
> hint, which of the conditions for a deadlock is changed by preemption?
> 
> > instead of psychoanlyzing
> > me or looking for financial motives?
> 
> If I had known, how easily people are offended by implying they could
> act out of financial interest, I hadn't made that comment. Sorry, but
> I'm just annoyed, how you attack any attempt to add realtime
> capabilities to the kernel, mostly with the argument that it sucks under
> IRIX. I people want to try it, let them. I prefer to see patches and if
> they should really suck, I would be first one to say so.

I'm annoyed that you take a comment in which I said that the Morton approach
was much preferrable to the preempt patch and respond by saying I "attack
any attempt to add realtime capabilities to the kernel". 
I'm all in favor of people trying all sorts of things. My original comment
was that the numbers I'd seen all favored the Morton patch and I still
haven't seen any evidence to the contrary.

I also made two very simple and specific comments:
	1) I don't see how processor specific caching, which seems
	essential for smp performance and will be more essential 
	with numa, works with this patch
	2) preempt seems to lead inescapably to priority inherit. If this
	is true, people better understand the ramifications now before they
	commit.

Of course, I think there are strong limits to what you can get for RT 
performance in the kernel - I think the RTLinux method is far superior.
Believe what you want - it won't change the numbers.

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Preemption doesn't solve of course every problem. It's mainly useful to
> get an event as fast as possible from kernel to user space. This can be
> the mouse click or the buffer your process is waiting for. Latencies can
> quickly sum up here to be sensible.

The pre-emption patch doesn't change the average latencies. Go run some real
benchmarks. Its lost in the noise after the low latency patch. A single inw
from some I/O cards can cost as much as the latency target we hit.

Its not a case of the 90% of the result with 10% of the work, the pre-empt
patch is firmly in the all pain no gain camp

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Rob Landley wrote:
> 
> On Friday 11 January 2002 09:50 pm, yodaiken@fsmlabs.com wrote:
> > On Fri, Jan 11, 2002 at 03:33:22PM -0500, Robert Love wrote:
> > > On Fri, 2002-01-11 at 07:37, Alan Cox wrote:
> > > The preemptible kernel plus the spinlock cleanup could really take us
> > > far.  Having locked at a lot of the long-held locks in the kernel, I am
> > > confident at least reasonable progress could be made.
> > >
> > > Beyond that, yah, we need a better locking construct.  Priority
> > > inversion could be solved with a priority-inheriting mutex, which we can
> > > tackle if and when we want to go that route.  Not now.
> >
> > Backing the car up to the edge of the cliff really gives us
> > good results. Beyond that, we could jump off the cliff
> > if we want to go that route.
> > Preempt leads to inheritance and inheritance leads to disaster.
> 
> I preempt leads to disaster than Linux can't do SMP.  Are you saying that's
> the case?

Victor is referring to priority inheritance, to solve priority inversion.

Priority inheritance seems undesirable for Linux - these applications are
already in the minority.   A realtime application on Linux should simply
avoid complex system calls which can lead to blockage on a SCHED_OTHER
thread.

If the app is well-designed, the only place in which it is likely to
be unexpectedly blocked inside the kernel is in the page allocator.
My approach to this problem is to cause non-SCHED_OTHER processes
to perform atomic (non-blocking) memory allocations, with a fallback
to non-atomic.

> The preempt patch is really "SMP on UP".  If pre-empt shows up a problem,
> then it's a problem SMP users will see too.  If we can't take advantage of
> the existing SMP locking infrastructure to improve latency and interactive
> feel on UP machines, than SMP for linux DOES NOT WORK.
> 
> > All the numbers I've seen show Morton's low latency just works better. Are
> > there other numbers I should look at.
> 
> This approach is basically a collection of heuristics.  The kernel has been
> profiled and everywhere a latency spike was found, a band-aid was put on it
> (an explicit scheduling point).  This doesn't say there aren't other latency
> spikes, just that with the collection of hardware and software being
> benchmarked, the latency spikes that were found have each had a band-aid
> individually applied to them.

The preempt patch needs all this as well.
 
> This isn't a BAD thing.  If the benchmarks used to find latency spikes are at
> all like real-world use, then it helps real-world applications.  But of
> COURSE the benchmarks are going to look good, since tuning the kernel to
> those benchmarks is the way the patch was developed!
> 
> The majority of the original low latency scheduling point work is handled
> automatically by the SMP on UP kernel.

No it is not.

The preempt code only obsoletes a handful of the low-latency patch's
resceduling.  The most trivial ones.  generic_file_read, generic_file_write
and a couple of /proc functions.

Of the sixty or so rescheduling points in the low-latency patch, about
fifty are inside locks.  Many of these are just lock_kernel().  About
half are not.

>  You don't NEED to insert scheduling
> points anywhere you aren't inside a spinlock.

I know of only four or five places in the kernel where large amount of
time are spent in unlocked code.  All the other problem areas are inside locks.

>  So the SMP on UP patch makes
> most of the explicit scheduling point patch go away,

s/most/a trivial minority/

> accomplishing the same
> thing in a less intrusive manner.

s/less/more/

> (Yes, it makes all kernels act like SMP
> kernels for debugging purposes.  But you can turn it off for debugging if you
> want to, that's just another toggle in the magic sysreq menu.  And this isn't
> entirely a bad thing: applying the enormous UP userbase to the remaining SMP
> bugs is bound to squeeze out one or two more obscure ones, but those bugs DO
> exist already on SMP.)

Saying "it's a config option" is a cop-out.  The kernel developers should
be aiming at producing a piece of software which can be shrink-wrap
deployed to millions of people.

Arguably, enabling it on UP and disabling it on SMP may be a sensible
approach, meraly because SMP tends to map onto applications which
do not require lower latencies.
  
> However, what's left of the explicit scheduling work is still very useful.
> When you ARE inside a spinlock, you can't just schedule, you have to save
> state, drop the lock(s), schedule, re-acquire the locks, and reload your
> state in case somebody else diddled with the structures you were using.  This
> is a lot harder than just scheduling, but breaking up long-held locks like
> this helps SMP scalability, AND helps latency in the SMP-on-UP case.

Yes, it _may_ help SMP scalability.  But a better approach is to replace
spinlocks with rwlocks when a lock is fond to have this access pattern.
 
> So the best approach is a combination of the two patches.  SMP-on-UP for
> everything outside of spinlocks, and then manually yielding locks that cause
> problems.

Well the ideal approach is to simply make the long-running locked code
faster, by better choice of algorithm and data structure.  Unfortunately,
in the majority of cases, this isn't possible.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Jussi Laako]

Andrew Morton wrote:
> 
> Priority inheritance seems undesirable for Linux - these applications are
> already in the minority.   A realtime application on Linux should simply
> avoid complex system calls which can lead to blockage on a SCHED_OTHER
> thread.

I think it's very common to have SCHED_FIFO thread communicating with
various other processes through pipe/fifo/socket or some other IPC
mechanism.

It would be great to have priority inheritance where process receiving data
through fifo from SCHED_FIFO process would have raised priority for transfer
time. (see QNX priority inheriting message queues) Too bad we don't have
message queues so we could have send/receive/reply time priority
inheritance.

So we could have

Process 1 at SCHED_FIFO sending data to two processes.
Process 2 at SCHED_FIFO receiving data from process 1.
Process 3 at SCHED_OTHER receiving data from process 1.
Process 4 at SCHED_OTHER sending data to process 5.
Process 5 at SCHED_OTHER receiving data from process 4.

And (2) would get the data first from (1) and then (3). And if (1) starts
sending data to (2) system would immediately start running (1/2) and even
pre-empt the ongoing system call of (3). Also (1/3) would take over/pre-empt
(4/5) because (3) inherits priority from sending process (1).

If this is currently _not_ done I think it's very strange.

But I think I have misunderstood the whole point of original message... :)


 - Jussi Laako

-- 
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Fri, 2002-01-11 at 15:22, Rob Landley wrote:

> So the best approach is a combination of the two patches.  SMP-on-UP for 
> everything outside of spinlocks, and then manually yielding locks that cause 
> problems.  Both Robert Love and Andrew Morton have come out in favor of each 
> other's patches on lkml just in the past few days.  The patches work together 
> quite well, and each wants to see the other's patch applied.

Right.  Here is what I want for 2.5 as a _general_ step towards a better
kernel that will yield better performance:

Merge the preemptible kernel patch.  A version is now out for
2.5.2-pre11 with support for Ingo's scheduler:

	ftp://ftp.kernel.org/pub/linux/kernel/people/rml/preempt-kernel

Next, make available a tool for profiling kernel latencies.  I have one
available now, preempt-stats, at the above url.  Andrew has some
excellent tools available at his website, too.  Something like this
could even be merged.  Daniel Phillips suggested a passive tool on IRC. 
Preempt-stats works like this.  It is off-by-default and, when enabled,
measures time between lock and unlock, reporting the top 20 worst-cases.

Begin working on the worst-case locks.  Solutions like Andrew's
low-latency and my lock-break are a start.  Better (at least in general)
solutions are to analyze the locks.  Localize them; make them finer
grained.  Analyze the algorithms.  Find the big problems.  Anyone look
at the tty layer lately?  Ugh.  Using the preemptive kernel as a base
and the analysis of the locks as a list of culprits, clean this cruft
up.  This would benefit SMP, too.  Perhaps a better locking construct is
useful.

The immediate result is good; the future is better.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sat, Jan 12, 2002 at 01:01:39AM -0500, Robert Love wrote:
> could even be merged.  Daniel Phillips suggested a passive tool on IRC. 
> Preempt-stats works like this.  It is off-by-default and, when enabled,
> measures time between lock and unlock, reporting the top 20 worst-cases.

I think one of the problems with this entire debate is lack of meaningful
numbers. Not for the first time, I propose that you test with something
that tests application benefits instead of internal numbers that may not
mean anything. For example, there is a simple test
		/* user code */
		get time.
		count = 200*3600; /* one hour */
		while(count--){
			read cycle timer
			clock_nanosleep(5 milliseconds)
			read cycle timer
			compute actual delay and difference from 5 milliseconds
			store the worst case
			}
		get time.
		printf("After one hour the worst deviation is %d clock ticks\n",worst);
		printf("This was supposed to take one hour and it took %d", compute_elapsed());

		

> 
> Begin working on the worst-case locks.  Solutions like Andrew's
> low-latency and my lock-break are a start.  Better (at least in general)
> solutions are to analyze the locks.  Localize them; make them finer
> grained.  Analyze the algorithms.  Find the big problems.  Anyone look

The theory that "fine grained = better" is not proved. It's obvious that
"fine grained = more time spent in the overhead of locking and unlocking locks and
		potentially more time spent in lock contention
		and lots more opportunities of cache ping-pong in real smp
		and much harder to debug"
But the performance gain that is supposed to balance that is often elusive.



> at the tty layer lately?  Ugh.  Using the preemptive kernel as a base
> and the analysis of the locks as a list of culprits, clean this cruft
> up.  This would benefit SMP, too.  Perhaps a better locking construct is
> useful.
> 
> The immediate result is good; the future is better.

Removing synchronization by removing contention 
is better engineering than fiddling about with synchronization
primitives, but it is much harder.


-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Right.  Here is what I want for 2.5 as a _general_ step towards a better
> kernel that will yield better performance:

I see absolutely _no_ evidence to support this repeated claim. I'm still
waiting to see any evidence that low latency patches are not sufficient, or
an explanation of who is going to fix all the drivers you break in subtle
ways

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sat, 2002-01-12 at 14:00, Alan Cox wrote:

> I see absolutely _no_ evidence to support this repeated claim. I'm still
> waiting to see any evidence that low latency patches are not sufficient, or
> an explanation of who is going to fix all the drivers you break in subtle
> ways

I'll work on fixing things the patch breaks.  I don't think it will be
that bad.  I've been working on preemption for a long long time, and
before me others have been working for a long long time, and I just
don't see the hordes of broken drivers or the tons of race-conditions
due to per-CPU data.  I have seen some, and I have fixed them.

For a solution to latency concerns, I'd much prefer to lay a framework
down that provides a proper solution and then work on fine tuning the
kernel to get the desired latency out of it.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> For a solution to latency concerns, I'd much prefer to lay a framework
> down that provides a proper solution and then work on fine tuning the
> kernel to get the desired latency out of it.

As the low latency patch proves, the framework has always been there, the
ll patches do the rest

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 13, 2002 02:41 am, Alan Cox wrote:
> [somebody wrote]
> > For a solution to latency concerns, I'd much prefer to lay a framework
> > down that provides a proper solution and then work on fine tuning the
> > kernel to get the desired latency out of it.
> 
> As the low latency patch proves, the framework has always been there, the
> ll patches do the rest

For that matter, the -preempt patch proves that the framework has always - 
i.e., since genesis of SMP - been there for a preemptible kernel.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[arjan]

In article <20020112042404.WCSI23959.femail47.sdc1.sfba.home.com@there> you wrote:

> The preempt patch is really "SMP on UP".

Ok I've seen this misconception quite a lot now. THIS IS NOT TRUE. For one,
constructs that are ok on SMP are not automatically ok with the -preempt
patch, like per-cpu data. And there's a LOT more of that than you think.
Basically with preempt you change the locking rules from under all existing
code. Most will work, even more will appear to work as preemption isn't
an event THAT common (by this I mean the chance of getting preempted in your
4 lines of C code where you have per cpu data).

Also, once you add locks around the  per-cpu data, for the core code it
might be close to smp. For drivers it's not though. Drivers assume that when
they do

outb(foo,bar);
outb(foo2,bar2);

that those happen "close" to eachother in time. Especially in initialisation
paths (where the driver thread is the only thread that can see the
datastructures/device) there's no spinlocks helt so preempt can trigger
here. Sure in the current situation you can get an interrupt but the linux
interrupt delay is not more than, say, 1ms while a schedule-out can take a
second or two easily. Do we know all devices can stand such delays ?
I dare to say we don't as the hardware requirements currently aren't coded
in the drivers.

Add to that that there's no actual benefit of -preempt over the -lowlat
patch latency wise (you REALLY need to combine them or -preempt sucks raw
eggs for latency)....

Greetings,
   Arjan van de Ven

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

Another example is in the network drivers. The 8390 core for one example
carefully disables an IRQ on the card so that it can avoid spinlocking on 
uniprocessor boxes.

So with pre-empt this happens

	driver magic
	disable_irq(dev->irq)
PRE-EMPT:
	[large periods of time running other code]
PRE-EMPT:
	We get back and we've missed 300 packets, the serial port sharing
	the IRQ has dropped our internet connection completely.

["Don't do that then" isnt a valid answer here. If I did hold a lock
 it would be for several milliseconds at a time anyway and would reliably
 trash performance this time]

There are numerous other examples in the kernel tree where the current code
knows that there is a small bounded time between two actions in kernel space
that do not have a sleep. They are not spin locked, and putting spin locks
everywhere will just trash performance. They are pure hardware interactions
so you can't automatically detect them.

That is why the pre-empt code is a much much bigger problem and task than the
low latency code.

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ed Sweetman]

> Another example is in the network drivers. The 8390 core for one example
> carefully disables an IRQ on the card so that it can avoid spinlocking on
> uniprocessor boxes.
>
> So with pre-empt this happens
>
> driver magic
> disable_irq(dev->irq)
> PRE-EMPT:
> [large periods of time running other code]
> PRE-EMPT:
> We get back and we've missed 300 packets, the serial port sharing
> the IRQ has dropped our internet connection completely.
>
> ["Don't do that then" isnt a valid answer here. If I did hold a lock
>  it would be for several milliseconds at a time anyway and would reliably
>  trash performance this time

> There are numerous other examples in the kernel tree where the current
code
> knows that there is a small bounded time between two actions in kernel
space
> that do not have a sleep. They are not spin locked, and putting spin locks
> everywhere will just trash performance. They are pure hardware
interactions
> so you can't automatically detect them.

hardware to hardware could have a higher priority than normal programs being
run.   That way they're not preempted by simple programs, it would have to
be purposely preempted by the user.

> That is why the pre-empt code is a much much bigger problem and task than
the
> low latency code.

Lowering the latency, sure the low latency code probably does nearly as well
as the preempt patch.  that's fine.  Shortening the time locks are held by
better code can help to a certain extent (unless a lot of the kernel code is
poorly written, which i doubt).  at it's present state though,  my idea to
fix the kernel would be to give parts of the kernel where locks are made,
that shouldn't be broken normally, higher priorities.  That way we can
distinguish between safe locks to preempt at and the ones that can do harm.
But those people who require their app to be treated special can run it
with -20 and preempt everything.   To me that makes sense.  Is there a
reason why it doesn't?  Besides ethstetics.   the only way the ethsetic
argument people are going to be pleased is if the kernel is designed from
the ground up to be better latency and lock-wise.   A lot of people would
like to not have to wait until that time in the meantime.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sat, Jan 12, 2002 at 02:23:00PM -0500, Ed Sweetman wrote:
> hardware to hardware could have a higher priority than normal programs being
> run.   That way they're not preempted by simple programs, it would have to
> be purposely preempted by the user.

Priority is currently, and sensibly, by process. A process may run user
code, do sys-calls, or field interrupts both soft and hard. Now do you want to
adjust the priority at every transition?

> Lowering the latency, sure the low latency code probably does nearly as well
> as the preempt patch.  that's fine.  Shortening the time locks are held by
> better code can help to a certain extent (unless a lot of the kernel code is
> poorly written, which i doubt).  at it's present state though,  my idea to
> fix the kernel would be to give parts of the kernel where locks are made,

"Fix" what? What is the objective of your fix?


> that shouldn't be broken normally, higher priorities.  That way we can
> distinguish between safe locks to preempt at and the ones that can do harm.
> But those people who require their app to be treated special can run it
> with -20 and preempt everything.   To me that makes sense.  Is there a

So:
	get semaphore on slab memory and raise priority
		get preempted by "treated special" app that then
			does an operation on the slab queues 

Is that what you want?
		
> reason why it doesn't?  Besides ethstetics.   the only way the ethsetic

It doesn't work? Is that a sufficient reason?


-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> hardware to hardware could have a higher priority than normal programs being
> run.   That way they're not preempted by simple programs, it would have to
> be purposely preempted by the user.

How do you know they are there. How do you detect the situation, or do you
plan to audit every driver ?

> Lowering the latency, sure the low latency code probably does nearly as well
> as the preempt patch.  that's fine.  Shortening the time locks are held by

Not nearly as well. The tests I've seen it runs _better_ than just pre-empt
and pre-empt + low latency is the same as pure low latency - 1mS

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Pavel Machek]

Hi!

> > hardware to hardware could have a higher priority than normal programs being
> > run.   That way they're not preempted by simple programs, it would have to
> > be purposely preempted by the user.
> 
> How do you know they are there. How do you detect the situation, or do you
> plan to audit every driver ?

Any driver which depends on timing is broken. 2.4.9 was happy to spend
two seconds in interrupt (console switch). So I doubt too much drivers
are broken like that. 

And... The drivers were broken already. That is not reason against the
patch!
									Pavel
-- 
(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sat, 2002-01-12 at 13:54, Alan Cox wrote:
> Another example is in the network drivers. The 8390 core for one example
> carefully disables an IRQ on the card so that it can avoid spinlocking on 
> uniprocessor boxes.
> 
> So with pre-empt this happens
> 
> 	driver magic
> 	disable_irq(dev->irq)
> PRE-EMPT:
> 	[large periods of time running other code]
> PRE-EMPT:
> 	We get back and we've missed 300 packets, the serial port sharing
> 	the IRQ has dropped our internet connection completely.

We don't preempt while IRQ are disabled.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sat, Jan 12, 2002 at 02:26:27PM -0500, Robert Love wrote:
> On Sat, 2002-01-12 at 13:54, Alan Cox wrote:
> > Another example is in the network drivers. The 8390 core for one example
> > carefully disables an IRQ on the card so that it can avoid spinlocking on 
> > uniprocessor boxes.
> > 
> > So with pre-empt this happens
> > 
> > 	driver magic
> > 	disable_irq(dev->irq)
> > PRE-EMPT:
> > 	[large periods of time running other code]
> > PRE-EMPT:
> > 	We get back and we've missed 300 packets, the serial port sharing
> > 	the IRQ has dropped our internet connection completely.
> 
> We don't preempt while IRQ are disabled.

You read the mask map? and somehow figure out which masked irqs correspond to 
active devices?

> 
> 	Robert Love
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> > PRE-EMPT:
> > 	We get back and we've missed 300 packets, the serial port sharing
> > 	the IRQ has dropped our internet connection completely.
> 
> We don't preempt while IRQ are disabled.

I must have missed that in the code. I can see you check __cli() status but
I didn't see anywhere you check disable_irq(). Even if you did it doesnt
help when I mask the irq on the chip rather than using disable_irq() calls.

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sat, 2002-01-12 at 15:07, Alan Cox wrote:

> > We don't preempt while IRQ are disabled.
> 
> I must have missed that in the code. I can see you check __cli() status but
> I didn't see anywhere you check disable_irq(). Even if you did it doesnt
> help when I mask the irq on the chip rather than using disable_irq() calls.

Well, if IRQs are disabled we won't have the timer... would not the
system panic anyhow if schedule() was called while in an interrupt
handler?

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> > I didn't see anywhere you check disable_irq(). Even if you did it doesnt
> > help when I mask the irq on the chip rather than using disable_irq() calls.
> 
> Well, if IRQs are disabled we won't have the timer... would not the
> system panic anyhow if schedule() was called while in an interrupt
> handler?

You completely misunderstand.

	disable_irq(n)

I disable a single specific interrupt, I don't disable the timer interrupt.
Your code doesn't seem to handle that. Its just one of the examples of where
you really need priority handling, and thats a horrible dark and slippery
slope

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sat, 2002-01-12 at 15:21, Alan Cox wrote:

> You completely misunderstand.
> 
> 	disable_irq(n)
> 
> I disable a single specific interrupt, I don't disable the timer interrupt.
> Your code doesn't seem to handle that.

It can if we increment the preempt_count in disable_irq_nosync and
decrement it on enable_irq.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Russell King]

On Sat, Jan 12, 2002 at 10:10:55PM -0500, Robert Love wrote:
> It can if we increment the preempt_count in disable_irq_nosync and
> decrement it on enable_irq.

Who says you're going to be enabling IRQs any time soon?  AFAIK, there is
nothing that requires enable_irq to be called after disable_irq_nosync.

In fact, you could well have the following in a driver:

	/* initial shutdown of device */

	disable_irq_nosync(i); /* or disable_irq(i); */

	/* other shutdown stuff */

	free_irq(i, private);

and you would have to audit all drivers to find out if they did this -
this would seriously damage your preempt_count.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 06:39, Russell King wrote:
> On Sat, Jan 12, 2002 at 10:10:55PM -0500, Robert Love wrote:
> > It can if we increment the preempt_count in disable_irq_nosync and
> > decrement it on enable_irq.
> 
> Who says you're going to be enabling IRQs any time soon?  AFAIK, there is
> nothing that requires enable_irq to be called after disable_irq_nosync.
> 
> In fact, you could well have the following in a driver:
> 
> 	/* initial shutdown of device */
> 
> 	disable_irq_nosync(i); /* or disable_irq(i); */
> 
> 	/* other shutdown stuff */
> 
> 	free_irq(i, private);
> 
> and you would have to audit all drivers to find out if they did this -
> this would seriously damage your preempt_count.

I wasn't thinking.  Anytime we are in an interrupt handler, preemption
is disabled.  Regardless of how (or even if) interrupts are disabled. 
We bump preempt_count on the entry path.  So, no problem.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Russell King]

On Sun, Jan 13, 2002 at 01:24:20PM -0500, Robert Love wrote:
> On Sun, 2002-01-13 at 06:39, Russell King wrote:
> > On Sat, Jan 12, 2002 at 10:10:55PM -0500, Robert Love wrote:
> > > It can if we increment the preempt_count in disable_irq_nosync and
> > > decrement it on enable_irq.
> > 
> > Who says you're going to be enabling IRQs any time soon?  AFAIK, there is
> > nothing that requires enable_irq to be called after disable_irq_nosync.
> > 
> > In fact, you could well have the following in a driver:
> > 
> > 	/* initial shutdown of device */
> > 
> > 	disable_irq_nosync(i); /* or disable_irq(i); */
> > 
> > 	/* other shutdown stuff */
> > 
> > 	free_irq(i, private);
> > 
> > and you would have to audit all drivers to find out if they did this -
> > this would seriously damage your preempt_count.
> 
> I wasn't thinking.  Anytime we are in an interrupt handler, preemption
> is disabled.  Regardless of how (or even if) interrupts are disabled. 
> We bump preempt_count on the entry path.  So, no problem.

Err.  This isn't *inside* an interrupt handler.  This could well be in
the driver shutdown code (eg, when fops->release is called).

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> I wasn't thinking.  Anytime we are in an interrupt handler, preemption
> is disabled.  Regardless of how (or even if) interrupts are disabled. 
> We bump preempt_count on the entry path.  So, no problem.

The code path isnt in an interrupt handler.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> > I disable a single specific interrupt, I don't disable the timer interrupt.
> > Your code doesn't seem to handle that.
> 
> It can if we increment the preempt_count in disable_irq_nosync and
> decrement it on enable_irq.

A driver that knows about how its irq is handled and that it is sole
user (eg ISA) may and some do leave it disabled for hours at a time

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 10:59, Alan Cox wrote:
> > I disable a single specific interrupt, I don't disable the timer interrupt.
> > Your code doesn't seem to handle that.
> 
> It can if we increment the preempt_count in disable_irq_nosync and
> decrement it on enable_irq.

OK, Alan, you spooked me with the disable_irq mess and admittedly my
initial solution wasn't ideal for a few reasons.

But it isn't a problem after all. In hw_irq.h we bump the count in the
interrupt path.  This should handle any handler, however we end up in
it.

I realized it because if we did not have a global solution to interrupt
request handlers, dropping spinlocks in the handler, even with IRQs
disabled, would cause a preemptive schedule.  All interrupts are
properly protected.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 13, 2002 04:59 pm, Alan Cox wrote:
> > > I disable a single specific interrupt, I don't disable the timer 
interrupt.
> > > Your code doesn't seem to handle that.
> > 
> > It can if we increment the preempt_count in disable_irq_nosync and
> > decrement it on enable_irq.
> 
> A driver that knows about how its irq is handled and that it is sole
> user (eg ISA) may and some do leave it disabled for hours at a time

Good point.  Preemption would be disabled for that thread if we mindlessly 
shut it off for every irq_disable.  For that driver we probably just want to 
leave preemption enabled, it can't hurt.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rob Landley]

On Monday 14 January 2002 12:59 am, Daniel Phillips wrote:
> On January 13, 2002 04:59 pm, Alan Cox wrote:
> > > > I disable a single specific interrupt, I don't disable the timer
>
> interrupt.
>
> > > > Your code doesn't seem to handle that.
> > >
> > > It can if we increment the preempt_count in disable_irq_nosync and
> > > decrement it on enable_irq.
> >
> > A driver that knows about how its irq is handled and that it is sole
> > user (eg ISA) may and some do leave it disabled for hours at a time
>
> Good point.  Preemption would be disabled for that thread if we mindlessly
> shut it off for every irq_disable.  For that driver we probably just want
> to leave preemption enabled, it can't hurt.

Once we return to user space, we can preempt again.  If preemption is still 
disabled upon return from the syscall, I'd say it's okay to switch it back on 
now. :)

Unless I'm missing something fundamental...?

Rob

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 12, 2002 09:21 pm, Alan Cox wrote:
> > > I didn't see anywhere you check disable_irq(). Even if you did it doesnt
> > > help when I mask the irq on the chip rather than using disable_irq() calls.
> > 
> > Well, if IRQs are disabled we won't have the timer... would not the
> > system panic anyhow if schedule() was called while in an interrupt
> > handler?
> 
> You completely misunderstand.
> 
> 	disable_irq(n)
> 
> I disable a single specific interrupt, I don't disable the timer interrupt.
> Your code doesn't seem to handle that. Its just one of the examples of where
> you really need priority handling, and thats a horrible dark and slippery
> slope

He just needs to disable preemption there, it's just a slight mod to 
disable/enable_irq.  You probably have a few more of those, though...

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Kenneth Johansson]

Alan Cox wrote:

> > > PRE-EMPT:
> > >     We get back and we've missed 300 packets, the serial port sharing
> > >     the IRQ has dropped our internet connection completely.
> >
> > We don't preempt while IRQ are disabled.
>
> I must have missed that in the code. I can see you check __cli() status but
> I didn't see anywhere you check disable_irq(). Even if you did it doesnt
> help when I mask the irq on the chip rather than using disable_irq() calls.
>
> Alan

But you get interrupted by other interrups then so you have the same problem
reagardless of any preemtion patch you hopefully lose the cpu for a much
shorter time but still the same problem.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sat, 2002-01-12 at 15:36, Kenneth Johansson wrote:

> > I must have missed that in the code. I can see you check __cli() status but
> > I didn't see anywhere you check disable_irq(). Even if you did it doesnt
> > help when I mask the irq on the chip rather than using disable_irq() calls.
> >
> > Alan
> 
> But you get interrupted by other interrups then so you have the same problem
> reagardless of any preemtion patch you hopefully lose the cpu for a much
> shorter time but still the same problem.

Agreed.  Further, you can't put _any_ upper bound on the number of
interrupts that could occur, preempt or not.  Sure, preempt can make it
worse, but I don't see it.  I have no bug reports to correlate.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[J Sloan]

Robert Love wrote:

>Agreed.  Further, you can't put _any_ upper bound on the number of interrupts that could occur, preempt or not.  Sure, preempt can make it worse, but I don't see it. I have no bug reports to correlate.
>

OTOH we do have a pile of user reports which
say the low latency patches give better results.

 From my view here, low latency provides a more
silky feel when e.g. playing RtCW or Q3A -

BTW I have checked out 2.4.18pre2-aa2 and
am now running 2.4.18-pre3 + mini low latency.

* -aa absolutely kicks major booty in benchmarks.

* -mini-low-latency seems to do no worse than
stock kernel benchmark-wise, but seems to be
somehow smoother. I played some mp3s while
running dbench 16 and heard no hitches. Also
the RtCW test was successful, e.g. movement
was fluid and I was victorious in most skirmishes
with win32 opponents.

Regards,

jjs

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> > But you get interrupted by other interrups then so you have the same problem
> > reagardless of any preemtion patch you hopefully lose the cpu for a much
> > shorter time but still the same problem.
> 
> Agreed.  Further, you can't put _any_ upper bound on the number of
> interrupts that could occur, preempt or not.  Sure, preempt can make it
> worse, but I don't see it.  I have no bug reports to correlate.

How may full benchmark sets have you done on an NE2000. Its quite obvious
from your earlier mail you hadn't even considered problems like this.

Let me ask you the _right_ question instead

-	Prove to me that there are no cases that pre-empt doesn't screw up
	like this.
-	Prove to me that pre-empt is better than the big low latency patch

All I have seen so far is benchmarks that say low latency is better as is,
and evidence that preempt patches cause far more problems than they solve
and have complex and subtle side effects nobody yet understands.

Furthermore its obvious that the only way to fix these side effects is to
implement full priority handling to avoid priority inversion issues (which
is precisely what the IRQ problem is) , that means implementing interrupt
handlers as threads, heavyweight locks and an end result I'm really not
interested in using.

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

Alan Cox wrote:

> All I have seen so far is benchmarks that say low latency is better as is,

If Andrew did a good job (what he obviously did), I don't doubt that.

> and evidence that preempt patches cause far more problems than they solve
> and have complex and subtle side effects nobody yet understands.

I'm aware of two side effects:
- preempt exposes already existing problems, which are worth fixing
independent of preempt.
- it can cause unexpected delays, which should be nonfatal, otherwise
worth fixing as well.

What somehow got lost in this discussion, that both patches don't
necessarily conflict with each other, they both attack the same problem
with different approaches, which complement each other. I prefer to get
the best of both patches.
The ll patch identifies problem, which preempt alone can't fix, on the
other hand the ll patch inserts schedule calls all over the place, where
preempt can handle this transparently. So far I haven't seen any
evidence, that preempt introduces any _new_ serious problems, so I'd
rather like to see to get the best out of both.

> Furthermore its obvious that the only way to fix these side effects is to
> implement full priority handling to avoid priority inversion issues (which
> is precisely what the IRQ problem is) , that means implementing interrupt
> handlers as threads, heavyweight locks and an end result I'm really not
> interested in using.

It's not really needed to go that far, it's generally a good idea to
keep interrupt handler as short as possible, we use bh or tasklets for
exactly that reason. I don't think we need to work around broken
hardware, but halfway decent hardware should not be a problem to get
decent latency.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Arjan van de Ven]

On Sun, Jan 13, 2002 at 04:18:29PM +0100, Roman Zippel wrote:

> What somehow got lost in this discussion, that both patches don't
> necessarily conflict with each other, they both attack the same problem
> with different approaches, which complement each other. I prefer to get
> the best of both patches.

If you do this (and I've seen the results of doing both at once vs only
either of then vs pure) then there's NO benifit for the preemption left.
Sure AVERAGE latency goes down slightly, however this is talking in the usec
range since worst case is already 1msec or less. Below the 1msec range it
really doesn't matter anymore however. 

At that point you're adding all the complexity for the negliable-to-no-gain
case...

Greetings,
   Arjan van de Ven

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 13, 2002 04:36 pm, Arjan van de Ven wrote:
> On Sun, Jan 13, 2002 at 04:18:29PM +0100, Roman Zippel wrote:
> 
> > What somehow got lost in this discussion, that both patches don't
> > necessarily conflict with each other, they both attack the same problem
> > with different approaches, which complement each other. I prefer to get
> > the best of both patches.
> 
> If you do this (and I've seen the results of doing both at once vs only
> either of then vs pure) then there's NO benifit for the preemption left.

Sorry, that's incorrect.  I stated why earlier in this thread and akpm signed 
off on it.  With preempt you get ASAP (i.e., as soon as the outermost 
spinlock is done) process scheduling.  With hand-coded scheduling points you 
get 'as soon as it happens to hit a scheduling point'.

That is not the only benefit, just the most obvious one.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Daniel Phillips wrote:
> 
> On January 13, 2002 04:36 pm, Arjan van de Ven wrote:
> > On Sun, Jan 13, 2002 at 04:18:29PM +0100, Roman Zippel wrote:
> >
> > > What somehow got lost in this discussion, that both patches don't
> > > necessarily conflict with each other, they both attack the same problem
> > > with different approaches, which complement each other. I prefer to get
> > > the best of both patches.
> >
> > If you do this (and I've seen the results of doing both at once vs only
> > either of then vs pure) then there's NO benifit for the preemption left.
> 
> Sorry, that's incorrect.  I stated why earlier in this thread and akpm signed
> off on it.  With preempt you get ASAP (i.e., as soon as the outermost
> spinlock is done) process scheduling.  With hand-coded scheduling points you
> get 'as soon as it happens to hit a scheduling point'.

With preempt it's "as soon as you hit a lock-break point".  They're equivalent,
for the inside-lock case, which is where most of the problems and complexity
lie.

> That is not the only benefit, just the most obvious one.

I'd say the most obvious benefit of preempt is that it catches some
of the cases which the explicit schedules do not - the stuff which
the developer didn't test for, and which is outside locks.

How useful this is, is moot.

But we can *make* it useful.  I believe that internal preemption is
the foundation to improve 2.5 kernel latency.  But first we need
consensus that we *want* linux to be a low-latency kernel.

Do we have that?

If we do, then as I've said before, holding a lock for more than N milliseconds
becomes a bug to be fixed.  We can put tools in the hands of testers to
locate those bugs.  Easy.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 14, 2002 06:09 am, Andrew Morton wrote:
> Daniel Phillips wrote:
> I believe that internal preemption is
> the foundation to improve 2.5 kernel latency.  But first we need
> consensus that we *want* linux to be a low-latency kernel.
> 
> Do we have that?

You have it from me, for what it's worth ;-)

> If we do, then as I've said before, holding a lock for more than N
> milliseconds becomes a bug to be fixed.  We can put tools in the hands of 
> testers to locate those bugs.  Easy.

Perhaps not a bug, but bad-acting.  Just as putting a huge object on the 
stack is not necessarily a bug, but deserves a quick larting nonetheless.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 06:03:43AM +0100, Daniel Phillips wrote:
> On January 13, 2002 04:36 pm, Arjan van de Ven wrote:
> > On Sun, Jan 13, 2002 at 04:18:29PM +0100, Roman Zippel wrote:
> > 
> > > What somehow got lost in this discussion, that both patches don't
> > > necessarily conflict with each other, they both attack the same problem
> > > with different approaches, which complement each other. I prefer to get
> > > the best of both patches.
> > 
> > If you do this (and I've seen the results of doing both at once vs only
> > either of then vs pure) then there's NO benifit for the preemption left.
> 
> Sorry, that's incorrect.  I stated why earlier in this thread and akpm signed 
> off on it.  With preempt you get ASAP (i.e., as soon as the outermost 
> spinlock is done) process scheduling.  With hand-coded scheduling points you 
> get 'as soon as it happens to hit a scheduling point'.
> 
> That is not the only benefit, just the most obvious one.

My understanding of this situation is as follows:
	The pure preempt measurements show some improvement on synthetic
	latency benchmarks that have not been shown to have any relationship
	to any real application

	The LL measurements show _better_ results on similar benchmarks.

	Some people find preempt improves "feel"
	Some people find LL improves "feel"

	The interactions of these improvements with Ingos scheduler, aa mm, and
	other recent patches are exceptionally murky

	We have one benchmark that shows that kernel compiles run on different 
	untarred trees show a slight advantage for preempt+Ingo via some
	unknown mechanism. This benchmark, aside from its dubious repeatability 
	tests something that seems to have no relationship to _anything_  at all
	by running a huge number of compile processes.

	Nobody has answered my question about the conflict between SMP per-cpu caching
	and preempt. Since NUMA is apparently the future of MP in the PC world and
	the future of Linux servers, it's interesting to consider this tradeoff.
	
	Nobody has answered the question about how to make sure all processes
	make progress with preempt.

	Nobody has offered a single benchmark of actual application code benefits
	from either preempt or LL.

	Nobody has clearly explained how to avoid what I claim to be the inevitable 
	result of preempt -- priority inheritance locks (not just semaphores).
	What we have is some "we'll figure that out when we get to it".

	It's not even clear how preempt is supposed to interact with SCHED_FIFO.


As far as your "most obvious" "benefit". It's neither obvious that it happens
or obvious that it is a benefit.  According to the measurements I've seen, Andrew
reduces latency _more_ than preempt. Andrews argument, as I understand it, is that
the longest latencies are within spinlocks anyways so increasing speed of preempt
outside those locks misses the problem. If he is correct, then if you are correct,
it doesn't matter - preempt is reducing already short latencies. 

BTW: there is a detailed explanation of how priority inherit works in Solaris in the
UNIX Internals book. It's worth reading and thinking about.

I'm not at all sure that putting preempt into 2.5 is a bad idea. I think that 2.4
has a long lifetime ahead of it and the debacle that will follow putting preempt into 2.5
will eventually discredit the entire idea for at least a year or two.
But
I think that there are some much more important scheduling issues that are being ignored to
"improve the feel" of DVD playing. The key one is some idea of being able to assure processes
of some rate of progress.  This is not classical RT, but it is important to multimedia and 
databases and also to some applications we are interested in looking at. 




---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

On Sun, 13 Jan 2002 yodaiken@fsmlabs.com wrote:

> 	Nobody has answered my question about the conflict between SMP per-cpu caching
> 	and preempt. Since NUMA is apparently the future of MP in the PC world and
> 	the future of Linux servers, it's interesting to consider this tradeoff.

Preempt is a UP feature so far.

> 	Nobody has answered the question about how to make sure all processes
> 	make progress with preempt.

The same way as without preempt.

> 	Nobody has clearly explained how to avoid what I claim to be the inevitable
> 	result of preempt -- priority inheritance locks (not just semaphores).
> 	What we have is some "we'll figure that out when we get to it".

So far you haven't given any reason, how preempt should lead to this.
(If I missed something, please explain it in a way a mere mortal can
understand it.)

> 	It's not even clear how preempt is supposed to interact with SCHED_FIFO.

The same way as without preempt.

More of other FUD deleted, Victor, could you please stop this?

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> More of other FUD deleted, Victor, could you please stop this?

Insulting people won't make problems go away Roman.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

On Mon, 14 Jan 2002, Alan Cox wrote:

> > More of other FUD deleted, Victor, could you please stop this?
>
> Insulting people won't make problems go away Roman.

I'm really trying to avoid this, I'm more than happy to discuss
theoretical or practical problems _if_ they are backed by arguments,
latter are very thin with Victor. Making pointless claims only triggers
above reaction. If I did really miss a major argument so far, I will
publicly apologize.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> I'm really trying to avoid this, I'm more than happy to discuss
> theoretical or practical problems _if_ they are backed by arguments,
> latter are very thin with Victor. Making pointless claims only triggers
> above reaction. If I did really miss a major argument so far, I will
> publicly apologize.

You seem to be missing the fact that latency guarantees only work if you
can make progress. If a low priority process is pre-empted owning a
resource (quite likely) then you won't get your good latency. To
handle those cases you get into priority boosting, and all sorts of lock
complexity - so that the task that owns the resource temporarily can borrow
your priority in order that you can make progress at your needed speed.
That gets horrendously complex, and you get huge chains of priority
dependancies including hardware level ones.

The low latency patches don't make that problem go away, but they achieve
equivalent real world latencies up to at least the point you have to do
priority handling of that kind. 

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

On Mon, 14 Jan 2002, Alan Cox wrote:

> You seem to be missing the fact that latency guarantees only work if you
> can make progress. If a low priority process is pre-empted owning a
> resource (quite likely) then you won't get your good latency. To
> handle those cases you get into priority boosting, and all sorts of lock
> complexity - so that the task that owns the resource temporarily can borrow
> your priority in order that you can make progress at your needed speed.
> That gets horrendously complex, and you get huge chains of priority
> dependancies including hardware level ones.

Any ll approach so far only addresses a single type of latency - the time
from waking up an important process until it really gets the cpu. What is
not handled by any patch are i/o latencies, that means the average time to
get access to a specific resource. (To be exact breaking up locks modifies
of course i/o latencies, but that's more a side effect.)
I/O latencies are only relevant for this discussion insofar, as to verify
they are not overly harmed by improving scheduling latencies. Preempting
does not modify the behaviour of the scheduler, all it does is to increase
the scheduling frequency. This means it can happen that a low priority
task locks a resource for a longer time, because it's interrupted by
another task. Nethertheless the current scheduler guarantees every process
gets its share of the cpu time(*), so the low priority task will continue
and release the resource within a guaranteed amount of time.
So the worst behaviour I see is that on a loaded system, a low priority
task can hold up another task, if that task should be our interactive
task, the interactivity is of course gone. But this problem is not really
new, as we have no guarantees regarding i/o latencies. So everyone using
any patch should be aware of that it's not a magical tool and for getting
better scheduling latencies, one has to trade something else, but so far
I haven't seen any evidence that it makes something else much worse.

(*) This of course assumes accurate cpu time accounting, but I mentioned
this problem before. On the other hand it's also fixable, the tickless
patch looks most interesting in this regard.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> So the worst behaviour I see is that on a loaded system, a low priority
> task can hold up another task, if that task should be our interactive
> task, the interactivity is of course gone. But this problem is not really
> new, as we have no guarantees regarding i/o latencies. So everyone using
> any patch should be aware of that it's not a magical tool and for getting
> better scheduling latencies, one has to trade something else, but so far
> I haven't seen any evidence that it makes something else much worse.

It doesn't make anything better is the issue. Its more complex than ll but
gains nothing

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

On Mon, 14 Jan 2002, Alan Cox wrote:

> It doesn't make anything better is the issue. Its more complex than ll but
> gains nothing

Please see my previous mail about maintenance costs.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rik van Riel]

On Mon, 14 Jan 2002, Roman Zippel wrote:

> Any ll approach so far only addresses a single type of latency - the
> time from waking up an important process until it really gets the cpu.
> What is not handled by any patch are i/o latencies, that means the
> average time to get access to a specific resource.

OK, suppose you have three tasks.

A is a SCHED_FIFO task
B is a nice 0 SCHED_OTHER task
C is a nice +19 SCHED_OTHER task

Task B is your standard CPU hog, running all the time, task C has
grabbed  an inode semaphore (no spinlock), task A wakes up, preempts
task C, tries to grab the inode semaphore and goes back to sleep.

Now task A has to wait for task B to give up the CPU before task C
can run again and release the semaphore.

Without preemption task C would not have been preempted and it would
have released the lock much sooner, meaning task A could have gotten
the resource earlier.

Using the low latency patch we'd insert some smart code into the
algorithm so task A also releases the lock before rescheduling.

Before you say this thing never happens in practice, I ran into
this thing in real life with the SCHED_IDLE patch. In fact, this
problem was so severe it convinced me to abandon SCHED_IDLE ;))

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

On Mon, 14 Jan 2002, Rik van Riel wrote:

> Without preemption task C would not have been preempted and it would
> have released the lock much sooner, meaning task A could have gotten
> the resource earlier.

Define "much sooner", nobody disputes that low priority tasks can be
delayed, that's actually the purpose of both patches.

> Using the low latency patch we'd insert some smart code into the
> algorithm so task A also releases the lock before rescheduling.

Could you please show me that "smart code"?

> Before you say this thing never happens in practice, I ran into
> this thing in real life with the SCHED_IDLE patch. In fact, this
> problem was so severe it convinced me to abandon SCHED_IDLE ;))

SCHED_IDLE is something completely different than preeempt. Rik, do I
really have to explain the difference?

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-14 at 09:35, Rik van Riel wrote:
> OK, suppose you have three tasks.
> 
> A is a SCHED_FIFO task
> B is a nice 0 SCHED_OTHER task
> C is a nice +19 SCHED_OTHER task
> 
> Task B is your standard CPU hog, running all the time, task C has
> grabbed  an inode semaphore (no spinlock), task A wakes up, preempts
> task C, tries to grab the inode semaphore and goes back to sleep.
> 
> Now task A has to wait for task B to give up the CPU before task C
> can run again and release the semaphore.
> 
> Without preemption task C would not have been preempted and it would
> have released the lock much sooner, meaning task A could have gotten
> the resource earlier.
> 
> Using the low latency patch we'd insert some smart code into the
> algorithm so task A also releases the lock before rescheduling.
> 
> Before you say this thing never happens in practice, I ran into
> this thing in real life with the SCHED_IDLE patch. In fact, this
> problem was so severe it convinced me to abandon SCHED_IDLE ;))

This isn't related.  The problem you described can happen nearly as
easily on a non-preemptible system.  We have plenty of semaphores held
across schedules and there is no reason to single out ones that acquire
and release the semaphore in short, non-preemptible, sequences.  We
always have this "problem."

SCHED_IDLE is much different, as you know, because the SCHED_IDLE task
holding the lock can _never_ get scheduled if there is a CPU hog on the
system!  With the preemptive case, we only worry about an increase in
this period, which is at the expense of fairness in running higher
priority tasks.  But I think you know this ...

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Alan Cox wrote:
> 
> > I'm really trying to avoid this, I'm more than happy to discuss
> > theoretical or practical problems _if_ they are backed by arguments,
> > latter are very thin with Victor. Making pointless claims only triggers
> > above reaction. If I did really miss a major argument so far, I will
> > publicly apologize.
> 
> You seem to be missing the fact that latency guarantees only work if you
> can make progress. If a low priority process is pre-empted owning a
> resource (quite likely) then you won't get your good latency. To
> handle those cases you get into priority boosting, and all sorts of lock
> complexity - so that the task that owns the resource temporarily can borrow
> your priority in order that you can make progress at your needed speed.
> That gets horrendously complex, and you get huge chains of priority
> dependancies including hardware level ones.
> 

It would be useful to define the scope and design guidelines of a "real
time task".   Obviously, if it tries to perform filesystem or network
I/O it can block for a long time.  If it acquires VFS locks it can suffer
bad priority inversion.

I have all along assumed that a well-designed RT application would delegate
all these operations to SCHED_OTHER worker processes, probably via shared
memory/shared mappings.  So in the simplest case, you'd have a SCHED_FIFO
task which talks to the hardware, and which has a helper task which reads
and writes stuff from and to disk.  With sufficient buffering and readahead
to cover the worst case IO latencies.

If this is generally workable, then it means that the areas of possible
priority inversion are quite small - basically device driver read/write
functions.  The main remaining area where priority inversion can 
happen is in the page allocator.  I'm experimenting/thinking about giving
non-SCHED_OTHER tasks a modified form of atomic allocation to defeat this.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> I have all along assumed that a well-designed RT application would delegate
> all these operations to SCHED_OTHER worker processes, probably via shared
> memory/shared mappings.  So in the simplest case, you'd have a SCHED_FIFO
> task which talks to the hardware, and which has a helper task which reads
> and writes stuff from and to disk.  With sufficient buffering and readahead
> to cover the worst case IO latencies.

A real RT task has hard guarantees and to all intents and purposes you may
deem the system failed if it ever misses one (arguably if you cannot verify
it will never miss one).

The stuff we care about is things like DVD players which tangle with
sockets, pipes, X11, memory allocation, and synchronization between multiple
hardware devices all running at slightly incorrect clocks.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Alan Cox wrote:
> 
> > I have all along assumed that a well-designed RT application would delegate
> > all these operations to SCHED_OTHER worker processes, probably via shared
> > memory/shared mappings.  So in the simplest case, you'd have a SCHED_FIFO
> > task which talks to the hardware, and which has a helper task which reads
> > and writes stuff from and to disk.  With sufficient buffering and readahead
> > to cover the worst case IO latencies.
> 
> A real RT task has hard guarantees and to all intents and purposes you may
> deem the system failed if it ever misses one (arguably if you cannot verify
> it will never miss one).

We know that :)  Here, "RT" means "Linux-RT": something which is non-SCHED_OTHER,
and which we'd prefer didn't completely suck.

> The stuff we care about is things like DVD players which tangle with
> sockets, pipes, X11, memory allocation, and synchronization between multiple
> hardware devices all running at slightly incorrect clocks.

Well, that's my point.  A well-designed DVD player would have two processes.
One which tangles with the sockets, pipes, disks, etc, and which feeds data into
and out of the SCHED_FIFO task via a shared, mlocked memory region.

What I'm trying to develop here is a set of guidelines which will allow
application developers to design these programs with a reasonable
degree of success.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Well, that's my point.  A well-designed DVD player would have two processes.
> One which tangles with the sockets, pipes, disks, etc, and which feeds data into
> and out of the SCHED_FIFO task via a shared, mlocked memory region.
> 
> What I'm trying to develop here is a set of guidelines which will allow
> application developers to design these programs with a reasonable
> degree of success.

What about the X server 8)

Given 1mS and vague fairness DVD is more than acceptable

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 12:14:47PM +0100, Roman Zippel wrote:
> Hi,
> 
> On Sun, 13 Jan 2002 yodaiken@fsmlabs.com wrote:
> 
> > 	Nobody has answered my question about the conflict between SMP per-cpu caching
> > 	and preempt. Since NUMA is apparently the future of MP in the PC world and
> > 	the future of Linux servers, it's interesting to consider this tradeoff.
> 
> Preempt is a UP feature so far.

I think this is a sufficient summary of your engineering approach.

 ...

> More of other FUD deleted, Victor, could you please stop this?

I guess that Andrew, Alan, Andrea and I all are raising objections that
you ignore because we  have some kind of shared bias.

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

On Mon, 14 Jan 2002 yodaiken@fsmlabs.com wrote:

> > > 	Nobody has answered my question about the conflict between SMP per-cpu caching
> > > 	and preempt. Since NUMA is apparently the future of MP in the PC world and
> > > 	the future of Linux servers, it's interesting to consider this tradeoff.
> >
> > Preempt is a UP feature so far.
>
> I think this is a sufficient summary of your engineering approach.

Would you please care to explain, what the hell you want?
Preempt on SMP has more problems than you mention above, so that the scope
of my arguments only included UP. Sorry, if I missed something, but
preempt on SMP is an entirely different dicussion.

> > More of other FUD deleted, Victor, could you please stop this?
>
> I guess that Andrew, Alan, Andrea and I all are raising objections that
> you ignore because we  have some kind of shared bias.

No, your sparse use of arguments makes the difference.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Guest section DW]

On Mon, Jan 14, 2002 at 12:14:47PM +0100, Roman Zippel wrote:

> On Sun, 13 Jan 2002 yodaiken@fsmlabs.com wrote:
> 
> > 	Nobody has answered the question about how to make sure all processes
> > 	make progress with preempt.
> 
> The same way as without preempt.
> 
> More of other FUD deleted, Victor, could you please stop this?

Roman, Victor asks meaningful questions.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Momchil Velikov]

>>>>> "yodaiken" == yodaiken  <yodaiken@fsmlabs.com> writes:
yodaiken> 	It's not even clear how preempt is supposed to interact with SCHED_FIFO.

How so ? The POSIX specification is not clear enough or it is not to be followed ?

Regards,
-velco

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Oliver Neukum]

On Monday 14 January 2002 13:17, Momchil Velikov wrote:
> >>>>> "yodaiken" == yodaiken  <yodaiken@fsmlabs.com> writes:
>
> yodaiken> 	It's not even clear how preempt is supposed to interact with
> SCHED_FIFO.
>
> How so ? The POSIX specification is not clear enough or it is not to be
> followed ?

You can have an rt task block on a lock held by a normal task that was 
preempted by a rt task of lower priority. The same problem as with the 
sched_idle patches.

	Regards
		Oliver

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Momchil Velikov]

>>>>> "Oliver" == Oliver Neukum <520047054719-0001@t-online.de> writes:

Oliver> On Monday 14 January 2002 13:17, Momchil Velikov wrote:
>> >>>>> "yodaiken" == yodaiken  <yodaiken@fsmlabs.com> writes:
>> 
yodaiken> It's not even clear how preempt is supposed to interact with
>> SCHED_FIFO.
>> 
>> How so ? The POSIX specification is not clear enough or it is not to be
>> followed ?

Oliver> You can have an rt task block on a lock held by a normal task that was 
Oliver> preempted by a rt task of lower priority. The same problem as with the 
Oliver> sched_idle patches.

This can happen with a non-preemptible kernel too. And it has nothing to
do with scheduling policy.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Oliver> You can have an rt task block on a lock held by a normal task that was 
> Oliver> preempted by a rt task of lower priority. The same problem as with the 
> Oliver> sched_idle patches.
> 
> This can happen with a non-preemptible kernel too. And it has nothing to
> do with scheduling policy.

So why bother adding pre-emption. As you keep saying - it doesnt
gain anything

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Momchil Velikov]

>>>>> "Alan" == Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

Oliver> You can have an rt task block on a lock held by a normal task that was 
Oliver> preempted by a rt task of lower priority. The same problem as with the 
Oliver> sched_idle patches.
>> 
>> This can happen with a non-preemptible kernel too. And it has nothing to
>> do with scheduling policy.

Alan> So why bother adding pre-emption. As you keep saying - it doesnt
Alan> gain anything

Nope. I don't. I said (at least in the above) it didn't hurt.

One can consider a non-preemptible kernel as a special kind of
priority inversion, preemptible kernel will eliminate _that_ case of
priority inversion.

Regards,
-velco

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Tue, Jan 15, 2002 at 12:34:01AM +0200, Momchil Velikov wrote:
> One can consider a non-preemptible kernel as a special kind of
> priority inversion, preemptible kernel will eliminate _that_ case of
> priority inversion.

The problem here is that priority means something very different in 
a time-shared system than in a hard real-time system. And even in real-time
systems, as Walpole and colleagues have pointed out, priority doesn't
really capture much of what is needed for good scheduling.

In a general purpose system,  priorities are dynamic and "fair". 
The priority of even the lowliest process increases while it waits
for time. In a raw real-time system, the low priority process can sit
forever and should wait until no higher priority thread needs the 
processor. So it's absurd to talk of priority inversion in a non RT
system. When a low priority process is delaying a higher priority task
for reasons of fairness, increased throughput, or any other valid
objective, that is not a scheduling error.


> 
> Regards,
> -velco

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Tue, Jan 15, 2002 at 10:15:11AM +0200, Momchil Velikov wrote:
> >>>>> "yodaiken" == yodaiken  <yodaiken@fsmlabs.com> writes:
> 
> yodaiken> On Tue, Jan 15, 2002 at 12:34:01AM +0200, Momchil Velikov wrote:
> >> One can consider a non-preemptible kernel as a special kind of
> >> priority inversion, preemptible kernel will eliminate _that_ case of
> >> priority inversion.
> 
> yodaiken> The problem here is that priority means something very different in 
> yodaiken> a time-shared system than in a hard real-time system. And even in real-time
> yodaiken> systems, as Walpole and colleagues have pointed out, priority doesn't
> yodaiken> really capture much of what is needed for good scheduling.
> 
> Well, maybe there are other policies (notably static scheduling ;),
> which may be preferrable in one or other case, anyway, I personally
> tend to think rate-monotonic scheduling is quite adequate in practice.

That's not what we see in real RT applications.

> Of course, anyone is free to prove me wrong by implementing earliest
> deadline or slack time scheduler in the kernel.

My impression is that these are of limited use in applications.

> 
> yodaiken> In a general purpose system,  priorities are dynamic and "fair". 
> yodaiken> The priority of even the lowliest process increases while it waits
> yodaiken> for time. In a raw real-time system, the low priority process can sit
> yodaiken> forever and should wait until no higher priority thread needs the 
> yodaiken> processor. 
> 
> *nod*
> 
> yodaiken>             So it's absurd to talk of priority inversion in a non RT
> yodaiken> system. When a low priority process is delaying a higher priority task
> yodaiken> for reasons of fairness, increased throughput, or any other valid
> yodaiken> objective, that is not a scheduling error.
> 
> What is a non real-time system?  E.g. are a desktop playing ogg or a
> server serving multimedia content over ATM, real-time systems?  The
> point is that one can view the system as real-time or not depending on
> the dominant _current_ load.  A system may have real-time load now and

I don't think so, and certainly the straightforward priority based model
of traditional RT seems grossly wrong. What's the highest priority process - 
is it the CD player or is it the niced sendmail that is running in the
background? The fact is we require that _both_ make progress. Therefore
at some time the sendmail process must have effectively higher priority
than the CD player. If we then factor in the many kernel threads that may
be running at any time, the calculation becomes even more problematic. 
In RTLinux we have a much simpler environment and we can use the rule:
	The highest priority runnable task should be the running task
	within some T and the OS gets better as T goes to 0.
If something starves, that's the user problem - our users are programmers
who we want to provide with as much control as possible. We certainly
do not want to have a situation where the stop command to the robot arm
is delayed because the OS needs to run bdflush ...


For Linux, even Linux doing the critical work of playing DVDs, this is 
not so simple. The target users are not programmers, liveness and 
responsiveness remain important criteria ... 
It's clear that much of the problem when people discuss this issue is
lack of clear goals. "Preemption" is a controversial implementation 
technique, not a goal in itself - I hope. Even "low latency" doesn't
really capture the intended goal - "low latency" to do what?  If it's
simply "low latency" switch to highest priority process, that's not
necesarily anything that is going to be useful in itself. The 
"resource reservation" work at CMU seems to me to be way off the point
too. The OGI people have some interesting thoughts about this stuff, but
I don't think they've come close to working out a comprehensive model.

What I think we need is a kind of interval real-time scheduling.
Something like:
	The system has a basic timing period of N milliseconds where
	N is at lease 500 and probably more.

	Over a N millisecond period each process gets a full scheduling quantum
	and, if it requests, a full I/O quantum.
	For a niced process there is some calculated interval greater than
	N. An I/O quantum should correspond somehow to a rate of I/0.

	RTLIMIT is used to set the max number of processes allowed to start
	and this determines the computation length of "one quantum"

There is this cliche about RT that RT is about deadlines, not speed, but I think
that's only partially true. To say that every task will run during a 1 second period
with today's PC/Server technology is fundamentally different than to say that 
every task will run during a 10 millisecond period.

> a non real-time one an hour later, both occasionally intermixed with
> tasks of the other class. It is unreasonable to say "This system is
> non-realtime so use that kernel and when you want to play vidoes,
> please reboot to something else, for example XX-Linux".

Yes, but it's not unreasonable to pop up a window and ask the user if
he/she wants to have the scheduling rates of other applications
stepped down to run the DVD player. 
	The Oracle logging server is designed to run at 1 second periods
	increasing the responsiveness of your current process will 
	reduce that to a 2 second period.
	Email delivery will be delayed up to 1 minute
	...
	YES NO

and so we can follow the most important OS design technique of all time:
	push complexity to user space.
 

---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[george anzinger]

yodaiken@fsmlabs.com wrote:
> 
> On Tue, Jan 15, 2002 at 10:15:11AM +0200, Momchil Velikov wrote:
> > >>>>> "yodaiken" == yodaiken  <yodaiken@fsmlabs.com> writes:
> >
> > yodaiken> On Tue, Jan 15, 2002 at 12:34:01AM +0200, Momchil Velikov wrote:
> > >> One can consider a non-preemptible kernel as a special kind of
> > >> priority inversion, preemptible kernel will eliminate _that_ case of
> > >> priority inversion.
> >
> > yodaiken> The problem here is that priority means something very different in
> > yodaiken> a time-shared system than in a hard real-time system. And even in real-time
> > yodaiken> systems, as Walpole and colleagues have pointed out, priority doesn't
> > yodaiken> really capture much of what is needed for good scheduling.
> >
> > Well, maybe there are other policies (notably static scheduling ;),
> > which may be preferrable in one or other case, anyway, I personally
> > tend to think rate-monotonic scheduling is quite adequate in practice.
> 
> That's not what we see in real RT applications.
> 
> > Of course, anyone is free to prove me wrong by implementing earliest
> > deadline or slack time scheduler in the kernel.
> 
> My impression is that these are of limited use in applications.
> 
> >
> > yodaiken> In a general purpose system,  priorities are dynamic and "fair".
> > yodaiken> The priority of even the lowliest process increases while it waits
> > yodaiken> for time. In a raw real-time system, the low priority process can sit
> > yodaiken> forever and should wait until no higher priority thread needs the
> > yodaiken> processor.
> >
> > *nod*
> >
> > yodaiken>             So it's absurd to talk of priority inversion in a non RT
> > yodaiken> system. When a low priority process is delaying a higher priority task
> > yodaiken> for reasons of fairness, increased throughput, or any other valid
> > yodaiken> objective, that is not a scheduling error.
> >
> > What is a non real-time system?  E.g. are a desktop playing ogg or a
> > server serving multimedia content over ATM, real-time systems?  The
> > point is that one can view the system as real-time or not depending on
> > the dominant _current_ load.  A system may have real-time load now and
> 
> I don't think so, and certainly the straightforward priority based model
> of traditional RT seems grossly wrong. What's the highest priority process -
> is it the CD player or is it the niced sendmail that is running in the
> background? The fact is we require that _both_ make progress. Therefore
> at some time the sendmail process must have effectively higher priority
> than the CD player. If we then factor in the many kernel threads that may
> be running at any time, the calculation becomes even more problematic.
> In RTLinux we have a much simpler environment and we can use the rule:
>         The highest priority runnable task should be the running task
>         within some T and the OS gets better as T goes to 0.
> If something starves, that's the user problem - our users are programmers
> who we want to provide with as much control as possible. We certainly
> do not want to have a situation where the stop command to the robot arm
> is delayed because the OS needs to run bdflush ...
> 
> For Linux, even Linux doing the critical work of playing DVDs, this is
> not so simple. The target users are not programmers, liveness and
> responsiveness remain important criteria ...
> It's clear that much of the problem when people discuss this issue is
> lack of clear goals. "Preemption" is a controversial implementation
> technique, not a goal in itself - I hope. Even "low latency" doesn't
> really capture the intended goal - "low latency" to do what?  If it's
> simply "low latency" switch to highest priority process, that's not
> necesarily anything that is going to be useful in itself. The
> "resource reservation" work at CMU seems to me to be way off the point
> too. The OGI people have some interesting thoughts about this stuff, but
> I don't think they've come close to working out a comprehensive model.
> 
> What I think we need is a kind of interval real-time scheduling.
> Something like:
>         The system has a basic timing period of N milliseconds where
>         N is at lease 500 and probably more.
> 
>         Over a N millisecond period each process gets a full scheduling quantum
>         and, if it requests, a full I/O quantum.
>         For a niced process there is some calculated interval greater than
>         N. An I/O quantum should correspond somehow to a rate of I/0.
> 
>         RTLIMIT is used to set the max number of processes allowed to start
>         and this determines the computation length of "one quantum"

Have you looked at SCHED_SPORADIC (see 1003.1d-1999)?
> 
> There is this cliche about RT that RT is about deadlines, not speed, but I think
> that's only partially true. To say that every task will run during a 1 second period
> with today's PC/Server technology is fundamentally different than to say that
> every task will run during a 10 millisecond period.
> 
> > a non real-time one an hour later, both occasionally intermixed with
> > tasks of the other class. It is unreasonable to say "This system is
> > non-realtime so use that kernel and when you want to play vidoes,
> > please reboot to something else, for example XX-Linux".
> 
> Yes, but it's not unreasonable to pop up a window and ask the user if
> he/she wants to have the scheduling rates of other applications
> stepped down to run the DVD player.
>         The Oracle logging server is designed to run at 1 second periods
>         increasing the responsiveness of your current process will
>         reduce that to a 2 second period.
>         Email delivery will be delayed up to 1 minute
>         ...
>         YES NO
> 
> and so we can follow the most important OS design technique of all time:
>         push complexity to user space.
> 
> 
> ---------------------------------------------------------
> Victor Yodaiken
> Finite State Machine Labs: The RTLinux Company.
>  www.fsmlabs.com  www.rtlinux.com
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sun, Jan 20, 2002 at 02:31:51AM -0800, george anzinger wrote:
> yodaiken@fsmlabs.com wrote:
> > What I think we need is a kind of interval real-time scheduling.
> > Something like:
> >         The system has a basic timing period of N milliseconds where
> >         N is at lease 500 and probably more.
> > 
> >         Over a N millisecond period each process gets a full scheduling quantum
> >         and, if it requests, a full I/O quantum.
> >         For a niced process there is some calculated interval greater than
> >         N. An I/O quantum should correspond somehow to a rate of I/0.
> > 
> >         RTLIMIT is used to set the max number of processes allowed to start
> >         and this determines the computation length of "one quantum"
> 
> Have you looked at SCHED_SPORADIC (see 1003.1d-1999)?

Yes, but I don't think it does much - although its description is quite
long and complex.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Oliver Neukum]

> >> How so ? The POSIX specification is not clear enough or it is not to be
> >> followed ?
>
> Oliver> You can have an rt task block on a lock held by a normal task that
> was Oliver> preempted by a rt task of lower priority. The same problem as
> with the Oliver> sched_idle patches.
>
> This can happen with a non-preemptible kernel too. And it has nothing to
> do with scheduling policy.

It can happen if you sleep with a lock held.
It can not happen at random points in the code.
Thus there is a relation to preemption in kernel mode.

To cure that problem tasks holding a lock would have to be given
the highest priority of all tasks blocking on that lock. The semaphore
code would get much more complex, even in the succesful code path,
which would hurt a lot.

If on the other hand sleeping in kernel mode is explicit, you can simply
give any task being woken up a timeslice and the scheduling requirements
are met. If that should be a problem.

	Regards
		Oliver

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-14 at 13:04, Oliver Neukum wrote:

> It can happen if you sleep with a lock held.
> It can not happen at random points in the code.
> Thus there is a relation to preemption in kernel mode.
> 
> To cure that problem tasks holding a lock would have to be given
> the highest priority of all tasks blocking on that lock. The semaphore
> code would get much more complex, even in the succesful code path,
> which would hurt a lot.

No, this isn't needed.  This same problem would occur without
preemption.  Our semaphores now have locking rules such that we aren't
going to have blatant priority inversion like this (1 holds A needs B, 2
holds B needs A).

When priority inversion begins to become a problem is if we intend to
start turning existing spinlocks into semaphores.  There the locking
rules are weaker, and thus we would need to do priority inheriting.  But
that's not now.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Oliver Neukum]

On Monday 14 January 2002 21:09, Robert Love wrote:
> On Mon, 2002-01-14 at 13:04, Oliver Neukum wrote:
> > It can happen if you sleep with a lock held.
> > It can not happen at random points in the code.
> > Thus there is a relation to preemption in kernel mode.
> >
> > To cure that problem tasks holding a lock would have to be given
> > the highest priority of all tasks blocking on that lock. The semaphore
> > code would get much more complex, even in the succesful code path,
> > which would hurt a lot.
>
> No, this isn't needed.  This same problem would occur without
> preemption.  Our semaphores now have locking rules such that we aren't
> going to have blatant priority inversion like this (1 holds A needs B, 2
> holds B needs A).

No this is a good old deadlock.
The problem with preemption and SCHED_FIFO is, that due to SCHED_FIFO
you have no guarantee that any task will make any progress at all.
Thus a semaphore could basically be held forever.
That can happen without preemption only if you do something that
might block.

	Regards
		Oliver

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-14 at 15:22, Oliver Neukum wrote:

> > No, this isn't needed.  This same problem would occur without
> > preemption.  Our semaphores now have locking rules such that we aren't
> > going to have blatant priority inversion like this (1 holds A needs B, 2
> > holds B needs A).
> 
> No this is a good old deadlock.
> The problem with preemption and SCHED_FIFO is, that due to SCHED_FIFO
> you have no guarantee that any task will make any progress at all.
> Thus a semaphore could basically be held forever.
> That can happen without preemption only if you do something that
> might block.

Well, semaphores block.  And we have these races right now with
SCHED_FIFO tasks.  I still contend preempt does not change the nature of
the problem and it certainly doesn't introduce a new one.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Oliver Neukum]

> Well, semaphores block.  And we have these races right now with
> SCHED_FIFO tasks.  I still contend preempt does not change the nature of
> the problem and it certainly doesn't introduce a new one.

But it does:

down(&sem);
do_something_that_cannot_block();
up(&sem);

Will stop a SCHED_FIFO task for a definite amount of time. Only
until it returns from the kernel to user space at worst.

If do_something_that_cannot_block() can be preempted, a SCHED_FIFO
task can block indefinitely long on the semaphore, because you have
no guarantee that the scheduler will ever again select the the preempted task.
In fact it must never again select the preempted task as long as there's
another runnable SCHED_FIFO task.

	Regards
		Oliver

Re: [2.4.17/18pre] VM and swap - it's really unusable

[george anzinger]

Oliver Neukum wrote:
> 
> > Well, semaphores block.  And we have these races right now with
> > SCHED_FIFO tasks.  I still contend preempt does not change the nature of
> > the problem and it certainly doesn't introduce a new one.
> 
> But it does:
> 
> down(&sem);
> do_something_that_cannot_block();
> up(&sem);
> 
> Will stop a SCHED_FIFO task for a definite amount of time. Only
> until it returns from the kernel to user space at worst.
> 
> If do_something_that_cannot_block() can be preempted, a SCHED_FIFO
> task can block indefinitely long on the semaphore, because you have
> no guarantee that the scheduler will ever again select the the preempted task.
> In fact it must never again select the preempted task as long as there's
> another runnable SCHED_FIFO task.
> 
This is not true, and if it is is a scheduler bug.  When a task (any
task) gets preempted it is not moved from the front of its queue, thus,
in this case, the FIFO task will still be the fitst task at its prioity
to run.  Also, it can only be preempted by another real time task of
higher priority.  Now it is possible that that task may block on the
same sem, but this is simple priority inversion and has nothing to do
with the sem holder being FIFO, RR or any thing else.  In other words
preemption does NOT change a FIFO (or any other) task's position in the
dispatch queue.


-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 02:17:46PM +0200, Momchil Velikov wrote:
> >>>>> "yodaiken" == yodaiken  <yodaiken@fsmlabs.com> writes:
> yodaiken> 	It's not even clear how preempt is supposed to interact with SCHED_FIFO.
> 
> How so ? The POSIX specification is not clear enough or it is not to be followed ?

POSIX makes no specification of how scheduling classes interact - unless something changed
in the new version.

But more than that, the problem of preemption is much more complex when you have
task that do not share the "goodness fade" with everything else. That is, given a
set of SCHED_OTHER processes at time T0, it is reasonable to design the scheduler so
that there is some D so that by time T0+D each process has become the highest priority
and has received cpu up to either a complete time slice or a I/O block. Linux kind of
has this property now, and I believe that making this more robust and easier to analyze
is going to be an enormously important issue.  However, once you add SCHED_FIFO in the
current scheme, this becomes more complex. And with preempt, you cannot even offer the
assurance that once a process gets the cpu it will make _any_ advance at all.



-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

I forgot the line that says: "Oliver pointed out the immediate problem but .."

On Mon, Jan 14, 2002 at 06:45:48AM -0700, yodaiken@fsmlabs.com wrote:
> On Mon, Jan 14, 2002 at 02:17:46PM +0200, Momchil Velikov wrote:
> > >>>>> "yodaiken" == yodaiken  <yodaiken@fsmlabs.com> writes:
> > yodaiken> 	It's not even clear how preempt is supposed to interact with SCHED_FIFO.
> > 
> > How so ? The POSIX specification is not clear enough or it is not to be followed ?
> 
> POSIX makes no specification of how scheduling classes interact - unless something changed
> in the new version.
> 
> But more than that, the problem of preemption is much more complex when you have
> task that do not share the "goodness fade" with everything else. That is, given a
> set of SCHED_OTHER processes at time T0, it is reasonable to design the scheduler so
> that there is some D so that by time T0+D each process has become the highest priority
> and has received cpu up to either a complete time slice or a I/O block. Linux kind of
> has this property now, and I believe that making this more robust and easier to analyze
> is going to be an enormously important issue.  However, once you add SCHED_FIFO in the
> current scheme, this becomes more complex. And with preempt, you cannot even offer the
> assurance that once a process gets the cpu it will make _any_ advance at all.
> 
> 
> 
> -- 
> ---------------------------------------------------------
> Victor Yodaiken 
> Finite State Machine Labs: The RTLinux Company.
>  www.fsmlabs.com  www.rtlinux.com

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

On Mon, 14 Jan 2002 yodaiken@fsmlabs.com wrote:

> is going to be an enormously important issue.  However, once you add SCHED_FIFO in the
> current scheme, this becomes more complex. And with preempt, you cannot even offer the
> assurance that once a process gets the cpu it will make _any_ advance at all.

I'm not sure if I understand you correctly, but how is this related to
preempt? A SCHED_FIFO tasks only delays SCHED_OTHER tasks, but it doesn't
consume their time slice, so the remaining tasks still get their
(previously assigned) time at the cpu, until all tasks have consumed
their share.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 03:56:05PM +0100, Roman Zippel wrote:
> Hi,
> 
> On Mon, 14 Jan 2002 yodaiken@fsmlabs.com wrote:
> 
> > is going to be an enormously important issue.  However, once you add SCHED_FIFO in the
> > current scheme, this becomes more complex. And with preempt, you cannot even offer the
> > assurance that once a process gets the cpu it will make _any_ advance at all.
> 
> I'm not sure if I understand you correctly, but how is this related to
> preempt?

It's pretty subtle. If there is no preempt, processes don't get preempted.
If there is preempt, they can be preempted. Amazing isn't it? 

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> > > is going to be an enormously important issue.  However, once you add SCHED_FIFO in the
> > > current scheme, this becomes more complex. And with preempt, you cannot even offer the
> > > assurance that once a process gets the cpu it will make _any_ advance at all.
> >
> > I'm not sure if I understand you correctly, but how is this related to
> > preempt?
> 
> It's pretty subtle. If there is no preempt, processes don't get preempted.
> If there is preempt, they can be preempted. Amazing isn't it?

I just can't win against such brilliant argumentation, I'm out.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Momchil Velikov]

>>>>> "yodaiken" == yodaiken  <yodaiken@fsmlabs.com> writes:
yodaiken> current scheme, this becomes more complex. And with preempt, you cannot even offer the
yodaiken> assurance that once a process gets the cpu it will make _any_ advance at all.

So? It either shouldn't have got the CPU anyway (maybe it CPU is
needed for other things) or user's priority setup is seriously borked.

The scheduling policies, algorithms, mechanisms, whatever ... do not
guarantee schedulability by themselves.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 14, 2002 10:09 am, yodaiken@fsmlabs.com wrote:
> UNIX generally tries to ensure liveness. So you know that
> 	cat lkarchive | grep feel | wc
> will complete and not just that, it will run pretty reasonably because
> for UNIX _every_ process is important and gets cpu and IO time.
> When you start trying to add special low latency tasks, you endanger
> liveness.  And preempt is especially corrosive because one of the 
> mechanisms UNIX uses to assure liveness is to make sure that once a 
> process starts it can do a significant chunk of work.

You're claiming that preemption by nature is not Unix-like?

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 07:43:59PM +0100, Daniel Phillips wrote:
> On January 14, 2002 10:09 am, yodaiken@fsmlabs.com wrote:
> > UNIX generally tries to ensure liveness. So you know that
> > 	cat lkarchive | grep feel | wc
> > will complete and not just that, it will run pretty reasonably because
> > for UNIX _every_ process is important and gets cpu and IO time.
> > When you start trying to add special low latency tasks, you endanger
> > liveness.  And preempt is especially corrosive because one of the 
> > mechanisms UNIX uses to assure liveness is to make sure that once a 
> > process starts it can do a significant chunk of work.
> 
> You're claiming that preemption by nature is not Unix-like?

Kernel preemption is not traditionally part of UNIX. 

> 
> --
> Daniel

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-14 at 13:39, yodaiken@fsmlabs.com wrote:

> > You're claiming that preemption by nature is not Unix-like?
> 
> Kernel preemption is not traditionally part of UNIX. 

True original AT&T was non-preemptible, but it also didn't originally
have paging.  Today, Solaris, IRIX, latest BSD (via BSDng), etc. are all
preemptible kernels.

Ask Core whether SMPng in FreeBSD 5.0 will include preempt, I think they
are still debating.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rick Stevens]

Daniel Phillips wrote:

> On January 14, 2002 10:09 am, yodaiken@fsmlabs.com wrote:
> 
>>UNIX generally tries to ensure liveness. So you know that
>>	cat lkarchive | grep feel | wc
>>will complete and not just that, it will run pretty reasonably because
>>for UNIX _every_ process is important and gets cpu and IO time.
>>When you start trying to add special low latency tasks, you endanger
>>liveness.  And preempt is especially corrosive because one of the 
>>mechanisms UNIX uses to assure liveness is to make sure that once a 
>>process starts it can do a significant chunk of work.
>>
> 
> You're claiming that preemption by nature is not Unix-like?


Unix started out life as a _time-sharing_ OS.  It never claimed to
be preemptive or real time.  For those, you waited a while, then
got to run MACH.
----------------------------------------------------------------------
- Rick Stevens, SSE, VitalStream, Inc.      rstevens@vitalstream.com -
- 949-743-2010 (Voice)                    http://www.vitalstream.com -
-                                                                    -
-        Change is inevitable, except from a vending machine.        -
----------------------------------------------------------------------

Re: [2.4.17/18pre] VM and swap - it's really unusable

[george anzinger]

Daniel Phillips wrote:
> 
> On January 14, 2002 10:09 am, yodaiken@fsmlabs.com wrote:
> > UNIX generally tries to ensure liveness. So you know that
> >       cat lkarchive | grep feel | wc
> > will complete and not just that, it will run pretty reasonably because
> > for UNIX _every_ process is important and gets cpu and IO time.
> > When you start trying to add special low latency tasks, you endanger
> > liveness.  And preempt is especially corrosive because one of the
> > mechanisms UNIX uses to assure liveness is to make sure that once a
> > process starts it can do a significant chunk of work.
> 
If I read this right, your complaint is not with preemption but with
scheduler policy.  Clearly both are needed to "assure liveness". 
Another way of looking at preemption is that is enables a more
responsive and nimble scheduler policy (afterall it is the scheduler
that decided that task A should give way to task B.  All preemption does
is to allow that to happen with greater dispatch.)  Given that, we can
then discuss what scheduler policy should be.
-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 15, 2002 04:07 am, george anzinger wrote:
> Daniel Phillips wrote:
> > 
> > On January 14, 2002 10:09 am, yodaiken@fsmlabs.com wrote:
> > > UNIX generally tries to ensure liveness. So you know that
> > >       cat lkarchive | grep feel | wc
> > > will complete and not just that, it will run pretty reasonably because
> > > for UNIX _every_ process is important and gets cpu and IO time.
> > > When you start trying to add special low latency tasks, you endanger
> > > liveness.  And preempt is especially corrosive because one of the
> > > mechanisms UNIX uses to assure liveness is to make sure that once a
> > > process starts it can do a significant chunk of work.
>
> If I read this right, your complaint is not with preemption but with
> scheduler policy.  Clearly both are needed to "assure liveness". 
> Another way of looking at preemption is that is enables a more
> responsive and nimble scheduler policy (afterall it is the scheduler
> that decided that task A should give way to task B.  All preemption does
> is to allow that to happen with greater dispatch.)  Given that, we can
> then discuss what scheduler policy should be.

You responded to the wrong person, however I'll take this opportunity to 
agree with you, on the basis of my years of experience with critical path 
scheduling.  For project schedules 'earlist completion' is the name of the 
game, within bounds of available resources.  When you delay an indvidual 
'task' (I'm using the project management term here) past the earliest time it 
can be scheduled, you are using up its 'float', and if the delay is longer 
than the task's float, the completion time of the schedule as a whole will be 
delayed.  This is no different for a computer than it is for a group of 
people, it is still a scheduling problem.  Delaying any random task risks 
delaying the schedule as a whole, and that risk approaches certainty as the 
number of delays approaches infinity.

N.B.: the above observation is aimed at project managers, who will know 
exactly what I'm talking about.  Otherwise, don't worry if it sounds like so 
much BS, it actually isn't ;-)

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 07:07:46PM -0800, george anzinger wrote:
> Daniel Phillips wrote:
> > 
> > On January 14, 2002 10:09 am, yodaiken@fsmlabs.com wrote:
> > > UNIX generally tries to ensure liveness. So you know that
> > >       cat lkarchive | grep feel | wc
> > > will complete and not just that, it will run pretty reasonably because
> > > for UNIX _every_ process is important and gets cpu and IO time.
> > > When you start trying to add special low latency tasks, you endanger
> > > liveness.  And preempt is especially corrosive because one of the
> > > mechanisms UNIX uses to assure liveness is to make sure that once a
> > > process starts it can do a significant chunk of work.
> > 
> If I read this right, your complaint is not with preemption but with
> scheduler policy.  Clearly both are needed to "assure liveness". 

You are right: I think however "preemption" is part of
a package of ideas about how the system should work. 
So it would probably be better to separate these issues out

> Another way of looking at preemption is that is enables a more
> responsive and nimble scheduler policy (afterall it is the scheduler
> that decided that task A should give way to task B.  All preemption does
> is to allow that to happen with greater dispatch.)  Given that, we can
> then discuss what scheduler policy should be.

If you would write this as 
	Another way of looking at preemption is that it is intended
	to enable a more responsive ...
then we would be off to a good start in narrowing the discussion.
My reservation about preemption as an implementation technique is that
it has costs, which seem to be not easily boundable, but not very 
clear benefits.


> -- 
> George           george@mvista.com
> High-res-timers: http://sourceforge.net/projects/high-res-timers/
> Real time sched: http://sourceforge.net/projects/rtsched/

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 15, 2002 01:39 pm, yodaiken@fsmlabs.com wrote:
> My reservation about preemption as an implementation technique is that
> it has costs, which seem to be not easily boundable, but not very 
> clear benefits.

To me the benefit is clear enough: ASAP scheduling of IO threads, a simple 
heuristic that improves both throughput and latency.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 04:38:59PM +0100, Daniel Phillips wrote:
> On January 15, 2002 01:39 pm, yodaiken@fsmlabs.com wrote:
> > My reservation about preemption as an implementation technique is that
> > it has costs, which seem to be not easily boundable, but not very 
> > clear benefits.
> 
> To me the benefit is clear enough: ASAP scheduling of IO threads, a simple 
> heuristic that improves both throughput and latency.

I think of "benefit", perhaps naiively, in terms of something that can
be measured or demonstrated rather than just announced.


-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 21, 2002 04:43 pm, yodaiken@fsmlabs.com wrote:
> On Mon, Jan 21, 2002 at 04:38:59PM +0100, Daniel Phillips wrote:
> > On January 15, 2002 01:39 pm, yodaiken@fsmlabs.com wrote:
> > > My reservation about preemption as an implementation technique is that
> > > it has costs, which seem to be not easily boundable, but not very 
> > > clear benefits.
> > 
> > To me the benefit is clear enough: ASAP scheduling of IO threads, a 
> > simple heuristic that improves both throughput and latency.
> 
> I think of "benefit", perhaps naiively, in terms of something that can
> be measured or demonstrated rather than just announced.

But you see why asap scheduling improves latency/throughput *in theory*, 
don't you?  As for the measured benefit, there have been a steady stream of 
postive reports on lkml.  My own experience is that the usability of my 
laptop with its small memory is much improved under heavy IO load.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 05:05:01PM +0100, Daniel Phillips wrote:
> > I think of "benefit", perhaps naiively, in terms of something that can
> > be measured or demonstrated rather than just announced.
> 
> But you see why asap scheduling improves latency/throughput *in theory*, 

Nope. And I don't even see a relationship between preemption and asap I/O
schedulding. What make you think that I/O threads won't be preempted by
other threads?

> don't you?  As for the measured benefit, there have been a steady stream of 
> postive reports on lkml. 

I have not seen a single well structured benchmark that shows a significant
difference. I've seen lots of benchmarks with odd mixes of different patches
showing something unknown. How about a simple clear dbench?

>My own experience is that the usability of my 
> laptop with its small memory is much improved under heavy IO load.

No comment.

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Peter Wächtler]

yodaiken@fsmlabs.com schrieb:
> 
> On Mon, Jan 21, 2002 at 05:05:01PM +0100, Daniel Phillips wrote:
> > > I think of "benefit", perhaps naiively, in terms of something that can
> > > be measured or demonstrated rather than just announced.
> >
> > But you see why asap scheduling improves latency/throughput *in theory*,
> 
> Nope. And I don't even see a relationship between preemption and asap I/O
> schedulding. What make you think that I/O threads won't be preempted by
> other threads?
> 

I/O intensive threads block early voluntarily - while CPU hogs don't.
Statistically there is a higher chance, that a CPU hog gets preempted
instead of an IO bound (that gives up the CPU in some useconds anyway)

The next IO request is hitting the device "earlier" - instead of waiting
for the next schedule() - that makes sense to me.

With this scenario the system CPU utilization gets "bigger" for the benefit
of "faster" IO. OTOH, seti@home needs longer to run.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 05:33:50PM +0100, Peter Wächtler wrote:
> yodaiken@fsmlabs.com schrieb:
> > 
> > On Mon, Jan 21, 2002 at 05:05:01PM +0100, Daniel Phillips wrote:
> > > > I think of "benefit", perhaps naiively, in terms of something that can
> > > > be measured or demonstrated rather than just announced.
> > >
> > > But you see why asap scheduling improves latency/throughput *in theory*,
> > 
> > Nope. And I don't even see a relationship between preemption and asap I/O
> > schedulding. What make you think that I/O threads won't be preempted by
> > other threads?
> > 
> 
> I/O intensive threads block early voluntarily - while CPU hogs don't.

Since the preemption patch only allows additional preemption in kernel
mode, I'm curious to know what the compute bound tasks are doing in 
kernel mode. Did Linux add in-kernel matrix multiplication while
I was not looking?


> Statistically there is a higher chance, that a CPU hog gets preempted
> instead of an IO bound (that gives up the CPU in some useconds anyway)


"Statistically"?  As far as I know, most I/O in Linux does not block.
When you say "statistically", you should have some analysis with clearly
stated assumptions. 

> 
> The next IO request is hitting the device "earlier" - instead of waiting
> for the next schedule() - that makes sense to me.
> 
> With this scenario the system CPU utilization gets "bigger" for the benefit
> of "faster" IO. OTOH, seti@home needs longer to run.

Sorry. No sale.


-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Peter Wächtler]

yodaiken@fsmlabs.com schrieb:
> 
> On Mon, Jan 21, 2002 at 05:33:50PM +0100, Peter Wächtler wrote:
> > yodaiken@fsmlabs.com schrieb:
> > >
> > > On Mon, Jan 21, 2002 at 05:05:01PM +0100, Daniel Phillips wrote:
> > > > > I think of "benefit", perhaps naiively, in terms of something that can
> > > > > be measured or demonstrated rather than just announced.
> > > >
> > > > But you see why asap scheduling improves latency/throughput *in theory*,
> > >
> > > Nope. And I don't even see a relationship between preemption and asap I/O
> > > schedulding. What make you think that I/O threads won't be preempted by
> > > other threads?
> > >
> >
> > I/O intensive threads block early voluntarily - while CPU hogs don't.
> 
> Since the preemption patch only allows additional preemption in kernel
> mode, I'm curious to know what the compute bound tasks are doing in
> kernel mode. Did Linux add in-kernel matrix multiplication while
> I was not looking?
> 

Dead right you are.
Then there are only slow system calls left. Umh, execve(), fork()
(with big address space) - what about page_launder etc.?


> > Statistically there is a higher chance, that a CPU hog gets preempted
> > instead of an IO bound (that gives up the CPU in some useconds anyway)
> 
> "Statistically"?  As far as I know, most I/O in Linux does not block.

You mean, the syscall returns without a reschedule?
Aehm, now it's time for some statistics where the kernel spents its time on ;-)

But what is a possible explanation for the people, who think their 
systems behave better with preemption - strong believe?

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 06:12:58PM +0100, Peter Wächtler wrote:
> > Since the preemption patch only allows additional preemption in kernel
> > mode, I'm curious to know what the compute bound tasks are doing in
> > kernel mode. Did Linux add in-kernel matrix multiplication while
> > I was not looking?
> > 
> 
> Dead right you are.
> Then there are only slow system calls left. Umh, execve(), fork()
> (with big address space) - what about page_launder etc.?

Those are, in some sense, I/O right? It's not clear to me
that preempting page_launder is sensible.

> But what is a possible explanation for the people, who think their 
> systems behave better with preemption - strong believe?

Beats me. Maybe it really does work - but maybe not and nobody
has advanced any analysis or numbers that make the case.



-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 21, 2002 05:06 pm, yodaiken@fsmlabs.com wrote:
> On Mon, Jan 21, 2002 at 05:05:01PM +0100, Daniel Phillips wrote:
> > > I think of "benefit", perhaps naiively, in terms of something that can
> > > be measured or demonstrated rather than just announced.
> > 
> > But you see why asap scheduling improves latency/throughput *in theory*, 
> 
> Nope. And I don't even see a relationship between preemption and asap I/O
> schedulding. What make you think that I/O threads won't be preempted by
> other threads?

Consider a thread reading from disk in such a way that readahead is no help, 
i.e., perhaps the disk is fragmented.  At each step the IO thread schedules a 
read and sleeps until the read completes, then schedules the next one.  At 
the same time there is a hog in the kernel, or perhaps there is 
competition from other tasks using the kernel.  In any event, it will 
frequently transpire that at the time the disk IO completes there is somebody 
in the kernel.  Without preemption the IO thread has to wait until the kernel 
hog blocks, hits a scheduling point or exits the kernel.

The result, without preemption, is:

         IO thread      Kernel hog       Disk
             |              .
             |--------------.-----------> .
             .              |             |
             .              |             |
             .              |             |
             .              |             |
             .              |<------------|
             .              |             .
             .              |             .
             .              |             .
             .<-------------|             .
             |--------------.-----------> .
             .              .             |
             .              .             |
             .              |             |
             .              |             |
             .              |             |
             .              |<------------|
             .              |             .
             .              |             .
             .              |             .
             .<-------------|             .
             |--------------.-----------> .
             .              .             |
             .              .             |
             .              |             |
             .              |             |
             .              |             |
             .              |<------------|
             .              |             .
             .              |             .
             .              |             .
             .<-------------|             .
             .

Whereas with preemption, we have:

         IO thread      Kernel hog       Disk
             |              .
             |--------------.-----------> .
             .              |             |
             .              |             |
             .              |             |
             .              |             |
             .<-------------|-------------|
             |--------------.-----------> .
             .              |             |
             .              |             |
             .              |             |
             .              |             |
             .<-------------|-------------|
             |--------------.-----------> .
             .              |             |
             .              |             |
             .              |             |
             .              |             |
             .<-------------|-------------|
             |

The disk and the IO thread are active a higher portion of the time, while the 
kernel hog gets the same amount of time.  So in this case we have improved 
both latency and throughput.

Naturally I constructed this case to show the effect most clearly.  There are 
many possible variations on the above scenario.  It does seem to explain the 
latency/throughput improvements that have been reported in practice.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 05:48:30PM +0100, Daniel Phillips wrote:
> On January 21, 2002 05:06 pm, yodaiken@fsmlabs.com wrote:
> > On Mon, Jan 21, 2002 at 05:05:01PM +0100, Daniel Phillips wrote:
> > > > I think of "benefit", perhaps naiively, in terms of something that can
> > > > be measured or demonstrated rather than just announced.
> > > 
> > > But you see why asap scheduling improves latency/throughput *in theory*, 
> > 
> > Nope. And I don't even see a relationship between preemption and asap I/O
> > schedulding. What make you think that I/O threads won't be preempted by
> > other threads?
> 
> Consider a thread reading from disk in such a way that readahead is no help, 
> i.e., perhaps the disk is fragmented.  At each step the IO thread schedules a 
> read and sleeps until the read completes, then schedules the next one.  At 
> the same time there is a hog in the kernel, or perhaps there is 
> competition from other tasks using the kernel.  In any event, it will 
> frequently transpire that at the time the disk IO completes there is somebody 
> in the kernel.  Without preemption the IO thread has to wait until the kernel 
> hog blocks, hits a scheduling point or exits the kernel.


So your claim is that:
	Preemption improves latency when there are both kernel cpu bound
	tasks and tasks that are I/O bound with very low cache hit
	rates?

Is that it?

Can you give me an example of a CPU bound task that runs
mostly in kernel? Doesn't that seem like a kernel bug?

> Naturally I constructed this case to show the effect most clearly.  There are 

How about a plausible case?

> many possible variations on the above scenario.  It does seem to explain the 
> latency/throughput improvements that have been reported in practice.

I still keep missing these reports. Can you help me here?
(Obviously "my laptop seems more effervescent" is not what I'm looking
 for.)





-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Chris Friesen]

yodaiken@fsmlabs.com wrote:

> So your claim is that:
>         Preemption improves latency when there are both kernel cpu bound
>         tasks and tasks that are I/O bound with very low cache hit
>         rates?
> 
> Is that it?
> 
> Can you give me an example of a CPU bound task that runs
> mostly in kernel? Doesn't that seem like a kernel bug?

cat /dev/urandom > /dev/null

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 12:32:57PM -0500, Chris Friesen wrote:
> yodaiken@fsmlabs.com wrote:
> 
> > So your claim is that:
> >         Preemption improves latency when there are both kernel cpu bound
> >         tasks and tasks that are I/O bound with very low cache hit
> >         rates?
> > 
> > Is that it?
> > 
> > Can you give me an example of a CPU bound task that runs
> > mostly in kernel? Doesn't that seem like a kernel bug?
> 
> cat /dev/urandom > /dev/null

Don't see any of Daniel's postulated long latencies there.  (Sorry, but
I'm having a hard time figuring out what is meant as a serious comment
here).


> 
> -- 
> Chris Friesen                    | MailStop: 043/33/F10  
> Nortel Networks                  | work: (613) 765-0557
> 3500 Carling Avenue              | fax:  (613) 765-2986
> Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Chris Friesen]

yodaiken@fsmlabs.com wrote:
> 
> On Mon, Jan 21, 2002 at 12:32:57PM -0500, Chris Friesen wrote:
> > yodaiken@fsmlabs.com wrote:
> >
> > > So your claim is that:
> > >         Preemption improves latency when there are both kernel cpu bound
> > >         tasks and tasks that are I/O bound with very low cache hit
> > >         rates?
> > >
> > > Is that it?
> > >
> > > Can you give me an example of a CPU bound task that runs
> > > mostly in kernel? Doesn't that seem like a kernel bug?
> >
> > cat /dev/urandom > /dev/null
> 
> Don't see any of Daniel's postulated long latencies there.  (Sorry, but
> I'm having a hard time figuring out what is meant as a serious comment
> here).

No, that one wasn't serious.  And while it is CPU bound and mostly in the
kernel, you're right that there are no long latencies to cause issues...

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Peter Wächtler]

yodaiken@fsmlabs.com schrieb:
> 
> On Mon, Jan 21, 2002 at 12:32:57PM -0500, Chris Friesen wrote:
> > yodaiken@fsmlabs.com wrote:
> >
> > > So your claim is that:
> > >         Preemption improves latency when there are both kernel cpu bound
> > >         tasks and tasks that are I/O bound with very low cache hit
> > >         rates?
> > >
> > > Is that it?
> > >
> > > Can you give me an example of a CPU bound task that runs
> > > mostly in kernel? Doesn't that seem like a kernel bug?
> >
> > cat /dev/urandom > /dev/null
> 
> Don't see any of Daniel's postulated long latencies there.  (Sorry, but
> I'm having a hard time figuring out what is meant as a serious comment
> here).
> 

This does not count for a "general" kernel:

zisofs reading
e2compr

I will try to compare with preemption kernel patch for out webbox like
device - but there I am mostly interested in "GUI feeling" - and will we
use e2compr on a 2.4.9ac kernel.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-21 at 11:50, yodaiken@fsmlabs.com wrote:
> On Mon, Jan 21, 2002 at 05:48:30PM +0100, Daniel Phillips wrote:

> > Consider a thread reading from disk in such a way that readahead is no help, 
> > i.e., perhaps the disk is fragmented.  At each step the IO thread schedules a 
> > read and sleeps until the read completes, then schedules the next one.  At 
> > the same time there is a hog in the kernel, or perhaps there is 
> > competition from other tasks using the kernel.  In any event, it will 
> > frequently transpire that at the time the disk IO completes there is somebody 
> > in the kernel.  Without preemption the IO thread has to wait until the kernel 
> > hog blocks, hits a scheduling point or exits the kernel.
> 
> 
> So your claim is that:
> 	Preemption improves latency when there are both kernel cpu bound
> 	tasks and tasks that are I/O bound with very low cache hit
> 	rates?
> 
> Is that it?
> 
> Can you give me an example of a CPU bound task that runs
> mostly in kernel? Doesn't that seem like a kernel bug?

It doesn't have to run mostly in the kernel.  It just has to be in the
kernel when the I/O-bound tasks awakes.  Further, there are plenty of
what we consider CPU-bound tasks that are interactive and/or
graphics-oriented and this adds much to their time in the kernel.

In a given period of time, a CPU bound task can run at any allotment
within it is given.  On the other hand, an I/O-bound task spends much
time blocked and thus can only run when I/O is available and it is
awake.  It is thus advantageous to schedule it within the bounds of the
I/O being available, and as tightly in those bounds as possible.  This
more fairly distributes scheduling to all tasks.  Same goes for RT
tasks, interactive tasks, etc.

The result is faster wake-up-to-run and thus higher throughput.  I just
sent some dbench scores to correlate this.

> I still keep missing these reports. Can you help me here?
> (Obviously "my laptop seems more effervescent" is not what I'm looking
>  for.)

While we certainly need tangible empirical benefits, users finding their
desktop experience smoother and thus more enjoyable is just about the
best thing we can ask for.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 04:22:58PM -0500, Robert Love wrote:
> On Mon, 2002-01-21 at 11:50, yodaiken@fsmlabs.com wrote:
> > On Mon, Jan 21, 2002 at 05:48:30PM +0100, Daniel Phillips wrote:
> 
> > > Consider a thread reading from disk in such a way that readahead is no help, 
> > > i.e., perhaps the disk is fragmented.  At each step the IO thread schedules a 
> > > read and sleeps until the read completes, then schedules the next one.  At 
> > > the same time there is a hog in the kernel, or perhaps there is 
> > > competition from other tasks using the kernel.  In any event, it will 
> > > frequently transpire that at the time the disk IO completes there is somebody 
> > > in the kernel.  Without preemption the IO thread has to wait until the kernel 
> > > hog blocks, hits a scheduling point or exits the kernel.
> > 
> > 
> > So your claim is that:
> > 	Preemption improves latency when there are both kernel cpu bound
> > 	tasks and tasks that are I/O bound with very low cache hit
> > 	rates?
> > 
> > Is that it?
> > 
> > Can you give me an example of a CPU bound task that runs
> > mostly in kernel? Doesn't that seem like a kernel bug?
> 
> It doesn't have to run mostly in the kernel.  It just has to be in the
> kernel when the I/O-bound tasks awakes.  Further, there are plenty of

How does that work? Won't the switch happen on exit from the kernel?

> what we consider CPU-bound tasks that are interactive and/or
> graphics-oriented and this adds much to their time in the kernel.

I'm not sure what an "interactive and/or graphics-oriented" CPU bound
task might be. Is there a definition?

> In a given period of time, a CPU bound task can run at any allotment
> within it is given.  On the other hand, an I/O-bound task spends much
> time blocked and thus can only run when I/O is available and it is
> awake.  It is thus advantageous to schedule it within the bounds of the
> I/O being available, and as tightly in those bounds as possible.  This
> more fairly distributes scheduling to all tasks.  Same goes for RT
> tasks, interactive tasks, etc.

So you think of an "I/O bound task" as  "an I/O bound task that spends
most of its timeblocked". Won't the latencies of such tasks already be
pretty high? I'd think that better caching and read-ahead is the correct
fix.


> The result is faster wake-up-to-run and thus higher throughput.  I just
> sent some dbench scores to correlate this.
> 
> > I still keep missing these reports. Can you help me here?
> > (Obviously "my laptop seems more effervescent" is not what I'm looking
> >  for.)
> 
> While we certainly need tangible empirical benefits, users finding their
> desktop experience smoother and thus more enjoyable is just about the
> best thing we can ask for.

It depends on what you want. 

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-21 at 16:54, yodaiken@fsmlabs.com wrote:

> > It doesn't have to run mostly in the kernel.  It just has to be in the
> > kernel when the I/O-bound tasks awakes.  Further, there are plenty of
> 
> How does that work? Won't the switch happen on exit from the kernel?

Sure, but we have essentially unbounded times in the kernel ...
I/O-bound tasks shouldn't have to wait for do_try_to_free_pages to
finish for some lower priority process.

> > what we consider CPU-bound tasks that are interactive and/or
> > graphics-oriented and this adds much to their time in the kernel.
> 
> I'm not sure what an "interactive and/or graphics-oriented" CPU bound
> task might be. Is there a definition?

I'm talking about today's GUI application.  It does computation and its
riddled with bloated GUI code.  So it is certainly CPU bound.  At the
same time it is interactive (blocking or polling on user input) and
involves some graphics output.  So it is involved in I/O, too.  What
would you consider it?

> So you think of an "I/O bound task" as  "an I/O bound task that spends
> most of its timeblocked". Won't the latencies of such tasks already be
> pretty high? I'd think that better caching and read-ahead is the correct
> fix.

I should correct myself, I didn't mean "most of its time" but a
statistically large portion.  All it needs to do is find itself woken
when something else is in the kernel.

> > While we certainly need tangible empirical benefits, users finding their
> > desktop experience smoother and thus more enjoyable is just about the
> > best thing we can ask for.
> 
> It depends on what you want. 

I want a better kernel.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Horst von Brand]

Robert Love <rml@tech9.net> said:

[...]

> It doesn't have to run mostly in the kernel.  It just has to be in the
> kernel when the I/O-bound tasks awakes.  Further, there are plenty of
> what we consider CPU-bound tasks that are interactive and/or
> graphics-oriented and this adds much to their time in the kernel.

Look, I don't know about you, but system (kernel) tieme around here is
rarely very high as a %. Perhaps 5% could be called "typical". And it is
during those 5% (i.e., something like 5% of the time) any of this stuff
will make a difference at all. This will be _hard_ to "feel" (if it is
possible to feel at all).

For the mostly positive (subjective) responses you see, there is something
called "psycology", which would predict that for _exactly_ the same "feel"
(whatever that may be) somebody who just made an effort downloading
patches, applying them, reconfiguring ad building a kernel "to make it feel
better" _will_ feel it better. I.e., nobody wants to have to say "Okay,
lots of work down the drain". Besides, those who see no difference will
shut up, those that delude themselves most will be vocal about it.

There was a famous experiment in determining productivity of people under
various circumstances. Whatever they changed, the productivity went
up. They finally had to conclude their results weren't due to the
environmental changes, but to the fact that the subjects felt important
(and motivated). Something similar might be happening here.

I'm not saying it _is_ like this, but as long as there are no reproducible
ways to put numbers to the "feel", and make controlled experiments, no one
will know for sure.

> In a given period of time, [...]

Yes, yes, we all know the theory, and it is obviously true. Question is
just, _how much_ is the change? Is it large enough to compensate for the
pain it causes?

[...]

> While we certainly need tangible empirical benefits, users finding their
> desktop experience smoother and thus more enjoyable is just about the
> best thing we can ask for.

It just isn't enough justification for wholesale redesign on basic
assumptions in the kernel, sorry.
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Chris Friesen]

Horst von Brand wrote:
> 
> Robert Love <rml@tech9.net> said:
> 
> [...]
> 
> > It doesn't have to run mostly in the kernel.  It just has to be in the
> > kernel when the I/O-bound tasks awakes.  Further, there are plenty of
> > what we consider CPU-bound tasks that are interactive and/or
> > graphics-oriented and this adds much to their time in the kernel.
> 
> Look, I don't know about you, but system (kernel) tieme around here is
> rarely very high as a %. Perhaps 5% could be called "typical". And it is
> during those 5% (i.e., something like 5% of the time) any of this stuff
> will make a difference at all. This will be _hard_ to "feel" (if it is
> possible to feel at all).

As a thought experiment...

1) top usually averages over 5 seconds
2) 5% of 5 seconds is 0.25 seconds


What I'm getting at is that it is possible that there are cases where we could
be taking significant amounts of time to respond to something, without the
overall average being too high.

If we have even a single 0.1 second delay, that's going to be noticeable to the
user without seriously bumping up system percentages.

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Mon, 21 Jan 2002, Horst von Brand wrote:

> Robert Love <rml@tech9.net> said:
> 
> [...]
> 
> > It doesn't have to run mostly in the kernel.  It just has to be in the
> > kernel when the I/O-bound tasks awakes.  Further, there are plenty of
> > what we consider CPU-bound tasks that are interactive and/or
> > graphics-oriented and this adds much to their time in the kernel.
	[ snip ]
> For the mostly positive (subjective) responses you see, there is something
> called "psycology", which would predict that for _exactly_ the same "feel"
> (whatever that may be) somebody who just made an effort downloading
> patches, applying them, reconfiguring ad building a kernel "to make it feel
> better" _will_ feel it better. I.e., nobody wants to have to say "Okay,
> lots of work down the drain". Besides, those who see no difference will
> shut up, those that delude themselves most will be vocal about it.

Ah, that's it, we're deluding ourselves. To the point that booting a
kernel without identification and having casual users watch the cursor
move while running a standard low will result in those users sharing our
delusion.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-21 at 11:48, Daniel Phillips wrote:

> The disk and the IO thread are active a higher portion of the time, while the 
> kernel hog gets the same amount of time.  So in this case we have improved 
> both latency and throughput.
> 
> Naturally I constructed this case to show the effect most clearly.  There are 
> many possible variations on the above scenario.  It does seem to explain the 
> latency/throughput improvements that have been reported in practice.

Well put.  I think this is exact the scenario we are observing.

Its the same benefit to latency.  We react quicker.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-21 at 11:06, yodaiken@fsmlabs.com wrote:

> I have not seen a single well structured benchmark that shows a significant
> difference. I've seen lots of benchmarks with odd mixes of different patches
> showing something unknown. How about a simple clear dbench?

I and many others have been posting benchmarks for months.

Here:

(average of 4 runs of `dbench 16')
2.5.3-pre1:		25.7608 MB/s
2.5.3-pre1-preempt:	32.341 MB/s

(old, average of 4 runs of `dbench 16')
2.5.2-pre11:		24.5364 MB/s
2.5.2-pre11-preempt:	27.5192 MB/s

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Robert Love wrote:
> 
> On Mon, 2002-01-21 at 11:06, yodaiken@fsmlabs.com wrote:
> 
> > I have not seen a single well structured benchmark that shows a significant
> > difference. I've seen lots of benchmarks with odd mixes of different patches
> > showing something unknown. How about a simple clear dbench?
> 
> I and many others have been posting benchmarks for months.
> 
> Here:
> 
> (average of 4 runs of `dbench 16')
> 2.5.3-pre1:             25.7608 MB/s
> 2.5.3-pre1-preempt:     32.341 MB/s
> 

Well I spent several hours last week trying to reproduce and
account for these observations and basically came up with
nothing.  The patch-induced variation was of the same order
as the inter-run variation.

You should publish the dbench dots.  They're most informative.
Look at these two:

16 clients started
..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................+++++++++++++++****************
Throughput 6.48209 MB/sec (NB=8.10261 MB/sec  64.8209 MBit/sec)

16 clients started
.................................................................................................................................................................................................................................+...................................................................................................................................................................................................................................................................+................................................................................+............................+++++++++++++****************
Throughput 7.76901 MB/sec (NB=9.71126 MB/sec  77.6901 MBit/sec)

These are identical runs.  Empty filesystem, 64 megs of memory.

Note how in the second run, a few clients terminated early,
and the throughput numbers increased quite markedly.

I don't know what causes this, and frankly I'm not really
interested.  It's some bizarre freaky dbench instability.

Similar effects occur with the `make -j12 bzImage' swapstorm
test.  After a while, all the `cc' instances start to
get synchronised at the onset of their peak memory demand.
The earlier and longer this happens, the worse the runtime.
It's an unstable system and tiny input perturbations create
large effects on the output.

Sorry, but these are not interesting, repeatable or stable workloads,
and I remain unconvinced about claims that low-latency or preempt
aid I/O throughput.  And even if a statistically significant
benefit _can_ be empirically demonstrated, it would be incautious
to claim a general benefit without a solid explanation of what
is causing it.  (Apart from the RAID5 kernel thread starvation
effect, which _is_ explained).

If someone can show me a sensible and repeatable I/O gain then
great, I can go in and work out exactly where it's coming from
and then we finally have a real, tangible, non-hand-wavy
explanation.  It may be there, but I just don't see it yet.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[J Sloan]

Remember, the -aa kernels are dbench champ -

They don't use preempt, but do use mini-ll...

Just a rant from the peanut gallery....

cu

jjs

Andrew Morton wrote:

>Robert Love wrote:
>
>>On Mon, 2002-01-21 at 11:06, yodaiken@fsmlabs.com wrote:
>>
>>>I have not seen a single well structured benchmark that shows a significant
>>>difference. I've seen lots of benchmarks with odd mixes of different patches
>>>showing something unknown. How about a simple clear dbench?
>>>
>>I and many others have been posting benchmarks for months.
>>
>>Here:
>>
>>(average of 4 runs of `dbench 16')
>>2.5.3-pre1:             25.7608 MB/s
>>2.5.3-pre1-preempt:     32.341 MB/s
>>
>
>Well I spent several hours last week trying to reproduce and
>account for these observations and basically came up with
>nothing.  The patch-induced variation was of the same order
>as the inter-run variation.
>
>You should publish the dbench dots.  They're most informative.
>Look at these two:
>
>16 clients started
>..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................+..............................................................................................+++++++++++++++****************
>Throughput 6.48209 MB/sec (NB=8.10261 MB/sec  64.8209 MBit/sec)
>
>16 clients started
>.................................................................................................................................................................................................................................+...................................................................................................................................................................................................................................................................+................................................................................+............................+++++++++++++****************
>Throughput 7.76901 MB/sec (NB=9.71126 MB/sec  77.6901 MBit/sec)
>
>These are identical runs.  Empty filesystem, 64 megs of memory.
>
>Note how in the second run, a few clients terminated early,
>and the throughput numbers increased quite markedly.
>
>I don't know what causes this, and frankly I'm not really
>interested.  It's some bizarre freaky dbench instability.
>
>Similar effects occur with the `make -j12 bzImage' swapstorm
>test.  After a while, all the `cc' instances start to
>get synchronised at the onset of their peak memory demand.
>The earlier and longer this happens, the worse the runtime.
>It's an unstable system and tiny input perturbations create
>large effects on the output.
>
>Sorry, but these are not interesting, repeatable or stable workloads,
>and I remain unconvinced about claims that low-latency or preempt
>aid I/O throughput.  And even if a statistically significant
>benefit _can_ be empirically demonstrated, it would be incautious
>to claim a general benefit without a solid explanation of what
>is causing it.  (Apart from the RAID5 kernel thread starvation
>effect, which _is_ explained).
>
>If someone can show me a sensible and repeatable I/O gain then
>great, I can go in and work out exactly where it's coming from
>and then we finally have a real, tangible, non-hand-wavy
>explanation.  It may be there, but I just don't see it yet.
>
>-
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 04:16:51PM -0500, Robert Love wrote:
> On Mon, 2002-01-21 at 11:06, yodaiken@fsmlabs.com wrote:
> 
> > I have not seen a single well structured benchmark that shows a significant
> > difference. I've seen lots of benchmarks with odd mixes of different patches
> > showing something unknown. How about a simple clear dbench?
> 
> I and many others have been posting benchmarks for months.
> 
> Here:
> 
> (average of 4 runs of `dbench 16')
> 2.5.3-pre1:		25.7608 MB/s
> 2.5.3-pre1-preempt:	32.341 MB/s
> 
> (old, average of 4 runs of `dbench 16')
> 2.5.2-pre11:		24.5364 MB/s
> 2.5.2-pre11-preempt:	27.5192 MB/s
> 

Robert, with all due respect, my tests of dbench show such high
variation that 4 miserable runs prove exactly nothing.
Did these even come on the same filesystem? 



---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-21 at 16:49, yodaiken@fsmlabs.com wrote:

> > (average of 4 runs of `dbench 16')
> > 2.5.3-pre1:		25.7608 MB/s
> > 2.5.3-pre1-preempt:	32.341 MB/s
> > 
> > (old, average of 4 runs of `dbench 16')
> > 2.5.2-pre11:		24.5364 MB/s
> > 2.5.2-pre11-preempt:	27.5192 MB/s

> Robert, with all due respect, my tests of dbench show such high
> variation that 4 miserable runs prove exactly nothing.

Well you asked for dbench.  Would you prefer 10 runs each?  There were,
however, no statistical anomalies and the variation was low enough such
that I suspect I could construct a reasonable confidence interval from
these 16 runs.

I've run these tests over and over again sufficiently that the
repeatability of obtaining improved marks under a preemptive kernel is
evident to me.

You can see very old (2.4.6) yet still positive results from Nigel, too:
http://kpreempt.sourceforge.net.

I guess the point is, everyone argues preemption is detrimental to
throughput.  I'm not going to argue that we aren't adding complexity,
because clearly we are.  But now we have tests showing throughput is
improved and people still argue.  I've seen the same behavior under
bonnie, timing kernel compiles, etc ...

> Did these even come on the same filesystem?

Yes, why would you suspect otherwise?

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Marcelo Tosatti]

On 21 Jan 2002, Robert Love wrote:

> On Mon, 2002-01-21 at 16:49, yodaiken@fsmlabs.com wrote:
> 
> > > (average of 4 runs of `dbench 16')
> > > 2.5.3-pre1:		25.7608 MB/s
> > > 2.5.3-pre1-preempt:	32.341 MB/s
> > > 
> > > (old, average of 4 runs of `dbench 16')
> > > 2.5.2-pre11:		24.5364 MB/s
> > > 2.5.2-pre11-preempt:	27.5192 MB/s
> 
> > Robert, with all due respect, my tests of dbench show such high
> > variation that 4 miserable runs prove exactly nothing.
> 
> Well you asked for dbench.  Would you prefer 10 runs each?  There were,
> however, no statistical anomalies and the variation was low enough such
> that I suspect I could construct a reasonable confidence interval from
> these 16 runs.
> 
> I've run these tests over and over again sufficiently that the
> repeatability of obtaining improved marks under a preemptive kernel is
> evident to me.
> 
> You can see very old (2.4.6) yet still positive results from Nigel, too:
> http://kpreempt.sourceforge.net.
> 
> I guess the point is, everyone argues preemption is detrimental to
> throughput.  I'm not going to argue that we aren't adding complexity,
> because clearly we are.  But now we have tests showing throughput is
> improved and people still argue.  I've seen the same behavior under
> bonnie, timing kernel compiles, etc ...

Sure, you've seen it. But _why_ it happens ?

That is the point.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-21 at 15:52, Marcelo Tosatti wrote:

> > I guess the point is, everyone argues preemption is detrimental to
> > throughput.  I'm not going to argue that we aren't adding complexity,
> > because clearly we are.  But now we have tests showing throughput is
> > improved and people still argue.  I've seen the same behavior under
> > bonnie, timing kernel compiles, etc ...
> 
> Sure, you've seen it. But _why_ it happens ?
> 
> That is the point.

Daniel just reiterated it, but I suspect we better multitask a mix of
tasks.  I/O-bound tasks that are woken can be run quicker and thus
throughput increases.

I'm not trying to tout preempt-kernel as a throughput solution.  I think
it is a neat and promising side-note to the patch, and one that
benchmarks are correlating.  Ignore it as a statistical error and
consider throughput untouched if you want. 

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 05:01:45PM -0500, Robert Love wrote:
> On Mon, 2002-01-21 at 16:49, yodaiken@fsmlabs.com wrote:
> 
> > > (average of 4 runs of `dbench 16')
> > > 2.5.3-pre1:		25.7608 MB/s
> > > 2.5.3-pre1-preempt:	32.341 MB/s
> > > 
> > > (old, average of 4 runs of `dbench 16')
> > > 2.5.2-pre11:		24.5364 MB/s
> > > 2.5.2-pre11-preempt:	27.5192 MB/s
> 
> > Robert, with all due respect, my tests of dbench show such high
> > variation that 4 miserable runs prove exactly nothing.
> 
> Well you asked for dbench.  Would you prefer 10 runs each?  There were,

50. And I'd like standard deviation as well as average, best, worst.

And then I'd like to see the same test done with a RT task running in
the background - since I assume preempt has as its main purpose
to enable SCHED_FIFO.

> I guess the point is, everyone argues preemption is detrimental to
> throughput.  I'm not going to argue that we aren't adding complexity,

I have not seen that argued - certainly I have not argued it myself.
My argument is:
	It makes the kernel _much_ more complex
	It has known costs e.g. by making the lockless 
		per-processor caching  more difficult if not impossible
	It seems to lead to a requirement for inheritance
	It has no demonstrated benefits.

> > Did these even come on the same filesystem?
> 
> Yes, why would you suspect otherwise?
> 

Because the prior cited benchmark was a kernel compile on different trees.
Was the filesystem unchanged between runs?
I'm not suggesting you are cheating, it's easy to overlook something 
critical when there are so many variables.


-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> I have not seen that argued - certainly I have not argued it myself.
> My argument is:

Which is your usual FUD and everything was discussed before, just
repeating claims without any arguments doesn't make it better. :-(

>         It makes the kernel _much_ more complex

It certainly adds complexity, but which new feature doesn't?
Please specify "_much_ more complex".

>         It has known costs e.g. by making the lockless
>                 per-processor caching  more difficult if not impossible

It would only expose the locking, which is currently hidden in the smp
infrastructure.
What's so "impossible" about it?

>         It seems to lead to a requirement for inheritance

Where? I still haven't seen any proof for this.

>         It has no demonstrated benefits.

http://kpreempt.sourceforge.net/benno/linux+kp-2.4.6/3x256.html

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Tue, Jan 22, 2002 at 01:45:37AM +0100, Roman Zippel wrote:
> Hi,
> 
> yodaiken@fsmlabs.com wrote:
> 
> > I have not seen that argued - certainly I have not argued it myself.
> > My argument is:
> 
> Which is your usual FUD and everything was discussed before, just
> repeating claims without any arguments doesn't make it better. :-(
> 

I was not talking to you and will not be.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

> I was not talking to you and will not be.

Thanks.
Somehow I have the feeling, you hadn't to tell me anything important
anyway. :-)

bye, Roman

PS: SCNR

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 22, 2002 12:56 am, yodaiken@fsmlabs.com wrote:
> I have not seen that argued - certainly I have not argued it myself.
> My argument is:
> 	It makes the kernel _much_ more complex

The patch itself is simple, so this must be an extended interpretation of the 
word 'complex'.

> 	It has known costs e.g. by making the lockless 
> 		per-processor caching  more difficult if not impossible

Not at all, the lazy man's way of dealing with this is to disable preemption 
around that code, an efficient operation.

> 	It seems to lead to a requirement for inheritance

I don't know about that.  From the (long) thread above, it looks like you 
haven't successfully proved the assertion that -preempt introduces any new 
inheritance requirement.

> 	It has no demonstrated benefits.

Demonstrated to who?  I have certainly demonstrated the benefits to myself, 
and others have attested to doing the same.

As far as arguments go, your main points don't seem to be rooted in firm 
ground at all.  On the other hand, the proponents of this patch have 
compelling arguments: it makes Linux feel smoother, it makes certain tests 
run faster, it doesn't slow anything down measurably, it's stable and so on.  
I even explained why it does what it does.  I don't understand why you're so 
vehemently opposed to this, especially as it's a config option.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Tue, Jan 22, 2002 at 03:10:42AM +0100, Daniel Phillips wrote:
> On January 22, 2002 12:56 am, yodaiken@fsmlabs.com wrote:
> > I have not seen that argued - certainly I have not argued it myself.
> > My argument is:
> > 	It makes the kernel _much_ more complex
> 
> The patch itself is simple, so this must be an extended interpretation of the 
> word 'complex'.

I'm at a loss here. You seem to be arguing that
there there is a relationship between the complexity of a patch
that changes the entire synchronization assumption of the
kernel and the complexity of the result. But that seems
unbelievable. Is there some component of your argument I've missed?

> 
> > 	It has known costs e.g. by making the lockless 
> > 		per-processor caching  more difficult if not impossible
> 
> Not at all, the lazy man's way of dealing with this is to disable preemption 
> around that code, an efficient operation.

Well, aside from how easy it will be to isolate all that information,
doesn't that defeat the purpose of the patch?  There is a big
difference in design between
	try to get fromverylightweight cache
		fallback to slow but fair and safe pool
and
	try to bound worst case times

In the first case we amortize worst case times by making average case
very low. This is a common design methodology in Linux kernel: semaphores
are the classic example.
So I'm sure that you can add any number of hacks to the kernel, but my
argument stands: per-processor caching is a common case optimization that
de-optimizes worst case. If the purpose of preemption is to reduce
latency, per-processor caching is at counter-purposes.


It's also worth pointing out that every use of cpu-specific information
is dangerous if preemption is extended to smp.
	x = smp_processor_id();
	//get preempted
	do_something(memslab[x]); // used to be safe since only current can
				// do this.

	
> > 	It seems to lead to a requirement for inheritance
> 
> I don't know about that.  From the (long) thread above, it looks like you 
> haven't successfully proved the assertion that -preempt introduces any new 
> inheritance requirement.

Oliver cited the trivial case. He was ignored.

> > 	It has no demonstrated benefits.
> 
> Demonstrated to who?  I have certainly demonstrated the benefits to myself, 
> and others have attested to doing the same.

I've heard similar arguments in favor of aromatherapy and Scientology.

What's amazing about all the arguments in favor of preemption is that we
don't see any published numbers of the obvious application: a periodic
SCHED_FIFO process. We've done these experiments and the results are
_dismal_. 
Even ignoring this, the repeated publication of numbers showing that
Andrew's patches get better results and the repeated statement by 
Andrew that the hard part of latency reduction
is _not_ solved by preemption alone
is continually met with repetitions of the same unsubstantiated chorus:
	But it is easier to maintain  



> As far as arguments go, your main points don't seem to be rooted in firm 
> ground at all.  On the other hand, the proponents of this patch have 
> compelling arguments: it makes Linux feel smoother, it makes certain tests 
> run faster, it doesn't slow anything down measurably, it's stable and so on.  
> I even explained why it does what it does.  I don't understand why you're so 
> vehemently opposed to this, especially as it's a config option.

What you proposed is a
claimed explanation of why a task that experienced regular unfixable
latencies of multiple milliseconds waiting for I/O would have additional
latencies possibly reduced by some unknown amount. You failed to make a
case that this is either something that actually happens or that it would
ever make any difference.

BTW: I've made no arguments that the patch should not be made an option.
I've argued that it is based on a very bad design premise. As for
whether it should be added to 2.5,  sold by other vendors, advertised on TV,
used when operating heavy machinery, or taken by pregnant women,
that's not up to me.


-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

I know you're not talking to me (you even removed me and only me from
the Cc list...) and personally I don't care, but other people might get
the impression you had something to say.

> > The patch itself is simple, so this must be an extended interpretation of the 
> > word 'complex'.
> 
> I'm at a loss here. You seem to be arguing that
> there there is a relationship between the complexity of a patch
> that changes the entire synchronization assumption of the
> kernel and the complexity of the result. But that seems
> unbelievable. Is there some component of your argument I've missed?

Which "entire synchronization assumption"? It changes timing
assumptions, which sometimes are used for synchronization. If something
breaks because of this, it was already broken before.
Maybe you could explain this "relationship" in a bit more detail?

> It's also worth pointing out that every use of cpu-specific information
> is dangerous if preemption is extended to smp.
>         x = smp_processor_id();
>         //get preempted
>         do_something(memslab[x]); // used to be safe since only current can
>                                 // do this.

This is a well known problem, which is also fixable. It's also known,
that these fixes would have a larger impact on the kernel, for that
reason preempt for smp is not a real option right now, so that it is
outside the scope of this discussion (or what's left it).

> > >     It seems to lead to a requirement for inheritance
> >
> > I don't know about that.  From the (long) thread above, it looks like you
> > haven't successfully proved the assertion that -preempt introduces any new
> > inheritance requirement.
> 
> Oliver cited the trivial case. He was ignored.

I really tried to find a detailed explanation of this case, but I
haven't found anything, it's possible I missed, so I'd be happy about
any pointer. From the hints I found that case seemed rather unlikely and
assuming a really bad scheduler.

> I've heard similar arguments in favor of aromatherapy and Scientology.

Now we are compared to this. Nice "arguments". :-(

> What's amazing about all the arguments in favor of preemption is that we
> don't see any published numbers of the obvious application: a periodic
> SCHED_FIFO process. We've done these experiments and the results are
> _dismal_.

url?

> Even ignoring this, the repeated publication of numbers showing that
> Andrew's patches get better results and the repeated statement by
> Andrew that the hard part of latency reduction
> is _not_ solved by preemption alone
> is continually met with repetitions of the same unsubstantiated chorus:
>         But it is easier to maintain

"repeated statement"? I've seen Andrew stating it once, repeated by you
and here is my (unanswered) response
http://marc.theaimsgroup.com/?l=linux-kernel&m=101100365432615&w=2.

> > As far as arguments go, your main points don't seem to be rooted in firm
> > ground at all.  On the other hand, the proponents of this patch have
> > compelling arguments: it makes Linux feel smoother, it makes certain tests
> > run faster, it doesn't slow anything down measurably, it's stable and so on.
> > I even explained why it does what it does.  I don't understand why you're so
> > vehemently opposed to this, especially as it's a config option.
> 
> What you proposed is a
> claimed explanation of why a task that experienced regular unfixable
> latencies of multiple milliseconds waiting for I/O would have additional
> latencies possibly reduced by some unknown amount. You failed to make a
> case that this is either something that actually happens or that it would
> ever make any difference.

You happily ignore the cases where it makes a difference. There might be
purely empirical, but they exist and you make no attempt at explaining
them.

> I've argued that it is based on a very bad design premise.

"Arguing" means for me to come up with arguments, which are really rare
from you. Maybe you can surprise us with some real arguments? I'd be
even happy about urls. So far I have only seen statements with very thin
backing. If I missed something, I still have an open offer for an
apology.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Jamie Lokier]

yodaiken@fsmlabs.com wrote:
> > > 	It has no demonstrated benefits.
> > 
> > Demonstrated to who?  I have certainly demonstrated the benefits to
> > myself, and others have attested to doing the same.
> 
> I've heard similar arguments in favor of aromatherapy and Scientology.
> 
> What's amazing about all the arguments in favor of preemption is that we
> don't see any published numbers of the obvious application: a periodic
> SCHED_FIFO process. We've done these experiments and the results are
> _dismal_. 

Hi Victor,

I am specifically interested in SCHED_FIFO performance (for a software
modem driver).  Can you publish thes results of yours of SCHED_FIFO
performance with & without the preempt patches?

Thanks,
-- Jamie

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Mon, 21 Jan 2002 yodaiken@fsmlabs.com wrote:

> I have not seen that argued - certainly I have not argued it myself.
> My argument is:
> 	It makes the kernel _much_ more complex
  It modifies a tiny fraction of a percent of the kernel, which is
currently simplistic rather than simple. Nearly everyone who looks at it
makes some improvement, be it preempt, low latency, etc.

> 	It has known costs e.g. by making the lockless 
> 		per-processor caching  more difficult if not impossible
  How much slowdown did you measure when you tested the effect of that?

> 	It seems to lead to a requirement for inheritance
  To the limited extent that I agree, so what?

> 	It has no demonstrated benefits.
  You have that backward. There are  many people who say they can see a
benefit, and no one has shown either a quantified bad impact or a single
user account which said it was worse. And I bet you looked, didn't you?

  I believe that a system will run better for a single user, and better
for a server with high interrupt rates, like DNS or web servers, where
many threads may be blocked on i/o, but there is significant CPU load as
well.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> > don't you?  As for the measured benefit, there have been a steady stream of
> > postive reports on lkml.
> 
> I have not seen a single well structured benchmark that shows a significant
> difference. I've seen lots of benchmarks with odd mixes of different patches
> showing something unknown. How about a simple clear dbench?

So let's explore the benchmark argument a bit. A usual some things need
to be clarified first.
It would be useful to know what we actually want to measure in the
benchmark. The main goal is to reduce scheduling latencies, so it would
make sense to test this. Results can be found at
http://www.gardena.net/benno/linux/audio/ and
http://kpreempt.sourceforge.net/. I haven't found a direct compare of
preemp+lockbreak and ll patch, but I wouldn't expect major differences
and it isn't really important. It is only important that both approaches
improve the scheduling latency considerably.
Now what in this thread is mostly mentioned are i/o benchmarks.
Improving i/o performance isn't really the main goal of the patches, so
it's only important that i/o performance isn't harmed. So far I haven't
seen any results indicating something like this, so case closed.
Victor, could you please explain, what are you trying to prove with the
benchmark argument??? I see you arguing a lot, but I don't really see
for what.
Finally my theory, why preempt performs better. If we assume that the
kernel spends too much time in kernel space, we only need to look at the
ll patch for possible latency problems. My favourites are two places -
copy_(to|from)_user and page table scan. Both are likely places for
loads like grep or kernel compile. I can see two possible effects here.
First lots of small i/o can be issued faster instead copying data
around. Second we spent too much time scanning the page tables looking
for freeable pages, while already enough i/o is scheduled, so instead
something useful is done and we just wait for the i/o to finish. Anyway,
anyone who really wants to know it, could modify the profiling code and
test where in kernel we schedule so often.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Mark Hahn]

> > > To me the benefit is clear enough: ASAP scheduling of IO threads, a 
> > > simple heuristic that improves both throughput and latency.
> > 
> > I think of "benefit", perhaps naiively, in terms of something that can
> > be measured or demonstrated rather than just announced.
> 
> But you see why asap scheduling improves latency/throughput *in theory*, 
> don't you?  

NO, IT DOES NOT. why can't you preempt-ophiles get that through your heads? 

	eager scheduling is NOT optimal in general.  

for instance, suppose my disk can only read a sector at a time.
scheduling my sequentially-reading process to wake eagerly
is most definitly PESSIMAL.  laziness is a cardinal virtue!
this doesn't preclude heuristics to sometimes short-cut the laziness.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Allan Sandfeld]

On Monday 21 January 2002 20:26, Mark Hahn wrote:
> > > > To me the benefit is clear enough: ASAP scheduling of IO threads, a
> > > > simple heuristic that improves both throughput and latency.
> > >
> > > I think of "benefit", perhaps naiively, in terms of something that can
> > > be measured or demonstrated rather than just announced.
> >
> > But you see why asap scheduling improves latency/throughput *in theory*,
> > don't you?
>
> NO, IT DOES NOT. why can't you preempt-ophiles get that through your heads?
>
> 	eager scheduling is NOT optimal in general.
>
> for instance, suppose my disk can only read a sector at a time.
> scheduling my sequentially-reading process to wake eagerly
> is most definitly PESSIMAL.  laziness is a cardinal virtue!
> this doesn't preclude heuristics to sometimes short-cut the laziness.
>
It's because your system is behaving wrongly for your dream to come true. If 
your want to handle several expected inputs from IO, you should ask it for an 
interrupt for every package. Rather you should rely on a timer function an 
periodically handle new data and in case of nothing new, go back to 
interrupts..

Eager scheduling is OPTIMAL for the sematics in your system. It thats is not 
optimal for throughput/whatever, it's the code that is wrong!

-Allan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Peter Wächtler]

Mark Hahn schrieb:
> 
> > > > To me the benefit is clear enough: ASAP scheduling of IO threads, a
> > > > simple heuristic that improves both throughput and latency.
> > >
> > > I think of "benefit", perhaps naiively, in terms of something that can
> > > be measured or demonstrated rather than just announced.
> >
> > But you see why asap scheduling improves latency/throughput *in theory*,
> > don't you?
> 
> NO, IT DOES NOT. why can't you preempt-ophiles get that through your heads?
> 
>         eager scheduling is NOT optimal in general.
> 
> for instance, suppose my disk can only read a sector at a time.
> scheduling my sequentially-reading process to wake eagerly
> is most definitly PESSIMAL.  laziness is a cardinal virtue!
> this doesn't preclude heuristics to sometimes short-cut the laziness.
> 

Do you think there are no other benefits besides the scheduling latency in
a realtime system?

In a realtime system you want your event handling code (outside of the
interrupt handler [on Linux: bottom halves/tasklets/sorftirq?) get running 
on the CPU as fast as possible. Therefore a realtime kernel is often fully 
preemptible (well, there are always critical sections that has to disable 
interrupts).

So the time between the interrupt handler wanting to schedule a specific 
task/thread and the next scheduling decision is crucial, right? 

I have no hard numbers, but I can imagine that this can also lead to
better IO (in terms of latency AND IO throughput but with the cost of 
cpu cycles [user space CPU throughput]).

I don't know the Linux kernel good enough right now, but if you shorten
the scheduling latency: that could be a win for faster IO. But there's always
a tradeoff: if you spent too much time in scheduling decisions/preparations
the overhead eats the lower latency (especially if your mutexes have to deal
with priority inversion, giving a lock holder at least the same priority as
the lock contender for the period it holds the lock).

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Mon, 21 Jan 2002 yodaiken@fsmlabs.com wrote:

> I think of "benefit", perhaps naiively, in terms of something that can
> be measured or demonstrated rather than just announced.

I guess there are people who assume that anything which can't be given a
number doesn't matter or possible doesn't exist. If software can't come up
with a number for the beauty of a sunset, cuteness of a baby, or taste of
a good wine, then obviously all that subjective stuff is meaningless.

However, since we have art, food, and wine critics making a living giving
their meaningless opinions, I guess the majority of us recognize that even
without a number produced by a benchmark there is "subjectively better." I
don't know of anyone who doesn't feel that the rmap patches, even with
some admited imprefections, make the system more responsive. I haven't
seen one person who questioned this after trying it.

Explaining responsiveness is like describing color to a blind person, any
quantifications totally miss the experience.

There are some responsemarks which may or may not be useful, feel free to
actually locate and run these and post the results instead of posting
multiple ways to ask for quantification. Linux people work on the things
that interest them, and most of us can tell a pea from a bowling ball
without a caliper.

Consider this a response to your other notes of similar nature.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 21, 2002 at 03:35:35PM -0500, Bill Davidsen wrote:
> However, since we have art, food, and wine critics making a living giving
> their meaningless opinions, I guess the majority of us recognize that even
> without a number produced by a benchmark there is "subjectively better." I
> don't know of anyone who doesn't feel that the rmap patches, even with
> some admited imprefections, make the system more responsive. I haven't
> seen one person who questioned this after trying it.

Thanks for stating the case so clearly.



-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Mark Hahn]

> Explaining responsiveness is like describing color to a blind person, any
> quantifications totally miss the experience.

you overestimate human uniqueness - we all have near-identical
perceptual hardware, and there *is* an absolute limit
beyond which no human can perceive.  for our purposes, let's
say it's 5ms.

> There are some responsemarks which may or may not be useful, feel free to
> actually locate and run these and post the results instead of posting

I posted "realfeel" last year, AKPM added some touches to it.
it's in his amlat bundle.  

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ken Brownfield]

I'm ranting, and not at you directly, Mark.  But I think it's important
to get perspective here.

On Mon, Jan 21, 2002 at 04:42:28PM -0500, Mark Hahn wrote:
| > Explaining responsiveness is like describing color to a blind person, any
| > quantifications totally miss the experience.
| 
| you overestimate human uniqueness - we all have near-identical
| perceptual hardware, and there *is* an absolute limit
| beyond which no human can perceive.  for our purposes, let's
| say it's 5ms.

A 5ms wavelength is 200Hz.  Whether xterm lines scroll with a 5ms or a
10ms latency will most likely not be perceivable by most people.  On the
other hand, audio and video processing is worthless with a 5ms latency,
especially when the current worst-case is atrocious.

5ms is clearly a poor choice for human latency, but hard/software
latencies can never be small enough, just as processor speeds can never
be fast enough.  And we certainly can't pull these debatably "absolute"
limits out of thin air.

Of course, Linux is just for people who don't actually want to USE an
operating system, only those who want to write kernel code and don't
want to have to worry about "needless" complexity.

It is an invalid statement that all kernel modifications need to have
benchmarks or human case studies to justify them.  There are many parts
and functions of these systems that are not well-specified by
benchmarks, not to mention that one person's benchmark is another
person's non-real-world invalid test.

Ultimately the maintainers need to make this decision, based on the
community's exploration of theory, practice, and from appropriate
benchmarks or other research.  Research like the O(1) scheduler, rmap,
preemptive, and others need SUPPORT from this community.  One random
person should not get in the way of progress in a kernel that is not
only intended for that one person's narrow view of the world.

That being said, this one person's narrow view of the world is quite
important.  But however cheesy it sounds, "the greater good" is the
priority.

IMHO, latency is a critical part of Linux's growth, as is SMP
scalability (and a hundred other projects).  These are, perhaps due to
hardware restrictions, not a top priority.  But work on this type of
advancement MUST occur, and it must occur without having to fend off
people whose sole basis for argument is that they don't need it, or that
it will be hard, or that they need proof from sources they are unwilling
to mention.

>From what I've seen of the preemption and latency discussions, the Linux
kernel is going to be constrained to old technology and old ideas for
the foreseeable future.  People unable or unwilling to see the light
insist on being able to see the chicken before the egg is hatched.

Can this community be open-source and research-oriented without seeing
the business model and profit forecasts first?

I hope so.
-- 
Ken.
brownfld@irridia.com


| > There are some responsemarks which may or may not be useful, feel free to
| > actually locate and run these and post the results instead of posting
| 
| I posted "realfeel" last year, AKPM added some touches to it.
| it's in his amlat bundle.  

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Mon, 21 Jan 2002, Mark Hahn wrote:

> you overestimate human uniqueness - we all have near-identical
> perceptual hardware, and there *is* an absolute limit
> beyond which no human can perceive.  for our purposes, let's
> say it's 5ms.

Let's say that if the original poster didn't see the difference (a) he has
NO functioning "perceptual hardware," or (b) he hasn't tried it, which I
invited him to do.
 
> > There are some responsemarks which may or may not be useful, feel free to
> > actually locate and run these and post the results instead of posting
> 
> I posted "realfeel" last year, AKPM added some touches to it.
> it's in his amlat bundle.  

I was making the point that no quantification is needed, rmap is at the
"wow that's better!" level overywhere I've tried it.

Now if someone could get the rmap performance with memory pressure, add
the -aa improvements to heavy i/o and large memory, and season with a
touch of J4 scheduler, I think we could have response which would blow
your fingers off the keyboard.

OT: has someone gotten 2.4.17 rmap-11c and J4 playing together? I looked
at it for about five minutes but had no time last night. 

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Jussi Laako]

Bill Davidsen wrote:
> 
> OT: has someone gotten 2.4.17 rmap-11c and J4 playing together? I looked
> at it for about five minutes but had no time last night.

It's in my -jl13 http://www.pp.song.fi/~visitor/linux/


 - Jussi Laako

-- 
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Tue, 22 Jan 2002, Jussi Laako wrote:

> Bill Davidsen wrote:
> > 
> > OT: has someone gotten 2.4.17 rmap-11c and J4 playing together? I looked
> > at it for about five minutes but had no time last night.
> 
> It's in my -jl13 http://www.pp.song.fi/~visitor/linux/

It was 15 by the time I tried it, but I'm still running it. What sched is
in pre7ac1? I looked at the post Alan put up and didn't see it on the
first pass. The Jn scheduler and and low latency seem to play well with
rmap, but the -aa changes to enable better bdflush tuning work well. I
don't want to tune for dbench, but I don't want to ignore it, either.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> It was 15 by the time I tried it, but I'm still running it. What sched is
> in pre7ac1? I looked at the post Alan put up and didn't see it on the
> first pass. The Jn scheduler and and low latency seem to play well with

I've not touched the scheduler at all. Its in somewhat of a bit of flux
and I'm trying to accumulate stuff to go to Marcelo sooner not later

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 14, 2002 02:45 pm, yodaiken@fsmlabs.com wrote:
> POSIX makes no specification of how scheduling classes interact - unless something changed
> in the new version.
> 
> But more than that, the problem of preemption is much more complex when you have
> task that do not share the "goodness fade" with everything else. That is, given a
> set of SCHED_OTHER processes at time T0, it is reasonable to design the scheduler so
> that there is some D so that by time T0+D each process has become the highest priority
> and has received cpu up to either a complete time slice or a I/O block. Linux kind of
> has this property now, and I believe that making this more robust and easier to analyze
> is going to be an enormously important issue.  However, once you add SCHED_FIFO in the
> current scheme, this becomes more complex. And with preempt, you cannot even offer the
> assurance that once a process gets the cpu it will make _any_ advance at all.

So the prediction here is that SCHED_FIFO + preempt can livelock some set of correctly
designed processes, is that it?  I don't see exactly how that could happen, though that
may simply mean I didn't read closely enough.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Russ Leighton]

This is getting silly ... feeback like "ll is better than PK", "feels 
smooth", "is reponsive",  "my kernel
compile is faster than yours", etc. is not getting us any closer to the 
"how" of making a better kernel.

What's the goal? How should SMP and NUMA behave? How is success measured?

It would be good to be very clear on the ultimate purpose before making 
radical changes. All of
these changes are dancing around some vague concept of 
reponsiveness...so define it!

These comments seem to set a better tone for this thread, perhaps we can 
concentrate  on _useful_ debate
around some well defined goal.

yodaiken@fsmlabs.com wrote:

> The key one is some idea of being able to assure processes
> of some rate of progress.  This is not classical RT, but it is important to multimedia and 
> databases and also to some applications we are interested in looking at. 


Andrew Morton wrote:

> But we can **make** it useful.  I believe that internal preemption is
> the foundation to improve 2.5 kernel latency.  But first we need
> consensus that we **want** linux to be a low-latency kernel.
> 
> Do we have that?
> 
> If we do, then as I've said before, holding a lock for more than N milliseconds
> becomes a bug to be fixed.  We can put tools in the hands of testers to
> locate those bugs.  Easy.
> 

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Arjan van de Ven]

On Mon, Jan 14, 2002 at 06:03:43AM +0100, Daniel Phillips wrote:
> Sorry, that's incorrect.  I stated why earlier in this thread and akpm signed 
> off on it.  With preempt you get ASAP (i.e., as soon as the outermost 
> spinlock is done) process scheduling.  With hand-coded scheduling points you 
> get 'as soon as it happens to hit a scheduling point'.
> 
> That is not the only benefit, just the most obvious one.

Big duh. So you get there 1 usec sooner. NOBODY will notice that. NOBODY.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> What somehow got lost in this discussion, that both patches don't
> necessarily conflict with each other, they both attack the same problem
> with different approaches, which complement each other. I prefer to get
> the best of both patches.

When you look at the benchmark there is no difference between ll and 
ll+pre-empt. ll alone takes you to the 1ms point. pre-empt takes you no
further and to get much out of pre-emption requires you go and do all the
hideously slow and complex priority inversion stuff.

> exactly that reason. I don't think we need to work around broken
> hardware, but halfway decent hardware should not be a problem to get
> decent latency.

We have to work around common hardware not designed for SMP - the 8390 isnt
a broken chip in that sense, its just from a different era, and there are a 
lot of them.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

Alan Cox wrote:

> > What somehow got lost in this discussion, that both patches don't
> > necessarily conflict with each other, they both attack the same problem
> > with different approaches, which complement each other. I prefer to get
> > the best of both patches.
> 
> When you look at the benchmark there is no difference between ll and
> ll+pre-empt. ll alone takes you to the 1ms point.

I don't doubt that, but would you seriously consider the ll patch for
inclusion into the main kernel?
It's a useful patch for anyone, who needs good latencies now, but it's
still a quick&dirty solution. Preempt offers a clean solution for a
certain part of the problem, as it's possible to cleanly localize the
needed changes for preemption (at least for UP). That means the ll patch
becomes smaller and future work on ll becomes simpler, since a certain
type of latency problems is handled automatically (and transparently),
so you do gain something by it.
The remaining places pointed out in the ll patch are worth a closer look
as well, as mostly now we hold a spinlock for too long. These should be
fixed as well, as they mean possible contention problems on SMP.

> pre-empt takes you no
> further and to get much out of pre-emption requires you go and do all the
> hideously slow and complex priority inversion stuff.

The possibility of priority inversion problems are not new, it was
already discussed before. It was considered not a serious problem, since
all processes will still make progress. Preempt now increases the
likeliness such a situation occurs, but nonetheless the processes will
still make progress. In the past I can't remember any report that
indicated a problem caused by priority inversion and so I simply can't
believe it should become a massive problem now with preempt.

> > exactly that reason. I don't think we need to work around broken
> > hardware, but halfway decent hardware should not be a problem to get
> > decent latency.
> 
> We have to work around common hardware not designed for SMP - the 8390 isnt
> a broken chip in that sense, its just from a different era, and there are a
> lot of them.

Please let me rephrase, I just don't expect terrible good latency
numbers with non dma hardware.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> I don't doubt that, but would you seriously consider the ll patch for
> inclusion into the main kernel?

The mini ll patch definitely. The full ll one needs some head scratching to
be sure its correct. pre-empt is a 2.5 thing which in some ways is easier
because it doesnt matter if it breaks something.

> Please let me rephrase, I just don't expect terrible good latency
> numbers with non dma hardware.

Expect the same with DMA hardware too at times.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Stephan von Krawczynski]

> > I don't doubt that, but would you seriously consider the ll patch 
for                                                                   
> > inclusion into the main kernel?                                   
>                                                                     
> The mini ll patch definitely.                                       
                                                                      
Huh?                                                                  
Can you point at anyone who experienced a significant benefit from it?
I can see a lot of interesting patches ahead if you let this go.      
Tell me honestly that the idea behind this patch is not _crap_. You   
can only make this basic idea work if you patch a tremendous lot of   
those conditional_schedules() through the kernel. We already saw it   
starting off in some graphics drivers, network drivers. Why not just  
all of it? You will not be far away in the end from the 'round 4000 I 
already stated in earlier post.                                       
I do believe Roberts' preempt is a lot cleaner in its idea _how_ to   
achieve basically the same goal. Although I am at least as sceptic as 
you about a race-free implementation.                                 
                                                                      
> The full ll one needs some head scratching to                       
> be sure its correct.                                                
                                                                      
You may simply call it _counting_ (the files to patch).               
                                                                      
> pre-empt is a 2.5 thing which in some ways is easier                
> because it doesnt matter if it breaks something.                    
                                                                      
So I understand you agree somehow with me in the answer to "what idea 
is really better?"...                                                 
                                                                      
Regards,                                                              
Stephan                                                               
                                                                      
                                                                      

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Tell me honestly that the idea behind this patch is not _crap_. You   
> can only make this basic idea work if you patch a tremendous lot of   
> those conditional_schedules() through the kernel. We already saw it   
> starting off in some graphics drivers, network drivers. Why not just  
> all of it? You will not be far away in the end from the 'round 4000 I 
> already stated in earlier post.                                       

There are very few places you need to touch to get a massive benefit. Most
of the kernel already behaves extremely well.

> So I understand you agree somehow with me in the answer to "what idea 
> is really better?"...                                                 

Do you want a clean simple solution or complex elegance ? For 2.4 I definitely
favour clean and simple. For 2.5 its an open debate

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 19:50, Alan Cox wrote:

> Do you want a clean simple solution or complex elegance ? For 2.4 I definitely
> favour clean and simple. For 2.5 its an open debate

Make no mistake, I do not intend to see preempt-kernel in 2.4.  I will,
however, continue to maintain the patch for endusers and such that use
it.  A proper in-kernel solution for 2.4 in my opinion in mini-ll,
perhaps extend with any other obviously-completely-utterly sane bits
from full-ll.

For 2.5, however, I tout preempt as the answer.  This does not mean just
preempt.  It means a preemptible kernel as a basis for beginning
low-latency works in manners other than explicit scheduling statements.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Stephan von Krawczynski]

On 13 Jan 2002 20:17:11 -0500
Robert Love <rml@tech9.net> wrote:

> For 2.5, however, I tout preempt as the answer.  This does not mean just
> preempt.  It means a preemptible kernel as a basis for beginning
> low-latency works in manners other than explicit scheduling statements.

Aha, exactly my thoughts...

Regards,
Stephan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Stephan von Krawczynski]

On Mon, 14 Jan 2002 00:50:54 +0000 (GMT)
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> > all of it? You will not be far away in the end from the 'round 4000 I 
> > already stated in earlier post.                                       
> 
> There are very few places you need to touch to get a massive benefit. Most
> of the kernel already behaves extremely well.

Just a short question: the last (add-on) patch to mini-ll I saw on the list
patches: drivers/net/3c59x.c
drivers/net/8139too.c
drivers/net/eepro100.c

Unfortunately me have neither of those. This would mean I cannot benefit from
_these_ patches, but instead would need _others_ (like tulip or
name-one-of-the-rest-of-the-drivers) to see _some_ effect you tell me I
_should_ see (I currently see _none_). How do you argue then against the
statement: we need patches for /drivers/net/*.c ?? I do not expect 3c59x.c to
be particularly bad in comparison to tulip/*.c or lets say via-rhine.c, do you?

> > So I understand you agree somehow with me in the answer to "what idea 
> > is really better?"...                                                 
> 
> Do you want a clean simple solution or complex elegance ? For 2.4 I
definitely> favour clean and simple. For 2.5 its an open debate

Hm, obviously the ll-patches look simple, but their pure required number makes
me think they are as well stupid as simple. This whole story looks like making
an old mac do real multitasking, just spread around scheduling points
throughout the code ... This is like drilling for water on top of the mountain.
I want the water too, but I state there must be a nice valley somewhere around
...

Regards,
Stephan
 

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Stephan von Krawczynski wrote:
> 
> ...
> Unfortunately me have neither of those. This would mean I cannot benefit from
> _these_ patches, but instead would need _others_ (like tulip or
> name-one-of-the-rest-of-the-drivers) to see _some_ effect you tell me I
> _should_ see (I currently see _none_). How do you argue then against the
> statement: we need patches for /drivers/net/*.c ?? I do not expect 3c59x.c to
> be particularly bad in comparison to tulip/*.c or lets say via-rhine.c, do you?
> 

In 3c59x.c, probably the biggest problem will be the call to issue_and_wait()
in boomerang_start_xmit().  On a LAN which is experiencing heavy collision rates
this can take as long as 2,000 PCI cycles (it's quite rare, and possibly an
erratum).  It is called under at least two spinlocks.

In via-rhine, wait_for_reset() can busywait for up to ten milliseconds.
via_rhine_tx_timeout() calls it from under a spinlock.

In eepro100.c, wait_for_cmd_done() can busywait for one millisecond
and is called multiple times under spinlock.



Preemption will help _some_ of this, but by no means all, or enough.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Stephan von Krawczynski]

On Mon, 14 Jan 2002 02:04:56 -0800
Andrew Morton <akpm@zip.com.au> wrote:

> Stephan von Krawczynski wrote:
> > 
> > ...
> > Unfortunately me have neither of those. This would mean I cannot benefit
from> > _these_ patches, but instead would need _others_ 
[...]
> 
> In 3c59x.c, probably the biggest problem will be the call to issue_and_wait()
> in boomerang_start_xmit().  On a LAN which is experiencing heavy collision
rates> this can take as long as 2,000 PCI cycles (it's quite rare, and possibly
an> erratum).  It is called under at least two spinlocks.
> 
> In via-rhine, wait_for_reset() can busywait for up to ten milliseconds.
> via_rhine_tx_timeout() calls it from under a spinlock.
> 
> In eepro100.c, wait_for_cmd_done() can busywait for one millisecond
> and is called multiple times under spinlock.

Did I get that right, as long as spinlocked no sense in conditional_schedule()
?

> Preemption will help _some_ of this, but by no means all, or enough.

Maybe we should really try to shorten the lock-times _first_. You mentioned a
way to find the bad guys?

Regards,
Stephan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> > In eepro100.c, wait_for_cmd_done() can busywait for one millisecond
> > and is called multiple times under spinlock.
> 
> Did I get that right, as long as spinlocked no sense in conditional_schedule()
> ?

No conditional schedule, no pre-emption. You would need to rewrite that code
to do something like try for 100uS then queue a 1 tick timer to retry
asynchronously. That makes the code vastly more complex for an error case and
for some drivers where irq mask is required during reset waits won't help.

Yet again there are basically 1mS limitations buried in the hardware.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Jussi Laako]

Alan Cox wrote:
> 
> > > In eepro100.c, wait_for_cmd_done() can busywait for one millisecond
> > > and is called multiple times under spinlock.
> >
> > Did I get that right, as long as spinlocked no sense in 
> > conditional_schedule()
> 
> No conditional schedule, no pre-emption. You would need to rewrite that 
> code to do something like try for 100uS then queue a 1 tick timer to 
> retry asynchronously. That makes the code vastly more complex for an 
> error case and for some drivers where irq mask is required during reset 
> waits won't help.

That wait_for_cmd_done() and similar functions in other drivers are called
let's say 3 times in interrupt handler or spinlocked routine and 20 times in
non-interrupts disabled nor spinlocked functions.

Spinlocked reqions are usually protected by spin_lock_irqsave().

So the code reads

	if (!spin_is_locked(sl))
		conditional_schedule();

This doesn't make the whole problem go away, but could make the situation a
little bit better for most of the time?


 - Jussi Laako

-- 
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Stephan von Krawczynski]

> Alan Cox wrote:                                                     
> >                                                                   
> > > > In eepro100.c, wait_for_cmd_done() can busywait for one       
millisecond                                                           
> > > > and is called multiple times under spinlock.                  
> > >                                                                 
> > > Did I get that right, as long as spinlocked no sense in         
> > > conditional_schedule()                                          
> >                                                                   
> > No conditional schedule, no pre-emption. You would need to rewrite
that                                                                  
> > code to do something like try for 100uS then queue a 1 tick timer 
to                                                                    
> > retry asynchronously. That makes the code vastly more complex for 
an                                                                    
> > error case and for some drivers where irq mask is required during 
reset                                                                 
> > waits won't help.                                                 
>                                                                     
> That wait_for_cmd_done() and similar functions in other drivers are 
called                                                                
> let's say 3 times in interrupt handler or spinlocked routine and 20 
times in                                                              
> non-interrupts disabled nor spinlocked functions.                   
>                                                                     
> Spinlocked reqions are usually protected by spin_lock_irqsave().    
                                                                      
Now I have really waited for this one: _usually_.                     
My servers work usually, except for the days they hit that other      
rather unusual part of the code...                                    
                                                                      
> So the code reads                                                   
>                                                                     
> 	if (!spin_is_locked(sl))                                           
> 		conditional_schedule();                                           
>                                                                     
> This doesn't make the whole problem go away, but could make the     
situation a                                                           
> little bit better for most of the time?                             
                                                                      
Time to take out these big hats and rename ourself to Gandalf or the  
like. What do you expect your server to do, having no problem "most of
the time"? Please read Albert E. Time can be pretty relative to your  
personal point of view...                                             
                                                                      
Regards,                                                              
Stephan                                                               
                                                                      
                                                                      

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Jussi Laako]

Stephan von Krawczynski wrote:
> 
> Time to take out these big hats and rename ourself to Gandalf or the
> like. What do you expect your server to do, having no problem "most of
> the time"? Please read Albert E. Time can be pretty relative to your
> personal point of view...

Is this flaming really necessary?

So we don't have care about how long some driver spends in it internal
loops? So we could as well start writing drivers like 

	while (!frame_received()) udelay(1000000);

Because one day we will have powershortage anyway and that will anyway cause
few hours latencypeak? And if the user pulls the ethernet plug we don't have
to do anything else?


 - Jussi Laako

-- 
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers

Re: [2.4.17/18pre] VM and swap - it's really unusable

[george anzinger]

Stephan von Krawczynski wrote:
> 
> On Mon, 14 Jan 2002 02:04:56 -0800
> Andrew Morton <akpm@zip.com.au> wrote:
> 
> > Stephan von Krawczynski wrote:
> > >
> > > ...
> > > Unfortunately me have neither of those. This would mean I cannot benefit
> from> > _these_ patches, but instead would need _others_
> [...]
> >
> > In 3c59x.c, probably the biggest problem will be the call to issue_and_wait()
> > in boomerang_start_xmit().  On a LAN which is experiencing heavy collision
> rates> this can take as long as 2,000 PCI cycles (it's quite rare, and possibly
> an> erratum).  It is called under at least two spinlocks.
> >
> > In via-rhine, wait_for_reset() can busywait for up to ten milliseconds.
> > via_rhine_tx_timeout() calls it from under a spinlock.
> >
> > In eepro100.c, wait_for_cmd_done() can busywait for one millisecond
> > and is called multiple times under spinlock.
> 
> Did I get that right, as long as spinlocked no sense in conditional_schedule()
> ?
> 
> > Preemption will help _some_ of this, but by no means all, or enough.
> 
> Maybe we should really try to shorten the lock-times _first_. You mentioned a
> way to find the bad guys?

Apply the preempt patch and then the preempt-stats patch.  Follow
instructions that come with the stats patch.  It will report on the
longest preempt disable times since the last report.  You need to
provide a load that will exercise the bad code, but it will tell you
which, where, and how bad.  Note: it measures preempt off time, NOT how
long it took to get to some task, i.e. it does not depend on requesting
preemption at the worst possible time.
-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Just a short question: the last (add-on) patch to mini-ll I saw on the list
> patches: drivers/net/3c59x.c
> drivers/net/8139too.c
> drivers/net/eepro100.c

I've seen multiple quite frankly bizarre patches like that. I've not applied
them and don't see the point

Re: [2.4.17/18pre] VM and swap - it's really unusable

[J.A. Magallon]

On 20020114 Stephan von Krawczynski wrote:
>
>Hm, obviously the ll-patches look simple, but their pure required number makes
>me think they are as well stupid as simple. This whole story looks like making
>an old mac do real multitasking, just spread around scheduling points

Yup. That remind me of...
Would there be any kernel call every driver is doing just to hide there
a conditional_schedule() so everyone does it even without knowledge of it ?
Just like Apple put the SystemTask() inside GetNextEvent()...

-- 
J.A. Magallon                           #  Let the source be with you...        
mailto:jamagallon@able.es
Mandrake Linux release 8.2 (Cooker) for i586
Linux werewolf 2.4.18-pre3-beo #5 SMP Sun Jan 13 02:14:04 CET 2002 i686

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Arjan van de Ven]

"J.A. Magallon" wrote:
> 
> On 20020114 Stephan von Krawczynski wrote:
> >
> >Hm, obviously the ll-patches look simple, but their pure required number makes
> >me think they are as well stupid as simple. This whole story looks like making
> >an old mac do real multitasking, just spread around scheduling points
> 
> Yup. That remind me of...
> Would there be any kernel call every driver is doing just to hide there
> a conditional_schedule() so everyone does it even without knowledge of it ?
> Just like Apple put the SystemTask() inside GetNextEvent()...


Well the preempt patch sort of does this in every spin_unlock*() .....

Re: [2.4.17/18pre] VM and swap - it's really unusable

[george anzinger]

Arjan van de Ven wrote:
> 
> "J.A. Magallon" wrote:
> >
> > On 20020114 Stephan von Krawczynski wrote:
> > >
> > >Hm, obviously the ll-patches look simple, but their pure required number makes
> > >me think they are as well stupid as simple. This whole story looks like making
> > >an old mac do real multitasking, just spread around scheduling points
> >
> > Yup. That remind me of...
> > Would there be any kernel call every driver is doing just to hide there
> > a conditional_schedule() so everyone does it even without knowledge of it ?
> > Just like Apple put the SystemTask() inside GetNextEvent()...
> 
> Well the preempt patch sort of does this in every spin_unlock*() .....
> -
Gosh, not really.  The nature of the preempt patch is to allow
preemption on completion of the interrupt that put the contending task
back in the run list.  This can not be done if a spin lock is held, so
yes, there is a test on exit from the spin lock, but the point is that
this is only needed when the lock is release, not in unlocked code.

The utility of most of the ll patches is that they address the problem
within locked regions.  This is why there is a lock-break patch that is
designed to augment the preempt patch.  It picks up several of the long
held spin locks and pops out of them early to allow preemption, and then
relocks and continues (after picking up the pieces, of course).
-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-14 at 10:02, J.A. Magallon wrote:

> Yup. That remind me of...
> Would there be any kernel call every driver is doing just to hide there
> a conditional_schedule() so everyone does it even without knowledge of it ?
> Just like Apple put the SystemTask() inside GetNextEvent()...

It's not nearly that easy.  If it were, we would all certainly switch to
the preemptive kernel design, and preempt whenever and wherever we
needed.

Instead, we have to worry about reentrancy and thus can not preempt
inside critical regions (denoted by spinlocks).  So we can't have
preempt there, and have more work to do -- thus this discussion.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rob Landley]

On Monday 14 January 2002 02:35 pm, Robert Love wrote:
> On Mon, 2002-01-14 at 10:02, J.A. Magallon wrote:
> > Yup. That remind me of...
> > Would there be any kernel call every driver is doing just to hide there
> > a conditional_schedule() so everyone does it even without knowledge of it
> > ? Just like Apple put the SystemTask() inside GetNextEvent()...
>
> It's not nearly that easy.  If it were, we would all certainly switch to
> the preemptive kernel design, and preempt whenever and wherever we
> needed.
>
> Instead, we have to worry about reentrancy and thus can not preempt
> inside critical regions (denoted by spinlocks).  So we can't have
> preempt there, and have more work to do -- thus this discussion.
>
> 	Robert Love

The real question is: what can get in.

Variants of the explicit scheduling points have been around for over a year, 
since Ingo's original version.  Just a few days ago Marcello once again said 
that if all the patch does is add scheduling points, he had no intention of 
integrating it.  Linus's opinion on the matter has pretty much been about the 
same since Ingo's version.

If explicit scheduling points ARE a better first step than preempt (which 
doesn't necessarily elminate preempt, it just lets us move forward while 
arguing), when the heck might they possibly appear in a mainline kernel we 
don't have to manually patch each new release of?

Rob

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Jussi Laako]

Stephan von Krawczynski wrote:
> 
> Just a short question: the last (add-on) patch to mini-ll I saw on the 

It was for full-ll, not for mini-ll.

> patches: drivers/net/3c59x.c
> drivers/net/8139too.c
> drivers/net/eepro100.c
> 
> Unfortunately me have neither of those. This would mean I cannot benefit 
> from _these_ patches, but instead would need _others_ (like tulip or
> name-one-of-the-rest-of-the-drivers) to see _some_ effect you tell me I
> _should_ see (I currently see _none_). How do you argue then against the
> statement: we need patches for /drivers/net/*.c ?? I do not expect 
> 3c59x.c to be particularly bad in comparison to tulip/*.c or lets say 
> via-rhine.c, do you?

I also checked the tulip driver (which is the one I use at home) and didn't
find need for "fixing" there. I will definitely take a closer look at that
driver in future.

WLAN drivers seem to need some hacking, but I'm not very interested in that
area. I think WLAN is one big security hole that noone should be using...


 - Jussi Laako

-- 
PGP key fingerprint: 161D 6FED 6A92 39E2 EB5B  39DD A4DE 63EB C216 1E4B
Available at PGP keyservers

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Stephan von Krawczynski]

> Stephan von Krawczynski wrote:                                      
                                                                      
> I also checked the tulip driver (which is the one I use at home) and
didn't                                                                
> find need for "fixing" there. I will definitely take a closer look  
at that                                                               
> driver in future.                                                   
                                                                      
Aha, I see, everything as expected...                                 
                                                                      
> WLAN drivers seem to need some hacking, but I'm not very interested 
in that                                                               
> area. I think WLAN is one big security hole that noone should be    
using...                                                              
                                                                      
Uhm, I got hot news: internet is insecure.                            
Come on, this is really silly, every network is a security problem,   
not speaking of the guys sitting in front of the screens...           
                                                                      
Regards,                                                              
Stephan                                                               
                                                                      
                                                                      

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sun, Jan 13, 2002 at 09:25:50PM +0100, Roman Zippel wrote:
> I don't doubt that, but would you seriously consider the ll patch for
> inclusion into the main kernel?
> It's a useful patch for anyone, who needs good latencies now, but it's
> still a quick&dirty solution. Preempt offers a clean solution for a
> certain part of the problem, as it's possible to cleanly localize the
> needed changes for preemption (at least for UP). That means the ll patch
> becomes smaller and future work on ll becomes simpler, since a certain

That is exactly what Andrew Morton disputes. So why do you think he is
wrong?

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> > It's a useful patch for anyone, who needs good latencies now, but it's
> > still a quick&dirty solution. Preempt offers a clean solution for a
> > certain part of the problem, as it's possible to cleanly localize the
> > needed changes for preemption (at least for UP). That means the ll patch
> > becomes smaller and future work on ll becomes simpler, since a certain
> 
> That is exactly what Andrew Morton disputes. So why do you think he is
> wrong?

Please explain, what do you mean?

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 01:41:35AM +0100, Roman Zippel wrote:
> Hi,
> 
> yodaiken@fsmlabs.com wrote:
> 
> > > It's a useful patch for anyone, who needs good latencies now, but it's
> > > still a quick&dirty solution. Preempt offers a clean solution for a
> > > certain part of the problem, as it's possible to cleanly localize the
> > > needed changes for preemption (at least for UP). That means the ll patch
> > > becomes smaller and future work on ll becomes simpler, since a certain
> > 
> > That is exactly what Andrew Morton disputes. So why do you think he is
> > wrong?
> 
> Please explain, what do you mean?

I mean, that these conversations are not very useful if you don't
read what the other people write.
Here's a prior response by Andrew to a post by you.

>From akpm@zip.com.au  Sat Jan 12 13:15:22 2002
Roman Zippel wrote:
> 
> Andrew's patch requires constant audition and Andrew can't audit all
> drivers for possible problems. That doesn't mean Andrew's work is
> wasted, since it identifies problems, which preempting can't solve, but
> it will always be a hunt for the worst cases, where preempting goes for
> the general case.

Guys,

I've heard this so many times, and it just ain't so.   The overwhelming
majority of problem areas are inside locks.  All the complexity and 
maintainability difficulties to which you refer exist in the preempt
patch as well.    There just is no difference.

> 
> bye, Roman

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

yodaiken@fsmlabs.com wrote:

> I mean, that these conversations are not very useful if you don't
> read what the other people write.

Oh, I do read everything that's for me, but it happens I'm not answering
everything, but thanks for your kind reminder.

> Here's a prior response by Andrew to a post by you.
>
> I've heard this so many times, and it just ain't so.   The overwhelming
> majority of problem areas are inside locks.  All the complexity and
> maintainability difficulties to which you refer exist in the preempt
> patch as well.    There just is no difference.

There is a difference. There is of course the maintenance work which have
both approaches in common, every kernel has to be tested for new latency
problems. What differs is the amount of problems that needs fixing, as
preempt can handle the problems outside of locks automatically, but
inserting schedule points for these cases is usually quite simple. (I
leave it open whether these problem are really only the minority, it
doesn't matter much, it only matters that they do exist.)
The remaining problems need to be examined again by either approach.
Breaking up the locks and inserting (implicit or explicit) schedule points
is often the simpler solution, but analyzing the problem and adjusting the
algorithm leads usually to the cleaner solution. Anyway, this is again
pretty much common for both approaches.
There is an additional cost for maintaining the explicit schedule points,
as they mean additional code all over the kernel, which has to be
maintained and to be verified overtime something is changed in that area.
This work has to be done by someone, if the ll patch would be included
into the standard kernel, the burden would be put onto every maintainer of
the systems that were changed. The easy approach by these maintainers
could be of course to just drop these schedule points, but then the ll
maintainer can start from zero and it adds up to the testing costs above.
This additional cost does not exist for all cases that are automatically
handled by preempting.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 19:41, Roman Zippel wrote:

> > That is exactly what Andrew Morton disputes. So why do you think he is
> > wrong?

Victor is saying that Andrew contends the hard parts of his low-latency
patch are just as hard to maintain with a preemptive kernel.  This is
true, for the places where spinlocks are held anyway, but it assumes we
continue to treat lock breaking and explicit scheduling as our only
solution.  It isn't under a preemptible kernel.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 10:18, Roman Zippel wrote:

> What somehow got lost in this discussion, that both patches don't
> necessarily conflict with each other, they both attack the same problem
> with different approaches, which complement each other. I prefer to get
> the best of both patches.
> The ll patch identifies problem, which preempt alone can't fix, on the
> other hand the ll patch inserts schedule calls all over the place, where
> preempt can handle this transparently. So far I haven't seen any
> evidence, that preempt introduces any _new_ serious problems, so I'd
> rather like to see to get the best out of both.

Good point.  In fact, I have an "ll patch" for preempt-kernel, it is
called lock-break and available at
	ftp://ftp.kernel.org/pub/linux/kernel/people/rml/lock-break

While I am not so sure this sort of explicit work is the answer -- I'd
much prefer to work on the algorithms to shorten lock time or lock into
different locks -- it does work.  The work is based heavily on Andrew's
ll patch but designed for use with preempt-kernel.  This means we can
drop some of the conditional schedules that aren't needed, and in others
we don't need to call schedule (just drop the locks).

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rik van Riel]

On Sun, 13 Jan 2002, Roman Zippel wrote:

> So far I haven't seen any evidence, that preempt introduces any _new_
> serious problems, so I'd rather like to see to get the best out of
> both.

Are you seriously suggesting you haven't read a single
email in this thread yet ?

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 20:50, Rik van Riel wrote:

> > So far I haven't seen any evidence, that preempt introduces any _new_
> > serious problems, so I'd rather like to see to get the best out of
> > both.
> 
> Are you seriously suggesting you haven't read a single
> email in this thread yet ?

No, I think he is suggesting he doesn't consider any of the problems
serious.  A lot of it is just smoke.  What is "bad" wrt 2.5?

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

On Sun, 13 Jan 2002, Rik van Riel wrote:

> > So far I haven't seen any evidence, that preempt introduces any _new_
> > serious problems, so I'd rather like to see to get the best out of
> > both.
>
> Are you seriously suggesting you haven't read a single
> email in this thread yet ?

Could you please be more explicit?

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Oliver Neukum]

On Tuesday 15 January 2002 08:10, george anzinger wrote:

> Yes, this is classic priority inversion.  It is here now, today with
> semaphors which are held by code that blocks.  If the code doesn't
> block, why not use a spin lock?  If it does, well the problem is here

Because eg. other code that holds the semaphore needs to sleep

> now.  I suppose we could set a preempt disable around a semaphore if it
> makes you feel better.  It doesn't fix the problem if the task blocks

It would make me feel better, but it would defeat the purpose.
There's a lot of code holding semaphores.

> AND it is legal to block while holding a preemption lock.

But it's easier to fix. If you can preempt only by explicitely
sleeping, you can beat priority invasion by changing basically
only wake_up. If you can be preempted at random, you need to know
who holds a semaphore.

	Regards
		Oliver

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> > I must have missed that in the code. I can see you check __cli() status but
> > I didn't see anywhere you check disable_irq(). Even if you did it doesnt
> > help when I mask the irq on the chip rather than using disable_irq() calls.
> 
> But you get interrupted by other interrups then so you have the same problem
> reagardless of any preemtion patch you hopefully lose the cpu for a much
> shorter time but still the same problem.

Interrupt paths are well sub millisecond, a pre empt might mean I don't get
the CPU back for measurable chunks of a second.  They are totally different
guarantees. 

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

Alan Cox wrote:

> So with pre-empt this happens
> 
>         driver magic
>         disable_irq(dev->irq)
> PRE-EMPT:
>         [large periods of time running other code]
> PRE-EMPT:
>         We get back and we've missed 300 packets, the serial port sharing
>         the IRQ has dropped our internet connection completely.

But it shouldn't deadlock as Victor is suggesting.

> There are numerous other examples in the kernel tree where the current code
> knows that there is a small bounded time between two actions in kernel space
> that do not have a sleep. They are not spin locked, and putting spin locks
> everywhere will just trash performance. They are pure hardware interactions
> so you can't automatically detect them.

Why should spin locks trash perfomance, while an expensive disable_irq()
doesn't?

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rob Landley]

On Saturday 12 January 2002 03:53 pm, Roman Zippel wrote:
> Hi,
>
> Alan Cox wrote:
> > So with pre-empt this happens
> >
> >         driver magic
> >         disable_irq(dev->irq)
> > PRE-EMPT:
> >         [large periods of time running other code]
> > PRE-EMPT:
> >         We get back and we've missed 300 packets, the serial port sharing
> >         the IRQ has dropped our internet connection completely.
>
> But it shouldn't deadlock as Victor is suggesting.

Um, hang on...

Obvioiusly, Alan, you know more about the networking stack than I do. :)  But 
could you define "large periods of time running other code"?

The real performance KILLER is when your code gets swapped out, and that 
simply doesn't happen in-kernel.  Yes, the niced app may get swapped out, but 
the syscall it's making is pinned in ram and it will only block on disk 
access when it returns.  So we're talking what kind of delay here, one second?

As for scheduling, even a nice 19 task will still get SOME time, and we can 
find out exactly what the worst case is since we hand out time slices and we 
don't hand out more until EVERYBODY exhausts theirs, including seriously 
niced processes.  So this isn't exactly non-deterministic behavior, is it?  
There IS an upper bound here...

There ISN'T an upper bound on interrupts.  We've got some nasty interrupts in 
the system.  How long does the PCI bus get tied up with spinning video cards 
flooding the bus to make their benchmarks look 5% better?  How long of a 
latency spike did we (until recently) get switching between graphics and text 
consoles?  (I heard that got fixed, moved into a tasklet or some such.  
Haven't looked at it yet.)  Without Andre's IDE patches, how much latency can 
the disk insert at random?

Yes, it's possible than if you have a fork bomb trying to take down your 
system, and you're using an old 10baseT ethernet driver built with some 
serious assumptions about how the kernel works, that you could drop some 
packets.  But I find it interesting that make -j can be considered a fairly 
unrealistic test intentionally overloading the system, yet an example with 
150 active threads all eating CPU time is NOT considered an example of how 
your process's receive buffer could easily fill up and drop packets no matter 
HOW fast the interrupt is working since even 10baseT feeds you 1.1 megabytes 
per second and with a 1 second delay we might have to swap stuff out to make 
room for them if we don't read from the socket in that long...

One other fun little thing about the scheduler: a process that is submitting 
network packets probably isn't entirely CPU bound, is it?  It's doing I/O.  
So even if it's niced, if it's competing with totally CPU bound tasks isn't 
it likely to get promoted?  How real-world is your overload-induced failure 
case example?

As for dropping 300 packets killing your connection, are you saying 802.11 
cards can't have a static burst that blocks your connection for half a 
second?  I've had full second gaps in network traffic on my cable modem, all 
time time, and the current overload behavior of most routers is dropping lots 
and lots of packets on the floor.  (My in-house network is still using an 
ancient 10baseT half-duplex hub.  I'm lazy, and it's still way faster than my 
upstream connection to the internet.)  Datagram delivery is not guaranteed.  
It never has been.  Maybe it will be once ECN comes in, but that's not yet.

What's one of the other examples you were worried about, besides NE2K (which 
can't do 100baseT, even on PCI, and a 100baseT PCI card is now $9 new.  Just 
a data point.)

Rob

(P.S.  The only behavior difference between preempt and SMP, apart from 
contention for per-cpu data, is the potential insertion of latency spikes in 
kernel code, which interrupts do anyway.  You're saying it can matter when 
something disables an interrupt.  Robert Love suggested the macro that 
disables an interrupt can count as a preemption guard just like a spinlock.  
Is this NOT enough to fix the objection?)

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Obvioiusly, Alan, you know more about the networking stack than I do. :)  But 
> could you define "large periods of time running other code"?

10ths of a second if there is a lot to let run instead of this thread. Even
1/100th is bad news. 

> There ISN'T an upper bound on interrupts.  We've got some nasty interrupts in 
> the system.  How long does the PCI bus get tied up with spinning video cards 
> flooding the bus to make their benchmarks look 5% better?  How long of a 

They dont flood the bus with interrupts, the lock the bus off for several
millseconds worst case. Which btw you'll note means that lowlatency already
achieves the best value you can get

> latency spike did we (until recently) get switching between graphics and text 
> consoles?  (I heard that got fixed, moved into a tasklet or some such.  
> Haven't looked at it yet.)  Without Andre's IDE patches, how much latency can 

Been fixed in -ac for ages, and finally made Linus tree.

> the disk insert at random?

IDE with or without Andre's changes can insert multiple millisecond delays
on the bus in some situations. Again pre-empt patch can offer you nothing 
because the hardware limit is easily met by low latency

> One other fun little thing about the scheduler: a process that is submitting 
> network packets probably isn't entirely CPU bound, is it?  It's doing I/O.  

Network packets get submitted from _outside_

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> > everywhere will just trash performance. They are pure hardware interactions
> > so you can't automatically detect them.
> 
> Why should spin locks trash perfomance, while an expensive disable_irq()
> doesn't?

disable_irq only blocks _one_ interrupt line, spin_lock_irqsave locks the
interrupt off on a uniprocessor, and  50% of the time off on a
dual processor. 

If I use a spin lock you can't run a modem and an NE2000 card together on
Linux 2.4. Thats why I had to do that work on the code. Its one of myriads
of basic obvious cases that the pre-empt patch gets wrong

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roman Zippel]

Hi,

Alan Cox wrote:

> disable_irq only blocks _one_ interrupt line, spin_lock_irqsave locks the
> interrupt off on a uniprocessor, and  50% of the time off on a
> dual processor.
> 
> If I use a spin lock you can't run a modem and an NE2000 card together on
> Linux 2.4. Thats why I had to do that work on the code. Its one of myriads
> of basic obvious cases that the pre-empt patch gets wrong

I wouldn't say it gets it wrong, the driver also has to take a non irq
spinlock anyway, so the window is quite small and even then the packet
is only delayed.
But now I really have to look at that driver and try a more optimistic
irq disabling approach, otherwise it will happily disable the most
important shared interrupt on my Amiga for ages.

bye, Roman

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> I wouldn't say it gets it wrong, the driver also has to take a non irq
> spinlock anyway, so the window is quite small and even then the packet
> is only delayed.

Or you lose a pile of them

> But now I really have to look at that driver and try a more optimistic
> irq disabling approach, otherwise it will happily disable the most
> important shared interrupt on my Amiga for ages.

If you play with the code remember that the irq delivery on x86 is
asynchronous. You can disable the irq on the chip, synchronize_irq() on 
the result and very occasionally get the irq delivered after all of that

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 12, 2002 07:54 pm, Alan Cox wrote:
> Another example is in the network drivers. The 8390 core for one example
> carefully disables an IRQ on the card so that it can avoid spinlocking on 
> uniprocessor boxes.
> 
> So with pre-empt this happens
> 
> 	driver magic
> 	disable_irq(dev->irq)
	inc task's preempt inhibit

> PRE-EMPT:
> 	[large periods of time running other code]
> PRE-EMPT:
> 	We get back and we've missed 300 packets, the serial port sharing
> 	the IRQ has dropped our internet connection completely.

	We are ok
	dec tasks's preempt inhibit
	jmp if nonzero

--
Daniel

Re: Alans example against preemtive kernel (Was: Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Roger Larsson]

On Saturday den 12 January 2002 19.54, Alan Cox wrote:
> Another example is in the network drivers. The 8390 core for one example
> carefully disables an IRQ on the card so that it can avoid spinlocking on
> uniprocessor boxes.
>
> So with pre-empt this happens
>
> 	driver magic
> 	disable_irq(dev->irq)
> PRE-EMPT:
> 	[large periods of time running other code]
> PRE-EMPT:
> 	We get back and we've missed 300 packets, the serial port sharing
> 	the IRQ has dropped our internet connection completely.
>
> ["Don't do that then" isnt a valid answer here. If I did hold a lock
>  it would be for several milliseconds at a time anyway and would reliably
>  trash performance this time]
>

./drivers/net/8390.c
I checked the code ./drivers/net/8390.c - this is how it REALLY looks like...

	/* Ugly but a reset can be slow, yet must be protected */
		
	disable_irq_nosync(dev->irq);
	spin_lock(&ei_local->page_lock);
		
	/* Try to restart the card.  Perhaps the user has fixed something. */
	ei_reset_8390(dev);
	NS8390_init(dev, 1);
		
	spin_unlock(&ei_local->page_lock);
	enable_irq(dev->irq);

This should be mostly OK for the preemptive kernel. Swapping the irq and spin 
lock lines should be preferred. But I think that is the case in SMP too...

Suppose two processors does the disable_irq_nosync - unlikely but possible...
One gets the spinlock, the other waits
The first runs through the code, exits the spin lock, enables irq
The second starts running the code - without irq disabled!!!

This would work in both cases.
	/* Ugly but a reset can be slow, yet must be protected */
		
	spin_lock(&ei_local->page_lock);
	disable_irq_nosync(dev->irq);
		
	/* Try to restart the card.  Perhaps the user has fixed something. */
	ei_reset_8390(dev);
	NS8390_init(dev, 1);
		
	enable_irq(dev->irq);
	spin_unlock(&ei_local->page_lock);

/RogerL



-- 
Roger Larsson
Skellefteå
Sweden

Re: Alans example against preemtive kernel (Was: Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Alan Cox]

> This should be mostly OK for the preemptive kernel. Swapping the irq an=
> d spin=20
> lock lines should be preferred. But I think that is the case in SMP too=
> =2E..

You deadlock if you swap the two lines over. In this case for pre-empt you
really have to go in and add non pre-emption places to the driver

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Helge Hafting]

Rob Landley wrote:
> 
> On Friday 11 January 2002 09:50 pm, yodaiken@fsmlabs.com wrote:
> > On Fri, Jan 11, 2002 at 03:33:22PM -0500, Robert Love wrote:
> > > On Fri, 2002-01-11 at 07:37, Alan Cox wrote:
> > > The preemptible kernel plus the spinlock cleanup could really take us
> > > far.  Having locked at a lot of the long-held locks in the kernel, I am
> > > confident at least reasonable progress could be made.
> > >
> > > Beyond that, yah, we need a better locking construct.  Priority
> > > inversion could be solved with a priority-inheriting mutex, which we can
> > > tackle if and when we want to go that route.  Not now.
> >
> > Backing the car up to the edge of the cliff really gives us
> > good results. Beyond that, we could jump off the cliff
> > if we want to go that route.
> > Preempt leads to inheritance and inheritance leads to disaster.
> 
> I preempt leads to disaster than Linux can't do SMP.  Are you saying that's
> the case?

There is a difference.  Preempt have the same locking requirements as
SMP, but there's also _timing_ requirements.

> The preempt patch is really "SMP on UP".  If pre-empt shows up a problem,
> then it's a problem SMP users will see too.  If we can't take advantage of
> the existing SMP locking infrastructure to improve latency and interactive
> feel on UP machines, than SMP for linux DOES NOT WORK.

One example where preempt may break and SMP does not:

Consider driver code.  Critical data structures is protected by
spinlocks,
but some of the access to the hardware device itself is outside those
locks (I can prove that the other processors can't get there with
the driver in that state anyway)

Now, hardware access has timing requirements.  That works on SMP because
you don't loose the CPU to anything but interrupts, and they are fast. 
You get it back almost immediately.  The device in question times out
after a much longer interval.

But preempt may decide to run a time-consuming higher priority task in
the 
middle of device access, cuasing the hardware to time out and fail.
Hardware access isn't necessarily in a interrupt handler.  It may be
done directly in a read/write/ioctl call if the device happens
to be available at the moment.

This is a case where SMP works even though preempt may fail.  I don't
know if this is an issue for existing drivers, but it is possible.

Helge Hafting



> 
> > All the numbers I've seen show Morton's low latency just works better. Are
> > there other numbers I should look at.
> 
> This approach is basically a collection of heuristics.  The kernel has been
> profiled and everywhere a latency spike was found, a band-aid was put on it
> (an explicit scheduling point).  This doesn't say there aren't other latency
> spikes, just that with the collection of hardware and software being
> benchmarked, the latency spikes that were found have each had a band-aid
> individually applied to them.
> 
> This isn't a BAD thing.  If the benchmarks used to find latency spikes are at
> all like real-world use, then it helps real-world applications.  But of
> COURSE the benchmarks are going to look good, since tuning the kernel to
> those benchmarks is the way the patch was developed!
> 
> The majority of the original low latency scheduling point work is handled
> automatically by the SMP on UP kernel.  You don't NEED to insert scheduling
> points anywhere you aren't inside a spinlock.  So the SMP on UP patch makes
> most of the explicit scheduling point patch go away, accomplishing the same
> thing in a less intrusive manner.  (Yes, it makes all kernels act like SMP
> kernels for debugging purposes.  But you can turn it off for debugging if you
> want to, that's just another toggle in the magic sysreq menu.  And this isn't
> entirely a bad thing: applying the enormous UP userbase to the remaining SMP
> bugs is bound to squeeze out one or two more obscure ones, but those bugs DO
> exist already on SMP.)
> 
> However, what's left of the explicit scheduling work is still very useful.
> When you ARE inside a spinlock, you can't just schedule, you have to save
> state, drop the lock(s), schedule, re-acquire the locks, and reload your
> state in case somebody else diddled with the structures you were using.  This
> is a lot harder than just scheduling, but breaking up long-held locks like
> this helps SMP scalability, AND helps latency in the SMP-on-UP case.
> 
> So the best approach is a combination of the two patches.  SMP-on-UP for
> everything outside of spinlocks, and then manually yielding locks that cause
> problems.  Both Robert Love and Andrew Morton have come out in favor of each
> other's patches on lkml just in the past few days.  The patches work together
> quite well, and each wants to see the other's patch applied.
> 
> Rob
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Preempt & how long it takes to interrupt (was Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Pavel Machek]

Hi!

> > I preempt leads to disaster than Linux can't do SMP.  Are you saying that's
> > the case?
> 
> There is a difference.  Preempt have the same locking requirements as
> SMP, but there's also _timing_ requirements.
...
> > The preempt patch is really "SMP on UP".  If pre-empt shows up a problem,
> > then it's a problem SMP users will see too.  If we can't take advantage of
> > the existing SMP locking infrastructure to improve latency and interactive
> > feel on UP machines, than SMP for linux DOES NOT WORK.
> 
> One example where preempt may break and SMP does not:
> 
> Consider driver code.  Critical data structures is protected by
> spinlocks,
> but some of the access to the hardware device itself is outside those
> locks (I can prove that the other processors can't get there with
> the driver in that state anyway)
> 
> Now, hardware access has timing requirements.  That works on SMP because
> you don't loose the CPU to anything but interrupts, and they are fast. 
> You get it back almost immediately.  The device in question times out
> after a much longer interval.

So... how long do you have to stay in interrupt for it to be a bug?

There's *no* requirement that says "it may not take second to handle
an interrupt". Actually I guess that some nasty conditions (UHCI needs
reset?) may take that long in interrupt. Oh and actually few releases
ago, console switching was done from interrupt and it *did* take 2
seconds for me.

If someone assumes interrupts are "short", he has broken code already.

									Pavel
-- 
(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa

Re: Preempt & how long it takes to interrupt (was Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Rob Landley]

On Friday 18 January 2002 05:41 pm, Pavel Machek wrote:

> So... how long do you have to stay in interrupt for it to be a bug?
>
> There's *no* requirement that says "it may not take second to handle
> an interrupt". Actually I guess that some nasty conditions (UHCI needs
> reset?) may take that long in interrupt. Oh and actually few releases
> ago, console switching was done from interrupt and it *did* take 2
> seconds for me.
>
> If someone assumes interrupts are "short", he has broken code already.

That kinda defeats the entire purpose of low-latency patches, doesn't it?

I'm not entirely certain what Alan's smoking if he's raising the straw man 
argument of a two second delay dropping 300 packets and causing connections 
to abort (my sister's on DSL, 5 second dropouts every time the phone rings, 
but connections continue just fine.  Wouldn't want to play quake under those 
circumstances, but would I really have the ~200 CPU-hog background threads 
while playing quake as per Alan's example, and even then the argument is just 
that either the network driver or the network card is bad...)

Still not as bad an example as Aunt Tillie, I'll grant you...

Rob

Re: Preempt & how long it takes to interrupt (was Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Alan Cox]

> I'm not entirely certain what Alan's smoking if he's raising the straw man 
> argument of a two second delay dropping 300 packets and causing connections 

Go read my original mail about the NE2000 driver. If you are going to accuse
me of smoking things you could at least read the posts you base it on

Re: Preempt & how long it takes to interrupt (was Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Rob Landley]

On Monday 21 January 2002 04:48 pm, Alan Cox wrote:
> > I'm not entirely certain what Alan's smoking if he's raising the straw
> > man argument of a two second delay dropping 300 packets and causing
> > connections
>
> Go read my original mail about the NE2000 driver. If you are going to
> accuse me of smoking things

For which I apologize,

> you could at least read the posts you base it on

I did.

Okay, let's review:

> On Sat, 12 Jan 2002 18:54:27 +0000 (GMT), Alan Cox spaketh thusly:
> 
>Another example is in the network drivers. The 8390 core for one example
>carefully disables an IRQ on the card so that it can avoid spinlocking on 
>uniprocessor boxes.

Sounds like a bit of a kludge, but it's not my code.  However, without 
preempt aren't spinlocks basically NOPs on uniprocessor boxes?  What did I 
miss?

And wasn't there discussion of using IRQ disabling as a preempt barrier (at 
least until the syscall returns to userspace or finished a module unload 
call, clueing us in that it won't reenable it any time soon.)

>So with pre-empt this happens
>
>        driver magic
>        disable_irq(dev->irq)
>PRE-EMPT:
>        [large periods of time running other code]
>PRE-EMPT:
>        We get back and we've missed 300 packets, the serial port sharing
>        the IRQ has dropped our internet connection completely.

Okay, please point out where I missed a curve here:

An NE2K cannot go faster than 10baseT.  (Never designed to.  It's an old ISA 
standard dragged along to PCI largely because they had these chips lying 
around and nobody wanted to come up with a new interface anyway.  But it can 
only handle packets one at a time, as far as I know.  I've got several of 
these suckers lying around in various drawers, some of which are ISA.  I'm 
considering throwing them out since a new 100baseT card is $9 retail.  But I 
digress...).

With 10baseT you've got a theoretical maximum throughput of 1.25 (decimal) 
megabytes/second.  Assuming 1500 byte packet sizes at 10 megabits per second 
on a saturated link, we're talking 833 packets/second so a little over a 
third of a second is the shortest amount of time in which you can drop 300 
packets.

So in a worst case scenario latency spike introduced by an overloaded system 
running make -j where niced down CPU bound processes are doing network I/O 
through a driver that's not doing the right locking for preempt to know it 
shouldn't be interrupted...  Yeah, it could lose 300 packets.  Why this is a 
bad thing when we're designing gigabit ethernet systems with interrupt 
mitigation so they intentionally drop thousands of packets at a time rather 
than livelocking...  Open question.  TCP/IP is designed to retransmit around 
this sort of thing, and even with ECN it's not going to forget how. 

But that wasn't really the bad thing.  The bad thing was the incidentally 
misconfigured serial connection (sharing the network card's IRQ) hanging up.  
Serial maxes out at 115,200 which is 14400 bytes/sec (assuming perfect 8 bit 
encoding with no overhead), and losing 1/3 of that means 4800 bytes, which is 
indeed noticeably more than a 16550a UART's 16 byte buffer.  And SLIP and PPP 
also tend to have smaller MTU, (around 256 bytes for latency reasons), 
meaning the loss here could be a whole 18 packets.  (Assuming your 56k modem 
that can't actually quite do 56k isn't the real bottleneck, but we won't go 
there...)

And I agree that's not good for playing quake, but again: playing quake with 
"make -j" running in the background isn't going to give you the world's 
greatest frame rates anyway.  The 1/3 second latency spike was ENTIRELY due 
to the computer being loaded up with other things to do and not scheduling 
back to you before then.  If your game of quake is experiencing those kind of 
SCHEDULING latency spikes, it's unplayable anyway.

As for hanging up, I've used slip at 2400 bps with no error correction, on 
noisy phone lines.  (It sucked, but it more or less worked.  Yes, it would 
hang up at times when the line noise made retransmission impossible for more 
than about fifteen seconds at a time, which is why PPP was invented.  PPP is 
designed to be MORE robust than slip, more intelligent than SLIP about 
retransmits, and above all not to give up nearly as easily.  It's been a 
couple years since I've messed with it in depth, but it seems to me the phone 
generally physically lost carrier before PPP gave up.  (Should PPP over 
Ethernet ever "hang up" and exit during a network storm?)  The modem itself 
doesn't care about the data being transmitted through it, that has no bearing 
on its carrier detect status.  And if pppd exited due to a 1/3 second dropout 
(producing at most 2 garbled packets: one cut off at the start and one cut 
off at the end, the rest simply dropped), then there would be something wrong 
with pppd.

So you've got a "gloom and doom" scenario that, even in this fairly 
pathlogical worst case, doesn't really seem all that bad.  And it's also a 
purely theoretical objection of a kind that I haven't heard anybody actually 
testing the patch complaining about, AND one that seems like it could be 
addressed by using IRQ disabling as a latency guard in addition to spinlocks.

>["Don't do that then" isnt a valid answer here. If I did hold a lock
> it would be for several milliseconds at a time anyway and would reliably
> trash performance this time]

If NE2K is holding the lock for several miliseconds at a time, how is it 
managing 833 packets/second?  (Is it NOT doing one per interrupt?)

If it's holding the lock for several miliseconds, the overhead of acquiring 
the lock in the first place isn't exactly a show-stopper, is it?

If spinlocks don't get compiled in on non-preempt UP boxes (and are basically 
just an increment in preempt), where is the killer overhead in the UP case?  
If you're saying spinlocks would kill SMP performance, on a lock which should 
basically have no contention at all (when we used to have 100baseT drivers 
using the Big Kernel Lock), 

And again, this is where the use of IRQ blocking as a preempt guard comes in 
handy.  (Which naturally expires when you return to userspace anyway, so 
hand-waving about unlimited blocking time is just that: there IS an upper 
bound here.  And an IRQ block that's part of a device shutdown is really a 
different call, which would probably mostly be confined to the module unload 
code anyway.)

>There are numerous other examples in the kernel tree where the current code
>knows that there is a small bounded time between two actions in kernel space
>that do not have a sleep.

Such as?

(And if the use of IRQ disabling as a preempt guard doesn't fix it, then the 
code is ALREADY hosed because interrupts can be arbitrarily long.  We're 
trying to keep them short now, but we used to switch consoles from interrupt 
context and that could take a LONG time if we were wandering between graphics 
and text consoles.  So you're saying this is code that didn't show up as a 
bug back then...)

>They are not spin locked, and putting spin locks
>everywhere will just trash performance. They are pure hardware interactions
>so you can't automatically detect them.

If they don't have IRQs blocked, they don't have any real latency guarantees 
anyway.  If the DO have IRQs blocked, they can be automatically detected.

>That is why the pre-empt code is a much much bigger problem and task than the
>low latency code.

I don't see it.  Care to point out what I've missed?

>Alan

Rob

Re: Preempt & how long it takes to interrupt (was Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Alan Cox]

> >carefully disables an IRQ on the card so that it can avoid spinlocking on 
> >uniprocessor boxes.
> 
> Sounds like a bit of a kludge, but it's not my code.  However, without 
> preempt aren't spinlocks basically NOPs on uniprocessor boxes?  What did I 
> miss?

spin lock is a nop on uniprocessor. That is much of the point of this. Most
ne2000's are in uniprocessor boxes so they are primary target

> An NE2K cannot go faster than 10baseT.  (Never designed to.  It's an old ISA 

Wrong. There are multiple 100Mbit NE2000 clones (notably PCMCIA ones). I
have one in my laptop for example.

> testing the patch complaining about, AND one that seems like it could be 
> addressed by using IRQ disabling as a latency guard in addition to spinlocks.

I dont believe anyone has tested the driver hard with pre-empt. Its not that
this driver can't be fixed. Its that this is one tiny example of maybe 
thousands of other similar flaws lurking. There is no obvious automated way
to find them either.

> If it's holding the lock for several miliseconds, the overhead of acquiring 
> the lock in the first place isn't exactly a show-stopper, is it?

I don't hold the lock with interrupts off for several milliseconds

Alan

Re: Preempt & how long it takes to interrupt (was Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Nigel Gamble]

On Sun, 27 Jan 2002, Alan Cox wrote:
> I dont believe anyone has tested the driver hard with pre-empt. Its not that
> this driver can't be fixed. Its that this is one tiny example of maybe
> thousands of other similar flaws lurking. There is no obvious automated way
> to find them either.

You could make the same argument against SMP, but Linux has SMP support
despite all the thousands of SMP flaws that once lurked with no obvious
automated way to find them.  Most of them have been found.

Actually, there is a way to help to automate the finding of preemption
problems:  you keep a log of kernel preemption events in a circular
buffer, and dump the log after something unexpected happens (like a
kernel oops).  Then you search the log for preemptions that happened in
suspicious places.  Kernel preemptions don't happen very often, so the
log usually goes back several seconds, which is usually plenty of time
to catch the preemption that happened in the wrong place.  (Since SMP
locking problems are also preemption problems, this technique can also
catch SMP problems.)

I have a patch to do this for earlier versions of the kernel preemption
patch - I need to bring it up to date and send it to Robert for use with
the latest versions of his patch.

Nigel Gamble                                    nigel@nrg.org
Mountain View, CA, USA.                         http://www.nrg.org/

Re: Preempt & how long it takes to interrupt (was Re: [2.4.17/18pre]u

[Alan Cox]

> You could make the same argument against SMP, but Linux has SMP support
> despite all the thousands of SMP flaws that once lurked with no obvious
> automated way to find them.  Most of them have been found.

We spent four years on that. It was also done in a very careful and 
precise manner starting with SMP that gave the same guarantees as non SMP
for the 2.0 kernel tree, then moving on to relaxing guarantees in certain
places -as they were audited- for 2.2 and with 2.4 increasing the coverage
to all major points of contention except the scsi layer

> I have a patch to do this for earlier versions of the kernel preemption
> patch - I need to bring it up to date and send it to Robert for use with
> the latest versions of his patch.

Nod - thats a productive approach

Re: Preempt & how long it takes to interrupt (was Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Robert Love]

On Fri, 2002-01-18 at 17:41, Pavel Machek wrote:

> So... how long do you have to stay in interrupt for it to be a bug?
> 
> There's *no* requirement that says "it may not take second to handle
> an interrupt". Actually I guess that some nasty conditions (UHCI needs
> reset?) may take that long in interrupt. Oh and actually few releases
> ago, console switching was done from interrupt and it *did* take 2
> seconds for me.
> 
> If someone assumes interrupts are "short", he has broken code already.

Agreed.  Conversely, however, writing code that introduces long
interrupt-off periods should be considered a BUG.

In other words, relying on short interrupt-off periods is bad form, but
so is writing gross code that rudely keeps them off.

I think we all considered the long off periods in VT switching and fbdev
a bug.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Fri, Jan 11, 2002 at 03:33:22PM -0500, Robert Love wrote:
> On Fri, 2002-01-11 at 07:37, Alan Cox wrote:
> 
> > Its more than a spinlock cleanup at that point. To do anything useful you have
> > to tackle both priority inversion and some kind of at least semi-formal 
> > validation of the code itself. At the point it comes down to validating the
> > code I'd much rather validate rtlinux than the entire kernel
> 
> The preemptible kernel plus the spinlock cleanup could really take us
> far.  Having locked at a lot of the long-held locks in the kernel, I am
> confident at least reasonable progress could be made.
> 
> Beyond that, yah, we need a better locking construct.  Priority
> inversion could be solved with a priority-inheriting mutex, which we can
> tackle if and when we want to go that route.  Not now.
> 
> I want to lay the groundwork for a better kernel.  The preempt-kernel
> patch gives real-world improvements, it provides a smoother user desktop
> experience -- just look at the positive feedback.  Most importantly,
> however, it provides a framework for superior response with our standard

I don't know how to tell you, positive feedback compared to mainline
kernel is totally irrelevant, mainline has broken read/write/sendfile
syscalls that can hang the machine etc... That was fixed ages ago in
many ways, current way is very lightweight, if you can get positive
feedback compared to -aa _that_ will matter.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[jogi]

On Sat, Jan 12, 2002 at 12:13:15PM +0100, Andrea Arcangeli wrote:
> On Fri, Jan 11, 2002 at 03:33:22PM -0500, Robert Love wrote:
> > On Fri, 2002-01-11 at 07:37, Alan Cox wrote:
> > 
> > > Its more than a spinlock cleanup at that point. To do anything useful you have
> > > to tackle both priority inversion and some kind of at least semi-formal 
> > > validation of the code itself. At the point it comes down to validating the
> > > code I'd much rather validate rtlinux than the entire kernel
> > 
> > The preemptible kernel plus the spinlock cleanup could really take us
> > far.  Having locked at a lot of the long-held locks in the kernel, I am
> > confident at least reasonable progress could be made.
> > 
> > Beyond that, yah, we need a better locking construct.  Priority
> > inversion could be solved with a priority-inheriting mutex, which we can
> > tackle if and when we want to go that route.  Not now.
> > 
> > I want to lay the groundwork for a better kernel.  The preempt-kernel
> > patch gives real-world improvements, it provides a smoother user desktop
> > experience -- just look at the positive feedback.  Most importantly,
> > however, it provides a framework for superior response with our standard
> 
> I don't know how to tell you, positive feedback compared to mainline
> kernel is totally irrelevant, mainline has broken read/write/sendfile
> syscalls that can hang the machine etc... That was fixed ages ago in
> many ways, current way is very lightweight, if you can get positive
> feedback compared to -aa _that_ will matter.

Hello Andrea,

I did my usual compile testings (untar kernel archive, apply patches,
make -j<value> ...

Here are some results (Wall time + Percent cpu) for each of the consecutive five runs:

        13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp
j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%
j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%
j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%
j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%
j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%

j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%
j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%
j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%
j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%
j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%

* build incomplete (OOM killer killed several cc1 ... )

So far 2.4.13-pre5aa1 had been the king of the block in compile times.
But this has changed. Now the (by far) fastest kernel is 2.4.18-pre
+ Ingos scheduler patch (s) + preemptive patch (p). I did not test
preemptive patch alone so far since I don't know if the one I have
applies cleanly against -pre3 without Ingos patch. I used the
following patches:

s: sched-O1-2.4.17-H6.patch
p: preempt-kernel-rml-2.4.18-pre3-ingo-1.patch

I hope this info is useful to someone.

Kind regards,

   Jogi

-- 

Well, yeah ... I suppose there's no point in getting greedy, is there?

    << Calvin & Hobbes >>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> On Sat, Jan 12, 2002 at 12:13:15PM +0100, Andrea Arcangeli wrote:
> > On Fri, Jan 11, 2002 at 03:33:22PM -0500, Robert Love wrote:
> > > On Fri, 2002-01-11 at 07:37, Alan Cox wrote:
> > > 
> > > > Its more than a spinlock cleanup at that point. To do anything useful you have
> > > > to tackle both priority inversion and some kind of at least semi-formal 
> > > > validation of the code itself. At the point it comes down to validating the
> > > > code I'd much rather validate rtlinux than the entire kernel
> > > 
> > > The preemptible kernel plus the spinlock cleanup could really take us
> > > far.  Having locked at a lot of the long-held locks in the kernel, I am
> > > confident at least reasonable progress could be made.
> > > 
> > > Beyond that, yah, we need a better locking construct.  Priority
> > > inversion could be solved with a priority-inheriting mutex, which we can
> > > tackle if and when we want to go that route.  Not now.
> > > 
> > > I want to lay the groundwork for a better kernel.  The preempt-kernel
> > > patch gives real-world improvements, it provides a smoother user desktop
> > > experience -- just look at the positive feedback.  Most importantly,
> > > however, it provides a framework for superior response with our standard
> > 
> > I don't know how to tell you, positive feedback compared to mainline
> > kernel is totally irrelevant, mainline has broken read/write/sendfile
> > syscalls that can hang the machine etc... That was fixed ages ago in
> > many ways, current way is very lightweight, if you can get positive
> > feedback compared to -aa _that_ will matter.
> 
> Hello Andrea,
> 
> I did my usual compile testings (untar kernel archive, apply patches,
> make -j<value> ...
> 
> Here are some results (Wall time + Percent cpu) for each of the consecutive five runs:
> 
>         13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp
> j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%
> j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%
> j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%
> j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%
> j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%
> 
> j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%
> j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%
> j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%
> j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%
> j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%
> 
> * build incomplete (OOM killer killed several cc1 ... )
> 
> So far 2.4.13-pre5aa1 had been the king of the block in compile times.
> But this has changed. Now the (by far) fastest kernel is 2.4.18-pre
> + Ingos scheduler patch (s) + preemptive patch (p). I did not test
> preemptive patch alone so far since I don't know if the one I have
> applies cleanly against -pre3 without Ingos patch. I used the
> following patches:
> 
> s: sched-O1-2.4.17-H6.patch
> p: preempt-kernel-rml-2.4.18-pre3-ingo-1.patch
> 
> I hope this info is useful to someone.

the improvement of "sp" compared to "s" is quite visible, not sure how
can a little different time spent in kernel make such a difference on
the final numbers, also given compilation is mostly an userspace task, I
assume you were swapping out or running out of cache at the very least,
right?

btw, I'd be curious if you could repeat the same test with -j1 or -j2?
(actually real world)

Still the other numbers remains interesting for a trashing machine, but
a few percent difference with a trashing box isn't a big difference, vm
changes can infulence those numbers more than any preempt or scheduler
number (of course if my guess that you're swapping out is really right :).
I guess "p" helps because we simply miss some schedule point in some vm
routine. Hints?

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[jogi]

On Sat, Jan 12, 2002 at 05:05:28PM +0100, Andrea Arcangeli wrote:
> On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:

[...]

> > Hello Andrea,
> > 
> > I did my usual compile testings (untar kernel archive, apply patches,
> > make -j<value> ...
> > 
> > Here are some results (Wall time + Percent cpu) for each of the consecutive five runs:
> > 
> >         13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp
> > j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%
> > j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%
> > j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%
> > j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%
> > j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%
> > 
> > j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%
> > j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%
> > j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%
> > j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%
> > j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%
> > 
> > * build incomplete (OOM killer killed several cc1 ... )
> > 
> > So far 2.4.13-pre5aa1 had been the king of the block in compile times.
> > But this has changed. Now the (by far) fastest kernel is 2.4.18-pre
> > + Ingos scheduler patch (s) + preemptive patch (p). I did not test
> > preemptive patch alone so far since I don't know if the one I have
> > applies cleanly against -pre3 without Ingos patch. I used the
> > following patches:
> > 
> > s: sched-O1-2.4.17-H6.patch
> > p: preempt-kernel-rml-2.4.18-pre3-ingo-1.patch
> > 
> > I hope this info is useful to someone.
> 
> the improvement of "sp" compared to "s" is quite visible, not sure how
> can a little different time spent in kernel make such a difference on
> the final numbers, also given compilation is mostly an userspace task, I
> assume you were swapping out or running out of cache at the very least,
> right?

The system is *heavily* swapping. Plain 2.4.18-pre3 can not even finish
the jobs because it runs out of memory. That's why I used j75 or j100
initially. Otherwise there was not even a difference between the 2.4.10+
vm and the 2.4.9-ac+ vm. All I want to test with this "benchmark" is how
well the system reacts when I throw *lots* of compilation jobs at it ...

> btw, I'd be curious if you could repeat the same test with -j1 or -j2?
> (actually real world)

Using just -j1 or -j2 will probably be no difference (I will test it anyway
and post the results). 

> Still the other numbers remains interesting for a trashing machine, but
> a few percent difference with a trashing box isn't a big difference, vm
> changes can infulence those numbers more than any preempt or scheduler
> number (of course if my guess that you're swapping out is really right :).
> I guess "p" helps because we simply miss some schedule point in some vm
> routine. Hints?

But what *I* like most about the preemptive results are that the results
for all runs do not vary that much. Looking at plain 2.4.18-pre3 there
is a huge difference in runtime between the fastest and the longest run.

Regards,

   Jogi

-- 

Well, yeah ... I suppose there's no point in getting greedy, is there?

    << Calvin & Hobbes >>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> I did my usual compile testings (untar kernel archive, apply patches,
> make -j<value> ...

If I understand your test, 
you are testing different loads - you are compiling kernels that may differ
in size and makefile organization, not to mention different layout on the
file system and disk.

What happens when you do the same test, compiling one kernel under multiple
different kernels?

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Sat, Jan 12, 2002 at 09:52:09AM -0700, yodaiken@fsmlabs.com wrote:
> On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> > I did my usual compile testings (untar kernel archive, apply patches,
> > make -j<value> ...
> 
> If I understand your test, 
> you are testing different loads - you are compiling kernels that may differ
> in size and makefile organization, not to mention different layout on the
> file system and disk.

Ouch, I assumed this wasn't the case indeed.

> 
> What happens when you do the same test, compiling one kernel under multiple
> different kernels?

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ed Sweetman]

> On Sat, Jan 12, 2002 at 09:52:09AM -0700, yodaiken@fsmlabs.com wrote:
> > On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> > > I did my usual compile testings (untar kernel archive, apply patches,
> > > make -j<value> ...
> >
> > If I understand your test,
> > you are testing different loads - you are compiling kernels that may
differ
> > in size and makefile organization, not to mention different layout on
the
> > file system and disk.

Can someone tell me why we're "testing" the preempt kernel by running
make -j on a build?  What exactly is this going to show us?  The only thing
i can think of is showing us that throughput is not damaged when you want to
run single apps by using preempt.  You dont get to see the effects of the
kernel preemption because all the damn thing is doing is preempting itself.

If you want to test the preempt kernel you're going to need something that
can find the mean latancy or "time to action" for a particular program or
all programs being run at the time and then run multiple programs that you
would find on various peoples' systems.   That is the "feel" people talk
about when they praise the preempt patch.  make -j'ing something and not
testing anything else but that will show you nothing important except "does
throughput get screwed by the preempt patch."   Perhaps checking the
latencies on a common program on people's systems like mozilla or konqueror
while doing a 'make -j N bzImage'  would be a better idea.

> Ouch, I assumed this wasn't the case indeed.
>
> >
> > What happens when you do the same test, compiling one kernel under
multiple
> > different kernels?
>
> Andrea


You should _always_ use the same kernel tree at the same point each time you
rerun the test under a different kernel.  Always make clean before rebooting
to the next kernel.  setting up the test bed should be pretty straight
forward.   make sure the build tree is clean then make dep it.   reboot to
the next kernel.   load up mozilla but nothing else (mozilla should be
modified a bit to display the time it takes to do certain functions such as
displaying drop down menus, loading, opening a new window. Also you should
make the homepage something on the drive or blank.).  start make -j 4
bzImage then load mozilla (no other gnome gtk libraries or having them
loaded via running gnome doesn't matter, just as long as it's the same each
time).  Mozilla should then output times it takes to do certain things and
that should give you a good idea of how the preempt patch is performing
assuming everything is running on the same priority and your memory isn't
being maxed out and your hdd isn't eating the majority of the cpu time.

But i really think make -j'ing and only testing that or reporting those
numbers is a complete waste of time if you're trying to look at the
preempt's patch performance.  I like using mozilla in this example because
it's a big bulky app that most people have (kde users possibly excluded)
where an improvement in latency or "time to action" is actually important to
people, and cant be easily ignored.

well those are just my two cents,  i'd do it myself but i'm waiting for
hardware to replace the broken crap i have now.  but if nobody has done it
by then i'll set that up.

-formerly safemode

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Ed Sweetman wrote:
> 
> If you want to test the preempt kernel you're going to need something that
> can find the mean latancy or "time to action" for a particular program or
> all programs being run at the time and then run multiple programs that you
> would find on various peoples' systems.   That is the "feel" people talk
> about when they praise the preempt patch.

Right.  And that is precisely why I created the "mini-ll" patch.  To
give the improved "feel" in a way which is acceptable for merging into
the 2.4 kernel.

And guess what?   Nobody has tested the damn thing, so it's going
nowhere.

Here it is again:



--- linux-2.4.18-pre3/fs/buffer.c	Fri Dec 21 11:19:14 2001
+++ linux-akpm/fs/buffer.c	Sat Jan 12 12:22:29 2002
@@ -249,12 +249,19 @@ static int wait_for_buffers(kdev_t dev, 
 	struct buffer_head * next;
 	int nr;
 
-	next = lru_list[index];
 	nr = nr_buffers_type[index];
+repeat:
+	next = lru_list[index];
 	while (next && --nr >= 0) {
 		struct buffer_head *bh = next;
 		next = bh->b_next_free;
 
+		if (dev == NODEV && current->need_resched) {
+			spin_unlock(&lru_list_lock);
+			conditional_schedule();
+			spin_lock(&lru_list_lock);
+			goto repeat;
+		}
 		if (!buffer_locked(bh)) {
 			if (refile)
 				__refile_buffer(bh);
@@ -1174,8 +1181,10 @@ struct buffer_head * bread(kdev_t dev, i
 
 	bh = getblk(dev, block, size);
 	touch_buffer(bh);
-	if (buffer_uptodate(bh))
+	if (buffer_uptodate(bh)) {
+		conditional_schedule();
 		return bh;
+	}
 	ll_rw_block(READ, 1, &bh);
 	wait_on_buffer(bh);
 	if (buffer_uptodate(bh))
--- linux-2.4.18-pre3/fs/dcache.c	Fri Dec 21 11:19:14 2001
+++ linux-akpm/fs/dcache.c	Sat Jan 12 12:22:29 2002
@@ -71,7 +71,7 @@ static inline void d_free(struct dentry 
  * d_iput() operation if defined.
  * Called with dcache_lock held, drops it.
  */
-static inline void dentry_iput(struct dentry * dentry)
+static void dentry_iput(struct dentry * dentry)
 {
 	struct inode *inode = dentry->d_inode;
 	if (inode) {
@@ -84,6 +84,7 @@ static inline void dentry_iput(struct de
 			iput(inode);
 	} else
 		spin_unlock(&dcache_lock);
+	conditional_schedule();
 }
 
 /* 
--- linux-2.4.18-pre3/fs/jbd/commit.c	Fri Dec 21 11:19:14 2001
+++ linux-akpm/fs/jbd/commit.c	Sat Jan 12 12:22:29 2002
@@ -212,6 +212,16 @@ write_out_data_locked:
 				__journal_remove_journal_head(bh);
 				refile_buffer(bh);
 				__brelse(bh);
+				if (current->need_resched) {
+					if (commit_transaction->t_sync_datalist)
+						commit_transaction->t_sync_datalist =
+							next_jh;
+					if (bufs)
+						break;
+					spin_unlock(&journal_datalist_lock);
+					conditional_schedule();
+					goto write_out_data;
+				}
 			}
 		}
 		if (bufs == ARRAY_SIZE(wbuf)) {
--- linux-2.4.18-pre3/fs/proc/array.c	Thu Oct 11 09:00:01 2001
+++ linux-akpm/fs/proc/array.c	Sat Jan 12 12:22:29 2002
@@ -415,6 +415,8 @@ static inline void statm_pte_range(pmd_t
 		pte_t page = *pte;
 		struct page *ptpage;
 
+		conditional_schedule();
+
 		address += PAGE_SIZE;
 		pte++;
 		if (pte_none(page))
--- linux-2.4.18-pre3/fs/proc/generic.c	Fri Sep  7 10:53:59 2001
+++ linux-akpm/fs/proc/generic.c	Sat Jan 12 12:22:29 2002
@@ -98,7 +98,9 @@ proc_file_read(struct file * file, char 
 				retval = n;
 			break;
 		}
-		
+
+		conditional_schedule();
+
 		/* This is a hack to allow mangling of file pos independent
  		 * of actual bytes read.  Simply place the data at page,
  		 * return the bytes, and set `start' to the desired offset
--- linux-2.4.18-pre3/include/linux/condsched.h	Thu Jan  1 00:00:00 1970
+++ linux-akpm/include/linux/condsched.h	Sat Jan 12 12:22:29 2002
@@ -0,0 +1,18 @@
+#ifndef _LINUX_CONDSCHED_H
+#define _LINUX_CONDSCHED_H
+
+#ifndef __LINUX_COMPILER_H
+#include <linux/compiler.h>
+#endif
+
+#ifndef __ASSEMBLY__
+#define conditional_schedule()				\
+do {							\
+	if (unlikely(current->need_resched)) {		\
+		__set_current_state(TASK_RUNNING);	\
+		schedule();				\
+	}						\
+} while(0)
+#endif
+
+#endif
--- linux-2.4.18-pre3/include/linux/sched.h	Fri Dec 21 11:19:23 2001
+++ linux-akpm/include/linux/sched.h	Sat Jan 12 12:22:29 2002
@@ -13,6 +13,7 @@ extern unsigned long event;
 #include <linux/times.h>
 #include <linux/timex.h>
 #include <linux/rbtree.h>
+#include <linux/condsched.h>
 
 #include <asm/system.h>
 #include <asm/semaphore.h>
--- linux-2.4.18-pre3/mm/filemap.c	Thu Jan 10 13:39:50 2002
+++ linux-akpm/mm/filemap.c	Sat Jan 12 12:22:29 2002
@@ -296,10 +296,7 @@ static int truncate_list_pages(struct li
 
 			page_cache_release(page);
 
-			if (current->need_resched) {
-				__set_current_state(TASK_RUNNING);
-				schedule();
-			}
+			conditional_schedule();
 
 			spin_lock(&pagecache_lock);
 			goto restart;
@@ -609,6 +606,7 @@ void filemap_fdatasync(struct address_sp
 			UnlockPage(page);
 
 		page_cache_release(page);
+		conditional_schedule();
 		spin_lock(&pagecache_lock);
 	}
 	spin_unlock(&pagecache_lock);
@@ -1392,6 +1390,9 @@ page_ok:
 		offset &= ~PAGE_CACHE_MASK;
 
 		page_cache_release(page);
+
+		conditional_schedule();
+
 		if (ret == nr && desc->count)
 			continue;
 		break;
@@ -3025,6 +3026,8 @@ unlock:
 		SetPageReferenced(page);
 		UnlockPage(page);
 		page_cache_release(page);
+
+		conditional_schedule();
 
 		if (status < 0)
 			break;
--- linux-2.4.18-pre3/drivers/block/ll_rw_blk.c	Thu Jan 10 13:39:49 2002
+++ linux-akpm/drivers/block/ll_rw_blk.c	Sat Jan 12 12:22:29 2002
@@ -917,6 +917,7 @@ void submit_bh(int rw, struct buffer_hea
 			kstat.pgpgin += count;
 			break;
 	}
+	conditional_schedule();
 }
 
 /**

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Erik Andersen]

On Sat Jan 12, 2002 at 12:23:09PM -0800, Andrew Morton wrote:
> Ed Sweetman wrote:
> > 
> > If you want to test the preempt kernel you're going to need something that
> > can find the mean latancy or "time to action" for a particular program or
> > all programs being run at the time and then run multiple programs that you
> > would find on various peoples' systems.   That is the "feel" people talk
> > about when they praise the preempt patch.
> 
> Right.  And that is precisely why I created the "mini-ll" patch.  To
> give the improved "feel" in a way which is acceptable for merging into
> the 2.4 kernel.
> 
> And guess what?   Nobody has tested the damn thing, so it's going
> nowhere.

I've tested it.  I've been running it on my box for the last
several days.  Works just great.  My box has been quite solid
with it and I've not seen anything to prevent your sending it 
to Marcelo for 2.4.18...

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Stephan von Krawczynski]

On Sat, 12 Jan 2002 14:02:13 -0700
Erik Andersen <andersen@codepoet.org> wrote:

> > And guess what?   Nobody has tested the damn thing, so it's going
> > nowhere.
> 
> I've tested it.  I've been running it on my box for the last
> several days.  Works just great.  My box has been quite solid
> with it and I've not seen anything to prevent your sending it 
> to Marcelo for 2.4.18...

Sorry for this dumb question:
What is the difference to vanilla exactly like in your setup? Better
interactive feeling? Throughput?

Regards,
Stephan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Erik Andersen]

On Sat Jan 12, 2002 at 10:18:35PM +0100, Stephan von Krawczynski wrote:
> On Sat, 12 Jan 2002 14:02:13 -0700
> Erik Andersen <andersen@codepoet.org> wrote:
> 
> > > And guess what?   Nobody has tested the damn thing, so it's going
> > > nowhere.
> > 
> > I've tested it.  I've been running it on my box for the last
> > several days.  Works just great.  My box has been quite solid
> > with it and I've not seen anything to prevent your sending it 
> > to Marcelo for 2.4.18...
> 
> Sorry for this dumb question:
> What is the difference to vanilla exactly like in your setup? Better
> interactive feeling? Throughput?

To be honest, not a _huge_ difference.  I've been mostly doing
development, and when I happen to have, for example,  a kernel
compile and a gcc compile going on, xmms isn't skipping for me
at all, while previously I would hear skips every so often.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Stephan von Krawczynski]

On Sat, 12 Jan 2002 12:23:09 -0800
Andrew Morton <akpm@zip.com.au> wrote:

> Ed Sweetman wrote:
> > 
> > If you want to test the preempt kernel you're going to need something that
> > can find the mean latancy or "time to action" for a particular program or
> > all programs being run at the time and then run multiple programs that you
> > would find on various peoples' systems.   That is the "feel" people talk
> > about when they praise the preempt patch.
> 
> Right.  And that is precisely why I created the "mini-ll" patch.  To
> give the improved "feel" in a way which is acceptable for merging into
> the 2.4 kernel.

Hm, I am not quite sure about what you expect to hear about it, but:

a) It applies cleanly to 2.4.18-pre3.
b) It compiles
c) During a load of around 150 produced by (of course :-) "make -j bzImage" and
concurrent XMMS playing while my mail-client and mozilla are open, I cannot
"feel" a real big difference in interactivity compared to vanilla kernel. XMMS
hickups sometimes, mouse does kangaroo'ing, switching around different
X-screens and screen refresh (especially mozilla of course) are no big hit.

This is a dual PIII-1GHz/2 GB RAM and some swap. During make no swapping is
going on.

Sorry, but I cannot see (feel) the difference in _this_ test (if this is really
a test for what you intend to do). Compile time btw makes no difference either.
Perhaps this try is rather something for ingo and the scheduler...

Regards,
Stephan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Francois Romieu]

[Cc: trimmed]

Andrew Morton <akpm@zip.com.au> :
[mini-ll]
> And guess what?   Nobody has tested the damn thing, so it's going
> nowhere.

It allows me to del^W read NFS-mounted mail behind a linux router while I 
copy files locally on the router. If I don't apply mini-ll to the router, 
it's a "server foo not responding, still trying" fest. You know what
"interactivity feel" means when it happens.

If someone suspects the hardware is crap, it's a PIV motherboard with 
built-in Promise20265 and four IBM IC35L060AVER07-0 on their own channel.
Each disk has been able to behave normally during RAID1 rebuild.

Without mini-ll:
  well choosen file I/O => no file I/O, no networking, no console, *big pain*.
With mini-ll:
  well choosen file I/O => *only* those I/O suck (less than before btw).

-- 
Ueimor

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> And guess what?   Nobody has tested the damn thing, so it's going
> nowhere.

I've been testing it. It works for me, its not as good as the full one,
it seems obviously correct. What else am I supposed to say.

I'm pretty much exclusively running Andre's new IDE code too. In fact I'm
back to a page long list of applied diffs versus 2.4.18pre3, most of which
I need to feed Marcelo

Re: [2.4.17/18pre] VM and swap - it's really unusable

[J.A. Magallon]

On 20020112 Andrew Morton wrote:
>Ed Sweetman wrote:
>> 
>> If you want to test the preempt kernel you're going to need something that
>> can find the mean latancy or "time to action" for a particular program or
>> all programs being run at the time and then run multiple programs that you
>> would find on various peoples' systems.   That is the "feel" people talk
>> about when they praise the preempt patch.
>
>Right.  And that is precisely why I created the "mini-ll" patch.  To
>give the improved "feel" in a way which is acceptable for merging into
>the 2.4 kernel.
>
>And guess what?   Nobody has tested the damn thing, so it's going
>nowhere.
>

I have been running mini-ll on -pre3 for a time. And have just booted pre3
with full-ll. I see no marvelous diff between them, but I am not pushing
my box to their knees. 
I can get numbers for you, but is there any test out there that gives them ?
Something like 'under this damned test your system just delayed as much as xxx us'.
That kind of 'my xmms does not skip' does not look like a very serious measure.

And could you tell me if some of this patches can interfere with results ?
This is what I am running just now:
- 2.4.18-pre3
- vm fixes from aa (vm-22, vm-raend, truncate-garbage)
- ext3-0.9.17 update
- ide-20011210 (hint: plz, make it in mainline for the time of .18...)
- irqrate-A1
- interrupts-seq-file
- spinlock-cacheline + fast-pte from -aa
- scalable timers
- sensors-cvs
- bproc 3.1.5

On that i have run full-ll+ll_fixes (from -aa) or mini-ll+ll_fixes.

(If someone is interested, patches are at
http://giga.cps.unizar.es/~magallon/linux/ )

TIA.

-- 
J.A. Magallon                           #  Let the source be with you...        
mailto:jamagallon@able.es
Mandrake Linux release 8.2 (Cooker) for i586
Linux werewolf 2.4.18-pre3-beo #5 SMP Sun Jan 13 02:14:04 CET 2002 i686

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rusty Russell]

On Sat, 12 Jan 2002 12:23:09 -0800
Andrew Morton <akpm@zip.com.au> wrote:
> And guess what?   Nobody has tested the damn thing, so it's going
> nowhere.

Haven't had latency problems, to be honest.  Maybe I should start playing mp3s
while I code?

1) conditional_schedule?  Hmmm... Why the __set_current_state?  I think I prefer
   an explicit "if (need_schedule()) schedule()", with
   #define need_schedule() unlikely(current->need_resched)

2) I hate condsched.h: Use sched.h please!

3) Why this:
   > +#ifndef __LINUX_COMPILER_H
   > +#include <linux/compiler.h>
   > +#endif

Other than that, I like this patch.  Linus?
Rusty.
-- 
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[jogi]

On Sat, Jan 12, 2002 at 12:23:09PM -0800, Andrew Morton wrote:
> Ed Sweetman wrote:
> > 
> > If you want to test the preempt kernel you're going to need something that
> > can find the mean latancy or "time to action" for a particular program or
> > all programs being run at the time and then run multiple programs that you
> > would find on various peoples' systems.   That is the "feel" people talk
> > about when they praise the preempt patch.
> 
> Right.  And that is precisely why I created the "mini-ll" patch.  To
> give the improved "feel" in a way which is acceptable for merging into
> the 2.4 kernel.
> 
> And guess what?   Nobody has tested the damn thing, so it's going
> nowhere.

Ok, as promised, here are the results:

        13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp       18-pre3minill  
j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%        *
j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%        *
j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%        *
j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%        *
j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%        *
		                                                                                          
j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%    7:07.56  77%
j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%    7:17.15  74%
j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%    6:47.48  80%
j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%    6:34.02  83%
j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%    7:01.39  77%

Regards,

   Jogi

-- 

Well, yeah ... I suppose there's no point in getting greedy, is there?

    << Calvin & Hobbes >>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 12:42, jogi@planetzork.ping.de wrote:

>         13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp       18-pre3minill  
> j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%        *
> j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%        *
> j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%        *
> j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%        *
> j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%        *
> 		                                                                                          
> j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%    7:07.56  77%
> j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%    7:17.15  74%
> j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%    6:47.48  80%
> j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%    6:34.02  83%
> j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%    7:01.39  77%

Again, preempt seems to reign supreme.  Where is all the information
correlating preempt is inferior?  To be fair, however, we should bench a
mini-ll+s test.

But I stand by my original point that none of this matters all too
much.  A preemptive kernel will allow for future latency reduction
_without_ using explicit scheduling points everywhere there is a
problem.  This means we can tackle the problem and not provide a million
bandaids.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> Again, preempt seems to reign supreme.  Where is all the information
> correlating preempt is inferior?  To be fair, however, we should bench a
> mini-ll+s test.

How about some actual latency numbers ?

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Sun, Jan 13, 2002 at 07:32:18PM +0000, Alan Cox wrote:
> > Again, preempt seems to reign supreme.  Where is all the information
> > correlating preempt is inferior?  To be fair, however, we should bench a
> > mini-ll+s test.
> 
> How about some actual latency numbers ?

with an huge rescheduling rate (huge swapout/swapin load) and the
scheduler walking over 100 tasks at each schedule it is insane to
deduct anything from those numbers (-preempt was using O(1)
scheduler!!!!). so please don't make any assumption by just looking at
those numbers.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[J Sloan]

The problem here is that when people report
that the low latency patch works better for them
than the preempt patch, they aren't talking about
bebnchmarking the time to compile a kernel, they
are talking about interactive feel and smoothness.

You're speaking to a peripheral issue.

I've no agenda other than wanting to see linux
as an attractive option for the multimedia and
gaming crowds - and in my experience, the low
latency patches simply give a much smoother
feel and a more pleasant experience. Kernel
compilation time is the farthest thing from my
mind when e.g. playing Q3A!

I'd be happy to check out the preempt patch
again and see if anything's changed, if the
problem of tux+preempt oopsing has been
dealt with -

Regards,

jjs

Robert Love wrote:

>On Sun, 2002-01-13 at 12:42, jogi@planetzork.ping.de wrote:
>
>>        13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp       18-pre3minill  
>>j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%        *
>>j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%        *
>>j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%        *
>>j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%        *
>>j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%        *
>>		                                                                                          
>>j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%    7:07.56  77%
>>j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%    7:17.15  74%
>>j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%    6:47.48  80%
>>j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%    6:34.02  83%
>>j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%    7:01.39  77%
>>
>
>Again, preempt seems to reign supreme.  Where is all the information
>correlating preempt is inferior?  To be fair, however, we should bench a
>mini-ll+s test.
>
>But I stand by my original point that none of this matters all too
>much.  A preemptive kernel will allow for future latency reduction
>_without_ using explicit scheduling points everywhere there is a
>problem.  This means we can tackle the problem and not provide a million
>bandaids.
>
>	Robert Love
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 13, 2002 08:35 pm, J Sloan wrote:
> The problem here is that when people report
> that the low latency patch works better for them
> than the preempt patch, they aren't talking about
> bebnchmarking the time to compile a kernel, they
> are talking about interactive feel and smoothness.

Nobody is claiming the low latency patch works better than 
-preempt+lock_break, only that low latency can equal -preempt+lock_break, 
which is a claim I'm skeptical of, but oh well.

> I've no agenda other than wanting to see linux
> as an attractive option for the multimedia and
> gaming crowds - and in my experience, the low
> latency patches simply give a much smoother
> feel and a more pleasant experience. Kernel
> compilation time is the farthest thing from my
> mind when e.g. playing Q3A!

You need to read the thread *way* more closely ;-)

> I'd be happy to check out the preempt patch
> again and see if anything's changed, if the
> problem of tux+preempt oopsing has been
> dealt with -

Right, useful.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[J Sloan]

Daniel Phillips wrote:

>On January 13, 2002 08:35 pm, J Sloan wrote:
>
>>The problem here is that when people report
>>that the low latency patch works better for them
>>than the preempt patch, they aren't talking about
>>bebnchmarking the time to compile a kernel, they
>>are talking about interactive feel and smoothness.
>>
>
>Nobody is claiming the low latency patch works better than 
>-preempt+lock_break, only that low latency can equal -preempt+lock_break, 
>which is a claim I'm skeptical of, but oh well.
>
AFAICT Alan Cox  et al are saying that low-latency
gives better latency than -preempt, but that if lock-break
is added to -preempt, the results are basically the same.

IOW lock-break + preempt =~ low-latency as far as the
latency question is concerned.

>>I've no agenda other than wanting to see linux
>>as an attractive option for the multimedia and
>>gaming crowds - and in my experience, the low
>>latency patches simply give a much smoother
>>feel and a more pleasant experience. Kernel
>>compilation time is the farthest thing from my
>>mind when e.g. playing Q3A!
>>
>
>You need to read the thread *way* more closely ;-)
>
Admittedly my observations have been more from
an "end-user" point of view, because at the end
of the day, what I experience while using Linux as
a multimedia/gaming platform is worth more than a
barrel of benchmarks - and while kernel compilation
time is of interest, it is just _one_  benchmark in the
greater scheme of things. (not to mention that that
benchmark result could probably be matched in a
non -preempt kernel via /proc tuning)


>>I'd be happy to check out the preempt patch
>>again and see if anything's changed, if the
>>problem of tux+preempt oopsing has been
>>dealt with -
>>
>
>Right, useful.
>
See my previous reply, or the archives -

Regards,

jjs

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Robert Love wrote:
> 
> Again, preempt seems to reign supreme.  Where is all the information
> correlating preempt is inferior?  To be fair, however, we should bench a
> mini-ll+s test.

I can't say that I have ever seen any significant change in throughput
of anything with any of this stuff.

Benchmarks are well and good, but until we have a solid explanation for
the throughput changes which people are seeing, it's risky to claim
that there is a general benefit.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 14:46, Andrew Morton wrote:

> I can't say that I have ever seen any significant change in throughput
> of anything with any of this stuff.

I can send you some numbers.  It is typically 5-10% throughput increase
under load.  Obviously this work won't help a single task on a single
user system.  But things like (ack!) dbench 16 show a marked
improvement.

> Benchmarks are well and good, but until we have a solid explanation for
> the throughput changes which people are seeing, it's risky to claim
> that there is a general benefit.

I have an explanation.  We can schedule quicker off a woken task.  When
an event occurs that allows an I/O-blocked task to run, its time-to-run
is shorter.  Same event/response improvement that helps interactivity.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Robert Love wrote:
> 
> ...
> > Benchmarks are well and good, but until we have a solid explanation for
> > the throughput changes which people are seeing, it's risky to claim
> > that there is a general benefit.
> 
> I have an explanation.  We can schedule quicker off a woken task.  When
> an event occurs that allows an I/O-blocked task to run, its time-to-run
> is shorter.  Same event/response improvement that helps interactivity.
> 

Sounds more like handwaving that an explanation :)

The way to speed up dbench is to allow the processes which want to delete
files to actually do that.  This reduces the total amount of IO which the
test performs.  Another way is to increase usable memory (or at least to
delay the onset of balance_dirty going synchronous).  Possibly it's something
to do with letting kswapd schedule earlier.  Or bdflush.

In the swapstorm case, it's again not clear to me.  Perhaps it's due to prompter
kswapd activity, perhaps due somehow to improved request merging.

As I say, without a precise and detailed understanding of the mechanisms
I wouldn't be prepared to claim more than "speeds up dbench and swapstorms
for some reason".

(I'd _like_ to know the complete reason - that way we can stare at it
and maybe make things even better.  Doing a binary search through the
various chunks of the mini-ll patch would be instructive).

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Sun, Jan 13, 2002 at 03:04:35PM -0500, Robert Love wrote:
> user system.  But things like (ack!) dbench 16 show a marked
> improvement.

please try again on top of -aa, and I've to specify this : benchmarked
in a way that can be trusted and compared, so we can make some use of
this information.  This mean with -18pre2aa2 alone and only -preempt on
top of -18pre2aa2.

NOTE: I'd be glad to say "preempt rules", "go preempt", "preempt is
cool" like you as soon as I have some proof it makes _THE_ difference
and that it is worth the mess on SMP (per-cpu, RCU locking, etc...), not
to tell about the other architectures, but at the moment there's only a
number of people running xmms on mainline with the broken scheduling
points and those numbers that cannot be compared in any sane way. I
repeat, I'm not against preempt, I just want to get some real world
proof and measurement and at the moment I think preempt doesn't worth,
but if you give us _any_ real world proof that a that low mean latency
of the order of 10/100 usec matters to get most of the cpu cycles out of
the cpu during trashing (as it could be possible to speculate from the
broken benchmark posted in this thread), and that there's no real
regression with the additional branches in the spin_unlock in 100%
system load, I may change my mind (an of course, only for anything above
2.5, and still I think there are more interesting optimizations to do
rahter than requiring everybody spending lots of time fixing drivers,
auditing, fixing smp, rcu locking etc... but ok if it is obviously good
thing [aka no real regression and only benefits long term] it would be
ok to do it early as well). I'm not particularly worried about the
preempt lock around the per-cpu stuff, that's a cacheline local, and it
could go into the schedule_data like I did for the rcu per-cpu
variables, so they're at zero cacheline cost (RCU_poll patch costs now
only 1 instruction per schedule and zero memory overhead [an incl
instruction precisely]).

> > Benchmarks are well and good, but until we have a solid explanation for
> > the throughput changes which people are seeing, it's risky to claim
> > that there is a general benefit.
> 
> I have an explanation.  We can schedule quicker off a woken task.  When
> an event occurs that allows an I/O-blocked task to run, its time-to-run
> is shorter.  Same event/response improvement that helps interactivity.

That's a nice speculation out of a broken comparison, it may be really
the case, there's no way to be sure, before that sum of usec you should
also sum the seconds spent walking the tasklist in the non O(1)
scheduler.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Mon, 2002-01-14 at 06:56, Andrea Arcangeli wrote:
> On Sun, Jan 13, 2002 at 03:04:35PM -0500, Robert Love wrote:
> > user system.  But things like (ack!) dbench 16 show a marked
> > improvement.
> 
> please try again on top of -aa, and I've to specify this : benchmarked
> in a way that can be trusted and compared, so we can make some use of
> this information.  This mean with -18pre2aa2 alone and only -preempt on
> top of -18pre2aa2.

I realize the test isn't directly comparing what we want, so I asked him
for ll+O(1) benchmark, which he gave.  Another set would be to do
preempt and ll alone.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Mon, Jan 14, 2002 at 08:38:54AM -0500, Robert Love wrote:
> On Mon, 2002-01-14 at 06:56, Andrea Arcangeli wrote:
> > On Sun, Jan 13, 2002 at 03:04:35PM -0500, Robert Love wrote:
> > > user system.  But things like (ack!) dbench 16 show a marked
> > > improvement.
> > 
> > please try again on top of -aa, and I've to specify this : benchmarked
> > in a way that can be trusted and compared, so we can make some use of
> > this information.  This mean with -18pre2aa2 alone and only -preempt on
> > top of -18pre2aa2.
> 
> I realize the test isn't directly comparing what we want, so I asked him
> for ll+O(1) benchmark, which he gave.  Another set would be to do
      ^^ actually mini-ll

right (I was still in the middle of the backlog of my emails, so I
didn't know he just produced the mini-ll+O(1)). The mini-ll+O(1) shows
that -preempt is still a bit faster (as expected not much faster
anymore). The reason it is faster it is probably really the sum of few
usec latency of userspace cpu cycles that you save. However given the
small difference in numbers in this patological case (-j1 obviously
cannot take advantage of the few usec less of reduced latency) still
makes me to think it doesn't worth the pain and the complexity, or at
least somebody should also proof that it doesn't visibly drop
performance in a 100% cpu bound _system_ (not user) time load (ala
pagecache_lock collision testcase with sendfile etc..), in general with
a single thread in the system.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[jogi]

On Sun, Jan 13, 2002 at 01:22:57PM -0500, Robert Love wrote:
> On Sun, 2002-01-13 at 12:42, jogi@planetzork.ping.de wrote:
> 
> >         13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp       18-pre3minill  
> > j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%        *
> > j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%        *
> > j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%        *
> > j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%        *
> > j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%        *
> > 		                                                                                          
> > j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%    7:07.56  77%
> > j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%    7:17.15  74%
> > j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%    6:47.48  80%
> > j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%    6:34.02  83%
> > j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%    7:01.39  77%
> 
> Again, preempt seems to reign supreme.  Where is all the information
> correlating preempt is inferior?  To be fair, however, we should bench a
> mini-ll+s test.

Your wish is granted. Here are the results for mini-ll + scheduler:

j100:   8:26.54
j100:   7:50.35
j100:   6:49.59
j100:   6:39.30
j100:   6:39.70
j75:    6:01.02
j75:    6:12.16
j75:    6:04.60
j75:    6:24.58
j75:    6:28.00

Jogi

-- 

Well, yeah ... I suppose there's no point in getting greedy, is there?

    << Calvin & Hobbes >>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[jogi]

On Sun, Jan 13, 2002 at 03:35:08PM -0500, Mark Hahn wrote:
> > > >         13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp       18-pre3minill  
> > > > j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%        *
> > > > j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%        *
> > > > j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%        *
> > > > j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%        *
> > > > j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%        *
> > > > 		                                                                                          
> > > > j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%    7:07.56  77%
> > > > j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%    7:17.15  74%
> > > > j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%    6:47.48  80%
> > > > j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%    6:34.02  83%
> > > > j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%    7:01.39  77%
> > > 
> > > Again, preempt seems to reign supreme.  Where is all the information
> > > correlating preempt is inferior?  To be fair, however, we should bench a
> > > mini-ll+s test.
> > 
> > Your wish is granted. Here are the results for mini-ll + scheduler:
> > 
> > j100:   8:26.54
> > j100:   7:50.35
> > j100:   6:49.59
> > j100:   6:39.30
> > j100:   6:39.70
> > j75:    6:01.02
> > j75:    6:12.16
> > j75:    6:04.60
> > j75:    6:24.58
> > j75:    6:28.00
> 
> how about a real benchmark like -j2 or so (is this a dual machine?)

Why does everybody think this is no *real* benchmark? When I remember
the good old days at the university the systems I tried to compile some
applications on were *always* overloaded. Would it make a difference for
you if I would run

for a in lots_of.srpm; do
  rpm --rebuild $a &
done

Basically this gives the same result: lots of compile jobs running in
parallel. All *I* am doing is doing it a little extreme since running
the compile with make -j2 does not make a *noticable* difference at all.
And as I said previously my idea was to get the system into high memory
pressure and test the different vms (AA and RvR) ...

Furthermore some people think this combination (sched+preempt) is only
good for latency (if at all) all I can say is that this works *very*
well for me latency wise. Since I don't know how to measure latency
exactly I tried to run my compile script (make -j50) while running my
usual desktop + xmms. Result: xmms was *not* skipping, although the
system was ~70MB into swap and the load was >50. Changing workspaces
worked immedeatly all the time. But I was able to get xmms to skip for
a short while by starting galeon, StarOffice, gimp with ~10 pictures
all at the same time. But when all applications came up xmms was not
skipping any more and the system was ~130MB into swap. This is the best
result for me so far but I have to admit that I did not test mini-ll
+sched in this area (I can test this earliest on wednesday, sorry).

Since it is a little while since I posted my system specs here they are:

- Athlon 1.2GHz (single proc)
- 256 MB
- IDE drive (quantum)

> also, I've often found the user/sys/elapsed components to all be interesting;
> how do they look?  (I'd expect preempt to have more sys, for instance.)

        13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp       18-pre3smini  
        (sys) (user)
j100:   30.78 297.07    32.40 294.38        *           27.74 296.02    27.55 292.95    28.30 297.67
j100:   30.92 297.11    33.04 295.15        *           29.14 296.25    26.88 292.77    28.13 296.44
j100:   29.58 297.90    34.01 294.16        *           27.56 295.76    26.25 293.79    27.96 296.47
j100:   30.62 297.13    32.00 294.30        *           28.47 296.46    27.64 293.42    27.50 297.47
j100:   30.48 299.43    32.28 295.42        *           27.77 296.44    27.53 292.10    27.23 297.24

As expected the system and the user times are almost identical. The "fastest"
compile results are always where the job gets the most %cpu time. So I guess
it would be more interesting to see how much cpu time e.g. kswapd gets.

Probably I have to enhance my script to run vmstat in the background ...
Would this provide useful data?


Regards,

   Jogi


-- 

Well, yeah ... I suppose there's no point in getting greedy, is there?

    << Calvin & Hobbes >>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

Well to start with:
	1) Maybe I should be more precise: The latency measures I've seen
	posted all favor Morton and not preempt. Since the claimed purpose
	of both patches is improving latency isn't that more interesting
	than measuremts of kernel compile?
	
	2) In these measurements
	the tree is different each time so the measurement doesn't
	seem very stable. It's not exactly a secret that file layout 
	can have an affect on performance.

	3) There is no measure of preempt without Ingo's scheduler

	4) this is what I want to see:
		Run the periodic SCHED_FIFO task I've posted multiple times
		Let's see worst case error
		Let's see effect on the background kernel compile

	All the rest is just so much talk about "interactive feel". I saw
	exactly the same claims from the people who wanted kernel graphics.




On Sun, Jan 13, 2002 at 01:22:57PM -0500, Robert Love wrote:
> On Sun, 2002-01-13 at 12:42, jogi@planetzork.ping.de wrote:
> 
> >         13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp       18-pre3minill  
> > j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%        *
> > j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%        *
> > j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%        *
> > j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%        *
> > j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%        *
> > 		                                                                                          
> > j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%    7:07.56  77%
> > j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%    7:17.15  74%
> > j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%    6:47.48  80%
> > j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%    6:34.02  83%
> > j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%    7:01.39  77%
> 
> Again, preempt seems to reign supreme.  Where is all the information
> correlating preempt is inferior?  To be fair, however, we should bench a
> mini-ll+s test.
> 
> But I stand by my original point that none of this matters all too
> much.  A preemptive kernel will allow for future latency reduction
> _without_ using explicit scheduling points everywhere there is a
> problem.  This means we can tackle the problem and not provide a million
> bandaids.
> 
> 	Robert Love

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Sun, Jan 13, 2002 at 01:22:57PM -0500, Robert Love wrote:
> On Sun, 2002-01-13 at 12:42, jogi@planetzork.ping.de wrote:
> 
> >         13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp       18-pre3minill  
> > j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%        *
> > j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%        *
> > j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%        *
> > j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%        *
> > j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%        *
> > 		                                                                                          
> > j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%    7:07.56  77%
> > j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%    7:17.15  74%
> > j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%    6:47.48  80%
> > j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%    6:34.02  83%
> > j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%    7:01.39  77%
> 
> Again, preempt seems to reign supreme.  Where is all the information

those comparison are totally flawed. There's nothing to compare in
there. 

minill misses the O(1) scheduler, and -aa has faster vm etc... there's
absolutely nothing to compare in the above numbers, all variables
changes at the same time.

I'm amazed I've to say this, but in short:

1) to compare minill with preempt, apply both patches to 18-pre3, as the
   only patch applied (no O(1) in the way of preempt!!!!)
2) to compare -aa with preempt, apply -preempt on top of -aa and see
   what difference it makes

If you don't follow exactly those simple rules you will change an huge
amount of variables at the same time, and it will be again impossible to
make any comparison or deduction from the numbers.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Sat, Jan 12, 2002 at 01:12:27PM -0800, J Sloan wrote:
> 
>    Ah - if it stands a chance of going into 2.4,
>    I'll test the heck out of it!
>    I'll give it the Q3A test, the RtCW test, the
>    xine/xmms/dbench tests, and more - glad
>    to be of service.
>    jjs
>    Andrew Morton wrote:
> 
> Ed Sweetman wrote:
> 
> If you want to test the preempt kernel you're going to need something that
> can find the mean latancy or "time to action" for a particular program or
> all programs being run at the time and then run multiple programs that you
> would find on various peoples' systems.   That is the "feel" people talk
> about when they praise the preempt patch.
> 
> Right.  And that is precisely why I created the "mini-ll" patch.  To
> give the improved "feel" in a way which is acceptable for merging into
> the 2.4 kernel.
> And guess what?   Nobody has tested the damn thing, so it's going
> nowhere.
> Here it is again:
> --- linux-2.4.18-pre3/fs/buffer.c       Fri Dec 21 11:19:14 2001
> +++ linux-akpm/fs/buffer.c      Sat Jan 12 12:22:29 2002
> @@ -249,12 +249,19 @@ static int wait_for_buffers(kdev_t dev, 
>         struct buffer_head * next;
>         int nr;
>  
> -       next = lru_list[index];
>         nr = nr_buffers_type[index];
> +repeat:
> +       next = lru_list[index];
>         while (next && --nr >= 0) {
>                 struct buffer_head *bh = next;
>                 next = bh->b_next_free;
>  
> +               if (dev == NODEV && current->need_resched) {
> +                       spin_unlock(&lru_list_lock);
> +                       conditional_schedule();
> +                       spin_lock(&lru_list_lock);
> +                       goto repeat;
> +               }
>                 if (!buffer_locke
> d(bh)) {

this introduces possibility of looping indefinitely, this is why I
rejected it while I merged the mini-ll other points into -aa, if you
want to do anything like that at the very least you should roll the head
of the list as well or something like that.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Andrea Arcangeli wrote:
> 
> > --- linux-2.4.18-pre3/fs/buffer.c       Fri Dec 21 11:19:14 2001
> > +++ linux-akpm/fs/buffer.c      Sat Jan 12 12:22:29 2002
> > @@ -249,12 +249,19 @@ static int wait_for_buffers(kdev_t dev,
> >         struct buffer_head * next;
> >         int nr;
> >
> > -       next = lru_list[index];
> >         nr = nr_buffers_type[index];
> > +repeat:
> > +       next = lru_list[index];
> >         while (next && --nr >= 0) {
> >                 struct buffer_head *bh = next;
> >                 next = bh->b_next_free;
> >
> > +               if (dev == NODEV && current->need_resched) {
> > +                       spin_unlock(&lru_list_lock);
> > +                       conditional_schedule();
> > +                       spin_lock(&lru_list_lock);
> > +                       goto repeat;
> > +               }
> >                 if (!buffer_locke
> > d(bh)) {
> 
> this introduces possibility of looping indefinitely, this is why I
> rejected it while I merged the mini-ll other points into -aa, if you
> want to do anything like that at the very least you should roll the head
> of the list as well or something like that.

I ended up deciding that the `NODEV' check here avoids livelocks.
Unless, of course, the scheduling pressure is so high that we can't
even run a few statements.  I which case the interrupt load will be so 
high that the machine stops anyway.  Possibly it needs to check `refile'
as well.

A technique I frequently use in the full-ll patch is to only reschedule
after we've executed the loop (say) 16 times before dropping out.  This
assures that forward progress is made.  There's a test mode in the full
ll patch - in this mode, it *always* assumes that need_resched is true.
If the patch runs OK in this mode without livelocking, we know that it
can't livelock.

Anyway, I'll revisit this.  It is a "must fix".  wait_for_buffers() is
possibly the worst cause of latency in the kernel.  The usual scenario
is where kupdate has written 10,000 buffers and then sleeps.  Next time
it wakes, it has 10,000 clean, unlocked buffers to move from BUF_LOCKED
onto BUF_CLEAN.  It does this with lru_list_lock held.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[jogi]

On Sat, Jan 12, 2002 at 02:00:17PM -0500, Ed Sweetman wrote:
> 
> 
> > On Sat, Jan 12, 2002 at 09:52:09AM -0700, yodaiken@fsmlabs.com wrote:
> > > On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> > > > I did my usual compile testings (untar kernel archive, apply patches,
> > > > make -j<value> ...
> > >
> > > If I understand your test,
> > > you are testing different loads - you are compiling kernels that may
> differ
> > > in size and makefile organization, not to mention different layout on
> the
> > > file system and disk.
> 
> Can someone tell me why we're "testing" the preempt kernel by running
> make -j on a build?  What exactly is this going to show us?  The only thing
> i can think of is showing us that throughput is not damaged when you want to
> run single apps by using preempt.  You dont get to see the effects of the
> kernel preemption because all the damn thing is doing is preempting itself.
> 
> If you want to test the preempt kernel you're going to need something that
> can find the mean latancy or "time to action" for a particular program or
> all programs being run at the time and then run multiple programs that you
> would find on various peoples' systems.   That is the "feel" people talk
> about when they praise the preempt patch.  make -j'ing something and not
> testing anything else but that will show you nothing important except "does
> throughput get screwed by the preempt patch."   Perhaps checking the
> latencies on a common program on people's systems like mozilla or konqueror
> while doing a 'make -j N bzImage'  would be a better idea.

That's the second test I am normally running. Just running xmms while
doing the kernel compile. I just wanted to check if the system slows
down because of preemption but instead it compiled the kernel even
faster :-) But so far I was not able to test the latency and furthermore
it is very difficult to "measure" skipping of xmms ...

> > Ouch, I assumed this wasn't the case indeed.

Sorry for not answering immedeatly but I am compiling the same kernel
source with the same .config and everything I could think of being the
same! I even do a 'rm -rf linux' after every run and untar the same
sources *every* time.

Regards,

   Jogi

-- 

Well, yeah ... I suppose there's no point in getting greedy, is there?

    << Calvin & Hobbes >>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[george anzinger]

jogi@planetzork.ping.de wrote:
> 
> On Sat, Jan 12, 2002 at 02:00:17PM -0500, Ed Sweetman wrote:
> >
> >
> > > On Sat, Jan 12, 2002 at 09:52:09AM -0700, yodaiken@fsmlabs.com wrote:
> > > > On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> > > > > I did my usual compile testings (untar kernel archive, apply patches,
> > > > > make -j<value> ...
> > > >
> > > > If I understand your test,
> > > > you are testing different loads - you are compiling kernels that may
> > differ
> > > > in size and makefile organization, not to mention different layout on
> > the
> > > > file system and disk.
> >
> > Can someone tell me why we're "testing" the preempt kernel by running
> > make -j on a build?  What exactly is this going to show us?  The only thing
> > i can think of is showing us that throughput is not damaged when you want to
> > run single apps by using preempt.  You dont get to see the effects of the
> > kernel preemption because all the damn thing is doing is preempting itself.
> >
> > If you want to test the preempt kernel you're going to need something that
> > can find the mean latancy or "time to action" for a particular program or
> > all programs being run at the time and then run multiple programs that you
> > would find on various peoples' systems.   That is the "feel" people talk
> > about when they praise the preempt patch.  make -j'ing something and not
> > testing anything else but that will show you nothing important except "does
> > throughput get screwed by the preempt patch."   Perhaps checking the
> > latencies on a common program on people's systems like mozilla or konqueror
> > while doing a 'make -j N bzImage'  would be a better idea.
> 
> That's the second test I am normally running. Just running xmms while
> doing the kernel compile. I just wanted to check if the system slows
> down because of preemption but instead it compiled the kernel even
> faster :-) 

This sort of thing is nice to hear, but, it does show up a problem in
the non-preempt kernel.  That preemption improves compile performance
implies that the kernel is not doing the right thing during a normal
compile and that preemption, to some extent, corrects the problem.  But
preemption adds the overhead of additional context switches.  It would
be nice to know where the time is coming from.  I.e. lets assume that
the actual compile takes about the same amount of execution time with or
without preemption.  Then for the preemptable kernel to do the job
faster something else must go up, idle time perhaps.  If this is the
case, then there is some place in the kernel that is wasting cpu time
and that is preemptable and the preemptable patch is moving this idle
time to the idle process.  

What ever the reason, while I do want to promote preemption, I think we
should look at this issue and, at the very least, explain it.

>           But so far I was not able to test the latency and furthermore
> it is very difficult to "measure" skipping of xmms ...
> 
> > > Ouch, I assumed this wasn't the case indeed.
> 
> Sorry for not answering immedeatly but I am compiling the same kernel
> source with the same .config and everything I could think of being the
> same! I even do a 'rm -rf linux' after every run and untar the same
> sources *every* time.
> 
> Regards,
> 
>    Jogi
> 
> --
> 
> Well, yeah ... I suppose there's no point in getting greedy, is there?
> 
>     << Calvin & Hobbes >>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
George           george@mvista.com
High-res-timers: http://sourceforge.net/projects/high-res-timers/
Real time sched: http://sourceforge.net/projects/rtsched/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[jogi]

On Sat, Jan 12, 2002 at 09:52:09AM -0700, yodaiken@fsmlabs.com wrote:
> On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> > I did my usual compile testings (untar kernel archive, apply patches,
> > make -j<value> ...
> 
> If I understand your test, 
> you are testing different loads - you are compiling kernels that may differ
> in size and makefile organization, not to mention different layout on the
> file system and disk.

No, I use a script which is run in single user mode after a reboot. So
there are only a few processes running when I start the script (see
attachment) and the jobs should start from the same environment.

> What happens when you do the same test, compiling one kernel under multiple
> different kernels?

That is exactly what I am doing. I even try to my best to have the exact
same starting environment ...

Regards,

   Jogi

-- 

Well, yeah ... I suppose there's no point in getting greedy, is there?

    << Calvin & Hobbes >>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sun, Jan 13, 2002 at 04:18:23PM +0100, jogi@planetzork.ping.de wrote:
> On Sat, Jan 12, 2002 at 09:52:09AM -0700, yodaiken@fsmlabs.com wrote:
> > On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> > > I did my usual compile testings (untar kernel archive, apply patches,
> > > make -j<value> ...
> > 
> > If I understand your test, 
> > you are testing different loads - you are compiling kernels that may differ
> > in size and makefile organization, not to mention different layout on the
> > file system and disk.
> 
> No, I use a script which is run in single user mode after a reboot. So
> there are only a few processes running when I start the script (see
> attachment) and the jobs should start from the same environment.

But your description makes it sound like you do
	untar kernel X
	apply patches Y
	make -j  Tree

I'm sorry if I'm getting you wrong, but each of these steps is
variable.
Even if X and Y are the same each time, "Tree" is different.

The test should be
	reboot
		N times
		make clean
		time make -j Tree

Am I misunderstaning your test?


> 
> > What happens when you do the same test, compiling one kernel under multiple
> > different kernels?
> 
> That is exactly what I am doing. I even try to my best to have the exact
> same starting environment ...
> 
> Regards,
> 
>    Jogi
> 
> -- 
> 
> Well, yeah ... I suppose there's no point in getting greedy, is there?
> 
>     << Calvin & Hobbes >>

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[jogi]

On Sun, Jan 13, 2002 at 10:51:04AM -0700, yodaiken@fsmlabs.com wrote:
> On Sun, Jan 13, 2002 at 04:18:23PM +0100, jogi@planetzork.ping.de wrote:
> > On Sat, Jan 12, 2002 at 09:52:09AM -0700, yodaiken@fsmlabs.com wrote:
> > > On Sat, Jan 12, 2002 at 04:07:14PM +0100, jogi@planetzork.ping.de wrote:
> > > > I did my usual compile testings (untar kernel archive, apply patches,
> > > > make -j<value> ...
> > > 
> > > If I understand your test, 
> > > you are testing different loads - you are compiling kernels that may differ
> > > in size and makefile organization, not to mention different layout on the
> > > file system and disk.
> > 
> > No, I use a script which is run in single user mode after a reboot. So
> > there are only a few processes running when I start the script (see
> > attachment) and the jobs should start from the same environment.
> 
> But your description makes it sound like you do
> 	untar kernel X
> 	apply patches Y
> 	make -j  Tree
> 
> I'm sorry if I'm getting you wrong, but each of these steps is
> variable.
> Even if X and Y are the same each time, "Tree" is different.

X and Y are the same. But I don't really get why this is still
"different" ... If you think this could be because of the fs
fragmentation then I will enhance my test. I think I have a spare
partition somewhere which I can format each time before untar the
kernel sources and so on. But why can I reproduce the results then?
Ok, not exactly but the results do get close ...

Furthermore I am timing not only the make -j<value> but also the
complete untar and applying of patches. So basically I am timing the
following:

tar xvf linux-x.y.z.tar
patch -p0 < some_patches
cd linux; cp ../config-x.y.z .config
make oldconfig dep
make -j $PAR bzImage modules

and afterwards

cd .. ; rm -rf linux

and start again. Its just the same as doing 'rpm --rebuild' with

MAKE=make -j $PAR

> The test should be
> 	reboot
> 		N times
> 		make clean
> 		time make -j Tree
> 
> Am I misunderstaning your test?

No, but I don't understand why this should make any difference. I do not
propose my way of testing as *the* benchmark. Its just a benchmark of
something which I do most of the time on my system (compiling) in an
extreme way ...

Kind regards,

   Jogi

-- 

Well, yeah ... I suppose there's no point in getting greedy, is there?

    << Calvin & Hobbes >>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 10:18, jogi@planetzork.ping.de wrote:

> No, I use a script which is run in single user mode after a reboot. So
> there are only a few processes running when I start the script (see
> attachment) and the jobs should start from the same environment.
> 
> > What happens when you do the same test, compiling one kernel under multiple
> > different kernels?
> 
> That is exactly what I am doing. I even try to my best to have the exact
> same starting environment ...

So there you go, his testing is accurate.  Now we have results that
preempt works and is best and it is still refuted.  Everyone is running
around with these "ll is best" or "preempt sucks throughput" and that is
not true.  Further, with preempt we can improve things cleanly, and I
don't think that necessarily implies priority inversion problems.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Sun, Jan 13, 2002 at 01:11:21PM -0500, Robert Love wrote:
> On Sun, 2002-01-13 at 10:18, jogi@planetzork.ping.de wrote:
> 
> > No, I use a script which is run in single user mode after a reboot. So
> > there are only a few processes running when I start the script (see
> > attachment) and the jobs should start from the same environment.
> > 
> > > What happens when you do the same test, compiling one kernel under multiple
> > > different kernels?
> > 
> > That is exactly what I am doing. I even try to my best to have the exact
> > same starting environment ...
> 
> So there you go, his testing is accurate.  Now we have results that
> preempt works and is best and it is still refuted.  Everyone is running
> around with these "ll is best" or "preempt sucks throughput" and that is

assuming the report can be trusted this is not the test where we can
measure a throughput regression, this is a VM intensive test and nothing
else.  Swap load.

In short, run top and check you've 100% system load and cpus are never
idle or in userspace, and _then_ it will most certainly get an interesting
benchmark for -preempt throughput.

Furthmore the whole comparison is flawed, just -O(1) is as broken as
mainline w.r.t. the scheduling point, and -aa has the right scheduling
point but not the -O(1) scheduler, so there's no way to compare those
numbers at all. If you want to make any real comparison you should apply
-preempt on top of -aa.

Assuming it is really -preempt that makes the numbers more repetable
(not the fact -O(1) alone has the broken rescheduling points), this
still doesn't proof anything yet, the lower numbers are most certainly
because those tasks getting the page faults get rescheduled faster, -aa
didn't do more cpu work, it just had the cpus more idle than -preempt
apparently, this may be the indication of an important scheduling point
missing somewhere, if somebody could run a lowlatency measurement during
a swap intensive load and send me the offending IP that could probably
be addressed with a one liner.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 12, 2002 04:07 pm, jogi@planetzork.ping.de wrote:
> Hello Andrea,
> 
> I did my usual compile testings (untar kernel archive, apply patches,
> make -j<value> ...
> 
> Here are some results (Wall time + Percent cpu) for each of the consecutive five runs:
> 
>         13-pre5aa1      18-pre2aa2      18-pre3         18-pre3s        18-pre3sp
> j100:   6:59.79  78%    7:07.62  76%        *           6:39.55  81%    6:24.79  83%
> j100:   7:03.39  77%    8:10.04  66%        *           8:07.13  66%    6:21.23  83%
> j100:   6:40.40  81%    7:43.15  70%        *           6:37.46  81%    6:03.68  87%
> j100:   7:45.12  70%    7:11.59  75%        *           7:14.46  74%    6:06.98  87%
> j100:   6:56.71  79%    7:36.12  71%        *           6:26.59  83%    6:11.30  86%
> 
> j75:    6:22.33  85%    6:42.50  81%    6:48.83  80%    6:01.61  89%    5:42.66  93%
> j75:    6:41.47  81%    7:19.79  74%    6:49.43  79%    5:59.82  89%    6:00.83  88%
> j75:    6:10.32  88%    6:44.98  80%    7:01.01  77%    6:02.99  88%    5:48.00  91%
> j75:    6:28.55  84%    6:44.21  80%    9:33.78  57%    6:19.83  85%    5:49.07  91%
> j75:    6:17.15  86%    6:46.58  80%    7:24.52  73%    6:23.50  84%    5:58.06  88%
> 
> * build incomplete (OOM killer killed several cc1 ... )
> 
> So far 2.4.13-pre5aa1 had been the king of the block in compile times.
> But this has changed. Now the (by far) fastest kernel is 2.4.18-pre
> + Ingos scheduler patch (s) + preemptive patch (p). I did not test
> preemptive patch alone so far since I don't know if the one I have
> applies cleanly against -pre3 without Ingos patch. I used the
> following patches:
> 
> s: sched-O1-2.4.17-H6.patch
> p: preempt-kernel-rml-2.4.18-pre3-ingo-1.patch
> 
> I hope this info is useful to someone.

I'd like to add my 'me too' to those who have requested a re-run of this test, building
the *identical* kernel tree every time, starting from the same initial conditions.
Maybe that's what you did, but it's not clear from your post.

Thanks,

Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Sun, 2002-01-13 at 17:55, Daniel Phillips wrote:

> I'd like to add my 'me too' to those who have requested a re-run of this test, building
> the *identical* kernel tree every time, starting from the same initial conditions.
> Maybe that's what you did, but it's not clear from your post.

He later said he did in fact build the same tree, from the same initial
condition, in single user mode, etc etc ... sounded like good testing
methodology to me.

I later asked for a test of Ingo's sched with ll (to compare to Ingo's
sched with preempt).  In this test, like the others, preempt gives the
best times.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sun, Jan 13, 2002 at 05:56:25PM -0500, Robert Love wrote:
> On Sun, 2002-01-13 at 17:55, Daniel Phillips wrote:
> 
> > I'd like to add my 'me too' to those who have requested a re-run of this test, building
> > the *identical* kernel tree every time, starting from the same initial conditions.
> > Maybe that's what you did, but it's not clear from your post.
> 
> He later said he did in fact build the same tree, from the same initial
> condition, in single user mode, etc etc ... sounded like good testing
> methodology to me.

Really? You think that 
		unpack a tar archive
		make

is a repeatable benchmark?

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Marian Jancar]

Daniel Phillips wrote:

>I'd like to add my 'me too' to those who have requested a re-run of this test, building
>the *identical* kernel tree every time, starting from the same initial conditions.
>Maybe that's what you did, but it's not clear from your post.
>

Its obvious its same source code under different kernels. It would have 
no sense to do othervise and it would require more effort than just boot 
other image and run same script.

Marian

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Mon, Jan 14, 2002 at 12:18:57PM +0100, Marian Jancar wrote:
> Daniel Phillips wrote:
> 
> >I'd like to add my 'me too' to those who have requested a re-run of this test, building
> >the *identical* kernel tree every time, starting from the same initial conditions.
> >Maybe that's what you did, but it's not clear from your post.
> >
> 
> Its obvious its same source code under different kernels. It would have 
> no sense to do othervise and it would require more effort than just boot 
> other image and run same script.

It's a different tree each time. Same contents. Different 
tree.



-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Pavel Machek]

Hi!

> > exhausted...)  What sound output device DOESN'T have this much cache?  (You 
> > mentioned USB speakers in your diary at one point, which seemed to be like 
> > those old "paralell port cable plus a few resistors equals sound output" 
> > hacks...)
> 
> Umm no USB audio is rather good. USB sends isosynchronous, time guaranteed
> sample streams down the USB bus, to the speakers where the A to D is clear
> of the machine proper.

Actually usb speakers are *very* sensitive to latency, and when they drop
out,they drop out for half a second...
									Pavel

-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ken Brownfield]

On Wed, Jan 09, 2002 at 12:10:38AM +0000, Alan Cox wrote:
| That is generally not true. Pe-emption is used in user space to prevent
| applications doing very stupid things. Pre-emption in a trusted environment
| can often be most efficient if done by the programs themselves.
| Userspace is not a trusted environment

That's true, but at some point in the future I think the work involved
in making sure all new additional kernel code and all new intra-kernel
interactions are "tuned" becomes larger than going preemptive all the
way down.

Apple had its arguments for cooperative, along the same lines as what
you've mentioned I believe.  And while I agree that the kernel is a much
_more_ trusted environment, I think the possibilities easily remain for
abuse given that there are A) more and more people contributing kernel
code every day, and B) countless unspeakably evil modules out there.

And the preempt tunability that has been mentioned sounds like it would
go a long way.

| Andrew's patches give you 1mS worst case latency for normal situations, that
| is below human perception, and below scheduling granularity. In other words
| without the efficiency loss and the debugging problems you can place the
| far enough latency below other effects that it isnt worth attacking any more.

It sounds like the LL patches are easier and less prone to locking
issues with a lot of the benefit.  But I can't help but feel that it's
not using the right tool for the job.  I think the end result of
stabilizing a preemptive kernel (in 2.5?) is worth the price, IMHO.
-- 
Ken.
brownfld@irridia.com

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> That's true, but at some point in the future I think the work involved
> in making sure all new additional kernel code and all new intra-kernel
> interactions are "tuned" becomes larger than going preemptive all the
> way down.

It makes no difference to the kernel the work is the same in all cases
because you cannot pre-empt while holding a lock. Therefore you have to do
all the lock analysis anyway

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Dieter Nützel]

On Wednesday, 9. January 2002 00:02, Luigi Genoni wrote:
> On Tue, 8 Jan 2002, Daniel Phillips wrote:
> > On January 8, 2002 04:29 pm, Andrea Arcangeli wrote:
[-]
> > > I also don't want to devaluate the preemptive kernel approch (the mean
> > > latency it can reach is lower than the one of the lowlat kernel,
> > > however I personally care only about worst case latency and this is why
> > > I don't feel the need of -preempt),
> >
> > This is exactly the case that -preempt handles well.  On the other hand,
> > trying to show that scheduling hacks satisfy any given latency bound is
> > equivalent to solving the halting problem.
> >
> > I thought you had done some real time work?
> >
> > > but I just wanted to make clear that the
> > > idea that is floating around that preemptive kernel is all goodness is
> > > very far from reality, you get very low mean latency but at a price.
> >
> > A price lots of people are willing to pay
>
> Probably sometimes they are not making a good business. In the reality
> preempt is good in many scenarios, as I said, and I agree that for
> desktops, and dedicated servers where just one application runs, and
> probably the CPU is idle the most of the time,

OK, good. You are much at the same line than I am.

Should we starting not only to differentiate between UP and SMP systems but 
allthought between desktop and (big) servers?
I remember one saying. "Think, this patch is worth only for ~0.05% of the 
Linux users..." (He meant the multi SMP system users.)

Allmost 99.95% of the Linux users running desktops and I am somewhat tiered 
of saying, "sorry, Linux is under development..."
Look at the imprint of the famous German ct magazine (they are not even known 
as Linux bashers...;-). It shows little penguins falling like domino stones 
(starting with 2.4.17).

Let me rephrase it:
I appreciate all your great work and I know "only" some (little) internals of 
it but we should do some interactivity improvements for the 2.4 kernel, too.
I know what it's worth Andrew's (lowlatency patch) and Robert's (George 
Anzinger's) preempt patch. In short the system (bigger desktop) flies.

The holly grail would be a combination of preempt+lock-break plus lowlatency 
and Ingo's O(1) scheduler.

My main focus lies on 3D graphics not kernel and I use KDE (yes, a little 
luxury:-) 'cause KDE is C++ and most visualization systems are c and later 
c++.

Without the above patches even my 1 GHz Athlon II, 640 MB, feels sluggish.
But I don't forget to think about throughput which is even usefull for 
"heavy" compiler runs...

> indeed users have a speed
> feeling. Please consider that on eavilly loaded servers, with 40 and more
> users, some are running gcc, others g77, others g++ compilations, someone
> runs pine or mutt or kmail, and netscape, and mozilla, and emacs (someone
> form xterm kde or gnome), and and
> and... You can have also 4/8 CPU butthey are not infinite ;) (but I talk
> mainly thinking of dualAthlon systems).
> there is a lot of memory and disk I/O.
> This is not a strange scenary on the interactive servers used at SNS.
> Here preempt has a too high price

That's why preempt is a compile time option, btw.

> > By the way, have you measured the cost of -preempt in practice?
>
> Yes, I did a lot of tests, and with current preempt patch definitelly
> I was seeing a too big performance loss.

Have you tried with stock 2.4.17 or with additional patches?
2.4.17-rc2aa2 (10_vm-21)?

The later make big differences in throughput for me (with and without 
preempt).

I am under preparation of some numbers.
Anybody want some special tests?
dbench (yes, I know...) with and without MP3 during run
latencytest0.42-png
bonnie++
getc_putc

Thank you for all your serious answers. This was definitely not intended as a 
flamewar start.

-Dieter
-- 
Dieter Nützel
Graduate Student, Computer Science

University of Hamburg
Department of Computer Science
@home: Dieter.Nuetzel@hamburg.de

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 9, 2002 12:02 am, Luigi Genoni wrote:
> On Tue, 8 Jan 2002, Daniel Phillips wrote:
> > On January 8, 2002 04:29 pm, Andrea Arcangeli wrote:
> > > but I just wanted to make clear that the
> > > idea that is floating around that preemptive kernel is all goodness is
> > > very far from reality, you get very low mean latency but at a price.
> >
> > A price lots of people are willing to pay
>
> Probably sometimes they are not making a good business.

Perhaps.  But they are happy customers and their music sounds better.

Note: the dominating cost of -preempt is not Robert's patch, but the fact 
that you need to have CONFIG_SMP enabled, even for uniprocessor, turning all 
those stub macros into real spinlocks.  For a dual processor you have to have 
this anyway and it just isn't an issue.

Personally, I don't intend to ever get another single-processor machine, 
except maybe a laptop, and that's only if Transmeta doesn't come up with a 
dual-processor laptop configuration.

> > By the way, have you measured the cost of -preempt in practice?
>
> Yes, I did a lot of tests, and with current preempt patch definitelly
> I was seeing a too big performance loss.

Was this on uniprocessor machines, or your dual Athlons?  How did you measure 
the performance?

--
Daniel

Preemtive kernel (Was: Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Roger Larsson]

(the subject has been wrong for some time now...)

On Wednesday den 9 January 2002 07.26, Daniel Phillips wrote:
> On January 9, 2002 12:02 am, Luigi Genoni wrote:
> > On Tue, 8 Jan 2002, Daniel Phillips wrote:
> > > On January 8, 2002 04:29 pm, Andrea Arcangeli wrote:
> > > > but I just wanted to make clear that the
> > > > idea that is floating around that preemptive kernel is all goodness
> > > > is very far from reality, you get very low mean latency but at a
> > > > price.
> > >
> > > A price lots of people are willing to pay
> >
> > Probably sometimes they are not making a good business.
>
> Perhaps.  But they are happy customers and their music sounds better.
>
> Note: the dominating cost of -preempt is not Robert's patch, but the fact
> that you need to have CONFIG_SMP enabled, even for uniprocessor, turning
> all those stub macros into real spinlocks.  For a dual processor you have
> to have this anyway and it just isn't an issue.
>

Well you don't - the first versions used the SMP spinlocks macros but
replaced them with own code. (basically an INC on entry and a DEC and test
when leaving)

Think about what happens on a UP
There are two cases
 - the processor is in the critical section, it can not be preempted = no
   other process can take the CPU away from it.
 - the processor is not in a critical section, no process can be executing
   inside it = can never be busy.
=> no real spinlocks needed on a UP

/RogerL

-- 
Roger Larsson
Skellefteå
Sweden

Re: Preemtive kernel (Was: Re: [2.4.17/18pre] VM and swap - it's really unusable)

[Daniel Phillips]

On January 9, 2002 08:25 am, Roger Larsson wrote:
> (the subject has been wrong for some time now...)
> 
> On Wednesday den 9 January 2002 07.26, Daniel Phillips wrote:
> > On January 9, 2002 12:02 am, Luigi Genoni wrote:
> > > On Tue, 8 Jan 2002, Daniel Phillips wrote:
> > > > On January 8, 2002 04:29 pm, Andrea Arcangeli wrote:
> > > > > but I just wanted to make clear that the
> > > > > idea that is floating around that preemptive kernel is all goodness
> > > > > is very far from reality, you get very low mean latency but at a
> > > > > price.
> > > >
> > > > A price lots of people are willing to pay
> > >
> > > Probably sometimes they are not making a good business.
> >
> > Perhaps.  But they are happy customers and their music sounds better.
> >
> > Note: the dominating cost of -preempt is not Robert's patch, but the fact
> > that you need to have CONFIG_SMP enabled, even for uniprocessor, turning
> > all those stub macros into real spinlocks.  For a dual processor you have
> > to have this anyway and it just isn't an issue.
> 
> Well you don't - the first versions used the SMP spinlocks macros but
> replaced them with own code. (basically an INC on entry and a DEC and test
> when leaving)
> 
> Think about what happens on a UP
> There are two cases
>  - the processor is in the critical section, it can not be preempted = no
>    other process can take the CPU away from it.
>  - the processor is not in a critical section, no process can be executing
>    inside it = can never be busy.
> => no real spinlocks needed on a UP

Right, thanks, it was immediately obvious when you pointed out that the 
macros are just used to find the bounds of the critical regions.  So the cost 
of -preempt is somewhat less than I had imagined.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 10:29, Andrea Arcangeli wrote:

> "extra schedule points all over the place", that's the -preempt kernel
> not the lowlatency kernel! (on yeah, you don't see them in the source
> but ask your CPU if it sees them)

How so?  The branch on drop of the last lock?  It's not a factor in
profiles I've seen.  And it is marked unlikely.  The other change is the
usual check for reschedule on return from interrupt, but that is the
case already, we just allow it when in-kernel, too.

This makes me think the end conclusion would be that preemptive
multitasking in general is bad.  Why don't we increase the timeslice and
and tick period, in that case?

One can argue the complexity degrades performance, but tests show
otherwise.  In throughput and latency.  Besides, like I always say, its
an option that uses existing kernel (SMP lock) infrastructure.  You
don't have to use it ;)

> > The preemptible approach is much less of a maintainance headache, since 
> > people don't have to be constantly doing audits to see if something changed, 
> > and going in to fiddle with scheduling points.
> 
> this yes, it requires less maintainance, but still you should keep in
> mind the details about the spinlocks, things like the checks the VM does
> in shrink_cache are needed also with preemptive kernel.

They impact SMP in the same way they impact preempt-kernel.  Long-held
locks are never good.  Weird locking rules are never good.

> The I/O pipeline is big enough that a few msec before or later in a
> submit_bh shouldn't make a difference, the batch logic in the
> ll_rw_block layer also try to reduce the reschedule, and last but not
> the least if the task is I/O bound preemptive kernel or not won't make
> any difference in the submit_bh latency because no task is eating cpu
> and latency will be the one of pure schedule call.

Yet throughput tests show marked increase.  I believe this is true with
Andrew's patch, too.  We multitask better.  We dispatch waiting tasks
faster.  Without the patch, a queued I/O task may stall for sometime
waiting for some hog to get out of the kernel.

Although the patch's goal is to improve interactivity (*), no one says
the higher priority task we preempt in favor of has to be CPU-bound.  A
woken up I/O-bound task is just as benefiting from the patch.

(*) this is, IMO, where we benefit most though.  By far the most
pleasing benchmark isn't to see some x% decrease in latency or y more
MB/s in bonnie, but to feel the improvement in interactivity.  On a
multitasking desktop, it is noticeable.

> I also don't want to devaluate the preemptive kernel approch (the mean
> latency it can reach is lower than the one of the lowlat kernel, however
> I personally care only about worst case latency and this is why I don't
> feel the need of -preempt), but I just wanted to make clear that the
> idea that is floating around that preemptive kernel is all goodness is
> very far from reality, you get very low mean latency but at a price.

Andrea, I don't want you or anyone to believe preemption is a free
ride.  On the other hand, the patch has a _huge_ userbase and you can't
question that.  You also can't question the benchmarks that show
improvements in average _and_ worst case latency _and_ throughput.

I don't expect you to use the patch.  If it were merged, it is an
option.  It provides a framework for continuing to improve latency.  It
is a solution to the problem (i.e. latency is poor because the kernel is
non-preemptible) instead of a hack.  I agree worst-case latency is
important, and I agree the patch shines more so in average case.  But we
do affect worse-case.  And now a framework exists for working on fixing
the worst-case latencies too.  And in the end, its just an option for
some, but a better kernel for others.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Tue, Jan 08, 2002 at 03:55:38PM -0500, Robert Love wrote:
> On Tue, 2002-01-08 at 10:29, Andrea Arcangeli wrote:
> 
> > "extra schedule points all over the place", that's the -preempt kernel
> > not the lowlatency kernel! (on yeah, you don't see them in the source
> > but ask your CPU if it sees them)
> 
> How so?  The branch on drop of the last lock?  It's not a factor in

exactly, this is the reschedule point I meant. Oh note that it's
unlikely also in the lowlatecy patch. Please count the number of time
you add this branch in the -preempt, and how many times we add this
branch in the lowlat and then tell me who is adding rescheduling points
in the kernel all over the place.

> This makes me think the end conclusion would be that preemptive
> multitasking in general is bad.  Why don't we increase the timeslice and
> and tick period, in that case?

that would increase performance, but we'd lost interactivity.

> One can argue the complexity degrades performance, but tests show
> otherwise.  In throughput and latency.  Besides, like I always say, its

which benchmarks? you should make sure the CPU spend all its cycles in
the kernel to benchmark the perfrormance degradation (this is the normal
case of webserving with a few gigabit ethernet cards using sendfile).

> ride.  On the other hand, the patch has a _huge_ userbase and you can't

I question this because it is too risky to apply. There is no way any
distribution or production system could ever consider applying the
preempt kernel and ship it in its next kernel update 2.4. You never know
if a driver will deadlock because it is doing a test and set bit busy
loop by hand instead of using spin_lock and you cannot audit all the
device drivers out there. It is not like the VM that is self contained
and that can be replaced without any caller noticing, this instead
impacts every single driver out there and you'd need to audit all of
them, which is not feasible I think and that should be done by giving
everybody the time to test. This is also what makes preempt config
option risky, if we go preempt we should force everybody to use it, at
least during 2.5, so we get the useful feedback from testers of all the
hardware, or nobody could trust -preempt.

NOTE: I trust your work with spinlocks, locks around per-cpu data
structures etc.. is perfect, I trust that part, as said it's the driver
doing test and set bit that you cannot audit that is the problem here
and that makes it potentially unstable, not your changes.  And also the
per-cpu data structures sounds a little risky (but for example for UP
that's not an issue).

> question that.  You also can't question the benchmarks that show
> improvements in average _and_ worst case latency _and_ throughput.

I don't question some benchmark is faster with -preempt, the interesting
thing is to find why because it shouldn't be the case, Andrew for
example mentioned software raid, there are good reasons for which
-preempt could be faster there, so we added a single sechdule point and
we just have that case covered in 18pre2aa1, we don't need reschedule
points all over the place like in -preempt to cover things like that.
It is good to find them out so we can fix those bugs, I consider them
bugs :).

Again: I'm not completly against preempt, it can reach an mean latency
much lower than mainline (it can reschedule immediatly in the middle of
long copy-users for example), so it definitely has a value, it's just
that I'm not sure if it worth it.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Ed Sweetman]

----- Original Message -----
From: "Andrea Arcangeli" <andrea@suse.de>
To: "Robert Love" <rml@tech9.net>
Cc: "Daniel Phillips" <phillips@bonn-fries.net>; "Anton Blanchard"
<anton@samba.org>; "Luigi Genoni" <kernel@Expansa.sns.it>; "Dieter N?tzel"
<Dieter.Nuetzel@hamburg.de>; "Marcelo Tosatti" <marcelo@conectiva.com.br>;
"Rik van Riel" <riel@conectiva.com.br>; "Linux Kernel List"
<linux-kernel@vger.kernel.org>; "Andrew Morton" <akpm@zip.com.au>
Sent: Wednesday, January 09, 2002 6:24 AM
Subject: Re: [2.4.17/18pre] VM and swap - it's really unusable


> On Tue, Jan 08, 2002 at 03:55:38PM -0500, Robert Love wrote:
> > On Tue, 2002-01-08 at 10:29, Andrea Arcangeli wrote:
> >
> > > "extra schedule points all over the place", that's the -preempt kernel
> > > not the lowlatency kernel! (on yeah, you don't see them in the source
> > > but ask your CPU if it sees them)
> >
> > How so?  The branch on drop of the last lock?  It's not a factor in
>
> exactly, this is the reschedule point I meant. Oh note that it's
> unlikely also in the lowlatecy patch. Please count the number of time
> you add this branch in the -preempt, and how many times we add this
> branch in the lowlat and then tell me who is adding rescheduling points
> in the kernel all over the place.
>
> > This makes me think the end conclusion would be that preemptive
> > multitasking in general is bad.  Why don't we increase the timeslice and
> > and tick period, in that case?
>
> that would increase performance, but we'd lost interactivity.
>
> > One can argue the complexity degrades performance, but tests show
> > otherwise.  In throughput and latency.  Besides, like I always say, its
>
> which benchmarks? you should make sure the CPU spend all its cycles in
> the kernel to benchmark the perfrormance degradation (this is the normal
> case of webserving with a few gigabit ethernet cards using sendfile).
 I haven't seen any interactive tests that showed worse results than the
vanilla kernel with the preempt patch.  The only cases where it gives a
worse performance is in a single tasking environment such as running bonnie
or dbench and apps like that that require to throttle the system.  This is
obviously expected behavior though.  Performance degradation might be seen
on a per app basis, but when looking at the system as a whole, performance
has never degraded with the patch as far as i've seen.  Better overall
performance is what has lead to better "benchmark" performance on the tests
being run by people.


> > ride.  On the other hand, the patch has a _huge_ userbase and you can't
>
> I question this because it is too risky to apply. There is no way any
> distribution or production system could ever consider applying the
> preempt kernel and ship it in its next kernel update 2.4. You never know
> if a driver will deadlock because it is doing a test and set bit busy
> loop by hand instead of using spin_lock and you cannot audit all the
> device drivers out there. It is not like the VM that is self contained
> and that can be replaced without any caller noticing, this instead
> impacts every single driver out there and you'd need to audit all of
> them, which is not feasible I think and that should be done by giving
> everybody the time to test. This is also what makes preempt config
> option risky, if we go preempt we should force everybody to use it, at
> least during 2.5, so we get the useful feedback from testers of all the
> hardware, or nobody could trust -preempt.
 I disagree.   Redhat shipped gcc 2.96 when it was producing incompatible
binaries and was buggy as all hell, why not ship a kernel that is "unstable"
and "risky" if it promises better performance.
If scheduling points are ugly in 2.4, then they'd be ugly in 2.5.  The only
solution to the problem you see with it is making 2.5 fully preemptible from
the ground up instead of having to add fixes.  If nobody wants to do things
the hard way (assuming there is a better way), is it better to leave it
unfixed rather than fix it?  Of course i'm assuming that the idea of a fully
preemptible kernel is better than the current version we have now.


> NOTE: I trust your work with spinlocks, locks around per-cpu data
> structures etc.. is perfect, I trust that part, as said it's the driver
> doing test and set bit that you cannot audit that is the problem here
> and that makes it potentially unstable, not your changes.  And also the
> per-cpu data structures sounds a little risky (but for example for UP
> that's not an issue).
>
> > question that.  You also can't question the benchmarks that show
> > improvements in average _and_ worst case latency _and_ throughput.
>
> I don't question some benchmark is faster with -preempt, the interesting
> thing is to find why because it shouldn't be the case, Andrew for
> example mentioned software raid, there are good reasons for which
> -preempt could be faster there, so we added a single sechdule point and
> we just have that case covered in 18pre2aa1, we don't need reschedule
> points all over the place like in -preempt to cover things like that.
> It is good to find them out so we can fix those bugs, I consider them
> bugs :).

  I think robert love is trying to give the kernel the highest flexibility.
Making it flexible in key areas will improve your worst cases but a lot of
the time during normal use it's the multitude of smaller cases that is
noticeable.

> Again: I'm not completly against preempt, it can reach an mean latency
> much lower than mainline (it can reschedule immediatly in the middle of
> long copy-users for example), so it definitely has a value, it's just
> that I'm not sure if it worth it.
>
> Andrea


Ok so the medicine is worse than the disease.   I take it that you only want
some key points made for rescheduling instead of the full preempt patch by
Robert.   That seems logical enough.   The only issue i see is that for the
most part people dont like the idea of needing to add scheduling points.  So
how would the kernel need to be fixed in order to not need them and still be
fully preemptible like it's getting in Robert's patch.  If it just cant then
is it really best to hang out somewhere on the edge of preemptible
multitasking because some people are in denial that the kernel needs to be
patched so much to work correctly and for the sake of single tasking
performance?

Now in just my own opinion i think of linux as a multitasking kernel and as
thus it should perform that function as best as possible.  If you want to
run a single program as fast as possible then absolutely dont run anything
else and nothing can preempt it to degrade it's performance.   The fact that
you can run multiple apps and run a single program as fast as possible
without degrading it's performance  is a bug if those other apps (at the
same priority) have to wait longer than they should if we want linux to be a
multitasking kernel.  Just to sum things up, if there is a way to be fully
preemptible without scheduling points linux, then perhaps that should be a
major focus for 2.5 instead of picking and choosing (ugly)scheduling points,
but if not then the argument about them not being elegant is mute because
then the kernel, itself, is far from elegant already, so what exactly are
you saving?

- Formerly safemode

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Wed, Jan 09, 2002 at 09:07:55AM -0500, Ed Sweetman wrote:
> Ok so the medicine is worse than the disease.   I take it that you only want
> some key points made for rescheduling instead of the full preempt patch by
> Robert.   That seems logical enough.   The only issue i see is that for the

My ideal is to have the kernel to be as low worst latency as -preempt,
but without being preemptive. that's possible to achieve, I don't think
we're that far.

mean latency is another matter, but I personally don't mind about mean
latency and I much prefer to save cpu cycles instead.

Andrea

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Arjan van de Ven]

Andrea Arcangeli wrote:
> 
> On Wed, Jan 09, 2002 at 09:07:55AM -0500, Ed Sweetman wrote:
> > Ok so the medicine is worse than the disease.   I take it that you only want
> > some key points made for rescheduling instead of the full preempt patch by
> > Robert.   That seems logical enough.   The only issue i see is that for the
> 
> My ideal is to have the kernel to be as low worst latency as -preempt,
> but without being preemptive. that's possible to achieve, I don't think
> we're that far.
> 
> mean latency is another matter, but I personally don't mind about mean
> latency and I much prefer to save cpu cycles instead.

hear hear!

The akpm patch is achieving a MUCH better latency than pure -preempt,
and only has 40 
or so coded preemption points instead of a few hundred (eg every
spin_unlock).... 

and if with 40 we can get <= 1ms then everybody will be happy; if you
want, say, 50 usec
latency instead you need RTLinux anyway. With 1ms _worst case_ latency
the "mean" latency 
is obviously also very good.......

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roger Larsson]

On Wednesday den 9 January 2002 15.51, Arjan van de Ven wrote:
> Andrea Arcangeli wrote:
> > On Wed, Jan 09, 2002 at 09:07:55AM -0500, Ed Sweetman wrote:
> > > Ok so the medicine is worse than the disease.   I take it that you only
> > > want some key points made for rescheduling instead of the full preempt
> > > patch by Robert.   That seems logical enough.   The only issue i see is
> > > that for the
> >
> > My ideal is to have the kernel to be as low worst latency as -preempt,
> > but without being preemptive. that's possible to achieve, I don't think
> > we're that far.
> >
> > mean latency is another matter, but I personally don't mind about mean
> > latency and I much prefer to save cpu cycles instead.
>
> hear hear!
>
> The akpm patch is achieving a MUCH better latency than pure -preempt,
> and only has 40
> or so coded preemption points instead of a few hundred (eg every
> spin_unlock)....

The difference is that the preemptive kernel mostly uses existing 
infrastructure. When SMP scalability gets better due to holding locks
for a shorter time then the preemptive kernel will improve as well!

AND it can be used on a UP computer to "simulate" SMP and that
should help the quality of the total code base...

This is my idea:
* Add the preemptive kernel
* "Remove" reschedule points from main kernel.
   note: that reschedule points that does nothing more than
   test and schedule can be NOOPed since they will never trigger in a
   preemptive kernel...

>
> and if with 40 we can get <= 1ms then everybody will be happy; if you
> want, say, 50 usec
> latency instead you need RTLinux anyway. With 1ms _worst case_ latency
> the "mean" latency
> is obviously also very good.......

Worst case latency... is VERY hard to prove if you rely on schedule points.
Since they are typically added after the fact...
If the code suddenly end up on a road less travelled...

With preemptive kernel your worst latency is the longest held spinlock. 
PERIOD.
(you can of cause be delayed by an even higher priority process)
* Make sure that there are no "infinite" loops inside any spinlock.

"infinite" == over ALL or ALL/x of something since someone, somewere
 will have ALL close to infinite... (infinity/x is still infinity... :-)
   example code is looping through LRU list to find a victim page...
   once it was not infinite due to the small number of pages...

Note: that akpm patches usually hava a - "do not do this list" with known
problem spots (ok, usually in a hard to break spinlocks).

/RogerL

-- 
Roger Larsson
Skellefteå
Sweden

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Arjan van de Ven]

On Wed, Jan 09, 2002 at 06:02:53PM +0100, Roger Larsson wrote:
> The difference is that the preemptive kernel mostly uses existing 
> infrastructure. When SMP scalability gets better due to holding locks
> for a shorter time then the preemptive kernel will improve as well!

Ehm. Holding locks for a shorter time is not guaranteed to improve smp
scalability. In fact it can completely kill it due to cacheline pingpong
effects.

> > and if with 40 we can get <= 1ms then everybody will be happy; if you
> > want, say, 50 usec
> > latency instead you need RTLinux anyway. With 1ms _worst case_ latency
> > the "mean" latency
> > is obviously also very good.......
> 
> Worst case latency... is VERY hard to prove if you rely on schedule points.

Agreed. It's "worst case" in the soft real time sence. But we've beaten the
kernel quite hard during such tests....

> With preemptive kernel your worst latency is the longest held spinlock. 
> PERIOD.

Yes and without the same stuff akpm does that's about 80 to 90 ms right now. 

> Note: that akpm patches usually hava a - "do not do this list" with known
> problem spots (ok, usually in a hard to break spinlocks).

Usually in hardware related parts. Even with -preempt you'll get this.
Hopefully only during hardware initialisation, but there are just cases
where you need to go WAAAY too far if you want to go below, say, 5ms during
device init.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 9, 2002 03:51 pm, Arjan van de Ven wrote:
> Andrea Arcangeli wrote:
> > 
> > On Wed, Jan 09, 2002 at 09:07:55AM -0500, Ed Sweetman wrote:
> > > Ok so the medicine is worse than the disease.   I take it that you only want
> > > some key points made for rescheduling instead of the full preempt patch by
> > > Robert.   That seems logical enough.   The only issue i see is that for the
> > 
> > My ideal is to have the kernel to be as low worst latency as -preempt,
> > but without being preemptive. that's possible to achieve, I don't think
> > we're that far.
> > 
> > mean latency is another matter, but I personally don't mind about mean
> > latency and I much prefer to save cpu cycles instead.
> 
> hear hear!

> The akpm patch is achieving a MUCH better latency than pure -preempt,

Can you please point us at the benchmark results that support your claim?

> and only has 40 
> or so coded preemption points instead of a few hundred (eg every
> spin_unlock).... 

So?  The cost of this is, in theory, a dec and a branch normally not taken.
Robert hasn't coded it that way in the current incarnation, and personally,
I'd rather see the correctness proven before the microoptimizations are
done, but that's where it's going in theory.  Big deal.

On the other hand, I just did a test for myself that pretty well makes up
my mind about this patch.  I'm typing this right now on a 64 Meg laptop with
a slow disk, dma turned off.  On this machine, debian apt-get dist-upgrade
is essentially a DoS - once it gets to unpacking packages and configuring,
for whatever reason, the machine becomes almost ununsable.  Changing windows
for example, can take 10-15 seconds.  Updatedb, while not quite as bad, is
definitely an irritant as far as interactive use goes.

With Robert's patch, the machine is a little sluggish during apt-get, but
quite usable.  This is a *huge* difference.  And during updatedb, well, I
hardly notice it's happening, except for the disk light.

So I like this patch.  What was your complaint again?  If you've got hard
numbers and repeatable benchmarks, please trot them out.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Daniel Phillips wrote:
> 
> On January 8, 2002 02:33 pm, Anton Blanchard wrote:
> > Andrea Arcangeli [apparently] wrote:
> > > So yes, mean latency will decrease with preemptive kernel, but your CPU
> > > is definitely paying something for it.
> >
> > And Andrew Morton's work suggests he can do a much better job of
> > reducing latency than -preempt.
> 
> That's not a particularly clueful comment, Anton.  Obviously, any
> latency-busting hacks that Andrew does could also be patched into a
> -preempt kernel.

Yes.  The important part is the implicit dropping of the BKL across
schedule().

> What a preemptible kernel can do that a non-preemptible kernel can't is:
> reschedule exactly as often as necessary, instead of having lots of extra
> schedule points inserted all over the place, firing when *they* think the
> time is right, which may well be earlier than necessary.

Nope.  `if (current->need_resched)' -> the time is right (beyond right,
actually).

> The preemptible approach is much less of a maintainance headache, since
> people don't have to be constantly doing audits to see if something changed,
> and going in to fiddle with scheduling points.

Except it doesn't work.  The full-on low-latency patch has ~60 rescheduling
points.  Of these, ~40 involve popping spinlocks.  Really, the only significant
latency sources which the preemptible kernel solves are generic_file_read()
and generic_file_write().

So preemptible kernel needs lock-break to be useful.  And then it's basically
the same thing, with the same maintainability problems.  And believe me, these
are considerable.  Mainly because the areas which needs busting up exactly
coincide with the areas where there has been most churn in the kernel.

> Finally, with preemption, rescheduling can be forced with essentially zero
> latency in response to an arbitrary interrupt such as IO completion, whereas
> the non-preemptive kernel will have to 'coast to a stop'.  In other words,
> the non-preemptive kernel will have little lags between successive IOs,
> whereas the preemptive kernel can submit the next IO immediately.  So there
> are bound to be loads where the preemptive kernel turns in better latency
> *and throughput* than the scheduling point hack.

Latency yes.  Throughout no.

I don't think the "preempt slows down the kernel" argument is very valid
really.  Let's invert the argument - Linux is multitasking, and that has a
cost.  There's no reason why certain bits of the kernel need to violate that
just to get a bit more throughput.  If it really worries you, set HZ=10 and
increase all the timeslices, etc.

Now, there *may* be overheads added due to losing the implicit locking which
per-CPU data gives you.

The main cost of preempt IMO is in complexity and stability risks.

(BTW: I took a weird oops testing the preempt patch on an SMP NFS client.
The fault address was 0x0aXXXXXX.  No useful backtrace, unfortunately).

> Mind you, I'm not devaluing Andrew's work, it's good and valuable.  However
> it's good to be aware of why that approach can't equal the latency-busting
> performance of the preemptive approach.

There's no point in just merging the preempt patch and saying "there,
that's done".  It doesn't do anything.

Instead, a decision needs to be made: "Linux will henceforth be a 
low-latency kernel".  Now, IF we can come to this decision, then
internal preemption is the way to do it.  But it affects ALL kernel
developers.  Because we'll need to introduce a new rule: "it is a
bug to spend more than five milliseconds holding any locks".

So.  Do we we want a low-latency kernel?  Are we prepared to mandate
the five-millisecond rule?   It can be done, but won't be easy, and
we'll never get complete coverage.  But I don't see the will around
here.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> low-latency kernel".  Now, IF we can come to this decision, then
> internal preemption is the way to do it.  But it affects ALL kernel

The pre-empt patches just make things much much harder to debug. They
remove some of the predictability and the normal call chain following
goes out of the window because you end up seeing crashes in a thread with
no idea what ran the microsecond before

Some of that happens now but this makes it vastly worse.

The low latency patches don't change the basic predictability and
debuggability but allow you to hit a 1mS pre-empt target for the general
case.

Alan

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Roger Larsson]

On Tuesdayen den 8 January 2002 21.13, Alan Cox wrote:
> > low-latency kernel".  Now, IF we can come to this decision, then
> > internal preemption is the way to do it.  But it affects ALL kernel
>
> The pre-empt patches just make things much much harder to debug. They
> remove some of the predictability and the normal call chain following
> goes out of the window because you end up seeing crashes in a thread with
> no idea what ran the microsecond before
>
> Some of that happens now but this makes it vastly worse.
>
> The low latency patches don't change the basic predictability and
> debuggability but allow you to hit a 1mS pre-empt target for the general
> case.
>

Yes, it does make things much much harder to debug - but:
* If you get a problem on a preemtive UP kernel, it is likely to be a problem
  on a SMP too - and those are hard to debug aswell. But the positive aspect
  is that you get more people that can do the debugging... :-)
  (One CPU gets delayed with handling a IRQ the other runs into the critical
   section)
* It is optional at compile time.
   And could even be made run time optional / CPU ! Just set a too big value
   on the counter and it will never reschedule...

/RogerL

-- 
Roger Larsson
Skellefteå
Sweden

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 8, 2002 08:47 pm, Andrew Morton wrote:
> There's no point in just merging the preempt patch and saying "there,
> that's done".  It doesn't do anything.
> 
> Instead, a decision needs to be made: "Linux will henceforth be a 
> low-latency kernel".

I thought the intention was to make it a config option?

> Now, IF we can come to this decision, then
> internal preemption is the way to do it.  But it affects ALL kernel
> developers.  Because we'll need to introduce a new rule: "it is a
> bug to spend more than five milliseconds holding any locks".
> 
> So.  Do we we want a low-latency kernel?  Are we prepared to mandate
> the five-millisecond rule?   It can be done, but won't be easy, and
> we'll never get complete coverage.  But I don't see the will around
> here.

At least the flaming has gotten a little less ;-)

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 15:18, Daniel Phillips wrote:

> > Instead, a decision needs to be made: "Linux will henceforth be a 
> > low-latency kernel".
> 
> I thought the intention was to make it a config option?

It was originally, it is now, and I intend it to be.

Further, since it uses the existing SMP locks, it doesn't introduce new
design decisions (the one being protection of implicitly locked per-CPU
data on preempt).

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Tue, 8 Jan 2002, Daniel Phillips wrote:

> On January 8, 2002 08:47 pm, Andrew Morton wrote:
> > There's no point in just merging the preempt patch and saying "there,
> > that's done".  It doesn't do anything.
> > 
> > Instead, a decision needs to be made: "Linux will henceforth be a 
> > low-latency kernel".
> 
> I thought the intention was to make it a config option?

Irrelevant, it has to be implemented in order to be an option, so the
amount of work involved is the same either way. And if you want to make it
a runtime setting you add a slight bit of work and overhead deciding if LL
is wanted.

I'm not advocating that, but it would allow admins to enable LL when the
system was slow and see if it really made a change. Rebooting is bound to
change the load ;-)
 
> > Now, IF we can come to this decision, then
> > internal preemption is the way to do it.  But it affects ALL kernel
> > developers.  Because we'll need to introduce a new rule: "it is a
> > bug to spend more than five milliseconds holding any locks".
> > 
> > So.  Do we we want a low-latency kernel?  Are we prepared to mandate
> > the five-millisecond rule?   It can be done, but won't be easy, and
> > we'll never get complete coverage.  But I don't see the will around
> > here.

Really? You have people working on low latency, people working on preempt,
and at least a few of us trying to characterize the problems with large
memory and i/o. I would say latency has become a real issue, and you only
need enough "will" to have one person write useful code, this is a
committee.

Since changes of this type don't need to be perfect and address all cases,
just help some and not make other worse, I think we will see improvement
in 2.4.xx without waiting for 2.5 or 2.6. No one is complaining that the
Linux overall thruput is bad, that network performance is bad, etc. But
responsiveness has become an issue, and I'm sure there's enough will to
solve it. "Solve" means getting most of the delays to be caused by
hardware capacity instead of kernel ineptitude.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 8, 2002 08:47 pm, Andrew Morton wrote:
> Daniel Phillips wrote:
> > What a preemptible kernel can do that a non-preemptible kernel can't is:
> > reschedule exactly as often as necessary, instead of having lots of extra
> > schedule points inserted all over the place, firing when *they* think the
> > time is right, which may well be earlier than necessary.
> 
> Nope.  `if (current->need_resched)' -> the time is right (beyond right,
> actually).

Oops, sorry, right.

The preemptible kernel can reschedule, on average, sooner than the 
scheduling-point kernel, which has to wait for a scheduling point to roll 
around.

And while I'm enumerating differences, the preemptable kernel (in this 
incarnation) has a slight per-spinlock cost, while the non-preemptable kernel 
has the fixed cost of checking for rescheduling, at intervals throughout all 
'interesting' kernel code, essentially all long-running loops.  But by clever 
coding it's possible to finesse away almost all the overhead of those loop 
checks, so in the end, the non-preemptible low-latency patch has a slight 
efficiency advantage here, with emphasis on 'slight'.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rik van Riel]

On Tue, 8 Jan 2002, Daniel Phillips wrote:

> The preemptible kernel can reschedule, on average, sooner than the
> scheduling-point kernel, which has to wait for a scheduling point to
> roll around.

The preemptible kernel ALSO has to wait for a scheduling point
to roll around, since it cannot preempt with spinlocks held.

Considering this, I don't see much of an advantage to adding
kernel preemption.

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 16:08, Rik van Riel wrote:

> The preemptible kernel ALSO has to wait for a scheduling point
> to roll around, since it cannot preempt with spinlocks held.
> 
> Considering this, I don't see much of an advantage to adding
> kernel preemption.

It only has to wait if locks are held and then only until the locks are
dropped.  Otherwise it will preempt on the next return from interrupt.

Future work would be to look into long-held locks and see what we can
do.

Without preempt-kernel, we have none of this: either run until
completion or explicit scheduling points. 

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Rik van Riel]

On 8 Jan 2002, Robert Love wrote:
> On Tue, 2002-01-08 at 16:08, Rik van Riel wrote:
>
> > The preemptible kernel ALSO has to wait for a scheduling point
> > to roll around, since it cannot preempt with spinlocks held.
> >
> > Considering this, I don't see much of an advantage to adding
> > kernel preemption.
>
> It only has to wait if locks are held and then only until the locks are
> dropped.  Otherwise it will preempt on the next return from interrupt.

So what exactly _is_ the difference between an explicit
preemption point and a place where we need to explicitly
drop a spinlock ?

>From what I can see, there really isn't a difference.

> Future work would be to look into long-held locks and see what we can
> do.

One thing we could do is download Andrew Morton's patch ;)

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 16:24, Rik van Riel wrote:

> So what exactly _is_ the difference between an explicit
> preemption point and a place where we need to explicitly
> drop a spinlock ?

In that case nothing, except that when we drop the lock and check it is
the earliest place where preemption is allowed.  In the normal scenario,
however, we have a check for reschedule on return from interrupt (e.g.
the timer) and thus preempt in the same manner as with user space and
that is the key.

> > Future work would be to look into long-held locks and see what we can
> > do.
> 
> One thing we could do is download Andrew Morton's patch ;)

That is certainly one option, and Andrew's patch is very good. 
Nonetheless, I think we need a more general framework that tackles the
problem itself.  Preemptible kernel does this, yields results now, and
allows for greater return later on.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Robert Love wrote:
> 
> On Tue, 2002-01-08 at 16:24, Rik van Riel wrote:
> 
> > So what exactly _is_ the difference between an explicit
> > preemption point and a place where we need to explicitly
> > drop a spinlock ?
> 
> In that case nothing, except that when we drop the lock and check it is
> the earliest place where preemption is allowed.  In the normal scenario,
> however, we have a check for reschedule on return from interrupt (e.g.
> the timer) and thus preempt in the same manner as with user space and
> that is the key.

One could do:

static inline void spin_unlock(spinlock_t *lock)
{
        __asm__ __volatile__(
                spin_unlock_string
        );

	if (--current->lock_depth == 0 &&
		current->need_resched &&
		current->state == TASK_RUNNING)
		schedule();
}

But I have generally avoided "global" solutions like this, in favour
of nailing the _specific_ code which is causing the problem.  Which
is a lot more work, but more useful.

The scheduling points in bread() and submit_bh() in the mini-ll patch
go against this (masochistic) philosophy.

> > > Future work would be to look into long-held locks and see what we can
> > > do.
> >
> > One thing we could do is download Andrew Morton's patch ;)
> 
> That is certainly one option, and Andrew's patch is very good.
> Nonetheless, I think we need a more general framework that tackles the
> problem itself.  Preemptible kernel does this, yields results now, and
> allows for greater return later on.

We need something which makes 2.4.x not suck.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 8, 2002 10:08 pm, Rik van Riel wrote:
> On Tue, 8 Jan 2002, Daniel Phillips wrote:
> 
> > The preemptible kernel can reschedule, on average, sooner than the
> > scheduling-point kernel, which has to wait for a scheduling point to
> > roll around.
> 
> The preemptible kernel ALSO has to wait for a scheduling point
> to roll around, since it cannot preempt with spinlocks held.

Think about the relative amount of time spent inside spinlocks vs regular 
kernel.

> Considering this, I don't see much of an advantage to adding
> kernel preemption.

And now?

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Daniel Phillips wrote:
> 
> On January 8, 2002 08:47 pm, Andrew Morton wrote:
> > Daniel Phillips wrote:
> > > What a preemptible kernel can do that a non-preemptible kernel can't is:
> > > reschedule exactly as often as necessary, instead of having lots of extra
> > > schedule points inserted all over the place, firing when *they* think the
> > > time is right, which may well be earlier than necessary.
> >
> > Nope.  `if (current->need_resched)' -> the time is right (beyond right,
> > actually).
> 
> Oops, sorry, right.
> 
> The preemptible kernel can reschedule, on average, sooner than the
> scheduling-point kernel, which has to wait for a scheduling point to roll
> around.

That's theory.  Practice (ie: instrumentation) says that the preempt
patch makes little improvement over conditional yields in generic_file_read()
and generic_file_write().  Four lines.  Additional yields in wait_for_buffers()
(where we move zillions of buffers from BUF_LOCKED to BUF_CLEAN) and in submit_bh()
and bread() are cream.

Preemptability is global in its impact, and in its effect.  It requires
global changes to make it useful.  If we're prepared to make those
changes (scan_swap_map, truncate_inode_pages, etc) then fine.  Go for
it.  We'll end up with a better kernel.

> And while I'm enumerating differences, the preemptable kernel (in this
> incarnation) has a slight per-spinlock cost, while the non-preemptable kernel
> has the fixed cost of checking for rescheduling, at intervals throughout all
> 'interesting' kernel code, essentially all long-running loops.  But by clever
> coding it's possible to finesse away almost all the overhead of those loop
> checks, so in the end, the non-preemptible low-latency patch has a slight
> efficiency advantage here, with emphasis on 'slight'.
> 

As I said: I don't buy the efficiency worries at all.  If scheduling pressure
is so high that either approach impacts performance, then scheduling pressure
is too high.  We need to fix the context switch rate and/or speed up context
switches.  The overhead of conditional_schedule() or preempt will be zilch.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 15:59, Daniel Phillips wrote:

> And while I'm enumerating differences, the preemptable kernel (in this 
> incarnation) has a slight per-spinlock cost, while the non-preemptable kernel 
> has the fixed cost of checking for rescheduling, at intervals throughout all 
> 'interesting' kernel code, essentially all long-running loops.  But by clever 
> coding it's possible to finesse away almost all the overhead of those loop 
> checks, so in the end, the non-preemptible low-latency patch has a slight 
> efficiency advantage here, with emphasis on 'slight'.

True (re spinlock weight in preemptible kernel) but how is that not
comparable to explicit scheduling points?  Worse, the preempt-kernel
typically does its preemption on a branch on return to interrupt
(similar to user space's preemption).  What better time to check and
reschedule if needed?

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 8, 2002 10:17 pm, Robert Love wrote:
> On Tue, 2002-01-08 at 15:59, Daniel Phillips wrote:
> 
> > And while I'm enumerating differences, the preemptable kernel (in this 
> > incarnation) has a slight per-spinlock cost, while the non-preemptable kernel 
> > has the fixed cost of checking for rescheduling, at intervals throughout all 
> > 'interesting' kernel code, essentially all long-running loops.  But by clever 
> > coding it's possible to finesse away almost all the overhead of those loop 
> > checks, so in the end, the non-preemptible low-latency patch has a slight 
> > efficiency advantage here, with emphasis on 'slight'.
> 
> True (re spinlock weight in preemptible kernel) but how is that not
> comparable to explicit scheduling points?  Worse, the preempt-kernel
> typically does its preemption on a branch on return to interrupt
> (similar to user space's preemption).  What better time to check and
> reschedule if needed?

The per-spinlock cost I was refering to is the cost of the inc/dec per 
spinlock.  I guess this cost is small enough as to be hard to measure, but
I have not tried so I don't know.  Curiously, none of the people I've heard
making pronouncements on the overhead of your preempt patch seem to have 
measured it either.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 16:57, Daniel Phillips wrote:

> > True (re spinlock weight in preemptible kernel) but how is that not
> > comparable to explicit scheduling points?  Worse, the preempt-kernel
> > typically does its preemption on a branch on return to interrupt
> > (similar to user space's preemption).  What better time to check and
> > reschedule if needed?
> 
> The per-spinlock cost I was refering to is the cost of the inc/dec per 
> spinlock.  I guess this cost is small enough as to be hard to measure, but
> I have not tried so I don't know.  Curiously, none of the people I've heard
> making pronouncements on the overhead of your preempt patch seem to have 
> measured it either.

;-)
 
If they did I suspect it would be minimal.  Andrew's point on complexity
and overhead in this manner is exact -- such thinks are just not an
issue.

I see two valid arguments against kernel preemption, and I'll be the
first to admit them:

- we introduce new problems with kernel programming.  specifically, the
issue with implicitly locked per-CPU data.  honestly, this isn't a huge
deal.  I've been working on preempt-kernel for awhile now and the
problems we have found and fixed are minimal.  admittedly, however,
especially wrt the future, preempt-kernel may introduce new concerns.  I
say let's rise to meet them.

- we don't do enough for the worst-case latency.  this is where future
work is useful and where preempt-kernel provides the framework for a
better kernel.

I want a better kernel.  Hell, I want the best kernel.  In my opinion,
one factor of that is having a preemptible kernel.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Tue, 8 Jan 2002, Robert Love wrote:

> On Tue, 2002-01-08 at 15:59, Daniel Phillips wrote:
> 
> > And while I'm enumerating differences, the preemptable kernel (in this 
> > incarnation) has a slight per-spinlock cost, while the non-preemptable kernel 
> > has the fixed cost of checking for rescheduling, at intervals throughout all 
> > 'interesting' kernel code, essentially all long-running loops.  But by clever 
> > coding it's possible to finesse away almost all the overhead of those loop 
> > checks, so in the end, the non-preemptible low-latency patch has a slight 
> > efficiency advantage here, with emphasis on 'slight'.
> 
> True (re spinlock weight in preemptible kernel) but how is that not
> comparable to explicit scheduling points?  Worse, the preempt-kernel
> typically does its preemption on a branch on return to interrupt
> (similar to user space's preemption).  What better time to check and
> reschedule if needed?

I'm not sure that preempt and low latency really are attacking the same
problem. What I am finding is the LL improves overall performance when a
process does something which is physically slow, like a find in a
directory with 20k files. On the other hand PK makes the response of the
system better to changes. In particular I see the DNS servers which have
other work running, even backups or reports, are more responsive with PK,
as are usenet news servers. I find it hard to measure "feels faster" with
either approach, although like the supreme court "I know it when I see
it."

I'd like to hope that some of each will get in the main kernel, PK has
been stable for me for a while, LL has never been unstable but I've run it
less.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Daniel Phillips wrote:
> 
> On January 8, 2002 08:47 pm, Andrew Morton wrote:
> > Daniel Phillips wrote:
> > > What a preemptible kernel can do that a non-preemptible kernel can't is:
> > > reschedule exactly as often as necessary, instead of having lots of extra
> > > schedule points inserted all over the place, firing when *they* think the
> > > time is right, which may well be earlier than necessary.
> >
> > Nope.  `if (current->need_resched)' -> the time is right (beyond right,
> > actually).
> 
> Oops, sorry, right.
> 
> The preemptible kernel can reschedule, on average, sooner than the
> scheduling-point kernel, which has to wait for a scheduling point to roll
> around.
> 

Yes.  It can also fix problematic areas which my testing
didn't cover.

Incidentally, there's the SMP problem.  Suppose we
have the code:

	lock_kernel();
	for (lots) {
		do(something sucky);
		if (current->need_resched)
			schedule();
	}
	unlock_kernel();

This works fine on UP, but not on SMP.  The scenario:

- CPU A runs this loop.

- CPU B is spinning on the lock.

- Interrupt occurs, kernel elects to run RT task on CPU B.
  CPU A doesn't have need_resched set, and just keeps 
  on going.  CPU B is stuck spinning on the lock.

This is only an issue for the low-latency patch - all the
other approaches still have sufficiently bad worse-case that
this scenario isn't worth worrying about.

I toyed with creating spin_lock_while_polling_resched(),
but ended up changing the scheduler to set need_resched
against _all_ CPUs if an RT task is being woken (yes, yuk).

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 8, 2002 11:21 pm, Andrew Morton wrote:
> Daniel Phillips wrote:
> > The preemptible kernel can reschedule, on average, sooner than the
> > scheduling-point kernel, which has to wait for a scheduling point to roll
> > around.
> 
> Yes.  It can also fix problematic areas which my testing
> didn't cover.

I bet, with a minor hack, it can help you *find* those problem areas too.  
You compile the two patches together and automatically log any event along 
with the execution address, where your explicit schedule points failed to 
reschedule in time.  Sort of like a profile but suited exactly to your 
problem.

This just detects the problem areas in normal kernel execution, not 
spinlocks, but that is probably where most of the maintainance will be anyway.

By the way, did you check for latency in directory operations?

> Incidentally, there's the SMP problem.  Suppose we
> have the code:
> 
> 	lock_kernel();
> 	for (lots) {
> 		do(something sucky);
> 		if (current->need_resched)
> 			schedule();
> 	}
> 	unlock_kernel();
> 
> This works fine on UP, but not on SMP.  The scenario:
> 
> - CPU A runs this loop.
> 
> - CPU B is spinning on the lock.
> 
> - Interrupt occurs, kernel elects to run RT task on CPU B.
>   CPU A doesn't have need_resched set, and just keeps 
>   on going.  CPU B is stuck spinning on the lock.
> 
> This is only an issue for the low-latency patch - all the
> other approaches still have sufficiently bad worse-case that
> this scenario isn't worth worrying about.
> 
> I toyed with creating spin_lock_while_polling_resched(),
> but ended up changing the scheduler to set need_resched
> against _all_ CPUs if an RT task is being woken (yes, yuk).

Heh, subtle.  Thanks for pointing that out and making my head hurt.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Daniel Phillips wrote:
> 
> On January 8, 2002 11:21 pm, Andrew Morton wrote:
> > Daniel Phillips wrote:
> > > The preemptible kernel can reschedule, on average, sooner than the
> > > scheduling-point kernel, which has to wait for a scheduling point to roll
> > > around.
> >
> > Yes.  It can also fix problematic areas which my testing
> > didn't cover.
> 
> I bet, with a minor hack, it can help you *find* those problem areas too.
> You compile the two patches together and automatically log any event along
> with the execution address, where your explicit schedule points failed to
> reschedule in time.  Sort of like a profile but suited exactly to your
> problem.

Well, one of the instrumentation patches which I use detects a
scheduling overrun at interrupt time and emits an all-CPU backtrace.
You just feed the trace into ksymoops or gdb then go stare at
the offending code.  

That's the easy part - the hard part is getting sufficient coverage.
There are surprising places.  close_files(), exit_notify(), ...

> This just detects the problem areas in normal kernel execution, not
> spinlocks, but that is probably where most of the maintainance will be anyway.
> 
> By the way, did you check for latency in directory operations?

Yes.  They can be very bad for really large directories.  Scheduling
on the found-in-cache case in bread() kills that one easily for most
local filesystems.  There may still be a problem in ext2.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 9, 2002 10:26 am, Andrew Morton wrote:
> Daniel Phillips wrote:
> > By the way, did you check for latency in directory operations?
> 
> Yes.  They can be very bad for really large directories.  Scheduling
> on the found-in-cache case in bread() kills that one easily for most
> local filesystems.  There may still be a problem in ext2.

A indexed directory won't have that problem - I'll get to finishing off the 
htree patch pretty soon[1].  In any event, the analogous technique will work: 
a schedule point in ext2_bread.

[1] Wli's hash work is happening at a convenient time.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Luigi Genoni]

On Tue, 8 Jan 2002, Daniel Phillips wrote:

> On January 8, 2002 08:47 pm, Andrew Morton wrote:
> > Daniel Phillips wrote:
> > > What a preemptible kernel can do that a non-preemptible kernel can't is:
> > > reschedule exactly as often as necessary, instead of having lots of extra
> > > schedule points inserted all over the place, firing when *they* think the
> > > time is right, which may well be earlier than necessary.
> >
> > Nope.  `if (current->need_resched)' -> the time is right (beyond right,
> > actually).
>
> Oops, sorry, right.
>
> The preemptible kernel can reschedule, on average, sooner than the
> scheduling-point kernel, which has to wait for a scheduling point to roll
> around.
>
mmhhh. At which cost? And then anyway if I have a spinlock, I still have
to wait for a scheduling point to roll around.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 9, 2002 12:26 am, Luigi Genoni wrote:
> On Tue, 8 Jan 2002, Daniel Phillips wrote:
> 
> > On January 8, 2002 08:47 pm, Andrew Morton wrote:
> > > Daniel Phillips wrote:
> > > > What a preemptible kernel can do that a non-preemptible kernel can't 
is:
> > > > reschedule exactly as often as necessary, instead of having lots of 
extra
> > > > schedule points inserted all over the place, firing when *they* think 
the
> > > > time is right, which may well be earlier than necessary.
> > >
> > > Nope.  `if (current->need_resched)' -> the time is right (beyond right,
> > > actually).
> >
> > Oops, sorry, right.
> >
> > The preemptible kernel can reschedule, on average, sooner than the
> > scheduling-point kernel, which has to wait for a scheduling point to roll
> > around.
>
> mmhhh. At which cost? And then anyway if I have a spinlock, I still have
> to wait for a scheduling point to roll around.

Did you read the thread?  Think about the relative amount of time spent in
spinlocks vs the amount of time spent in the regular kernel.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Robert Love]

On Tue, 2002-01-08 at 14:47, Andrew Morton wrote:

> > What a preemptible kernel can do that a non-preemptible kernel can't is:
> > reschedule exactly as often as necessary, instead of having lots of extra
> > schedule points inserted all over the place, firing when *they* think the
> > time is right, which may well be earlier than necessary.
> 
> Nope.  `if (current->need_resched)' -> the time is right (beyond right,
> actually).

Eh, I disagree here.  The right time is the moment a high-priority task
becomes runnable.  Given your HZ, only a fully preemptible kernel can
come close to meeting that.

> > Finally, with preemption, rescheduling can be forced with essentially zero
> > latency in response to an arbitrary interrupt such as IO completion, whereas
> > the non-preemptive kernel will have to 'coast to a stop'.  In other words,
> > the non-preemptive kernel will have little lags between successive IOs,
> > whereas the preemptive kernel can submit the next IO immediately.  So there
> > are bound to be loads where the preemptive kernel turns in better latency
> > *and throughput* than the scheduling point hack.
> 
> Latency yes.  Throughout no.

I bet in _many_ (most?) workloads the preemptible kernel turns in better
throughput.  Anytime there is load on the system, there should be a
benefit.  I bet the same goes for your patch.  I've certainly verified
it for both in various loads myself.

> I don't think the "preempt slows down the kernel" argument is very valid
> really.  Let's invert the argument - Linux is multitasking, and that has a
> cost.  There's no reason why certain bits of the kernel need to violate that
> just to get a bit more throughput.  If it really worries you, set HZ=10 and
> increase all the timeslices, etc.

Very well said.  I always find an answer to the "more complexity, more
context switching, blah blah" arguments as ultimately being arguments
tantamount to "preemptive multitasking sucks".

> Now, there *may* be overheads added due to losing the implicit locking which
> per-CPU data gives you.

Perhaps, but note what preempt enable and disable statements effectively
are: an inc and a dec.  Not even atomic.

Yes, there is a branch on reenable.  This may be an interesting change
to look into.  FWIW, we have a construct that doesn't check for
reschedule on reenable, too.

> The main cost of preempt IMO is in complexity and stability risks.
> 
> (BTW: I took a weird oops testing the preempt patch on an SMP NFS client.
> The fault address was 0x0aXXXXXX.  No useful backtrace, unfortunately).

Should of sent me the oops :)

> > Mind you, I'm not devaluing Andrew's work, it's good and valuable.  However
> > it's good to be aware of why that approach can't equal the latency-busting
> > performance of the preemptive approach.
> 
> There's no point in just merging the preempt patch and saying "there,
> that's done".  It doesn't do anything.
> 
> Instead, a decision needs to be made: "Linux will henceforth be a 
> low-latency kernel".  Now, IF we can come to this decision, then
> internal preemption is the way to do it.  But it affects ALL kernel
> developers.  Because we'll need to introduce a new rule: "it is a
> bug to spend more than five milliseconds holding any locks".
> 
> So.  Do we we want a low-latency kernel?  Are we prepared to mandate
> the five-millisecond rule?   It can be done, but won't be easy, and
> we'll never get complete coverage.  But I don't see the will around
> here.

I agree here, but then I do have three points:

- proper lock use that benefits SMP scalability benefits preempt-kernel
induced latency improvements.  In other words, things like short lock
durations, fine-grained locking, and ditching the BKL benefit both
worlds.

- with the preemptible kernel in place, we can look at the long-held
locks and figure ways to combat them.  The preempt-stats patch helps us
find them.  Then, we can take a lock-break approach.  We can look into
finer grained locks.  We can localize lock the lock if it is global or
the BKL.  Or we can do something radical like make long-held spinlocks
priority-inheriting when preempt-kernel is enabled.  In other words,
preempt-kernel becomes a framework for proper solutions in the future.

- finally, the usual: it's an option.

	Robert Love

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Oliver Xymoron]

On Tue, 8 Jan 2002, Andrew Morton wrote:

> > What a preemptible kernel can do that a non-preemptible kernel can't is:
> > reschedule exactly as often as necessary, instead of having lots of extra
> > schedule points inserted all over the place, firing when *they* think the
> > time is right, which may well be earlier than necessary.
>
> Nope.  `if (current->need_resched)' -> the time is right (beyond right,
> actually).

Have we ever considered making rescheduling work like get_user? That is,
make current->need_resched be a pointer, and if we need to reschedule,
make it an INVALID pointer that causes us to fault and call schedule in
its fault path?

Orthogonally, for rescheduling points with locks, we could build a version
of the spinlocks that know when they're blocking other processes and can
do a spin_yield(&lock) in places where they can safely give up a lock. On
single processor, spin_yield could translate to a scheduling point.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Helge Hafting]

Oliver Xymoron wrote:
[...]
> Have we ever considered making rescheduling work like get_user? That is,
> make current->need_resched be a pointer, and if we need to reschedule,
> make it an INVALID pointer that causes us to fault and call schedule in
> its fault path?

Elegant perhaps, but now you take the time to do a completely
unnecessary
page fault when rescheduling.  This has a cost which is high on
some architectures.  But the point of rescheduling was to improve
interactive performance and io latency.  
Every page fault may have to check for this case.

Helge Hafting

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Luigi Genoni]

On Tue, 8 Jan 2002, Andrea Arcangeli wrote:

> On Tue, Jan 08, 2002 at 11:55:59AM +0100, Luigi Genoni wrote:
> >
> >
> > On Tue, 8 Jan 2002, Dieter [iso-8859-15] Nützel wrote (passim):
> >
> > > Is it possible to decide, now what should go into 2.4.18 (maybe -pre3) -aa or
> > > -rmap?
> > [...]
> > > Maybe preemption? It is disengageable so nobody should be harmed but we get
> > > the chance for wider testing.
> > >
> > > Any comments?
> > preemption?? this is eventually 2.5 stuff, and should not be integrated
>
> indeed ("eventually" in the italian sense btw, obvious to me, but not
> for l-k).
>
> I'm not against preemption (I can see the benefits about the mean
> latency for real time DSP) but the claims about preemption making the
> kernel faster doesn't make sense to me. more frequent scheduling,
> overhead of branches in the locks (you've to conditional_schedule after
> the last preemption lock is released and the cachelines for the per-cpu
> preemption locks) and the other preemption stuff can only make the
> kernel slower.  Furthmore for multimedia playback any sane kernel out
> there with lowlatency fixes applied will work as well as a preemption
> kernel that pays for all the preemption overhead.
I would add that preemption simply gives a felling of more speed with
interactive usage (with one single user on the system), and also has some
advantages for dedicated servers, but except of those conditions it never
showed in my experience to be a real and decisive advantage.
Of course we are supposing that the preemptive scheduler is very well
done, because otherway (bad working scheduler) there is nothing worse than
preemption.
>
> About the other claim that as the kernel becomes more granular
> performance will increase with preemption in kernel, that's obviously
> wrong as well, it's clearly the other way around. Maybe it was meant
> "latency will decrease further", that's right, but also performance will
> decrease if something.
>
> So yes, mean latency will decrease with preemptive kernel, but your CPU
> is definitely paying something for it.
agreed. Obviously this choice depends on what you want to do with your
system. If you have more than a couple of interactive users (and here I
have also 50 interactive users at the same time on every single system),
preemption is not what you want, period. If you have a desktop system,
well, it is a different situation.
>
> > into 2.4 stable tree. Of course a backport is possible, when/if it will be
> > quite well tested and well working on 2.5
> >
> Andrea


Luigi

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Tue, 8 Jan 2002, Andrea Arcangeli wrote:

> I'm not against preemption (I can see the benefits about the mean
> latency for real time DSP) but the claims about preemption making the
> kernel faster doesn't make sense to me. more frequent scheduling,
> overhead of branches in the locks (you've to conditional_schedule after
> the last preemption lock is released and the cachelines for the per-cpu
> preemption locks) and the other preemption stuff can only make the
> kernel slower.  Furthmore for multimedia playback any sane kernel out
> there with lowlatency fixes applied will work as well as a preemption
> kernel that pays for all the preemption overhead.

I'm not sure I have seen claims that it makes the kernel faster, but it
sure makes the latency lower, and improves performance for systems doing a
lot of network activity (DNS servers) with anything else going on in the
systems, such as daily reports and backups.

I will try the low latency kernel stuff, but I think intrinsically that if
you want to service the incoming requests quickly you have to dispatch to
them quickly, not at the end of a time slice. Preempt is a way to avoid
having to play with RT processes, and I think it's desirable in general as
an option where the load will benefit from such behaviour.

I'm not sure it "competes" with low latency, since many of the thing LL is
doing are "good things" in general.

Finally, I doubt that any of this will address my biggest problem with
Linux, which is that as memory gets cheap a program doing significant disk
writing can get buffers VERY full (perhaps a while CD worth) before the
kernel decides to do the write, at which point the system becomes
non-responsive for seconds at a time while the disk light comes on and
stays on. That's another problem, and I did play with some patches this
weekend without making myself really happy :-( Another topic,
unfortunately.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[yodaiken]

On Sun, Jan 13, 2002 at 07:46:54PM -0500, Bill Davidsen wrote:
> Finally, I doubt that any of this will address my biggest problem with
> Linux, which is that as memory gets cheap a program doing significant disk
> writing can get buffers VERY full (perhaps a while CD worth) before the
> kernel decides to do the write, at which point the system becomes
> non-responsive for seconds at a time while the disk light comes on and
> stays on. That's another problem, and I did play with some patches this
> weekend without making myself really happy :-( Another topic,
> unfortunately.

I think this is a critical problem. I'd like to be able to have some
assurance that a task with a buffer of size N doing read-disk->write-disk
will maintain data flow at some minimal rate over intervals of 1 or 2
seconds or something like that.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrew Morton]

Bill Davidsen wrote:
> 
> Finally, I doubt that any of this will address my biggest problem with
> Linux, which is that as memory gets cheap a program doing significant disk
> writing can get buffers VERY full (perhaps a while CD worth) before the
> kernel decides to do the write, at which point the system becomes
> non-responsive for seconds at a time while the disk light comes on and
> stays on. That's another problem, and I did play with some patches this
> weekend without making myself really happy :-( Another topic,
> unfortunately.

/proc/sys/vm/bdflush: Decreasing the kupdate interval from five
seconds, decreasing the nfract and nfract_sync setting in there
should smooth this out.  The -aa patches add start and stop
levels for bdflush as well, which means that bdflush can be the
one who blocks on IO rather than your process.  And it means that
the request queue doesn't get 100% drained as soon as the writer
hits nfract_sync.

All very interesting and it will be fun to play with when it
*finally* gets merged.

But with the current elevator design, disk read latencies will
still be painful.

-

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Bill Davidsen]

On Sun, 13 Jan 2002, Andrew Morton wrote:

> Bill Davidsen wrote:
> > 
> > Finally, I doubt that any of this will address my biggest problem with
> > Linux, which is that as memory gets cheap a program doing significant disk
> > writing can get buffers VERY full (perhaps a while CD worth) before the
> > kernel decides to do the write, at which point the system becomes
> > non-responsive for seconds at a time while the disk light comes on and
> > stays on.

> /proc/sys/vm/bdflush: Decreasing the kupdate interval from five
> seconds, decreasing the nfract and nfract_sync setting in there
> should smooth this out.  The -aa patches add start and stop
> levels for bdflush as well, which means that bdflush can be the
> one who blocks on IO rather than your process.  And it means that
> the request queue doesn't get 100% drained as soon as the writer
> hits nfract_sync.

Been there, done that. Makes it "less bad" if the right settings are
chosen. I will try -aa on 2.4.18-pre3 (or 4 if the patch is out), I've
been trying -ac this morning. Looking at the code, it doesn't look as if
the logic is what I want to see, no matter how tuned. last night I tried a
patch, and several of the "unued" elements in the bdflush were reused, but
I froze w/o any io for seconds, so I don't have it right.

What I want is a smooth increase in how mush is written as the dirty
buffers increase. Percentage of {anything} may be the wrong thing to use,
the problem is dirty buffers vs. disk write bandwidth, on a 2GB machine
it's a smaller percentage than 128M machine, but the absolute numbers seem
to be similar.
 
> All very interesting and it will be fun to play with when it
> *finally* gets merged.
> 
> But with the current elevator design, disk read latencies will
> still be painful.

There are a few patches around to change that. Note I didn't say "fix"
just change.

Finally, one of my goals is to be able to keep a large free page pool. I
have two apps which will suddenly need another 8MB or so, and if they have
to wait for disk they become unpleasant. With current memory prices I
don't mind "wasting" 256MB or so, if it means I get better response when I
want it. This is all part of tuning the system to application, I certainly
wouldn't use it on other machines.

Better doc of bdflush wouldn't be amiss, either. When/if it stops changing
I will clean up my notes from a stack of 3x5 cards to something useful and
make them available. If there's a doc giving what the bdflush values do
(in current kernels) and what happens when you change them, and what to
tune first if you have this problem, I haven't found it. 

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 14, 2002 01:46 am, Bill Davidsen wrote:
> Finally, I doubt that any of this will address my biggest problem with
> Linux, which is that as memory gets cheap a program doing significant disk
> writing can get buffers VERY full (perhaps a while CD worth) before the
> kernel decides to do the write, at which point the system becomes
> non-responsive for seconds at a time while the disk light comes on and
> stays on.  That's another problem, and I did play with some patches this
> weekend without making myself really happy :-( Another topic,
> unfortunately.

Patience, this is understood, a solution is known and a fix is in the 
pipeline.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 14, 2002 01:46 am, Bill Davidsen wrote:
> Finally, I doubt that any of this will address my biggest problem with
> Linux, which is that as memory gets cheap a program doing significant disk
> writing can get buffers VERY full (perhaps a while CD worth) before the
> kernel decides to do the write, at which point the system becomes
> non-responsive for seconds at a time while the disk light comes on and
> stays on.  That's another problem, and I did play with some patches this
> weekend without making myself really happy :-( Another topic,
> unfortunately.

Patience, this is understood, a solution is known and a fix is in the 
pipeline.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 14, 2002 01:46 am, Bill Davidsen wrote:
> Finally, I doubt that any of this will address my biggest problem with
> Linux, which is that as memory gets cheap a program doing significant disk
> writing can get buffers VERY full (perhaps a while CD worth) before the
> kernel decides to do the write, at which point the system becomes
> non-responsive for seconds at a time while the disk light comes on and
> stays on.  That's another problem, and I did play with some patches this
> weekend without making myself really happy :-( Another topic,
> unfortunately.

Patience, this is understood, a solution is known and a fix is in the 
pipeline.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 14, 2002 01:46 am, Bill Davidsen wrote:
> Finally, I doubt that any of this will address my biggest problem with
> Linux, which is that as memory gets cheap a program doing significant disk
> writing can get buffers VERY full (perhaps a while CD worth) before the
> kernel decides to do the write, at which point the system becomes
> non-responsive for seconds at a time while the disk light comes on and
> stays on.  That's another problem, and I did play with some patches this
> weekend without making myself really happy :-( Another topic,
> unfortunately.

Patience, the problem is understood and there will be a fix in the 2.5 
timeframe.

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Alan Cox]

> > stays on.  That's another problem, and I did play with some patches this
> > weekend without making myself really happy :-( Another topic,
> > unfortunately.
> 
> Patience, the problem is understood and there will be a fix in the 2.5 
> timeframe.

Without a fix in the 2.4 timeframe everyone has to run 2.2. That strikes
me as decidedly non optimal. If you are having VM problems try both the
Andrea -aa and the Rik rmap-11b patches (*not together*) and report back

Re: [2.4.17/18pre] VM and swap - it's really unusable

[J Sloan]

Alan Cox wrote:

>>>stays on.  That's another problem, and I did play with some patches this
>>>weekend without making myself really happy :-( Another topic,
>>>unfortunately.
>>>
>>Patience, the problem is understood and there will be a fix in the 2.5 
>>timeframe.
>>
>
>Without a fix in the 2.4 timeframe everyone has to run 2.2. That strikes
>me as decidedly non optimal. If you are having VM problems try both the
>Andrea -aa and the Rik rmap-11b patches (*not together*) and report back
>
Easiest is to grab 2.4.17 and apply 2.4.18pre2 and 2.4.18pre2-aa2 -

pre2-aa2 has all the fixes and tweaks I had been doing by hand.

cu

jjs

>

Re: [2.4.17/18pre] VM and swap - it's really unusable

[J.A. Magallon]

On 20020108 Dieter Nützel wrote:
>Is it possible to decide, now what should go into 2.4.18 (maybe -pre3) -aa or 
>-rmap?
>Andrew Morten`s read-latency.patch is a clear winner for me, too.
>What about 00_nanosleep-5 and bootmem?
>The O(1) scheduler?
>Maybe preemption? It is disengageable so nobody should be harmed but we get 
>the chance for wider testing.
>
>Any comments?
>

I would pefer the ton of small, usefull and safe bits in Andrea's kernel
(vm-21, cache-aligned-spinlocks, compiler, gcc3, rwsem, highmen fixes...)

-- 
J.A. Magallon                           #  Let the source be with you...        
mailto:jamagallon@able.es
Mandrake Linux release 8.2 (Cooker) for i586
Linux werewolf 2.4.18-pre2-beo #1 SMP Tue Jan 8 03:18:18 CET 2002 i686

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Daniel Phillips]

On January 9, 2002 02:55 pm, Andrea Arcangeli wrote:
> On Wed, Jan 09, 2002 at 12:56:50PM +0100, Daniel Phillips wrote:
> > BTW, I find your main argument confusing.  First you don't want -preempt with
> > CONFIG_EXERIMENTAL because it might not get wide enough testing, so you want 
> > to enable it by default in the mainline kernel, then you argue it's too risky 
> > because everybody will use it and it might break some obscure driver.  Sorry, 
> > you lost me back there.
> 
> the point I am making is very simple: _if_ we include it, it should _not_
> be a config option.

That doesn't make any sense to me.  Why should _SMP be a config option and not
_PREEMPT?

--
Daniel

Re: [2.4.17/18pre] VM and swap - it's really unusable

[Andrea Arcangeli]

On Wed, Jan 09, 2002 at 03:07:40PM +0100, Daniel Phillips wrote:
> On January 9, 2002 02:55 pm, Andrea Arcangeli wrote:
> > On Wed, Jan 09, 2002 at 12:56:50PM +0100, Daniel Phillips wrote:
> > > BTW, I find your main argument confusing.  First you don't want -preempt with
> > > CONFIG_EXERIMENTAL because it might not get wide enough testing, so you want 
> > > to enable it by default in the mainline kernel, then you argue it's too risky 
> > > because everybody will use it and it might break some obscure driver.  Sorry, 
> > > you lost me back there.
> > 
> > the point I am making is very simple: _if_ we include it, it should _not_
> > be a config option.
> 
> That doesn't make any sense to me.  Why should _SMP be a config option and not

getting the drivers tested with preempt enable makes lots of sense to
me.

> _PREEMPT?

SMP in 2.1 wasn't a config option.

Andrea