Process priority in 2.4.18 (RedHat 7.3)

Process priority in 2.4.18 (RedHat 7.3)

[Paul Faure]

Just installed RedHat 7.3 with kernel 2.4.18 and noticed that my network 
no longer functions when my CPU usage is at 100%.  Looking at the 
priorities of the application that I was running showed a priority of 35 
(thats what `nice ./test` produces).  Then looking at the priorities of 
all the other processes shows a range from 15 to 35.

All the main kernel processes run at 15 except ksoftirqd_CPU0 and 
ksoftirqd_CPU1 which run at 35.

None of the documentation on my system (`man nice` etc...) mentions
anything about priority levels between 15 and 35, they all say -20 to +20
(as it used to be).

So, why the change the priority levels?  And why my test app (at any 
priority level) starves out my network?

I am running an SMP box.

Thanks.

-- 
Paul N. Faure					613.266.3286
EngSoc Administrator            		paul-at-engsoc-dot-org
Chief Technical Officer, CertainKey Inc.	paul-at-certainkey-dot-com
Carleton University Systems Eng. 4th Year	paul-at-faure-dot-ca

Re: Process priority in 2.4.18 (RedHat 7.3)

[Nerijus Baliunas]

On Thu, 16 May 2002 16:08:15 -0400 (EDT) Paul Faure <paul@engsoc.org> wrote:

> Just installed RedHat 7.3 with kernel 2.4.18 and noticed that my network 
> no longer functions when my CPU usage is at 100%.  Looking at the 
> priorities of the application that I was running showed a priority of 35 
> (thats what `nice ./test` produces).  Then looking at the priorities of 
> all the other processes shows a range from 15 to 35.

> I am running an SMP box.

Didn't care to look at RedHat errata?

Regards,
Nerijus

Re: Process priority in 2.4.18 (RedHat 7.3)

[Paul Faure]

On Thu, 16 May 2002, Andrew Morton wrote:

> Paul Faure wrote:
> > 
> > Just installed RedHat 7.3 with kernel 2.4.18 and noticed that my network
> > no longer functions when my CPU usage is at 100%.
> 
> ugh.  Which NIC are you using?

ne.o
Its an ISA card.

-- 
Paul N. Faure					613.266.3286
EngSoc Administrator            		paul-at-engsoc-dot-org
Chief Technical Officer, CertainKey Inc.	paul-at-certainkey-dot-com
Carleton University Systems Eng. 4th Year	paul-at-faure-dot-ca

Re: Process priority in 2.4.18 (RedHat 7.3)

[Andrew Morton]

Paul Faure wrote:
> 
> On Thu, 16 May 2002, Andrew Morton wrote:
> 
> > Paul Faure wrote:
> > >
> > > Just installed RedHat 7.3 with kernel 2.4.18 and noticed that my network
> > > no longer functions when my CPU usage is at 100%.
> >
> > ugh.  Which NIC are you using?
> 
> ne.o
> Its an ISA card.
> 

It's possible that this particular driver is flunking on a lot
of transmit attempts and is relying on ksoftirqd to transmit.

Are you able to test with a moderately respectable NIC?
tulip, 3com, realtek, eepro?

-

Re: Process priority in 2.4.18 (RedHat 7.3)

[Andrea Arcangeli]

On Thu, May 16, 2002 at 02:06:10PM -0700, Andrew Morton wrote:
> of transmit attempts and is relying on ksoftirqd to transmit.

ksoftirqd or not the softirq are guaranteed to keep running even if
there's a task in loop with SCHED_FIFO, ksoftirqd only enhance/polish
the case of a recursive softirq, or a very big flood of softirq events,
it is not required to run softirqs.

Andrea

Re: Process priority in 2.4.18 (RedHat 7.3)

[Paul Faure]

It would seem that it only occurs when running the application (that takes
100% of the CPU) as root.

As for testing it with other cards, I only have this one card.

Thanks for your time... its not a big issue now since I can run my
application as a non privileged user.

On Thu, 16 May 2002, Andrea Arcangeli wrote:

> On Thu, May 16, 2002 at 02:06:10PM -0700, Andrew Morton wrote:
> > of transmit attempts and is relying on ksoftirqd to transmit.
> 
> ksoftirqd or not the softirq are guaranteed to keep running even if
> there's a task in loop with SCHED_FIFO, ksoftirqd only enhance/polish
> the case of a recursive softirq, or a very big flood of softirq events,
> it is not required to run softirqs.
> 
> Andrea
> 

-- 
Paul N. Faure					613.266.3286
EngSoc Administrator            		paul-at-engsoc-dot-org
Chief Technical Officer, CertainKey Inc.	paul-at-certainkey-dot-com
Carleton University Systems Eng. 4th Year	paul-at-faure-dot-ca

Re: Process priority in 2.4.18 (RedHat 7.3)

[Andrew Morton]

Paul Faure wrote:
> 
> It would seem that it only occurs when running the application (that takes
> 100% of the CPU) as root.

That's because without root, the application cannot raise its
scheuling priority and it cannot change to realtime policy.

So the problem would appear to be that your networking *requires*
ksoftirqd services to function.  Either:

1) The driver is bust - its hard_start_xmit() function is failing
   frequently, and relying on ksoftirqd to get things done (I think;
   it's been a while).  Or

2) Something is wrong with the ksoftirqd design.  Or

3) Red Hat fiddled with ksoftirqd and broke it.

I'd be inclined to suspect 1).

> As for testing it with other cards, I only have this one card.
> 

That's a shame.

-

Re: Process priority in 2.4.18 (RedHat 7.3)

[Andrea Arcangeli]

On Fri, May 17, 2002 at 01:49:21PM +0100, Alan Cox wrote:
> I think its mostly #2. We invoke ksoftirq far far too easily.

ksoftirqd + SCHED_FIFO is like no ksoftirqd at all, provided the ne card
is irq driven (it is) everything works like it was working in 2.4.0.

Andrea

Re: Process priority in 2.4.18 (RedHat 7.3)

[Alan Cox]

> So the problem would appear to be that your networking *requires*
> ksoftirqd services to function.  Either:
> 
> 1) The driver is bust - its hard_start_xmit() function is failing
>    frequently, and relying on ksoftirqd to get things done (I think;
>    it's been a while).  Or

The ne2k card has little buffering. 

> 2) Something is wrong with the ksoftirqd design.  Or

I think its mostly #2. We invoke ksoftirq far far too easily.

Re: Process priority in 2.4.18 (RedHat 7.3)

[Andrea Arcangeli]

On Fri, May 17, 2002 at 02:01:30PM +0100, Alan Cox wrote:
> > On Fri, May 17, 2002 at 01:49:21PM +0100, Alan Cox wrote:
> > > I think its mostly #2. We invoke ksoftirq far far too easily.
> > 
> > ksoftirqd + SCHED_FIFO is like no ksoftirqd at all, provided the ne card
> > is irq driven (it is) everything works like it was working in 2.4.0.
> 
> For a 10Mbit ne2k it ought to be if its done with sched fifo. For serious
> devices its not. The ksoftirqd bounce blows everything out of cache and is
> easily measured

if you're under a flood of irq ksoftirqd or not won't make differences
to the softirq handling, and yes in such case ksoftirq cannot help
because you are under a flood of do_softirq anyways run from irq context
and it is only a minor scheduler overhead in such case, but it gets
right all and polishes all the other "recursion" cases like NAPI.

But that has nothing to do with this case, here the userspace runs with
SCHED_FIFO in a loop so ksoftirqd cannot make any difference compared to
2.4.0 if the device is irq driven, so I don't see your point in
mentioning minor performance regressions while under a flood of irqs due
the minor scheduler overhead, here the minor scheduler overhead cannot
apply because ksoftirqd has not a chance to run at all.

Also I'd be nice if he could try with mainline (or 2.4.19pre8aa3) too
just in case, we didn't had any confirm that such proggy uses SCHED_FIFO
or SCHED_RR, even if I of course agree about the supposions made by
Andrew without having access to additional informations.

Andrea

Re: Process priority in 2.4.18 (RedHat 7.3)

[Alan Cox]

> On Fri, May 17, 2002 at 01:49:21PM +0100, Alan Cox wrote:
> > I think its mostly #2. We invoke ksoftirq far far too easily.
> 
> ksoftirqd + SCHED_FIFO is like no ksoftirqd at all, provided the ne card
> is irq driven (it is) everything works like it was working in 2.4.0.

For a 10Mbit ne2k it ought to be if its done with sched fifo. For serious
devices its not. The ksoftirqd bounce blows everything out of cache and is
easily measured

Re: Process priority in 2.4.18 (RedHat 7.3)

[Andrea Arcangeli]

On Fri, May 17, 2002 at 03:51:20PM +0100, Alan Cox wrote:
> > > For a 10Mbit ne2k it ought to be if its done with sched fifo. For serious
> > > devices its not. The ksoftirqd bounce blows everything out of cache and is
> > > easily measured
> > 
> > if you're under a flood of irq ksoftirqd or not won't make differences
> 
> I didnt mention a flood of irqs. If stuff falls back to softirqd it 
> materially harms throughput

You did implicitly becuse if there's not a flood of irq or recursive
softirqs it cannot fall to sofitrqd.

Andrea

Re: Process priority in 2.4.18 (RedHat 7.3)

[Alan Cox]

> > For a 10Mbit ne2k it ought to be if its done with sched fifo. For serious
> > devices its not. The ksoftirqd bounce blows everything out of cache and is
> > easily measured
> 
> if you're under a flood of irq ksoftirqd or not won't make differences

I didnt mention a flood of irqs. If stuff falls back to softirqd it 
materially harms throughput

Re: Process priority in 2.4.18 (RedHat 7.3)

[Alan Cox]

> > I didnt mention a flood of irqs. If stuff falls back to softirqd it 
> > materially harms throughput
> 
> You did implicitly becuse if there's not a flood of irq or recursive
> softirqs it cannot fall to sofitrqd.

Hardly takes a flood of IRQ's. A tiny burst of interrupts will happily
trigger it 

Re: Process priority in 2.4.18 (RedHat 7.3)

[Andrea Arcangeli]

On Fri, May 17, 2002 at 03:57:15PM +0100, Alan Cox wrote:
> Hardly takes a flood of IRQ's. A tiny burst of interrupts will happily
> trigger it 

hmm, tiny != burst. of course sometime ksoftirqd will kick in when it
notices a burst. But it is irrelevant to this thread about SCHED_FIFO +
ksoftirqd.

If there's SCHED_FIFO app in loop, ksoftirqd never runs and we only rely
on the support from irq that we had in 2.4.0 and previous too.

returning to the other issue about the fake burst that stops immediatly,
it should be easy to write a small heuristic to avoid scheduling
ksoftirqd in those cases, two different ways could be to simply count
the number of do_softirq invocations outside ksofitrqd per jiffy and to
skip waking up ksoftirqd if the frequency is high enough, this would fix
the minor schedule overhead during the fake burst, however this may not
be optimal for NAPI if for whatever reason there's an irq flood from
some other device in background, so I would actually prefer to detect
when ksoftirqd stops immediatly, it's a bit more complicated but more
generic, if it always stops immediatly then we make it harder to kick it
in. Note that I also benchmarked the effect softirqd in the early days
and I couldn't notice any performance drawback with using netpipe on
100Mbit tulip both on the small fragments (higher burst) and on the big
fragments, so whatever regression it is it is a minor issue and it
definitely pays off the polishing of all the other important cases so
cleanly (NAPI, rcu-poll, ppa, irq flood even on 10Mbit, even if you
block the hard irq with ratelimit you still can hang in the softirq code
without ksofitrqd otherwise you risk to introduce a 10msec latency to
the softirq processing, you don't know how long the softirq load takes
etc..).  It is true that even with ksoftirqd you may hang if the softirq
load takes exactly the same frequency of the do_softirq calls, but
that's not a pratical case, you cannot stay exactly on the border for a
relevant time, while you can pratically go over the border as long as
you want.

Andrea

Re: Process priority in 2.4.18 (RedHat 7.3)

[Paul Faure]

A little more information:

On a dual system:
Running as root -> Single process app (50% total CPU)-> locks out network
Running as root -> Multi process app (100% total CPU) -> locks out network
Running unprivileged -> Single process(50%) -> Network works fine
Running unprivileged -> Multi process(100%) -> locks out network

Going to try a few other network cards if I find one.

On Fri, 17 May 2002, Alan Cox wrote:

> > So the problem would appear to be that your networking *requires*
> > ksoftirqd services to function.  Either:
> > 
> > 1) The driver is bust - its hard_start_xmit() function is failing
> >    frequently, and relying on ksoftirqd to get things done (I think;
> >    it's been a while).  Or
> 
> The ne2k card has little buffering. 
> 
> > 2) Something is wrong with the ksoftirqd design.  Or
> 
> I think its mostly #2. We invoke ksoftirq far far too easily.
> 

-- 
Paul N. Faure					613.266.3286
EngSoc Administrator            		paul-at-engsoc-dot-org
Chief Technical Officer, CertainKey Inc.	paul-at-certainkey-dot-com
Carleton University Systems Eng. 4th Year	paul-at-faure-dot-ca

Re: Process priority in 2.4.18 (RedHat 7.3)

[Alan Cox]

> hmm, tiny != burst. of course sometime ksoftirqd will kick in when it
> notices a burst. But it is irrelevant to this thread about SCHED_FIFO +
> ksoftirqd.

Agreed

> If there's SCHED_FIFO app in loop, ksoftirqd never runs and we only rely
> on the support from irq that we had in 2.4.0 and previous too.

Yes

Re: Process priority in 2.4.18 (RedHat 7.3)

[Paul Faure]

My brother dropped off a network card (smc-ultra) and it works fine with 
the 2.4.18-4 redhat kernel. If you are still interested in testing the 
ne.o card with the latest official kernel, I can do it.

Thanks for all your help.

On Fri, 17 May 2002, Alan Cox wrote:

> > hmm, tiny != burst. of course sometime ksoftirqd will kick in when it
> > notices a burst. But it is irrelevant to this thread about SCHED_FIFO +
> > ksoftirqd.
> 
> Agreed
> 
> > If there's SCHED_FIFO app in loop, ksoftirqd never runs and we only rely
> > on the support from irq that we had in 2.4.0 and previous too.
> 
> Yes
> 

-- 
Paul N. Faure					613.266.3286
EngSoc Administrator            		paul-at-engsoc-dot-org
Chief Technical Officer, CertainKey Inc.	paul-at-certainkey-dot-com
Carleton University Systems Eng. 4th Year	paul-at-faure-dot-ca

Re: Process priority in 2.4.18 (RedHat 7.3)

[Paul Faure]

My brother dropped of a network card (smc-ultra) and it works fine with 
the 2.4.18-4 redhat kernel. If you are still interested in testing the 
ne.o card with the latest official kernel, I can do so.

Thanks for all your help.

On Fri, 17 May 2002, Alan Cox wrote:

> > hmm, tiny != burst. of course sometime ksoftirqd will kick in when it
> > notices a burst. But it is irrelevant to this thread about SCHED_FIFO +
> > ksoftirqd.
> 
> Agreed
> 
> > If there's SCHED_FIFO app in loop, ksoftirqd never runs and we only rely
> > on the support from irq that we had in 2.4.0 and previous too.
> 
> Yes
> 

-- 
Paul N. Faure					613.266.3286
EngSoc Administrator            		paul-at-engsoc-dot-org
Chief Technical Officer, CertainKey Inc.	paul-at-certainkey-dot-com
Carleton University Systems Eng. 4th Year	paul-at-faure-dot-ca