* RE: [PATCH] turn off irqdebug by default on Altix
2004-02-04 15:28 [PATCH] turn off irqdebug by default on Altix Martin Hicks
@ 2004-02-04 19:04 ` Chen, Kenneth W
2004-02-04 19:08 ` Martin Hicks
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Chen, Kenneth W @ 2004-02-04 19:04 UTC (permalink / raw)
To: linux-ia64
This is really confusing with two parameters that do the same thing.
What will happen if someone pass noirqdebug and irqdebug at the same
time?
What's wrong with boot time parameter of noirqdebug?
- Ken
-----Original Message-----
From: linux-ia64-owner@vger.kernel.org
[mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of Martin Hicks
Sent: Wednesday, February 04, 2004 7:29 AM
To: linux-ia64@vger.kernel.org; akpm@osdl.org
Cc: steiner@sgi.com; jbarnes@sgi.com
Subject: [PATCH] turn off irqdebug by default on Altix
Hello,
A while ago there was a discussion, started by Jack Steiner, about a hot
cache line in the irqdebug code that makes booting on large machines
very slow. The original thread can be found here:
http://marc.theaimsgroup.com/?l=linux-kernel&m\x106858073229831&w=2
The end result was that it was felt that irqdebug sould remain on by
default on small machines, and that large machines should turn it off be
default if it was causing a problem.
Here is a patch that turns it off for sn2. It applies cleanly against
the linux-2.5 bk tree from this morning.
mh
--
Martin Hicks || mort@bork.org || PGP/GnuPG: 0x4C7F2BEE
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] turn off irqdebug by default on Altix
2004-02-04 15:28 [PATCH] turn off irqdebug by default on Altix Martin Hicks
2004-02-04 19:04 ` Chen, Kenneth W
@ 2004-02-04 19:08 ` Martin Hicks
2004-02-04 19:17 ` Andrew Morton
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Martin Hicks @ 2004-02-04 19:08 UTC (permalink / raw)
To: linux-ia64
On Wed, Feb 04, 2004 at 11:04:43AM -0800, Chen, Kenneth W wrote:
> This is really confusing with two parameters that do the same thing.
> What will happen if someone pass noirqdebug and irqdebug at the same
> time?
>
> What's wrong with boot time parameter of noirqdebug?
It just means that if you forget to pass this option then your machine
never boots. Well, it will boot. Eventually. Jack is a patient man,
and he said he never waited long enough for it to boot.
mh
--
Martin Hicks || mort@bork.org || PGP/GnuPG: 0x4C7F2BEE
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] turn off irqdebug by default on Altix
2004-02-04 15:28 [PATCH] turn off irqdebug by default on Altix Martin Hicks
2004-02-04 19:04 ` Chen, Kenneth W
2004-02-04 19:08 ` Martin Hicks
@ 2004-02-04 19:17 ` Andrew Morton
2004-02-04 19:40 ` Jack Steiner
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Andrew Morton @ 2004-02-04 19:17 UTC (permalink / raw)
To: linux-ia64
Martin Hicks <mort@bork.org> wrote:
>
> A while ago there was a discussion, started by Jack Steiner, about a hot
> cache line in the irqdebug code that makes booting on large machines
> very slow. The original thread can be found here:
>
> http://marc.theaimsgroup.com/?l=linux-kernel&m\x106858073229831&w=2
>
> The end result was that it was felt that irqdebug sould remain on by
> default on small machines, and that large machines should turn it off be
> default if it was causing a problem.
Yup.
> Here is a patch that turns it off for sn2. It applies cleanly against
> the linux-2.5 bk tree from this morning.
I think it would be better to kill noirqdebug altogether, and to simply
require irqdebug=N.
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] turn off irqdebug by default on Altix
2004-02-04 15:28 [PATCH] turn off irqdebug by default on Altix Martin Hicks
` (2 preceding siblings ...)
2004-02-04 19:17 ` Andrew Morton
@ 2004-02-04 19:40 ` Jack Steiner
2004-02-04 21:32 ` Chen, Kenneth W
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Jack Steiner @ 2004-02-04 19:40 UTC (permalink / raw)
To: linux-ia64
On Wed, Feb 04, 2004 at 02:08:05PM -0500, Martin Hicks wrote:
>
>
> On Wed, Feb 04, 2004 at 11:04:43AM -0800, Chen, Kenneth W wrote:
> > This is really confusing with two parameters that do the same thing.
> > What will happen if someone pass noirqdebug and irqdebug at the same
> > time?
> >
> > What's wrong with boot time parameter of noirqdebug?
>
> It just means that if you forget to pass this option then your machine
> never boots. Well, it will boot. Eventually. Jack is a patient man,
> and he said he never waited long enough for it to boot.
Actually, I determined that a big system would never boot. Each cpu would have to
do about 1500usec of work every 1000usec. (highly dependent on the
system interconnect).
Each clock tick, the desc->irq_count is incremented by each cpu. On a big
system, this creates an incredibly hot cache line. On a big system, it takes
more than a millisecond to do the update. At that point, another clock
interrupt is already pending. Forward progress stops.
Another possible approach might be to special case note_interrupt() to
skip the update for the timer tick. (I havent actually looked at the code, but
this should be possible).
>
> mh
>
> --
> Martin Hicks || mort@bork.org || PGP/GnuPG: 0x4C7F2BEE
--
Thanks
Jack Steiner (steiner@sgi.com) 651-683-5302
Principal Engineer SGI - Silicon Graphics, Inc.
^ permalink raw reply [flat|nested] 9+ messages in thread* RE: [PATCH] turn off irqdebug by default on Altix
2004-02-04 15:28 [PATCH] turn off irqdebug by default on Altix Martin Hicks
` (3 preceding siblings ...)
2004-02-04 19:40 ` Jack Steiner
@ 2004-02-04 21:32 ` Chen, Kenneth W
2004-02-05 18:37 ` Martin Hicks
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Chen, Kenneth W @ 2004-02-04 21:32 UTC (permalink / raw)
To: linux-ia64
Don't take me wrong, I do want these irqdebug off too on one of our
large performance setup here as well :-) Even reading from variable
noirqdebug is costing us some cache misses. Can this be made compile
time option? So when it is off, it is really off unless someone need to
turn this debug option on and off dynamically.
- Ken
-----Original Message-----
From: linux-ia64-owner@vger.kernel.org
[mailto:linux-ia64-owner@vger.kernel.org] On Behalf Of Jack Steiner
Sent: Wednesday, February 04, 2004 11:41 AM
To: Martin Hicks
Cc: Chen, Kenneth W; linux-ia64@vger.kernel.org; akpm@osdl.org;
jbarnes@sgi.com
Subject: Re: [PATCH] turn off irqdebug by default on Altix
On Wed, Feb 04, 2004 at 02:08:05PM -0500, Martin Hicks wrote:
>
>
> On Wed, Feb 04, 2004 at 11:04:43AM -0800, Chen, Kenneth W wrote:
> > This is really confusing with two parameters that do the same thing.
> > What will happen if someone pass noirqdebug and irqdebug at the same
> > time?
> >
> > What's wrong with boot time parameter of noirqdebug?
>
> It just means that if you forget to pass this option then your machine
> never boots. Well, it will boot. Eventually. Jack is a patient man,
> and he said he never waited long enough for it to boot.
Actually, I determined that a big system would never boot. Each cpu
would have to
do about 1500usec of work every 1000usec. (highly dependent on the
system interconnect).
Each clock tick, the desc->irq_count is incremented by each cpu. On a
big
system, this creates an incredibly hot cache line. On a big system, it
takes
more than a millisecond to do the update. At that point, another clock
interrupt is already pending. Forward progress stops.
Another possible approach might be to special case note_interrupt() to
skip the update for the timer tick. (I havent actually looked at the
code, but
this should be possible).
>
> mh
>
> --
> Martin Hicks || mort@bork.org || PGP/GnuPG: 0x4C7F2BEE
--
Thanks
Jack Steiner (steiner@sgi.com) 651-683-5302
Principal Engineer SGI - Silicon Graphics, Inc.
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] turn off irqdebug by default on Altix
2004-02-04 15:28 [PATCH] turn off irqdebug by default on Altix Martin Hicks
` (4 preceding siblings ...)
2004-02-04 21:32 ` Chen, Kenneth W
@ 2004-02-05 18:37 ` Martin Hicks
2004-02-10 22:43 ` David Mosberger
2004-02-11 21:19 ` Martin Hicks
7 siblings, 0 replies; 9+ messages in thread
From: Martin Hicks @ 2004-02-05 18:37 UTC (permalink / raw)
To: linux-ia64
[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]
On Wed, Feb 04, 2004 at 01:40:47PM -0600, Jack Steiner wrote:
> On Wed, Feb 04, 2004 at 02:08:05PM -0500, Martin Hicks wrote:
> >
> >
> > On Wed, Feb 04, 2004 at 11:04:43AM -0800, Chen, Kenneth W wrote:
> > > This is really confusing with two parameters that do the same thing.
> > > What will happen if someone pass noirqdebug and irqdebug at the same
> > > time?
> > >
> > > What's wrong with boot time parameter of noirqdebug?
> >
> > It just means that if you forget to pass this option then your machine
> > never boots. Well, it will boot. Eventually. Jack is a patient man,
> > and he said he never waited long enough for it to boot.
>
> Actually, I determined that a big system would never boot. Each cpu would have to
> do about 1500usec of work every 1000usec. (highly dependent on the
> system interconnect).
>
> Each clock tick, the desc->irq_count is incremented by each cpu. On a big
> system, this creates an incredibly hot cache line. On a big system, it takes
> more than a millisecond to do the update. At that point, another clock
> interrupt is already pending. Forward progress stops.
>
> Another possible approach might be to special case note_interrupt() to
> skip the update for the timer tick. (I havent actually looked at the code, but
> this should be possible).
Okay. This is certainly less messy, and is IA64 specific.
How about this patch? I just tested it on a 512p machine and it booted.
mh
--
Martin Hicks || Silicon Graphics Inc. || mort@sgi.com
[-- Attachment #2: no-timer-interrupt-in-note-interrupt.diff --]
[-- Type: text/plain, Size: 1080 bytes --]
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
# ChangeSet 1.1543 -> 1.1544
# arch/ia64/kernel/irq.c 1.34 -> 1.35
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 04/02/04 mort@green.i.bork.org 1.1544
# Don't debug the timer interrupt in note_interrupt(). This
# should allow forward progress on large machine with irqdebug
# enabled.
# --------------------------------------------
#
diff -Nru a/arch/ia64/kernel/irq.c b/arch/ia64/kernel/irq.c
--- a/arch/ia64/kernel/irq.c Wed Feb 4 15:43:12 2004
+++ b/arch/ia64/kernel/irq.c Wed Feb 4 15:43:12 2004
@@ -311,6 +311,10 @@
*/
static void note_interrupt(int irq, irq_desc_t *desc, irqreturn_t action_ret)
{
+ /* Don't debug the timer interrupt */
+ if (irq == IA64_TIMER_VECTOR)
+ return;
+
if (action_ret != IRQ_HANDLED) {
desc->irqs_unhandled++;
if (action_ret != IRQ_NONE)
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] turn off irqdebug by default on Altix
2004-02-04 15:28 [PATCH] turn off irqdebug by default on Altix Martin Hicks
` (5 preceding siblings ...)
2004-02-05 18:37 ` Martin Hicks
@ 2004-02-10 22:43 ` David Mosberger
2004-02-11 21:19 ` Martin Hicks
7 siblings, 0 replies; 9+ messages in thread
From: David Mosberger @ 2004-02-10 22:43 UTC (permalink / raw)
To: linux-ia64
>>>>> On Thu, 5 Feb 2004 13:37:28 -0500, Martin Hicks <mort@sgi.com> said:
Martin> How about this patch? I just tested it on a 512p machine
Martin> and it booted.
I suspect a better fix would be to change do_IRQ() such that it
doesn't call note_interrupt() for IRQ_PER_CPU interrupts. For per-CPU
interrupts, the note_interrupt() code is racy and rather bogus anyhow.
--david
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH] turn off irqdebug by default on Altix
2004-02-04 15:28 [PATCH] turn off irqdebug by default on Altix Martin Hicks
` (6 preceding siblings ...)
2004-02-10 22:43 ` David Mosberger
@ 2004-02-11 21:19 ` Martin Hicks
7 siblings, 0 replies; 9+ messages in thread
From: Martin Hicks @ 2004-02-11 21:19 UTC (permalink / raw)
To: linux-ia64
[-- Attachment #1: Type: text/plain, Size: 642 bytes --]
On Tue, Feb 10, 2004 at 02:43:14PM -0800, David Mosberger wrote:
> >>>>> On Thu, 5 Feb 2004 13:37:28 -0500, Martin Hicks <mort@sgi.com> said:
>
> Martin> How about this patch? I just tested it on a 512p machine
> Martin> and it booted.
>
> I suspect a better fix would be to change do_IRQ() such that it
> doesn't call note_interrupt() for IRQ_PER_CPU interrupts. For per-CPU
> interrupts, the note_interrupt() code is racy and rather bogus anyhow.
Okay....Here it is.
I didn't test it on a big machine yet, but it certainly seems like it
takes care of the problem.
mh
--
Martin Hicks || mort@bork.org || PGP/GnuPG: 0x4C7F2BEE
[-- Attachment #2: no-note-interrupt.patch --]
[-- Type: text/plain, Size: 977 bytes --]
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
# ChangeSet 1.1609 -> 1.1610
# arch/ia64/kernel/irq.c 1.34 -> 1.35
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 04/02/11 mort@green.i.bork.org 1.1610
# Don't call note_interrupt() for per-cpu interrupts.
# --------------------------------------------
#
diff -Nru a/arch/ia64/kernel/irq.c b/arch/ia64/kernel/irq.c
--- a/arch/ia64/kernel/irq.c Wed Feb 11 16:17:22 2004
+++ b/arch/ia64/kernel/irq.c Wed Feb 11 16:17:22 2004
@@ -465,8 +465,6 @@
desc->handler->ack(irq);
action_ret = handle_IRQ_event(irq, regs, desc->action);
desc->handler->end(irq);
- if (!noirqdebug)
- note_interrupt(irq, desc, action_ret);
} else {
spin_lock(&desc->lock);
desc->handler->ack(irq);
^ permalink raw reply [flat|nested] 9+ messages in thread