[ANN] IRQ Latency tool 0.1.3 release.

[ANN] IRQ Latency tool 0.1.3 release.

[Iain Sandoe]

Hi list...

Well, here is a tool that should allow identification of rogue IRQ blocks
(ints held off for a long time).

This is important to those of us trying to do audio on LinuxPPC - it would
be a great shame if we couldn't succeed where MacOS 9 already performs...

Many thanks to Jun Sun for the code that was the inspiration (and basis) of
this...

It is well-tested on my system but needs other input...

It needs some development input before it's going to work on Gemini/APUS/8xx
- because the code is too hairy to do without a system to test on (well for
me it is anyway :-)

Take a look, have a go, tell me what happens....

http://www.drfruitcake.com/linux/irq_blk.html

BTW: on my system ll_rw_blk.c gets up to 750 us quite often when the system
is loaded... this **IS** a potential problem for audio use.

log entry : 3  (16  history entries)
        Peak Block Time  :        430.0 us  (    7185) [ OK ]
        startFilename    : ll_rw_blk.c                    Line : 167
        endFileName      : ll_rw_blk.c                    Line : 179

I still haven't found the "hard-block" that is effecting the overall audio
latency performance... ah well, back to the drawing board...

ciao,
Iain.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Geert Uytterhoeven]

On Mon, 24 Jul 2000, Iain Sandoe wrote:
> BTW: on my system ll_rw_blk.c gets up to 750 us quite often when the system
> is loaded... this **IS** a potential problem for audio use.
>
> log entry : 3  (16  history entries)
>         Peak Block Time  :        430.0 us  (    7185) [ OK ]
>         startFilename    : ll_rw_blk.c                    Line : 167
>         endFileName      : ll_rw_blk.c                    Line : 179
>
> I still haven't found the "hard-block" that is effecting the overall audio
> latency performance... ah well, back to the drawing board...

Do you have IDE? The IDE driver turned out to be the major interrupt messing
problem on m68k, since it can disable interrupts for quite a while.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Jun Sun]

Geert Uytterhoeven wrote:
>
> On Mon, 24 Jul 2000, Iain Sandoe wrote:
> > BTW: on my system ll_rw_blk.c gets up to 750 us quite often when the system
> > is loaded... this **IS** a potential problem for audio use.
> >
> > log entry : 3  (16  history entries)
> >         Peak Block Time  :        430.0 us  (    7185) [ OK ]
> >         startFilename    : ll_rw_blk.c                    Line : 167
> >         endFileName      : ll_rw_blk.c                    Line : 179
> >
> > I still haven't found the "hard-block" that is effecting the overall audio
> > latency performance... ah well, back to the drawing board...
>
> Do you have IDE? The IDE driver turned out to be the major interrupt messing
> problem on m68k, since it can disable interrupts for quite a while.
>
> Gr{oetje,eeting}s,
>
>                                                 Geert

The IDE driver in v2.2.xx has a run-away cli(), which causes long
interrupt blocking times.  See my postings in linux-kernel mailing list
a while back (around March), if you can still find them.  A IDE driver
guys suggested a fix, but I am not sure if it there yet.  The function
is  do_rw_dis().

log entry : 3
(unplug_device in ll_rw_blk.c, calling dev->request_fn(),
 do_rw_disk() in ide-disk.c)
        blockingTime     : 39955
        startFileName    : ide-disk.c
        startFileLine    : 390
        startCount       : 3433912489
        endFileName      : ll_rw_blk.c
        endFileLine      : 172
        endCount         : 3433952444
        unused1,2        : 0, 0

I believe there is a user-land workaround, which involves using "hdparm"
to change some bits.

Jun

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Benjamin Herrenschmidt]

>> I still haven't found the "hard-block" that is effecting the overall audio
>> latency performance... ah well, back to the drawing board...
>
>Do you have IDE? The IDE driver turned out to be the major interrupt messing
>problem on m68k, since it can disable interrupts for quite a while.

We have some great IRQ blockers in Linux currently. IDE is one,
especially in PIO mode where it disables IRQs by default during
transferts (AFAIK), but I think also in the fbdevs where, I think, a
bunch of lenghtly things like scrolling are done with IRQs disabled.

This has caused me troubles in the past since the PMU on new machines is
very sensitive to timeouts. If IRQ get blocked too long while an
asynchronous PMU request is beeing serviced, the PMU will time out and
shut down the machine abruptly. This happened several times in the past
during IDE probe on PowerBooks while some backlight or ADB requests where
beeing handled.

In my latest kernels, I had to add a pmu_safe_poll() function that polls
until the PMU has finished servicing any pending request in order to
avoid crashes in xmon while scrolling the screen for example.

I'm wondering if we could eventually turn ON the IDE unmask interrupt
options on a per-arch or per-machine basis. It's really not necessary
with recent chipsets anyway.

I don't know how the fbdev case can be improved, I beleive James has some
plans about that for 2.5...

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Geert Uytterhoeven]

On Tue, 25 Jul 2000, Benjamin Herrenschmidt wrote:
> >> I still haven't found the "hard-block" that is effecting the overall audio
> >> latency performance... ah well, back to the drawing board...
> >
> >Do you have IDE? The IDE driver turned out to be the major interrupt messing
> >problem on m68k, since it can disable interrupts for quite a while.
>
> We have some great IRQ blockers in Linux currently. IDE is one,
> especially in PIO mode where it disables IRQs by default during
> transferts (AFAIK), but I think also in the fbdevs where, I think, a
> bunch of lenghtly things like scrolling are done with IRQs disabled.

> I don't know how the fbdev case can be improved, I beleive James has some
> plans about that for 2.5...

There's indeed a global lock on the console, to prevent multiple accesses to
the console at once. This solves some problems, one being that some graphics
hardware crashes when both the CPU and the graphics accelerator touch the video
RAM at the same time.

Interrupts in fbdev would be nice for some high-end hardware, which can
trigger an interrupt when a bunch of graphics processing was done.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Takashi Oe]

On Wed, 26 Jul 2000, Geert Uytterhoeven wrote:

> There's indeed a global lock on the console, to prevent multiple accesses to
> the console at once. This solves some problems, one being that some graphics
> hardware crashes when both the CPU and the graphics accelerator touch the video
> RAM at the same time.

Would it be difficult to make the lock per fbdev thing?  If the
CPU/accelerator clash is the only reason for the global lock, we can
certainly remove that for hardware which doesn't have that kind of
problem, can't we?


Takashi Oe


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Geert Uytterhoeven]

On Fri, 28 Jul 2000, Takashi Oe wrote:
> On Wed, 26 Jul 2000, Geert Uytterhoeven wrote:
> > There's indeed a global lock on the console, to prevent multiple accesses to
> > the console at once. This solves some problems, one being that some graphics
> > hardware crashes when both the CPU and the graphics accelerator touch the video
> > RAM at the same time.
>
> Would it be difficult to make the lock per fbdev thing?  If the
> CPU/accelerator clash is the only reason for the global lock, we can
> certainly remove that for hardware which doesn't have that kind of
> problem, can't we?

There could indeed be separate locks per fbdev. But the global console lock
(console_lock) would still be there. It is needed because the console subsystem
is not re-entrant (yet). Since fbdev is called from the console subsystem, it
runs under the same lock (i.e. with interrupts disabled).

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [linux-fbdev] Re: [ANN] IRQ Latency tool 0.1.3 release.

[James Simmons]

> > Would it be difficult to make the lock per fbdev thing?  If the
> > CPU/accelerator clash is the only reason for the global lock, we can
> > certainly remove that for hardware which doesn't have that kind of
> > problem, can't we?
>
> There could indeed be separate locks per fbdev. But the global console lock
> (console_lock) would still be there. It is needed because the console subsystem
> is not re-entrant (yet). Since fbdev is called from the console subsystem, it
> runs under the same lock (i.e. with interrupts disabled).

I will be testing for re-entrant ability next week with code I have in
CVS. Their should be a semaphore for each struct console. Also their
should be a semaphore for each VT where on semephore is shared between the
VT console and struct console. Why a semephore. Eventually we will run
into hardware that runs only by interrupts. So a hardware independent why
to lock the console system is needed.

Q: Why did they deprecate a.out support in linux?
A: Because a nasty coff is bad for your elf.

James Simmons  [jsimmons@linux-fbdev.org]               ____/|
fbdev/console/gfx developer                             \ o.O|
http://www.linux-fbdev.org                               =(_)=
http://linuxgfx.sourceforge.net                            U
http://linuxconsole.sourceforge.net

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Iain Sandoe]

On Tue, Jul 25, 2000, Geert Uytterhoeven wrote:
> On Mon, 24 Jul 2000, Iain Sandoe wrote:
[...]
>> I still haven't found the "hard-block" that is effecting the overall audio
>> latency performance... ah well, back to the drawing board...
>
> Do you have IDE? The IDE driver turned out to be the major interrupt messing
> problem on m68k, since it can disable interrupts for quite a while.

I do, but the audio latency test (which is what promoted this interest in
doing the IRQ measurement) doesn't stress the IDE (I only have a CDROM on
the IDE bus).  All the disk activity is via the AppleSCSI card.

However, I would like others with different configurations to try out the
test(s)... it's not much use if acceptable performance sound only happens on
*my* machine :-)

It's also true that all the 'little' IRQ blocks do add up - as I said when
my system is stressed up to 5% of the total elapsed time can be blocked... a
good motivation for driver writers to tune things methinks...

Iain.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Iain Sandoe]

>>> I still haven't found the "hard-block" that is effecting the overall audio
>>> latency performance... ah well, back to the drawing board...
>>
>>Do you have IDE? The IDE driver turned out to be the major interrupt messing
>>problem on m68k, since it can disable interrupts for quite a while.
>
> We have some great IRQ blockers in Linux currently. IDE is one,
> especially in PIO mode where it disables IRQs by default during

[...]

Does this mean that there are already other tools to measure the IRQ
blocking?

To go back to the original problem:

running a sched_fifo based audio latency test (Benno Senoner's x86 version
with PPC patches) I was seeing 8-10ms 'inexplicable' scheduling spikes.

Someone postulated that this could be caused by IRQ blocking.

Hence the tool.  However, on my set-up, using the same latency test the
spikes show - but there are no IRQ blocks of that length showing in the
measurement.

I therefore have to conclude that:

(a) The tool is missing something - please peruse and suggest if you have
time... I think that most stuff is covered (and I did cover the DataAccess &
InstructionAccess exceptions - although that is currently not enabled in the
patch I've posted).

(b) The effect is caused by some other phenomenon (which is my current
guess).

So, I'm going to have another look at the latency test to see what I can do
to narrow down the source of the problem (including checking what the test
does :-)

Iain,

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: [ANN] IRQ Latency tool 0.1.3 release.

[Iain Sandoe]

On  Wed, Jul 26, 2000, Geert Uytterhoeven wrote:
> On Tue, 25 Jul 2000, Benjamin Herrenschmidt wrote:
>> >> I still haven't found the "hard-block" that is effecting the overall audio
>> >> latency performance... ah well, back to the drawing board...
>> >
>> >Do you have IDE? The IDE driver turned out to be the major interrupt messing
>> >problem on m68k, since it can disable interrupts for quite a while.
>>
>> We have some great IRQ blockers in Linux currently. IDE is one,
>> especially in PIO mode where it disables IRQs by default during
>> transferts (AFAIK), but I think also in the fbdevs where, I think, a
>> bunch of lenghtly things like scrolling are done with IRQs disabled.
>
>> I don't know how the fbdev case can be improved, I beleive James has some
>> plans about that for 2.5...
>
> There's indeed a global lock on the console, to prevent multiple accesses to
> the console at once. This solves some problems, one being that some graphics
> hardware crashes when both the CPU and the graphics accelerator touch the
video
> RAM at the same time.
>
> Interrupts in fbdev would be nice for some high-end hardware, which can
> trigger an interrupt when a bunch of graphics processing was done.

but, unless the test code is c**p, that is not what's causing my problem...

on the x86 (over on the linux-audio-dev list) they've come to the conclusion
that the spikes are caused by APM introduced at 2.2.16.

I'm not very aware of whether this is an issue on ppc (I though APM was an
x86-specific feature).

please correct me if I'm wrong..

Iain.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/