* 2.6 vs 2.4 regression when running gnomemeeting
@ 2003-12-19 20:11 Christian Meder
2003-12-19 20:32 ` William Lee Irwin III
2003-12-20 19:34 ` Marc Schiffbauer
0 siblings, 2 replies; 35+ messages in thread
From: Christian Meder @ 2003-12-19 20:11 UTC (permalink / raw)
To: linux-kernel
Hi,
I've got a longstanding regression in gnomemeeting usage when switching
between 2.4 and 2.6 kernels.
Phenomenon:
Without load gnomemeeting VOIP connections are fine. As soon as some
load like a kernel compile is put on the laptop the gnomemeeting audio
stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
even the slightest distortion in the audio stream under load. I played
around with different nice levels with no success. The problem persisted
during the whole 2.6.0-test series no matter whether I used -mm kernels
or pristine Linus kernels. Even when nicing the kernel compile to +19
the distortions start right away. I tried Nick Piggin's scheduler which
fared slightly better after changing the nice level of gnomemeeting to
-10 but it's still a far cry from the 2.4.2x feeling without any
fiddling with nice values.
Any hints where to start looking are greatly appreciated.
Christian Meder
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-19 20:11 2.6 vs 2.4 regression when running gnomemeeting Christian Meder
@ 2003-12-19 20:32 ` William Lee Irwin III
2003-12-19 23:30 ` Christian Meder
2003-12-20 19:34 ` Marc Schiffbauer
1 sibling, 1 reply; 35+ messages in thread
From: William Lee Irwin III @ 2003-12-19 20:32 UTC (permalink / raw)
To: Christian Meder; +Cc: linux-kernel
On Fri, Dec 19, 2003 at 09:11:50PM +0100, Christian Meder wrote:
> I've got a longstanding regression in gnomemeeting usage when switching
> between 2.4 and 2.6 kernels.
> Phenomenon:
> Without load gnomemeeting VOIP connections are fine. As soon as some
> load like a kernel compile is put on the laptop the gnomemeeting audio
> stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
> even the slightest distortion in the audio stream under load. I played
> around with different nice levels with no success. The problem persisted
> during the whole 2.6.0-test series no matter whether I used -mm kernels
> or pristine Linus kernels. Even when nicing the kernel compile to +19
> the distortions start right away. I tried Nick Piggin's scheduler which
> fared slightly better after changing the nice level of gnomemeeting to
> -10 but it's still a far cry from the 2.4.2x feeling without any
> fiddling with nice values.
> Any hints where to start looking are greatly appreciated.
Please instrument your workload with the following, and send logs of the
output (preferably compressed) to me and possibly others:
top b d 5
vmstat 5
while true; do cat /proc/vmstat; sleep 5; done
while true; do cat /proc/meminfo; sleep 5; done
A good way to log commands like this is:
(command) > /home/foo.log.1 2>&1 &
where parentheses surround the command in the actual shell input.
-- wli
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-19 20:32 ` William Lee Irwin III
@ 2003-12-19 23:30 ` Christian Meder
2003-12-20 0:21 ` Nick Piggin
0 siblings, 1 reply; 35+ messages in thread
From: Christian Meder @ 2003-12-19 23:30 UTC (permalink / raw)
To: William Lee Irwin III; +Cc: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 2028 bytes --]
On Fri, 2003-12-19 at 21:32, William Lee Irwin III wrote:
> On Fri, Dec 19, 2003 at 09:11:50PM +0100, Christian Meder wrote:
> > I've got a longstanding regression in gnomemeeting usage when switching
> > between 2.4 and 2.6 kernels.
> > Phenomenon:
> > Without load gnomemeeting VOIP connections are fine. As soon as some
> > load like a kernel compile is put on the laptop the gnomemeeting audio
> > stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
> > even the slightest distortion in the audio stream under load. I played
> > around with different nice levels with no success. The problem persisted
> > during the whole 2.6.0-test series no matter whether I used -mm kernels
> > or pristine Linus kernels. Even when nicing the kernel compile to +19
> > the distortions start right away. I tried Nick Piggin's scheduler which
> > fared slightly better after changing the nice level of gnomemeeting to
> > -10 but it's still a far cry from the 2.4.2x feeling without any
> > fiddling with nice values.
> > Any hints where to start looking are greatly appreciated.
>
> Please instrument your workload with the following, and send logs of the
> output (preferably compressed) to me and possibly others:
>
> top b d 5
> vmstat 5
> while true; do cat /proc/vmstat; sleep 5; done
> while true; do cat /proc/meminfo; sleep 5; done
>
> A good way to log commands like this is:
>
> (command) > /home/foo.log.1 2>&1 &
>
> where parentheses surround the command in the actual shell input.
Hi,
I've attached the tarred output of a gnomemeeting run without load and
without distortions and another tarred output of a gnomemeeting run
while compiling a kernel with severe distortions in the audio stream.
Christian Meder
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
[-- Attachment #2: noload.tar.bz2 --]
[-- Type: application/x-bzip, Size: 8350 bytes --]
[-- Attachment #3: load.tar.bz2 --]
[-- Type: application/x-bzip, Size: 9520 bytes --]
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-19 23:30 ` Christian Meder
@ 2003-12-20 0:21 ` Nick Piggin
2003-12-20 0:37 ` Christian Meder
0 siblings, 1 reply; 35+ messages in thread
From: Nick Piggin @ 2003-12-20 0:21 UTC (permalink / raw)
To: Christian Meder; +Cc: William Lee Irwin III, linux-kernel
Christian Meder wrote:
>On Fri, 2003-12-19 at 21:32, William Lee Irwin III wrote:
>
>>On Fri, Dec 19, 2003 at 09:11:50PM +0100, Christian Meder wrote:
>>
>>>I've got a longstanding regression in gnomemeeting usage when switching
>>>between 2.4 and 2.6 kernels.
>>>Phenomenon:
>>>Without load gnomemeeting VOIP connections are fine. As soon as some
>>>load like a kernel compile is put on the laptop the gnomemeeting audio
>>>stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
>>>even the slightest distortion in the audio stream under load. I played
>>>around with different nice levels with no success. The problem persisted
>>>during the whole 2.6.0-test series no matter whether I used -mm kernels
>>>or pristine Linus kernels. Even when nicing the kernel compile to +19
>>>the distortions start right away. I tried Nick Piggin's scheduler which
>>>fared slightly better after changing the nice level of gnomemeeting to
>>>-10 but it's still a far cry from the 2.4.2x feeling without any
>>>fiddling with nice values.
>>>Any hints where to start looking are greatly appreciated.
>>>
>>Please instrument your workload with the following, and send logs of the
>>output (preferably compressed) to me and possibly others:
>>
>>top b d 5
>>vmstat 5
>>while true; do cat /proc/vmstat; sleep 5; done
>>while true; do cat /proc/meminfo; sleep 5; done
>>
>>A good way to log commands like this is:
>>
>>(command) > /home/foo.log.1 2>&1 &
>>
>>where parentheses surround the command in the actual shell input.
>>
>
>Hi,
>
>I've attached the tarred output of a gnomemeeting run without load and
>without distortions and another tarred output of a gnomemeeting run
>while compiling a kernel with severe distortions in the audio stream.
>
You're getting a lot fewer interrupts in the loaded case. Maybe its
the sound card driver that has the regression from 2.4? It could be
that 2.6 allows a smaller sound fragment size which is more stressful.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 0:21 ` Nick Piggin
@ 2003-12-20 0:37 ` Christian Meder
2003-12-20 0:48 ` Nick Piggin
0 siblings, 1 reply; 35+ messages in thread
From: Christian Meder @ 2003-12-20 0:37 UTC (permalink / raw)
To: Nick Piggin; +Cc: William Lee Irwin III, linux-kernel
On Sat, 2003-12-20 at 01:21, Nick Piggin wrote:
> Christian Meder wrote:
>
> >On Fri, 2003-12-19 at 21:32, William Lee Irwin III wrote:
> >
> >>On Fri, Dec 19, 2003 at 09:11:50PM +0100, Christian Meder wrote:
> >>
> >>>I've got a longstanding regression in gnomemeeting usage when switching
> >>>between 2.4 and 2.6 kernels.
> >>>Phenomenon:
> >>>Without load gnomemeeting VOIP connections are fine. As soon as some
> >>>load like a kernel compile is put on the laptop the gnomemeeting audio
> >>>stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
> >>>even the slightest distortion in the audio stream under load. I played
> >>>around with different nice levels with no success. The problem persisted
> >>>during the whole 2.6.0-test series no matter whether I used -mm kernels
> >>>or pristine Linus kernels. Even when nicing the kernel compile to +19
> >>>the distortions start right away. I tried Nick Piggin's scheduler which
> >>>fared slightly better after changing the nice level of gnomemeeting to
> >>>-10 but it's still a far cry from the 2.4.2x feeling without any
> >>>fiddling with nice values.
> >>>Any hints where to start looking are greatly appreciated.
> >>>
> >>Please instrument your workload with the following, and send logs of the
> >>output (preferably compressed) to me and possibly others:
> >>
> >>top b d 5
> >>vmstat 5
> >>while true; do cat /proc/vmstat; sleep 5; done
> >>while true; do cat /proc/meminfo; sleep 5; done
> >>
> >>A good way to log commands like this is:
> >>
> >>(command) > /home/foo.log.1 2>&1 &
> >>
> >>where parentheses surround the command in the actual shell input.
> >>
> >
> >Hi,
> >
> >I've attached the tarred output of a gnomemeeting run without load and
> >without distortions and another tarred output of a gnomemeeting run
> >while compiling a kernel with severe distortions in the audio stream.
> >
>
> You're getting a lot fewer interrupts in the loaded case. Maybe its
> the sound card driver that has the regression from 2.4? It could be
> that 2.6 allows a smaller sound fragment size which is more stressful.
>
Well I had the same problem with the OSS driver on 2.6. Now I use the
ALSA driver because I thought that could possibly improve things. The
ALSA driver is better indeed but it doesn't change this particular
phenomenon. Additionally I'd guess that the latest ALSA driver in 2.4
and 2.6 doesn't differ significantly and 2.4.2x with the latest ALSA
works great while 2.6 doesn't.
Christian Meder
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 0:37 ` Christian Meder
@ 2003-12-20 0:48 ` Nick Piggin
2003-12-20 1:11 ` Christian Meder
0 siblings, 1 reply; 35+ messages in thread
From: Nick Piggin @ 2003-12-20 0:48 UTC (permalink / raw)
To: Christian Meder; +Cc: William Lee Irwin III, linux-kernel
Christian Meder wrote:
>On Sat, 2003-12-20 at 01:21, Nick Piggin wrote:
>
>>Christian Meder wrote:
>>
>>
>>>On Fri, 2003-12-19 at 21:32, William Lee Irwin III wrote:
>>>
>>>
>>>>On Fri, Dec 19, 2003 at 09:11:50PM +0100, Christian Meder wrote:
>>>>
>>>>
>>>>>I've got a longstanding regression in gnomemeeting usage when switching
>>>>>between 2.4 and 2.6 kernels.
>>>>>Phenomenon:
>>>>>Without load gnomemeeting VOIP connections are fine. As soon as some
>>>>>load like a kernel compile is put on the laptop the gnomemeeting audio
>>>>>stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
>>>>>even the slightest distortion in the audio stream under load. I played
>>>>>around with different nice levels with no success. The problem persisted
>>>>>during the whole 2.6.0-test series no matter whether I used -mm kernels
>>>>>or pristine Linus kernels. Even when nicing the kernel compile to +19
>>>>>the distortions start right away. I tried Nick Piggin's scheduler which
>>>>>fared slightly better after changing the nice level of gnomemeeting to
>>>>>-10 but it's still a far cry from the 2.4.2x feeling without any
>>>>>fiddling with nice values.
>>>>>Any hints where to start looking are greatly appreciated.
>>>>>
>>>>>
>>>>Please instrument your workload with the following, and send logs of the
>>>>output (preferably compressed) to me and possibly others:
>>>>
>>>>top b d 5
>>>>vmstat 5
>>>>while true; do cat /proc/vmstat; sleep 5; done
>>>>while true; do cat /proc/meminfo; sleep 5; done
>>>>
>>>>A good way to log commands like this is:
>>>>
>>>>(command) > /home/foo.log.1 2>&1 &
>>>>
>>>>where parentheses surround the command in the actual shell input.
>>>>
>>>>
>>>Hi,
>>>
>>>I've attached the tarred output of a gnomemeeting run without load and
>>>without distortions and another tarred output of a gnomemeeting run
>>>while compiling a kernel with severe distortions in the audio stream.
>>>
>>>
>>You're getting a lot fewer interrupts in the loaded case. Maybe its
>>the sound card driver that has the regression from 2.4? It could be
>>that 2.6 allows a smaller sound fragment size which is more stressful.
>>
>>
>
>Well I had the same problem with the OSS driver on 2.6. Now I use the
>ALSA driver because I thought that could possibly improve things. The
>ALSA driver is better indeed but it doesn't change this particular
>phenomenon. Additionally I'd guess that the latest ALSA driver in 2.4
>and 2.6 doesn't differ significantly and 2.4.2x with the latest ALSA
>works great while 2.6 doesn't.
>
>
Sounds reasonable. Maybe its large interrupt or scheduling latency
caused somewhere else. Does disk activity alone cause a problem?
find / -type f | xargs cat > /dev/null
how about
dd if=/dev/zero of=./deleteme bs=1M count=256
You said it faired slightly better with my scheduler when renicing
gnome meeting to -10. How much better is that?
whats your /proc/cpuinfo?
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 0:48 ` Nick Piggin
@ 2003-12-20 1:11 ` Christian Meder
2003-12-20 1:26 ` Nick Piggin
0 siblings, 1 reply; 35+ messages in thread
From: Christian Meder @ 2003-12-20 1:11 UTC (permalink / raw)
To: Nick Piggin; +Cc: William Lee Irwin III, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 4399 bytes --]
On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
> Christian Meder wrote:
>
> >On Sat, 2003-12-20 at 01:21, Nick Piggin wrote:
> >
> >>Christian Meder wrote:
> >>
> >>
> >>>On Fri, 2003-12-19 at 21:32, William Lee Irwin III wrote:
> >>>
> >>>
> >>>>On Fri, Dec 19, 2003 at 09:11:50PM +0100, Christian Meder wrote:
> >>>>
> >>>>
> >>>>>I've got a longstanding regression in gnomemeeting usage when switching
> >>>>>between 2.4 and 2.6 kernels.
> >>>>>Phenomenon:
> >>>>>Without load gnomemeeting VOIP connections are fine. As soon as some
> >>>>>load like a kernel compile is put on the laptop the gnomemeeting audio
> >>>>>stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
> >>>>>even the slightest distortion in the audio stream under load. I played
> >>>>>around with different nice levels with no success. The problem persisted
> >>>>>during the whole 2.6.0-test series no matter whether I used -mm kernels
> >>>>>or pristine Linus kernels. Even when nicing the kernel compile to +19
> >>>>>the distortions start right away. I tried Nick Piggin's scheduler which
> >>>>>fared slightly better after changing the nice level of gnomemeeting to
> >>>>>-10 but it's still a far cry from the 2.4.2x feeling without any
> >>>>>fiddling with nice values.
> >>>>>Any hints where to start looking are greatly appreciated.
> >>>>>
> >>>>>
> >>>>Please instrument your workload with the following, and send logs of the
> >>>>output (preferably compressed) to me and possibly others:
> >>>>
> >>>>top b d 5
> >>>>vmstat 5
> >>>>while true; do cat /proc/vmstat; sleep 5; done
> >>>>while true; do cat /proc/meminfo; sleep 5; done
> >>>>
> >>>>A good way to log commands like this is:
> >>>>
> >>>>(command) > /home/foo.log.1 2>&1 &
> >>>>
> >>>>where parentheses surround the command in the actual shell input.
> >>>>
> >>>>
> >>>Hi,
> >>>
> >>>I've attached the tarred output of a gnomemeeting run without load and
> >>>without distortions and another tarred output of a gnomemeeting run
> >>>while compiling a kernel with severe distortions in the audio stream.
> >>>
> >>>
> >>You're getting a lot fewer interrupts in the loaded case. Maybe its
> >>the sound card driver that has the regression from 2.4? It could be
> >>that 2.6 allows a smaller sound fragment size which is more stressful.
> >>
> >>
> >
> >Well I had the same problem with the OSS driver on 2.6. Now I use the
> >ALSA driver because I thought that could possibly improve things. The
> >ALSA driver is better indeed but it doesn't change this particular
> >phenomenon. Additionally I'd guess that the latest ALSA driver in 2.4
> >and 2.6 doesn't differ significantly and 2.4.2x with the latest ALSA
> >works great while 2.6 doesn't.
> >
> >
>
> Sounds reasonable. Maybe its large interrupt or scheduling latency
> caused somewhere else. Does disk activity alone cause a problem?
> find / -type f | xargs cat > /dev/null
> how about
> dd if=/dev/zero of=./deleteme bs=1M count=256
Ok. I've attached the logs from a run with a call with only an
additional dd. The quality was almost undisturbed only very slightly
worse than the unloaded case.
> You said it faired slightly better with my scheduler when renicing
> gnome meeting to -10. How much better is that?
Worse than unloaded and worse than the disk loaded case from above. But
all (CPU) loaded cases were producing almost complete audio dropouts
while with your scheduler and renicing to -10 I got at least a
stuttering audio stream (a regular pattern of very short slices of audio
mixed with very short slices of silence).
> whats your /proc/cpuinfo?
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 1
cpu MHz : 451.193
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca
cmov pat pse36 mmx fxsr sse
bogomips : 447.48
Christian Meder
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
[-- Attachment #2: diskload.tar.bz2 --]
[-- Type: application/x-bzip, Size: 9421 bytes --]
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 1:11 ` Christian Meder
@ 2003-12-20 1:26 ` Nick Piggin
2003-12-20 1:52 ` Christian Meder
0 siblings, 1 reply; 35+ messages in thread
From: Nick Piggin @ 2003-12-20 1:26 UTC (permalink / raw)
To: Christian Meder; +Cc: William Lee Irwin III, linux-kernel
Christian Meder wrote:
>On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
>
>>Sounds reasonable. Maybe its large interrupt or scheduling latency
>>caused somewhere else. Does disk activity alone cause a problem?
>>find / -type f | xargs cat > /dev/null
>>how about
>>dd if=/dev/zero of=./deleteme bs=1M count=256
>>
>
>Ok. I've attached the logs from a run with a call with only an
>additional dd. The quality was almost undisturbed only very slightly
>worse than the unloaded case.
>
OK, its probably not that then. Try the find command though, it would
be closer to what make/gcc is doing.
>
>>You said it faired slightly better with my scheduler when renicing
>>gnome meeting to -10. How much better is that?
>>
>
>Worse than unloaded and worse than the disk loaded case from above. But
>all (CPU) loaded cases were producing almost complete audio dropouts
>while with your scheduler and renicing to -10 I got at least a
>stuttering audio stream (a regular pattern of very short slices of audio
>mixed with very short slices of silence).
>
So it does sound like scheduling latency then. Its difficult to find
out what is happening with top and vmstat because they don't give you
an idea of individual scheduling events, which is what is important
for things like this. I'll have a look into making up a patch to gather
what I want to know.
In the meantime, I have a newer scheduler patch against 2.6.0 you could
try: http://www.kerneltrap.org/~npiggin/v28p1.gz
Try nicing the compile to +19 if it still stutters.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 1:26 ` Nick Piggin
@ 2003-12-20 1:52 ` Christian Meder
2003-12-20 2:38 ` Nick Piggin
0 siblings, 1 reply; 35+ messages in thread
From: Christian Meder @ 2003-12-20 1:52 UTC (permalink / raw)
To: Nick Piggin; +Cc: William Lee Irwin III, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 2272 bytes --]
On Sat, 2003-12-20 at 02:26, Nick Piggin wrote:
> Christian Meder wrote:
>
> >On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
> >
> >>Sounds reasonable. Maybe its large interrupt or scheduling latency
> >>caused somewhere else. Does disk activity alone cause a problem?
> >>find / -type f | xargs cat > /dev/null
> >>how about
> >>dd if=/dev/zero of=./deleteme bs=1M count=256
> >>
> >
> >Ok. I've attached the logs from a run with a call with only an
> >additional dd. The quality was almost undisturbed only very slightly
> >worse than the unloaded case.
> >
>
> OK, its probably not that then. Try the find command though, it would
> be closer to what make/gcc is doing.
Ok. I've attached the find load log. Doesn't feel different from the dd
disk load case. Quality is almost undisturbed.
>
> >
> >>You said it faired slightly better with my scheduler when renicing
> >>gnome meeting to -10. How much better is that?
> >>
> >
> >Worse than unloaded and worse than the disk loaded case from above. But
> >all (CPU) loaded cases were producing almost complete audio dropouts
> >while with your scheduler and renicing to -10 I got at least a
> >stuttering audio stream (a regular pattern of very short slices of audio
> >mixed with very short slices of silence).
> >
>
> So it does sound like scheduling latency then. Its difficult to find
> out what is happening with top and vmstat because they don't give you
> an idea of individual scheduling events, which is what is important
> for things like this. I'll have a look into making up a patch to gather
> what I want to know.
Ok. I will try it then.
>
> In the meantime, I have a newer scheduler patch against 2.6.0 you could
> try: http://www.kerneltrap.org/~npiggin/v28p1.gz
Is this patch supposed to be different schedulerwise from the scheduler
rollup patches against 2.6.0-testx
> Try nicing the compile to +19 if it still stutters.
Ok. I'll try it but I'm not too optimistic about the outcome ;-)
Christian Meder
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
[-- Attachment #2: findload.tar.bz2 --]
[-- Type: application/x-bzip, Size: 8963 bytes --]
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 1:52 ` Christian Meder
@ 2003-12-20 2:38 ` Nick Piggin
2003-12-20 2:55 ` Con Kolivas
0 siblings, 1 reply; 35+ messages in thread
From: Nick Piggin @ 2003-12-20 2:38 UTC (permalink / raw)
To: Christian Meder; +Cc: William Lee Irwin III, linux-kernel
Christian Meder wrote:
>On Sat, 2003-12-20 at 02:26, Nick Piggin wrote:
>
>>Christian Meder wrote:
>>
>>
>>>On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
>>>
>>>
>>>>Sounds reasonable. Maybe its large interrupt or scheduling latency
>>>>caused somewhere else. Does disk activity alone cause a problem?
>>>>find / -type f | xargs cat > /dev/null
>>>>how about
>>>>dd if=/dev/zero of=./deleteme bs=1M count=256
>>>>
>>>>
>>>Ok. I've attached the logs from a run with a call with only an
>>>additional dd. The quality was almost undisturbed only very slightly
>>>worse than the unloaded case.
>>>
>>>
>>OK, its probably not that then. Try the find command though, it would
>>be closer to what make/gcc is doing.
>>
>
>Ok. I've attached the find load log. Doesn't feel different from the dd
>disk load case. Quality is almost undisturbed.
>
OK I guess that rules that out then.
>
>>In the meantime, I have a newer scheduler patch against 2.6.0 you could
>>try: http://www.kerneltrap.org/~npiggin/v28p1.gz
>>
>
>Is this patch supposed to be different schedulerwise from the scheduler
>rollup patches against 2.6.0-testx
>
Its just a newer version I haven't released yet. It has a few improvements.
>
>>Try nicing the compile to +19 if it still stutters.
>>
>
>Ok. I'll try it but I'm not too optimistic about the outcome ;-)
>
>
I don't know what it could be if that doesn't work. Sound driver or maybe
some change in 2.6 causing gnomemeeting to do silly things. scheduling
latency is typically very low...
Running a not niced make -j3 bzImage while in X playing a video causes
my (not niced) latency measurement program to average 14us latency,
with 14 samples over 2ms and 136 over 100us out of 145 000 (maximum 7ms).
Running the latency measurement at -10 gives an average of 12us scheduling
latency, with one sample over 100us (305us).
Thats with my scheduler and a PIII-1000. I haven't tried plain 2.6 for a
while, but it wouldn't be much worse (if any). I haven't tried 2.4 either
and I would be surprised if it were any better.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 2:38 ` Nick Piggin
@ 2003-12-20 2:55 ` Con Kolivas
2003-12-20 3:32 ` Christian Meder
0 siblings, 1 reply; 35+ messages in thread
From: Con Kolivas @ 2003-12-20 2:55 UTC (permalink / raw)
To: Nick Piggin, Christian Meder
Cc: linux kernel mailing list, William Lee Irwin III
On Sat, 20 Dec 2003 13:38, Nick Piggin wrote:
> Christian Meder wrote:
> >On Sat, 2003-12-20 at 02:26, Nick Piggin wrote:
> >>Christian Meder wrote:
> >>>On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
> >>>>Sounds reasonable. Maybe its large interrupt or scheduling latency
> >>>>caused somewhere else. Does disk activity alone cause a problem?
> >>>>find / -type f | xargs cat > /dev/null
> >>>>how about
> >>>>dd if=/dev/zero of=./deleteme bs=1M count=256
> >>>
> >>>Ok. I've attached the logs from a run with a call with only an
> >>>additional dd. The quality was almost undisturbed only very slightly
> >>>worse than the unloaded case.
Since so many things have actually changed it's going to be hard to extract
what role the cpu scheduler has in this setting, but lets do our best.
Is there a reason you're running gnomemeeting niced -10? It is hardly using
any cpu and the problem is actually audio in your case, not the cpu
gnomemeeting is getting. Running dependant things (gnomemeeting, audio
server, gnome etc) at different nice levels is not a great idea as it can
lead to priority inversion scenarios if those apps aren't coded carefully.
What happens if you run gnomemeeting at nice 0?
How is your dma working on your disks?
What happens if you don't use an audio server (I'm not sure what the audio
server is in gnome); or if you're not using one what happens when you do?
Renice the audio server instead?
You've already tried different audio drivers right?
Nice the compile instead of -nicing the other stuff.
Try the minor interactivity fix I posted only yesterday for different nice
level latencies:
http://ck.kolivas.org/patches/2.6/2.6.0/patch-2.6.0-O21int
Is your network responsible and the audio unrelated? Some have reported
strange problems with ppp or certain network card drivers?
As you see it's not a straight forward problem but there's some things for you
to get your teeth stuck into. As it stands the cpu scheduler from your top
output appears to be giving appropriate priorities to the different factors
in your equation.
Con
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 2:55 ` Con Kolivas
@ 2003-12-20 3:32 ` Christian Meder
2003-12-20 3:50 ` Nick Piggin
2003-12-22 10:54 ` Andrew McGregor
0 siblings, 2 replies; 35+ messages in thread
From: Christian Meder @ 2003-12-20 3:32 UTC (permalink / raw)
To: Con Kolivas; +Cc: Nick Piggin, linux kernel mailing list, William Lee Irwin III
On Sat, 2003-12-20 at 03:55, Con Kolivas wrote:
> On Sat, 20 Dec 2003 13:38, Nick Piggin wrote:
> > Christian Meder wrote:
> > >On Sat, 2003-12-20 at 02:26, Nick Piggin wrote:
> > >>Christian Meder wrote:
> > >>>On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
> > >>>>Sounds reasonable. Maybe its large interrupt or scheduling latency
> > >>>>caused somewhere else. Does disk activity alone cause a problem?
> > >>>>find / -type f | xargs cat > /dev/null
> > >>>>how about
> > >>>>dd if=/dev/zero of=./deleteme bs=1M count=256
> > >>>
> > >>>Ok. I've attached the logs from a run with a call with only an
> > >>>additional dd. The quality was almost undisturbed only very slightly
> > >>>worse than the unloaded case.
>
> Since so many things have actually changed it's going to be hard to extract
> what role the cpu scheduler has in this setting, but lets do our best.
>
> Is there a reason you're running gnomemeeting niced -10? It is hardly using
> any cpu and the problem is actually audio in your case, not the cpu
> gnomemeeting is getting. Running dependant things (gnomemeeting, audio
> server, gnome etc) at different nice levels is not a great idea as it can
> lead to priority inversion scenarios if those apps aren't coded carefully.
>
> What happens if you run gnomemeeting at nice 0?
Exactly the same. It was only reniced to -10 because I tried it and
forgot to set it back. With your scheduler renicing doesn't make a
difference. No matter if I renice the compile to 19 or gnomemeeting to
-10. With Nick's scheduler renicing gnomemeeting to -10 improves the
situation.
>
> How is your dma working on your disks?
/dev/hda:
multcount = 0 (off)
IO_support = 0 (default 16-bit)
unmaskirq = 0 (off)
using_dma = 1 (on)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 256 (on)
geometry = 65535/16/63, sectors = 117210240, start = 0
> What happens if you don't use an audio server (I'm not sure what the audio
> server is in gnome); or if you're not using one what happens when you do?
esd was running but I'm not sure gnomemeeting with ALSA support was
using it. After killing esd and retrying there was no difference.
> Renice the audio server instead?
gnomemeeting without audio server is showing the same phenomenon like
gnomemeeting with esd.
> You've already tried different audio drivers right?
Yes, the phenomenon occurs for the OSS and the ALSA driver.
> Nice the compile instead of -nicing the other stuff.
Tried it with same result (see above).
> Try the minor interactivity fix I posted only yesterday for different nice
> level latencies:
> http://ck.kolivas.org/patches/2.6/2.6.0/patch-2.6.0-O21int
Actually all the posted results were on a 2.6.0-test11-mm1 with your
patch added on top. So the patch didn't change anything for me.
> Is your network responsible and the audio unrelated? Some have reported
> strange problems with ppp or certain network card drivers?
The problem occurs whether I use my WLAN PCMCIA card or my PCMCIA
Ethernet card.
> As you see it's not a straight forward problem but there's some things for you
> to get your teeth stuck into. As it stands the cpu scheduler from your top
> output appears to be giving appropriate priorities to the different factors
> in your equation.
I know that the problem isn't straight forward that's why I refrained a
long time before posting to linux-kernel trying to rule out different
scenarios. As it stands I tried different gnomemeeting versions,
different audio drivers, different nice levels, different schedulers,
preemption on and off, ACPI on and off, -mm kernels and pristine Linus
kernels with no luck. If I put CPU load on my box the gnomemeeting
audiostream gets badly mutilated (unusable). There's not much left I can
think of that's why I'm finally posting to linux-kernel.
Christian Meder
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 3:32 ` Christian Meder
@ 2003-12-20 3:50 ` Nick Piggin
2003-12-20 4:16 ` Christian Meder
2003-12-20 22:20 ` Matthias Andree
2003-12-22 10:54 ` Andrew McGregor
1 sibling, 2 replies; 35+ messages in thread
From: Nick Piggin @ 2003-12-20 3:50 UTC (permalink / raw)
To: Christian Meder
Cc: Con Kolivas, linux kernel mailing list, William Lee Irwin III
Christian Meder wrote:
>On Sat, 2003-12-20 at 03:55, Con Kolivas wrote:
>
>>On Sat, 20 Dec 2003 13:38, Nick Piggin wrote:
>>
>>>Christian Meder wrote:
>>>
>>>>On Sat, 2003-12-20 at 02:26, Nick Piggin wrote:
>>>>
>>>>>Christian Meder wrote:
>>>>>
>>>>>>On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
>>>>>>
>>>>>>>Sounds reasonable. Maybe its large interrupt or scheduling latency
>>>>>>>caused somewhere else. Does disk activity alone cause a problem?
>>>>>>>find / -type f | xargs cat > /dev/null
>>>>>>>how about
>>>>>>>dd if=/dev/zero of=./deleteme bs=1M count=256
>>>>>>>
>>>>>>Ok. I've attached the logs from a run with a call with only an
>>>>>>additional dd. The quality was almost undisturbed only very slightly
>>>>>>worse than the unloaded case.
>>>>>>
>>Since so many things have actually changed it's going to be hard to extract
>>what role the cpu scheduler has in this setting, but lets do our best.
>>
>>Is there a reason you're running gnomemeeting niced -10? It is hardly using
>>any cpu and the problem is actually audio in your case, not the cpu
>>gnomemeeting is getting. Running dependant things (gnomemeeting, audio
>>server, gnome etc) at different nice levels is not a great idea as it can
>>lead to priority inversion scenarios if those apps aren't coded carefully.
>>
>>What happens if you run gnomemeeting at nice 0?
>>
>
>Exactly the same. It was only reniced to -10 because I tried it and
>forgot to set it back. With your scheduler renicing doesn't make a
>difference. No matter if I renice the compile to 19 or gnomemeeting to
>-10. With Nick's scheduler renicing gnomemeeting to -10 improves the
>situation.
>
(although not much Con)
>
>>How is your dma working on your disks?
>>
>
>/dev/hda:
> multcount = 0 (off)
> IO_support = 0 (default 16-bit)
> unmaskirq = 0 (off)
> using_dma = 1 (on)
> keepsettings = 0 (off)
> readonly = 0 (off)
> readahead = 256 (on)
> geometry = 65535/16/63, sectors = 117210240, start = 0
>
This might be a problem - try turning unmaskirq on, and possibly
32-bit IO support on (hdparm -u1 -c1 /dev/hda). I think there is
a remote possibility that doing this will corrupt your data just
to let you know.
>
>>What happens if you don't use an audio server (I'm not sure what the audio
>>server is in gnome); or if you're not using one what happens when you do?
>>
>
>esd was running but I'm not sure gnomemeeting with ALSA support was
>using it. After killing esd and retrying there was no difference.
>
So the 1 gnomemeeting process is doing everything? (except display of
course)
>
>>Renice the audio server instead?
>>
>
>gnomemeeting without audio server is showing the same phenomenon like
>gnomemeeting with esd.
>
>
>>You've already tried different audio drivers right?
>>
>
>Yes, the phenomenon occurs for the OSS and the ALSA driver.
>
>
>>Nice the compile instead of -nicing the other stuff.
>>
>
>Tried it with same result (see above).
>
>
>>Try the minor interactivity fix I posted only yesterday for different nice
>>level latencies:
>>http://ck.kolivas.org/patches/2.6/2.6.0/patch-2.6.0-O21int
>>
>
>Actually all the posted results were on a 2.6.0-test11-mm1 with your
>patch added on top. So the patch didn't change anything for me.
>
>
>>Is your network responsible and the audio unrelated? Some have reported
>>strange problems with ppp or certain network card drivers?
>>
>
>The problem occurs whether I use my WLAN PCMCIA card or my PCMCIA
>Ethernet card.
>
>
>>As you see it's not a straight forward problem but there's some things for you
>>to get your teeth stuck into. As it stands the cpu scheduler from your top
>>output appears to be giving appropriate priorities to the different factors
>>in your equation.
>>
>
>I know that the problem isn't straight forward that's why I refrained a
>long time before posting to linux-kernel trying to rule out different
>scenarios. As it stands I tried different gnomemeeting versions,
>different audio drivers, different nice levels, different schedulers,
>preemption on and off, ACPI on and off, -mm kernels and pristine Linus
>kernels with no luck. If I put CPU load on my box the gnomemeeting
>audiostream gets badly mutilated (unusable). There's not much left I can
>think of that's why I'm finally posting to linux-kernel.
>
Thanks for your effort.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 3:50 ` Nick Piggin
@ 2003-12-20 4:16 ` Christian Meder
2003-12-20 4:32 ` Nick Piggin
2003-12-20 22:20 ` Matthias Andree
1 sibling, 1 reply; 35+ messages in thread
From: Christian Meder @ 2003-12-20 4:16 UTC (permalink / raw)
To: Nick Piggin; +Cc: Con Kolivas, linux kernel mailing list, William Lee Irwin III
[-- Attachment #1: Type: text/plain, Size: 5544 bytes --]
On Sat, 2003-12-20 at 04:50, Nick Piggin wrote:
> Christian Meder wrote:
>
> >On Sat, 2003-12-20 at 03:55, Con Kolivas wrote:
> >
> >>On Sat, 20 Dec 2003 13:38, Nick Piggin wrote:
> >>
> >>>Christian Meder wrote:
> >>>
> >>>>On Sat, 2003-12-20 at 02:26, Nick Piggin wrote:
> >>>>
> >>>>>Christian Meder wrote:
> >>>>>
> >>>>>>On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
> >>>>>>
> >>>>>>>Sounds reasonable. Maybe its large interrupt or scheduling latency
> >>>>>>>caused somewhere else. Does disk activity alone cause a problem?
> >>>>>>>find / -type f | xargs cat > /dev/null
> >>>>>>>how about
> >>>>>>>dd if=/dev/zero of=./deleteme bs=1M count=256
> >>>>>>>
> >>>>>>Ok. I've attached the logs from a run with a call with only an
> >>>>>>additional dd. The quality was almost undisturbed only very slightly
> >>>>>>worse than the unloaded case.
> >>>>>>
> >>Since so many things have actually changed it's going to be hard to extract
> >>what role the cpu scheduler has in this setting, but lets do our best.
> >>
> >>Is there a reason you're running gnomemeeting niced -10? It is hardly using
> >>any cpu and the problem is actually audio in your case, not the cpu
> >>gnomemeeting is getting. Running dependant things (gnomemeeting, audio
> >>server, gnome etc) at different nice levels is not a great idea as it can
> >>lead to priority inversion scenarios if those apps aren't coded carefully.
> >>
> >>What happens if you run gnomemeeting at nice 0?
> >>
> >
> >Exactly the same. It was only reniced to -10 because I tried it and
> >forgot to set it back. With your scheduler renicing doesn't make a
> >difference. No matter if I renice the compile to 19 or gnomemeeting to
> >-10. With Nick's scheduler renicing gnomemeeting to -10 improves the
> >situation.
> >
>
> (although not much Con)
right. Ok I'm running now 2.6.0 with Nick's v28p1: The results without
load and with kernel compile load are attached. On nice level 0 I get
now the stuttering sound which I described in the previous mail. When I
renice gnomemeeting to -10 it's actually usable but not as good as in
2.4.2x. It's still sensitive to window movement and X activity. Two
subjective observations are that the nice levels haven't got such a big
impact in Nick's scheduler they used to have and that the default
behaviour gnomemeetingwise is better than in earlier Nick schedulers.
> >
> >>How is your dma working on your disks?
> >>
> >
> >/dev/hda:
> > multcount = 0 (off)
> > IO_support = 0 (default 16-bit)
> > unmaskirq = 0 (off)
> > using_dma = 1 (on)
> > keepsettings = 0 (off)
> > readonly = 0 (off)
> > readahead = 256 (on)
> > geometry = 65535/16/63, sectors = 117210240, start = 0
> >
>
> This might be a problem - try turning unmaskirq on, and possibly
> 32-bit IO support on (hdparm -u1 -c1 /dev/hda). I think there is
> a remote possibility that doing this will corrupt your data just
> to let you know.
Tried it and doesn't make a difference.
> >
> >>What happens if you don't use an audio server (I'm not sure what the audio
> >>server is in gnome); or if you're not using one what happens when you do?
> >>
> >
> >esd was running but I'm not sure gnomemeeting with ALSA support was
> >using it. After killing esd and retrying there was no difference.
> >
>
> So the 1 gnomemeeting process is doing everything? (except display of
> course)
AFAIK yes.
Christian
> >
> >>Renice the audio server instead?
> >>
> >
> >gnomemeeting without audio server is showing the same phenomenon like
> >gnomemeeting with esd.
> >
> >
> >>You've already tried different audio drivers right?
> >>
> >
> >Yes, the phenomenon occurs for the OSS and the ALSA driver.
> >
> >
> >>Nice the compile instead of -nicing the other stuff.
> >>
> >
> >Tried it with same result (see above).
> >
> >
> >>Try the minor interactivity fix I posted only yesterday for different nice
> >>level latencies:
> >>http://ck.kolivas.org/patches/2.6/2.6.0/patch-2.6.0-O21int
> >>
> >
> >Actually all the posted results were on a 2.6.0-test11-mm1 with your
> >patch added on top. So the patch didn't change anything for me.
> >
> >
> >>Is your network responsible and the audio unrelated? Some have reported
> >>strange problems with ppp or certain network card drivers?
> >>
> >
> >The problem occurs whether I use my WLAN PCMCIA card or my PCMCIA
> >Ethernet card.
> >
> >
> >>As you see it's not a straight forward problem but there's some things for you
> >>to get your teeth stuck into. As it stands the cpu scheduler from your top
> >>output appears to be giving appropriate priorities to the different factors
> >>in your equation.
> >>
> >
> >I know that the problem isn't straight forward that's why I refrained a
> >long time before posting to linux-kernel trying to rule out different
> >scenarios. As it stands I tried different gnomemeeting versions,
> >different audio drivers, different nice levels, different schedulers,
> >preemption on and off, ACPI on and off, -mm kernels and pristine Linus
> >kernels with no luck. If I put CPU load on my box the gnomemeeting
> >audiostream gets badly mutilated (unusable). There's not much left I can
> >think of that's why I'm finally posting to linux-kernel.
> >
>
> Thanks for your effort.
>
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
[-- Attachment #2: nickload.tar.bz2 --]
[-- Type: application/x-bzip, Size: 8166 bytes --]
[-- Attachment #3: nicknoload.tar.bz2 --]
[-- Type: application/x-bzip, Size: 7283 bytes --]
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 4:16 ` Christian Meder
@ 2003-12-20 4:32 ` Nick Piggin
2003-12-20 5:15 ` Christian Meder
0 siblings, 1 reply; 35+ messages in thread
From: Nick Piggin @ 2003-12-20 4:32 UTC (permalink / raw)
To: Christian Meder
Cc: Con Kolivas, linux kernel mailing list, William Lee Irwin III
Christian Meder wrote:
>On Sat, 2003-12-20 at 04:50, Nick Piggin wrote:
>
>>(although not much Con)
>>
>
>right. Ok I'm running now 2.6.0 with Nick's v28p1: The results without
>load and with kernel compile load are attached. On nice level 0 I get
>now the stuttering sound which I described in the previous mail. When I
>renice gnomemeeting to -10 it's actually usable but not as good as in
>2.4.2x. It's still sensitive to window movement and X activity. Two
>subjective observations are that the nice levels haven't got such a big
>impact in Nick's scheduler they used to have and that the default
>behaviour gnomemeetingwise is better than in earlier Nick schedulers.
>
No, nice levels don't have such a big impact. That is the last big
think I have to fix, but thats another story...
At nice -10, there is basically nothing more the scheduler can do
for it (nice -20 will be a tiny bit better again).
I'd say its due to either sound drivers or your app doing something
different when running in 2.6.
>
>
>>This might be a problem - try turning unmaskirq on, and possibly
>>32-bit IO support on (hdparm -u1 -c1 /dev/hda). I think there is
>>a remote possibility that doing this will corrupt your data just
>>to let you know.
>>
>
>Tried it and doesn't make a difference.
>
dang
>
>>So the 1 gnomemeeting process is doing everything? (except display of
>>course)
>>
>
>AFAIK yes.
>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 4:32 ` Nick Piggin
@ 2003-12-20 5:15 ` Christian Meder
2003-12-20 8:31 ` Con Kolivas
2003-12-20 11:19 ` Ingo Molnar
0 siblings, 2 replies; 35+ messages in thread
From: Christian Meder @ 2003-12-20 5:15 UTC (permalink / raw)
To: Nick Piggin; +Cc: Con Kolivas, linux kernel mailing list, William Lee Irwin III
On Sat, 2003-12-20 at 05:32, Nick Piggin wrote:
> Christian Meder wrote:
>
> >On Sat, 2003-12-20 at 04:50, Nick Piggin wrote:
> >
> >>(although not much Con)
> >>
> >
> >right. Ok I'm running now 2.6.0 with Nick's v28p1: The results without
> >load and with kernel compile load are attached. On nice level 0 I get
> >now the stuttering sound which I described in the previous mail. When I
> >renice gnomemeeting to -10 it's actually usable but not as good as in
> >2.4.2x. It's still sensitive to window movement and X activity. Two
> >subjective observations are that the nice levels haven't got such a big
> >impact in Nick's scheduler they used to have and that the default
> >behaviour gnomemeetingwise is better than in earlier Nick schedulers.
> >
>
> No, nice levels don't have such a big impact. That is the last big
> think I have to fix, but thats another story...
>
> At nice -10, there is basically nothing more the scheduler can do
> for it (nice -20 will be a tiny bit better again).
>
> I'd say its due to either sound drivers or your app doing something
> different when running in 2.6.
I just tried hammering on the sound drivers on the playback side. So I
put on a kernel compile, a find | cat >/dev/null and ogg123 playback.
Playback performed largely unimpressed from the load level, no skips or
whatever. Even adding a gnomemeeting connection didn't decrease audio
playback. My guess is that the audio drivers are ok even more so because
otherwise OSS _and_ ALSA would be broken for my soundcard.
That would leave me with two possibilities: 2.6. is doing something
different in the gnomemeeting case or gnomemeeting is doing something
different in the 2.6 case. A cursory look at the gnomemeeting sources
didn't give me the impression that it's doing anything which would be
affected by 2.6 deployment but I'll ask on the gnomemeeting-devel list
for advice.
Thanks for all your help, I hope I can nail it soon,
Christian
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 5:15 ` Christian Meder
@ 2003-12-20 8:31 ` Con Kolivas
2003-12-20 11:19 ` Ingo Molnar
1 sibling, 0 replies; 35+ messages in thread
From: Con Kolivas @ 2003-12-20 8:31 UTC (permalink / raw)
To: Christian Meder, Nick Piggin
Cc: linux kernel mailing list, William Lee Irwin III
On Sat, 20 Dec 2003 16:15, Christian Meder wrote:
> I just tried hammering on the sound drivers on the playback side. So I
> put on a kernel compile, a find | cat >/dev/null and ogg123 playback.
> Playback performed largely unimpressed from the load level, no skips or
> whatever. Even adding a gnomemeeting connection didn't decrease audio
> playback.
Great, this is more the performance I'm used to hearing about.
> My guess is that the audio drivers are ok even more so because
> otherwise OSS _and_ ALSA would be broken for my soundcard.
>
> That would leave me with two possibilities: 2.6. is doing something
> different in the gnomemeeting case or gnomemeeting is doing something
> different in the 2.6 case. A cursory look at the gnomemeeting sources
> didn't give me the impression that it's doing anything which would be
> affected by 2.6 deployment but I'll ask on the gnomemeeting-devel list
> for advice.
Threads perhaps?
Sounds more like a resource collision of some sort. IRQ conflict? Spurious
interrupt? Were you facing Mecca when you ran it?
> Thanks for all your help, I hope I can nail it soon,
Good luck and keep us informed.
Con
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 5:15 ` Christian Meder
2003-12-20 8:31 ` Con Kolivas
@ 2003-12-20 11:19 ` Ingo Molnar
2003-12-20 16:17 ` Christian Meder
2003-12-20 16:49 ` Christian Meder
1 sibling, 2 replies; 35+ messages in thread
From: Ingo Molnar @ 2003-12-20 11:19 UTC (permalink / raw)
To: Christian Meder; +Cc: linux-kernel
* Christian Meder <chris@onestepahead.de> wrote:
> That would leave me with two possibilities: 2.6. is doing something
> different in the gnomemeeting case or gnomemeeting is doing something
> different in the 2.6 case. A cursory look at the gnomemeeting sources
> didn't give me the impression that it's doing anything which would be
> affected by 2.6 deployment but I'll ask on the gnomemeeting-devel list
> for advice.
yep, i've looked at the source too and it doesnt do anything that
changed in 2.6 from an interactivity POV.
To analyze the precise workload that hurts gnomemeeting, could you try
the following workload:
main()
{
for (;;) sched_yield();
}
and run 1-2 copies of such a load-generator - does it degrade
gnome-meeting audio just as much as eg. a kernel compile does?
as a next step, does the following degrade gnomemeeting?:
main()
{
for (;;) ;
}
my guess would be that if the yield() one degrades interactivity too
then this is unlikely to be somehow related to the scheduler proper.
If it doesnt degrade but the simple non-yield loop above does, then it's
probably something scheduling related in the sound architecture. (eg.
use of yield() by some codepath of the sound drivers - although they
dont seem to be doing anything like this.)
If neither of these workloads degrades gnomemeeting, but a kernel-make
does, then it's the interactivity estimator.
Ingo
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 11:19 ` Ingo Molnar
@ 2003-12-20 16:17 ` Christian Meder
2003-12-20 16:49 ` Christian Meder
1 sibling, 0 replies; 35+ messages in thread
From: Christian Meder @ 2003-12-20 16:17 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel
On Sat, 2003-12-20 at 12:19, Ingo Molnar wrote:
> * Christian Meder <chris@onestepahead.de> wrote:
>
> > That would leave me with two possibilities: 2.6. is doing something
> > different in the gnomemeeting case or gnomemeeting is doing something
> > different in the 2.6 case. A cursory look at the gnomemeeting sources
> > didn't give me the impression that it's doing anything which would be
> > affected by 2.6 deployment but I'll ask on the gnomemeeting-devel list
> > for advice.
>
> yep, i've looked at the source too and it doesnt do anything that
> changed in 2.6 from an interactivity POV.
>
> To analyze the precise workload that hurts gnomemeeting, could you try
> the following workload:
>
> main()
> {
> for (;;) sched_yield();
> }
>
> and run 1-2 copies of such a load-generator - does it degrade
> gnome-meeting audio just as much as eg. a kernel compile does?
>
> as a next step, does the following degrade gnomemeeting?:
>
> main()
> {
> for (;;) ;
> }
>
> my guess would be that if the yield() one degrades interactivity too
> then this is unlikely to be somehow related to the scheduler proper.
>
> If it doesnt degrade but the simple non-yield loop above does, then it's
> probably something scheduling related in the sound architecture. (eg.
> use of yield() by some codepath of the sound drivers - although they
> dont seem to be doing anything like this.)
>
> If neither of these workloads degrades gnomemeeting, but a kernel-make
> does, then it's the interactivity estimator.
Ok, the results are in. Running up to three yield-loops doesn't degrade
gnomemeeting. Running one non-yield loop does degrade gnomemeeting
_very_ slightly. Adding a second non-yield loop has approximately the
same stuttering effect as a kernel compile. Adding a third non-yield
loop makes gnomemeeting totally unusable.
All these tests were done with Nick's scheduler patch. If I should retry
with stock 2.6.0 just tell me.
Sound driver is by the way snd_es1968 in ALSA and maestro in OSS.
Christian
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 11:19 ` Ingo Molnar
2003-12-20 16:17 ` Christian Meder
@ 2003-12-20 16:49 ` Christian Meder
2003-12-20 17:42 ` Ingo Molnar
2003-12-20 23:29 ` Nick Piggin
1 sibling, 2 replies; 35+ messages in thread
From: Christian Meder @ 2003-12-20 16:49 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel
On Sat, 2003-12-20 at 12:19, Ingo Molnar wrote:
> * Christian Meder <chris@onestepahead.de> wrote:
>
> > That would leave me with two possibilities: 2.6. is doing something
> > different in the gnomemeeting case or gnomemeeting is doing something
> > different in the 2.6 case. A cursory look at the gnomemeeting sources
> > didn't give me the impression that it's doing anything which would be
> > affected by 2.6 deployment but I'll ask on the gnomemeeting-devel list
> > for advice.
>
> yep, i've looked at the source too and it doesnt do anything that
> changed in 2.6 from an interactivity POV.
Stefan Bruens pointed out on the gnomemeeting-devel list that pwlib
which gnomemeeting is using executes sched_yield and that perhaps there
is a problem akin to the openoffice busy-loop on sched_yield() problem
earlier this year. I found the following sched_yield code in pwlib 1.5.2
in src/ptlib/unix/tlibthrd.cxx:
> static BOOL PAssertThreadOp(int retval,
> unsigned & retry,
> const char * funcname,
> const char * file,
> unsigned line)
> {
> if (retval == 0) {
> PTRACE_IF(2, retry > 0, "PWLib\t" << funcname << " required " << retry << "
> retries!");
> return FALSE;
> }
>
> if (errno == EINTR || errno == EAGAIN) {
> if (++retry < 1000) {
> #if defined(P_RTEMS)
> sched_yield();
> #else
> usleep(10000); // Basically just swap out thread to try and clear blockage
> #endif
> return TRUE; // Return value to try again
> }
> // Give up and assert
> }
>
> PAssertFunc(file, line, NULL, psprintf("Function %s failed", funcname));
> return FALSE;
> }
Is this obviously broken for 2.6 usage ?
Christian
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 16:49 ` Christian Meder
@ 2003-12-20 17:42 ` Ingo Molnar
2003-12-21 1:40 ` Christian Meder
2003-12-20 23:29 ` Nick Piggin
1 sibling, 1 reply; 35+ messages in thread
From: Ingo Molnar @ 2003-12-20 17:42 UTC (permalink / raw)
To: Christian Meder; +Cc: linux-kernel
* Christian Meder <chris@onestepahead.de> wrote:
> > yep, i've looked at the source too and it doesnt do anything that
> > changed in 2.6 from an interactivity POV.
>
> Stefan Bruens pointed out on the gnomemeeting-devel list that pwlib
> which gnomemeeting is using executes sched_yield and that perhaps
> there is a problem akin to the openoffice busy-loop on sched_yield()
> problem earlier this year. I found the following sched_yield code in
> pwlib 1.5.2 in src/ptlib/unix/tlibthrd.cxx:
ah! I suspected something like this, that's why i looked at the source,
but i didnt check dependent libs ...
> if (++retry < 1000) {
> #if defined(P_RTEMS)
> sched_yield();
> Is this obviously broken for 2.6 usage ?
yes, very definitely broken. For one, it does not provide predictable
timing - 1000 loops of sched_yield() can be very different on different
CPUs. But the main problem is that on 2.6 sched_yield() is much more
agressive. Something like this was fixed in OpenOffice earlier this
year, and it improved interactivity quite visibly. Could you remove the
sched_yield() and replace it with a 20 msec nanosleep (and keep the rety
loop to 100)? Does that make any difference?
Ingo
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-19 20:11 2.6 vs 2.4 regression when running gnomemeeting Christian Meder
2003-12-19 20:32 ` William Lee Irwin III
@ 2003-12-20 19:34 ` Marc Schiffbauer
2003-12-21 1:49 ` Christian Meder
1 sibling, 1 reply; 35+ messages in thread
From: Marc Schiffbauer @ 2003-12-20 19:34 UTC (permalink / raw)
To: linux-kernel
* Christian Meder schrieb am 19.12.03 um 21:11 Uhr:
> Hi,
>
> I've got a longstanding regression in gnomemeeting usage when switching
> between 2.4 and 2.6 kernels.
>
> Phenomenon:
> Without load gnomemeeting VOIP connections are fine. As soon as some
> load like a kernel compile is put on the laptop the gnomemeeting audio
> stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
> even the slightest distortion in the audio stream under load. I played
> around with different nice levels with no success. The problem persisted
> during the whole 2.6.0-test series no matter whether I used -mm kernels
> or pristine Linus kernels. Even when nicing the kernel compile to +19
> the distortions start right away. I tried Nick Piggin's scheduler which
> fared slightly better after changing the nice level of gnomemeeting to
> -10 but it's still a far cry from the 2.4.2x feeling without any
> fiddling with nice values.
>
> Any hints where to start looking are greatly appreciated.
>
Hi Christian,
is it true, that your X-Server runs with a higher priority? Perhaps
you are a Debian user?
If thats the case try to make your X-Server run with normal
priority. That fixed sound problems here too. I had the problem that
I could not play an mp3 file without distortion while compiling a
kernel.
-Marc
--
8AAC 5F46 83B4 DB70 8317 3723 296C 6CCA 35A6 4134
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 3:50 ` Nick Piggin
2003-12-20 4:16 ` Christian Meder
@ 2003-12-20 22:20 ` Matthias Andree
2003-12-21 19:23 ` Jens Axboe
1 sibling, 1 reply; 35+ messages in thread
From: Matthias Andree @ 2003-12-20 22:20 UTC (permalink / raw)
To: linux kernel mailing list
On Sat, 20 Dec 2003, Nick Piggin wrote:
> >/dev/hda:
> >multcount = 0 (off)
> >IO_support = 0 (default 16-bit)
> >unmaskirq = 0 (off)
> >using_dma = 1 (on)
> >keepsettings = 0 (off)
> >readonly = 0 (off)
> >readahead = 256 (on)
> >geometry = 65535/16/63, sectors = 117210240, start = 0
> >
>
> This might be a problem - try turning unmaskirq on, and possibly
> 32-bit IO support on (hdparm -u1 -c1 /dev/hda). I think there is
> a remote possibility that doing this will corrupt your data just
Does 32-bit I/O support make a difference for DMA transfers at all or
just for PIO?
--
Matthias Andree
Encrypt your mail: my GnuPG key ID is 0x052E7D95
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 16:49 ` Christian Meder
2003-12-20 17:42 ` Ingo Molnar
@ 2003-12-20 23:29 ` Nick Piggin
1 sibling, 0 replies; 35+ messages in thread
From: Nick Piggin @ 2003-12-20 23:29 UTC (permalink / raw)
To: Christian Meder; +Cc: Ingo Molnar, linux-kernel
Christian Meder wrote:
>On Sat, 2003-12-20 at 12:19, Ingo Molnar wrote:
>
>>* Christian Meder <chris@onestepahead.de> wrote:
>>
>>
>>>That would leave me with two possibilities: 2.6. is doing something
>>>different in the gnomemeeting case or gnomemeeting is doing something
>>>different in the 2.6 case. A cursory look at the gnomemeeting sources
>>>didn't give me the impression that it's doing anything which would be
>>>affected by 2.6 deployment but I'll ask on the gnomemeeting-devel list
>>>for advice.
>>>
>>yep, i've looked at the source too and it doesnt do anything that
>>changed in 2.6 from an interactivity POV.
>>
>
>Stefan Bruens pointed out on the gnomemeeting-devel list that pwlib
>which gnomemeeting is using executes sched_yield and that perhaps there
>is a problem akin to the openoffice busy-loop on sched_yield() problem
>earlier this year. I found the following sched_yield code in pwlib 1.5.2
>in src/ptlib/unix/tlibthrd.cxx:
>
>
>
>>static BOOL PAssertThreadOp(int retval,
>> unsigned & retry,
>> const char * funcname,
>> const char * file,
>> unsigned line)
>>{
>> if (retval == 0) {
>> PTRACE_IF(2, retry > 0, "PWLib\t" << funcname << " required " << retry << "
>>retries!");
>> return FALSE;
>> }
>>
>> if (errno == EINTR || errno == EAGAIN) {
>> if (++retry < 1000) {
>>#if defined(P_RTEMS)
>> sched_yield();
>>#else
>> usleep(10000); // Basically just swap out thread to try and clear blockage
>>#endif
>> return TRUE; // Return value to try again
>> }
>> // Give up and assert
>> }
>>
>> PAssertFunc(file, line, NULL, psprintf("Function %s failed", funcname));
>> return FALSE;
>>}
>>
>
>Is this obviously broken for 2.6 usage ?
>
Most probably, yes. That will cause the program to give up its entire
timeslice, so then all processes of any priority will be able to use
their entire timeslices before it.
If it gets called often, that would be the cause of your problem. Try
just using the usleep. You could even reduce the usleep timeout to
1000 or so, which 2.6 can take advantage of.
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 17:42 ` Ingo Molnar
@ 2003-12-21 1:40 ` Christian Meder
2003-12-21 8:57 ` Ingo Molnar
0 siblings, 1 reply; 35+ messages in thread
From: Christian Meder @ 2003-12-21 1:40 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel
On Sat, 2003-12-20 at 18:42, Ingo Molnar wrote:
> * Christian Meder <chris@onestepahead.de> wrote:
>
> > > yep, i've looked at the source too and it doesnt do anything that
> > > changed in 2.6 from an interactivity POV.
> >
> > Stefan Bruens pointed out on the gnomemeeting-devel list that pwlib
> > which gnomemeeting is using executes sched_yield and that perhaps
> > there is a problem akin to the openoffice busy-loop on sched_yield()
> > problem earlier this year. I found the following sched_yield code in
> > pwlib 1.5.2 in src/ptlib/unix/tlibthrd.cxx:
>
> ah! I suspected something like this, that's why i looked at the source,
> but i didnt check dependent libs ...
>
> > if (++retry < 1000) {
> > #if defined(P_RTEMS)
> > sched_yield();
>
> > Is this obviously broken for 2.6 usage ?
>
>
> yes, very definitely broken. For one, it does not provide predictable
> timing - 1000 loops of sched_yield() can be very different on different
> CPUs. But the main problem is that on 2.6 sched_yield() is much more
> agressive. Something like this was fixed in OpenOffice earlier this
> year, and it improved interactivity quite visibly. Could you remove the
> sched_yield() and replace it with a 20 msec nanosleep (and keep the rety
> loop to 100)? Does that make any difference?
I tried to verify your suggestion and found that the P_RTEMS symbol is
not defined on Linux. It seems to be some other kind of realtime
operating system. So the code in question already uses usleep. Now I'm
still digging for other occurances of sched_yield in the pwlib sources.
I'll keep you posted with further facts.
Christian
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 19:34 ` Marc Schiffbauer
@ 2003-12-21 1:49 ` Christian Meder
0 siblings, 0 replies; 35+ messages in thread
From: Christian Meder @ 2003-12-21 1:49 UTC (permalink / raw)
To: Marc Schiffbauer; +Cc: linux-kernel
On Sat, 2003-12-20 at 20:34, Marc Schiffbauer wrote:
> * Christian Meder schrieb am 19.12.03 um 21:11 Uhr:
> > Hi,
> >
> > I've got a longstanding regression in gnomemeeting usage when switching
> > between 2.4 and 2.6 kernels.
> >
> > Phenomenon:
> > Without load gnomemeeting VOIP connections are fine. As soon as some
> > load like a kernel compile is put on the laptop the gnomemeeting audio
> > stream is cut to pieces and gets unintelligible . On 2.4.2x I don't get
> > even the slightest distortion in the audio stream under load. I played
> > around with different nice levels with no success. The problem persisted
> > during the whole 2.6.0-test series no matter whether I used -mm kernels
> > or pristine Linus kernels. Even when nicing the kernel compile to +19
> > the distortions start right away. I tried Nick Piggin's scheduler which
> > fared slightly better after changing the nice level of gnomemeeting to
> > -10 but it's still a far cry from the 2.4.2x feeling without any
> > fiddling with nice values.
> >
> > Any hints where to start looking are greatly appreciated.
> >
>
> Hi Christian,
>
> is it true, that your X-Server runs with a higher priority? Perhaps
> you are a Debian user?
>
> If thats the case try to make your X-Server run with normal
> priority. That fixed sound problems here too. I had the problem that
> I could not play an mp3 file without distortion while compiling a
> kernel.
Oh I tried gnomemeeting, the Xserver and the kernel compile at different
nice levels without significant changes. Additionally I don't have any
problems with audio playback under load, it's just that gnomemeeting is
performing worse than in 2.4.2x.
Christian
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-21 1:40 ` Christian Meder
@ 2003-12-21 8:57 ` Ingo Molnar
2003-12-22 1:19 ` Christian Meder
0 siblings, 1 reply; 35+ messages in thread
From: Ingo Molnar @ 2003-12-21 8:57 UTC (permalink / raw)
To: Christian Meder; +Cc: linux-kernel
* Christian Meder <chris@onestepahead.de> wrote:
> I tried to verify your suggestion and found that the P_RTEMS symbol is
> not defined on Linux. It seems to be some other kind of realtime
> operating system. So the code in question already uses usleep. Now I'm
> still digging for other occurances of sched_yield in the pwlib
> sources.
could you try to strace -f gnomemeeting? Maybe there's no sched_yield()
at all. Could you also try to run the non-yielding loop code via:
nice -19 ./loop &
do a couple of such loops still degrade gnomemeeting?
Ingo
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 22:20 ` Matthias Andree
@ 2003-12-21 19:23 ` Jens Axboe
0 siblings, 0 replies; 35+ messages in thread
From: Jens Axboe @ 2003-12-21 19:23 UTC (permalink / raw)
To: linux kernel mailing list
On Sat, Dec 20 2003, Matthias Andree wrote:
> On Sat, 20 Dec 2003, Nick Piggin wrote:
>
> > >/dev/hda:
> > >multcount = 0 (off)
> > >IO_support = 0 (default 16-bit)
> > >unmaskirq = 0 (off)
> > >using_dma = 1 (on)
> > >keepsettings = 0 (off)
> > >readonly = 0 (off)
> > >readahead = 256 (on)
> > >geometry = 65535/16/63, sectors = 117210240, start = 0
> > >
> >
> > This might be a problem - try turning unmaskirq on, and possibly
> > 32-bit IO support on (hdparm -u1 -c1 /dev/hda). I think there is
> > a remote possibility that doing this will corrupt your data just
>
> Does 32-bit I/O support make a difference for DMA transfers at all or
> just for PIO?
It's just for PIO.
--
Jens Axboe
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-21 8:57 ` Ingo Molnar
@ 2003-12-22 1:19 ` Christian Meder
2003-12-22 1:47 ` Con Kolivas
2003-12-22 8:48 ` Ingo Molnar
0 siblings, 2 replies; 35+ messages in thread
From: Christian Meder @ 2003-12-22 1:19 UTC (permalink / raw)
To: Ingo Molnar; +Cc: linux-kernel, gnomemeeting-devel-list
On Sun, 2003-12-21 at 09:57, Ingo Molnar wrote:
> * Christian Meder <chris@onestepahead.de> wrote:
>
> > I tried to verify your suggestion and found that the P_RTEMS symbol is
> > not defined on Linux. It seems to be some other kind of realtime
> > operating system. So the code in question already uses usleep. Now I'm
> > still digging for other occurances of sched_yield in the pwlib
> > sources.
>
> could you try to strace -f gnomemeeting? Maybe there's no sched_yield()
> at all. Could you also try to run the non-yielding loop code via:
>
> nice -19 ./loop &
>
> do a couple of such loops still degrade gnomemeeting?
I found the culprit. It's sched_yield again. When I straced gnomemeeting
even without load I saw a lot of sched_yields. So I googled around for
2.6 and sched_yield and found among others
http://www.hpl.hp.com/research/linux/kernel/o1-openmp.php by David
Mosberger. I tried gnomemeeting with the romp hack at the end of the
article which changes all sched_yields to noops via library preloading.
The difference was _really_ impressive. No matter how many non-yield
loops and kernel compiles I ran gnomemeeting didn't even skip once.
So the questionable code in pwlib is probably:
> BOOL PSemaphore::Wait(const PTimeInterval & waitTime)
> {
> if (waitTime == PMaxTimeInterval) {
> Wait();
> return TRUE;
> }
>
> // create absolute finish time
> PTime finishTime;
> finishTime += waitTime;
>
> #ifdef P_HAS_SEMAPHORES
>
> // loop until timeout, or semaphore becomes available
> // don't use a PTimer, as this causes the housekeeping
> // thread to get very busy
> do {
> if (sem_trywait(&semId) == 0)
> return TRUE;
>
> PThread::Yield(); // One time slice
> } while (PTime() < finishTime);
>
> return FALSE;
Defining Yield to noop and building a new libpt solved the problem
permanently for me.
It seems that not all people have got problems with gnomemeeting and
2.6. Damien Sandras (the gnomemeeting maintainer) for example reported
that he hasn't got any problems with gnomemeeting on 2.6 while compiling
in parallel. So I guess it's depending on the frequency of sched_yields
one gets in gnomemeeting. Which is probably depending on the processor
speed, etc.
That just leaves the question what is the proper fix, to send it
upstream and to note the phenomenon down in a faq.
Thanks to all who helped me with debugging advice and if anybody needs
further information just ask.
Christian
--
Christian Meder, email: chris@onestepahead.de
What's the railroad to me ?
I never go to see
Where it ends.
It fills a few hollows,
And makes banks for the swallows,
It sets the sand a-blowing,
And the blackberries a-growing.
(Henry David Thoreau)
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-22 1:19 ` Christian Meder
@ 2003-12-22 1:47 ` Con Kolivas
2003-12-22 8:48 ` Ingo Molnar
1 sibling, 0 replies; 35+ messages in thread
From: Con Kolivas @ 2003-12-22 1:47 UTC (permalink / raw)
To: Christian Meder, Ingo Molnar; +Cc: linux-kernel, gnomemeeting-devel-list
On Mon, 22 Dec 2003 12:19, Christian Meder wrote:
> On Sun, 2003-12-21 at 09:57, Ingo Molnar wrote:
> > * Christian Meder <chris@onestepahead.de> wrote:
> > > I tried to verify your suggestion and found that the P_RTEMS symbol is
> > > not defined on Linux. It seems to be some other kind of realtime
> > > operating system. So the code in question already uses usleep. Now I'm
> > > still digging for other occurances of sched_yield in the pwlib
> > > sources.
> >
> > could you try to strace -f gnomemeeting? Maybe there's no sched_yield()
> > at all. Could you also try to run the non-yielding loop code via:
> >
> > nice -19 ./loop &
> >
> > do a couple of such loops still degrade gnomemeeting?
>
> I found the culprit. It's sched_yield again. When I straced gnomemeeting
> even without load I saw a lot of sched_yields. So I googled around for
> 2.6 and sched_yield and found among others
> http://www.hpl.hp.com/research/linux/kernel/o1-openmp.php by David
> Mosberger. I tried gnomemeeting with the romp hack at the end of the
> article which changes all sched_yields to noops via library preloading.
> The difference was _really_ impressive. No matter how many non-yield
> loops and kernel compiles I ran gnomemeeting didn't even skip once.
It's a valuable lesson to us that the sched_yield() issue is one we should be
suspicious of. I'm very happy that it performs well for you once the coding
is corrected.
[SNIP]
> Defining Yield to noop and building a new libpt solved the problem
> permanently for me.
>
> It seems that not all people have got problems with gnomemeeting and
> 2.6. Damien Sandras (the gnomemeeting maintainer) for example reported
> that he hasn't got any problems with gnomemeeting on 2.6 while compiling
> in parallel. So I guess it's depending on the frequency of sched_yields
> one gets in gnomemeeting. Which is probably depending on the processor
> speed, etc.
It probably hits that code path less frequently in slower cpus ironically.
> That just leaves the question what is the proper fix, to send it
> upstream and to note the phenomenon down in a faq.
The sched_yield issue is noted in faqs, but it should be highlighted in the
interactivity section.
> Thanks to all who helped me with debugging advice and if anybody needs
> further information just ask.
No, thank you for tracking this down fully. No amount of reassuring that it's
not 2.6's fault will suffice for us unless we can find the real cause and
solutions for these issues.
Regards,
Con
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-22 1:19 ` Christian Meder
2003-12-22 1:47 ` Con Kolivas
@ 2003-12-22 8:48 ` Ingo Molnar
1 sibling, 0 replies; 35+ messages in thread
From: Ingo Molnar @ 2003-12-22 8:48 UTC (permalink / raw)
To: Christian Meder; +Cc: linux-kernel, gnomemeeting-devel-list
* Christian Meder <chris@onestepahead.de> wrote:
> > nice -19 ./loop &
> >
> > do a couple of such loops still degrade gnomemeeting?
>
> I found the culprit. It's sched_yield again. When I straced
> gnomemeeting even without load I saw a lot of sched_yields. [...]
this is definitely broken code. Such code already causes big CPU
overhead in certain circumstances (under 2.4 too) - but in 2.6 it also
shows up as an interactivity problem. So 2.4 hid the problem, 2.6
exposes it.
> So the questionable code in pwlib is probably:
> > BOOL PSemaphore::Wait(const PTimeInterval & waitTime)
yeah. pwlib should be fixed. The quick fix is, instead of sched_yield(),
to do:
{
struct timespec timer = { 0, 1 };
nanosleep (&timer, NULL);
}
this does what pwlib really wants to do: sleep for the shortest amount
of time posssible, because its semaphore implementation is polling
based.
(but pwlib should perhaps use sem_timedwait(sem, abs_timeout) instead -
which does exactly what PSemaphore::Wait() tries to implement.)
Ingo
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-20 3:32 ` Christian Meder
2003-12-20 3:50 ` Nick Piggin
@ 2003-12-22 10:54 ` Andrew McGregor
2003-12-22 11:15 ` Con Kolivas
` (2 more replies)
1 sibling, 3 replies; 35+ messages in thread
From: Andrew McGregor @ 2003-12-22 10:54 UTC (permalink / raw)
To: Christian Meder, Con Kolivas
Cc: Nick Piggin, linux kernel mailing list, William Lee Irwin III
Hmm. Gnomemeeting has a history of strange threading issues (actually, all
OpenH323 derived projects do). Is there a threading change that might
explain this?
Andrew
--On Saturday, 20 December 2003 4:32 a.m. +0100 Christian Meder
<chris@onestepahead.de> wrote:
> On Sat, 2003-12-20 at 03:55, Con Kolivas wrote:
>> On Sat, 20 Dec 2003 13:38, Nick Piggin wrote:
>> > Christian Meder wrote:
>> > > On Sat, 2003-12-20 at 02:26, Nick Piggin wrote:
>> > >> Christian Meder wrote:
>> > >>> On Sat, 2003-12-20 at 01:48, Nick Piggin wrote:
>> > >>>> Sounds reasonable. Maybe its large interrupt or scheduling latency
>> > >>>> caused somewhere else. Does disk activity alone cause a problem?
>> > >>>> find / -type f | xargs cat > /dev/null
>> > >>>> how about
>> > >>>> dd if=/dev/zero of=./deleteme bs=1M count=256
>> > >>>
>> > >>> Ok. I've attached the logs from a run with a call with only an
>> > >>> additional dd. The quality was almost undisturbed only very
>> > >>> slightly worse than the unloaded case.
>>
>> Since so many things have actually changed it's going to be hard to
>> extract what role the cpu scheduler has in this setting, but lets do
>> our best.
>>
>> Is there a reason you're running gnomemeeting niced -10? It is hardly
>> using any cpu and the problem is actually audio in your case, not the
>> cpu gnomemeeting is getting. Running dependant things (gnomemeeting,
>> audio server, gnome etc) at different nice levels is not a great idea
>> as it can lead to priority inversion scenarios if those apps aren't
>> coded carefully.
>>
>> What happens if you run gnomemeeting at nice 0?
>
> Exactly the same. It was only reniced to -10 because I tried it and
> forgot to set it back. With your scheduler renicing doesn't make a
> difference. No matter if I renice the compile to 19 or gnomemeeting to
> -10. With Nick's scheduler renicing gnomemeeting to -10 improves the
> situation.
>
>>
>> How is your dma working on your disks?
>
> /dev/hda:
> multcount = 0 (off)
> IO_support = 0 (default 16-bit)
> unmaskirq = 0 (off)
> using_dma = 1 (on)
> keepsettings = 0 (off)
> readonly = 0 (off)
> readahead = 256 (on)
> geometry = 65535/16/63, sectors = 117210240, start = 0
>
>> What happens if you don't use an audio server (I'm not sure what the
>> audio server is in gnome); or if you're not using one what happens when
>> you do?
>
> esd was running but I'm not sure gnomemeeting with ALSA support was
> using it. After killing esd and retrying there was no difference.
>
>> Renice the audio server instead?
>
> gnomemeeting without audio server is showing the same phenomenon like
> gnomemeeting with esd.
>
>> You've already tried different audio drivers right?
>
> Yes, the phenomenon occurs for the OSS and the ALSA driver.
>
>> Nice the compile instead of -nicing the other stuff.
>
> Tried it with same result (see above).
>
>> Try the minor interactivity fix I posted only yesterday for different
>> nice level latencies:
>> http://ck.kolivas.org/patches/2.6/2.6.0/patch-2.6.0-O21int
>
> Actually all the posted results were on a 2.6.0-test11-mm1 with your
> patch added on top. So the patch didn't change anything for me.
>
>> Is your network responsible and the audio unrelated? Some have reported
>> strange problems with ppp or certain network card drivers?
>
> The problem occurs whether I use my WLAN PCMCIA card or my PCMCIA
> Ethernet card.
>
>> As you see it's not a straight forward problem but there's some things
>> for you to get your teeth stuck into. As it stands the cpu scheduler
>> from your top output appears to be giving appropriate priorities to the
>> different factors in your equation.
>
> I know that the problem isn't straight forward that's why I refrained a
> long time before posting to linux-kernel trying to rule out different
> scenarios. As it stands I tried different gnomemeeting versions,
> different audio drivers, different nice levels, different schedulers,
> preemption on and off, ACPI on and off, -mm kernels and pristine Linus
> kernels with no luck. If I put CPU load on my box the gnomemeeting
> audiostream gets badly mutilated (unusable). There's not much left I can
> think of that's why I'm finally posting to linux-kernel.
>
>
>
> Christian Meder
>
>
> --
> Christian Meder, email: chris@onestepahead.de
>
> What's the railroad to me ?
> I never go to see
> Where it ends.
> It fills a few hollows,
> And makes banks for the swallows,
> It sets the sand a-blowing,
> And the blackberries a-growing.
> (Henry David Thoreau)
>
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
>
---------
Andrew McGregor
Director, Scientific Advisor
IndraNet Technologies Ltd
http://www.indranet-technologies.com/
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-22 10:54 ` Andrew McGregor
@ 2003-12-22 11:15 ` Con Kolivas
2003-12-22 12:51 ` Ingo Molnar
2003-12-22 13:25 ` Nick Piggin
2 siblings, 0 replies; 35+ messages in thread
From: Con Kolivas @ 2003-12-22 11:15 UTC (permalink / raw)
To: Andrew McGregor; +Cc: linux kernel mailing list
On Mon, 22 Dec 2003 21:54, Andrew McGregor wrote:
> Hmm. Gnomemeeting has a history of strange threading issues (actually, all
> OpenH323 derived projects do). Is there a threading change that might
> explain this?
[cc list stripped]
It's been sorted. It was a sched_yield() issue.
Con
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-22 10:54 ` Andrew McGregor
2003-12-22 11:15 ` Con Kolivas
@ 2003-12-22 12:51 ` Ingo Molnar
2003-12-22 13:25 ` Nick Piggin
2 siblings, 0 replies; 35+ messages in thread
From: Ingo Molnar @ 2003-12-22 12:51 UTC (permalink / raw)
To: Andrew McGregor; +Cc: linux kernel mailing list
* Andrew McGregor <andrew@indranet.co.nz> wrote:
> Hmm. Gnomemeeting has a history of strange threading issues
> (actually, all OpenH323 derived projects do). Is there a threading
> change that might explain this?
we tracked down the bug to pwlib's sched_yield() usage. Other code that
uses sched_yield() to 'sleep a bit' could be affected as well.
Ingo
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: 2.6 vs 2.4 regression when running gnomemeeting
2003-12-22 10:54 ` Andrew McGregor
2003-12-22 11:15 ` Con Kolivas
2003-12-22 12:51 ` Ingo Molnar
@ 2003-12-22 13:25 ` Nick Piggin
2 siblings, 0 replies; 35+ messages in thread
From: Nick Piggin @ 2003-12-22 13:25 UTC (permalink / raw)
To: Andrew McGregor
Cc: Christian Meder, Con Kolivas, linux kernel mailing list,
William Lee Irwin III
Andrew McGregor wrote:
> Hmm. Gnomemeeting has a history of strange threading issues
> (actually, all OpenH323 derived projects do). Is there a threading
> change that might explain this?
Yes, changes in the way the scheduler deals with sched_yield.
^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2003-12-22 13:25 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-12-19 20:11 2.6 vs 2.4 regression when running gnomemeeting Christian Meder
2003-12-19 20:32 ` William Lee Irwin III
2003-12-19 23:30 ` Christian Meder
2003-12-20 0:21 ` Nick Piggin
2003-12-20 0:37 ` Christian Meder
2003-12-20 0:48 ` Nick Piggin
2003-12-20 1:11 ` Christian Meder
2003-12-20 1:26 ` Nick Piggin
2003-12-20 1:52 ` Christian Meder
2003-12-20 2:38 ` Nick Piggin
2003-12-20 2:55 ` Con Kolivas
2003-12-20 3:32 ` Christian Meder
2003-12-20 3:50 ` Nick Piggin
2003-12-20 4:16 ` Christian Meder
2003-12-20 4:32 ` Nick Piggin
2003-12-20 5:15 ` Christian Meder
2003-12-20 8:31 ` Con Kolivas
2003-12-20 11:19 ` Ingo Molnar
2003-12-20 16:17 ` Christian Meder
2003-12-20 16:49 ` Christian Meder
2003-12-20 17:42 ` Ingo Molnar
2003-12-21 1:40 ` Christian Meder
2003-12-21 8:57 ` Ingo Molnar
2003-12-22 1:19 ` Christian Meder
2003-12-22 1:47 ` Con Kolivas
2003-12-22 8:48 ` Ingo Molnar
2003-12-20 23:29 ` Nick Piggin
2003-12-20 22:20 ` Matthias Andree
2003-12-21 19:23 ` Jens Axboe
2003-12-22 10:54 ` Andrew McGregor
2003-12-22 11:15 ` Con Kolivas
2003-12-22 12:51 ` Ingo Molnar
2003-12-22 13:25 ` Nick Piggin
2003-12-20 19:34 ` Marc Schiffbauer
2003-12-21 1:49 ` Christian Meder
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox