tvtime and the Linux 2.6 scheduler

tvtime and the Linux 2.6 scheduler

[Billy Biggs]

  I am the author of tvtime, a TV application with advanced image
processing algorithms.  Some users are complaining about poor
performance under Linux 2.6, and I would like more information about how
tvtime will be treated by the scheduler.  Here is an example of the
intended usage:

  - Program running as root and SCHED_FIFO
  - NTSC, input ~30 fps, each field processed for an output of ~60 fps
  - CPU intensive processing, say 9 ms per field on my P3-733
  - with a typical AGP card, the X driver takes 4 ms to draw
  - Wait using /dev/rtc set to 1024 Hz

  for(;;)
      9 ms : process frame
      4 ms : draw frame
      3 ms : wait until next field time using /dev/rtc
      9 ms : process frame
      4 ms : draw frame
      3 ms : block on /dev/video0 for next frame
     -----
     33 ms : time per NTSC frame

  The theory is that Linux classifies this as a CPU hog regardless of
its priority, and preempts tvtime with other processes.  Oswald
Buddenhagen describes the effect as this:

  "[...] it starts up fine, but after a few seconds (when the scheduler
gathered some stats) ... well, it looks funny: the scene goes roughly
exponentially into slow motion, then there is a frame drop and the
process starts over.  this behaviour can be observed at any priority,
which is clearly against the claim "no normally priorized interactive
process will preempt a highly priorized cpu-hog" that i've read
somewhere.  the xserver priority does not change anything, either;"

  Avoiding root/SCHED_FIFO and using usleep() instead of /dev/rtc seems
to exhibit the same behavior.

  Thoughts?

  -Billy

Re: tvtime and the Linux 2.6 scheduler

[Jose Luis Domingo Lopez]

On Sunday, 23 May 2004, at 10:48:59 -0500,
Billy Biggs wrote:

>   "[...] it starts up fine, but after a few seconds (when the scheduler
> gathered some stats) ... well, it looks funny: the scene goes roughly
> exponentially into slow motion, then there is a frame drop and the
> process starts over.  this behaviour can be observed at any priority,
> which is clearly against the claim "no normally priorized interactive
> process will preempt a highly priorized cpu-hog" that i've read
> somewhere.  the xserver priority does not change anything, either;"
> 
I am currently using tvtime 0.9.12-2 from Debian SID on a Linux kernel
version 2.6.6 compiled with Sid's gcc 3.3.3 and haven't seen such
problems so far. I use tvtime often to watch TV on my monitor, and it
works OK, no artifacts and no slowdown (even when I download my email
and spamassassin starts to eat CPU cicles like mad).

With my current configuration tvtime at full screen (1024x768) takes
about 40% CPU from a AMD XP 1477 MHz (1700 rating). I have just tried
launching three copies of "yes" on several aterm's and tvtime still
looks as smooth as before, althoug CPU is always at 100%

Please ask for more information should you need it.

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.6.6)

Re: tvtime and the Linux 2.6 scheduler

[Con Kolivas]

Hi

On Mon, 24 May 2004 01:48 am, Billy Biggs wrote:
>   I am the author of tvtime, a TV application with advanced image
> processing algorithms.  Some users are complaining about poor
> performance under Linux 2.6, and I would like more information about how
> tvtime will be treated by the scheduler.  Here is an example of the
> intended usage:
>
>   - Program running as root and SCHED_FIFO

snip

>      33 ms : time per NTSC frame

snip

The followup email from someone describing good performance may help us 
understand what's going on. Your example of poor performance is one when the 
cpu performance is marginal to get exactly 30 fps processed and on the 
screen. The cpu overhead in 2.6 is slightly higher than 2.4 so a borderline 
case may be just pushed over. 

A program running as sched_fifo it will preempt absolutely everything 
regardless of how it behaves. It sounds like it's giving X less cpu time to 
draw the frame each time until eventually the processing fails to capture the 
frame and then X smooths out again. I cant pretend to understand how your 
application blocks (as you say) between X and tvtime, but does tvtime not try 
to schedule until X has finished using up cpu or will it just run off the 
timer and preempt X away? You say changing priorities doesnt help, but I cant 
tell if you tried this: run the processing sched_normal at lower priority 
than X.

Con

Re: tvtime and the Linux 2.6 scheduler

[Billy Biggs]

  Hi Con, thanks for your reply.

Con Kolivas (kernel@kolivas.org):

> A program running as sched_fifo it will preempt absolutely everything
> regardless of how it behaves. It sounds like it's giving X less cpu
> time to draw the frame each time until eventually the processing fails
> to capture the frame and then X smooths out again. I cant pretend to
> understand how your application blocks (as you say) between X and
> tvtime, but does tvtime not try to schedule until X has finished using
> up cpu or will it just run off the timer and preempt X away? You say
> changing priorities doesnt help, but I cant tell if you tried this:
> run the processing sched_normal at lower priority than X.

  We call XvShmPutImage and then do an XSync, which will wait until X
sends a message indicating it has processed all outstanding requests.
The XvShmPutImage uploads the image to video memory immediately, so the
call to XSync blocks us until it is complete.  We are not preempting the
X server.

  I will get accurate information about the priorities of all processes
involved here.

  -Billy

Re: tvtime and the Linux 2.6 scheduler

[Oswald Buddenhagen]

On Mon, May 24, 2004 at 02:54:19AM +1000, Con Kolivas wrote:
> Your example of poor performance is one when the cpu performance is
> marginal to get exactly 30 fps processed and on the screen. The cpu
> overhead in 2.6 is slightly higher than 2.4 so a borderline case may
> be just pushed over. 
> 
"his" example is in fact mine. 2.4 ghz chew 25 fps. the total cpu load
is at 30-40% (depending on the selected deinterlacer algorithm).

> A program running as sched_fifo it will preempt absolutely everything 
> regardless of how it behaves.
>
errm ... right. i paid too little attention to this "tiny" detail.
*blush*

now i hacked tvtime to simply ignore v4l2 ... and guess what? it works
...
so the new theory is, that tvtime has some problem with v4l2 input
buffering ...

greetings

-- 
Hi! I'm a .signature virus! Copy me into your ~/.signature, please!
--
Chaos, panic, and disorder - my work here is done.

Re: tvtime and the Linux 2.6 scheduler

[Ingo Molnar]

* Con Kolivas <kernel@kolivas.org> wrote:

> >      33 ms : time per NTSC frame
> 
> snip
> 
> The followup email from someone describing good performance may help
> us understand what's going on. Your example of poor performance is one
> when the cpu performance is marginal to get exactly 30 fps processed
> and on the screen. The cpu overhead in 2.6 is slightly higher than 2.4
> so a borderline case may be just pushed over. 

most of the cpu overhead comes from HZ=1000. Especial with SCHED_FIFO
there should be minimal (if any) impact from the scheduler changes -
SCHED_FIFO tasks get all CPU time, no ifs and whens.

could people who experience tvtime performance problems apply the patch
below to change HZ back to 100? Does it have any impact?

	Ingo

--- linux/include/asm-i386/param.h.orig	
+++ linux/include/asm-i386/param.h	
@@ -2,7 +2,7 @@
 #define _ASMi386_PARAM_H
 
 #ifdef __KERNEL__
-# define HZ		1000		/* Internal kernel timer frequency */
+# define HZ		100		/* Internal kernel timer frequency */
 # define USER_HZ	100		/* .. some user interfaces are in "ticks" */
 # define CLOCKS_PER_SEC		(USER_HZ)	/* like times() */
 #endif

Re: tvtime and the Linux 2.6 scheduler

[Nick Piggin]

Ingo Molnar wrote:
> * Con Kolivas <kernel@kolivas.org> wrote:
> 
> 
>>>     33 ms : time per NTSC frame
>>
>>snip
>>
>>The followup email from someone describing good performance may help
>>us understand what's going on. Your example of poor performance is one
>>when the cpu performance is marginal to get exactly 30 fps processed
>>and on the screen. The cpu overhead in 2.6 is slightly higher than 2.4
>>so a borderline case may be just pushed over. 
> 
> 
> most of the cpu overhead comes from HZ=1000. Especial with SCHED_FIFO
> there should be minimal (if any) impact from the scheduler changes -
> SCHED_FIFO tasks get all CPU time, no ifs and whens.
> 
> could people who experience tvtime performance problems apply the patch
> below to change HZ back to 100? Does it have any impact?
> 

Just one other thing - realtime scheduling was basically broken up
until around 2.6.5. Before starting any tests, please ensure first
that you are using at least the 2.6.5 kernel. Thanks.

Re: tvtime and the Linux 2.6 scheduler

[Ingo Molnar]

* Nick Piggin <nickpiggin@yahoo.com.au> wrote:

> Just one other thing - realtime scheduling was basically broken up
> until around 2.6.5. Before starting any tests, please ensure first
> that you are using at least the 2.6.5 kernel. Thanks.

you mean the spurious 'queue to end of prio-queue' bug noticed by Joe
Korty? tvtime should not be affected by this one. This bug only hits if
there are multiple SCHED_FIFO tasks on the same priority level - tvtime
is a single-process application.

	Ingo

Re: tvtime and the Linux 2.6 scheduler

[Nick Piggin]

Ingo Molnar wrote:
> * Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> 
> 
>>Just one other thing - realtime scheduling was basically broken up
>>until around 2.6.5. Before starting any tests, please ensure first
>>that you are using at least the 2.6.5 kernel. Thanks.
> 
> 
> you mean the spurious 'queue to end of prio-queue' bug noticed by Joe
> Korty?

Yes I did mean that one.

  tvtime should not be affected by this one. This bug only hits if
> there are multiple SCHED_FIFO tasks on the same priority level - tvtime
> is a single-process application.
> 

Yes I see.

Re: tvtime and the Linux 2.6 scheduler

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> you mean the spurious 'queue to end of prio-queue' bug noticed by Joe
> Korty? tvtime should not be affected by this one. This bug only hits
> if there are multiple SCHED_FIFO tasks on the same priority level -
> tvtime is a single-process application.

i checked out the source and it uses multiple threads if recording is
done to disk - but the disk writer thread is at normal priority, only
the capture thread is using SCHED_FIFO.

	Ingo

Re: tvtime and the Linux 2.6 scheduler

[szonyi calin]

 --- Billy Biggs <vektor@dumbterm.net> a écrit : >   I am the
author of tvtime, a TV application with advanced
> image
> processing algorithms.  Some users are complaining about poor
> performance under Linux 2.6, and I would like more information
> about how
> tvtime will be treated by the scheduler.  Here is an example
> of the
> intended usage:
> 
>   - Program running as root and SCHED_FIFO
>   - NTSC, input ~30 fps, each field processed for an output of
> ~60 fps
>   - CPU intensive processing, say 9 ms per field on my P3-733
>   - with a typical AGP card, the X driver takes 4 ms to draw
>   - Wait using /dev/rtc set to 1024 Hz
> 
>   for(;;)
>       9 ms : process frame
>       4 ms : draw frame
>       3 ms : wait until next field time using /dev/rtc
>       9 ms : process frame
>       4 ms : draw frame
>       3 ms : block on /dev/video0 for next frame
>      -----
>      33 ms : time per NTSC frame
> 
>   The theory is that Linux classifies this as a CPU hog
> regardless of
> its priority, and preempts tvtime with other processes. 
> Oswald
> Buddenhagen describes the effect as this:
> 
>   "[...] it starts up fine, but after a few seconds (when the
> scheduler
> gathered some stats) ... well, it looks funny: the scene goes
> roughly
> exponentially into slow motion, then there is a frame drop and
> the
> process starts over.  this behaviour can be observed at any
> priority,
> which is clearly against the claim "no normally priorized
> interactive
> process will preempt a highly priorized cpu-hog" that i've
> read
> somewhere.  the xserver priority does not change anything,
> either;"
> 
>   Avoiding root/SCHED_FIFO and using usleep() instead of
> /dev/rtc seems
> to exhibit the same behavior.
> 
>   Thoughts?
> 

For me it works ok
Running tvtime with 2.6.6-mm5 load average 26 (twenty six) it
works 
almost perfect (no frame skipped -- at least not reported by
tvtime)
at full rate (50 frames/sec). Computer: AMD Duron 700 MHz, 256MB
Ram,
VIA KT133 chipset, AGP 4x, Ati Radeon 7200 

Maybe your users are renicing tvtime ?
AFAIK it is not recommended to renice a process which uses X
server and 
wants realtime processing 

Ohhhhhhhhhh
Take a look at this. Is tvtime's fault (X server does'n have the
chance to display what tvtime processed because tvtime gets
almost all the cpu)

sony@grinch -01:40:31- 0 jobs, ver 2.05b.0 3 
 /~ $ tvtime -s -v
Running tvtime 0.9.12.
Reading configuration from /usr/local/etc/tvtime/tvtime.xml
Reading configuration from /home/sony/.tvtime/tvtime.xml
cpuinfo: CPU AMD Duron(tm) Processor, family 6, model 3,
stepping 1.
cpuinfo: CPU measured at 700,579MHz.
tvtime: Cannot set priority to -19: Permission denied.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

xcommon: Display :0.0, vendor The XFree86 Project, Inc, XFree86
4.4.0
xfullscreen: Single-head detected, pixel aspect will be
calculated.
xfullscreen: Pixel aspect ratio on the primary head is: 75/76 ==
0,99.
xfullscreen: Using the XFree86-VidModeExtension to calculate
fullscreen size.
xfullscreen: Fullscreen to 0,0 with size 1024x768.
xcommon: Have XTest, will use it to ping the screensaver.
xcommon: Pixel aspect ratio 75:76.
xcommon: No window properties found for EWMH.
xcommon: Window manager is not EWMH compliant.
xcommon: Pixel aspect ratio 75:76.
xcommon: Displaying in a 778x576 window inside 778x576 space.
xvoutput: Using XVIDEO adaptor 69: ATI Radeon Video Overlay.
speedycode: Using MMXEXT optimized functions.
station: Reading stationlist from
/home/sony/.tvtime/stationlist.xml
videoinput: Using video4linux2 driver 'bttv', card 'BT878 video
(AVerMedia TVCaptur' (bus PCI:0000:00:09.0).
videoinput: Version is 2318, capabilities 5010015.
tvtime: Sampling input at 720 pixels per scanline.
xcommon: Pixel aspect ratio 75:76.
xcommon: Displaying in a 778x576 window inside 778x576 space.
xcommon: Received a map, marking window as visible (50)

I hope the line above answers your question

>   -Billy
> 
Calin

P.S. Load average is that high because i run freenet
 http://freenet.sourceforge.net -- lots of java processes

=====
--
A mouse is a device used to point at 
the xterm you want to type in.
Kim Alm on a.s.r.


	

	
		
Yahoo! Mail : votre e-mail personnel et gratuit qui vous suit partout ! 
Créez votre Yahoo! Mail sur http://fr.benefits.yahoo.com/

Dialoguez en direct avec vos amis grâce à Yahoo! Messenger !Téléchargez Yahoo! Messenger sur http://fr.messenger.yahoo.com

Re: tvtime and the Linux 2.6 scheduler

[Bill Davidsen]

szonyi calin wrote:
>  --- Billy Biggs <vektor@dumbterm.net> a écrit : >   I am the
> author of tvtime, a TV application with advanced
> 
>>image
>>processing algorithms.  Some users are complaining about poor
>>performance under Linux 2.6, and I would like more information
>>about how
>>tvtime will be treated by the scheduler.  Here is an example
>>of the
>>intended usage:
>>
>>  - Program running as root and SCHED_FIFO
>>  - NTSC, input ~30 fps, each field processed for an output of ~60 fps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
>>  - CPU intensive processing, say 9 ms per field on my P3-733
>>  - with a typical AGP card, the X driver takes 4 ms to draw
>>  - Wait using /dev/rtc set to 1024 Hz
>>
>>  for(;;)
>>      9 ms : process frame
>>      4 ms : draw frame
>>      3 ms : wait until next field time using /dev/rtc
>>      9 ms : process frame
>>      4 ms : draw frame
>>      3 ms : block on /dev/video0 for next frame
>>     -----
>>     33 ms : time per NTSC frame
	[___snip___]
> For me it works ok
> Running tvtime with 2.6.6-mm5 load average 26 (twenty six) it
> works 
> almost perfect (no frame skipped -- at least not reported by
> tvtime)
> at full rate (50 frames/sec). Computer: AMD Duron 700 MHz, 256MB
~~~~~~~~~~~~~~~~^^^
> Ram,
> VIA KT133 chipset, AGP 4x, Ati Radeon 7200 

There have been pretty good explanations of this, so I'll just say that 
it might be that someone with a machine of limited capacity would be 
able to generate 50fps and not 60fps. In much of Europe the TV flickers 
at the same frequency as the lights ;-)

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: tvtime and the Linux 2.6 scheduler

[Tobias Diedrich]

Bill Davidsen wrote:

> There have been pretty good explanations of this, so I'll just say that 
> it might be that someone with a machine of limited capacity would be 
> able to generate 50fps and not 60fps. In much of Europe the TV flickers 
> at the same frequency as the lights ;-)

Rather the other way around.

768*288*50*2 = 22118400
640*240*60000*2/1001 = 18413586

So the amount of data to process is higher for PAL (in square pixel
resolution).

-- 
Tobias						PGP: http://9ac7e0bc.2ya.com
He said he hadn't had a byte in three days.
I had a short, so I split it with him.

Re: tvtime and the Linux 2.6 scheduler

[Ingo Molnar]

* Billy Biggs <vektor@dumbterm.net> wrote:

>   for(;;)
>       9 ms : process frame
>       4 ms : draw frame
>       3 ms : wait until next field time using /dev/rtc
>       9 ms : process frame
>       4 ms : draw frame
>       3 ms : block on /dev/video0 for next frame
>      -----
>      33 ms : time per NTSC frame
> 
>   The theory is that Linux classifies this as a CPU hog regardless of
> its priority, and preempts tvtime with other processes. [...]

this would indicate a pretty broken scheduler. To prove (or exclude)
this possibility, could you apply the attached debugging patch? The
patch checks whether we ever switch away from a still running (and
hence, on-runqueue) RT task to a non-RT task.

(NOTE: dont run this patch on an SMP kernel, the debugging message will
deadlock there due to spinlock recursion.)

	Ingo

--- linux/kernel/sched.c.orig	
+++ linux/kernel/sched.c	
@@ -2279,6 +2279,7 @@ switch_tasks:
 	prev->timestamp = now;
 
 	if (likely(prev != next)) {
+		WARN_ON(rt_task(prev) && (prev->state == TASK_RUNNING) && !rt_task(next));
 		next->timestamp = now;
 		rq->nr_switches++;
 		rq->curr = next;

Re: tvtime and the Linux 2.6 scheduler

[Oswald Buddenhagen]

On Mon, May 24, 2004 at 11:20:57AM +0200, Ingo Molnar wrote:
> * Billy Biggs <vektor@dumbterm.net> wrote:
> >   The theory is that Linux classifies this as a CPU hog regardless of
> > its priority, and preempts tvtime with other processes. [...]
> 
> this would indicate a pretty broken scheduler. To prove (or exclude)
> this possibility, could you apply the attached debugging patch?
>
done. no messages in the kernel syslog.

greetings

-- 
Hi! I'm a .signature virus! Copy me into your ~/.signature, please!
--
Chaos, panic, and disorder - my work here is done.

Re: tvtime and the Linux 2.6 scheduler

[Redeeman]

using nicksched solves this, now tvtime only uses 1% again! yay!

On Sun, 2004-05-23 at 17:48, Billy Biggs wrote:
>   I am the author of tvtime, a TV application with advanced image
> processing algorithms.  Some users are complaining about poor
> performance under Linux 2.6, and I would like more information about how
> tvtime will be treated by the scheduler.  Here is an example of the
> intended usage:
> 
>   - Program running as root and SCHED_FIFO
>   - NTSC, input ~30 fps, each field processed for an output of ~60 fps
>   - CPU intensive processing, say 9 ms per field on my P3-733
>   - with a typical AGP card, the X driver takes 4 ms to draw
>   - Wait using /dev/rtc set to 1024 Hz
> 
>   for(;;)
>       9 ms : process frame
>       4 ms : draw frame
>       3 ms : wait until next field time using /dev/rtc
>       9 ms : process frame
>       4 ms : draw frame
>       3 ms : block on /dev/video0 for next frame
>      -----
>      33 ms : time per NTSC frame
> 
>   The theory is that Linux classifies this as a CPU hog regardless of
> its priority, and preempts tvtime with other processes.  Oswald
> Buddenhagen describes the effect as this:
> 
>   "[...] it starts up fine, but after a few seconds (when the scheduler
> gathered some stats) ... well, it looks funny: the scene goes roughly
> exponentially into slow motion, then there is a frame drop and the
> process starts over.  this behaviour can be observed at any priority,
> which is clearly against the claim "no normally priorized interactive
> process will preempt a highly priorized cpu-hog" that i've read
> somewhere.  the xserver priority does not change anything, either;"
> 
>   Avoiding root/SCHED_FIFO and using usleep() instead of /dev/rtc seems
> to exhibit the same behavior.
> 
>   Thoughts?
> 
>   -Billy
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Regards, Redeeman
redeeman@metanurb.dk