2.6.33-7-rt30 and 2.6.38 comparison

2.6.33-7-rt30 and 2.6.38 comparison

[Madovsky]

Just tried to compare real time app
at work with 2.6.33.7-rt30 and 2.6.38 and
the result is 2.6.38 is very impressive...

Re: 2.6.33-7-rt30 and 2.6.38 comparison

[Ilyes Gouta]

Hi,

Would you please elaborate a bit more?

Thanks,

-Ilyes

On Fri, Mar 18, 2011 at 9:15 PM, Madovsky <infos@madovsky.org> wrote:
> Just tried to compare real time app
> at work with 2.6.33.7-rt30 and 2.6.38 and
> the result is 2.6.38 is very impressive...
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.33-7-rt30 and 2.6.38 comparison

[Madovsky]

not a lot, I'm only a rt user, not linux developer.
so my compare is only a view of for example
latency in audio/video....


> ----- Original Message ----- 
> From: "Ilyes Gouta" <ilyes.gouta@gmail.com>
> To: "Madovsky" <infos@madovsky.org>
> Cc: <linux-rt-users@vger.kernel.org>
> Sent: Friday, March 18, 2011 4:16 PM
> Subject: Re: 2.6.33-7-rt30 and 2.6.38 comparison
>
>
>> Hi,
>>
>> Would you please elaborate a bit more?
>>
>> Thanks,
>>
>> -Ilyes
>>
>> On Fri, Mar 18, 2011 at 9:15 PM, Madovsky <infos@madovsky.org> wrote:
>>> Just tried to compare real time app
>>> at work with 2.6.33.7-rt30 and 2.6.38 and
>>> the result is 2.6.38 is very impressive...
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe 
>>> linux-rt-users" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" 
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: 2.6.33-7-rt30 and 2.6.38 comparison

[Madovsky]

I compiled 2.6.38 with exactly the same
config so I made different audio/video task
and very stable with short latency


----- Original Message ----- 
From: "Madovsky" <infos@madovsky.org>
To: "linux-rt-users" <linux-rt-users@vger.kernel.org>
Sent: Friday, March 18, 2011 4:28 PM
Subject: Re: 2.6.33-7-rt30 and 2.6.38 comparison


> not a lot, I'm only a rt user, not linux developer.
> so my compare is only a view of for example
> latency in audio/video....
>
>
>> ----- Original Message ----- 
>> From: "Ilyes Gouta" <ilyes.gouta@gmail.com>
>> To: "Madovsky" <infos@madovsky.org>
>> Cc: <linux-rt-users@vger.kernel.org>
>> Sent: Friday, March 18, 2011 4:16 PM
>> Subject: Re: 2.6.33-7-rt30 and 2.6.38 comparison
>>
>>
>>> Hi,
>>>
>>> Would you please elaborate a bit more?
>>>
>>> Thanks,
>>>
>>> -Ilyes
>>>
>>> On Fri, Mar 18, 2011 at 9:15 PM, Madovsky <infos@madovsky.org> wrote:
>>>> Just tried to compare real time app
>>>> at work with 2.6.33.7-rt30 and 2.6.38 and
>>>> the result is 2.6.38 is very impressive...
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>> linux-rt-users" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe 
>>> linux-rt-users" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 

Re: 2.6.33-7-rt30 and 2.6.38 comparison

[Niccolò Belli]

:/ @#!*##@
I thought they released 2.6.38-rt :'(

RE: 2.6.33-7-rt30 and 2.6.38 comparison

[Victor henri]

Hi

I have observed similar behavior with newer non-RT kernels but there were something strange to me ; 

Same results (very few X-runs, if any) with :

  
 -  on one hand, on an Asus laptop with a Core i3  : latency 2.3 msec 
with the 2.6.33.7.2-rt kernel and 23 msec with a 2.6.37 preempt kernel;

   
 -  on the other hand, on a Lenovo Thinkpad Core i5 :  latency 2.3 msec 
with the 2.6.33.7.2-rt kernel and 2.3 msec (yes the same) with a 2.6.37 
preempt kernel !!

Of course the Lenovo Thinkpad is a very good laptop, but how come that there is such a big difference? 

So, I have observed the same quality of performance as Madovsky but not 
on every machine I've used... I don't have a clue why....

Victor

----------------------------------------
> From: infos@madovsky.org
> To: linux-rt-users@vger.kernel.org
> Subject: Re: 2.6.33-7-rt30 and 2.6.38 comparison
> Date: Fri, 18 Mar 2011 16:34:51 -0400
>
> I compiled 2.6.38 with exactly the same
> config so I made different audio/video task
> and very stable with short latency
>
>
> ----- Original Message -----
> From: "Madovsky" 
> To: "linux-rt-users" 
> Sent: Friday, March 18, 2011 4:28 PM
> Subject: Re: 2.6.33-7-rt30 and 2.6.38 comparison
>
>
> > not a lot, I'm only a rt user, not linux developer.
> > so my compare is only a view of for example
> > latency in audio/video....
> >
> >
> >> ----- Original Message -----
> >> From: "Ilyes Gouta" 
> >> To: "Madovsky" 
> >> Cc: 
> >> Sent: Friday, March 18, 2011 4:16 PM
> >> Subject: Re: 2.6.33-7-rt30 and 2.6.38 comparison
> >>
> >>
> >>> Hi,
> >>>
> >>> Would you please elaborate a bit more?
> >>>
> >>> Thanks,
> >>>
> >>> -Ilyes
> >>>
> >>> On Fri, Mar 18, 2011 at 9:15 PM, Madovsky  wrote:
> >>>> Just tried to compare real time app
> >>>> at work with 2.6.33.7-rt30 and 2.6.38 and
> >>>> the result is 2.6.38 is very impressive...
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe
> >>>> linux-rt-users" in
> >>>> the body of a message to majordomo@vger.kernel.org
> >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>>>
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe
> >>> linux-rt-users" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
 		 	   		  --
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.33-7-rt30 and 2.6.38 comparison

[jordan]

Hi Victor,

> Same results (very few X-runs, if any) with :
>
>  -  on one hand, on an Asus laptop with a Core i3  : latency 2.3 msec
> with the 2.6.33.7.2-rt kernel and 23 msec with a 2.6.37 preempt kernel;

23msec with 2.6.37 preemptive kernel is terrible. Not one of my
machines would perform that badly (and most use 2.6.37.3). Some are
also considerably older (like 5years older) than your asus laptop..

>  -  on the other hand, on a Lenovo Thinkpad Core i5 :  latency 2.3 msec
> with the 2.6.33.7.2-rt kernel and 2.3 msec (yes the same) with a 2.6.37
> preempt kernel !!

that's not surprising, that is what i would expect to see. 2.6.33-rt
is getting quite dated, and old. For industrial / embedded
applications it might be the only option but for Linux audio / video
editing, there are preemptive kernels that work very well.

> Of course the Lenovo Thinkpad is a very good laptop, but how come that there is such a big difference?

someone can probably answer this better, but i think it comes down to
hardware / software design / implementation - some PCs work better
because of the quality of parts, the software and how well they
integrate.

*** ....and how well-supported each component is under Linux. ***

> So, I have observed the same quality of performance as Madovsky but not
> on every machine I've used... I don't have a clue why....

I also have observed the same thing.  i actually only use RT on 1
computer right now. On the Rest of my machines, 2.6.37.3-zen (PREEMPT,
BFS and BFQ v2.0 + optimizations) works wonders.
In every use i have 2.6.37 is better ( i am a pro-audio person ).

I will say between 2.6.33 and 2.6.38 they have been many MANY changes,
so i am not surprised that 2.6.38 performs as well as 2.6.33-rt 9or
even just close). 2.6.38 (i believe) is the First kernel to have BKL
(Big kernel Lock) completely removed ....and also CFS (completely fair
scheduler) have changed quite a bit (since .33) and has been more
optimized. those 2 factors alone, should reduce latency. Coupled with
other fixes and changes in the kernel - should explain these kinds of
observations.

just my 2 cents

jordan
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: 2.6.33-7-rt30 and 2.6.38 comparison

[Victor henri]

Hello Jordan

> > - on one hand, on an Asus laptop with a Core i3 : latency 2.3 msec
> > with the 2.6.33.7.2-rt kernel and 23 msec with a 2.6.37 preempt kernel;
>
> 23msec with 2.6.37 preemptive kernel is terrible.

For sure 23 msec is terrible; I was so surprised to see that!

 Not one of my
> machines would perform that badly (and most use 2.6.37.3). Some are
> also considerably older (like 5years older) than your asus laptop..
>
> > - on the other hand, on a Lenovo Thinkpad Core i5 : latency 2.3 msec
> > with the 2.6.33.7.2-rt kernel and 2.3 msec (yes the same) with a 2.6.37
> > preempt kernel !!
>
> that's not surprising, that is what i would expect to see. 2.6.33-rt
> is getting quite dated, and old. For industrial / embedded
> applications it might be the only option but for Linux audio / video
> editing, there are preemptive kernels that work very well.
>
> > Of course the Lenovo Thinkpad is a very good laptop, but how come that there is such a big difference?
>
> someone can probably answer this better, but i think it comes down to
> hardware / software design / implementation - some PCs work better
> because of the quality of parts, the software and how well they
> integrate.
>
> *** ....and how well-supported each component is under Linux. ***

Yes Lenovo Thinkpad are famous for having good performance... and furthermore the T410s is "Ubuntu certified". However I experience hang on with the 2.6.33.7.2-rt kernel, but I have a touchscreen (input = 4 fingers multitouch working natively under Maverick... really great) and I suspect the mutitouch screen is responsible of this. Hopefully the performaces og the preempt are great!

>
> > So, I have observed the same quality of performance as Madovsky but not
> > on every machine I've used... I don't have a clue why....
>
> I also have observed the same thing. i actually only use RT on 1
> computer right now. On the Rest of my machines, 2.6.37.3-zen (PREEMPT,
> BFS and BFQ v2.0 + optimizations) works wonders.
> In every use i have 2.6.37 is better ( i am a pro-audio person ).
>
> I will say between 2.6.33 and 2.6.38 they have been many MANY changes,
> so i am not surprised that 2.6.38 performs as well as 2.6.33-rt 9or
> even just close). 2.6.38 (i believe) is the First kernel to have BKL
> (Big kernel Lock) completely removed ....and also CFS (completely fair
> scheduler) have changed quite a bit (since .33) and has been more
> optimized. those 2 factors alone, should reduce latency. Coupled with
> other fixes and changes in the kernel - should explain these kinds of
> observations.
>
> just my 2 cents
>
> jordan

Thank you for those explanations

Victor
 		 	   		  

Re: 2.6.33-7-rt30 and 2.6.38 comparison

[jordan]

Hey Victor,

> For sure 23 msec is terrible; I was so surprised to see that!

So was I man :)  On my old Dell inspiron 6400, coreduo with 2.6.37, i
am running quite stable at 2.3ms (as in no xruns, at all).  it's a
semi-slow machine, with 1 sketchy PCI bus, too. It has been recycled
into a rackmount, and has "on/off" operation. It's still good enough
though, to run several VSTi's at once, some effects chains,
sooperlooper and a sampler. I never have problems with it. i use it
live (with my band). RT runs like crap on it, so i used 2.6.37 with
BFS.

I'm very impressed with recent kernels, using a non-RT kernel seems to
not be a problem.
although, most of the time, they do require tweaking.

i won't be excited about RT-kernels, until i see 2.6.38-rt ~ that
should be a good one!

I recycle junk laptops - ie: busted display / keyboard / failed GPU /
etc ~ useless to most people, but they are easily converted into
dedicated music gear. (synth-modules, VSTi hosts, FX rack).
So i am often working with "sub-standard" equipment. surprisingly most
of the time you can tune
them quite well ~ and get very reasonable performance. RT can
misbehave on some gear so i use BFS scheduler in those corner cases. I
mostly will use Zen-kernel (www.zen-kernel.org) as that kernel can be
tuned up very well, and has all of the bellz and whistles.

> Yes Lenovo Thinkpad are famous for having good performance... and furthermore the T410s is "Ubuntu certified". However I experience hang on with the 2.6.33.7.2-rt kernel, but I have a touchscreen (input = 4 fingers multitouch working natively under Maverick... really great) and I suspect the mutitouch screen is responsible of this. Hopefully the performaces og the preempt are great!

I have heard lenovo laptops are quite nice for running linux. :)
...in this case it could be your multi-touch, but it also could be
your video card. . On my one machine if i use a compositor (compiz,
metacity, etc) it randomly will have soft lockups. All of the audio
stuff still will work but the display goes down the drain.  On another
machine, an older version of BFQ ( I/O scheduler ), would potentially
cause lockups. (although i had thought that it was my ATI card) both
of those machines are quite stable now though. :)

jordan

RE: 2.6.33-7-rt30 and 2.6.38 comparison

[Victor henri]

Hello Jordan

>
> Hi Victor,
>
> > Same results (very few X-runs, if any) with :
> >
> >  -  on one hand, on an Asus laptop with a Core i3  : latency 2.3 msec
> > with the 2.6.33.7.2-rt kernel and 23 msec with a 2.6.37 preempt kernel;
>
> 23msec with 2.6.37 preemptive kernel is terrible. Not one of my
> machines would perform that badly (and most use 2.6.37.3). Some are
> also considerably older (like 5years older) than your asus laptop..
>
> >  -  on the other hand, on a Lenovo Thinkpad Core i5 :  latency 2.3 msec
> > with the 2.6.33.7.2-rt kernel and 2.3 msec (yes the same) with a 2.6.37
> > preempt kernel !!
>
> that's not surprising, that is what i would expect to see. 2.6.33-rt
> is getting quite dated, and old. For industrial / embedded
> applications it might be the only option but for Linux audio / video
> editing, there are preemptive kernels that work very well.
>
> > Of course the Lenovo Thinkpad is a very good laptop, but how come that there is such a big difference?
>
> someone can probably answer this better, but i think it comes down to
> hardware / software design / implementation - some PCs work better
> because of the quality of parts, the software and how well they
> integrate.
>
> *** ....and how well-supported each component is under Linux. ***
>
> > So, I have observed the same quality of performance as Madovsky but not
> > on every machine I've used... I don't have a clue why....
>
> I also have observed the same thing. i actually only use RT on 1
> computer right now. On the Rest of my machines, 2.6.37.3-zen (PREEMPT,
> BFS and BFQ v2.0 + optimizations) works wonders.
> In every use i have 2.6.37 is better ( i am a pro-audio person ).

Could you briefly summarize why use zen-kernel instead of vanilla? As 
far as I understand, the zen-kernel brings features that may not (yet) 
be available on vanilla kernel, (such as those scheduler BFS and BFQ?) Could please say wich optimization is needed for the best performances? I'd like to give it a try actually. Or is there a place where I can find all these recommendations; I'm talking about the special features that you can find specifically in a zen-kernel, and that can help further for lower latency for music performance.

On the other hand, I've checked the www.zen-kernel.org website; the most recent kernel patch seem to be for 2.6.36 kernel; where did you find a 2.6.37 zen kernel? 

>
> I will say between 2.6.33 and 2.6.38 they have been many MANY changes,
> so i am not surprised that 2.6.38 performs as well as 2.6.33-rt 9or
> even just close). 2.6.38 (i believe) is the First kernel to have BKL
> (Big kernel Lock) completely removed ....and also CFS (completely fair
> scheduler) have changed quite a bit (since .33) and has been more
> optimized. those 2 factors alone, should reduce latency. Coupled with
> other fixes and changes in the kernel - should explain these kinds of
> observations.
>
> just my 2 cents
>
> jordan
> --

Victor
 		 	   		  

Re: 2.6.33-7-rt30 and 2.6.38 comparison

[jordan]

Hey Victor,

Sorry for the delayed response, i've been a little busy.

> Could you briefly summarize why use zen-kernel instead of vanilla? As
> far as I understand, the zen-kernel brings features that may not (yet)
> be available on vanilla kernel, (such as those scheduler BFS and BFQ?)

I use zen-kernel over vanilla, for the simple fact that it works, and
i don't have to apply multiple patches to a vanilla source. BFS will
likely never be upstream (which is old news), and BFQ i am not sure
about.

> Could please say wich optimization is needed for the best performances? I'd like to give it a try actually. Or is there a place where I can find all these recommendations; I'm talking about the special features that you can find specifically in a zen-kernel, and that can help further for lower latency for music performance.

BFS is the obvious patch set that helps with low-latency and cpu
scheduling. I do use BFQ io scheduler as well, and i have been
experimenting with the latest BFQ Cgroups stuff too. An example of
that can be found in the archlinux AUR ~ kernel26-ck.

I prefer to use non-vanilla, because often i am using more than a few
patches, on my kernel. Zen-kernel, -ck, etc already had other
tuning/fixes done to them.  then i can do the fine-tuning.

Other tunings you can do..? well, there are lots. When compiling a
kernel, use CFLAG optimizations.
Off hand, easy one's that are safe - CFLAGS='-march=native
-mtune=native' (those 2 will compile the code for your CPU). but there
are many that you can use. read through GCC documentation, Gentoo,
sourcemage and archlinux resources on compiling a kernel. (in fact
gentoo, and Arch have the best documentation on any Linux subject -
including guides on maximizing performance).

you could also only compile modules that you need (saves on compiling time too).

after getting a new kernel to run that is decent. You can muddle
through kernel subsystems;

disk scheduler tunables:

/sys/block/sda/queue
/sys/block/sda/queue/iosched

Cpu scheduler tunables:

/proc/sys/kernel

VM subsystem

/proc/sys/vm

I would have to go into way too much detail here, to explain these
and which things can be tuned.
But i think you will find that google is your friend in this case.
The idea is that you can adjust certain tunables to adjust the system
to better suit your needs. sometimes, reducing latency, or possibly
tuning scheduler performance, etc.

You then can store your changes in /etc/rc.local and /etc/sysctl.conf
- to be run on boot.

There are plenty of tools for benchmarking, latency-tests, etc.
...again, google.

There are also other daemons and tools that could be used...
Ulatencyd, verynice, schedtools, Cgroups(although, i wouldn't
recommend Cgroups just yet),etc. While not are meant for RT-related
Jack applications (actually some will hinder jackd performance).. Some
can be used to control/limit other processes that may be interferring,
introducing latency or causing xruns.  In most cases, you will require
testing/benchmarking, and also just as importantly - "real world
tests" (as in using your system).

tuning all hardware is also very important. using patched drivers, if
required. tuning your disk with tools like sdparm, hdparm. editing
your fstab, etc. I also tend to disable any hardware not being used.
(no kernel support/disable in bios)... I would run a stripped down
desktop too. Although, i do run gnome, i've optimized it.  Tuning your
IRQ manually, over using irqbalance, or rtirq...

> On the other hand, I've checked the www.zen-kernel.org website; the most recent kernel patch seem to be for 2.6.36 kernel; where did you find a 2.6.37 zen kernel?

master branch of Zen-stable (using Git). I have it locally, and update
the tree, when I am about to compile the kernel. I don't download
snapshots, or Zen-kernel patches, git only.

sorry that i don't have more specific info, but your asked a very
broad question.

cheerz

jordan