Re: [ANNOUNCE] 3.0-rc7-rt0

["Madovsky" <infos@madovsky.org>]

Hi Thomas,

I just compiled the kernel with the patch on my whole cluster
of 6 fedora10 and the results are very stable and smooth.
the only little thing is the cluster software heartbeat / pacemaker
takes 20% of cpu that 11% before on a dual xeon nocona with
linux command "top".

Thanks

Franck


----- Original Message ----- 
From: "Armin Steinhoff" <armin@steinhoff.de>
To: "Thomas Gleixner" <tglx@linutronix.de>
Cc: "LKML" <linux-kernel@vger.kernel.org>; "linux-rt-users" 
<linux-rt-users@vger.kernel.org>; "Peter Zijlstra" <peterz@infradead.org>; 
"Ingo Molnar" <mingo@elte.hu>; "Carsten Emde" <ce@ceag.ch>; "Clark Williams" 
<williams@redhat.com>; "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>; 
"Kumar Gala" <galak@gate.crashing.org>; "Ralf Baechle" <ralf@linux-mips.org>
Sent: Wednesday, July 20, 2011 7:08 AM
Subject: Re: [ANNOUNCE] 3.0-rc7-rt0


>
> Thomas,
>
> congratulations !
>
> The results are really amazing !!
>
> https://www.osadl.org/Latency-plot-of-system-in-rack-4-slot.qa-latencyplot-r4s6.0.html?latencies=&showno=&slider=57
>
> --Armin
>
>
>
> Thomas Gleixner wrote:
>> Dear RT Folks,
>>
>> I'm pleased to announce the first drop of the 3.0-rc7 based RT
>> patch.
>>
>> It's been quite a while since 2.6.33-rt, but I went through a very
>> painful experience while trying to get a 2.6.38-rt stabilized. The
>> beast insisted on destroying filesystems with reproduction times
>> measured in days and the total refusal to reveal at least a
>> minimalistic hint to debug the root cause. Staring into completely
>> useless traces for months is not a very pleasant pastime.
>>
>> That's the very first problem in the RT history which I gave up on.
>>
>> [The truth: Linus avoiding the final 2.6.42 release made all my
>>   ultimate plans go down the drain ... ]
>>
>> Though while trying to analyse the problem I had plenty of time to
>> twist my brain around the existing RT approach and its shortcomings.
>>
>> The main issue which RT is fighting with is the ever growing per cpu
>> variable usage and the assumptions which are built around it. The
>> existing RT approach to work around this with PER_CPU_LOCKED
>> constructs and hand the CPU number around simply does not work anymore
>> because the number of sites which need to be patched is way too large
>> and the resulting mess in the code is neither acceptable nor
>> maintainable.
>>
>> After lenghty and fruitful discussions with Peter Zijlstra - thanks a
>> lot Peter! - we finally agreed on trying a totally different approach
>> to tackle these issues: disabling migration over spinlock and get_cpu
>> sections. This had been discussed before, but nobody ever considered
>> to sit down and make it work.
>>
>> This keeps the semantics which are expected by the per cpu users,
>> while keeping the regions preemptible. As a side effect, it allows us
>> to run softirq handlers directly from irq threads on local_bh_enable
>> which was a long desired feature to lower the performance impact of
>> RT.
>>
>> Changing this required a major refactoring of the RT patch queue,
>> which took some time as I had to go through every single patch, fold
>> fixes back into the right places and sort them into various categories:
>>
>>   - Mainline ready (raw lock annotations, infrastructure patches, code
>>     restructuring...)
>>
>>   - Preparatory (_rt()/_nort() variants of preempt_*(), local_irq_*(),
>>     BUG*(), WARN*() and the annotations in various places)
>>
>>   - Base patches (Reworking the slab/page_alloc code, bit_spinlock
>>     replacements, migrate disable infrastructure ...)
>>
>>   - Full RT patches (sleeping spinlocks and the resulting fixups here
>>     and there)
>>
>> In course of that exercise I weeded out a lot of historically grown
>> hackery and dropped stuff which was not essential for getting it up
>> and running. Thanks to Carsten for reintegrating the tracer addons
>> which he's using for the OSADL test farm:
>>
>>    https://www.osadl.org/?id=1042
>>
>> I probably have missed a few bits and pieces, but the overall outcome
>> is stable and survived testing on various systems. The latency
>> behaviour with cyclictest is on par with 33-rt at least on x86_64/32.
>>
>> The overall patch size has shrunk significantly and the readability
>> (except for the missing changelogs in various patches) is at an
>> acceptable level.
>>
>> If you download the quilt tarball, you'll find various sections:
>>
>> - upstream fixes: Stuff broken upstream which we managed to trip
>>    over. This section contains real weird stuff from simple fixes, over
>>    mainline code which claims to contain (complete bogus) RT support up
>>    to an archaeologic bug in the floppy driver code.
>>
>>    8 patches (size 8892)
>>    7 files changed, 59 insertions(+), 51 deletions(-)
>>
>> - upstream submitted: Stuff which is on LKML already and needs some
>>    follow up.
>>
>>    4 patches (size 9741)
>>    4 files changed, 81 insertions(+), 119 deletions(-)
>>
>> - upstream ready: Stuff which needs a bit polishing and upstream
>>    submission
>>
>>    79 patches (size 232566)
>>    192 files changed, 1204 insertions(+), 1097 deletions(-)
>>
>> - upstream needs work: Stuff which should go upstream, but needs some
>>    or lots of care.
>>
>>    7 patches (size 164120)
>>    49 files changed, 3292 insertions(+), 253 deletions(-)
>>
>> - the real rt stuff:
>>
>>    125 patches (size 280665)
>>    162 files changed, 4327 insertions(+), 592 deletions(-)
>>
>> The overall patch is now:
>>    223 patches (size 680054)
>>    374 files changed, 8950 insertions(+), 2099 deletions(-)
>>
>> Compared that to 2.6.33-rt:
>>    462 patches (size 1396505)
>>    690 files changed, 15994 insertions(+), 5123 deletions(-)
>>
>> That's a significant reduction in size and impact. Some of it is due
>> to the new approach, but we also got quite a lot of the infrastructure
>> patches upstream in the last few kernel releases. Thanks to all folks
>> who have helped to get that done, especially to Peter Zijlstra for
>> getting the preemptible mmu gather problem and lots of the scheduler
>> issues which we discovered in RT over time sorted out!!!
>>
>> What's new in 3.0-rt ?
>>
>>   - No more split soft interrupt threads. We need to analyze whether
>>     this is a good decision.
>>
>>   - softirq handling from the end of interrupt threads and on all
>>     thread sites where a nested local_bh disabled section ends
>>
>>   - SPARSE interrupts and IOMMU interrupt remapping work now
>>
>>   - Split config option CONFIG_PREEMPT_RT into CONFIG_PREEMPT_RT_BASE
>>     and CONFIG_PREEMPT_RT_FULL. RT_BASE covers some of the more complex
>>     changes (e.g. mm/* where we substitute interrupt disabled sections
>>     with per cpu locks and the bit_spinlock to spinlock conversion).
>>     RT_BASE allows us to test and verify these changes independently of
>>     the big RT_FULL modifications. That's mainly a debugability and
>>     maintainability issue.
>>
>> What's the state:
>>
>>     We've done quite some testing on x86 32/64 bit and basic tests on
>>     some ARM/MIPS/POWERPC platforms. Thank God, no file system eating so
>>     far :)
>>
>>     Given the fact that it is a major rewrite it's amazinlgy stable and
>>     I consider it to be the best -rt1 release we ever had. That doesn't
>>     mean that there are no bugs, since it has not had the proper test
>>     coverage yet.
>>
>>     Thanks to Carsten, Clark and Peter for all the help to get this far!
>>
>> Want to help?
>>
>>     Many people offered help in the past and I had to turn them down so
>>     far as refactoring that stuff really is not a task which can be
>>     shared easily. Though now is the point where I can use all the help
>>     you promised to provide.
>>
>>     What's needed?
>>
>>     - Testing, testing, testing ... you know the drill (good bug
>>       reports are 98% of the solution)
>>
>>     - Compare and analyze the performance/troughput impact of the new
>>       approach with 33-rt
>>
>>     - Help mainlining the "upstream ready section"
>>
>>       That means reviewing the patches, cleaning them up, fixing the
>>       changelogs, submitting them through the proper channels ...
>>
>>       Please do not blindly pick any of these patches and submit them
>>       to mailing lists w/o doing the above. Also please coordinate on
>>       the #linux-rt IRC channel on oftc.net so redundant and
>>       conflicting work can be avoided
>>
>>     - Help getting the "upstream needs work" section into shape
>>
>>       All of these patches need a close look and (especially the
>>       hwlatency detector) major cleanups. Please coordinate with the
>>       patch authors and lookout for previous discussions of some of
>>       those on LKML.
>>
>>     - Tend to the FIXME annotations in the RT stuff section
>>
>>       I have annotated some places with /* FIXME ... comments. These
>>       sections are not for the faint hearted and need some serious
>>       review and thought.
>>
>>     - Help with the RCU modifications
>>
>>       That's an easy one. We have a volunteer signed up for this
>>       involuntarily already. Thanks Paul!
>>
>>     - Twist your brain around the schedulability impact of the
>>       migrate_disable() approach.
>>
>>       A really interesting research topic for our friends from the
>>       academic universe. Relevant and conclusive (even short notice)
>>       papers and/or talks on that topic have a reserved slot in the
>>       Kernel developers track at the Realtime Linux Workshop in Prague
>>       in October this year.
>>
>> Enough marketing, here comes the real stuff.
>>
>>    Patch against 3.0-rc7 can be found here:
>>
>> 
>> http://www.kernel.org/pub/linux/kernel/projects/rt/patch-3.0-rc7-rt0.patch.bz2
>>
>>    The split quilt queue is available at:
>>
>> 
>> http://www.kernel.org/pub/linux/kernel/projects/rt/patches-3.0-rc7-rt0.tar.gz
>>
>> There is no git tree for now.
>>
>> I'm not yet convinced that moving RT to git was a good idea as quilt
>> allows me to move stuff around in a way more flexible manner. So for
>> now no git version until someone comes up with a brilliant idea which
>> allows me to keep my workflow sane (do not even try to suggest stgit&
>> co!).
>>
>> That said, have fun and make sure that you have the fire extinguisher
>> ready when you start using this!
>>
>> Thanks,
>>
>> tglx
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" 
>> in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" 
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html