realtime-preempt for MIPS - compile problem with rwsem

realtime-preempt for MIPS - compile problem with rwsem

[Ryan McAvoy]

Hello,

I have been trying to get the realtime-preempt patches working on a
MIPs processor with the 2.6.15 (and subsequently the 2.6.16) kernel.

One problem I am having is as follows:

- The mips configuration specifically disables
CONFIG_RWSEM_GENERIC_SPINLOCK when CONFIG_PREEMPT_RT is on
- This causes the include/asm-mips/rwsem.h to be included by
include/linux/rwsem.h.
- include/asm-mips/rwsem.h calls rwsem_down_read_failed which is
implemented in lib/rwsem.c
- rwsem.c is only compiled if CONFIG_RWSEM_XCHGADD_ALGORITHM is on.
This option is also disabled if CONFIG_PREEMPT_RT is on.

To summarise, if CONFIG_PREEMPT_RT is on, then the mips specific
rwsem.h is included, but at the same time, it prevents inclusion of
lib/rwsem.c which is needed by the mips rwsem.h.

Does anyone have a solution to this.
Thanks,
Ryan McAvoy

Re: realtime-preempt for MIPS - compile problem with rwsem

[Steven Rostedt]

Hi Ryan,

First, whenever sending mail about the -rt patch, always CC Ingo (and
perhaps Thomas Gliexner and myself).

On Tue, 20 Jun 2006, Ryan McAvoy wrote:

> I have been trying to get the realtime-preempt patches working on a
> MIPs processor with the 2.6.15 (and subsequently the 2.6.16) kernel.

2.6.15 had lots of problems with -rt.  Mainly the patch went through some
major rework, and Ingo was busy getting mutexes into mainline.  So the
2.6.15-rtX was sort of neglected.  It would be best to use 2.6.16-rtX and
maybe even 2.6.17-rtX

Which also comes the question: Which -rt patch are you actually trying?

>
> One problem I am having is as follows:
>
> - The mips configuration specifically disables
> CONFIG_RWSEM_GENERIC_SPINLOCK when CONFIG_PREEMPT_RT is on

That's a bug.  It should enable it not disable it.

> - This causes the include/asm-mips/rwsem.h to be included by
> include/linux/rwsem.h.
> - include/asm-mips/rwsem.h calls rwsem_down_read_failed which is
> implemented in lib/rwsem.c
> - rwsem.c is only compiled if CONFIG_RWSEM_XCHGADD_ALGORITHM is on.
> This option is also disabled if CONFIG_PREEMPT_RT is on.
>
> To summarise, if CONFIG_PREEMPT_RT is on, then the mips specific
> rwsem.h is included, but at the same time, it prevents inclusion of
> lib/rwsem.c which is needed by the mips rwsem.h.
>
> Does anyone have a solution to this.

Yep, try the following patch: (completey untested since I don't have a
mips machine).

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux-2.6.16-rt25/arch/mips/Kconfig
===================================================================
--- linux-2.6.16-rt25.orig/arch/mips/Kconfig	2006-06-21 04:02:54.000000000 -0400
+++ linux-2.6.16-rt25/arch/mips/Kconfig	2006-06-21 04:03:29.000000000 -0400
@@ -798,7 +798,7 @@ source "kernel/Kconfig.preempt"

 config RWSEM_GENERIC_SPINLOCK
 	bool
-	depends on !PREEMPT_RT
+	depends on PREEMPT_RT
 	default y

 config RWSEM_XCHGADD_ALGORITHM

Re: realtime-preempt for MIPS - compile problem with rwsem

[Ryan McAvoy]

Hi Steve,

Thanks for your reply.

Steven Rostedt wrote:

> First, whenever sending mail about the -rt patch, always CC Ingo (and
> perhaps Thomas Gliexner and myself).

Thanks, I will do that.

> 2.6.15 had lots of problems with -rt.  Mainly the patch went through some
> major rework, and Ingo was busy getting mutexes into mainline.  So the
> 2.6.15-rtX was sort of neglected.  It would be best to use 2.6.16-rtX and
> maybe even 2.6.17-rtX
>
> Which also comes the question: Which -rt patch are you actually trying?

patch-2.6.15-rt21 .  I have also tried using the 2.6.16 kernel and
patch-2.6.16-rt29 and have the same problems with it.  (I may be
constrained in having to use a 2.6.15 kernel ... but would be happy to
get 2.6.16 working /stable as a starting point.)

> Yep, try the following patch: (completey untested since I don't have a
> mips machine).
>
>  config RWSEM_GENERIC_SPINLOCK
>         bool
> -       depends on !PREEMPT_RT
> +       depends on PREEMPT_RT
>         default y

I did just that when I first started with these patches and did
succeed in getting it compiling and booting.  The resulting kernel,
however, is very unstable and hangs frequently with no output.  (It
will hang within hours if left idle.  I can hang it more quickly by
attempting to use it).  I have deadlock detection turned on and have
confirmed that it does produce output at least for some deadlocks:
http://groups.google.com/group/linux.kernel/browse_frm/thread/1559667001b7da2d/2558b539a5adc660?lnk=st&q=realtime+preempt+mips&rnum=2&hl=en#2558b539a5adc660
In the more common hangs though, I get no output.

I decided to review the changes I made in getting it to compile and
was hoping that this one may be the cause of the instability.  I
thought that perhaps this change was incorrect because
include/asm-mips/rwsem.h is introduced by the rt-preempt patch and
would only be used if RWSEM_GENERIC_SPINLOCK was off.  [As well, it
seemed like something fundamental enough to account for the general
instability I am seeing.]

Ryan

Re: realtime-preempt for MIPS - compile problem with rwsem

[Steven Rostedt]

On Wed, 21 Jun 2006, Ryan McAvoy wrote:

> > First, whenever sending mail about the -rt patch, always CC Ingo (and
> > perhaps Thomas Gliexner and myself).
>
> Thanks, I will do that.

Ingo is the important one.  Thomas and I are just the janitors
Although Thomas is chief janitor :)

>
> > 2.6.15 had lots of problems with -rt.  Mainly the patch went through some
> > major rework, and Ingo was busy getting mutexes into mainline.  So the
> > 2.6.15-rtX was sort of neglected.  It would be best to use 2.6.16-rtX and
> > maybe even 2.6.17-rtX
> >
> > Which also comes the question: Which -rt patch are you actually trying?
>
> patch-2.6.15-rt21 .  I have also tried using the 2.6.16 kernel and

Hmm, 15-rt21 may have stabilized a bit. But I do still think the 16 series
is much more stable.

> patch-2.6.16-rt29 and have the same problems with it.  (I may be
> constrained in having to use a 2.6.15 kernel ... but would be happy to
> get 2.6.16 working /stable as a starting point.)
>
> > Yep, try the following patch: (completey untested since I don't have a
> > mips machine).
> >
> >  config RWSEM_GENERIC_SPINLOCK
> >         bool
> > -       depends on !PREEMPT_RT
> > +       depends on PREEMPT_RT
> >         default y
>
> I did just that when I first started with these patches and did
> succeed in getting it compiling and booting.  The resulting kernel,
> however, is very unstable and hangs frequently with no output.  (It
> will hang within hours if left idle.  I can hang it more quickly by
> attempting to use it).  I have deadlock detection turned on and have
> confirmed that it does produce output at least for some deadlocks:
> http://groups.google.com/group/linux.kernel/browse_frm/thread/1559667001b7da2d/2558b539a5adc660?lnk=st&q=realtime+preempt+mips&rnum=2&hl=en#2558b539a5adc660
> In the more common hangs though, I get no output.

That output looks like it had a deadlock on the serial output of sysrq
key.  But that back trace looks screwy.

>
> I decided to review the changes I made in getting it to compile and
> was hoping that this one may be the cause of the instability.  I
> thought that perhaps this change was incorrect because
> include/asm-mips/rwsem.h is introduced by the rt-preempt patch and

Ha, you're right!  (added John Cooper to this so he can clean up this mess
;)

> would only be used if RWSEM_GENERIC_SPINLOCK was off.  [As well, it
> seemed like something fundamental enough to account for the general
> instability I am seeing.]
>

Perhaps you can post all the changes you made as a patch to see if
something else is wrong.  It might also be best to see if you can get the
latest working (2.6.17-rtX) and work your way backwards to the kernel
version you really need.

-- Steve

Re: realtime-preempt for MIPS - compile problem with rwsem

[Steven Rostedt]

On Wed, 21 Jun 2006, Steven Rostedt wrote:

>
> >
> > I decided to review the changes I made in getting it to compile and
> > was hoping that this one may be the cause of the instability.  I
> > thought that perhaps this change was incorrect because
> > include/asm-mips/rwsem.h is introduced by the rt-preempt patch and
>
> Ha, you're right!  (added John Cooper to this so he can clean up this mess
> ;)
>

Forget John, I guess he's no longer at TimeSys.  Just got a bounce from
his email address.

-- Steve

Re: realtime-preempt for MIPS - compile problem with rwsem

[Ryan McAvoy]

On 6/21/06, Steven Rostedt wrote:

http://groups.google.com/group/linux.kernel/browse_frm/thread/1559667001b7da2d/2558b539a5adc660?lnk=st&q=realtime+preempt+mips&rnum=2&hl=en#2558b539a5adc660
> > In the more common hangs though, I get no output.
>
> That output looks like it had a deadlock on the serial output of sysrq
> key.  But that back trace looks screwy.
>

That problem actually went away with 2.6.16.  All the others remained
unfortunately.  (I was not that concerned about that one ... I can
avoid it just by not sending a sysrq ;-).  I posted it though since I
did actually have output with that hang).

>
> Perhaps you can post all the changes you made as a patch to see if
> something else is wrong.  It might also be best to see if you can get the
> latest working (2.6.17-rtX) and work your way backwards to the kernel
> version you really need.
>

I will post the other changes soon.  (I am not at the office where I
am working on this yet this morning).  I will also try 2.6.17.
Thanks.

Ryan

Re: realtime-preempt for MIPS - compile problem with rwsem

[john cooper]

Steven Rostedt wrote:
> On Wed, 21 Jun 2006, Ryan McAvoy wrote:

>> I did just that when I first started with these patches and did
>> succeed in getting it compiling and booting.  The resulting kernel,
>> however, is very unstable and hangs frequently with no output.  (It
>> will hang within hours if left idle.  I can hang it more quickly by
>> attempting to use it).

I also had stability issues with 2.6.14+RT+MIPS.
The only point I found it to be stable was under 2.6.13
where it successfully passed a standard battery of QA
tests.  To my knowledge no one has updated MIPS RT
since that time so I'd assume it is either in the
same state as then (best case) or worse (likely) due to
languishing.

Note the MIPS patch under discussion also required a
patch from linux-mips to produce a functional kernel
on the malta 4Kc which was the only MIPS target known
to be supported by that combination.

>> I have deadlock detection turned on and have
>> confirmed that it does produce output at least for some deadlocks:
>> http://groups.google.com/group/linux.kernel/browse_frm/thread/1559667001b7da2d/2558b539a5adc660?lnk=st&q=realtime+preempt+mips&rnum=2&hl=en#2558b539a5adc660
>> In the more common hangs though, I get no output.
> 
> That output looks like it had a deadlock on the serial output of sysrq
> key.  But that back trace looks screwy.

Well, not for MIPS.  There isn't any easy/painless method of
backtracing the stack so the MIPS version of dump_stack()
doesn't even try.  Instead it scans the entirety of the
stack and prints out anything and everything which smells
like a valid kernel text address, hence the screwy output.

>> I decided to review the changes I made in getting it to compile and
>> was hoping that this one may be the cause of the instability.  I
>> thought that perhaps this change was incorrect because
>> include/asm-mips/rwsem.h is introduced by the rt-preempt patch and
> 
> Ha, you're right!  (added John Cooper to this so he can clean up this mess
> ;)

Deselection of RWSEM_GENERIC_SPINLOCK in arch/mips/Kconfig is
a bug.   Unfortunately I don't have an available mips target
ATM and few if any spare cycles, but I'd sure like to get MIPS
through this nagging impasse.

That said, keep in mind due to limitations in the MIPS ABI
which carry over to gcc, latency instrumentation as it
currently exists (or may ever exist) is effectively
unsupportable.  IMHO this is a serious stumbling block for
realistic support of the RT work on this architecture.

-john

-- 
john.cooper@member.fsf.org