Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Wed, Jan 30, 2008 at 11:45:01AM +0100, Wolfgang Grandegger wrote:
> Paul E. McKenney wrote:
> > On Wed, Jan 30, 2008 at 09:18:49AM +0100, Wolfgang Grandegger wrote:
> >> Paul E. McKenney wrote:
> >>> On Tue, Jan 29, 2008 at 02:38:04PM +0100, Wolfgang Grandegger wrote:
> >>>> Luotao Fu wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Wolfgang Grandegger wrote:
> >>>>> ..........
> >>>>>> Do you still get high latencies with:
> >>>>>>
> >>>>>>   CONFIG_PREEMPT_RCU_BOOST=y
> >>>>>>   CONFIG_RCU_TRACE=y
> >>>>>>   CONFIG_NO_HZ is not set
> >>>>>>
> >>>>>> With this setting I have not yet realized latencies > 150us. Could you
> >>>>>> please give it a try? If I change one of the parameters above, latency
> >>>>>> increases in short time.
> >>>>> I played through some combination of the RCU options and can back your
> >>>>> observation this time: With the rcu Tracer or the priority boost turned
> >>>>> off I also could measure reliably extraordinory high latencies. If they
> >>>>> are both turned on, no high latencies could be measured. Turning on the
> >>>>> dynamic ticker however doesn't seem to cause high latencies during my
> >>>>> test runs. Seemed like an rcu issue here.
> >>>> I'm just making a long test run on my TQM5200 module with my good
> >>>> settings. After more than 4.5 hours under load, cyclictest shows a
> >>>> maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.
> >>>>
> >>>>> Further such results only appear if the target board is booted with
> >>>>> nfsrootfs. (As I already have mentioned several times before), which
> >>>>> leads my suspection to rcu usage in nfs implementation. In this case
> >>>>> this problem might even be platformindependent. I'd have to do some
> >>>>> tests on one of our arm boards later to test this. Since there're no
> >>>>> reports like this for other architecture as powerpc till now, I doubt
> >>>>> quite if this is verifiable.
> >>>> It's also my suspicion that the high latencies are related to the RCU
> >>>> usage in the network layer, where it's heavily used. What is really
> >>>> wired is that switching off CONFIG_RCU_TRACE has a negative impact on
> >>>> the latency. As I see it, it just adds some trace points, but I might
> >>>> have missed something.
> >>> I would expect that CONFIG_RCU_TRACE=n (as in "no" rather than "module")
> >>> would have low latencies rather than high latencies.  So I am quite
> >>> surprised by your result.  I will dig into this more.
> >> Thanks a lot. To be clear. I need "CONFIG_PREEMPT_RCU_BOOST=y" *and*
> >> "CONFIG_RCU_TRACE=y" to achieve reasonable latencies below 180us. With
> >> CONFIG_PREEMPT_RCU_BOOST or CONFIG_RCU_TRACE not set or
> >> CONFIG_RCU_TRACE=m is rmeasure latencies up to 600us within a minute or so.
> > 
> > OK, thank you for the confirmation.
> > 
> > The large latencies were from cyclictest, correct?  Did other tests
> > also show these latencies?  In either case, could you please send the
> > exact command line you used to run the test?
> 
> Yes, I used "$ cyclictest -n -t1 -p80 -i1000" to measure the latency. So
> far, I have not done other tests. Any recommendation?
> As no-rt load I used "while ls; do ls /bin; done" in one telnet window
> and "while ./hackbench 10; do ./calibrator 400 32M cali; sleep 30; done"
> in another. But already "while ls; do ls /bin; done" is enough to
> trigger the high latencies quickly. Note also, that I work on a root
> files-ystem mounted via NFS resulting in a lot of network traffic and
> utilization.

I have to ask...

Did you see large latencies when -not- running on NFS?

							Thanx, Paul