* Re: 2.6.24-rc8-rt1
2008-01-17 10:13 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
@ 2008-01-17 12:46 ` Luotao Fu
2008-01-17 16:17 ` 2.6.24-rc8-rt1 Daniel Walker
` (2 subsequent siblings)
3 siblings, 0 replies; 49+ messages in thread
From: Luotao Fu @ 2008-01-17 12:46 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]
Hi Wolfgang,
On Thu, Jan 17, 2008 at 11:13:26AM +0100, Wolfgang Grandegger wrote:
> Steven Rostedt wrote:
> It builds and runs fine on my Icecube-MPC5200 board, now also with the
> latency tracer enabled. That's great. Still, "cyclictest -n -p80 -i1000"
> reports latencies up to 400 us and therefore I tried to trigger and save
> a high latency trace using:
>
> # ./cyclictest -n -p80 -i1000 -b400
> 1.21 0.33 0.11 4/42 1048
>
> T: 0 ( 914) P:80 I:1000 C: 38726 Min: 61 Act: 107 Avg: 106
> Max: 377
> [ 91.042169] ( cyclictest-914 |#0): new 39733427 us user-latency.
> bash-3.00# cat /proc/latency_trace > trace.log
>
I tested 2.6.24-rc8-rt1 on our phytec phyCORE-MPC5200B-tiny. I runned cyclictest
with exactly the same parameter as you used and made some non-rt Workload with
"hackbench 5" in a while 1 loop and a "ping -f" from a host machine. I run the
test for about 3 hours and could not push the latency to higher than 220 us.
(The latency tracer is also turned on in kernel). Could you please send me your
kernelconfig privately?
cheers
Luotao Fu
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1
2008-01-17 10:13 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
2008-01-17 12:46 ` 2.6.24-rc8-rt1 Luotao Fu
@ 2008-01-17 16:17 ` Daniel Walker
2008-01-17 18:17 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
2008-01-17 21:11 ` 2.6.24-rc8-rt1 Robert Schwebel
2008-01-23 14:53 ` 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc Luotao Fu
3 siblings, 1 reply; 49+ messages in thread
From: Daniel Walker @ 2008-01-17 16:17 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
On Thu, 2008-01-17 at 11:13 +0100, Wolfgang Grandegger wrote:
> Steven Rostedt wrote:
> > We are pleased to announce the 2.6.24-rc8-rt1 tree, which can be
> > downloaded from the location:
> >
> > http://rt.et.redhat.com/download/
>
> It builds and runs fine on my Icecube-MPC5200 board, now also with the
> latency tracer enabled. That's great. Still, "cyclictest -n -p80 -i1000"
> reports latencies up to 400 us and therefore I tried to trigger and save
> a high latency trace using:
Do you happen to have a boot log (dmesg) for this system?
Daniel
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1
2008-01-17 16:17 ` 2.6.24-rc8-rt1 Daniel Walker
@ 2008-01-17 18:17 ` Wolfgang Grandegger
2008-01-17 18:30 ` 2.6.24-rc8-rt1 Daniel Walker
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-17 18:17 UTC (permalink / raw)
To: Daniel Walker; +Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
[-- Attachment #1: Type: text/plain, Size: 621 bytes --]
Daniel Walker wrote:
> On Thu, 2008-01-17 at 11:13 +0100, Wolfgang Grandegger wrote:
>> Steven Rostedt wrote:
>>> We are pleased to announce the 2.6.24-rc8-rt1 tree, which can be
>>> downloaded from the location:
>>>
>>> http://rt.et.redhat.com/download/
>> It builds and runs fine on my Icecube-MPC5200 board, now also with the
>> latency tracer enabled. That's great. Still, "cyclictest -n -p80 -i1000"
>> reports latencies up to 400 us and therefore I tried to trigger and save
>> a high latency trace using:
>
> Do you happen to have a boot log (dmesg) for this system?
Yes, of course, see attachment.
Wolfgang.
[-- Attachment #2: boot.log --]
[-- Type: text/x-log, Size: 6493 bytes --]
=> boot
Using FEC ETHERNET device
TFTP from server 10.0.30.2; our IP address is 10.0.30.199
Filename 'icecube/uImage-rt-lat'.
Load address: 0x400000
Loading: #################################################################
####################################
done
Bytes transferred = 1474093 (167e2d hex)
## Booting image at 00400000 ...
Image Name: Linux-2.6.24-rc8-rt1
Created: 2008-01-17 9:59:11 UTC
Image Type: PowerPC Linux Kernel Image (gzip compressed)
Data Size: 1474029 Bytes = 1.4 MB
Load Address: 00000000
Entry Point: 00000000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
Booting using the fdt at 0xff850000
Loading Device Tree to 007fc000, end 007fefff ... OK
[ 0.000000] Using lite5200 machine description
[ 0.000000] Linux version 2.6.24-rc8-rt1 (wolf@lancy.lan.de) (gcc version 4.0.0 (DENX ELDK 4.1 4.0.0)) #6 PREEMPT RT Thu Jan 17 10:59:02 CET 2008
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0 -> 16384
[ 0.000000] Normal 16384 -> 16384
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[1] active PFN ranges
[ 0.000000] 0: 0 -> 16384
[ 0.000000] Real-Time Preemption Support (C) 2004-2007 Ingo Molnar
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 16256
[ 0.000000] Kernel command line: root=/dev/nfs rw nfsroot=10.0.30.2:/data//ELDK/v4.1/ppc_6xx ip=10.0.30.199:10.0.30.2::255.255.255.0:lite5200:eth0:off console=ttyPSC0,115200
[ 0.000000] WARNING: experimental RCU implementation.
[ 0.000000] MPC52xx PIC is up and running!
[ 0.000000] PID hash table entries: 256 (order: 8, 1024 bytes)
[ 0.000019] clocksource: timebase mult[79364d9] shift[22] registered
[ 0.000345] console [ttyPSC0] enabled
[ 0.098764] num_possible_cpus(): 1
[ 0.147047] CPU#0: allocated 2097152 bytes trace buffer.
[ 0.197391] CPU#0: allocated 2097152 bytes max-trace buffer.
[ 0.248029] allocated 2097152 bytes out-trace buffer.
[ 0.253223] tracer: a total of 6291456 bytes allocated.
[ 0.258951] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
[ 0.266485] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.283562] Memory: 55256k/65536k available (3024k kernel code, 10220k reserved, 172k data, 236k bss, 176k init)
[ 0.387337] Mount-cache hash table entries: 512
[ 0.405903] net_namespace: 76 bytes
[ 0.414103] NET: Registered protocol family 16
[ 0.458695] PCI: Probing PCI hardware
[ 0.468126] DMA: MPC52xx BestComm driver
[ 0.473000] DMA: MPC52xx BestComm engine @f0001200 ok !
[ 0.499454] SCSI subsystem initialized
[ 0.567145] NET: Registered protocol family 2
[ 0.715232] IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.725291] TCP established hash table entries: 2048 (order: 2, 16384 bytes)
[ 0.733248] TCP bind hash table entries: 2048 (order: 3, 57344 bytes)
[ 0.741132] TCP: Hash tables configured (established 2048 bind 2048)
[ 0.747981] TCP reno registered
[ 0.805896] krcupreemptd setsched 0
[ 0.809657] prio = 98
[ 0.816864] io scheduler noop registered
[ 0.821149] io scheduler anticipatory registered (default)
[ 0.827137] io scheduler deadline registered
[ 0.831849] io scheduler cfq registered
[ 1.532740] Serial: MPC52xx PSC UART driver
[ 1.539120] f0002000.serial: ttyPSC0 at MMIO 0xf0002000 (irq = 129) is a MPC52xx PSC
[ 1.568537] RAMDISK driver initialized: 16 RAM disks of 32768K size 1024 blocksize
[ 1.586580] loop: module loaded
[ 1.631604] mpc52xx MII bus: probed
[ 1.640809] ata: MPC52xx IDE/ATA libata driver
[ 1.649069] scsi0 : mpc52xx_ata
[ 1.653845] ata1: PATA max PIO4 ata_regs 0xf0003a00 irq 135
[ 1.843308] TCP cubic registered
[ 1.846923] NET: Registered protocol family 1
[ 1.851717] NET: Registered protocol family 17
[ 1.858127] RPC: Registered udp transport module.
[ 1.863271] RPC: Registered tcp transport module.
[ 2.381436] net eth0: attached phy 0 to driver Generic PHY
[ 3.392504] IP-Config: Complete:
[ 3.395845] device=eth0, addr=10.0.30.199, mask=255.255.255.0, gw=255.255.255.255,
[ 3.404378] host=lite5200, domain=, nis-domain=(none),
[ 3.410215] bootserver=10.0.30.2, rootserver=10.0.30.2, rootpath=
[ 3.419137] Looking up port of RPC 100003/2 on 10.0.30.2
[ 5.378639] PHY: f0003000:00 - Link is Up - 100/Full
[ 5.427768] Looking up port of RPC 100005/1 on 10.0.30.2
[ 5.486821] VFS: Mounted root (nfs filesystem).
[ 5.492025] *****************************************************************************
[ 5.500722] * *
[ 5.509440] * REMINDER, the following debugging options are turned on in your .config: *
[ 5.518136] * *
[ 5.527066] * CONFIG_CRITICAL_PREEMPT_TIMING *
[ 5.535708] * CONFIG_FUNCTION_TRACE *
[ 5.544334] * *
[ 5.552957] * they may increase runtime overhead and latencies. *
[ 5.561574] * *
[ 5.570202] *****************************************************************************
[ 5.578909] Freeing unused kernel memory: 176k init
INIT: version 2.85 booting
Welcome to DENX Embedded Linux Environment
Press 'I' to enter interactive startup.
Building the cache [ OK ]
storage network audio done[ OK ]
Cannot access the Hardware Clock via any known method.
Use the --debug option to see the details of our search for an access method.
Setting clock : Thu Jan 1 00:00:11 UTC 1970 [ OK ]
Setting hostname lite5200: [ OK ]
Mounting local filesystems: [ OK ]
Enabling swap space: [ OK ]
INIT: Entering runlevel: 3
Entering non-interactive startup
Bringing up loopback interface: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Starting portmap: portmap: fork: No such device[ OK ]
Mounting NFS filesystems: [ OK ]
Mounting other filesystems: [ OK ]
Starting xinetd: [ OK ]
DENX ELDK version 4.1 build 2007-01-19
Linux 2.6.24-rc8-rt1 on a ppc
lite5200 login:
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1
2008-01-17 18:17 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
@ 2008-01-17 18:30 ` Daniel Walker
2008-01-17 18:44 ` 2.6.24-rc8-rt1 Steven Rostedt
` (2 more replies)
0 siblings, 3 replies; 49+ messages in thread
From: Daniel Walker @ 2008-01-17 18:30 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
On Thu, 2008-01-17 at 19:17 +0100, Wolfgang Grandegger wrote:
> [ 0.733248] TCP bind hash table entries: 2048 (order: 3, 57344
> bytes)
> [ 0.741132] TCP: Hash tables configured (established 2048 bind
> 2048)
> [ 0.747981] TCP reno registered
> [ 0.805896] krcupreemptd setsched 0
> [ 0.809657] prio = 98
That's interesting .. You chould try running cyclictest at priority 99
to eliminate other threads that might get involved (using -p99 instead
of -p80 , I think) ..
Daniel
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1
2008-01-17 18:30 ` 2.6.24-rc8-rt1 Daniel Walker
@ 2008-01-17 18:44 ` Steven Rostedt
2008-01-17 18:45 ` 2.6.24-rc8-rt1 Steven Rostedt
2008-01-17 18:46 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
2 siblings, 0 replies; 49+ messages in thread
From: Steven Rostedt @ 2008-01-17 18:44 UTC (permalink / raw)
To: Daniel Walker; +Cc: Wolfgang Grandegger, LKML, RT, Ingo Molnar, Thomas Gleixner
On Thu, 17 Jan 2008, Daniel Walker wrote:
>
> On Thu, 2008-01-17 at 19:17 +0100, Wolfgang Grandegger wrote:
> > [ 0.733248] TCP bind hash table entries: 2048 (order: 3, 57344
> > bytes)
> > [ 0.741132] TCP: Hash tables configured (established 2048 bind
> > 2048)
> > [ 0.747981] TCP reno registered
> > [ 0.805896] krcupreemptd setsched 0
> > [ 0.809657] prio = 98
>
> That's interesting .. You chould try running cyclictest at priority 99
> to eliminate other threads that might get involved (using -p99 instead
> of -p80 , I think) ..
No that prio is the internal prio where smaller number is higher priority.
The krcupreemptd runs at RT prio 1, which is 98 internally.
-- Steve
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1
2008-01-17 18:30 ` 2.6.24-rc8-rt1 Daniel Walker
2008-01-17 18:44 ` 2.6.24-rc8-rt1 Steven Rostedt
@ 2008-01-17 18:45 ` Steven Rostedt
2008-01-17 20:01 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
2008-01-17 18:46 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
2 siblings, 1 reply; 49+ messages in thread
From: Steven Rostedt @ 2008-01-17 18:45 UTC (permalink / raw)
To: Daniel Walker; +Cc: Wolfgang Grandegger, LKML, RT, Ingo Molnar, Thomas Gleixner
On Thu, 17 Jan 2008, Daniel Walker wrote:
>
> On Thu, 2008-01-17 at 19:17 +0100, Wolfgang Grandegger wrote:
> > [ 0.733248] TCP bind hash table entries: 2048 (order: 3, 57344
> > bytes)
> > [ 0.741132] TCP: Hash tables configured (established 2048 bind
> > 2048)
> > [ 0.747981] TCP reno registered
> > [ 0.805896] krcupreemptd setsched 0
> > [ 0.809657] prio = 98
>
> That's interesting .. You chould try running cyclictest at priority 99
> to eliminate other threads that might get involved (using -p99 instead
> of -p80 , I think) ..
>
But, anotherthing to try is disabling CONFIG_PREEMPT_RCU_BOOST, and see if
that fixes anything.
-- Steve
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1
2008-01-17 18:45 ` 2.6.24-rc8-rt1 Steven Rostedt
@ 2008-01-17 20:01 ` Wolfgang Grandegger
0 siblings, 0 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-17 20:01 UTC (permalink / raw)
To: Steven Rostedt; +Cc: Daniel Walker, LKML, RT, Ingo Molnar, Thomas Gleixner
Steven Rostedt wrote:
> On Thu, 17 Jan 2008, Daniel Walker wrote:
>
>> On Thu, 2008-01-17 at 19:17 +0100, Wolfgang Grandegger wrote:
>>> [ 0.733248] TCP bind hash table entries: 2048 (order: 3, 57344
>>> bytes)
>>> [ 0.741132] TCP: Hash tables configured (established 2048 bind
>>> 2048)
>>> [ 0.747981] TCP reno registered
>>> [ 0.805896] krcupreemptd setsched 0
>>> [ 0.809657] prio = 98
>> That's interesting .. You chould try running cyclictest at priority 99
>> to eliminate other threads that might get involved (using -p99 instead
>> of -p80 , I think) ..
>>
>
> But, anotherthing to try is disabling CONFIG_PREEMPT_RCU_BOOST, and see if
> that fixes anything.
It did not really help:
/cyclictest -n -p99 -i1000
3.89 3.46 1.77 1/43 3830
T: 0 ( 916) P:99 I:1000 C: 520752 Min: 52 Act: 112 Avg: 105
Max: 960
Any other idea?
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1
2008-01-17 18:30 ` 2.6.24-rc8-rt1 Daniel Walker
2008-01-17 18:44 ` 2.6.24-rc8-rt1 Steven Rostedt
2008-01-17 18:45 ` 2.6.24-rc8-rt1 Steven Rostedt
@ 2008-01-17 18:46 ` Wolfgang Grandegger
2 siblings, 0 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-17 18:46 UTC (permalink / raw)
To: Daniel Walker; +Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Daniel Walker wrote:
> On Thu, 2008-01-17 at 19:17 +0100, Wolfgang Grandegger wrote:
>> [ 0.733248] TCP bind hash table entries: 2048 (order: 3, 57344
>> bytes)
>> [ 0.741132] TCP: Hash tables configured (established 2048 bind
>> 2048)
>> [ 0.747981] TCP reno registered
>> [ 0.805896] krcupreemptd setsched 0
>> [ 0.809657] prio = 98
>
> That's interesting .. You chould try running cyclictest at priority 99
> to eliminate other threads that might get involved (using -p99 instead
> of -p80 , I think) ..
I already tried that but I did not realize any difference. If I run
"while ls; do ls /bin; done" on the console I even get:
#./cyclictest -n -p99 -i1000
2.82 1.70 0.90 2/45 3168
T: 0 ( 2426) P:99 I:1000 C: 135995 Min: 57 Act: 110 Avg: 106
Max: 911
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1
2008-01-17 10:13 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
2008-01-17 12:46 ` 2.6.24-rc8-rt1 Luotao Fu
2008-01-17 16:17 ` 2.6.24-rc8-rt1 Daniel Walker
@ 2008-01-17 21:11 ` Robert Schwebel
2008-01-17 21:36 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
2008-01-23 14:53 ` 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc Luotao Fu
3 siblings, 1 reply; 49+ messages in thread
From: Robert Schwebel @ 2008-01-17 21:11 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
On Thu, Jan 17, 2008 at 11:13:26AM +0100, Wolfgang Grandegger wrote:
> # ./cyclictest -n -p80 -i1000 -b400
>
> [...]
>
> Well, I'm not sure if this is the correct way to do it. Is there some
> doc on how to use the latency tracer and interpret the results?
Could you put in this kind of log information inline next time? It makes
it easier to review.
Let's see ...
> preemption latency trace v1.1.5 on 2.6.24-rc8-rt1
> --------------------------------------------------------------------
> latency: 39733427 us, #65536/1202801, CPU#0 | (M:rt VP:0, KP:0, SP:1 HP:1)
> -----------------
> | task: cyclictest-914 (uid:0 nice:0 policy:1 rt_prio:80)
> -----------------
Ok, so cyclictest-914 triggered the tracer. No big surprise at -t 1 :-)
> cyclicte-914 0D..1 40us+: enqueue_hrtimer+0x50/0x1a0 ( 89 5aa9862 c3a6dea8)
cyclictest-914 goes to sleep for next period. We expect it to wake up
again at 40 us + 1000 us = 1040 us.
> ls-1032 0DNh2 999us+: __trace_start_sched_wakeup+0x154/0x18c <cyclicte-914> (19 -1)
> ls-1032 0DNh2 1001us+: try_to_wake_up+0x194/0x1b4 <cyclicte-914> (180 0)
> ls-1032 0DNh. 1008us+: clockevents_program_event+0x84/0x1c0 ( 89 5ad6bbf 18e8e)
> cyclicte-914 0D..2 1033us+: __schedule+0x21c/0x444 <ls-1032> (0 180)
> cyclicte-914 0D..1 1071us+: enqueue_hrtimer+0x50/0x1a0 ( 89 5b9daa2 c3a6dea8)
Looks good, it happened. Next wakeup at about 1071 us + 1000 us = 2071
us.
> cyclicte-914 0D..2 2031us+: __schedule+0x21c/0x444 <IRQ-129-682> (150 180)
> cyclicte-914 0D..1 2072us+: enqueue_hrtimer+0x50/0x1a0 ( 89 5c91ce2 c3a6dea8)
Bingo. Let's get a cup of coffee in the meantime ...
> cyclicte-914 0D..1 1905066us+: enqueue_hrtimer+0x50/0x1a0 ( 91 66210 c3a6dea8)
Expected arival at 1905066 us + 1000 us = 1906066 us.
> ls-1048 0DNh3 1906017us+: __trace_start_sched_wakeup+0x154/0x18c <cyclicte-914> (19 -1)
> ls-1048 0DNh3 1906021us : __trace_start_sched_wakeup+0x154/0x18c <cyclicte-914> (19 -1)
> ls-1048 0DNh3 1906023us+: try_to_wake_up+0x194/0x1b4 <cyclicte-914> (180 0)
It's trying hard to wake up ...
> ls-1048 0DNh1 1906028us!: clockevents_program_event+0x84/0x1c0 ( 91 391676 367eff)
> ls-1048 0DNh2 1906274us+: activate_task+0x60/0xa4 <IRQ-193-681> (150 6)
Between these it should have happened.
> ls-1048 0DNh2 1906285us+: __trace_start_sched_wakeup+0x154/0x18c <IRQ-193-681> (49 -1)
> ls-1048 0DNh2 1906293us+: __trace_start_sched_wakeup+0x154/0x18c <IRQ-193-681> (49 -1)
> ls-1048 0DNh2 1906295us+: try_to_wake_up+0x194/0x1b4 <IRQ-193-681> (150 0)
> cyclicte-914 0D..2 1906332us : __schedule+0x21c/0x444 <ls-1048> (0 180)
What is this IRQ-193-681? Can you post your /proc/interrupts and the
output of
ps axHo user,pid,%cpu,%mem,nice,rtprio,policy,tty,stat,start_time,bsdtime,args
Robert
--
Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
Pengutronix - Linux Solutions for Science and Industry
Handelsregister: Amtsgericht Hildesheim, HRA 2686
Hannoversche Str. 2, 31134 Hildesheim, Germany
Phone: +49-5121-206917-0 | Fax: +49-5121-206917-9
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1
2008-01-17 21:11 ` 2.6.24-rc8-rt1 Robert Schwebel
@ 2008-01-17 21:36 ` Wolfgang Grandegger
0 siblings, 0 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-17 21:36 UTC (permalink / raw)
To: Robert Schwebel; +Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Robert Schwebel wrote:
> On Thu, Jan 17, 2008 at 11:13:26AM +0100, Wolfgang Grandegger wrote:
>> # ./cyclictest -n -p80 -i1000 -b400
>>
>> [...]
>>
>> Well, I'm not sure if this is the correct way to do it. Is there some
>> doc on how to use the latency tracer and interpret the results?
>
> Could you put in this kind of log information inline next time? It makes
> it easier to review.
OK.
>
> Let's see ...
>
>> preemption latency trace v1.1.5 on 2.6.24-rc8-rt1
>> --------------------------------------------------------------------
>> latency: 39733427 us, #65536/1202801, CPU#0 | (M:rt VP:0, KP:0, SP:1 HP:1)
>> -----------------
>> | task: cyclictest-914 (uid:0 nice:0 policy:1 rt_prio:80)
>> -----------------
>
> Ok, so cyclictest-914 triggered the tracer. No big surprise at -t 1 :-)
>
>> cyclicte-914 0D..1 40us+: enqueue_hrtimer+0x50/0x1a0 ( 89 5aa9862 c3a6dea8)
>
> cyclictest-914 goes to sleep for next period. We expect it to wake up
> again at 40 us + 1000 us = 1040 us.
>
>> ls-1032 0DNh2 999us+: __trace_start_sched_wakeup+0x154/0x18c <cyclicte-914> (19 -1)
>> ls-1032 0DNh2 1001us+: try_to_wake_up+0x194/0x1b4 <cyclicte-914> (180 0)
>> ls-1032 0DNh. 1008us+: clockevents_program_event+0x84/0x1c0 ( 89 5ad6bbf 18e8e)
>> cyclicte-914 0D..2 1033us+: __schedule+0x21c/0x444 <ls-1032> (0 180)
>> cyclicte-914 0D..1 1071us+: enqueue_hrtimer+0x50/0x1a0 ( 89 5b9daa2 c3a6dea8)
>
> Looks good, it happened. Next wakeup at about 1071 us + 1000 us = 2071
> us.
>
>> cyclicte-914 0D..2 2031us+: __schedule+0x21c/0x444 <IRQ-129-682> (150 180)
>> cyclicte-914 0D..1 2072us+: enqueue_hrtimer+0x50/0x1a0 ( 89 5c91ce2 c3a6dea8)
>
> Bingo. Let's get a cup of coffee in the meantime ...
>
>> cyclicte-914 0D..1 1905066us+: enqueue_hrtimer+0x50/0x1a0 ( 91 66210 c3a6dea8)
>
> Expected arival at 1905066 us + 1000 us = 1906066 us.
>
>> ls-1048 0DNh3 1906017us+: __trace_start_sched_wakeup+0x154/0x18c <cyclicte-914> (19 -1)
>> ls-1048 0DNh3 1906021us : __trace_start_sched_wakeup+0x154/0x18c <cyclicte-914> (19 -1)
>> ls-1048 0DNh3 1906023us+: try_to_wake_up+0x194/0x1b4 <cyclicte-914> (180 0)
>
> It's trying hard to wake up ...
>
>> ls-1048 0DNh1 1906028us!: clockevents_program_event+0x84/0x1c0 ( 91 391676 367eff)
>> ls-1048 0DNh2 1906274us+: activate_task+0x60/0xa4 <IRQ-193-681> (150 6)
>
> Between these it should have happened.
>
>> ls-1048 0DNh2 1906285us+: __trace_start_sched_wakeup+0x154/0x18c <IRQ-193-681> (49 -1)
>> ls-1048 0DNh2 1906293us+: __trace_start_sched_wakeup+0x154/0x18c <IRQ-193-681> (49 -1)
>> ls-1048 0DNh2 1906295us+: try_to_wake_up+0x194/0x1b4 <IRQ-193-681> (150 0)
>> cyclicte-914 0D..2 1906332us : __schedule+0x21c/0x444 <ls-1048> (0 180)
>
> What is this IRQ-193-681? Can you post your /proc/interrupts and the
bash-3.00# cat /proc/interrupts
CPU0
129: 179 MPC52xx Peripherals Edge mpc52xx_psc_uart
133: 0 MPC52xx Peripherals Edge mpc52xx-fec_ctrl
135: 0 MPC52xx Peripherals Edge mpc52xx_ata
192: 14853 MPC52xx SDMA Edge mpc52xx-fec_rx
193: 9488 MPC52xx SDMA Edge mpc52xx-fec_tx
BAD: 0
> output of
>
> ps axHo user,pid,%cpu,%mem,nice,rtprio,policy,tty,stat,start_time,bsdtime,args
time,args# ps axHo
user,pid,%cpu,%mem,nice,rtprio,policy,tty,stat,start_time,bsd
USER PID %CPU %MEM NI RTPRIO POL TT STAT START TIME COMMAND
root 1 1.0 1.0 0 - TS ? Ss 00:00 0:00 init
[3]
root 2 0.0 0.0 -5 - TS ? S< 00:00 0:00
[kthreadd]
root 3 0.0 0.0 - 99 FF ? S< 00:00 0:00
[posix_cpu_t]
root 4 0.0 0.0 - 50 FF ? S< 00:00 0:00
[softirq-hig]
root 5 1.4 0.0 - 50 FF ? S< 00:00 0:01
[softirq-tim]
root 6 0.7 0.0 - 50 FF ? S< 00:00 0:00
[softirq-net]
root 7 2.4 0.0 - 50 FF ? S< 00:00 0:02
[softirq-net]
root 8 0.0 0.0 - 50 FF ? S< 00:00 0:00
[softirq-blo]
root 9 0.0 0.0 - 50 FF ? S< 00:00 0:00
[softirq-tas]
root 10 0.0 0.0 - 50 FF ? S< 00:00 0:00
[softirq-sch]
root 11 0.0 0.0 - 50 FF ? S< 00:00 0:00
[softirq-hrt]
root 12 0.4 0.0 - 50 FF ? S< 00:00 0:00
[softirq-rcu]
root 13 0.0 0.0 - 99 FF ? S< 00:00 0:00
[watchdog/0]
root 14 0.0 0.0 -10 - TS ? S< 00:00 0:00
[desched/0]
root 15 0.0 0.0 - 1 FF ? S< 00:00 0:00
[events/0]
root 16 0.2 0.0 -5 - TS ? S< 00:00 0:00 [khelper]
root 68 0.0 0.0 -5 - TS ? S< 00:00 0:00
[kblockd/0]
root 75 0.0 0.0 -5 - TS ? S< 00:00 0:00 [ata/0]
root 76 0.0 0.0 -5 - TS ? S< 00:00 0:00 [ata_aux]
root 93 0.0 0.0 - 1 FF ? S< 00:00 0:00
[krcupreempt]
root 94 0.0 0.0 0 - TS ? S 00:00 0:00 [pdflush]
root 95 0.0 0.0 0 - TS ? S 00:00 0:00 [pdflush]
root 96 0.0 0.0 -5 - TS ? S< 00:00 0:00 [kswapd0]
root 97 0.0 0.0 -5 - TS ? S< 00:00 0:00 [aio/0]
root 675 0.0 0.0 - 50 FF ? S< 00:00 0:00 [IRQ-135]
root 676 0.0 0.0 -5 - TS ? S< 00:00 0:00
[scsi_eh_0]
root 678 0.1 0.0 -5 - TS ? S< 00:00 0:00
[rpciod/0]
root 679 0.0 0.0 - 50 FF ? S< 00:00 0:00 [IRQ-133]
root 680 1.1 0.0 - 50 FF ? S< 00:00 0:01 [IRQ-192]
root 681 0.9 0.0 - 50 FF ? S< 00:00 0:00 [IRQ-193]
root 682 0.0 0.0 - 50 FF ? S< 00:00 0:00 [IRQ-129]
root 848 0.1 1.3 0 - TS ? Ss 00:00 0:00
syslogd -m 0
root 850 0.0 0.6 0 - TS ? Ss 00:00 0:00 klogd -x
root 884 0.1 1.6 0 - TS ? Ss 00:00 0:00
xinetd -staya
root 890 0.2 2.2 0 - TS ? Ss 00:00 0:00 login
-- root
root 891 0.3 2.8 0 - TS ? Ss 00:00 0:00 -bash
root 913 5.2 1.6 0 - TS ? Ss 00:00 0:03
in.telnetd: 1
root 914 0.1 2.2 0 - TS ? Ss 00:00 0:00 login
-- root
root 915 1.4 2.8 0 - TS ttyp0 Ss 00:00 0:00 -bash
root 938 8.4 1.1 0 - TS ttyp0 Sl+ 00:00 0:03
./cyclictest
root 938 5.5 1.1 - 99 FF ttyp0 Sl+ 00:00 0:02
./cyclictest
root 941 0.0 1.7 0 - TS ? R+ 00:01 0:00 ps
axHo user,
bash-3.00#
Wolfgang.
> Robert
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc
2008-01-17 10:13 ` 2.6.24-rc8-rt1 Wolfgang Grandegger
` (2 preceding siblings ...)
2008-01-17 21:11 ` 2.6.24-rc8-rt1 Robert Schwebel
@ 2008-01-23 14:53 ` Luotao Fu
2008-01-23 15:50 ` Daniel Walker
2008-01-23 16:36 ` Wolfgang Grandegger
3 siblings, 2 replies; 49+ messages in thread
From: Luotao Fu @ 2008-01-23 14:53 UTC (permalink / raw)
To: Steven Rostedt, Wolfgang Grandegger
Cc: LKML, RT, Ingo Molnar, Thomas Gleixner
[-- Attachment #1.1: Type: text/plain, Size: 2587 bytes --]
Hi folks,
On Thu, Jan 17, 2008 at 11:13:26AM +0100, Wolfgang Grandegger wrote:
> It builds and runs fine on my Icecube-MPC5200 board, now also with the
> latency tracer enabled. That's great. Still, "cyclictest -n -p80 -i1000"
> reports latencies up to 400 us and therefore I tried to trigger and save
> a high latency trace using:
>
> # ./cyclictest -n -p80 -i1000 -b400
> 1.21 0.33 0.11 4/42 1048
>
> T: 0 ( 914) P:80 I:1000 C: 38726 Min: 61 Act: 107 Avg: 106
> Max: 377
> [ 91.042169] ( cyclictest-914 |#0): new 39733427 us user-latency.
> bash-3.00# cat /proc/latency_trace > trace.log
>
I was doing some tests on my mpc5200b Board to reproduce the high latency as
measured by wolfgang.
I ran some tests with
while [ 1 ]; do ls /bin;done
as non-rt workload, as in Wolfgangs Scenario.
Now I also got some strange values. My latency lies at round about 100 and the
max. latency keep pending normally at about 150us-200us. However the max. value
will occasionally break out to very high values. I got a max. about 850us after
some rounds of measurement, which is definitively too high for the processor. I
made some traces and attached the last "interesting" path to this mail.
trace_600_1.log and trace_600_2.log are both taken with -b600. For comparation I
also added a "normal" trace taken with -b150. In the traces with abnormal long
latency there're a big "hole" between the last call, which is
clockevents_program_event() in both long traces and the actual schedule()
call. The holes are both about 600 us long, which is the main part of the
latency actually.
Two important things I also noted during my tests:
1. I got the unusual latencies on a system booted with nfsrootfs. I ran the same
test scenario on system booted from flash and got no extraordinory results.
After serveral hours test my max. latency lies at round about 200us.
2. Even on a nfsrootfs system I could not get the high latencies if I run
hackbench as non-rt workload.
Hence I suppose the unusual results are caused by network/Filesystemaccess.
However I have no idea what could be the reason for the "hole"s in the trace.
Looks almost like the cpu is doing nothing. As I don't have a trace on other
architecture at hand at the moment. I can't say for 100 procent if the tracer is
"missing" anything.
Any comments, ideas?
cheers
Luotao Fu
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
[-- Attachment #1.2: trace_600_1.log --]
[-- Type: text/plain, Size: 4621 bytes --]
cyclicte-16669 0D..2 2203085us+: __schedule+0x1f4/0x40c <IRQ-131-151> (150 180)
cyclicte-16669 0D..1 2203115us+: enqueue_hrtimer+0x48/0x194 ( 394 1ca956ec c3a11e98)
cyclicte-16669 0D..2 2203129us+: deactivate_task+0x58/0x9c <cyclicte-16669> (180 5)
IRQ-131-151 0D..2 2203135us+: __schedule+0x1f4/0x40c <cyclicte-16669> (180 150)
IRQ-131-151 0D..3 2203160us+: task_setprio+0xc0/0x258 <cyclicte-16668> (0 150)
IRQ-131-151 0D..3 2203164us+: task_setprio+0x13c/0x258 ( 0 0 0)
IRQ-131-151 0D.h1 2203173us+: hrtimer_interrupt+0xa8/0x2d0 ( 394 1c9cb6c7 0)
IRQ-131-151 0D.h2 2203176us+: hrtimer_interrupt+0x128/0x2d0 ( 394 1c9c3800 c02d6788)
IRQ-131-151 0D.h2 2203199us+: activate_task+0x58/0x9c <softirq--5> (150 4)
IRQ-131-151 0D.h2 2203217us+: activate_task+0x58/0x9c <softirq--12> (150 5)
IRQ-131-151 0D.h2 2203224us+: enqueue_hrtimer+0x48/0x194 ( 394 1d34ce80 c02d6788)
IRQ-131-151 0D.h1 2203231us+: clockevents_program_event+0x7c/0x1d0 ( 394 1ca956ec bc1d7)
IRQ-131-151 0D..2 2203250us+: deactivate_task+0x58/0x9c <IRQ-131-151> (150 6)
cyclicte-16668 0D..2 2203259us+: __schedule+0x1f4/0x40c <IRQ-131-151> (150 150)
cyclicte-16668 0D..2 2203272us+: activate_task+0x58/0x9c <IRQ-131-151> (150 5)
cyclicte-16668 0D..2 2203282us+: task_setprio+0x184/0x258 <cyclicte-16668> (150 0)
cyclicte-16668 0DN.2 2203291us+: task_setprio+0x13c/0x258 ( 0 1 0)
softirq--5 0D..2 2203306us+: __schedule+0x1f4/0x40c <cyclicte-16668> (0 150)
softirq--5 0D..2 2203340us+: deactivate_task+0x58/0x9c <softirq--5> (150 6)
softirq--12 0D..2 2203346us+: __schedule+0x1f4/0x40c <softirq--5> (150 150)
softirq--12 0D..2 2203361us+: deactivate_task+0x58/0x9c <softirq--12> (150 5)
IRQ-131-151 0D..2 2203367us+: __schedule+0x1f4/0x40c <softirq--12> (150 150)
IRQ-131-151 0D..1 2203401us+: activate_task+0x58/0x9c <softirq--9> (150 4)
IRQ-131-151 0D..2 2203428us+: deactivate_task+0x58/0x9c <IRQ-131-151> (150 5)
softirq--9 0D..2 2203435us+: __schedule+0x1f4/0x40c <IRQ-131-151> (150 150)
softirq--9 0D..2 2203454us+: deactivate_task+0x58/0x9c <softirq--9> (150 4)
cyclicte-16668 0D..2 2203465us+: __schedule+0x1f4/0x40c <softirq--9> (150 0)
cyclicte-16668 0D.h2 2203491us+: activate_task+0x58/0x9c <IRQ-131-151> (150 3)
cyclicte-16668 0D.h2 2203494us+: __trace_start_sched_wakeup+0x14c/0x184 <IRQ-131-151> (49 -1)
cyclicte-16668 0DNh2 2203500us+: __trace_start_sched_wakeup+0x14c/0x184 <IRQ-131-151> (49 -1)
cyclicte-16668 0DNh2 2203501us+: try_to_wake_up+0x184/0x1a4 <IRQ-131-151> (150 0)
IRQ-131-151 0D..2 2203519us+: __schedule+0x1f4/0x40c <cyclicte-16668> (0 150)
IRQ-131-151 0D..3 2203539us+: task_setprio+0xc0/0x258 <cyclicte-16668> (0 150)
IRQ-131-151 0D..3 2203544us+: task_setprio+0x13c/0x258 ( 0 0 0)
IRQ-131-151 0D..2 2203554us+: deactivate_task+0x58/0x9c <IRQ-131-151> (150 4)
cyclicte-16668 0D..2 2203559us+: __schedule+0x1f4/0x40c <IRQ-131-151> (150 150)
cyclicte-16668 0D..2 2203570us+: activate_task+0x58/0x9c <IRQ-131-151> (150 3)
cyclicte-16668 0D..2 2203577us+: task_setprio+0x184/0x258 <cyclicte-16668> (150 0)
cyclicte-16668 0DN.2 2203583us+: task_setprio+0x13c/0x258 ( 0 1 0)
IRQ-131-151 0D..2 2203594us+: __schedule+0x1f4/0x40c <cyclicte-16668> (0 150)
IRQ-131-151 0D..1 2203605us+: activate_task+0x58/0x9c <softirq--9> (150 4)
IRQ-131-151 0D..2 2203631us+: deactivate_task+0x58/0x9c <IRQ-131-151> (150 5)
softirq--9 0D..2 2203636us+: __schedule+0x1f4/0x40c <IRQ-131-151> (150 150)
softirq--9 0D..2 2203654us+: deactivate_task+0x58/0x9c <softirq--9> (150 4)
cyclicte-16668 0D..2 2203663us+: __schedule+0x1f4/0x40c <softirq--9> (150 0)
cyclicte-16668 0D..1 2203711us+: enqueue_hrtimer+0x48/0x194 ( 394 1d3d76bb c3a15ea8)
cyclicte-16668 0D..2 2203726us+: deactivate_task+0x58/0x9c <cyclicte-16668> (0 3)
sh-19600 0D..2 2203742us!: __schedule+0x1f4/0x40c <cyclicte-16668> (0 0)
sh-19600 0D.h1 2204020us+: hrtimer_interrupt+0xa8/0x2d0 ( 394 1ca99383 0)
sh-19600 0D.h2 2204025us+: hrtimer_interrupt+0x128/0x2d0 ( 394 1ca956ec c3a11e98)
sh-19600 0D.h3 2204039us+: activate_task+0x58/0x9c <cyclicte-16669> (180 2)
sh-19600 0D.h3 2204044us+: __trace_start_sched_wakeup+0x14c/0x184 <cyclicte-16669> (19 -1)
sh-19600 0DNh3 2204050us+: __trace_start_sched_wakeup+0x14c/0x184 <cyclicte-16669> (19 -1)
sh-19600 0DNh3 2204051us+: try_to_wake_up+0x184/0x1a4 <cyclicte-16669> (180 0)
sh-19600 0DNh1 2204058us!: clockevents_program_event+0x7c/0x1d0 ( 394 1d34ce80 8a9639)
cyclicte-16669 0D..2 2204680us : __schedule+0x1f4/0x40c <sh-19600> (0 180)
[-- Attachment #1.3: trace_600_2.log --]
[-- Type: text/plain, Size: 2789 bytes --]
cyclicte-1193 0D..2 2211546us+: __schedule+0x1f4/0x40c <softirq--6> (150 180)
cyclicte-1193 0D..1 2211575us+: enqueue_hrtimer+0x48/0x194 ( 1846 4569198 c3a33e98)
cyclicte-1193 0D..1 2211583us+: clockevents_program_event+0x7c/0x1d0 ( 1846 4569198 dbc5e)
cyclicte-1193 0D..2 2211597us+: deactivate_task+0x58/0x9c <cyclicte-1193> (180 4)
softirq--6 0D..2 2211603us+: __schedule+0x1f4/0x40c <cyclicte-1193> (180 150)
softirq--6 0D..2 2211625us+: deactivate_task+0x58/0x9c <softirq--6> (150 3)
sh-1902 0D..2 2211637us+: __schedule+0x1f4/0x40c <softirq--6> (150 0)
sh-1902 0D..2 2211689us+: deactivate_task+0x58/0x9c <sh-1902> (0 2)
sh-449 0D..2 2211704us+: __schedule+0x1f4/0x40c <sh-1902> (0 0)
sh-449 0D.h2 2211762us+: activate_task+0x58/0x9c <IRQ-192-149> (150 1)
sh-449 0D.h2 2211767us+: __trace_start_sched_wakeup+0x14c/0x184 <IRQ-192-149> (49 -1)
sh-449 0DNh2 2211772us+: __trace_start_sched_wakeup+0x14c/0x184 <IRQ-192-149> (49 -1)
sh-449 0DNh2 2211773us+: try_to_wake_up+0x184/0x1a4 <IRQ-192-149> (150 0)
IRQ-192-149 0D..2 2211792us+: __schedule+0x1f4/0x40c <sh-449> (0 150)
IRQ-192-149 0D..1 2211821us+: activate_task+0x58/0x9c <softirq--7> (150 2)
IRQ-192-149 0D..2 2211839us+: deactivate_task+0x58/0x9c <IRQ-192-149> (150 3)
softirq--7 0D..2 2211846us!: __schedule+0x1f4/0x40c <IRQ-192-149> (150 150)
softirq--7 0D..1 2211957us+: activate_task+0x58/0x9c <sh-1902> (0 2)
softirq--7 0D.h2 2212039us+: activate_task+0x58/0x9c <IRQ-193-150> (150 3)
softirq--7 0D..2 2212072us+: deactivate_task+0x58/0x9c <softirq--7> (150 4)
IRQ-193-150 0D..2 2212083us+: __schedule+0x1f4/0x40c <softirq--7> (150 150)
IRQ-193-150 0D..1 2212098us+: activate_task+0x58/0x9c <softirq--6> (150 3)
IRQ-193-150 0D..2 2212111us+: deactivate_task+0x58/0x9c <IRQ-193-150> (150 4)
softirq--6 0D..2 2212118us+: __schedule+0x1f4/0x40c <IRQ-193-150> (150 150)
softirq--6 0D..2 2212139us+: deactivate_task+0x58/0x9c <softirq--6> (150 3)
sh-1902 0D..2 2212152us!: __schedule+0x1f4/0x40c <softirq--6> (150 0)
sh-1902 0D.h1 2212501us+: hrtimer_interrupt+0xa8/0x2d0 ( 1846 456cfc8 0)
sh-1902 0D.h2 2212507us+: hrtimer_interrupt+0x128/0x2d0 ( 1846 4569198 c3a33e98)
sh-1902 0D.h3 2212521us+: activate_task+0x58/0x9c <cyclicte-1193> (180 2)
sh-1902 0D.h3 2212526us+: __trace_start_sched_wakeup+0x14c/0x184 <cyclicte-1193> (19 -1)
sh-1902 0DNh3 2212533us+: __trace_start_sched_wakeup+0x14c/0x184 <cyclicte-1193> (19 -1)
sh-1902 0DNh3 2212534us+: try_to_wake_up+0x184/0x1a4 <cyclicte-1193> (180 0)
sh-1902 0DNh1 2212541us!: clockevents_program_event+0x7c/0x1d0 ( 1846 4820f3a 2a9658)
cyclicte-1193 0D..2 2213164us : __schedule+0x1f4/0x40c <sh-1902> (0 180)
[-- Attachment #1.4: trace_150.log --]
[-- Type: text/plain, Size: 2708 bytes --]
cyclicte-12651 0D..2 2241836us+: __schedule+0x1f4/0x40c <sh-12822> (0 180)
cyclicte-12651 0D..1 2241863us+: enqueue_hrtimer+0x48/0x194 ( 2038 2ae2bf80 c3b57e98)
cyclicte-12651 0D..1 2241871us+: clockevents_program_event+0x7c/0x1d0 ( 2038 2ae2bf80 d8949)
cyclicte-12651 0D..2 2241886us+: deactivate_task+0x58/0x9c <cyclicte-12651> (180 2)
sh-12822 0D..2 2241898us!: __schedule+0x1f4/0x40c <cyclicte-12651> (180 0)
sh-12822 0D..1 2242158us+: activate_task+0x58/0x9c <telnetd-423> (0 1)
sh-12822 0DN.1 2242172us+: try_to_wake_up+0x184/0x1a4 <telnetd-423> (0 0)
telnetd-423 0D..2 2242198us+: __schedule+0x1f4/0x40c <sh-12822> (0 0)
telnetd-423 0D..2 2242253us+: deactivate_task+0x58/0x9c <telnetd-423> (0 2)
sh-12822 0D..2 2242264us+: __schedule+0x1f4/0x40c <telnetd-423> (0 0)
sh-12822 0D..2 2242274us+: activate_task+0x58/0x9c <telnetd-423> (0 1)
sh-12822 0DN.2 2242281us+: try_to_wake_up+0x184/0x1a4 <telnetd-423> (0 0)
telnetd-423 0D..2 2242293us+: __schedule+0x1f4/0x40c <sh-12822> (0 0)
telnetd-423 0D..2 2242339us+: deactivate_task+0x58/0x9c <telnetd-423> (0 2)
sh-12822 0D..2 2242351us!: __schedule+0x1f4/0x40c <telnetd-423> (0 0)
sh-12822 0D..3 2242490us+: activate_task+0x58/0x9c <telnetd-423> (0 1)
sh-12822 0DN.3 2242499us+: try_to_wake_up+0x184/0x1a4 <telnetd-423> (0 0)
telnetd-423 0D..2 2242516us!: __schedule+0x1f4/0x40c <sh-12822> (0 0)
telnetd-423 0D..2 2242774us+: deactivate_task+0x58/0x9c <telnetd-423> (0 2)
sh-12822 0D..2 2242796us+: __schedule+0x1f4/0x40c <telnetd-423> (0 0)
sh-12822 0D.h2 2242819us+: activate_task+0x58/0x9c <IRQ-193-150> (150 1)
sh-12822 0D.h2 2242824us+: __trace_start_sched_wakeup+0x14c/0x184 <IRQ-193-150> (49 -1)
sh-12822 0DNh2 2242828us+: __trace_start_sched_wakeup+0x14c/0x184 <IRQ-193-150> (49 -1)
sh-12822 0DNh2 2242830us+: try_to_wake_up+0x184/0x1a4 <IRQ-193-150> (150 0)
IRQ-193-150 0D..2 2242846us+: __schedule+0x1f4/0x40c <sh-12822> (0 150)
IRQ-193-150 0D.h. 2242860us+: hrtimer_interrupt+0xa8/0x2d0 ( 2038 2ae44ab9 0)
IRQ-193-150 0D.h1 2242863us+: hrtimer_interrupt+0x128/0x2d0 ( 2038 2ae2bf80 c3b57e98)
IRQ-193-150 0D.h2 2242871us+: activate_task+0x58/0x9c <cyclicte-12651> (180 2)
IRQ-193-150 0D.h2 2242875us+: __trace_start_sched_wakeup+0x14c/0x184 <cyclicte-12651> (19 -1)
IRQ-193-150 0DNh2 2242877us : __trace_start_sched_wakeup+0x14c/0x184 <cyclicte-12651> (19 -1)
IRQ-193-150 0DNh2 2242878us+: try_to_wake_up+0x184/0x1a4 <cyclicte-12651> (180 150)
IRQ-193-150 0DNh. 2242885us+: clockevents_program_event+0x7c/0x1d0 ( 2038 2aea5400 5a227)
cyclicte-12651 0D..2 2242900us : __schedule+0x1f4/0x40c <IRQ-193-150> (150 180)
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc
2008-01-23 14:53 ` 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc Luotao Fu
@ 2008-01-23 15:50 ` Daniel Walker
2008-01-23 16:36 ` Wolfgang Grandegger
1 sibling, 0 replies; 49+ messages in thread
From: Daniel Walker @ 2008-01-23 15:50 UTC (permalink / raw)
To: Luotao Fu
Cc: Steven Rostedt, Wolfgang Grandegger, LKML, RT, Ingo Molnar,
Thomas Gleixner
On Wed, 2008-01-23 at 15:53 +0100, Luotao Fu wrote:
> I was doing some tests on my mpc5200b Board to reproduce the high latency as
> measured by wolfgang.
>
> I ran some tests with
> while [ 1 ]; do ls /bin;done
> as non-rt workload, as in Wolfgangs Scenario.
>
Can you try enabling CRITICAL_PREEMPT_TIMING only, and not latency
tracing .. You won't get a trace, but you will get some indication of
the start/stop points of the latency if it's preempt latency, and a
measurement of the longest latency..
Daniel
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc
2008-01-23 14:53 ` 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc Luotao Fu
2008-01-23 15:50 ` Daniel Walker
@ 2008-01-23 16:36 ` Wolfgang Grandegger
2008-01-24 10:53 ` Wolfgang Grandegger
2008-01-28 15:11 ` Luotao Fu
1 sibling, 2 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-23 16:36 UTC (permalink / raw)
To: Luotao Fu
Cc: Steven Rostedt, Wolfgang Grandegger, LKML, RT, Ingo Molnar,
Thomas Gleixner
Hi Fu,
Luotao Fu wrote:
> Hi folks,
>
> On Thu, Jan 17, 2008 at 11:13:26AM +0100, Wolfgang Grandegger wrote:
>> It builds and runs fine on my Icecube-MPC5200 board, now also with the
>> latency tracer enabled. That's great. Still, "cyclictest -n -p80 -i1000"
>> reports latencies up to 400 us and therefore I tried to trigger and save
>> a high latency trace using:
>>
>> # ./cyclictest -n -p80 -i1000 -b400
>> 1.21 0.33 0.11 4/42 1048
>>
>> T: 0 ( 914) P:80 I:1000 C: 38726 Min: 61 Act: 107 Avg: 106
>> Max: 377
>> [ 91.042169] ( cyclictest-914 |#0): new 39733427 us user-latency.
>> bash-3.00# cat /proc/latency_trace > trace.log
>>
>
> I was doing some tests on my mpc5200b Board to reproduce the high latency as
> measured by wolfgang.
>
> I ran some tests with
> while [ 1 ]; do ls /bin;done
> as non-rt workload, as in Wolfgangs Scenario.
I also did some more measurements and made, by chance, interesting
observations. I will summarize in more detail later on. Here are some
preliminary results. My high latencies of up to 570us (without latency
tracer) seem to be caused mainly by the following setting:
CONFIG_RCU_TRACE=m
which is the default if CONFIG_MODULES=y. With CONFIG_RCU_TRACE=y
latencies go down significantly. I furthermore realized some bad impact
of CONFIG_NO_HZ and CONFIG_PPC_BESTCOMM_GEN_BD. With the following
settings the latencies did not yet exceed 140 us.
CONFIG_PREEMPT_RCU_BOOST=y
CONFIG_RCU_TRACE=y
# CONFIG_PPC_BESTCOMM_GEN_BD is not set
# CONFIG_NO_HZ is not set
With CONFIG_NO_HZ=y or CONFIG_PPC_BESTCOMM_GEN_BD=y the latency
increases by approx. 100..150us, each.
> Now I also got some strange values. My latency lies at round about 100 and the
> max. latency keep pending normally at about 150us-200us. However the max. value
> will occasionally break out to very high values. I got a max. about 850us after
> some rounds of measurement, which is definitively too high for the processor. I
> made some traces and attached the last "interesting" path to this mail.
> trace_600_1.log and trace_600_2.log are both taken with -b600. For comparation I
> also added a "normal" trace taken with -b150. In the traces with abnormal long
> latency there're a big "hole" between the last call, which is
> clockevents_program_event() in both long traces and the actual schedule()
> call. The holes are both about 600 us long, which is the main part of the
> latency actually.
>
> Two important things I also noted during my tests:
> 1. I got the unusual latencies on a system booted with nfsrootfs. I ran the same
> test scenario on system booted from flash and got no extraordinory results.
> After serveral hours test my max. latency lies at round about 200us.
> 2. Even on a nfsrootfs system I could not get the high latencies if I run
> hackbench as non-rt workload.
>
> Hence I suppose the unusual results are caused by network/Filesystemaccess.
> However I have no idea what could be the reason for the "hole"s in the trace.
> Looks almost like the cpu is doing nothing. As I don't have a trace on other
> architecture at hand at the moment. I can't say for 100 procent if the tracer is
> "missing" anything.
>
> Any comments, ideas?
Could you check your .config and try the __good__ settings mentioned
above? Can you back my observations? Don't ask me why. Maybe somebody
else could shed some light on this.
Wolfgang,
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc
2008-01-23 16:36 ` Wolfgang Grandegger
@ 2008-01-24 10:53 ` Wolfgang Grandegger
[not found] ` <20080124112847.GE4776@unix.sh>
2008-01-28 15:11 ` Luotao Fu
1 sibling, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-24 10:53 UTC (permalink / raw)
To: Luotao Fu; +Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
[-- Attachment #1: Type: text/plain, Size: 4474 bytes --]
Wolfgang Grandegger wrote:
> Hi Fu,
>
> Luotao Fu wrote:
>> Hi folks,
>>
>> On Thu, Jan 17, 2008 at 11:13:26AM +0100, Wolfgang Grandegger wrote:
>>> It builds and runs fine on my Icecube-MPC5200 board, now also with the
>>> latency tracer enabled. That's great. Still, "cyclictest -n -p80 -i1000"
>>> reports latencies up to 400 us and therefore I tried to trigger and save
>>> a high latency trace using:
>>>
>>> # ./cyclictest -n -p80 -i1000 -b400
>>> 1.21 0.33 0.11 4/42 1048
>>>
>>> T: 0 ( 914) P:80 I:1000 C: 38726 Min: 61 Act: 107 Avg: 106
>>> Max: 377
>>> [ 91.042169] ( cyclictest-914 |#0): new 39733427 us user-latency.
>>> bash-3.00# cat /proc/latency_trace > trace.log
>>>
>> I was doing some tests on my mpc5200b Board to reproduce the high latency as
>> measured by wolfgang.
>>
>> I ran some tests with
>> while [ 1 ]; do ls /bin;done
>> as non-rt workload, as in Wolfgangs Scenario.
>
> I also did some more measurements and made, by chance, interesting
> observations. I will summarize in more detail later on. Here are some
> preliminary results. My high latencies of up to 570us (without latency
> tracer) seem to be caused mainly by the following setting:
OK, here is the full report.
I had the occasion to test Linux 2.6.24-rc8-rt1 on a TQM5200 module with
a MPC5200 Rev.B processor and, what surprise, the latencies measured
were much better:
CPU: MPC5200B v2.2, Core v1.4 at 396 MHz
Bus 132 MHz, IPB 132 MHz, PCI 66 MHz
Board: TQM5200 (TQ-Components GmbH)
on a STK52xx carrier board
bash-3.00# ./cyclictest -n -p80 -t1 -i1000
2.91 4.81 13.72 1/50 23887
T: 0 ( 976) P:80 I:1000 C:1634520 Min: 15 Act: 45 Avg: 68 Max: 138
So far I used my very old Icecube eval board with a MPC5200 *Rev. *A for
testing, This made me curious and I made measurements on a TQM5200
module with a MPC5200 Rev.A processor as well booting the same kernel,
but the results were the same:
CPU: MPC5200 v1.2, Core v1.1 at 396 MHz
Bus 132 MHz, IPB 132 MHz, PCI 66 MHz
Board: TQM5200 (TQ-Components GmbH)
on a STK52xx carrier board
bash-3.00# ./cyclictest -n -p80 -i1000
52.31 96.08 61.61 2/51 9129
T: 0 ( 976) P:80 I:1000 C: 795180 Min: 14 Act: 75 Avg: 69 Max: 134
Hm, then I used that config for my Icecube and also on that board, the
latency did not yet exceed 150us. With my old config, the latency jumped
up to 500..600us within a minute or so. Therefore I started to study the
impact of the different kernel options on the latency. Let's start from
the *good* settings (see attached .config) giving the following results:
CPU: MPC5200 v1.0, Core v1.1 at 198 MHz
Bus 132 MHz, IPB 66 MHz, PCI 33 MHz
Board: Motorola MPC5200 (IceCube)
bash-3.00# ./cyclictest -n -p80 -t1 -i1000
291.40 143.72 115.43 5/447 28227
T: 0 ( 938) P:80 I:1000 C:3613444 Min: 17 Act: 68 Avg: 65 Max: 143
Note: this test was running for *1* hour.
Then I repeated the measurements with the following configs:
- .config and "CONFIG_RCU_TRACE=m":
bash-3.00# ./cyclictest -n -p80 -t1 -i1000
2.49 1.19 0.45 2/46 2846
T: 0 ( 1144) P:80 I:1000 C: 130362 Min: 17 Act: 57 Avg: 73 Max: 522
Note: High latencies show up quickly.
- .config and CONFIG_NO_HZ=y:
bash-3.00# ./cyclictest -n -p80 -t1 -i1000
15.83 79.48 83.66 3/47 16861
T: 0 ( 913) P:80 I:1000 C:1924513 Min: 20 Act: 81 Avg: 84 Max: 226
- .config and CONFIG_PPC_BESTCOMM_GEN_BD=y
bash-3.00# ./cyclictest -n -p80 -t1 -i1000
87.81 116.12 85.31 3/47 11858
T: 0 ( 914) P:80 I:1000 C:1389487 Min: 17 Act: 80 Avg: 67 Max: 147
- .config and CONFIG_PREEMPT_RCU_BOOST is not set:
bash-3.00# ./cyclictest -n -p80 -t1 -i10000
81.36 103.93 67.34 3/46 9593
T: 0 ( 913) P:80 I:10000 C: 91915 Min: 25 Act: 64 Avg: 72 Max: 298
And here is my summary:
- CONFIG_RCU_TRACE=m causes high latencies up to 500..600us.
- CONFIG_NO_HZ=y increases the latency by 90 us, at least.
- CONFIG_PPC_BESTCOMM_GEN_BD=y seems not to harm.
- CONFIG_PREEMPT_RCU_BOOST is not set increases the latency by 150us, at
least
For the tests I used a root file-system mounted via NFS and "while ls;
do ls/bin; done" in one telnet window and "while ./hackbench 10; do
./calibrator 400 32M cali; sleep 30; done" in another telnet window as
non-rt load.
I don't know if "my" .config is already optimal but a latency below
150us seems more reasonable for that processor.
Please comment.
Wolfgang.
[-- Attachment #2: uImage-rt-test1.config --]
[-- Type: text/plain, Size: 21323 bytes --]
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.24-rc8-rt1
# Wed Jan 23 11:40:51 2008
#
# CONFIG_PPC64 is not set
#
# Processor support
#
CONFIG_6xx=y
# CONFIG_PPC_85xx is not set
# CONFIG_PPC_8xx is not set
# CONFIG_40x is not set
# CONFIG_44x is not set
# CONFIG_E200 is not set
CONFIG_PPC_FPU=y
# CONFIG_ALTIVEC is not set
CONFIG_PPC_STD_MMU=y
CONFIG_PPC_STD_MMU_32=y
# CONFIG_PPC_MM_SLICES is not set
# CONFIG_SMP is not set
CONFIG_PPC32=y
CONFIG_WORD_SIZE=32
CONFIG_PPC_MERGE=y
CONFIG_MMU=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_IRQ_PER_CPU=y
CONFIG_ARCH_HAS_ILOG2_U32=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
# CONFIG_ARCH_NO_VIRT_TO_BUS is not set
CONFIG_PPC=y
CONFIG_EARLY_PRINTK=y
CONFIG_GENERIC_NVRAM=y
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_PPC_OF=y
CONFIG_OF=y
# CONFIG_PPC_UDBG_16550 is not set
# CONFIG_GENERIC_TBSYNC is not set
CONFIG_AUDIT_ARCH=y
CONFIG_GENERIC_BUG=y
# CONFIG_DEFAULT_UIMAGE is not set
# CONFIG_PPC_DCR_NATIVE is not set
# CONFIG_PPC_DCR_MMIO is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_AUDIT is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=14
# CONFIG_CGROUPS is not set
# CONFIG_FAIR_GROUP_SCHED is not set
CONFIG_SYSFS_DEPRECATED=y
# CONFIG_RELAY is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
# CONFIG_RADIX_TREE_CONCURRENT is not set
CONFIG_EMBEDDED=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
# CONFIG_LOGBUFFER is not set
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
# CONFIG_EPOLL is not set
CONFIG_SIGNALFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_KMOD is not set
CONFIG_BLOCK=y
# CONFIG_LBD is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_LSF is not set
# CONFIG_BLK_DEV_BSG is not set
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_AS=y
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="anticipatory"
#
# Platform support
#
CONFIG_PPC_MULTIPLATFORM=y
# CONFIG_PPC_82xx is not set
# CONFIG_PPC_83xx is not set
# CONFIG_PPC_86xx is not set
CONFIG_CLASSIC32=y
# CONFIG_PPC_CHRP is not set
CONFIG_PPC_MPC52xx=y
CONFIG_PPC_MPC5200=y
CONFIG_PPC_MPC5200_BUGFIX=y
# CONFIG_PPC_EFIKA is not set
CONFIG_PPC_LITE5200=y
# CONFIG_PPC_MOTIONPRO is not set
# CONFIG_PPC_TQM5200 is not set
# CONFIG_PPC_CM5200 is not set
# CONFIG_PPC_PMAC is not set
# CONFIG_PPC_CELL is not set
# CONFIG_PPC_CELL_NATIVE is not set
# CONFIG_PQ2ADS is not set
# CONFIG_EMBEDDED6xx is not set
# CONFIG_MPIC is not set
# CONFIG_MPIC_WEIRD is not set
# CONFIG_PPC_I8259 is not set
# CONFIG_PPC_RTAS is not set
# CONFIG_MMIO_NVRAM is not set
# CONFIG_PPC_MPC106 is not set
# CONFIG_PPC_970_NAP is not set
# CONFIG_PPC_INDIRECT_IO is not set
# CONFIG_GENERIC_IOMAP is not set
# CONFIG_CPU_FREQ is not set
# CONFIG_TAU is not set
# CONFIG_CPM2 is not set
# CONFIG_FSL_ULI1575 is not set
CONFIG_PPC_BESTCOMM=y
CONFIG_PPC_BESTCOMM_ATA=y
CONFIG_PPC_BESTCOMM_FEC=y
# CONFIG_PPC_BESTCOMM_GEN_BD is not set
#
# Kernel options
#
# CONFIG_HIGHMEM is not set
CONFIG_TICK_ONESHOT=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_BKL=y
# CONFIG_CLASSIC_RCU is not set
CONFIG_PREEMPT_RCU=y
CONFIG_PREEMPT_RCU_BOOST=y
CONFIG_RCU_TRACE=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_ASM_SEMAPHORES=y
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_MISC is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
# CONFIG_KEXEC is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
# CONFIG_SPARSEMEM_VMEMMAP_ENABLE is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_PROC_DEVICETREE=y
# CONFIG_CMDLINE_BOOL is not set
CONFIG_PM=y
# CONFIG_PM_LEGACY is not set
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND_UP_POSSIBLE=y
CONFIG_SUSPEND=y
CONFIG_HIBERNATION_UP_POSSIBLE=y
# CONFIG_HIBERNATION is not set
CONFIG_SECCOMP=y
CONFIG_WANT_DEVICE_TREE=y
CONFIG_DEVICE_TREE=""
CONFIG_ISA_DMA_API=y
#
# Bus options
#
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
# CONFIG_PPC_INDIRECT_PCI is not set
CONFIG_FSL_SOC=y
CONFIG_PCI=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCI_SYSCALL=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set
#
# Advanced setup
#
# CONFIG_ADVANCED_OPTIONS is not set
#
# Default settings for advanced configuration options are used
#
CONFIG_HIGHMEM_START=0xfe000000
CONFIG_LOWMEM_SIZE=0x30000000
CONFIG_KERNEL_START=0xc0000000
CONFIG_TASK_SIZE=0xc0000000
CONFIG_BOOT_LOAD=0x00800000
#
# Networking
#
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
CONFIG_IP_PNP_BOOTP=y
# CONFIG_IP_PNP_RARP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
CONFIG_INET_XFRM_MODE_TRANSPORT=y
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
# CONFIG_INET_LRO is not set
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_INET6_XFRM_TUNNEL is not set
# CONFIG_INET6_TUNNEL is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETFILTER is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_SCHED is not set
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
#
# Wireless
#
# CONFIG_CFG80211 is not set
# CONFIG_WIRELESS_EXT is not set
# CONFIG_MAC80211 is not set
# CONFIG_IEEE80211 is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
# CONFIG_FW_LOADER is not set
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
CONFIG_OF_DEVICE=y
# CONFIG_PARPORT is not set
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=32768
CONFIG_BLK_DEV_RAM_BLOCKSIZE=1024
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_MISC_DEVICES=y
# CONFIG_PHANTOM is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_IDE is not set
#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
# CONFIG_SCSI_PROC_FS is not set
#
# SCSI support type (disk, tape, CD-ROM)
#
# CONFIG_BLK_DEV_SD is not set
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
# CONFIG_CHR_DEV_SG is not set
# CONFIG_CHR_DEV_SCH is not set
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
# CONFIG_SATA_AHCI is not set
# CONFIG_SATA_SVW is not set
# CONFIG_ATA_PIIX is not set
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIL24 is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
CONFIG_PATA_MPC52xx=y
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_PLATFORM is not set
# CONFIG_MD is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set
# CONFIG_ARCNET is not set
CONFIG_PHYLIB=y
#
# MII PHY device drivers
#
# CONFIG_MARVELL_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_ICPLUS_PHY is not set
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set
CONFIG_NET_ETHERNET=y
# CONFIG_MII is not set
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_NET_PCI is not set
# CONFIG_B44 is not set
CONFIG_FEC_MPC52xx=y
CONFIG_FEC_MPC52xx_MDIO=y
# CONFIG_NETDEV_1000 is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set
#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
#
# Input device support
#
# CONFIG_INPUT is not set
#
# Hardware I/O ports
#
# CONFIG_SERIO is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#
# CONFIG_VT is not set
# CONFIG_SERIAL_NONSTANDARD is not set
#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
#
# Non-8250 serial port support
#
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_MPC52xx=y
CONFIG_SERIAL_MPC52xx_CONSOLE=y
CONFIG_SERIAL_MPC52xx_CONSOLE_BAUD=9600
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
# CONFIG_GEN_RTC is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_TCG_TPM is not set
CONFIG_RMEM=m
CONFIG_ALLOC_RTSJ_MEM=m
CONFIG_DEVPORT=y
# CONFIG_I2C is not set
#
# SPI support
#
# CONFIG_SPI is not set
# CONFIG_SPI_MASTER is not set
# CONFIG_W1 is not set
# CONFIG_POWER_SUPPLY is not set
# CONFIG_HWMON is not set
# CONFIG_WATCHDOG is not set
#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
#
# Multifunction device drivers
#
# CONFIG_MFD_SM501 is not set
#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
# CONFIG_DAB is not set
#
# Graphics support
#
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=m
# CONFIG_FB is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set
#
# Sound
#
# CONFIG_SOUND is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
# CONFIG_USB is not set
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#
#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set
# CONFIG_MMC is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
# CONFIG_RTC_CLASS is not set
#
# Userspace I/O
#
# CONFIG_UIO is not set
#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
# CONFIG_EXT4DEV_FS is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set
#
# DOS/FAT/NT Filesystems
#
# CONFIG_MSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLB_PAGE is not set
# CONFIG_CONFIGFS_FS is not set
#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
# CONFIG_NFS_V3 is not set
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_DIRECTIO is not set
# CONFIG_NFSD is not set
CONFIG_ROOT_NFS=y
CONFIG_LOCKD=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_SUNRPC_BIND34 is not set
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_NLS is not set
# CONFIG_DLM is not set
# CONFIG_UCC_SLOW is not set
#
# Library routines
#
CONFIG_BITREVERSE=y
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
# CONFIG_CRC_ITU_T is not set
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
# CONFIG_LIBCRC32C is not set
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
# CONFIG_INSTRUMENTATION is not set
CONFIG_MARKERS=y
#
# Kernel hacking
#
CONFIG_PRINTK_TIME=y
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
# CONFIG_MAGIC_SYSRQ is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_SLAB is not set
CONFIG_DEBUG_PREEMPT=y
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_PREEMPT_TRACE=y
# CONFIG_EVENT_TRACE is not set
# CONFIG_FUNCTION_TRACE is not set
# CONFIG_WAKEUP_TIMING is not set
# CONFIG_CRITICAL_PREEMPT_TIMING is not set
# CONFIG_CRITICAL_IRQSOFF_TIMING is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_BUGVERBOSE is not set
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
CONFIG_FORCED_INLINING=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_HAVE_MCOUNT=y
# CONFIG_SAMPLES is not set
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_DEBUGGER is not set
# CONFIG_VIRQ_DEBUG is not set
# CONFIG_BDI_SWITCH is not set
# CONFIG_BOOTX_TEXT is not set
# CONFIG_PPC_EARLY_DEBUG is not set
#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
# CONFIG_CRYPTO is not set
CONFIG_PPC_CLOCK=y
CONFIG_PPC_LIB_RHEAP=y
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc
2008-01-23 16:36 ` Wolfgang Grandegger
2008-01-24 10:53 ` Wolfgang Grandegger
@ 2008-01-28 15:11 ` Luotao Fu
2008-01-28 15:38 ` Wolfgang Grandegger
1 sibling, 1 reply; 49+ messages in thread
From: Luotao Fu @ 2008-01-28 15:11 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Hi,
I took some time today and went through Wolfgangs scenarios partly. Now
some results from my side. I ran my tests on a 2.6.24-rt1
Wolfgang Grandegger wrote:
> I also did some more measurements and made, by chance, interesting
> observations. I will summarize in more detail later on. Here are some
> preliminary results. My high latencies of up to 570us (without latency
> tracer) seem to be caused mainly by the following setting:
>
> CONFIG_RCU_TRACE=m
>
I also got high latencies without CONFIG_RCU_TRACE set at all. setting
CONFIG_RCU_TRACE to y or m causes also high latency though in my test
runs. Hence I doubt the rcu Tracer is really the problem.
As I mentioned in my last mail. Only thing I could reproduce reliably is
that the measurement results depend heavily on kind of the non-rt
Workload. For with hackbench or cache calibrator I couldn't produce
abnormal high latencies. Nor could I produce the hight latencies on a
system booted via flash. Hence my suspects stays on the fec irq thread
and filesystem access routines.
> With CONFIG_NO_HZ=y or CONFIG_PPC_BESTCOMM_GEN_BD=y the latency
> increases by approx. 100..150us, each.
>
Since I was producing high latencies independently from the rcu
settings. I didn't spend a lot of time playing around with the GEN_BD
and dynamic clock. All I can say is that disabling them also cause high
latency. ;-) The average results don't differ significantly in my test runs.
cheers
Luotao Fu
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc
2008-01-28 15:11 ` Luotao Fu
@ 2008-01-28 15:38 ` Wolfgang Grandegger
2008-01-29 12:13 ` 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue? Luotao Fu
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-28 15:38 UTC (permalink / raw)
To: Luotao Fu; +Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Hi Fu,
Luotao Fu wrote:
> Hi,
>
> I took some time today and went through Wolfgangs scenarios partly. Now
> some results from my side. I ran my tests on a 2.6.24-rt1
>
> Wolfgang Grandegger wrote:
> > I also did some more measurements and made, by chance, interesting
>> observations. I will summarize in more detail later on. Here are some
>> preliminary results. My high latencies of up to 570us (without latency
>> tracer) seem to be caused mainly by the following setting:
>>
>> CONFIG_RCU_TRACE=m
>>
>
> I also got high latencies without CONFIG_RCU_TRACE set at all. setting
> CONFIG_RCU_TRACE to y or m causes also high latency though in my test
> runs. Hence I doubt the rcu Tracer is really the problem.
>
> As I mentioned in my last mail. Only thing I could reproduce reliably is
> that the measurement results depend heavily on kind of the non-rt
> Workload. For with hackbench or cache calibrator I couldn't produce
> abnormal high latencies. Nor could I produce the hight latencies on a
> system booted via flash. Hence my suspects stays on the fec irq thread
> and filesystem access routines.
>
>> With CONFIG_NO_HZ=y or CONFIG_PPC_BESTCOMM_GEN_BD=y the latency
>> increases by approx. 100..150us, each.
>>
>
> Since I was producing high latencies independently from the rcu
> settings. I didn't spend a lot of time playing around with the GEN_BD
> and dynamic clock. All I can say is that disabling them also cause high
> latency. ;-) The average results don't differ significantly in my test
> runs.
In the meantime I have measured the impact more carefully and posted the
results to the list:
- CONFIG_RCU_TRACE or CONFIG_PREEMPT_RCU_BOOST not set causes high
latencies up to 500..600us.
- CONFIG_NO_HZ=y increases the latency by 90 us, at least.
- CONFIG_PPC_BESTCOMM_GEN_BD=y seems not to harm.
Do you still get high latencies with:
CONFIG_PREEMPT_RCU_BOOST=y
CONFIG_RCU_TRACE=y
CONFIG_NO_HZ is not set
With this setting I have not yet realized latencies > 150us. Could you
please give it a try? If I change one of the parameters above, latency
increases in short time.
Thanks.
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-28 15:38 ` Wolfgang Grandegger
@ 2008-01-29 12:13 ` Luotao Fu
2008-01-29 13:38 ` Wolfgang Grandegger
0 siblings, 1 reply; 49+ messages in thread
From: Luotao Fu @ 2008-01-29 12:13 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Hi,
Wolfgang Grandegger wrote:
..........
> Do you still get high latencies with:
>
> CONFIG_PREEMPT_RCU_BOOST=y
> CONFIG_RCU_TRACE=y
> CONFIG_NO_HZ is not set
>
> With this setting I have not yet realized latencies > 150us. Could you
> please give it a try? If I change one of the parameters above, latency
> increases in short time.
I played through some combination of the RCU options and can back your
observation this time: With the rcu Tracer or the priority boost turned
off I also could measure reliably extraordinory high latencies. If they
are both turned on, no high latencies could be measured. Turning on the
dynamic ticker however doesn't seem to cause high latencies during my
test runs. Seemed like an rcu issue here.
Further such results only appear if the target board is booted with
nfsrootfs. (As I already have mentioned several times before), which
leads my suspection to rcu usage in nfs implementation. In this case
this problem might even be platformindependent. I'd have to do some
tests on one of our arm boards later to test this. Since there're no
reports like this for other architecture as powerpc till now, I doubt
quite if this is verifiable.
regards
Luotao Fu
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-29 12:13 ` 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue? Luotao Fu
@ 2008-01-29 13:38 ` Wolfgang Grandegger
2008-01-30 1:07 ` Paul E. McKenney
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-29 13:38 UTC (permalink / raw)
To: Luotao Fu; +Cc: Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Luotao Fu wrote:
> Hi,
>
> Wolfgang Grandegger wrote:
> ..........
>> Do you still get high latencies with:
>>
>> CONFIG_PREEMPT_RCU_BOOST=y
>> CONFIG_RCU_TRACE=y
>> CONFIG_NO_HZ is not set
>>
>> With this setting I have not yet realized latencies > 150us. Could you
>> please give it a try? If I change one of the parameters above, latency
>> increases in short time.
>
> I played through some combination of the RCU options and can back your
> observation this time: With the rcu Tracer or the priority boost turned
> off I also could measure reliably extraordinory high latencies. If they
> are both turned on, no high latencies could be measured. Turning on the
> dynamic ticker however doesn't seem to cause high latencies during my
> test runs. Seemed like an rcu issue here.
I'm just making a long test run on my TQM5200 module with my good
settings. After more than 4.5 hours under load, cyclictest shows a
maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.
> Further such results only appear if the target board is booted with
> nfsrootfs. (As I already have mentioned several times before), which
> leads my suspection to rcu usage in nfs implementation. In this case
> this problem might even be platformindependent. I'd have to do some
> tests on one of our arm boards later to test this. Since there're no
> reports like this for other architecture as powerpc till now, I doubt
> quite if this is verifiable.
It's also my suspicion that the high latencies are related to the RCU
usage in the network layer, where it's heavily used. What is really
wired is that switching off CONFIG_RCU_TRACE has a negative impact on
the latency. As I see it, it just adds some trace points, but I might
have missed something.
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-29 13:38 ` Wolfgang Grandegger
@ 2008-01-30 1:07 ` Paul E. McKenney
2008-01-30 8:18 ` Wolfgang Grandegger
0 siblings, 1 reply; 49+ messages in thread
From: Paul E. McKenney @ 2008-01-30 1:07 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
On Tue, Jan 29, 2008 at 02:38:04PM +0100, Wolfgang Grandegger wrote:
> Luotao Fu wrote:
> > Hi,
> >
> > Wolfgang Grandegger wrote:
> > ..........
> >> Do you still get high latencies with:
> >>
> >> CONFIG_PREEMPT_RCU_BOOST=y
> >> CONFIG_RCU_TRACE=y
> >> CONFIG_NO_HZ is not set
> >>
> >> With this setting I have not yet realized latencies > 150us. Could you
> >> please give it a try? If I change one of the parameters above, latency
> >> increases in short time.
> >
> > I played through some combination of the RCU options and can back your
> > observation this time: With the rcu Tracer or the priority boost turned
> > off I also could measure reliably extraordinory high latencies. If they
> > are both turned on, no high latencies could be measured. Turning on the
> > dynamic ticker however doesn't seem to cause high latencies during my
> > test runs. Seemed like an rcu issue here.
>
> I'm just making a long test run on my TQM5200 module with my good
> settings. After more than 4.5 hours under load, cyclictest shows a
> maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.
>
> > Further such results only appear if the target board is booted with
> > nfsrootfs. (As I already have mentioned several times before), which
> > leads my suspection to rcu usage in nfs implementation. In this case
> > this problem might even be platformindependent. I'd have to do some
> > tests on one of our arm boards later to test this. Since there're no
> > reports like this for other architecture as powerpc till now, I doubt
> > quite if this is verifiable.
>
> It's also my suspicion that the high latencies are related to the RCU
> usage in the network layer, where it's heavily used. What is really
> wired is that switching off CONFIG_RCU_TRACE has a negative impact on
> the latency. As I see it, it just adds some trace points, but I might
> have missed something.
I would expect that CONFIG_RCU_TRACE=n (as in "no" rather than "module")
would have low latencies rather than high latencies. So I am quite
surprised by your result. I will dig into this more.
Thanx, Paul
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-30 1:07 ` Paul E. McKenney
@ 2008-01-30 8:18 ` Wolfgang Grandegger
2008-01-30 10:22 ` Paul E. McKenney
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-30 8:18 UTC (permalink / raw)
To: paulmck; +Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Paul E. McKenney wrote:
> On Tue, Jan 29, 2008 at 02:38:04PM +0100, Wolfgang Grandegger wrote:
>> Luotao Fu wrote:
>>> Hi,
>>>
>>> Wolfgang Grandegger wrote:
>>> ..........
>>>> Do you still get high latencies with:
>>>>
>>>> CONFIG_PREEMPT_RCU_BOOST=y
>>>> CONFIG_RCU_TRACE=y
>>>> CONFIG_NO_HZ is not set
>>>>
>>>> With this setting I have not yet realized latencies > 150us. Could you
>>>> please give it a try? If I change one of the parameters above, latency
>>>> increases in short time.
>>> I played through some combination of the RCU options and can back your
>>> observation this time: With the rcu Tracer or the priority boost turned
>>> off I also could measure reliably extraordinory high latencies. If they
>>> are both turned on, no high latencies could be measured. Turning on the
>>> dynamic ticker however doesn't seem to cause high latencies during my
>>> test runs. Seemed like an rcu issue here.
>> I'm just making a long test run on my TQM5200 module with my good
>> settings. After more than 4.5 hours under load, cyclictest shows a
>> maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.
>>
>>> Further such results only appear if the target board is booted with
>>> nfsrootfs. (As I already have mentioned several times before), which
>>> leads my suspection to rcu usage in nfs implementation. In this case
>>> this problem might even be platformindependent. I'd have to do some
>>> tests on one of our arm boards later to test this. Since there're no
>>> reports like this for other architecture as powerpc till now, I doubt
>>> quite if this is verifiable.
>> It's also my suspicion that the high latencies are related to the RCU
>> usage in the network layer, where it's heavily used. What is really
>> wired is that switching off CONFIG_RCU_TRACE has a negative impact on
>> the latency. As I see it, it just adds some trace points, but I might
>> have missed something.
>
> I would expect that CONFIG_RCU_TRACE=n (as in "no" rather than "module")
> would have low latencies rather than high latencies. So I am quite
> surprised by your result. I will dig into this more.
Thanks a lot. To be clear. I need "CONFIG_PREEMPT_RCU_BOOST=y" *and*
"CONFIG_RCU_TRACE=y" to achieve reasonable latencies below 180us. With
CONFIG_PREEMPT_RCU_BOOST or CONFIG_RCU_TRACE not set or
CONFIG_RCU_TRACE=m is rmeasure latencies up to 600us within a minute or so.
Wolfgang.
> Thanx, Paul
> -
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-30 8:18 ` Wolfgang Grandegger
@ 2008-01-30 10:22 ` Paul E. McKenney
2008-01-30 10:45 ` Wolfgang Grandegger
0 siblings, 1 reply; 49+ messages in thread
From: Paul E. McKenney @ 2008-01-30 10:22 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
On Wed, Jan 30, 2008 at 09:18:49AM +0100, Wolfgang Grandegger wrote:
> Paul E. McKenney wrote:
> > On Tue, Jan 29, 2008 at 02:38:04PM +0100, Wolfgang Grandegger wrote:
> >> Luotao Fu wrote:
> >>> Hi,
> >>>
> >>> Wolfgang Grandegger wrote:
> >>> ..........
> >>>> Do you still get high latencies with:
> >>>>
> >>>> CONFIG_PREEMPT_RCU_BOOST=y
> >>>> CONFIG_RCU_TRACE=y
> >>>> CONFIG_NO_HZ is not set
> >>>>
> >>>> With this setting I have not yet realized latencies > 150us. Could you
> >>>> please give it a try? If I change one of the parameters above, latency
> >>>> increases in short time.
> >>> I played through some combination of the RCU options and can back your
> >>> observation this time: With the rcu Tracer or the priority boost turned
> >>> off I also could measure reliably extraordinory high latencies. If they
> >>> are both turned on, no high latencies could be measured. Turning on the
> >>> dynamic ticker however doesn't seem to cause high latencies during my
> >>> test runs. Seemed like an rcu issue here.
> >> I'm just making a long test run on my TQM5200 module with my good
> >> settings. After more than 4.5 hours under load, cyclictest shows a
> >> maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.
> >>
> >>> Further such results only appear if the target board is booted with
> >>> nfsrootfs. (As I already have mentioned several times before), which
> >>> leads my suspection to rcu usage in nfs implementation. In this case
> >>> this problem might even be platformindependent. I'd have to do some
> >>> tests on one of our arm boards later to test this. Since there're no
> >>> reports like this for other architecture as powerpc till now, I doubt
> >>> quite if this is verifiable.
> >> It's also my suspicion that the high latencies are related to the RCU
> >> usage in the network layer, where it's heavily used. What is really
> >> wired is that switching off CONFIG_RCU_TRACE has a negative impact on
> >> the latency. As I see it, it just adds some trace points, but I might
> >> have missed something.
> >
> > I would expect that CONFIG_RCU_TRACE=n (as in "no" rather than "module")
> > would have low latencies rather than high latencies. So I am quite
> > surprised by your result. I will dig into this more.
>
> Thanks a lot. To be clear. I need "CONFIG_PREEMPT_RCU_BOOST=y" *and*
> "CONFIG_RCU_TRACE=y" to achieve reasonable latencies below 180us. With
> CONFIG_PREEMPT_RCU_BOOST or CONFIG_RCU_TRACE not set or
> CONFIG_RCU_TRACE=m is rmeasure latencies up to 600us within a minute or so.
OK, thank you for the confirmation.
The large latencies were from cyclictest, correct? Did other tests
also show these latencies? In either case, could you please send the
exact command line you used to run the test?
Thanx, Paul
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-30 10:22 ` Paul E. McKenney
@ 2008-01-30 10:45 ` Wolfgang Grandegger
2008-01-30 10:57 ` Paul E. McKenney
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-30 10:45 UTC (permalink / raw)
To: paulmck; +Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Paul E. McKenney wrote:
> On Wed, Jan 30, 2008 at 09:18:49AM +0100, Wolfgang Grandegger wrote:
>> Paul E. McKenney wrote:
>>> On Tue, Jan 29, 2008 at 02:38:04PM +0100, Wolfgang Grandegger wrote:
>>>> Luotao Fu wrote:
>>>>> Hi,
>>>>>
>>>>> Wolfgang Grandegger wrote:
>>>>> ..........
>>>>>> Do you still get high latencies with:
>>>>>>
>>>>>> CONFIG_PREEMPT_RCU_BOOST=y
>>>>>> CONFIG_RCU_TRACE=y
>>>>>> CONFIG_NO_HZ is not set
>>>>>>
>>>>>> With this setting I have not yet realized latencies > 150us. Could you
>>>>>> please give it a try? If I change one of the parameters above, latency
>>>>>> increases in short time.
>>>>> I played through some combination of the RCU options and can back your
>>>>> observation this time: With the rcu Tracer or the priority boost turned
>>>>> off I also could measure reliably extraordinory high latencies. If they
>>>>> are both turned on, no high latencies could be measured. Turning on the
>>>>> dynamic ticker however doesn't seem to cause high latencies during my
>>>>> test runs. Seemed like an rcu issue here.
>>>> I'm just making a long test run on my TQM5200 module with my good
>>>> settings. After more than 4.5 hours under load, cyclictest shows a
>>>> maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.
>>>>
>>>>> Further such results only appear if the target board is booted with
>>>>> nfsrootfs. (As I already have mentioned several times before), which
>>>>> leads my suspection to rcu usage in nfs implementation. In this case
>>>>> this problem might even be platformindependent. I'd have to do some
>>>>> tests on one of our arm boards later to test this. Since there're no
>>>>> reports like this for other architecture as powerpc till now, I doubt
>>>>> quite if this is verifiable.
>>>> It's also my suspicion that the high latencies are related to the RCU
>>>> usage in the network layer, where it's heavily used. What is really
>>>> wired is that switching off CONFIG_RCU_TRACE has a negative impact on
>>>> the latency. As I see it, it just adds some trace points, but I might
>>>> have missed something.
>>> I would expect that CONFIG_RCU_TRACE=n (as in "no" rather than "module")
>>> would have low latencies rather than high latencies. So I am quite
>>> surprised by your result. I will dig into this more.
>> Thanks a lot. To be clear. I need "CONFIG_PREEMPT_RCU_BOOST=y" *and*
>> "CONFIG_RCU_TRACE=y" to achieve reasonable latencies below 180us. With
>> CONFIG_PREEMPT_RCU_BOOST or CONFIG_RCU_TRACE not set or
>> CONFIG_RCU_TRACE=m is rmeasure latencies up to 600us within a minute or so.
>
> OK, thank you for the confirmation.
>
> The large latencies were from cyclictest, correct? Did other tests
> also show these latencies? In either case, could you please send the
> exact command line you used to run the test?
Yes, I used "$ cyclictest -n -t1 -p80 -i1000" to measure the latency. So
far, I have not done other tests. Any recommendation?
As no-rt load I used "while ls; do ls /bin; done" in one telnet window
and "while ./hackbench 10; do ./calibrator 400 32M cali; sleep 30; done"
in another. But already "while ls; do ls /bin; done" is enough to
trigger the high latencies quickly. Note also, that I work on a root
files-ystem mounted via NFS resulting in a lot of network traffic and
utilization.
Thanks,
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-30 10:45 ` Wolfgang Grandegger
@ 2008-01-30 10:57 ` Paul E. McKenney
2008-01-30 11:15 ` Luotao Fu
2008-01-30 11:22 ` Wolfgang Grandegger
0 siblings, 2 replies; 49+ messages in thread
From: Paul E. McKenney @ 2008-01-30 10:57 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
On Wed, Jan 30, 2008 at 11:45:01AM +0100, Wolfgang Grandegger wrote:
> Paul E. McKenney wrote:
> > On Wed, Jan 30, 2008 at 09:18:49AM +0100, Wolfgang Grandegger wrote:
> >> Paul E. McKenney wrote:
> >>> On Tue, Jan 29, 2008 at 02:38:04PM +0100, Wolfgang Grandegger wrote:
> >>>> Luotao Fu wrote:
> >>>>> Hi,
> >>>>>
> >>>>> Wolfgang Grandegger wrote:
> >>>>> ..........
> >>>>>> Do you still get high latencies with:
> >>>>>>
> >>>>>> CONFIG_PREEMPT_RCU_BOOST=y
> >>>>>> CONFIG_RCU_TRACE=y
> >>>>>> CONFIG_NO_HZ is not set
> >>>>>>
> >>>>>> With this setting I have not yet realized latencies > 150us. Could you
> >>>>>> please give it a try? If I change one of the parameters above, latency
> >>>>>> increases in short time.
> >>>>> I played through some combination of the RCU options and can back your
> >>>>> observation this time: With the rcu Tracer or the priority boost turned
> >>>>> off I also could measure reliably extraordinory high latencies. If they
> >>>>> are both turned on, no high latencies could be measured. Turning on the
> >>>>> dynamic ticker however doesn't seem to cause high latencies during my
> >>>>> test runs. Seemed like an rcu issue here.
> >>>> I'm just making a long test run on my TQM5200 module with my good
> >>>> settings. After more than 4.5 hours under load, cyclictest shows a
> >>>> maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.
> >>>>
> >>>>> Further such results only appear if the target board is booted with
> >>>>> nfsrootfs. (As I already have mentioned several times before), which
> >>>>> leads my suspection to rcu usage in nfs implementation. In this case
> >>>>> this problem might even be platformindependent. I'd have to do some
> >>>>> tests on one of our arm boards later to test this. Since there're no
> >>>>> reports like this for other architecture as powerpc till now, I doubt
> >>>>> quite if this is verifiable.
> >>>> It's also my suspicion that the high latencies are related to the RCU
> >>>> usage in the network layer, where it's heavily used. What is really
> >>>> wired is that switching off CONFIG_RCU_TRACE has a negative impact on
> >>>> the latency. As I see it, it just adds some trace points, but I might
> >>>> have missed something.
> >>> I would expect that CONFIG_RCU_TRACE=n (as in "no" rather than "module")
> >>> would have low latencies rather than high latencies. So I am quite
> >>> surprised by your result. I will dig into this more.
> >> Thanks a lot. To be clear. I need "CONFIG_PREEMPT_RCU_BOOST=y" *and*
> >> "CONFIG_RCU_TRACE=y" to achieve reasonable latencies below 180us. With
> >> CONFIG_PREEMPT_RCU_BOOST or CONFIG_RCU_TRACE not set or
> >> CONFIG_RCU_TRACE=m is rmeasure latencies up to 600us within a minute or so.
> >
> > OK, thank you for the confirmation.
> >
> > The large latencies were from cyclictest, correct? Did other tests
> > also show these latencies? In either case, could you please send the
> > exact command line you used to run the test?
>
> Yes, I used "$ cyclictest -n -t1 -p80 -i1000" to measure the latency. So
> far, I have not done other tests. Any recommendation?
> As no-rt load I used "while ls; do ls /bin; done" in one telnet window
> and "while ./hackbench 10; do ./calibrator 400 32M cali; sleep 30; done"
> in another. But already "while ls; do ls /bin; done" is enough to
> trigger the high latencies quickly. Note also, that I work on a root
> files-ystem mounted via NFS resulting in a lot of network traffic and
> utilization.
I have to ask...
Did you see large latencies when -not- running on NFS?
Thanx, Paul
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-30 10:57 ` Paul E. McKenney
@ 2008-01-30 11:15 ` Luotao Fu
2008-07-01 14:27 ` Wolfgang Grandegger
2008-01-30 11:22 ` Wolfgang Grandegger
1 sibling, 1 reply; 49+ messages in thread
From: Luotao Fu @ 2008-01-30 11:15 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Wolfgang Grandegger, Luotao Fu, Steven Rostedt, LKML, RT,
Ingo Molnar, Thomas Gleixner
[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]
Hi,
On Wed, Jan 30, 2008 at 02:57:16AM -0800, Paul E. McKenney wrote:
......
> > Yes, I used "$ cyclictest -n -t1 -p80 -i1000" to measure the latency. So
> > far, I have not done other tests. Any recommendation?
> > As no-rt load I used "while ls; do ls /bin; done" in one telnet window
> > and "while ./hackbench 10; do ./calibrator 400 32M cali; sleep 30; done"
> > in another. But already "while ls; do ls /bin; done" is enough to
> > trigger the high latencies quickly. Note also, that I work on a root
> > files-ystem mounted via NFS resulting in a lot of network traffic and
> > utilization.
>
> I have to ask...
>
> Did you see large latencies when -not- running on NFS?
>
I cannot speak for Wolfgang but I myself did not get extraordinary high
latencies running tests on system booted from flash. Neither I could produce
high latencies on nfs booted system. If my non-rt workload doesn't do heavy
filesystem/network accesses. i.E. running only hackbench. Hence we were
wondering if the problem is caused by rcu's in network layer or nfs
implementation.
regards
Luotao Fu
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-30 11:15 ` Luotao Fu
@ 2008-07-01 14:27 ` Wolfgang Grandegger
2008-07-01 15:11 ` Steven Rostedt
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-01 14:27 UTC (permalink / raw)
To: Paul E. McKenney, Wolfgang Grandegger, Luotao Fu, Steven Rostedt,
LKML
Hi,
I continue this thread because it's still not understood why enabling
CONFIG_RCU_TRACE is necessary to get reasonable latencies on the
MPC5200. It might also explain, why I get much worse latencies with
2.6.25.8-rt7.
Luotao Fu wrote:
> Hi,
>
> On Wed, Jan 30, 2008 at 02:57:16AM -0800, Paul E. McKenney wrote:
> ......
>>> Yes, I used "$ cyclictest -n -t1 -p80 -i1000" to measure the latency. So
>>> far, I have not done other tests. Any recommendation?
>>> As no-rt load I used "while ls; do ls /bin; done" in one telnet window
>>> and "while ./hackbench 10; do ./calibrator 400 32M cali; sleep 30; done"
>>> in another. But already "while ls; do ls /bin; done" is enough to
>>> trigger the high latencies quickly. Note also, that I work on a root
>>> files-ystem mounted via NFS resulting in a lot of network traffic and
>>> utilization.
>> I have to ask...
>>
>> Did you see large latencies when -not- running on NFS?
>>
>
> I cannot speak for Wolfgang but I myself did not get extraordinary high
> latencies running tests on system booted from flash. Neither I could produce
> high latencies on nfs booted system. If my non-rt workload doesn't do heavy
> filesystem/network accesses. i.E. running only hackbench. Hence we were
> wondering if the problem is caused by rcu's in network layer or nfs
> implementation.
To recapitulate, with CONFIG_RCU_TRACE enabled, cyclictest reports max.
latencies of approx. 130 us with 2.6.24.4-rt4 on my MPC5200 PowerPC
board. If I disable it, the latency goes up to 600 us. Obviously, the
trace_mark() calls in rcupreempt*.c have some positive impact on the
latency. I narrowed down, that the 2 calls in __rcu_preempt_boost() in
rcupreempt-boost.c are the important one:
void __rcu_preempt_unboost(void)
{
struct task_struct *curr = current;
struct rcu_boost_dat *rbd;
int prio;
unsigned long flags;
trace_mark(unboost_called, "NULL");
/* if not boosted, then ignore */
if (likely(!rcu_is_boosted(curr)))
return;
/*
* Need to be very careful with NMIs.
* If we take the lock and an NMI comes in
* and it may try to unboost us if curr->rcub_rbdp
* is still set. So we zero it before grabbing the lock.
* But this also means that we might be boosted again
* so the boosting code needs to be aware of this.
*/
rbd = curr->rcub_rbdp;
curr->rcub_rbdp = NULL;
/*
* Now an NMI might have came in after we grab
* the below lock. This check makes sure that
* the NMI doesn't try grabbing the lock
* while we already have it.
*/
if (unlikely(!rbd))
return;
spin_lock_irqsave(&rbd->rbs_lock, flags);
/*
* It is still possible that an NMI came in
* between the "is_boosted" check and setting
* the rcu_rbdp to NULL. This would mean that
* the NMI already dequeued us.
*/
if (unlikely(!rcu_is_boosted(curr)))
goto out;
list_del_init(&curr->rcub_entry);
trace_mark(unboosted, "NULL");
curr->rcu_prio = MAX_PRIO;
spin_lock(&curr->pi_lock);
prio = rt_mutex_getprio(curr);
task_setprio(curr, prio);
curr->rcub_rbdp = NULL;
spin_unlock(&curr->pi_lock);
out:
spin_unlock_irqrestore(&rbd->rbs_lock, flags);
}
With them and all other trace_mark() calls commented out, the latency is
still OK. The first one has a bigger impact.
In 2.6.25.8-rt7, trace_mark() is not used any more but a function
incrementing the corresponding counter directly and I suspect that's the
reason why I'm seeing high latencies with both, CONFIG_RCU_TRACE enabled
and disabled.
I hope this observation sheds some light on the issue.
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-01 14:27 ` Wolfgang Grandegger
@ 2008-07-01 15:11 ` Steven Rostedt
2008-07-01 16:11 ` Wolfgang Grandegger
2008-07-02 8:09 ` Wolfgang Grandegger
0 siblings, 2 replies; 49+ messages in thread
From: Steven Rostedt @ 2008-07-01 15:11 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Paul E. McKenney, Luotao Fu, LKML, RT, Ingo Molnar,
Thomas Gleixner
On Tue, 1 Jul 2008, Wolfgang Grandegger wrote:
>
> I continue this thread because it's still not understood why enabling
> CONFIG_RCU_TRACE is necessary to get reasonable latencies on the
> MPC5200. It might also explain, why I get much worse latencies with
> 2.6.25.8-rt7.
Have you tried turning on ftrace?
>
> To recapitulate, with CONFIG_RCU_TRACE enabled, cyclictest reports max.
> latencies of approx. 130 us with 2.6.24.4-rt4 on my MPC5200 PowerPC
> board. If I disable it, the latency goes up to 600 us. Obviously, the
> trace_mark() calls in rcupreempt*.c have some positive impact on the
> latency. I narrowed down, that the 2 calls in __rcu_preempt_boost() in
> rcupreempt-boost.c are the important one:
>
> void __rcu_preempt_unboost(void)
> {
> struct task_struct *curr = current;
> struct rcu_boost_dat *rbd;
> int prio;
> unsigned long flags;
>
> trace_mark(unboost_called, "NULL");
>
> /* if not boosted, then ignore */
> if (likely(!rcu_is_boosted(curr)))
> return;
I wonder if the "likely" is faulty on the PPC code generation. Have you
tried removing that "likely" statement.
>
> /*
> * Need to be very careful with NMIs.
> * If we take the lock and an NMI comes in
> * and it may try to unboost us if curr->rcub_rbdp
> * is still set. So we zero it before grabbing the lock.
> * But this also means that we might be boosted again
> * so the boosting code needs to be aware of this.
> */
> rbd = curr->rcub_rbdp;
> curr->rcub_rbdp = NULL;
>
> /*
> * Now an NMI might have came in after we grab
> * the below lock. This check makes sure that
> * the NMI doesn't try grabbing the lock
> * while we already have it.
> */
> if (unlikely(!rbd))
> return;
Actually, remove all "likely" and "unlikely". The marker code could be
making it work better. But still, this shouldn't cause 600us latencies.
>
> spin_lock_irqsave(&rbd->rbs_lock, flags);
> /*
> * It is still possible that an NMI came in
> * between the "is_boosted" check and setting
> * the rcu_rbdp to NULL. This would mean that
> * the NMI already dequeued us.
> */
> if (unlikely(!rcu_is_boosted(curr)))
> goto out;
>
> list_del_init(&curr->rcub_entry);
>
> trace_mark(unboosted, "NULL");
>
> curr->rcu_prio = MAX_PRIO;
>
> spin_lock(&curr->pi_lock);
> prio = rt_mutex_getprio(curr);
> task_setprio(curr, prio);
>
> curr->rcub_rbdp = NULL;
>
> spin_unlock(&curr->pi_lock);
> out:
> spin_unlock_irqrestore(&rbd->rbs_lock, flags);
> }
>
> With them and all other trace_mark() calls commented out, the latency is
> still OK. The first one has a bigger impact.
>
> In 2.6.25.8-rt7, trace_mark() is not used any more but a function
> incrementing the corresponding counter directly and I suspect that's the
> reason why I'm seeing high latencies with both, CONFIG_RCU_TRACE enabled
> and disabled.
>
> I hope this observation sheds some light on the issue.
It is still a mystery to me. Maybe looking at the different assembly
outputs with the different configurations.
-- Steve
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-01 15:11 ` Steven Rostedt
@ 2008-07-01 16:11 ` Wolfgang Grandegger
2008-07-01 21:11 ` Luotao Fu
2008-07-02 11:03 ` Wolfgang Grandegger
2008-07-02 8:09 ` Wolfgang Grandegger
1 sibling, 2 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-01 16:11 UTC (permalink / raw)
To: Steven Rostedt
Cc: Paul E. McKenney, Luotao Fu, LKML, RT, Ingo Molnar,
Thomas Gleixner
Steven Rostedt wrote:
> On Tue, 1 Jul 2008, Wolfgang Grandegger wrote:
>> I continue this thread because it's still not understood why enabling
>> CONFIG_RCU_TRACE is necessary to get reasonable latencies on the
>> MPC5200. It might also explain, why I get much worse latencies with
>> 2.6.25.8-rt7.
>
> Have you tried turning on ftrace?
Not yet.
>
>> To recapitulate, with CONFIG_RCU_TRACE enabled, cyclictest reports max.
>> latencies of approx. 130 us with 2.6.24.4-rt4 on my MPC5200 PowerPC
>> board. If I disable it, the latency goes up to 600 us. Obviously, the
>> trace_mark() calls in rcupreempt*.c have some positive impact on the
>> latency. I narrowed down, that the 2 calls in __rcu_preempt_boost() in
>> rcupreempt-boost.c are the important one:
>>
>> void __rcu_preempt_unboost(void)
>> {
>> struct task_struct *curr = current;
>> struct rcu_boost_dat *rbd;
>> int prio;
>> unsigned long flags;
>>
>> trace_mark(unboost_called, "NULL");
To make it clear: If I just comment out the line above and ...
>>
>> /* if not boosted, then ignore */
>> if (likely(!rcu_is_boosted(curr)))
>> return;
>
> I wonder if the "likely" is faulty on the PPC code generation. Have you
> tried removing that "likely" statement.
>
>> /*
>> * Need to be very careful with NMIs.
>> * If we take the lock and an NMI comes in
>> * and it may try to unboost us if curr->rcub_rbdp
>> * is still set. So we zero it before grabbing the lock.
>> * But this also means that we might be boosted again
>> * so the boosting code needs to be aware of this.
>> */
>> rbd = curr->rcub_rbdp;
>> curr->rcub_rbdp = NULL;
>>
>> /*
>> * Now an NMI might have came in after we grab
>> * the below lock. This check makes sure that
>> * the NMI doesn't try grabbing the lock
>> * while we already have it.
>> */
>> if (unlikely(!rbd))
>> return;
>
> Actually, remove all "likely" and "unlikely". The marker code could be
> making it work better. But still, this shouldn't cause 600us latencies.
>
>> spin_lock_irqsave(&rbd->rbs_lock, flags);
>> /*
>> * It is still possible that an NMI came in
>> * between the "is_boosted" check and setting
>> * the rcu_rbdp to NULL. This would mean that
>> * the NMI already dequeued us.
>> */
>> if (unlikely(!rcu_is_boosted(curr)))
>> goto out;
>>
>> list_del_init(&curr->rcub_entry);
>>
>> trace_mark(unboosted, "NULL");
.. this one as well, then the latency goes *up* to 600us. The first one
has more influence, though.
>>
>> curr->rcu_prio = MAX_PRIO;
>>
>> spin_lock(&curr->pi_lock);
>> prio = rt_mutex_getprio(curr);
>> task_setprio(curr, prio);
>>
>> curr->rcub_rbdp = NULL;
>>
>> spin_unlock(&curr->pi_lock);
>> out:
>> spin_unlock_irqrestore(&rbd->rbs_lock, flags);
>> }
>>
>> With them and all other trace_mark() calls commented out, the latency is
>> still OK. The first one has a bigger impact.
>>
>> In 2.6.25.8-rt7, trace_mark() is not used any more but a function
>> incrementing the corresponding counter directly and I suspect that's the
>> reason why I'm seeing high latencies with both, CONFIG_RCU_TRACE enabled
>> and disabled.
>>
>> I hope this observation sheds some light on the issue.
>
> It is still a mystery to me. Maybe looking at the different assembly
> outputs with the different configurations.
There seems to be something in trace_mark() keeping latency low:
http://lxr.linux.no/linux+v2.6.24.4/include/linux/marker.h#L52
I will follow your suggestions.
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-01 16:11 ` Wolfgang Grandegger
@ 2008-07-01 21:11 ` Luotao Fu
2008-07-02 11:03 ` Wolfgang Grandegger
1 sibling, 0 replies; 49+ messages in thread
From: Luotao Fu @ 2008-07-01 21:11 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Steven Rostedt, Paul E. McKenney, Luotao Fu, LKML, RT,
Ingo Molnar, Thomas Gleixner
[-- Attachment #1: Type: text/plain, Size: 616 bytes --]
Hi Wolfgang,
On Tue, Jul 01, 2008 at 06:11:42PM +0200, Wolfgang Grandegger wrote:
[...]
I'm out of office till next week. I will try to narrow down the issue you
reported on my board asap. Unfortunately I was not able to produce such worst
case results in my test runs till now. Anyhow I'll also have a look at the code
segment you found and play a little around there. Hopefully I'll find something
useful. ;-)
Cheers
Luotao Fu
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-01 16:11 ` Wolfgang Grandegger
2008-07-01 21:11 ` Luotao Fu
@ 2008-07-02 11:03 ` Wolfgang Grandegger
2008-07-06 0:42 ` Steven Rostedt
2008-08-01 21:09 ` Paul E. McKenney
1 sibling, 2 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-02 11:03 UTC (permalink / raw)
To: Steven Rostedt
Cc: Paul E. McKenney, Luotao Fu, LKML, RT, Ingo Molnar,
Thomas Gleixner
Wolfgang Grandegger wrote:
> Steven Rostedt wrote:
>> On Tue, 1 Jul 2008, Wolfgang Grandegger wrote:
>>> I continue this thread because it's still not understood why enabling
>>> CONFIG_RCU_TRACE is necessary to get reasonable latencies on the
>>> MPC5200. It might also explain, why I get much worse latencies with
>>> 2.6.25.8-rt7.
>>
>> Have you tried turning on ftrace?
>
> Not yet.
>
>>
>>> To recapitulate, with CONFIG_RCU_TRACE enabled, cyclictest reports max.
>>> latencies of approx. 130 us with 2.6.24.4-rt4 on my MPC5200 PowerPC
>>> board. If I disable it, the latency goes up to 600 us. Obviously, the
>>> trace_mark() calls in rcupreempt*.c have some positive impact on the
>>> latency. I narrowed down, that the 2 calls in __rcu_preempt_boost() in
>>> rcupreempt-boost.c are the important one:
>>>
>>> void __rcu_preempt_unboost(void)
>>> {
>>> struct task_struct *curr = current;
>>> struct rcu_boost_dat *rbd;
>>> int prio;
>>> unsigned long flags;
>>>
>>> trace_mark(unboost_called, "NULL");
>
> To make it clear: If I just comment out the line above and ...
>
>>>
>>> /* if not boosted, then ignore */
>>> if (likely(!rcu_is_boosted(curr)))
>>> return;
>>
>> I wonder if the "likely" is faulty on the PPC code generation. Have you
>> tried removing that "likely" statement.
>>
>>> /*
>>> * Need to be very careful with NMIs.
>>> * If we take the lock and an NMI comes in
>>> * and it may try to unboost us if curr->rcub_rbdp
>>> * is still set. So we zero it before grabbing the lock.
>>> * But this also means that we might be boosted again
>>> * so the boosting code needs to be aware of this.
>>> */
>>> rbd = curr->rcub_rbdp;
>>> curr->rcub_rbdp = NULL;
>>>
>>> /*
>>> * Now an NMI might have came in after we grab
>>> * the below lock. This check makes sure that
>>> * the NMI doesn't try grabbing the lock
>>> * while we already have it.
>>> */
>>> if (unlikely(!rbd))
>>> return;
>>
>> Actually, remove all "likely" and "unlikely". The marker code could be
>> making it work better. But still, this shouldn't cause 600us latencies.
>>
>>> spin_lock_irqsave(&rbd->rbs_lock, flags);
>>> /*
>>> * It is still possible that an NMI came in
>>> * between the "is_boosted" check and setting
>>> * the rcu_rbdp to NULL. This would mean that
>>> * the NMI already dequeued us.
>>> */
>>> if (unlikely(!rcu_is_boosted(curr)))
>>> goto out;
>>>
>>> list_del_init(&curr->rcub_entry);
>>>
>>> trace_mark(unboosted, "NULL");
>
> .. this one as well, then the latency goes *up* to 600us. The first one
> has more influence, though.
>
>>>
>>> curr->rcu_prio = MAX_PRIO;
>>>
>>> spin_lock(&curr->pi_lock);
>>> prio = rt_mutex_getprio(curr);
>>> task_setprio(curr, prio);
>>>
>>> curr->rcub_rbdp = NULL;
>>>
>>> spin_unlock(&curr->pi_lock);
>>> out:
>>> spin_unlock_irqrestore(&rbd->rbs_lock, flags);
>>> }
>>>
>>> With them and all other trace_mark() calls commented out, the latency is
>>> still OK. The first one has a bigger impact.
>>>
>>> In 2.6.25.8-rt7, trace_mark() is not used any more but a function
>>> incrementing the corresponding counter directly and I suspect that's the
>>> reason why I'm seeing high latencies with both, CONFIG_RCU_TRACE enabled
>>> and disabled.
>>>
>>> I hope this observation sheds some light on the issue.
>>
>> It is still a mystery to me. Maybe looking at the different assembly
>> outputs with the different configurations.
>
> There seems to be something in trace_mark() keeping latency low:
>
> http://lxr.linux.no/linux+v2.6.24.4/include/linux/marker.h#L52
>
> I will follow your suggestions.
I removed all "likely" and "unlikely" macros, but the latencies did not
improve. Then I added
preempt_disable();
preempt_enable();
at the two locations mentioned above, like trace_mark() does, and
disabled CONFIG_RCU_TRACE. That helped to keep the latencies low, and it
did for 2.6.25.8-rt7 as well. As I see it, adding preemption points seem
to prevent high latencies.
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-02 11:03 ` Wolfgang Grandegger
@ 2008-07-06 0:42 ` Steven Rostedt
2008-07-06 9:41 ` Wolfgang Grandegger
2008-08-01 21:09 ` Paul E. McKenney
1 sibling, 1 reply; 49+ messages in thread
From: Steven Rostedt @ 2008-07-06 0:42 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Paul E. McKenney, Luotao Fu, LKML, RT, Ingo Molnar,
Thomas Gleixner
On Wed, 2 Jul 2008, Wolfgang Grandegger wrote:
>
> I removed all "likely" and "unlikely" macros, but the latencies did not
> improve. Then I added
>
> preempt_disable();
> preempt_enable();
>
> at the two locations mentioned above, like trace_mark() does, and
> disabled CONFIG_RCU_TRACE. That helped to keep the latencies low, and it
> did for 2.6.25.8-rt7 as well. As I see it, adding preemption points seem
> to prevent high latencies.
This to me sounds like we have preempt_enable_noresched someplace that
shouldn't. In otherwords, we enabled preemption without checking if
preemption is needed, which is a serious bug in -rt.
-- Steve
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-06 0:42 ` Steven Rostedt
@ 2008-07-06 9:41 ` Wolfgang Grandegger
2008-07-08 15:08 ` Luotao Fu
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-06 9:41 UTC (permalink / raw)
To: Steven Rostedt
Cc: Paul E. McKenney, Luotao Fu, LKML, RT, Ingo Molnar,
Thomas Gleixner
Steven Rostedt wrote:
> On Wed, 2 Jul 2008, Wolfgang Grandegger wrote:
>> I removed all "likely" and "unlikely" macros, but the latencies did not
>> improve. Then I added
>>
>> preempt_disable();
>> preempt_enable();
>>
>> at the two locations mentioned above, like trace_mark() does, and
>> disabled CONFIG_RCU_TRACE. That helped to keep the latencies low, and it
>> did for 2.6.25.8-rt7 as well. As I see it, adding preemption points seem
>> to prevent high latencies.
>
> This to me sounds like we have preempt_enable_noresched someplace that
> shouldn't. In otherwords, we enabled preemption without checking if
> preemption is needed, which is a serious bug in -rt.
Yep, in 2.6.25.8-rt7 trace_mark() is not used any more, maybe for that
reason. Nevertheless, this "bug" kept the latency on my MPC5200 box low.
I will try with the old tracer to understand what the real cause of the
high latencies is (with CONFIG_RCU_TRACE disabled).
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-06 9:41 ` Wolfgang Grandegger
@ 2008-07-08 15:08 ` Luotao Fu
2008-07-08 19:43 ` Wolfgang Grandegger
0 siblings, 1 reply; 49+ messages in thread
From: Luotao Fu @ 2008-07-08 15:08 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Steven Rostedt, Paul E. McKenney, Luotao Fu, LKML, RT,
Ingo Molnar, Thomas Gleixner
[-- Attachment #1.1: Type: text/plain, Size: 1548 bytes --]
Hi,
I found some time and played with my mpc5200B Board again on this issue
(2.6.25.8-rt7, same test environment)
On Sun, Jul 06, 2008 at 11:41:28AM +0200, Wolfgang Grandegger wrote:
....
> Yep, in 2.6.25.8-rt7 trace_mark() is not used any more, maybe for that
> reason. Nevertheless, this "bug" kept the latency on my MPC5200 box low.
> I will try with the old tracer to understand what the real cause of the
> high latencies is (with CONFIG_RCU_TRACE disabled).
>
As you said, in 2.6.25.8-rt7 there're rcu_trace_boost_unboost_called() and
rcu_trace_boost_unboosted() are used instead of trace_mark. I still haven't had
time to take a closer look waht these routines exactly do. Seemed however to me,
that they possibly do the same thing. I played a little bit around with
the RCU_TRACE and RCU_BOOST Option. Also I tried to comment out the
rcu_trace_boost_unboost_called() and rcu_trace_boost_unboosted() calls. My Tests
last appr. 30 Minutes with non-rt Payload "while [ 1 ]; do ls /;done". None of
my test showed extraordinary results. My worstcase values stay at about 220 us
with all my test combinations. I will attache a config with RCU_BOOST and
RCU_TRACE turned off to this mail, which provide a worst case value at 223 us
after about 40 Minutes test run on my board.
Are you using the FEC all a pci network card?
cheers
Luotao Fu
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
[-- Attachment #1.2: kernelconfig.target --]
[-- Type: text/plain, Size: 33797 bytes --]
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.25.8-rt7-mig-trunk
# Tue Jul 8 16:59:34 2008
#
# CONFIG_PPC64 is not set
#
# Processor support
#
CONFIG_6xx=y
# CONFIG_PPC_85xx is not set
# CONFIG_PPC_8xx is not set
# CONFIG_40x is not set
# CONFIG_44x is not set
# CONFIG_E200 is not set
CONFIG_PPC_FPU=y
# CONFIG_ALTIVEC is not set
CONFIG_PPC_STD_MMU=y
CONFIG_PPC_STD_MMU_32=y
# CONFIG_PPC_MM_SLICES is not set
# CONFIG_SMP is not set
CONFIG_PPC32=y
CONFIG_WORD_SIZE=32
CONFIG_PPC_MERGE=y
CONFIG_MMU=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_HARDIRQS=y
# CONFIG_HAVE_SETUP_PER_CPU_AREA is not set
CONFIG_IRQ_PER_CPU=y
CONFIG_ARCH_HAS_ILOG2_U32=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
# CONFIG_ARCH_NO_VIRT_TO_BUS is not set
CONFIG_PPC=y
CONFIG_EARLY_PRINTK=y
CONFIG_GENERIC_NVRAM=y
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_PPC_OF=y
CONFIG_OF=y
# CONFIG_PPC_UDBG_16550 is not set
# CONFIG_GENERIC_TBSYNC is not set
CONFIG_AUDIT_ARCH=y
CONFIG_GENERIC_BUG=y
CONFIG_DEFAULT_UIMAGE=y
# CONFIG_PPC_DCR_NATIVE is not set
# CONFIG_PPC_DCR_MMIO is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set
CONFIG_IKCONFIG=y
# CONFIG_IKCONFIG_PROC is not set
CONFIG_LOG_BUF_SHIFT=14
# CONFIG_CGROUPS is not set
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
# CONFIG_RELAY is not set
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
# CONFIG_RADIX_TREE_CONCURRENT is not set
# CONFIG_EMBEDDED is not set
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_COMPAT_BRK=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
# CONFIG_MARKERS is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_KMOD is not set
CONFIG_BLOCK=y
# CONFIG_LBD is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_LSF is not set
# CONFIG_BLK_DEV_BSG is not set
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
# CONFIG_IOSCHED_DEADLINE is not set
# CONFIG_IOSCHED_CFQ is not set
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
CONFIG_DEFAULT_NOOP=y
CONFIG_DEFAULT_IOSCHED="noop"
# CONFIG_CLASSIC_RCU is not set
#
# Platform support
#
CONFIG_PPC_MULTIPLATFORM=y
# CONFIG_PPC_82xx is not set
# CONFIG_PPC_83xx is not set
# CONFIG_PPC_86xx is not set
CONFIG_CLASSIC32=y
# CONFIG_PPC_CHRP is not set
# CONFIG_PPC_MPC512x is not set
# CONFIG_PPC_MPC5121 is not set
# CONFIG_MPC5121_ADS is not set
CONFIG_PPC_MPC52xx=y
# CONFIG_PPC_MPC5200_SIMPLE is not set
# CONFIG_PPC_EFIKA is not set
# CONFIG_PPC_LITE5200 is not set
CONFIG_PPC_MIG=y
# CONFIG_PPC_MPC5200_BUGFIX is not set
# CONFIG_PPC_PMAC is not set
# CONFIG_PPC_CELL is not set
# CONFIG_PPC_CELL_NATIVE is not set
# CONFIG_PQ2ADS is not set
# CONFIG_EMBEDDED6xx is not set
# CONFIG_IPIC is not set
# CONFIG_MPIC is not set
# CONFIG_MPIC_WEIRD is not set
# CONFIG_PPC_I8259 is not set
# CONFIG_PPC_RTAS is not set
# CONFIG_MMIO_NVRAM is not set
# CONFIG_PPC_MPC106 is not set
# CONFIG_PPC_970_NAP is not set
# CONFIG_PPC_INDIRECT_IO is not set
# CONFIG_GENERIC_IOMAP is not set
# CONFIG_CPU_FREQ is not set
# CONFIG_TAU is not set
# CONFIG_FSL_ULI1575 is not set
CONFIG_PPC_BESTCOMM=y
CONFIG_PPC_BESTCOMM_ATA=y
CONFIG_PPC_BESTCOMM_FEC=y
CONFIG_PPC_BESTCOMM_GEN_BD=y
#
# Kernel options
#
# CONFIG_HIGHMEM is not set
CONFIG_TICK_ONESHOT=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
# CONFIG_SCHED_HRTICK is not set
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT_DESKTOP is not set
CONFIG_PREEMPT_RT=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_RCU=y
# CONFIG_PREEMPT_RCU_BOOST is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_ASM_SEMAPHORES=y
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_MISC is not set
# CONFIG_IOMMU_HELPER is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_HAS_WALK_MEMORY=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
# CONFIG_KEXEC is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
# CONFIG_SPARSEMEM_VMEMMAP_ENABLE is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_PROC_DEVICETREE=y
# CONFIG_CMDLINE_BOOL is not set
CONFIG_PM=y
# CONFIG_PM_LEGACY is not set
# CONFIG_PM_DEBUG is not set
CONFIG_SECCOMP=y
CONFIG_ISA_DMA_API=y
#
# Bus options
#
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
# CONFIG_PPC_INDIRECT_PCI is not set
CONFIG_FSL_SOC=y
CONFIG_PCI=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCI_SYSCALL=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set
#
# Advanced setup
#
CONFIG_ADVANCED_OPTIONS=y
CONFIG_HIGHMEM_START=0xfe000000
# CONFIG_LOWMEM_SIZE_BOOL is not set
CONFIG_LOWMEM_SIZE=0x30000000
# CONFIG_KERNEL_START_BOOL is not set
CONFIG_KERNEL_START=0xc0000000
CONFIG_TASK_SIZE_BOOL=y
CONFIG_TASK_SIZE=0x80000000
CONFIG_BOOT_LOAD=0x00800000
#
# Networking
#
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
# CONFIG_IP_PNP_BOOTP is not set
# CONFIG_IP_PNP_RARP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
# CONFIG_INET_LRO is not set
# CONFIG_INET_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
# CONFIG_INET6_XFRM_TUNNEL is not set
# CONFIG_INET6_TUNNEL is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
#
# Core Netfilter Configuration
#
# CONFIG_NETFILTER_NETLINK_QUEUE is not set
# CONFIG_NETFILTER_NETLINK_LOG is not set
# CONFIG_NF_CONNTRACK is not set
CONFIG_NETFILTER_XTABLES=m
# CONFIG_NETFILTER_XT_TARGET_CLASSIFY is not set
# CONFIG_NETFILTER_XT_TARGET_DSCP is not set
# CONFIG_NETFILTER_XT_TARGET_MARK is not set
# CONFIG_NETFILTER_XT_TARGET_NFQUEUE is not set
# CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
# CONFIG_NETFILTER_XT_TARGET_RATEEST is not set
# CONFIG_NETFILTER_XT_TARGET_TCPMSS is not set
# CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP is not set
# CONFIG_NETFILTER_XT_MATCH_COMMENT is not set
# CONFIG_NETFILTER_XT_MATCH_DCCP is not set
# CONFIG_NETFILTER_XT_MATCH_DSCP is not set
# CONFIG_NETFILTER_XT_MATCH_ESP is not set
# CONFIG_NETFILTER_XT_MATCH_IPRANGE is not set
# CONFIG_NETFILTER_XT_MATCH_LENGTH is not set
# CONFIG_NETFILTER_XT_MATCH_LIMIT is not set
# CONFIG_NETFILTER_XT_MATCH_MAC is not set
# CONFIG_NETFILTER_XT_MATCH_MARK is not set
# CONFIG_NETFILTER_XT_MATCH_OWNER is not set
# CONFIG_NETFILTER_XT_MATCH_MULTIPORT is not set
# CONFIG_NETFILTER_XT_MATCH_PKTTYPE is not set
# CONFIG_NETFILTER_XT_MATCH_QUOTA is not set
# CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
# CONFIG_NETFILTER_XT_MATCH_REALM is not set
# CONFIG_NETFILTER_XT_MATCH_SCTP is not set
# CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set
# CONFIG_NETFILTER_XT_MATCH_STRING is not set
# CONFIG_NETFILTER_XT_MATCH_TCPMSS is not set
# CONFIG_NETFILTER_XT_MATCH_TIME is not set
# CONFIG_NETFILTER_XT_MATCH_U32 is not set
# CONFIG_NETFILTER_XT_MATCH_HASHLIMIT is not set
#
# IP: Netfilter Configuration
#
# CONFIG_IP_NF_QUEUE is not set
CONFIG_IP_NF_IPTABLES=m
# CONFIG_IP_NF_MATCH_RECENT is not set
# CONFIG_IP_NF_MATCH_ECN is not set
# CONFIG_IP_NF_MATCH_AH is not set
# CONFIG_IP_NF_MATCH_TTL is not set
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
# CONFIG_IP_NF_TARGET_ULOG is not set
CONFIG_IP_NF_MANGLE=m
# CONFIG_IP_NF_TARGET_ECN is not set
# CONFIG_IP_NF_TARGET_TTL is not set
# CONFIG_IP_NF_RAW is not set
# CONFIG_IP_NF_ARPTABLES is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_SCHED is not set
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
CONFIG_CAN=y
CONFIG_CAN_RAW=m
CONFIG_CAN_BCM=m
#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=m
# CONFIG_CAN_DEBUG_DEVICES is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
#
# Wireless
#
# CONFIG_CFG80211 is not set
# CONFIG_WIRELESS_EXT is not set
# CONFIG_MAC80211 is not set
# CONFIG_IEEE80211 is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
# CONFIG_FW_LOADER is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_CONNECTOR is not set
CONFIG_MTD=y
# CONFIG_MTD_DEBUG is not set
# CONFIG_MTD_CONCAT is not set
CONFIG_MTD_PARTITIONS=y
# CONFIG_MTD_REDBOOT_PARTS is not set
CONFIG_MTD_CMDLINE_PARTS=y
# CONFIG_MTD_OF_PARTS is not set
#
# User Modules And Translation Layers
#
CONFIG_MTD_CHAR=y
CONFIG_MTD_BLKDEVS=y
CONFIG_MTD_BLOCK=y
# CONFIG_FTL is not set
# CONFIG_NFTL is not set
# CONFIG_INFTL is not set
# CONFIG_RFD_FTL is not set
# CONFIG_SSFDC is not set
# CONFIG_MTD_OOPS is not set
#
# RAM/ROM/Flash chip drivers
#
CONFIG_MTD_CFI=y
# CONFIG_MTD_JEDECPROBE is not set
CONFIG_MTD_GEN_PROBE=y
# CONFIG_MTD_CFI_ADV_OPTIONS is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
CONFIG_MTD_CFI_INTELEXT=y
# CONFIG_MTD_CFI_AMDSTD is not set
# CONFIG_MTD_CFI_STAA is not set
CONFIG_MTD_CFI_UTIL=y
# CONFIG_MTD_RAM is not set
# CONFIG_MTD_ROM is not set
# CONFIG_MTD_ABSENT is not set
#
# Mapping drivers for chip access
#
CONFIG_MTD_COMPLEX_MAPPINGS=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_START=0x0
CONFIG_MTD_PHYSMAP_LEN=0x0
CONFIG_MTD_PHYSMAP_BANKWIDTH=2
# CONFIG_MTD_PHYSMAP_OF is not set
# CONFIG_MTD_PCI is not set
# CONFIG_MTD_INTEL_VR_NOR is not set
# CONFIG_MTD_PLATRAM is not set
#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_DATAFLASH is not set
# CONFIG_MTD_M25P80 is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
# CONFIG_MTD_MTDRAM is not set
# CONFIG_MTD_BLOCK2MTD is not set
#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOC2000 is not set
# CONFIG_MTD_DOC2001 is not set
# CONFIG_MTD_DOC2001PLUS is not set
# CONFIG_MTD_NAND is not set
# CONFIG_MTD_ONENAND is not set
#
# UBI - Unsorted block images
#
# CONFIG_MTD_UBI is not set
CONFIG_OF_DEVICE=y
# CONFIG_PARPORT is not set
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_MISC_DEVICES=y
# CONFIG_PHANTOM is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_ENCLOSURE_SERVICES is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set
#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=m
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
# CONFIG_CHR_DEV_SG is not set
# CONFIG_CHR_DEV_SCH is not set
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set
# CONFIG_ATA is not set
# CONFIG_MD is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set
# CONFIG_ARCNET is not set
CONFIG_PHYLIB=y
#
# MII PHY device drivers
#
# CONFIG_MARVELL_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_ICPLUS_PHY is not set
# CONFIG_REALTEK_PHY is not set
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_ENC28J60 is not set
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_EEPRO100 is not set
CONFIG_E100=m
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
CONFIG_FEC_MPC52xx=y
CONFIG_FEC_MPC52xx_MDIO=y
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set
# CONFIG_E1000E_ENABLED is not set
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_GIANFAR is not set
# CONFIG_MV643XX_ETH is not set
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
CONFIG_NETDEV_10000=y
# CONFIG_CHELSIO_T1 is not set
# CONFIG_CHELSIO_T3 is not set
# CONFIG_IXGBE is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set
# CONFIG_MYRI10GE is not set
# CONFIG_NETXEN_NIC is not set
# CONFIG_NIU is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_TEHUTI is not set
# CONFIG_BNX2X is not set
# CONFIG_TR is not set
#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
# CONFIG_SERIO_I8042 is not set
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
#
# Non-8250 serial port support
#
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_MPC52xx=y
CONFIG_SERIAL_MPC52xx_CONSOLE=y
CONFIG_SERIAL_MPC52xx_CONSOLE_BAUD=115200
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
# CONFIG_GEN_RTC is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_TCG_TPM is not set
CONFIG_RMEM=m
CONFIG_ALLOC_RTSJ_MEM=m
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=y
#
# I2C Algorithms
#
# CONFIG_I2C_ALGOBIT is not set
# CONFIG_I2C_ALGOPCF is not set
# CONFIG_I2C_ALGOPCA is not set
#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_I810 is not set
# CONFIG_I2C_PIIX4 is not set
CONFIG_I2C_MPC=y
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_TINY_USB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
# CONFIG_I2C_VOODOO3 is not set
#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
CONFIG_SENSORS_EEPROM=m
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_PCF8575 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_TPS65010 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_ST24CXX is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
#
# SPI support
#
CONFIG_SPI=y
CONFIG_SPI_MASTER=y
#
# SPI Master Controller Drivers
#
CONFIG_SPI_BITBANG=m
CONFIG_SPI_MPC52xx_PSC=y
CONFIG_SPI_MPC5200=m
#
# SPI Protocol Masters
#
# CONFIG_SPI_AT25 is not set
# CONFIG_SPI_SPIDEV is not set
# CONFIG_SPI_TLE62X0 is not set
CONFIG_W1=m
#
# 1-wire Bus Masters
#
# CONFIG_W1_MASTER_MATROX is not set
# CONFIG_W1_MASTER_DS2490 is not set
# CONFIG_W1_MASTER_DS2482 is not set
#
# 1-wire Slaves
#
# CONFIG_W1_SLAVE_THERM is not set
# CONFIG_W1_SLAVE_SMEM is not set
CONFIG_W1_SLAVE_DS2433=m
# CONFIG_W1_SLAVE_DS2433_CRC is not set
# CONFIG_W1_SLAVE_DS2760 is not set
# CONFIG_POWER_SUPPLY is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
# CONFIG_WATCHDOG is not set
#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
#
# Multifunction device drivers
#
# CONFIG_MFD_SM501 is not set
#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
CONFIG_DAB=y
# CONFIG_USB_DABUSB is not set
#
# Graphics support
#
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
# CONFIG_FB is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_DUMMY_CONSOLE=y
#
# Sound
#
# CONFIG_SOUND is not set
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set
#
# USB Input Devices
#
CONFIG_USB_HID=m
# CONFIG_USB_HIDINPUT_POWERBOOK is not set
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y
#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set
#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_DEVICE_CLASS=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_PERSIST is not set
# CONFIG_USB_OTG is not set
#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_EHCI_FSL is not set
CONFIG_USB_EHCI_HCD_PPC_OF=y
# CONFIG_USB_ISP116X_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_HCD_PPC_SOC is not set
CONFIG_USB_OHCI_HCD_PPC_OF=y
CONFIG_USB_OHCI_HCD_PPC_OF_BE=y
# CONFIG_USB_OHCI_HCD_PPC_OF_LE is not set
CONFIG_USB_OHCI_HCD_PCI=y
CONFIG_USB_OHCI_BIG_ENDIAN_DESC=y
CONFIG_USB_OHCI_BIG_ENDIAN_MMIO=y
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#
#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_KARMA is not set
CONFIG_USB_LIBUSUAL=y
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_MON is not set
#
# USB port drivers
#
# CONFIG_USB_SERIAL is not set
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_GADGET is not set
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set
#
# MMC/SD Card Drivers
#
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_BOUNCE=y
# CONFIG_SDIO_UART is not set
#
# MMC/SD Host Controller Drivers
#
# CONFIG_MMC_SDHCI is not set
# CONFIG_MMC_WBSD is not set
# CONFIG_MMC_TIFM_SD is not set
CONFIG_MMC_SPI=m
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=m
CONFIG_RTC_CLASS=m
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_DS1307 is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
CONFIG_RTC_DRV_RS5C372=m
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_S35390A is not set
#
# SPI RTC drivers
#
# CONFIG_RTC_DRV_MAX6902 is not set
# CONFIG_RTC_DRV_R9701 is not set
# CONFIG_RTC_DRV_RS5C348 is not set
#
# Platform RTC drivers
#
# CONFIG_RTC_DRV_CMOS is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_V3020 is not set
#
# on-CPU RTC drivers
#
# CONFIG_DMADEVICES is not set
#
# Userspace I/O
#
# CONFIG_UIO is not set
#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
# CONFIG_EXT4DEV_FS is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
# CONFIG_MSDOS_FS is not set
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLB_PAGE is not set
# CONFIG_CONFIGFS_FS is not set
#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_JFFS2_FS=y
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_JFFS2_FS_WRITEBUFFER=y
# CONFIG_JFFS2_FS_WBUF_VERIFY is not set
# CONFIG_JFFS2_SUMMARY is not set
# CONFIG_JFFS2_FS_XATTR is not set
# CONFIG_JFFS2_COMPRESSION_OPTIONS is not set
CONFIG_JFFS2_ZLIB=y
# CONFIG_JFFS2_LZO is not set
CONFIG_JFFS2_RTIME=y
# CONFIG_JFFS2_RUBIN is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_DIRECTIO is not set
# CONFIG_NFSD is not set
CONFIG_ROOT_NFS=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
# CONFIG_SUNRPC_BIND34 is not set
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=y
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=y
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_UTF8 is not set
# CONFIG_DLM is not set
#
# Library routines
#
CONFIG_BITREVERSE=y
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC7=m
# CONFIG_LIBCRC32C is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
#
# Kernel hacking
#
# CONFIG_PRINTK_TIME is not set
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
# CONFIG_MAGIC_SYSRQ is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_KERNEL is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_EVENT_TRACER is not set
# CONFIG_WAKEUP_LATENCY_HIST is not set
# CONFIG_SAMPLES is not set
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_VIRQ_DEBUG is not set
# CONFIG_BOOTX_TEXT is not set
# CONFIG_PPC_EARLY_DEBUG is not set
#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
# CONFIG_CRYPTO is not set
CONFIG_PPC_CLOCK=y
CONFIG_PPC_LIB_RHEAP=y
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-08 15:08 ` Luotao Fu
@ 2008-07-08 19:43 ` Wolfgang Grandegger
2008-07-09 12:53 ` Luotao Fu
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-08 19:43 UTC (permalink / raw)
To: Wolfgang Grandegger, Steven Rostedt, Paul E. McKenney, Luotao Fu,
LKML
Hi Fun,
Luotao Fu wrote:
> Hi,
>
> I found some time and played with my mpc5200B Board again on this issue
> (2.6.25.8-rt7, same test environment)
>
> On Sun, Jul 06, 2008 at 11:41:28AM +0200, Wolfgang Grandegger wrote:
> ....
>> Yep, in 2.6.25.8-rt7 trace_mark() is not used any more, maybe for that
>> reason. Nevertheless, this "bug" kept the latency on my MPC5200 box low.
>> I will try with the old tracer to understand what the real cause of the
>> high latencies is (with CONFIG_RCU_TRACE disabled).
>>
> As you said, in 2.6.25.8-rt7 there're rcu_trace_boost_unboost_called() and
> rcu_trace_boost_unboosted() are used instead of trace_mark. I still haven't had
> time to take a closer look waht these routines exactly do. Seemed however to me,
> that they possibly do the same thing. I played a little bit around with
> the RCU_TRACE and RCU_BOOST Option. Also I tried to comment out the
> rcu_trace_boost_unboost_called() and rcu_trace_boost_unboosted() calls. My Tests
> last appr. 30 Minutes with non-rt Payload "while [ 1 ]; do ls /;done". None of
> my test showed extraordinary results. My worstcase values stay at about 220 us
> with all my test combinations. I will attache a config with RCU_BOOST and
> RCU_TRACE turned off to this mail, which provide a worst case value at 223 us
> after about 40 Minutes test run on my board.
I think I understood why CONFIG_RCU_TRACEin 2.6.24-rt helped to keep
latencies low. See:
ttp://marc.info/?l=linux-rt-users&m=121499677026236&w=4
But as Steven pointed out, the preempt_enable_noresched at that place
might be illegal, anyway. In 2.6.25-rt, trace_mark() is no longer used
and I do not see an improvement with CONFIG_RCU_TRAC any more. Latencies
go up to 600us. I also tried with CONFIG_RCU_BOOST disabled, but it did
not improve the latencies. Actually, I still measure significantly
different latencies with 2.6.24.4-rt4, 2.6.24.7-rt14 and 2.6.25.8-rt7,
which is quite frustrating. As you seem not be able to reproduce my high
latencies, I wonder if there are some toolchain or glibc related issues.
Using ftrace to localized the latency spot would be the next step. But
it's not yet available for PowerPC. Looking into that now.
> Are you using the FEC all a pci network card?
I use only the FEC. There is no PCI device.
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-08 19:43 ` Wolfgang Grandegger
@ 2008-07-09 12:53 ` Luotao Fu
2008-07-09 13:15 ` Wolfgang Grandegger
0 siblings, 1 reply; 49+ messages in thread
From: Luotao Fu @ 2008-07-09 12:53 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Steven Rostedt, Paul E. McKenney, Luotao Fu, LKML, RT,
Ingo Molnar, Thomas Gleixner
Hi Wolfgang,
Wolfgang Grandegger wrote:
> Hi Fun,
What's so funny? ;-)
[...]
> But as Steven pointed out, the preempt_enable_noresched at that place
> might be illegal, anyway. In 2.6.25-rt, trace_mark() is no longer
> used and I do not see an improvement with CONFIG_RCU_TRAC any more.
> Latencies go up to 600us. I also tried with CONFIG_RCU_BOOST
> disabled, but it did not improve the latencies. Actually, I still
> measure significantly different latencies with 2.6.24.4-rt4,
> 2.6.24.7-rt14 and 2.6.25.8-rt7, which is quite frustrating. As you
> seem not be able to reproduce my high latencies, I wonder if there
> are some toolchain or glibc related issues.
I took a quick look into the rcppreempt boost stuff in 2.6.24-rt and
2.6.25-rt. As you have mentioned, trace_mark is no more used in
2.6.24-rt. Instead of that flags are used to detect if the task is
preempted. It might indeed probably be some toolchain issue.
My powerpc toolchain I'm using here contains gcc 4.1.2 and glibc 2.5
Cheers
Fu (without n)
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-09 12:53 ` Luotao Fu
@ 2008-07-09 13:15 ` Wolfgang Grandegger
2008-07-09 14:52 ` Luotao Fu
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-09 13:15 UTC (permalink / raw)
To: Luotao Fu
Cc: Steven Rostedt, Paul E. McKenney, LKML, RT, Ingo Molnar,
Thomas Gleixner
Hi Fu (without n)
Luotao Fu wrote:
> Hi Wolfgang,
>
> Wolfgang Grandegger wrote:
>> Hi Fun,
>
> What's so funny? ;-)
Ah, oh, just a funny typo ;-).
> [...]
>
>> But as Steven pointed out, the preempt_enable_noresched at that place
>> might be illegal, anyway. In 2.6.25-rt, trace_mark() is no longer
>> used and I do not see an improvement with CONFIG_RCU_TRAC any more.
>> Latencies go up to 600us. I also tried with CONFIG_RCU_BOOST
>> disabled, but it did not improve the latencies. Actually, I still
>> measure significantly different latencies with 2.6.24.4-rt4,
>> 2.6.24.7-rt14 and 2.6.25.8-rt7, which is quite frustrating. As you
>> seem not be able to reproduce my high latencies, I wonder if there
>> are some toolchain or glibc related issues.
>
> I took a quick look into the rcppreempt boost stuff in 2.6.24-rt and
> 2.6.25-rt. As you have mentioned, trace_mark is no more used in
> 2.6.24-rt. Instead of that flags are used to detect if the task is
> preempted. It might indeed probably be some toolchain issue.
> My powerpc toolchain I'm using here contains gcc 4.1.2 and glibc 2.5
OK, in the past you have been able to reproduce the high latencies with
2.6.24-rt1 and CONFIG_RCU_TRACE disabled, IIRC. Did you use a different
toolchain at that time?
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-09 13:15 ` Wolfgang Grandegger
@ 2008-07-09 14:52 ` Luotao Fu
2008-07-10 7:50 ` Wolfgang Grandegger
0 siblings, 1 reply; 49+ messages in thread
From: Luotao Fu @ 2008-07-09 14:52 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Luotao Fu, Steven Rostedt, Paul E. McKenney, LKML, RT,
Ingo Molnar, Thomas Gleixner
[-- Attachment #1: Type: text/plain, Size: 767 bytes --]
Hi Wolfgang,
On Wed, Jul 09, 2008 at 03:15:01PM +0200, Wolfgang Grandegger wrote:
> Hi Fu (without n)
>
....
> OK, in the past you have been able to reproduce the high latencies with
> 2.6.24-rt1 and CONFIG_RCU_TRACE disabled, IIRC. Did you use a different
> toolchain at that time?
>
Nope. As mentioned above, trace_mark() does some "real" works (what ever it is.),
while the new mechahnismen use flags to remember the state of preemption. Maybe
something here got optimized away? I take for grant, that you use gcc in your
toolchain. Which version do you have?
cheers
Fu
--
Dipl.-Ing. Luotao Fu | Phone: +49-5121-206917-3
Pengutronix - Linux Solutions for Science and Industry
Entwicklungszentrum Nord http://www.pengutronix.de
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-09 14:52 ` Luotao Fu
@ 2008-07-10 7:50 ` Wolfgang Grandegger
2008-08-01 21:09 ` Paul E. McKenney
0 siblings, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-10 7:50 UTC (permalink / raw)
To: Wolfgang Grandegger, Luotao Fu, Steven Rostedt, Paul E. McKenney,
LKML
Luotao Fu wrote:
> Hi Wolfgang,
> On Wed, Jul 09, 2008 at 03:15:01PM +0200, Wolfgang Grandegger wrote:
>> Hi Fu (without n)
>>
> ....
>> OK, in the past you have been able to reproduce the high latencies with
>> 2.6.24-rt1 and CONFIG_RCU_TRACE disabled, IIRC. Did you use a different
>> toolchain at that time?
>>
>
> Nope. As mentioned above, trace_mark() does some "real" works (what ever it is.),
> while the new mechahnismen use flags to remember the state of preemption. Maybe
I don't known what you refer to, but in __rcu_preempt_unboost() of 2.6.25.8-rt7,
the trace code simply increments a counter:
static void rcu_trace_boost_##type(struct rcu_boost_dat *rbd) \
{ \
rbd->rbs_stat_##type++; \
}
and that's the reason why latency is not affected by switching CONFIG_RCU_TRACE
on (while trace_mark uses preempt_disable/preempt_enable around).
> something here got optimized away? I take for grant, that you use gcc in your
> toolchain. Which version do you have?
The ELDK v4.2 uses:
ppc_6xx-gcc (GCC) 4.2.2
and
GLIBC v2.6
But I measured the same latencies with ELDK v4.1:
ppc_6xx-gcc (GCC) 4.0.0 (DENX ELDK 4.1 4.0.0)
GLIBC v2.3.5
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-10 7:50 ` Wolfgang Grandegger
@ 2008-08-01 21:09 ` Paul E. McKenney
0 siblings, 0 replies; 49+ messages in thread
From: Paul E. McKenney @ 2008-08-01 21:09 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
On Thu, Jul 10, 2008 at 09:50:47AM +0200, Wolfgang Grandegger wrote:
> Luotao Fu wrote:
>> Hi Wolfgang,
>> On Wed, Jul 09, 2008 at 03:15:01PM +0200, Wolfgang Grandegger wrote:
>>> Hi Fu (without n)
>>>
>> ....
>>> OK, in the past you have been able to reproduce the high latencies with
>>> 2.6.24-rt1 and CONFIG_RCU_TRACE disabled, IIRC. Did you use a different
>>> toolchain at that time?
>>>
>> Nope. As mentioned above, trace_mark() does some "real" works (what ever
>> it is.),
>> while the new mechahnismen use flags to remember the state of preemption.
>> Maybe
>
> I don't known what you refer to, but in __rcu_preempt_unboost() of
> 2.6.25.8-rt7, the trace code simply increments a counter:
>
> static void rcu_trace_boost_##type(struct rcu_boost_dat *rbd) \
> { \
> rbd->rbs_stat_##type++; \
> }
>
> and that's the reason why latency is not affected by switching
> CONFIG_RCU_TRACE
> on (while trace_mark uses preempt_disable/preempt_enable around).
This changed -- preempt_disable()/preempt_enable() pair
was added for rcu_trace_boost_boost_called_preempt() and
rcu_trace_boost_unboost_called_preempt() later to suppress a warning
(and also make that statistic accurate in face of preemption).
Thanx, Paul
>> something here got optimized away? I take for grant, that you use gcc in
>> your
>> toolchain. Which version do you have?
>
> The ELDK v4.2 uses:
>
> ppc_6xx-gcc (GCC) 4.2.2
>
> and
> GLIBC v2.6
>
> But I measured the same latencies with ELDK v4.1:
>
> ppc_6xx-gcc (GCC) 4.0.0 (DENX ELDK 4.1 4.0.0)
>
> GLIBC v2.3.5
>
> Wolfgang.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users"
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-02 11:03 ` Wolfgang Grandegger
2008-07-06 0:42 ` Steven Rostedt
@ 2008-08-01 21:09 ` Paul E. McKenney
2008-08-05 15:40 ` Wolfgang Grandegger
1 sibling, 1 reply; 49+ messages in thread
From: Paul E. McKenney @ 2008-08-01 21:09 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Steven Rostedt, Luotao Fu, LKML, RT, Ingo Molnar, Thomas Gleixner
On Wed, Jul 02, 2008 at 01:03:27PM +0200, Wolfgang Grandegger wrote:
> Wolfgang Grandegger wrote:
>> Steven Rostedt wrote:
>> There seems to be something in trace_mark() keeping latency low:
>> http://lxr.linux.no/linux+v2.6.24.4/include/linux/marker.h#L52
>> I will follow your suggestions.
>
> I removed all "likely" and "unlikely" macros, but the latencies did not
> improve. Then I added
>
> preempt_disable();
> preempt_enable();
>
> at the two locations mentioned above, like trace_mark() does, and disabled
> CONFIG_RCU_TRACE. That helped to keep the latencies low, and it did for
> 2.6.25.8-rt7 as well. As I see it, adding preemption points seem to prevent
> high latencies.
Interesting. Perhaps a preempt_enable_no_resched() somewhere that
should be preempt_enable()?
Thanx, Paul
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-08-01 21:09 ` Paul E. McKenney
@ 2008-08-05 15:40 ` Wolfgang Grandegger
0 siblings, 0 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-08-05 15:40 UTC (permalink / raw)
To: paulmck; +Cc: Steven Rostedt, Luotao Fu, LKML, RT, Ingo Molnar, Thomas Gleixner
Paul E. McKenney wrote:
> On Wed, Jul 02, 2008 at 01:03:27PM +0200, Wolfgang Grandegger wrote:
>> Wolfgang Grandegger wrote:
>>> Steven Rostedt wrote:
>>> There seems to be something in trace_mark() keeping latency low:
>>> http://lxr.linux.no/linux+v2.6.24.4/include/linux/marker.h#L52
>>> I will follow your suggestions.
>> I removed all "likely" and "unlikely" macros, but the latencies did not
>> improve. Then I added
>>
>> preempt_disable();
>> preempt_enable();
>>
>> at the two locations mentioned above, like trace_mark() does, and disabled
>> CONFIG_RCU_TRACE. That helped to keep the latencies low, and it did for
>> 2.6.25.8-rt7 as well. As I see it, adding preemption points seem to prevent
>> high latencies.
>
> Interesting. Perhaps a preempt_enable_no_resched() somewhere that
> should be preempt_enable()?
FYI, Thomas fixed the real problem a week ago and the patch went into
2.6.24.7-rt16. See his mail with the subject "2.6.24.7-rt16".
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-01 15:11 ` Steven Rostedt
2008-07-01 16:11 ` Wolfgang Grandegger
@ 2008-07-02 8:09 ` Wolfgang Grandegger
2008-07-06 0:39 ` Steven Rostedt
1 sibling, 1 reply; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-02 8:09 UTC (permalink / raw)
To: Steven Rostedt
Cc: Paul E. McKenney, Luotao Fu, LKML, RT, Ingo Molnar,
Thomas Gleixner
Steven Rostedt wrote:
> On Tue, 1 Jul 2008, Wolfgang Grandegger wrote:
>> I continue this thread because it's still not understood why enabling
>> CONFIG_RCU_TRACE is necessary to get reasonable latencies on the
>> MPC5200. It might also explain, why I get much worse latencies with
>> 2.6.25.8-rt7.
>
> Have you tried turning on ftrace?
Hm, is it available for PowerPC? I have not found yet how to enable it.
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-02 8:09 ` Wolfgang Grandegger
@ 2008-07-06 0:39 ` Steven Rostedt
2008-07-06 9:34 ` Wolfgang Grandegger
0 siblings, 1 reply; 49+ messages in thread
From: Steven Rostedt @ 2008-07-06 0:39 UTC (permalink / raw)
To: Wolfgang Grandegger
Cc: Paul E. McKenney, Luotao Fu, LKML, RT, Ingo Molnar,
Thomas Gleixner
On Wed, 2 Jul 2008, Wolfgang Grandegger wrote:
>
> Steven Rostedt wrote:
> > On Tue, 1 Jul 2008, Wolfgang Grandegger wrote:
> >> I continue this thread because it's still not understood why enabling
> >> CONFIG_RCU_TRACE is necessary to get reasonable latencies on the
> >> MPC5200. It might also explain, why I get much worse latencies with
> >> 2.6.25.8-rt7.
> >
> > Have you tried turning on ftrace?
>
> Hm, is it available for PowerPC? I have not found yet how to enable it.
Hmm, should be. But I need to test it on my PPC box first before I can
tell you for sure. If you don't see a way to enable it, perhaps I haven't
updated the config options for PPC yet.
-- Steve
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-07-06 0:39 ` Steven Rostedt
@ 2008-07-06 9:34 ` Wolfgang Grandegger
0 siblings, 0 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-07-06 9:34 UTC (permalink / raw)
To: Steven Rostedt
Cc: Paul E. McKenney, Luotao Fu, LKML, RT, Ingo Molnar,
Thomas Gleixner
Steven Rostedt wrote:
> On Wed, 2 Jul 2008, Wolfgang Grandegger wrote:
>
>> Steven Rostedt wrote:
>>> On Tue, 1 Jul 2008, Wolfgang Grandegger wrote:
>>>> I continue this thread because it's still not understood why enabling
>>>> CONFIG_RCU_TRACE is necessary to get reasonable latencies on the
>>>> MPC5200. It might also explain, why I get much worse latencies with
>>>> 2.6.25.8-rt7.
>>> Have you tried turning on ftrace?
>> Hm, is it available for PowerPC? I have not found yet how to enable it.
>
> Hmm, should be. But I need to test it on my PPC box first before I can
> tell you for sure. If you don't see a way to enable it, perhaps I haven't
> updated the config options for PPC yet.
I added HAVE_FTRACE and HAVE_DYNAMIC_FTRACE to "config PPC" but more
seems to be missing, e.g. arch/powerpc/kernel/ftrace.c. Nevertheless,
how would I trigger high latency events with ftrace. I there a
description somewhere?
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: 2.6.24-rc8-rt1: Strange latencies on mpc5200 powerpc - RCU issue?
2008-01-30 10:57 ` Paul E. McKenney
2008-01-30 11:15 ` Luotao Fu
@ 2008-01-30 11:22 ` Wolfgang Grandegger
1 sibling, 0 replies; 49+ messages in thread
From: Wolfgang Grandegger @ 2008-01-30 11:22 UTC (permalink / raw)
To: paulmck; +Cc: Luotao Fu, Steven Rostedt, LKML, RT, Ingo Molnar, Thomas Gleixner
Paul E. McKenney wrote:
> On Wed, Jan 30, 2008 at 11:45:01AM +0100, Wolfgang Grandegger wrote:
>> Paul E. McKenney wrote:
>>> On Wed, Jan 30, 2008 at 09:18:49AM +0100, Wolfgang Grandegger wrote:
>>>> Paul E. McKenney wrote:
>>>>> On Tue, Jan 29, 2008 at 02:38:04PM +0100, Wolfgang Grandegger wrote:
>>>>>> Luotao Fu wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Wolfgang Grandegger wrote:
>>>>>>> ..........
>>>>>>>> Do you still get high latencies with:
>>>>>>>>
>>>>>>>> CONFIG_PREEMPT_RCU_BOOST=y
>>>>>>>> CONFIG_RCU_TRACE=y
>>>>>>>> CONFIG_NO_HZ is not set
>>>>>>>>
>>>>>>>> With this setting I have not yet realized latencies > 150us. Could you
>>>>>>>> please give it a try? If I change one of the parameters above, latency
>>>>>>>> increases in short time.
>>>>>>> I played through some combination of the RCU options and can back your
>>>>>>> observation this time: With the rcu Tracer or the priority boost turned
>>>>>>> off I also could measure reliably extraordinory high latencies. If they
>>>>>>> are both turned on, no high latencies could be measured. Turning on the
>>>>>>> dynamic ticker however doesn't seem to cause high latencies during my
>>>>>>> test runs. Seemed like an rcu issue here.
>>>>>> I'm just making a long test run on my TQM5200 module with my good
>>>>>> settings. After more than 4.5 hours under load, cyclictest shows a
>>>>>> maximum latency of 177 us. I'm going to re-check the effect of CONFIG_NO_HZ.
>>>>>>
>>>>>>> Further such results only appear if the target board is booted with
>>>>>>> nfsrootfs. (As I already have mentioned several times before), which
>>>>>>> leads my suspection to rcu usage in nfs implementation. In this case
>>>>>>> this problem might even be platformindependent. I'd have to do some
>>>>>>> tests on one of our arm boards later to test this. Since there're no
>>>>>>> reports like this for other architecture as powerpc till now, I doubt
>>>>>>> quite if this is verifiable.
>>>>>> It's also my suspicion that the high latencies are related to the RCU
>>>>>> usage in the network layer, where it's heavily used. What is really
>>>>>> wired is that switching off CONFIG_RCU_TRACE has a negative impact on
>>>>>> the latency. As I see it, it just adds some trace points, but I might
>>>>>> have missed something.
>>>>> I would expect that CONFIG_RCU_TRACE=n (as in "no" rather than "module")
>>>>> would have low latencies rather than high latencies. So I am quite
>>>>> surprised by your result. I will dig into this more.
>>>> Thanks a lot. To be clear. I need "CONFIG_PREEMPT_RCU_BOOST=y" *and*
>>>> "CONFIG_RCU_TRACE=y" to achieve reasonable latencies below 180us. With
>>>> CONFIG_PREEMPT_RCU_BOOST or CONFIG_RCU_TRACE not set or
>>>> CONFIG_RCU_TRACE=m is rmeasure latencies up to 600us within a minute or so.
>>> OK, thank you for the confirmation.
>>>
>>> The large latencies were from cyclictest, correct? Did other tests
>>> also show these latencies? In either case, could you please send the
>>> exact command line you used to run the test?
>> Yes, I used "$ cyclictest -n -t1 -p80 -i1000" to measure the latency. So
>> far, I have not done other tests. Any recommendation?
>> As no-rt load I used "while ls; do ls /bin; done" in one telnet window
>> and "while ./hackbench 10; do ./calibrator 400 32M cali; sleep 30; done"
>> in another. But already "while ls; do ls /bin; done" is enough to
>> trigger the high latencies quickly. Note also, that I work on a root
>> files-ystem mounted via NFS resulting in a lot of network traffic and
>> utilization.
>
> I have to ask...
>
> Did you see large latencies when -not- running on NFS?
I have not made test without NFS, but Fu has and he said:
"Further such results only appear if the target board is booted with
nfsrootfs."
Wolfgang.
^ permalink raw reply [flat|nested] 49+ messages in thread