From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: From: henning.richter@domain.hid Date: Tue, 5 May 2009 15:12:35 +0200 MIME-Version: 1.0 Content-type: multipart/alternative; Boundary="0__=4EBBFF3EDFDB89BD8f9e8a93df938690918c4EBBFF3EDFDB89BD" Content-Disposition: inline Subject: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: xenomai@xenomai.org --0__=4EBBFF3EDFDB89BD8f9e8a93df938690918c4EBBFF3EDFDB89BD Content-type: text/plain; charset=US-ASCII Hi, running a xenomai task in which I am sending a single Frame (size ~500 byte) from my EtherCAT Master to some Beckhoff slaves has latencies around 200 us. While running that task, top shows that gatekeeper/0 (with RT priority) has 30% CPU usage. What is the task of this gatekeeper and can this cause my high latencies? Henning --0__=4EBBFF3EDFDB89BD8f9e8a93df938690918c4EBBFF3EDFDB89BD Content-type: text/html; charset=US-ASCII Content-Disposition: inline

Hi,

running a xenomai task in which I am sending a single Frame (size ~500 byte) from my EtherCAT Master to some Beckhoff slaves has latencies around 200 us.
While running that task, top shows that gatekeeper/0 (with RT priority) has 30% CPU usage.
What is the task of this gatekeeper and can this cause my high latencies?

Henning --0__=4EBBFF3EDFDB89BD8f9e8a93df938690918c4EBBFF3EDFDB89BD-- From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A0040D9.60907@domain.hid> Date: Tue, 05 May 2009 15:36:25 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > Hi, > > running a xenomai task in which I am sending a single Frame (size ~500 > byte) from my EtherCAT Master to some Beckhoff slaves has latencies around > 200 us. > While running that task, top shows that gatekeeper/0 (with RT priority) has > 30% CPU usage. > What is the task of this gatekeeper and can this cause my high latencies? The gatekeeper is used when a taks is migrating from secondary mode to primary mode. The gatekeeper itself can not cause the high latencies, but the fact that a task runs in secondary mode can. To check if this is what happens, check the MSW column in /proc/xenomai/stat for the thread which has latencies issues. If MSW is increasing then you have a problem. To find where your thread switches to secondary mode, use the technique demonstrated in examples/native/sigxcpu.c. -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A029F81.1010404@domain.hid> Date: Thu, 07 May 2009 10:44:49 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > Running latency is fine. And switchtest ? -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 In-Reply-To: <4A029F81.1010404@domain.hid> Message-ID: From: henning.richter@domain.hid Date: Thu, 7 May 2009 10:54:52 +0200 MIME-Version: 1.0 Content-type: multipart/alternative; Boundary="0__=4EBBFF3CDFA3DD7C8f9e8a93df938690918c4EBBFF3CDFA3DD7C" Content-Disposition: inline Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Gilles Chanteperdrix Cc: xenomai@xenomai.org --0__=4EBBFF3CDFA3DD7C8f9e8a93df938690918c4EBBFF3CDFA3DD7C Content-type: text/plain; charset=US-ASCII My machine only has one CPU. So switchtest doesn't work. Henning henning.richter@domain.hid wrote: > Running latency is fine. And switchtest ? -- Gilles. --0__=4EBBFF3CDFA3DD7C8f9e8a93df938690918c4EBBFF3CDFA3DD7C Content-type: text/html; charset=US-ASCII Content-Disposition: inline

My machine only has one CPU.
So switchtest doesn't work.

Henning

henning.richter@domain.hid wrote: > Running latency is fine. And switchtest ? -- Gilles.
--0__=4EBBFF3CDFA3DD7C8f9e8a93df938690918c4EBBFF3CDFA3DD7C-- From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A02A232.9070300@domain.hid> Date: Thu, 07 May 2009 10:56:18 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > My machine only has one CPU. > So switchtest doesn't work. switchtest works on machine with only one CPU. What is the exact error message you get ? -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A02A38C.4040203@domain.hid> Date: Thu, 07 May 2009 11:02:04 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > Outputs of the switchtest: > > == FPU check routines: OK!. > switchtest: Unabale to open switchtest device. > (modprobe xeno_switchtest ? ) > > modeprobe xeno_switchtest leads to FATAL: Module xeno_switchtest not found. I do not see anything in this message related to the number of CPUs in your system. You should simply enable the switchtest module in Xenomai configuration (in the kernel configuration menus). -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A02A514.7000009@domain.hid> Date: Thu, 07 May 2009 11:08:36 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > In which submenu is this option hidden? It is not hidden. Besides xconfig has a "search" functionality. -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A02AEC3.70401@domain.hid> Date: Thu, 07 May 2009 11:49:55 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > Beside I actually don't use xconfig because QT3 is not installed > my application runs well in console mode and I am able to gather > information from /proc/xenomai/stat etc. In a previous mail you told us that you were running your application in graphical mode. If you are running in console mode, you should see the oops (though it may not be easy to capture, and you may need to actually increase the console resolution through kernel command line to see the message entirely, some people in the past send us numerical photos of the screen to show us their oops...). As for not being able to use switchtest, sorry, but if you do not want to use the graphical tool, you have to learn how to search in text mode, this means find . -name 'Kconfig*' | xargs grep switchtest... -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A02D381.8020506@domain.hid> Date: Thu, 07 May 2009 14:26:41 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > Things are strange. Now my application works without crashing. I haven't > changed anything. > Btw. what should be a "normal" value of /proc/xenomai/latency ? > I've got 1676 /proc/xenomai/latency is the anticipation of the Xenomai timer. It is actually a value for you to tweak with a default value that does not cause any pathological behaviour. > > Back to my main problem: > Sending / Receiving Frames is still too slow. At a framesize of around 700 > byte it takes around > 350 us before the next frame is send. And this at a given cycle time of 250 > us at a priority of 90. > All printf's and other logs are removed. No Modeswitches are done during > run of my application. > Btw. what counts CSW at /proc/xenomai/stat ? CSW is context switches. But IMO, first things should be done first, and you should investigate the crashes. And it would be still be interesting to know if switchtest works on your platform. As for the latency, as I think I already told you, you should run the latency test at the same frequency as the one of your application with which you see problems. If you see problems with the latency test, then Xenomai has a problem on your platform, otherwise, the problem is in your application. > > > ethercat@domain.hid:~$ cat /proc/xenomai/sched > > CPU PID PRI PERIOD TIMEOUT TIMEBASE > STAT NAME > 0 0 -1 0 0 master R > ROOT/0 > 0 0 98 0 0 master W > rtnet-stack > 0 0 0 0 0 master > W rtnet-rtpc > 0 5259 0 0 0 > master X master_test > 0 5260 90 249752 1109638 master > w myRtTask > > > > ethercat@domain.hid:~$ cat /proc/xenomai/stat > CPU PID MSW CSW PF STAT %CPU NAME > 0 0 0 436897 0 00500080 74.9 > ROOT/0 > 0 0 0 435422 0 00000082 1.0 > rtnet-stack > 0 0 0 1 0 00000082 0.0 > rtnet-rtpc > 0 5259 1 1 0 00300380 0.0 > master_test > 0 5260 24 59775 0 00300186 0.1 > myRtTask > 0 0 0 870784 0 00000000 3.6 > IRQ6: rt_eepro100 > 0 0 0 527744 0 00000000 2.6 > IRQ233: [timer] > > I think I will start tracing with the ipipe tracer. The I-pipe tracer will not help you understand what happens in user-space. For this, you are probably better of measuring things with rt_timer_tsc() or clock_gettime(). -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A02E251.6030909@domain.hid> Date: Thu, 07 May 2009 15:29:53 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: Xenomai help henning.richter@domain.hid wrote: > So I will configure the kernel once more. This time with the switchtest > option. > But what I am still wondering about is that the kernel oops is not > regularly when I am running > my application. Sometimes there is no problem running the application with > prio 99 a few times (e.g. 10 times) > and gathering information from /xenomai/stat and/or /sched etc. (also in > graphical mode) > (I also run my application in console mode and nothing went wrong.) You mean that you get crashes only in graphic mode ? -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A044932.1010005@domain.hid> Date: Fri, 08 May 2009 17:01:06 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: Xenomai help henning.richter@domain.hid wrote: > So here are some more results. > The switchtest is now working with the enabled kernel option. > My adeos Ipipe version is 2.2-06. > I run both, latency and switchtest, in graphical mode. > I send you my .config in a seperate mail, as there where problem the last > time. Ok. The working switchtest is a good news. The latency results are a bit high, if they are taken on an idle system (only latencies taken on a busy system are meaningful), do not you get a message in kernel console talking about SMIs ? In any case, we do not see the same results as your application, so, the cause of the latencies observed by your application is not Xenomai latencies. and: > Here the .config as plain text. Nope, your mailer still sends your mail with twice its body: once in plain text, once in HTML. This is pure bandwith waste for the few hundred people subscribed to the list, and this makes people think that your mail is spam, because spam is usually HTML too. Please switch your mailer to plain text instead. As for the config itself you should disable CONFIG_X86_PM_TIMER. and also: > Things are still strange. > The 'freeze' still appears sporadically, but only in graphical mode. > When I login via console I don't get any freeze. Ok. As I already told you, if when you see freezes the keyboard leds blink, it means that what you have is a BUG, an oops, or a kernel panic. In any case, the kernel console will tell us what happens. Try to get the kernel console through a serial cable, or if not possible using the less reliable netconsole. -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A081829.5090901@domain.hid> Date: Mon, 11 May 2009 14:20:57 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: Xenomai help henning.richter@domain.hid wrote: > So here is what I got via serial console when my application causes that > freeze: Ok. Is it the first bug? If yes, could you send us a disassembly of the profile_pc function? (run objdump vmlinux, then look for profile_pc, and only copy this function in the mail). -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A0949E7.2050309@domain.hid> Date: Tue, 12 May 2009 12:05:27 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: Xenomai help henning.richter@domain.hid wrote: > > It really is the first bug. > > [ 2510.572237] BUG: unable to handle kernel paging request at b807a7fc > [ 2510.576044] IP: [] profile_pc+0x46/0x50 > [ 2510.576044] Oops: 0000 [#1] SMP > [ 2510.576044] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat e100 > rt_eepro100 rtpacket rtnet af_packet i915 drm bridge stp bnep rfcomm sco > l2cap bluetooth ppdev ipv6 iptable_filter ip_tables x_tables parport_pc lp > parport sr_mod cdrom serio_raw evdev psmouse com20020_pci com20020 arcnet > iTCO_wdt iTCO_vendor_support shpchp intel_agp pci_hotplug agpgart ext3 jbd > mbcache sd_mod crc_t10dif sg usb_storage libusual ata_piix ata_generic > libata mii uhci_hcd ehci_hcd scsi_mod dock usbcore fuse [last unloaded: > e100] > > > here the corresponding disassembly: > > c0107080 : > c0107080: 55 push %ebp > c0107081: 89 e5 mov %esp,%ebp > c0107083: 83 ec 08 sub $0x8,%esp > c0107086: 89 1c 24 mov %ebx,(%esp) > c0107089: 89 74 24 04 mov %esi,0x4(%esp) Ok. So profile_pc reserves some room on the stack for mcount arguments... > c010708d: e8 da 24 01 00 call c011956c > c0107092: f6 40 36 02 testb $0x2,0x36(%eax) > c0107096: 8b 70 2c mov 0x2c(%eax),%esi > c0107099: 89 c3 mov %eax,%ebx > c010709b: 75 0d jne c01070aa > c010709d: 8b 40 30 mov 0x30(%eax),%eax > c01070a0: 25 fc 00 00 00 and $0xfc,%eax > c01070a5: 83 f8 60 cmp $0x60,%eax > c01070a8: 74 0e je c01070b8 > c01070aa: 89 f0 mov %esi,%eax > c01070ac: 8b 1c 24 mov (%esp),%ebx > c01070af: 8b 74 24 04 mov 0x4(%esp),%esi And then tries to access the same room on the stack believing that the frame pointer or pc is stored there. Game over. Modify profile_pc function declaration to add the "notrace" qualifier. This bug is a red herring, this is a simple effect of enabling the tracing. However, it would be nice if you could recompile the kernel without the following configuration options: CONFIG_PROFILING CONFIG_MARKERS CONFIG_OPROFILE CONFIG_KPROBES CONFIG_KRETPROBES -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A09645D.4090003@domain.hid> Date: Tue, 12 May 2009 13:58:21 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: Xenomai help henning.richter@domain.hid wrote: >> And then tries to access the same room on the stack believing that the >> frame pointer or pc is stored there. Game over. >> >> Modify profile_pc function declaration to add the "notrace" qualifier. >> >> This bug is a red herring, this is a simple effect of enabling the > tracing. > > -- > Gilles. > > Can you give me a hint how to do that or is there sth. to read about? > > Henning Try: diff --git a/arch/x86/include/asm/ptrace.h b/arch/x86/include/asm/ptrace.h index 6d34d95..abfd422 100644 --- a/arch/x86/include/asm/ptrace.h +++ b/arch/x86/include/asm/ptrace.h @@ -134,7 +134,7 @@ struct pt_regs { struct cpuinfo_x86; struct task_struct; -extern unsigned long profile_pc(struct pt_regs *regs); +extern unsigned long notrace profile_pc(struct pt_regs *regs); extern unsigned long convert_ip_to_linear(struct task_struct *child, struct pt_regs *regs); If that is not enough, add notrace where profile_pc is implemented too. -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A096496.2080809@domain.hid> Date: Tue, 12 May 2009 13:59:18 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: Xenomai help henning.richter@domain.hid wrote: >> And then tries to access the same room on the stack believing that the >> frame pointer or pc is stored there. Game over. >> >> Modify profile_pc function declaration to add the "notrace" qualifier. >> >> This bug is a red herring, this is a simple effect of enabling the > tracing. > > -- > Gilles. > > Can you give me a hint how to do that or is there sth. to read about? But you can also try to disable all the tracing and profiling options I gave you in two different mails and that slow down your system anyway. -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A096821.9090609@domain.hid> Date: Tue, 12 May 2009 14:14:25 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: <4A0949E7.2050309@domain.hid> In-Reply-To: <4A0949E7.2050309@domain.hid> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: Xenomai help Gilles Chanteperdrix wrote: > henning.richter@domain.hid wrote: >> It really is the first bug. >> >> [ 2510.572237] BUG: unable to handle kernel paging request at b807a7fc >> [ 2510.576044] IP: [] profile_pc+0x46/0x50 >> [ 2510.576044] Oops: 0000 [#1] SMP >> [ 2510.576044] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat e100 >> rt_eepro100 rtpacket rtnet af_packet i915 drm bridge stp bnep rfcomm sco >> l2cap bluetooth ppdev ipv6 iptable_filter ip_tables x_tables parport_pc lp >> parport sr_mod cdrom serio_raw evdev psmouse com20020_pci com20020 arcnet >> iTCO_wdt iTCO_vendor_support shpchp intel_agp pci_hotplug agpgart ext3 jbd >> mbcache sd_mod crc_t10dif sg usb_storage libusual ata_piix ata_generic >> libata mii uhci_hcd ehci_hcd scsi_mod dock usbcore fuse [last unloaded: >> e100] >> >> >> here the corresponding disassembly: >> >> c0107080 : >> c0107080: 55 push %ebp >> c0107081: 89 e5 mov %esp,%ebp >> c0107083: 83 ec 08 sub $0x8,%esp >> c0107086: 89 1c 24 mov %ebx,(%esp) >> c0107089: 89 74 24 04 mov %esi,0x4(%esp) > > Ok. So profile_pc reserves some room on the stack for mcount arguments... > >> c010708d: e8 da 24 01 00 call c011956c >> c0107092: f6 40 36 02 testb $0x2,0x36(%eax) >> c0107096: 8b 70 2c mov 0x2c(%eax),%esi >> c0107099: 89 c3 mov %eax,%ebx >> c010709b: 75 0d jne c01070aa >> c010709d: 8b 40 30 mov 0x30(%eax),%eax >> c01070a0: 25 fc 00 00 00 and $0xfc,%eax >> c01070a5: 83 f8 60 cmp $0x60,%eax >> c01070a8: 74 0e je c01070b8 >> c01070aa: 89 f0 mov %esi,%eax >> c01070ac: 8b 1c 24 mov (%esp),%ebx >> c01070af: 8b 74 24 04 mov 0x4(%esp),%esi > > And then tries to access the same room on the stack believing that the > frame pointer or pc is stored there. Game over. This is bullsh*t, I tried to give an answer too fast, so please try to disable the options I gave you. -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A0ABB32.8080706@domain.hid> Date: Wed, 13 May 2009 14:21:06 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Gilles Chanteperdrix Cc: Xenomai help Somehow, the xenomai-help mailing list does not receive your mails. So, I am resending your mail to the list. Hi Gilles, Hi list, nearly everything is working fine. I rebuild the kernel again without the given options and it seems to work. Even the latencies look quite much better. There is only one problem left that bothers me. Every time I load the rt_eepro100 for my real-time device I get the message: [ 95.753152] *** Receiver lock-up bug detected *** [ 95.753154] Your device may not work reliably! and sometimes the systems hangs (and can not be woken up) at this point. I know that this is a hint for a irq problem but I don't know where this should be. My eth0 device (irq6) is not sharing this irq with an other deveice. IRQs after boot: CPU0 0: 40 XT-PIC-XT timer 1: 52 XT-PIC-XT i8042 2: 0 XT-PIC-XT cascade 3: 2 XT-PIC-XT 4: 280 XT-PIC-XT serial 5: 6179 XT-PIC-XT uhci_hcd:usb2, eth3, i915@domain.hid 6: 44 XT-PIC-XT eth0 7: 1293 XT-PIC-XT ehci_hcd:usb1 8: 2 XT-PIC-XT rtc0 9: 11 XT-PIC-XT acpi 10: 2 XT-PIC-XT 11: 2 XT-PIC-XT 12: 951 XT-PIC-XT i8042 14: 7734 XT-PIC-XT ata_piix 15: 40 XT-PIC-XT uhci_hcd:usb3, ata_piix, eth1, eth2 IRQs after I added the real-time device: CPU0 0: 40 XT-PIC-XT timer 1: 131 XT-PIC-XT i8042 2: 0 XT-PIC-XT cascade 3: 2 XT-PIC-XT 4: 280 XT-PIC-XT serial 5: 21773 XT-PIC-XT uhci_hcd:usb2, i915@domain.hid, eth3 6: 71 XT-PIC-XT 7: 2334 XT-PIC-XT ehci_hcd:usb1 8: 2 XT-PIC-XT rtc0 9: 17 XT-PIC-XT acpi 10: 2 XT-PIC-XT 11: 2 XT-PIC-XT 12: 3771 XT-PIC-XT i8042 14: 11960 XT-PIC-XT ata_piix 15: 66 XT-PIC-XT uhci_hcd:usb3, ata_piix, eth1, eth2 cat /proc/xenomai/irq IRQ CPU0 6: 1 rt_eepro100 230: 0 [IPI] 233: 170615 [timer] 234: 0 [critical sync] 258: 2 [virtual] From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A0C1B0D.1050104@domain.hid> Date: Thu, 14 May 2009 15:22:21 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: Xenomai help henning.richter@domain.hid wrote: > Hi Gilles, Hi list, > > nearly everything is working fine. I rebuild the kernel again without the > given options and it seems to work. > Even the latencies look quite much better. There is only one problem left > that bothers me. > > Every time I load the rt_eepro100 for my real-time device I get the > message: > > [ 95.753152] *** Receiver lock-up bug detected *** > [ 95.753154] Your device may not work reliably! > > and sometimes the systems hangs (and can not be woken up) at this point. > > I know that this is a hint for a irq problem but I don't know where this > should be. > My eth0 device (irq6) is not sharing this irq with an other deveice. > > IRQs after boot: Hi, when your question is new, please post a new message instead of replying to an old thread. The person likely to answer your question probably did not even read your message because it is buried so deep in a thread which I handled until now. Regards. -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 In-Reply-To: <4A0040D9.60907@domain.hid> Message-ID: From: henning.richter@domain.hid Date: Tue, 5 May 2009 15:44:16 +0200 MIME-Version: 1.0 Content-type: multipart/alternative; Boundary="0__=4EBBFF3EDFD883358f9e8a93df938690918c4EBBFF3EDFD88335" Content-Disposition: inline Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Gilles Chanteperdrix Cc: xenomai@xenomai.org --0__=4EBBFF3EDFD883358f9e8a93df938690918c4EBBFF3EDFD88335 Content-type: text/plain; charset=US-ASCII henning.richter@domain.hid wrote: >> Hi, >> >> running a xenomai task in which I am sending a single Frame (size ~500 >> byte) from my EtherCAT Master to some Beckhoff slaves has latencies around >> 200 us. >> While running that task, top shows that gatekeeper/0 (with RT priority) has >> 30% CPU usage. >> What is the task of this gatekeeper and can this cause my high latencies? > > The gatekeeper is used when a taks is migrating from secondary mode to > primary mode. The gatekeeper itself can not cause the high latencies, > but the fact that a task runs in secondary mode can. > > To check if this is what happens, check the MSW column in > /proc/xenomai/stat for the thread which has latencies issues. If MSW is > increasing then you have a problem. > To find where your thread switches to secondary mode, use the technique > demonstrated in examples/native/sigxcpu.c. -- Gilles. So thank your for defining my problem. Now I am sure that there my thread is switching to secondary mode as the MSW column increases rapidly while running my application. Can I also use the ipipe tracer or LTTng to find the place where it switches the modes as I don't have the xenomai examples installed. Henning --0__=4EBBFF3EDFD883358f9e8a93df938690918c4EBBFF3EDFD88335 Content-type: text/html; charset=US-ASCII Content-Disposition: inline

henning.richter@domain.hid wrote: >> Hi, >> >> running a xenomai task in which I am sending a single Frame (size ~500 >> byte) from my EtherCAT Master to some Beckhoff slaves has latencies around >> 200 us. >> While running that task, top shows that gatekeeper/0 (with RT priority) has >> 30% CPU usage. >> What is the task of this gatekeeper and can this cause my high latencies? > > The gatekeeper is used when a taks is migrating from secondary mode to > primary mode. The gatekeeper itself can not cause the high latencies, > but the fact that a task runs in secondary mode can. > > To check if this is what happens, check the MSW column in > /proc/xenomai/stat for the thread which has latencies issues. If MSW is > increasing then you have a problem. > To find where your thread switches to secondary mode, use the technique > demonstrated in examples/native/sigxcpu.c. -- Gilles.
So thank your for defining my problem.
Now I am sure that there my thread is switching to secondary mode
as the MSW column increases rapidly while running my application.
Can I also use the ipipe tracer or LTTng to find the place where it switches the modes as I don't have the
xenomai examples installed.

Henning --0__=4EBBFF3EDFD883358f9e8a93df938690918c4EBBFF3EDFD88335-- From mboxrd@z Thu Jan 1 00:00:00 1970 In-Reply-To: <4A004385.7020200@domain.hid> Message-ID: From: henning.richter@domain.hid Date: Wed, 6 May 2009 09:21:30 +0200 MIME-Version: 1.0 Content-type: multipart/alternative; Boundary="0__=4EBBFF3DDFB665418f9e8a93df938690918c4EBBFF3DDFB66541" Content-Disposition: inline Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Gilles Chanteperdrix Cc: xenomai@xenomai.org --0__=4EBBFF3DDFB665418f9e8a93df938690918c4EBBFF3DDFB66541 Content-type: text/plain; charset=US-ASCII Hi Gilles, I tried to run, after a simple make, sigxcpu. The result: ./sigxcpu[0x80488337] [0xb800c400] /lib/tls/i686/cmov/libphtread.so.0[0xb7fcc50f] /lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb7f417ee] This is where my system freezes or simply reboots. Am I doing sth wrong? --0__=4EBBFF3DDFB665418f9e8a93df938690918c4EBBFF3DDFB66541 Content-type: text/html; charset=US-ASCII Content-Disposition: inline

Hi Gilles,

I tried to run, after a simple make, sigxcpu.
The result:
./sigxcpu[0x80488337]
[0xb800c400]
/lib/tls/i686/cmov/libphtread.so.0[0xb7fcc50f]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb7f417ee]

This is where my system freezes or simply reboots.

Am I doing sth wrong? --0__=4EBBFF3DDFB665418f9e8a93df938690918c4EBBFF3DDFB66541-- From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A013B94.5050000@domain.hid> Date: Wed, 06 May 2009 09:26:12 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > Hi Gilles, > > I tried to run, after a simple make, sigxcpu. > The result: > ./sigxcpu[0x80488337] > [0xb800c400] > /lib/tls/i686/cmov/libphtread.so.0[0xb7fcc50f] > /lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb7f417ee] > > This is where my system freezes or simply reboots. > > Am I doing sth wrong? You are probably setting the warning bit too early and/or for the wrong thread. You should set the bit only for the thread which is expected to run in primary mode, and only after it has done its non primary mode duties (like creating new threads). However, you should not get freezes or reboots. Any trace on the console ? -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A015D0A.1030404@domain.hid> Date: Wed, 06 May 2009 11:48:58 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > Hi Gilles, > > I tried to run, after a simple make, sigxcpu. > The result: > ./sigxcpu[0x80488337] > [0xb800c400] > /lib/tls/i686/cmov/libphtread.so.0[0xb7fcc50f] > /lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb7f417ee] > > This is where my system freezes or simply reboots. > > Am I doing sth wrong? What version of Xenomai are you using ? -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A004385.7020200@domain.hid> Date: Tue, 05 May 2009 15:47:49 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > > > henning.richter@domain.hid wrote: >>> Hi, >>> >>> running a xenomai task in which I am sending a single Frame (size ~500 >>> byte) from my EtherCAT Master to some Beckhoff slaves has latencies > around >>> 200 us. >>> While running that task, top shows that gatekeeper/0 (with RT priority) > has >>> 30% CPU usage. >>> What is the task of this gatekeeper and can this cause my high > latencies? >> The gatekeeper is used when a taks is migrating from secondary mode to >> primary mode. The gatekeeper itself can not cause the high latencies, >> but the fact that a task runs in secondary mode can. >> >> To check if this is what happens, check the MSW column in >> /proc/xenomai/stat for the thread which has latencies issues. If MSW is >> increasing then you have a problem. > >> To find where your thread switches to secondary mode, use the technique >> demonstrated in examples/native/sigxcpu.c. > > -- > Gilles. > > So thank your for defining my problem. > Now I am sure that there my thread is switching to secondary mode > as the MSW column increases rapidly while running my application. > Can I also use the ipipe tracer or LTTng to find the place where it > switches the modes as I don't have the > xenomai examples installed. Yes, you can use the I-pipe tracer, but reading the output might be a bit complicated. Examples are part of Xenomai sources freely available on Xenomai web site. -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A028D87.1000808@domain.hid> Date: Thu, 07 May 2009 09:28:07 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > Now I'm using xenomai 2.4.7 on kernel 2.6.27.19. > > The sigxcpu example from xenomai is now working correctly. > > After removing some outputs there are no more modeswitches in my task, > apart from some debugging output while initilization. > > But now my system is no longer stable. It freezes. Show us /proc/interrupts, /proc/xenomai/irq, your .config if it changed. What kind of freeze is this? Do you run in graphical or in text mode? If in graphical mode, try and get the console output using serial console or netconsole. Does the system restart when you hit the keyboard? Does the system freeze when using the latency test alone? Same question with switchtest? -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 In-Reply-To: <4A028D87.1000808@domain.hid> Message-ID: From: henning.richter@domain.hid Date: Thu, 7 May 2009 09:57:07 +0200 MIME-Version: 1.0 Content-type: multipart/alternative; Boundary="0__=4EBBFF3CDFBA3BF58f9e8a93df938690918c4EBBFF3CDFBA3BF5" Content-Disposition: inline Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Gilles Chanteperdrix Cc: xenomai@xenomai.org --0__=4EBBFF3CDFBA3BF58f9e8a93df938690918c4EBBFF3CDFBA3BF5 Content-type: text/plain; charset=US-ASCII I'm running in graphical mode (Kubuntu 8.10 with Kde4). Now it freezes while 'cat /proc/xenomai/sched' The Scroll/Lock -LED and the Caps-Lock-Led were flashing. cat /proc/xenomai/irq (while running my rt task) IRQ CPU0 6: 19144 rt_eepro_100 230: 0 [IPI] 233: 68798 [timer] 234: 0 [critical sync] 258: 83 [virtual] cat /proc/interrupts (while running my rt task) CPU0 0: 40 XT-PIC-XT timer 1: 411 XT-PIC-XT i8042 2: 0 XT-PIC-XT cascade 3: 2 XT-PIC-XT 4: 2 XT-PIC-XT 5: 54863 XT-PIC-XT uhci_hcd:usb2, i915@domain.hid, eth3 6: 140 XT-PIC-XT 7: 6380 XT-PIC-XT ehci_hcd:usb1 8: 2 XT-PIC-XT rtc0 9: 20 XT-PIC-XT acpi 10: 2 XT-PIC-XT 11: 2 XT-PIC-XT 12: 4403 XT-PIC-XT i8042 14: 9713 XT-PIC-XT ata_piix 15: 27142 XT-PIC-XT uhci_hcd:usb3, ata_piix, eth1, eth2 NMI: 0 Non-maskable interrupts LOC: 61661 Local timer interrupts RES: 0 Rescheduling interrupts CAL: 0 function call interrupts TLB: 0 TLB shootdowns SPU 0 Spurious interrupts ERR: 0 MIS: 0 A few seconds after i gathered this information the system freezes again. The .config is the same as yesterday with one change -> ACPI is now enabled and the ACPI_PROCESSOR disabled. I now try to gather more information via netconsole. Henning henning.richter@domain.hid wrote: > Now I'm using xenomai 2.4.7 on kernel 2.6.27.19. > > The sigxcpu example from xenomai is now working correctly. > > After removing some outputs there are no more modeswitches in my task, > apart from some debugging output while initilization. > > But now my system is no longer stable. It freezes. Show us /proc/interrupts, /proc/xenomai/irq, your .config if it changed. What kind of freeze is this? Do you run in graphical or in text mode? If in graphical mode, try and get the console output using serial console or netconsole. Does the system restart when you hit the keyboard? Does the system freeze when using the latency test alone? Same question with switchtest? -- Gilles. --0__=4EBBFF3CDFBA3BF58f9e8a93df938690918c4EBBFF3CDFBA3BF5 Content-type: text/html; charset=US-ASCII Content-Disposition: inline

I'm running in graphical mode (Kubuntu 8.10 with Kde4).
Now it freezes while 'cat /proc/xenomai/sched'
The Scroll/Lock -LED and the Caps-Lock-Led were flashing.

cat /proc/xenomai/irq (while running my rt task)
IRQ CPU0
6: 19144 rt_eepro_100
230: 0 [IPI]
233: 68798 [timer]
234: 0 [critical sync]
258: 83 [virtual]

cat /proc/interrupts (while running my rt task)
CPU0
0: 40 XT-PIC-XT timer
1: 411 XT-PIC-XT i8042
2: 0 XT-PIC-XT cascade
3: 2 XT-PIC-XT
4: 2 XT-PIC-XT
5: 54863 XT-PIC-XT uhci_hcd:usb2, i915@domain.hid, eth3
6: 140 XT-PIC-XT
7: 6380 XT-PIC-XT ehci_hcd:usb1
8: 2 XT-PIC-XT rtc0
9: 20 XT-PIC-XT acpi
10: 2 XT-PIC-XT
11: 2 XT-PIC-XT
12: 4403 XT-PIC-XT i8042
14: 9713 XT-PIC-XT ata_piix
15: 27142 XT-PIC-XT uhci_hcd:usb3, ata_piix, eth1, eth2
NMI: 0 Non-maskable interrupts
LOC: 61661 Local timer interrupts
RES: 0 Rescheduling interrupts
CAL: 0 function call interrupts
TLB: 0 TLB shootdowns
SPU 0 Spurious interrupts
ERR: 0
MIS: 0

A few seconds after i gathered this information the system freezes again.
The .config is the same as yesterday with one change -> ACPI is now enabled and the
ACPI_PROCESSOR disabled.
I now try to gather more information via netconsole.

Henning

henning.richter@domain.hid wrote: > Now I'm using xenomai 2.4.7 on kernel 2.6.27.19. > > The sigxcpu example from xenomai is now working correctly. > > After removing some outputs there are no more modeswitches in my task, > apart from some debugging output while initilization. > > But now my system is no longer stable. It freezes. Show us /proc/interrupts, /proc/xenomai/irq, your .config if it changed. What kind of freeze is this? Do you run in graphical or in text mode? If in graphical mode, try and get the console output using serial console or netconsole. Does the system restart when you hit the keyboard? Does the system freeze when using the latency test alone? Same question with switchtest? -- Gilles.
--0__=4EBBFF3CDFBA3BF58f9e8a93df938690918c4EBBFF3CDFBA3BF5-- From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4A0296B2.5050906@domain.hid> Date: Thu, 07 May 2009 10:07:14 +0200 From: Gilles Chanteperdrix MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: henning.richter@domain.hid Cc: xenomai@xenomai.org henning.richter@domain.hid wrote: > I'm running in graphical mode (Kubuntu 8.10 with Kde4). > Now it freezes while 'cat /proc/xenomai/sched' > The Scroll/Lock -LED and the Caps-Lock-Led were flashing. Ok. It is a kernel oops then. Do you get it when simply running the latency or switchtest test instead of your application ? -- Gilles. From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4D35F6C8.5020306@domain.hid> Date: Tue, 18 Jan 2011 15:23:36 -0500 From: Waldemar Valdas Bancewicz MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Subject: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: xenomai@xenomai.org Hi, I am getting an abnormally high cpu utilization (30 - 85%) for the gatekeeper/0 thread. I'm using Xenomai 2.4.10 with a Debian linux kernel 2.6.32. The problem appears after upgrading the kernel with newer debian patches. Any ideas? Thanks. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Philippe Gerum In-Reply-To: <4D35F6C8.5020306@domain.hid> References: <4D35F6C8.5020306@domain.hid> Content-Type: text/plain; charset="UTF-8" Date: Wed, 19 Jan 2011 12:31:11 +0100 Message-ID: <1295436671.1857.129.camel@domain.hid> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] gatekeeper/0 List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Waldemar Valdas Bancewicz Cc: xenomai@xenomai.org On Tue, 2011-01-18 at 15:23 -0500, Waldemar Valdas Bancewicz wrote: > Hi, > > I am getting an abnormally high cpu utilization (30 - 85%) for the > gatekeeper/0 thread. I'm using Xenomai 2.4.10 with a Debian linux kernel > 2.6.32. The problem appears after upgrading the kernel with newer debian > patches. Any ideas? That is an interesting question, but I'm unsure that you will get a definite answer, at least because of the following reasons: - we have absolutely no clue about what those "newer debian patches" are, we track the mainline kernel only. - we don't know what your application does, what were the CPU figures before you changed your configuration, etc. - you are running 2.4.10, which is seriously outdated, and you did not tell anything about the I-pipe release you are running, albeit this is likely crucial for your issue. Actually we don't even know what architecture you are running. Wild guess: - your RT code is probably doing a massive number of mode switches to secondary mode. You may want to check whether the value of the CSW counter from /proc/xenomai/stat increases at the same pace depending on the version you are testing. - those patches somehow broke the host clock tick accounting code of the I-pipe, and the gatekeeper is unduly charged with all Linux timer ticks which happened while your system was running in primary mode, as soon as it returns to secondary mode. In such a case, the longer the time spent in primary mode, the higher the likeliness to have deferred Linux ticks charged this way. > > Thanks. > > _______________________________________________ > Xenomai-help mailing list > Xenomai-help@domain.hid > https://mail.gna.org/listinfo/xenomai-help -- Philippe.