From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from omx.cbeyond.com (omx.cbeyond.com [50.20.30.10]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 6C41EE014A9 for ; Fri, 8 Mar 2013 14:37:09 -0800 (PST) X-SBRS: 5.3 X-HAT: Sender Group CBEYOND_SYSTEMS, Policy $CBEYOND_RELAY applied. X-Hostname: omx02bay.sys.cbeyond.net X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AuIAADVnOlEyFB4LmWdsb2JhbABDhBUjwBiBcw4BAQEBAQgLCwcUIQeCLgUtXgEqViYBBBuICwGbAaEKjluDF2EDqk+CJw X-IronPort-AV: E=Sophos;i="4.84,810,1355115600"; d="scan'208,217";a="13237445" Received: from mts.cbeyond.com (HELO EXHUB101.mmeprod.cbeyond) ([50.20.30.11]) by omx.cbeyond.com with ESMTP; 08 Mar 2013 17:37:07 -0500 Received: from EXHUB104.mmeprod.cbeyond (10.128.38.103) by EXHUB101.mmeprod.cbeyond (10.128.38.100) with Microsoft SMTP Server (TLS) id 14.2.298.4; Fri, 8 Mar 2013 17:36:37 -0500 Received: from EXMBX202A.mmeprod.cbeyond ([fe80::b0ca:4d9b:ba7c:8227]) by EXHUB104.mmeprod.cbeyond ([::1]) with mapi id 14.02.0298.004; Fri, 8 Mar 2013 17:36:37 -0500 From: David Mulder To: "yocto@yoctoproject.org" Thread-Topic: Can I disable RT throttling? Thread-Index: Ac4cTWBMCcKVjca/TdSAdJObHV69ZQ== Date: Fri, 8 Mar 2013 22:36:36 +0000 Message-ID: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [69.198.178.141] MIME-Version: 1.0 Subject: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Mar 2013 22:37:09 -0000 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_19F0379280417C4A8FE3BD23AA7298E501347DE7EXMBX202Ammepro_" --_000_19F0379280417C4A8FE3BD23AA7298E501347DE7EXMBX202Ammepro_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi. I'm running a 10us control loop by (under vxWorks) setting one thread's pri= ority to max and not yielding ever (letting other tasks run on other cores)= , but Linux seems to thwart that capability: Ubuntu swaps out my thread occ= asionally for hundreds of microseconds; Yocto prints a "[sched_delayed] sch= ed: RT throttling activated" message as soon as I start my thread, so it se= ems likely that it will swap out my thread periodically (I'm not quite able= to confirm that yet), and even the fastest thread swap that I've heard of = is too slow. I tried changing the kernel's Preemption Model to "No Forced Preemption (Se= rver)", but that didn't remove the RT throttling message. I browsed the re= st of menuconfig but nothing looked related. I read an article from 2008 t= hat talked about the kernel reserving 5% of the CPU for non-SCHED_FIFO task= s, so that's the direction and terminology I looked for in menuconfig. So maybe disabling RT throttling is all I need to do, maybe some other appr= oach is needed. Hopefully someone will know more than I do. Thanks! Dave Mulder --_000_19F0379280417C4A8FE3BD23AA7298E501347DE7EXMBX202Ammepro_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi.

 

I’m running a 10us control loop by (under vxWo= rks) setting one thread’s priority to max and not yielding ever (lett= ing other tasks run on other cores), but Linux seems to thwart that capabil= ity: Ubuntu swaps out my thread occasionally for hundreds of microseconds; Yocto prints a “[sched_delayed] sched: RT = throttling activated” message as soon as I start my thread, so it see= ms likely that it will swap out my thread periodically (I’m not quite= able to confirm that yet), and even the fastest thread swap that I’ve heard of is too slow.

 

I tried changing the kernel’s Preemption Model= to “No Forced Preemption (Server)”, but that didn’t remo= ve the RT throttling message.  I browsed the rest of menuconfig but no= thing looked related.  I read an article from 2008 that talked about the kernel reserving 5% of the CPU for non-SCHED_FIFO tasks, so that= ’s the direction and terminology I looked for in menuconfig.

 

So maybe disabling RT throttling is all I need to do= , maybe some other approach is needed.  Hopefully someone will know mo= re than I do.

 

Thanks!

Dave Mulder

 

--_000_19F0379280417C4A8FE3BD23AA7298E501347DE7EXMBX202Ammepro_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.windriver.com (mail.windriver.com [147.11.1.11]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id B9C75E014CF for ; Fri, 8 Mar 2013 19:28:41 -0800 (PST) Received: from ALA-HCA.corp.ad.wrs.com (ala-hca.corp.ad.wrs.com [147.11.189.40]) by mail.windriver.com (8.14.5/8.14.3) with ESMTP id r293Sc4O012734 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Fri, 8 Mar 2013 19:28:38 -0800 (PST) Received: from bruce-ashfields-macbook.local (128.224.21.250) by ALA-HCA.corp.ad.wrs.com (147.11.189.50) with Microsoft SMTP Server id 14.2.342.3; Fri, 8 Mar 2013 19:28:38 -0800 Message-ID: <513AAC65.1050002@windriver.com> Date: Fri, 8 Mar 2013 22:28:37 -0500 From: Bruce Ashfield User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: David Mulder References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> In-Reply-To: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> Cc: "yocto@yoctoproject.org" Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 03:28:42 -0000 Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 8bit On 13-03-08 5:36 PM, David Mulder wrote: > Hi. > > I’m running a 10us control loop by (under vxWorks) setting one thread’s > priority to max and not yielding ever (letting other tasks run on other > cores), but Linux seems to thwart that capability: Ubuntu swaps out my > thread occasionally for hundreds of microseconds; Yocto prints a > “[sched_delayed] sched: RT throttling activated” message as soon as I > start my thread, so it seems likely that it will swap out my thread > periodically (I’m not quite able to confirm that yet), and even the > fastest thread swap that I’ve heard of is too slow. You are looking for kernel functionality that doesn't exist yet. In any SMP system there are sources of cross cpu interference that can't be removed, if you do, the global state machines of the kernel will break and the system will eventually come to a halt. The system is trying to save you from yourself, by throttling the RT task from taking the entire system down. There is work in the mainline kernel and -rt communities around cpu and cpu isolation (some of which we'll try and make available via the yocto meta-virtualization or meta-realtime layer, when they are ready), it goes by names such as task_nohz or adaptive_nohz. It's a complex problem to solve (interrupts, rcu, lapic), and there's not likely to be anything available in the short term. That's the mainline/scalable point of view, there are plenty of "custom" and one off solutions to the problem, such as hotplugging the cpu out of the system and running in an AMP configuration where a bare metal, or RTOS can monopolize a CPU since the global state machines don't interfere. These have their advantages and disadvantages, but if you are coming from an existing RTOS application, they might be the closest options to get you the performance you are looking for. Alternatively, there's the preempt-rt kernel (that we make available in linux-yocto, if that suits) that if your control loop is interrupt driven, you'll be closer to your needs since the kernel is far more preemptible and deterministic and might be able to meet your deadlines. There are experimental patches like sched_deadline (EDF) that might also be applicable, depending on your application architecture. Searching for "making Linux hard realtime", you'll find plenty of talks and research over the years for yet more custom ways to get things done. So that's my summary, as you can see, it's a long standing, evolving and complex story .. and not something that likely has an 'out of the box' solution at the moment. Cheers, Bruce > > I 0tried changing the kernel’s Preemption Model to “No Forced Preemption > (Server)”, but that didn’t remove the RT throttling message. I browsed > the rest of menuconfig but nothing looked related. I read an article > from 2008 that talked about the kernel reserving 5% of the CPU for > non-SCHED_FIFO tasks, so that’s the direction and terminology I looked > for in menuconfig. > > So maybe disabling RT throttling is all I need to do, maybe some other > approach is needed. Hopefully someone will know more than I do. > > Thanks! > > Dave Mulder > > > > _______________________________________________ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ve0-f179.google.com (mail-ve0-f179.google.com [209.85.128.179]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 560EFE014E1 for ; Sat, 9 Mar 2013 09:12:55 -0800 (PST) Received: by mail-ve0-f179.google.com with SMTP id da11so1870551veb.24 for ; Sat, 09 Mar 2013 09:12:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=2MELJXvud7/Pb1bCLyS42oU/N+BW2i9sdRozPKyZG3w=; b=N8Zut8LmMZPS/RQ7rG/0E0t0cfD1xJ9napFsja0/bGEfcpjO5qlm6MGuuRmi7KpQWA 7cETbYyQMVPhwDOopJu/1gk5262E4svK9UgKofFV0tu383V7nTiPpgCg0RZR1ANX7wcc fCeE3zKP/rBt0nBvSXrojSEnpAx0QgbsLcV/prF3r5rC39+hLL45yiODuvd9z8s5b6VI Y8Cj699WGSQ1mt7//2gfC73wcxYVn0UbG4XXQtMFnkjEt/BwIwl5tz+Rq84ZUU3GOciL wGYXZ1+QKgmrjfP2nWtZ7VZhoHqAzXuHKjuKfxjCS83kR5UUlZp91/yUMY88IIhKYk96 jk/g== MIME-Version: 1.0 X-Received: by 10.220.151.76 with SMTP id b12mr2569234vcw.16.1362849174406; Sat, 09 Mar 2013 09:12:54 -0800 (PST) Sender: ethersoft@gmail.com Received: by 10.58.196.236 with HTTP; Sat, 9 Mar 2013 09:12:54 -0800 (PST) In-Reply-To: <513AAC65.1050002@windriver.com> References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> <513AAC65.1050002@windriver.com> Date: Sat, 9 Mar 2013 12:12:54 -0500 X-Google-Sender-Auth: AQCawxEmyEJSabds6vJs7OsX1t0 Message-ID: From: Vin Shelton To: Bruce Ashfield Cc: "yocto@yoctoproject.org" , David Mulder Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 17:12:55 -0000 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On Fri, Mar 8, 2013 at 10:28 PM, Bruce Ashfield wrote: > On 13-03-08 5:36 PM, David Mulder wrote: >> I=92m running a 10us control loop by (under vxWorks) setting one thread= =92s >> priority to max and not yielding ever (letting other tasks run on other >> cores), but Linux seems to thwart that capability: Ubuntu swaps out my >> thread occasionally for hundreds of microseconds; Yocto prints a >> =93[sched_delayed] sched: RT throttling activated=94 message as soon as = I >> start my thread, so it seems likely that it will swap out my thread >> periodically (I=92m not quite able to confirm that yet), and even the >> fastest thread swap that I=92ve heard of is too slow. > > > You are looking for kernel functionality that doesn't exist yet. In > any SMP system there are sources of cross cpu interference that can't > be removed, if you do, the global state machines of the kernel will > break and the system will eventually come to a halt. > > The system is trying to save you from yourself, by throttling the > RT task from taking the entire system down. > > There is work in the mainline kernel and -rt communities around cpu > and cpu isolation (some of which we'll try and make available via > the yocto meta-virtualization or meta-realtime layer, when they are > ready), it goes by names such as task_nohz or adaptive_nohz. It's > a complex problem to solve (interrupts, rcu, lapic), and there's not > likely to be anything available in the short term. At ELC, Steve Rostedt said that this feature is currently targeted at the June mainline kernel release, but there's a lot of work still to be done, so that's not a firm date. Regards, Vin Shelton From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f43.google.com (mail-la0-f43.google.com [209.85.215.43]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 280FDE014DB for ; Sat, 9 Mar 2013 19:09:40 -0800 (PST) Received: by mail-la0-f43.google.com with SMTP id ek20so2877866lab.30 for ; Sat, 09 Mar 2013 19:09:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=lWWMIysZDIZJrnnRQ4ShIVvxvYGcPyy+4BmRhz4QaEk=; b=FQkYCX9haejFPxc7rAyvnmnoiacJjchz5L27QzMr5tQYhltyb8oPEkJEuhB2cormc8 AV584IkJsTECViFf5AL3fvexhay2hFs29IMH6D8VKosUxHqfDaO9oBBZRAgz5n0vhnxn zNDVUpMpw0tuQVefboyknaQDClKiYcjZIeXtV38nU74tYZMWAEcXi5lO8moThXS66CZo GEgHscVowmGlpdXfSX1rNY4uGWi69g2w0iG+VKc8lotQxESvb1RH9yJwFHS/ZgYct4+5 xndJcEpbe6F+SQLW+b+2LCDyrS2uUh9BXev8qjkGrwndAAblMDLRK9cy24tf7E447zzs KtyQ== MIME-Version: 1.0 X-Received: by 10.112.50.169 with SMTP id d9mr2874016lbo.57.1362884979424; Sat, 09 Mar 2013 19:09:39 -0800 (PST) Received: by 10.112.61.3 with HTTP; Sat, 9 Mar 2013 19:09:39 -0800 (PST) In-Reply-To: References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> <513AAC65.1050002@windriver.com> Date: Sat, 9 Mar 2013 22:09:39 -0500 Message-ID: From: Bruce Ashfield To: Vin Shelton Cc: "yocto@yoctoproject.org" , David Mulder Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2013 03:09:41 -0000 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On Sat, Mar 9, 2013 at 12:12 PM, Vin Shelton wro= te: > On Fri, Mar 8, 2013 at 10:28 PM, Bruce Ashfield > wrote: >> On 13-03-08 5:36 PM, David Mulder wrote: >>> I=92m running a 10us control loop by (under vxWorks) setting one thread= =92s >>> priority to max and not yielding ever (letting other tasks run on other >>> cores), but Linux seems to thwart that capability: Ubuntu swaps out my >>> thread occasionally for hundreds of microseconds; Yocto prints a >>> =93[sched_delayed] sched: RT throttling activated=94 message as soon as= I >>> start my thread, so it seems likely that it will swap out my thread >>> periodically (I=92m not quite able to confirm that yet), and even the >>> fastest thread swap that I=92ve heard of is too slow. >> >> >> You are looking for kernel functionality that doesn't exist yet. In >> any SMP system there are sources of cross cpu interference that can't >> be removed, if you do, the global state machines of the kernel will >> break and the system will eventually come to a halt. >> >> The system is trying to save you from yourself, by throttling the >> RT task from taking the entire system down. >> >> There is work in the mainline kernel and -rt communities around cpu >> and cpu isolation (some of which we'll try and make available via >> the yocto meta-virtualization or meta-realtime layer, when they are >> ready), it goes by names such as task_nohz or adaptive_nohz. It's >> a complex problem to solve (interrupts, rcu, lapic), and there's not >> likely to be anything available in the short term. > > At ELC, Steve Rostedt said that this feature is currently targeted at > the June mainline kernel release, but there's a lot of work still to > be done, so that's not a firm date. Absolutely. There's no such thing as a firm date for anything, I (among others) are following the development closely, testing, and helping whereve= r possible. Cheers, Bruce > > Regards, > Vin Shelton > _______________________________________________ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto --=20 "Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end" From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vb0-f49.google.com (mail-vb0-f49.google.com [209.85.212.49]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 28D0AE014F5 for ; Sun, 10 Mar 2013 04:43:11 -0700 (PDT) Received: by mail-vb0-f49.google.com with SMTP id s24so1220094vbi.8 for ; Sun, 10 Mar 2013 04:43:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=l8+IKmLc5rKL8XGshzBnaG4PSdu+9W4lsICcCQjYXs0=; b=g1dSYzw3mxtzzLP5sCepAUkvb5X/jqnV/8zHytj0tt7uuk/Bzij61xvYS4GmkDLVwG 6HO9kWIaqnk7Ck6Qx+JRfQdHq41YhktG2COUZybQbwDY/QuDLswU9aU1V8Fo+//5BuTf ScaS47YyS73d40BjiDlQNu1JUV21+VfPz+xxK0mava+8kW4DQ9xXOJAcYOjdnUPHIXGt qC9YJs8Quw9Hr2x9AU9ztBt4Mda65yuqzi+8OhJgjDR94l9CtBSlGo2z6/xT8aXwrfDX HttvO+qMsrtUlMQrv5jiQEPIKoKR/klazw3i1Q4HjYgFog663YEXeVsii1l+5JxPlAky VGXg== X-Received: by 10.220.142.71 with SMTP id p7mr3391296vcu.3.1362915790161; Sun, 10 Mar 2013 04:43:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.59.9.227 with HTTP; Sun, 10 Mar 2013 04:42:40 -0700 (PDT) In-Reply-To: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> From: =?ISO-8859-1?Q?Fredrik_Markstr=F6m?= Date: Sun, 10 Mar 2013 12:42:40 +0100 Message-ID: To: David Mulder Cc: "yocto@yoctoproject.org" Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2013 11:43:11 -0000 Content-Type: multipart/alternative; boundary=f46d0434bf0c8848fb04d7908f5c --f46d0434bf0c8848fb04d7908f5c Content-Type: text/plain; charset=ISO-8859-1 The RT-throttling can be disabled with: echo -1 >/proc/sys/kernel/sched_rt_runtime_us echo -1 >/proc/sys/kernel/sched_rt_period_us But as mentioned in the other answers things like the system tick, ipi:s etc. will interfere. But significantly less then the rt-throttling :) /Fredrik --f46d0434bf0c8848fb04d7908f5c Content-Type: text/html; charset=ISO-8859-1
The RT-throttling can be disabled with:

echo -1 >/proc/sys/kernel/sched_rt_runtime_us
echo -1 >/proc/sys/kernel/sched_rt_period_us

But as mentioned in the other answers things like the system tick, ipi:s etc. will interfere. But significantly less then the rt-throttling :)

/Fredrik
--f46d0434bf0c8848fb04d7908f5c-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f175.google.com (mail-lb0-f175.google.com [209.85.217.175]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 24D30E01488 for ; Sun, 10 Mar 2013 11:00:39 -0700 (PDT) Received: by mail-lb0-f175.google.com with SMTP id n3so2578167lbo.34 for ; Sun, 10 Mar 2013 11:00:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=qFBc3O1/DD85/IWrCksf3+n9ky3z82/qD8TEsZd9pJM=; b=JsIvlxr/1pFTqgpQsXLiFSM7JW1xQ5z0L54ArdgQKKssP9R1v3Y8dplN+B/Neu1G8d aRwbjt2XPbO4IFKA1ugYwJlMAuWMxaNyMo259jZJv4z0ikPnnZ6hiT6vy4D7/wExiwnw qE2dWqseFRKXt8cWBPLnXFelB2EklnVXMZKhXsMVkaFrQt+ohZBtOMse7l+bqS0BIoF9 fPqnecOvCk2SSDIVVxLWt+ANQzg/QnC8FPEfid9cdxuuBAEHwH6nF/l8/r7FgM7XqcHh e/OZpQmlsNs4nfjms9K0fMZvssy4EopV/NQt4L8Ty8GmrncLWWJTzruyO+dkhpODm2lT rc7A== MIME-Version: 1.0 X-Received: by 10.112.8.9 with SMTP id n9mr3530577lba.71.1362938438385; Sun, 10 Mar 2013 11:00:38 -0700 (PDT) Received: by 10.112.61.3 with HTTP; Sun, 10 Mar 2013 11:00:38 -0700 (PDT) In-Reply-To: References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> Date: Sun, 10 Mar 2013 14:00:38 -0400 Message-ID: From: Bruce Ashfield To: =?ISO-8859-1?Q?Fredrik_Markstr=F6m?= Cc: "yocto@yoctoproject.org" , David Mulder Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2013 18:00:40 -0000 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Sun, Mar 10, 2013 at 7:42 AM, Fredrik Markstr=F6m wrote: > The RT-throttling can be disabled with: > > echo -1 >/proc/sys/kernel/sched_rt_runtime_us > echo -1 >/proc/sys/kernel/sched_rt_period_us > > But as mentioned in the other answers things like the system tick, ipi:s > etc. will interfere. But significantly less then the rt-throttling :) Absolutely! And as long as you really know what you are doing, you won't kill the system .. turning off the throttling definitely gives a bit more rope to that end as well :P Cheers, Bruce > > /Fredrik > > _______________________________________________ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto > --=20 "Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end" From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from omx.cbeyond.com (omx.cbeyond.com [50.20.30.10]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 4974AE00709 for ; Mon, 11 Mar 2013 09:15:47 -0700 (PDT) X-SBRS: 3.9 X-HAT: Sender Group CBEYOND_SYSTEMS, Policy $CBEYOND_RELAY applied. X-Hostname: omx02bay.sys.cbeyond.net X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqwBAPsBPlEyFB4Li2dsb2JhbABDhDrAG4FxDgEBARULGyEHgikBAQEDATpPAgEIGAoUECERJQEBBBMIE4dmAwkGAbMtDYlEF4xGghc4gl9hA5R1jTyII4Io X-IronPort-AV: E=Sophos;i="4.84,824,1355115600"; d="scan'208";a="13490548" Received: from mts.cbeyond.com (HELO EXHUB102.mmeprod.cbeyond) ([50.20.30.11]) by omx.cbeyond.com with ESMTP; 11 Mar 2013 12:15:46 -0400 Received: from EXMBX202A.mmeprod.cbeyond ([fe80::b0ca:4d9b:ba7c:8227]) by EXHUB102.mmeprod.cbeyond ([::1]) with mapi id 14.02.0298.004; Mon, 11 Mar 2013 12:15:45 -0400 From: David Mulder To: "yocto@yoctoproject.org" Thread-Topic: [yocto] Can I disable RT throttling? Thread-Index: Ac4cTWBMCcKVjca/TdSAdJObHV69ZQAUrgaAAHMmm5A= Date: Mon, 11 Mar 2013 16:15:45 +0000 Message-ID: <19F0379280417C4A8FE3BD23AA7298E501347F1C@EXMBX202A.mmeprod.cbeyond> References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> <513AAC65.1050002@windriver.com> In-Reply-To: <513AAC65.1050002@windriver.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [69.198.178.141] MIME-Version: 1.0 Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2013 16:15:47 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Wow! Thanks, Bruce, Fredrik, and everyone, for all this useful info! > > I'm running a 10us control loop by (under vxWorks) setting one thread's > > priority to max and not yielding ever >=20 > wrote: > You are looking for kernel functionality that doesn't exist yet. In > any SMP system there are sources of cross cpu interference that can't > be removed, if you do, the global state machines of the kernel will > break and the system will eventually come to a halt. >=20 > That's the mainline/scalable point of view, there are plenty of "custom" > and one off solutions to the problem, such as hotplugging the cpu out > of the system and running in an AMP configuration where a bare metal, > or RTOS can monopolize a CPU since the global state machines don't > interfere. These have their advantages and disadvantages, but if you > are coming from an existing RTOS application, they might be the > closest options to get you the performance you are looking for. An AMP configuration does seem like the closest thing to what the RTOS gave= me. Do you have more info about that? Specifically, I see that I can rem= ove a core from the scheduler ($ echo 0 > /sys/devices/system/cpu//online), but how can I make my run my code on that core? Do I get the= code running on that core first and then remove the core, or vice-versa, o= r some other concept? I do PCIe reads/writes - are those affected? I use = atomic read-write-modify locks for critical sections - will those still fun= ction correctly (I can't imagine they would break, but I thought I'd ask)? > > So maybe disabling RT throttling is all I need to do, maybe some other > > approach is needed. Hopefully someone will know more than I do. > > > wrote: > The RT-throttling can be disabled with: > > echo -1 >/proc/sys/kernel/sched_rt_runtime_us > echo -1 >/proc/sys/kernel/sched_rt_period_us > > But as mentioned in the other answers things like the system tick, ipi:s > etc. will interfere. But significantly less then the rt-throttling :) Thanks, Fredrick! I will try this. Dave Mulder From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-we0-f179.google.com (mail-we0-f179.google.com [74.125.82.179]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 84F30E0070E for ; Mon, 11 Mar 2013 10:32:51 -0700 (PDT) Received: by mail-we0-f179.google.com with SMTP id p43so3878206wea.10 for ; Mon, 11 Mar 2013 10:32:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=RQHBWDJnFXLJbRAfuIAJgfUFWfLW7h3VaYUzHUvZM8g=; b=LO8RNES0exCHmY9RvDtO0H3CUiN5jh3Nf6QzyN8CyEKVX0e6UwMchX/oFihSNPBYZs 7H4HU0CDpuDWkuDN3N3NZya9h8re5gbPMlO5EILG7A5vH4p9KhMIPQ/vY2F9BJzyOsT5 30LdIt140orEN1X7djK1TAgYG16UFLWJwNedNNDH9TCv/MXNUJpDlkUHyXqFJlspGh1Y ahLk+9nB6MCN2docOuEMvD3Nody39sZtLtQlpn6bD6ZbBvyUB3SDbsLdpdF2Z2CoBua+ uZtwrr2gpqtZ0hXETmOyFaf5IVJ6uEAIcDh2IGqy0sK0SQm9A0QLkKR4rVbVEpp1tUD7 jBVQ== MIME-Version: 1.0 X-Received: by 10.180.97.132 with SMTP id ea4mr14273832wib.23.1363023170349; Mon, 11 Mar 2013 10:32:50 -0700 (PDT) Received: by 10.194.45.100 with HTTP; Mon, 11 Mar 2013 10:32:50 -0700 (PDT) In-Reply-To: <19F0379280417C4A8FE3BD23AA7298E501347F1C@EXMBX202A.mmeprod.cbeyond> References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> <513AAC65.1050002@windriver.com> <19F0379280417C4A8FE3BD23AA7298E501347F1C@EXMBX202A.mmeprod.cbeyond> Date: Mon, 11 Mar 2013 13:32:50 -0400 Message-ID: From: Trevor Woerner To: David Mulder Cc: "yocto@yoctoproject.org" Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2013 17:32:51 -0000 Content-Type: text/plain; charset=ISO-8859-1 On Mon, Mar 11, 2013 at 12:15 PM, David Mulder wrote: > but how can I make my run my code on that core? You can call sched_setaffinity() before fork()ing your task, or use taskset on the cmdline. (sorry if this has already been answered but I'm not seeing any responses due to mailing list delays) From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from omx.cbeyond.com (omx.cbeyond.com [50.20.30.10]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 14229E00719 for ; Mon, 11 Mar 2013 10:38:06 -0700 (PDT) X-SBRS: 3.9 X-HAT: Sender Group CBEYOND_SYSTEMS, Policy $CBEYOND_RELAY applied. X-Hostname: omx06bay.sys.cbeyond.net X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtEBAL4VPlEyFB4Li2dsb2JhbABDhDyDJrx+gXMOAQEBFQsbKIIpAQEBAwE6PwULAgEIDgcNFAkHIREUEQEBBA4FCId5AwkGAbQ6DYlbjEaCFzEHgl9hA5R1jTyII4Io X-IronPort-AV: E=Sophos;i="4.84,825,1355115600"; d="scan'208";a="13127882" Received: from mts.cbeyond.com (HELO EXHUB101.mmeprod.cbeyond) ([50.20.30.11]) by omx.cbeyond.com with ESMTP; 11 Mar 2013 13:38:05 -0400 Received: from EXHUB104.mmeprod.cbeyond (10.128.38.103) by EXHUB101.mmeprod.cbeyond (10.128.38.100) with Microsoft SMTP Server (TLS) id 14.2.298.4; Mon, 11 Mar 2013 13:38:03 -0400 Received: from EXMBX202A.mmeprod.cbeyond ([fe80::b0ca:4d9b:ba7c:8227]) by EXHUB104.mmeprod.cbeyond ([::1]) with mapi id 14.02.0298.004; Mon, 11 Mar 2013 13:38:03 -0400 From: David Mulder To: Trevor Woerner Thread-Topic: [yocto] Can I disable RT throttling? Thread-Index: Ac4cTWBMCcKVjca/TdSAdJObHV69ZQAUrgaAAHMmm5AADNILAAAITIEg Date: Mon, 11 Mar 2013 17:38:02 +0000 Message-ID: <19F0379280417C4A8FE3BD23AA7298E501347F52@EXMBX202A.mmeprod.cbeyond> References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> <513AAC65.1050002@windriver.com> <19F0379280417C4A8FE3BD23AA7298E501347F1C@EXMBX202A.mmeprod.cbeyond> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [69.198.178.141] MIME-Version: 1.0 Cc: "yocto@yoctoproject.org" Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2013 17:38:06 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable > From: Trevor Woerner [mailto:twoerner@gmail.com] > You can call sched_setaffinity() before fork()ing your task, or use > taskset on the cmdline. Will that work on a core that's offline? From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail1.windriver.com (mail1.windriver.com [147.11.146.13]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 81FE9E0070E for ; Mon, 11 Mar 2013 10:45:45 -0700 (PDT) Received: from ALA-HCA.corp.ad.wrs.com (ala-hca.corp.ad.wrs.com [147.11.189.40]) by mail1.windriver.com (8.14.5/8.14.3) with ESMTP id r2BHjfEV013055 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Mon, 11 Mar 2013 10:45:41 -0700 (PDT) Received: from [128.224.22.49] (128.224.22.49) by ALA-HCA.corp.ad.wrs.com (147.11.189.50) with Microsoft SMTP Server id 14.2.342.3; Mon, 11 Mar 2013 10:45:40 -0700 Message-ID: <513E1843.3070604@windriver.com> Date: Mon, 11 Mar 2013 13:45:39 -0400 From: Bruce Ashfield User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130221 Thunderbird/17.0.3 MIME-Version: 1.0 To: David Mulder References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> <513AAC65.1050002@windriver.com> <19F0379280417C4A8FE3BD23AA7298E501347F1C@EXMBX202A.mmeprod.cbeyond> <19F0379280417C4A8FE3BD23AA7298E501347F52@EXMBX202A.mmeprod.cbeyond> In-Reply-To: <19F0379280417C4A8FE3BD23AA7298E501347F52@EXMBX202A.mmeprod.cbeyond> Cc: "yocto@yoctoproject.org" Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2013 17:45:45 -0000 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit On 13-03-11 01:38 PM, David Mulder wrote: >> From: Trevor Woerner [mailto:twoerner@gmail.com] >> You can call sched_setaffinity() before fork()ing your task, or use >> taskset on the cmdline. > > Will that work on a core that's offline? Nope. Only with an online core controlled by the Linux scheduler. If you do end up trying to get AMP working, you need to plumbing to load the other OS/kernel in a reserved memory location, set the program counter and start the OS. But that secondary OS has to know how to behave in a system that it doesn't control, and you'll need ways to communicate with it from Linux. remoteproc/rpmsg can solve some of the issues that I mention, but it is far from out of the box. That's why there's more interest in running a single task with exclusive CPU in userspace. The work and scaffolding required to get an AMP system up and running is non trivial. Cheers, Bruce > _______________________________________________ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from omx.cbeyond.com (omx.cbeyond.com [50.20.30.10]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 85266E003AB for ; Mon, 11 Mar 2013 11:13:49 -0700 (PDT) X-SBRS: 3.9 X-HAT: Sender Group CBEYOND_SYSTEMS, Policy $CBEYOND_RELAY applied. X-Hostname: omx05bay.sys.cbeyond.net X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Aq0BACwePlEyFB4Li2dsb2JhbABDhD7AJIF1DgEBARULGyiCKQEBAQMBOj8QAgEIIhQQMiUCBA4NiAUGAb5ljl0xB4JfYQOqVIIo X-IronPort-AV: E=Sophos;i="4.84,825,1355115600"; d="scan'208";a="12795061" Received: from mts.cbeyond.com (HELO EXHUB102.mmeprod.cbeyond) ([50.20.30.11]) by omx.cbeyond.com with ESMTP; 11 Mar 2013 14:13:48 -0400 Received: from EXHUB104.mmeprod.cbeyond (10.128.38.103) by EXHUB102.mmeprod.cbeyond (10.128.38.101) with Microsoft SMTP Server (TLS) id 14.2.298.4; Mon, 11 Mar 2013 14:13:47 -0400 Received: from EXMBX202A.mmeprod.cbeyond ([fe80::b0ca:4d9b:ba7c:8227]) by EXHUB104.mmeprod.cbeyond ([::1]) with mapi id 14.02.0298.004; Mon, 11 Mar 2013 14:13:47 -0400 From: David Mulder To: Bruce Ashfield Thread-Topic: [yocto] Can I disable RT throttling? Thread-Index: Ac4cTWBMCcKVjca/TdSAdJObHV69ZQAUrgaAAHMmm5AADNILAAAITIEg///BMICAAEI7cA== Date: Mon, 11 Mar 2013 18:13:46 +0000 Message-ID: <19F0379280417C4A8FE3BD23AA7298E501347F87@EXMBX202A.mmeprod.cbeyond> References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> <513AAC65.1050002@windriver.com> <19F0379280417C4A8FE3BD23AA7298E501347F1C@EXMBX202A.mmeprod.cbeyond> <19F0379280417C4A8FE3BD23AA7298E501347F52@EXMBX202A.mmeprod.cbeyond> <513E1843.3070604@windriver.com> In-Reply-To: <513E1843.3070604@windriver.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [69.198.178.141] MIME-Version: 1.0 Cc: "yocto@yoctoproject.org" Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2013 18:13:49 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable > > Will that work on a core that's offline? >=20 > Nope. Only with an online core controlled by the Linux scheduler. > If you do end up trying to get AMP working, you need to plumbing > to load the other OS/kernel in a reserved memory location, set the > program counter and start the OS. >=20 > But that secondary OS has to know how to behave in a system that > it doesn't control, and you'll need ways to communicate with it > from Linux. >=20 > remoteproc/rpmsg can solve some of the issues that I mention, but > it is far from out of the box. >=20 > That's why there's more interest in running a single task with > exclusive CPU in userspace. The work and scaffolding required to > get an AMP system up and running is non trivial. I kinda thought so, but I was hopeful. After speaking with some co-workers, I have a new perspective on these dela= ys: yes, we are trying to do a 10us control loop, but if we miss a step or= two occasionally we can accept that. And looking online I see people indi= cating context switch times well below 10us (Core-i cpus), which is better = than I had anticipated, and should be workable. So I'm going to approach t= his problem by just trying to squeeze the kernel as much as I can. Some th= ings that I see to squeeze are /dev/cpu_dma_latency (should be 0) or max_cs= tate (should be as low as possible (0, maybe 1)), possibly idle=3Dpoll. Ar= e there other kernel parameters that can minimize kernel interference/time?= And perhaps hints about how to set them in Yocto or menuconfig? Thanks! Dave From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.windriver.com (mail.windriver.com [147.11.1.11]) by yocto-www.yoctoproject.org (Postfix) with ESMTP id 0ED2CE00738 for ; Mon, 11 Mar 2013 17:14:23 -0700 (PDT) Received: from ALA-HCA.corp.ad.wrs.com (ala-hca.corp.ad.wrs.com [147.11.189.40]) by mail.windriver.com (8.14.5/8.14.3) with ESMTP id r2C0EIFh017399 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Mon, 11 Mar 2013 17:14:18 -0700 (PDT) Received: from bruce-ashfields-macbook.local (128.224.22.23) by ALA-HCA.corp.ad.wrs.com (147.11.189.50) with Microsoft SMTP Server id 14.2.342.3; Mon, 11 Mar 2013 17:14:18 -0700 Message-ID: <513E7359.6060205@windriver.com> Date: Mon, 11 Mar 2013 20:14:17 -0400 From: Bruce Ashfield User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: David Mulder References: <19F0379280417C4A8FE3BD23AA7298E501347DE7@EXMBX202A.mmeprod.cbeyond> <513AAC65.1050002@windriver.com> <19F0379280417C4A8FE3BD23AA7298E501347F1C@EXMBX202A.mmeprod.cbeyond> <19F0379280417C4A8FE3BD23AA7298E501347F52@EXMBX202A.mmeprod.cbeyond> <513E1843.3070604@windriver.com> <19F0379280417C4A8FE3BD23AA7298E501347F87@EXMBX202A.mmeprod.cbeyond> In-Reply-To: <19F0379280417C4A8FE3BD23AA7298E501347F87@EXMBX202A.mmeprod.cbeyond> Cc: "yocto@yoctoproject.org" Subject: Re: Can I disable RT throttling? X-BeenThere: yocto@yoctoproject.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: Discussion of all things Yocto Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Mar 2013 00:14:23 -0000 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit On 13-03-11 2:13 PM, David Mulder wrote: >>> Will that work on a core that's offline? >> >> Nope. Only with an online core controlled by the Linux scheduler. >> If you do end up trying to get AMP working, you need to plumbing >> to load the other OS/kernel in a reserved memory location, set the >> program counter and start the OS. >> >> But that secondary OS has to know how to behave in a system that >> it doesn't control, and you'll need ways to communicate with it >> from Linux. >> >> remoteproc/rpmsg can solve some of the issues that I mention, but >> it is far from out of the box. >> >> That's why there's more interest in running a single task with >> exclusive CPU in userspace. The work and scaffolding required to >> get an AMP system up and running is non trivial. > > I kinda thought so, but I was hopeful. > > After speaking with some co-workers, I have a new perspective on these delays: yes, we are trying to do a 10us control loop, but if we miss a step or two occasionally we can accept that. And looking online I see people indicating context switch times well below 10us (Core-i cpus), which is better than I had anticipated, and should be workable. So I'm going to approach this problem by just trying to squeeze the kernel as much as I can. Some things that I see to squeeze are /dev/cpu_dma_latency (should be 0) or max_cstate (should be as low as possible (0, maybe 1)), possibly idle=poll. Are there other kernel parameters that can minimize kernel interference/time? And perhaps hints about how to set them in Yocto or menuconfig? There's nothing "out of the box" that I can recommend, outside of saying "it depends on your platform". It's a matter of knowing your devices, their interrupts, and the configuration of your kernel. Using things like CONFIG_NOHZ will remove the timer tick, and hence ticks that you may not need, you want to move device interrupts off your core, except for the one that you want. Use cgroups/cpusets to control resources and the scheduler off your core with "other" tasks. pin/lock memory to avoid page faults, etc. If you check out the preempt-rt wiki page on kernel.org, a lot of the information there applies to making sure that your prioritized thread gets the most run time that it can. As we progress with the meta-realtime layer, scripts for the above, system configuration, services and will be part of the layer and might be of use. Cheers, Bruce > > Thanks! > Dave