Re: [Xen-devel] VCPUOP_set_periodic

On 15/11/13 11:24, Simon Martin wrote:

Hi Andrew,

The operating system I am trying to port to Xen is an industrial servo= controller. We are currently running at 125 microsecond servo update rate= on our bespoke hardware (at the moment MIPS64 and ARM) (hence the 125 micr= osecond ideal interrupt period). We will be driving EtherCAT servo drives= that need to be updated at 500 microseconds (hence at least 500 microsecon= d interrupt period). If we can achieve this on a single core ARM11 clocked= at 532 MHz, it must be achievable on PC hardware. As I said before= the idea is to dedicate a CPU core to this with all other functions runnin= g on the other cores.

Luckily we can accept a reasonable amount of jitter on the EtherCAT= network as we can hand over clock synchronisation to a slave. This gives= us a little leeway.

Regards.

It certainly is achievable on PC hardware= , but not under the standard assumptions in virtualisation. You want= to use VCPUOP_set_singleshot_timer, but be aware than there are still no= normal guarantees that your domain vcpu will be run when the timer expires= ; the credit scheduler will pick the highest priority vcpu to run, which= might not be the one which is expecting a timer interrupt.

You shou= ld investigate using the arinc653 scheduler in Xen, which is a realtime = scheduler, and might be more appropriate for your usecase.

~Andrew

------ Original Message ------

From: "Andrew Cooper" <andrew.cooper3@citrix.com>

To: "Simon Martin" <smartin@milliways.cl>; mailto:xen-devel@lists.xen.or= g <xen-devel@lists.xen.org>

Sent: 14/11/2013 18:39:21

Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer

On 14/11/2013 21:18, Simon Martin wrote:

Hi all,

I need a periodic timer running at ideally at 125 microseconds and = at least 500 microseconds. I've just found the VCPUOP_set_periodic_timer,= however there is a comment saying "periods less than one millisecond may= not be supported".

I will be running on an x64 machine. Is this supported? If not, is = there any alternate means of generating a fast interrupt?

Regards.

What is the usecase here? 125us= is very short indeed. Xen certainly cant guarantee anything more = accurate than 50us. Unless the affected vcpu is running uncontested= on the hardware, there is very little chance that the vcpu will indeed = be woken up again in 125us.

It sounds as if you are looking for some= pseudo realtime system, at which point you might want to consider a differ= ent scheduler.

~Andrew

--------=_MBA738298C-F2E0-401E-ADC0-734323E03E2B-- --===============4793090721494988134== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4793090721494988134==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 12:17:02 +0000 Message-ID: <528610BE.2080802@citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3735914663010010382==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin Cc: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org --===============3735914663010010382== Content-Type: multipart/alternative; boundary="------------040403040603030405080706" --------------040403040603030405080706 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit On 15/11/13 12:02, Simon Martin wrote: > Thanks Andrew, > > I've tried to load the arinc653 scheduler, however Xen crashes > whenever I create a CPU pool with this scheduler. I reported this and > it was confirmed by Seeing as I've been fighting to get my little port > up, running and communicating via shared pages with a Windows domU > I've kind of ignored this. However before I go and invest more time > and effort with this I just wanted to confirm that I'm not barking up > the wrong tree. > > Regards. The cpupool issue with the arinc653 scheduler is now fixed (or at least believed to be) in xen-unstable. ~Andrew --------------040403040603030405080706 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: 8bit
On 15/11/13 12:02, Simon Martin wrote:

Thanks Andrew,

I've tried to load the arinc653 scheduler, however Xen crashes whenever I create a CPU pool with this scheduler. I reported this and it was confirmed by Seeing as I've been fighting to get my little port up, running and communicating via shared pages with a Windows domU I've kind of ignored this. However before I go and invest more time and effort with this I just wanted to confirm that I'm not barking up the wrong tree.

Regards.

The cpupool issue with the arinc653 scheduler is now fixed (or at least believed to be) in xen-unstable.

~Andrew

--------------040403040603030405080706-- --===============3735914663010010382== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============3735914663010010382==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 12:37:38 +0000 Message-ID: References: Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1874753798418603913==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Keir Fraser , Andrew Cooper Cc: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org --===============1874753798418603913== Content-Type: multipart/related; boundary="------=_MB2FEAE09A-F635-4CF7-B6BB-272E23A2EE13" --------=_MB2FEAE09A-F635-4CF7-B6BB-272E23A2EE13 Content-Type: multipart/alternative; boundary="------=_MBBC0FB126-FF03-480F-B0BF-0DE9CBDC9860" --------=_MBBC0FB126-FF03-480F-B0BF-0DE9CBDC9860 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; format=flowed; charset=utf-8 Latency isn't a big problem. Jitter could be. CPU offline shouldn't be a= =20 problem as the system is simple and static, at power up I create 2 CPU=20 pools, one for the realtime machine domU running on a single core, and=20 one for dom0 and a Windows machine running on the remaining cores. Any idea on the timings I can expect? For example, assuming there are no= =20 CPU contention issues, if I request a 125 microsecond single shot timer=20 interrupt, will I get 125 microseconds +/- 5 microseconds (perfectly=20 acceptable) or 500 microseconds (because that is the minimum=20 granularity) +/- 50 microseconds (totally unacceptable)? ------ Original Message ------ From: "Keir Fraser" To: "Simon Martin" ; "Andrew Cooper"=20 Cc: "xen-devel@lists.xen.org" Sent: 15/11/2013 08:56:18 Subject: Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer >If you have dedicated a core to the vcpu, so you can discount=20 >scheduling crosstalk, you shouldn=E2=80=99t miss deadlines by more than= a=20 >microsecond or two. Just enough time to wake up the CPU, handle the=20 >timer interrupt, and wake up the vcpu through the scheduler. There are=20 >infrequent operations that can take out all CPUs for a period: taking a= =20 >CPU offline for example. You could avoid doing such things in your=20 >setup perhaps. Ultimately you will need to measure and confirm=20 >deadline-to-notification latency distribution if it is critical to you. > > -- Keir --------=_MBBC0FB126-FF03-480F-B0BF-0DE9CBDC9860 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer
Latency isn't a big problem. Jitter could be. CPU offline shouldn't= be a problem as the system is simple and static, at power up I create 2= CPU pools, one for the realtime machine domU running on a single core, = and one for dom0 and a Windows machine running on the remaining cores.

Any idea on the timings I can expect? For example, assuming there are= no CPU contention issues, if I request a 125 microsecond single shot timer= interrupt, will I get 125 microseconds +/- 5 microseconds (perfectly accep= table) or 500 microseconds (because that is the minimum granularity) +/-= 50 microseconds (totally unacceptable)?

------ Original Message ------

From: "Keir Fraser" <keir.xen= @gmail.com>

To: "Simon Martin" <smartin= @milliways.cl>; "Andrew Cooper" <andrew.cooper3@citrix.com>

Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>

Sent: 15/11/2013 08:56:18

Subject: Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer

If you have dedicated a core to the vcpu, so you can discount= scheduling crosstalk, you shouldn=E2=80=99t miss deadlines by more than= a microsecond or two. Just enough time to wake up the CPU, handle the time= r interrupt, and wake up the vcpu through the scheduler. There are infreque= nt operations that can take out all CPUs for a period: taking a CPU offline= for example. You could avoid doing such things in your setup perhaps. Ultimately you will need to measure and confirm= deadline-to-notification latency distribution if it is critical to you.=

-- Keir
--------=_MBBC0FB126-FF03-480F-B0BF-0DE9CBDC9860-- --------=_MB2FEAE09A-F635-4CF7-B6BB-272E23A2EE13 Content-Disposition: inline Content-Id: Content-Transfer-Encoding: base64 Content-Type: image/png iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAJXSURBVDhPnZBPSJNxGMefQ5d2SDpUdCkIVl26tIiKUsOgWLEO6WHuUAgtoogORVDZ IQwkCLpIvEqZSvjiIZcQSb3OmZvCHCKLoQ7McKi1rXzpH72v2/v0fH9bq4OnBt993+f5fp7vCy8t dBNleoiWe4k+6uTK9pFPpIkMUbzsmH3IwYHHHTOXHsoFbpH+Jbxv1vpw1yzmu2z++tyBY8YeObi1 CtyL+vqhH6kLOc5rzFlRros531NyzLJHDg58pWDuMbnmO0n/ngzklkb8/KZ1h9Ji9BI7y4+U/9kh Bwced6og3U6+xYHdaef9RV4yvJx5fZozRj0vRwLszDcrx4w9cnDgcacKUm2kfZusN4tTp9gcPc6F gsWeoMZm7AwXU+eUY8YeOTjwuFMFUw/JsN757UK8jlfCRxQY1BK8MlzHhcRJ5ZixV7lw4HGnCuL3 KV5INjirsWouqZZXx6Cj/0hm7MsMeNypgmgLGT/HvbY9coinn+5k++1hUTXbozWi2pJjlv10l+TC gcedKjBuk/bp5UHTGjrA/Tc3sH69il+1bKt8eWjw3nbuu7GR+29VMTjwuFMFoavki7ZuTlvDNZwN 7edQ8yaeScYk+/vDPHBni8rBgcedRETPzpOru4n0mSd7clbkBOcHvTyh7eXIg13q7fBEu4c/yx45 OPC4UwUdAaL2RnJ3nl03lOrw5KyxANsTQbYnL4uulFxm7JGDA4+7SoHmJ2prILdIf3Ft6+xc7zHT DDfav8abHDhm7JGDA79WAeQS+USayBDFy44Ze+SKrxTg7//F9Bs8U4bjEi0nagAAAABJRU5ErkJg gg== --------=_MB2FEAE09A-F635-4CF7-B6BB-272E23A2EE13-- --===============1874753798418603913== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============1874753798418603913==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nate Studer Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 07:46:10 -0500 Message-ID: <52861792.8090905@dornerworks.com> References: <528610BE.2080802@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <528610BE.2080802@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Andrew Cooper , Simon Martin Cc: George Dunlap , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On 11/15/2013 7:17 AM, Andrew Cooper wrote: > > The cpupool issue with the arinc653 scheduler is now fixed (or at least believed > to be) in xen-unstable. The crashes are fixed in unstable, but pools still do not work in xen-unstable unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus parameter is set to 1, and cpu0 is never removed from Pool-0. I have pools working locally without these restrictions, but I was unable to get a patchset around before the feature freeze was declared, so I believe it is too late to try and get it into unstable. (George, correct me if I am wrong.) However, I am more than happy to provide Simon with my patches to get arinc653 cpu-pools completely working with the understanding that they may change before they make it into xen-unstable. Also, the arinc653 scheduler currently only supports a scheduling resolution of 1ms, but it is a trivial task to change it some other resolution. > > ~Andrew > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 12:52:02 +0000 Message-ID: <528618F2.70902@citrix.com> References: <528610BE.2080802@citrix.com> <52861792.8090905@dornerworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <52861792.8090905@dornerworks.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Nate Studer Cc: George Dunlap , Simon Martin , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On 15/11/13 12:46, Nate Studer wrote: > On 11/15/2013 7:17 AM, Andrew Cooper wrote: >> The cpupool issue with the arinc653 scheduler is now fixed (or at least believed >> to be) in xen-unstable. > The crashes are fixed in unstable, but pools still do not work in xen-unstable > unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus > parameter is set to 1, and cpu0 is never removed from Pool-0. > > I have pools working locally without these restrictions, but I was unable to get > a patchset around before the feature freeze was declared, so I believe it is too > late to try and get it into unstable. (George, correct me if I am wrong.) > > However, I am more than happy to provide Simon with my patches to get arinc653 > cpu-pools completely working with the understanding that they may change before > they make it into xen-unstable. > > Also, the arinc653 scheduler currently only supports a scheduling resolution of > 1ms, but it is a trivial task to change it some other resolution. Absolutely post the patches. The worst that can happen is that they are labelled as differed, but at least they are available for people to use. It is George's call, but if they are small and fairly contained then they might be fine to go in. ~Andrew From mboxrd@z Thu Jan 1 00:00:00 1970 From: George Dunlap Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 12:54:17 +0000 Message-ID: <52861979.6020801@eu.citrix.com> References: <528610BE.2080802@citrix.com> <52861792.8090905@dornerworks.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <52861792.8090905@dornerworks.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Nate Studer , Andrew Cooper , Simon Martin Cc: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On 15/11/13 12:46, Nate Studer wrote: > On 11/15/2013 7:17 AM, Andrew Cooper wrote: >> The cpupool issue with the arinc653 scheduler is now fixed (or at least believed >> to be) in xen-unstable. > The crashes are fixed in unstable, but pools still do not work in xen-unstable > unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus > parameter is set to 1, and cpu0 is never removed from Pool-0. > > I have pools working locally without these restrictions, but I was unable to get > a patchset around before the feature freeze was declared, so I believe it is too > late to try and get it into unstable. (George, correct me if I am wrong.) Bug fixes are not features. :-) Bug fixes are accepted very late in the release process. The only reason they might ever be rejected is if we think there may be a risk that the fix will break other working functionality -- particularly if we think that breakage may not be discovered until after the release. If the fixes involve generic scheduling code, I imagine they would be accepted well into RC3 if they were not terribly complicated; if they were limited to the arinc653 scheduler itself, they could probably be accepted until just before the release. -George From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 13:10:14 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1444022889777413094==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin , Andrew Cooper Cc: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============1444022889777413094== Content-type: multipart/related; boundary="B_3467365822_113883289" > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3467365822_113883289 Content-type: multipart/alternative; boundary="B_3467365822_113886149" --B_3467365822_113886149 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable You=B9ll have to measure to confirm, but if the core is dedicated to your vcp= u then there is no reason there should be 50us jitter. Also there is not much point playing with different schedulers =8B if there are no other vcpus schedulable on the cpu, the scheduler can=B9t have multiple runnable vcpus to choose between. -- Keir On 15/11/2013 12:37, "Simon Martin" wrote: > Latency isn't a big problem. Jitter could be. CPU offline shouldn't be a > problem as the system is simple and static, at power up I create 2 CPU po= ols, > one for the realtime machine domU running on a single core, and one for d= om0 > and a Windows machine running on the remaining cores. > =20 > Any idea on the timings I can expect? For example, assuming there are no = CPU > contention issues, if I request a 125 microsecond single shot timer inter= rupt, > will I get 125 microseconds +/- 5 microseconds (perfectly acceptable) or = 500 > microseconds (because that is the minimum granularity) +/- 50 microsecond= s > (totally unacceptable)? > =20 > =20 > ------ Original Message ------ > From: "Keir Fraser" > To: "Simon Martin" ; "Andrew Cooper" > > Cc: "xen-devel@lists.xen.org" > Sent: 15/11/2013 08:56:18 > Subject: Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer > =20 >> If you have dedicated a core to the vcpu, so you can discount scheduling >> crosstalk, you shouldn=B9t miss deadlines by more than a microsecond or tw= o. >> Just enough time to wake up the CPU, handle the timer interrupt, and wak= e up >> the vcpu through the scheduler. There are infrequent operations that can= take >> out all CPUs for a period: taking a CPU offline for example. You could a= void >> doing such things in your setup perhaps. Ultimately you will need to me= asure >> and confirm deadline-to-notification latency distribution if it is criti= cal >> to you.=20 >>=20 >> -- Keir >=20 --B_3467365822_113886149 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Re: Re[4]: [Xen-devel] VCPUOP_set_periodic_timer You’ll have to measure to confirm, but if the core is dedicated to y= our vcpu then there is no reason there should be 50us jitter. Also there is = not much point playing with different schedulers — if there are no oth= er vcpus schedulable on the cpu, the scheduler can’t have multiple run= nable vcpus to choose between.

-- Keir

On 15/11/2013 12:37, "Simon Martin" <smartin@milliways.cl> wrote:

<= FONT SIZE=3D"2">Latency isn't a big problem. Jitt= er could be. CPU offline shouldn't be a problem as the system is simple and = static, at power up I create 2 CPU pools, one for the realtime machine domU = running on a single core, and one for dom0 and a Windows machine running on = the remaining cores.

Any idea on the timings I can expect? For example, assuming there are no CP= U contention issues, if I request a 125 microsecond single shot timer interr= upt, will I get 125 microseconds +/- 5 microseconds (perfectly acceptable) o= r 500 microseconds (because that is the minimum granularity) +/- 50 microsec= onds (totally unacceptable)?

------ Original Message ------
From: "Keir Fraser" <keir.xen@gma= il.com>
To: "Simon Martin" <smartin@mil= liways.cl>; "Andrew Cooper" <andrew.cooper3@citrix.com>
Cc: "xen-devel@lists.xen.org&quo= t; <xen-devel@lists.xen.org>
Sent: 15/11/2013 08:56:18
Subject: Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer

If you have dedicated a core to the vcpu,= so you can discount scheduling crosstalk, you shouldn’t miss deadline= s by more than a microsecond or two. Just enough time to wake up the CPU, ha= ndle the timer interrupt, and wake up the vcpu through the scheduler. There = are infrequent operations that can take out all CPUs for a period: taking a = CPU offline for example. You could avoid doing such things in your setup per= haps. Ultimately you will need to meas= ure and confirm deadline-to-notification latency distribution if it is criti= cal to you.

-- Keir

=

--B_3467365822_113886149-- --B_3467365822_113883289 Content-Type: image/png; name="image.png" Content-ID: <3467365814_113877269> Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1B AACxjwv8YQUAAAJXSURBVDhPnZBPSJNxGMefQ5d2SDpUdCkIVl26tIiKUsOgWLEO6WHuUAgt oogORVDZIQwkCLpIvEqZSvjiIZcQSb3OmZvCHCKLoQ7McKi1rXzpH72v2/v0fH9bq4OnBt99 3+f5fp7vCy8tdBNleoiWe4k+6uTK9pFPpIkMUbzsmH3IwYHHHTOXHsoFbpH+Jbxv1vpw1yzm u2z++tyBY8YeObi1CtyL+vqhH6kLOc5rzFlRros531NyzLJHDg58pWDuMbnmO0n/ngzklkb8 /KZ1h9Ji9BI7y4+U/9khBwced6og3U6+xYHdaef9RV4yvJx5fZozRj0vRwLszDcrx4w9cnDg cacKUm2kfZusN4tTp9gcPc6FgsWeoMZm7AwXU+eUY8YeOTjwuFMFUw/JsN757UK8jlfCRxQY 1BK8MlzHhcRJ5ZixV7lw4HGnCuL3KV5INjirsWouqZZXx6Cj/0hm7MsMeNypgmgLGT/HvbY9 coinn+5k++1hUTXbozWi2pJjlv10l+TCgcedKjBuk/bp5UHTGjrA/Tc3sH69il+1bKt8eWjw 3nbuu7GR+29VMTjwuFMFoavki7ZuTlvDNZwN7edQ8yaeScYk+/vDPHBni8rBgcedRETPzpOr u4n0mSd7clbkBOcHvTyh7eXIg13q7fBEu4c/yx45OPC4UwUdAaL2RnJ3nl03lOrw5KyxANsT QbYnL4uulFxm7JGDA4+7SoHmJ2prILdIf3Ft6+xc7zHTDDfav8abHDhm7JGDA79WAeQS+USa yBDFy44Ze+SKrxTg7//F9Bs8U4bjEi0nagAAAABJRU5ErkJggg== --B_3467365822_113883289-- --===============1444022889777413094== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============1444022889777413094==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 13:13:12 +0000 Message-ID: <52861DE8.5060208@citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2493604012922610266==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Keir Fraser Cc: Simon Martin , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org --===============2493604012922610266== Content-Type: multipart/alternative; boundary="------------060109040602010102080805" --------------060109040602010102080805 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit On 15/11/13 13:10, Keir Fraser wrote: > Re: Re[4]: [Xen-devel] VCPUOP_set_periodic_timer You'll have to > measure to confirm, but if the core is dedicated to your vcpu then > there is no reason there should be 50us jitter. Also there is not much > point playing with different schedulers --- if there are no other > vcpus schedulable on the cpu, the scheduler can't have multiple > runnable vcpus to choose between. > > -- Keir The deadline timer code as up to 50us of slop when calculating when to wake up and when timers expire. It is somewhat poor, and is certainly an area in need of improvement. ~Andrew --------------060109040602010102080805 Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit
On 15/11/13 13:10, Keir Fraser wrote:

Re: Re[4]: [Xen-devel] VCPUOP_set_periodic_timer You’ll have to measure to confirm, but if the core is dedicated to your vcpu then there is no reason there should be 50us jitter. Also there is not much point playing with different schedulers — if there are no other vcpus schedulable on the cpu, the scheduler can’t have multiple runnable vcpus to choose between.

-- Keir

The deadline timer code as up to 50us of slop when calculating when to wake up and when timers expire. It is somewhat poor, and is certainly an area in need of improvement.

~Andrew
--------------060109040602010102080805-- --===============2493604012922610266== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2493604012922610266==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 13:39:31 +0000 Message-ID: References: <52861DE8.5060208@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <52861DE8.5060208@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Andrew Cooper Cc: Simon Martin , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On 15/11/2013 13:13, "Andrew Cooper" wrote: > = > On 15/11/13 13:10, Keir Fraser wrote: > = > = >> Re: Re[4]: [Xen-devel] VCPUOP_set_periodic_timer You=B9ll have to meas= ure to >> confirm, but if the core is dedicated to your vcpu then there is no reas= on >> there should be 50us jitter. Also there is not much point playing with >> different schedulers =8B if there are no other vcpus schedulable on the = cpu, >> the scheduler can=B9t have multiple runnable vcpus to choose between. >> = >> -- Keir >> = > = > The deadline timer code as up to 50us of slop when calculating when to w= ake > up and when timers expire. It is somewhat poor, and is certainly an area= in > need of improvement. Forgot about that. For real-time applications, timer_slop=3D0 on the Xen command line will turn it off. But there could certainly be other gotchas lurking; actual performance needs to be measured. -- Keir > ~Andrew > = From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: VCPUOP_set_periodic_timer Date: Fri, 15 Nov 2013 22:10:25 +0100 Message-ID: <1384549825.3896.169.camel@Abyss> References: <52860878.8010406@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4899604362579190209==" Return-path: In-Reply-To: <52860878.8010406@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Andrew Cooper Cc: Roland Heusser , Simon Martin , Sisu Xi , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============4899604362579190209== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-OUPr11vrOfzC5kEPYjeg" --=-OUPr11vrOfzC5kEPYjeg Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On ven, 2013-11-15 at 11:41 +0000, Andrew Cooper wrote: > On 15/11/13 11:24, Simon Martin wrote:=20 > > The operating system I am trying to port to Xen is an industrial > > servo controller. We are currently running at 125 microsecond servo > > update rate on our bespoke hardware (at the moment MIPS64 and ARM) > > (hence the 125 microsecond ideal interrupt period). We will be > > driving EtherCAT servo drives that need to be updated at 500 > > microseconds (hence at least 500 microsecond interrupt period). If > > we can achieve this on a single core ARM11 clocked at 532 MHz, it > > must be achievable on PC hardware. As I said before the idea is to > > dedicate a CPU core to this with all other functions running on the > > other cores. > > =20 > > Luckily we can accept a reasonable amount of jitter on the EtherCAT > > network as we can hand over clock synchronisation to a slave. This > > gives us a little leeway. > > =20 > > Regards. > > =20 >=20 > You should investigate using the arinc653 scheduler in Xen, which is a > realtime scheduler, and might be more appropriate for your usecase. >=20 Right, or even SEDF, if only it were functional! :-/ That's another area where we really need to do better. Luckily enough, we've got Roland, Joshua and Drek (from GVSU and HsH) working on trying to 'resurrect' the SEDF scheduler. Coming to Simon's situation, I think that, as it has already been said, if you can partition the system in such a way that the real-time VM/OS gets one full core, without any other interference (either via cpupool or pinning), there is few point in changing scheduler or scheduling parameters, and that's probably the setup that would guarantee the most reliable performances. If that is not the case, you sure should check arinc, which is probably a rather "static" algorithm, but I'm quite ignorante (sigh) about how it actually works, so I can't say anything about it (and I see Nathan is on it already, so you're in good hands :-)). If you need something more advanced, while waiting for SEDF to be 'restored', you perhaps can check at RT-Xen (Sisu, one of the main developers, is also in Cc). Basically, with that, you can enable some SEDF-alike schedulers, with even more complex and advanced functionalities. https://sites.google.com/site/realtimexen/ These schedulers allow you to specify, for each vcpu, a period and a certain amount of cpu time it will get every period (right Sisu?). This may turn out to be handy in your case, because you then will have the guarantee that the vcpu will get a chance to run at least once every time units. At this point coming up with a period that guarantees the time granularity that you need shouldn't be too hard, especially is the system is after all pretty simple, as it looks like. Of course, it remains to be verified whether the scheduling parameters can tolerate being set at the small values that your workload requires... But you cannot get along with a RT-ish system without doing some measurements, right? :-) Regards, Dario > ~Andrew >=20 >=20 > > ------ Original Message ------ > > From: "Andrew Cooper" > > To: "Simon Martin" ; "xen-devel@lists.xen.org" > > > > Sent: 14/11/2013 18:39:21 > > Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer > > =20 > > > On 14/11/2013 21:18, Simon Martin wrote: > > >=20 > > > > Hi all, > > > > =20 > > > > I need a periodic timer running at ideally at 125 microseconds > > > > and at least 500 microseconds. I've just found the > > > > VCPUOP_set_periodic_timer, however there is a comment saying > > > > "periods less than one millisecond may not be supported". > > > > =20 > > > > I will be running on an x64 machine. Is this supported? If not, > > > > is there any alternate means of generating a fast interrupt? > > > > =20 > > > > Regards. > > >=20 > > > What is the usecase here? 125us is very short indeed. Xen > > > certainly cant guarantee anything more accurate than 50us. Unless > > > the affected vcpu is running uncontested on the hardware, there is > > > very little chance that the vcpu will indeed be woken up again in > > > 125us. > > >=20 > > > It sounds as if you are looking for some pseudo realtime system, > > > at which point you might want to consider a different scheduler. > > >=20 > > > ~Andrew > > >=20 >=20 > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-OUPr11vrOfzC5kEPYjeg Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKGjcEACgkQk4XaBE3IOsSNCgCfadrlRuvo0MpzgKpqFqFeG/iM 514AoKiSpi+0+alk7AC9MJFNJ8XzsOiS =yJrF -----END PGP SIGNATURE----- --=-OUPr11vrOfzC5kEPYjeg-- --===============4899604362579190209== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4899604362579190209==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: VCPUOP_set_periodic_timer Date: Sat, 16 Nov 2013 20:37:49 +0000 Message-ID: References: <1384549825.3896.169.camel@Abyss> Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4330673627768364558==" Return-path: In-Reply-To: <1384549825.3896.169.camel@Abyss> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dario Faggioli , Andrew Cooper Cc: Roland Heusser , Sisu Xi , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --===============4330673627768364558== Content-Type: multipart/related; boundary="=_wombat.nemo.cl-14759-1384634317-0001-2" This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_wombat.nemo.cl-14759-1384634317-0001-2 Content-Type: multipart/alternative; boundary="=_wombat.nemo.cl-14759-1384634317-0001-3" This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_wombat.nemo.cl-14759-1384634317-0001-3 Content-Type: text/plain; format=flowed; charset=utf-8 Content-Transfer-Encoding: quoted-printable >Coming to Simon's situation, I think that, as it has already been said, >if you can partition the system in such a way that the real-time VM/OS >gets one full core, without any other interference (either via cpupool >or pinning), there is few point in changing scheduler or scheduling >parameters, and that's probably the setup that would guarantee the most >reliable performances. This is the configuration I am working on. The system is reasonably=20 simple, a single flat memory space with it's own simple scheduler, so I=20 can't see why it shouldn't work, as long as I can get the granularity=20 out of the underlying system. > >If that is not the case, you sure should check arinc, which is probably >a rather "static" algorithm, but I'm quite ignorante (sigh) about how=20 >it >actually works, so I can't say anything about it (and I see Nathan is=20 >on >it already, so you're in good hands ). In this kind of environment, a static algorithm is good. It is much=20 better to be predictable than flexible. In an industrial setting where=20 this is controlling machinery that can potentially kill the operators=20 the principle of least surprise is what you need. > >If you need something more advanced, while waiting for SEDF to be >'restored', you perhaps can check at RT-Xen (Sisu, one of the main >developers, is also in Cc). Basically, with that, you can enable some >SEDF-alike schedulers, with even more complex and advanced >functionalities. Before I embarked on this I looked at RealTime Xen. Very interesting=20 project, however from my understanding of the Xen architecture I thought= =20 that the vanilla Xen would be sufficient. This is still my fallback=20 position. > >Of course, it remains to be verified whether the scheduling parameters >can tolerate being set at the small values that your workload >requires... But you cannot get along with a RT-ish system without doing >some measurements, right? This is what is missing. There is a steep learning curve on how to write= =20 a PV guest. The mini-os is a good starting point, and so is David=20 Chisnall's book, however in many ways the mini-os is overkill for what I= =20 want as a proof of concept where I want to do the bare minimum to=20 interface with Xen and just work on the fixed flat memory space that it=20 initialises with. Once I have this working smoothly I'll look at timings, however I really= =20 needed to know whether what I am asking is feasible or not. From all the= =20 answers I think that it is more than feasible, not only due to the Xen=20 infrastructure, but also because of the good will of the people=20 involved. Thanks very much. --=_wombat.nemo.cl-14759-1384634317-0001-3 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Coming to Simon's situation, I think that, as it has already been said= ,

if you can partition the system in such a way that the real-time VM/OS=

gets one full core, without any other interference (either via cpupool=

or pinning), there is few point in changing scheduler or scheduling=

parameters, and that's probably the setup that would guarantee the = most

reliable performances.

This is the configuration I am working on. The system is rea= sonably simple, a single flat memory space with it's own simple scheduler,= so I can't see why it shouldn't work, as long as I can get the granularity= out of the underlying system.

If that is not the case, you sure should check arinc, which is probabl= y

a rather "static" algorithm, but I'm quite ignorante (sigh) about how= it

actually works, so I can't say anything about it (and I see Nathan = is on

it already, so you're in good hands ).=

In this kind of environment, a static algorithm is good. It is= much better to be predictable than flexible. In an industrial setting wher= e this is controlling machinery that can potentially kill the operators = the principle of least surprise is what you need.

If you need something more advanced, while waiting for SEDF to be
'restored', you perhaps can check at RT-Xen (Sisu, one of the main =

developers, is also in Cc). Basically, with that, you can enable some=

SEDF-alike schedulers, with even more complex and advanced

functionalities.

Before I embarked on this I looked at RealTime Xen. Very interest= ing project, however from my understanding of the Xen architecture I though= t that the vanilla Xen would be sufficient. This is still my fallback posit= ion.

Of course, it remains to be verified whether the scheduling parameters=

can tolerate being set at the small values that your workload

requires... But you cannot get along with a RT-ish system without doin= g

some measurements, right?

This is what is missing. There is a steep learning curve on how= to write a PV guest. The mini-os is a good starting point, and so is David= Chisnall's book, however in many ways the mini-os is overkill for what = I want as a proof of concept where I want to do the bare minimum to interfa= ce with Xen and just work on the fixed flat memory space that it initialise= s with.

Once I have this working smoothly I'll look at timings, however I real= ly needed to know whether what I am asking is feasible or not. From all = the answers I think that it is more than feasible, not only due to the Xen= infrastructure, but also because of the good will of the people involved.<= /DIV>

Thanks very much.

--=_wombat.nemo.cl-14759-1384634317-0001-3-- --=_wombat.nemo.cl-14759-1384634317-0001-2 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-Id: iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAJXSURBVDhPnZBPSJNxGMefQ5d2SDpUdCkIVl26tIiKUsOgWLEO6WHuUAgtoogORVDZ IQwkCLpIvEqZSvjiIZcQSb3OmZvCHCKLoQ7McKi1rXzpH72v2/v0fH9bq4OnBt993+f5fp7vCy8t dBNleoiWe4k+6uTK9pFPpIkMUbzsmH3IwYHHHTOXHsoFbpH+Jbxv1vpw1yzmu2z++tyBY8YeObi1 CtyL+vqhH6kLOc5rzFlRros531NyzLJHDg58pWDuMbnmO0n/ngzklkb8/KZ1h9Ji9BI7y4+U/9kh Bwced6og3U6+xYHdaef9RV4yvJx5fZozRj0vRwLszDcrx4w9cnDgcacKUm2kfZusN4tTp9gcPc6F gsWeoMZm7AwXU+eUY8YeOTjwuFMFUw/JsN757UK8jlfCRxQY1BK8MlzHhcRJ5ZixV7lw4HGnCuL3 KV5INjirsWouqZZXx6Cj/0hm7MsMeNypgmgLGT/HvbY9coinn+5k++1hUTXbozWi2pJjlv10l+TC gcedKjBuk/bp5UHTGjrA/Tc3sH69il+1bKt8eWjw3nbuu7GR+29VMTjwuFMFoavki7ZuTlvDNZwN 7edQ8yaeScYk+/vDPHBni8rBgcedRETPzpOru4n0mSd7clbkBOcHvTyh7eXIg13q7fBEu4c/yx45 OPC4UwUdAaL2RnJ3nl03lOrw5KyxANsTQbYnL4uulFxm7JGDA4+7SoHmJ2prILdIf3Ft6+xc7zHT DDfav8abHDhm7JGDA79WAeQS+USayBDFy44Ze+SKrxTg7//F9Bs8U4bjEi0nagAAAABJRU5ErkJg gg== --=_wombat.nemo.cl-14759-1384634317-0001-2 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-Id: iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAJXSURBVDhPnZBPSJNxGMefQ5d2SDpUdCkIVl26tIiKUsOgWLEO6WHuUAgtoogORVDZ IQwkCLpIvEqZSvjiIZcQSb3OmZvCHCKLoQ7McKi1rXzpH72v2/v0fH9bq4OnBt993+f5fp7vCy8t dBNleoiWe4k+6uTK9pFPpIkMUbzsmH3IwYHHHTOXHsoFbpH+Jbxv1vpw1yzmu2z++tyBY8YeObi1 CtyL+vqhH6kLOc5rzFlRros531NyzLJHDg58pWDuMbnmO0n/ngzklkb8/KZ1h9Ji9BI7y4+U/9kh Bwced6og3U6+xYHdaef9RV4yvJx5fZozRj0vRwLszDcrx4w9cnDgcacKUm2kfZusN4tTp9gcPc6F gsWeoMZm7AwXU+eUY8YeOTjwuFMFUw/JsN757UK8jlfCRxQY1BK8MlzHhcRJ5ZixV7lw4HGnCuL3 KV5INjirsWouqZZXx6Cj/0hm7MsMeNypgmgLGT/HvbY9coinn+5k++1hUTXbozWi2pJjlv10l+TC gcedKjBuk/bp5UHTGjrA/Tc3sH69il+1bKt8eWjw3nbuu7GR+29VMTjwuFMFoavki7ZuTlvDNZwN 7edQ8yaeScYk+/vDPHBni8rBgcedRETPzpOru4n0mSd7clbkBOcHvTyh7eXIg13q7fBEu4c/yx45 OPC4UwUdAaL2RnJ3nl03lOrw5KyxANsTQbYnL4uulFxm7JGDA4+7SoHmJ2prILdIf3Ft6+xc7zHT DDfav8abHDhm7JGDA79WAeQS+USayBDFy44Ze+SKrxTg7//F9Bs8U4bjEi0nagAAAABJRU5ErkJg gg== --=_wombat.nemo.cl-14759-1384634317-0001-2-- --===============4330673627768364558== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4330673627768364558==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: VCPUOP_set_periodic_timer Date: Mon, 18 Nov 2013 19:28:51 +0100 Message-ID: <1384799331.16918.182.camel@Solace> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2756954294053945444==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============2756954294053945444== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-VahuWBvL9IDQHZpwdcei" --=-VahuWBvL9IDQHZpwdcei Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On sab, 2013-11-16 at 20:37 +0000, Simon Martin wrote: > This is the configuration I am working on. The system is reasonably > simple, a single flat memory space with it's own simple scheduler, so > I can't see why it shouldn't work, as long as I can get the > granularity out of the underlying system. > It indeed should work. It's the fact that the "underlying system" is an hypervisor that makes things fun! :-) There is quite some interest in enabling at least a certain level of real-time behavior in/on top of Xen these days (mostly thanks to the ARM port), which means a lot of people will be looking forward of what you'll be able to achieve and how. > Before I embarked on this I looked at RealTime Xen. Very interesting > project, however from my understanding of the Xen architecture I > thought that the vanilla Xen would be sufficient. This is still my > fallback position. > Yep, it's good to have alternatives. Long term, I think we should aim at both fixing SEDF and having something like RT-Xen merged upstream (people are working on both these projects already). =20 > Once I have this working smoothly I'll look at timings, however I > really needed to know whether what I am asking is feasible or not. > From all the answers I think that it is more than feasible, not only > due to the Xen infrastructure, but also because of the good will of > the people involved. > HeHe... Nice to hear you feel comfortable with us. :-) Keep us posted on how it goes. Thanks and Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-VahuWBvL9IDQHZpwdcei Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKKXGMACgkQk4XaBE3IOsSLAwCghqA2bw8On2SOFnjoN5FjYS8l FloAnju9VcVXx2GmujB9tm6ict/zrbDG =+dIM -----END PGP SIGNATURE----- --=-VahuWBvL9IDQHZpwdcei-- --===============2756954294053945444== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2756954294053945444==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: PV guest timings Date: Tue, 26 Nov 2013 14:50:32 +0000 Message-ID: References: <1384799331.16918.182.camel@Solace> Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2963359528140991402==" Return-path: In-Reply-To: <1384799331.16918.182.camel@Solace> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "xen-devel@lists.xen.org" Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Dario Faggioli , Joshua Whitehead , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============2963359528140991402== Content-Type: multipart/alternative; boundary="------=_MB465DB48F-3A55-4586-9A48-D98CBC65FFDD" --------=_MB465DB48F-3A55-4586-9A48-D98CBC65FFDD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; format=flowed; charset=utf-8 Hi all, I have finally got my little PV guest running. My critical error was not= =20 setting __XEN_INTERFACE_VERSION__. After tearing to pieces the code and=20 put it back together a few times I decided to debug my Makefile script=20 comparing it to Mini-OS. I am now starting to do some simple timings and the numbers are so bad=20 I'm wondering what could be wrong with my test. As mentioned previously I have created a CPU pool with one CPU and one=20 domU. I am using the standard credit scheduler. I set timer_slop=3D0 on=20 the Xen command line. I initialise console, traps, events, and TSC clock in my PV and then=20 start a periodic operation running. I then calculate latency as (clock=20 time - deadline) and period as (clock time - previous deadline).At the=20 moment I'm just displaying min/max values. I also increment a tick count= =20 on every cycle. Looking at the statistics I see that I get the expected number of ticks=20 per second, however I get latencies in the range [-1ms, +25us] and the=20 periods in the range of [3us, 1.02ms]. The upper bounds look OK, but the= =20 lower bounds are all over the place. This makes me think that I'm not the only thing using VIRQ_TIMER. I seem= =20 to remember that this is called nominally every 10ms by the Hypervisor. Is there any way to recognize the origin of the timer? Can anyone suggest other things to try? Regards. --------=_MB465DB48F-3A55-4586-9A48-D98CBC65FFDD Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Hi all,

I have finally got my little PV guest running. My critical error was= not setting __XEN_INTERFACE_VERSION__. After tearing to pieces the code= and put it back together a few times I decided to debug my Makefile script= comparing it to Mini-OS.

I am now starting to do some simple timings and the numbers are so = bad I'm wondering what could be wrong with my test.

As mentioned previously I have created a CPU pool with one CPU and = one domU. I am using the standard credit scheduler. I set timer_slop=3D0= on the Xen command line.

I initialise console, traps, events, and TSC clock in my PV and= then start a periodic operation running. I then calculate latency as (cloc= k time - deadline) and period as (clock time - previous deadline).At the= moment I'm just displaying min/max values. I also increment a tick count on every cycle.

Looking at the statistics I see that I get the expected number= of ticks per second, however I get latencies in the range [-1ms, +25us]= and the periods in the range of [3us, 1.02ms]. The upper bounds look OK,= but the lower bounds are all over the place.

This makes me think that I'm not the only thing using VIRQ_TIMER= . I seem to remember that this is called nominally every 10ms by the Hyperv= isor.

Is there any way to recognize the origin of the timer?

Can anyone suggest other things to try?

Regards.

--------=_MB465DB48F-3A55-4586-9A48-D98CBC65FFDD-- --===============2963359528140991402== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2963359528140991402==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: PV guest timings Date: Tue, 26 Nov 2013 15:11:40 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin , "xen-devel@lists.xen.org" Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Dario Faggioli , Joshua Whitehead , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org On 26/11/2013 14:50, "Simon Martin" wrote: > Hi all, > > I have finally got my little PV guest running. My critical error was not > setting __XEN_INTERFACE_VERSION__. After tearing to pieces the code and put it > back together a few times I decided to debug my Makefile script comparing it > to Mini-OS. > > I am now starting to do some simple timings and the numbers are so bad I'm > wondering what could be wrong with my test. > > As mentioned previously I have created a CPU pool with one CPU and one domU. I > am using the standard credit scheduler. I set timer_slop=0 on the Xen command > line. > > I initialise console, traps, events, and TSC clock in my PV and then start a > periodic operation running. I then calculate latency as (clock time - > deadline) and period as (clock time - previous deadline).At the moment I'm > just displaying min/max values. I also increment a tick count on every cycle. > > Looking at the statistics I see that I get the expected number of ticks per > second, however I get latencies in the range [-1ms, +25us] and the periods in > the range of [3us, 1.02ms]. The upper bounds look OK, but the lower bounds are > all over the place. > > This makes me think that I'm not the only thing using VIRQ_TIMER. I seem to > remember that this is called nominally every 10ms by the Hypervisor. > > Is there any way to recognize the origin of the timer? > > Can anyone suggest other things to try? Are you using the single-shot timer or the periodic timer? How are you calculating clock time? -- Keir > Regards. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: PV guest timings Date: Tue, 26 Nov 2013 15:38:14 +0000 Message-ID: References: Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5484798294758171603==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Keir Fraser , "xen-devel@lists.xen.org" Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Dario Faggioli , Joshua Whitehead , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============5484798294758171603== Content-Type: multipart/alternative; boundary="------=_MB963FD55B-013F-41DD-9E47-1B9F19FAE1F7" --------=_MB963FD55B-013F-41DD-9E47-1B9F19FAE1F7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; format=flowed; charset=utf-8 Hi Keir, >Are you using the single-shot timer or the periodic timer? I'm using the single-shot timer. > >How are you calculating clock time? Copied code in from Mini-OS. To make sure I don't get drift between=20 deadlines as this is a single shot timer I read monotonic_clock at=20 startup and then calculate every deadline by incrementing this register=20 by the period on each timer call. As an additional comment. I decided to filter out the samples with=20 negative latency. This seems to happen 100 times a second, i.e. the 10=20 ms Hypervisor timer event. This improves the timings considerably. So it= =20 looks like I'm on the right track looking for ways to distinguish=20 between timer events. --------=_MB963FD55B-013F-41DD-9E47-1B9F19FAE1F7 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Hi Keir,

Are you using the single-shot timer or the periodic timer?

I'm using the single-shot timer.

How are you calculating clock time?

Copied code in from Mini-OS. To make sure I don't get drift between= deadlines as this is a single shot timer I read monotonic_clock at startup= and then calculate every deadline by incrementing this register by the = period on each timer call.

As an additional comment. I decided to filter out the samples with = negative latency. This seems to happen 100 times a second, i.e. the 10 ms= Hypervisor timer event. This improves the timings considerably. So it look= s like I'm on the right track looking for ways to distinguish between timer= events.

--------=_MB963FD55B-013F-41DD-9E47-1B9F19FAE1F7-- --===============5484798294758171603== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============5484798294758171603==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 00:31:04 +0100 Message-ID: <1385508664.15201.49.camel@Solace> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6911889471780603018==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============6911889471780603018== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-dD7u9Dga7DKK53kW7Ozz" --=-dD7u9Dga7DKK53kW7Ozz Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mar, 2013-11-26 at 14:50 +0000, Simon Martin wrote: > Hi all, > =20 Hi Simon! > I have finally got my little PV guest running. My critical error was > not setting __XEN_INTERFACE_VERSION__. After tearing to pieces the > code and put it back together a few times I decided to debug my > Makefile script comparing it to Mini-OS. > =20 Cool! :-) > I am now starting to do some simple timings and the numbers are so bad > I'm wondering what could be wrong with my test.=20 > Not so cool! :-( =20 > As mentioned previously I have created a CPU pool with one CPU and one > domU. I am using the standard credit scheduler. I set timer_slop=3D0 on > the Xen command line. > =20 > I initialise console, traps, events, and TSC clock in my PV and then > start a periodic operation running. I then calculate latency as (clock > time - deadline) and period as (clock time - previous deadline).At the > moment I'm just displaying min/max values. I also increment a tick > count on every cycle. > =20 > Looking at the statistics I see that I get the expected number of > ticks per second, however I get latencies in the range [-1ms, +25us] > and the periods in the range of [3us, 1.02ms]. The upper bounds look > OK, but the lower bounds are all over the place. > Ok. Well, I'm not super expert in that area (yet), but even for the ones that are, I think it's pretty hard to guess what could be going on without seeing the actual code. Is there any chance that you can share/show it? At least the relevant chunks... Some questions: - "looking at the statistics", what statistics? How do you collect=20 them? - you say you get latencies in some range and periods in some other range. It may be my fault, but I'm not sure I understand what you mean with these two terms in this context, could you clarify? =20 > This makes me think that I'm not the only thing using VIRQ_TIMER. I > seem to remember that this is called nominally every 10ms by the > Hypervisor. > =20 > Is there any way to recognize the origin of the timer? > I'm not sure if fits your exact needs, and you probably know it already, but, just in case, have you looked at xentrace and xenalyze? There's a blog post that could be a nice introduction to them: http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze= / If the exact information you're after is not among the already existing tracepoints, it shouldn't be to hard to add one or more of those, in the spots where you need them. But again, it's very hard to judge without seeing the code... Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-dD7u9Dga7DKK53kW7Ozz Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKVLzgACgkQk4XaBE3IOsTFFgCdHrPrXgC0y8l2YOcIBwJ9a2LH 5XUAn3kW6GtRdog6+3Z/jX1y3EZl5Skk =VIWo -----END PGP SIGNATURE----- --=-dD7u9Dga7DKK53kW7Ozz-- --===============6911889471780603018== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============6911889471780603018==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 00:33:19 +0100 Message-ID: <1385508799.15201.50.camel@Solace> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5486771147919856845==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Keir Fraser , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============5486771147919856845== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-EFb681VzYCpKtim4kWMW" --=-EFb681VzYCpKtim4kWMW Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote: > As an additional comment. I decided to filter out the samples with > negative latency. This seems to happen 100 times a second, i.e. the 10 > ms Hypervisor timer event. > Again, what's "negative latency" ? Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-EFb681VzYCpKtim4kWMW Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKVL78ACgkQk4XaBE3IOsQqfACfQ6XioOg+eHBxdklZP2jLVKA2 QScAoJzaAeY4WIHRjYG85RVPCNzKTKwp =6N3h -----END PGP SIGNATURE----- --=-EFb681VzYCpKtim4kWMW-- --===============5486771147919856845== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============5486771147919856845==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 02:32:01 +0000 Message-ID: References: <1385508799.15201.50.camel@Solace> Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4196795148451328357==" Return-path: In-Reply-To: <1385508799.15201.50.camel@Solace> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dario Faggioli Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Keir Fraser , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============4196795148451328357== Content-Type: multipart/alternative; boundary="------=_MB36EBA0F0-7D8C-46F7-ABBF-11F7BDA15227" --------=_MB36EBA0F0-7D8C-46F7-ABBF-11F7BDA15227 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; format=flowed; charset=utf-8 >On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote: > > >> As an additional comment. I decided to filter out the samples with >> negative latency. This seems to happen 100 times a second, i.e. the=20 >>10 >> ms Hypervisor timer event. >> >Again, what's "negative latency" ? Latency is the time between the deadline and the handler being called.=20 Negative latency means the handler was called before the next deadline. Regards --------=_MB36EBA0F0-7D8C-46F7-ABBF-11F7BDA15227 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote:

As an additional comment. I decided to filter out the samples= with

negative latency. This seems to happen 100 times a second, i.e.= the 10

ms Hypervisor timer event.

Again, what's "negative latency" ?

Latency is the time between the deadline and the handler being called.= Negative latency means the handler was called before the next deadline.

Regards

--------=_MB36EBA0F0-7D8C-46F7-ABBF-11F7BDA15227-- --===============4196795148451328357== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4196795148451328357==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 02:36:59 +0000 Message-ID: References: <1385508664.15201.49.camel@Solace> Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6326657108019929054==" Return-path: In-Reply-To: <1385508664.15201.49.camel@Solace> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dario Faggioli Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============6326657108019929054== Content-Type: multipart/alternative; boundary="------=_MBCEC41DB3-F183-4116-B8DE-5D909EF83FD1" --------=_MBCEC41DB3-F183-4116-B8DE-5D909EF83FD1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; format=flowed; charset=utf-8 >> >> Looking at the statistics I see that I get the expected number of >> ticks per second, however I get latencies in the range [-1ms, +25us] >> and the periods in the range of [3us, 1.02ms]. The upper bounds look >> OK, but the lower bounds are all over the place. >> >Ok. Well, I'm not super expert in that area (yet), but even for the=20 >ones >that are, I think it's pretty hard to guess what could be going on >without seeing the actual code. > >Is there any chance that you can share/show it? At least the relevant >chunks... No problems, I can share the complete code it you want, there is nothing= =20 proprietary in there yet. What's the best way to do it? Put it on=20 github? Send a tarball? >Some questions: > - "looking at the statistics", what statistics? How do you collect > them? At every cycle I compare the current time with the expected time and=20 calculate latency (time from when I expect the event to when it actually= =20 happens) and period. > - you say you get latencies in some range and periods in some other > range. It may be my fault, but I'm not sure I understand what you > mean with these two terms in this context, could you clarify? > Latency =3D difference between expected time and actual time of the event. Period =3D time between two consecutive events. >> This makes me think that I'm not the only thing using VIRQ_TIMER. I >> seem to remember that this is called nominally every 10ms by the >> Hypervisor. >> >> Is there any way to recognize the origin of the timer? >> >I'm not sure if fits your exact needs, and you probably know it=20 >already, >but, just in case, have you looked at xentrace and xenalyze? There's a >blog post that could be a nice introduction to them: >http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyz= e/ > I hadn't read that. I'll look at it tomorrow (it's almost midnight=20 here). >If the exact information you're after is not among the already existing >tracepoints, it shouldn't be to hard to add one or more of those, in=20 >the >spots where you need them. > >But again, it's very hard to judge without seeing the code... I totally understand. Thanks for the effort. > >Regards, >Dario > >-- ><> (Raistlin Majere) >----------------------------------------------------------------- >Dario Faggioli, Ph.D, http://about.me/dario.faggioli >Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) > --------=_MBCEC41DB3-F183-4116-B8DE-5D909EF83FD1 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8



Looking at the statistics I see that I get the expected number= of

ticks per second, however I get latencies in the range [-1ms,= +25us]

and the periods in the range of [3us, 1.02ms]. The upper bounds= look

OK, but the lower bounds are all over the place.

Ok. Well, I'm not super expert in that area (yet), but even for the= ones

that are, I think it's pretty hard to guess what could be going on =

without seeing the actual code.

Is there any chance that you can share/show it? At least the relevant=

chunks...

No problems, I can share the complete code it you want, there is nothi= ng proprietary in there yet. What's the best way to do it? Put it on github= ? Send a tarball?

Some questions:

- "looking at the statistics", what statistics? How do you colle= ct

   them?

At every cycle I compare the current time with the expected time= and calculate latency (time from when I expect the event to when it actual= ly happens) and period.

- you say you get latencies in some range and periods in some= other

   range. It may be my fault, but I'm not sure I unders= tand what you

   mean with these two terms in this context, could = you clarify?

Latency =3D difference between expected time and actual time of= the event.

Period =3D time between two consecutive events.

This makes me think that I'm not the only thing using VIRQ_TIMER= . I

seem to remember that this is called nominally every 10ms by = the

Hypervisor.



Is there any way to recognize the origin of the timer?

I'm not sure if fits your exact needs, and you probably know it alread= y,

but, just in case, have you looked at xentrace and xenalyze? There's= a

blog post that could be a nice introduction to them:

http://blog.xen.org/index.php/2012/09/27/tracing-with-xe= ntrace-and-xenalyze/

I hadn't read that. I'll look at it tomorrow (it's almost midnight = here).

If the exact information you're after is not among the already existin= g

tracepoints, it shouldn't be to hard to add one or more of those, in= the

spots where you need them.

But again, it's very hard to judge without seeing the code...

I totally understand. Thanks for the effort.

Regards,

Dario

--

<<This happens because I choose it to happen!>> (Raistlin= Majere)

-----------------------------------------------------------------
Dario Faggioli, Ph.D, http:= //about.me/dario.faggioli

Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)=

--------=_MBCEC41DB3-F183-4116-B8DE-5D909EF83FD1-- --===============6326657108019929054== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============6326657108019929054==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 09:46:35 +0100 Message-ID: <1385541995.15201.197.camel@Solace> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5809251732361524834==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Keir Fraser , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============5809251732361524834== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-LgumgyNo5KufUEHEfpSr" --=-LgumgyNo5KufUEHEfpSr Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mer, 2013-11-27 at 02:32 +0000, Simon Martin wrote: > =20 > > On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote:=20 > > =20 > > =20 > > > As an additional comment. I decided to filter out the samples > > > with=20 > > > negative latency. This seems to happen 100 times a second, i.e. > > > the 10=20 > > > ms Hypervisor timer event.=20 > > > =20 > > Again, what's "negative latency" ?=20 > > Latency is the time between the deadline and the handler being called. > Ok, that's a reasonable definition of latency. However, if you don't mind another question, 'deadline' here is basically the deadline of the periodic instance x-1, and you expect a timer firing at that time in order to activate instance x? Or is that a proper deadline (and the above is just called period) and you want to have some handler running when it's missed? > Negative latency means the handler was called before the next > deadline. > =20 Oh, wow... I see. So you're saying that you have timers firing early than their expiry time? Is that expected? Why is that happening? :-O Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-LgumgyNo5KufUEHEfpSr Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKVsWsACgkQk4XaBE3IOsS3JQCgjcGg2qie5GgDWnuZnj5PPnST C+sAoIRmwLDxHpoPk8Xns6x48i0vd2VB =Vuij -----END PGP SIGNATURE----- --=-LgumgyNo5KufUEHEfpSr-- --===============5809251732361524834== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============5809251732361524834==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 09:56:26 +0100 Message-ID: <1385542586.15201.204.camel@Solace> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2694789469019738671==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============2694789469019738671== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-CctRwEFA8jsRmp6AdjoJ" --=-CctRwEFA8jsRmp6AdjoJ Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mer, 2013-11-27 at 02:36 +0000, Simon Martin wrote: > > =20 > > Is there any chance that you can share/show it? At least the > > relevant=20 > > chunks...=20 > No problems, I can share the complete code it you want, there is > nothing proprietary in there yet. What's the best way to do it? Put it > on github? Send a tarball? > Well, I think a repo somewhere would be preferrable over a tarball, espacially if you have it in a (although private) repo already, so that we gt to see the history, the commit changelogs, etc. Also, if you go for github, or whatever similar service, people would be able to both checkout your code, or browse it on-line, depending on what they like most. If you don't, I still personally prefer repositories, but, really, whatever you're more comfortable with would do. > > Some questions:=20 > > - "looking at the statistics", what statistics? How do you collect=20 > > them?=20 > At every cycle I compare the current time with the expected time and > calculate latency (time from when I expect the event to when it > actually happens) and period. > =20 Sure, that part I got. I was more asking how you get these numbers out, i.e., you print them online, store somewhere and print later/on request, etc. (I guess I could have specified this more clearly, sorry about that :-) ) > > - you say you get latencies in some range and periods in some > > other=20 > > range. It may be my fault, but I'm not sure I understand what > > you=20 > > mean with these two terms in this context, could you clarify?=20 > > =20 > Latency =3D difference between expected time and actual time of the > event. > =20 > Period =3D time between two consecutive events.=20 > Ok, I think I see now. Just to confirm that I do, this means that the values/ranges you get by measuring both are related, i.e., not completely independent, right? Again, I'm not saying it's not useful to have both, just trying to double check I understood what you're doing. > > I'm not sure if fits your exact needs, and you probably know it > > already,=20 > > but, just in case, have you looked at xentrace and xenalyze? There's > > a=20 > > blog post that could be a nice introduction to them:=20 > > http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xena= lyze/=20 > > =20 > I hadn't read that. I'll look at it tomorrow (it's almost midnight > here). > =20 Ask again if you need anything. Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-CctRwEFA8jsRmp6AdjoJ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKVs7oACgkQk4XaBE3IOsSIdgCfQ2u6LeSjH/lARUQFUAOJwaIL U7EAoI2Kopqh91lFKCPsYd0OPA8WvuV8 =SRYK -----END PGP SIGNATURE----- --=-CctRwEFA8jsRmp6AdjoJ-- --===============2694789469019738671== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2694789469019738671==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Vrabel Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 10:38:43 +0000 Message-ID: <5295CBB3.8090609@cantab.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin , Keir Fraser , "xen-devel@lists.xen.org" Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Dario Faggioli , Joshua Whitehead , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org On 26/11/2013 15:38, Simon Martin wrote: > Hi Keir, > >> Are you using the single-shot timer or the periodic timer? > I'm using the single-shot timer. > >> >> How are you calculating clock time? > Copied code in from Mini-OS. To make sure I don't get drift between > deadlines as this is a single shot timer I read monotonic_clock at > startup and then calculate every deadline by incrementing this register > by the period on each timer call. > > As an additional comment. I decided to filter out the samples with > negative latency. This seems to happen 100 times a second, i.e. the 10 > ms Hypervisor timer event. This improves the timings considerably. So it > looks like I'm on the right track looking for ways to distinguish > between timer events. You should disable the periodic timer if you don't need it. VCPUOP_stop_periodic_timer David From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 12:04:15 +0000 Message-ID: References: <1385541995.15201.197.camel@Solace> Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4378718241731434636==" Return-path: In-Reply-To: <1385541995.15201.197.camel@Solace> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dario Faggioli Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Keir Fraser , Nate Studer List-Id: xen-devel@lists.xenproject.org This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --===============4378718241731434636== Content-Type: multipart/related; boundary="=_wombat.nemo.cl-25000-1385554065-0001-2" This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_wombat.nemo.cl-25000-1385554065-0001-2 Content-Type: multipart/alternative; boundary="=_wombat.nemo.cl-25000-1385554065-0001-3" This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_wombat.nemo.cl-25000-1385554065-0001-3 Content-Type: text/plain; format=flowed; charset=utf-8 Content-Transfer-Encoding: quoted-printable >> Latency is the time between the deadline and the handler being=20 >>called. >> >Ok, that's a reasonable definition of latency. However, if you don't >mind another question, 'deadline' here is basically the deadline of the >periodic instance x-1, and you expect a timer firing at that time in >order to activate instance x? Or is that a proper deadline (and the >above is just called period) and you want to have some handler running >when it's missed? Each time the timer handler is called it calculates the deadline for the= =20 next handler and calls VCPUOP_set_singleshot_timer specifying the=20 deadline. The deadline is calculated as the "previous deadline + the=20 period". So handler instance 'n' will set the deadline for handler=20 instance 'n+1' as "deadline n + period". I'm not sure if that answers the question, but that is what is=20 happening. > >> Negative latency means the handler was called before the next >> deadline. >> >Oh, wow... I see. So you're saying that you have timers firing early >than their expiry time? Is that expected? Why is that happening? Yes. As I said I think this is the 100 Hz Hypervisor periodic timer. I=20 need to be able to distinguish between the different timer interrupt=20 sources. Regards. --=_wombat.nemo.cl-25000-1385554065-0001-3 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

Latency is the time between the deadline and the handler being= called.

Ok, that's a reasonable definition of latency. However, if you don't=

mind another question, 'deadline' here is basically the deadline of= the

periodic instance x-1, and you expect a timer firing at that time in=

order to activate instance x? Or is that a proper deadline (and the=

above is just called period) and you want to have some handler running=

when it's missed?

Each time the timer handler is called it calculates the deadline for= the next handler and calls VCPUOP_set_singleshot_timer specifying the dead= line. The deadline is calculated as the "previous deadline + the period".= So handler instance 'n' will set the deadline for handler instance 'n+1'= as "deadline n + period".

I'm not sure if that answers the question, but that is what is happeni= ng.

  Negative latency means the handler was called before the= next

deadline.



Oh, wow... I see. So you're saying that you have timers firing early=

than their expiry time? Is that expected? Why is that happening?

Yes. As I said I think this is the 100 Hz Hypervisor periodic timer.= I need to be able to distinguish between the different timer interrupt = sources.

Regards.

--=_wombat.nemo.cl-25000-1385554065-0001-3-- --=_wombat.nemo.cl-25000-1385554065-0001-2 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-Id: iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAJeSURBVDhPnZI9aBNxGMbfwaUBcRAEQUGFiIsgxMHPNlJBrRIHmyHJoBSsiEsHxaUu KhKkQwejXouWNkiPDlIFBevlq80HhiKlErDBtmJIY3sxPdSKd03u9f/8E4NDJ4fn3rzP83ufwHH0 ZYSoECYqjRJ9VcmxMkYeIUVIE8o2JnYPcnDgccfM9R+NAqeQWokenDM/3zZq5WGLvz+3MbHDRw5u owJnUW2JrOWu6FxWmFeE9GHmcrg+sQsfOTjwzYL5J+RYHCL152xAX0r4+G1wj1QxeY3t0iM5/3rI wYHHnSzID5Cn+HJf3l64yktaBxcmznNB6+RSPMD24i05scNHDg487mRBLkTKj/edRm3mHBtTp7ha NdnVrbCRusC13CU5scNHDg487mTBTD9p5gefVc2282r0uAS7lWlejbVzdfqsnNjhy1xw4HEnC7L3 KVud9drrqVauy83raejEPxI7/AYDHneyIHmXtF+ZDstKHGErcZStyWNCrWxNtQm56xM7fOSCA487 WaD1krL86rBhRg6xGRNA3M1jNzfz6zs75Zt/c28Xj/dulT5ycOBxJwvGe8iTDG7Lm7E2NqMCmjzD 4Z4W1peLXKlU5FRvbJG+zAUHHney4Nllcox0kfrx6X7djJ9mK+UV/76DPyUfsLX2jRfSIZ4I7pY+ cnDgcScLBgNEA35yDl3cFMkNunQzHRBv3s+Zhwc42reX3z12sSG+BfjIwYHHXbNA8RGFvOQUUl9c 3z43P3rSMKJ+63emy8bEDh85OPAbFUAOIY+QIqQJZRsTO3zkkm8W4PH/YvoDDBGJP7l1I0EAAAAA SUVORK5CYII= --=_wombat.nemo.cl-25000-1385554065-0001-2-- --===============4378718241731434636== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4378718241731434636==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 13:00:51 +0000 Message-ID: References: <1385542586.15201.204.camel@Solace> Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2260375508145872141==" Return-path: In-Reply-To: <1385542586.15201.204.camel@Solace> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dario Faggioli Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --===============2260375508145872141== Content-Type: multipart/related; boundary="=_wombat.nemo.cl-27289-1385557318-0001-2" This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_wombat.nemo.cl-27289-1385557318-0001-2 Content-Type: multipart/alternative; boundary="=_wombat.nemo.cl-27289-1385557318-0001-3" This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_wombat.nemo.cl-27289-1385557318-0001-3 Content-Type: text/plain; format=flowed; charset=utf-8 Content-Transfer-Encoding: quoted-printable ------ Original Message ------ From: "Dario Faggioli" To: "Simon Martin" Cc: "xen-devel@lists.xen.org" ; "Andrew Cooper"= =20 ; "Roland Heusser" ;=20 "Sisu Xi" ; "Joshua Whitehead"=20 ; "Drek Darkover" ; "Nate=20 Studer" Sent: 27/11/2013 05:56:26 Subject: Re: Re[2]: [Xen-devel] PV guest timings >On mer, 2013-11-27 at 02:36 +0000, Simon Martin wrote: > >> > >> > Is there any chance that you can share/show it? At least the >> > relevant >> > chunks... >> No problems, I can share the complete code it you want, there is >> nothing proprietary in there yet. What's the best way to do it? Put=20 >>it >> on github? Send a tarball? >> >Well, I think a repo somewhere would be preferrable over a tarball, >espacially if you have it in a (although private) repo already, so that >we gt to see the history, the commit changelogs, etc. Here's the URL. https://github.com/FurryFuttock/micro-pv. There is no=20 commit history as I mostly use SVN. I just created this and will keep it= =20 up to date as I carry on playing with it. >> > Some questions: >> > - "looking at the statistics", what statistics? How do you collect >> > them? >> At every cycle I compare the current time with the expected time and >> calculate latency (time from when I expect the event to when it >> actually happens) and period. >> >Sure, that part I got. I was more asking how you get these numbers out, >i.e., you print them online, store somewhere and print later/on=20 >request, >etc. (I guess I could have specified this more clearly, sorry about >that ) Here's an example. There are 2 lines per sample. Line 0 has format=20 , line 1 has , timeofday=3D12:56:41 - 1000 1043,47287,955121,2001800 timeofday=3D12:56:42 - 1000 1024,30989,970082,2000885 timeofday=3D12:56:43 - 1000 1042,30888,970167,2000947 timeofday=3D12:56:44 - 1000 1042,30910,970661,2001063 timeofday=3D12:56:45 - 1000 1043,30909,970162,2000057 timeofday=3D12:56:46 - 1000 1043,30883,970183,2000049 timeofday=3D12:56:47 - 1000 1066,30923,970160,2001268 timeofday=3D12:56:48 - 1000 1048,30922,970140,2000031 timeofday=3D12:56:49 - 1000 1041,30864,970211,2001090 timeofday=3D12:56:50 - 1000 1043,30898,970162,2000880 timeofday=3D12:56:51 - 1000 1042,30948,970106,2000947 timeofday=3D12:56:52 - 1000 1044,30777,970752,2000058 > >> > - you say you get latencies in some range and periods in some >> > other >> > range. It may be my fault, but I'm not sure I understand what >> > you >> > mean with these two terms in this context, could you clarify? >> > >> Latency =3D difference between expected time and actual time of the >> event. >> >> Period =3D time between two consecutive events. >> >Ok, I think I see now. Just to confirm that I do, this means that the >values/ranges you get by measuring both are related, i.e., not >completely independent, right? Again, I'm not saying it's not useful to >have both, just trying to double check I understood what you're doing. They're not really related. Period is time between events, Latency is=20 time from event to handler. > >> > I'm not sure if fits your exact needs, and you probably know it >> > already, >> > but, just in case, have you looked at xentrace and xenalyze?=20 >>There's >> > a >> > blog post that could be a nice introduction to them: >> >=20 >>http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenaly= ze/ >> > >> I hadn't read that. I'll look at it tomorrow (it's almost midnight >> here). >> >Ask again if you need anything. I will ! Thanks. Regards. --=_wombat.nemo.cl-27289-1385557318-0001-3 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

------ Original Message ------

From: "Dario Faggioli" <dario.faggioli@citrix.com>

To: "Simon Martin" <smartin= @milliways.cl>

Cc: "xen-devel@lists.xen.or= g" <xen-devel@lists.xen.o= rg>; "Andrew Cooper" <andrew.cooper3@citrix.com>; "Roland Heusser" <heusserr@mail.gvsu.edu>; "Sisu Xi" <xisisu@gmail.com>; "Joshua Whitehe= ad" <whitehej@mail.gvsu.edu>; "Drek Darkover" <wackerei@g= mail.com>; "Nate Studer" <nate.studer@dornerworks.com>

Sent: 27/11/2013 05:56:26

Subject: Re: Re[2]: [Xen-devel] PV guest timings

On mer, 2013-11-27 at 02:36 +0000, Simon Martin wrote:

>

> Is there any chance that you can share/show it? At least= the

> relevant

> chunks...

No problems, I can share the complete code it you want, there= is

nothing proprietary in there yet. What's the best way to do it?= Put it

on github? Send a tarball?

Well, I think a repo somewhere would be preferrable over a tarball,=

espacially if you have it in a (although private) repo already, so = that

we gt to see the history, the commit changelogs, etc.

Here's the URL. h= ttps://github.com/FurryFuttock/micro-pv. There is no commit histor= y as I mostly use SVN. I just created this and will keep it up to date as= I carry on playing with it.



> Some questions:

> - "looking at the statistics", what statistics? How do you= collect

> them?

At every cycle I compare the current time with the expected time= and

calculate latency (time from when I expect the event to when = it

actually happens) and period.



Sure, that part I got. I was more asking how you get these numbers = out,

i.e., you print them online, store somewhere and print later/on reques= t,

etc. (I guess I could have specified this more clearly, sorry about=

that )

Here's an example. There are 2 lines per sample. Line 0 has forma= t <time> <ticks per second>, line 1 has <latency min/max>= ,<period min/max>

timeofday=3D12:56:41 - 1000
1043,47287,955121,2001800
timeofday= =3D12:56:42 - 1000
1024,30989,970082,2000885
timeofday=3D12:56:43 = - 1000
1042,30888,970167,2000947
timeofday=3D12:56:44 - 1000
1042,= 30910,970661,2001063
timeofday=3D12:56:45 - 1000
1043,30909,970162,20= 00057
timeofday=3D12:56:46 - 1000
1043,30883,970183,2000049
timeof= day=3D12:56:47 - 1000
1066,30923,970160,2001268
timeofday=3D12:56:48= - 1000
1048,30922,970140,2000031
timeofday=3D12:56:49 - 1000
1041= ,30864,970211,2001090
timeofday=3D12:56:50 - 1000
1043,30898,970162,2= 000880
timeofday=3D12:56:51 - 1000
1042,30948,970106,2000947
timeo= fday=3D12:56:52 - 1000
1044,30777,970752,2000058

> - you say you get latencies in some range and periods in= some

> other

> range. It may be my fault, but I'm not sure I understand= what

> you

> mean with these two terms in this context, could you clarif= y?

>

Latency =3D difference between expected time and actual time = of the

event.



Period =3D time between two consecutive events.

Ok, I think I see now. Just to confirm that I do, this means that the=

values/ranges you get by measuring both are related, i.e., not

completely independent, right? Again, I'm not saying it's not useful= to

have both, just trying to double check I understood what you're doing.=

They're not really related. Period is time between events, Latency = is time from event to handler.

> I'm not sure if fits your exact needs, and you probably = know it

> already,

> but, just in case, have you looked at xentrace and xenalyze= ? There's

> a

> blog post that could be a nice introduction to them:

> http://blog.xen.org/index.php/2012/09/27/trac= ing-with-xentrace-and-xenalyze/

>

I hadn't read that. I'll look at it tomorrow (it's almost midnig= ht

here).



Ask again if you need anything.

I will ! Thanks.

Regards.

--=_wombat.nemo.cl-27289-1385557318-0001-3-- --=_wombat.nemo.cl-27289-1385557318-0001-2 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-Id: iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAJXSURBVDhPnZBPSJNxGMefQ5d2SDpUdCkIVl26tIiKUsOgWLEO6WHuUAgtoogORVDZ IQwkCLpIvEqZSvjiIZcQSb3OmZvCHCKLoQ7McKi1rXzpH72v2/v0fH9bq4OnBt993+f5fp7vCy8t dBNleoiWe4k+6uTK9pFPpIkMUbzsmH3IwYHHHTOXHsoFbpH+Jbxv1vpw1yzmu2z++tyBY8YeObi1 CtyL+vqhH6kLOc5rzFlRros531NyzLJHDg58pWDuMbnmO0n/ngzklkb8/KZ1h9Ji9BI7y4+U/9kh Bwced6og3U6+xYHdaef9RV4yvJx5fZozRj0vRwLszDcrx4w9cnDgcacKUm2kfZusN4tTp9gcPc6F gsWeoMZm7AwXU+eUY8YeOTjwuFMFUw/JsN757UK8jlfCRxQY1BK8MlzHhcRJ5ZixV7lw4HGnCuL3 KV5INjirsWouqZZXx6Cj/0hm7MsMeNypgmgLGT/HvbY9coinn+5k++1hUTXbozWi2pJjlv10l+TC gcedKjBuk/bp5UHTGjrA/Tc3sH69il+1bKt8eWjw3nbuu7GR+29VMTjwuFMFoavki7ZuTlvDNZwN 7edQ8yaeScYk+/vDPHBni8rBgcedRETPzpOru4n0mSd7clbkBOcHvTyh7eXIg13q7fBEu4c/yx45 OPC4UwUdAaL2RnJ3nl03lOrw5KyxANsTQbYnL4uulFxm7JGDA4+7SoHmJ2prILdIf3Ft6+xc7zHT DDfav8abHDhm7JGDA79WAeQS+USayBDFy44Ze+SKrxTg7//F9Bs8U4bjEi0nagAAAABJRU5ErkJg gg== --=_wombat.nemo.cl-27289-1385557318-0001-2 Content-Type: image/png Content-Transfer-Encoding: base64 Content-Disposition: inline Content-Id: iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAJXSURBVDhPnZBPSJNxGMefQ5d2SDpUdCkIVl26tIiKUsOgWLEO6WHuUAgtoogORVDZ IQwkCLpIvEqZSvjiIZcQSb3OmZvCHCKLoQ7McKi1rXzpH72v2/v0fH9bq4OnBt993+f5fp7vCy8t dBNleoiWe4k+6uTK9pFPpIkMUbzsmH3IwYHHHTOXHsoFbpH+Jbxv1vpw1yzmu2z++tyBY8YeObi1 CtyL+vqhH6kLOc5rzFlRros531NyzLJHDg58pWDuMbnmO0n/ngzklkb8/KZ1h9Ji9BI7y4+U/9kh Bwced6og3U6+xYHdaef9RV4yvJx5fZozRj0vRwLszDcrx4w9cnDgcacKUm2kfZusN4tTp9gcPc6F gsWeoMZm7AwXU+eUY8YeOTjwuFMFUw/JsN757UK8jlfCRxQY1BK8MlzHhcRJ5ZixV7lw4HGnCuL3 KV5INjirsWouqZZXx6Cj/0hm7MsMeNypgmgLGT/HvbY9coinn+5k++1hUTXbozWi2pJjlv10l+TC gcedKjBuk/bp5UHTGjrA/Tc3sH69il+1bKt8eWjw3nbuu7GR+29VMTjwuFMFoavki7ZuTlvDNZwN 7edQ8yaeScYk+/vDPHBni8rBgcedRETPzpOru4n0mSd7clbkBOcHvTyh7eXIg13q7fBEu4c/yx45 OPC4UwUdAaL2RnJ3nl03lOrw5KyxANsTQbYnL4uulFxm7JGDA4+7SoHmJ2prILdIf3Ft6+xc7zHT DDfav8abHDhm7JGDA79WAeQS+USayBDFy44Ze+SKrxTg7//F9Bs8U4bjEi0nagAAAABJRU5ErkJg gg== --=_wombat.nemo.cl-27289-1385557318-0001-2-- --===============2260375508145872141== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2260375508145872141==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: PV guest timings Date: Wed, 27 Nov 2013 14:07:11 +0000 Message-ID: References: <5295CBB3.8090609@cantab.net> Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1787283152748637630==" Return-path: In-Reply-To: <5295CBB3.8090609@cantab.net> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: David Vrabel , Keir Fraser , "xen-devel@lists.xen.org" Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Dario Faggioli , Joshua Whitehead , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============1787283152748637630== Content-Type: multipart/alternative; boundary="------=_MB35C99BEF-614A-4CFB-B038-CF4D48E04036" --------=_MB35C99BEF-614A-4CFB-B038-CF4D48E04036 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; format=flowed; charset=utf-8 >You should disable the periodic timer if you don't need it. > >VCPUOP_stop_periodic_timer > >David Perfect. Thanks David, that got rid of my strange counts. I now only=20 have one VIRQ_TIMER source. Regards. --------=_MB35C99BEF-614A-4CFB-B038-CF4D48E04036 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

You should disable the periodic timer if you don't need it.

VCPUOP_stop_periodic_timer

David

Perfect. Thanks David, that got rid of my strange counts. I now only= have one VIRQ_TIMER source.

Regards.
--------=_MB35C99BEF-614A-4CFB-B038-CF4D48E04036-- --===============1787283152748637630== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============1787283152748637630==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: PV guest timings Date: Thu, 28 Nov 2013 12:16:40 +0100 Message-ID: <1385637400.20797.84.camel@Abyss> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4262477613050033886==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Simon Martin Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============4262477613050033886== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-BqBIEYoZySpqN6hHqx4a" --=-BqBIEYoZySpqN6hHqx4a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mer, 2013-11-27 at 13:00 +0000, Simon Martin wrote: ------ Original Message ------ > From: "Dario Faggioli" > > >=20 > > Well, I think a repo somewhere would be preferrable over a tarball,=20 > > espacially if you have it in a (although private) repo already, so > > that=20 > > we gt to see the history, the commit changelogs, etc.=20 > Here's the URL. https://github.com/FurryFuttock/micro-pv. There is no > commit history as I mostly use SVN. I just created this and will keep > it up to date as I carry on playing with it. > Once more, thanks for this. It looks really interesting and, if you ask me, I'll sure it will be useful! I took a quick look at it, and added to my TODO list to properly look into it. Hopefully, it won't take that long. :-) > > Sure, that part I got. I was more asking how you get these numbers > > out,=20 > > i.e., you print them online, store somewhere and print later/on > > request,=20 > > etc. (I guess I could have specified this more clearly, sorry about=20 > > that :-) )=20 > Here's an example. There are 2 lines per sample. Line 0 has format > , line 1 has , min/max> > =20 > timeofday=3D12:56:41 - 1000 > 1043,47287,955121,2001800 > timeofday=3D12:56:42 - 1000 > 1024,30989,970082,2000885 > timeofday=3D12:56:43 - 1000 > 1042,30888,970167,2000947 > ... > Ok, I was wondering whether, e.g., the printing could interfere with the measurements, and stuff like that, but I checked this out in your code, and it doesn't look like it is happening. > > Ok, I think I see now. Just to confirm that I do, this means that > > the=20 > > values/ranges you get by measuring both are related, i.e., not=20 > > completely independent, right? Again, I'm not saying it's not useful > > to=20 > > have both, just trying to double check I understood what you're > > doing.=20 > They're not really related. Period is time between events, Latency is > time from event to handler. > "Period is time between events", what events? I was thinking the events to be two subsequent invocation of the handler. In an ideal world, that would be exactly equal to the PERIOD that you choose, which also mean latency would be 0. OTOH, if you get some latency in invoking the handler, you'll see different values for the actual time difference between different invocations... Otherwise, how could period vary over time? I mean, if at t0 you set the next event to occur at t1=3Dt0+T, still in the ideal world, you'll get the event both triggered and handled exactly at t1, i.e., period: p1=3Dt0+T-t0=3DT, latency: l1=3Dt1-t1=3D0. If, OTOH, you get some latency, i.e., you are processing the event in the handler at t1'=3Dt0+T+l1, that means latency: t1'-t1=3Dt0+T+l1-t0-T=3Dl1, and that also makes the distance between the two handling event p1'=3Dt1'-t0=3Dt0+T+l1-t0=3DT+l1. Going further on, you get t2=3Dt1+T=3Dt0+T+T=3Dt0+2T t2'=3Dt2+l2 and thus p2'=3Dt2'-t1'=3Dt0+2T+l2-t0-T-l1=3DT+l2-l1 That's what I meant with "related". Checking the code (but very quickly, so bear with me if I'm getting something wrong), _BUT_, I totally understand that this relationship is only useful if you perform the calculation a bit differently, and the way you're doing them allows you to monitor both latency and jitter good enough anyway, so I'm cool with it. :-) Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-BqBIEYoZySpqN6hHqx4a Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKXJhgACgkQk4XaBE3IOsRfaQCfa+jLyS1sFtbimVQtzoPQZtMF w5oAoJRH3u0pA2T3Yk5ms+QxNYexn9up =SDt7 -----END PGP SIGNATURE----- --=-BqBIEYoZySpqN6hHqx4a-- --===============4262477613050033886== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4262477613050033886==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Simon Martin" Subject: Re: PV guest timings Date: Thu, 28 Nov 2013 11:57:52 +0000 Message-ID: References: <1385637400.20797.84.camel@Abyss> Reply-To: Simon Martin Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8346425723979536700==" Return-path: In-Reply-To: <1385637400.20797.84.camel@Abyss> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Dario Faggioli Cc: Roland Heusser , Sisu Xi , Andrew Cooper , Joshua Whitehead , "xen-devel@lists.xen.org" , Drek Darkover , Nate Studer List-Id: xen-devel@lists.xenproject.org --===============8346425723979536700== Content-Type: multipart/related; boundary="------=_MBC21F3AE3-0969-4401-A51A-626B22D31C33" --------=_MBC21F3AE3-0969-4401-A51A-626B22D31C33 Content-Type: multipart/alternative; boundary="------=_MB53047D67-6560-4F80-8538-8ADF8D7D1E1C" --------=_MB53047D67-6560-4F80-8538-8ADF8D7D1E1C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; format=flowed; charset=utf-8 >Once more, thanks for this. It looks really interesting and, if you ask >me, I'll sure it will be useful! > >I took a quick look at it, and added to my TODO list to properly look >into it. Hopefully, it won't take that long. Thanks. All feedback is welcome. > >> >> timeofday=3D12:56:41 - 1000 >> 1043,47287,955121,2001800 >> timeofday=3D12:56:42 - 1000 >> 1024,30989,970082,2000885 >> timeofday=3D12:56:43 - 1000 >> 1042,30888,970167,2000947 >> ... >> >Ok, I was wondering whether, e.g., the printing could interfere with=20 >the >measurements, and stuff like that, but I checked this out in your code, >and it doesn't look like it is happening. It shouldn't. I calculate numbers in the periodic interrupt and then=20 print them from the normal run loop. > >> > Ok, I think I see now. Just to confirm that I do, this means that >> > the >> > values/ranges you get by measuring both are related, i.e., not >> > completely independent, right? Again, I'm not saying it's not=20 >>useful >> > to >> > have both, just trying to double check I understood what you're >> > doing. >> They're not really related. Period is time between events, Latency is >> time from event to handler. >> >"Period is time between events", what events? As you say, events in this context are subsequent invocations of the=20 handler. >In an ideal world, that >would be exactly equal to the PERIOD that you choose, which also mean >latency would be 0. OTOH, if you get some latency in invoking the >handler, you'll see different values for the actual time difference >between different invocations... Otherwise, how could period vary over >time? Not quite. Let us say that latency is always 5 =C2=B5s, no jitter. That = means=20 that the measured period will always be exactly the same, just slightly=20 delayed in time. This is fine. However there IS jitter in the latency (+/- 30 =C2=B5s) so the period varie= s=20 likewise. One event comes in late, the next comes in on time, period is=20 less than expected. One comes in on time the next comes in late, period=20 is longer than expected. > >I mean, if at t0 you set the next event to occur at > > t1=3Dt0+T, > >still in the ideal world, you'll get the event both triggered and >handled exactly at t1, i.e., > > period: p1=3Dt0+T-t0=3DT, > latency: l1=3Dt1-t1=3D0. > >If, OTOH, you get some latency, i.e., you are processing the event in >the handler at > > t1'=3Dt0+T+l1, > >that means > > latency: t1'-t1=3Dt0+T+l1-t0-T=3Dl1, > >and that also makes the distance between the two handling event > > p1'=3Dt1'-t0=3Dt0+T+l1-t0=3DT+l1. > >Going further on, you get > > t2=3Dt1+T=3Dt0+T+T=3Dt0+2T > t2'=3Dt2+l2 > >and thus > > p2'=3Dt2'-t1'=3Dt0+2T+l2-t0-T-l1=3DT+l2-l1 > >That's what I meant with "related". I agree, it is not a direct relationship, but yes they are related. > >Checking the code (but very quickly, so bear with me if I'm getting >something wrong), _BUT_, I totally understand that this relationship is >only useful if you perform the calculation a bit differently, and the >way you're doing them allows you to monitor both latency and jitter=20 >good >enough anyway, so I'm cool with it. I'm glad I haven't got it too wrong at the first attempt. --------=_MB53047D67-6560-4F80-8538-8ADF8D7D1E1C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

Once more, thanks for this. It looks really interesting and, if you= ask

me, I'll sure it will be useful!

I took a quick look at it, and added to my TODO list to properly look=

into it. Hopefully, it won't take that long.

Thanks. All feedback is welcome.



timeofday=3D12:56:41 - 1000

1043,47287,955121,2001800

timeofday=3D12:56:42 - 1000

1024,30989,970082,2000885

timeofday=3D12:56:43 - 1000

1042,30888,970167,2000947

...

Ok, I was wondering whether, e.g., the printing could interfere with= the

measurements, and stuff like that, but I checked this out in your code= ,

and it doesn't look like it is happening.

It shouldn't. I calculate numbers in the periodic interrupt and then= print them from the normal run loop.

> Ok, I think I see now. Just to confirm that I do, this mean= s that

> the

> values/ranges you get by measuring both are related, i.e.,= not

> completely independent, right? Again, I'm not saying it's= not useful

> to

> have both, just trying to double check I understood what= you're

> doing.

They're not really related. Period is time between events, Laten= cy is

time from event to handler.

"Period is time between events", what events?

As you say, events in this context are subsequent invocations of= the handler.

In an ideal world, that

would be exactly equal to the PERIOD that you choose, which also mean=

latency would be 0. OTOH, if you get some latency in invoking the
handler, you'll see different values for the actual time difference=

between different invocations... Otherwise, how could period vary over=

time?

Not quite. Let us say that latency is always 5 =C2=B5s, no jitter. = That means that the measured period will always be exactly the same, just= slightly delayed in time. This is fine.

However there IS jitter in the latency (+/- 30 =C2=B5s) so the period varies likewise. One = event comes in late, the next comes in on time, period is less than expecte= d. One comes in on time the next comes in late, period is longer than expec= ted.

I mean, if at t0 you set the next event to occur at

t1=3Dt0+T,

still in the ideal world, you'll get the event both triggered and
handled exactly at t1, i.e.,

period: p1=3Dt0+T-t0=3DT,

latency: l1=3Dt1-t1=3D0.

If, OTOH, you get some latency, i.e., you are processing the event = in

the handler at

t1'=3Dt0+T+l1,

that means

latency: t1'-t1=3Dt0+T+l1-t0-T=3Dl1,

and that also makes the distance between the two handling event

p1'=3Dt1'-t0=3Dt0+T+l1-t0=3DT+l1.

Going further on, you get

t2=3Dt1+T=3Dt0+T+T=3Dt0+2T

t2'=3Dt2+l2

and thus

p2'=3Dt2'-t1'=3Dt0+2T+l2-t0-T-l1=3DT+l2-l1

That's what I meant with "related".

I agree, it is not a direct relationship, but yes they are related.

Checking the code (but very quickly, so bear with me if I'm getting=

something wrong), _BUT_, I totally understand that this relationship= is

only useful if you perform the calculation a bit differently, and the=

way you're doing them allows you to monitor both latency and jitter= good

enough anyway, so I'm cool with it. =

I'm glad I haven't got it too wrong at the first attempt.

--------=_MB53047D67-6560-4F80-8538-8ADF8D7D1E1C-- --------=_MBC21F3AE3-0969-4401-A51A-626B22D31C33 Content-Disposition: inline Content-Id: Content-Transfer-Encoding: base64 Content-Type: image/png iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAJXSURBVDhPnZBPSJNxGMefQ5d2SDpUdCkIVl26tIiKUsOgWLEO6WHuUAgtoogORVDZ IQwkCLpIvEqZSvjiIZcQSb3OmZvCHCKLoQ7McKi1rXzpH72v2/v0fH9bq4OnBt993+f5fp7vCy8t dBNleoiWe4k+6uTK9pFPpIkMUbzsmH3IwYHHHTOXHsoFbpH+Jbxv1vpw1yzmu2z++tyBY8YeObi1 CtyL+vqhH6kLOc5rzFlRros531NyzLJHDg58pWDuMbnmO0n/ngzklkb8/KZ1h9Ji9BI7y4+U/9kh Bwced6og3U6+xYHdaef9RV4yvJx5fZozRj0vRwLszDcrx4w9cnDgcacKUm2kfZusN4tTp9gcPc6F gsWeoMZm7AwXU+eUY8YeOTjwuFMFUw/JsN757UK8jlfCRxQY1BK8MlzHhcRJ5ZixV7lw4HGnCuL3 KV5INjirsWouqZZXx6Cj/0hm7MsMeNypgmgLGT/HvbY9coinn+5k++1hUTXbozWi2pJjlv10l+TC gcedKjBuk/bp5UHTGjrA/Tc3sH69il+1bKt8eWjw3nbuu7GR+29VMTjwuFMFoavki7ZuTlvDNZwN 7edQ8yaeScYk+/vDPHBni8rBgcedRETPzpOru4n0mSd7clbkBOcHvTyh7eXIg13q7fBEu4c/yx45 OPC4UwUdAaL2RnJ3nl03lOrw5KyxANsTQbYnL4uulFxm7JGDA4+7SoHmJ2prILdIf3Ft6+xc7zHT DDfav8abHDhm7JGDA79WAeQS+USayBDFy44Ze+SKrxTg7//F9Bs8U4bjEi0nagAAAABJRU5ErkJg gg== --------=_MBC21F3AE3-0969-4401-A51A-626B22D31C33 Content-Disposition: inline Content-Id: Content-Transfer-Encoding: base64 Content-Type: image/png iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAARnQU1BAACx jwv8YQUAAAJXSURBVDhPnZBPSJNxGMefQ5d2SDpUdCkIVl26tIiKUsOgWLEO6WHuUAgtoogORVDZ IQwkCLpIvEqZSvjiIZcQSb3OmZvCHCKLoQ7McKi1rXzpH72v2/v0fH9bq4OnBt993+f5fp7vCy8t dBNleoiWe4k+6uTK9pFPpIkMUbzsmH3IwYHHHTOXHsoFbpH+Jbxv1vpw1yzmu2z++tyBY8YeObi1 CtyL+vqhH6kLOc5rzFlRros531NyzLJHDg58pWDuMbnmO0n/ngzklkb8/KZ1h9Ji9BI7y4+U/9kh Bwced6og3U6+xYHdaef9RV4yvJx5fZozRj0vRwLszDcrx4w9cnDgcacKUm2kfZusN4tTp9gcPc6F gsWeoMZm7AwXU+eUY8YeOTjwuFMFUw/JsN757UK8jlfCRxQY1BK8MlzHhcRJ5ZixV7lw4HGnCuL3 KV5INjirsWouqZZXx6Cj/0hm7MsMeNypgmgLGT/HvbY9coinn+5k++1hUTXbozWi2pJjlv10l+TC gcedKjBuk/bp5UHTGjrA/Tc3sH69il+1bKt8eWjw3nbuu7GR+29VMTjwuFMFoavki7ZuTlvDNZwN 7edQ8yaeScYk+/vDPHBni8rBgcedRETPzpOru4n0mSd7clbkBOcHvTyh7eXIg13q7fBEu4c/yx45 OPC4UwUdAaL2RnJ3nl03lOrw5KyxANsTQbYnL4uulFxm7JGDA4+7SoHmJ2prILdIf3Ft6+xc7zHT DDfav8abHDhm7JGDA79WAeQS+USayBDFy44Ze+SKrxTg7//F9Bs8U4bjEi0nagAAAABJRU5ErkJg gg== --------=_MBC21F3AE3-0969-4401-A51A-626B22D31C33-- --===============8346425723979536700== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============8346425723979536700==--