From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Shan, Haitao" Subject: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 30 Oct 2007 22:28:09 +0800 Message-ID: <823A93EED437D048963A3697DB0E35DED56B99@pdsmsx414.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C81B01.18CE7E05" Return-path: Content-class: urn:content-classes:message List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. ------_=_NextPart_001_01C81B01.18CE7E05 Content-Type: multipart/alternative; boundary="----_=_NextPart_002_01C81B01.18CE7E05" ------_=_NextPart_002_01C81B01.18CE7E05 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, Keir, This patch adds a new timer mode, in which no missed ticks is calculated. This can be used with latest x86_64 linux guest, since it can pick up missed ticks themselves. <>=20 Best Regards Haitao Shan ------_=_NextPart_002_01C81B01.18CE7E05 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable [PATCH] Add a timer mode that disables pending missed = ticks

Hi, Keir,

This patch adds a new timer mode, in = which no missed ticks is calculated. This can be used with latest x86_64 = linux guest, since it can pick up missed ticks themselves.

= <<no_missed_ticks.patch>>

Best = Regards
Haitao Shan

------_=_NextPart_002_01C81B01.18CE7E05-- ------_=_NextPart_001_01C81B01.18CE7E05 Content-Type: application/octet-stream; name="no_missed_ticks.patch" Content-Transfer-Encoding: base64 Content-Description: no_missed_ticks.patch Content-Disposition: attachment; filename="no_missed_ticks.patch" ZGlmZiAtciA0YjRjNzVjYjZjMGYgeGVuL2FyY2gveDg2L2h2bS92cHQuYwotLS0gYS94ZW4vYXJj aC94ODYvaHZtL3ZwdC5jCVR1ZSBPY3QgMzAgMTE6MDU6MzYgMjAwNyArMDAwMAorKysgYi94ZW4v YXJjaC94ODYvaHZtL3ZwdC5jCVR1ZSBPY3QgMzAgMDU6NDU6NTcgMjAwNyArMDgwMApAQCAtMjUs OCArMjUsMTYgQEAKIAogc3RhdGljIGludCBwdF9zdXBwb3J0X3RpbWVfZnJvemVuKHN0cnVjdCBk b21haW4gKmQpCiB7Ci0gICAgcmV0dXJuIChkLT5hcmNoLmh2bV9kb21haW4ucGFyYW1zW0hWTV9Q QVJBTV9USU1FUl9NT0RFXSA9PSAKLSAgICAgICAgICAgIEhWTVBUTV9kZWxheV9mb3JfbWlzc2Vk X3RpY2tzKTsKKyAgICByZXR1cm4gKCBkLT5hcmNoLmh2bV9kb21haW4ucGFyYW1zW0hWTV9QQVJB TV9USU1FUl9NT0RFXSA9PQorICAgICAgICAgICAgSFZNUFRNX2RlbGF5X2Zvcl9taXNzZWRfdGlj a3MKKyAgICAgICAgICB8fCBkLT5hcmNoLmh2bV9kb21haW4ucGFyYW1zW0hWTV9QQVJBTV9USU1F Ul9NT0RFXSA9PQorICAgICAgICAgICAgSFZNUFRNX2RlbGF5X2Zvcl9taXNzZWRfdGlja3MgKTsK K30KKworc3RhdGljIGludCBwdF9zdXBwb3J0X25vX21pc3NlZF90aWNrcyhzdHJ1Y3QgZG9tYWlu ICpkKQoreworICAgIHJldHVybiAoZC0+YXJjaC5odm1fZG9tYWluLnBhcmFtc1tIVk1fUEFSQU1f VElNRVJfTU9ERV0gPT0KKyAgICAgICAgICAgIEhWTVBUTV9ub19taXNzZWRfdGlja3NfcGVuZGlu Zyk7CiB9CiAKIHN0YXRpYyB2b2lkIHB0X2xvY2soc3RydWN0IHBlcmlvZGljX3RpbWUgKnB0KQpA QCAtMTE1LDcgKzEyMyw4IEBAIHZvaWQgcHRfcmVzdG9yZV90aW1lcihzdHJ1Y3QgdmNwdSAqdikK IAogICAgIGxpc3RfZm9yX2VhY2hfZW50cnkgKCBwdCwgaGVhZCwgbGlzdCApCiAgICAgewotICAg ICAgICBtaXNzZWRfdGlja3MocHQpOworICAgICAgICBpZiAoICFwdF9zdXBwb3J0X25vX21pc3Nl ZF90aWNrcyh2LT5kb21haW4pICkKKyAgICAgICAgICAgIG1pc3NlZF90aWNrcyhwdCk7CiAgICAg ICAgIHNldF90aW1lcigmcHQtPnRpbWVyLCBwdC0+c2NoZWR1bGVkKTsKICAgICB9CiAKQEAgLTEz Niw3ICsxNDUsMTMgQEAgc3RhdGljIHZvaWQgcHRfdGltZXJfZm4odm9pZCAqZGF0YSkKICAgICBp ZiAoICFwdC0+b25lX3Nob3QgKQogICAgIHsKICAgICAgICAgcHQtPnNjaGVkdWxlZCArPSBwdC0+ cGVyaW9kOwotICAgICAgICBtaXNzZWRfdGlja3MocHQpOworICAgICAgICBpZiAoICFwdF9zdXBw b3J0X25vX21pc3NlZF90aWNrcyhwdC0+dmNwdS0+ZG9tYWluKSApCisgICAgICAgICAgICBtaXNz ZWRfdGlja3MocHQpOworICAgICAgICBlbHNlIGlmICggTk9XKCkgLSBwdC0+c2NoZWR1bGVkID49 IDAgICkKKyAgICAgICAgeworICAgICAgICAgICAgcHQtPnBlbmRpbmdfaW50cl9ucisrOworICAg ICAgICAgICAgcHQtPnNjaGVkdWxlZCA9IE5PVygpICsgcHQtPnBlcmlvZDsKKyAgICAgICAgfQog ICAgICAgICBzZXRfdGltZXIoJnB0LT50aW1lciwgcHQtPnNjaGVkdWxlZCk7CiAgICAgfQogCkBA IC0yMzMsNyArMjQ4LDEwIEBAIHZvaWQgcHRfaW50cl9wb3N0KHN0cnVjdCB2Y3B1ICp2LCBzdHJ1 Y3QKICAgICBlbHNlCiAgICAgewogICAgICAgICBwdC0+cGVuZGluZ19pbnRyX25yLS07Ci0gICAg ICAgIHB0LT5sYXN0X3BsdF9ndGltZSArPSBwdC0+cGVyaW9kX2N5Y2xlczsKKyAgICAgICAgaWYg KCBwdF9zdXBwb3J0X25vX21pc3NlZF90aWNrcyh2LT5kb21haW4pICkKKyAgICAgICAgICAgIHB0 LT5sYXN0X3BsdF9ndGltZSA9IGh2bV9nZXRfZ3Vlc3RfdGltZSh2KTsKKyAgICAgICAgZWxzZQor ICAgICAgICAgICAgcHQtPmxhc3RfcGx0X2d0aW1lICs9IHB0LT5wZXJpb2RfY3ljbGVzOwogICAg IH0KIAogICAgIGlmICggcHRfc3VwcG9ydF90aW1lX2Zyb3plbih2LT5kb21haW4pICYmCmRpZmYg LXIgNGI0Yzc1Y2I2YzBmIHhlbi9pbmNsdWRlL3B1YmxpYy9odm0vcGFyYW1zLmgKLS0tIGEveGVu L2luY2x1ZGUvcHVibGljL2h2bS9wYXJhbXMuaAlUdWUgT2N0IDMwIDExOjA1OjM2IDIwMDcgKzAw MDAKKysrIGIveGVuL2luY2x1ZGUvcHVibGljL2h2bS9wYXJhbXMuaAlUdWUgT2N0IDMwIDA1OjEz OjMyIDIwMDcgKzA4MDAKQEAgLTcwLDYgKzcwLDcgQEAKICNkZWZpbmUgSFZNX1BBUkFNX1RJTUVS X01PREUgICAxMAogI2RlZmluZSBIVk1QVE1fZGVsYXlfZm9yX21pc3NlZF90aWNrcyAgICAwCiAj ZGVmaW5lIEhWTVBUTV9ub19kZWxheV9mb3JfbWlzc2VkX3RpY2tzIDEKKyNkZWZpbmUgSFZNUFRN X25vX21pc3NlZF90aWNrc19wZW5kaW5nICAgMgogCiAjZGVmaW5lIEhWTV9OUl9QQVJBTVMgICAg ICAgICAgMTEKIAo= ------_=_NextPart_001_01C81B01.18CE7E05 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ------_=_NextPart_001_01C81B01.18CE7E05-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 30 Oct 2007 16:12:54 +0000 Message-ID: References: <823A93EED437D048963A3697DB0E35DED56B99@pdsmsx414.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0813682813==" Return-path: In-Reply-To: <823A93EED437D048963A3697DB0E35DED56B99@pdsmsx414.ccr.corp.intel.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Shan, Haitao" Cc: Dave Winchell , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --===============0813682813== Content-type: multipart/alternative; boundary="B_3276605575_62589428" > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3276605575_62589428 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Applied as c/s 16274. Please take a look and make sure the mode works as you expect. -- Keir On 30/10/07 14:28, "Shan, Haitao" wrote: > Hi, Keir, > > This patch adds a new timer mode, in which no missed ticks is calculated. This > can be used with latest x86_64 linux guest, since it can pick up missed ticks > themselves. > > <> > > Best Regards > Haitao Shan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel --B_3276605575_62589428 Content-type: text/html; charset="US-ASCII" Content-transfer-encoding: quoted-printable Re: [Xen-devel] [PATCH] Add a timer mode that disables pending misse= d ticks
Applied as c/s 16274. Please take a look and make sure the mode works as yo= u expect.

 -- Keir

On 30/10/07 14:28, "Shan, Haitao" <haitao.shan@intel.com> w= rote:

Hi, Keir,

This patch adds a new timer mode, in which no mi= ssed ticks is calculated. This can be used with latest x86_64 linux guest, s= ince it can pick up missed ticks themselves.

 <<no_missed_ticks.patch>>

Best Regards
Haitao Shan

____________________= ___________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/x= en-devel

 --B_3276605575_62589428-- --===============0813682813== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0813682813==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Shan, Haitao" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 31 Oct 2007 11:10:17 +0800 Message-ID: <823A93EED437D048963A3697DB0E35DED56D05@pdsmsx414.ccr.corp.intel.com> References: <823A93EED437D048963A3697DB0E35DED56B99@pdsmsx414.ccr.corp.intel.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C81B6B.9040EA89" Return-path: Content-class: urn:content-classes:message In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: Dave Winchell , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. ------_=_NextPart_001_01C81B6B.9040EA89 Content-Type: multipart/alternative; boundary="----_=_NextPart_002_01C81B6B.9040EA89" ------_=_NextPart_002_01C81B6B.9040EA89 Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: quoted-printable Hi, Keir, Dave =20 Sorry, I made a silly mistake for my last patch. The attached will fix = it. =20 This piece of code should actually be added in pt_restore_timer rather = than pt_timer_fn. + else if ( (NOW() - pt->scheduled) >=3D 0 ) + { + pt->pending_intr_nr++; + pt->scheduled =3D NOW() + pt->period; + } Best Regards=20 Haitao Shan=20 =20 ________________________________ From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]=20 Sent: 2007=C4=EA10=D4=C231=C8=D5 0:13 To: Shan, Haitao Cc: xen-devel@lists.xensource.com; Dong, Eddie; Jiang, Yunhong; Dave = Winchell Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending = missed ticks Applied as c/s 16274. Please take a look and make sure the mode works as = you expect. -- Keir On 30/10/07 14:28, "Shan, Haitao" wrote: Hi, Keir,=20 =09 This patch adds a new timer mode, in which no missed ticks is = calculated. This can be used with latest x86_64 linux guest, since it = can pick up missed ticks themselves. =09 <>=20 =09 Best Regards=20 Haitao Shan=20 =09 =09 ________________________________ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel =09 ------_=_NextPart_002_01C81B6B.9040EA89 Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable Re: [Xen-devel] [PATCH] Add a timer mode that = disables pending missed ticks
Hi, Keir, Dave
 
Sorry, I made a silly mistake for my last = patch. The=20 attached will fix it.
 
This piece of code should actually be added in=20 pt_restore_timer rather than pt_timer_fn.
+        = else if (=20 (NOW() - pt->scheduled) >=3D 0=20 )
+       =20 {
+            = pt->pending_intr_nr++;
+       &= nbsp;   =20 pt->scheduled =3D NOW() +=20 pt->period;
+       =20 }

Best=20 Regards
Haitao Shan

 

From: Keir Fraser=20 [mailto:Keir.Fraser@cl.cam.ac.uk]
Sent: = 2007=C4=EA10=D4=C231=C8=D5=20 0:13
To: Shan, Haitao
Cc: = xen-devel@lists.xensource.com;=20 Dong, Eddie; Jiang, Yunhong; Dave Winchell
Subject: Re: = [Xen-devel]=20 [PATCH] Add a timer mode that disables pending missed = ticks


Applied as c/s 16274. Please take a look = and make=20 sure the mode works as you expect.

 -- Keir

On = 30/10/07=20 14:28, "Shan, Haitao" <haitao.shan@intel.com> = wrote:
Hi,=20 Keir, =

This patch adds a new timer mode, in which no missed = ticks is=20 calculated. This can be used with latest x86_64 linux guest, since it = can pick=20 up missed ticks themselves.

 <<no_missed_ticks.patch>>=20

Best = Regards
Haitao Shan

_______________________________________________
Xen-devel=20 mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/= xen-devel

 ------_=_NextPart_002_01C81B6B.9040EA89-- ------_=_NextPart_001_01C81B6B.9040EA89 Content-Type: application/octet-stream; name="no_missed_ticks_fix.patch" Content-Transfer-Encoding: base64 Content-Description: no_missed_ticks_fix.patch Content-Disposition: attachment; filename="no_missed_ticks_fix.patch" ZGlmZiAtciA3ZWI2OGQ5OTVhYTcgeGVuL2FyY2gveDg2L2h2bS9odm0uYwotLS0gYS94ZW4vYXJj aC94ODYvaHZtL2h2bS5jCVR1ZSBPY3QgMzAgMTY6MjU6NTggMjAwNyArMDAwMAorKysgYi94ZW4v YXJjaC94ODYvaHZtL2h2bS5jCVR1ZSBPY3QgMzAgMTk6NTM6MzMgMjAwNyArMDgwMApAQCAtMTg0 OSw3ICsxODQ5LDggQEAgbG9uZyBkb19odm1fb3AodW5zaWduZWQgbG9uZyBvcCwgWEVOX0dVRQog ICAgICAgICAgICAgY2FzZSBIVk1fUEFSQU1fVElNRVJfTU9ERToKICAgICAgICAgICAgICAgICBy YyA9IC1FSU5WQUw7CiAgICAgICAgICAgICAgICAgaWYgKCAoYS52YWx1ZSAhPSBIVk1QVE1fZGVs YXlfZm9yX21pc3NlZF90aWNrcykgJiYKLSAgICAgICAgICAgICAgICAgICAgIChhLnZhbHVlICE9 IEhWTVBUTV9ub19kZWxheV9mb3JfbWlzc2VkX3RpY2tzKSApCisgICAgICAgICAgICAgICAgICAg ICAoYS52YWx1ZSAhPSBIVk1QVE1fbm9fZGVsYXlfZm9yX21pc3NlZF90aWNrcykgJiYKKyAgICAg ICAgICAgICAgICAgICAgIChhLnZhbHVlICE9IEhWTVBUTV9ub19taXNzZWRfdGlja19hY2NvdW50 aW5nKSApCiAgICAgICAgICAgICAgICAgICAgIGdvdG8gcGFyYW1fZmFpbDsKICAgICAgICAgICAg ICAgICBicmVhazsKICAgICAgICAgICAgIH0KZGlmZiAtciA3ZWI2OGQ5OTVhYTcgeGVuL2FyY2gv eDg2L2h2bS92cHQuYwotLS0gYS94ZW4vYXJjaC94ODYvaHZtL3ZwdC5jCVR1ZSBPY3QgMzAgMTY6 MjU6NTggMjAwNyArMDAwMAorKysgYi94ZW4vYXJjaC94ODYvaHZtL3ZwdC5jCVR1ZSBPY3QgMzAg MTk6MDQ6MTIgMjAwNyArMDgwMApAQCAtMTE5LDYgKzExOSwxMSBAQCB2b2lkIHB0X3Jlc3RvcmVf dGltZXIoc3RydWN0IHZjcHUgKnYpCiAgICAgewogICAgICAgICBpZiAoICFtb2RlX2lzKHYtPmRv bWFpbiwgbm9fbWlzc2VkX3RpY2tfYWNjb3VudGluZykgKQogICAgICAgICAgICAgcHRfcHJvY2Vz c19taXNzZWRfdGlja3MocHQpOworICAgICAgICBlbHNlIGlmICggKE5PVygpIC0gcHQtPnNjaGVk dWxlZCkgPj0gMCApCisgICAgICAgIHsKKyAgICAgICAgICAgIHB0LT5wZW5kaW5nX2ludHJfbnIr KzsKKyAgICAgICAgICAgIHB0LT5zY2hlZHVsZWQgPSBOT1coKSArIHB0LT5wZXJpb2Q7CisgICAg ICAgIH0KICAgICAgICAgc2V0X3RpbWVyKCZwdC0+dGltZXIsIHB0LT5zY2hlZHVsZWQpOwogICAg IH0KIApAQCAtMTQxLDExICsxNDYsNiBAQCBzdGF0aWMgdm9pZCBwdF90aW1lcl9mbih2b2lkICpk YXRhKQogICAgICAgICBpZiAoICFtb2RlX2lzKHB0LT52Y3B1LT5kb21haW4sIG5vX21pc3NlZF90 aWNrX2FjY291bnRpbmcpICkKICAgICAgICAgewogICAgICAgICAgICAgcHRfcHJvY2Vzc19taXNz ZWRfdGlja3MocHQpOwotICAgICAgICB9Ci0gICAgICAgIGVsc2UgaWYgKCAoTk9XKCkgLSBwdC0+ c2NoZWR1bGVkKSA+PSAwICkKLSAgICAgICAgewotICAgICAgICAgICAgcHQtPnBlbmRpbmdfaW50 cl9ucisrOwotICAgICAgICAgICAgcHQtPnNjaGVkdWxlZCA9IE5PVygpICsgcHQtPnBlcmlvZDsK ICAgICAgICAgfQogICAgICAgICBzZXRfdGltZXIoJnB0LT50aW1lciwgcHQtPnNjaGVkdWxlZCk7 CiAgICAgfQo= ------_=_NextPart_001_01C81B6B.9040EA89 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ------_=_NextPart_001_01C81B6B.9040EA89-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 31 Oct 2007 07:09:42 +0000 Message-ID: References: <47279F1F.6050904@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47279F1F.6050904@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Dong, Eddie" , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org I'm going to apply Haitao's no_missed_ticks_fix.patch. If you think fixes are needed apart from that, please provide a unified diff. -- Keir On 30/10/07 21:16, "Dave Winchell" wrote: > Keir, > > Here are my comments on your change. > I've attached an updated vpt.c with the changes. > > 1. For the no_missed_tick_accounting method, we still need the update > to pt->scheduled taking into account the time that has elapsed when > missed ticks are calculated. missed_ticks (pt_process_missed_ticks) > is called from pt_restore_timer() and pt_timer_fn() and, thus, its > easiest to put the check for no_missed_tick_accounting method in > missed_ticks() itself. > > 2. In pt_timer_fn you don't want to increment pending_intr_nr beyond 1 > for no_missed_tick_accounting option. > > thanks, > Dave > > > Keir Fraser wrote: > >> >> Applied as c/s 16274. Please take a look and make sure the mode works >> as you expect. >> >> -- Keir >> >> On 30/10/07 14:28, "Shan, Haitao" wrote: >> >> Hi, Keir, >> >> This patch adds a new timer mode, in which no missed ticks is >> calculated. This can be used with latest x86_64 linux guest, since >> it can pick up missed ticks themselves. >> >> <> >> >> Best Regards >> Haitao Shan >> >> ------------------------------------------------------------------------ >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >> > > *** vpt.c.new.c 2007-10-30 16:45:26.000000000 -0400 > --- vpt.c 2007-10-30 15:30:57.000000000 -0400 > *************** > *** 57,73 **** > return; > > missed_ticks = missed_ticks / (s_time_t) pt->period + 1; > ! > ! if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { > ! if ( missed_ticks > 1000 ) > ! { > ! /* TODO: Adjust guest time together */ > ! pt->pending_intr_nr++; > ! } > ! else > ! { > ! pt->pending_intr_nr += missed_ticks; > ! } > } > > pt->scheduled += missed_ticks * pt->period; > --- 57,70 ---- > return; > > missed_ticks = missed_ticks / (s_time_t) pt->period + 1; > ! if ( missed_ticks > 1000 ) > ! { > ! /* TODO: Adjust guest time together */ > ! pt->pending_intr_nr++; > ! } > ! else > ! { > ! pt->pending_intr_nr += missed_ticks; > } > > pt->scheduled += missed_ticks * pt->period; > *************** > *** 120,126 **** > > list_for_each_entry ( pt, head, list ) > { > ! pt_process_missed_ticks(pt); > set_timer(&pt->timer, pt->scheduled); > } > > --- 117,124 ---- > > list_for_each_entry ( pt, head, list ) > { > ! if ( !mode_is(v->domain, no_missed_tick_accounting) ) > ! pt_process_missed_ticks(pt); > set_timer(&pt->timer, pt->scheduled); > } > > *************** > *** 135,151 **** > > pt_lock(pt); > > ! if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) { > ! if(!pt->pending_intr_nr) > ! pt->pending_intr_nr++; > ! } > ! else > ! pt->pending_intr_nr++; > > if ( !pt->one_shot ) > { > pt->scheduled += pt->period; > ! pt_process_missed_ticks(pt); > set_timer(&pt->timer, pt->scheduled); > } > > --- 133,152 ---- > > pt_lock(pt); > > ! pt->pending_intr_nr++; > > if ( !pt->one_shot ) > { > pt->scheduled += pt->period; > ! if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) > ! { > ! pt_process_missed_ticks(pt); > ! } > ! else if ( (NOW() - pt->scheduled) >= 0 ) > ! { > ! pt->pending_intr_nr++; > ! pt->scheduled = NOW() + pt->period; > ! } > set_timer(&pt->timer, pt->scheduled); > } > > /* > * vpt.c: Virtual Platform Timer > * > * Copyright (c) 2006, Xiaowei Yang, Intel Corporation. > * > * This program is free software; you can redistribute it and/or modify it > * under the terms and conditions of the GNU General Public License, > * version 2, as published by the Free Software Foundation. > * > * This program is distributed in the hope it will be useful, but WITHOUT > * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or > * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for > * more details. > * > * You should have received a copy of the GNU General Public License along > with > * this program; if not, write to the Free Software Foundation, Inc., 59 > Temple > * Place - Suite 330, Boston, MA 02111-1307 USA. > * > */ > > #include > #include > #include > #include > > #define mode_is(d, name) \ > ((d)->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] == HVMPTM_##name) > > static void pt_lock(struct periodic_time *pt) > { > struct vcpu *v; > > for ( ; ; ) > { > v = pt->vcpu; > spin_lock(&v->arch.hvm_vcpu.tm_lock); > if ( likely(pt->vcpu == v) ) > break; > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > } > } > > static void pt_unlock(struct periodic_time *pt) > { > spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock); > } > > static void pt_process_missed_ticks(struct periodic_time *pt) > { > s_time_t missed_ticks; > > if ( pt->one_shot ) > return; > > missed_ticks = NOW() - pt->scheduled; > if ( missed_ticks <= 0 ) > return; > > missed_ticks = missed_ticks / (s_time_t) pt->period + 1; > > if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { > if ( missed_ticks > 1000 ) > { > /* TODO: Adjust guest time together */ > pt->pending_intr_nr++; > } > else > { > pt->pending_intr_nr += missed_ticks; > } > } > > pt->scheduled += missed_ticks * pt->period; > } > > static void pt_freeze_time(struct vcpu *v) > { > if ( !mode_is(v->domain, delay_for_missed_ticks) ) > return; > > v->arch.hvm_vcpu.guest_time = hvm_get_guest_time(v); > } > > static void pt_thaw_time(struct vcpu *v) > { > if ( !mode_is(v->domain, delay_for_missed_ticks) ) > return; > > if ( v->arch.hvm_vcpu.guest_time == 0 ) > return; > > hvm_set_guest_time(v, v->arch.hvm_vcpu.guest_time); > v->arch.hvm_vcpu.guest_time = 0; > } > > void pt_save_timer(struct vcpu *v) > { > struct list_head *head = &v->arch.hvm_vcpu.tm_list; > struct periodic_time *pt; > > if ( test_bit(_VPF_blocked, &v->pause_flags) ) > return; > > spin_lock(&v->arch.hvm_vcpu.tm_lock); > > list_for_each_entry ( pt, head, list ) > stop_timer(&pt->timer); > > pt_freeze_time(v); > > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > } > > void pt_restore_timer(struct vcpu *v) > { > struct list_head *head = &v->arch.hvm_vcpu.tm_list; > struct periodic_time *pt; > > spin_lock(&v->arch.hvm_vcpu.tm_lock); > > list_for_each_entry ( pt, head, list ) > { > pt_process_missed_ticks(pt); > set_timer(&pt->timer, pt->scheduled); > } > > pt_thaw_time(v); > > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > } > > static void pt_timer_fn(void *data) > { > struct periodic_time *pt = data; > > pt_lock(pt); > > if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) { > if(!pt->pending_intr_nr) > pt->pending_intr_nr++; > } > else > pt->pending_intr_nr++; > > if ( !pt->one_shot ) > { > pt->scheduled += pt->period; > pt_process_missed_ticks(pt); > set_timer(&pt->timer, pt->scheduled); > } > > vcpu_kick(pt->vcpu); > > pt_unlock(pt); > } > > void pt_update_irq(struct vcpu *v) > { > struct list_head *head = &v->arch.hvm_vcpu.tm_list; > struct periodic_time *pt; > uint64_t max_lag = -1ULL; > int irq = -1; > > spin_lock(&v->arch.hvm_vcpu.tm_lock); > > list_for_each_entry ( pt, head, list ) > { > if ( !is_isa_irq_masked(v, pt->irq) && pt->pending_intr_nr && > ((pt->last_plt_gtime + pt->period_cycles) < max_lag) ) > { > max_lag = pt->last_plt_gtime + pt->period_cycles; > irq = pt->irq; > } > } > > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > > if ( is_lvtt(v, irq) ) > { > vlapic_set_irq(vcpu_vlapic(v), irq, 0); > } > else if ( irq >= 0 ) > { > hvm_isa_irq_deassert(v->domain, irq); > hvm_isa_irq_assert(v->domain, irq); > } > } > > static struct periodic_time *is_pt_irq( > struct vcpu *v, struct hvm_intack intack) > { > struct list_head *head = &v->arch.hvm_vcpu.tm_list; > struct periodic_time *pt; > struct RTCState *rtc = &v->domain->arch.hvm_domain.pl_time.vrtc; > int vector; > > list_for_each_entry ( pt, head, list ) > { > if ( !pt->pending_intr_nr ) > continue; > > if ( is_lvtt(v, pt->irq) ) > { > if ( pt->irq != intack.vector ) > continue; > return pt; > } > > vector = get_isa_irq_vector(v, pt->irq, intack.source); > > /* RTC irq need special care */ > if ( (intack.vector != vector) || > ((pt->irq == 8) && !is_rtc_periodic_irq(rtc)) ) > continue; > > return pt; > } > > return NULL; > } > > void pt_intr_post(struct vcpu *v, struct hvm_intack intack) > { > struct periodic_time *pt; > time_cb *cb; > void *cb_priv; > > spin_lock(&v->arch.hvm_vcpu.tm_lock); > > pt = is_pt_irq(v, intack); > if ( pt == NULL ) > { > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > return; > } > > if ( pt->one_shot ) > { > pt->enabled = 0; > list_del(&pt->list); > } > else > { > pt->pending_intr_nr--; > if ( mode_is(v->domain, no_missed_tick_accounting) ) > pt->last_plt_gtime = hvm_get_guest_time(v); > else > pt->last_plt_gtime += pt->period_cycles; > } > > if ( mode_is(v->domain, delay_for_missed_ticks) && > (hvm_get_guest_time(v) < pt->last_plt_gtime) ) > hvm_set_guest_time(v, pt->last_plt_gtime); > > cb = pt->cb; > cb_priv = pt->priv; > > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > > if ( cb != NULL ) > cb(v, cb_priv); > } > > void pt_reset(struct vcpu *v) > { > struct list_head *head = &v->arch.hvm_vcpu.tm_list; > struct periodic_time *pt; > > spin_lock(&v->arch.hvm_vcpu.tm_lock); > > list_for_each_entry ( pt, head, list ) > { > pt->pending_intr_nr = 0; > pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu); > pt->scheduled = NOW() + pt->period; > set_timer(&pt->timer, pt->scheduled); > } > > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > } > > void pt_migrate(struct vcpu *v) > { > struct list_head *head = &v->arch.hvm_vcpu.tm_list; > struct periodic_time *pt; > > spin_lock(&v->arch.hvm_vcpu.tm_lock); > > list_for_each_entry ( pt, head, list ) > migrate_timer(&pt->timer, v->processor); > > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > } > > void create_periodic_time( > struct vcpu *v, struct periodic_time *pt, uint64_t period, > uint8_t irq, char one_shot, time_cb *cb, void *data) > { > destroy_periodic_time(pt); > > spin_lock(&v->arch.hvm_vcpu.tm_lock); > > pt->enabled = 1; > pt->pending_intr_nr = 0; > > /* Periodic timer must be at least 0.9ms. */ > if ( (period < 900000) && !one_shot ) > { > gdprintk(XENLOG_WARNING, > "HVM_PlatformTime: program too small period %"PRIu64"\n", > period); > period = 900000; > } > > pt->period = period; > pt->vcpu = v; > pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu); > pt->irq = irq; > pt->period_cycles = (u64)period * cpu_khz / 1000000L; > pt->one_shot = one_shot; > pt->scheduled = NOW() + period; > /* > * Offset LAPIC ticks from other timer ticks. Otherwise guests which use > * LAPIC ticks for process accounting can see long sequences of process > * ticks incorrectly accounted to interrupt processing. > */ > if ( is_lvtt(v, irq) ) > pt->scheduled += period >> 1; > pt->cb = cb; > pt->priv = data; > > list_add(&pt->list, &v->arch.hvm_vcpu.tm_list); > > init_timer(&pt->timer, pt_timer_fn, pt, v->processor); > set_timer(&pt->timer, pt->scheduled); > > spin_unlock(&v->arch.hvm_vcpu.tm_lock); > } > > void destroy_periodic_time(struct periodic_time *pt) > { > if ( !pt->enabled ) > return; > > pt_lock(pt); > pt->enabled = 0; > list_del(&pt->list); > pt_unlock(pt); > > /* > * pt_timer_fn() can run until this kill_timer() returns. We must do this > * outside pt_lock() otherwise we can deadlock with pt_timer_fn(). > */ > kill_timer(&pt->timer); > } From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 02 Nov 2007 09:40:37 +0000 Message-ID: References: <472A41CA.7080405@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <472A41CA.7080405@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Dong, Eddie" , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org On 1/11/07 21:14, "Dave Winchell" wrote: > 1. Increase missed ticks threshold to 100 seconds. Per the comment in > the patch, I assume that's the magic number 100000 in the patch? I think removing the check entirely would be better than an arbitrarily high threshold that we're baiscally hoping will never trigger. > 3. Call pt_process_missed_ticks() unconditionally and place the > test for no_missed_tick_accounting inside pt_process_missed_ticks(). > This returns the calculation for the next timer expiration > to the original method. The original method is important > as it sets up a theoretical time space at t=0 where expirations > occur only at n*period, where n is any integer. This, in turn, removes > rounding errors. Why do 'rounding errors' matter? I thought that no_missed_tick_accounting was for guests which sampled the TSC on tick interrupt and used that to determine elapsed wall time, in which case why would it matter when exactly the tick interrupt is delivered? -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 02 Nov 2007 16:14:50 +0000 Message-ID: References: <472B4772.8070109@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <472B4772.8070109@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Dong, Eddie" , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Thanks for the explanation. I think you are mis-describing the current behaviour of vpt.c though (If I understand it correctly myself, which I may not!). As I understand it, we do *not* unconditionally deliver a tick and reset the time space when a vcpu is scheduled onto a cpu. We only do that if (NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has passed. Otherwise we wait to deliver the next tick exactly one period after the previous tick was delivered. The remainder of your explanation seems to be predicated on the (impossible?) case that we deliver a tick less than one period after the previous one. Am I confused? :-) Also, what version of Linux are we talking about here, and what periodic timer, and x86/64 or i386? There are lots of different Linux timer-handling logics out there in the wild! -- Keir On 2/11/07 15:51, "Dave Winchell" wrote: > Let D be the time that the clock vcpu is descheduled and P be > the clock period. When D < P, I think there is an issue. > > The reason is that Linux's offset calculation, which affects the > last clock interrupt tsc that is recorded, doesn't kick in if the > time since the last interrupt is less than P. In this case it sets > offset to zero. Linux records the tsc of the current (last) clock interrupt > as (current tsc - offset). > > Interrupt delivery method AS delivers a clock interrupt > at context switch (in) time. When D < P, Linux records the > interrupt delivery time as current tsc and bumps jiffies. > This results in a gain of time equal to P-D over wall time. > > For method S this never happens because the interrupt isn't > delivered until the next boundary. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 02 Nov 2007 16:35:37 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser , Dave Winchell Cc: "Shan, Haitao" , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org By the way I checked in the other bits we agree on as changeset 16312. It now looks quite sane to me. -- Keir On 2/11/07 16:14, "Keir Fraser" wrote: > Thanks for the explanation. I think you are mis-describing the current > behaviour of vpt.c though (If I understand it correctly myself, which I may > not!). As I understand it, we do *not* unconditionally deliver a tick and > reset the time space when a vcpu is scheduled onto a cpu. We only do that if > (NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has > passed. Otherwise we wait to deliver the next tick exactly one period after > the previous tick was delivered. The remainder of your explanation seems to > be predicated on the (impossible?) case that we deliver a tick less than one > period after the previous one. > > Am I confused? :-) Also, what version of Linux are we talking about here, > and what periodic timer, and x86/64 or i386? There are lots of different > Linux timer-handling logics out there in the wild! > > -- Keir > > On 2/11/07 15:51, "Dave Winchell" wrote: > >> Let D be the time that the clock vcpu is descheduled and P be >> the clock period. When D < P, I think there is an issue. >> >> The reason is that Linux's offset calculation, which affects the >> last clock interrupt tsc that is recorded, doesn't kick in if the >> time since the last interrupt is less than P. In this case it sets >> offset to zero. Linux records the tsc of the current (last) clock interrupt >> as (current tsc - offset). >> >> Interrupt delivery method AS delivers a clock interrupt >> at context switch (in) time. When D < P, Linux records the >> interrupt delivery time as current tsc and bumps jiffies. >> This results in a gain of time equal to P-D over wall time. >> >> For method S this never happens because the interrupt isn't >> delivered until the next boundary. > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Sat, 03 Nov 2007 22:31:47 +0000 Message-ID: References: <472CE57A.9030804@virtualiron.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="B_3276973911_87054626" Return-path: In-Reply-To: <472CE57A.9030804@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Dong, Eddie" , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3276973911_87054626 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit On 3/11/07 21:17, "Dave Winchell" wrote: > Thanks for applying the fixes in the last submit. > In moving the test for no_missed_tick_accounting into > pt_process_missed_ticks() > you forgot to add the scheduling part. Actually it was deliberate, but clearly it was one code simplification too far: thanks for spotting it! I'll go the async route, but we do need to set pending_intr_nr to 1. We can't leave that out -- the point of the async route is to send a tick to the vcpu immediately, since it hasn't had one for more than a tick period. If we wait for the timeout to do that then we have to wait a whole extra period after the vcpu is re-scheduled. Attached is my proposed patch. I think it's quite neat. :-) -- Keir --B_3276973911_87054626 Content-type: application/octet-stream; name="timer.patch" Content-disposition: attachment; filename="timer.patch" Content-transfer-encoding: base64 ZGlmZiAtciA2NTBjYWRkMWIyODMgeGVuL2FyY2gveDg2L2h2bS92cHQuYwotLS0gYS94ZW4v YXJjaC94ODYvaHZtL3ZwdC5jCUZyaSBOb3YgMDIgMTY6Mzg6MTEgMjAwNyArMDAwMAorKysg Yi94ZW4vYXJjaC94ODYvaHZtL3ZwdC5jCVNhdCBOb3YgMDMgMjI6MjI6MjMgMjAwNyArMDAw MApAQCAtNDcsMjEgKzQ3LDI2IEBAIHN0YXRpYyB2b2lkIHB0X3VubG9jayhzdHJ1Y3QgcGVy aW9kaWNfdGkKIAogc3RhdGljIHZvaWQgcHRfcHJvY2Vzc19taXNzZWRfdGlja3Moc3RydWN0 IHBlcmlvZGljX3RpbWUgKnB0KQogewotICAgIHNfdGltZV90IG1pc3NlZF90aWNrczsKKyAg ICBzX3RpbWVfdCBtaXNzZWRfdGlja3MsIG5vdyA9IE5PVygpOworCisgICAgaWYgKCBwdC0+ b25lX3Nob3QgKQorICAgICAgICByZXR1cm47CisKKyAgICBtaXNzZWRfdGlja3MgPSBub3cg LSBwdC0+c2NoZWR1bGVkOworICAgIGlmICggbWlzc2VkX3RpY2tzIDw9IDAgKQorICAgICAg ICByZXR1cm47CiAKICAgICBpZiAoIG1vZGVfaXMocHQtPnZjcHUtPmRvbWFpbiwgbm9fbWlz c2VkX3RpY2tfYWNjb3VudGluZykgKQotICAgICAgICByZXR1cm47Ci0KLSAgICBpZiAoIHB0 LT5vbmVfc2hvdCApCi0gICAgICAgIHJldHVybjsKLQotICAgIG1pc3NlZF90aWNrcyA9IE5P VygpIC0gcHQtPnNjaGVkdWxlZDsKLSAgICBpZiAoIG1pc3NlZF90aWNrcyA8PSAwICkKLSAg ICAgICAgcmV0dXJuOwotCi0gICAgbWlzc2VkX3RpY2tzID0gbWlzc2VkX3RpY2tzIC8gKHNf dGltZV90KSBwdC0+cGVyaW9kICsgMTsKLSAgICBwdC0+cGVuZGluZ19pbnRyX25yICs9IG1p c3NlZF90aWNrczsKLSAgICBwdC0+c2NoZWR1bGVkICs9IG1pc3NlZF90aWNrcyAqIHB0LT5w ZXJpb2Q7CisgICAgeworICAgICAgICBwdC0+cGVuZGluZ19pbnRyID0gMTsKKyAgICAgICAg cHQtPnNjaGVkdWxlZCA9IG5vdyArIHB0LT5zY2hlZHVsZWQ7CisgICAgfQorICAgIGVsc2UK KyAgICB7CisgICAgICAgIG1pc3NlZF90aWNrcyA9IG1pc3NlZF90aWNrcyAvIChzX3RpbWVf dCkgcHQtPnBlcmlvZCArIDE7CisgICAgICAgIHB0LT5wZW5kaW5nX2ludHJfbnIgKz0gbWlz c2VkX3RpY2tzOworICAgICAgICBwdC0+c2NoZWR1bGVkICs9IG1pc3NlZF90aWNrcyAqIHB0 LT5wZXJpb2Q7CisgICAgfQogfQogCiBzdGF0aWMgdm9pZCBwdF9mcmVlemVfdGltZShzdHJ1 Y3QgdmNwdSAqdikK --B_3276973911_87054626 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --B_3276973911_87054626-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 07 Nov 2007 14:39:46 +0000 Message-ID: References: <4731CE04.6050105@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4731CE04.6050105@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Dong, Eddie" , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Oh dear, that was a silly typo on my part. Thanks for tracking it down! Well, I'm surprised that SYNC vs ASYNC differs, if the guest is really tracking time via the TSC value. But it sounds like the actual time-tracking algorithm in the guest must be a bit more complicated than that? I'd be very happy to help with any further analysis. -- Keir On 7/11/07 14:39, "Dave Winchell" wrote: > Keir, > > Attached is a fix for pt_process_missed_ticks(). > Without this fix, systems run very erratically and some > guests panic on boot complaining that timer interrupts > are not working. As you can imagine. > > Also, I have some longer term measurements of the > accuracy of the sync and async methods. > The hardware is an eight cpu AMD box. Two eight > vcpu guests, rh4u4-64 and sles9sp3-64. > All vcpus running loads. > > Method Test duration Clock errors > > SYNC 56000 sec 6.4, 6.7 sec (.012%) > ASYNC 52000 sec 13, 19 sec (.036%) > > More testing should be done to validate the significance > of this difference. > > regards, > Dave > > > Dave Winchell wrote: > >> >> Keir Fraser wrote: >> >>> On 3/11/07 21:17, "Dave Winchell" wrote: >>> >>> >>> >>>> Thanks for applying the fixes in the last submit. >>>> In moving the test for no_missed_tick_accounting into >>>> pt_process_missed_ticks() >>>> you forgot to add the scheduling part. >>>> >>> >>> >>> Actually it was deliberate, but clearly it was one code >>> simplification too >>> far: thanks for spotting it! I'll go the async route, but we do need >>> to set >>> pending_intr_nr to 1. We can't leave that out -- the point of the async >>> route is to send a tick to the vcpu immediately, since it hasn't had >>> one for >>> more than a tick period. If we wait for the timeout to do that then >>> we have >>> to wait a whole extra period after the vcpu is re-scheduled. >>> >>> Attached is my proposed patch. I think it's quite neat. :-) >>> >>> >> It looks good to me. >> >> thanks, >> Dave >> >>> -- Keir >>> >>> >>> >> > > diff -r dfe9c0c10a2c xen/arch/x86/hvm/vpt.c > --- a/xen/arch/x86/hvm/vpt.c Mon Nov 05 13:23:55 2007 +0000 > +++ b/xen/arch/x86/hvm/vpt.c Wed Nov 07 08:55:45 2007 -0500 > @@ -59,7 +59,7 @@ static void pt_process_missed_ticks(stru > if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) > { > pt->pending_intr_nr = 1; > - pt->scheduled = now + pt->scheduled; > + pt->scheduled = now + pt->period; > } > else > { From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 07 Nov 2007 17:10:40 +0000 Message-ID: References: <4731E679.9000307@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4731E679.9000307@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Dong, Eddie" , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org On 7/11/07 16:23, "Dave Winchell" wrote: > I'm concerned about the way delay is blown off in the final calculation > > vxtime.last_tsc = tsc - > (((long) offset << 32) / vxtime.tsc_quot) - 1; > > If an interrupt is right on time, and the delay is the same as last time, > then the offset should be zero in my mind. In the code above, offset will > equal delay for this example. > > What do you think? I don't think this code likes the ASYNC method *at all*. The reason is that it estimates how late the tick interrupt is by reading the PIT counter. It knows the interrupt should have been triggered when the counter wrapped at zero. But of course, if we are delivering ticks in ASYNC mode then quickly the time we deliver an interrupt has *nothing* to do with the current PIT counter value! Hence the lines that effectively fold max(offset, delay) into last_tsc are just not going to work properly, because delay is a uniform random variable, and hence we probably lose time. Let me see if I can come up with a patch that gets the best of ASYNC and SYNC in one handy mode... -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 07 Nov 2007 17:29:17 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser , Dave Winchell Cc: "Shan, Haitao" , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org On 7/11/07 17:10, "Keir Fraser" wrote: > I don't think this code likes the ASYNC method *at all*. The reason is that > it estimates how late the tick interrupt is by reading the PIT counter. It > knows the interrupt should have been triggered when the counter wrapped at > zero. But of course, if we are delivering ticks in ASYNC mode then quickly > the time we deliver an interrupt has *nothing* to do with the current PIT > counter value! Hence the lines that effectively fold max(offset, delay) into > last_tsc are just not going to work properly, because delay is a uniform > random variable, and hence we probably lose time. Actually, no, I don't understand why this code is less accurate with ASYNC mode. My comments on 'delay' above are not true I think. Both 'delay' and 'offset' are accurate estimates of the time elapsed since the last actual tick threshold, when the PIT counter rolled over. And hence the logic that is being applied *should* work... I wonder if something silly like the following is happening: Because we send unsynchronised tick interrupts, they can actually be delivered when the PIT is just about to roll over (but hasn't yet). If that happens we could get a worst-case disagreement between offset (derived from TSC) and delay (derived from PIT). Let's say that when we read the TSC it looks like we have just passed a tick rollover. In that case we will count some whole number of lost ticks and offset will end up being a very small number (very little 'slop' after a whole number of ticks is accounted). However, when we read the PIT it looks like we are just about to roll over (but haven't yet) and so delay will be a very large number: about as large as it can be. In this case we will subtract a lot from last_tsc, because it looks like we've had nearly a full tick of delay. But actually the amount of delay that is getting subtracted from last_tsc has already been accounted for elsewhere as a full tick! If this analysis is the truth of the matter then we would actually expect the ASYNC mode to cause the guest timeofday to run *fast* rather than slow. Anyhow, it's bound to be something silly like this. :-) The SYNC/ASYNC method I was going to propose by the way was going to be, in pt_process_missed_ticks(): pt->scheduled += missed_ticks * pt->period; if ( mode_is(no_missed_tick_accounting) ) pt->pending_intr_nr = 1; else pt->pending_intr_nr += missed_ticks; So, you can see we send an interrupt immediately (and ASYNC) if any ticks have been missed, but then successive ticks are delivered 'on the beat'. A possible middleground? Or perhaps we should just go with SYNC after all... -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 07 Nov 2007 17:47:33 +0000 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser , Dave Winchell Cc: "Shan, Haitao" , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org On 7/11/07 17:29, "Keir Fraser" wrote: > So, you can see we send an interrupt immediately (and ASYNC) if any ticks > have been missed, but then successive ticks are delivered 'on the beat'. A > possible middleground? Or perhaps we should just go with SYNC after all... How do these Linux x64 guests fare with the original and default timer mode, by the way? I would expect that time should be accounted pretty accurately in that mode, albeit with more interrupts than you'd like. Or is the lack of synchronisation of TSCs across VCPUs causing issues that you're trying to avoid? -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 08 Nov 2007 08:07:48 +0000 Message-ID: References: <47321450.8090107@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47321450.8090107@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Shan, Haitao" , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org On 7/11/07 19:38, "Dave Winchell" wrote: > My feeling is that we should go full SYNC. Yes, in theory the > guests should be able to handle ASYNC, but in reality it appears that > some do not. Since it is easy for us to give them SYNC, > lets just do it and not stress them out. One problem with pure SYNC is there's a fair chance you won't deliver any ticks at all for a long time, if the guest only runs in short bursts (e.g., I/O bound) and happens not to be running on any tick boundary. I'm not sure how much that matters. It could cause time goes backwards if the time extrapolation via the TSC is not perfectly accurate, or cause problems if there are any assumptions that TSC delta since last tick fits in 32 bits (less likely in x64 code I suppose). Anyway, my point is that only testing VCPUs under full load may cause us to optimise in ways that have nasty unexpected effects for other workloads. > For default mode as checked into unstable is now, > 64 bit guests should run quite fast as missed is calculated and then a bunch > of additional interrupts are delivered. On the other hand > 32bit guests very well in default mode. > > For the original code, before we put in the constant tsc offset business, > 64bit guests run poorly and 32bit quests very well time-wise. The default mode hasn't changed. Are you under the impression that missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 guests run badly with that because they treat every one of the missed ticks they receive as a full tick. -- Keir >> Or is the lack of >> synchronization of TSCs across VCPUs causing issues that you're trying to >> avoid? > > This does cause issues, but its not the only contributor to poor timing. > Having TSCs synchronized across vcpus will help some of the time going > backwards problems we have seen, I think. > > Regards, > Dave > > Keir Fraser wrote: > >> On 7/11/07 17:29, "Keir Fraser" wrote: >> >> >> >>> So, you can see we send an interrupt immediately (and ASYNC) if any ticks >>> have been missed, but then successive ticks are delivered 'on the beat'. A >>> possible middleground? Or perhaps we should just go with SYNC after all... >>> >>> >> >> How do these Linux x64 guests fare with the original and default timer mode, >> by the way? I would expect that time should be accounted pretty accurately >> in that mode, albeit with more interrupts than you'd like. Or is the lack of >> synchronisation of TSCs across VCPUs causing issues that you're trying to >> avoid? >> >> -- Keir >> >> >> >> > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 08 Nov 2007 14:53:11 +0000 Message-ID: References: <47332084.8090305@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47332084.8090305@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Shan, Haitao" , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org On 8/11/07 14:43, "Dave Winchell" wrote: > I agree that this could be a problem. I have an idea that could give us full > SYNC and eliminate the long periods without clock interrupts. > In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1. > In pt_save_timer(): > > list_for_each_entry ( pt, head, list ) > if(!pt->run_timer) > stop_timer(&pt->timer); > > And in pt_timer_fn(): > > pt->run_timer = 0; > > So, for a guest that misses a tick, we will interrupt him once from the > descheduled state and then leave him alone in the descheduled state. Well, I'd rather not complicate the code if it's avoidable. I checked in a SYNC/ASYNC combo and code simplification as changeset 16341, and it'd be interesting to know how that fares against your suggested scheme. I suppose as long as we're better than ntpd's tolerance it doesn't actually matter all that much. -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Sat, 10 Nov 2007 10:55:09 +0000 Message-ID: References: <4734B36F.1080408@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4734B36F.1080408@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Shan, Haitao" , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org On 9/11/07 19:22, "Dave Winchell" wrote: > Since I had a high error (~.03%) for the ASYNC method a couple of days ago, > I ran another ASYNC test. I think there may have been something > wrong with the code I used a couple of days ago for ASYNC. It may have been > missing the immediate delivery of interrupt after context switch in. > > My results indicate that either SYNC or ASYNC give acceptable accuracy, > each running consistently around or under .01%. MIXED has a fairly high > error of > greater than .03%. Probably too close to .05% ntp threshold for comfort. > I don't have an overnight run with SYNC. I plan to leave SYNC running > over the weekend. If you'd rather I can leave MIXED running instead. > > It may be too early to pick the protocol and I can run more overnight tests > next week. I'm a bit worried about any unwanted side effects of the SYNC+run_timer approach -- e.g., whether timer wakeups will cause higher system-wide CPU contention. I find it easier to think through the implications of ASYNC. I'm surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it delivers more timer interrupts than the other approaches, and each interrupt event causes a small accumulated error? Overall I would consider MIXED and ASYNC as favourites and if the latter is actually more accurate then I can simply revert the changeset that implemented MIXED. Perhaps rather than running more of the same workloads you could try idle VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We don't have any data on workloads that aren't CPU bound, so that's really an obvious place to put any further effort imo. -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 30 Oct 2007 17:16:15 -0400 Message-ID: <47279F1F.6050904@virtualiron.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------050007040101080506050902" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------050007040101080506050902 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Keir, Here are my comments on your change. I've attached an updated vpt.c with the changes. 1. For the no_missed_tick_accounting method, we still need the update to pt->scheduled taking into account the time that has elapsed when missed ticks are calculated. missed_ticks (pt_process_missed_ticks) is called from pt_restore_timer() and pt_timer_fn() and, thus, its easiest to put the check for no_missed_tick_accounting method in missed_ticks() itself. 2. In pt_timer_fn you don't want to increment pending_intr_nr beyond 1 for no_missed_tick_accounting option. thanks, Dave Keir Fraser wrote: > > Applied as c/s 16274. Please take a look and make sure the mode works > as you expect. > > -- Keir > > On 30/10/07 14:28, "Shan, Haitao" wrote: > > Hi, Keir, > > This patch adds a new timer mode, in which no missed ticks is > calculated. This can be used with latest x86_64 linux guest, since > it can pick up missed ticks themselves. > > <> > > Best Regards > Haitao Shan > > ------------------------------------------------------------------------ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > --------------050007040101080506050902 Content-Type: text/plain; name="diff.vpt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="diff.vpt" *** vpt.c.new.c 2007-10-30 16:45:26.000000000 -0400 --- vpt.c 2007-10-30 15:30:57.000000000 -0400 *************** *** 57,73 **** return; missed_ticks = missed_ticks / (s_time_t) pt->period + 1; ! ! if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { ! if ( missed_ticks > 1000 ) ! { ! /* TODO: Adjust guest time together */ ! pt->pending_intr_nr++; ! } ! else ! { ! pt->pending_intr_nr += missed_ticks; ! } } pt->scheduled += missed_ticks * pt->period; --- 57,70 ---- return; missed_ticks = missed_ticks / (s_time_t) pt->period + 1; ! if ( missed_ticks > 1000 ) ! { ! /* TODO: Adjust guest time together */ ! pt->pending_intr_nr++; ! } ! else ! { ! pt->pending_intr_nr += missed_ticks; } pt->scheduled += missed_ticks * pt->period; *************** *** 120,126 **** list_for_each_entry ( pt, head, list ) { ! pt_process_missed_ticks(pt); set_timer(&pt->timer, pt->scheduled); } --- 117,124 ---- list_for_each_entry ( pt, head, list ) { ! if ( !mode_is(v->domain, no_missed_tick_accounting) ) ! pt_process_missed_ticks(pt); set_timer(&pt->timer, pt->scheduled); } *************** *** 135,151 **** pt_lock(pt); ! if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) { ! if(!pt->pending_intr_nr) ! pt->pending_intr_nr++; ! } ! else ! pt->pending_intr_nr++; if ( !pt->one_shot ) { pt->scheduled += pt->period; ! pt_process_missed_ticks(pt); set_timer(&pt->timer, pt->scheduled); } --- 133,152 ---- pt_lock(pt); ! pt->pending_intr_nr++; if ( !pt->one_shot ) { pt->scheduled += pt->period; ! if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) ! { ! pt_process_missed_ticks(pt); ! } ! else if ( (NOW() - pt->scheduled) >= 0 ) ! { ! pt->pending_intr_nr++; ! pt->scheduled = NOW() + pt->period; ! } set_timer(&pt->timer, pt->scheduled); } --------------050007040101080506050902 Content-Type: text/x-csrc; name="vpt.c" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="vpt.c" /* * vpt.c: Virtual Platform Timer * * Copyright (c) 2006, Xiaowei Yang, Intel Corporation. * * This program is free software; you can redistribute it and/or modify it * under the terms and conditions of the GNU General Public License, * version 2, as published by the Free Software Foundation. * * This program is distributed in the hope it will be useful, but WITHOUT * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for * more details. * * You should have received a copy of the GNU General Public License along with * this program; if not, write to the Free Software Foundation, Inc., 59 Temple * Place - Suite 330, Boston, MA 02111-1307 USA. * */ #include #include #include #include #define mode_is(d, name) \ ((d)->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] == HVMPTM_##name) static void pt_lock(struct periodic_time *pt) { struct vcpu *v; for ( ; ; ) { v = pt->vcpu; spin_lock(&v->arch.hvm_vcpu.tm_lock); if ( likely(pt->vcpu == v) ) break; spin_unlock(&v->arch.hvm_vcpu.tm_lock); } } static void pt_unlock(struct periodic_time *pt) { spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock); } static void pt_process_missed_ticks(struct periodic_time *pt) { s_time_t missed_ticks; if ( pt->one_shot ) return; missed_ticks = NOW() - pt->scheduled; if ( missed_ticks <= 0 ) return; missed_ticks = missed_ticks / (s_time_t) pt->period + 1; if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { if ( missed_ticks > 1000 ) { /* TODO: Adjust guest time together */ pt->pending_intr_nr++; } else { pt->pending_intr_nr += missed_ticks; } } pt->scheduled += missed_ticks * pt->period; } static void pt_freeze_time(struct vcpu *v) { if ( !mode_is(v->domain, delay_for_missed_ticks) ) return; v->arch.hvm_vcpu.guest_time = hvm_get_guest_time(v); } static void pt_thaw_time(struct vcpu *v) { if ( !mode_is(v->domain, delay_for_missed_ticks) ) return; if ( v->arch.hvm_vcpu.guest_time == 0 ) return; hvm_set_guest_time(v, v->arch.hvm_vcpu.guest_time); v->arch.hvm_vcpu.guest_time = 0; } void pt_save_timer(struct vcpu *v) { struct list_head *head = &v->arch.hvm_vcpu.tm_list; struct periodic_time *pt; if ( test_bit(_VPF_blocked, &v->pause_flags) ) return; spin_lock(&v->arch.hvm_vcpu.tm_lock); list_for_each_entry ( pt, head, list ) stop_timer(&pt->timer); pt_freeze_time(v); spin_unlock(&v->arch.hvm_vcpu.tm_lock); } void pt_restore_timer(struct vcpu *v) { struct list_head *head = &v->arch.hvm_vcpu.tm_list; struct periodic_time *pt; spin_lock(&v->arch.hvm_vcpu.tm_lock); list_for_each_entry ( pt, head, list ) { pt_process_missed_ticks(pt); set_timer(&pt->timer, pt->scheduled); } pt_thaw_time(v); spin_unlock(&v->arch.hvm_vcpu.tm_lock); } static void pt_timer_fn(void *data) { struct periodic_time *pt = data; pt_lock(pt); if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) { if(!pt->pending_intr_nr) pt->pending_intr_nr++; } else pt->pending_intr_nr++; if ( !pt->one_shot ) { pt->scheduled += pt->period; pt_process_missed_ticks(pt); set_timer(&pt->timer, pt->scheduled); } vcpu_kick(pt->vcpu); pt_unlock(pt); } void pt_update_irq(struct vcpu *v) { struct list_head *head = &v->arch.hvm_vcpu.tm_list; struct periodic_time *pt; uint64_t max_lag = -1ULL; int irq = -1; spin_lock(&v->arch.hvm_vcpu.tm_lock); list_for_each_entry ( pt, head, list ) { if ( !is_isa_irq_masked(v, pt->irq) && pt->pending_intr_nr && ((pt->last_plt_gtime + pt->period_cycles) < max_lag) ) { max_lag = pt->last_plt_gtime + pt->period_cycles; irq = pt->irq; } } spin_unlock(&v->arch.hvm_vcpu.tm_lock); if ( is_lvtt(v, irq) ) { vlapic_set_irq(vcpu_vlapic(v), irq, 0); } else if ( irq >= 0 ) { hvm_isa_irq_deassert(v->domain, irq); hvm_isa_irq_assert(v->domain, irq); } } static struct periodic_time *is_pt_irq( struct vcpu *v, struct hvm_intack intack) { struct list_head *head = &v->arch.hvm_vcpu.tm_list; struct periodic_time *pt; struct RTCState *rtc = &v->domain->arch.hvm_domain.pl_time.vrtc; int vector; list_for_each_entry ( pt, head, list ) { if ( !pt->pending_intr_nr ) continue; if ( is_lvtt(v, pt->irq) ) { if ( pt->irq != intack.vector ) continue; return pt; } vector = get_isa_irq_vector(v, pt->irq, intack.source); /* RTC irq need special care */ if ( (intack.vector != vector) || ((pt->irq == 8) && !is_rtc_periodic_irq(rtc)) ) continue; return pt; } return NULL; } void pt_intr_post(struct vcpu *v, struct hvm_intack intack) { struct periodic_time *pt; time_cb *cb; void *cb_priv; spin_lock(&v->arch.hvm_vcpu.tm_lock); pt = is_pt_irq(v, intack); if ( pt == NULL ) { spin_unlock(&v->arch.hvm_vcpu.tm_lock); return; } if ( pt->one_shot ) { pt->enabled = 0; list_del(&pt->list); } else { pt->pending_intr_nr--; if ( mode_is(v->domain, no_missed_tick_accounting) ) pt->last_plt_gtime = hvm_get_guest_time(v); else pt->last_plt_gtime += pt->period_cycles; } if ( mode_is(v->domain, delay_for_missed_ticks) && (hvm_get_guest_time(v) < pt->last_plt_gtime) ) hvm_set_guest_time(v, pt->last_plt_gtime); cb = pt->cb; cb_priv = pt->priv; spin_unlock(&v->arch.hvm_vcpu.tm_lock); if ( cb != NULL ) cb(v, cb_priv); } void pt_reset(struct vcpu *v) { struct list_head *head = &v->arch.hvm_vcpu.tm_list; struct periodic_time *pt; spin_lock(&v->arch.hvm_vcpu.tm_lock); list_for_each_entry ( pt, head, list ) { pt->pending_intr_nr = 0; pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu); pt->scheduled = NOW() + pt->period; set_timer(&pt->timer, pt->scheduled); } spin_unlock(&v->arch.hvm_vcpu.tm_lock); } void pt_migrate(struct vcpu *v) { struct list_head *head = &v->arch.hvm_vcpu.tm_list; struct periodic_time *pt; spin_lock(&v->arch.hvm_vcpu.tm_lock); list_for_each_entry ( pt, head, list ) migrate_timer(&pt->timer, v->processor); spin_unlock(&v->arch.hvm_vcpu.tm_lock); } void create_periodic_time( struct vcpu *v, struct periodic_time *pt, uint64_t period, uint8_t irq, char one_shot, time_cb *cb, void *data) { destroy_periodic_time(pt); spin_lock(&v->arch.hvm_vcpu.tm_lock); pt->enabled = 1; pt->pending_intr_nr = 0; /* Periodic timer must be at least 0.9ms. */ if ( (period < 900000) && !one_shot ) { gdprintk(XENLOG_WARNING, "HVM_PlatformTime: program too small period %"PRIu64"\n", period); period = 900000; } pt->period = period; pt->vcpu = v; pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu); pt->irq = irq; pt->period_cycles = (u64)period * cpu_khz / 1000000L; pt->one_shot = one_shot; pt->scheduled = NOW() + period; /* * Offset LAPIC ticks from other timer ticks. Otherwise guests which use * LAPIC ticks for process accounting can see long sequences of process * ticks incorrectly accounted to interrupt processing. */ if ( is_lvtt(v, irq) ) pt->scheduled += period >> 1; pt->cb = cb; pt->priv = data; list_add(&pt->list, &v->arch.hvm_vcpu.tm_list); init_timer(&pt->timer, pt_timer_fn, pt, v->processor); set_timer(&pt->timer, pt->scheduled); spin_unlock(&v->arch.hvm_vcpu.tm_lock); } void destroy_periodic_time(struct periodic_time *pt) { if ( !pt->enabled ) return; pt_lock(pt); pt->enabled = 0; list_del(&pt->list); pt_unlock(pt); /* * pt_timer_fn() can run until this kill_timer() returns. We must do this * outside pt_lock() otherwise we can deadlock with pt_timer_fn(). */ kill_timer(&pt->timer); } --------------050007040101080506050902 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------050007040101080506050902-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 01 Nov 2007 17:14:50 -0400 Message-ID: <472A41CA.7080405@virtualiron.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------010105000402010406090500" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------010105000402010406090500 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Keir, Haitao, Eddie: Here is the patch for timekeeping. There are three components. 1. Increase missed ticks threshold to 100 seconds. Per the comment in the patch, I have seen scheduling delays of 5 seconds in a test lasting 30 minutes. The result is an immediate error of 5 seconds in guest time that persists. 2. In pt_timer_fn for no_missed_tick_accounting option, only increment pt->pending_intr_nr if its found to be zero. This deals with the case where the guest has interrupts disabled for longer than a tick period. 3. Call pt_process_missed_ticks() unconditionally and place the test for no_missed_tick_accounting inside pt_process_missed_ticks(). This returns the calculation for the next timer expiration to the original method. The original method is important as it sets up a theoretical time space at t=0 where expirations occur only at n*period, where n is any integer. This, in turn, removes rounding errors. Accuracy tests: A. 32bit Linux guest (Item '1' above relevant) Test: 2 64bit Linux guests and 1 32bit Linux guest stacked. 8 vcpus for each guest. 2 physical processors on the node. All vcpus heavily loaded with work. Test duration: 3 hours. Result: clock error < .02%. (Before this patch guest time was 16 seconds slow in a 3.5 minute test.) B. 64bit Linux guest (Items '2' and '3' above relevant) Test: 2 64bit Linux guests. 8 vcpus for each guest. 2 physical processors on the node. All vcpus heavily loaded with work. Test duration: 3 hours. Result: clock error .02%. (Before this patch clock error was 1.3%. The contributions of items '2' and '3' above to the improvement in accuracy are .4% and .9% respectively. Of course, these tests are without ntpd running in the guest. Ntpd requires a local clock error of < .05%. With the .02% accuracy, one can used ntpd to synchronize the clock. thanks, Dave Keir Fraser wrote: >I'm going to apply Haitao's no_missed_ticks_fix.patch. If you think fixes >are needed apart from that, please provide a unified diff. > > -- Keir > >On 30/10/07 21:16, "Dave Winchell" wrote: > > > >>Keir, >> >>Here are my comments on your change. >>I've attached an updated vpt.c with the changes. >> >>1. For the no_missed_tick_accounting method, we still need the update >> to pt->scheduled taking into account the time that has elapsed when >> missed ticks are calculated. missed_ticks (pt_process_missed_ticks) >> is called from pt_restore_timer() and pt_timer_fn() and, thus, its >> easiest to put the check for no_missed_tick_accounting method in >> missed_ticks() itself. >> >>2. In pt_timer_fn you don't want to increment pending_intr_nr beyond 1 >> for no_missed_tick_accounting option. >> >>thanks, >>Dave >> >> >>Keir Fraser wrote: >> >> >> >>>Applied as c/s 16274. Please take a look and make sure the mode works >>>as you expect. >>> >>> -- Keir >>> >>>On 30/10/07 14:28, "Shan, Haitao" wrote: >>> >>> Hi, Keir, >>> >>> This patch adds a new timer mode, in which no missed ticks is >>> calculated. This can be used with latest x86_64 linux guest, since >>> it can pick up missed ticks themselves. >>> >>> <> >>> >>> Best Regards >>> Haitao Shan >>> >>> ------------------------------------------------------------------------ >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >>> >>> >>> >>*** vpt.c.new.c 2007-10-30 16:45:26.000000000 -0400 >>--- vpt.c 2007-10-30 15:30:57.000000000 -0400 >>*************** >>*** 57,73 **** >> return; >> >> missed_ticks = missed_ticks / (s_time_t) pt->period + 1; >>! >>! if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { >>! if ( missed_ticks > 1000 ) >>! { >>! /* TODO: Adjust guest time together */ >>! pt->pending_intr_nr++; >>! } >>! else >>! { >>! pt->pending_intr_nr += missed_ticks; >>! } >> } >> >> pt->scheduled += missed_ticks * pt->period; >>--- 57,70 ---- >> return; >> >> missed_ticks = missed_ticks / (s_time_t) pt->period + 1; >>! if ( missed_ticks > 1000 ) >>! { >>! /* TODO: Adjust guest time together */ >>! pt->pending_intr_nr++; >>! } >>! else >>! { >>! pt->pending_intr_nr += missed_ticks; >> } >> >> pt->scheduled += missed_ticks * pt->period; >>*************** >>*** 120,126 **** >> >> list_for_each_entry ( pt, head, list ) >> { >>! pt_process_missed_ticks(pt); >> set_timer(&pt->timer, pt->scheduled); >> } >> >>--- 117,124 ---- >> >> list_for_each_entry ( pt, head, list ) >> { >>! if ( !mode_is(v->domain, no_missed_tick_accounting) ) >>! pt_process_missed_ticks(pt); >> set_timer(&pt->timer, pt->scheduled); >> } >> >>*************** >>*** 135,151 **** >> >> pt_lock(pt); >> >>! if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) { >>! if(!pt->pending_intr_nr) >>! pt->pending_intr_nr++; >>! } >>! else >>! pt->pending_intr_nr++; >> >> if ( !pt->one_shot ) >> { >> pt->scheduled += pt->period; >>! pt_process_missed_ticks(pt); >> set_timer(&pt->timer, pt->scheduled); >> } >> >>--- 133,152 ---- >> >> pt_lock(pt); >> >>! pt->pending_intr_nr++; >> >> if ( !pt->one_shot ) >> { >> pt->scheduled += pt->period; >>! if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) >>! { >>! pt_process_missed_ticks(pt); >>! } >>! else if ( (NOW() - pt->scheduled) >= 0 ) >>! { >>! pt->pending_intr_nr++; >>! pt->scheduled = NOW() + pt->period; >>! } >> set_timer(&pt->timer, pt->scheduled); >> } >> >>/* >> * vpt.c: Virtual Platform Timer >> * >> * Copyright (c) 2006, Xiaowei Yang, Intel Corporation. >> * >> * This program is free software; you can redistribute it and/or modify it >> * under the terms and conditions of the GNU General Public License, >> * version 2, as published by the Free Software Foundation. >> * >> * This program is distributed in the hope it will be useful, but WITHOUT >> * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or >> * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for >> * more details. >> * >> * You should have received a copy of the GNU General Public License along >>with >> * this program; if not, write to the Free Software Foundation, Inc., 59 >>Temple >> * Place - Suite 330, Boston, MA 02111-1307 USA. >> * >> */ >> >>#include >>#include >>#include >>#include >> >>#define mode_is(d, name) \ >> ((d)->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] == HVMPTM_##name) >> >>static void pt_lock(struct periodic_time *pt) >>{ >> struct vcpu *v; >> >> for ( ; ; ) >> { >> v = pt->vcpu; >> spin_lock(&v->arch.hvm_vcpu.tm_lock); >> if ( likely(pt->vcpu == v) ) >> break; >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >> } >>} >> >>static void pt_unlock(struct periodic_time *pt) >>{ >> spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock); >>} >> >>static void pt_process_missed_ticks(struct periodic_time *pt) >>{ >> s_time_t missed_ticks; >> >> if ( pt->one_shot ) >> return; >> >> missed_ticks = NOW() - pt->scheduled; >> if ( missed_ticks <= 0 ) >> return; >> >> missed_ticks = missed_ticks / (s_time_t) pt->period + 1; >> >> if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { >>if ( missed_ticks > 1000 ) >> { >>/* TODO: Adjust guest time together */ >>pt->pending_intr_nr++; >> } >>else >> { >>pt->pending_intr_nr += missed_ticks; >> } >> } >> >> pt->scheduled += missed_ticks * pt->period; >>} >> >>static void pt_freeze_time(struct vcpu *v) >>{ >> if ( !mode_is(v->domain, delay_for_missed_ticks) ) >> return; >> >> v->arch.hvm_vcpu.guest_time = hvm_get_guest_time(v); >>} >> >>static void pt_thaw_time(struct vcpu *v) >>{ >> if ( !mode_is(v->domain, delay_for_missed_ticks) ) >> return; >> >> if ( v->arch.hvm_vcpu.guest_time == 0 ) >> return; >> >> hvm_set_guest_time(v, v->arch.hvm_vcpu.guest_time); >> v->arch.hvm_vcpu.guest_time = 0; >>} >> >>void pt_save_timer(struct vcpu *v) >>{ >> struct list_head *head = &v->arch.hvm_vcpu.tm_list; >> struct periodic_time *pt; >> >> if ( test_bit(_VPF_blocked, &v->pause_flags) ) >> return; >> >> spin_lock(&v->arch.hvm_vcpu.tm_lock); >> >> list_for_each_entry ( pt, head, list ) >> stop_timer(&pt->timer); >> >> pt_freeze_time(v); >> >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >>} >> >>void pt_restore_timer(struct vcpu *v) >>{ >> struct list_head *head = &v->arch.hvm_vcpu.tm_list; >> struct periodic_time *pt; >> >> spin_lock(&v->arch.hvm_vcpu.tm_lock); >> >> list_for_each_entry ( pt, head, list ) >> { >>pt_process_missed_ticks(pt); >> set_timer(&pt->timer, pt->scheduled); >> } >> >> pt_thaw_time(v); >> >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >>} >> >>static void pt_timer_fn(void *data) >>{ >> struct periodic_time *pt = data; >> >> pt_lock(pt); >> >> if (mode_is(pt->vcpu->domain, no_missed_tick_accounting)) { >>if(!pt->pending_intr_nr) >> pt->pending_intr_nr++; >> } >> else >>pt->pending_intr_nr++; >> >> if ( !pt->one_shot ) >> { >> pt->scheduled += pt->period; >>pt_process_missed_ticks(pt); >> set_timer(&pt->timer, pt->scheduled); >> } >> >> vcpu_kick(pt->vcpu); >> >> pt_unlock(pt); >>} >> >>void pt_update_irq(struct vcpu *v) >>{ >> struct list_head *head = &v->arch.hvm_vcpu.tm_list; >> struct periodic_time *pt; >> uint64_t max_lag = -1ULL; >> int irq = -1; >> >> spin_lock(&v->arch.hvm_vcpu.tm_lock); >> >> list_for_each_entry ( pt, head, list ) >> { >> if ( !is_isa_irq_masked(v, pt->irq) && pt->pending_intr_nr && >> ((pt->last_plt_gtime + pt->period_cycles) < max_lag) ) >> { >> max_lag = pt->last_plt_gtime + pt->period_cycles; >> irq = pt->irq; >> } >> } >> >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >> >> if ( is_lvtt(v, irq) ) >> { >> vlapic_set_irq(vcpu_vlapic(v), irq, 0); >> } >> else if ( irq >= 0 ) >> { >> hvm_isa_irq_deassert(v->domain, irq); >> hvm_isa_irq_assert(v->domain, irq); >> } >>} >> >>static struct periodic_time *is_pt_irq( >> struct vcpu *v, struct hvm_intack intack) >>{ >> struct list_head *head = &v->arch.hvm_vcpu.tm_list; >> struct periodic_time *pt; >> struct RTCState *rtc = &v->domain->arch.hvm_domain.pl_time.vrtc; >> int vector; >> >> list_for_each_entry ( pt, head, list ) >> { >> if ( !pt->pending_intr_nr ) >> continue; >> >> if ( is_lvtt(v, pt->irq) ) >> { >> if ( pt->irq != intack.vector ) >> continue; >> return pt; >> } >> >> vector = get_isa_irq_vector(v, pt->irq, intack.source); >> >> /* RTC irq need special care */ >> if ( (intack.vector != vector) || >> ((pt->irq == 8) && !is_rtc_periodic_irq(rtc)) ) >> continue; >> >> return pt; >> } >> >> return NULL; >>} >> >>void pt_intr_post(struct vcpu *v, struct hvm_intack intack) >>{ >> struct periodic_time *pt; >> time_cb *cb; >> void *cb_priv; >> >> spin_lock(&v->arch.hvm_vcpu.tm_lock); >> >> pt = is_pt_irq(v, intack); >> if ( pt == NULL ) >> { >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >> return; >> } >> >> if ( pt->one_shot ) >> { >> pt->enabled = 0; >> list_del(&pt->list); >> } >> else >> { >> pt->pending_intr_nr--; >> if ( mode_is(v->domain, no_missed_tick_accounting) ) >> pt->last_plt_gtime = hvm_get_guest_time(v); >> else >> pt->last_plt_gtime += pt->period_cycles; >> } >> >> if ( mode_is(v->domain, delay_for_missed_ticks) && >> (hvm_get_guest_time(v) < pt->last_plt_gtime) ) >> hvm_set_guest_time(v, pt->last_plt_gtime); >> >> cb = pt->cb; >> cb_priv = pt->priv; >> >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >> >> if ( cb != NULL ) >> cb(v, cb_priv); >>} >> >>void pt_reset(struct vcpu *v) >>{ >> struct list_head *head = &v->arch.hvm_vcpu.tm_list; >> struct periodic_time *pt; >> >> spin_lock(&v->arch.hvm_vcpu.tm_lock); >> >> list_for_each_entry ( pt, head, list ) >> { >> pt->pending_intr_nr = 0; >> pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu); >> pt->scheduled = NOW() + pt->period; >> set_timer(&pt->timer, pt->scheduled); >> } >> >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >>} >> >>void pt_migrate(struct vcpu *v) >>{ >> struct list_head *head = &v->arch.hvm_vcpu.tm_list; >> struct periodic_time *pt; >> >> spin_lock(&v->arch.hvm_vcpu.tm_lock); >> >> list_for_each_entry ( pt, head, list ) >> migrate_timer(&pt->timer, v->processor); >> >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >>} >> >>void create_periodic_time( >> struct vcpu *v, struct periodic_time *pt, uint64_t period, >> uint8_t irq, char one_shot, time_cb *cb, void *data) >>{ >> destroy_periodic_time(pt); >> >> spin_lock(&v->arch.hvm_vcpu.tm_lock); >> >> pt->enabled = 1; >> pt->pending_intr_nr = 0; >> >> /* Periodic timer must be at least 0.9ms. */ >> if ( (period < 900000) && !one_shot ) >> { >> gdprintk(XENLOG_WARNING, >> "HVM_PlatformTime: program too small period %"PRIu64"\n", >> period); >> period = 900000; >> } >> >> pt->period = period; >> pt->vcpu = v; >> pt->last_plt_gtime = hvm_get_guest_time(pt->vcpu); >> pt->irq = irq; >> pt->period_cycles = (u64)period * cpu_khz / 1000000L; >> pt->one_shot = one_shot; >> pt->scheduled = NOW() + period; >> /* >> * Offset LAPIC ticks from other timer ticks. Otherwise guests which use >> * LAPIC ticks for process accounting can see long sequences of process >> * ticks incorrectly accounted to interrupt processing. >> */ >> if ( is_lvtt(v, irq) ) >> pt->scheduled += period >> 1; >> pt->cb = cb; >> pt->priv = data; >> >> list_add(&pt->list, &v->arch.hvm_vcpu.tm_list); >> >> init_timer(&pt->timer, pt_timer_fn, pt, v->processor); >> set_timer(&pt->timer, pt->scheduled); >> >> spin_unlock(&v->arch.hvm_vcpu.tm_lock); >>} >> >>void destroy_periodic_time(struct periodic_time *pt) >>{ >> if ( !pt->enabled ) >> return; >> >> pt_lock(pt); >> pt->enabled = 0; >> list_del(&pt->list); >> pt_unlock(pt); >> >> /* >> * pt_timer_fn() can run until this kill_timer() returns. We must do this >> * outside pt_lock() otherwise we can deadlock with pt_timer_fn(). >> */ >> kill_timer(&pt->timer); >>} >> >> > > > > --------------010105000402010406090500 Content-Type: text/x-patch; name="vpt.xen.un.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="vpt.xen.un.patch" diff -r 2717128cbdd1 xen/arch/x86/hvm/vpt.c --- a/xen/arch/x86/hvm/vpt.c Wed Oct 31 10:07:42 2007 +0000 +++ b/xen/arch/x86/hvm/vpt.c Thu Nov 01 16:19:06 2007 -0400 @@ -57,14 +57,23 @@ static void pt_process_missed_ticks(stru return; missed_ticks = missed_ticks / (s_time_t) pt->period + 1; - if ( missed_ticks > 1000 ) - { - /* TODO: Adjust guest time together */ - pt->pending_intr_nr++; - } - else - { - pt->pending_intr_nr += missed_ticks; + + if( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { + /* When many guests are stacked on a node and the total number of vcpus + * running loads is 10 timed the number of physical processors, + * scheduling delays of 5 seconds are common. So set the threshold at + * 100 seconds to be safe. It would be reasonable to remove this + * check against threshold completely. + */ + if ( missed_ticks > 100000 ) + { + /* TODO: Adjust guest time together */ + pt->pending_intr_nr++; + } + else + { + pt->pending_intr_nr += missed_ticks; + } } pt->scheduled += missed_ticks * pt->period; @@ -117,15 +126,7 @@ void pt_restore_timer(struct vcpu *v) list_for_each_entry ( pt, head, list ) { - if ( !mode_is(v->domain, no_missed_tick_accounting) ) - { - pt_process_missed_ticks(pt); - } - else if ( (NOW() - pt->scheduled) >= 0 ) - { - pt->pending_intr_nr++; - pt->scheduled = NOW() + pt->period; - } + pt_process_missed_ticks(pt); set_timer(&pt->timer, pt->scheduled); } @@ -140,13 +141,14 @@ static void pt_timer_fn(void *data) pt_lock(pt); - pt->pending_intr_nr++; + if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) || !pt->pending_intr_nr ) { + pt->pending_intr_nr++; + } if ( !pt->one_shot ) { pt->scheduled += pt->period; - if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) - pt_process_missed_ticks(pt); + pt_process_missed_ticks(pt); set_timer(&pt->timer, pt->scheduled); } --------------010105000402010406090500 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------010105000402010406090500-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 01 Nov 2007 17:21:36 -0400 Message-ID: <472A4360.90200@virtualiron.com> References: <472A41CA.7080405@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <472A41CA.7080405@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , "Shan, Haitao" , xen-devel@lists.xensource.com, "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org In the previous mail this (Before this patch clock error was 1.3%. The contributions of items '2' and '3' above to the improvement in accuracy are .4% and .9% respectively. should be (Before this patch clock error was 1.3%. The contributions of items '2' and '3' above to the improvement in accuracy are .9% and .4% respectively. thanks, Dave From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 02 Nov 2007 11:51:14 -0400 Message-ID: <472B4772.8070109@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Keir, I agree with removing the check. Sorry, I was a bit vague with the rounding term. My experiments show that delivering clock interrupts in a constant N*period time space results in a .4% improvement in accuracy over delivering interrupts at context switch (in) time and resetting the N*period time space. Lets call the former technique method S (synch) and the later AS (async). Now I will offer an explanation without proof. If you would like proof, I can instrument the hypervisor to see if this explanation is correct. Let D be the time that the clock vcpu is descheduled and P be the clock period. When D < P, I think there is an issue. The reason is that Linux's offset calculation, which affects the last clock interrupt tsc that is recorded, doesn't kick in if the time since the last interrupt is less than P. In this case it sets offset to zero. Linux records the tsc of the current (last) clock interrupt as (current tsc - offset). Interrupt delivery method AS delivers a clock interrupt at context switch (in) time. When D < P, Linux records the interrupt delivery time as current tsc and bumps jiffies. This results in a gain of time equal to P-D over wall time. For method S this never happens because the interrupt isn't delivered until the next boundary. regards, Dave Keir Fraser wrote: >On 1/11/07 21:14, "Dave Winchell" wrote: > > > >>1. Increase missed ticks threshold to 100 seconds. Per the comment in >>the patch, >> >> > >I assume that's the magic number 100000 in the patch? I think removing the >check entirely would be better than an arbitrarily high threshold that we're >baiscally hoping will never trigger. > > > > >>3. Call pt_process_missed_ticks() unconditionally and place the >> test for no_missed_tick_accounting inside pt_process_missed_ticks(). >> This returns the calculation for the next timer expiration >> to the original method. The original method is important >> as it sets up a theoretical time space at t=0 where expirations >> occur only at n*period, where n is any integer. This, in turn, removes >> rounding errors. >> >> > >Why do 'rounding errors' matter? I thought that no_missed_tick_accounting >was for guests which sampled the TSC on tick interrupt and used that to >determine elapsed wall time, in which case why would it matter when exactly >the tick interrupt is delivered? > > > -- Keir > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 02 Nov 2007 14:05:37 -0400 Message-ID: <472B66F1.20201@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org oops, you're right. I missed the ( (NOW() - pt->scheduled) >= 0 ). This means I will have to come up with another explanation. -Dave Keir Fraser wrote: >Thanks for the explanation. I think you are mis-describing the current >behaviour of vpt.c though (If I understand it correctly myself, which I may >not!). As I understand it, we do *not* unconditionally deliver a tick and >reset the time space when a vcpu is scheduled onto a cpu. We only do that if >(NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has >passed. Otherwise we wait to deliver the next tick exactly one period after >the previous tick was delivered. The remainder of your explanation seems to >be predicated on the (impossible?) case that we deliver a tick less than one >period after the previous one. > >Am I confused? :-) Also, what version of Linux are we talking about here, >and what periodic timer, and x86/64 or i386? There are lots of different >Linux timer-handling logics out there in the wild! > > -- Keir > >On 2/11/07 15:51, "Dave Winchell" wrote: > > > >>Let D be the time that the clock vcpu is descheduled and P be >>the clock period. When D < P, I think there is an issue. >> >>The reason is that Linux's offset calculation, which affects the >>last clock interrupt tsc that is recorded, doesn't kick in if the >>time since the last interrupt is less than P. In this case it sets >>offset to zero. Linux records the tsc of the current (last) clock interrupt >>as (current tsc - offset). >> >>Interrupt delivery method AS delivers a clock interrupt >>at context switch (in) time. When D < P, Linux records the >>interrupt delivery time as current tsc and bumps jiffies. >>This results in a gain of time equal to P-D over wall time. >> >>For method S this never happens because the interrupt isn't >>delivered until the next boundary. >> >> > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Sat, 03 Nov 2007 17:17:46 -0400 Message-ID: <472CE57A.9030804@virtualiron.com> References: <472B66F1.20201@virtualiron.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------030403080509070900000408" Return-path: In-Reply-To: <472B66F1.20201@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , "Shan, Haitao" , xen-devel@lists.xensource.com, "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------030403080509070900000408 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Keir, Thanks for applying the fixes in the last submit. In moving the test for no_missed_tick_accounting into pt_process_missed_ticks() you forgot to add the scheduling part. This patch adds the scheduling. It also defines two options for scheduling, PT_SCHEDULE_SYNC and PT_SCHEDULE_ASYNC. The default is PT_SCHEDULE_SYNC. I am not providing any means for selecting between these two options. My purpose is to have something to discuss and you can do as you like with this patch or tell me what you want. We have been discussing the difference between SYNC and ASYNC scheduling. I found the reason the code checked in as of 10.31 had an error of .23%. In the following fragment from pt_restore_timer() if ( !mode_is(v->domain, no_missed_tick_accounting) ) { pt_process_missed_ticks(pt); } else if ( (NOW() - pt->scheduled) >= 0 ) { pt->pending_intr_nr++; pt->scheduled = NOW() + pt->period; } set_timer(&pt->timer, pt->scheduled); the pt->pending_intr_nr++; line is the problem because if the value of pt->pending_intr_nr is non-zero before the increment, then you get extra timer injections. We can either test for zero, set unconditionally to 1, or leave the line out altogether and let the timeout take care of it. I have chosen the later in the patch. The result with this fixed, and with the patch submitted here is as follows with the usual test I have been describing. For both SYNC and ASYNC methods, with a sles9sp3-64 and rh4u4-64 guest running at the same time, the errors were less than .028%. I have run some tests with single guests and big loads and found sles to be in error by .1% with the ASYNC method. All of the SYNC tests have been under .03%. But my tests are short, usually 10 to 20 minutes. And there is quite a bit of variability. Looking at the timer code for the two guests, they look identical to me. In my review of the Linux code, I see no reason why the interrupt deliveries need to be "SYNC". This has been your intuition all along. So, I leave the choice up to you. thanks, Dave Dave Winchell wrote: > oops, you're right. I missed the ( (NOW() - pt->scheduled) >= 0 ). > > This means I will have to come up with another explanation. > > -Dave > > > Keir Fraser wrote: > >> Thanks for the explanation. I think you are mis-describing the current >> behaviour of vpt.c though (If I understand it correctly myself, which >> I may >> not!). As I understand it, we do *not* unconditionally deliver a tick >> and >> reset the time space when a vcpu is scheduled onto a cpu. We only do >> that if >> (NOW() - pt->scheduled) >= 0 -- that is if more than a tick period has >> passed. Otherwise we wait to deliver the next tick exactly one period >> after >> the previous tick was delivered. The remainder of your explanation >> seems to >> be predicated on the (impossible?) case that we deliver a tick less >> than one >> period after the previous one. >> >> Am I confused? :-) Also, what version of Linux are we talking about >> here, >> and what periodic timer, and x86/64 or i386? There are lots of different >> Linux timer-handling logics out there in the wild! >> >> -- Keir >> >> On 2/11/07 15:51, "Dave Winchell" wrote: >> >> >> >>> Let D be the time that the clock vcpu is descheduled and P be >>> the clock period. When D < P, I think there is an issue. >>> >>> The reason is that Linux's offset calculation, which affects the >>> last clock interrupt tsc that is recorded, doesn't kick in if the >>> time since the last interrupt is less than P. In this case it sets >>> offset to zero. Linux records the tsc of the current (last) clock >>> interrupt >>> as (current tsc - offset). >>> >>> Interrupt delivery method AS delivers a clock interrupt >>> at context switch (in) time. When D < P, Linux records the >>> interrupt delivery time as current tsc and bumps jiffies. >>> This results in a gain of time equal to P-D over wall time. >>> >>> For method S this never happens because the interrupt isn't >>> delivered until the next boundary. >>> >> >> >> >> >> > --------------030403080509070900000408 Content-Type: text/plain; name="vpt.patch.11.03" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="vpt.patch.11.03" diff -r 650cadd1b283 xen/arch/x86/hvm/vpt.c --- a/xen/arch/x86/hvm/vpt.c Fri Nov 02 16:38:11 2007 +0000 +++ b/xen/arch/x86/hvm/vpt.c Sat Nov 03 16:20:04 2007 -0400 @@ -45,12 +45,20 @@ static void pt_unlock(struct periodic_ti spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock); } +#define PT_SCHEDULE_SYNC 0 +#define PT_SCHEDULE_ASYNC 1 +int pt_missed_option = PT_SCHEDULE_SYNC; static void pt_process_missed_ticks(struct periodic_time *pt) { s_time_t missed_ticks; - if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) - return; + if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) && (pt_missed_option == PT_SCHEDULE_ASYNC)) { + if ( pt->one_shot ) + return; + if ( (NOW() - pt->scheduled) >= 0 ) + pt->scheduled = NOW() + pt->period; + return; + } if ( pt->one_shot ) return; @@ -60,7 +68,8 @@ static void pt_process_missed_ticks(stru return; missed_ticks = missed_ticks / (s_time_t) pt->period + 1; - pt->pending_intr_nr += missed_ticks; + if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) + pt->pending_intr_nr += missed_ticks; pt->scheduled += missed_ticks * pt->period; } --------------030403080509070900000408 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------030403080509070900000408-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Mon, 05 Nov 2007 09:36:43 -0500 Message-ID: <472F2A7B.4020308@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Keir Fraser wrote: >On 3/11/07 21:17, "Dave Winchell" wrote: > > > >>Thanks for applying the fixes in the last submit. >>In moving the test for no_missed_tick_accounting into >>pt_process_missed_ticks() >>you forgot to add the scheduling part. >> >> > >Actually it was deliberate, but clearly it was one code simplification too >far: thanks for spotting it! I'll go the async route, but we do need to set >pending_intr_nr to 1. We can't leave that out -- the point of the async >route is to send a tick to the vcpu immediately, since it hasn't had one for >more than a tick period. If we wait for the timeout to do that then we have >to wait a whole extra period after the vcpu is re-scheduled. > >Attached is my proposed patch. I think it's quite neat. :-) > > It looks good to me. thanks, Dave > -- Keir > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 07 Nov 2007 09:39:00 -0500 Message-ID: <4731CE04.6050105@virtualiron.com> References: <472F2A7B.4020308@virtualiron.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------020700000803030308010602" Return-path: In-Reply-To: <472F2A7B.4020308@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , "Shan, Haitao" , xen-devel@lists.xensource.com, "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------020700000803030308010602 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Keir, Attached is a fix for pt_process_missed_ticks(). Without this fix, systems run very erratically and some guests panic on boot complaining that timer interrupts are not working. As you can imagine. Also, I have some longer term measurements of the accuracy of the sync and async methods. The hardware is an eight cpu AMD box. Two eight vcpu guests, rh4u4-64 and sles9sp3-64. All vcpus running loads. Method Test duration Clock errors SYNC 56000 sec 6.4, 6.7 sec (.012%) ASYNC 52000 sec 13, 19 sec (.036%) More testing should be done to validate the significance of this difference. regards, Dave Dave Winchell wrote: > > Keir Fraser wrote: > >> On 3/11/07 21:17, "Dave Winchell" wrote: >> >> >> >>> Thanks for applying the fixes in the last submit. >>> In moving the test for no_missed_tick_accounting into >>> pt_process_missed_ticks() >>> you forgot to add the scheduling part. >>> >> >> >> Actually it was deliberate, but clearly it was one code >> simplification too >> far: thanks for spotting it! I'll go the async route, but we do need >> to set >> pending_intr_nr to 1. We can't leave that out -- the point of the async >> route is to send a tick to the vcpu immediately, since it hasn't had >> one for >> more than a tick period. If we wait for the timeout to do that then >> we have >> to wait a whole extra period after the vcpu is re-scheduled. >> >> Attached is my proposed patch. I think it's quite neat. :-) >> >> > It looks good to me. > > thanks, > Dave > >> -- Keir >> >> >> > --------------020700000803030308010602 Content-Type: text/plain; name="vpt.patch.11.07" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="vpt.patch.11.07" diff -r dfe9c0c10a2c xen/arch/x86/hvm/vpt.c --- a/xen/arch/x86/hvm/vpt.c Mon Nov 05 13:23:55 2007 +0000 +++ b/xen/arch/x86/hvm/vpt.c Wed Nov 07 08:55:45 2007 -0500 @@ -59,7 +59,7 @@ static void pt_process_missed_ticks(stru if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { pt->pending_intr_nr = 1; - pt->scheduled = now + pt->scheduled; + pt->scheduled = now + pt->period; } else { --------------020700000803030308010602 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------020700000803030308010602-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 07 Nov 2007 11:23:21 -0500 Message-ID: <4731E679.9000307@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Dong, Eddie" , Dave Winchell , xen-devel@lists.xensource.com, "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Hi Keir, You can help me analyze the Linux code. Here is the code from sles (linux-2.6.5-7.244) x86_64. static irqreturn_t timer_interrupt(int irq, void *dev_id, struct pt_regs *regs) spin_lock(&i8253_lock); outb_p(0x00, 0x43); delay = inb_p(0x40); delay |= inb(0x40) << 8; spin_unlock(&i8253_lock); delay = LATCH - 1 - delay; rdtscll_sync(&tsc); offset = (((tsc - vxtime.last_tsc) * vxtime.tsc_quot) >> 32) - (USEC_PER_SEC / HZ); if (offset < 0) offset = 0; if (offset > (USEC_PER_SEC / HZ)) { lost = offset / (USEC_PER_SEC / HZ); offset %= (USEC_PER_SEC / HZ); } monotonic_base += (tsc - vxtime.last_tsc)*1000000/cpu_khz ; vxtime.last_tsc = tsc - vxtime.quot * delay / vxtime.tsc_quot; if ((((tsc - vxtime.last_tsc) * vxtime.tsc_quot) >> 32) < offset) vxtime.last_tsc = tsc - (((long) offset << 32) / vxtime.tsc_quot) - 1; if (lost > 0) { if (report_lost_ticks) { printk(KERN_WARNING "time.c: Lost %ld timer " "tick(s)! ", lost); print_symbol("rip %s)\n", regs->rip); } jiffies += lost; } do_timer(regs); Now as 'offset' gets modified it represents the remainder in the lost calculation. Our data indicates that when offset is close to zero, timekeeping is more accurate. So this leads us to this line of code: if ((((tsc - vxtime.last_tsc) * vxtime.tsc_quot) >> 32) < offset) vxtime.last_tsc = tsc - (((long) offset << 32) / vxtime.tsc_quot) - 1; Thus the problem seems to be the way the code switches from adjusting last_tsc by the delay which is the normal mode when interrupts are timely or SYNCed to handling offsets larger than the delay. I'm concerned about the way delay is blown off in the final calculation vxtime.last_tsc = tsc - (((long) offset << 32) / vxtime.tsc_quot) - 1; If an interrupt is right on time, and the delay is the same as last time, then the offset should be zero in my mind. In the code above, offset will equal delay for this example. What do you think? thanks, Dave Keir Fraser wrote: >Oh dear, that was a silly typo on my part. Thanks for tracking it down! > >Well, I'm surprised that SYNC vs ASYNC differs, if the guest is really >tracking time via the TSC value. But it sounds like the actual time-tracking >algorithm in the guest must be a bit more complicated than that? I'd be very >happy to help with any further analysis. > > -- Keir > >On 7/11/07 14:39, "Dave Winchell" wrote: > > > >>Keir, >> >>Attached is a fix for pt_process_missed_ticks(). >>Without this fix, systems run very erratically and some >>guests panic on boot complaining that timer interrupts >>are not working. As you can imagine. >> >>Also, I have some longer term measurements of the >>accuracy of the sync and async methods. >>The hardware is an eight cpu AMD box. Two eight >>vcpu guests, rh4u4-64 and sles9sp3-64. >>All vcpus running loads. >> >>Method Test duration Clock errors >> >>SYNC 56000 sec 6.4, 6.7 sec (.012%) >>ASYNC 52000 sec 13, 19 sec (.036%) >> >>More testing should be done to validate the significance >>of this difference. >> >>regards, >>Dave >> >> >>Dave Winchell wrote: >> >> >> >>>Keir Fraser wrote: >>> >>> >>> >>>>On 3/11/07 21:17, "Dave Winchell" wrote: >>>> >>>> >>>> >>>> >>>> >>>>>Thanks for applying the fixes in the last submit. >>>>>In moving the test for no_missed_tick_accounting into >>>>>pt_process_missed_ticks() >>>>>you forgot to add the scheduling part. >>>>> >>>>> >>>>> >>>>Actually it was deliberate, but clearly it was one code >>>>simplification too >>>>far: thanks for spotting it! I'll go the async route, but we do need >>>>to set >>>>pending_intr_nr to 1. We can't leave that out -- the point of the async >>>>route is to send a tick to the vcpu immediately, since it hasn't had >>>>one for >>>>more than a tick period. If we wait for the timeout to do that then >>>>we have >>>>to wait a whole extra period after the vcpu is re-scheduled. >>>> >>>>Attached is my proposed patch. I think it's quite neat. :-) >>>> >>>> >>>> >>>> >>>It looks good to me. >>> >>>thanks, >>>Dave >>> >>> >>> >>>>-- Keir >>>> >>>> >>>> >>>> >>>> >>diff -r dfe9c0c10a2c xen/arch/x86/hvm/vpt.c >>--- a/xen/arch/x86/hvm/vpt.c Mon Nov 05 13:23:55 2007 +0000 >>+++ b/xen/arch/x86/hvm/vpt.c Wed Nov 07 08:55:45 2007 -0500 >>@@ -59,7 +59,7 @@ static void pt_process_missed_ticks(stru >> if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) >> { >> pt->pending_intr_nr = 1; >>- pt->scheduled = now + pt->scheduled; >>+ pt->scheduled = now + pt->period; >> } >> else >> { >> >> > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 08 Nov 2007 09:43:16 -0500 Message-ID: <47332084.8090305@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Shan, Haitao" , Dave Winchell , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Hi Keir, I've added comments below. See my next mail on some interesting performance numbers. thanks, Dave Keir Fraser wrote: >On 7/11/07 19:38, "Dave Winchell" wrote: > > > >>My feeling is that we should go full SYNC. Yes, in theory the >>guests should be able to handle ASYNC, but in reality it appears that >>some do not. Since it is easy for us to give them SYNC, >>lets just do it and not stress them out. >> >> > >One problem with pure SYNC is there's a fair chance you won't deliver any >ticks at all for a long time, if the guest only runs in short bursts (e.g., >I/O bound) and happens not to be running on any tick boundary. I'm not sure >how much that matters. It could cause time goes backwards if the time >extrapolation via the TSC is not perfectly accurate, or cause problems if >there are any assumptions that TSC delta since last tick fits in 32 bits >(less likely in x64 code I suppose). Anyway, my point is that only testing >VCPUs under full load may cause us to optimise in ways that have nasty >unexpected effects for other workloads. > > I agree that this could be a problem. I have an idea that could give us full SYNC and eliminate the long periods without clock interrupts. In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1. In pt_save_timer(): list_for_each_entry ( pt, head, list ) if(!pt->run_timer) stop_timer(&pt->timer); And in pt_timer_fn(): pt->run_timer = 0; So, for a guest that misses a tick, we will interrupt him once from the descheduled state and then leave him alone in the descheduled state. > > >>For default mode as checked into unstable is now, >>64 bit guests should run quite fast as missed is calculated and then a bunch >>of additional interrupts are delivered. On the other hand >>32bit guests very well in default mode. >> >>For the original code, before we put in the constant tsc offset business, >>64bit guests run poorly and 32bit quests very well time-wise. >> >> > >The default mode hasn't changed. Are you under the impression that >missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 guests >run badly with that because they treat every one of the missed ticks they >receive as a full tick. > > Sorry, I was confused. However, the default mode will still run poorly for 64 bit guests because of the pending_nr's accumulated while the guest has interrupts disabled. As I recall, the effect is quite large, on the order of 10% error. I'll get you a number later today. > -- Keir > > > >>>Or is the lack of >>>synchronization of TSCs across VCPUs causing issues that you're trying to >>>avoid? >>> >>> >>This does cause issues, but its not the only contributor to poor timing. >>Having TSCs synchronized across vcpus will help some of the time going >>backwards problems we have seen, I think. >> >>Regards, >>Dave >> >>Keir Fraser wrote: >> >> >> >>>On 7/11/07 17:29, "Keir Fraser" wrote: >>> >>> >>> >>> >>> >>>>So, you can see we send an interrupt immediately (and ASYNC) if any ticks >>>>have been missed, but then successive ticks are delivered 'on the beat'. A >>>>possible middleground? Or perhaps we should just go with SYNC after all... >>>> >>>> >>>> >>>> >>>How do these Linux x64 guests fare with the original and default timer mode, >>>by the way? I would expect that time should be accounted pretty accurately >>>in that mode, albeit with more interrupts than you'd like. Or is the lack of >>>synchronisation of TSCs across VCPUs causing issues that you're trying to >>>avoid? >>> >>>-- Keir >>> >>> >>> >>> >>> >>> > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 07 Nov 2007 14:38:56 -0500 Message-ID: <47321450.8090107@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Shan, Haitao" , Dave Winchell , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Hi Keir, My feeling is that we should go full SYNC. Yes, in theory the guests should be able to handle ASYNC, but in reality it appears that some do not. Since it is easy for us to give them SYNC, lets just do it and not stress them out. In terms of sending the one ASYNC interrupt, I would say no. Lets simply give it to him on the next boundary. If we give it to him ASYNC he will have a big offset to deal with, exactly what gives him trouble. > How do these Linux x64 guests fare with the original and default timer mode, > by the way? I would expect that time should be accounted pretty accurately > in that mode, albeit with more interrupts than you'd like. For default mode as checked into unstable is now, 64 bit guests should run quite fast as missed is calculated and then a bunch of additional interrupts are delivered. On the other hand 32bit guests very well in default mode. For the original code, before we put in the constant tsc offset business, 64bit guests run poorly and 32bit quests very well time-wise. > Or is the lack of > synchronization of TSCs across VCPUs causing issues that you're trying to > avoid? This does cause issues, but its not the only contributor to poor timing. Having TSCs synchronized across vcpus will help some of the time going backwards problems we have seen, I think. Regards, Dave Keir Fraser wrote: >On 7/11/07 17:29, "Keir Fraser" wrote: > > > >>So, you can see we send an interrupt immediately (and ASYNC) if any ticks >>have been missed, but then successive ticks are delivered 'on the beat'. A >>possible middleground? Or perhaps we should just go with SYNC after all... >> >> > >How do these Linux x64 guests fare with the original and default timer mode, >by the way? I would expect that time should be accounted pretty accurately >in that mode, albeit with more interrupts than you'd like. Or is the lack of >synchronisation of TSCs across VCPUs causing issues that you're trying to >avoid? > > -- Keir > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 08 Nov 2007 10:08:23 -0500 Message-ID: <47332667.1090701@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Shan, Haitao" , Dave Winchell , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org ok, I'll try your new code and let you know what I find. -Dave Keir Fraser wrote: >On 8/11/07 14:43, "Dave Winchell" wrote: > > > >>I agree that this could be a problem. I have an idea that could give us full >>SYNC and eliminate the long periods without clock interrupts. >>In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1. >>In pt_save_timer(): >> >> list_for_each_entry ( pt, head, list ) >> if(!pt->run_timer) >> stop_timer(&pt->timer); >> >>And in pt_timer_fn(): >> >> pt->run_timer = 0; >> >>So, for a guest that misses a tick, we will interrupt him once from the >>descheduled state and then leave him alone in the descheduled state. >> >> > >Well, I'd rather not complicate the code if it's avoidable. I checked in a >SYNC/ASYNC combo and code simplification as changeset 16341, and it'd be >interesting to know how that fares against your suggested scheme. I suppose >as long as we're better than ntpd's tolerance it doesn't actually matter all >that much. > > -- Keir > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 08 Nov 2007 09:57:57 -0500 Message-ID: <473323F5.4090906@virtualiron.com> References: <47332084.8090305@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47332084.8090305@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Shan, Haitao" , Dave Winchell , "Dong, Eddie" , xen-devel@lists.xensource.com, "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Keir, I ran a 24 hour (23hr:40min) test. The usual setup. Protocol was ASYNC. Errors: sles9sp3-64: -4.96 sec -.0058% rh4u4-64: +4.42 sec +.0052% So, lets leave it ASYNC unless someone produces some test cases where the error gets up to close to .05%. I'll do some testing here with overnight runs or, perhaps, different loads. thanks, Dave Dave Winchell wrote: > Hi Keir, > > I've added comments below. > See my next mail on some interesting performance numbers. > > thanks, > Dave > > Keir Fraser wrote: > >> On 7/11/07 19:38, "Dave Winchell" wrote: >> >> >> >>> My feeling is that we should go full SYNC. Yes, in theory the >>> guests should be able to handle ASYNC, but in reality it appears that >>> some do not. Since it is easy for us to give them SYNC, >>> lets just do it and not stress them out. >>> >> >> >> One problem with pure SYNC is there's a fair chance you won't deliver >> any >> ticks at all for a long time, if the guest only runs in short bursts >> (e.g., >> I/O bound) and happens not to be running on any tick boundary. I'm >> not sure >> how much that matters. It could cause time goes backwards if the time >> extrapolation via the TSC is not perfectly accurate, or cause >> problems if >> there are any assumptions that TSC delta since last tick fits in 32 bits >> (less likely in x64 code I suppose). Anyway, my point is that only >> testing >> VCPUs under full load may cause us to optimise in ways that have nasty >> unexpected effects for other workloads. >> >> > I agree that this could be a problem. I have an idea that could give > us full > SYNC and eliminate the long periods without clock interrupts. > In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1. > In pt_save_timer(): > > list_for_each_entry ( pt, head, list ) > if(!pt->run_timer) > stop_timer(&pt->timer); > > And in pt_timer_fn(): > > pt->run_timer = 0; > > So, for a guest that misses a tick, we will interrupt him once from the > descheduled state and then leave him alone in the descheduled state. > >> >> >>> For default mode as checked into unstable is now, >>> 64 bit guests should run quite fast as missed is calculated and then >>> a bunch >>> of additional interrupts are delivered. On the other hand >>> 32bit guests very well in default mode. >>> >>> For the original code, before we put in the constant tsc offset >>> business, >>> 64bit guests run poorly and 32bit quests very well time-wise. >>> >> >> >> The default mode hasn't changed. Are you under the impression that >> missed-ticks-but-no-delay-of-tsc is the default mode now? I know x64 >> guests >> run badly with that because they treat every one of the missed ticks >> they >> receive as a full tick. >> >> > Sorry, I was confused. > However, the default mode will still run poorly for 64 bit guests because > of the pending_nr's accumulated while the guest has interrupts disabled. > As I recall, the effect is quite large, on the order of 10% error. > I'll get you a number later today. > >> -- Keir >> >> >> >>>> Or is the lack of >>>> synchronization of TSCs across VCPUs causing issues that you're >>>> trying to >>>> avoid? >>>> >>> >>> This does cause issues, but its not the only contributor to poor >>> timing. >>> Having TSCs synchronized across vcpus will help some of the time going >>> backwards problems we have seen, I think. >>> >>> Regards, >>> Dave >>> >>> Keir Fraser wrote: >>> >>> >>> >>>> On 7/11/07 17:29, "Keir Fraser" wrote: >>>> >>>> >>>> >>>> >>>> >>>>> So, you can see we send an interrupt immediately (and ASYNC) if >>>>> any ticks >>>>> have been missed, but then successive ticks are delivered 'on the >>>>> beat'. A >>>>> possible middleground? Or perhaps we should just go with SYNC >>>>> after all... >>>>> >>>>> >>>> >>>> How do these Linux x64 guests fare with the original and default >>>> timer mode, >>>> by the way? I would expect that time should be accounted pretty >>>> accurately >>>> in that mode, albeit with more interrupts than you'd like. Or is >>>> the lack of >>>> synchronisation of TSCs across VCPUs causing issues that you're >>>> trying to >>>> avoid? >>>> >>>> -- Keir >>>> >>>> >>>> >>>> >>>> >>> >> >> >> >> > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 09 Nov 2007 14:22:23 -0500 Message-ID: <4734B36F.1080408@virtualiron.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------040201020300020309030903" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Shan, Haitao" , Dave Winchell , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------040201020300020309030903 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Keir, Here are the results of the testing I've done recently. There are three protocols tested. I'll call them ASYNC for the code as it stood just before your last check-in, MIXED (for ASYNC+SYNC) for the code you recently checked in, and SYNC for the method I proposed in the last mail. Date Duration Protocol sles, rhat error 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% Each protocol was run a couple of times to gage the repeatability. Since I had a high error (~.03%) for the ASYNC method a couple of days ago, I ran another ASYNC test. I think there may have been something wrong with the code I used a couple of days ago for ASYNC. It may have been missing the immediate delivery of interrupt after context switch in. My results indicate that either SYNC or ASYNC give acceptable accuracy, each running consistently around or under .01%. MIXED has a fairly high error of greater than .03%. Probably too close to .05% ntp threshold for comfort. I don't have an overnight run with SYNC. I plan to leave SYNC running over the weekend. If you'd rather I can leave MIXED running instead. It may be too early to pick the protocol and I can run more overnight tests next week. I have attached a patch based on what's checked-in now that shows what I tested for SYNC and ASYNC. For MIXED I used the code that is checked-in now. The patch looks more complicated than the change really is as it backs out the MIXED code you put in. Regards, Dave Keir Fraser wrote: >On 8/11/07 14:43, "Dave Winchell" wrote: > > > >>I agree that this could be a problem. I have an idea that could give us full >>SYNC and eliminate the long periods without clock interrupts. >>In pt_process_missed_ticks() when missed_ticks > 0 set pt->run_timer = 1. >>In pt_save_timer(): >> >> list_for_each_entry ( pt, head, list ) >> if(!pt->run_timer) >> stop_timer(&pt->timer); >> >>And in pt_timer_fn(): >> >> pt->run_timer = 0; >> >>So, for a guest that misses a tick, we will interrupt him once from the >>descheduled state and then leave him alone in the descheduled state. >> >> > >Well, I'd rather not complicate the code if it's avoidable. I checked in a >SYNC/ASYNC combo and code simplification as changeset 16341, and it'd be >interesting to know how that fares against your suggested scheme. I suppose >as long as we're better than ntpd's tolerance it doesn't actually matter all >that much. > > -- Keir > > > > --------------040201020300020309030903 Content-Type: text/x-patch; name="vpt.11.09.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="vpt.11.09.patch" diff -r c0bdfda5183d xen/arch/x86/hvm/vpt.c --- a/xen/arch/x86/hvm/vpt.c Thu Nov 08 14:50:01 2007 +0000 +++ b/xen/arch/x86/hvm/vpt.c Fri Nov 09 09:08:59 2007 -0500 @@ -45,6 +45,26 @@ static void pt_unlock(struct periodic_ti spin_unlock(&pt->vcpu->arch.hvm_vcpu.tm_lock); } +#include +#define PT_MISSED_OPTION_SYNC 0 +#define PT_MISSED_OPTION_ASYNC 1 +int pt_missed_option = 0; +static void pt_missed_option_fn(unsigned char key) +{ + pt_missed_option = pt_missed_option ^ 1; + if(pt_missed_option == PT_MISSED_OPTION_SYNC) + printk("pt_missed_option = 0 (SYNC)\n"); + else + printk("pt_missed_option = 1 (ASYNC)\n"); +} +static int __init pt_missed_option_init(void) +{ + register_keyhandler('D', pt_missed_option_fn, "pt_missed_option"); + return 0; +} + +__initcall(pt_missed_option_init); + static void pt_process_missed_ticks(struct periodic_time *pt) { s_time_t missed_ticks, now = NOW(); @@ -56,9 +76,20 @@ static void pt_process_missed_ticks(stru if ( missed_ticks <= 0 ) return; - missed_ticks = missed_ticks / (s_time_t) pt->period + 1; - pt->pending_intr_nr += missed_ticks; - pt->scheduled += missed_ticks * pt->period; + if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) && (pt_missed_option == PT_MISSED_OPTION_ASYNC) ) + { + pt->pending_intr_nr = 1; + pt->scheduled = now + pt->period; + } + else + { + missed_ticks = missed_ticks / (s_time_t) pt->period + 1; + if ( !mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) + pt->pending_intr_nr += missed_ticks; + else + pt->run_timer = 1; + pt->scheduled += missed_ticks * pt->period; + } } static void pt_freeze_time(struct vcpu *v) @@ -91,8 +122,10 @@ void pt_save_timer(struct vcpu *v) spin_lock(&v->arch.hvm_vcpu.tm_lock); - list_for_each_entry ( pt, head, list ) - stop_timer(&pt->timer); + list_for_each_entry ( pt, head, list ) { + if(!pt->run_timer) + stop_timer(&pt->timer); + } pt_freeze_time(v); @@ -123,7 +156,12 @@ static void pt_timer_fn(void *data) pt_lock(pt); - pt->pending_intr_nr++; + if ( mode_is(pt->vcpu->domain, no_missed_tick_accounting) ) { + pt->pending_intr_nr = 1; + pt->run_timer = 0; + } + else + pt->pending_intr_nr++; if ( !pt->one_shot ) { @@ -224,16 +262,11 @@ void pt_intr_post(struct vcpu *v, struct } else { + pt->pending_intr_nr--; if ( mode_is(v->domain, no_missed_tick_accounting) ) - { pt->last_plt_gtime = hvm_get_guest_time(v); - pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */ - } else - { pt->last_plt_gtime += pt->period_cycles; - pt->pending_intr_nr--; - } } if ( mode_is(v->domain, delay_for_missed_ticks) && diff -r c0bdfda5183d xen/include/asm-x86/hvm/vpt.h --- a/xen/include/asm-x86/hvm/vpt.h Thu Nov 08 14:50:01 2007 +0000 +++ b/xen/include/asm-x86/hvm/vpt.h Fri Nov 09 08:53:14 2007 -0500 @@ -84,6 +84,7 @@ struct periodic_time { struct timer timer; /* ac_timer */ time_cb *cb; void *priv; /* point back to platform time source */ + u32 run_timer; }; --------------040201020300020309030903 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------040201020300020309030903-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Mon, 12 Nov 2007 10:37:30 -0500 Message-ID: <4738733A.2090407@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Shan, Haitao" , Dave Winchell , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Keir, I'll run the I/O and idle tests you suggested. Since the tests will be run in the evenings, they won't be completed until the end of the week. Below is the table of tests updated with the weekend SYNC with cpu load test, where the error was less than .01% for both linux guests. > I'm a bit worried about any unwanted side effects of the SYNC+run_timer > approach -- e.g., whether timer wakeups will cause higher system-wide CPU > contention. I agree. If the SYNC model turns out to be desirable from an accuracy standpoint, then the significance of the additional wakeups should characterized. Regards, Dave Date Duration Protocol sles, rhat error load 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu Keir Fraser wrote: >On 9/11/07 19:22, "Dave Winchell" wrote: > > > >>Since I had a high error (~.03%) for the ASYNC method a couple of days ago, >>I ran another ASYNC test. I think there may have been something >>wrong with the code I used a couple of days ago for ASYNC. It may have been >>missing the immediate delivery of interrupt after context switch in. >> >>My results indicate that either SYNC or ASYNC give acceptable accuracy, >>each running consistently around or under .01%. MIXED has a fairly high >>error of >>greater than .03%. Probably too close to .05% ntp threshold for comfort. >>I don't have an overnight run with SYNC. I plan to leave SYNC running >>over the weekend. If you'd rather I can leave MIXED running instead. >> >>It may be too early to pick the protocol and I can run more overnight tests >>next week. >> >> > >I'm a bit worried about any unwanted side effects of the SYNC+run_timer >approach -- e.g., whether timer wakeups will cause higher system-wide CPU >contention. I find it easier to think through the implications of ASYNC. I'm >surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it >delivers more timer interrupts than the other approaches, and each interrupt >event causes a small accumulated error? > >Overall I would consider MIXED and ASYNC as favourites and if the latter is >actually more accurate then I can simply revert the changeset that >implemented MIXED. > >Perhaps rather than running more of the same workloads you could try idle >VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We >don't have any data on workloads that aren't CPU bound, so that's really an >obvious place to put any further effort imo. > > -- Keir > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Mon, 26 Nov 2007 15:57:24 -0500 Message-ID: <474B3334.8010906@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "Shan, Haitao" , Dave Winchell , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Keir, The accuracy data I've collected for i/o loads for the various time protocols follows. In addition, the data for cpu loads is shown. The loads labeled cpu and i/o-8 are on an 8 processor AMD box. Two guests, red hat and sles 64 bit, 8 vcpu each. The cpu load is usex -e36 on each guest. (usex is available at http://people.redhat.com/anderson/usex.) i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. The loads labeled i/o-32 are 32 instances of dd. Also, these are run on 4 cpu AMD box. In addition, there is an idle rh-32bit guest. All three guests are 8vcpu. The loads labeled i/o-4/32 are the same as i/o-32 except that the redhat-64 guest has 4 instances of dd. Date Duration Protocol sles, rhat error load 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 Overhead measurements: Progress in terms of number of passes through a fixed system workload on an 8 vcpu red hat with an 8 vcpu sles idle. The workload was usex -b48. ASYNC 167 min 145 passes .868 passes/min SYNC 167 min 144 passes .862 passes/min SYNC 1065 min 919 passes .863 passes/min MIXED 221 min 196 passes .887 passes/min Conclusions: The only protocol which meets the .05% accuracy requirement for ntp tracking under the loads above is the SYNC protocol. The worst case accuracies for SYNC, MIXED, and ASYNC are .022%, .12%, and .14%, respectively. We could reduce the cost of the SYNC method by only scheduling the extra wakeups if a certain number of ticks are missed. Regards, Dave Keir Fraser wrote: >On 9/11/07 19:22, "Dave Winchell" wrote: > > > >>Since I had a high error (~.03%) for the ASYNC method a couple of days ago, >>I ran another ASYNC test. I think there may have been something >>wrong with the code I used a couple of days ago for ASYNC. It may have been >>missing the immediate delivery of interrupt after context switch in. >> >>My results indicate that either SYNC or ASYNC give acceptable accuracy, >>each running consistently around or under .01%. MIXED has a fairly high >>error of >>greater than .03%. Probably too close to .05% ntp threshold for comfort. >>I don't have an overnight run with SYNC. I plan to leave SYNC running >>over the weekend. If you'd rather I can leave MIXED running instead. >> >>It may be too early to pick the protocol and I can run more overnight tests >>next week. >> >> > >I'm a bit worried about any unwanted side effects of the SYNC+run_timer >approach -- e.g., whether timer wakeups will cause higher system-wide CPU >contention. I find it easier to think through the implications of ASYNC. I'm >surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it >delivers more timer interrupts than the other approaches, and each interrupt >event causes a small accumulated error? > >Overall I would consider MIXED and ASYNC as favourites and if the latter is >actually more accurate then I can simply revert the changeset that >implemented MIXED. > >Perhaps rather than running more of the same workloads you could try idle >VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We >don't have any data on workloads that aren't CPU bound, so that's really an >obvious place to put any further effort imo. > > -- Keir > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 06 Dec 2007 11:57:10 +0000 Message-ID: References: <474B3334.8010906@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <474B3334.8010906@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "Shan, Haitao" , xen-devel@lists.xensource.com, "Dong, Eddie" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Please take a look at xen-unstable changeset 16545. -- Keir On 26/11/07 20:57, "Dave Winchell" wrote: > Keir, > > The accuracy data I've collected for i/o loads for the > various time protocols follows. In addition, the data > for cpu loads is shown. > > The loads labeled cpu and i/o-8 are on an 8 processor AMD box. > Two guests, red hat and sles 64 bit, 8 vcpu each. > The cpu load is usex -e36 on each guest. > (usex is available at http://people.redhat.com/anderson/usex.) > i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. > > The loads labeled i/o-32 are 32 instances of dd. > Also, these are run on 4 cpu AMD box. > In addition, there is an idle rh-32bit guest. > All three guests are 8vcpu. > > The loads labeled i/o-4/32 are the same as i/o-32 > except that the redhat-64 guest has 4 instances of dd. > > Date Duration Protocol sles, rhat error load > > 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu > 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu > > 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu > 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu > 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu > > 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu > 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu > > > 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 > 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 > > 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 > 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 > > 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 > 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 > > > > 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 > 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 > 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 > > 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 > 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 > > > Overhead measurements: > > Progress in terms of number of passes through a fixed system workload > on an 8 vcpu red hat with an 8 vcpu sles idle. > The workload was usex -b48. > > > ASYNC 167 min 145 passes .868 passes/min > SYNC 167 min 144 passes .862 passes/min > SYNC 1065 min 919 passes .863 passes/min > MIXED 221 min 196 passes .887 passes/min > > > Conclusions: > > The only protocol which meets the .05% accuracy requirement for ntp > tracking under the loads > above is the SYNC protocol. The worst case accuracies for SYNC, MIXED, > and ASYNC > are .022%, .12%, and .14%, respectively. > > We could reduce the cost of the SYNC method by only scheduling the extra > wakeups if a certain number > of ticks are missed. > > Regards, > Dave > > Keir Fraser wrote: > >> On 9/11/07 19:22, "Dave Winchell" wrote: >> >> >> >>> Since I had a high error (~.03%) for the ASYNC method a couple of days ago, >>> I ran another ASYNC test. I think there may have been something >>> wrong with the code I used a couple of days ago for ASYNC. It may have been >>> missing the immediate delivery of interrupt after context switch in. >>> >>> My results indicate that either SYNC or ASYNC give acceptable accuracy, >>> each running consistently around or under .01%. MIXED has a fairly high >>> error of >>> greater than .03%. Probably too close to .05% ntp threshold for comfort. >>> I don't have an overnight run with SYNC. I plan to leave SYNC running >>> over the weekend. If you'd rather I can leave MIXED running instead. >>> >>> It may be too early to pick the protocol and I can run more overnight tests >>> next week. >>> >>> >> >> I'm a bit worried about any unwanted side effects of the SYNC+run_timer >> approach -- e.g., whether timer wakeups will cause higher system-wide CPU >> contention. I find it easier to think through the implications of ASYNC. I'm >> surprised that MIXED loses time, and is less accurate than ASYNC. Perhaps it >> delivers more timer interrupts than the other approaches, and each interrupt >> event causes a small accumulated error? >> >> Overall I would consider MIXED and ASYNC as favourites and if the latter is >> actually more accurate then I can simply revert the changeset that >> implemented MIXED. >> >> Perhaps rather than running more of the same workloads you could try idle >> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads to /dev/null)? We >> don't have any data on workloads that aren't CPU bound, so that's really an >> obvious place to put any further effort imo. >> >> -- Keir >> >> >> >> > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 19 Dec 2007 11:57:00 -0700 Message-ID: <20071219115700531.00000002296@djm-pc> References: Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser , Dave Winchell Cc: "Dong, Eddie" , "xen-devel@lists.xensource.com" , "Shan, Haitao" , "Jiang, Yunhong" List-Id: xen-devel@lists.xenproject.org Anyone make measurements on the final patch? I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no= load. This was xen-unstable tip today with no options specified. 32-bit = was about 0.01%. I think I missed something... how do I run the various accounting choices a= nd which ones are known to be appropriate for which kernels? Thanks, Dan > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser > Sent: Thursday, December 06, 2007 4:57 AM > To: Dave Winchell > Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, > Yunhong > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Please take a look at xen-unstable changeset 16545. > = > -- Keir > = > On 26/11/07 20:57, "Dave Winchell" wrote: > = > > Keir, > > > > The accuracy data I've collected for i/o loads for the > > various time protocols follows. In addition, the data > > for cpu loads is shown. > > > > The loads labeled cpu and i/o-8 are on an 8 processor AMD box. > > Two guests, red hat and sles 64 bit, 8 vcpu each. > > The cpu load is usex -e36 on each guest. > > (usex is available at http://people.redhat.com/anderson/usex.) > > i/o load is 8 instances of dd if=3D/dev/hda6 of=3D/dev/null. > > > > The loads labeled i/o-32 are 32 instances of dd. > > Also, these are run on 4 cpu AMD box. > > In addition, there is an idle rh-32bit guest. > > All three guests are 8vcpu. > > > > The loads labeled i/o-4/32 are the same as i/o-32 > > except that the redhat-64 guest has 4 instances of dd. > > > > Date Duration Protocol sles, rhat error load > > > > 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu > > 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu > > > > 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu > > 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu > > 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu > > > > 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu > > 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu > > > > > > 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 > > 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 > > > > 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 > > 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 > > > > 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 > > 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 > > > > > > > > 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 > > 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 > > 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 > > > > 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 > > 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 > > > > > > Overhead measurements: > > > > Progress in terms of number of passes through a fixed = > system workload > > on an 8 vcpu red hat with an 8 vcpu sles idle. > > The workload was usex -b48. > > > > > > ASYNC 167 min 145 passes .868 passes/min > > SYNC 167 min 144 passes .862 passes/min > > SYNC 1065 min 919 passes .863 passes/min > > MIXED 221 min 196 passes .887 passes/min > > > > > > Conclusions: > > > > The only protocol which meets the .05% accuracy requirement for ntp > > tracking under the loads > > above is the SYNC protocol. The worst case accuracies for = > SYNC, MIXED, > > and ASYNC > > are .022%, .12%, and .14%, respectively. > > > > We could reduce the cost of the SYNC method by only = > scheduling the extra > > wakeups if a certain number > > of ticks are missed. > > > > Regards, > > Dave > > > > Keir Fraser wrote: > > > >> On 9/11/07 19:22, "Dave Winchell" = > wrote: > >> > >> > >> > >>> Since I had a high error (~.03%) for the ASYNC method a = > couple of days ago, > >>> I ran another ASYNC test. I think there may have been something > >>> wrong with the code I used a couple of days ago for = > ASYNC. It may have been > >>> missing the immediate delivery of interrupt after context = > switch in. > >>> > >>> My results indicate that either SYNC or ASYNC give = > acceptable accuracy, > >>> each running consistently around or under .01%. MIXED has = > a fairly high > >>> error of > >>> greater than .03%. Probably too close to .05% ntp = > threshold for comfort. > >>> I don't have an overnight run with SYNC. I plan to leave = > SYNC running > >>> over the weekend. If you'd rather I can leave MIXED = > running instead. > >>> > >>> It may be too early to pick the protocol and I can run = > more overnight tests > >>> next week. > >>> > >>> > >> > >> I'm a bit worried about any unwanted side effects of the = > SYNC+run_timer > >> approach -- e.g., whether timer wakeups will cause higher = > system-wide CPU > >> contention. I find it easier to think through the = > implications of ASYNC. I'm > >> surprised that MIXED loses time, and is less accurate than = > ASYNC. Perhaps it > >> delivers more timer interrupts than the other approaches, = > and each interrupt > >> event causes a small accumulated error? > >> > >> Overall I would consider MIXED and ASYNC as favourites and = > if the latter is > >> actually more accurate then I can simply revert the changeset that > >> implemented MIXED. > >> > >> Perhaps rather than running more of the same workloads you = > could try idle > >> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads = > to /dev/null)? We > >> don't have any data on workloads that aren't CPU bound, so = > that's really an > >> obvious place to put any further effort imo. > >> > >> -- Keir > >> > >> > >> > >> > > > = > = > = > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > = > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 3 Jan 2008 15:57:52 -0700 Message-ID: <20080103155752531.00000000500@djm-pc> References: <476971DE.4010102@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <476971DE.4010102@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Hi Dave -- Did you get your correction ported? If so, it would be nice to see this ge= t into 3.1.3. Note that I just did some very limited testing with timer_mode=3D2(=3DSYNC=3D= no missed ticks pending) on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've = seen so far is 0.012%. But I haven't tried any exotic loads, just LTP. Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Wednesday, December 19, 2007 12:33 PM > To: dan.magenheimer@oracle.com > Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong, > Eddie; Jiang, Yunhong; Dave Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Dan, > = > I did some testing with the constant tsc offset SYNC method = > (now called > no_missed_ticks_pending) > and found the error to be very high, much larger than 1 %, as = > I recall. > I have not had a chance to submit a correction. I will try to = > do it later > this week or the first week in January. My version of constant tsc > offset SYNC method > produces .02 % error, so I just need to port that into the = > current code. > = > The error you got for both of those kernels is what I would expect > for the default mode, delay_for_missed_ticks. > = > I'll let Keir answer on how to set the time mode. > = > Regards, > Dave > = > Dan Magenheimer wrote: > = > >Anyone make measurements on the final patch? > > > >I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of = > about 0.2% with no load. This was xen-unstable tip today = > with no options specified. 32-bit was about 0.01%. > > > >I think I missed something... how do I run the various = > accounting choices and which ones are known to be appropriate = > for which kernels? > > > >Thanks, > >Dan > > > > > > > >>-----Original Message----- > >>From: xen-devel-bounces@lists.xensource.com > >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of = > Keir Fraser > >>Sent: Thursday, December 06, 2007 4:57 AM > >>To: Dave Winchell > >>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, > >>Yunhong > >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>disables pending > >>missed ticks > >> > >> > >>Please take a look at xen-unstable changeset 16545. > >> > >> -- Keir > >> > >>On 26/11/07 20:57, "Dave Winchell" = > wrote: > >> > >> > >> > >>>Keir, > >>> > >>>The accuracy data I've collected for i/o loads for the > >>>various time protocols follows. In addition, the data > >>>for cpu loads is shown. > >>> > >>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. > >>>Two guests, red hat and sles 64 bit, 8 vcpu each. > >>>The cpu load is usex -e36 on each guest. > >>>(usex is available at http://people.redhat.com/anderson/usex.) > >>>i/o load is 8 instances of dd if=3D/dev/hda6 of=3D/dev/null. > >>> > >>>The loads labeled i/o-32 are 32 instances of dd. > >>>Also, these are run on 4 cpu AMD box. > >>>In addition, there is an idle rh-32bit guest. > >>>All three guests are 8vcpu. > >>> > >>>The loads labeled i/o-4/32 are the same as i/o-32 > >>>except that the redhat-64 guest has 4 instances of dd. > >>> > >>>Date Duration Protocol sles, rhat error load > >>> > >>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu > >>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu > >>> > >>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu > >>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu > >>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu > >>> > >>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu > >>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu > >>> > >>> > >>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 > >>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 > >>> > >>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 > >>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 > >>> > >>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 > >>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 > >>> > >>> > >>> > >>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 > >>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 > >>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 > >>> > >>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 > >>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 > >>> > >>> > >>>Overhead measurements: > >>> > >>>Progress in terms of number of passes through a fixed > >>> > >>> > >>system workload > >> > >> > >>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>The workload was usex -b48. > >>> > >>> > >>>ASYNC 167 min 145 passes .868 passes/min > >>>SYNC 167 min 144 passes .862 passes/min > >>>SYNC 1065 min 919 passes .863 passes/min > >>>MIXED 221 min 196 passes .887 passes/min > >>> > >>> > >>>Conclusions: > >>> > >>>The only protocol which meets the .05% accuracy requirement for ntp > >>>tracking under the loads > >>>above is the SYNC protocol. The worst case accuracies for > >>> > >>> > >>SYNC, MIXED, > >> > >> > >>>and ASYNC > >>>are .022%, .12%, and .14%, respectively. > >>> > >>>We could reduce the cost of the SYNC method by only > >>> > >>> > >>scheduling the extra > >> > >> > >>>wakeups if a certain number > >>>of ticks are missed. > >>> > >>>Regards, > >>>Dave > >>> > >>>Keir Fraser wrote: > >>> > >>> > >>> > >>>>On 9/11/07 19:22, "Dave Winchell" > >>>> > >>>> > >> wrote: > >> > >> > >>>> > >>>> > >>>> > >>>>>Since I had a high error (~.03%) for the ASYNC method a > >>>>> > >>>>> > >>couple of days ago, > >> > >> > >>>>>I ran another ASYNC test. I think there may have been something > >>>>>wrong with the code I used a couple of days ago for > >>>>> > >>>>> > >>ASYNC. It may have been > >> > >> > >>>>>missing the immediate delivery of interrupt after context > >>>>> > >>>>> > >>switch in. > >> > >> > >>>>>My results indicate that either SYNC or ASYNC give > >>>>> > >>>>> > >>acceptable accuracy, > >> > >> > >>>>>each running consistently around or under .01%. MIXED has > >>>>> > >>>>> > >>a fairly high > >> > >> > >>>>>error of > >>>>>greater than .03%. Probably too close to .05% ntp > >>>>> > >>>>> > >>threshold for comfort. > >> > >> > >>>>>I don't have an overnight run with SYNC. I plan to leave > >>>>> > >>>>> > >>SYNC running > >> > >> > >>>>>over the weekend. If you'd rather I can leave MIXED > >>>>> > >>>>> > >>running instead. > >> > >> > >>>>>It may be too early to pick the protocol and I can run > >>>>> > >>>>> > >>more overnight tests > >> > >> > >>>>>next week. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>I'm a bit worried about any unwanted side effects of the > >>>> > >>>> > >>SYNC+run_timer > >> > >> > >>>>approach -- e.g., whether timer wakeups will cause higher > >>>> > >>>> > >>system-wide CPU > >> > >> > >>>>contention. I find it easier to think through the > >>>> > >>>> > >>implications of ASYNC. I'm > >> > >> > >>>>surprised that MIXED loses time, and is less accurate than > >>>> > >>>> > >>ASYNC. Perhaps it > >> > >> > >>>>delivers more timer interrupts than the other approaches, > >>>> > >>>> > >>and each interrupt > >> > >> > >>>>event causes a small accumulated error? > >>>> > >>>>Overall I would consider MIXED and ASYNC as favourites and > >>>> > >>>> > >>if the latter is > >> > >> > >>>>actually more accurate then I can simply revert the changeset that > >>>>implemented MIXED. > >>>> > >>>>Perhaps rather than running more of the same workloads you > >>>> > >>>> > >>could try idle > >> > >> > >>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads > >>>> > >>>> > >>to /dev/null)? We > >> > >> > >>>>don't have any data on workloads that aren't CPU bound, so > >>>> > >>>> > >>that's really an > >> > >> > >>>>obvious place to put any further effort imo. > >>>> > >>>>-- Keir > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >> > >>_______________________________________________ > >>Xen-devel mailing list > >>Xen-devel@lists.xensource.com > >>http://lists.xensource.com/xen-devel > >> > >> > >> > >> > > > > > > > = > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 08 Jan 2008 14:33:04 +0000 Message-ID: References: <477EC012.6040206@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <477EC012.6040206@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Applied as c/s 16690, although the checked-in patch is smaller. I think the only important fix is to pt_intr_post() and the only bit of the patch I totally omitted was the change to pt_process_missed_ticks(). I don't think that change can be important, but let's see what happens to the error percentage... -- Keir On 4/1/08 23:24, "Dave Winchell" wrote: > Hi Dan and Keir, > > Attached is a patch that fixes some issues with the SYNC policy > (no_missed_ticks_pending). > I have not tried to make the change the minimal one, but, rather, just > ported into > the new code what I know to work well. The error for > no_missed_ticks_pending goes from > over 3% to .03% with this change according to my testing. > > Regards, > Dave > > Dan Magenheimer wrote: > >> Hi Dave -- >> >> Did you get your correction ported? If so, it would be nice to see this get >> into 3.1.3. >> >> Note that I just did some very limited testing with timer_mode=2(=SYNC=no >> missed ticks pending) >> on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've >> seen so far >> is 0.012%. But I haven't tried any exotic loads, just LTP. >> >> Thanks, >> Dan >> >> >> >>> -----Original Message----- >>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>> Sent: Wednesday, December 19, 2007 12:33 PM >>> To: dan.magenheimer@oracle.com >>> Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>> Eddie; Jiang, Yunhong; Dave Winchell >>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>> disables pending >>> missed ticks >>> >>> >>> Dan, >>> >>> I did some testing with the constant tsc offset SYNC method >>> (now called >>> no_missed_ticks_pending) >>> and found the error to be very high, much larger than 1 %, as >>> I recall. >>> I have not had a chance to submit a correction. I will try to >>> do it later >>> this week or the first week in January. My version of constant tsc >>> offset SYNC method >>> produces .02 % error, so I just need to port that into the >>> current code. >>> >>> The error you got for both of those kernels is what I would expect >>> for the default mode, delay_for_missed_ticks. >>> >>> I'll let Keir answer on how to set the time mode. >>> >>> Regards, >>> Dave >>> >>> Dan Magenheimer wrote: >>> >>> >>> >>>> Anyone make measurements on the final patch? >>>> >>>> I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>>> >>>> >>> about 0.2% with no load. This was xen-unstable tip today >>> with no options specified. 32-bit was about 0.01%. >>> >>> >>>> I think I missed something... how do I run the various >>>> >>>> >>> accounting choices and which ones are known to be appropriate >>> for which kernels? >>> >>> >>>> Thanks, >>>> Dan >>>> >>>> >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: xen-devel-bounces@lists.xensource.com >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>> >>>>> >>> Keir Fraser >>> >>> >>>>> Sent: Thursday, December 06, 2007 4:57 AM >>>>> To: Dave Winchell >>>>> Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, >>>>> Yunhong >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>> disables pending >>>>> missed ticks >>>>> >>>>> >>>>> Please take a look at xen-unstable changeset 16545. >>>>> >>>>> -- Keir >>>>> >>>>> On 26/11/07 20:57, "Dave Winchell" >>>>> >>>>> >>> wrote: >>> >>> >>>>> >>>>> >>>>> >>>>>> Keir, >>>>>> >>>>>> The accuracy data I've collected for i/o loads for the >>>>>> various time protocols follows. In addition, the data >>>>>> for cpu loads is shown. >>>>>> >>>>>> The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>>>> Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>> The cpu load is usex -e36 on each guest. >>>>>> (usex is available at http://people.redhat.com/anderson/usex.) >>>>>> i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>>>>> >>>>>> The loads labeled i/o-32 are 32 instances of dd. >>>>>> Also, these are run on 4 cpu AMD box. >>>>>> In addition, there is an idle rh-32bit guest. >>>>>> All three guests are 8vcpu. >>>>>> >>>>>> The loads labeled i/o-4/32 are the same as i/o-32 >>>>>> except that the redhat-64 guest has 4 instances of dd. >>>>>> >>>>>> Date Duration Protocol sles, rhat error load >>>>>> >>>>>> 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu >>>>>> 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu >>>>>> >>>>>> 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>>>> 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>>>> 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>>>>> >>>>>> 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>>>> 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu >>>>>> >>>>>> >>>>>> 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 >>>>>> 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 >>>>>> >>>>>> 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 >>>>>> 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>>>>> >>>>>> 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 >>>>>> 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 >>>>>> >>>>>> >>>>>> >>>>>> 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>>>> 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 >>>>>> 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>>>>> >>>>>> 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 >>>>>> 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 >>>>>> >>>>>> >>>>>> Overhead measurements: >>>>>> >>>>>> Progress in terms of number of passes through a fixed >>>>>> >>>>>> >>>>>> >>>>>> >>>>> system workload >>>>> >>>>> >>>>> >>>>> >>>>>> on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>> The workload was usex -b48. >>>>>> >>>>>> >>>>>> ASYNC 167 min 145 passes .868 passes/min >>>>>> SYNC 167 min 144 passes .862 passes/min >>>>>> SYNC 1065 min 919 passes .863 passes/min >>>>>> MIXED 221 min 196 passes .887 passes/min >>>>>> >>>>>> >>>>>> Conclusions: >>>>>> >>>>>> The only protocol which meets the .05% accuracy requirement for ntp >>>>>> tracking under the loads >>>>>> above is the SYNC protocol. The worst case accuracies for >>>>>> >>>>>> >>>>>> >>>>>> >>>>> SYNC, MIXED, >>>>> >>>>> >>>>> >>>>> >>>>>> and ASYNC >>>>>> are .022%, .12%, and .14%, respectively. >>>>>> >>>>>> We could reduce the cost of the SYNC method by only >>>>>> >>>>>> >>>>>> >>>>>> >>>>> scheduling the extra >>>>> >>>>> >>>>> >>>>> >>>>>> wakeups if a certain number >>>>>> of ticks are missed. >>>>>> >>>>>> Regards, >>>>>> Dave >>>>>> >>>>>> Keir Fraser wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On 9/11/07 19:22, "Dave Winchell" >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Since I had a high error (~.03%) for the ASYNC method a >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> couple of days ago, >>>>> >>>>> >>>>> >>>>> >>>>>>>> I ran another ASYNC test. I think there may have been something >>>>>>>> wrong with the code I used a couple of days ago for >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> ASYNC. It may have been >>>>> >>>>> >>>>> >>>>> >>>>>>>> missing the immediate delivery of interrupt after context >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> switch in. >>>>> >>>>> >>>>> >>>>> >>>>>>>> My results indicate that either SYNC or ASYNC give >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> acceptable accuracy, >>>>> >>>>> >>>>> >>>>> >>>>>>>> each running consistently around or under .01%. MIXED has >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> a fairly high >>>>> >>>>> >>>>> >>>>> >>>>>>>> error of >>>>>>>> greater than .03%. Probably too close to .05% ntp >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> threshold for comfort. >>>>> >>>>> >>>>> >>>>> >>>>>>>> I don't have an overnight run with SYNC. I plan to leave >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> SYNC running >>>>> >>>>> >>>>> >>>>> >>>>>>>> over the weekend. If you'd rather I can leave MIXED >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> running instead. >>>>> >>>>> >>>>> >>>>> >>>>>>>> It may be too early to pick the protocol and I can run >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> more overnight tests >>>>> >>>>> >>>>> >>>>> >>>>>>>> next week. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> I'm a bit worried about any unwanted side effects of the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> SYNC+run_timer >>>>> >>>>> >>>>> >>>>> >>>>>>> approach -- e.g., whether timer wakeups will cause higher >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> system-wide CPU >>>>> >>>>> >>>>> >>>>> >>>>>>> contention. I find it easier to think through the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> implications of ASYNC. I'm >>>>> >>>>> >>>>> >>>>> >>>>>>> surprised that MIXED loses time, and is less accurate than >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> ASYNC. Perhaps it >>>>> >>>>> >>>>> >>>>> >>>>>>> delivers more timer interrupts than the other approaches, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> and each interrupt >>>>> >>>>> >>>>> >>>>> >>>>>>> event causes a small accumulated error? >>>>>>> >>>>>>> Overall I would consider MIXED and ASYNC as favourites and >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> if the latter is >>>>> >>>>> >>>>> >>>>> >>>>>>> actually more accurate then I can simply revert the changeset that >>>>>>> implemented MIXED. >>>>>>> >>>>>>> Perhaps rather than running more of the same workloads you >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> could try idle >>>>> >>>>> >>>>> >>>>> >>>>>>> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> to /dev/null)? We >>>>> >>>>> >>>>> >>>>> >>>>>>> don't have any data on workloads that aren't CPU bound, so >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> that's really an >>>>> >>>>> >>>>> >>>>> >>>>>>> obvious place to put any further effort imo. >>>>>>> >>>>>>> -- Keir >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> _______________________________________________ >>>>> Xen-devel mailing list >>>>> Xen-devel@lists.xensource.com >>>>> http://lists.xensource.com/xen-devel >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >> > > diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > --- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 > +++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 > @@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru > > missed_ticks = missed_ticks / (s_time_t) pt->period + 1; > if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) > - pt->do_not_freeze = !pt->pending_intr_nr; > + pt->do_not_freeze = 1; > else > pt->pending_intr_nr += missed_ticks; > pt->scheduled += missed_ticks * pt->period; > @@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > > pt_lock(pt); > > - pt->pending_intr_nr++; > + if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { > + pt->pending_intr_nr = 1; > + pt->do_not_freeze = 0; > + } > + else > + pt->pending_intr_nr++; > > if ( !pt->one_shot ) > { > @@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct > return; > } > > - pt->do_not_freeze = 0; > - > if ( pt->one_shot ) > { > pt->enabled = 0; > @@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct > pt->last_plt_gtime = hvm_get_guest_time(v); > pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */ > } > + else if ( mode_is(v->domain, no_missed_ticks_pending) ) { > + pt->pending_intr_nr--; > + pt->last_plt_gtime = hvm_get_guest_time(v); > + } > else > { > pt->last_plt_gtime += pt->period_cycles; From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 9 Jan 2008 10:19:00 -0700 Message-ID: <20080109101900265.00000003216@djm-pc> References: <4784FBF3.3040503@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <4784FBF3.3040503@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , Keir Fraser Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Thanks very much Dave and Keir! Keir, if its not too late already, can this patch be added for 3.1.3? Which reminds me, when will 3.1.3 go final? Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Wednesday, January 09, 2008 9:53 AM > To: Keir Fraser > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave > Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Hi Keir, > = > The latest change, c/s 16690, looks fine. > I agree that the code in c/s 16690 is equivalent to > the code I submitted. Also, your version is more > concise. > = > The error tests confirm the equivalence. With overnight cpu loads, > the checked in version was accurate to +.048% for sles > and +.038% for red hat. My version was +.046% and +.032% in a = > 2 hour test. > I don't think the difference is significant. > = > i/o loads produced errors of +.01%. > = > Thanks for all your efforts on this issue. > = > Regards, > Dave > = > = > = > Keir Fraser wrote: > = > >Applied as c/s 16690, although the checked-in patch is = > smaller. I think the > >only important fix is to pt_intr_post() and the only bit of = > the patch I > >totally omitted was the change to pt_process_missed_ticks(). = > I don't think > >that change can be important, but let's see what happens to the error > >percentage... > > > > -- Keir > > > >On 4/1/08 23:24, "Dave Winchell" wrote: > > > > > > > >>Hi Dan and Keir, > >> > >>Attached is a patch that fixes some issues with the SYNC policy > >>(no_missed_ticks_pending). > >>I have not tried to make the change the minimal one, but, = > rather, just > >>ported into > >>the new code what I know to work well. The error for > >>no_missed_ticks_pending goes from > >>over 3% to .03% with this change according to my testing. > >> > >>Regards, > >>Dave > >> > >>Dan Magenheimer wrote: > >> > >> > >> > >>>Hi Dave -- > >>> > >>>Did you get your correction ported? If so, it would be = > nice to see this get > >>>into 3.1.3. > >>> > >>>Note that I just did some very limited testing with = > timer_mode=3D2(=3DSYNC=3Dno > >>>missed ticks pending) > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the = > worst error I've > >>>seen so far > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. > >>> > >>>Thanks, > >>>Dan > >>> > >>> > >>> > >>> > >>> > >>>>-----Original Message----- > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>To: dan.magenheimer@oracle.com > >>>>Cc: Keir Fraser; Shan, Haitao; = > xen-devel@lists.xensource.com; Dong, > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>disables pending > >>>>missed ticks > >>>> > >>>> > >>>>Dan, > >>>> > >>>>I did some testing with the constant tsc offset SYNC method > >>>>(now called > >>>>no_missed_ticks_pending) > >>>>and found the error to be very high, much larger than 1 %, as > >>>>I recall. > >>>>I have not had a chance to submit a correction. I will try to > >>>>do it later > >>>>this week or the first week in January. My version of constant tsc > >>>>offset SYNC method > >>>>produces .02 % error, so I just need to port that into the > >>>>current code. > >>>> > >>>>The error you got for both of those kernels is what I would expect > >>>>for the default mode, delay_for_missed_ticks. > >>>> > >>>>I'll let Keir answer on how to set the time mode. > >>>> > >>>>Regards, > >>>>Dave > >>>> > >>>>Dan Magenheimer wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>>Anyone make measurements on the final patch? > >>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>about 0.2% with no load. This was xen-unstable tip today > >>>>with no options specified. 32-bit was about 0.01%. > >>>> > >>>> > >>>> > >>>> > >>>>>I think I missed something... how do I run the various > >>>>> > >>>>> > >>>>> > >>>>> > >>>>accounting choices and which ones are known to be appropriate > >>>>for which kernels? > >>>> > >>>> > >>>> > >>>> > >>>>>Thanks, > >>>>>Dan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>-----Original Message----- > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>Keir Fraser > >>>> > >>>> > >>>> > >>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>>>To: Dave Winchell > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, = > Eddie; Jiang, > >>>>>>Yunhong > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>>>disables pending > >>>>>>missed ticks > >>>>>> > >>>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. > >>>>>> > >>>>>>-- Keir > >>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>> wrote: > >>>> > >>>> > >>>> > >>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>Keir, > >>>>>>> > >>>>>>>The accuracy data I've collected for i/o loads for the > >>>>>>>various time protocols follows. In addition, the data > >>>>>>>for cpu loads is shown. > >>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.) > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 of=3D/dev/null. > >>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>>>>All three guests are 8vcpu. > >>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > >>>>>>>except that the redhat-64 guest has 4 instances of dd. > >>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, = > +.005% cpu > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, = > +.012% cpu > >>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu > >>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, = > -.031% cpu > >>>>>>> > >>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, = > -.09% i/o-8 > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% = > -.14% i/o-8 > >>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, = > -.022% i/o-8 > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 > >>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, = > -.029% i/o-8 > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, = > -.031% i/o-8 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, = > -.005% i/o-32 > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 > >>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, = > .003% i/o-4/32 > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, = > .01% i/o-4/32 > >>>>>>> > >>>>>>> > >>>>>>>Overhead measurements: > >>>>>>> > >>>>>>>Progress in terms of number of passes through a fixed > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>system workload > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>>>>>The workload was usex -b48. > >>>>>>> > >>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>>>> > >>>>>>> > >>>>>>>Conclusions: > >>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy = > requirement for ntp > >>>>>>>tracking under the loads > >>>>>>>above is the SYNC protocol. The worst case accuracies for > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>SYNC, MIXED, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>and ASYNC > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>>>> > >>>>>>>We could reduce the cost of the SYNC method by only > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>scheduling the extra > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>wakeups if a certain number > >>>>>>>of ticks are missed. > >>>>>>> > >>>>>>>Regards, > >>>>>>>Dave > >>>>>>> > >>>>>>>Keir Fraser wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>couple of days ago, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have = > been something > >>>>>>>>>wrong with the code I used a couple of days ago for > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>ASYNC. It may have been > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>missing the immediate delivery of interrupt after context > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>switch in. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>acceptable accuracy, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>each running consistently around or under .01%. MIXED has > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>a fairly high > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>error of > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>threshold for comfort. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>SYNC running > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>running instead. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>It may be too early to pick the protocol and I can run > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>more overnight tests > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>next week. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side effects of the > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>SYNC+run_timer > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will cause higher > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>system-wide CPU > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>contention. I find it easier to think through the > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>implications of ASYNC. I'm > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>surprised that MIXED loses time, and is less accurate than > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>ASYNC. Perhaps it > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>delivers more timer interrupts than the other approaches, > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>and each interrupt > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>event causes a small accumulated error? > >>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>if the latter is > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>actually more accurate then I can simply revert the = > changeset that > >>>>>>>>implemented MIXED. > >>>>>>>> > >>>>>>>>Perhaps rather than running more of the same workloads you > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>could try idle > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>to /dev/null)? We > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>don't have any data on workloads that aren't CPU bound, so > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>that's really an > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>>>>> > >>>>>>>>-- Keir > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>_______________________________________________ > >>>>>>Xen-devel mailing list > >>>>>>Xen-devel@lists.xensource.com > >>>>>>http://lists.xensource.com/xen-devel > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> > >>> > >>> > >>> > >>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru > >> > >> missed_ticks =3D missed_ticks / (s_time_t) pt->period + 1; > >> if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>+ pt->do_not_freeze =3D 1; > >> else > >> pt->pending_intr_nr +=3D missed_ticks; > >> pt->scheduled +=3D missed_ticks * pt->period; > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > >> > >> pt_lock(pt); > >> > >>- pt->pending_intr_nr++; > >>+ if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { > >>+ pt->pending_intr_nr =3D 1; > >>+ pt->do_not_freeze =3D 0; > >>+ } > >>+ else > >>+ pt->pending_intr_nr++; > >> > >> if ( !pt->one_shot ) > >> { > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct > >> return; > >> } > >> > >>- pt->do_not_freeze =3D 0; > >>- > >> if ( pt->one_shot ) > >> { > >> pt->enabled =3D 0; > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct > >> pt->last_plt_gtime =3D hvm_get_guest_time(v); > >> pt->pending_intr_nr =3D 0; /* 'collapse' all = > missed ticks */ > >> } > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { > >>+ pt->pending_intr_nr--; > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>+ } > >> else > >> { > >> pt->last_plt_gtime +=3D pt->period_cycles; > >> > >> > > > > > > > > > = > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 09 Jan 2008 19:14:06 +0000 Message-ID: References: <20080109101900265.00000003216@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080109101900265.00000003216@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" , Dave Winchell Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Yes, it'll be added. 3.1.3 won't be for a couple of weeks. I want to get 3.2.0 squared away first. -- Keir On 9/1/08 17:19, "Dan Magenheimer" wrote: > Thanks very much Dave and Keir! > > Keir, if its not too late already, can this patch be added for 3.1.3? > > Which reminds me, when will 3.1.3 go final? > > Thanks, > Dan > >> -----Original Message----- >> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >> Sent: Wednesday, January 09, 2008 9:53 AM >> To: Keir Fraser >> Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave >> Winchell >> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >> disables pending >> missed ticks >> >> >> Hi Keir, >> >> The latest change, c/s 16690, looks fine. >> I agree that the code in c/s 16690 is equivalent to >> the code I submitted. Also, your version is more >> concise. >> >> The error tests confirm the equivalence. With overnight cpu loads, >> the checked in version was accurate to +.048% for sles >> and +.038% for red hat. My version was +.046% and +.032% in a >> 2 hour test. >> I don't think the difference is significant. >> >> i/o loads produced errors of +.01%. >> >> Thanks for all your efforts on this issue. >> >> Regards, >> Dave >> >> >> >> Keir Fraser wrote: >> >>> Applied as c/s 16690, although the checked-in patch is >> smaller. I think the >>> only important fix is to pt_intr_post() and the only bit of >> the patch I >>> totally omitted was the change to pt_process_missed_ticks(). >> I don't think >>> that change can be important, but let's see what happens to the error >>> percentage... >>> >>> -- Keir >>> >>> On 4/1/08 23:24, "Dave Winchell" wrote: >>> >>> >>> >>>> Hi Dan and Keir, >>>> >>>> Attached is a patch that fixes some issues with the SYNC policy >>>> (no_missed_ticks_pending). >>>> I have not tried to make the change the minimal one, but, >> rather, just >>>> ported into >>>> the new code what I know to work well. The error for >>>> no_missed_ticks_pending goes from >>>> over 3% to .03% with this change according to my testing. >>>> >>>> Regards, >>>> Dave >>>> >>>> Dan Magenheimer wrote: >>>> >>>> >>>> >>>>> Hi Dave -- >>>>> >>>>> Did you get your correction ported? If so, it would be >> nice to see this get >>>>> into 3.1.3. >>>>> >>>>> Note that I just did some very limited testing with >> timer_mode=2(=SYNC=no >>>>> missed ticks pending) >>>>> on tip of xen-3.1-testing (64-bit Linux hv guest) and the >> worst error I've >>>>> seen so far >>>>> is 0.012%. But I haven't tried any exotic loads, just LTP. >>>>> >>>>> Thanks, >>>>> Dan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>> Sent: Wednesday, December 19, 2007 12:33 PM >>>>>> To: dan.magenheimer@oracle.com >>>>>> Cc: Keir Fraser; Shan, Haitao; >> xen-devel@lists.xensource.com; Dong, >>>>>> Eddie; Jiang, Yunhong; Dave Winchell >>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>> disables pending >>>>>> missed ticks >>>>>> >>>>>> >>>>>> Dan, >>>>>> >>>>>> I did some testing with the constant tsc offset SYNC method >>>>>> (now called >>>>>> no_missed_ticks_pending) >>>>>> and found the error to be very high, much larger than 1 %, as >>>>>> I recall. >>>>>> I have not had a chance to submit a correction. I will try to >>>>>> do it later >>>>>> this week or the first week in January. My version of constant tsc >>>>>> offset SYNC method >>>>>> produces .02 % error, so I just need to port that into the >>>>>> current code. >>>>>> >>>>>> The error you got for both of those kernels is what I would expect >>>>>> for the default mode, delay_for_missed_ticks. >>>>>> >>>>>> I'll let Keir answer on how to set the time mode. >>>>>> >>>>>> Regards, >>>>>> Dave >>>>>> >>>>>> Dan Magenheimer wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Anyone make measurements on the final patch? >>>>>>> >>>>>>> I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> about 0.2% with no load. This was xen-unstable tip today >>>>>> with no options specified. 32-bit was about 0.01%. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> I think I missed something... how do I run the various >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> accounting choices and which ones are known to be appropriate >>>>>> for which kernels? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> Dan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: xen-devel-bounces@lists.xensource.com >>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> Keir Fraser >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>> Sent: Thursday, December 06, 2007 4:57 AM >>>>>>>> To: Dave Winchell >>>>>>>> Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, >> Eddie; Jiang, >>>>>>>> Yunhong >>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>> disables pending >>>>>>>> missed ticks >>>>>>>> >>>>>>>> >>>>>>>> Please take a look at xen-unstable changeset 16545. >>>>>>>> >>>>>>>> -- Keir >>>>>>>> >>>>>>>> On 26/11/07 20:57, "Dave Winchell" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Keir, >>>>>>>>> >>>>>>>>> The accuracy data I've collected for i/o loads for the >>>>>>>>> various time protocols follows. In addition, the data >>>>>>>>> for cpu loads is shown. >>>>>>>>> >>>>>>>>> The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>>>>>>> Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>>>>> The cpu load is usex -e36 on each guest. >>>>>>>>> (usex is available at http://people.redhat.com/anderson/usex.) >>>>>>>>> i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>>>>>>>> >>>>>>>>> The loads labeled i/o-32 are 32 instances of dd. >>>>>>>>> Also, these are run on 4 cpu AMD box. >>>>>>>>> In addition, there is an idle rh-32bit guest. >>>>>>>>> All three guests are 8vcpu. >>>>>>>>> >>>>>>>>> The loads labeled i/o-4/32 are the same as i/o-32 >>>>>>>>> except that the redhat-64 guest has 4 instances of dd. >>>>>>>>> >>>>>>>>> Date Duration Protocol sles, rhat error load >>>>>>>>> >>>>>>>>> 11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, >> +.005% cpu >>>>>>>>> 11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, >> +.012% cpu >>>>>>>>> >>>>>>>>> 11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>>>>>>> 11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>>>>>>> 11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>>>>>>>> >>>>>>>>> 11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>>>>>>> 11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, >> -.031% cpu >>>>>>>>> >>>>>>>>> >>>>>>>>> 11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, >> -.09% i/o-8 >>>>>>>>> 11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% >> -.14% i/o-8 >>>>>>>>> >>>>>>>>> 11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, >> -.022% i/o-8 >>>>>>>>> 11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>>>>>>>> >>>>>>>>> 11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, >> -.029% i/o-8 >>>>>>>>> 11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, >> -.031% i/o-8 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>>>>>>> 11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, >> -.005% i/o-32 >>>>>>>>> 11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>>>>>>>> >>>>>>>>> 11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, >> .003% i/o-4/32 >>>>>>>>> 11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, >> .01% i/o-4/32 >>>>>>>>> >>>>>>>>> >>>>>>>>> Overhead measurements: >>>>>>>>> >>>>>>>>> Progress in terms of number of passes through a fixed >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> system workload >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>>> The workload was usex -b48. >>>>>>>>> >>>>>>>>> >>>>>>>>> ASYNC 167 min 145 passes .868 passes/min >>>>>>>>> SYNC 167 min 144 passes .862 passes/min >>>>>>>>> SYNC 1065 min 919 passes .863 passes/min >>>>>>>>> MIXED 221 min 196 passes .887 passes/min >>>>>>>>> >>>>>>>>> >>>>>>>>> Conclusions: >>>>>>>>> >>>>>>>>> The only protocol which meets the .05% accuracy >> requirement for ntp >>>>>>>>> tracking under the loads >>>>>>>>> above is the SYNC protocol. The worst case accuracies for >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> SYNC, MIXED, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> and ASYNC >>>>>>>>> are .022%, .12%, and .14%, respectively. >>>>>>>>> >>>>>>>>> We could reduce the cost of the SYNC method by only >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> scheduling the extra >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> wakeups if a certain number >>>>>>>>> of ticks are missed. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Dave >>>>>>>>> >>>>>>>>> Keir Fraser wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 9/11/07 19:22, "Dave Winchell" >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Since I had a high error (~.03%) for the ASYNC method a >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> couple of days ago, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> I ran another ASYNC test. I think there may have >> been something >>>>>>>>>>> wrong with the code I used a couple of days ago for >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> ASYNC. It may have been >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> missing the immediate delivery of interrupt after context >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> switch in. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> My results indicate that either SYNC or ASYNC give >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> acceptable accuracy, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> each running consistently around or under .01%. MIXED has >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> a fairly high >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> error of >>>>>>>>>>> greater than .03%. Probably too close to .05% ntp >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> threshold for comfort. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> I don't have an overnight run with SYNC. I plan to leave >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> SYNC running >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> over the weekend. If you'd rather I can leave MIXED >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> running instead. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> It may be too early to pick the protocol and I can run >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>> more overnight tests >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> next week. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> I'm a bit worried about any unwanted side effects of the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> SYNC+run_timer >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> approach -- e.g., whether timer wakeups will cause higher >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> system-wide CPU >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> contention. I find it easier to think through the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> implications of ASYNC. I'm >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> surprised that MIXED loses time, and is less accurate than >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> ASYNC. Perhaps it >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> delivers more timer interrupts than the other approaches, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> and each interrupt >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> event causes a small accumulated error? >>>>>>>>>> >>>>>>>>>> Overall I would consider MIXED and ASYNC as favourites and >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> if the latter is >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> actually more accurate then I can simply revert the >> changeset that >>>>>>>>>> implemented MIXED. >>>>>>>>>> >>>>>>>>>> Perhaps rather than running more of the same workloads you >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> could try idle >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> to /dev/null)? We >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> don't have any data on workloads that aren't CPU bound, so >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> that's really an >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>> obvious place to put any further effort imo. >>>>>>>>>> >>>>>>>>>> -- Keir >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Xen-devel mailing list >>>>>>>> Xen-devel@lists.xensource.com >>>>>>>> http://lists.xensource.com/xen-devel >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>> --- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 >>>> +++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 >>>> @@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru >>>> >>>> missed_ticks = missed_ticks / (s_time_t) pt->period + 1; >>>> if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) >>>> - pt->do_not_freeze = !pt->pending_intr_nr; >>>> + pt->do_not_freeze = 1; >>>> else >>>> pt->pending_intr_nr += missed_ticks; >>>> pt->scheduled += missed_ticks * pt->period; >>>> @@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >>>> >>>> pt_lock(pt); >>>> >>>> - pt->pending_intr_nr++; >>>> + if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { >>>> + pt->pending_intr_nr = 1; >>>> + pt->do_not_freeze = 0; >>>> + } >>>> + else >>>> + pt->pending_intr_nr++; >>>> >>>> if ( !pt->one_shot ) >>>> { >>>> @@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct >>>> return; >>>> } >>>> >>>> - pt->do_not_freeze = 0; >>>> - >>>> if ( pt->one_shot ) >>>> { >>>> pt->enabled = 0; >>>> @@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct >>>> pt->last_plt_gtime = hvm_get_guest_time(v); >>>> pt->pending_intr_nr = 0; /* 'collapse' all >> missed ticks */ >>>> } >>>> + else if ( mode_is(v->domain, no_missed_ticks_pending) ) { >>>> + pt->pending_intr_nr--; >>>> + pt->last_plt_gtime = hvm_get_guest_time(v); >>>> + } >>>> else >>>> { >>>> pt->last_plt_gtime += pt->period_cycles; >>>> >>>> >>> >>> >>> >>> >> >> > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 19 Dec 2007 14:32:46 -0500 Message-ID: <476971DE.4010102@virtualiron.com> References: <20071219115700531.00000002296@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20071219115700531.00000002296@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , "Shan, Haitao" , "Jiang, Yunhong" , "Dong, Eddie" , Dave Winchell List-Id: xen-devel@lists.xenproject.org Dan, I did some testing with the constant tsc offset SYNC method (now called no_missed_ticks_pending) and found the error to be very high, much larger than 1 %, as I recall. I have not had a chance to submit a correction. I will try to do it later this week or the first week in January. My version of constant tsc offset SYNC method produces .02 % error, so I just need to port that into the current code. The error you got for both of those kernels is what I would expect for the default mode, delay_for_missed_ticks. I'll let Keir answer on how to set the time mode. Regards, Dave Dan Magenheimer wrote: >Anyone make measurements on the final patch? > >I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no load. This was xen-unstable tip today with no options specified. 32-bit was about 0.01%. > >I think I missed something... how do I run the various accounting choices and which ones are known to be appropriate for which kernels? > >Thanks, >Dan > > > >>-----Original Message----- >>From: xen-devel-bounces@lists.xensource.com >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser >>Sent: Thursday, December 06, 2007 4:57 AM >>To: Dave Winchell >>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, >>Yunhong >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Please take a look at xen-unstable changeset 16545. >> >> -- Keir >> >>On 26/11/07 20:57, "Dave Winchell" wrote: >> >> >> >>>Keir, >>> >>>The accuracy data I've collected for i/o loads for the >>>various time protocols follows. In addition, the data >>>for cpu loads is shown. >>> >>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>The cpu load is usex -e36 on each guest. >>>(usex is available at http://people.redhat.com/anderson/usex.) >>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>> >>>The loads labeled i/o-32 are 32 instances of dd. >>>Also, these are run on 4 cpu AMD box. >>>In addition, there is an idle rh-32bit guest. >>>All three guests are 8vcpu. >>> >>>The loads labeled i/o-4/32 are the same as i/o-32 >>>except that the redhat-64 guest has 4 instances of dd. >>> >>>Date Duration Protocol sles, rhat error load >>> >>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu >>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu >>> >>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>> >>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu >>> >>> >>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 >>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 >>> >>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 >>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>> >>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 >>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 >>> >>> >>> >>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 >>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>> >>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 >>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 >>> >>> >>>Overhead measurements: >>> >>>Progress in terms of number of passes through a fixed >>> >>> >>system workload >> >> >>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>The workload was usex -b48. >>> >>> >>>ASYNC 167 min 145 passes .868 passes/min >>>SYNC 167 min 144 passes .862 passes/min >>>SYNC 1065 min 919 passes .863 passes/min >>>MIXED 221 min 196 passes .887 passes/min >>> >>> >>>Conclusions: >>> >>>The only protocol which meets the .05% accuracy requirement for ntp >>>tracking under the loads >>>above is the SYNC protocol. The worst case accuracies for >>> >>> >>SYNC, MIXED, >> >> >>>and ASYNC >>>are .022%, .12%, and .14%, respectively. >>> >>>We could reduce the cost of the SYNC method by only >>> >>> >>scheduling the extra >> >> >>>wakeups if a certain number >>>of ticks are missed. >>> >>>Regards, >>>Dave >>> >>>Keir Fraser wrote: >>> >>> >>> >>>>On 9/11/07 19:22, "Dave Winchell" >>>> >>>> >> wrote: >> >> >>>> >>>> >>>> >>>>>Since I had a high error (~.03%) for the ASYNC method a >>>>> >>>>> >>couple of days ago, >> >> >>>>>I ran another ASYNC test. I think there may have been something >>>>>wrong with the code I used a couple of days ago for >>>>> >>>>> >>ASYNC. It may have been >> >> >>>>>missing the immediate delivery of interrupt after context >>>>> >>>>> >>switch in. >> >> >>>>>My results indicate that either SYNC or ASYNC give >>>>> >>>>> >>acceptable accuracy, >> >> >>>>>each running consistently around or under .01%. MIXED has >>>>> >>>>> >>a fairly high >> >> >>>>>error of >>>>>greater than .03%. Probably too close to .05% ntp >>>>> >>>>> >>threshold for comfort. >> >> >>>>>I don't have an overnight run with SYNC. I plan to leave >>>>> >>>>> >>SYNC running >> >> >>>>>over the weekend. If you'd rather I can leave MIXED >>>>> >>>>> >>running instead. >> >> >>>>>It may be too early to pick the protocol and I can run >>>>> >>>>> >>more overnight tests >> >> >>>>>next week. >>>>> >>>>> >>>>> >>>>> >>>>I'm a bit worried about any unwanted side effects of the >>>> >>>> >>SYNC+run_timer >> >> >>>>approach -- e.g., whether timer wakeups will cause higher >>>> >>>> >>system-wide CPU >> >> >>>>contention. I find it easier to think through the >>>> >>>> >>implications of ASYNC. I'm >> >> >>>>surprised that MIXED loses time, and is less accurate than >>>> >>>> >>ASYNC. Perhaps it >> >> >>>>delivers more timer interrupts than the other approaches, >>>> >>>> >>and each interrupt >> >> >>>>event causes a small accumulated error? >>>> >>>>Overall I would consider MIXED and ASYNC as favourites and >>>> >>>> >>if the latter is >> >> >>>>actually more accurate then I can simply revert the changeset that >>>>implemented MIXED. >>>> >>>>Perhaps rather than running more of the same workloads you >>>> >>>> >>could try idle >> >> >>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>> >>>> >>to /dev/null)? We >> >> >>>>don't have any data on workloads that aren't CPU bound, so >>>> >>>> >>that's really an >> >> >>>>obvious place to put any further effort imo. >>>> >>>>-- Keir >>>> >>>> >>>> >>>> >>>> >>>> >> >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xensource.com >>http://lists.xensource.com/xen-devel >> >> >> >> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 19 Dec 2007 14:40:00 -0500 Message-ID: <47697390.5000801@virtualiron.com> References: <20071219115700531.00000002296@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20071219115700531.00000002296@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , "Shan, Haitao" , "Jiang, Yunhong" , "Dong, Eddie" , Dave Winchell List-Id: xen-devel@lists.xenproject.org > and which ones are known to be appropriate for which kernels? The default, delay_for_missed_ticks is good for 32 bit Linux. no_missed_ticks_pending is the most accurate for 64 bit Linux, but the version of no_missed_ticks_pending checked in has some errors in it, as I mentioned. Last I checked, before the most recent changeset, the version now called one_missed_tick_pending had an error > .05 % for some loads. This is called the MIXED method in the data I've published. Regards, Dave Dan Magenheimer wrote: >Anyone make measurements on the final patch? > >I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of about 0.2% with no load. This was xen-unstable tip today with no options specified. 32-bit was about 0.01%. > >I think I missed something... how do I run the various accounting choices and which ones are known to be appropriate for which kernels? > >Thanks, >Dan > > > >>-----Original Message----- >>From: xen-devel-bounces@lists.xensource.com >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of Keir Fraser >>Sent: Thursday, December 06, 2007 4:57 AM >>To: Dave Winchell >>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, >>Yunhong >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Please take a look at xen-unstable changeset 16545. >> >> -- Keir >> >>On 26/11/07 20:57, "Dave Winchell" wrote: >> >> >> >>>Keir, >>> >>>The accuracy data I've collected for i/o loads for the >>>various time protocols follows. In addition, the data >>>for cpu loads is shown. >>> >>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>The cpu load is usex -e36 on each guest. >>>(usex is available at http://people.redhat.com/anderson/usex.) >>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>> >>>The loads labeled i/o-32 are 32 instances of dd. >>>Also, these are run on 4 cpu AMD box. >>>In addition, there is an idle rh-32bit guest. >>>All three guests are 8vcpu. >>> >>>The loads labeled i/o-4/32 are the same as i/o-32 >>>except that the redhat-64 guest has 4 instances of dd. >>> >>>Date Duration Protocol sles, rhat error load >>> >>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu >>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu >>> >>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>> >>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu >>> >>> >>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 >>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 >>> >>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 >>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>> >>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 >>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 >>> >>> >>> >>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 >>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>> >>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 >>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 >>> >>> >>>Overhead measurements: >>> >>>Progress in terms of number of passes through a fixed >>> >>> >>system workload >> >> >>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>The workload was usex -b48. >>> >>> >>>ASYNC 167 min 145 passes .868 passes/min >>>SYNC 167 min 144 passes .862 passes/min >>>SYNC 1065 min 919 passes .863 passes/min >>>MIXED 221 min 196 passes .887 passes/min >>> >>> >>>Conclusions: >>> >>>The only protocol which meets the .05% accuracy requirement for ntp >>>tracking under the loads >>>above is the SYNC protocol. The worst case accuracies for >>> >>> >>SYNC, MIXED, >> >> >>>and ASYNC >>>are .022%, .12%, and .14%, respectively. >>> >>>We could reduce the cost of the SYNC method by only >>> >>> >>scheduling the extra >> >> >>>wakeups if a certain number >>>of ticks are missed. >>> >>>Regards, >>>Dave >>> >>>Keir Fraser wrote: >>> >>> >>> >>>>On 9/11/07 19:22, "Dave Winchell" >>>> >>>> >> wrote: >> >> >>>> >>>> >>>> >>>>>Since I had a high error (~.03%) for the ASYNC method a >>>>> >>>>> >>couple of days ago, >> >> >>>>>I ran another ASYNC test. I think there may have been something >>>>>wrong with the code I used a couple of days ago for >>>>> >>>>> >>ASYNC. It may have been >> >> >>>>>missing the immediate delivery of interrupt after context >>>>> >>>>> >>switch in. >> >> >>>>>My results indicate that either SYNC or ASYNC give >>>>> >>>>> >>acceptable accuracy, >> >> >>>>>each running consistently around or under .01%. MIXED has >>>>> >>>>> >>a fairly high >> >> >>>>>error of >>>>>greater than .03%. Probably too close to .05% ntp >>>>> >>>>> >>threshold for comfort. >> >> >>>>>I don't have an overnight run with SYNC. I plan to leave >>>>> >>>>> >>SYNC running >> >> >>>>>over the weekend. If you'd rather I can leave MIXED >>>>> >>>>> >>running instead. >> >> >>>>>It may be too early to pick the protocol and I can run >>>>> >>>>> >>more overnight tests >> >> >>>>>next week. >>>>> >>>>> >>>>> >>>>> >>>>I'm a bit worried about any unwanted side effects of the >>>> >>>> >>SYNC+run_timer >> >> >>>>approach -- e.g., whether timer wakeups will cause higher >>>> >>>> >>system-wide CPU >> >> >>>>contention. I find it easier to think through the >>>> >>>> >>implications of ASYNC. I'm >> >> >>>>surprised that MIXED loses time, and is less accurate than >>>> >>>> >>ASYNC. Perhaps it >> >> >>>>delivers more timer interrupts than the other approaches, >>>> >>>> >>and each interrupt >> >> >>>>event causes a small accumulated error? >>>> >>>>Overall I would consider MIXED and ASYNC as favourites and >>>> >>>> >>if the latter is >> >> >>>>actually more accurate then I can simply revert the changeset that >>>>implemented MIXED. >>>> >>>>Perhaps rather than running more of the same workloads you >>>> >>>> >>could try idle >> >> >>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>> >>>> >>to /dev/null)? We >> >> >>>>don't have any data on workloads that aren't CPU bound, so >>>> >>>> >>that's really an >> >> >>>>obvious place to put any further effort imo. >>>> >>>>-- Keir >>>> >>>> >>>> >>>> >>>> >>>> >> >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xensource.com >>http://lists.xensource.com/xen-devel >> >> >> >> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 03 Jan 2008 18:24:19 -0500 Message-ID: <477D6EA3.3090102@virtualiron.com> References: <20080103155752531.00000000500@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080103155752531.00000000500@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Dan, I had to put that port aside with other pressing issues here. I will try to get the port done in the next couple of days. Is that soon enough for 3.1.3? To see the error with timer_mode=2(=SYNC=no missed ticks pending) you probably need to over commit the physical processors with virtual processors, say 2:1, and then load up all the virtual processors. Regards, Dave Dan Magenheimer wrote: >Hi Dave -- > >Did you get your correction ported? If so, it would be nice to see this get into 3.1.3. > >Note that I just did some very limited testing with timer_mode=2(=SYNC=no missed ticks pending) >on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've seen so far >is 0.012%. But I haven't tried any exotic loads, just LTP. > >Thanks, >Dan > > > >>-----Original Message----- >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>Sent: Wednesday, December 19, 2007 12:33 PM >>To: dan.magenheimer@oracle.com >>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>Eddie; Jiang, Yunhong; Dave Winchell >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Dan, >> >>I did some testing with the constant tsc offset SYNC method >>(now called >>no_missed_ticks_pending) >>and found the error to be very high, much larger than 1 %, as >>I recall. >>I have not had a chance to submit a correction. I will try to >>do it later >>this week or the first week in January. My version of constant tsc >>offset SYNC method >>produces .02 % error, so I just need to port that into the >>current code. >> >>The error you got for both of those kernels is what I would expect >>for the default mode, delay_for_missed_ticks. >> >>I'll let Keir answer on how to set the time mode. >> >>Regards, >>Dave >> >>Dan Magenheimer wrote: >> >> >> >>>Anyone make measurements on the final patch? >>> >>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>> >>> >>about 0.2% with no load. This was xen-unstable tip today >>with no options specified. 32-bit was about 0.01%. >> >> >>>I think I missed something... how do I run the various >>> >>> >>accounting choices and which ones are known to be appropriate >>for which kernels? >> >> >>>Thanks, >>>Dan >>> >>> >>> >>> >>> >>>>-----Original Message----- >>>>From: xen-devel-bounces@lists.xensource.com >>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>> >>>> >>Keir Fraser >> >> >>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>To: Dave Winchell >>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, >>>>Yunhong >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>disables pending >>>>missed ticks >>>> >>>> >>>>Please take a look at xen-unstable changeset 16545. >>>> >>>>-- Keir >>>> >>>>On 26/11/07 20:57, "Dave Winchell" >>>> >>>> >> wrote: >> >> >>>> >>>> >>>> >>>>>Keir, >>>>> >>>>>The accuracy data I've collected for i/o loads for the >>>>>various time protocols follows. In addition, the data >>>>>for cpu loads is shown. >>>>> >>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>The cpu load is usex -e36 on each guest. >>>>>(usex is available at http://people.redhat.com/anderson/usex.) >>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>>>> >>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>Also, these are run on 4 cpu AMD box. >>>>>In addition, there is an idle rh-32bit guest. >>>>>All three guests are 8vcpu. >>>>> >>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>except that the redhat-64 guest has 4 instances of dd. >>>>> >>>>>Date Duration Protocol sles, rhat error load >>>>> >>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu >>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu >>>>> >>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>>>> >>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu >>>>> >>>>> >>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 >>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 >>>>> >>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 >>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>>>> >>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 >>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 >>>>> >>>>> >>>>> >>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 >>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>>>> >>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 >>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 >>>>> >>>>> >>>>>Overhead measurements: >>>>> >>>>>Progress in terms of number of passes through a fixed >>>>> >>>>> >>>>> >>>>> >>>>system workload >>>> >>>> >>>> >>>> >>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>The workload was usex -b48. >>>>> >>>>> >>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>SYNC 167 min 144 passes .862 passes/min >>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>MIXED 221 min 196 passes .887 passes/min >>>>> >>>>> >>>>>Conclusions: >>>>> >>>>>The only protocol which meets the .05% accuracy requirement for ntp >>>>>tracking under the loads >>>>>above is the SYNC protocol. The worst case accuracies for >>>>> >>>>> >>>>> >>>>> >>>>SYNC, MIXED, >>>> >>>> >>>> >>>> >>>>>and ASYNC >>>>>are .022%, .12%, and .14%, respectively. >>>>> >>>>>We could reduce the cost of the SYNC method by only >>>>> >>>>> >>>>> >>>>> >>>>scheduling the extra >>>> >>>> >>>> >>>> >>>>>wakeups if a certain number >>>>>of ticks are missed. >>>>> >>>>>Regards, >>>>>Dave >>>>> >>>>>Keir Fraser wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>> >>>>>> >>>>>> >>>>>> >>>> wrote: >>>> >>>> >>>> >>>> >>>>>> >>>>>> >>>>>> >>>>>>>Since I had a high error (~.03%) for the ASYNC method a >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>couple of days ago, >>>> >>>> >>>> >>>> >>>>>>>I ran another ASYNC test. I think there may have been something >>>>>>>wrong with the code I used a couple of days ago for >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>ASYNC. It may have been >>>> >>>> >>>> >>>> >>>>>>>missing the immediate delivery of interrupt after context >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>switch in. >>>> >>>> >>>> >>>> >>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>acceptable accuracy, >>>> >>>> >>>> >>>> >>>>>>>each running consistently around or under .01%. MIXED has >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>a fairly high >>>> >>>> >>>> >>>> >>>>>>>error of >>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>threshold for comfort. >>>> >>>> >>>> >>>> >>>>>>>I don't have an overnight run with SYNC. I plan to leave >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>SYNC running >>>> >>>> >>>> >>>> >>>>>>>over the weekend. If you'd rather I can leave MIXED >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>running instead. >>>> >>>> >>>> >>>> >>>>>>>It may be too early to pick the protocol and I can run >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>more overnight tests >>>> >>>> >>>> >>>> >>>>>>>next week. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>I'm a bit worried about any unwanted side effects of the >>>>>> >>>>>> >>>>>> >>>>>> >>>>SYNC+run_timer >>>> >>>> >>>> >>>> >>>>>>approach -- e.g., whether timer wakeups will cause higher >>>>>> >>>>>> >>>>>> >>>>>> >>>>system-wide CPU >>>> >>>> >>>> >>>> >>>>>>contention. I find it easier to think through the >>>>>> >>>>>> >>>>>> >>>>>> >>>>implications of ASYNC. I'm >>>> >>>> >>>> >>>> >>>>>>surprised that MIXED loses time, and is less accurate than >>>>>> >>>>>> >>>>>> >>>>>> >>>>ASYNC. Perhaps it >>>> >>>> >>>> >>>> >>>>>>delivers more timer interrupts than the other approaches, >>>>>> >>>>>> >>>>>> >>>>>> >>>>and each interrupt >>>> >>>> >>>> >>>> >>>>>>event causes a small accumulated error? >>>>>> >>>>>>Overall I would consider MIXED and ASYNC as favourites and >>>>>> >>>>>> >>>>>> >>>>>> >>>>if the latter is >>>> >>>> >>>> >>>> >>>>>>actually more accurate then I can simply revert the changeset that >>>>>>implemented MIXED. >>>>>> >>>>>>Perhaps rather than running more of the same workloads you >>>>>> >>>>>> >>>>>> >>>>>> >>>>could try idle >>>> >>>> >>>> >>>> >>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>>>> >>>>>> >>>>>> >>>>>> >>>>to /dev/null)? We >>>> >>>> >>>> >>>> >>>>>>don't have any data on workloads that aren't CPU bound, so >>>>>> >>>>>> >>>>>> >>>>>> >>>>that's really an >>>> >>>> >>>> >>>> >>>>>>obvious place to put any further effort imo. >>>>>> >>>>>>-- Keir >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>_______________________________________________ >>>>Xen-devel mailing list >>>>Xen-devel@lists.xensource.com >>>>http://lists.xensource.com/xen-devel >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 04 Jan 2008 18:24:02 -0500 Message-ID: <477EC012.6040206@virtualiron.com> References: <20080103155752531.00000000500@djm-pc> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------000708050109000100010403" Return-path: In-Reply-To: <20080103155752531.00000000500@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" , Keir Fraser Cc: Dave Winchell , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------000708050109000100010403 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Dan and Keir, Attached is a patch that fixes some issues with the SYNC policy (no_missed_ticks_pending). I have not tried to make the change the minimal one, but, rather, just ported into the new code what I know to work well. The error for no_missed_ticks_pending goes from over 3% to .03% with this change according to my testing. Regards, Dave Dan Magenheimer wrote: >Hi Dave -- > >Did you get your correction ported? If so, it would be nice to see this get into 3.1.3. > >Note that I just did some very limited testing with timer_mode=2(=SYNC=no missed ticks pending) >on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've seen so far >is 0.012%. But I haven't tried any exotic loads, just LTP. > >Thanks, >Dan > > > >>-----Original Message----- >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>Sent: Wednesday, December 19, 2007 12:33 PM >>To: dan.magenheimer@oracle.com >>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>Eddie; Jiang, Yunhong; Dave Winchell >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Dan, >> >>I did some testing with the constant tsc offset SYNC method >>(now called >>no_missed_ticks_pending) >>and found the error to be very high, much larger than 1 %, as >>I recall. >>I have not had a chance to submit a correction. I will try to >>do it later >>this week or the first week in January. My version of constant tsc >>offset SYNC method >>produces .02 % error, so I just need to port that into the >>current code. >> >>The error you got for both of those kernels is what I would expect >>for the default mode, delay_for_missed_ticks. >> >>I'll let Keir answer on how to set the time mode. >> >>Regards, >>Dave >> >>Dan Magenheimer wrote: >> >> >> >>>Anyone make measurements on the final patch? >>> >>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>> >>> >>about 0.2% with no load. This was xen-unstable tip today >>with no options specified. 32-bit was about 0.01%. >> >> >>>I think I missed something... how do I run the various >>> >>> >>accounting choices and which ones are known to be appropriate >>for which kernels? >> >> >>>Thanks, >>>Dan >>> >>> >>> >>> >>> >>>>-----Original Message----- >>>>From: xen-devel-bounces@lists.xensource.com >>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>> >>>> >>Keir Fraser >> >> >>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>To: Dave Winchell >>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, >>>>Yunhong >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>disables pending >>>>missed ticks >>>> >>>> >>>>Please take a look at xen-unstable changeset 16545. >>>> >>>>-- Keir >>>> >>>>On 26/11/07 20:57, "Dave Winchell" >>>> >>>> >> wrote: >> >> >>>> >>>> >>>> >>>>>Keir, >>>>> >>>>>The accuracy data I've collected for i/o loads for the >>>>>various time protocols follows. In addition, the data >>>>>for cpu loads is shown. >>>>> >>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>The cpu load is usex -e36 on each guest. >>>>>(usex is available at http://people.redhat.com/anderson/usex.) >>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>>>> >>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>Also, these are run on 4 cpu AMD box. >>>>>In addition, there is an idle rh-32bit guest. >>>>>All three guests are 8vcpu. >>>>> >>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>except that the redhat-64 guest has 4 instances of dd. >>>>> >>>>>Date Duration Protocol sles, rhat error load >>>>> >>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu >>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu >>>>> >>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>>>> >>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu >>>>> >>>>> >>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 >>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 >>>>> >>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 >>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>>>> >>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 >>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 >>>>> >>>>> >>>>> >>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 >>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>>>> >>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 >>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 >>>>> >>>>> >>>>>Overhead measurements: >>>>> >>>>>Progress in terms of number of passes through a fixed >>>>> >>>>> >>>>> >>>>> >>>>system workload >>>> >>>> >>>> >>>> >>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>The workload was usex -b48. >>>>> >>>>> >>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>SYNC 167 min 144 passes .862 passes/min >>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>MIXED 221 min 196 passes .887 passes/min >>>>> >>>>> >>>>>Conclusions: >>>>> >>>>>The only protocol which meets the .05% accuracy requirement for ntp >>>>>tracking under the loads >>>>>above is the SYNC protocol. The worst case accuracies for >>>>> >>>>> >>>>> >>>>> >>>>SYNC, MIXED, >>>> >>>> >>>> >>>> >>>>>and ASYNC >>>>>are .022%, .12%, and .14%, respectively. >>>>> >>>>>We could reduce the cost of the SYNC method by only >>>>> >>>>> >>>>> >>>>> >>>>scheduling the extra >>>> >>>> >>>> >>>> >>>>>wakeups if a certain number >>>>>of ticks are missed. >>>>> >>>>>Regards, >>>>>Dave >>>>> >>>>>Keir Fraser wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>> >>>>>> >>>>>> >>>>>> >>>> wrote: >>>> >>>> >>>> >>>> >>>>>> >>>>>> >>>>>> >>>>>>>Since I had a high error (~.03%) for the ASYNC method a >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>couple of days ago, >>>> >>>> >>>> >>>> >>>>>>>I ran another ASYNC test. I think there may have been something >>>>>>>wrong with the code I used a couple of days ago for >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>ASYNC. It may have been >>>> >>>> >>>> >>>> >>>>>>>missing the immediate delivery of interrupt after context >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>switch in. >>>> >>>> >>>> >>>> >>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>acceptable accuracy, >>>> >>>> >>>> >>>> >>>>>>>each running consistently around or under .01%. MIXED has >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>a fairly high >>>> >>>> >>>> >>>> >>>>>>>error of >>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>threshold for comfort. >>>> >>>> >>>> >>>> >>>>>>>I don't have an overnight run with SYNC. I plan to leave >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>SYNC running >>>> >>>> >>>> >>>> >>>>>>>over the weekend. If you'd rather I can leave MIXED >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>running instead. >>>> >>>> >>>> >>>> >>>>>>>It may be too early to pick the protocol and I can run >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>more overnight tests >>>> >>>> >>>> >>>> >>>>>>>next week. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>I'm a bit worried about any unwanted side effects of the >>>>>> >>>>>> >>>>>> >>>>>> >>>>SYNC+run_timer >>>> >>>> >>>> >>>> >>>>>>approach -- e.g., whether timer wakeups will cause higher >>>>>> >>>>>> >>>>>> >>>>>> >>>>system-wide CPU >>>> >>>> >>>> >>>> >>>>>>contention. I find it easier to think through the >>>>>> >>>>>> >>>>>> >>>>>> >>>>implications of ASYNC. I'm >>>> >>>> >>>> >>>> >>>>>>surprised that MIXED loses time, and is less accurate than >>>>>> >>>>>> >>>>>> >>>>>> >>>>ASYNC. Perhaps it >>>> >>>> >>>> >>>> >>>>>>delivers more timer interrupts than the other approaches, >>>>>> >>>>>> >>>>>> >>>>>> >>>>and each interrupt >>>> >>>> >>>> >>>> >>>>>>event causes a small accumulated error? >>>>>> >>>>>>Overall I would consider MIXED and ASYNC as favourites and >>>>>> >>>>>> >>>>>> >>>>>> >>>>if the latter is >>>> >>>> >>>> >>>> >>>>>>actually more accurate then I can simply revert the changeset that >>>>>>implemented MIXED. >>>>>> >>>>>>Perhaps rather than running more of the same workloads you >>>>>> >>>>>> >>>>>> >>>>>> >>>>could try idle >>>> >>>> >>>> >>>> >>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>>>> >>>>>> >>>>>> >>>>>> >>>>to /dev/null)? We >>>> >>>> >>>> >>>> >>>>>>don't have any data on workloads that aren't CPU bound, so >>>>>> >>>>>> >>>>>> >>>>>> >>>>that's really an >>>> >>>> >>>> >>>> >>>>>>obvious place to put any further effort imo. >>>>>> >>>>>>-- Keir >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>_______________________________________________ >>>>Xen-devel mailing list >>>>Xen-devel@lists.xensource.com >>>>http://lists.xensource.com/xen-devel >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > > --------------000708050109000100010403 Content-Type: text/plain; name="p.1.04.sync.time" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="p.1.04.sync.time" diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c --- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 +++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 @@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru missed_ticks = missed_ticks / (s_time_t) pt->period + 1; if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) - pt->do_not_freeze = !pt->pending_intr_nr; + pt->do_not_freeze = 1; else pt->pending_intr_nr += missed_ticks; pt->scheduled += missed_ticks * pt->period; @@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) pt_lock(pt); - pt->pending_intr_nr++; + if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { + pt->pending_intr_nr = 1; + pt->do_not_freeze = 0; + } + else + pt->pending_intr_nr++; if ( !pt->one_shot ) { @@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct return; } - pt->do_not_freeze = 0; - if ( pt->one_shot ) { pt->enabled = 0; @@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct pt->last_plt_gtime = hvm_get_guest_time(v); pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */ } + else if ( mode_is(v->domain, no_missed_ticks_pending) ) { + pt->pending_intr_nr--; + pt->last_plt_gtime = hvm_get_guest_time(v); + } else { pt->last_plt_gtime += pt->period_cycles; --------------000708050109000100010403 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------000708050109000100010403-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 09 Jan 2008 11:53:07 -0500 Message-ID: <4784FBF3.3040503@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Keir, The latest change, c/s 16690, looks fine. I agree that the code in c/s 16690 is equivalent to the code I submitted. Also, your version is more concise. The error tests confirm the equivalence. With overnight cpu loads, the checked in version was accurate to +.048% for sles and +.038% for red hat. My version was +.046% and +.032% in a 2 hour test. I don't think the difference is significant. i/o loads produced errors of +.01%. Thanks for all your efforts on this issue. Regards, Dave Keir Fraser wrote: >Applied as c/s 16690, although the checked-in patch is smaller. I think the >only important fix is to pt_intr_post() and the only bit of the patch I >totally omitted was the change to pt_process_missed_ticks(). I don't think >that change can be important, but let's see what happens to the error >percentage... > > -- Keir > >On 4/1/08 23:24, "Dave Winchell" wrote: > > > >>Hi Dan and Keir, >> >>Attached is a patch that fixes some issues with the SYNC policy >>(no_missed_ticks_pending). >>I have not tried to make the change the minimal one, but, rather, just >>ported into >>the new code what I know to work well. The error for >>no_missed_ticks_pending goes from >>over 3% to .03% with this change according to my testing. >> >>Regards, >>Dave >> >>Dan Magenheimer wrote: >> >> >> >>>Hi Dave -- >>> >>>Did you get your correction ported? If so, it would be nice to see this get >>>into 3.1.3. >>> >>>Note that I just did some very limited testing with timer_mode=2(=SYNC=no >>>missed ticks pending) >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the worst error I've >>>seen so far >>>is 0.012%. But I haven't tried any exotic loads, just LTP. >>> >>>Thanks, >>>Dan >>> >>> >>> >>> >>> >>>>-----Original Message----- >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>To: dan.magenheimer@oracle.com >>>>Cc: Keir Fraser; Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>disables pending >>>>missed ticks >>>> >>>> >>>>Dan, >>>> >>>>I did some testing with the constant tsc offset SYNC method >>>>(now called >>>>no_missed_ticks_pending) >>>>and found the error to be very high, much larger than 1 %, as >>>>I recall. >>>>I have not had a chance to submit a correction. I will try to >>>>do it later >>>>this week or the first week in January. My version of constant tsc >>>>offset SYNC method >>>>produces .02 % error, so I just need to port that into the >>>>current code. >>>> >>>>The error you got for both of those kernels is what I would expect >>>>for the default mode, delay_for_missed_ticks. >>>> >>>>I'll let Keir answer on how to set the time mode. >>>> >>>>Regards, >>>>Dave >>>> >>>>Dan Magenheimer wrote: >>>> >>>> >>>> >>>> >>>> >>>>>Anyone make measurements on the final patch? >>>>> >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>>>> >>>>> >>>>> >>>>> >>>>about 0.2% with no load. This was xen-unstable tip today >>>>with no options specified. 32-bit was about 0.01%. >>>> >>>> >>>> >>>> >>>>>I think I missed something... how do I run the various >>>>> >>>>> >>>>> >>>>> >>>>accounting choices and which ones are known to be appropriate >>>>for which kernels? >>>> >>>> >>>> >>>> >>>>>Thanks, >>>>>Dan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>-----Original Message----- >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>> >>>>>> >>>>>> >>>>>> >>>>Keir Fraser >>>> >>>> >>>> >>>> >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>To: Dave Winchell >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, Eddie; Jiang, >>>>>>Yunhong >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>disables pending >>>>>>missed ticks >>>>>> >>>>>> >>>>>>Please take a look at xen-unstable changeset 16545. >>>>>> >>>>>>-- Keir >>>>>> >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>> >>>>>> >>>>>> >>>>>> >>>> wrote: >>>> >>>> >>>> >>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>Keir, >>>>>>> >>>>>>>The accuracy data I've collected for i/o loads for the >>>>>>>various time protocols follows. In addition, the data >>>>>>>for cpu loads is shown. >>>>>>> >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.) >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. >>>>>>> >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>All three guests are 8vcpu. >>>>>>> >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>>>except that the redhat-64 guest has 4 instances of dd. >>>>>>> >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>> >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, +.005% cpu >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, +.012% cpu >>>>>>> >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu >>>>>>> >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, -.031% cpu >>>>>>> >>>>>>> >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, -.09% i/o-8 >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% -.14% i/o-8 >>>>>>> >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, -.022% i/o-8 >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 >>>>>>> >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, -.029% i/o-8 >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, -.031% i/o-8 >>>>>>> >>>>>>> >>>>>>> >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, -.005% i/o-32 >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 >>>>>>> >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, .003% i/o-4/32 >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, .01% i/o-4/32 >>>>>>> >>>>>>> >>>>>>>Overhead measurements: >>>>>>> >>>>>>>Progress in terms of number of passes through a fixed >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>system workload >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>The workload was usex -b48. >>>>>>> >>>>>>> >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>> >>>>>>> >>>>>>>Conclusions: >>>>>>> >>>>>>>The only protocol which meets the .05% accuracy requirement for ntp >>>>>>>tracking under the loads >>>>>>>above is the SYNC protocol. The worst case accuracies for >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>SYNC, MIXED, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>and ASYNC >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>> >>>>>>>We could reduce the cost of the SYNC method by only >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>scheduling the extra >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>wakeups if a certain number >>>>>>>of ticks are missed. >>>>>>> >>>>>>>Regards, >>>>>>>Dave >>>>>>> >>>>>>>Keir Fraser wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>couple of days ago, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>I ran another ASYNC test. I think there may have been something >>>>>>>>>wrong with the code I used a couple of days ago for >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>ASYNC. It may have been >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>missing the immediate delivery of interrupt after context >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>switch in. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>acceptable accuracy, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>each running consistently around or under .01%. MIXED has >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>a fairly high >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>error of >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>threshold for comfort. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>SYNC running >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>over the weekend. If you'd rather I can leave MIXED >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>running instead. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>It may be too early to pick the protocol and I can run >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>more overnight tests >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>>next week. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>I'm a bit worried about any unwanted side effects of the >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>SYNC+run_timer >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>approach -- e.g., whether timer wakeups will cause higher >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>system-wide CPU >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>contention. I find it easier to think through the >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>implications of ASYNC. I'm >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>surprised that MIXED loses time, and is less accurate than >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>ASYNC. Perhaps it >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>delivers more timer interrupts than the other approaches, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>and each interrupt >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>event causes a small accumulated error? >>>>>>>> >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>if the latter is >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>actually more accurate then I can simply revert the changeset that >>>>>>>>implemented MIXED. >>>>>>>> >>>>>>>>Perhaps rather than running more of the same workloads you >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>could try idle >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>to /dev/null)? We >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>don't have any data on workloads that aren't CPU bound, so >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>that's really an >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>>obvious place to put any further effort imo. >>>>>>>> >>>>>>>>-- Keir >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>_______________________________________________ >>>>>>Xen-devel mailing list >>>>>>Xen-devel@lists.xensource.com >>>>>>http://lists.xensource.com/xen-devel >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>> >>> >>> >>> >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru >> >> missed_ticks = missed_ticks / (s_time_t) pt->period + 1; >> if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) >>- pt->do_not_freeze = !pt->pending_intr_nr; >>+ pt->do_not_freeze = 1; >> else >> pt->pending_intr_nr += missed_ticks; >> pt->scheduled += missed_ticks * pt->period; >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >> >> pt_lock(pt); >> >>- pt->pending_intr_nr++; >>+ if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { >>+ pt->pending_intr_nr = 1; >>+ pt->do_not_freeze = 0; >>+ } >>+ else >>+ pt->pending_intr_nr++; >> >> if ( !pt->one_shot ) >> { >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct >> return; >> } >> >>- pt->do_not_freeze = 0; >>- >> if ( pt->one_shot ) >> { >> pt->enabled = 0; >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct >> pt->last_plt_gtime = hvm_get_guest_time(v); >> pt->pending_intr_nr = 0; /* 'collapse' all missed ticks */ >> } >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { >>+ pt->pending_intr_nr--; >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>+ } >> else >> { >> pt->last_plt_gtime += pt->period_cycles; >> >> > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 25 Jan 2008 16:50:36 -0700 Message-ID: <20080125165036078.00000002440@djm-pc> References: <4784FBF3.3040503@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=-------72c38c1b72c38c1b Return-path: In-Reply-To: <4784FBF3.3040503@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , Keir Fraser Cc: "akira.ijuin@oracle.com" , "xen-devel@lists.xensource.com" , "deepak.patel@oracle.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format ---------72c38c1b72c38c1b Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Sorry for the very late followup on this but we finally were able to get our testing set up again on stable 3.1 bits and have seen some very bad results on 3.1.3-rc1, on the order of 1%. Test enviroment was a 4-socket dual core machine with 24GB of memory running six two-vcpu 2GB domains, four hvm plus two pv. All six guests were running LTP simultaneously. The four hvm guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests. All four hvm guests experienced skew around -1%, even the 32-bit guest. Less intensive testing didn't exhibit much skew at all. A representative graph is attached. Dave, I wonder if some portion of your patches didn't end up in the xen trees? (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer 14.318MHz HPET.) Thanks, Dan P.S. Many thanks to Deepak and Akira for running tests. > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of = > Dave Winchell > Sent: Wednesday, January 09, 2008 9:53 AM > To: Keir Fraser > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave > Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Hi Keir, > = > The latest change, c/s 16690, looks fine. > I agree that the code in c/s 16690 is equivalent to > the code I submitted. Also, your version is more > concise. > = > The error tests confirm the equivalence. With overnight cpu loads, > the checked in version was accurate to +.048% for sles > and +.038% for red hat. My version was +.046% and +.032% in a = > 2 hour test. > I don't think the difference is significant. > = > i/o loads produced errors of +.01%. > = > Thanks for all your efforts on this issue. > = > Regards, > Dave > = > = > = > Keir Fraser wrote: > = > >Applied as c/s 16690, although the checked-in patch is = > smaller. I think the > >only important fix is to pt_intr_post() and the only bit of = > the patch I > >totally omitted was the change to pt_process_missed_ticks(). = > I don't think > >that change can be important, but let's see what happens to the error > >percentage... > > > > -- Keir > > > >On 4/1/08 23:24, "Dave Winchell" wrote: > > > > > > > >>Hi Dan and Keir, > >> > >>Attached is a patch that fixes some issues with the SYNC policy > >>(no_missed_ticks_pending). > >>I have not tried to make the change the minimal one, but, = > rather, just > >>ported into > >>the new code what I know to work well. The error for > >>no_missed_ticks_pending goes from > >>over 3% to .03% with this change according to my testing. > >> > >>Regards, > >>Dave > >> > >>Dan Magenheimer wrote: > >> > >> > >> > >>>Hi Dave -- > >>> > >>>Did you get your correction ported? If so, it would be = > nice to see this get > >>>into 3.1.3. > >>> > >>>Note that I just did some very limited testing with = > timer_mode=3D2(=3DSYNC=3Dno > >>>missed ticks pending) > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the = > worst error I've > >>>seen so far > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. > >>> > >>>Thanks, > >>>Dan > >>> > >>> > >>> > >>> > >>> > >>>>-----Original Message----- > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>To: dan.magenheimer@oracle.com > >>>>Cc: Keir Fraser; Shan, Haitao; = > xen-devel@lists.xensource.com; Dong, > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>disables pending > >>>>missed ticks > >>>> > >>>> > >>>>Dan, > >>>> > >>>>I did some testing with the constant tsc offset SYNC method > >>>>(now called > >>>>no_missed_ticks_pending) > >>>>and found the error to be very high, much larger than 1 %, as > >>>>I recall. > >>>>I have not had a chance to submit a correction. I will try to > >>>>do it later > >>>>this week or the first week in January. My version of constant tsc > >>>>offset SYNC method > >>>>produces .02 % error, so I just need to port that into the > >>>>current code. > >>>> > >>>>The error you got for both of those kernels is what I would expect > >>>>for the default mode, delay_for_missed_ticks. > >>>> > >>>>I'll let Keir answer on how to set the time mode. > >>>> > >>>>Regards, > >>>>Dave > >>>> > >>>>Dan Magenheimer wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>>Anyone make measurements on the final patch? > >>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>about 0.2% with no load. This was xen-unstable tip today > >>>>with no options specified. 32-bit was about 0.01%. > >>>> > >>>> > >>>> > >>>> > >>>>>I think I missed something... how do I run the various > >>>>> > >>>>> > >>>>> > >>>>> > >>>>accounting choices and which ones are known to be appropriate > >>>>for which kernels? > >>>> > >>>> > >>>> > >>>> > >>>>>Thanks, > >>>>>Dan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>-----Original Message----- > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>Keir Fraser > >>>> > >>>> > >>>> > >>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>>>To: Dave Winchell > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, = > Eddie; Jiang, > >>>>>>Yunhong > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>>>disables pending > >>>>>>missed ticks > >>>>>> > >>>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. > >>>>>> > >>>>>>-- Keir > >>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>> wrote: > >>>> > >>>> > >>>> > >>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>Keir, > >>>>>>> > >>>>>>>The accuracy data I've collected for i/o loads for the > >>>>>>>various time protocols follows. In addition, the data > >>>>>>>for cpu loads is shown. > >>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.) > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 of=3D/dev/null. > >>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>>>>All three guests are 8vcpu. > >>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > >>>>>>>except that the redhat-64 guest has 4 instances of dd. > >>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, = > +.005% cpu > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, = > +.012% cpu > >>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu > >>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, = > -.031% cpu > >>>>>>> > >>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, = > -.09% i/o-8 > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% = > -.14% i/o-8 > >>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, = > -.022% i/o-8 > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 > >>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, = > -.029% i/o-8 > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, = > -.031% i/o-8 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, = > -.005% i/o-32 > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 > >>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, = > .003% i/o-4/32 > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, = > .01% i/o-4/32 > >>>>>>> > >>>>>>> > >>>>>>>Overhead measurements: > >>>>>>> > >>>>>>>Progress in terms of number of passes through a fixed > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>system workload > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>>>>>The workload was usex -b48. > >>>>>>> > >>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>>>> > >>>>>>> > >>>>>>>Conclusions: > >>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy = > requirement for ntp > >>>>>>>tracking under the loads > >>>>>>>above is the SYNC protocol. The worst case accuracies for > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>SYNC, MIXED, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>and ASYNC > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>>>> > >>>>>>>We could reduce the cost of the SYNC method by only > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>scheduling the extra > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>wakeups if a certain number > >>>>>>>of ticks are missed. > >>>>>>> > >>>>>>>Regards, > >>>>>>>Dave > >>>>>>> > >>>>>>>Keir Fraser wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>couple of days ago, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have = > been something > >>>>>>>>>wrong with the code I used a couple of days ago for > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>ASYNC. It may have been > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>missing the immediate delivery of interrupt after context > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>switch in. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>acceptable accuracy, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>each running consistently around or under .01%. MIXED has > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>a fairly high > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>error of > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>threshold for comfort. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>SYNC running > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>running instead. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>It may be too early to pick the protocol and I can run > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>more overnight tests > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>next week. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side effects of the > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>SYNC+run_timer > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will cause higher > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>system-wide CPU > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>contention. I find it easier to think through the > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>implications of ASYNC. I'm > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>surprised that MIXED loses time, and is less accurate than > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>ASYNC. Perhaps it > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>delivers more timer interrupts than the other approaches, > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>and each interrupt > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>event causes a small accumulated error? > >>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>if the latter is > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>actually more accurate then I can simply revert the = > changeset that > >>>>>>>>implemented MIXED. > >>>>>>>> > >>>>>>>>Perhaps rather than running more of the same workloads you > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>could try idle > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>to /dev/null)? We > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>don't have any data on workloads that aren't CPU bound, so > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>that's really an > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>>>>> > >>>>>>>>-- Keir > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>_______________________________________________ > >>>>>>Xen-devel mailing list > >>>>>>Xen-devel@lists.xensource.com > >>>>>>http://lists.xensource.com/xen-devel > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> > >>> > >>> > >>> > >>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru > >> > >> missed_ticks =3D missed_ticks / (s_time_t) pt->period + 1; > >> if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>+ pt->do_not_freeze =3D 1; > >> else > >> pt->pending_intr_nr +=3D missed_ticks; > >> pt->scheduled +=3D missed_ticks * pt->period; > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > >> > >> pt_lock(pt); > >> > >>- pt->pending_intr_nr++; > >>+ if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { > >>+ pt->pending_intr_nr =3D 1; > >>+ pt->do_not_freeze =3D 0; > >>+ } > >>+ else > >>+ pt->pending_intr_nr++; > >> > >> if ( !pt->one_shot ) > >> { > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct > >> return; > >> } > >> > >>- pt->do_not_freeze =3D 0; > >>- > >> if ( pt->one_shot ) > >> { > >> pt->enabled =3D 0; > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct > >> pt->last_plt_gtime =3D hvm_get_guest_time(v); > >> pt->pending_intr_nr =3D 0; /* 'collapse' all = > missed ticks */ > >> } > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { > >>+ pt->pending_intr_nr--; > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>+ } > >> else > >> { > >> pt->last_plt_gtime +=3D pt->period_cycles; > >> > >> > > > > > > > > > = > = > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > ---------72c38c1b72c38c1b Content-Type: application/octet-stream; name="el4u5-32-hvm-ltp.png" Content-Disposition: attachment; filename="el4u5-32-hvm-ltp.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAaMAAAGkEAAAAADpa8nTAAAACXBIWXMAAABIAAAASABGyWs+ AAAACXZwQWcAAAGjAAABpADnhMYnAAAeqElEQVR42u2dPXKsvNaFu+om91TdhDphJ7fIvipn BLcHQNCJQ3JHlGeAh0B5BkyBxANgCkyBKTAFf17vtkrQDd3qFgIE6wnOsbux+JEWkra29j58 E0IsOSx9AYT4D2VEiDWUEfk+XJBl8qn8PzV1HcdTlZ4kVZVlh8Vb8eIXMA1VlSRoAFFUlkPf xjG+TdO2vfyuruUvk6Sux8tHVSnQCID8Xf8zTZ6H4eEQhmONpWn6lV8UUTR+B9fHozEeDkHQ LT/LggDXMnYnl2Uo8GSqDk0jn7qRUZblebf069oZqpO2TVPcXRR1PwsCqRspx831mrAJGVWV ND8R02UzxLd4axVFEERR/7u6DoI4Ln+IoiC4Fpkiy6JINTJVjXGMUvuf6ePRTORNOVy5adpt 0jguSXAdQ3dwfXzT4F6qKs91+Sgjz6tq/E5Q9tDnw9foSkZRBJkqsV7XjtybqhM5Cn8VBEUh d6c+K8s01TJaslfahIySJAxVw4njS6mEoeoriuJw6Dd4/KX8hDfbeLOBZC4/u9XMggAVDPAO vS4NPVW30iFm/e3lHVwfjyYjd4weCD+1rboe/ZMGfRfe5cvLqGn6d3ddO3g1iFDQe6Ln6tZc 9+7SFC8cymgiokg3wsuHWdfDb/ff2+80lSS5bL7dM1w2KVTnWLl4w1bV9c/6GrMMA5luWbr8 6+ZweTyEpe5Yld89j/5W0TTXZQw/hdufoofoDrfiuKrQT2Io2TT9YVdRYHiphp7qNZfnIgwp fah29CtCP5fhupFXoDwvNezGneOq7g1wp2UTMuqCN3f3d7zb2h+uB17/3H5PRuNvMzQc9Ai6 QchgRM1/+oOovhCGG+n4u/PyDu6VKT93vx8re+xzUxmpARjEJNeIJ6KGZHgSZYmfdPPGs5Gh rerNlcik9KHawaAO8ySZDUm/JCOFuq4q/aQxANZ3BRHjRYJycF49LJxDSBuTER6lvO0UMjuR N1UYXj7SKFJvOQzqxho2BhcyMsc7Tt70GGbIfAyl99/+z8sITefyDu6VOYWM+kivdn3d+u/V dF7fOb6Tv+sOsrrzNjz7plGvCH3V17UDcQnqSUCG0g92h3RFcXk+dbSaHaJW1fDaJRuSUVGg bxiqekzGZVR+OfkuS/1+HZs5yJBITWshHwxD0JD0QKU/NHlWRvJOH56RuJVR31In9zXWG5Wl foZ9ofR/0oMzPcspCnW8vurL2lF10jUYaUlpo4qaDw/JSF/50Nx0ejYio6bBGBxj4stv8HBV E9dTVo0ICIOGLMObsmmqCy6tXkMNvf9Zt7kOTfgvjwF1jesYuoOh45cZ1OFpiXE6DLsznH7Z w1N+OVKbsPVVX9aONjroAe61MGAUHzvf7ZmmCzYhI4ymMega+g7vNt00b1mfkgQV010h6g5x Oo/srozumRhAv3pheA/DMQldH/+oiWGojNt3dOtZST+B523aG+Flkudtq2d98pdDtTMkgq6J R5kUVH3fkxF7I0PwfhqbSEoFys9oan27UJapkTPehcNClArqDupwrq4h4NqQftvgrcrUv4Wh NtmPX0P/N23wlvtUBvvuHd8qQ2MqoyS57APvyag/NyoKPU+R74Zqp98bydwVVsb+Z2GoauTe 3Oh6qWJ6NiEjNZZW4DP94NVSKGYe6rM4lgpW43DMq4bf4EAWBJWJQZoC/hILt93P9DllBC/m B9VM1Dn1Vamf0YBwdP8Oxo+X5nF/+bXb+K7L0OVfzo2ULa3/qRhwcJ2ySIxmfE9G+imgMWNe petM/vK6dobmRiIHdWZcoTaAd89XlnJVYv6RATstdUagH+mDT7trJJj19FcvtIWpLC+/Gz6H rIl0nXu0A5IWij6ncgbS/ULfntdt0ujNru9g/Hhg4gx0uU50OVtR5V+eXT6/viblkCPNXI65 LSOIWj/driG/+5eXNaAMPmJ8UPcrT1tmj2r1qXu+ukY5yl9PnkW3BJdsQEZknSznVeDKjenG GZe5UbJ9KCNCrKGMCLEGPnXLnDmOx2yurqCMCLGGMiLEGsqIEGsoI0KsoYwIsYYyIsQayogQ aygjQqyhjAixhjIixBonMlKu7xIHU8J0DMcsJWQLOJARNmCpIBXYrJWmUYRwSVE0R4wWQubH gYywk1H9BB9ftdtefiNkezht2BjcdcUzHNyDEN9xJiPZdo04l8MyatuPj/SHz1++vipCVsbX l2qfaKsfH2Ozeycyalts2UpTyRwwLKOPjz9/jj+cfnl/z0Y5n9/eMkOOR9Mj397O52VLfXl5 fTU99nQyPfL19eVl2VK3U1/v76p9oq3++fPxMZuMYExIEhX+aGxulKbHo2mJiBxjeuzfv6ZH PpIPx02pp9Pnp+mx5lvgPj9Pp2VL3Wp9HY9jRjIHMkIuoO7vw5Y6yogyAj7V16wykvC0/bBM 1+tGj1TLY2f3p9RHmps5brLOucpl51N9jb/2FjNBU0aUEfCpviijVZZKGflVX5TRKkuljPyq rxXK6Ovr/d1FuW4iNrsptWlceBm2rbaSrr1Uv+rr/f3ra/ibxWS0ZPp1Qp5hfPRAGRFiCGVE iDWUESHWUEaEWEMZEWINZUSINZQRIdZQRoRYQxkRYg1lRIg1lBEh1lBGhFizShlJKImlzk/I I0iolhXK6O0NIYyWOj8hj4C2ihY7/C0HdYQYsspBHWVE/IIyIsQayogQaygjQqyhjAixhjIi xBrKiBBrKCNCrKGMCLGGMiLEGsqIEGsoI0KsoYwIsYYyIsSaRWU0nLSSMiK+saiMhlMoU0bE NxaV0eEgJ6+qQ+dslBHxjQVl1BWPEpR8ThkRv6CMCLFmlTL617/+/e8giOP//vd//4vjus4y ydi5vf+Lgv9/fxeFf//XdRz/3w9x/Pfvf/4TBKucGxVF/ENZur8GQuxZsaWurvF9lrnIyU3I lKx83aht8zwM05RR68ia8cKLoaqSJIpkPkHI+vBCRqCuw7BplromQm7hjYzwOWZSS10VIeN4 JKPv7zzvmiIIWQteyej7O0nEck/ImvBMRjCO09RA1oZnMhJTA2dIZF14J6Pv77KM46WujZAh PJSR5Ddb6uoIucZLGX1/xzFNDWQ9eCojmhrImvBURjA1cDGWrAVvZYRdH0my1DUS0sVjGWGj RZ4vdZWEaLyW0fd3FNHUQJbHcxnB1EAhkaXxXEYUElkDq5TR21tVme93pZDIkqCtosUOf7ug jM7nx/wUKCSyHGir5/MKZfS4qw+FRJZklYO6ZzzmKCSyHJuREYVElmNDMqKQyFJsSkYUElmG jcmIvt9kCTYnIwgJ4fOXunqyRzYoI26iIHOzSRlhEwXjNZD52KiMGK+BzMlmZYR4DcyPROZh wzJq2zCkqYHMwYZlRFMDmYtNy4jxGsg8bFxGjNdA5mDzMoJXA1NeErfMLKMsC8PDIQgk2+v9 3K9T0DTM00fcMquMiiIIcLqmiSLMWW5nIp8O5Omb/m4IUcwqo6pS3tdVdfgp/3CQk8tv+qjp F06zjHn6iDsWmhthcNcVjxIUcCEj5ukjLllARk2TpkFQ1+MyOh7//hD/Ms0SKjdQkCmpa9U+ 0VaPR4cyOvTAJ22bZTApYMo/b2/EPH3EHbP2RugRkkRbzeabGwllSa8G4oJZZZRlfYvZXJY6 TZ7T1ECmZ1YZxXF/kDfPulEfejWQ6dm8F8Ml6Pu4gYJMy+5kxA0UZHp2KCNuoCBTs0sZMVYD mZadyoixGsiU7FZGsBvSPYhMw45lRPcgMhU7lhHdg8hU7FpGdA8i07BzGcFmR/cgYsvuZQT3 INrsiB2U0TdtdsQWyuibqVyILZTRP8Bmx+hB5FmMZVT9wxxWrSVkJNGDaLMjz3FXRm2b51Gk dwmFYZq6DZ+4jIwYqJg8zx0ZYb9qlpWlmjm0bVXleZK4nEtU1dsber75HwZtduRx0FbRYoe/ Pcghw182jUsZnc9LOY7SZkceBW31fH7AxFDXc0zDlxrUAfrZkWcwMjGUJYIGlyXmRu7f1kvK iDY78gxGMgrDPJcYBlUVhq4vaVkZcW8seRwjGUkUH4kld3C+nrS0jGizI49i2BuJfU6LySXL y4g2O/IYRjIqCsyKEJYqTd2/p9cgo+9vhuEi5hh6MdS12K+Kwv2sYR0yYhguYg596kahqYGY ckdGhyvcB6Zai4y4pY+YckdG4pCa50GQ5/K/+xnDemTEiN/EDKNBnc7lvYd1oy7MY05MMF43 GvrZDWuSEfOYExOMZCT5w8HeeiO5HpoayG2MZJRlQVAUyCMeBO6b+NpkxORi5B6GBm9kDseW vTka+PpkRFMDuQ3XjYygqYHcwtAZKAj6GcVtaFs111oiaeWzwNTAGRIZxtA1FfEXFPanVBnI 50+hbANMDUtfA1knDxu8bcEyrpKR+h+f6CPWKiM8LJoayBCGvdFUKyd1DQ9xkU9XPEpQYL0y +v5OEsZqINcYyaiuEerDflCH4RsEeU9GLy+nH7Jf1rT4yVgNRGga1T7RVl9eDGQUx13nVPNT 9Z1adaLIezJ6ff38Qcl2XRN7+n0TgFBzAtrq6+uMBu9LOfo2NxKYfplcYjSoa1v4MRwOU/kw +Gmp0zD9Mulj6AwUReIMhBiq9idVMvJp3agPt5iTLoauqWqa3zRB4PqSfJARt5iTLobrRqq3 2EtkoPvQ1EA0RjJKkjRFf9Q0+4kMdB/6fROFoYlBpWaZ4w3si4zo900Uxh7edV1V88wG/JER /b6JYCgjWOnwP2J5u74kf2REv28iGO9+leFLHLufD/gkI2Tb4GIsMTR462x7NHhfwsVYQoP3 BHCGtHceNnjvKWqqKU3DNaR9Y2jwVk6lNHgPw4xI+8bY4N00NHjfgmtIe4aRgSaCG/r2jHGc OmyUwDvXfVPxU0b0stszxhslkIccy6/0qRuHXnZ75YGNEvtJofw8DHmyTx4IsNXNR+4Sn2XE GdI+MVw3QsPGIuw8GyVOpzj21cWGM6S9gbZ6OhmuGyEUPlJWct3oHnnu9/WTx+G6kQPimJEa 9sVD60Z1PUfoRf9lxH1Ie8NIRmWJHBAweR8O7i1R/stIhESb3X4wjOGN7XpYO9pj0srnoJD2 hLHBW5m6afA2hULaD4a9UVWJ/wLXjR4jTenXsAcMs+1hVgTrEwNsPQqFtAcMLXV1LcbuouC6 0aNQSNvnjoySZGgFBLld3I36tyYjPOQkoWfDlrkjI2wbD4I0xWGgKODtHYYulxe3JyP04nQR 2jIGg7qmyfM4llzkSKHieoV+izKikLYNd7/OBoW0XSijGaGQtgplNCsQ0tLXQKaHMpoZbjTf IjPLqCyR4kXlkPU3aaUNXEfaHkYygnVuijF9VcFTXPIwIKqbrymUbaGQtoahjLD3NUny3G6/ kWxG75ziN5VyVXU99bYvIzxROq1uiQd2vxYF5GQzRcZgTg3q2rYrHiUosAcZ0ft7WxjLCI07 SYLARkaYB2Ewh9DxOPGYjF5fP3+oftmmiZhC8ht0AwLa6uurgYwQCB8uQY86ph564HcV7wc7 acdl9PJy+iH7ZY6N60vAYFw+A79SAW315cVov1EQ2M+MkAdIywjpxvY7NxJgaqGQtoDhoE5m RnBKtbExYd+SHtTt11KnqWsKaQs8sG6E5h1Fdkkrsww2vzDc87pRH4aH3AKGofAxO0LjnyP+ 2r5kxBRjW8BIRlGU5/MNPfYmIwxu93bHW4M+dasAwcuWvgbyPEwTtgralqYGn2GasJVQ13Ok GSBuYJqw1UBTg78wTdiKwJr40tdAnmGVacL225jimH52PmIkI6YJmwv62fkJ04StDHo1+Ihh b6R/cr9vc98ygqnB17y3+8XQi0GEBL9s95Ft9i4jmhr8wzCjBEbsSXI4zFG9lBFNDb5hODfC FgdkOZrjkigjmhp844FN5HO5q1BGgFv6fOKOjA4DuL4kykigzc4f7sioGsD1JVFGCgTHpJB8 wGBQV9cSg0Fl3HNNVZ1OcUyjL6Cf3fpBWz2dbsoI/gtin5Pt3nO8HdkbdckyRlhdP3d6IwQd QV9U10ihjMVXLr/OTZoiDAxZM3dkJFskZMcR/m9bu5AmJlBGfRgYcv3ctdTJLxIQq/uJOyij SzC0pvF7zRj1RthlJIexN1oGxrNbN3dkhMFcVSEbuTqc+42WgZvM18wdGUlAxiCQsTnkxJAm S8GEl+vloQBbdT3H+5AyGoMJL9cK49R5BY3f64Qy8goYv+fxsyePQBl5Bjy/t5rxyV8oI++o Kjqsrg3GYvAQ5Jpa+hpIF8Zi8BLmoFgXjMXgKcxBsSYYi8FTmINiTcwci6EsJWmlvEmZtNIG bjJfD7PGYmgacXFF5nEYbZlC2Q7ujV0Ls8ZigHyUjPC/8hvH792jKCNTGBhyHRgO6qpKnFPz 3G4YEUXSo8He1xWPEpSci03DHAaGXAOG2faCQHy54thmxaJtswzzq7rOMoTWH5PR6+vnD6r3 4/j/Ftwbuwxtq9on2urrq1G2PWVceGzb3uWcShtpETpqXEYvL6cfsl/o+nIbCmkJmka1T7TV lxejbHuqR7DLtoewKPKTZJLl3GgaKKSlMcy2l6boE5omTW3ix8EyJ4M6sc3RUjcViNbALRTL YZxtT5kGbGYqmBsFAXbTZhnK4brRlMwR/IwMw2x7G4JCWgrD3kj3Iu4viTJ6HgppGQwN3pjC Yu1Ix6tzB2VkA4IVc4lgboxkpGKnYmjHOHVrBy87CmlejGQ0lcHbDMrIFgppbh42eDPcow8w pt28GJoYtC8cE7P4AWPazYmxwbuuafD2iyShZ8NcMGnlZmE28/lgCuUNgzwUNDXMAePUbRp4 0i99DXvggTh1bcu5kX8wd+wc3JVRnoehNGtsZ5gjxw5lNC00NbjnjozyHPtetQ8DnYH8g6YG 99yRkQqFpcCk1fUlUUZTQ1ODawxTKHc+pqXOQ2hqcMtdGfXfYvSp8xWaGlxyR0ZJ0m/S86RQ Pp8Zf216aGpwA9rq+XxTRnUdBGkqE9S6niuF8tvbHN4Se4OmBjegraLFDn/7O3hDInnlvxDH c1QDB3WugKmB4cpcYLj8OmfYRcrIHczU5wY6A+0Mhs93AWW0O2izmx7KaIfQZjc1lNEOgc2O KS+nhDLaJUx5OS2U0U5hysspoYx2C/zsKKRpoIx2DKMHTQVltGvSlM96CiijncPcsVNAGe0c Gr+ngDLaPUx5aQ9lRCgka2aQUXfHbFGE4eEQhlJpTFq5FigkOxzLqGmQ20DJqCyDAKerqiDA eJwplNcDhWSDYxlBEuhx5Lc4VgLJMmQ0Pxzk5IiA1/+bpR/LHqGQnmeGQZ0WiZKNfNYVj/6G MloODLIppGegjEgPJmF+hklllGXdzBNKDI/L6Hj8+0P8Cz2R54VCMgExSgS01eNxxt6IcyM/ oJAeZdZBHS11vkAhPcasMuK6kT+kaZ4vfQ3+QC8GMgj3xz4CZURGwP7Ypa/BFygjMgojqZtC GZEbxDFjqZtAGZEbNA0TjJlAGZGbMFixCZQRuQNjrN6HMiJ3wPI407nchjIid0E6l6WvYd1Q RsQAmr5vQxkRI5jw8haUETGCMb9vQRkRQxiqeBzKiBgTxwwMOQxlRIyh1/cYlBF5AM6QhqGM yEPQOWiIVcro7a2q6Fm8VriG1AdtFS12+NsFZXQ+s6rWDJO5dEFbPZ9XKCNKaN3Ay46mhi6r HNRRRmunrrkPqQtlRJ4C7qoUkoIyIk9CrwYNZUSehgmYFZQRsYB5YwXKiFhAm51AGREr6B4E KCNiSVkiO8i+oYyINVm297D5lBGxhhsoKCMyAXsPm08ZkUnYtzMxZUQmIor2u7mFMiIT0TT7 NX1TRmQy9utlN4OM2lbnfs0y5H4NAsn2ytyvW2OvsYMcy6hpiiKKlIyKQjKRo/vHjn5mIt8a MH3vcWDnWEaQBHoc9ZvafCzZyQ8HOXk3Vzll5Df79GmYYVDXF4k6bRh2P1eCkuMpI59J0/35 NCwgo6ZJ0yCo63EZHY9/f4h/2ff6uH/sIx9SXav2ibZ6PE4ooyw7dFB9SlcubYtj0hSPmb3R VtlfPqRZeyO8p5JEv6k4N9oqe/NpmFVGWdZ/S9FSt132tS92VhnFcXfQx3WjLdM0eGmm6T7E RC8G4pC63oeYKCPinO2LiTIiMyFiimPsla2qbfk6UEZkVtq2qvI8y+I4DOMYke6qyv9VJsqI LAjcw0RSWebzkI8yIqugLOEgliR57mPfRBmRFdE0eZ4k/vVNlBFZIdI3wRzhx8Z0yoisFuxW S1Mf5EQZkdXTl9MaBUUZEW+AnLIsSeI4iiAqyAqzqOVXoSgj4i3Ys4ZZFJZ1l92VRhmRDYCo rUu2GsqIbAQs4y41uKOMyGYoS0T4WOLMlBHZEAjdtkTrWaWM3t7WadYkPjD34A5tFS12+NsF ZXQ+720vP5kSDO7mcyZCWz2fVygjSojY0bZYU5qvT1rloI4yIvZgcIc9t0Xhfk2JMiKbBpsE k0R23brzdqCMyC5oGvg7oH8SFyJsZJ/OjEUZkZ2BbexoY8o7D8JSKRqehTIixBrKiBBrKCNC rKGMCLGGMiLEGsqIEGsoI0KsoYwIsWaFMvr6en93Ua4bzyo3pTaNC7eVtnURkdRNqX7V1/v7 19fwN4vJ6PPzdHJRrptE825KdROXzU0/72r04FN9nU6fn8PfUEYLlkoZ+VVfM8iobQ+H/u9B II1kOGklZUQZAZ/qy7GMEKAvivoyyjKVgXw4hTJlRBkBn+rLsYzwiNHj6E/goK5kpP7XSZZB mh6PpuU/0tz+/n3kqpctdbxarjFvGI+8oNyUutX6Oh67HUEXB5nIYSdJEiWf7udKUIAyooyA T/U1q4wwfINx9LaMPj7+/Dn+cPrl/T0b5Xx+e8sMOR5Nj3x7k6Aqy5X68vL6anrs6WR65Ovr y8uypW6nvt7fVftEW/3z5+NjMhlh1qNRWtZywem1aMZk1LYfH+kPn798fVWErIyvL9U+0VY/ PsbW+Rz0RnHcldnY3IiQ7eBkbvRb9E1LHSHbYQYZDa8bEbIdOMwixBrKiBBrKCNCrKGMCLFm ERlNbXTIsjA8HIJAlZdlQTBV6UWh18OmKVXKUWlFigLXHoZ2oQjhN4InmiSy00aecBDY+MGl qVrlu6wvm9L7peJJqIyuNq1i3DHapt5Qiv7tVr0tIqNpTeBFIY8MyaPghIQbxIa4JLEvva6x CqYe2xSlIhEwyklTXGtZyrVXVRDYJBlRabNQujzhJMFGu2flWdfatfi6vp4t/bLUOIa3C8SE pvlsq7jtGP1svbVtWeLVpEu8VW+LyGjaBdmqUpUp5SF68zSlI/FHWUopU5UaBN2dmapUyYzw fKnKmwTNR36XJ/xsqUXRbfCX9fVs6f1SdQ4I+ezZVnHbMfrZemuafqm3620BGY25B9mDwV23 TNvS4xjJ5PtNx65UlNYdZOiy7OSpHbRQvbosm1KHmjU+syv9+umhlKaxaxVjjtF29da/01v1 thkZNQ0GB3hjTCUj8Q2cXkaoCAxAMciYSkbIo4ABUZZhiOGTjDAkz3PbVjHmGD2ljG7V2yZk hOaD25SgG9M0+LKUWcb0MlLld/0N7Rp80xwOavLff7OvW0Z1HYYqq/hUMuo7Rk8pI/lpuN42 MDfC+ydJdNyaaWYxfT/2qpqmVG1RwiR1qrkRrknLqDvPsCl1+rlRv1SUhn7o8ptnnvC4Y7RN vXX7uNv1tgFLnbJOKaa01OlHOVWpcawGB6iIqSx1YuwXE7I8YTtLHZjeUtcvNYq6IrJrFeMe nTb11i31dr1tYN3o8v0z7bpRd7gwRamoUFlzcbtupM7yfJm6wV+vGz1fereP6/b2dq1iXEY2 9dYt9Xa90YuBEGsoI0KsoYwIsYYyIsQayogQaygjQqyhjAixhjJaiP5qF1Y5nnUyuhWj1Gzt vqrur6l0/UTIJZTRonSbuUljvgar/re+NZGmiYDhDrXMM/IBymhR7L0K89w+04NZPwinmrme i29QRoty7deM/bbYPAD3FXE/EXeTMQecKBJfPPVX8P3SQaGlfF1m3/tZbXNUx8tm/DCUo+oa e0rhYCS9ZJoyW+8YlNGiXMtINoiJh7n6H9+KT7FyjdRgY4T8j+aOv8b/cOfXEdRVmfi0uyOr 77GNM0Gy6hwiWXi6iZDznMO6MSijRRmWUfcb+R/e1HKUOOpf//3l/ppLGUmPMrxdW34PQ+UD XZY4G7zGu8M4xmAfhw9mUUxlhH/7XuyXf39PRt2jhmXUtxzKfuIwDALlNU4ZjcMHsyimMqpr vTGvaw6wk5EK16J6I7XjqWkgHLEcYmex9H+U0Th8MItiKiO1bUwFeNLIbOcxGWG4JnMgJSMI CNsfRT5i1lDmDAkUw7nRLSijRTGXkVjqrrefKUvdIzJSW87KUoQBWx12oo5Z6kSotNSNQxl5 zhTrRmZw3WgcyshzbnsxTEdZMsnbOJSR90hsOtfQp+4WlBEh1lBGhFjz/zxZBG8Q/BSEAAAA PHRFWHRDb21tZW50ACBJbWFnZSBnZW5lcmF0ZWQgYnkgRVNQIEdob3N0c2NyaXB0IChkZXZp Y2U9cG5tcmF3KQqV01S1AAAAInRFWHRwczpIaVJlc0JvdW5kaW5nQm94ADQxOXg0MjArNzIr MTg0PNQ4sQAAABx0RVh0cHM6TGV2ZWwAQWRvYmUtMy4wIEVQU0YtMy4wCptwu+MAAAAASUVO RK5CYII= ---------72c38c1b72c38c1b Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ---------72c38c1b72c38c1b-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Sun, 27 Jan 2008 17:29:33 -0700 Message-ID: <20080127172933828.00000002440@djm-pc> References: Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1588774336==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , Keir Fraser Cc: "akira.ijuin@oracle.com" , "xen-devel@lists.xensource.com" , "deepak.patel@oracle.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format --===============1588774336== Content-Type: multipart/alternative; boundary=-------5775352157753521 This is a multi-part message in MIME format ---------5775352157753521 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable RE: [Xen-devel] [PATCH] Add a timer mode that disables pending missed ticks= Thanks, I hadn't realized that! No wonder we didn't see the same improveme= nt you saw! > Try specifying clock=3Dpit on the linux boot line... I'm confused... do you mean "clocksource=3Dpit" on the Xen command line or "nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the guest (or dom0?) comm= and line? Or both places? Since the tests take awhile, it would be nice to get this right the first time. Do the Xen and guest clocksources need to be the same? Thanks, Dan = -----Original Message----- From: Dave Winchell [mailto:dwinchell@virtualiron.com] Sent: Sunday, January 27, 2008 2:22 PM To: dan.magenheimer@oracle.com; Keir Fraser Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com; akira.ijuin@ora= cle.com; Dave Winchell Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending mis= sed ticks Hi Dan, Hpet timer does have a fairly large error, as I was trying this one recen= tly. I don't remember what I got for error, but 1% sounds about right. The problem is that hpet is not built on top of vpt.c, the module Keir an= d I did all the recent work in, for its periodic timer needs. Try specifying cloc= k=3Dpit on the linux boot line. If it still picks the hpet, which it might, let m= e know and I'll tell you how to get around this. Regards, Dave ---------------------------------------------------------------------------= --- From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] Sent: Fri 1/25/2008 6:50 PM To: Dave Winchell; Keir Fraser Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com; akira.ijuin@o= racle.com Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending m= issed ticks Sorry for the very late followup on this but we finally were able to get our testing set up again on stable 3.1 bits and have seen some very bad results on 3.1.3-rc1, on the order of 1%. Test enviroment was a 4-socket dual core machine with 24GB of memory running six two-vcpu 2GB domains, four hvm plus two pv. All six guests were running LTP simultaneously. The four hvm guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests. All four hvm guests experienced skew around -1%, even the 32-bit guest. Less intensive testing didn't exhibit much skew at all. A representative graph is attached. Dave, I wonder if some portion of your patches didn't end up in the xen trees? (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer 14.318MHz HPET.) Thanks, Dan P.S. Many thanks to Deepak and Akira for running tests. > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > Dave Winchell > Sent: Wednesday, January 09, 2008 9:53 AM > To: Keir Fraser > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave > Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > disables pending > missed ticks > > > Hi Keir, > > The latest change, c/s 16690, looks fine. > I agree that the code in c/s 16690 is equivalent to > the code I submitted. Also, your version is more > concise. > > The error tests confirm the equivalence. With overnight cpu loads, > the checked in version was accurate to +.048% for sles > and +.038% for red hat. My version was +.046% and +.032% in a > 2 hour test. > I don't think the difference is significant. > > i/o loads produced errors of +.01%. > > Thanks for all your efforts on this issue. > > Regards, > Dave > > > > Keir Fraser wrote: > > >Applied as c/s 16690, although the checked-in patch is > smaller. I think the > >only important fix is to pt_intr_post() and the only bit of > the patch I > >totally omitted was the change to pt_process_missed_ticks(). > I don't think > >that change can be important, but let's see what happens to the error > >percentage... > > > > -- Keir > > > >On 4/1/08 23:24, "Dave Winchell" wrote: > > > > > > > >>Hi Dan and Keir, > >> > >>Attached is a patch that fixes some issues with the SYNC policy > >>(no_missed_ticks_pending). > >>I have not tried to make the change the minimal one, but, > rather, just > >>ported into > >>the new code what I know to work well. The error for > >>no_missed_ticks_pending goes from > >>over 3% to .03% with this change according to my testing. > >> > >>Regards, > >>Dave > >> > >>Dan Magenheimer wrote: > >> > >> > >> > >>>Hi Dave -- > >>> > >>>Did you get your correction ported? If so, it would be > nice to see this get > >>>into 3.1.3. > >>> > >>>Note that I just did some very limited testing with > timer_mode=3D2(=3DSYNC=3Dno > >>>missed ticks pending) > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the > worst error I've > >>>seen so far > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. > >>> > >>>Thanks, > >>>Dan > >>> > >>> > >>> > >>> > >>> > >>>>-----Original Message----- > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>To: dan.magenheimer@oracle.com > >>>>Cc: Keir Fraser; Shan, Haitao; > xen-devel@lists.xensource.com; Dong, > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>disables pending > >>>>missed ticks > >>>> > >>>> > >>>>Dan, > >>>> > >>>>I did some testing with the constant tsc offset SYNC method > >>>>(now called > >>>>no_missed_ticks_pending) > >>>>and found the error to be very high, much larger than 1 %, as > >>>>I recall. > >>>>I have not had a chance to submit a correction. I will try to > >>>>do it later > >>>>this week or the first week in January. My version of constant tsc > >>>>offset SYNC method > >>>>produces .02 % error, so I just need to port that into the > >>>>current code. > >>>> > >>>>The error you got for both of those kernels is what I would expect > >>>>for the default mode, delay_for_missed_ticks. > >>>> > >>>>I'll let Keir answer on how to set the time mode. > >>>> > >>>>Regards, > >>>>Dave > >>>> > >>>>Dan Magenheimer wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>>Anyone make measurements on the final patch? > >>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>about 0.2% with no load. This was xen-unstable tip today > >>>>with no options specified. 32-bit was about 0.01%. > >>>> > >>>> > >>>> > >>>> > >>>>>I think I missed something... how do I run the various > >>>>> > >>>>> > >>>>> > >>>>> > >>>>accounting choices and which ones are known to be appropriate > >>>>for which kernels? > >>>> > >>>> > >>>> > >>>> > >>>>>Thanks, > >>>>>Dan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>-----Original Message----- > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>Keir Fraser > >>>> > >>>> > >>>> > >>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>>>To: Dave Winchell > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, > Eddie; Jiang, > >>>>>>Yunhong > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>>>disables pending > >>>>>>missed ticks > >>>>>> > >>>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. > >>>>>> > >>>>>>-- Keir > >>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>> wrote: > >>>> > >>>> > >>>> > >>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>Keir, > >>>>>>> > >>>>>>>The accuracy data I've collected for i/o loads for the > >>>>>>>various time protocols follows. In addition, the data > >>>>>>>for cpu loads is shown. > >>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.) > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 of=3D/dev/null. > >>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>>>>All three guests are 8vcpu. > >>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > >>>>>>>except that the redhat-64 guest has 4 instances of dd. > >>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, > +.005% cpu > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, > +.012% cpu > >>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu > >>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, > -.031% cpu > >>>>>>> > >>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, > -.09% i/o-8 > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% > -.14% i/o-8 > >>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, > -.022% i/o-8 > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 > >>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, > -.029% i/o-8 > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, > -.031% i/o-8 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, > -.005% i/o-32 > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 > >>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, > .003% i/o-4/32 > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, > .01% i/o-4/32 > >>>>>>> > >>>>>>> > >>>>>>>Overhead measurements: > >>>>>>> > >>>>>>>Progress in terms of number of passes through a fixed > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>system workload > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>>>>>The workload was usex -b48. > >>>>>>> > >>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>>>> > >>>>>>> > >>>>>>>Conclusions: > >>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy > requirement for ntp > >>>>>>>tracking under the loads > >>>>>>>above is the SYNC protocol. The worst case accuracies for > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>SYNC, MIXED, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>and ASYNC > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>>>> > >>>>>>>We could reduce the cost of the SYNC method by only > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>scheduling the extra > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>wakeups if a certain number > >>>>>>>of ticks are missed. > >>>>>>> > >>>>>>>Regards, > >>>>>>>Dave > >>>>>>> > >>>>>>>Keir Fraser wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>couple of days ago, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have > been something > >>>>>>>>>wrong with the code I used a couple of days ago for > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>ASYNC. It may have been > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>missing the immediate delivery of interrupt after context > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>switch in. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>acceptable accuracy, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>each running consistently around or under .01%. MIXED has > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>a fairly high > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>error of > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>threshold for comfort. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>SYNC running > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>running instead. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>It may be too early to pick the protocol and I can run > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>more overnight tests > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>next week. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side effects of the > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>SYNC+run_timer > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will cause higher > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>system-wide CPU > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>contention. I find it easier to think through the > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>implications of ASYNC. I'm > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>surprised that MIXED loses time, and is less accurate than > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>ASYNC. Perhaps it > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>delivers more timer interrupts than the other approaches, > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>and each interrupt > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>event causes a small accumulated error? > >>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>if the latter is > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>actually more accurate then I can simply revert the > changeset that > >>>>>>>>implemented MIXED. > >>>>>>>> > >>>>>>>>Perhaps rather than running more of the same workloads you > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>could try idle > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>to /dev/null)? We > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>don't have any data on workloads that aren't CPU bound, so > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>that's really an > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>>>>> > >>>>>>>>-- Keir > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>_______________________________________________ > >>>>>>Xen-devel mailing list > >>>>>>Xen-devel@lists.xensource.com > >>>>>>http://lists.xensource.com/xen-devel > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> > >>> > >>> > >>> > >>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru > >> > >> missed_ticks =3D missed_ticks / (s_time_t) pt->period + 1; > >> if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>+ pt->do_not_freeze =3D 1; > >> else > >> pt->pending_intr_nr +=3D missed_ticks; > >> pt->scheduled +=3D missed_ticks * pt->period; > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > >> > >> pt_lock(pt); > >> > >>- pt->pending_intr_nr++; > >>+ if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { > >>+ pt->pending_intr_nr =3D 1; > >>+ pt->do_not_freeze =3D 0; > >>+ } > >>+ else > >>+ pt->pending_intr_nr++; > >> > >> if ( !pt->one_shot ) > >> { > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct > >> return; > >> } > >> > >>- pt->do_not_freeze =3D 0; > >>- > >> if ( pt->one_shot ) > >> { > >> pt->enabled =3D 0; > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct > >> pt->last_plt_gtime =3D hvm_get_guest_time(v); > >> pt->pending_intr_nr =3D 0; /* 'collapse' all > missed ticks */ > >> } > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { > >>+ pt->pending_intr_nr--; > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>+ } > >> else > >> { > >> pt->last_plt_gtime +=3D pt->period_cycles; > >> > >> > > > > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > ---------5775352157753521 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable RE: [Xen-devel] [PATCH] Add a timer mode that = disables pending missed ticks
Thanks, I hadn't realized that!  No wonder we didn't see the = same = improvement you saw!
 
> = Try specifying clock=3Dpit on the linux boot line...
 
I'm = confused... do you mean "clocksource=3Dpit" on the Xen command line = or
"nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the guest (or do= m0?) = command
line?  Or both places?  Since the tests take awhile, it = would = be nice
to get = this right the first time.  Do the Xen and guest clocksources = need
to be = the same?
 
Thanks,
Dan
 
 -----Original Message-----
Fr= om: = Dave Winchell [mailto:dwinchell@virtualiron.com]
Sent: Sunday, Ja= nuary = 27, 2008 2:22 PM
To: dan.magenheimer@oracle.com; Keir = Fraser
Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com= ; = akira.ijuin@oracle.com; Dave Winchell
Subject: RE: [Xen-devel] [P= ATCH] = Add a timer mode that disables pending missed ticks

Hi Dan,=
 
Hpet timer does have a fairly = large = error, as I was trying this one recently.
I don't remember what I got fo= r error, = but 1% sounds about right.
The problem is that hpet = is not = built on top of vpt.c, the module Keir and I did
all the recent work in, f= or its = periodic timer needs. Try specifying = clock=3Dpit
on the linux boot line. If it = still picks = the hpet, which it might, let me = know
and I'll tell you how to get a= round = this.
 
Regards,
Dave
 
 
 

From: Dan Magenheimer = [mailto:dan.magenheimer@oracle.com]
Sent: Fri 1/25/2008 6:50 = PM
To: Dave Winchell; Keir Fraser
Cc: = xen-devel@lists.xensource.com; deepak.patel@oracle.com; = akira.ijuin@oracle.com
Subject: RE: [Xen-devel] [PATCH] Add a t= imer = mode that disables pending missed ticks

Sorry for the very late followup on this but we finally= were = able
to get our testing set up again on stable 3.1 bits and have
se= en = some very bad results on 3.1.3-rc1, on the order of 1%.

Test envir= oment = was a 4-socket dual core machine with 24GB of
memory running six two-v= cpu = 2GB domains, four hvm plus two pv.
All six guests were running LTP = simultaneously.  The four hvm
guests were: RHEL5u1-64, RHEL4u5-32= , = RHEL5-64, and RHEL4u5-64.
Timer_mode was set to 2 for 64-bit guests an= d 0 = for 32-bit guests.
All four hvm guests experienced skew around -1%, ev= en = the 32-bit
guest.  Less intensive testing didn't exhibit much ske= w at = all.

A representative graph is attached.

Dave, I wonder if = some = portion of your patches didn't end up in
the xen trees?

(xm dme= sg = shows 8x Xeon 3.2GHz stepping 04, Platform timer
14.318MHz = HPET.)

Thanks,
Dan

P.S. Many thanks to Deepak and Akira = for = running tests.

> -----Original Message-----
> From: = xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bo= unces@lists.xensource.com]On = Behalf Of
> Dave Winchell
> Sent: Wednesday, January 09, 2008= 9:53 = AM
> To: Keir Fraser
> Cc: dan.magenheimer@oracle.com; = xen-devel@lists.xensource.com; Dave
> Winchell
> Subject: Re:= = [Xen-devel] [PATCH] Add a timer mode that
> disables pending
>= ; = missed ticks
>
>
> Hi Keir,
>
> The latest = = change, c/s 16690, looks fine.
> I agree that the code in c/s 16690= is = equivalent to
> the code I submitted. Also, your version is more> = concise.
>
> The error tests confirm the equivalence. With = overnight cpu loads,
> the checked in version was accurate to +.048= % for = sles
> and +.038% for red hat. My version was +.046% and +.032% in = = a
> 2 hour test.
> I don't think the difference is = significant.
>
> i/o loads produced errors of = +.01%.
>
> Thanks for all your efforts on this = issue.
>
> Regards,
> Dave
>
>
>
&= gt; = Keir Fraser wrote:
>
> >Applied as c/s 16690, although the= = checked-in patch is
> smaller. I think the
> >only importa= nt = fix is to pt_intr_post() and the only bit of
> the patch I
> = = >totally omitted was the change to pt_process_missed_ticks().
> = I = don't think
> >that change can be important, but let's see what = = happens to the error
> >percentage...
> >
> > = -- = Keir
> >
> >On 4/1/08 23:24, "Dave Winchell" = <dwinchell@virtualiron.com> wrote:
> >
> >
>= ; = >
> >>Hi Dan and Keir,
> >>
> = >>Attached is a patch that fixes some issues with the SYNC = policy
> >>(no_missed_ticks_pending).
> >>I have = not = tried to make the change the minimal one, but,
> rather, just
&g= t; = >>ported into
> >>the new code what I know to work well= . The = error for
> >>no_missed_ticks_pending goes from
> = >>over 3% to .03% with this change according to my testing.
>= = >>
> >>Regards,
> >>Dave
> = >>
> >>Dan Magenheimer wrote:
> >>
> = = >>
> >>
> >>>Hi Dave --
> = >>>
> >>>Did you get your correction ported? = ; If = so, it would be
> nice to see this get
> >>>into = 3.1.3.
> >>>
> >>>Note that I just did some= very = limited testing with
> timer_mode=3D2(=3DSYNC=3Dno
> >>= >missed = ticks pending)
> >>>on tip of xen-3.1-testing (64-bit Linu= x hv = guest) and the
> worst error I've
> >>>seen so = far
> >>>is 0.012%.  But I haven't tried any exotic l= oads, = just LTP.
> >>>
> >>>Thanks,
> = >>>Dan
> >>>
> >>>
> = >>>
> >>>
> >>>
> = >>>>-----Original Message-----
> >>>>From: = Dave = Winchell [mailto:dwinchell@virtualiron.co= m]
> = >>>>Sent: Wednesday, December 19, 2007 12:33 PM
> = >>>>To: dan.magenheimer@oracle.com
> >>>>Cc= : = Keir Fraser; Shan, Haitao;
> xen-devel@lists.xensource.com; = Dong,
> >>>>Eddie; Jiang, Yunhong; Dave Winchell
>= ; = >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that> = >>>>disables pending
> >>>>missed ticks
= > = >>>>
> >>>>
> >>>>Dan,> = >>>>
> >>>>I did some testing with the cons= tant = tsc offset SYNC method
> >>>>(now called
> = >>>>no_missed_ticks_pending)
> >>>>and foun= d the = error to be very high, much larger than 1 %, as
> >>>>I= = recall.
> >>>>I have not had a chance to submit a = correction. I will try to
> >>>>do it later
> = >>>>this week or the first week in January. My version of con= stant = tsc
> >>>>offset SYNC method
> = >>>>produces .02 % error, so I just need to port that into = the
> >>>>current code.
> >>>>
>= ; = >>>>The error you got for both of those kernels is what I wou= ld = expect
> >>>>for the default mode, = delay_for_missed_ticks.
> >>>>
> >>>>= I'll = let Keir answer on how to set the time mode.
> >>>>
= > = >>>>Regards,
> >>>>Dave
> = >>>>
> >>>>Dan Magenheimer wrote:
> = >>>>
> >>>>
> >>>>
>= ; = >>>>
> >>>>
> >>>>>Any= one = make measurements on the final patch?
> >>>>>
>= ; = >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss= = of
> >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>about 0.2% with no load.  This was xen-unstable tip = = today
> >>>>with no options specified.  32-bit was= = about 0.01%.
> >>>>
> >>>>
> = >>>>
> >>>>
> >>>>>I t= hink = I missed something... how do I run the various
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>accounting choices and which ones are known to be = appropriate
> >>>>for which kernels?
> = >>>>
> >>>>
> >>>>
>= ; = >>>>
> >>>>>Thanks,
> = >>>>>Dan
> >>>>>
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>>>-----Original Message-----
> = >>>>>>From: xen-devel-bounces@lists.xensource.com
&g= t; = >>>>>>[mailto:xen-devel-bo= unces@lists.xensource.com]On = Behalf Of
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>Keir Fraser
> = >>>>
> >>>>
> >>>>
>= ; = >>>>
> >>>>>>Sent: Thursday, December= 06, = 2007 4:57 AM
> >>>>>>To: Dave Winchell
> = >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; = = Dong,
> Eddie; Jiang,
> >>>>>>Yunhong
&g= t; = >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode= = that
> >>>>>>disables pending
> = >>>>>>missed ticks
> >>>>>>
= > = >>>>>>
> >>>>>>Please take a lo= ok at = xen-unstable changeset 16545.
> >>>>>>
> = >>>>>>-- Keir
> >>>>>>
> = = >>>>>>On 26/11/07 20:57, "Dave Winchell"
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>><dwinchell@virtualiron.com> wrote:
> = >>>>
> >>>>
> >>>>
>= ; = >>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>Keir,
>= ; = >>>>>>>
> >>>>>>>The accu= racy = data I've collected for i/o loads for the
> = >>>>>>>various time protocols follows. In addition, = the = data
> >>>>>>>for cpu loads is shown.
> = = >>>>>>>
> >>>>>>>The load= s = labeled cpu and i/o-8 are on an 8 processor AMD box.
> = >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu = each.
> >>>>>>>The cpu load is usex -e36 on ea= ch = guest.
> >>>>>>>(usex is available at http://people.redhat.com/= anderson/usex.)
> = >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 = = of=3D/dev/null.
> >>>>>>>
> = >>>>>>>The loads labeled i/o-32 are 32 instances of = = dd.
> >>>>>>>Also, these are run on 4 cpu AMD = = box.
> >>>>>>>In addition, there is an idle = rh-32bit guest.
> >>>>>>>All three guests are = = 8vcpu.
> >>>>>>>
> = >>>>>>>The loads labeled i/o-4/32 are the same as = i/o-32
> >>>>>>>except that the redhat-64 gues= t has = 4 instances of dd.
> >>>>>>>
> = >>>>>>>Date Duration Protocol sles, rhat error = load
> >>>>>>>
> = >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 se= c = -.006%,
> +.005% cpu
> >>>>>>>11/09 3 hr= s 19 = min ASYNC -.13 sec, +1.44 sec, -.001%,
> +.012% cpu
> = >>>>>>>
> >>>>>>>11/08 2 = hrs = 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> = >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.= 005%, = -.005% cpu
> >>>>>>>11/12 65 hrs 40 min SYNC -= 18 = sec, -8 sec, -.008%, -.003% cpu
> >>>>>>>
&= gt; = >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%,= = -.040% cpu
> >>>>>>>11/08 15 hrs 39 min MIXED = -19. = sec,-17.4 sec, -.034%,
> -.031% cpu
> = >>>>>>>
> >>>>>>>
>= = >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec,= = -.01%,
> -.09% i/o-8
> >>>>>>>11/15 2 hr= s 44 = min ASYNC -1.47 sec,-14.0 sec, -.015%
> -.14% i/o-8
> = >>>>>>>
> >>>>>>>11/13 15= hrs = 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
> -.022% i/o-8
> = >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017= %, = -.018% i/o-8
> >>>>>>>
> = >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, = = -.020%,
> -.029% i/o-8
> >>>>>>>11/20 16= hrs = 2 min MIXED -13.4 sec,-18.1 sec, -.023%,
> -.031% i/o-8
> = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>>11/21 28= min = MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> = >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, = -.011%,
> -.005% i/o-32
> >>>>>>>11/21 4= 0 min = ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> = >>>>>>>
> >>>>>>>11/26 11= 3 hrs = 46 min MIXED -297. sec, 13. sec -.07%,
> .003% i/o-4/32
> = >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, = = -.017%,
> .01% i/o-4/32
> >>>>>>>
>= ; = >>>>>>>
> >>>>>>>Overhead= = measurements:
> >>>>>>>
> = >>>>>>>Progress in terms of number of passes through= a = fixed
> >>>>>>>
> = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>system = workload
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>on an 8 vcpu= red = hat with an 8 vcpu sles idle.
> >>>>>>>The wor= kload = was usex -b48.
> >>>>>>>
> = >>>>>>>
> >>>>>>>ASYNC 16= 7 min = 145 passes .868 passes/min
> >>>>>>>SYNC 167 m= in = 144 passes .862 passes/min
> >>>>>>>SYNC 1065 = min = 919 passes .863 passes/min
> >>>>>>>MIXED 221 = min = 196 passes .887 passes/min
> >>>>>>>
> = >>>>>>>
> = >>>>>>>Conclusions:
> = >>>>>>>
> >>>>>>>The only= = protocol which meets the .05% accuracy
> requirement for ntp
>= ; = >>>>>>>tracking under the loads
> = >>>>>>>above is the SYNC protocol. The worst case = accuracies for
> >>>>>>>
> = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>SYNC, = MIXED,
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>and ASYNC> = >>>>>>>are .022%, .12%, and .14%, respectively.
&= gt; = >>>>>>>
> >>>>>>>We could= = reduce the cost of the SYNC method by only
> = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>>
>= = >>>>>>scheduling the extra
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>wakeups if a certain number
> = >>>>>>>of ticks are missed.
> = >>>>>>>
> = >>>>>>>Regards,
> = >>>>>>>Dave
> >>>>>>>
= > = >>>>>>>Keir Fraser wrote:
> = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>>
>= = >>>>>>>
> >>>>>>>>On = 9/11/07 19:22, "Dave Winchell"
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>><dwinchell@virtualiron.com> wrote:
> = = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>>Since I had a high error (~.03%) for = the = ASYNC method a
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>coup= le of = days ago,
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>>I ra= n = another ASYNC test. I think there may have
> been something
>= = >>>>>>>>>wrong with the code I used a couple o= f = days ago for
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>ASYN= C. It = may have been
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>>miss= ing = the immediate delivery of interrupt after context
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>swit= ch = in.
> >>>>>>
> >>>>>>
= > = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>My results indicate that either SYNC = or = ASYNC give
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>acceptable accuracy,
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>each running consistently around or u= nder = .01%. MIXED has
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>a fa= irly = high
> >>>>>>
> >>>>>>> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>error of
> = >>>>>>>>>greater than .03%. Probably too close= to = .05% ntp
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>thre= shold = for comfort.
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>>I do= n't = have an overnight run with SYNC. I plan to leave
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>SYNC= = running
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>>over= the = weekend. If you'd rather I can leave MIXED
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>runn= ing = instead.
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>>It m= ay be = too early to pick the protocol and I can run
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>more= = overnight tests
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>>next= = week.
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>I'm a bit worried about any unwanted side= = effects of the
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>SYNC+run_timer
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>approach -- e.g., whether timer wakeups w= ill = cause higher
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>system-w= ide = CPU
> >>>>>>
> >>>>>>
= > = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>contention. I find it easier to think thr= ough = the
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>implicat= ions = of ASYNC. I'm
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>surprise= d = that MIXED loses time, and is less accurate than
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>ASYNC. = Perhaps it
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>delivers= more = timer interrupts than the other approaches,
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>and each= = interrupt
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>event ca= uses = a small accumulated error?
> >>>>>>>>
&g= t; = >>>>>>>>Overall I would consider MIXED and ASYNC = as = favourites and
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>if the l= atter = is
> >>>>>>
> >>>>>>
&= gt; = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>actually more accurate then I can simply = = revert the
> changeset that
> = >>>>>>>>implemented MIXED.
> = >>>>>>>>
> = >>>>>>>>Perhaps rather than running more of the s= ame = workloads you
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>could tr= y = idle
> >>>>>>
> >>>>>>> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated= = large disc reads
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>to = /dev/null)? We
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>don't ha= ve = any data on workloads that aren't CPU bound, so
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>that's r= eally = an
> >>>>>>
> >>>>>>
&= gt; = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>obvious place to put any further effort = imo.
> >>>>>>>>
> = >>>>>>>>-- Keir
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>_______________________________________________> = >>>>>>Xen-devel mailing list
> = >>>>>>Xen-devel@lists.xensource.com
> = >>>>>>http://lists.xensource.com/= xen-devel
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>
> >>>>
> >>>>
>= ; = >>>>
> >>>
> >>>
> = >>>
> >>>
> >>diff -r cfdbdca5b831 = xen/arch/x86/hvm/vpt.c
> >>--- a/xen/arch/x86/hvm/vpt.c Thu D= ec 06 = 15:36:07 2007 +0000
> >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan = 04 = 17:58:16 2008 -0500
> >>@@ -58,7 +58,7 @@ static void = pt_process_missed_ticks(stru
> >>
> = >>     missed_ticks =3D missed_ticks / (s_time_= t) = pt->period + 1;
> >>     if ( = mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
> = >>-        pt->do_not_freeze = =3D = !pt->pending_intr_nr;
> = >>+        pt->do_not_freeze = =3D = 1;
> >>     else
> = >>         = pt->pending_intr_nr +=3D missed_ticks;
> = >>     pt->scheduled +=3D missed_ticks * = pt->period;
> >>@@ -127,7 +127,12 @@ static void = pt_timer_fn(void *data)
> >>
> = >>     pt_lock(pt);
> >>
> = >>-    pt->pending_intr_nr++;
> = >>+    if ( mode_is(pt->vcpu->domain, = no_missed_ticks_pending) ) {
> = >>+        pt->pending_intr_n= r =3D = 1;
> >>+ pt->do_not_freeze =3D 0;
> = >>+    }
> >>+    else> = >>+ pt->pending_intr_nr++;
> >>
> = >>     if ( !pt->one_shot )
> = >>     {
> >>@@ -221,8 +226,6 @@ vo= id = pt_intr_post(struct vcpu *v, struct
> = >>         return;
> = = >>     }
> >>
> = >>-    pt->do_not_freeze =3D 0;
> = >>-
> >>     if ( pt->one_shot = )
> >>     {
> = >>         pt->enabled =3D= = 0;
> >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v,= = struct
> = >>           = ;  = pt->last_plt_gtime =3D hvm_get_guest_time(v);
> = >>           = ;  = pt->pending_intr_nr =3D 0; /* 'collapse' all
> missed ticks */> = >>         }
> >&g= t;+ = else if ( mode_is(v->domain, no_missed_ticks_pending) ) {
> = >>+     pt->pending_intr_nr--;
> = >>+     pt->last_plt_gtime =3D = hvm_get_guest_time(v);
> >>+ }
> = >>         else
> = >>         {
> = >>           = ;  = pt->last_plt_gtime +=3D pt->period_cycles;
> >>
>= = >>
> >
> >
> >
> = >
>
>
> = _______________________________________________
> Xen-devel mailing= = list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/= xen-devel
>

---------5775352157753521-- --===============1588774336== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============1588774336==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dave Winchell" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Sun, 27 Jan 2008 16:21:50 -0500 Message-ID: References: <20080125165036078.00000002440@djm-pc> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0992105506==" Return-path: Content-class: urn:content-classes:message List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Mime-version: 1.0 Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: dan.magenheimer@oracle.com, Keir Fraser Cc: akira.ijuin@oracle.com, Dave Winchell , xen-devel@lists.xensource.com, deepak.patel@oracle.com List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============0992105506== Content-class: urn:content-classes:message Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C8612A.A1A3A9FC" This is a multi-part message in MIME format. ------_=_NextPart_001_01C8612A.A1A3A9FC Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Dan, =20 Hpet timer does have a fairly large error, as I was trying this one = recently. I don't remember what I got for error, but 1% sounds about right. The problem is that hpet is not built on top of vpt.c, the module Keir = and I did all the recent work in, for its periodic timer needs. Try specifying = clock=3Dpit on the linux boot line. If it still picks the hpet, which it might, let = me know and I'll tell you how to get around this. =20 Regards, Dave =20 =20 =20 ________________________________ From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] Sent: Fri 1/25/2008 6:50 PM To: Dave Winchell; Keir Fraser Cc: xen-devel@lists.xensource.com; deepak.patel@oracle.com; = akira.ijuin@oracle.com Subject: RE: [Xen-devel] [PATCH] Add a timer mode that disables pending = missed ticks Sorry for the very late followup on this but we finally were able to get our testing set up again on stable 3.1 bits and have seen some very bad results on 3.1.3-rc1, on the order of 1%. Test enviroment was a 4-socket dual core machine with 24GB of memory running six two-vcpu 2GB domains, four hvm plus two pv. All six guests were running LTP simultaneously. The four hvm guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests. All four hvm guests experienced skew around -1%, even the 32-bit guest. Less intensive testing didn't exhibit much skew at all. A representative graph is attached. Dave, I wonder if some portion of your patches didn't end up in the xen trees? (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer 14.318MHz HPET.) Thanks, Dan P.S. Many thanks to Deepak and Akira for running tests. > -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > Dave Winchell > Sent: Wednesday, January 09, 2008 9:53 AM > To: Keir Fraser > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave > Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > disables pending > missed ticks > > > Hi Keir, > > The latest change, c/s 16690, looks fine. > I agree that the code in c/s 16690 is equivalent to > the code I submitted. Also, your version is more > concise. > > The error tests confirm the equivalence. With overnight cpu loads, > the checked in version was accurate to +.048% for sles > and +.038% for red hat. My version was +.046% and +.032% in a > 2 hour test. > I don't think the difference is significant. > > i/o loads produced errors of +.01%. > > Thanks for all your efforts on this issue. > > Regards, > Dave > > > > Keir Fraser wrote: > > >Applied as c/s 16690, although the checked-in patch is > smaller. I think the > >only important fix is to pt_intr_post() and the only bit of > the patch I > >totally omitted was the change to pt_process_missed_ticks(). > I don't think > >that change can be important, but let's see what happens to the error > >percentage... > > > > -- Keir > > > >On 4/1/08 23:24, "Dave Winchell" wrote: > > > > > > > >>Hi Dan and Keir, > >> > >>Attached is a patch that fixes some issues with the SYNC policy > >>(no_missed_ticks_pending). > >>I have not tried to make the change the minimal one, but, > rather, just > >>ported into > >>the new code what I know to work well. The error for > >>no_missed_ticks_pending goes from > >>over 3% to .03% with this change according to my testing. > >> > >>Regards, > >>Dave > >> > >>Dan Magenheimer wrote: > >> > >> > >> > >>>Hi Dave -- > >>> > >>>Did you get your correction ported? If so, it would be > nice to see this get > >>>into 3.1.3. > >>> > >>>Note that I just did some very limited testing with > timer_mode=3D2(=3DSYNC=3Dno > >>>missed ticks pending) > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the > worst error I've > >>>seen so far > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. > >>> > >>>Thanks, > >>>Dan > >>> > >>> > >>> > >>> > >>> > >>>>-----Original Message----- > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>To: dan.magenheimer@oracle.com > >>>>Cc: Keir Fraser; Shan, Haitao; > xen-devel@lists.xensource.com; Dong, > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>disables pending > >>>>missed ticks > >>>> > >>>> > >>>>Dan, > >>>> > >>>>I did some testing with the constant tsc offset SYNC method > >>>>(now called > >>>>no_missed_ticks_pending) > >>>>and found the error to be very high, much larger than 1 %, as > >>>>I recall. > >>>>I have not had a chance to submit a correction. I will try to > >>>>do it later > >>>>this week or the first week in January. My version of constant tsc > >>>>offset SYNC method > >>>>produces .02 % error, so I just need to port that into the > >>>>current code. > >>>> > >>>>The error you got for both of those kernels is what I would expect > >>>>for the default mode, delay_for_missed_ticks. > >>>> > >>>>I'll let Keir answer on how to set the time mode. > >>>> > >>>>Regards, > >>>>Dave > >>>> > >>>>Dan Magenheimer wrote: > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>>Anyone make measurements on the final patch? > >>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>about 0.2% with no load. This was xen-unstable tip today > >>>>with no options specified. 32-bit was about 0.01%. > >>>> > >>>> > >>>> > >>>> > >>>>>I think I missed something... how do I run the various > >>>>> > >>>>> > >>>>> > >>>>> > >>>>accounting choices and which ones are known to be appropriate > >>>>for which kernels? > >>>> > >>>> > >>>> > >>>> > >>>>>Thanks, > >>>>>Dan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>-----Original Message----- > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>Keir Fraser > >>>> > >>>> > >>>> > >>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>>>To: Dave Winchell > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, > Eddie; Jiang, > >>>>>>Yunhong > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>>>disables pending > >>>>>>missed ticks > >>>>>> > >>>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. > >>>>>> > >>>>>>-- Keir > >>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>> wrote: > >>>> > >>>> > >>>> > >>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>Keir, > >>>>>>> > >>>>>>>The accuracy data I've collected for i/o loads for the > >>>>>>>various time protocols follows. In addition, the data > >>>>>>>for cpu loads is shown. > >>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD box. > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>>>>(usex is available at http://people.redhat.com/anderson/usex.) > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 of=3D/dev/null. > >>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>>>>All three guests are 8vcpu. > >>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > >>>>>>>except that the redhat-64 guest has 4 instances of dd. > >>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, > +.005% cpu > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, > +.012% cpu > >>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu > >>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, > -.031% cpu > >>>>>>> > >>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, > -.09% i/o-8 > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% > -.14% i/o-8 > >>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, > -.022% i/o-8 > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 > >>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, > -.029% i/o-8 > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, > -.031% i/o-8 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, > -.005% i/o-32 > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 > >>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, > .003% i/o-4/32 > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, > .01% i/o-4/32 > >>>>>>> > >>>>>>> > >>>>>>>Overhead measurements: > >>>>>>> > >>>>>>>Progress in terms of number of passes through a fixed > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>system workload > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>>>>>The workload was usex -b48. > >>>>>>> > >>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>>>> > >>>>>>> > >>>>>>>Conclusions: > >>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy > requirement for ntp > >>>>>>>tracking under the loads > >>>>>>>above is the SYNC protocol. The worst case accuracies for > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>SYNC, MIXED, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>and ASYNC > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>>>> > >>>>>>>We could reduce the cost of the SYNC method by only > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>scheduling the extra > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>wakeups if a certain number > >>>>>>>of ticks are missed. > >>>>>>> > >>>>>>>Regards, > >>>>>>>Dave > >>>>>>> > >>>>>>>Keir Fraser wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>couple of days ago, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have > been something > >>>>>>>>>wrong with the code I used a couple of days ago for > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>ASYNC. It may have been > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>missing the immediate delivery of interrupt after context > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>switch in. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>acceptable accuracy, > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>each running consistently around or under .01%. MIXED has > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>a fairly high > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>error of > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>threshold for comfort. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>SYNC running > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>running instead. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>It may be too early to pick the protocol and I can run > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>more overnight tests > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>>next week. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side effects of the > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>SYNC+run_timer > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will cause higher > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>system-wide CPU > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>contention. I find it easier to think through the > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>implications of ASYNC. I'm > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>surprised that MIXED loses time, and is less accurate than > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>ASYNC. Perhaps it > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>delivers more timer interrupts than the other approaches, > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>and each interrupt > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>event causes a small accumulated error? > >>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>if the latter is > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>actually more accurate then I can simply revert the > changeset that > >>>>>>>>implemented MIXED. > >>>>>>>> > >>>>>>>>Perhaps rather than running more of the same workloads you > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>could try idle > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>to /dev/null)? We > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>don't have any data on workloads that aren't CPU bound, so > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>that's really an > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>>>>> > >>>>>>>>-- Keir > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>_______________________________________________ > >>>>>>Xen-devel mailing list > >>>>>>Xen-devel@lists.xensource.com > >>>>>>http://lists.xensource.com/xen-devel > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> > >>> > >>> > >>> > >>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru > >> > >> missed_ticks =3D missed_ticks / (s_time_t) pt->period + 1; > >> if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>+ pt->do_not_freeze =3D 1; > >> else > >> pt->pending_intr_nr +=3D missed_ticks; > >> pt->scheduled +=3D missed_ticks * pt->period; > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > >> > >> pt_lock(pt); > >> > >>- pt->pending_intr_nr++; > >>+ if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { > >>+ pt->pending_intr_nr =3D 1; > >>+ pt->do_not_freeze =3D 0; > >>+ } > >>+ else > >>+ pt->pending_intr_nr++; > >> > >> if ( !pt->one_shot ) > >> { > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct > >> return; > >> } > >> > >>- pt->do_not_freeze =3D 0; > >>- > >> if ( pt->one_shot ) > >> { > >> pt->enabled =3D 0; > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct > >> pt->last_plt_gtime =3D hvm_get_guest_time(v); > >> pt->pending_intr_nr =3D 0; /* 'collapse' all > missed ticks */ > >> } > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { > >>+ pt->pending_intr_nr--; > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>+ } > >> else > >> { > >> pt->last_plt_gtime +=3D pt->period_cycles; > >> > >> > > > > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > ------_=_NextPart_001_01C8612A.A1A3A9FC Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable RE: [Xen-devel] [PATCH] Add a timer mode = that disables pending missed ticks=0A= =0A= =0A= =0A=
=0A=
Hi = Dan,
=0A=
 
=0A=
Hpet timer does have a fairly = large error, as I was trying this one recently.
=0A=
I don't remember what I got = for error, but 1% sounds about right.
=0A=
The problem is that hpet = is not built on top of vpt.c, the module Keir and I did
=0A=
all the recent work = in, for its periodic timer needs. Try specifying clock=3Dpit
=0A=
on the linux boot line. If it = still picks the hpet, which it might, = let me know
=0A=
and I'll tell you how to get = around this.
=0A=
 
=0A=
Regards,
=0A=
Dave
=0A=
 
=0A=
 
=0A=
 
=0A=

=0A=
=0A= From: Dan Magenheimer = [mailto:dan.magenheimer@oracle.com]
Sent: Fri 1/25/2008 6:50 = PM
To: Dave Winchell; Keir Fraser
Cc: = xen-devel@lists.xensource.com; deepak.patel@oracle.com; = akira.ijuin@oracle.com
Subject: RE: [Xen-devel] [PATCH] Add a = timer mode that disables pending missed ticks

=0A=
=0A=

Sorry for the very late followup on this but we = finally were able
to get our testing set up again on stable 3.1 bits = and have
seen some very bad results on 3.1.3-rc1, on the order of = 1%.

Test enviroment was a 4-socket dual core machine with 24GB = of
memory running six two-vcpu 2GB domains, four hvm plus two = pv.
All six guests were running LTP simultaneously.  The four = hvm
guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and = RHEL4u5-64.
Timer_mode was set to 2 for 64-bit guests and 0 for = 32-bit guests.
All four hvm guests experienced skew around -1%, even = the 32-bit
guest.  Less intensive testing didn't exhibit much = skew at all.

A representative graph is attached.

Dave, I = wonder if some portion of your patches didn't end up in
the xen = trees?

(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform = timer
14.318MHz HPET.)

Thanks,
Dan

P.S. Many thanks = to Deepak and Akira for running tests.

> -----Original = Message-----
> From: xen-devel-bounces@lists.xensource.com
> = [mailto:xen-devel-bo= unces@lists.xensource.com]On Behalf Of
> Dave Winchell
> = Sent: Wednesday, January 09, 2008 9:53 AM
> To: Keir = Fraser
> Cc: dan.magenheimer@oracle.com; = xen-devel@lists.xensource.com; Dave
> Winchell
> Subject: = Re: [Xen-devel] [PATCH] Add a timer mode that
> disables = pending
> missed ticks
>
>
> Hi = Keir,
>
> The latest change, c/s 16690, looks fine.
> = I agree that the code in c/s 16690 is equivalent to
> the code I = submitted. Also, your version is more
> concise.
>
> = The error tests confirm the equivalence. With overnight cpu = loads,
> the checked in version was accurate to +.048% for = sles
> and +.038% for red hat. My version was +.046% and +.032% in = a
> 2 hour test.
> I don't think the difference is = significant.
>
> i/o loads produced errors of = +.01%.
>
> Thanks for all your efforts on this = issue.
>
> Regards,
> = Dave
>
>
>
> Keir Fraser wrote:
>
> = >Applied as c/s 16690, although the checked-in patch is
> = smaller. I think the
> >only important fix is to pt_intr_post() = and the only bit of
> the patch I
> >totally omitted was = the change to pt_process_missed_ticks().
> I don't think
> = >that change can be important, but let's see what happens to the = error
> >percentage...
> >
> > -- = Keir
> >
> >On 4/1/08 23:24, "Dave Winchell" = <dwinchell@virtualiron.com> wrote:
> >
> = >
> >
> >>Hi Dan and Keir,
> = >>
> >>Attached is a patch that fixes some issues with = the SYNC policy
> >>(no_missed_ticks_pending).
> = >>I have not tried to make the change the minimal one, = but,
> rather, just
> >>ported into
> = >>the new code what I know to work well. The error for
> = >>no_missed_ticks_pending goes from
> >>over 3% to = .03% with this change according to my testing.
> >>
> = >>Regards,
> >>Dave
> >>
> = >>Dan Magenheimer wrote:
> >>
> >>
> = >>
> >>>Hi Dave --
> >>>
> = >>>Did you get your correction ported?  If so, it would = be
> nice to see this get
> >>>into 3.1.3.
> = >>>
> >>>Note that I just did some very limited = testing with
> timer_mode=3D2(=3DSYNC=3Dno
> = >>>missed ticks pending)
> >>>on tip of = xen-3.1-testing (64-bit Linux hv guest) and the
> worst error = I've
> >>>seen so far
> >>>is = 0.012%.  But I haven't tried any exotic loads, just LTP.
> = >>>
> >>>Thanks,
> >>>Dan
> = >>>
> >>>
> >>>
> = >>>
> >>>
> >>>>-----Original = Message-----
> >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.co= m]
> >>>>Sent: Wednesday, December 19, 2007 12:33 = PM
> >>>>To: dan.magenheimer@oracle.com
> = >>>>Cc: Keir Fraser; Shan, Haitao;
> = xen-devel@lists.xensource.com; Dong,
> >>>>Eddie; = Jiang, Yunhong; Dave Winchell
> >>>>Subject: Re: = [Xen-devel] [PATCH] Add a timer mode that
> = >>>>disables pending
> >>>>missed = ticks
> >>>>
> >>>>
> = >>>>Dan,
> >>>>
> >>>>I = did some testing with the constant tsc offset SYNC method
> = >>>>(now called
> = >>>>no_missed_ticks_pending)
> >>>>and = found the error to be very high, much larger than 1 %, as
> = >>>>I recall.
> >>>>I have not had a = chance to submit a correction. I will try to
> >>>>do = it later
> >>>>this week or the first week in January. = My version of constant tsc
> >>>>offset SYNC = method
> >>>>produces .02 % error, so I just need to = port that into the
> >>>>current code.
> = >>>>
> >>>>The error you got for both of = those kernels is what I would expect
> >>>>for the = default mode, delay_for_missed_ticks.
> >>>>
> = >>>>I'll let Keir answer on how to set the time = mode.
> >>>>
> >>>>Regards,
> = >>>>Dave
> >>>>
> = >>>>Dan Magenheimer wrote:
> >>>>
> = >>>>
> >>>>
> = >>>>
> >>>>
> = >>>>>Anyone make measurements on the final patch?
> = >>>>>
> >>>>>I just ran a 64-bit = RHEL5.1 pvm kernel and saw a loss of
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>about 0.2% with no load.  This was xen-unstable tip = today
> >>>>with no options specified.  32-bit = was about 0.01%.
> >>>>
> = >>>>
> >>>>
> = >>>>
> >>>>>I think I missed = something... how do I run the various
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>accounting choices and which ones are known to be = appropriate
> >>>>for which kernels?
> = >>>>
> >>>>
> = >>>>
> >>>>
> = >>>>>Thanks,
> >>>>>Dan
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>>
> >>>>>>-----Original = Message-----
> >>>>>>From: = xen-devel-bounces@lists.xensource.com
> = >>>>>>[mailto:xen-devel-bo= unces@lists.xensource.com]On Behalf Of
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>Keir Fraser
> >>>>
> = >>>>
> >>>>
> = >>>>
> >>>>>>Sent: Thursday, = December 06, 2007 4:57 AM
> >>>>>>To: Dave = Winchell
> >>>>>>Cc: Shan, Haitao; = xen-devel@lists.xensource.com; Dong,
> Eddie; Jiang,
> = >>>>>>Yunhong
> >>>>>>Subject: = Re: [Xen-devel] [PATCH] Add a timer mode that
> = >>>>>>disables pending
> = >>>>>>missed ticks
> = >>>>>>
> >>>>>>
> = >>>>>>Please take a look at xen-unstable changeset = 16545.
> >>>>>>
> = >>>>>>-- Keir
> >>>>>>
> = >>>>>>On 26/11/07 20:57, "Dave Winchell"
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>><dwinchell@virtualiron.com> wrote:
> = >>>>
> >>>>
> = >>>>
> >>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>Keir,
> = >>>>>>>
> >>>>>>>The = accuracy data I've collected for i/o loads for the
> = >>>>>>>various time protocols follows. In addition, = the data
> >>>>>>>for cpu loads is = shown.
> >>>>>>>
> = >>>>>>>The loads labeled cpu and i/o-8 are on an 8 = processor AMD box.
> >>>>>>>Two guests, red = hat and sles 64 bit, 8 vcpu each.
> = >>>>>>>The cpu load is usex -e36 on each = guest.
> >>>>>>>(usex is available at http://people.redhat.com/= anderson/usex.)
> >>>>>>>i/o load is 8 = instances of dd if=3D/dev/hda6 of=3D/dev/null.
> = >>>>>>>
> >>>>>>>The = loads labeled i/o-32 are 32 instances of dd.
> = >>>>>>>Also, these are run on 4 cpu AMD = box.
> >>>>>>>In addition, there is an idle = rh-32bit guest.
> >>>>>>>All three guests are = 8vcpu.
> >>>>>>>
> = >>>>>>>The loads labeled i/o-4/32 are the same as = i/o-32
> >>>>>>>except that the redhat-64 = guest has 4 instances of dd.
> = >>>>>>>
> >>>>>>>Date = Duration Protocol sles, rhat error load
> = >>>>>>>
> >>>>>>>11/07 = 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%,
> +.005% = cpu
> >>>>>>>11/09 3 hrs 19 min ASYNC -.13 = sec, +1.44 sec, -.001%,
> +.012% cpu
> = >>>>>>>
> >>>>>>>11/08 2 = hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, -.004% cpu
> = >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, = -.005%, -.005% cpu
> >>>>>>>11/12 65 hrs 40 = min SYNC -18 sec, -8 sec, -.008%, -.003% cpu
> = >>>>>>>
> >>>>>>>11/08 = 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu
> = >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 = sec, -.034%,
> -.031% cpu
> = >>>>>>>
> = >>>>>>>
> >>>>>>>11/14 = 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%,
> -.09% = i/o-8
> >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 = sec,-14.0 sec, -.015%
> -.14% i/o-8
> = >>>>>>>
> >>>>>>>11/13 = 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%,
> -.022% = i/o-8
> >>>>>>>11/14 48 min SYNC - .46 sec, - = .48 sec, -.017%, -.018% i/o-8
> = >>>>>>>
> >>>>>>>11/14 4 = hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%,
> -.029% = i/o-8
> >>>>>>>11/20 16 hrs 2 min MIXED -13.4 = sec,-18.1 sec, -.023%,
> -.031% i/o-8
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> >>>>>>>11/21 = 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32
> = >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, = -.011%,
> -.005% i/o-32
> >>>>>>>11/21 = 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32
> = >>>>>>>
> >>>>>>>11/26 = 113 hrs 46 min MIXED -297. sec, 13. sec -.07%,
> .003% = i/o-4/32
> >>>>>>>11/26 4 hrs 50 min SYNC = -3.21 sec, 1.44 sec, -.017%,
> .01% i/o-4/32
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>Overhead measurements:
> = >>>>>>>
> = >>>>>>>Progress in terms of number of passes = through a fixed
> >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> >>>>>>system = workload
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>on an 8 = vcpu red hat with an 8 vcpu sles idle.
> = >>>>>>>The workload was usex -b48.
> = >>>>>>>
> = >>>>>>>
> >>>>>>>ASYNC = 167 min 145 passes .868 passes/min
> = >>>>>>>SYNC 167 min 144 passes .862 = passes/min
> >>>>>>>SYNC 1065 min 919 passes = .863 passes/min
> >>>>>>>MIXED 221 min 196 = passes .887 passes/min
> >>>>>>>
> = >>>>>>>
> = >>>>>>>Conclusions:
> = >>>>>>>
> >>>>>>>The = only protocol which meets the .05% accuracy
> requirement for = ntp
> >>>>>>>tracking under the loads
> = >>>>>>>above is the SYNC protocol. The worst case = accuracies for
> >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> >>>>>>SYNC, = MIXED,
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>and = ASYNC
> >>>>>>>are .022%, .12%, and .14%, = respectively.
> >>>>>>>
> = >>>>>>>We could reduce the cost of the SYNC method = by only
> >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> >>>>>>scheduling = the extra
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>wakeups if = a certain number
> >>>>>>>of ticks are = missed.
> >>>>>>>
> = >>>>>>>Regards,
> = >>>>>>>Dave
> = >>>>>>>
> >>>>>>>Keir = Fraser wrote:
> >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> = >>>>>>>
> >>>>>>>>On = 9/11/07 19:22, "Dave Winchell"
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>><dwinchell@virtualiron.com> wrote:
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>>Since I had a high error (~.03%) for = the ASYNC method a
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>couple of days ago,
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>I ran another ASYNC test. I think = there may have
> been something
> = >>>>>>>>>wrong with the code I used a couple = of days ago for
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>ASYNC. It may have been
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>missing the immediate delivery of = interrupt after context
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>switch in.
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>My results indicate that either SYNC = or ASYNC give
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>acceptable accuracy,
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>each running consistently around or = under .01%. MIXED has
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> >>>>>>a = fairly high
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> = >>>>>>>>>error of
> = >>>>>>>>>greater than .03%. Probably too = close to .05% ntp
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>threshold for comfort.
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>I don't have an overnight run with = SYNC. I plan to leave
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>SYNC running
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>over the weekend. If you'd rather I = can leave MIXED
> >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>running instead.
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>It may be too early to pick the = protocol and I can run
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>more overnight tests
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>>next week.
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>>
> = >>>>>>>>I'm a bit worried about any unwanted = side effects of the
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>SYNC+run_timer
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>approach -- e.g., whether timer wakeups = will cause higher
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>system-wide CPU
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>contention. I find it easier to think = through the
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>implications of ASYNC. I'm
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>>>surprised that MIXED loses time, and is = less accurate than
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>ASYNC. = Perhaps it
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> = >>>>>>>>delivers more timer interrupts than the = other approaches,
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>and = each interrupt
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>event = causes a small accumulated error?
> = >>>>>>>>
> = >>>>>>>>Overall I would consider MIXED and ASYNC = as favourites and
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>if the = latter is
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> = >>>>>>>>actually more accurate then I can simply = revert the
> changeset that
> = >>>>>>>>implemented MIXED.
> = >>>>>>>>
> = >>>>>>>>Perhaps rather than running more of the = same workloads you
> >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>could = try idle
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>VCPUs = and I/O bound VCPUs (e.g., repeated large disc reads
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>to = /dev/null)? We
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>don't = have any data on workloads that aren't CPU bound, so
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> >>>>>>that's = really an
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>>>obvious = place to put any further effort imo.
> = >>>>>>>>
> = >>>>>>>>-- Keir
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>>>
> = >>>>>>_______________________________________________> >>>>>>Xen-devel mailing list
> = >>>>>>Xen-devel@lists.xensource.com
> = >>>>>>http://lists.xensource.com/= xen-devel
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>>
> = >>>>>>
> >>>>>
> = >>>>>
> >>>>>
> = >>>>>
> >>>>
> = >>>>
> >>>>
> = >>>>
> >>>
> >>>
> = >>>
> >>>
> >>diff -r cfdbdca5b831 = xen/arch/x86/hvm/vpt.c
> >>--- a/xen/arch/x86/hvm/vpt.c Thu = Dec 06 15:36:07 2007 +0000
> >>+++ b/xen/arch/x86/hvm/vpt.c = Fri Jan 04 17:58:16 2008 -0500
> >>@@ -58,7 +58,7 @@ static = void pt_process_missed_ticks(stru
> >>
> = >>     missed_ticks =3D missed_ticks / = (s_time_t) pt->period + 1;
> >>     = if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) )
> = >>-        pt->do_not_freeze = =3D !pt->pending_intr_nr;
> = >>+        pt->do_not_freeze = =3D 1;
> >>     else
> = >>         = pt->pending_intr_nr +=3D missed_ticks;
> = >>     pt->scheduled +=3D missed_ticks * = pt->period;
> >>@@ -127,7 +127,12 @@ static void = pt_timer_fn(void *data)
> >>
> = >>     pt_lock(pt);
> >>
> = >>-    pt->pending_intr_nr++;
> = >>+    if ( mode_is(pt->vcpu->domain, = no_missed_ticks_pending) ) {
> = >>+        = pt->pending_intr_nr =3D 1;
> >>+ pt->do_not_freeze =3D = 0;
> >>+    }
> = >>+    else
> >>+ = pt->pending_intr_nr++;
> >>
> = >>     if ( !pt->one_shot )
> = >>     {
> >>@@ -221,8 +226,6 @@ = void pt_intr_post(struct vcpu *v, struct
> = >>         return;
> = >>     }
> >>
> = >>-    pt->do_not_freeze =3D 0;
> = >>-
> >>     if ( pt->one_shot = )
> >>     {
> = >>         pt->enabled = =3D 0;
> >>@@ -235,6 +238,10 @@ void pt_intr_post(struct = vcpu *v, struct
> = >>           = ;  pt->last_plt_gtime =3D hvm_get_guest_time(v);
> = >>           = ;  pt->pending_intr_nr =3D 0; /* 'collapse' all
> missed = ticks */
> = >>         }
> = >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) = {
> >>+     = pt->pending_intr_nr--;
> >>+     = pt->last_plt_gtime =3D hvm_get_guest_time(v);
> >>+ = }
> >>         = else
> >>         = {
> = >>           = ;  pt->last_plt_gtime +=3D pt->period_cycles;
> = >>
> >>
> >
> >
> >
> = >
>
>
> = _______________________________________________
> Xen-devel = mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/= xen-devel
>

------_=_NextPart_001_01C8612A.A1A3A9FC-- --===============0992105506== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0992105506==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Mon, 28 Jan 2008 10:21:19 -0500 Message-ID: <479DF2EF.8090601@virtualiron.com> References: <20080127172933828.00000002440@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080127172933828.00000002440@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "akira.ijuin@oracle.com" , "xen-devel@lists.xensource.com" , "deepak.patel@oracle.com" , Dave Winchell List-Id: xen-devel@lists.xenproject.org Dan, I guess I'm a bit out of date calling for clock= usage. Looking at linux 2.6.20.4 sources, I think you should specify "clocksource=pit nohpet" on the linux guest bootline. You can leave the xen and dom0 bootlines as they are. The xen and guest clocksources do not need to be the same. In my tests, xen is using the hpet for its timekeeping and that appears to be the default. When you boot the guests you should see time.c: Using PIT/TSC based timekeeping. on the rh4u5-64 guest, and something similar on the others. > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > 14.318MHz HPET.) This appears to be the xen state, which is fine. I was wrongly assuming that this was the guest state. You might want to look in your guest logs and see what they were picking for a clock source. Regards, Dave Dan Magenheimer wrote: > Thanks, I hadn't realized that! No wonder we didn't see the same > improvement you saw! > >> Try specifying clock=pit on the linux boot line... > > I'm confused... do you mean "clocksource=pit" on the Xen command line or > "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or dom0?) command > line? Or both places? Since the tests take awhile, it would be nice > to get this right the first time. Do the Xen and guest clocksources need > to be the same? > > Thanks, > Dan > > -----Original Message----- > *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] > *Sent:* Sunday, January 27, 2008 2:22 PM > *To:* dan.magenheimer@oracle.com; Keir Fraser > *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > akira.ijuin@oracle.com; Dave Winchell > *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables > pending missed ticks > > Hi Dan, > > Hpet timer does have a fairly large error, as I was trying this > one recently. > I don't remember what I got for error, but 1% sounds about right. > The problem is that hpet is not built on top of vpt.c, the module > Keir and I did > all the recent work in, for its periodic timer needs. Try > specifying clock=pit > on the linux boot line. If it still picks the hpet, which it > might, let me know > and I'll tell you how to get around this. > > Regards, > Dave > > > > > ------------------------------------------------------------------------ > *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > *Sent:* Fri 1/25/2008 6:50 PM > *To:* Dave Winchell; Keir Fraser > *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > akira.ijuin@oracle.com > *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables > pending missed ticks > > Sorry for the very late followup on this but we finally were able > to get our testing set up again on stable 3.1 bits and have > seen some very bad results on 3.1.3-rc1, on the order of 1%. > > Test enviroment was a 4-socket dual core machine with 24GB of > memory running six two-vcpu 2GB domains, four hvm plus two pv. > All six guests were running LTP simultaneously. The four hvm > guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. > Timer_mode was set to 2 for 64-bit guests and 0 for 32-bit guests. > All four hvm guests experienced skew around -1%, even the 32-bit > guest. Less intensive testing didn't exhibit much skew at all. > > A representative graph is attached. > > Dave, I wonder if some portion of your patches didn't end up in > the xen trees? > > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > 14.318MHz HPET.) > > Thanks, > Dan > > P.S. Many thanks to Deepak and Akira for running tests. > > > -----Original Message----- > > From: xen-devel-bounces@lists.xensource.com > > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > > Dave Winchell > > Sent: Wednesday, January 09, 2008 9:53 AM > > To: Keir Fraser > > Cc: dan.magenheimer@oracle.com; xen-devel@lists.xensource.com; Dave > > Winchell > > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > disables pending > > missed ticks > > > > > > Hi Keir, > > > > The latest change, c/s 16690, looks fine. > > I agree that the code in c/s 16690 is equivalent to > > the code I submitted. Also, your version is more > > concise. > > > > The error tests confirm the equivalence. With overnight cpu loads, > > the checked in version was accurate to +.048% for sles > > and +.038% for red hat. My version was +.046% and +.032% in a > > 2 hour test. > > I don't think the difference is significant. > > > > i/o loads produced errors of +.01%. > > > > Thanks for all your efforts on this issue. > > > > Regards, > > Dave > > > > > > > > Keir Fraser wrote: > > > > >Applied as c/s 16690, although the checked-in patch is > > smaller. I think the > > >only important fix is to pt_intr_post() and the only bit of > > the patch I > > >totally omitted was the change to pt_process_missed_ticks(). > > I don't think > > >that change can be important, but let's see what happens to the > error > > >percentage... > > > > > > -- Keir > > > > > >On 4/1/08 23:24, "Dave Winchell" wrote: > > > > > > > > > > > >>Hi Dan and Keir, > > >> > > >>Attached is a patch that fixes some issues with the SYNC policy > > >>(no_missed_ticks_pending). > > >>I have not tried to make the change the minimal one, but, > > rather, just > > >>ported into > > >>the new code what I know to work well. The error for > > >>no_missed_ticks_pending goes from > > >>over 3% to .03% with this change according to my testing. > > >> > > >>Regards, > > >>Dave > > >> > > >>Dan Magenheimer wrote: > > >> > > >> > > >> > > >>>Hi Dave -- > > >>> > > >>>Did you get your correction ported? If so, it would be > > nice to see this get > > >>>into 3.1.3. > > >>> > > >>>Note that I just did some very limited testing with > > timer_mode=2(=SYNC=no > > >>>missed ticks pending) > > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the > > worst error I've > > >>>seen so far > > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. > > >>> > > >>>Thanks, > > >>>Dan > > >>> > > >>> > > >>> > > >>> > > >>> > > >>>>-----Original Message----- > > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > > >>>>To: dan.magenheimer@oracle.com > > >>>>Cc: Keir Fraser; Shan, Haitao; > > xen-devel@lists.xensource.com; Dong, > > >>>>Eddie; Jiang, Yunhong; Dave Winchell > > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > >>>>disables pending > > >>>>missed ticks > > >>>> > > >>>> > > >>>>Dan, > > >>>> > > >>>>I did some testing with the constant tsc offset SYNC method > > >>>>(now called > > >>>>no_missed_ticks_pending) > > >>>>and found the error to be very high, much larger than 1 %, as > > >>>>I recall. > > >>>>I have not had a chance to submit a correction. I will try to > > >>>>do it later > > >>>>this week or the first week in January. My version of > constant tsc > > >>>>offset SYNC method > > >>>>produces .02 % error, so I just need to port that into the > > >>>>current code. > > >>>> > > >>>>The error you got for both of those kernels is what I would > expect > > >>>>for the default mode, delay_for_missed_ticks. > > >>>> > > >>>>I'll let Keir answer on how to set the time mode. > > >>>> > > >>>>Regards, > > >>>>Dave > > >>>> > > >>>>Dan Magenheimer wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>>>Anyone make measurements on the final patch? > > >>>>> > > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>about 0.2% with no load. This was xen-unstable tip today > > >>>>with no options specified. 32-bit was about 0.01%. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>>>I think I missed something... how do I run the various > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>accounting choices and which ones are known to be appropriate > > >>>>for which kernels? > > >>>> > > >>>> > > >>>> > > >>>> > > >>>>>Thanks, > > >>>>>Dan > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>>>-----Original Message----- > > >>>>>>From: xen-devel-bounces@lists.xensource.com > > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>Keir Fraser > > >>>> > > >>>> > > >>>> > > >>>> > > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > > >>>>>>To: Dave Winchell > > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, > > Eddie; Jiang, > > >>>>>>Yunhong > > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > >>>>>>disables pending > > >>>>>>missed ticks > > >>>>>> > > >>>>>> > > >>>>>>Please take a look at xen-unstable changeset 16545. > > >>>>>> > > >>>>>>-- Keir > > >>>>>> > > >>>>>>On 26/11/07 20:57, "Dave Winchell" > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>> wrote: > > >>>> > > >>>> > > >>>> > > >>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>Keir, > > >>>>>>> > > >>>>>>>The accuracy data I've collected for i/o loads for the > > >>>>>>>various time protocols follows. In addition, the data > > >>>>>>>for cpu loads is shown. > > >>>>>>> > > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 processor AMD > box. > > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > > >>>>>>>The cpu load is usex -e36 on each guest. > > >>>>>>>(usex is available at > http://people.redhat.com/anderson/usex.) > > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 of=/dev/null. > > >>>>>>> > > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > > >>>>>>>Also, these are run on 4 cpu AMD box. > > >>>>>>>In addition, there is an idle rh-32bit guest. > > >>>>>>>All three guests are 8vcpu. > > >>>>>>> > > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > > >>>>>>>except that the redhat-64 guest has 4 instances of dd. > > >>>>>>> > > >>>>>>>Date Duration Protocol sles, rhat error load > > >>>>>>> > > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, > > +.005% cpu > > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, > > +.012% cpu > > >>>>>>> > > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, > -.004% cpu > > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, -.005%, -.005% cpu > > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, -.008%, -.003% cpu > > >>>>>>> > > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, -.040% cpu > > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, > > -.031% cpu > > >>>>>>> > > >>>>>>> > > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, > > -.09% i/o-8 > > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% > > -.14% i/o-8 > > >>>>>>> > > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, > > -.022% i/o-8 > > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, -.017%, -.018% i/o-8 > > >>>>>>> > > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, > > -.029% i/o-8 > > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, > > -.031% i/o-8 > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, -.04% i/o-32 > > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, > > -.005% i/o-32 > > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, -.11% i/o-32 > > >>>>>>> > > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, > > .003% i/o-4/32 > > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, > > .01% i/o-4/32 > > >>>>>>> > > >>>>>>> > > >>>>>>>Overhead measurements: > > >>>>>>> > > >>>>>>>Progress in terms of number of passes through a fixed > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>system workload > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > > >>>>>>>The workload was usex -b48. > > >>>>>>> > > >>>>>>> > > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > > >>>>>>>SYNC 167 min 144 passes .862 passes/min > > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > > >>>>>>>MIXED 221 min 196 passes .887 passes/min > > >>>>>>> > > >>>>>>> > > >>>>>>>Conclusions: > > >>>>>>> > > >>>>>>>The only protocol which meets the .05% accuracy > > requirement for ntp > > >>>>>>>tracking under the loads > > >>>>>>>above is the SYNC protocol. The worst case accuracies for > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>SYNC, MIXED, > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>and ASYNC > > >>>>>>>are .022%, .12%, and .14%, respectively. > > >>>>>>> > > >>>>>>>We could reduce the cost of the SYNC method by only > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>scheduling the extra > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>wakeups if a certain number > > >>>>>>>of ticks are missed. > > >>>>>>> > > >>>>>>>Regards, > > >>>>>>>Dave > > >>>>>>> > > >>>>>>>Keir Fraser wrote: > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>> wrote: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>>>Since I had a high error (~.03%) for the ASYNC method a > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>couple of days ago, > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>I ran another ASYNC test. I think there may have > > been something > > >>>>>>>>>wrong with the code I used a couple of days ago for > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>ASYNC. It may have been > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>missing the immediate delivery of interrupt after context > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>switch in. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>My results indicate that either SYNC or ASYNC give > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>acceptable accuracy, > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>each running consistently around or under .01%. MIXED has > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>a fairly high > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>error of > > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>threshold for comfort. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>I don't have an overnight run with SYNC. I plan to leave > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>SYNC running > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>running instead. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>It may be too early to pick the protocol and I can run > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>more overnight tests > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>>next week. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>I'm a bit worried about any unwanted side effects of the > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>SYNC+run_timer > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>approach -- e.g., whether timer wakeups will cause higher > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>system-wide CPU > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>contention. I find it easier to think through the > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>implications of ASYNC. I'm > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>surprised that MIXED loses time, and is less accurate than > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>ASYNC. Perhaps it > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>delivers more timer interrupts than the other approaches, > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>and each interrupt > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>event causes a small accumulated error? > > >>>>>>>> > > >>>>>>>>Overall I would consider MIXED and ASYNC as favourites and > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>if the latter is > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>actually more accurate then I can simply revert the > > changeset that > > >>>>>>>>implemented MIXED. > > >>>>>>>> > > >>>>>>>>Perhaps rather than running more of the same workloads you > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>could try idle > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated large disc reads > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>to /dev/null)? We > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>don't have any data on workloads that aren't CPU bound, so > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>that's really an > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>>>obvious place to put any further effort imo. > > >>>>>>>> > > >>>>>>>>-- Keir > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>_______________________________________________ > > >>>>>>Xen-devel mailing list > > >>>>>>Xen-devel@lists.xensource.com > > >>>>>>http://lists.xensource.com/xen-devel > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>> > > >>> > > >>> > > >>> > > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 > > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 > > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru > > >> > > >> missed_ticks = missed_ticks / (s_time_t) pt->period + 1; > > >> if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) > > >>- pt->do_not_freeze = !pt->pending_intr_nr; > > >>+ pt->do_not_freeze = 1; > > >> else > > >> pt->pending_intr_nr += missed_ticks; > > >> pt->scheduled += missed_ticks * pt->period; > > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > > >> > > >> pt_lock(pt); > > >> > > >>- pt->pending_intr_nr++; > > >>+ if ( mode_is(pt->vcpu->domain, no_missed_ticks_pending) ) { > > >>+ pt->pending_intr_nr = 1; > > >>+ pt->do_not_freeze = 0; > > >>+ } > > >>+ else > > >>+ pt->pending_intr_nr++; > > >> > > >> if ( !pt->one_shot ) > > >> { > > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct > > >> return; > > >> } > > >> > > >>- pt->do_not_freeze = 0; > > >>- > > >> if ( pt->one_shot ) > > >> { > > >> pt->enabled = 0; > > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu *v, struct > > >> pt->last_plt_gtime = hvm_get_guest_time(v); > > >> pt->pending_intr_nr = 0; /* 'collapse' all > > missed ticks */ > > >> } > > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { > > >>+ pt->pending_intr_nr--; > > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); > > >>+ } > > >> else > > >> { > > >> pt->last_plt_gtime += pt->period_cycles; > > >> > > >> > > > > > > > > > > > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 29 Jan 2008 15:34:58 -0700 Message-ID: <20080129153458531.00000002384@djm-pc> References: <479DF2EF.8090601@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=-------6184412361844123 Return-path: In-Reply-To: <479DF2EF.8090601@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "akira.ijuin@oracle.com" , "xen-devel@lists.xensource.com" , "deepak.patel@oracle.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format ---------6184412361844123 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hi Dave (Keir, see suggestion below) -- Thanks! Turning off vhpet certainly helps a lot (though see below). I wonder if timekeeping with vhpet is so bad that it should be turned off by default (in 3.1, 3.2, and unstable) until it is fixed? (I have a patch that defaults it off, can post it if there is agreement on the above point.) The whole point of an HPET is to provide more precise timekeeping and if vhpet is worse than vpit, it can only confuse users. Comments? In your testing, are you just measuring % skew over a long period of time? We are graphing the skew continuously and seeing periodic behavior that is unsettling, even with pit. See attached. Though your algorithm recovers, the "cliffs" could still cause real user problems. I wonder if there is anything that can be done to make the "recovery" more responsive? We are looking into what part(s) of LTP is causing the cliffs. Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Monday, January 28, 2008 8:21 AM > To: dan.magenheimer@oracle.com > Cc: Keir Fraser; xen-devel@lists.xensource.com; = > deepak.patel@oracle.com; > akira.ijuin@oracle.com; Dave Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Dan, > = > I guess I'm a bit out of date calling for clock=3D usage. > Looking at linux 2.6.20.4 sources, I think you should specify > "clocksource=3Dpit nohpet" on the linux guest bootline. > = > You can leave the xen and dom0 bootlines as they are. > The xen and guest clocksources do not need to be the same. > In my tests, xen is using the hpet for its timekeeping and > that appears to be the default. > = > When you boot the guests you should see > time.c: Using PIT/TSC based timekeeping. > on the rh4u5-64 guest, and something similar on the others. > = > > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > > 14.318MHz HPET.) > = > This appears to be the xen state, which is fine. > I was wrongly assuming that this was the guest state. > You might want to look in your guest logs and see what they = > were picking > for a clock source. > = > Regards, > Dave > = > = > = > = > Dan Magenheimer wrote: > = > > Thanks, I hadn't realized that! No wonder we didn't see the same > > improvement you saw! > > > >> Try specifying clock=3Dpit on the linux boot line... > > > > I'm confused... do you mean "clocksource=3Dpit" on the Xen = > command line or > > "nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the guest (or = > dom0?) command > > line? Or both places? Since the tests take awhile, it = > would be nice > > to get this right the first time. Do the Xen and guest = > clocksources need > > to be the same? > > > > Thanks, > > Dan > > > > -----Original Message----- > > *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] > > *Sent:* Sunday, January 27, 2008 2:22 PM > > *To:* dan.magenheimer@oracle.com; Keir Fraser > > *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > > akira.ijuin@oracle.com; Dave Winchell > > *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables > > pending missed ticks > > > > Hi Dan, > > > > Hpet timer does have a fairly large error, as I was trying this > > one recently. > > I don't remember what I got for error, but 1% sounds = > about right. > > The problem is that hpet is not built on top of vpt.c, = > the module > > Keir and I did > > all the recent work in, for its periodic timer needs. Try > > specifying clock=3Dpit > > on the linux boot line. If it still picks the hpet, which it > > might, let me know > > and I'll tell you how to get around this. > > > > Regards, > > Dave > > > > > > > > > > = > -------------------------------------------------------------- > ---------- > > *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > > *Sent:* Fri 1/25/2008 6:50 PM > > *To:* Dave Winchell; Keir Fraser > > *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > > akira.ijuin@oracle.com > > *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode = > that disables > > pending missed ticks > > > > Sorry for the very late followup on this but we finally = > were able > > to get our testing set up again on stable 3.1 bits and have > > seen some very bad results on 3.1.3-rc1, on the order of 1%. > > > > Test enviroment was a 4-socket dual core machine with 24GB of > > memory running six two-vcpu 2GB domains, four hvm plus two pv. > > All six guests were running LTP simultaneously. The four hvm > > guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. > > Timer_mode was set to 2 for 64-bit guests and 0 for = > 32-bit guests. > > All four hvm guests experienced skew around -1%, even the 32-bit > > guest. Less intensive testing didn't exhibit much skew at all. > > > > A representative graph is attached. > > > > Dave, I wonder if some portion of your patches didn't end up in > > the xen trees? > > > > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > > 14.318MHz HPET.) > > > > Thanks, > > Dan > > > > P.S. Many thanks to Deepak and Akira for running tests. > > > > > -----Original Message----- > > > From: xen-devel-bounces@lists.xensource.com > > > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > > > Dave Winchell > > > Sent: Wednesday, January 09, 2008 9:53 AM > > > To: Keir Fraser > > > Cc: dan.magenheimer@oracle.com; = > xen-devel@lists.xensource.com; Dave > > > Winchell > > > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > > disables pending > > > missed ticks > > > > > > > > > Hi Keir, > > > > > > The latest change, c/s 16690, looks fine. > > > I agree that the code in c/s 16690 is equivalent to > > > the code I submitted. Also, your version is more > > > concise. > > > > > > The error tests confirm the equivalence. With = > overnight cpu loads, > > > the checked in version was accurate to +.048% for sles > > > and +.038% for red hat. My version was +.046% and +.032% in a > > > 2 hour test. > > > I don't think the difference is significant. > > > > > > i/o loads produced errors of +.01%. > > > > > > Thanks for all your efforts on this issue. > > > > > > Regards, > > > Dave > > > > > > > > > > > > Keir Fraser wrote: > > > > > > >Applied as c/s 16690, although the checked-in patch is > > > smaller. I think the > > > >only important fix is to pt_intr_post() and the only bit of > > > the patch I > > > >totally omitted was the change to pt_process_missed_ticks(). > > > I don't think > > > >that change can be important, but let's see what = > happens to the > > error > > > >percentage... > > > > > > > > -- Keir > > > > > > > >On 4/1/08 23:24, "Dave Winchell" = > wrote: > > > > > > > > > > > > > > > >>Hi Dan and Keir, > > > >> > > > >>Attached is a patch that fixes some issues with the = > SYNC policy > > > >>(no_missed_ticks_pending). > > > >>I have not tried to make the change the minimal one, but, > > > rather, just > > > >>ported into > > > >>the new code what I know to work well. The error for > > > >>no_missed_ticks_pending goes from > > > >>over 3% to .03% with this change according to my testing. > > > >> > > > >>Regards, > > > >>Dave > > > >> > > > >>Dan Magenheimer wrote: > > > >> > > > >> > > > >> > > > >>>Hi Dave -- > > > >>> > > > >>>Did you get your correction ported? If so, it would be > > > nice to see this get > > > >>>into 3.1.3. > > > >>> > > > >>>Note that I just did some very limited testing with > > > timer_mode=3D2(=3DSYNC=3Dno > > > >>>missed ticks pending) > > > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the > > > worst error I've > > > >>>seen so far > > > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. > > > >>> > > > >>>Thanks, > > > >>>Dan > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>>>-----Original Message----- > > > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > > > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > > > >>>>To: dan.magenheimer@oracle.com > > > >>>>Cc: Keir Fraser; Shan, Haitao; > > > xen-devel@lists.xensource.com; Dong, > > > >>>>Eddie; Jiang, Yunhong; Dave Winchell > > > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > > >>>>disables pending > > > >>>>missed ticks > > > >>>> > > > >>>> > > > >>>>Dan, > > > >>>> > > > >>>>I did some testing with the constant tsc offset = > SYNC method > > > >>>>(now called > > > >>>>no_missed_ticks_pending) > > > >>>>and found the error to be very high, much larger = > than 1 %, as > > > >>>>I recall. > > > >>>>I have not had a chance to submit a correction. I = > will try to > > > >>>>do it later > > > >>>>this week or the first week in January. My version of > > constant tsc > > > >>>>offset SYNC method > > > >>>>produces .02 % error, so I just need to port that into the > > > >>>>current code. > > > >>>> > > > >>>>The error you got for both of those kernels is = > what I would > > expect > > > >>>>for the default mode, delay_for_missed_ticks. > > > >>>> > > > >>>>I'll let Keir answer on how to set the time mode. > > > >>>> > > > >>>>Regards, > > > >>>>Dave > > > >>>> > > > >>>>Dan Magenheimer wrote: > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>Anyone make measurements on the final patch? > > > >>>>> > > > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>about 0.2% with no load. This was xen-unstable tip today > > > >>>>with no options specified. 32-bit was about 0.01%. > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>I think I missed something... how do I run the various > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>accounting choices and which ones are known to be = > appropriate > > > >>>>for which kernels? > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>Thanks, > > > >>>>>Dan > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>>>-----Original Message----- > > > >>>>>>From: xen-devel-bounces@lists.xensource.com > > > = > >>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>Keir Fraser > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > > > >>>>>>To: Dave Winchell > > > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, > > > Eddie; Jiang, > > > >>>>>>Yunhong > > > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > > > >>>>>>disables pending > > > >>>>>>missed ticks > > > >>>>>> > > > >>>>>> > > > >>>>>>Please take a look at xen-unstable changeset 16545. > > > >>>>>> > > > >>>>>>-- Keir > > > >>>>>> > > > >>>>>>On 26/11/07 20:57, "Dave Winchell" > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>> wrote: > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>Keir, > > > >>>>>>> > > > >>>>>>>The accuracy data I've collected for i/o loads for the > > > >>>>>>>various time protocols follows. In addition, the data > > > >>>>>>>for cpu loads is shown. > > > >>>>>>> > > > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 = > processor AMD > > box. > > > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > > > >>>>>>>The cpu load is usex -e36 on each guest. > > > >>>>>>>(usex is available at > > http://people.redhat.com/anderson/usex.) > > > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 = > of=3D/dev/null. > > > >>>>>>> > > > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > > > >>>>>>>Also, these are run on 4 cpu AMD box. > > > >>>>>>>In addition, there is an idle rh-32bit guest. > > > >>>>>>>All three guests are 8vcpu. > > > >>>>>>> > > > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > > > >>>>>>>except that the redhat-64 guest has 4 instances of dd. > > > >>>>>>> > > > >>>>>>>Date Duration Protocol sles, rhat error load > > > >>>>>>> > > > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, > > > +.005% cpu > > > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, > > > +.012% cpu > > > >>>>>>> > > > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, > > -.004% cpu > > > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, = > -.005%, -.005% cpu > > > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, = > -.008%, -.003% cpu > > > >>>>>>> > > > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, = > -.040% cpu > > > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, > > > -.031% cpu > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, > > > -.09% i/o-8 > > > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% > > > -.14% i/o-8 > > > >>>>>>> > > > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, > > > -.022% i/o-8 > > > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, = > -.017%, -.018% i/o-8 > > > >>>>>>> > > > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, > > > -.029% i/o-8 > > > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, > > > -.031% i/o-8 > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, = > -.04% i/o-32 > > > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, > > > -.005% i/o-32 > > > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, = > -.11% i/o-32 > > > >>>>>>> > > > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, > > > .003% i/o-4/32 > > > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, > > > .01% i/o-4/32 > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>Overhead measurements: > > > >>>>>>> > > > >>>>>>>Progress in terms of number of passes through a fixed > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>system workload > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > > > >>>>>>>The workload was usex -b48. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > > > >>>>>>>SYNC 167 min 144 passes .862 passes/min > > > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > > > >>>>>>>MIXED 221 min 196 passes .887 passes/min > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>Conclusions: > > > >>>>>>> > > > >>>>>>>The only protocol which meets the .05% accuracy > > > requirement for ntp > > > >>>>>>>tracking under the loads > > > >>>>>>>above is the SYNC protocol. The worst case = > accuracies for > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>SYNC, MIXED, > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>and ASYNC > > > >>>>>>>are .022%, .12%, and .14%, respectively. > > > >>>>>>> > > > >>>>>>>We could reduce the cost of the SYNC method by only > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>scheduling the extra > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>wakeups if a certain number > > > >>>>>>>of ticks are missed. > > > >>>>>>> > > > >>>>>>>Regards, > > > >>>>>>>Dave > > > >>>>>>> > > > >>>>>>>Keir Fraser wrote: > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>>>Since I had a high error (~.03%) for the = > ASYNC method a > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>couple of days ago, > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>I ran another ASYNC test. I think there may have > > > been something > > > >>>>>>>>>wrong with the code I used a couple of days ago for > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>ASYNC. It may have been > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>missing the immediate delivery of interrupt = > after context > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>switch in. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>My results indicate that either SYNC or ASYNC give > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>acceptable accuracy, > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>each running consistently around or under = > .01%. MIXED has > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>a fairly high > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>error of > > > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>threshold for comfort. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>I don't have an overnight run with SYNC. I = > plan to leave > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>SYNC running > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>running instead. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>It may be too early to pick the protocol and = > I can run > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>more overnight tests > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>>next week. > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>I'm a bit worried about any unwanted side = > effects of the > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>SYNC+run_timer > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>approach -- e.g., whether timer wakeups will = > cause higher > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>system-wide CPU > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>contention. I find it easier to think through the > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>implications of ASYNC. I'm > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>surprised that MIXED loses time, and is less = > accurate than > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>ASYNC. Perhaps it > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>delivers more timer interrupts than the other = > approaches, > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>and each interrupt > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>event causes a small accumulated error? > > > >>>>>>>> > > > >>>>>>>>Overall I would consider MIXED and ASYNC as = > favourites and > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>if the latter is > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>actually more accurate then I can simply revert the > > > changeset that > > > >>>>>>>>implemented MIXED. > > > >>>>>>>> > > > >>>>>>>>Perhaps rather than running more of the same = > workloads you > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>could try idle > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated = > large disc reads > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>to /dev/null)? We > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>don't have any data on workloads that aren't = > CPU bound, so > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>that's really an > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>>>obvious place to put any further effort imo. > > > >>>>>>>> > > > >>>>>>>>-- Keir > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>_______________________________________________ > > > >>>>>>Xen-devel mailing list > > > >>>>>>Xen-devel@lists.xensource.com > > > >>>>>>http://lists.xensource.com/xen-devel > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > > > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 > > > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 > > > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru > > > >> > > > >> missed_ticks =3D missed_ticks / (s_time_t) = > pt->period + 1; > > > >> if ( mode_is(pt->vcpu->domain, = > no_missed_ticks_pending) ) > > > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > > > >>+ pt->do_not_freeze =3D 1; > > > >> else > > > >> pt->pending_intr_nr +=3D missed_ticks; > > > >> pt->scheduled +=3D missed_ticks * pt->period; > > > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > > > >> > > > >> pt_lock(pt); > > > >> > > > >>- pt->pending_intr_nr++; > > > >>+ if ( mode_is(pt->vcpu->domain, = > no_missed_ticks_pending) ) { > > > >>+ pt->pending_intr_nr =3D 1; > > > >>+ pt->do_not_freeze =3D 0; > > > >>+ } > > > >>+ else > > > >>+ pt->pending_intr_nr++; > > > >> > > > >> if ( !pt->one_shot ) > > > >> { > > > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct > > > >> return; > > > >> } > > > >> > > > >>- pt->do_not_freeze =3D 0; > > > >>- > > > >> if ( pt->one_shot ) > > > >> { > > > >> pt->enabled =3D 0; > > > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu = > *v, struct > > > >> pt->last_plt_gtime =3D hvm_get_guest_time(v); > > > >> pt->pending_intr_nr =3D 0; /* 'collapse' all > > > missed ticks */ > > > >> } > > > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { > > > >>+ pt->pending_intr_nr--; > > > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > > > >>+ } > > > >> else > > > >> { > > > >> pt->last_plt_gtime +=3D pt->period_cycles; > > > >> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xensource.com > > > http://lists.xensource.com/xen-devel > > > > > > = > ---------6184412361844123 Content-Type: application/octet-stream; name="el5u1-64-hvm-const-ltp.png" Content-Disposition: attachment; filename="el5u1-64-hvm-const-ltp.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAZoAAAGkEAAAAAB9jWAhAAAACXBIWXMAAABIAAAASABGyWs+ AAAACXZwQWcAAAGaAAABpADE8ZBCAAAjOElEQVR42u2dvYrlRvqHlQ5/lkUYs0GDzSoweJlg 1wq2YcAOVkGzyyQLAm/WwSI63ElWDY42GBAdG4NiZ0r6AhTYFyDwFegWdAvz79c15SrpqKSq bn28Uv2eoE+fL52q0vtIpfpS8AEA4ESwdwIAOBqQBgBHIM2JyfNgcv8WRZ7vk7KmSZL1th6G T4EdrJc3VtJ0XZ5HURCEYZZ13eW7WRaGQRDHTTP+/SzTC6ooaFtBkKZta/5NCiyJ3JHqm2O/ JN6NorGd0rZpav7mZRrHPj+1ffGdwHKvSWmaJtCoa/m+qSTHSq6uk4ReG9sz8/uo6/r5yfOi sMuDbU5VrqoqTZU0+jtLwUoa2i2UybIMwzgevhvHYViWdU2PYxpQaKgdQwFD26oq+rx5R+d5 HNcfETu6KCg05DeHvyS3Kx7773UdpbuqzL/ZT+PY56e2L8gyV2nqOgiqSuZSposC2fStYcnR FtLUtGfm9xEdGvTcxPHUgcyd/p4vS/XKGmccRtJQQMnjj9jN+rtlGQQiqLvusiAo9Ok4qF6n s5X4j47M5uNaktBxSSeK5Ctj31TbpWNq/z3STQTD2Dcv0zj2+antU2rFOcCuRKU049W0shwP p7GSiyJ5Flb7YWyL4/tInLnU87adEu856FuPIhLdE2mqSq86DDObplMFTQHQPzar/8ckU8Tx 8L0koSPVeCpIZplG/X/5TVVT1/83pfHy89Pbp/DPczofj+WkbUXVSFWElDRjZSeqhGkqgkyE PP3eZcnR4ax/CBtnfB81TRj2z7BFIQ8osjqe5yINTSOqqyoPZZkkXUd5pk+J1+gcSJ+is7Qo J4HcgtpzqkpK25mvPNvBSBodKggVuoQosqZR1YuRzPRO0mFIISBq2ebKABVhf8dJ6JvDo2r/ mD1Ualg9nEvj5eentz+15bYNwyiiqhHlXJwr5Cfp/CTDRWpIVUP5SL9C35dXAcOSozNi94Ss wJoY20ddJw5Lel7kFY+sCBZFGFKok14kgqicqjzQYayuKQcUE7KqSJVAOlNTukSFWnxeCCl+ T75DV8tCRdIniqYiYh6W0lDhyePfbwkN6Dgmjhum80b/HXHh2r/0HUIVEFELp1DRzw3im5dV LK7SkOAyEGQ1SX5SVI7EVYosjbKUlTA6w1dVkqizxLDkRHCLV6LILM7YPpJnOfVK24prKb0K S1sXksq9Tq9QfuhRHD7leU/lv23zXORZbT2K+q+oBgH9DKqqoM+BnTRUyRjbNSqE6bg3ro3+ eprScY8CJUnEkXP812Sxi1BTlRA6klFK+uc7vtLQMbr+DVFWqnom80UX/yKIs0zllY7hVIUy lRxth7YurkbMzSqX+4iEFNtVeZHXUuL81c+DCmWplJ5bdfZI06rSvyu3rq6WxqRRZ5ex60V7 mElDJ+rLipKe+aks6ztGP79Q9YQKtB4w/JXLIB3+kr4LL6+VlpXGdC02vuVgQP+YfPltveWM rjj0C/1hydF3pGJTzSqX+4iqXVJjqiZR2MorisvU9fN7mQf5PuksznoyJfIddbU0Js1cGdrC ShrKiqlXRb9gnw/IsSLSe2TGq230fTr3qLPc8Jde0hAwTONzGgLMuVeXydOfFK9VlV49EW1y QouxkptuopnaR5cqq4busTON3hxvOtOofSGvctQ7qinbkzMNHcPMNU1qNZL/6/XvXmZ+K9T+ bjZ9vl+U8kqgH9bDbz6/yXmYxuc0Ocs0X76aZeoqsCioxUl+st+oIhrUs0y9Qldz1K4kql1j JScDmLjsDLDbRzLf6lpK72KgsldpkLnUr8vUNvJcRUlfDXm1dPnOaa9pRCVAVZ5EOCWJyCpl lE711N+rLlD7R/N+C42qmZt3M7UZxfFlmxN9UxzJ5PFXjRZQnY9yl6s00hWD6KyUITyVxrHP T21flZP+TGy/3/LUr5LoZSGuXVRwytAV5WsqOVkm1LlpHgAzto+G+aarEfkaXTPqWzW1nvW3 QYKp/SN+RTQj68OClDSixVDUM07XekZFOKyVU6blTpJt+Ekid8ewx6IfkKIPQG9mHeOyd0P2 Hujf1H9JDnNRZxKVRtUTIHfKVBrHPj+9fZE6fYtq+5d9HPKTaniS+B3Zl0HEsdy2aEUzldyw R2Wcy300zHd/FMJ8P81Y9UymT/2KUELvf9EPHKSWOHOJa6GT9tMAsCwvu/TvA2mAF0AasCtU zbnkZVWetYE0ADhCY8+W2hakAcARSAOAI5AGAEcgDQCOQBoAHIE0ADgCaQBwBNIA4AikAcAR SAOAIwtJI9cCkWu4zK3ECMBxWUCapqHBcFKaLItjWu4njl8yNw4AviwgDa0uoqSR/9G02L0z B8AaLBTYY6pMrTcGwHHZSJquu7/Pnnj4yONjDQAzHh9lfFKs3t8b13fbRpr7+1evrp64/sjd Xb4I19fLbKfP1dUaW725ub1dfqtv375+vfxWb29vbtYog3VKdqkouLuT8Umx+urV/f0m0piu abLs6mqZX+qzzq2BPvlkja2ucaeUDx8eHq6vl98qLUSxRhmsU7LrRMHVlakpa3FpxlvPIA2k ISCNvplgup9mnV27Fmve2m5p1grvdThSyV5fPzyMv7NRszCkWQtIsxaQxokj7VpIsxaQxokj 7VpIsxa7S/P4eHe3dyHYw3sFrz5dt+wtX9flSCV7d/f4OP7ORtIc63gIwFRrJ6QBYBRIA4Aj kAYARyANAI5AGgAcgTQAOAJpAHAE0gDgCKQBwBFIA4AjkAYARyANAI5AGgAcgTQAOAJpAHAE 0gDgCKQBwBFIA4AjkAYARxhIc3tLS0zvXRAA2ECxShE7/u5m0ogFtfcuDABsoFi9udldGggD jgWD6hmkAccC0gDgCKQBwBFIA4AjkAYARyANAI5AGgAcgTQAOAJpAHAE0gDgCKQBwBFIA4Aj kAYARyANAI5AGgAcgTQAOAJpAHAE0gDgCKQBwBFIA4AjkAYARyANAI5sKE3XZVnwRJZ1nXoV 0oCjsaE0WRbH3RNxnGXqVV2aritLXahx2jbPi+Ll6cGqnuB5bChNEIifqutA27aUpmlIqixL U/079G6aJon4ZtsWRRwnSVnGcVmKT5BoaRqG8rl6pWnUdpqmKJRo9FtRlKb9cx4Admwmja6K 1Ee8/p//lGXyRFWJBFEokyxJQhKVZdO0bZKQTnFcFG1Ln6LzVVXlefxEnpMeaUpnKdKFXmnb poki+mZZCkHoMU1pu7RV8VskH63NS7+ZJFFE35QatW1dFwWJTJ+hX6x/oyzzXEmqXm9b+tQx Rax7UNlKkt+IojCkvSJKz18YSPPq1R/+8Je/0Aq5d3cU7jc3b95QUL5/LwQpS3qsqv7zpvn6 6/fvKUDF8677/PPPPiOpxPMPH7777ve/T5I8f/dOfu/770kX+T49UnDc3Lx/T+kpy19+yfMw /L//e/OGQoTSQ6+/f//mDQkl0ifSSaJ9+im9TkEkXv/6a/G9L78Uj59//ssvKr30KHLt+vju 3bff0vn2yy+/+oq2++mnQUDBK55H0aefitflc/k5eozjIPj00y++CALxPfW5y8ebG/odyp94 fPfu4YEEevfu8VFPz/v333xD5/svv/z226qi9x8eyifevfvhB5v8yOXu5x9//JGU/eoJSvlX X/3jH7SNhydovz8+/vij/vnHx3//m9L97bfffEP5CYLf/Y7+fvGFyB9tx/RI3/vmm7HXxfbp kuDbb6lcrq//9Kc//nF3ab7++n//o93zww+Pj3ScbhpxrOb++NNPQgrT+9999+c/03/qdSGP 6+Pt7X//W9dNY/+9n3+m3fzc37N9/Omn77+nYLu9FQeN29u//51k+utfxUHjn/+8u6uqfrrp /Ny24vn8448/Zhl9Sz0vijy/u/vXv2j7b96Ig9abN3d3VIX/7DMqJ1JNyGT/O7/8Ir7388/D 36c4pedUg6DtkrL393/7G5NrmjNS10ny8sqauTLAG1HJo4q1fKVpgkAEpO0WbGJDVo23yxmr 1rPz0TQv1+ao0kjoelL8lz3hsre5xgb6aVamaehQ8ZItHF2arksSaqpp2yj68CGK7EuDa2xg RMAGOXyZNkeXhg4cUURVNTrj6O2Oc3CNDUizAdS0/Xxtji+N6CKg84w839jBNTYgzSZQL9Fz v3sGaUQ/l/gvSWy/wzU2IM1G0EXw8755DmkUkObFcC2Y5aHxCM/53vmk0Qc4TcE1NiDNZtDg m+d872zS2OeHa2xAms2o6+dV0CANNyDNhsTxc74FabgBaTbkedI891qIK3luO0qaa2xAmg1J U9tLYB05m+gs0OBOu09yjQ1IsyEuveGKs0ljv8e5xgak2ZDn5RXScAPSbAjNQHX/FqThBqTZ lOc0BSTJlnNF1qeu++tATH2SZ2xAmk15zuya55ydOGN/vuUaG5BmU57T53I2aexzxDU2IM2m 6AtJ2QJpuAFpNoVWXHP9DqThBqTZGPemgPNJE1jGFq09s3dax4A0G+PeFHA+aWxzZD92YFsg zca4NwVAGm5Amo1xn1UDabgBaTamaWy79iRnlMauuxbSGPBNGlr5y+3z55Mmy+yqqJDGQF1f X9PKvHsXxHa4NgWcr2xsr+s4SkOxen29uzT8CmZdXJsCfJaG5009GJxpfJPGtSnAZ2l4ju+G NJvj2hRwRmns9jmkMeCfNK5NAeeTxnafQxoDPkrj1hQAabgBaXbALRggDTcgzQ7QbSfsP31G aezGekMaAz5K07YuTQHnk8Z27iakMeCjNG5NAa4jCI4ApHkRfkrjsliG7eyTIwFpXoSf0rj0 dUMabkCaXXBpCjijNHZ5gjQG/JTGZdnAM0pjN+kb0hjwUxqXNrEw3Dute+Ue0hjwVRp7Fc7X 5AxpXoiv0tircE5pbFoPIY0Bf6WxvVfNGaWxm7sJaQz4Ko19QJxRGrvcH16a+lfcl++ew19p bG/wdEZp7JYBPKw0XVcUcRz8RhTl+ZK3fvBVGvuemjNKY7fXDypNnpMkVSXr321L2Y3jLFvq nOOrNPb5Pqc0NkNWDyqN6c22Xeps46s09t2bZ5TGLlcHlaZP0yx/Ty5fpbGX4ZzS2IzdPrQ0 ZUldcVVF1zSu9ydWjYtdl2W0hX7Vzl9pbLs3zymNzUCaQ0sTRaRKHFdVXbvM7miaPA8C+QNZ FsfdE3RFpD7jrzR+n2ls1kk4tDQ0ZLDrxMBBl+GDtMKXkkb+V9f6NnyWxq5785zS2HRvHlqa KKrroqD2jrZ1HXM7pooSyWdpbEPinNLY5D5NDyxNWdK1CE2biiLXNe8hjQnb7s1zSmPTvZkk B5aGrk5EVaIozDXRoId6dU6aq6tPnkg+Yjsi6/jYdm+eUxqbgyUfaZpGxifF6tXVymPPcE1j wjbnZ5Vmvs7CR5o+M2ea4ALXXYjWMxO23ZvnlMYmXweVRgzTzPMwLApqDAhD15sfKGnQTzPE b2nmOy8OKo3MnvyQWz+NDT5LY1eWZ5Vmvnvz0NIEhv+XwGdpbPoquu6s0sx3b7rfPn4brKQJ Q5xp1sCm/ayuzyrN/CGDa86tpKFrmrKsaxqDtnSI+ywNNYrMfea80swvmMg155b9NDSvRkxA WzoBPktjs8DEeaWZP89yzTnWCNiV+VEB55Vmvsmda86tpwZc9vUvg9/SNM3cnVrOK818+xnX nFs2OdNFm2TZBPgtzXyz85nLJ02nK6eHlmbN1YTPHBQ2zF3VnLl85poCDi1NFC0/zVly5qCw YW6I/JnLZy5vh5aGxnhSkzOqZ8sz1xRw5vKZ67g9tDRJMjbofxnOHBRL5P/c5TN9RXdoadbk 3EFhk//p9rNzl0+aTs2fOrQ0XUdjAoJg+fEAZw8KG6ZD49zlM105PbQ0tKamGEYTxxhGszTT VZRzl8907g4tTRjK1rO2Xfq+XOcOCht8PtOcWJogkEO05UJOy3HuoLBhuqfm3OVzYmnSNMto x7ZtlrmuRjPHuYPChumemnOXz/QgoUNLQ0PYRXMzzfJfNgHnDgobpi+Gz10+0z01h5aGaJq6 XmN5pXMHxctL4Ozlc2JpaBVnqp65Ln8+z9mDwqYEpnpqzl4+p5WGpqDR0Lq2RZPzGkwFx9nL 57TSqCbnrkOT8/JMlenZy2dqTs2hpUGT87r4fKaZiqdDS6M3OS+dkbq+vV1+7PSxmFrB+uzS HK16RrFKETv+bq/JWY5zXqPJ+eYmz88dGHNM9dRAGl5QrN7cWDY5032d0eS8DpDG/b09wdQA BkxJc/az8JQYSzc6LYX1umc0NWCNhUIhje/SmOsva65N8RKspwbQvZ1pKVGMPVueqQUmzi7N 1OK0h5ZG9NPot6tdDkgzLcbZpZk6yx5aGnVfZ0izBlMLtEIaflj209COoy5OVM/WYKoM5pcJ PzanlYb6aWgBdLp5IBoClmdamnN3/E4dFA4tDYF+mvXwWxpz3udvRLJXmh36aZpm+ZU2IY3f 0kzl/dCdm3TXALpcpQra0jNqIM30jBpIww/LtZxJFeqrwe0D12Bqpjyk4Yd1k7NsbEaT8xpA mjEOLQ3dEr0oqLG5bSHNGvgsjbkT49DSlCVdzVDTYBShn2YN/JVm6iaCh5aG2s1Ec3NRoJ9m DfyVZirvB5UmTce6nuhOkUv12UAaAtK4vbMvM9LQBOcwzLKiEDd0KgpamSZJltuRkIYwzxyB NPywqJ61bZ7L6c5JkufLjguANITP0pj7/Q8szbpAGsLn6pm5PRbSGIA0hM/S4EzjDKQhfJbm lNc06wJpCEjj9s6+rCoNtbTRnTqzjHp3aAobNSaIZxJIQ0Aat3f2xUqaLKsq905NGhst7jUQ xzSOIMtoqUG6140+phfSEOZGfB+kMU04Obg0dMZI06JwmU9DN7aV/1ELSRCInxLP1KcgzdSa LOeXJk1NOTy0NATdmyZNw/A5s+momqarIvUhIA1h3g3mkDoL5rwfXhoKburidJVGjCmg+6iZ pHn79uGJ+iNLj207BlOBA2n2p+tkfFKsvn1rIQ3JEkVzVzZBD/FTeU4X/lSpM0vz+vX1E/lH lp9QfQQgzXje906bhEbFCChWX7+2WiyQlClLt4CmS/40Vd/BNY0ZSDOe973T5priXvWMrmio OYDksd90vyqH1jMz5oWMfJDGFAEHl4ag8I4i+5Xc5SBPWV1DP42ZqcBZY+EsTpjXFz20NKIJ IIrWWO0R0hDHC5zlMEcA17xbSRPHRbHW8Q7STJcC18DxOe8Ye8aC4wXO+nlvGq55x02dWOC3 NOONS1Orwe0LburEAp+lMa1Hc3BpcFOntTEvTMs1cJbklNLgpk5rYw4QroGzJOPdGAeXBjd1 Wh+fpTnlmQY3dVofv6UZ69A4uDQEbuq0Lj5LMx6EfCPD8kyj/rMfe2YH36LZFr+lGbvrEd/I sBwRILSpqudNQpuCb9Fsi8/SjMcA38iwvGtAHDcNzdssiqUTwLdotsU0EBbS8MPymoZutmFe AOEl8C2abfFZGmpounyVb2RYNwTQ2WaNqch8i2ZbfK6ejeeSb2TMSBOMsGwC+BbNtvgtzdid XPlGxow09QjLJoBv0WyL39Kc6kxDNI2olsm7oS0L36LZFr+lSdPL2OIbGTPS0Ix+sSCGmKq8 /HUN36LZFr+lGQtDvpExIw2NNSNNmoZuVEtdm+jcXAfTAhpL94vxZKx7k29kzEgjJgWolWW6 zn5hDTv4Fs22mBamXbrhhSdjUcA3MmZbz8STOJYZQOvZOph2BKThh9WZhmbRiI+1Lc406+C3 NGPdm3wjY0YaqpbRrEKpCq5p1sJvacYaPPhGxmzrGbWahaG4TKOlApceSlPXNzfmpfL8AdIM X+EpDcXqzY3DEk5rrOlf17e3y3eZHg/TQox+NDmPHRx4SkOxShFryMVWieBYNNtTFOPl4Is0 lwNp+EYGFgtkgqkcfJHmKNUzAtIwAdIMX+EbGZCGCZBm+ArfyMAaAUwwLRcIafiBNQKYYFru 2x9phuOc+UYG1ghgg9/SXI694xsZWCOADX5LcxmIfCMDawSwAdL0X+EbGVgjgA1j8+R9lsbU 3bs/WCOADZBm+ArXyLCsnlUVfYxujL50AiCNxPfq2bCR6eDS0K3QaThh26rJaEsBaSR+S3NZ GTu4NGpCAKY7r4ff0lzGwcGlods5if9wJ7T1GL9LC6Thh+Wd0LKMzjVtm2VL70RIIxlfWgPS 8MP6TmiiuRnrnq3H+K7wWZrxaXn7gzuhscF3aYZ3czWH5t5YnmnyPAxprYDlAxzSSPyW5nLA 6sGloTVpyrKuaSgNmpzX4rKngvPNWpfnZNKoJmese7YeVTW2YB6k4QeanNkwvsokpOGHc5Pz 8HLtpUAaCaTpPz+4NOKGG+5NzmUZRUEQRWLEmlh4MAiyTN8GpJH4Ls1wwOrBpSGaxrXJuW3p 9hy044OAzlNZRsqRgPpseEgjGRPEp9IZXi0fVpqXTA3oOvFZkob+k4uo03P1KZ/CYppjLQK+ PKepnr10EhoJQhU0Ekh9U+ojPuFPWMwBaXQOK80StG2eU+XOJM3V1SdPJB9ZY9TBUYA0Opyk oa5XAcXq1ZX1umdtO72whvl8VFVRhDPNPJBGh5M0tikL5AfCUHwkz4NA3IHTdtNynmdVkTC4 ppnjKCvnr8NwtaPDSkPKlKUUpa5dhtFI3Wi+J/XuoPVsjsuFGH0qnTTth+JhpRHTnBV1Pb78 g2nj1E8ThqJnBv00c/hdPRuG4mGluawwYBjNekCaqed8mJWmfw2DsWdrAmmmnvNhRpo07e+0 PMfYs/VIkuOsMrk8p5GmaeiKRPSdNA3d5XnpfhSfwmKOIy3Nukbu+6vqHVYa0a0j+17W6Hr0 KSzm8F2aYa3msNIQNAhmjTs7Ez6FxRx+SzPM68GlWROfwmKOy/VXfCodSGONT2Exx+VKXz6V DqSxxqewmOOyLHwqHUhjjU9hMQek0Z9DGiM+hcUcl3d49ql0hjNXh2PR+ABpGHE5d9Ov0unn /rKrlwuQhhVDafjeQm/93EMaI5BG53JxCZ9KB9JYAml0Lmcv+lQ6kMYSSKOTJP1xF35J05+E B2mMQBqd4Y2d/JKmP+0E0hiBNDqXw+OXv582X4bVs73TYwLSsOI4w+PXANJYAml0hrfbgDQc gTSsOM5QkjXoL+IEaYxAGp3hnSf9kqbfDAJpjECaPv1Q8Uuafm4hjRFI06ffVwFpOMJAmutr Wlh674LgQn8xRkjDDYrV6+vdpcGZRsfv6pkeCzylEemENKzotyD5JU0/FiCNEUjTp79DIA1H IA0zII0E0hiBNH3yvCj0Z35Jo/dSQRojkKZPvzz8kqY/3RvSGIE0fXyWpi8KpDECafq0rR4s kIYjkIYd+pgA36TR8w5pjECaIfqwRd+k0RcWgTRGIM0QfU6Nb9KgemYFpBlC97+W/0MajkAa hsSxHEozXGjj7EAaKyDNJWqlAL4rsqyDnl9IYwTSXEK3bKx/JY79kkYPR0hjBNKMkST5r6xx l1POQBorIA1QQBorIA1Q6Ou+QRojkAYoylJFw/AOCnyANIARejQEG0WgO5AGMALSWAFpgALS WAFpgIJ6qOT/HkvTdWEofqLrsix4Isv0GxdBGqADaX79gSAQP5Flcdx1NBxRv/E3pAE6kOZD UdS1lEY+0ivqE5AG6ChpPO2naRpaXUTIoqsi9SEgDdBRUeKlNFQRoyHuc9K8ffvwRP2R/o1a gW+ouUScpOk6GZ8Uq2/fLiZN0EOtzDsnzevX10/kH9EXZQX+wbN61rYyPilWX79e7UyTJH2J cE0D5uEpTZ8N+mnQegbsUQvAQ5oP6KcBNqSpDEivpZkG0gAdFZCQxgikATqQxgJIA3QgjQWQ BujoK/HsnRZzGiENYERRyHiANEYgDdBR8QBpjEAaoANpLIA0QAfSWABpgA6ksQDSAB014RnS GIE0oA+kmQXSgD6QZhZIA/pAmlkgDegj525CGiOQBvSRUxQhjRFIA/qgejYLpAF9IM0skAb0 gTSzQBrQJ0nEdHhIYwTSgD7iDs/6UujcgDSAGSIk6xrSGIE0oA+kmQXSgD6QZhZIA/rkeVVB mknq+vaWFpbeuyAAF8Qa4FyloViliB1/dzNpbm7kUukAyLoHV2koVm9udpcGwgAdKQ3fuGBQ PeNbOGAPIM0snAsH7AGkmYVz4YA9EFcznOMC0gB2QJoZOBcO2AdIMwPnwgH7AGlm4Fw4YB8g zQycCwfsA60SwDkuIA1gB61HwzkuIA1gB6pnM3AuHLAPkGYGzoUD9gHSzMC5cMA+0CoBnOMC 0gB2UFByjgtIA9gBaWbgXDhgH2jCM+e4gDSAHVWVZZzjAtIAhkQR57iANIAhWVYUfONid2ke H+/u9i4Ee5pm7xTY03Vtu3ca7OmXbFXFMV9p7u4eH8ff2Uiah4fr670LwR6eK6SMc6xzeL9k uy4I+Kb++vrhYfwdSDMCpFmLYcmmKd/UrypNkgQfoQLouiyj/7NM3EpBAGnW4tjSlCXf1K8q TRDoF0xZFsfdE3GcZepVSLMWx5ambfmmfkVp2jbobUMqVNf661l2dbVGxtYJ708+WWOr5taY l7DO4WgtFS9LdokyWScKrq70w77Oi6UhOYgwpAqZrop+BoI0kIZYp2QPKE2eU6Nn21KFzCTN /f2rV1dPXH/k7i5fhOvrZbbT5+pqja3e3NzeLr/Vt29fv15+q7e3Yu3tpVmnZJeKgrs7GZ8U q69e3d8vJE3QQ3+nqkiTcWm67v4+e+LhI4+PNQDMeHyU8Umxen+vN2UteqYpS6lGVYWh6ZoG gPPw4sAuijAkTah6RvXg8dYzAM7DAmeDPI+iIIgicek43k8DwHlAFQoARyANAI5AGgAcgTQA OLKBNLybBmhOh566shTNGmUp3s/zMOSV9rYV7ZWXJTtM+76Ikg1DObagX5K8oqJp0pRSk6Zi xo9InUp7v2Q3kIZzIzSNnCsK0WCepqKviQKyrsOQ7nNPhdW2XZemfNJOu1NI0y/ZYdr3hVIh uiKiiEp4WJK8okJOhstzWmGaUpemNL1PaDIs2Q2k4dzdqUbZitQliTou0ogm+ZxP2mkEmyzR fskO074vw5kyw5LkFRVyMlye91PXjwL5fPUEmwbWcIN6m/QUinSr5zzSLuafiLQMS3aY9n2h qo2snlEFrF+S3KKCZFFzwlTqhlHw8fnayeFWPGNQDTYMqTbLW5qmoSrNMaShAJQVXyW6TCu3 qKgqUrvraC02SGMBXaBmmViegrc0caxXy7hLI6uINJCXtzR0ZSubJ4KgbXeXhlvtdQhdkKo1 Unhf0+jjy3+tWzO+poljJQ0N5OV8TUNpUNJQuna+puHWTtKHdqje5HmE1jO1Szm3npVlv3rG u/VMTKEU1XSRup1bz3i1yPdRF4BydhD/fholDe9+mv5A3qP109DzHftpADgXkAYARyANAI5A GgAcgTQAOAJpAHAE0gDgCKTZBLVIvEAf9OLG1Dqddr3rdT3fM5KmR7rrzdZAmg3Rg9omdC+h HvSpd21EtNG1rvcfiMMXSLMhLx9ntcTt9uzOcTToZatyORqQZkMux/Y2TZKUpRheIgZuiIEa w2EckjgWo8rkt5JEDEbRB0OqbYpXh8M65ef7w1xo2oEYSCLOgFnG9yYYewNpNuRSGnpFBLB6 pHfFuFo52FFBA9fFIwU3fZsexSracvtym/SqPkeoP76YfokElb8hBKURYULbokAFzQSk2ZBx afR3xCONrhWfEsPqL78/nPA8lEacLfrD3PvSRJEcXVxV9GtqTtFlWkEfFMyG2Eoj7/kzvDOD vTT6p8alGd79oW2zLIrCUI7zhTRmUDAbYitN06hJUfpF+8ukETMo1ZlGzrlpW9JEtObRhF9x boM0ZlAwG2IrDV3TyElRtLCUQlyluElDFS9x7SKlIV1o2Qshi2h8kI0OYoERXNNMAWk2xF4a 0Xp2OUVLtp65SCMnUFWV0IDaz2hOpan1TGiJ1jMzkOZQLNFPYwf6acxAmkMxPSJgOaqKw8x9 rkCagyFW5lobjD2bAtIA4AikAcCR/wcHSqeKqv8QEwAAADx0RVh0Q29tbWVudAAgSW1hZ2Ug Z2VuZXJhdGVkIGJ5IEVTUCBHaG9zdHNjcmlwdCAoZGV2aWNlPXBubXJhdykKldNUtQAAACJ0 RVh0cHM6SGlSZXNCb3VuZGluZ0JveAA0MTB4NDIwKzgxKzE4NBeLyiUAAAAcdEVYdHBzOkxl dmVsAEFkb2JlLTMuMCBFUFNGLTMuMAqbcLvjAAAAAElFTkSuQmCC ---------6184412361844123 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ---------6184412361844123-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 30 Jan 2008 10:25:00 -0500 Message-ID: <47A096CC.2070907@virtualiron.com> References: <20080129153458531.00000002384@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080129153458531.00000002384@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "akira.ijuin@oracle.com" , "xen-devel@lists.xensource.com" , "deepak.patel@oracle.com" , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Dan, I agree with you on disabling and the need for fixing hpet. If no one else wants to fix it, I could give it a try. I think we could make the hpet work better than pit/tsc. While I monitor the skew over time, I'm not as methodical as you. Its possible that I missed this behaviour. I've been thinking about what you've seen and have no clue as to the cause. Let me ask you a few questions: Is the graph for RHEL5u1-64? (I've never tested this one.) What was the behaviour of the other guests running? If they had spikes, were they at the same wall time? Were the other guests running ltp as well? How are you measuring skew? Are you running ntpd? Anything that you can discover that would be in sync with the spikes would be very helpful! The code that I test with is our product code, which is based on 3.1. So it is possible that something in 3.2 other than vpt.c is the cause. I can test with 3.2, if necessary. thanks, Dave Dan Magenheimer wrote: >Hi Dave (Keir, see suggestion below) -- > >Thanks! > >Turning off vhpet certainly helps a lot (though see below). > >I wonder if timekeeping with vhpet is so bad that it should be >turned off by default (in 3.1, 3.2, and unstable) until it is >fixed? (I have a patch that defaults it off, can post it if >there is agreement on the above point.) The whole point of an >HPET is to provide more precise timekeeping and if vhpet is >worse than vpit, it can only confuse users. Comments? > > >In your testing, are you just measuring % skew over a long >period of time? > > We are graphing the skew continuously and >seeing periodic behavior that is unsettling, even with pit. >See attached. Though your algorithm recovers, the "cliffs" >could still cause real user problems. I wonder if there is >anything that can be done to make the "recovery" more >responsive? > >We are looking into what part(s) of LTP is causing the cliffs. > >Thanks, >Dan > > > >>-----Original Message----- >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>Sent: Monday, January 28, 2008 8:21 AM >>To: dan.magenheimer@oracle.com >>Cc: Keir Fraser; xen-devel@lists.xensource.com; >>deepak.patel@oracle.com; >>akira.ijuin@oracle.com; Dave Winchell >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Dan, >> >>I guess I'm a bit out of date calling for clock= usage. >>Looking at linux 2.6.20.4 sources, I think you should specify >>"clocksource=pit nohpet" on the linux guest bootline. >> >>You can leave the xen and dom0 bootlines as they are. >>The xen and guest clocksources do not need to be the same. >>In my tests, xen is using the hpet for its timekeeping and >>that appears to be the default. >> >>When you boot the guests you should see >> time.c: Using PIT/TSC based timekeeping. >>on the rh4u5-64 guest, and something similar on the others. >> >> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >> > 14.318MHz HPET.) >> >>This appears to be the xen state, which is fine. >>I was wrongly assuming that this was the guest state. >>You might want to look in your guest logs and see what they >>were picking >>for a clock source. >> >>Regards, >>Dave >> >> >> >> >>Dan Magenheimer wrote: >> >> >> >>>Thanks, I hadn't realized that! No wonder we didn't see the same >>>improvement you saw! >>> >>> >>> >>>>Try specifying clock=pit on the linux boot line... >>>> >>>> >>>I'm confused... do you mean "clocksource=pit" on the Xen >>> >>> >>command line or >> >> >>>"nohpet" / "clock=pit" / "clocksource=pit" on the guest (or >>> >>> >>dom0?) command >> >> >>>line? Or both places? Since the tests take awhile, it >>> >>> >>would be nice >> >> >>>to get this right the first time. Do the Xen and guest >>> >>> >>clocksources need >> >> >>>to be the same? >>> >>>Thanks, >>>Dan >>> >>> -----Original Message----- >>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>*Sent:* Sunday, January 27, 2008 2:22 PM >>>*To:* dan.magenheimer@oracle.com; Keir Fraser >>>*Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>akira.ijuin@oracle.com; Dave Winchell >>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables >>>pending missed ticks >>> >>> Hi Dan, >>> >>> Hpet timer does have a fairly large error, as I was trying this >>> one recently. >>> I don't remember what I got for error, but 1% sounds >>> >>> >>about right. >> >> >>> The problem is that hpet is not built on top of vpt.c, >>> >>> >>the module >> >> >>> Keir and I did >>> all the recent work in, for its periodic timer needs. Try >>> specifying clock=pit >>> on the linux boot line. If it still picks the hpet, which it >>> might, let me know >>> and I'll tell you how to get around this. >>> >>> Regards, >>> Dave >>> >>> >>> >>> >>> >>> >>> >>-------------------------------------------------------------- >>---------- >> >> >>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>> *Sent:* Fri 1/25/2008 6:50 PM >>> *To:* Dave Winchell; Keir Fraser >>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>> akira.ijuin@oracle.com >>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>> >>> >>that disables >> >> >>> pending missed ticks >>> >>> Sorry for the very late followup on this but we finally >>> >>> >>were able >> >> >>> to get our testing set up again on stable 3.1 bits and have >>> seen some very bad results on 3.1.3-rc1, on the order of 1%. >>> >>> Test enviroment was a 4-socket dual core machine with 24GB of >>> memory running six two-vcpu 2GB domains, four hvm plus two pv. >>> All six guests were running LTP simultaneously. The four hvm >>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. >>> Timer_mode was set to 2 for 64-bit guests and 0 for >>> >>> >>32-bit guests. >> >> >>> All four hvm guests experienced skew around -1%, even the 32-bit >>> guest. Less intensive testing didn't exhibit much skew at all. >>> >>> A representative graph is attached. >>> >>> Dave, I wonder if some portion of your patches didn't end up in >>> the xen trees? >>> >>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>> 14.318MHz HPET.) >>> >>> Thanks, >>> Dan >>> >>> P.S. Many thanks to Deepak and Akira for running tests. >>> >>> > -----Original Message----- >>> > From: xen-devel-bounces@lists.xensource.com >>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>> > Dave Winchell >>> > Sent: Wednesday, January 09, 2008 9:53 AM >>> > To: Keir Fraser >>> > Cc: dan.magenheimer@oracle.com; >>> >>> >>xen-devel@lists.xensource.com; Dave >> >> >>> > Winchell >>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>> > disables pending >>> > missed ticks >>> > >>> > >>> > Hi Keir, >>> > >>> > The latest change, c/s 16690, looks fine. >>> > I agree that the code in c/s 16690 is equivalent to >>> > the code I submitted. Also, your version is more >>> > concise. >>> > >>> > The error tests confirm the equivalence. With >>> >>> >>overnight cpu loads, >> >> >>> > the checked in version was accurate to +.048% for sles >>> > and +.038% for red hat. My version was +.046% and +.032% in a >>> > 2 hour test. >>> > I don't think the difference is significant. >>> > >>> > i/o loads produced errors of +.01%. >>> > >>> > Thanks for all your efforts on this issue. >>> > >>> > Regards, >>> > Dave >>> > >>> > >>> > >>> > Keir Fraser wrote: >>> > >>> > >Applied as c/s 16690, although the checked-in patch is >>> > smaller. I think the >>> > >only important fix is to pt_intr_post() and the only bit of >>> > the patch I >>> > >totally omitted was the change to pt_process_missed_ticks(). >>> > I don't think >>> > >that change can be important, but let's see what >>> >>> >>happens to the >> >> >>> error >>> > >percentage... >>> > > >>> > > -- Keir >>> > > >>> > >On 4/1/08 23:24, "Dave Winchell" >>> >>> >> wrote: >> >> >>> > > >>> > > >>> > > >>> > >>Hi Dan and Keir, >>> > >> >>> > >>Attached is a patch that fixes some issues with the >>> >>> >>SYNC policy >> >> >>> > >>(no_missed_ticks_pending). >>> > >>I have not tried to make the change the minimal one, but, >>> > rather, just >>> > >>ported into >>> > >>the new code what I know to work well. The error for >>> > >>no_missed_ticks_pending goes from >>> > >>over 3% to .03% with this change according to my testing. >>> > >> >>> > >>Regards, >>> > >>Dave >>> > >> >>> > >>Dan Magenheimer wrote: >>> > >> >>> > >> >>> > >> >>> > >>>Hi Dave -- >>> > >>> >>> > >>>Did you get your correction ported? If so, it would be >>> > nice to see this get >>> > >>>into 3.1.3. >>> > >>> >>> > >>>Note that I just did some very limited testing with >>> > timer_mode=2(=SYNC=no >>> > >>>missed ticks pending) >>> > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the >>> > worst error I've >>> > >>>seen so far >>> > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. >>> > >>> >>> > >>>Thanks, >>> > >>>Dan >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>>>-----Original Message----- >>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>> > >>>>To: dan.magenheimer@oracle.com >>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>> > xen-devel@lists.xensource.com; Dong, >>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>> > >>>>disables pending >>> > >>>>missed ticks >>> > >>>> >>> > >>>> >>> > >>>>Dan, >>> > >>>> >>> > >>>>I did some testing with the constant tsc offset >>> >>> >>SYNC method >> >> >>> > >>>>(now called >>> > >>>>no_missed_ticks_pending) >>> > >>>>and found the error to be very high, much larger >>> >>> >>than 1 %, as >> >> >>> > >>>>I recall. >>> > >>>>I have not had a chance to submit a correction. I >>> >>> >>will try to >> >> >>> > >>>>do it later >>> > >>>>this week or the first week in January. My version of >>> constant tsc >>> > >>>>offset SYNC method >>> > >>>>produces .02 % error, so I just need to port that into the >>> > >>>>current code. >>> > >>>> >>> > >>>>The error you got for both of those kernels is >>> >>> >>what I would >> >> >>> expect >>> > >>>>for the default mode, delay_for_missed_ticks. >>> > >>>> >>> > >>>>I'll let Keir answer on how to set the time mode. >>> > >>>> >>> > >>>>Regards, >>> > >>>>Dave >>> > >>>> >>> > >>>>Dan Magenheimer wrote: >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>>>Anyone make measurements on the final patch? >>> > >>>>> >>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>about 0.2% with no load. This was xen-unstable tip today >>> > >>>>with no options specified. 32-bit was about 0.01%. >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>>>I think I missed something... how do I run the various >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>accounting choices and which ones are known to be >>> >>> >>appropriate >> >> >>> > >>>>for which kernels? >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>>>Thanks, >>> > >>>>>Dan >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>>>-----Original Message----- >>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>> > >>> >>> >>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>> >>>>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>Keir Fraser >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>> > >>>>>>To: Dave Winchell >>> > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>> > Eddie; Jiang, >>> > >>>>>>Yunhong >>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>> > >>>>>>disables pending >>> > >>>>>>missed ticks >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>Please take a look at xen-unstable changeset 16545. >>> > >>>>>> >>> > >>>>>>-- Keir >>> > >>>>>> >>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>> wrote: >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>Keir, >>> > >>>>>>> >>> > >>>>>>>The accuracy data I've collected for i/o loads for the >>> > >>>>>>>various time protocols follows. In addition, the data >>> > >>>>>>>for cpu loads is shown. >>> > >>>>>>> >>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>> >>> >>processor AMD >> >> >>> box. >>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>> > >>>>>>>The cpu load is usex -e36 on each guest. >>> > >>>>>>>(usex is available at >>> http://people.redhat.com/anderson/usex.) >>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>> >>> >>of=/dev/null. >> >> >>> > >>>>>>> >>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>> > >>>>>>>All three guests are 8vcpu. >>> > >>>>>>> >>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>> > >>>>>>>except that the redhat-64 guest has 4 instances of dd. >>> > >>>>>>> >>> > >>>>>>>Date Duration Protocol sles, rhat error load >>> > >>>>>>> >>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, >>> > +.005% cpu >>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, >>> > +.012% cpu >>> > >>>>>>> >>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, >>> -.004% cpu >>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>> >>> >>-.005%, -.005% cpu >> >> >>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>> >>> >>-.008%, -.003% cpu >> >> >>> > >>>>>>> >>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>> >>> >>-.040% cpu >> >> >>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, >>> > -.031% cpu >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, >>> > -.09% i/o-8 >>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% >>> > -.14% i/o-8 >>> > >>>>>>> >>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, >>> > -.022% i/o-8 >>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>> >>> >>-.017%, -.018% i/o-8 >> >> >>> > >>>>>>> >>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, >>> > -.029% i/o-8 >>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, >>> > -.031% i/o-8 >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>> >>> >>-.04% i/o-32 >> >> >>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, >>> > -.005% i/o-32 >>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>> >>> >>-.11% i/o-32 >> >> >>> > >>>>>>> >>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, >>> > .003% i/o-4/32 >>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, >>> > .01% i/o-4/32 >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>>Overhead measurements: >>> > >>>>>>> >>> > >>>>>>>Progress in terms of number of passes through a fixed >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>system workload >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>> > >>>>>>>The workload was usex -b48. >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>>Conclusions: >>> > >>>>>>> >>> > >>>>>>>The only protocol which meets the .05% accuracy >>> > requirement for ntp >>> > >>>>>>>tracking under the loads >>> > >>>>>>>above is the SYNC protocol. The worst case >>> >>> >>accuracies for >> >> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>SYNC, MIXED, >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>and ASYNC >>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>> > >>>>>>> >>> > >>>>>>>We could reduce the cost of the SYNC method by only >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>scheduling the extra >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>wakeups if a certain number >>> > >>>>>>>of ticks are missed. >>> > >>>>>>> >>> > >>>>>>>Regards, >>> > >>>>>>>Dave >>> > >>>>>>> >>> > >>>>>>>Keir Fraser wrote: >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>> >>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>> wrote: >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>>>Since I had a high error (~.03%) for the >>> >>> >>ASYNC method a >> >> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>couple of days ago, >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>I ran another ASYNC test. I think there may have >>> > been something >>> > >>>>>>>>>wrong with the code I used a couple of days ago for >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>ASYNC. It may have been >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>missing the immediate delivery of interrupt >>> >>> >>after context >> >> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>switch in. >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>acceptable accuracy, >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>each running consistently around or under >>> >>> >>.01%. MIXED has >> >> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>a fairly high >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>error of >>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>threshold for comfort. >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>> >>> >>plan to leave >> >> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>SYNC running >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>running instead. >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>It may be too early to pick the protocol and >>> >>> >>I can run >> >> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>more overnight tests >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>>next week. >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>I'm a bit worried about any unwanted side >>> >>> >>effects of the >> >> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>SYNC+run_timer >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>> >>> >>cause higher >> >> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>system-wide CPU >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>contention. I find it easier to think through the >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>implications of ASYNC. I'm >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>surprised that MIXED loses time, and is less >>> >>> >>accurate than >> >> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>ASYNC. Perhaps it >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>delivers more timer interrupts than the other >>> >>> >>approaches, >> >> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>and each interrupt >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>event causes a small accumulated error? >>> > >>>>>>>> >>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>> >>> >>favourites and >> >> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>if the latter is >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>actually more accurate then I can simply revert the >>> > changeset that >>> > >>>>>>>>implemented MIXED. >>> > >>>>>>>> >>> > >>>>>>>>Perhaps rather than running more of the same >>> >>> >>workloads you >> >> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>could try idle >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>> >>> >>large disc reads >> >> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>to /dev/null)? We >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>don't have any data on workloads that aren't >>> >>> >>CPU bound, so >> >> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>that's really an >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>>>>obvious place to put any further effort imo. >>> > >>>>>>>> >>> > >>>>>>>>-- Keir >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>>> >>> > >>>>>>_______________________________________________ >>> > >>>>>>Xen-devel mailing list >>> > >>>>>>Xen-devel@lists.xensource.com >>> > >>>>>>http://lists.xensource.com/xen-devel >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 >>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 >>> > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru >>> > >> >>> > >> missed_ticks = missed_ticks / (s_time_t) >>> >>> >>pt->period + 1; >> >> >>> > >> if ( mode_is(pt->vcpu->domain, >>> >>> >>no_missed_ticks_pending) ) >> >> >>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>> > >>+ pt->do_not_freeze = 1; >>> > >> else >>> > >> pt->pending_intr_nr += missed_ticks; >>> > >> pt->scheduled += missed_ticks * pt->period; >>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >>> > >> >>> > >> pt_lock(pt); >>> > >> >>> > >>- pt->pending_intr_nr++; >>> > >>+ if ( mode_is(pt->vcpu->domain, >>> >>> >>no_missed_ticks_pending) ) { >> >> >>> > >>+ pt->pending_intr_nr = 1; >>> > >>+ pt->do_not_freeze = 0; >>> > >>+ } >>> > >>+ else >>> > >>+ pt->pending_intr_nr++; >>> > >> >>> > >> if ( !pt->one_shot ) >>> > >> { >>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct >>> > >> return; >>> > >> } >>> > >> >>> > >>- pt->do_not_freeze = 0; >>> > >>- >>> > >> if ( pt->one_shot ) >>> > >> { >>> > >> pt->enabled = 0; >>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>> >>> >>*v, struct >> >> >>> > >> pt->last_plt_gtime = hvm_get_guest_time(v); >>> > >> pt->pending_intr_nr = 0; /* 'collapse' all >>> > missed ticks */ >>> > >> } >>> > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { >>> > >>+ pt->pending_intr_nr--; >>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>> > >>+ } >>> > >> else >>> > >> { >>> > >> pt->last_plt_gtime += pt->period_cycles; >>> > >> >>> > >> >>> > > >>> > > >>> > > >>> > > >>> > >>> > >>> > _______________________________________________ >>> > Xen-devel mailing list >>> > Xen-devel@lists.xensource.com >>> > http://lists.xensource.com/xen-devel >>> > >>> >>> >>> From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 1 Feb 2008 15:31:11 -0700 Message-ID: <20080201153111687.00000002384@djm-pc> References: <47A0EFC7.3020102@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=-------3c0352933c035293 Return-path: In-Reply-To: <47A0EFC7.3020102@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format ---------3c0352933c035293 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable OK, Deepak repeated the test without ntpd and using ntpdate -b before the test. The attached graph shows his results: el5u1-64 (best=3D~0.07%), el4u5-64 (middle=3D~0.2%), and el4u5-32 (worst=3D~0.3%). We will continue to look at LTP to try to isolate. Thanks, Dan P.S. elXuY is essentially RHEL XuY with some patches. > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Wednesday, January 30, 2008 2:45 PM > To: Deepak Patel > Cc: dan.magenheimer@oracle.com; Keir Fraser; > xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Dan, Deeepak, > = > It may be that the underlying clock error is too great for ntp > to handle. It would be useful if you did not run ntpd > and, instead did ntpdate -b at the start of the test > for each guest. Then capture the data as you have been doing. > If the drift is greater than .05%, then we need to address that. > = > Another option is, when running ntpd, to enable loop statistics in > /etc/ntp.conf > by adding this to the file: > = > statistics loopstats > statsdir /var/lib/ntp/ > = > Then you will see loop data in that directory. > Correlating the data in the loopstats files with the > peaks in skew would be interesting. You will see entries of the form > = > 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10 > = > Where the second to last column is the Allan Deviation. When that > gets over 1000, ntpd is working pretty hard. However, I have = > not seen ntpd > completely loose it like you have. > = > I'm on vacation until Monday, and won't be reading > email. > = > Thanks for all your work on this! > = > -Dave > = > Deepak Patel wrote: > = > > > >> > >> Is the graph for RHEL5u1-64? (I've never tested this one.) > > > > > > I do not know which graph was attached with this. But I saw this > > behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I > > was running ltp tests continuously. > > > >> What was the behaviour of the other guests running? > > > > > > All pvm guests are fine. But behavior of most of the hvm guests were > > as described. > > > >> If they had spikes, were they at the same wall time? > > > > > > No. They are not at the same wall time. > > > >> Were the other guests running ltp as well? > >> > > Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp > > continuously. > > > >> How are you measuring skew? > > > > > > I was collecting output of "ntpdate -q every = > 300 seconds > > (5 minutes) and have created graph based on that. > > > >> > >> Are you running ntpd? > >> > > Yes. ntp was running on all the guests. > > > > I am investigating what causes this spikes and let everyone = > know what > > are my findings. > > > > Thanks, > > Deepak > > > >> Anything that you can discover that would be in sync with > >> the spikes would be very helpful! > >> > >> The code that I test with is our product code, which is based > >> on 3.1. So it is possible that something in 3.2 other than vpt.c > >> is the cause. I can test with 3.2, if necessary. > >> > >> thanks, > >> Dave > >> > >> > >> > >> Dan Magenheimer wrote: > >> > >>> Hi Dave (Keir, see suggestion below) -- > >>> > >>> Thanks! > >>> > >>> Turning off vhpet certainly helps a lot (though see below). > >>> > >>> I wonder if timekeeping with vhpet is so bad that it should be > >>> turned off by default (in 3.1, 3.2, and unstable) until it is > >>> fixed? (I have a patch that defaults it off, can post it if > >>> there is agreement on the above point.) The whole point of an > >>> HPET is to provide more precise timekeeping and if vhpet is > >>> worse than vpit, it can only confuse users. Comments? > >>> > >>> > >>> In your testing, are you just measuring % skew over a long > >>> period of time? > >>> We are graphing the skew continuously and > >>> seeing periodic behavior that is unsettling, even with pit. > >>> See attached. Though your algorithm recovers, the "cliffs" > >>> could still cause real user problems. I wonder if there is > >>> anything that can be done to make the "recovery" more > >>> responsive? > >>> > >>> We are looking into what part(s) of LTP is causing the cliffs. > >>> > >>> Thanks, > >>> Dan > >>> > >>> > >>> > >>>> -----Original Message----- > >>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>> Sent: Monday, January 28, 2008 8:21 AM > >>>> To: dan.magenheimer@oracle.com > >>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; > >>>> deepak.patel@oracle.com; > >>>> akira.ijuin@oracle.com; Dave Winchell > >>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables > >>>> pending > >>>> missed ticks > >>>> > >>>> > >>>> Dan, > >>>> > >>>> I guess I'm a bit out of date calling for clock=3D usage. > >>>> Looking at linux 2.6.20.4 sources, I think you should specify > >>>> "clocksource=3Dpit nohpet" on the linux guest bootline. > >>>> > >>>> You can leave the xen and dom0 bootlines as they are. > >>>> The xen and guest clocksources do not need to be the same. > >>>> In my tests, xen is using the hpet for its timekeeping and > >>>> that appears to be the default. > >>>> > >>>> When you boot the guests you should see > >>>> time.c: Using PIT/TSC based timekeeping. > >>>> on the rh4u5-64 guest, and something similar on the others. > >>>> > >>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > >>>> > 14.318MHz HPET.) > >>>> > >>>> This appears to be the xen state, which is fine. > >>>> I was wrongly assuming that this was the guest state. > >>>> You might want to look in your guest logs and see what they were > >>>> picking > >>>> for a clock source. > >>>> > >>>> Regards, > >>>> Dave > >>>> > >>>> > >>>> > >>>> > >>>> Dan Magenheimer wrote: > >>>> > >>>> > >>>> > >>>>> Thanks, I hadn't realized that! No wonder we didn't = > see the same > >>>>> improvement you saw! > >>>>> > >>>>> > >>>>> > >>>>>> Try specifying clock=3Dpit on the linux boot line... > >>>>>> > >>>>> > >>>>> > >>>>> I'm confused... do you mean "clocksource=3Dpit" on the Xen > >>>> > >>>> > >>>> command line or > >>>> > >>>> > >>>>> "nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the guest (or > >>>> > >>>> > >>>> dom0?) command > >>>> > >>>> > >>>>> line? Or both places? Since the tests take awhile, it > >>>> > >>>> > >>>> would be nice > >>>> > >>>> > >>>>> to get this right the first time. Do the Xen and guest > >>>> > >>>> > >>>> clocksources need > >>>> > >>>> > >>>>> to be the same? > >>>>> > >>>>> Thanks, > >>>>> Dan > >>>>> > >>>>> -----Original Message----- > >>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>> *Sent:* Sunday, January 27, 2008 2:22 PM > >>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser > >>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > >>>>> akira.ijuin@oracle.com; Dave Winchell > >>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode = > that disables > >>>>> pending missed ticks > >>>>> > >>>>> Hi Dan, > >>>>> > >>>>> Hpet timer does have a fairly large error, as I was = > trying this > >>>>> one recently. > >>>>> I don't remember what I got for error, but 1% sounds > >>>> > >>>> > >>>> about right. > >>>> > >>>> > >>>>> The problem is that hpet is not built on top of vpt.c, > >>>> > >>>> > >>>> the module > >>>> > >>>> > >>>>> Keir and I did > >>>>> all the recent work in, for its periodic timer needs. Try > >>>>> specifying clock=3Dpit > >>>>> on the linux boot line. If it still picks the hpet, which it > >>>>> might, let me know > >>>>> and I'll tell you how to get around this. > >>>>> > >>>>> Regards, > >>>>> Dave > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> -------------------------------------------------------------- > >>>> ---------- > >>>> > >>>> > >>>>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > >>>>> *Sent:* Fri 1/25/2008 6:50 PM > >>>>> *To:* Dave Winchell; Keir Fraser > >>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; > >>>>> akira.ijuin@oracle.com > >>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>> > >>>> > >>>> that disables > >>>> > >>>> > >>>>> pending missed ticks > >>>>> > >>>>> Sorry for the very late followup on this but we finally > >>>> > >>>> > >>>> were able > >>>> > >>>> > >>>>> to get our testing set up again on stable 3.1 bits and have > >>>>> seen some very bad results on 3.1.3-rc1, on the order of 1%. > >>>>> > >>>>> Test enviroment was a 4-socket dual core machine with 24GB of > >>>>> memory running six two-vcpu 2GB domains, four hvm = > plus two pv. > >>>>> All six guests were running LTP simultaneously. The four hvm > >>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and = > RHEL4u5-64. > >>>>> Timer_mode was set to 2 for 64-bit guests and 0 for > >>>> > >>>> > >>>> 32-bit guests. > >>>> > >>>> > >>>>> All four hvm guests experienced skew around -1%, = > even the 32-bit > >>>>> guest. Less intensive testing didn't exhibit much = > skew at all. > >>>>> > >>>>> A representative graph is attached. > >>>>> > >>>>> Dave, I wonder if some portion of your patches = > didn't end up in > >>>>> the xen trees? > >>>>> > >>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > >>>>> 14.318MHz HPET.) > >>>>> > >>>>> Thanks, > >>>>> Dan > >>>>> > >>>>> P.S. Many thanks to Deepak and Akira for running tests. > >>>>> > >>>>> > -----Original Message----- > >>>>> > From: xen-devel-bounces@lists.xensource.com > >>>>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>> > Dave Winchell > >>>>> > Sent: Wednesday, January 09, 2008 9:53 AM > >>>>> > To: Keir Fraser > >>>>> > Cc: dan.magenheimer@oracle.com; > >>>> > >>>> > >>>> xen-devel@lists.xensource.com; Dave > >>>> > >>>> > >>>>> > Winchell > >>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>> > disables pending > >>>>> > missed ticks > >>>>> > > >>>>> > > >>>>> > Hi Keir, > >>>>> > > >>>>> > The latest change, c/s 16690, looks fine. > >>>>> > I agree that the code in c/s 16690 is equivalent to > >>>>> > the code I submitted. Also, your version is more > >>>>> > concise. > >>>>> > > >>>>> > The error tests confirm the equivalence. With > >>>> > >>>> > >>>> overnight cpu loads, > >>>> > >>>> > >>>>> > the checked in version was accurate to +.048% for sles > >>>>> > and +.038% for red hat. My version was +.046% and = > +.032% in a > >>>>> > 2 hour test. > >>>>> > I don't think the difference is significant. > >>>>> > > >>>>> > i/o loads produced errors of +.01%. > >>>>> > > >>>>> > Thanks for all your efforts on this issue. > >>>>> > > >>>>> > Regards, > >>>>> > Dave > >>>>> > > >>>>> > > >>>>> > > >>>>> > Keir Fraser wrote: > >>>>> > > >>>>> > >Applied as c/s 16690, although the checked-in patch is > >>>>> > smaller. I think the > >>>>> > >only important fix is to pt_intr_post() and the = > only bit of > >>>>> > the patch I > >>>>> > >totally omitted was the change to = > pt_process_missed_ticks(). > >>>>> > I don't think > >>>>> > >that change can be important, but let's see what > >>>> > >>>> > >>>> happens to the > >>>> > >>>> > >>>>> error > >>>>> > >percentage... > >>>>> > > > >>>>> > > -- Keir > >>>>> > > > >>>>> > >On 4/1/08 23:24, "Dave Winchell" > >>>> > >>>> > >>>> wrote: > >>>> > >>>> > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > >>Hi Dan and Keir, > >>>>> > >> > >>>>> > >>Attached is a patch that fixes some issues with the > >>>> > >>>> > >>>> SYNC policy > >>>> > >>>> > >>>>> > >>(no_missed_ticks_pending). > >>>>> > >>I have not tried to make the change the minimal one, but, > >>>>> > rather, just > >>>>> > >>ported into > >>>>> > >>the new code what I know to work well. The error for > >>>>> > >>no_missed_ticks_pending goes from > >>>>> > >>over 3% to .03% with this change according to my testing. > >>>>> > >> > >>>>> > >>Regards, > >>>>> > >>Dave > >>>>> > >> > >>>>> > >>Dan Magenheimer wrote: > >>>>> > >> > >>>>> > >> > >>>>> > >> > >>>>> > >>>Hi Dave -- > >>>>> > >>> > >>>>> > >>>Did you get your correction ported? If so, it would be > >>>>> > nice to see this get > >>>>> > >>>into 3.1.3. > >>>>> > >>> > >>>>> > >>>Note that I just did some very limited testing with > >>>>> > timer_mode=3D2(=3DSYNC=3Dno > >>>>> > >>>missed ticks pending) > >>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv = > guest) and the > >>>>> > worst error I've > >>>>> > >>>seen so far > >>>>> > >>>is 0.012%. But I haven't tried any exotic = > loads, just LTP. > >>>>> > >>> > >>>>> > >>>Thanks, > >>>>> > >>>Dan > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>>>-----Original Message----- > >>>>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>> > >>>>To: dan.magenheimer@oracle.com > >>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; > >>>>> > xen-devel@lists.xensource.com; Dong, > >>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>> > >>>>disables pending > >>>>> > >>>>missed ticks > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>Dan, > >>>>> > >>>> > >>>>> > >>>>I did some testing with the constant tsc offset > >>>> > >>>> > >>>> SYNC method > >>>> > >>>> > >>>>> > >>>>(now called > >>>>> > >>>>no_missed_ticks_pending) > >>>>> > >>>>and found the error to be very high, much larger > >>>> > >>>> > >>>> than 1 %, as > >>>> > >>>> > >>>>> > >>>>I recall. > >>>>> > >>>>I have not had a chance to submit a correction. I > >>>> > >>>> > >>>> will try to > >>>> > >>>> > >>>>> > >>>>do it later > >>>>> > >>>>this week or the first week in January. My version of > >>>>> constant tsc > >>>>> > >>>>offset SYNC method > >>>>> > >>>>produces .02 % error, so I just need to port = > that into the > >>>>> > >>>>current code. > >>>>> > >>>> > >>>>> > >>>>The error you got for both of those kernels is > >>>> > >>>> > >>>> what I would > >>>> > >>>> > >>>>> expect > >>>>> > >>>>for the default mode, delay_for_missed_ticks. > >>>>> > >>>> > >>>>> > >>>>I'll let Keir answer on how to set the time mode. > >>>>> > >>>> > >>>>> > >>>>Regards, > >>>>> > >>>>Dave > >>>>> > >>>> > >>>>> > >>>>Dan Magenheimer wrote: > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>Anyone make measurements on the final patch? > >>>>> > >>>>> > >>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and = > saw a loss of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>about 0.2% with no load. This was = > xen-unstable tip today > >>>>> > >>>>with no options specified. 32-bit was about 0.01%. > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>I think I missed something... how do I run the various > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>accounting choices and which ones are known to be > >>>> > >>>> > >>>> appropriate > >>>> > >>>> > >>>>> > >>>>for which kernels? > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>Thanks, > >>>>> > >>>>>Dan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>-----Original Message----- > >>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>> > > >>>>> > >>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>Keir Fraser > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>> > >>>>>>To: Dave Winchell > >>>>> > >>>>>>Cc: Shan, Haitao; = > xen-devel@lists.xensource.com; Dong, > >>>>> > Eddie; Jiang, > >>>>> > >>>>>>Yunhong > >>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer = > mode that > >>>>> > >>>>>>disables pending > >>>>> > >>>>>>missed ticks > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. > >>>>> > >>>>>> > >>>>> > >>>>>>-- Keir > >>>>> > >>>>>> > >>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>> wrote: > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>Keir, > >>>>> > >>>>>>> > >>>>> > >>>>>>>The accuracy data I've collected for i/o = > loads for the > >>>>> > >>>>>>>various time protocols follows. In = > addition, the data > >>>>> > >>>>>>>for cpu loads is shown. > >>>>> > >>>>>>> > >>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 > >>>> > >>>> > >>>> processor AMD > >>>> > >>>> > >>>>> box. > >>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. > >>>>> > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>> > >>>>>>>(usex is available at > >>>>> http://people.redhat.com/anderson/usex.) > >>>>> > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 > >>>> > >>>> > >>>> of=3D/dev/null. > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > >>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>> > >>>>>>>All three guests are 8vcpu. > >>>>> > >>>>>>> > >>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 > >>>>> > >>>>>>>except that the redhat-64 guest has 4 = > instances of dd. > >>>>> > >>>>>>> > >>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 = > sec -.006%, > >>>>> > +.005% cpu > >>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 = > sec, -.001%, > >>>>> > +.012% cpu > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, > >>>>> -.004% cpu > >>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, > >>>> > >>>> > >>>> -.005%, -.005% cpu > >>>> > >>>> > >>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, > >>>> > >>>> > >>>> -.008%, -.003% cpu > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, > >>>> > >>>> > >>>> -.040% cpu > >>>> > >>>> > >>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 = > sec, -.034%, > >>>>> > -.031% cpu > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, > >>>>> > -.09% i/o-8 > >>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% > >>>>> > -.14% i/o-8 > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, > >>>>> > -.022% i/o-8 > >>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, > >>>> > >>>> > >>>> -.017%, -.018% i/o-8 > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, > >>>>> > -.029% i/o-8 > >>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 = > sec, -.023%, > >>>>> > -.031% i/o-8 > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, > >>>> > >>>> > >>>> -.04% i/o-32 > >>>> > >>>> > >>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, > >>>>> > -.005% i/o-32 > >>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, > >>>> > >>>> > >>>> -.11% i/o-32 > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, > >>>>> > .003% i/o-4/32 > >>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, > >>>>> > .01% i/o-4/32 > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>Overhead measurements: > >>>>> > >>>>>>> > >>>>> > >>>>>>>Progress in terms of number of passes = > through a fixed > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>system workload > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>>> > >>>>>>>The workload was usex -b48. > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>Conclusions: > >>>>> > >>>>>>> > >>>>> > >>>>>>>The only protocol which meets the .05% accuracy > >>>>> > requirement for ntp > >>>>> > >>>>>>>tracking under the loads > >>>>> > >>>>>>>above is the SYNC protocol. The worst case > >>>> > >>>> > >>>> accuracies for > >>>> > >>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>SYNC, MIXED, > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>and ASYNC > >>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>> > >>>>>>> > >>>>> > >>>>>>>We could reduce the cost of the SYNC method by only > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>scheduling the extra > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>wakeups if a certain number > >>>>> > >>>>>>>of ticks are missed. > >>>>> > >>>>>>> > >>>>> > >>>>>>>Regards, > >>>>> > >>>>>>>Dave > >>>>> > >>>>>>> > >>>>> > >>>>>>>Keir Fraser wrote: > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>> > >>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>> wrote: > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>>Since I had a high error (~.03%) for the > >>>> > >>>> > >>>> ASYNC method a > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>couple of days ago, > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have > >>>>> > been something > >>>>> > >>>>>>>>>wrong with the code I used a couple of = > days ago for > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>ASYNC. It may have been > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>missing the immediate delivery of interrupt > >>>> > >>>> > >>>> after context > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>switch in. > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>acceptable accuracy, > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>each running consistently around or under > >>>> > >>>> > >>>> .01%. MIXED has > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>a fairly high > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>error of > >>>>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>threshold for comfort. > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I > >>>> > >>>> > >>>> plan to leave > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>SYNC running > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>over the weekend. If you'd rather I can = > leave MIXED > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>running instead. > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>It may be too early to pick the protocol and > >>>> > >>>> > >>>> I can run > >>>> > >>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>more overnight tests > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>>next week. > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>> > >>>>> > >>>>>>>>I'm a bit worried about any unwanted side > >>>> > >>>> > >>>> effects of the > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>SYNC+run_timer > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will > >>>> > >>>> > >>>> cause higher > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>system-wide CPU > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>contention. I find it easier to think through the > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>implications of ASYNC. I'm > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>surprised that MIXED loses time, and is less > >>>> > >>>> > >>>> accurate than > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>ASYNC. Perhaps it > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>delivers more timer interrupts than the other > >>>> > >>>> > >>>> approaches, > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>and each interrupt > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>event causes a small accumulated error? > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as > >>>> > >>>> > >>>> favourites and > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>if the latter is > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>actually more accurate then I can simply revert the > >>>>> > changeset that > >>>>> > >>>>>>>>implemented MIXED. > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>Perhaps rather than running more of the same > >>>> > >>>> > >>>> workloads you > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>could try idle > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated > >>>> > >>>> > >>>> large disc reads > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>to /dev/null)? We > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>don't have any data on workloads that aren't > >>>> > >>>> > >>>> CPU bound, so > >>>> > >>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>that's really an > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>> > >>>>>>>> > >>>>> > >>>>>>>>-- Keir > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>>> > >>>>> > >>>>>>_______________________________________________ > >>>>> > >>>>>>Xen-devel mailing list > >>>>> > >>>>>>Xen-devel@lists.xensource.com > >>>>> > >>>>>>http://lists.xensource.com/xen-devel > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>> > >>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 = > 2007 +0000 > >>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 = > 2008 -0500 > >>>>> > >>@@ -58,7 +58,7 @@ static void = > pt_process_missed_ticks(stru > >>>>> > >> > >>>>> > >> missed_ticks =3D missed_ticks / (s_time_t) > >>>> > >>>> > >>>> pt->period + 1; > >>>> > >>>> > >>>>> > >> if ( mode_is(pt->vcpu->domain, > >>>> > >>>> > >>>> no_missed_ticks_pending) ) > >>>> > >>>> > >>>>> > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>>>> > >>+ pt->do_not_freeze =3D 1; > >>>>> > >> else > >>>>> > >> pt->pending_intr_nr +=3D missed_ticks; > >>>>> > >> pt->scheduled +=3D missed_ticks * pt->period; > >>>>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) > >>>>> > >> > >>>>> > >> pt_lock(pt); > >>>>> > >> > >>>>> > >>- pt->pending_intr_nr++; > >>>>> > >>+ if ( mode_is(pt->vcpu->domain, > >>>> > >>>> > >>>> no_missed_ticks_pending) ) { > >>>> > >>>> > >>>>> > >>+ pt->pending_intr_nr =3D 1; > >>>>> > >>+ pt->do_not_freeze =3D 0; > >>>>> > >>+ } > >>>>> > >>+ else > >>>>> > >>+ pt->pending_intr_nr++; > >>>>> > >> > >>>>> > >> if ( !pt->one_shot ) > >>>>> > >> { > >>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct = > vcpu *v, struct > >>>>> > >> return; > >>>>> > >> } > >>>>> > >> > >>>>> > >>- pt->do_not_freeze =3D 0; > >>>>> > >>- > >>>>> > >> if ( pt->one_shot ) > >>>>> > >> { > >>>>> > >> pt->enabled =3D 0; > >>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu > >>>> > >>>> > >>>> *v, struct > >>>> > >>>> > >>>>> > >> pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>>>> > >> pt->pending_intr_nr =3D 0; /* 'collapse' all > >>>>> > missed ticks */ > >>>>> > >> } > >>>>> > >>+ else if ( mode_is(v->domain, = > no_missed_ticks_pending) ) { > >>>>> > >>+ pt->pending_intr_nr--; > >>>>> > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>>>> > >>+ } > >>>>> > >> else > >>>>> > >> { > >>>>> > >> pt->last_plt_gtime +=3D pt->period_cycles; > >>>>> > >> > >>>>> > >> > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > >>>>> > > >>>>> > _______________________________________________ > >>>>> > Xen-devel mailing list > >>>>> > Xen-devel@lists.xensource.com > >>>>> > http://lists.xensource.com/xen-devel > >>>>> > > >>>>> > >>>>> > >>>> > >>>> > >> > = > ---------3c0352933c035293 Content-Type: application/octet-stream; name="hvm-compare.png" Content-Disposition: attachment; filename="hvm-compare.png" Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAAaMAAAGkEAAAAADpa8nTAAAACXBIWXMAAABIAAAASABGyWs+ AAAACXZwQWcAAAGjAAABpADnhMYnAAAhMElEQVR42u2dv46syNnG0SdL1soROpYsqyXLQnJi TeJFljrcgGCs1YRIG05goQk3Yy4BzR0QOyOZC+AWuAVugVs43zxbU8ufgaZoKKoKnl9w+nQ3 00B1PV1Vb71/vO+EkJV4pi+AEPehjAhZDWW0E56XpqavgeiCMtoJyujIUEY7QRkdGcpoJyij I0MZ7QRklKa+73lBIAQVBHEs300S3//+Pc+jqKqiyPN8P03LMgw9b05+aRoE7Wd+/15VcYy/ al+JorJMErwWRXWNM3leGFYV3uufsWnwWtMkCT7T8+K4PappcFT3DHFclqZb1RYoo50Q3bj4 AN0xzyEAz5MdF51YvBKGOAZd3ffzvCzRZXH0OBAmBIe/xCdUle/jE8QrSSLO7Pvo8nkuRFwU +F8QiL8XghPHC1lDaDhz/6g4xlHfv9e177f3IWRGKKOdQHeVopFdXgokz0WHRHcVv/Dt/+TR Y9S152WZ+H+ShKGQgDiL+Iy6xpmjqH2l/Xz5KD8d/8dV5HlRtK/IRylljFTy/GHYjqfnhjLa ia4Y5P/ltC6OIYG223b/d2tVlWVyPJP4vhiBgBRZXyj9/7UjYl+UEHlRYFI5PApnKD/BVNF0 u9oBm2EnxmQkZIDxRvzW35JRXZcDmqZ71NezyGdzMhoeXxRiZRSGXRm1x/Qx3a52wGbYiTEZ 4fc/zzGlE7/1t2SEV/qU5dho1J4F8lwyGonjyxJrKrHmGRNb9wxEQhntxJiMxOoiDOVEbOmk Dqurdm2ENUsc99dGkMOcjPpro+6Zx2QUx+3aKE3bKeS5oYx2YlxGGE/ksn+5jIRs5i11t2WE 91tLHUYjYWIQ14b/d68GZ4ClTqyMpDHi7FBGOzEuI0ykur/uS2UkTN63943mZCT2p9p9oywL PoApQRjehcTaM8ozQK6mW9UWKKNT89VIQe6BjXhqKKNtYCOeGspoG9iIpwbecqav4QhQRoSs hjIiZDWUESGroYwIWQ1lRMhqKCNCVkMZEbIayoiQ1VBGhKyGMiJkNTvICAmb4FifJP1ITUKO wg4yQs4aZBxoozwJORY7yEhGdyKu0vTtEqID7R27K542XJqQI2FMRk3zn//8/MHbJ+/vJSGW 8f4u+2fywevr1OremIxeX//vgz/84Y9//NOf/vGPf//75eWf//z5518++PXXdJTHx+fndBGX y7Lj0/R6XXb8w8PTk94zLL+L5+fHR71nWH4XT08PD8v+Qv+3rdJOLy/XTy4f/PDD66shGU2t jZLkchGv/ve/v/4aRUHw5z//618//fTzz3/5C1Lmfv/+44/RB74v80Rfr7/8kqZFoW7v+/Zt 6bUuDWK7Xt/e9J5h+V0gBYneMyy/i7e363XZX4iM4TrvYnk7XS5TRjJjljopoxbkBc2yNEXm NuRvu16RGRQJBsX7f/vb21ue4xdCPP/rXyGzOH5/F8/f3mTW63sbljLSdReU0WrG943mGraq IBmMUkIsafrLL8iOBrGJI0QjtI1xvcZx9BviPN8+wDOZEFFOEuQZ0hQ5QtftZS3/spejP8xb /xmWd1kb72L6R9OYCXrZ7xOWe5CVFFaSQARi8jeNaFj5Bf74o1g2yub+8UeMfWEon889jlVP oIzUoIw0sXyY7wI5ZFkUhSFWS1NyWtKwbTmS6XN+fY0yUoMy0sQ6GUmaBl+QGKHEJK377pKG rWus4OaPa0uXAMpIDcpIE+/v0lSwDRBUlsGcEYZyuresFhymePNH5Xl37Ktr/X6C+iva6T9D 08xNwF24i5cXac4aYkxG+n6fmibPYWyYLvU4hdp4RM7K9NzjgDISoFhwGOr5DaTYzskJZSTO cd/4Mm20EMgK3eRcnFRG9ybXzTJsAU9LiaPROTmtjMQm633Xx+zWpM+JZYSqcNt/Kid1Z+TE Mlq/nzA2mrUV8sh5OLWM1nZ5OAxx9CEnl1EUrTV7w+JHIZFTy2gLh52q6stoD+8FYhunlhE8 w7f+zCy7z/5HXObUMqoqJvYiW3BqGcFXbpvPga8eJ3PnhTLaiKJASAacXrFWEoGEbeNWlQgM lFIbPtfv5Ux0cnIZ6fCBgzSlTKQ8pKymnsvIKHk1eA7a47uiI7ZxchkhQtbUfc6B3BJSOJBd kvRlhiwUHMXswEoZiSxhe5yrKPaT7LaI8Y4yMo/InWehjJ6f0UX2OFdVjaUjWUdd7925Odkz Cfoqeuz4u6eY1G1pZJDk+fLo2nXcE89LtsTKSd2eMsIqw/Vf86aJY9aIMsnpZYTR4wh+cWpp V4geKKPvWCEhq912v+ZLsomTI0AZfZLnQbCV8XuYFY8cHcrod5oGiSFdH0dQbYej4d5QRj2Q 6AQTPFPn3wKRLbbdqN0jf+vZoYxG2CIQ3BZXVexquP7DYD+U0QhbZP6pKjtkJOC+kl4ooxGY QIssgzIaIUm2WU24vxtF1KCMRthqUb69o9E6OMrqgjIaIU2PuZaABc/ewBCXoYxGcDd8Yg5k n5BCQuA7Ypbk1FOEXpi+QjehjDRfQRzbGhM0DGQXham/RuBOPcfxKF1tk0XSFJTRCFW13SrC VhHdS2vIHwbCIzoXfiBTgfJD2R4JymiUrRfjR+w6Y3RlUpZ92cAHvS8789/zVlBGo2yd0H59 muPjgYoe5r/pbaCMRrHNVH1UjhDpBXaWUZoGgef5vojVbJok8T7oR27aICPusOzDUcboXWWU 576P09V1GCKVCEoZNx+EYTcJsA0yiuOtTb/H3Inaitbs7ia7yqgsZWcqS+/j8z1PnFw8a48y L6PtgwuYIP8WiD922QxjaG2EyV1XPFJQwA4ZcfTYF2TEMH0N92NARnWdJL6PXYQpGV0u3z6I ov52354c14/BXorCpZUS9hYF6KuXi0YZeT3wStOkKUwKaDCbR6Ptr4GONqq4WGht19EIxoSu c4y9a6Mt/RjkJ1JGahRFGLo2pd5VRig53H1ur6WOJm+T1LVrrb+rjKKoP8mzd99oez8GsgTX 7Hb0YphAR25v0/dEdEEZTbD9tIIyOi6U0QQ66vCRo0IZTcANWLM0jUvh7pTRBE0TBNsuc/Pc pe1F06D93ZkPUEaTbF3qxK1devNUlTtCooxucJRoGFfRUVJUD5TRzSsJAnhNJcn4hIwiIwLK SIGqkllxxgiCMBT/Q1B0nsN7bkxgdU13oPuw39xAGW165fAMB0JeyJMDhLT+94HpK3QTuIvZ bTeljDTSNJCPTDQFcYUhi6TcA4Rkcwk3ymh3soxVw5fTNO36VP4smb6mFspoR+R9IdKTQrof mQFPPEOxUZFoUr6//8YCZbQj7fyePhJb08oKYRYC8byNUx1/XtcY2dqceWIqvizMkjIyAnIj mb4GIhjKRsiqnTRiG7ibVHkMysgQdH11ibrO825o6RDKaEfwOyf/XxS3vhbiFpTRrnS3EWlm OA6UkTHSlCkgXWPqh48yMgYsSiLVGHGFKa9/K2V0vbbmyKPRNy0URRDQ9O0SQTD84UNfvV4t lNGRR6Phl9A0cUxjgzuU5dgPvJWj0ZFlNAa87TgmuUKef10hUUZW3CHymjMznrtQRkYQU7v+ r1qSMB7JVSgjg8hiZEJOWcaJnatQRhYgCiyX5d//LkYpGVkjq3i31byJnVBG1tA0P/0kW0A8 ynaQj7KeeRQJWVFctkAZWcQyIwPibLqPrdzkV6r6SNZCGVmEHG2WIkel4eil+ihl2H9ESTfT LeIKlJFFpKlNGXAQh4NH96oN7Q9lZBE2V5yVoiJjUEYWYXOWUJR0M30N9kIZWQU9GdyEMrIK rkLchDKyCrsdgpAV1vQ12AllZBVZZnNELLOQT0EZWcWZ791lKCOrOPO9uwxlZBVlabtZeetS nsfAgIyaxvdlgECSeB/0U8OfWUb2ewxQRGMYkFGaep44aZIgVxuKbnR/g88sI5q83WR3GWVZ WUoZyUe80h5BGdlNXXNEGrKzjIS7i5BPVzxSUIAyshv8EJq+BtvYVUaYviEUYE5GDw/XD2Re /3MlRPSMmXbIEtq6E+irDw8aZeT1EFXSxOu3ZfT09PaBLJdxrikEC7a4QVvOBX316WnH0SiK +rLi2ugr9k/q8A2da4Ywj5F9I1rqpnFBRnnO2kx9jMqI+0ZfkelKiEvQi8Eyooh2MPegjCxj +guxibE81mdGWUb7Wc0oI9PXME+W0cjQZVZGTZNlYdja14JAd2gZZWT6GshSZmSUpmGItE/S MgNreZbFcRTps9WcXUauZPKuKk7sJDMymnqzrikjPeS5K3dvczqwvVlkYqiqPWbE55bRue/e VZRkVBSIECoKrI30TznO3ZFcu/ui4FpOUUZBkGXwNUCT6c+k5lpHOvfd1zWzBSnKCB5vTSP8 3vR7ILvWkba+e3szp5IpFEcjYZ9rxaSTc8uoLbPiDvCLNH0NZlGSUZ5jVYTBO0n0/1aeW0Zu OKf2Ofs3pmypqyph4N7DCeTsX4p7MiL0qbMOBu65x4yMvC/o/608u4zcDCM/t+F71osBZJnv I5EFHvWbN88uIzcndTr9WuxHaVIXhm3RXe4b6cZNGZ0b5X2jsf/rgTIyfQVkKUoyksmCORrt wb31yE1z5u9NSUZp6vt5XpZ57vv6m+rMXweIYzcX603jpvy3QNHgnaZBgJC9PTp4WV6vUXTe qQ0D99wCffV65b6RZVBG7qHoDOT73TSNeqGMXJXReTeOFV1TkX9BovuSKCOb67/ewlX5r2ex wVs/Z5cRg7PdQ3E02tMGc3YZnf3+XURJRlUVRTB4c1K3B+7ev7urOn133pHRsBKEXtztRme/ //PW4WOghHWU5Xn3zFxFSUZNAz8Gz9vDh4Eyoledeyg6A4WhcAZCDlXdl0QZuSqjsuzWqToT iq6p0lJX176v+5IoI1dl1DRnjTlS3DeSS0dmBtoD/V70ZFuUZBTHSYLxqK6ZGWgP9I/4ZFsU TQyyNAtqteq+JMpIVmx3D1eno2tRNnhXVVnuM/OljODk6WZKX66NhvRkBCsdHpHLW/clUUYY /4PgrF3SRZSjX4XXcRTpN2lSRqCqzht24B6KBu+22h4N3nvh4grJnVqBW983Dd7WEoauTezO 6lW32OC9zhZTFLD5SaeipkkS2P+SpNv4lJGEEztXUDR4Sx/vdQbvshSpuuoahcdQnwKfB3N6 d8VFGbXEsWvj0TlRNnjX9XqDdxz3BeJ54uRl2Z0qUkYtReGal9o5d452DZTAZE5O6pqmKx4p KEAZdXHNMeicueqU89QhUALTsDUjEqaFmMzVNTzFKSMV0tTNjdhzoRwogTrk2H5d4lPXL+mC 53LIx6dNy+hy+fZB9Mm5VwdVxVqwNoLECgL01ctFOVBifQnlMGxlhP0nro1UCAKXjMhIxWb6 GvZnQYKtbj3y+0AN2XZSR0udGlnmUnu4JPntUNw3wheJTdi1gRL9XODcN1IB/nXnXLi7g/K+ Ebo/1jZ0Td0fJjmxnV33jdSgjL7iTkLic+ZjWLRvVFV7TC4oo69g/Sg2Cc659rAdJRnBrlaW MFJ7nn4PXsponKZBdm9Mr6MIq1UUtQYUlnkUc3gjXA97RyxaaQeIRYYFD0QRLJ0IUSgVMX31 x0PZ4C1N3QyUsBGII1UE2wzYwcOIVhTbj2VnNIcojkb47YOpm/FGRwEmI7T0Hgk8j49itT2s iuDbxQRbxyNN45jrq3UoWuqqShi785z7RscDSaW5wbuGGRnF8Zh/cV3rjLmnjPYHpqOtdgXP WC1wRkYIG/d92IGEjSfPMZsOAp3O+5SRCRCwfm5f+jUoTOrqOsuiSNQih2lVd/wLZWSG89bK Ww/LhJFPMNMwfQ2uQhmRT7Zr9/Ol8qeMyCd1fcaN022gjMjvMCvevVBG5He28pdERLPpe9kX JRnBOrdfw1BGptgqKPN8MUeKMkLsaxxnGeONjsw505FswYLo1zyHnPTPnykjUzAn3r0oywid O459fw8ZPT8zKsYEW+4c1fVZ8uuhr6LHjr/bkRES4cMlaA/HVFzW46OLtX3cZ9t5wFlci9BX Hx+V4o18f6+VESd15kAGKNPX4CaKkzqxMoJTKotWHpmtp+xnqb23YN9IxEqyaOWR2VpGIpFn 04gJnnw8Hoqp8LE6Qq7TPSw5lJE59KTzxFym+1hV4htWfaxr2y2ISjIKwyzb73eEMjKHncES UkaIdzN9LePQGYh0sHvnqK5lKgPRQ+yR1a5lwtSgjMzhVgA4shvhMc9Faua90mN/RWuZsPug jMzhdtvLTXskvsTjPsmygZKMtioTptoYLn+VbnOMnSMpJ5l4Rz7Gcf+5nMKW5Vq57VomTLUR KCNzHEFGqsjRSo5e9+cZ2blMmAqUkUnOJKPtYJkw0uN8eRSG3FNJimXCSA+ORvf4jiqORu3/ 6FN3bCije1D0YhBCQrkwhu0dmz2m7faztA0UK0ogMW0ce94eHZwyMkkc2+gOtDdRtGxip7g2 QmkWVDna4xYoI5PY6VVnOwuCyLerOXAbysgklNE9zMjIG0H3JVFGJtFZcsct0lR9Yjcjo60L 8BYF9p9kYRfY/SBMEdwloYxMwoT4kqpSNzQoTOqky4SsuHc/de15OF1Zeh4+E2HpyLCJci/t UZSRSdj69zAjI/gvCPucGDek6fs+IB8pIzyKf8Xz7lH8Is3B1r+HGRlhvMC4UVUooYxJ2Lrt 1zAU6yvsPnXFIwUF+EWapCzPkl9uHvW+PiMjESIhIo7w2DRrfK6aJk0xMUSMPZyLpmR0uXz7 IPrkqGkw7OQYoRLbcMsOUFWyf6KvXi4zljrxJAzlCLHEUje08CH4T7xTFGHI0chOKKPlKI1G iDISh60bjTAxFP8TsbRcG9kIZbScGRlhMocyG1I8yON9/8mw0hKTOmGbo6XORvTvDLqCek+c tdTBQuf7YksOclqzUsHaCKlRfD9NRRpA7hvZB0cjSdOobsAuSrC1ZEPqfigjs1BGy2GeOjKA oRLLoYzIAIZKtKgGTFBGZAB9vFvqWm1kpozIAMpoOczFQAbYncd7f1QSbjMXAxnAUIk+KvZp 5mIgA9j+Y9yWEnMxkAFs/zFuT+2Yi4EMgPOX6WtwDeZiIAPqmn4MS9k5F4MKlJFpKKNxsmyq 9ytO6mR5wCxjKvzjQxmNU9fwaRD/9lGstuf7Is9+FHHf6PiwqsQtiuJrCjLFanvSuLAubE8N ysg0+vcGj4ZitT05lWO1vTPASd1SFKvtJYmYFSaJ/iamjExDGd3ma2lm5Wp7MjEWTQzHR5Ya JuPIWrEtrLZHvtA0e221HwXF0ajNoaD/kigj8yDpDGNg1VE0eIchBjK4qOrv4mV5vSKNnumm OTdZRpegabqTXvTV61XJ4C2XVHVNg/dZ4AppmmGZZRq8yQTIH8gVkhqLDd76E6VTRraAFZLp a3ADRRNDWwmCBu8zwdp7aigbvKuKBu/zgZhn09fgAgyUIDcJAhq+52HYHrkJp3UqME8duUld c1o3z4I8dU3DtdEZEWVLyS1mZZRlQSC6NYp57ZEmnTKyiywTIZtkmhkZZRniXlsfhn2cgSgj m8B2h+lrsJ0ZGQVBPxFtVQWB7kuijGyDZoY5Zi11X16mpe50YDyi2fsWszLqNx996s5JmvI7 ucWMjOK433zrSiirQRnZBwL5UGmCNrtxZmRUVb4vsxdX1doSympQRjZS10WRplEUBFE0DBMg swbvqpJ5GGDu3mPniDKym7rOsjCMIhgeypJrJqC4/Qpfur0ajDJyAfSJNEWmKPzMIg1onp83 OonOQGQT8K3FMaZ8AGtqMVqV5RkmgDvIqGvdy/MgQKUksQ+BEpiYLCZJd6SjjNwH8kHVPhB9 AoklCV4piqONW5plVNdIgyJlhKKXOF1Z+j62dZME+xHYleimz6CMjgvkBVOFGLfSFDUajjBa aZaRmD9LGaHh5GmR+cfzxMnhrdf/G9PNQvQjVlfS+gfcLd68w6SuFYmUjXitK572HcrovGCc CkPMTVxzPrJSRk9Pbx/IaFuaVM9GVaVpENjtVy6s1wB99elpQxmlaTdKVo4py2X08HD9IP3k CHNnshz9TtBrqGvZP9FXHx52HI24NiLqJIn+vB9bseukjpY6og5WSqavQZVdZcR9I6IOygGZ vgZV6MVArMWdKCfKiFiLO/tIlBGxFndWR5QRsRZ3kqlQRsRiXFkdUUbEYooCwev2b8BTRsRq mgabJLYHVlBGxHqQHdFuIVFGxAEgJBFTWxQ2rpYoI+IIKFQnshOJUAqbVkyUEXEQ9BER7ocI WvOCooyIw9R1WWYZQtLNZiaijMghkHKSGR72XUFRRuRQICo1y7pTvj2SplBG5LC0khKEoe9H PbZaV1FG5LQ0TZtGed0nUUbk5MBMEUXrAtYpI0K+I8Prmr+njAj5GJHWBaxTRoR8X5vOy0oZ PT6ySCLZl/vTeaGvPj5aKKPnZ2SjNHV+ckbuXx2hr6LHjr/LSR05EetWR1ZO6igjsj9rVkeU ESG/sSbZMWVEyG+s2TuijAj5DaTzujfUgjIi5JOqujehF2VEyO/k+X32OsqIkA6IULrnrygj Qn6naYJg+cSOMiKkR553i9apQRkRMmB59BFlRMgAhPEt+wvKiJAvxPGy8YgyIuQLSx1VKSNC RljmYUcZETJCXYehevItyoiQUYoCybfUxiTKiJBJVIs47yCjpvF+/6w0DQLP8/0kwU5x0ySJ 94F4JqGMiC1UVRyrHKdZRnWd52EoZZTnvo/TYd6Jy0sSeNTCQb27b0wZEXtQs9hplhEkgRFH PpNJXssSr3meOLl41v0bU41GSB+10PIdJnV9kcjTBkH3dSkocTxlRGzBitEIDGVU10ni+yhB SBkR24ljlYjYTWWUpl4HKYauXJoGxyQJLPLTMrpcvn0gi2fYXYOaHJspgVSV7J/oq5fLjqMR jAlx3G5qcW1EbAf1keaP2nVSl6Zh2H2HljpiO2q9cVcZRVF30sd9I2I/mLzNH0UvBkJuQhkR shrfnz+GMiLkJhyNCFmNSuQRZUTITaYlonIMZUTId7UU+ZQRITdR6Y+UESE3UUlvQhkRMgNl RMhq5mOOKCNCZuBoRMhq5oN1KCNCZpjfOaKMCJkhTYti7gjKiJCbzG/AUkaEzDDfIykjQmYo y7mkj5QRIbPMmbwpI0Jm6ecQ+QplRMgsc34MVsro+bkslxaxJUQftyZ16KvosePvGpTR42Oa ckQi9nDLjwF99fHRQhlRQsQu5vwYrJzUUUbELub8GCgjQmaZ82OgjAiZZa5PUkaEzDLnx0AZ EaLAbT8GyogQBW77MVBGhChw24+BMiJEAU7qCFnN7XwMlBEhCtz2Y6CMCFHgth8DZUSIAlUV hnk+9S5lRIgSKPWdpt0qxS2UESHKpGkQQDJDMVFGhCygabIsjqOoP8GzUEbv7y8vus8xl0x2 PXU9Pvy7dRf6z9A0de3+Xby8vL+Pv2NMRm9v16vuc6iUxV2HSqlD++9C/xn2mHvov4vr9e1t /B3KaBWUkRqUkSJN43n9574vuljTJIn3QZJ0J0CUkT13QRmpoVlGdZ3nYdiXUZp6nuhiSRKG TQNDYpK071NG9twFZaSGZhmhkTDitK9kWVlKGclHvNIekSSXy7KzLO+y374tvZOlX8V0w251 huV3sbzL6m+n5T+a+r/t5e10uXQHgi6bTeq6IqkqRBEK+XRfl4IClJGuu6CM1LBcRpi+wbx5 W0avrz/8cPng+snLSzrD4+Pz89wxfS6XZcen6fW67PiHh6cnvWdYfhfPzyIDoL4zLL+Lp6eH h2V/of/bVmmnlxfZP9FXf/jh9XUzGWHV0yIV3coFp29FMyWjpnl9TT54++T9vSTEMt7fZf9E X319ndol1DAaRVFXZlNrI0KOg5a10edH37TUEXIcdpDR+L4RIceB0yxCVkMZEbIayoiQ1VBG hKzGiIx0GR267rF5HgSeFwQy8CpNfX/dGXHV+IwwFJEt4i58X+6cDc94D7evert2KwrZUjra SW52CF+H4VUP2+0+xHVGkfjM7dtpuDd6+9s2IiMdJvC+e2xRCP/ysvR95HrBTSPELo7vP2OS RBG8MyAmfD1JEscIRxNNOTzjfffgefhbWD1xpuFVb9VudY29PV3tVJb9FL7Dq+63232kKTxl 0LXhdqarnUBVqXzbRmSkY0O27x4bRfJXI03xmyifrzljnsv4SnH98i76Z5DP76Fp2nbB/4ZX vVW7RZEcjXS0E8Kvu8+HV91vt/vw/W6sq652wjcSBH0H6/Fv24CMptyDtvzk9nPFa+3z9WcU Y0V7ruEZ1n15+GtMFvAL2L/qrdotSYpi2OG2bCfpLCYykA6vethu97ZRdxqnp53EvQx/Vsa/ bcpoIXnu+1k237BrqOs0RRfU0T3yHGOFThnlOaY9mHAhsbwuGUFAdS2mbbpkVNdiQkcZbSqj qgoCOcjrlBHm3uiC23cPzO3RMXTKSAJTw9er3kpGsp2+XvV2/StJvjpeWyMjfc6qXffY7ef8 +FuMQ8O72G5tlKZy0S26x/ZzfjFlbC1QOtopTUUWIMioO6ZutzZqLbJY6utaG8Hg0+Yzsm5t pM9Ztfsrtb0FKgy7ItJhqcN8H5+ByYpeC5RsKR3tFEWYcGFSJyx2Oix14hxoJ3RnPe2UZV2L o4WWOl37Rt1fn+33Q7q/42hCfG1b7xshXyc+U16nrn2jtqX07K/hKtHtxq562G73nUN+hr52 iuPuFd7+tunFQMhqKCNCVkMZEbIayoiQ1VBGhKyGMiJkNZQRIauhjAzRT0OGnah7HVdu5RdV 28f/WlfuK3Gsv0KRu1BGRul2c5XO/BXs1d96V0WaKgIuS/2p5t2FMjLKeq/CLFtfq0FtHITD zV7t4hqUkVG+eiNXFSqOCtcW4X4i3E2mXGjCUHjwyb+Ct1mbFFp8fvuZ3bTQ8l0xvcQ7whUJ 5YPxLorbd516Wn9nMoQyMspXGYmQNBH8Jh/xrvApls6YLdILWXR3/DUe4SHeZlCXn4lXp32u cSZIVp5DSBb+aULIWcZp3RSUkVHGZdR9RzzCs1gcJUIDvv79sJrUUEZiROmHv/dlFATSH1pE O8FBtjuNYw72adgwRlGVUT9OaOzv52TUPWpcRn3LIaSL+FXfj2OR94AymoYNYxRVGVWVHE/6 trd1MpKJTeRoJOOk6lpkUcAZmwZBCMNrJX3YMEZRlVE3GG6Yd0eMFUtkhOmaWANJGUFASFol 5CPMGtKcAcMDHrk2moYyMoq6jISl7msomrTULZGRDDkrCiEM2OoQ2TtlqRNCpaVuGsrIcbbY N1KD+0bTUEaOc9uLYTuKgkXepqGMnCdN702hsgT61N2CMiJkNZQRIav5fwHBRRvFdXjTAAAA PHRFWHRDb21tZW50ACBJbWFnZSBnZW5lcmF0ZWQgYnkgRVNQIEdob3N0c2NyaXB0IChkZXZp Y2U9cG5tcmF3KQqV01S1AAAAInRFWHRwczpIaVJlc0JvdW5kaW5nQm94ADQxOXg0MjArNzIr MTg0PNQ4sQAAABx0RVh0cHM6TGV2ZWwAQWRvYmUtMy4wIEVQU0YtMy4wCptwu+MAAAAASUVO RK5CYII= ---------3c0352933c035293 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ---------3c0352933c035293-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Deepak Patel Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 30 Jan 2008 13:04:16 -0800 Message-ID: <47A0E650.3070506@oracle.com> References: <20080129153458531.00000002384@djm-pc> <47A096CC.2070907@virtualiron.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------020507050506060802020406" Return-path: In-Reply-To: <47A096CC.2070907@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "akira.ijuin@oracle.com" , "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------020507050506060802020406 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit > > Is the graph for RHEL5u1-64? (I've never tested this one.) I do not know which graph was attached with this. But I saw this behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I was running ltp tests continuously. > What was the behaviour of the other guests running? All pvm guests are fine. But behavior of most of the hvm guests were as described. > If they had spikes, were they at the same wall time? No. They are not at the same wall time. > Were the other guests running ltp as well? > Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp continuously. > How are you measuring skew? I was collecting output of "ntpdate -q every 300 seconds (5 minutes) and have created graph based on that. > > Are you running ntpd? > Yes. ntp was running on all the guests. I am investigating what causes this spikes and let everyone know what are my findings. Thanks, Deepak > Anything that you can discover that would be in sync with > the spikes would be very helpful! > > The code that I test with is our product code, which is based > on 3.1. So it is possible that something in 3.2 other than vpt.c > is the cause. I can test with 3.2, if necessary. > > thanks, > Dave > > > > Dan Magenheimer wrote: > >> Hi Dave (Keir, see suggestion below) -- >> >> Thanks! >> >> Turning off vhpet certainly helps a lot (though see below). >> >> I wonder if timekeeping with vhpet is so bad that it should be >> turned off by default (in 3.1, 3.2, and unstable) until it is >> fixed? (I have a patch that defaults it off, can post it if >> there is agreement on the above point.) The whole point of an >> HPET is to provide more precise timekeeping and if vhpet is >> worse than vpit, it can only confuse users. Comments? >> >> >> In your testing, are you just measuring % skew over a long >> period of time? >> We are graphing the skew continuously and >> seeing periodic behavior that is unsettling, even with pit. >> See attached. Though your algorithm recovers, the "cliffs" >> could still cause real user problems. I wonder if there is >> anything that can be done to make the "recovery" more >> responsive? >> >> We are looking into what part(s) of LTP is causing the cliffs. >> >> Thanks, >> Dan >> >> >> >>> -----Original Message----- >>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>> Sent: Monday, January 28, 2008 8:21 AM >>> To: dan.magenheimer@oracle.com >>> Cc: Keir Fraser; xen-devel@lists.xensource.com; >>> deepak.patel@oracle.com; >>> akira.ijuin@oracle.com; Dave Winchell >>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending >>> missed ticks >>> >>> >>> Dan, >>> >>> I guess I'm a bit out of date calling for clock= usage. >>> Looking at linux 2.6.20.4 sources, I think you should specify >>> "clocksource=pit nohpet" on the linux guest bootline. >>> >>> You can leave the xen and dom0 bootlines as they are. >>> The xen and guest clocksources do not need to be the same. >>> In my tests, xen is using the hpet for its timekeeping and >>> that appears to be the default. >>> >>> When you boot the guests you should see >>> time.c: Using PIT/TSC based timekeeping. >>> on the rh4u5-64 guest, and something similar on the others. >>> >>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>> > 14.318MHz HPET.) >>> >>> This appears to be the xen state, which is fine. >>> I was wrongly assuming that this was the guest state. >>> You might want to look in your guest logs and see what they were >>> picking >>> for a clock source. >>> >>> Regards, >>> Dave >>> >>> >>> >>> >>> Dan Magenheimer wrote: >>> >>> >>> >>>> Thanks, I hadn't realized that! No wonder we didn't see the same >>>> improvement you saw! >>>> >>>> >>>> >>>>> Try specifying clock=pit on the linux boot line... >>>>> >>>> >>>> I'm confused... do you mean "clocksource=pit" on the Xen >>> >>> command line or >>> >>> >>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or >>> >>> dom0?) command >>> >>> >>>> line? Or both places? Since the tests take awhile, it >>> >>> would be nice >>> >>> >>>> to get this right the first time. Do the Xen and guest >>> >>> clocksources need >>> >>> >>>> to be the same? >>>> >>>> Thanks, >>>> Dan >>>> >>>> -----Original Message----- >>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>> *Sent:* Sunday, January 27, 2008 2:22 PM >>>> *To:* dan.magenheimer@oracle.com; Keir Fraser >>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>> akira.ijuin@oracle.com; Dave Winchell >>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables >>>> pending missed ticks >>>> >>>> Hi Dan, >>>> >>>> Hpet timer does have a fairly large error, as I was trying this >>>> one recently. >>>> I don't remember what I got for error, but 1% sounds >>> >>> about right. >>> >>> >>>> The problem is that hpet is not built on top of vpt.c, >>> >>> the module >>> >>> >>>> Keir and I did >>>> all the recent work in, for its periodic timer needs. Try >>>> specifying clock=pit >>>> on the linux boot line. If it still picks the hpet, which it >>>> might, let me know >>>> and I'll tell you how to get around this. >>>> >>>> Regards, >>>> Dave >>>> >>>> >>>> >>>> >>>> >>> >>> -------------------------------------------------------------- >>> ---------- >>> >>> >>>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>> *Sent:* Fri 1/25/2008 6:50 PM >>>> *To:* Dave Winchell; Keir Fraser >>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>> akira.ijuin@oracle.com >>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>> >>> that disables >>> >>> >>>> pending missed ticks >>>> >>>> Sorry for the very late followup on this but we finally >>> >>> were able >>> >>> >>>> to get our testing set up again on stable 3.1 bits and have >>>> seen some very bad results on 3.1.3-rc1, on the order of 1%. >>>> >>>> Test enviroment was a 4-socket dual core machine with 24GB of >>>> memory running six two-vcpu 2GB domains, four hvm plus two pv. >>>> All six guests were running LTP simultaneously. The four hvm >>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. >>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>> >>> 32-bit guests. >>> >>> >>>> All four hvm guests experienced skew around -1%, even the 32-bit >>>> guest. Less intensive testing didn't exhibit much skew at all. >>>> >>>> A representative graph is attached. >>>> >>>> Dave, I wonder if some portion of your patches didn't end up in >>>> the xen trees? >>>> >>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>> 14.318MHz HPET.) >>>> >>>> Thanks, >>>> Dan >>>> >>>> P.S. Many thanks to Deepak and Akira for running tests. >>>> >>>> > -----Original Message----- >>>> > From: xen-devel-bounces@lists.xensource.com >>>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>> > Dave Winchell >>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>> > To: Keir Fraser >>>> > Cc: dan.magenheimer@oracle.com; >>> >>> xen-devel@lists.xensource.com; Dave >>> >>> >>>> > Winchell >>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>> > disables pending >>>> > missed ticks >>>> > >>>> > >>>> > Hi Keir, >>>> > >>>> > The latest change, c/s 16690, looks fine. >>>> > I agree that the code in c/s 16690 is equivalent to >>>> > the code I submitted. Also, your version is more >>>> > concise. >>>> > >>>> > The error tests confirm the equivalence. With >>> >>> overnight cpu loads, >>> >>> >>>> > the checked in version was accurate to +.048% for sles >>>> > and +.038% for red hat. My version was +.046% and +.032% in a >>>> > 2 hour test. >>>> > I don't think the difference is significant. >>>> > >>>> > i/o loads produced errors of +.01%. >>>> > >>>> > Thanks for all your efforts on this issue. >>>> > >>>> > Regards, >>>> > Dave >>>> > >>>> > >>>> > >>>> > Keir Fraser wrote: >>>> > >>>> > >Applied as c/s 16690, although the checked-in patch is >>>> > smaller. I think the >>>> > >only important fix is to pt_intr_post() and the only bit of >>>> > the patch I >>>> > >totally omitted was the change to pt_process_missed_ticks(). >>>> > I don't think >>>> > >that change can be important, but let's see what >>> >>> happens to the >>> >>> >>>> error >>>> > >percentage... >>>> > > >>>> > > -- Keir >>>> > > >>>> > >On 4/1/08 23:24, "Dave Winchell" >>> >>> wrote: >>> >>> >>>> > > >>>> > > >>>> > > >>>> > >>Hi Dan and Keir, >>>> > >> >>>> > >>Attached is a patch that fixes some issues with the >>> >>> SYNC policy >>> >>> >>>> > >>(no_missed_ticks_pending). >>>> > >>I have not tried to make the change the minimal one, but, >>>> > rather, just >>>> > >>ported into >>>> > >>the new code what I know to work well. The error for >>>> > >>no_missed_ticks_pending goes from >>>> > >>over 3% to .03% with this change according to my testing. >>>> > >> >>>> > >>Regards, >>>> > >>Dave >>>> > >> >>>> > >>Dan Magenheimer wrote: >>>> > >> >>>> > >> >>>> > >> >>>> > >>>Hi Dave -- >>>> > >>> >>>> > >>>Did you get your correction ported? If so, it would be >>>> > nice to see this get >>>> > >>>into 3.1.3. >>>> > >>> >>>> > >>>Note that I just did some very limited testing with >>>> > timer_mode=2(=SYNC=no >>>> > >>>missed ticks pending) >>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the >>>> > worst error I've >>>> > >>>seen so far >>>> > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. >>>> > >>> >>>> > >>>Thanks, >>>> > >>>Dan >>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>>>-----Original Message----- >>>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>> > >>>>To: dan.magenheimer@oracle.com >>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>> > xen-devel@lists.xensource.com; Dong, >>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>> > >>>>disables pending >>>> > >>>>missed ticks >>>> > >>>> >>>> > >>>> >>>> > >>>>Dan, >>>> > >>>> >>>> > >>>>I did some testing with the constant tsc offset >>> >>> SYNC method >>> >>> >>>> > >>>>(now called >>>> > >>>>no_missed_ticks_pending) >>>> > >>>>and found the error to be very high, much larger >>> >>> than 1 %, as >>> >>> >>>> > >>>>I recall. >>>> > >>>>I have not had a chance to submit a correction. I >>> >>> will try to >>> >>> >>>> > >>>>do it later >>>> > >>>>this week or the first week in January. My version of >>>> constant tsc >>>> > >>>>offset SYNC method >>>> > >>>>produces .02 % error, so I just need to port that into the >>>> > >>>>current code. >>>> > >>>> >>>> > >>>>The error you got for both of those kernels is >>> >>> what I would >>> >>> >>>> expect >>>> > >>>>for the default mode, delay_for_missed_ticks. >>>> > >>>> >>>> > >>>>I'll let Keir answer on how to set the time mode. >>>> > >>>> >>>> > >>>>Regards, >>>> > >>>>Dave >>>> > >>>> >>>> > >>>>Dan Magenheimer wrote: >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>>>Anyone make measurements on the final patch? >>>> > >>>>> >>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>about 0.2% with no load. This was xen-unstable tip today >>>> > >>>>with no options specified. 32-bit was about 0.01%. >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>>>I think I missed something... how do I run the various >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>accounting choices and which ones are known to be >>> >>> appropriate >>> >>> >>>> > >>>>for which kernels? >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>>>Thanks, >>>> > >>>>>Dan >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>>>-----Original Message----- >>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>> > >>>> >>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>>> >>>>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>Keir Fraser >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>> > >>>>>>To: Dave Winchell >>>> > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>>> > Eddie; Jiang, >>>> > >>>>>>Yunhong >>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>> > >>>>>>disables pending >>>> > >>>>>>missed ticks >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>Please take a look at xen-unstable changeset 16545. >>>> > >>>>>> >>>> > >>>>>>-- Keir >>>> > >>>>>> >>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>> wrote: >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>Keir, >>>> > >>>>>>> >>>> > >>>>>>>The accuracy data I've collected for i/o loads for the >>>> > >>>>>>>various time protocols follows. In addition, the data >>>> > >>>>>>>for cpu loads is shown. >>>> > >>>>>>> >>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>> >>> processor AMD >>> >>> >>>> box. >>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>> > >>>>>>>(usex is available at >>>> http://people.redhat.com/anderson/usex.) >>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>> >>> of=/dev/null. >>> >>> >>>> > >>>>>>> >>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>> > >>>>>>>All three guests are 8vcpu. >>>> > >>>>>>> >>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>> > >>>>>>>except that the redhat-64 guest has 4 instances of dd. >>>> > >>>>>>> >>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>> > >>>>>>> >>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, >>>> > +.005% cpu >>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, >>>> > +.012% cpu >>>> > >>>>>>> >>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, >>>> -.004% cpu >>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>> >>> -.005%, -.005% cpu >>> >>> >>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>> >>> -.008%, -.003% cpu >>> >>> >>>> > >>>>>>> >>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>> >>> -.040% cpu >>> >>> >>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, >>>> > -.031% cpu >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, >>>> > -.09% i/o-8 >>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% >>>> > -.14% i/o-8 >>>> > >>>>>>> >>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, >>>> > -.022% i/o-8 >>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>> >>> -.017%, -.018% i/o-8 >>> >>> >>>> > >>>>>>> >>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, >>>> > -.029% i/o-8 >>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, >>>> > -.031% i/o-8 >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>> >>> -.04% i/o-32 >>> >>> >>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, >>>> > -.005% i/o-32 >>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>> >>> -.11% i/o-32 >>> >>> >>>> > >>>>>>> >>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, >>>> > .003% i/o-4/32 >>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, >>>> > .01% i/o-4/32 >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>>Overhead measurements: >>>> > >>>>>>> >>>> > >>>>>>>Progress in terms of number of passes through a fixed >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>system workload >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>> > >>>>>>>The workload was usex -b48. >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>>Conclusions: >>>> > >>>>>>> >>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>> > requirement for ntp >>>> > >>>>>>>tracking under the loads >>>> > >>>>>>>above is the SYNC protocol. The worst case >>> >>> accuracies for >>> >>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>SYNC, MIXED, >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>and ASYNC >>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>> > >>>>>>> >>>> > >>>>>>>We could reduce the cost of the SYNC method by only >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>scheduling the extra >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>wakeups if a certain number >>>> > >>>>>>>of ticks are missed. >>>> > >>>>>>> >>>> > >>>>>>>Regards, >>>> > >>>>>>>Dave >>>> > >>>>>>> >>>> > >>>>>>>Keir Fraser wrote: >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>> >>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>> wrote: >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>> >>> ASYNC method a >>> >>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>couple of days ago, >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>I ran another ASYNC test. I think there may have >>>> > been something >>>> > >>>>>>>>>wrong with the code I used a couple of days ago for >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>ASYNC. It may have been >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>missing the immediate delivery of interrupt >>> >>> after context >>> >>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>switch in. >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>acceptable accuracy, >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>each running consistently around or under >>> >>> .01%. MIXED has >>> >>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>a fairly high >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>error of >>>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>threshold for comfort. >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>> >>> plan to leave >>> >>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>SYNC running >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>running instead. >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>It may be too early to pick the protocol and >>> >>> I can run >>> >>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>more overnight tests >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>>next week. >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>> >>>> > >>>>>>>>I'm a bit worried about any unwanted side >>> >>> effects of the >>> >>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>SYNC+run_timer >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>> >>> cause higher >>> >>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>system-wide CPU >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>contention. I find it easier to think through the >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>implications of ASYNC. I'm >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>surprised that MIXED loses time, and is less >>> >>> accurate than >>> >>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>ASYNC. Perhaps it >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>delivers more timer interrupts than the other >>> >>> approaches, >>> >>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>and each interrupt >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>event causes a small accumulated error? >>>> > >>>>>>>> >>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>> >>> favourites and >>> >>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>if the latter is >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>actually more accurate then I can simply revert the >>>> > changeset that >>>> > >>>>>>>>implemented MIXED. >>>> > >>>>>>>> >>>> > >>>>>>>>Perhaps rather than running more of the same >>> >>> workloads you >>> >>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>could try idle >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>> >>> large disc reads >>> >>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>to /dev/null)? We >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>don't have any data on workloads that aren't >>> >>> CPU bound, so >>> >>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>that's really an >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>>>>obvious place to put any further effort imo. >>>> > >>>>>>>> >>>> > >>>>>>>>-- Keir >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>>> >>>> > >>>>>>_______________________________________________ >>>> > >>>>>>Xen-devel mailing list >>>> > >>>>>>Xen-devel@lists.xensource.com >>>> > >>>>>>http://lists.xensource.com/xen-devel >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 >>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 >>>> > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru >>>> > >> >>>> > >> missed_ticks = missed_ticks / (s_time_t) >>> >>> pt->period + 1; >>> >>> >>>> > >> if ( mode_is(pt->vcpu->domain, >>> >>> no_missed_ticks_pending) ) >>> >>> >>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>> > >>+ pt->do_not_freeze = 1; >>>> > >> else >>>> > >> pt->pending_intr_nr += missed_ticks; >>>> > >> pt->scheduled += missed_ticks * pt->period; >>>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >>>> > >> >>>> > >> pt_lock(pt); >>>> > >> >>>> > >>- pt->pending_intr_nr++; >>>> > >>+ if ( mode_is(pt->vcpu->domain, >>> >>> no_missed_ticks_pending) ) { >>> >>> >>>> > >>+ pt->pending_intr_nr = 1; >>>> > >>+ pt->do_not_freeze = 0; >>>> > >>+ } >>>> > >>+ else >>>> > >>+ pt->pending_intr_nr++; >>>> > >> >>>> > >> if ( !pt->one_shot ) >>>> > >> { >>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct >>>> > >> return; >>>> > >> } >>>> > >> >>>> > >>- pt->do_not_freeze = 0; >>>> > >>- >>>> > >> if ( pt->one_shot ) >>>> > >> { >>>> > >> pt->enabled = 0; >>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>> >>> *v, struct >>> >>> >>>> > >> pt->last_plt_gtime = hvm_get_guest_time(v); >>>> > >> pt->pending_intr_nr = 0; /* 'collapse' all >>>> > missed ticks */ >>>> > >> } >>>> > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { >>>> > >>+ pt->pending_intr_nr--; >>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>> > >>+ } >>>> > >> else >>>> > >> { >>>> > >> pt->last_plt_gtime += pt->period_cycles; >>>> > >> >>>> > >> >>>> > > >>>> > > >>>> > > >>>> > > >>>> > >>>> > >>>> > _______________________________________________ >>>> > Xen-devel mailing list >>>> > Xen-devel@lists.xensource.com >>>> > http://lists.xensource.com/xen-devel >>>> > >>>> >>>> >>> > --------------020507050506060802020406 Content-Type: text/x-vcard; charset=utf-8; name="deepak.patel.vcf" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="deepak.patel.vcf" begin:vcard fn:Deepak Patel n:Patel;Deepak email;internet:deepak.patel@oracle.com version:2.1 end:vcard --------------020507050506060802020406 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------020507050506060802020406-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 30 Jan 2008 16:44:39 -0500 Message-ID: <47A0EFC7.3020102@virtualiron.com> References: <20080129153458531.00000002384@djm-pc> <47A096CC.2070907@virtualiron.com> <47A0E650.3070506@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47A0E650.3070506@oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Deepak Patel Cc: "akira.ijuin@oracle.com" , "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Dave Winchell List-Id: xen-devel@lists.xenproject.org Dan, Deeepak, It may be that the underlying clock error is too great for ntp to handle. It would be useful if you did not run ntpd and, instead did ntpdate -b at the start of the test for each guest. Then capture the data as you have been doing. If the drift is greater than .05%, then we need to address that. Another option is, when running ntpd, to enable loop statistics in /etc/ntp.conf by adding this to the file: statistics loopstats statsdir /var/lib/ntp/ Then you will see loop data in that directory. Correlating the data in the loopstats files with the peaks in skew would be interesting. You will see entries of the form 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10 Where the second to last column is the Allan Deviation. When that gets over 1000, ntpd is working pretty hard. However, I have not seen ntpd completely loose it like you have. I'm on vacation until Monday, and won't be reading email. Thanks for all your work on this! -Dave Deepak Patel wrote: > >> >> Is the graph for RHEL5u1-64? (I've never tested this one.) > > > I do not know which graph was attached with this. But I saw this > behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I > was running ltp tests continuously. > >> What was the behaviour of the other guests running? > > > All pvm guests are fine. But behavior of most of the hvm guests were > as described. > >> If they had spikes, were they at the same wall time? > > > No. They are not at the same wall time. > >> Were the other guests running ltp as well? >> > Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp > continuously. > >> How are you measuring skew? > > > I was collecting output of "ntpdate -q every 300 seconds > (5 minutes) and have created graph based on that. > >> >> Are you running ntpd? >> > Yes. ntp was running on all the guests. > > I am investigating what causes this spikes and let everyone know what > are my findings. > > Thanks, > Deepak > >> Anything that you can discover that would be in sync with >> the spikes would be very helpful! >> >> The code that I test with is our product code, which is based >> on 3.1. So it is possible that something in 3.2 other than vpt.c >> is the cause. I can test with 3.2, if necessary. >> >> thanks, >> Dave >> >> >> >> Dan Magenheimer wrote: >> >>> Hi Dave (Keir, see suggestion below) -- >>> >>> Thanks! >>> >>> Turning off vhpet certainly helps a lot (though see below). >>> >>> I wonder if timekeeping with vhpet is so bad that it should be >>> turned off by default (in 3.1, 3.2, and unstable) until it is >>> fixed? (I have a patch that defaults it off, can post it if >>> there is agreement on the above point.) The whole point of an >>> HPET is to provide more precise timekeeping and if vhpet is >>> worse than vpit, it can only confuse users. Comments? >>> >>> >>> In your testing, are you just measuring % skew over a long >>> period of time? >>> We are graphing the skew continuously and >>> seeing periodic behavior that is unsettling, even with pit. >>> See attached. Though your algorithm recovers, the "cliffs" >>> could still cause real user problems. I wonder if there is >>> anything that can be done to make the "recovery" more >>> responsive? >>> >>> We are looking into what part(s) of LTP is causing the cliffs. >>> >>> Thanks, >>> Dan >>> >>> >>> >>>> -----Original Message----- >>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>> Sent: Monday, January 28, 2008 8:21 AM >>>> To: dan.magenheimer@oracle.com >>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>> deepak.patel@oracle.com; >>>> akira.ijuin@oracle.com; Dave Winchell >>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>> pending >>>> missed ticks >>>> >>>> >>>> Dan, >>>> >>>> I guess I'm a bit out of date calling for clock= usage. >>>> Looking at linux 2.6.20.4 sources, I think you should specify >>>> "clocksource=pit nohpet" on the linux guest bootline. >>>> >>>> You can leave the xen and dom0 bootlines as they are. >>>> The xen and guest clocksources do not need to be the same. >>>> In my tests, xen is using the hpet for its timekeeping and >>>> that appears to be the default. >>>> >>>> When you boot the guests you should see >>>> time.c: Using PIT/TSC based timekeeping. >>>> on the rh4u5-64 guest, and something similar on the others. >>>> >>>> > (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>> > 14.318MHz HPET.) >>>> >>>> This appears to be the xen state, which is fine. >>>> I was wrongly assuming that this was the guest state. >>>> You might want to look in your guest logs and see what they were >>>> picking >>>> for a clock source. >>>> >>>> Regards, >>>> Dave >>>> >>>> >>>> >>>> >>>> Dan Magenheimer wrote: >>>> >>>> >>>> >>>>> Thanks, I hadn't realized that! No wonder we didn't see the same >>>>> improvement you saw! >>>>> >>>>> >>>>> >>>>>> Try specifying clock=pit on the linux boot line... >>>>>> >>>>> >>>>> >>>>> I'm confused... do you mean "clocksource=pit" on the Xen >>>> >>>> >>>> command line or >>>> >>>> >>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or >>>> >>>> >>>> dom0?) command >>>> >>>> >>>>> line? Or both places? Since the tests take awhile, it >>>> >>>> >>>> would be nice >>>> >>>> >>>>> to get this right the first time. Do the Xen and guest >>>> >>>> >>>> clocksources need >>>> >>>> >>>>> to be the same? >>>>> >>>>> Thanks, >>>>> Dan >>>>> >>>>> -----Original Message----- >>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>> *Sent:* Sunday, January 27, 2008 2:22 PM >>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser >>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>> akira.ijuin@oracle.com; Dave Winchell >>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode that disables >>>>> pending missed ticks >>>>> >>>>> Hi Dan, >>>>> >>>>> Hpet timer does have a fairly large error, as I was trying this >>>>> one recently. >>>>> I don't remember what I got for error, but 1% sounds >>>> >>>> >>>> about right. >>>> >>>> >>>>> The problem is that hpet is not built on top of vpt.c, >>>> >>>> >>>> the module >>>> >>>> >>>>> Keir and I did >>>>> all the recent work in, for its periodic timer needs. Try >>>>> specifying clock=pit >>>>> on the linux boot line. If it still picks the hpet, which it >>>>> might, let me know >>>>> and I'll tell you how to get around this. >>>>> >>>>> Regards, >>>>> Dave >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -------------------------------------------------------------- >>>> ---------- >>>> >>>> >>>>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>> *Sent:* Fri 1/25/2008 6:50 PM >>>>> *To:* Dave Winchell; Keir Fraser >>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>> akira.ijuin@oracle.com >>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>> >>>> >>>> that disables >>>> >>>> >>>>> pending missed ticks >>>>> >>>>> Sorry for the very late followup on this but we finally >>>> >>>> >>>> were able >>>> >>>> >>>>> to get our testing set up again on stable 3.1 bits and have >>>>> seen some very bad results on 3.1.3-rc1, on the order of 1%. >>>>> >>>>> Test enviroment was a 4-socket dual core machine with 24GB of >>>>> memory running six two-vcpu 2GB domains, four hvm plus two pv. >>>>> All six guests were running LTP simultaneously. The four hvm >>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and RHEL4u5-64. >>>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>>> >>>> >>>> 32-bit guests. >>>> >>>> >>>>> All four hvm guests experienced skew around -1%, even the 32-bit >>>>> guest. Less intensive testing didn't exhibit much skew at all. >>>>> >>>>> A representative graph is attached. >>>>> >>>>> Dave, I wonder if some portion of your patches didn't end up in >>>>> the xen trees? >>>>> >>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>> 14.318MHz HPET.) >>>>> >>>>> Thanks, >>>>> Dan >>>>> >>>>> P.S. Many thanks to Deepak and Akira for running tests. >>>>> >>>>> > -----Original Message----- >>>>> > From: xen-devel-bounces@lists.xensource.com >>>>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>> > Dave Winchell >>>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>>> > To: Keir Fraser >>>>> > Cc: dan.magenheimer@oracle.com; >>>> >>>> >>>> xen-devel@lists.xensource.com; Dave >>>> >>>> >>>>> > Winchell >>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>> > disables pending >>>>> > missed ticks >>>>> > >>>>> > >>>>> > Hi Keir, >>>>> > >>>>> > The latest change, c/s 16690, looks fine. >>>>> > I agree that the code in c/s 16690 is equivalent to >>>>> > the code I submitted. Also, your version is more >>>>> > concise. >>>>> > >>>>> > The error tests confirm the equivalence. With >>>> >>>> >>>> overnight cpu loads, >>>> >>>> >>>>> > the checked in version was accurate to +.048% for sles >>>>> > and +.038% for red hat. My version was +.046% and +.032% in a >>>>> > 2 hour test. >>>>> > I don't think the difference is significant. >>>>> > >>>>> > i/o loads produced errors of +.01%. >>>>> > >>>>> > Thanks for all your efforts on this issue. >>>>> > >>>>> > Regards, >>>>> > Dave >>>>> > >>>>> > >>>>> > >>>>> > Keir Fraser wrote: >>>>> > >>>>> > >Applied as c/s 16690, although the checked-in patch is >>>>> > smaller. I think the >>>>> > >only important fix is to pt_intr_post() and the only bit of >>>>> > the patch I >>>>> > >totally omitted was the change to pt_process_missed_ticks(). >>>>> > I don't think >>>>> > >that change can be important, but let's see what >>>> >>>> >>>> happens to the >>>> >>>> >>>>> error >>>>> > >percentage... >>>>> > > >>>>> > > -- Keir >>>>> > > >>>>> > >On 4/1/08 23:24, "Dave Winchell" >>>> >>>> >>>> wrote: >>>> >>>> >>>>> > > >>>>> > > >>>>> > > >>>>> > >>Hi Dan and Keir, >>>>> > >> >>>>> > >>Attached is a patch that fixes some issues with the >>>> >>>> >>>> SYNC policy >>>> >>>> >>>>> > >>(no_missed_ticks_pending). >>>>> > >>I have not tried to make the change the minimal one, but, >>>>> > rather, just >>>>> > >>ported into >>>>> > >>the new code what I know to work well. The error for >>>>> > >>no_missed_ticks_pending goes from >>>>> > >>over 3% to .03% with this change according to my testing. >>>>> > >> >>>>> > >>Regards, >>>>> > >>Dave >>>>> > >> >>>>> > >>Dan Magenheimer wrote: >>>>> > >> >>>>> > >> >>>>> > >> >>>>> > >>>Hi Dave -- >>>>> > >>> >>>>> > >>>Did you get your correction ported? If so, it would be >>>>> > nice to see this get >>>>> > >>>into 3.1.3. >>>>> > >>> >>>>> > >>>Note that I just did some very limited testing with >>>>> > timer_mode=2(=SYNC=no >>>>> > >>>missed ticks pending) >>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv guest) and the >>>>> > worst error I've >>>>> > >>>seen so far >>>>> > >>>is 0.012%. But I haven't tried any exotic loads, just LTP. >>>>> > >>> >>>>> > >>>Thanks, >>>>> > >>>Dan >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>>>-----Original Message----- >>>>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>> > >>>>To: dan.magenheimer@oracle.com >>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>>> > xen-devel@lists.xensource.com; Dong, >>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>> > >>>>disables pending >>>>> > >>>>missed ticks >>>>> > >>>> >>>>> > >>>> >>>>> > >>>>Dan, >>>>> > >>>> >>>>> > >>>>I did some testing with the constant tsc offset >>>> >>>> >>>> SYNC method >>>> >>>> >>>>> > >>>>(now called >>>>> > >>>>no_missed_ticks_pending) >>>>> > >>>>and found the error to be very high, much larger >>>> >>>> >>>> than 1 %, as >>>> >>>> >>>>> > >>>>I recall. >>>>> > >>>>I have not had a chance to submit a correction. I >>>> >>>> >>>> will try to >>>> >>>> >>>>> > >>>>do it later >>>>> > >>>>this week or the first week in January. My version of >>>>> constant tsc >>>>> > >>>>offset SYNC method >>>>> > >>>>produces .02 % error, so I just need to port that into the >>>>> > >>>>current code. >>>>> > >>>> >>>>> > >>>>The error you got for both of those kernels is >>>> >>>> >>>> what I would >>>> >>>> >>>>> expect >>>>> > >>>>for the default mode, delay_for_missed_ticks. >>>>> > >>>> >>>>> > >>>>I'll let Keir answer on how to set the time mode. >>>>> > >>>> >>>>> > >>>>Regards, >>>>> > >>>>Dave >>>>> > >>>> >>>>> > >>>>Dan Magenheimer wrote: >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>>>Anyone make measurements on the final patch? >>>>> > >>>>> >>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and saw a loss of >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>about 0.2% with no load. This was xen-unstable tip today >>>>> > >>>>with no options specified. 32-bit was about 0.01%. >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>>>I think I missed something... how do I run the various >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>accounting choices and which ones are known to be >>>> >>>> >>>> appropriate >>>> >>>> >>>>> > >>>>for which kernels? >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>>>Thanks, >>>>> > >>>>>Dan >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>>>-----Original Message----- >>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>> > >>>>> >>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>Keir Fraser >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>> > >>>>>>To: Dave Winchell >>>>> > >>>>>>Cc: Shan, Haitao; xen-devel@lists.xensource.com; Dong, >>>>> > Eddie; Jiang, >>>>> > >>>>>>Yunhong >>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>> > >>>>>>disables pending >>>>> > >>>>>>missed ticks >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. >>>>> > >>>>>> >>>>> > >>>>>>-- Keir >>>>> > >>>>>> >>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>> wrote: >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>Keir, >>>>> > >>>>>>> >>>>> > >>>>>>>The accuracy data I've collected for i/o loads for the >>>>> > >>>>>>>various time protocols follows. In addition, the data >>>>> > >>>>>>>for cpu loads is shown. >>>>> > >>>>>>> >>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>> >>>> >>>> processor AMD >>>> >>>> >>>>> box. >>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>>> > >>>>>>>(usex is available at >>>>> http://people.redhat.com/anderson/usex.) >>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>> >>>> >>>> of=/dev/null. >>>> >>>> >>>>> > >>>>>>> >>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>>> > >>>>>>>All three guests are 8vcpu. >>>>> > >>>>>>> >>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>> > >>>>>>>except that the redhat-64 guest has 4 instances of dd. >>>>> > >>>>>>> >>>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>>> > >>>>>>> >>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 sec -.006%, >>>>> > +.005% cpu >>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 sec, -.001%, >>>>> > +.012% cpu >>>>> > >>>>>>> >>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, >>>>> -.004% cpu >>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>> >>>> >>>> -.005%, -.005% cpu >>>> >>>> >>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>> >>>> >>>> -.008%, -.003% cpu >>>> >>>> >>>>> > >>>>>>> >>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>> >>>> >>>> -.040% cpu >>>> >>>> >>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 sec, -.034%, >>>>> > -.031% cpu >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, >>>>> > -.09% i/o-8 >>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% >>>>> > -.14% i/o-8 >>>>> > >>>>>>> >>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, >>>>> > -.022% i/o-8 >>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>> >>>> >>>> -.017%, -.018% i/o-8 >>>> >>>> >>>>> > >>>>>>> >>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, >>>>> > -.029% i/o-8 >>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 sec, -.023%, >>>>> > -.031% i/o-8 >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>>> >>>> >>>> -.04% i/o-32 >>>> >>>> >>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, >>>>> > -.005% i/o-32 >>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>>> >>>> >>>> -.11% i/o-32 >>>> >>>> >>>>> > >>>>>>> >>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, >>>>> > .003% i/o-4/32 >>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, >>>>> > .01% i/o-4/32 >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>>Overhead measurements: >>>>> > >>>>>>> >>>>> > >>>>>>>Progress in terms of number of passes through a fixed >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>system workload >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>> > >>>>>>>The workload was usex -b48. >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>>Conclusions: >>>>> > >>>>>>> >>>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>>> > requirement for ntp >>>>> > >>>>>>>tracking under the loads >>>>> > >>>>>>>above is the SYNC protocol. The worst case >>>> >>>> >>>> accuracies for >>>> >>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>SYNC, MIXED, >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>and ASYNC >>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>>> > >>>>>>> >>>>> > >>>>>>>We could reduce the cost of the SYNC method by only >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>scheduling the extra >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>wakeups if a certain number >>>>> > >>>>>>>of ticks are missed. >>>>> > >>>>>>> >>>>> > >>>>>>>Regards, >>>>> > >>>>>>>Dave >>>>> > >>>>>>> >>>>> > >>>>>>>Keir Fraser wrote: >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>> >>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>> wrote: >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>>> >>>> >>>> ASYNC method a >>>> >>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>couple of days ago, >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have >>>>> > been something >>>>> > >>>>>>>>>wrong with the code I used a couple of days ago for >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>ASYNC. It may have been >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>missing the immediate delivery of interrupt >>>> >>>> >>>> after context >>>> >>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>switch in. >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>acceptable accuracy, >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>each running consistently around or under >>>> >>>> >>>> .01%. MIXED has >>>> >>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>a fairly high >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>error of >>>>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>threshold for comfort. >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>>> >>>> >>>> plan to leave >>>> >>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>SYNC running >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>over the weekend. If you'd rather I can leave MIXED >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>running instead. >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>It may be too early to pick the protocol and >>>> >>>> >>>> I can run >>>> >>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>more overnight tests >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>>next week. >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>> >>>>> > >>>>>>>>I'm a bit worried about any unwanted side >>>> >>>> >>>> effects of the >>>> >>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>SYNC+run_timer >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>>> >>>> >>>> cause higher >>>> >>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>system-wide CPU >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>contention. I find it easier to think through the >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>implications of ASYNC. I'm >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>surprised that MIXED loses time, and is less >>>> >>>> >>>> accurate than >>>> >>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>ASYNC. Perhaps it >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>delivers more timer interrupts than the other >>>> >>>> >>>> approaches, >>>> >>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>and each interrupt >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>event causes a small accumulated error? >>>>> > >>>>>>>> >>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>>> >>>> >>>> favourites and >>>> >>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>if the latter is >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>actually more accurate then I can simply revert the >>>>> > changeset that >>>>> > >>>>>>>>implemented MIXED. >>>>> > >>>>>>>> >>>>> > >>>>>>>>Perhaps rather than running more of the same >>>> >>>> >>>> workloads you >>>> >>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>could try idle >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>> >>>> >>>> large disc reads >>>> >>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>to /dev/null)? We >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>don't have any data on workloads that aren't >>>> >>>> >>>> CPU bound, so >>>> >>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>that's really an >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>>>>obvious place to put any further effort imo. >>>>> > >>>>>>>> >>>>> > >>>>>>>>-- Keir >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>>> >>>>> > >>>>>>_______________________________________________ >>>>> > >>>>>>Xen-devel mailing list >>>>> > >>>>>>Xen-devel@lists.xensource.com >>>>> > >>>>>>http://lists.xensource.com/xen-devel >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 2007 +0000 >>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 2008 -0500 >>>>> > >>@@ -58,7 +58,7 @@ static void pt_process_missed_ticks(stru >>>>> > >> >>>>> > >> missed_ticks = missed_ticks / (s_time_t) >>>> >>>> >>>> pt->period + 1; >>>> >>>> >>>>> > >> if ( mode_is(pt->vcpu->domain, >>>> >>>> >>>> no_missed_ticks_pending) ) >>>> >>>> >>>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>> > >>+ pt->do_not_freeze = 1; >>>>> > >> else >>>>> > >> pt->pending_intr_nr += missed_ticks; >>>>> > >> pt->scheduled += missed_ticks * pt->period; >>>>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >>>>> > >> >>>>> > >> pt_lock(pt); >>>>> > >> >>>>> > >>- pt->pending_intr_nr++; >>>>> > >>+ if ( mode_is(pt->vcpu->domain, >>>> >>>> >>>> no_missed_ticks_pending) ) { >>>> >>>> >>>>> > >>+ pt->pending_intr_nr = 1; >>>>> > >>+ pt->do_not_freeze = 0; >>>>> > >>+ } >>>>> > >>+ else >>>>> > >>+ pt->pending_intr_nr++; >>>>> > >> >>>>> > >> if ( !pt->one_shot ) >>>>> > >> { >>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct vcpu *v, struct >>>>> > >> return; >>>>> > >> } >>>>> > >> >>>>> > >>- pt->do_not_freeze = 0; >>>>> > >>- >>>>> > >> if ( pt->one_shot ) >>>>> > >> { >>>>> > >> pt->enabled = 0; >>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>> >>>> >>>> *v, struct >>>> >>>> >>>>> > >> pt->last_plt_gtime = hvm_get_guest_time(v); >>>>> > >> pt->pending_intr_nr = 0; /* 'collapse' all >>>>> > missed ticks */ >>>>> > >> } >>>>> > >>+ else if ( mode_is(v->domain, no_missed_ticks_pending) ) { >>>>> > >>+ pt->pending_intr_nr--; >>>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>> > >>+ } >>>>> > >> else >>>>> > >> { >>>>> > >> pt->last_plt_gtime += pt->period_cycles; >>>>> > >> >>>>> > >> >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > Xen-devel mailing list >>>>> > Xen-devel@lists.xensource.com >>>>> > http://lists.xensource.com/xen-devel >>>>> > >>>>> >>>>> >>>> >>>> >> From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Mon, 04 Feb 2008 15:07:17 -0500 Message-ID: <47A77075.3010707@virtualiron.com> References: <20080201153111687.00000002384@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080201153111687.00000002384@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , Deepak Patel , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Dan, Deepak: Thanks for the data. Those drifts are severe - no wonder ntp couldn't keep then in synch. I'll try to reproduce that behaviour here, with my code base. If I can't reproduce it, I'll try 3.2. If you can isolate what ltp is doing during the cliffs, that would be very helpful. thanks, Dave Dan Magenheimer wrote: >OK, Deepak repeated the test without ntpd and using ntpdate -b before >the test. > >The attached graph shows his results: el5u1-64 (best=~0.07%), >el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%). > >We will continue to look at LTP to try to isolate. > >Thanks, >Dan > >P.S. elXuY is essentially RHEL XuY with some patches. > > > >>-----Original Message----- >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>Sent: Wednesday, January 30, 2008 2:45 PM >>To: Deepak Patel >>Cc: dan.magenheimer@oracle.com; Keir Fraser; >>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Dan, Deeepak, >> >>It may be that the underlying clock error is too great for ntp >>to handle. It would be useful if you did not run ntpd >>and, instead did ntpdate -b at the start of the test >>for each guest. Then capture the data as you have been doing. >>If the drift is greater than .05%, then we need to address that. >> >>Another option is, when running ntpd, to enable loop statistics in >>/etc/ntp.conf >>by adding this to the file: >> >>statistics loopstats >>statsdir /var/lib/ntp/ >> >>Then you will see loop data in that directory. >>Correlating the data in the loopstats files with the >>peaks in skew would be interesting. You will see entries of the form >> >>54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10 >> >>Where the second to last column is the Allan Deviation. When that >>gets over 1000, ntpd is working pretty hard. However, I have >>not seen ntpd >>completely loose it like you have. >> >>I'm on vacation until Monday, and won't be reading >>email. >> >>Thanks for all your work on this! >> >>-Dave >> >>Deepak Patel wrote: >> >> >> >>>>Is the graph for RHEL5u1-64? (I've never tested this one.) >>>> >>>> >>>I do not know which graph was attached with this. But I saw this >>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I >>>was running ltp tests continuously. >>> >>> >>> >>>>What was the behaviour of the other guests running? >>>> >>>> >>>All pvm guests are fine. But behavior of most of the hvm guests were >>>as described. >>> >>> >>> >>>>If they had spikes, were they at the same wall time? >>>> >>>> >>>No. They are not at the same wall time. >>> >>> >>> >>>>Were the other guests running ltp as well? >>>> >>>> >>>> >>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp >>>continuously. >>> >>> >>> >>>>How are you measuring skew? >>>> >>>> >>>I was collecting output of "ntpdate -q every >>> >>> >>300 seconds >> >> >>>(5 minutes) and have created graph based on that. >>> >>> >>> >>>>Are you running ntpd? >>>> >>>> >>>> >>>Yes. ntp was running on all the guests. >>> >>>I am investigating what causes this spikes and let everyone >>> >>> >>know what >> >> >>>are my findings. >>> >>>Thanks, >>>Deepak >>> >>> >>> >>>>Anything that you can discover that would be in sync with >>>>the spikes would be very helpful! >>>> >>>>The code that I test with is our product code, which is based >>>>on 3.1. So it is possible that something in 3.2 other than vpt.c >>>>is the cause. I can test with 3.2, if necessary. >>>> >>>>thanks, >>>>Dave >>>> >>>> >>>> >>>>Dan Magenheimer wrote: >>>> >>>> >>>> >>>>>Hi Dave (Keir, see suggestion below) -- >>>>> >>>>>Thanks! >>>>> >>>>>Turning off vhpet certainly helps a lot (though see below). >>>>> >>>>>I wonder if timekeeping with vhpet is so bad that it should be >>>>>turned off by default (in 3.1, 3.2, and unstable) until it is >>>>>fixed? (I have a patch that defaults it off, can post it if >>>>>there is agreement on the above point.) The whole point of an >>>>>HPET is to provide more precise timekeeping and if vhpet is >>>>>worse than vpit, it can only confuse users. Comments? >>>>> >>>>> >>>>>In your testing, are you just measuring % skew over a long >>>>>period of time? >>>>> We are graphing the skew continuously and >>>>>seeing periodic behavior that is unsettling, even with pit. >>>>>See attached. Though your algorithm recovers, the "cliffs" >>>>>could still cause real user problems. I wonder if there is >>>>>anything that can be done to make the "recovery" more >>>>>responsive? >>>>> >>>>>We are looking into what part(s) of LTP is causing the cliffs. >>>>> >>>>>Thanks, >>>>>Dan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>-----Original Message----- >>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>Sent: Monday, January 28, 2008 8:21 AM >>>>>>To: dan.magenheimer@oracle.com >>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>>>>deepak.patel@oracle.com; >>>>>>akira.ijuin@oracle.com; Dave Winchell >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>>>>pending >>>>>>missed ticks >>>>>> >>>>>> >>>>>>Dan, >>>>>> >>>>>>I guess I'm a bit out of date calling for clock= usage. >>>>>>Looking at linux 2.6.20.4 sources, I think you should specify >>>>>>"clocksource=pit nohpet" on the linux guest bootline. >>>>>> >>>>>>You can leave the xen and dom0 bootlines as they are. >>>>>>The xen and guest clocksources do not need to be the same. >>>>>>In my tests, xen is using the hpet for its timekeeping and >>>>>>that appears to be the default. >>>>>> >>>>>>When you boot the guests you should see >>>>>> time.c: Using PIT/TSC based timekeeping. >>>>>>on the rh4u5-64 guest, and something similar on the others. >>>>>> >>>>>> >>>>>> >>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>14.318MHz HPET.) >>>>>>> >>>>>>> >>>>>>This appears to be the xen state, which is fine. >>>>>>I was wrongly assuming that this was the guest state. >>>>>>You might want to look in your guest logs and see what they were >>>>>>picking >>>>>>for a clock source. >>>>>> >>>>>>Regards, >>>>>>Dave >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>Dan Magenheimer wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>Thanks, I hadn't realized that! No wonder we didn't >>>>>>> >>>>>>> >>see the same >> >> >>>>>>>improvement you saw! >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>Try specifying clock=pit on the linux boot line... >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>I'm confused... do you mean "clocksource=pit" on the Xen >>>>>>> >>>>>>> >>>>>>command line or >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the guest (or >>>>>>> >>>>>>> >>>>>>dom0?) command >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>line? Or both places? Since the tests take awhile, it >>>>>>> >>>>>>> >>>>>>would be nice >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>to get this right the first time. Do the Xen and guest >>>>>>> >>>>>>> >>>>>>clocksources need >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>to be the same? >>>>>>> >>>>>>>Thanks, >>>>>>>Dan >>>>>>> >>>>>>>-----Original Message----- >>>>>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM >>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser >>>>>>>*Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>>>>akira.ijuin@oracle.com; Dave Winchell >>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>> >>>>>>> >>that disables >> >> >>>>>>>pending missed ticks >>>>>>> >>>>>>> Hi Dan, >>>>>>> >>>>>>> Hpet timer does have a fairly large error, as I was >>>>>>> >>>>>>> >>trying this >> >> >>>>>>> one recently. >>>>>>> I don't remember what I got for error, but 1% sounds >>>>>>> >>>>>>> >>>>>>about right. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> The problem is that hpet is not built on top of vpt.c, >>>>>>> >>>>>>> >>>>>>the module >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Keir and I did >>>>>>> all the recent work in, for its periodic timer needs. Try >>>>>>> specifying clock=pit >>>>>>> on the linux boot line. If it still picks the hpet, which it >>>>>>> might, let me know >>>>>>> and I'll tell you how to get around this. >>>>>>> >>>>>>> Regards, >>>>>>> Dave >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>-------------------------------------------------------------- >>>>>>---------- >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>> *Sent:* Fri 1/25/2008 6:50 PM >>>>>>> *To:* Dave Winchell; Keir Fraser >>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>>>> akira.ijuin@oracle.com >>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>> >>>>>>> >>>>>>that disables >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> pending missed ticks >>>>>>> >>>>>>> Sorry for the very late followup on this but we finally >>>>>>> >>>>>>> >>>>>>were able >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> to get our testing set up again on stable 3.1 bits and have >>>>>>> seen some very bad results on 3.1.3-rc1, on the order of 1%. >>>>>>> >>>>>>> Test enviroment was a 4-socket dual core machine with 24GB of >>>>>>> memory running six two-vcpu 2GB domains, four hvm >>>>>>> >>>>>>> >>plus two pv. >> >> >>>>>>> All six guests were running LTP simultaneously. The four hvm >>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and >>>>>>> >>>>>>> >>RHEL4u5-64. >> >> >>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>>>>>> >>>>>>> >>>>>>32-bit guests. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> All four hvm guests experienced skew around -1%, >>>>>>> >>>>>>> >>even the 32-bit >> >> >>>>>>> guest. Less intensive testing didn't exhibit much >>>>>>> >>>>>>> >>skew at all. >> >> >>>>>>> A representative graph is attached. >>>>>>> >>>>>>> Dave, I wonder if some portion of your patches >>>>>>> >>>>>>> >>didn't end up in >> >> >>>>>>> the xen trees? >>>>>>> >>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>> 14.318MHz HPET.) >>>>>>> >>>>>>> Thanks, >>>>>>> Dan >>>>>>> >>>>>>> P.S. Many thanks to Deepak and Akira for running tests. >>>>>>> >>>>>>> > -----Original Message----- >>>>>>> > From: xen-devel-bounces@lists.xensource.com >>>>>>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>> > Dave Winchell >>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>>>>> > To: Keir Fraser >>>>>>> > Cc: dan.magenheimer@oracle.com; >>>>>>> >>>>>>> >>>>>>xen-devel@lists.xensource.com; Dave >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > Winchell >>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>> > disables pending >>>>>>> > missed ticks >>>>>>> > >>>>>>> > >>>>>>> > Hi Keir, >>>>>>> > >>>>>>> > The latest change, c/s 16690, looks fine. >>>>>>> > I agree that the code in c/s 16690 is equivalent to >>>>>>> > the code I submitted. Also, your version is more >>>>>>> > concise. >>>>>>> > >>>>>>> > The error tests confirm the equivalence. With >>>>>>> >>>>>>> >>>>>>overnight cpu loads, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > the checked in version was accurate to +.048% for sles >>>>>>> > and +.038% for red hat. My version was +.046% and >>>>>>> >>>>>>> >>+.032% in a >> >> >>>>>>> > 2 hour test. >>>>>>> > I don't think the difference is significant. >>>>>>> > >>>>>>> > i/o loads produced errors of +.01%. >>>>>>> > >>>>>>> > Thanks for all your efforts on this issue. >>>>>>> > >>>>>>> > Regards, >>>>>>> > Dave >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > Keir Fraser wrote: >>>>>>> > >>>>>>> > >Applied as c/s 16690, although the checked-in patch is >>>>>>> > smaller. I think the >>>>>>> > >only important fix is to pt_intr_post() and the >>>>>>> >>>>>>> >>only bit of >> >> >>>>>>> > the patch I >>>>>>> > >totally omitted was the change to >>>>>>> >>>>>>> >>pt_process_missed_ticks(). >> >> >>>>>>> > I don't think >>>>>>> > >that change can be important, but let's see what >>>>>>> >>>>>>> >>>>>>happens to the >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> error >>>>>>> > >percentage... >>>>>>> > > >>>>>>> > > -- Keir >>>>>>> > > >>>>>>> > >On 4/1/08 23:24, "Dave Winchell" >>>>>>> >>>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > >>Hi Dan and Keir, >>>>>>> > >> >>>>>>> > >>Attached is a patch that fixes some issues with the >>>>>>> >>>>>>> >>>>>>SYNC policy >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>(no_missed_ticks_pending). >>>>>>> > >>I have not tried to make the change the minimal one, but, >>>>>>> > rather, just >>>>>>> > >>ported into >>>>>>> > >>the new code what I know to work well. The error for >>>>>>> > >>no_missed_ticks_pending goes from >>>>>>> > >>over 3% to .03% with this change according to my testing. >>>>>>> > >> >>>>>>> > >>Regards, >>>>>>> > >>Dave >>>>>>> > >> >>>>>>> > >>Dan Magenheimer wrote: >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >>>Hi Dave -- >>>>>>> > >>> >>>>>>> > >>>Did you get your correction ported? If so, it would be >>>>>>> > nice to see this get >>>>>>> > >>>into 3.1.3. >>>>>>> > >>> >>>>>>> > >>>Note that I just did some very limited testing with >>>>>>> > timer_mode=2(=SYNC=no >>>>>>> > >>>missed ticks pending) >>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv >>>>>>> >>>>>>> >>guest) and the >> >> >>>>>>> > worst error I've >>>>>>> > >>>seen so far >>>>>>> > >>>is 0.012%. But I haven't tried any exotic >>>>>>> >>>>>>> >>loads, just LTP. >> >> >>>>>>> > >>> >>>>>>> > >>>Thanks, >>>>>>> > >>>Dan >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>>>-----Original Message----- >>>>>>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>>>> > >>>>To: dan.magenheimer@oracle.com >>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>>>>> > xen-devel@lists.xensource.com; Dong, >>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>> > >>>>disables pending >>>>>>> > >>>>missed ticks >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>>Dan, >>>>>>> > >>>> >>>>>>> > >>>>I did some testing with the constant tsc offset >>>>>>> >>>>>>> >>>>>>SYNC method >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>(now called >>>>>>> > >>>>no_missed_ticks_pending) >>>>>>> > >>>>and found the error to be very high, much larger >>>>>>> >>>>>>> >>>>>>than 1 %, as >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>I recall. >>>>>>> > >>>>I have not had a chance to submit a correction. I >>>>>>> >>>>>>> >>>>>>will try to >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>do it later >>>>>>> > >>>>this week or the first week in January. My version of >>>>>>> constant tsc >>>>>>> > >>>>offset SYNC method >>>>>>> > >>>>produces .02 % error, so I just need to port >>>>>>> >>>>>>> >>that into the >> >> >>>>>>> > >>>>current code. >>>>>>> > >>>> >>>>>>> > >>>>The error you got for both of those kernels is >>>>>>> >>>>>>> >>>>>>what I would >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> expect >>>>>>> > >>>>for the default mode, delay_for_missed_ticks. >>>>>>> > >>>> >>>>>>> > >>>>I'll let Keir answer on how to set the time mode. >>>>>>> > >>>> >>>>>>> > >>>>Regards, >>>>>>> > >>>>Dave >>>>>>> > >>>> >>>>>>> > >>>>Dan Magenheimer wrote: >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>>>Anyone make measurements on the final patch? >>>>>>> > >>>>> >>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and >>>>>>> >>>>>>> >>saw a loss of >> >> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>about 0.2% with no load. This was >>>>>>> >>>>>>> >>xen-unstable tip today >> >> >>>>>>> > >>>>with no options specified. 32-bit was about 0.01%. >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>>>I think I missed something... how do I run the various >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>accounting choices and which ones are known to be >>>>>>> >>>>>>> >>>>>>appropriate >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>for which kernels? >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>>>Thanks, >>>>>>> > >>>>>Dan >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>>>-----Original Message----- >>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>> > >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>Keir Fraser >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>> > >>>>>>To: Dave Winchell >>>>>>> > >>>>>>Cc: Shan, Haitao; >>>>>>> >>>>>>> >>xen-devel@lists.xensource.com; Dong, >> >> >>>>>>> > Eddie; Jiang, >>>>>>> > >>>>>>Yunhong >>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer >>>>>>> >>>>>>> >>mode that >> >> >>>>>>> > >>>>>>disables pending >>>>>>> > >>>>>>missed ticks >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. >>>>>>> > >>>>>> >>>>>>> > >>>>>>-- Keir >>>>>>> > >>>>>> >>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>> wrote: >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>Keir, >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>The accuracy data I've collected for i/o >>>>>>> >>>>>>> >>loads for the >> >> >>>>>>> > >>>>>>>various time protocols follows. In >>>>>>> >>>>>>> >>addition, the data >> >> >>>>>>> > >>>>>>>for cpu loads is shown. >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>>>>> >>>>>>> >>>>>>processor AMD >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> box. >>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>> > >>>>>>>(usex is available at >>>>>>> http://people.redhat.com/anderson/usex.) >>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>>>>> >>>>>>> >>>>>>of=/dev/null. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>> > >>>>>>>All three guests are 8vcpu. >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>>> > >>>>>>>except that the redhat-64 guest has 4 >>>>>>> >>>>>>> >>instances of dd. >> >> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 >>>>>>> >>>>>>> >>sec -.006%, >> >> >>>>>>> > +.005% cpu >>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 >>>>>>> >>>>>>> >>sec, -.001%, >> >> >>>>>>> > +.012% cpu >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, >>>>>>> -.004% cpu >>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>>>>> >>>>>>> >>>>>>-.005%, -.005% cpu >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>>>>> >>>>>>> >>>>>>-.008%, -.003% cpu >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>>>>> >>>>>>> >>>>>>-.040% cpu >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 >>>>>>> >>>>>>> >>sec, -.034%, >> >> >>>>>>> > -.031% cpu >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, >>>>>>> > -.09% i/o-8 >>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% >>>>>>> > -.14% i/o-8 >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, >>>>>>> > -.022% i/o-8 >>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>>>>> >>>>>>> >>>>>>-.017%, -.018% i/o-8 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, >>>>>>> > -.029% i/o-8 >>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 >>>>>>> >>>>>>> >>sec, -.023%, >> >> >>>>>>> > -.031% i/o-8 >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>>>>>> >>>>>>> >>>>>>-.04% i/o-32 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, >>>>>>> > -.005% i/o-32 >>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>>>>>> >>>>>>> >>>>>>-.11% i/o-32 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, >>>>>>> > .003% i/o-4/32 >>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, >>>>>>> > .01% i/o-4/32 >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>Overhead measurements: >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>Progress in terms of number of passes >>>>>>> >>>>>>> >>through a fixed >> >> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>system workload >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>> > >>>>>>>The workload was usex -b48. >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>Conclusions: >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>>>>> > requirement for ntp >>>>>>> > >>>>>>>tracking under the loads >>>>>>> > >>>>>>>above is the SYNC protocol. The worst case >>>>>>> >>>>>>> >>>>>>accuracies for >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>SYNC, MIXED, >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>and ASYNC >>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>We could reduce the cost of the SYNC method by only >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>scheduling the extra >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>wakeups if a certain number >>>>>>> > >>>>>>>of ticks are missed. >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>Regards, >>>>>>> > >>>>>>>Dave >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>Keir Fraser wrote: >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>> >>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>> wrote: >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>>>>>> >>>>>>> >>>>>>ASYNC method a >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>couple of days ago, >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have >>>>>>> > been something >>>>>>> > >>>>>>>>>wrong with the code I used a couple of >>>>>>> >>>>>>> >>days ago for >> >> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>ASYNC. It may have been >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt >>>>>>> >>>>>>> >>>>>>after context >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>switch in. >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>acceptable accuracy, >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>each running consistently around or under >>>>>>> >>>>>>> >>>>>>.01%. MIXED has >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>a fairly high >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>error of >>>>>>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>threshold for comfort. >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>>>>>> >>>>>>> >>>>>>plan to leave >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>SYNC running >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can >>>>>>> >>>>>>> >>leave MIXED >> >> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>running instead. >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>It may be too early to pick the protocol and >>>>>>> >>>>>>> >>>>>>I can run >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>more overnight tests >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>>next week. >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>> >>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side >>>>>>> >>>>>>> >>>>>>effects of the >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>SYNC+run_timer >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>>>>>> >>>>>>> >>>>>>cause higher >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>system-wide CPU >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>contention. I find it easier to think through the >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>implications of ASYNC. I'm >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less >>>>>>> >>>>>>> >>>>>>accurate than >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>ASYNC. Perhaps it >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>delivers more timer interrupts than the other >>>>>>> >>>>>>> >>>>>>approaches, >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>and each interrupt >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>event causes a small accumulated error? >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>>>>>> >>>>>>> >>>>>>favourites and >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>if the latter is >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>actually more accurate then I can simply revert the >>>>>>> > changeset that >>>>>>> > >>>>>>>>implemented MIXED. >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>>Perhaps rather than running more of the same >>>>>>> >>>>>>> >>>>>>workloads you >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>could try idle >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>>>>> >>>>>>> >>>>>>large disc reads >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>to /dev/null)? We >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>don't have any data on workloads that aren't >>>>>>> >>>>>>> >>>>>>CPU bound, so >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>that's really an >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>>>>obvious place to put any further effort imo. >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>>-- Keir >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>>> >>>>>>> > >>>>>>_______________________________________________ >>>>>>> > >>>>>>Xen-devel mailing list >>>>>>> > >>>>>>Xen-devel@lists.xensource.com >>>>>>> > >>>>>>http://lists.xensource.com/xen-devel >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>> >>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 >>>>>>> >>>>>>> >>2007 +0000 >> >> >>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 >>>>>>> >>>>>>> >>2008 -0500 >> >> >>>>>>> > >>@@ -58,7 +58,7 @@ static void >>>>>>> >>>>>>> >>pt_process_missed_ticks(stru >> >> >>>>>>> > >> >>>>>>> > >> missed_ticks = missed_ticks / (s_time_t) >>>>>>> >>>>>>> >>>>>>pt->period + 1; >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >> if ( mode_is(pt->vcpu->domain, >>>>>>> >>>>>>> >>>>>>no_missed_ticks_pending) ) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>>>> > >>+ pt->do_not_freeze = 1; >>>>>>> > >> else >>>>>>> > >> pt->pending_intr_nr += missed_ticks; >>>>>>> > >> pt->scheduled += missed_ticks * pt->period; >>>>>>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >>>>>>> > >> >>>>>>> > >> pt_lock(pt); >>>>>>> > >> >>>>>>> > >>- pt->pending_intr_nr++; >>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, >>>>>>> >>>>>>> >>>>>>no_missed_ticks_pending) ) { >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >>+ pt->pending_intr_nr = 1; >>>>>>> > >>+ pt->do_not_freeze = 0; >>>>>>> > >>+ } >>>>>>> > >>+ else >>>>>>> > >>+ pt->pending_intr_nr++; >>>>>>> > >> >>>>>>> > >> if ( !pt->one_shot ) >>>>>>> > >> { >>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct >>>>>>> >>>>>>> >>vcpu *v, struct >> >> >>>>>>> > >> return; >>>>>>> > >> } >>>>>>> > >> >>>>>>> > >>- pt->do_not_freeze = 0; >>>>>>> > >>- >>>>>>> > >> if ( pt->one_shot ) >>>>>>> > >> { >>>>>>> > >> pt->enabled = 0; >>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>>>>> >>>>>>> >>>>>>*v, struct >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> > >> pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>> > >> pt->pending_intr_nr = 0; /* 'collapse' all >>>>>>> > missed ticks */ >>>>>>> > >> } >>>>>>> > >>+ else if ( mode_is(v->domain, >>>>>>> >>>>>>> >>no_missed_ticks_pending) ) { >> >> >>>>>>> > >>+ pt->pending_intr_nr--; >>>>>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>> > >>+ } >>>>>>> > >> else >>>>>>> > >> { >>>>>>> > >> pt->last_plt_gtime += pt->period_cycles; >>>>>>> > >> >>>>>>> > >> >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > >>>>>>> > >>>>>>> > _______________________________________________ >>>>>>> > Xen-devel mailing list >>>>>>> > Xen-devel@lists.xensource.com >>>>>>> > http://lists.xensource.com/xen-devel >>>>>>> > >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 14 Feb 2008 09:21:14 -0700 Message-ID: <20080214092114625.00000001516@djm-pc> References: <47B4656F.40306@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <47B4656F.40306@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org Hi Dave -- Thanks for continuing to run tests! Hmmm... I thought I had noticed that even though Linux will acknowledge the existence of the pmtimer, it still prints: time.c: Using PIT/TSC based timekeeping. I will check again, but assuming the clocksource for our tests is indeed pit, the huge difference in the results (yours vs ours) is baffling. I wonder if the difference may be the underlying hardware. Maybe we will try to ensure we can duplicate the results on a different box. So your testing was with stock 3.2.0 xen bits (what cset?) without any of your [quote from below] "clock related tweaks that I haven't submitted, because I'm still characterizing them"? Could you also send detail on the rhel4u4-64 kernel you are testing with, just to ensure we are not comparing apples and oranges? (Perhaps there's some way we can even share the identical disk image and vm.cfg file?) And if our problem is indeed the pmtimer, I will need to submit another patch to Keir to add an hvm pmtimer platform variable. (Hmmm... I don't think he's even accepted the hpet variable patch yet. I'll have to check.) Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Thursday, February 14, 2008 9:00 AM > To: dan.magenheimer@oracle.com > Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak > Patel > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Hi Dan, > = > I ran the ltp tests with 3.2 and found the errors > for a 16 hour run to be: > = > rh4u564 -9.9 sec (-.017%) > rh4u464 -7.3 sec (-.013%) > = > There were no cliffs and the drift was linear. > = > I think the problem you had may be due to the use of the > pm timer. If you still have the boot log, it would tell you. > = > When I first tried a guest on 3.2 with "clocksource=3Dpit nohpet" > I noticed that it picked the pm timer. Adding "nopmtimer", the > guest will pick the pit. > = > The reason I didn't have the problem with our 3.1 base is that > I had disabled the hpet and the pmtimer by not advertising them > in the acpi tables. I did this so long ago, I forgot that I had to > disable pmtimer as well as hpet. > = > So, can you re-run your test with "clocksource=3Dpit nohpet nopmtimer"? > You should see this in the boot messages: > = > time.c: Using PIT/TSC based timekeeping. > = > Thanks, > Dave > = > = > Dave Winchell wrote: > = > > Hi Dan, > > > > Over the weekend the drift was +18 seconds for each guest (no ntp). > > The duration was 3900 minutes, so the error for each was +.0077%. > > Looking back through the data, it appears to drift linearly at > > this rate. I've attached a plot for rh4u5-64. > > > > This accuracy is better than what I've seen before (.03-.05%). > > This may be due to the different load (ltp vs usex) or to one of the > > changes I've made recently. I'll do some experimentation to see if > > there is > > a fix I should propose. > > > > This still doesn't address the radical drift you saw. > > The next step for me is to run 3.2 and see if I can reproduce it. > > > > Regards, > > Dave > > > > > > > > > > > > Dave Winchell wrote: > > > >> Hi Dan, > >> > >> Sorry it took me so long, but I finally ran an ltp test today. > >> Its on rh4u4-64. I'm using the defaults for ltp and using a script > >> called runltp. I had a usex load on rh4u5-64. No ntpd. > >> virtual processors / physical processors =3D 2. > >> > >> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes > >> for -.005% and .008%. > >> > >> I'm running a 3.1 based hypervisor with some clock related = > tweaks that > >> I haven't submitted, because I'm still characterizing them. > >> > >> I'm stopping the usex load on 4u5-64 now and replacing it with ltp > >> and will leave the two guests running ltp over the weekend. > >> > >> Regards, > >> Dave > >> > >> > >> Dave Winchell wrote: > >> > >>> Hi Dan, Deepak: > >>> > >>> Thanks for the data. Those drifts are severe - no wonder = > ntp couldn't > >>> keep then in synch. I'll try to reproduce that behaviour = > here, with > >>> my code base. > >>> If I can't reproduce it, I'll try 3.2. > >>> > >>> If you can isolate what ltp is doing during the cliffs, that would > >>> be very > >>> helpful. > >>> > >>> thanks, > >>> Dave > >>> > >>> > >>> > >>> > >>> Dan Magenheimer wrote: > >>> > >>>> OK, Deepak repeated the test without ntpd and using = > ntpdate -b before > >>>> the test. > >>>> > >>>> The attached graph shows his results: el5u1-64 (best=3D~0.07%), > >>>> el4u5-64 (middle=3D~0.2%), and el4u5-32 (worst=3D~0.3%). > >>>> > >>>> We will continue to look at LTP to try to isolate. > >>>> > >>>> Thanks, > >>>> Dan > >>>> > >>>> P.S. elXuY is essentially RHEL XuY with some patches. > >>>> > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>> Sent: Wednesday, January 30, 2008 2:45 PM > >>>>> To: Deepak Patel > >>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser; > >>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; = > Dave Winchell > >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables > >>>>> pending > >>>>> missed ticks > >>>>> > >>>>> > >>>>> Dan, Deeepak, > >>>>> > >>>>> It may be that the underlying clock error is too great for ntp > >>>>> to handle. It would be useful if you did not run ntpd > >>>>> and, instead did ntpdate -b at the start = > of the test > >>>>> for each guest. Then capture the data as you have been doing. > >>>>> If the drift is greater than .05%, then we need to address that. > >>>>> > >>>>> Another option is, when running ntpd, to enable loop = > statistics in > >>>>> /etc/ntp.conf > >>>>> by adding this to the file: > >>>>> > >>>>> statistics loopstats > >>>>> statsdir /var/lib/ntp/ > >>>>> > >>>>> Then you will see loop data in that directory. > >>>>> Correlating the data in the loopstats files with the > >>>>> peaks in skew would be interesting. You will see = > entries of the form > >>>>> > >>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 = > 239.735511 10 > >>>>> > >>>>> Where the second to last column is the Allan Deviation. = > When that > >>>>> gets over 1000, ntpd is working pretty hard. However, I have not > >>>>> seen ntpd > >>>>> completely loose it like you have. > >>>>> > >>>>> I'm on vacation until Monday, and won't be reading > >>>>> email. > >>>>> > >>>>> Thanks for all your work on this! > >>>>> > >>>>> -Dave > >>>>> > >>>>> Deepak Patel wrote: > >>>>> > >>>>> > >>>>> > >>>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.) > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> I do not know which graph was attached with this. But = > I saw this > >>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm = > guests when I > >>>>>> was running ltp tests continuously. > >>>>>> > >>>>>> > >>>>>> > >>>>>>> What was the behaviour of the other guests running? > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> All pvm guests are fine. But behavior of most of the = > hvm guests were > >>>>>> as described. > >>>>>> > >>>>>> > >>>>>> > >>>>>>> If they had spikes, were they at the same wall time? > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> No. They are not at the same wall time. > >>>>>> > >>>>>> > >>>>>> > >>>>>>> Were the other guests running ltp as well? > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp > >>>>>> continuously. > >>>>>> > >>>>>> > >>>>>> > >>>>>>> How are you measuring skew? > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> I was collecting output of "ntpdate -q every > >>>>> > >>>>> > >>>>> > >>>>> 300 seconds > >>>>> > >>>>> > >>>>>> (5 minutes) and have created graph based on that. > >>>>>> > >>>>>> > >>>>>> > >>>>>>> Are you running ntpd? > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> Yes. ntp was running on all the guests. > >>>>>> > >>>>>> I am investigating what causes this spikes and let everyone > >>>>> > >>>>> > >>>>> > >>>>> know what > >>>>> > >>>>> > >>>>>> are my findings. > >>>>>> > >>>>>> Thanks, > >>>>>> Deepak > >>>>>> > >>>>>> > >>>>>> > >>>>>>> Anything that you can discover that would be in sync with > >>>>>>> the spikes would be very helpful! > >>>>>>> > >>>>>>> The code that I test with is our product code, which is based > >>>>>>> on 3.1. So it is possible that something in 3.2 other = > than vpt.c > >>>>>>> is the cause. I can test with 3.2, if necessary. > >>>>>>> > >>>>>>> thanks, > >>>>>>> Dave > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Dan Magenheimer wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Hi Dave (Keir, see suggestion below) -- > >>>>>>>> > >>>>>>>> Thanks! > >>>>>>>> > >>>>>>>> Turning off vhpet certainly helps a lot (though see below). > >>>>>>>> > >>>>>>>> I wonder if timekeeping with vhpet is so bad that it = > should be > >>>>>>>> turned off by default (in 3.1, 3.2, and unstable) until it is > >>>>>>>> fixed? (I have a patch that defaults it off, can post it if > >>>>>>>> there is agreement on the above point.) The whole = > point of an > >>>>>>>> HPET is to provide more precise timekeeping and if vhpet is > >>>>>>>> worse than vpit, it can only confuse users. Comments? > >>>>>>>> > >>>>>>>> > >>>>>>>> In your testing, are you just measuring % skew over a long > >>>>>>>> period of time? > >>>>>>>> We are graphing the skew continuously and > >>>>>>>> seeing periodic behavior that is unsettling, even with pit. > >>>>>>>> See attached. Though your algorithm recovers, the "cliffs" > >>>>>>>> could still cause real user problems. I wonder if there is > >>>>>>>> anything that can be done to make the "recovery" more > >>>>>>>> responsive? > >>>>>>>> > >>>>>>>> We are looking into what part(s) of LTP is causing = > the cliffs. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Dan > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM > >>>>>>>>> To: dan.magenheimer@oracle.com > >>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; > >>>>>>>>> deepak.patel@oracle.com; > >>>>>>>>> akira.ijuin@oracle.com; Dave Winchell > >>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode = > that disables > >>>>>>>>> pending > >>>>>>>>> missed ticks > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Dan, > >>>>>>>>> > >>>>>>>>> I guess I'm a bit out of date calling for clock=3D usage. > >>>>>>>>> Looking at linux 2.6.20.4 sources, I think you = > should specify > >>>>>>>>> "clocksource=3Dpit nohpet" on the linux guest bootline. > >>>>>>>>> > >>>>>>>>> You can leave the xen and dom0 bootlines as they are. > >>>>>>>>> The xen and guest clocksources do not need to be the same. > >>>>>>>>> In my tests, xen is using the hpet for its timekeeping and > >>>>>>>>> that appears to be the default. > >>>>>>>>> > >>>>>>>>> When you boot the guests you should see > >>>>>>>>> time.c: Using PIT/TSC based timekeeping. > >>>>>>>>> on the rh4u5-64 guest, and something similar on the others. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > >>>>>>>>>> 14.318MHz HPET.) > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> This appears to be the xen state, which is fine. > >>>>>>>>> I was wrongly assuming that this was the guest state. > >>>>>>>>> You might want to look in your guest logs and see = > what they were > >>>>>>>>> picking > >>>>>>>>> for a clock source. > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Dave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Dan Magenheimer wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Thanks, I hadn't realized that! No wonder we didn't > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> see the same > >>>>> > >>>>> > >>>>>>>>>> improvement you saw! > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> Try specifying clock=3Dpit on the linux boot line... > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> I'm confused... do you mean "clocksource=3Dpit" on the Xen > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> command line or > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> "nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the guest (o= r > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> dom0?) command > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> line? Or both places? Since the tests take awhile, it > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> would be nice > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> to get this right the first time. Do the Xen and guest > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> clocksources need > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> to be the same? > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Dan > >>>>>>>>>> > >>>>>>>>>> -----Original Message----- > >>>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM > >>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser > >>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; = > deepak.patel@oracle.com; > >>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell > >>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> that disables > >>>>> > >>>>> > >>>>>>>>>> pending missed ticks > >>>>>>>>>> > >>>>>>>>>> Hi Dan, > >>>>>>>>>> > >>>>>>>>>> Hpet timer does have a fairly large error, as I > >>>>>>>>>> was > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> trying this > >>>>> > >>>>> > >>>>>>>>>> one recently. > >>>>>>>>>> I don't remember what I got for error, but 1% sounds > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> about right. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> The problem is that hpet is not built on top of vpt.c, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> the module > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Keir and I did > >>>>>>>>>> all the recent work in, for its periodic timer needs. Try > >>>>>>>>>> specifying clock=3Dpit > >>>>>>>>>> on the linux boot line. If it still picks the = > hpet, which it > >>>>>>>>>> might, let me know > >>>>>>>>>> and I'll tell you how to get around this. > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Dave > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> = > -------------------------------------------------------------- > >>>>>>>>> ---------- > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> *From:* Dan Magenheimer = > [mailto:dan.magenheimer@oracle.com] > >>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM > >>>>>>>>>> *To:* Dave Winchell; Keir Fraser > >>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; = > deepak.patel@oracle.com; > >>>>>>>>>> akira.ijuin@oracle.com > >>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> that disables > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> pending missed ticks > >>>>>>>>>> > >>>>>>>>>> Sorry for the very late followup on this but we finally > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> were able > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> to get our testing set up again on stable 3.1 = > bits and have > >>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the = > order of 1%. > >>>>>>>>>> > >>>>>>>>>> Test enviroment was a 4-socket dual core machine = > with 24GB of > >>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> plus two pv. > >>>>> > >>>>> > >>>>>>>>>> All six guests were running LTP simultaneously. = > The four hvm > >>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> RHEL4u5-64. > >>>>> > >>>>> > >>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> 32-bit guests. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> All four hvm guests experienced skew around -1%, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> even the 32-bit > >>>>> > >>>>> > >>>>>>>>>> guest. Less intensive testing didn't exhibit much > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> skew at all. > >>>>> > >>>>> > >>>>>>>>>> A representative graph is attached. > >>>>>>>>>> > >>>>>>>>>> Dave, I wonder if some portion of your patches > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> didn't end up in > >>>>> > >>>>> > >>>>>>>>>> the xen trees? > >>>>>>>>>> > >>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, = > Platform timer > >>>>>>>>>> 14.318MHz HPET.) > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Dan > >>>>>>>>>> > >>>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests. > >>>>>>>>>> > >>>>>>>>>> > -----Original Message----- > >>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com > >>>>>>>>>> > = > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>>>>>> > Dave Winchell > >>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM > >>>>>>>>>> > To: Keir Fraser > >>>>>>>>>> > Cc: dan.magenheimer@oracle.com; > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> xen-devel@lists.xensource.com; Dave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > Winchell > >>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>>>>>>> > disables pending > >>>>>>>>>> > missed ticks > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > Hi Keir, > >>>>>>>>>> > > >>>>>>>>>> > The latest change, c/s 16690, looks fine. > >>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to > >>>>>>>>>> > the code I submitted. Also, your version is more > >>>>>>>>>> > concise. > >>>>>>>>>> > > >>>>>>>>>> > The error tests confirm the equivalence. With > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> overnight cpu loads, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > the checked in version was accurate to +.048% for sles > >>>>>>>>>> > and +.038% for red hat. My version was +.046% > >>>>>>>>>> and > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> +.032% in a > >>>>> > >>>>> > >>>>>>>>>> > 2 hour test. > >>>>>>>>>> > I don't think the difference is significant. > >>>>>>>>>> > > >>>>>>>>>> > i/o loads produced errors of +.01%. > >>>>>>>>>> > > >>>>>>>>>> > Thanks for all your efforts on this issue. > >>>>>>>>>> > > >>>>>>>>>> > Regards, > >>>>>>>>>> > Dave > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > Keir Fraser wrote: > >>>>>>>>>> > > >>>>>>>>>> > >Applied as c/s 16690, although the checked-in patch is > >>>>>>>>>> > smaller. I think the > >>>>>>>>>> > >only important fix is to pt_intr_post() and the > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> only bit of > >>>>> > >>>>> > >>>>>>>>>> > the patch I > >>>>>>>>>> > >totally omitted was the change to > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> pt_process_missed_ticks(). > >>>>> > >>>>> > >>>>>>>>>> > I don't think > >>>>>>>>>> > >that change can be important, but let's see what > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> happens to the > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> error > >>>>>>>>>> > >percentage... > >>>>>>>>>> > > > >>>>>>>>>> > > -- Keir > >>>>>>>>>> > > > >>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > >>Hi Dan and Keir, > >>>>>>>>>> > >> > >>>>>>>>>> > >>Attached is a patch that fixes some issues with the > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> SYNC policy > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>(no_missed_ticks_pending). > >>>>>>>>>> > >>I have not tried to make the change the = > minimal one, but, > >>>>>>>>>> > rather, just > >>>>>>>>>> > >>ported into > >>>>>>>>>> > >>the new code what I know to work well. The error for > >>>>>>>>>> > >>no_missed_ticks_pending goes from > >>>>>>>>>> > >>over 3% to .03% with this change according = > to my testing. > >>>>>>>>>> > >> > >>>>>>>>>> > >>Regards, > >>>>>>>>>> > >>Dave > >>>>>>>>>> > >> > >>>>>>>>>> > >>Dan Magenheimer wrote: > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> > >>>Hi Dave -- > >>>>>>>>>> > >>> > >>>>>>>>>> > >>>Did you get your correction ported? If so, = > it would be > >>>>>>>>>> > nice to see this get > >>>>>>>>>> > >>>into 3.1.3. > >>>>>>>>>> > >>> > >>>>>>>>>> > >>>Note that I just did some very limited testing with > >>>>>>>>>> > timer_mode=3D2(=3DSYNC=3Dno > >>>>>>>>>> > >>>missed ticks pending) > >>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> guest) and the > >>>>> > >>>>> > >>>>>>>>>> > worst error I've > >>>>>>>>>> > >>>seen so far > >>>>>>>>>> > >>>is 0.012%. But I haven't tried any exotic > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> loads, just LTP. > >>>>> > >>>>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>>Thanks, > >>>>>>>>>> > >>>Dan > >>>>>>>>>> > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>>>-----Original Message----- > >>>>>>>>>> > >>>>From: Dave Winchell = > [mailto:dwinchell@virtualiron.com] > >>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com > >>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; > >>>>>>>>>> > xen-devel@lists.xensource.com; Dong, > >>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a = > timer mode that > >>>>>>>>>> > >>>>disables pending > >>>>>>>>>> > >>>>missed ticks > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>Dan, > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>I did some testing with the constant tsc offset > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> SYNC method > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>(now called > >>>>>>>>>> > >>>>no_missed_ticks_pending) > >>>>>>>>>> > >>>>and found the error to be very high, much larger > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> than 1 %, as > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>I recall. > >>>>>>>>>> > >>>>I have not had a chance to submit a correction. I > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> will try to > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>do it later > >>>>>>>>>> > >>>>this week or the first week in January. My = > version of > >>>>>>>>>> constant tsc > >>>>>>>>>> > >>>>offset SYNC method > >>>>>>>>>> > >>>>produces .02 % error, so I just need to port > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> that into the > >>>>> > >>>>> > >>>>>>>>>> > >>>>current code. > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>The error you got for both of those kernels is > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> what I would > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> expect > >>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode. > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>Regards, > >>>>>>>>>> > >>>>Dave > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>Dan Magenheimer wrote: > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>>Anyone make measurements on the final patch? > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> saw a loss of > >>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>about 0.2% with no load. This was > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> xen-unstable tip today > >>>>> > >>>>> > >>>>>>>>>> > >>>>with no options specified. 32-bit was about 0.01%. > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>>I think I missed something... how do I = > run the various > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>accounting choices and which ones are known to be > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> appropriate > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>for which kernels? > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>>Thanks, > >>>>>>>>>> > >>>>>Dan > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>>>-----Original Message----- > >>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>>>>> > > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> = > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>Keir Fraser > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>>>>>>> > >>>>>>To: Dave Winchell > >>>>>>>>>> > >>>>>>Cc: Shan, Haitao; > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> xen-devel@lists.xensource.com; Dong, > >>>>> > >>>>> > >>>>>>>>>> > Eddie; Jiang, > >>>>>>>>>> > >>>>>>Yunhong > >>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> mode that > >>>>> > >>>>> > >>>>>>>>>> > >>>>>>disables pending > >>>>>>>>>> > >>>>>>missed ticks > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>Please take a look at xen-unstable = > changeset 16545. > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>-- Keir > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>> wrote: > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>Keir, > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> loads for the > >>>>> > >>>>> > >>>>>>>>>> > >>>>>>>various time protocols follows. In > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> addition, the data > >>>>> > >>>>> > >>>>>>>>>> > >>>>>>>for cpu loads is shown. > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> processor AMD > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> box. > >>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 = > vcpu each. > >>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>>>>>>> > >>>>>>>(usex is available at > >>>>>>>>>> http://people.redhat.com/anderson/usex.) > >>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> of=3D/dev/null. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. > >>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>>>>>>> > >>>>>>>All three guests are 8vcpu. > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same = > as i/o-32 > >>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> instances of dd. > >>>>> > >>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, > >>>>>>>>>> +4.42 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> sec -.006%, > >>>>> > >>>>> > >>>>>>>>>> > +.005% cpu > >>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> sec, -.001%, > >>>>> > >>>>> > >>>>>>>>>> > +.012% cpu > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 = > sec, -.009%, > >>>>>>>>>> -.004% cpu > >>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -.005%, -.005% cpu > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -.008%, -.003% cpu > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -.040% cpu > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> sec, -.034%, > >>>>> > >>>>> > >>>>>>>>>> > -.031% cpu > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 = > sec,-55.7 sec, -.01%, > >>>>>>>>>> > -.09% i/o-8 > >>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 = > sec,-14.0 sec, -.015% > >>>>>>>>>> > -.14% i/o-8 > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 = > sec, -.017%, > >>>>>>>>>> > -.022% i/o-8 > >>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -.017%, -.018% i/o-8 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 = > sec, -.020%, > >>>>>>>>>> > -.029% i/o-8 > >>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> sec, -.023%, > >>>>> > >>>>> > >>>>>>>>>> > -.031% i/o-8 > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -.04% i/o-32 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 = > sec, -.011%, > >>>>>>>>>> > -.005% i/o-32 > >>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -.11% i/o-32 > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, = > 13. sec -.07%, > >>>>>>>>>> > .003% i/o-4/32 > >>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 = > sec, -.017%, > >>>>>>>>>> > .01% i/o-4/32 > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>Overhead measurements: > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>Progress in terms of number of passes > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> through a fixed > >>>>> > >>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>system workload > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>>>>>>>> > >>>>>>>The workload was usex -b48. > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>Conclusions: > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy > >>>>>>>>>> > requirement for ntp > >>>>>>>>>> > >>>>>>>tracking under the loads > >>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> accuracies for > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>SYNC, MIXED, > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>and ASYNC > >>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC = > method by only > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>scheduling the extra > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>wakeups if a certain number > >>>>>>>>>> > >>>>>>>of ticks are missed. > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>Regards, > >>>>>>>>>> > >>>>>>>Dave > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>Keir Fraser wrote: > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>> > >>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>> wrote: > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ASYNC method a > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>couple of days ago, > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think = > there may have > >>>>>>>>>> > been something > >>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> days ago for > >>>>> > >>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>ASYNC. It may have been > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> after context > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>switch in. > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC = > or ASYNC give > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>acceptable accuracy, > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>each running consistently around or under > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> .01%. MIXED has > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>a fairly high > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>error of > >>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close = > to .05% ntp > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>threshold for comfort. > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> plan to leave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>SYNC running > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> leave MIXED > >>>>> > >>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>running instead. > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> I can run > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>more overnight tests > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>>next week. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> effects of the > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>SYNC+run_timer > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> cause higher > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>system-wide CPU > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>contention. I find it easier to think = > through the > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>implications of ASYNC. I'm > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> accurate than > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>ASYNC. Perhaps it > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> approaches, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>and each interrupt > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>event causes a small accumulated error? > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> favourites and > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>if the latter is > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>actually more accurate then I can = > simply revert the > >>>>>>>>>> > changeset that > >>>>>>>>>> > >>>>>>>>implemented MIXED. > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> workloads you > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>could try idle > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> large disc reads > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>to /dev/null)? We > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> CPU bound, so > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>that's really an > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>>-- Keir > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>>>> > >>>>>>_______________________________________________ > >>>>>>>>>> > >>>>>>Xen-devel mailing list > >>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com > >>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>> > >>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> 2007 +0000 > >>>>> > >>>>> > >>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> 2008 -0500 > >>>>> > >>>>> > >>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> pt_process_missed_ticks(stru > >>>>> > >>>>> > >>>>>>>>>> > >> > >>>>>>>>>> > >> missed_ticks =3D missed_ticks / (s_time_t) > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> pt->period + 1; > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >> if ( mode_is(pt->vcpu->domain, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> no_missed_ticks_pending) ) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>>>>>>>>> > >>+ pt->do_not_freeze =3D 1; > >>>>>>>>>> > >> else > >>>>>>>>>> > >> pt->pending_intr_nr +=3D missed_ticks; > >>>>>>>>>> > >> pt->scheduled +=3D missed_ticks * pt->period; > >>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void = > pt_timer_fn(void *data) > >>>>>>>>>> > >> > >>>>>>>>>> > >> pt_lock(pt); > >>>>>>>>>> > >> > >>>>>>>>>> > >>- pt->pending_intr_nr++; > >>>>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> no_missed_ticks_pending) ) { > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>+ pt->pending_intr_nr =3D 1; > >>>>>>>>>> > >>+ pt->do_not_freeze =3D 0; > >>>>>>>>>> > >>+ } > >>>>>>>>>> > >>+ else > >>>>>>>>>> > >>+ pt->pending_intr_nr++; > >>>>>>>>>> > >> > >>>>>>>>>> > >> if ( !pt->one_shot ) > >>>>>>>>>> > >> { > >>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> vcpu *v, struct > >>>>> > >>>>> > >>>>>>>>>> > >> return; > >>>>>>>>>> > >> } > >>>>>>>>>> > >> > >>>>>>>>>> > >>- pt->do_not_freeze =3D 0; > >>>>>>>>>> > >>- > >>>>>>>>>> > >> if ( pt->one_shot ) > >>>>>>>>>> > >> { > >>>>>>>>>> > >> pt->enabled =3D 0; > >>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> *v, struct > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >> pt->last_plt_gtime =3D = > hvm_get_guest_time(v); > >>>>>>>>>> > >> pt->pending_intr_nr =3D 0; /* = > 'collapse' all > >>>>>>>>>> > missed ticks */ > >>>>>>>>>> > >> } > >>>>>>>>>> > >>+ else if ( mode_is(v->domain, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>> no_missed_ticks_pending) ) { > >>>>> > >>>>> > >>>>>>>>>> > >>+ pt->pending_intr_nr--; > >>>>>>>>>> > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>>>>>>>>> > >>+ } > >>>>>>>>>> > >> else > >>>>>>>>>> > >> { > >>>>>>>>>> > >> pt->last_plt_gtime +=3D pt->period_cycles; > >>>>>>>>>> > >> > >>>>>>>>>> > >> > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > _______________________________________________ > >>>>>>>>>> > Xen-devel mailing list > >>>>>>>>>> > Xen-devel@lists.xensource.com > >>>>>>>>>> > http://lists.xensource.com/xen-devel > >>>>>>>>>> > > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>> > >> > > > > > > = > -------------------------------------------------------------- > ---------- > > > = > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 15 Feb 2008 09:46:50 -0700 Message-ID: <20080215094650484.00000001516@djm-pc> References: <47B480AD.4050308@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <47B480AD.4050308@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org Hi Dave -- No new results yet but one other question: The problems we've seen with our testing have been with a heavily oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests running LTP simultaneously. Was your LTP testing oversubscribed or just a single guest? Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Thursday, February 14, 2008 10:56 AM > To: dan.magenheimer@oracle.com > Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave > Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Dan, > = > Here are some boot snipets for rh4u564 on xen 3.2. > = > = > #1: > = > Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro > root=3DLABEL=3D/ console=3DttyS0 clocksource=3Dpit nohpet) > Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp > (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 > (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 > ... > Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=3DLABEL=3D/ > console=3DttyS0 clocksource=3Dpit nohpet > Feb 14 10:44:59 vs076 kernel: Initializing CPU#0 > Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11, > 65536 bytes) > Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer. > Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor. > ... > Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8 > CPUs: passed. > Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs > Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use = > of PM timer > Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping. > = > = > = > #2: > = > Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro > root=3DLABEL=3D/ console=3DttyS0 clocksource=3Dpit nohpet nopmtimer) > Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp > (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 > (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 > ... > Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=3DLABEL=3D/ > console=3DttyS0 clocksource=3Dpit nohpet nopmtimer > Feb 14 10:47:58 vs076 kernel: Initializing CPU#0 > Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11, > 65536 bytes) > Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer. > Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor. > ... > Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8 > CPUs: passed. > Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs > Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping. > = > = > As you can see, I only get the pit if I specify nopmtimer. > = > Dan Magenheimer wrote: > = > >Hi Dave -- > > > >Thanks for continuing to run tests! > > > >Hmmm... I thought I had noticed that even though Linux will = > acknowledge > >the existence of the pmtimer, it still prints: > > > >time.c: Using PIT/TSC based timekeeping. > > > >I will check again, but assuming the clocksource for our tests is > >indeed pit, the huge difference in the results (yours vs ours) is > >baffling. I wonder if the difference may be the underlying hardware. > >Maybe we will try to ensure we can duplicate the results on = > a different > >box. > > > > > >So your testing was with stock 3.2.0 xen bits (what cset?) without > >any of your [quote from below] "clock related tweaks that I haven't > >submitted, because I'm still characterizing them"? > > > > > None of the tweaks I mentioned are in this test. > It was stock with some patches. However, none of the patches are time > related to > my knowledge and I checked vpt.c to make sure that it is the same as > what's in unstable. > The only difference is in pt_intr_post, where I set the timer mode. > I don't have timer mode tied into our config process yet, which > is different than official xen method. > = > = > (In pt_intr_post) > else > { > + if(v->arch.paging.mode->guest_levels =3D=3D 4) > + v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =3D > HVMPTM_no_missed_ticks_pending; > + else > + v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =3D > HVMPTM_delay_for_missed_ticks; > if ( mode_is(v->domain, one_missed_tick_pending) || > mode_is(v->domain, no_missed_ticks_pending) ) > { > = > >Could you also send detail on the rhel4u4-64 kernel you > >are testing with, just to ensure we are not comparing apples > >and oranges? (Perhaps there's some way we can even share the > >identical disk image and vm.cfg file?) > > > >And if our problem is indeed the pmtimer, I will need to submit > >another patch to Keir to add an hvm pmtimer platform variable. > >(Hmmm... I don't think he's even accepted the hpet variable patch > >yet. I'll have to check.) > > > >Thanks, > >Dan > > > > > > > > > >>-----Original Message----- > >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>Sent: Thursday, February 14, 2008 9:00 AM > >>To: dan.magenheimer@oracle.com > >>Cc: Dave Winchell; Keir Fraser; = > xen-devel@lists.xensource.com; Deepak > >>Patel > >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>disables pending > >>missed ticks > >> > >> > >>Hi Dan, > >> > >>I ran the ltp tests with 3.2 and found the errors > >>for a 16 hour run to be: > >> > >>rh4u564 -9.9 sec (-.017%) > >>rh4u464 -7.3 sec (-.013%) > >> > >>There were no cliffs and the drift was linear. > >> > >>I think the problem you had may be due to the use of the > >>pm timer. If you still have the boot log, it would tell you. > >> > >>When I first tried a guest on 3.2 with "clocksource=3Dpit nohpet" > >>I noticed that it picked the pm timer. Adding "nopmtimer", the > >>guest will pick the pit. > >> > >>The reason I didn't have the problem with our 3.1 base is that > >>I had disabled the hpet and the pmtimer by not advertising them > >>in the acpi tables. I did this so long ago, I forgot that I had to > >>disable pmtimer as well as hpet. > >> > >>So, can you re-run your test with "clocksource=3Dpit nohpet = > nopmtimer"? > >>You should see this in the boot messages: > >> > >>time.c: Using PIT/TSC based timekeeping. > >> > >>Thanks, > >>Dave > >> > >> > >>Dave Winchell wrote: > >> > >> > >> > >>>Hi Dan, > >>> > >>>Over the weekend the drift was +18 seconds for each guest (no ntp). > >>>The duration was 3900 minutes, so the error for each was +.0077%. > >>>Looking back through the data, it appears to drift linearly at > >>>this rate. I've attached a plot for rh4u5-64. > >>> > >>>This accuracy is better than what I've seen before (.03-.05%). > >>>This may be due to the different load (ltp vs usex) or to = > one of the > >>>changes I've made recently. I'll do some experimentation to see if > >>>there is > >>>a fix I should propose. > >>> > >>>This still doesn't address the radical drift you saw. > >>>The next step for me is to run 3.2 and see if I can reproduce it. > >>> > >>>Regards, > >>>Dave > >>> > >>> > >>> > >>> > >>> > >>>Dave Winchell wrote: > >>> > >>> > >>> > >>>>Hi Dan, > >>>> > >>>>Sorry it took me so long, but I finally ran an ltp test today. > >>>>Its on rh4u4-64. I'm using the defaults for ltp and using a script > >>>>called runltp. I had a usex load on rh4u5-64. No ntpd. > >>>>virtual processors / physical processors =3D 2. > >>>> > >>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes > >>>>for -.005% and .008%. > >>>> > >>>>I'm running a 3.1 based hypervisor with some clock related > >>>> > >>>> > >>tweaks that > >> > >> > >>>>I haven't submitted, because I'm still characterizing them. > >>>> > >>>>I'm stopping the usex load on 4u5-64 now and replacing it with ltp > >>>>and will leave the two guests running ltp over the weekend. > >>>> > >>>>Regards, > >>>>Dave > >>>> > >>>> > >>>>Dave Winchell wrote: > >>>> > >>>> > >>>> > >>>>>Hi Dan, Deepak: > >>>>> > >>>>>Thanks for the data. Those drifts are severe - no wonder > >>>>> > >>>>> > >>ntp couldn't > >> > >> > >>>>>keep then in synch. I'll try to reproduce that behaviour > >>>>> > >>>>> > >>here, with > >> > >> > >>>>>my code base. > >>>>>If I can't reproduce it, I'll try 3.2. > >>>>> > >>>>>If you can isolate what ltp is doing during the cliffs, = > that would > >>>>>be very > >>>>>helpful. > >>>>> > >>>>>thanks, > >>>>>Dave > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>Dan Magenheimer wrote: > >>>>> > >>>>> > >>>>> > >>>>>>OK, Deepak repeated the test without ntpd and using > >>>>>> > >>>>>> > >>ntpdate -b before > >> > >> > >>>>>>the test. > >>>>>> > >>>>>>The attached graph shows his results: el5u1-64 (best=3D~0.07%), > >>>>>>el4u5-64 (middle=3D~0.2%), and el4u5-32 (worst=3D~0.3%). > >>>>>> > >>>>>>We will continue to look at LTP to try to isolate. > >>>>>> > >>>>>>Thanks, > >>>>>>Dan > >>>>>> > >>>>>>P.S. elXuY is essentially RHEL XuY with some patches. > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>-----Original Message----- > >>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM > >>>>>>>To: Deepak Patel > >>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser; > >>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; > >>>>>>> > >>>>>>> > >>Dave Winchell > >> > >> > >>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables > >>>>>>>pending > >>>>>>>missed ticks > >>>>>>> > >>>>>>> > >>>>>>>Dan, Deeepak, > >>>>>>> > >>>>>>>It may be that the underlying clock error is too great for ntp > >>>>>>>to handle. It would be useful if you did not run ntpd > >>>>>>>and, instead did ntpdate -b at the start > >>>>>>> > >>>>>>> > >>of the test > >> > >> > >>>>>>>for each guest. Then capture the data as you have been doing. > >>>>>>>If the drift is greater than .05%, then we need to = > address that. > >>>>>>> > >>>>>>>Another option is, when running ntpd, to enable loop > >>>>>>> > >>>>>>> > >>statistics in > >> > >> > >>>>>>>/etc/ntp.conf > >>>>>>>by adding this to the file: > >>>>>>> > >>>>>>>statistics loopstats > >>>>>>>statsdir /var/lib/ntp/ > >>>>>>> > >>>>>>>Then you will see loop data in that directory. > >>>>>>>Correlating the data in the loopstats files with the > >>>>>>>peaks in skew would be interesting. You will see > >>>>>>> > >>>>>>> > >>entries of the form > >> > >> > >>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776 > >>>>>>> > >>>>>>> > >>239.735511 10 > >> > >> > >>>>>>>Where the second to last column is the Allan Deviation. > >>>>>>> > >>>>>>> > >>When that > >> > >> > >>>>>>>gets over 1000, ntpd is working pretty hard. However, = > I have not > >>>>>>>seen ntpd > >>>>>>>completely loose it like you have. > >>>>>>> > >>>>>>>I'm on vacation until Monday, and won't be reading > >>>>>>>email. > >>>>>>> > >>>>>>>Thanks for all your work on this! > >>>>>>> > >>>>>>>-Dave > >>>>>>> > >>>>>>>Deepak Patel wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested this one.) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>I do not know which graph was attached with this. But > >>>>>>>> > >>>>>>>> > >>I saw this > >> > >> > >>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm > >>>>>>>> > >>>>>>>> > >>guests when I > >> > >> > >>>>>>>>was running ltp tests continuously. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>What was the behaviour of the other guests running? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>All pvm guests are fine. But behavior of most of the > >>>>>>>> > >>>>>>>> > >>hvm guests were > >> > >> > >>>>>>>>as described. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>If they had spikes, were they at the same wall time? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>No. They are not at the same wall time. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>Were the other guests running ltp as well? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp > >>>>>>>>continuously. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>How are you measuring skew? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>I was collecting output of "ntpdate -q every > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>>300 seconds > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>(5 minutes) and have created graph based on that. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>Are you running ntpd? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>Yes. ntp was running on all the guests. > >>>>>>>> > >>>>>>>>I am investigating what causes this spikes and let everyone > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>>know what > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>are my findings. > >>>>>>>> > >>>>>>>>Thanks, > >>>>>>>>Deepak > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>Anything that you can discover that would be in sync with > >>>>>>>>>the spikes would be very helpful! > >>>>>>>>> > >>>>>>>>>The code that I test with is our product code, which is based > >>>>>>>>>on 3.1. So it is possible that something in 3.2 other > >>>>>>>>> > >>>>>>>>> > >>than vpt.c > >> > >> > >>>>>>>>>is the cause. I can test with 3.2, if necessary. > >>>>>>>>> > >>>>>>>>>thanks, > >>>>>>>>>Dave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>Dan Magenheimer wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>>Hi Dave (Keir, see suggestion below) -- > >>>>>>>>>> > >>>>>>>>>>Thanks! > >>>>>>>>>> > >>>>>>>>>>Turning off vhpet certainly helps a lot (though see below). > >>>>>>>>>> > >>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it > >>>>>>>>>> > >>>>>>>>>> > >>should be > >> > >> > >>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable) = > until it is > >>>>>>>>>>fixed? (I have a patch that defaults it off, can post it if > >>>>>>>>>>there is agreement on the above point.) The whole > >>>>>>>>>> > >>>>>>>>>> > >>point of an > >> > >> > >>>>>>>>>>HPET is to provide more precise timekeeping and if vhpet is > >>>>>>>>>>worse than vpit, it can only confuse users. Comments? > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>In your testing, are you just measuring % skew over a long > >>>>>>>>>>period of time? > >>>>>>>>>>We are graphing the skew continuously and > >>>>>>>>>>seeing periodic behavior that is unsettling, even with pit. > >>>>>>>>>>See attached. Though your algorithm recovers, the "cliffs" > >>>>>>>>>>could still cause real user problems. I wonder if there is > >>>>>>>>>>anything that can be done to make the "recovery" more > >>>>>>>>>>responsive? > >>>>>>>>>> > >>>>>>>>>>We are looking into what part(s) of LTP is causing > >>>>>>>>>> > >>>>>>>>>> > >>the cliffs. > >> > >> > >>>>>>>>>>Thanks, > >>>>>>>>>>Dan > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>-----Original Message----- > >>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM > >>>>>>>>>>>To: dan.magenheimer@oracle.com > >>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; > >>>>>>>>>>>deepak.patel@oracle.com; > >>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell > >>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>> > >>>>>>>>>>> > >>that disables > >> > >> > >>>>>>>>>>>pending > >>>>>>>>>>>missed ticks > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>Dan, > >>>>>>>>>>> > >>>>>>>>>>>I guess I'm a bit out of date calling for clock=3D usage. > >>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you > >>>>>>>>>>> > >>>>>>>>>>> > >>should specify > >> > >> > >>>>>>>>>>>"clocksource=3Dpit nohpet" on the linux guest bootline. > >>>>>>>>>>> > >>>>>>>>>>>You can leave the xen and dom0 bootlines as they are. > >>>>>>>>>>>The xen and guest clocksources do not need to be the same. > >>>>>>>>>>>In my tests, xen is using the hpet for its timekeeping and > >>>>>>>>>>>that appears to be the default. > >>>>>>>>>>> > >>>>>>>>>>>When you boot the guests you should see > >>>>>>>>>>> time.c: Using PIT/TSC based timekeeping. > >>>>>>>>>>>on the rh4u5-64 guest, and something similar on the others. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer > >>>>>>>>>>>>14.318MHz HPET.) > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>This appears to be the xen state, which is fine. > >>>>>>>>>>>I was wrongly assuming that this was the guest state. > >>>>>>>>>>>You might want to look in your guest logs and see > >>>>>>>>>>> > >>>>>>>>>>> > >>what they were > >> > >> > >>>>>>>>>>>picking > >>>>>>>>>>>for a clock source. > >>>>>>>>>>> > >>>>>>>>>>>Regards, > >>>>>>>>>>>Dave > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>Dan Magenheimer wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>>Thanks, I hadn't realized that! No wonder we didn't > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>see the same > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>improvement you saw! > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>Try specifying clock=3Dpit on the linux boot line... > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>I'm confused... do you mean "clocksource=3Dpit" on the Xen > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>command line or > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>>"nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the = > guest (or > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>dom0?) command > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>>line? Or both places? Since the tests take awhile, it > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>would be nice > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>>to get this right the first time. Do the Xen and guest > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>clocksources need > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>>to be the same? > >>>>>>>>>>>> > >>>>>>>>>>>>Thanks, > >>>>>>>>>>>>Dan > >>>>>>>>>>>> > >>>>>>>>>>>>-----Original Message----- > >>>>>>>>>>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM > >>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser > >>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com; > >>>>>>>>>>>> > >>>>>>>>>>>> > >>deepak.patel@oracle.com; > >> > >> > >>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell > >>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>that disables > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>pending missed ticks > >>>>>>>>>>>> > >>>>>>>>>>>> Hi Dan, > >>>>>>>>>>>> > >>>>>>>>>>>> Hpet timer does have a fairly large error, as I > >>>>>>>>>>>>was > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>trying this > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> one recently. > >>>>>>>>>>>> I don't remember what I got for error, but 1% sounds > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>about right. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> The problem is that hpet is not built on top of vpt.c, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>the module > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> Keir and I did > >>>>>>>>>>>> all the recent work in, for its periodic timer = > needs. Try > >>>>>>>>>>>> specifying clock=3Dpit > >>>>>>>>>>>> on the linux boot line. If it still picks the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>hpet, which it > >> > >> > >>>>>>>>>>>> might, let me know > >>>>>>>>>>>> and I'll tell you how to get around this. > >>>>>>>>>>>> > >>>>>>>>>>>> Regards, > >>>>>>>>>>>> Dave > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>-------------------------------------------------------------- > >> > >> > >>>>>>>>>>>---------- > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> *From:* Dan Magenheimer > >>>>>>>>>>>> > >>>>>>>>>>>> > >>[mailto:dan.magenheimer@oracle.com] > >> > >> > >>>>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM > >>>>>>>>>>>> *To:* Dave Winchell; Keir Fraser > >>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; > >>>>>>>>>>>> > >>>>>>>>>>>> > >>deepak.patel@oracle.com; > >> > >> > >>>>>>>>>>>> akira.ijuin@oracle.com > >>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>that disables > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> pending missed ticks > >>>>>>>>>>>> > >>>>>>>>>>>> Sorry for the very late followup on this but we finally > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>were able > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> to get our testing set up again on stable 3.1 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>bits and have > >> > >> > >>>>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>order of 1%. > >> > >> > >>>>>>>>>>>> Test enviroment was a 4-socket dual core machine > >>>>>>>>>>>> > >>>>>>>>>>>> > >>with 24GB of > >> > >> > >>>>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>plus two pv. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> All six guests were running LTP simultaneously. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>The four hvm > >> > >> > >>>>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>RHEL4u5-64. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>32-bit guests. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> All four hvm guests experienced skew around -1%, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>even the 32-bit > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> guest. Less intensive testing didn't exhibit much > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>skew at all. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> A representative graph is attached. > >>>>>>>>>>>> > >>>>>>>>>>>> Dave, I wonder if some portion of your patches > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>didn't end up in > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> the xen trees? > >>>>>>>>>>>> > >>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>Platform timer > >> > >> > >>>>>>>>>>>> 14.318MHz HPET.) > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Dan > >>>>>>>>>>>> > >>>>>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests. > >>>>>>>>>>>> > >>>>>>>>>>>> > -----Original Message----- > >>>>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com > >>>>>>>>>>>> > > >>>>>>>>>>>> > >>>>>>>>>>>> > >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >> > >> > >>>>>>>>>>>> > Dave Winchell > >>>>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM > >>>>>>>>>>>> > To: Keir Fraser > >>>>>>>>>>>> > Cc: dan.magenheimer@oracle.com; > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>xen-devel@lists.xensource.com; Dave > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > Winchell > >>>>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>>>>>>>>> > disables pending > >>>>>>>>>>>> > missed ticks > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > Hi Keir, > >>>>>>>>>>>> > > >>>>>>>>>>>> > The latest change, c/s 16690, looks fine. > >>>>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to > >>>>>>>>>>>> > the code I submitted. Also, your version is more > >>>>>>>>>>>> > concise. > >>>>>>>>>>>> > > >>>>>>>>>>>> > The error tests confirm the equivalence. With > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>overnight cpu loads, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > the checked in version was accurate to +.048% for sles > >>>>>>>>>>>> > and +.038% for red hat. My version was +.046% > >>>>>>>>>>>>and > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>+.032% in a > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > 2 hour test. > >>>>>>>>>>>> > I don't think the difference is significant. > >>>>>>>>>>>> > > >>>>>>>>>>>> > i/o loads produced errors of +.01%. > >>>>>>>>>>>> > > >>>>>>>>>>>> > Thanks for all your efforts on this issue. > >>>>>>>>>>>> > > >>>>>>>>>>>> > Regards, > >>>>>>>>>>>> > Dave > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > Keir Fraser wrote: > >>>>>>>>>>>> > > >>>>>>>>>>>> > >Applied as c/s 16690, although the = > checked-in patch is > >>>>>>>>>>>> > smaller. I think the > >>>>>>>>>>>> > >only important fix is to pt_intr_post() and the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>only bit of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > the patch I > >>>>>>>>>>>> > >totally omitted was the change to > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>pt_process_missed_ticks(). > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > I don't think > >>>>>>>>>>>> > >that change can be important, but let's see what > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>happens to the > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> error > >>>>>>>>>>>> > >percentage... > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > -- Keir > >>>>>>>>>>>> > > > >>>>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > >>Hi Dan and Keir, > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >>Attached is a patch that fixes some issues with the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>SYNC policy > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>(no_missed_ticks_pending). > >>>>>>>>>>>> > >>I have not tried to make the change the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>minimal one, but, > >> > >> > >>>>>>>>>>>> > rather, just > >>>>>>>>>>>> > >>ported into > >>>>>>>>>>>> > >>the new code what I know to work well. The error for > >>>>>>>>>>>> > >>no_missed_ticks_pending goes from > >>>>>>>>>>>> > >>over 3% to .03% with this change according > >>>>>>>>>>>> > >>>>>>>>>>>> > >>to my testing. > >> > >> > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >>Regards, > >>>>>>>>>>>> > >>Dave > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >>Dan Magenheimer wrote: > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >>>Hi Dave -- > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>>Did you get your correction ported? If so, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>it would be > >> > >> > >>>>>>>>>>>> > nice to see this get > >>>>>>>>>>>> > >>>into 3.1.3. > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>>Note that I just did some very limited testing with > >>>>>>>>>>>> > timer_mode=3D2(=3DSYNC=3Dno > >>>>>>>>>>>> > >>>missed ticks pending) > >>>>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>guest) and the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > worst error I've > >>>>>>>>>>>> > >>>seen so far > >>>>>>>>>>>> > >>>is 0.012%. But I haven't tried any exotic > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>loads, just LTP. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>>Thanks, > >>>>>>>>>>>> > >>>Dan > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>>>-----Original Message----- > >>>>>>>>>>>> > >>>>From: Dave Winchell > >>>>>>>>>>>> > >>>>>>>>>>>> > >>[mailto:dwinchell@virtualiron.com] > >> > >> > >>>>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com > >>>>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; > >>>>>>>>>>>> > xen-devel@lists.xensource.com; Dong, > >>>>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a > >>>>>>>>>>>> > >>>>>>>>>>>> > >>timer mode that > >> > >> > >>>>>>>>>>>> > >>>>disables pending > >>>>>>>>>>>> > >>>>missed ticks > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>Dan, > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>I did some testing with the constant tsc offset > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>SYNC method > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>(now called > >>>>>>>>>>>> > >>>>no_missed_ticks_pending) > >>>>>>>>>>>> > >>>>and found the error to be very high, much larger > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>than 1 %, as > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>I recall. > >>>>>>>>>>>> > >>>>I have not had a chance to submit a correction. I > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>will try to > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>do it later > >>>>>>>>>>>> > >>>>this week or the first week in January. My > >>>>>>>>>>>> > >>>>>>>>>>>> > >>version of > >> > >> > >>>>>>>>>>>> constant tsc > >>>>>>>>>>>> > >>>>offset SYNC method > >>>>>>>>>>>> > >>>>produces .02 % error, so I just need to port > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>that into the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>current code. > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>The error you got for both of those kernels is > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>what I would > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> expect > >>>>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode. > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>Regards, > >>>>>>>>>>>> > >>>>Dave > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>Dan Magenheimer wrote: > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>>Anyone make measurements on the final patch? > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>saw a loss of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>about 0.2% with no load. This was > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>xen-unstable tip today > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>with no options specified. 32-bit was = > about 0.01%. > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>>I think I missed something... how do I > >>>>>>>>>>>> > >>>>>>>>>>>> > >>run the various > >> > >> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>accounting choices and which ones are known to be > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>appropriate > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>for which kernels? > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>>Thanks, > >>>>>>>>>>>> > >>>>>Dan > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>>>-----Original Message----- > >>>>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>>>>>>> > > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>Keir Fraser > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>>>>>>>>> > >>>>>>To: Dave Winchell > >>>>>>>>>>>> > >>>>>>Cc: Shan, Haitao; > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>xen-devel@lists.xensource.com; Dong, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > Eddie; Jiang, > >>>>>>>>>>>> > >>>>>>Yunhong > >>>>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>mode that > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>disables pending > >>>>>>>>>>>> > >>>>>>missed ticks > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>Please take a look at xen-unstable > >>>>>>>>>>>> > >>>>>>>>>>>> > >>changeset 16545. > >> > >> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>-- Keir > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>> wrote: > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>Keir, > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>loads for the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>various time protocols follows. In > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>addition, the data > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>for cpu loads is shown. > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>processor AMD > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> box. > >>>>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>vcpu each. > >> > >> > >>>>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>>>>>>>>> > >>>>>>>(usex is available at > >>>>>>>>>>>> http://people.redhat.com/anderson/usex.) > >>>>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>of=3D/dev/null. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 = > instances of dd. > >>>>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>>>>>>>>> > >>>>>>>All three guests are 8vcpu. > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same > >>>>>>>>>>>> > >>>>>>>>>>>> > >>as i/o-32 > >> > >> > >>>>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>instances of dd. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, > >>>>>>>>>>>>+4.42 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>sec -.006%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > +.005% cpu > >>>>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>sec, -.001%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > +.012% cpu > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>sec, -.009%, > >> > >> > >>>>>>>>>>>> -.004% cpu > >>>>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>-.005%, -.005% cpu > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>-.008%, -.003% cpu > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>-.040% cpu > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>sec, -.034%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > -.031% cpu > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>sec,-55.7 sec, -.01%, > >> > >> > >>>>>>>>>>>> > -.09% i/o-8 > >>>>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>sec,-14.0 sec, -.015% > >> > >> > >>>>>>>>>>>> > -.14% i/o-8 > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>sec, -.017%, > >> > >> > >>>>>>>>>>>> > -.022% i/o-8 > >>>>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>-.017%, -.018% i/o-8 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>sec, -.020%, > >> > >> > >>>>>>>>>>>> > -.029% i/o-8 > >>>>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>sec, -.023%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > -.031% i/o-8 > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>-.04% i/o-32 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>sec, -.011%, > >> > >> > >>>>>>>>>>>> > -.005% i/o-32 > >>>>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>-.11% i/o-32 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>13. sec -.07%, > >> > >> > >>>>>>>>>>>> > .003% i/o-4/32 > >>>>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>sec, -.017%, > >> > >> > >>>>>>>>>>>> > .01% i/o-4/32 > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>Overhead measurements: > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>Progress in terms of number of passes > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>through a fixed > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>system workload > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. > >>>>>>>>>>>> > >>>>>>>The workload was usex -b48. > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>Conclusions: > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy > >>>>>>>>>>>> > requirement for ntp > >>>>>>>>>>>> > >>>>>>>tracking under the loads > >>>>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>accuracies for > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>SYNC, MIXED, > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>and ASYNC > >>>>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC > >>>>>>>>>>>> > >>>>>>>>>>>> > >>method by only > >> > >> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>scheduling the extra > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>wakeups if a certain number > >>>>>>>>>>>> > >>>>>>>of ticks are missed. > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>Regards, > >>>>>>>>>>>> > >>>>>>>Dave > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>Keir Fraser wrote: > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>> wrote: > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>ASYNC method a > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>couple of days ago, > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think > >>>>>>>>>>>> > >>>>>>>>>>>> > >>there may have > >> > >> > >>>>>>>>>>>> > been something > >>>>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>days ago for > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>ASYNC. It may have been > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>after context > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>switch in. > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC > >>>>>>>>>>>> > >>>>>>>>>>>> > >>or ASYNC give > >> > >> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>acceptable accuracy, > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>each running consistently around or under > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>.01%. MIXED has > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>a fairly high > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>error of > >>>>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close > >>>>>>>>>>>> > >>>>>>>>>>>> > >>to .05% ntp > >> > >> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>threshold for comfort. > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>plan to leave > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>SYNC running > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>leave MIXED > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>running instead. > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>I can run > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>more overnight tests > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>>next week. > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>effects of the > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>SYNC+run_timer > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>cause higher > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>system-wide CPU > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>contention. I find it easier to think > >>>>>>>>>>>> > >>>>>>>>>>>> > >>through the > >> > >> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>implications of ASYNC. I'm > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>accurate than > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>ASYNC. Perhaps it > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>approaches, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>and each interrupt > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>event causes a small accumulated error? > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>favourites and > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>if the latter is > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>actually more accurate then I can > >>>>>>>>>>>> > >>>>>>>>>>>> > >>simply revert the > >> > >> > >>>>>>>>>>>> > changeset that > >>>>>>>>>>>> > >>>>>>>>implemented MIXED. > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>workloads you > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>could try idle > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>large disc reads > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>to /dev/null)? We > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>CPU bound, so > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>that's really an > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>>-- Keir > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>_______________________________________________ > >>>>>>>>>>>> > >>>>>>Xen-devel mailing list > >>>>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com > >>>>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>> > >>>>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>>>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>2007 +0000 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>2008 -0500 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>pt_process_missed_ticks(stru > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >> missed_ticks =3D missed_ticks / (s_time_t) > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>pt->period + 1; > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >> if ( mode_is(pt->vcpu->domain, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>no_missed_ticks_pending) ) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>>>>>>>>>>> > >>+ pt->do_not_freeze =3D 1; > >>>>>>>>>>>> > >> else > >>>>>>>>>>>> > >> pt->pending_intr_nr +=3D missed_ticks; > >>>>>>>>>>>> > >> pt->scheduled +=3D missed_ticks * pt->period; > >>>>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void > >>>>>>>>>>>> > >>>>>>>>>>>> > >>pt_timer_fn(void *data) > >> > >> > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >> pt_lock(pt); > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >>- pt->pending_intr_nr++; > >>>>>>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>no_missed_ticks_pending) ) { > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>+ pt->pending_intr_nr =3D 1; > >>>>>>>>>>>> > >>+ pt->do_not_freeze =3D 0; > >>>>>>>>>>>> > >>+ } > >>>>>>>>>>>> > >>+ else > >>>>>>>>>>>> > >>+ pt->pending_intr_nr++; > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >> if ( !pt->one_shot ) > >>>>>>>>>>>> > >> { > >>>>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>vcpu *v, struct > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >> return; > >>>>>>>>>>>> > >> } > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >>- pt->do_not_freeze =3D 0; > >>>>>>>>>>>> > >>- > >>>>>>>>>>>> > >> if ( pt->one_shot ) > >>>>>>>>>>>> > >> { > >>>>>>>>>>>> > >> pt->enabled =3D 0; > >>>>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>*v, struct > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >> pt->last_plt_gtime =3D > >>>>>>>>>>>> > >>>>>>>>>>>> > >>hvm_get_guest_time(v); > >> > >> > >>>>>>>>>>>> > >> pt->pending_intr_nr =3D 0; /* > >>>>>>>>>>>> > >>>>>>>>>>>> > >>'collapse' all > >> > >> > >>>>>>>>>>>> > missed ticks */ > >>>>>>>>>>>> > >> } > >>>>>>>>>>>> > >>+ else if ( mode_is(v->domain, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>no_missed_ticks_pending) ) { > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>> > >>+ pt->pending_intr_nr--; > >>>>>>>>>>>> > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>>>>>>>>>>> > >>+ } > >>>>>>>>>>>> > >> else > >>>>>>>>>>>> > >> { > >>>>>>>>>>>> > >> pt->last_plt_gtime +=3D = > pt->period_cycles; > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > >> > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > _______________________________________________ > >>>>>>>>>>>> > Xen-devel mailing list > >>>>>>>>>>>> > Xen-devel@lists.xensource.com > >>>>>>>>>>>> > http://lists.xensource.com/xen-devel > >>>>>>>>>>>> > > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>> > >>> > >>> > >>-------------------------------------------------------------- > >>---------- > >> > >> > >> > >> > > > > > > > = > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 08 Feb 2008 16:21:16 -0500 Message-ID: <47ACC7CC.5030103@virtualiron.com> References: <20080201153111687.00000002384@djm-pc> <47A77075.3010707@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47A77075.3010707@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: Dave Winchell , Deepak Patel , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Hi Dan, Sorry it took me so long, but I finally ran an ltp test today. Its on rh4u4-64. I'm using the defaults for ltp and using a script called runltp. I had a usex load on rh4u5-64. No ntpd. virtual processors / physical processors = 2. The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes for -.005% and .008%. I'm running a 3.1 based hypervisor with some clock related tweaks that I haven't submitted, because I'm still characterizing them. I'm stopping the usex load on 4u5-64 now and replacing it with ltp and will leave the two guests running ltp over the weekend. Regards, Dave Dave Winchell wrote: > Hi Dan, Deepak: > > Thanks for the data. Those drifts are severe - no wonder ntp couldn't > keep then in synch. I'll try to reproduce that behaviour here, with my > code base. > If I can't reproduce it, I'll try 3.2. > > If you can isolate what ltp is doing during the cliffs, that would be > very > helpful. > > thanks, > Dave > > > > > Dan Magenheimer wrote: > >> OK, Deepak repeated the test without ntpd and using ntpdate -b before >> the test. >> >> The attached graph shows his results: el5u1-64 (best=~0.07%), >> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%). >> >> We will continue to look at LTP to try to isolate. >> >> Thanks, >> Dan >> >> P.S. elXuY is essentially RHEL XuY with some patches. >> >> >> >>> -----Original Message----- >>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>> Sent: Wednesday, January 30, 2008 2:45 PM >>> To: Deepak Patel >>> Cc: dan.magenheimer@oracle.com; Keir Fraser; >>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell >>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending >>> missed ticks >>> >>> >>> Dan, Deeepak, >>> >>> It may be that the underlying clock error is too great for ntp >>> to handle. It would be useful if you did not run ntpd >>> and, instead did ntpdate -b at the start of the test >>> for each guest. Then capture the data as you have been doing. >>> If the drift is greater than .05%, then we need to address that. >>> >>> Another option is, when running ntpd, to enable loop statistics in >>> /etc/ntp.conf >>> by adding this to the file: >>> >>> statistics loopstats >>> statsdir /var/lib/ntp/ >>> >>> Then you will see loop data in that directory. >>> Correlating the data in the loopstats files with the >>> peaks in skew would be interesting. You will see entries of the form >>> >>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10 >>> >>> Where the second to last column is the Allan Deviation. When that >>> gets over 1000, ntpd is working pretty hard. However, I have not >>> seen ntpd >>> completely loose it like you have. >>> >>> I'm on vacation until Monday, and won't be reading >>> email. >>> >>> Thanks for all your work on this! >>> >>> -Dave >>> >>> Deepak Patel wrote: >>> >>> >>> >>>>> Is the graph for RHEL5u1-64? (I've never tested this one.) >>>>> >>>> >>>> I do not know which graph was attached with this. But I saw this >>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I >>>> was running ltp tests continuously. >>>> >>>> >>>> >>>>> What was the behaviour of the other guests running? >>>>> >>>> >>>> All pvm guests are fine. But behavior of most of the hvm guests were >>>> as described. >>>> >>>> >>>> >>>>> If they had spikes, were they at the same wall time? >>>>> >>>> >>>> No. They are not at the same wall time. >>>> >>>> >>>> >>>>> Were the other guests running ltp as well? >>>>> >>>>> >>>> >>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp >>>> continuously. >>>> >>>> >>>> >>>>> How are you measuring skew? >>>>> >>>> >>>> I was collecting output of "ntpdate -q every >>> >>> 300 seconds >>> >>> >>>> (5 minutes) and have created graph based on that. >>>> >>>> >>>> >>>>> Are you running ntpd? >>>>> >>>>> >>>> >>>> Yes. ntp was running on all the guests. >>>> >>>> I am investigating what causes this spikes and let everyone >>> >>> know what >>> >>> >>>> are my findings. >>>> >>>> Thanks, >>>> Deepak >>>> >>>> >>>> >>>>> Anything that you can discover that would be in sync with >>>>> the spikes would be very helpful! >>>>> >>>>> The code that I test with is our product code, which is based >>>>> on 3.1. So it is possible that something in 3.2 other than vpt.c >>>>> is the cause. I can test with 3.2, if necessary. >>>>> >>>>> thanks, >>>>> Dave >>>>> >>>>> >>>>> >>>>> Dan Magenheimer wrote: >>>>> >>>>> >>>>> >>>>>> Hi Dave (Keir, see suggestion below) -- >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Turning off vhpet certainly helps a lot (though see below). >>>>>> >>>>>> I wonder if timekeeping with vhpet is so bad that it should be >>>>>> turned off by default (in 3.1, 3.2, and unstable) until it is >>>>>> fixed? (I have a patch that defaults it off, can post it if >>>>>> there is agreement on the above point.) The whole point of an >>>>>> HPET is to provide more precise timekeeping and if vhpet is >>>>>> worse than vpit, it can only confuse users. Comments? >>>>>> >>>>>> >>>>>> In your testing, are you just measuring % skew over a long >>>>>> period of time? >>>>>> We are graphing the skew continuously and >>>>>> seeing periodic behavior that is unsettling, even with pit. >>>>>> See attached. Though your algorithm recovers, the "cliffs" >>>>>> could still cause real user problems. I wonder if there is >>>>>> anything that can be done to make the "recovery" more >>>>>> responsive? >>>>>> >>>>>> We are looking into what part(s) of LTP is causing the cliffs. >>>>>> >>>>>> Thanks, >>>>>> Dan >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>> Sent: Monday, January 28, 2008 8:21 AM >>>>>>> To: dan.magenheimer@oracle.com >>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>>>>> deepak.patel@oracle.com; >>>>>>> akira.ijuin@oracle.com; Dave Winchell >>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>>>>> pending >>>>>>> missed ticks >>>>>>> >>>>>>> >>>>>>> Dan, >>>>>>> >>>>>>> I guess I'm a bit out of date calling for clock= usage. >>>>>>> Looking at linux 2.6.20.4 sources, I think you should specify >>>>>>> "clocksource=pit nohpet" on the linux guest bootline. >>>>>>> >>>>>>> You can leave the xen and dom0 bootlines as they are. >>>>>>> The xen and guest clocksources do not need to be the same. >>>>>>> In my tests, xen is using the hpet for its timekeeping and >>>>>>> that appears to be the default. >>>>>>> >>>>>>> When you boot the guests you should see >>>>>>> time.c: Using PIT/TSC based timekeeping. >>>>>>> on the rh4u5-64 guest, and something similar on the others. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>> 14.318MHz HPET.) >>>>>>>> >>>>>>> >>>>>>> This appears to be the xen state, which is fine. >>>>>>> I was wrongly assuming that this was the guest state. >>>>>>> You might want to look in your guest logs and see what they were >>>>>>> picking >>>>>>> for a clock source. >>>>>>> >>>>>>> Regards, >>>>>>> Dave >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Dan Magenheimer wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Thanks, I hadn't realized that! No wonder we didn't >>>>>>> >>> see the same >>> >>> >>>>>>>> improvement you saw! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Try specifying clock=pit on the linux boot line... >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen >>>>>>>> >>>>>>> >>>>>>> command line or >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or >>>>>>>> >>>>>>> >>>>>>> dom0?) command >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> line? Or both places? Since the tests take awhile, it >>>>>>>> >>>>>>> >>>>>>> would be nice >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> to get this right the first time. Do the Xen and guest >>>>>>>> >>>>>>> >>>>>>> clocksources need >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> to be the same? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dan >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM >>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser >>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>>>>> akira.ijuin@oracle.com; Dave Winchell >>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>> >>> that disables >>> >>> >>>>>>>> pending missed ticks >>>>>>>> >>>>>>>> Hi Dan, >>>>>>>> >>>>>>>> Hpet timer does have a fairly large error, as I was >>>>>>> >>> trying this >>> >>> >>>>>>>> one recently. >>>>>>>> I don't remember what I got for error, but 1% sounds >>>>>>>> >>>>>>> >>>>>>> about right. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> The problem is that hpet is not built on top of vpt.c, >>>>>>>> >>>>>>> >>>>>>> the module >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Keir and I did >>>>>>>> all the recent work in, for its periodic timer needs. Try >>>>>>>> specifying clock=pit >>>>>>>> on the linux boot line. If it still picks the hpet, which it >>>>>>>> might, let me know >>>>>>>> and I'll tell you how to get around this. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Dave >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -------------------------------------------------------------- >>>>>>> ---------- >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM >>>>>>>> *To:* Dave Winchell; Keir Fraser >>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>>>>> akira.ijuin@oracle.com >>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>> >>>>>>> >>>>>>> that disables >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> pending missed ticks >>>>>>>> >>>>>>>> Sorry for the very late followup on this but we finally >>>>>>>> >>>>>>> >>>>>>> were able >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> to get our testing set up again on stable 3.1 bits and have >>>>>>>> seen some very bad results on 3.1.3-rc1, on the order of 1%. >>>>>>>> >>>>>>>> Test enviroment was a 4-socket dual core machine with 24GB of >>>>>>>> memory running six two-vcpu 2GB domains, four hvm >>>>>>> >>> plus two pv. >>> >>> >>>>>>>> All six guests were running LTP simultaneously. The four hvm >>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and >>>>>>> >>> RHEL4u5-64. >>> >>> >>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>>>>>>> >>>>>>> >>>>>>> 32-bit guests. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> All four hvm guests experienced skew around -1%, >>>>>>> >>> even the 32-bit >>> >>> >>>>>>>> guest. Less intensive testing didn't exhibit much >>>>>>> >>> skew at all. >>> >>> >>>>>>>> A representative graph is attached. >>>>>>>> >>>>>>>> Dave, I wonder if some portion of your patches >>>>>>> >>> didn't end up in >>> >>> >>>>>>>> the xen trees? >>>>>>>> >>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>> 14.318MHz HPET.) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dan >>>>>>>> >>>>>>>> P.S. Many thanks to Deepak and Akira for running tests. >>>>>>>> >>>>>>>> > -----Original Message----- >>>>>>>> > From: xen-devel-bounces@lists.xensource.com >>>>>>>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>> > Dave Winchell >>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>>>>>> > To: Keir Fraser >>>>>>>> > Cc: dan.magenheimer@oracle.com; >>>>>>>> >>>>>>> >>>>>>> xen-devel@lists.xensource.com; Dave >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > Winchell >>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>> > disables pending >>>>>>>> > missed ticks >>>>>>>> > >>>>>>>> > >>>>>>>> > Hi Keir, >>>>>>>> > >>>>>>>> > The latest change, c/s 16690, looks fine. >>>>>>>> > I agree that the code in c/s 16690 is equivalent to >>>>>>>> > the code I submitted. Also, your version is more >>>>>>>> > concise. >>>>>>>> > >>>>>>>> > The error tests confirm the equivalence. With >>>>>>>> >>>>>>> >>>>>>> overnight cpu loads, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > the checked in version was accurate to +.048% for sles >>>>>>>> > and +.038% for red hat. My version was +.046% and >>>>>>> >>> +.032% in a >>> >>> >>>>>>>> > 2 hour test. >>>>>>>> > I don't think the difference is significant. >>>>>>>> > >>>>>>>> > i/o loads produced errors of +.01%. >>>>>>>> > >>>>>>>> > Thanks for all your efforts on this issue. >>>>>>>> > >>>>>>>> > Regards, >>>>>>>> > Dave >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > Keir Fraser wrote: >>>>>>>> > >>>>>>>> > >Applied as c/s 16690, although the checked-in patch is >>>>>>>> > smaller. I think the >>>>>>>> > >only important fix is to pt_intr_post() and the >>>>>>> >>> only bit of >>> >>> >>>>>>>> > the patch I >>>>>>>> > >totally omitted was the change to >>>>>>> >>> pt_process_missed_ticks(). >>> >>> >>>>>>>> > I don't think >>>>>>>> > >that change can be important, but let's see what >>>>>>>> >>>>>>> >>>>>>> happens to the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> error >>>>>>>> > >percentage... >>>>>>>> > > >>>>>>>> > > -- Keir >>>>>>>> > > >>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" >>>>>>>> >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > >>Hi Dan and Keir, >>>>>>>> > >> >>>>>>>> > >>Attached is a patch that fixes some issues with the >>>>>>>> >>>>>>> >>>>>>> SYNC policy >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>(no_missed_ticks_pending). >>>>>>>> > >>I have not tried to make the change the minimal one, but, >>>>>>>> > rather, just >>>>>>>> > >>ported into >>>>>>>> > >>the new code what I know to work well. The error for >>>>>>>> > >>no_missed_ticks_pending goes from >>>>>>>> > >>over 3% to .03% with this change according to my testing. >>>>>>>> > >> >>>>>>>> > >>Regards, >>>>>>>> > >>Dave >>>>>>>> > >> >>>>>>>> > >>Dan Magenheimer wrote: >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >>>Hi Dave -- >>>>>>>> > >>> >>>>>>>> > >>>Did you get your correction ported? If so, it would be >>>>>>>> > nice to see this get >>>>>>>> > >>>into 3.1.3. >>>>>>>> > >>> >>>>>>>> > >>>Note that I just did some very limited testing with >>>>>>>> > timer_mode=2(=SYNC=no >>>>>>>> > >>>missed ticks pending) >>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv >>>>>>> >>> guest) and the >>> >>> >>>>>>>> > worst error I've >>>>>>>> > >>>seen so far >>>>>>>> > >>>is 0.012%. But I haven't tried any exotic >>>>>>> >>> loads, just LTP. >>> >>> >>>>>>>> > >>> >>>>>>>> > >>>Thanks, >>>>>>>> > >>>Dan >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>>>-----Original Message----- >>>>>>>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>>>>> > >>>>To: dan.magenheimer@oracle.com >>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>>>>>> > xen-devel@lists.xensource.com; Dong, >>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>> > >>>>disables pending >>>>>>>> > >>>>missed ticks >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>>Dan, >>>>>>>> > >>>> >>>>>>>> > >>>>I did some testing with the constant tsc offset >>>>>>>> >>>>>>> >>>>>>> SYNC method >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>(now called >>>>>>>> > >>>>no_missed_ticks_pending) >>>>>>>> > >>>>and found the error to be very high, much larger >>>>>>>> >>>>>>> >>>>>>> than 1 %, as >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>I recall. >>>>>>>> > >>>>I have not had a chance to submit a correction. I >>>>>>>> >>>>>>> >>>>>>> will try to >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>do it later >>>>>>>> > >>>>this week or the first week in January. My version of >>>>>>>> constant tsc >>>>>>>> > >>>>offset SYNC method >>>>>>>> > >>>>produces .02 % error, so I just need to port >>>>>>> >>> that into the >>> >>> >>>>>>>> > >>>>current code. >>>>>>>> > >>>> >>>>>>>> > >>>>The error you got for both of those kernels is >>>>>>>> >>>>>>> >>>>>>> what I would >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> expect >>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. >>>>>>>> > >>>> >>>>>>>> > >>>>I'll let Keir answer on how to set the time mode. >>>>>>>> > >>>> >>>>>>>> > >>>>Regards, >>>>>>>> > >>>>Dave >>>>>>>> > >>>> >>>>>>>> > >>>>Dan Magenheimer wrote: >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>>>Anyone make measurements on the final patch? >>>>>>>> > >>>>> >>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and >>>>>>> >>> saw a loss of >>> >>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>about 0.2% with no load. This was >>>>>>> >>> xen-unstable tip today >>> >>> >>>>>>>> > >>>>with no options specified. 32-bit was about 0.01%. >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>>>I think I missed something... how do I run the various >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>accounting choices and which ones are known to be >>>>>>>> >>>>>>> >>>>>>> appropriate >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>for which kernels? >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>>>Thanks, >>>>>>>> > >>>>>Dan >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>>>-----Original Message----- >>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>Keir Fraser >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>>> > >>>>>>To: Dave Winchell >>>>>>>> > >>>>>>Cc: Shan, Haitao; >>>>>>> >>> xen-devel@lists.xensource.com; Dong, >>> >>> >>>>>>>> > Eddie; Jiang, >>>>>>>> > >>>>>>Yunhong >>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer >>>>>>>> >>>>>>> >>> mode that >>> >>> >>>>>>>> > >>>>>>disables pending >>>>>>>> > >>>>>>missed ticks >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>-- Keir >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>> wrote: >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>Keir, >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>The accuracy data I've collected for i/o >>>>>>> >>> loads for the >>> >>> >>>>>>>> > >>>>>>>various time protocols follows. In >>>>>>> >>> addition, the data >>> >>> >>>>>>>> > >>>>>>>for cpu loads is shown. >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>>>>>> >>>>>>> >>>>>>> processor AMD >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> box. >>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>> > >>>>>>>(usex is available at >>>>>>>> http://people.redhat.com/anderson/usex.) >>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>>>>>> >>>>>>> >>>>>>> of=/dev/null. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>> > >>>>>>>All three guests are 8vcpu. >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 >>>>>>> >>> instances of dd. >>> >>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 >>>>>>> >>> sec -.006%, >>> >>> >>>>>>>> > +.005% cpu >>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 >>>>>>> >>> sec, -.001%, >>> >>> >>>>>>>> > +.012% cpu >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, >>>>>>>> -.004% cpu >>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>>>>>> >>>>>>> >>>>>>> -.005%, -.005% cpu >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>>>>>> >>>>>>> >>>>>>> -.008%, -.003% cpu >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>>>>>> >>>>>>> >>>>>>> -.040% cpu >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 >>>>>>> >>> sec, -.034%, >>> >>> >>>>>>>> > -.031% cpu >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, >>>>>>>> > -.09% i/o-8 >>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% >>>>>>>> > -.14% i/o-8 >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, >>>>>>>> > -.022% i/o-8 >>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>>>>>> >>>>>>> >>>>>>> -.017%, -.018% i/o-8 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, >>>>>>>> > -.029% i/o-8 >>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 >>>>>>> >>> sec, -.023%, >>> >>> >>>>>>>> > -.031% i/o-8 >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>>>>>>> >>>>>>> >>>>>>> -.04% i/o-32 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, >>>>>>>> > -.005% i/o-32 >>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>>>>>>> >>>>>>> >>>>>>> -.11% i/o-32 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, >>>>>>>> > .003% i/o-4/32 >>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, >>>>>>>> > .01% i/o-4/32 >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>Overhead measurements: >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>Progress in terms of number of passes >>>>>>> >>> through a fixed >>> >>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>system workload >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>> > >>>>>>>The workload was usex -b48. >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>Conclusions: >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>>>>>> > requirement for ntp >>>>>>>> > >>>>>>>tracking under the loads >>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case >>>>>>>> >>>>>>> >>>>>>> accuracies for >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>SYNC, MIXED, >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>and ASYNC >>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>We could reduce the cost of the SYNC method by only >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>scheduling the extra >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>wakeups if a certain number >>>>>>>> > >>>>>>>of ticks are missed. >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>Regards, >>>>>>>> > >>>>>>>Dave >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>Keir Fraser wrote: >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>> >>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>> wrote: >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>>>>>>> >>>>>>> >>>>>>> ASYNC method a >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>couple of days ago, >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have >>>>>>>> > been something >>>>>>>> > >>>>>>>>>wrong with the code I used a couple of >>>>>>> >>> days ago for >>> >>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>ASYNC. It may have been >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt >>>>>>>> >>>>>>> >>>>>>> after context >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>switch in. >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>acceptable accuracy, >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>each running consistently around or under >>>>>>>> >>>>>>> >>>>>>> .01%. MIXED has >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>a fairly high >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>error of >>>>>>>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>threshold for comfort. >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>>>>>>> >>>>>>> >>>>>>> plan to leave >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>SYNC running >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can >>>>>>> >>> leave MIXED >>> >>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>running instead. >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and >>>>>>>> >>>>>>> >>>>>>> I can run >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>more overnight tests >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>>next week. >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>> >>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side >>>>>>>> >>>>>>> >>>>>>> effects of the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>SYNC+run_timer >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>>>>>>> >>>>>>> >>>>>>> cause higher >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>system-wide CPU >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>contention. I find it easier to think through the >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>implications of ASYNC. I'm >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less >>>>>>>> >>>>>>> >>>>>>> accurate than >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>ASYNC. Perhaps it >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>delivers more timer interrupts than the other >>>>>>>> >>>>>>> >>>>>>> approaches, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>and each interrupt >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>event causes a small accumulated error? >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>>>>>>> >>>>>>> >>>>>>> favourites and >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>if the latter is >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>actually more accurate then I can simply revert the >>>>>>>> > changeset that >>>>>>>> > >>>>>>>>implemented MIXED. >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>>Perhaps rather than running more of the same >>>>>>>> >>>>>>> >>>>>>> workloads you >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>could try idle >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>>>>>> >>>>>>> >>>>>>> large disc reads >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>to /dev/null)? We >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>don't have any data on workloads that aren't >>>>>>>> >>>>>>> >>>>>>> CPU bound, so >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>that's really an >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>>>>obvious place to put any further effort imo. >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>>-- Keir >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>>> >>>>>>>> > >>>>>>_______________________________________________ >>>>>>>> > >>>>>>Xen-devel mailing list >>>>>>>> > >>>>>>Xen-devel@lists.xensource.com >>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 >>>>>>>> >>>>>>> >>> 2007 +0000 >>> >>> >>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 >>>>>>>> >>>>>>> >>> 2008 -0500 >>> >>> >>>>>>>> > >>@@ -58,7 +58,7 @@ static void >>>>>>> >>> pt_process_missed_ticks(stru >>> >>> >>>>>>>> > >> >>>>>>>> > >> missed_ticks = missed_ticks / (s_time_t) >>>>>>>> >>>>>>> >>>>>>> pt->period + 1; >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >> if ( mode_is(pt->vcpu->domain, >>>>>>>> >>>>>>> >>>>>>> no_missed_ticks_pending) ) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>>>>> > >>+ pt->do_not_freeze = 1; >>>>>>>> > >> else >>>>>>>> > >> pt->pending_intr_nr += missed_ticks; >>>>>>>> > >> pt->scheduled += missed_ticks * pt->period; >>>>>>>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >>>>>>>> > >> >>>>>>>> > >> pt_lock(pt); >>>>>>>> > >> >>>>>>>> > >>- pt->pending_intr_nr++; >>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, >>>>>>>> >>>>>>> >>>>>>> no_missed_ticks_pending) ) { >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >>+ pt->pending_intr_nr = 1; >>>>>>>> > >>+ pt->do_not_freeze = 0; >>>>>>>> > >>+ } >>>>>>>> > >>+ else >>>>>>>> > >>+ pt->pending_intr_nr++; >>>>>>>> > >> >>>>>>>> > >> if ( !pt->one_shot ) >>>>>>>> > >> { >>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct >>>>>>> >>> vcpu *v, struct >>> >>> >>>>>>>> > >> return; >>>>>>>> > >> } >>>>>>>> > >> >>>>>>>> > >>- pt->do_not_freeze = 0; >>>>>>>> > >>- >>>>>>>> > >> if ( pt->one_shot ) >>>>>>>> > >> { >>>>>>>> > >> pt->enabled = 0; >>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>>>>>> >>>>>>> >>>>>>> *v, struct >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> > >> pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>> > >> pt->pending_intr_nr = 0; /* 'collapse' all >>>>>>>> > missed ticks */ >>>>>>>> > >> } >>>>>>>> > >>+ else if ( mode_is(v->domain, >>>>>>> >>> no_missed_ticks_pending) ) { >>> >>> >>>>>>>> > >>+ pt->pending_intr_nr--; >>>>>>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>> > >>+ } >>>>>>>> > >> else >>>>>>>> > >> { >>>>>>>> > >> pt->last_plt_gtime += pt->period_cycles; >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > >>>>>>>> > >>>>>>>> > _______________________________________________ >>>>>>>> > Xen-devel mailing list >>>>>>>> > Xen-devel@lists.xensource.com >>>>>>>> > http://lists.xensource.com/xen-devel >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Mon, 11 Feb 2008 11:52:54 -0500 Message-ID: <47B07D66.7030307@virtualiron.com> References: <20080201153111687.00000002384@djm-pc> <47A77075.3010707@virtualiron.com> <47ACC7CC.5030103@virtualiron.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------070202080006050007040301" Return-path: In-Reply-To: <47ACC7CC.5030103@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: Dave Winchell , Deepak Patel , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------070202080006050007040301 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Dan, Over the weekend the drift was +18 seconds for each guest (no ntp). The duration was 3900 minutes, so the error for each was +.0077%. Looking back through the data, it appears to drift linearly at this rate. I've attached a plot for rh4u5-64. This accuracy is better than what I've seen before (.03-.05%). This may be due to the different load (ltp vs usex) or to one of the changes I've made recently. I'll do some experimentation to see if there is a fix I should propose. This still doesn't address the radical drift you saw. The next step for me is to run 3.2 and see if I can reproduce it. Regards, Dave Dave Winchell wrote: > Hi Dan, > > Sorry it took me so long, but I finally ran an ltp test today. > Its on rh4u4-64. I'm using the defaults for ltp and using a script > called runltp. I had a usex load on rh4u5-64. No ntpd. > virtual processors / physical processors = 2. > > The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes > for -.005% and .008%. > > I'm running a 3.1 based hypervisor with some clock related tweaks that > I haven't submitted, because I'm still characterizing them. > > I'm stopping the usex load on 4u5-64 now and replacing it with ltp > and will leave the two guests running ltp over the weekend. > > Regards, > Dave > > > Dave Winchell wrote: > >> Hi Dan, Deepak: >> >> Thanks for the data. Those drifts are severe - no wonder ntp couldn't >> keep then in synch. I'll try to reproduce that behaviour here, with >> my code base. >> If I can't reproduce it, I'll try 3.2. >> >> If you can isolate what ltp is doing during the cliffs, that would be >> very >> helpful. >> >> thanks, >> Dave >> >> >> >> >> Dan Magenheimer wrote: >> >>> OK, Deepak repeated the test without ntpd and using ntpdate -b before >>> the test. >>> >>> The attached graph shows his results: el5u1-64 (best=~0.07%), >>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%). >>> >>> We will continue to look at LTP to try to isolate. >>> >>> Thanks, >>> Dan >>> >>> P.S. elXuY is essentially RHEL XuY with some patches. >>> >>> >>> >>>> -----Original Message----- >>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>> Sent: Wednesday, January 30, 2008 2:45 PM >>>> To: Deepak Patel >>>> Cc: dan.magenheimer@oracle.com; Keir Fraser; >>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell >>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>> pending >>>> missed ticks >>>> >>>> >>>> Dan, Deeepak, >>>> >>>> It may be that the underlying clock error is too great for ntp >>>> to handle. It would be useful if you did not run ntpd >>>> and, instead did ntpdate -b at the start of the test >>>> for each guest. Then capture the data as you have been doing. >>>> If the drift is greater than .05%, then we need to address that. >>>> >>>> Another option is, when running ntpd, to enable loop statistics in >>>> /etc/ntp.conf >>>> by adding this to the file: >>>> >>>> statistics loopstats >>>> statsdir /var/lib/ntp/ >>>> >>>> Then you will see loop data in that directory. >>>> Correlating the data in the loopstats files with the >>>> peaks in skew would be interesting. You will see entries of the form >>>> >>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10 >>>> >>>> Where the second to last column is the Allan Deviation. When that >>>> gets over 1000, ntpd is working pretty hard. However, I have not >>>> seen ntpd >>>> completely loose it like you have. >>>> >>>> I'm on vacation until Monday, and won't be reading >>>> email. >>>> >>>> Thanks for all your work on this! >>>> >>>> -Dave >>>> >>>> Deepak Patel wrote: >>>> >>>> >>>> >>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.) >>>>>> >>>>> >>>>> >>>>> I do not know which graph was attached with this. But I saw this >>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I >>>>> was running ltp tests continuously. >>>>> >>>>> >>>>> >>>>>> What was the behaviour of the other guests running? >>>>>> >>>>> >>>>> >>>>> All pvm guests are fine. But behavior of most of the hvm guests were >>>>> as described. >>>>> >>>>> >>>>> >>>>>> If they had spikes, were they at the same wall time? >>>>>> >>>>> >>>>> >>>>> No. They are not at the same wall time. >>>>> >>>>> >>>>> >>>>>> Were the other guests running ltp as well? >>>>>> >>>>>> >>>>> >>>>> >>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp >>>>> continuously. >>>>> >>>>> >>>>> >>>>>> How are you measuring skew? >>>>>> >>>>> >>>>> >>>>> I was collecting output of "ntpdate -q every >>>> >>>> >>>> 300 seconds >>>> >>>> >>>>> (5 minutes) and have created graph based on that. >>>>> >>>>> >>>>> >>>>>> Are you running ntpd? >>>>>> >>>>>> >>>>> >>>>> >>>>> Yes. ntp was running on all the guests. >>>>> >>>>> I am investigating what causes this spikes and let everyone >>>> >>>> >>>> know what >>>> >>>> >>>>> are my findings. >>>>> >>>>> Thanks, >>>>> Deepak >>>>> >>>>> >>>>> >>>>>> Anything that you can discover that would be in sync with >>>>>> the spikes would be very helpful! >>>>>> >>>>>> The code that I test with is our product code, which is based >>>>>> on 3.1. So it is possible that something in 3.2 other than vpt.c >>>>>> is the cause. I can test with 3.2, if necessary. >>>>>> >>>>>> thanks, >>>>>> Dave >>>>>> >>>>>> >>>>>> >>>>>> Dan Magenheimer wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Hi Dave (Keir, see suggestion below) -- >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Turning off vhpet certainly helps a lot (though see below). >>>>>>> >>>>>>> I wonder if timekeeping with vhpet is so bad that it should be >>>>>>> turned off by default (in 3.1, 3.2, and unstable) until it is >>>>>>> fixed? (I have a patch that defaults it off, can post it if >>>>>>> there is agreement on the above point.) The whole point of an >>>>>>> HPET is to provide more precise timekeeping and if vhpet is >>>>>>> worse than vpit, it can only confuse users. Comments? >>>>>>> >>>>>>> >>>>>>> In your testing, are you just measuring % skew over a long >>>>>>> period of time? >>>>>>> We are graphing the skew continuously and >>>>>>> seeing periodic behavior that is unsettling, even with pit. >>>>>>> See attached. Though your algorithm recovers, the "cliffs" >>>>>>> could still cause real user problems. I wonder if there is >>>>>>> anything that can be done to make the "recovery" more >>>>>>> responsive? >>>>>>> >>>>>>> We are looking into what part(s) of LTP is causing the cliffs. >>>>>>> >>>>>>> Thanks, >>>>>>> Dan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>> Sent: Monday, January 28, 2008 8:21 AM >>>>>>>> To: dan.magenheimer@oracle.com >>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>>>>>> deepak.patel@oracle.com; >>>>>>>> akira.ijuin@oracle.com; Dave Winchell >>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>>>>>> pending >>>>>>>> missed ticks >>>>>>>> >>>>>>>> >>>>>>>> Dan, >>>>>>>> >>>>>>>> I guess I'm a bit out of date calling for clock= usage. >>>>>>>> Looking at linux 2.6.20.4 sources, I think you should specify >>>>>>>> "clocksource=pit nohpet" on the linux guest bootline. >>>>>>>> >>>>>>>> You can leave the xen and dom0 bootlines as they are. >>>>>>>> The xen and guest clocksources do not need to be the same. >>>>>>>> In my tests, xen is using the hpet for its timekeeping and >>>>>>>> that appears to be the default. >>>>>>>> >>>>>>>> When you boot the guests you should see >>>>>>>> time.c: Using PIT/TSC based timekeeping. >>>>>>>> on the rh4u5-64 guest, and something similar on the others. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>>> 14.318MHz HPET.) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> This appears to be the xen state, which is fine. >>>>>>>> I was wrongly assuming that this was the guest state. >>>>>>>> You might want to look in your guest logs and see what they were >>>>>>>> picking >>>>>>>> for a clock source. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Dave >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Dan Magenheimer wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Thanks, I hadn't realized that! No wonder we didn't >>>>>>>> >>>>>>>> >>>> see the same >>>> >>>> >>>>>>>>> improvement you saw! >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Try specifying clock=pit on the linux boot line... >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> command line or >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> dom0?) command >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> line? Or both places? Since the tests take awhile, it >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> would be nice >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> to get this right the first time. Do the Xen and guest >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> clocksources need >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> to be the same? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Dan >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM >>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser >>>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>>>>>> akira.ijuin@oracle.com; Dave Winchell >>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>> >>>>>>>> >>>> that disables >>>> >>>> >>>>>>>>> pending missed ticks >>>>>>>>> >>>>>>>>> Hi Dan, >>>>>>>>> >>>>>>>>> Hpet timer does have a fairly large error, as I was >>>>>>>> >>>>>>>> >>>> trying this >>>> >>>> >>>>>>>>> one recently. >>>>>>>>> I don't remember what I got for error, but 1% sounds >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> about right. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> The problem is that hpet is not built on top of vpt.c, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> the module >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Keir and I did >>>>>>>>> all the recent work in, for its periodic timer needs. Try >>>>>>>>> specifying clock=pit >>>>>>>>> on the linux boot line. If it still picks the hpet, which it >>>>>>>>> might, let me know >>>>>>>>> and I'll tell you how to get around this. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Dave >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -------------------------------------------------------------- >>>>>>>> ---------- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM >>>>>>>>> *To:* Dave Winchell; Keir Fraser >>>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>>>>>> akira.ijuin@oracle.com >>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> that disables >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> pending missed ticks >>>>>>>>> >>>>>>>>> Sorry for the very late followup on this but we finally >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> were able >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> to get our testing set up again on stable 3.1 bits and have >>>>>>>>> seen some very bad results on 3.1.3-rc1, on the order of 1%. >>>>>>>>> >>>>>>>>> Test enviroment was a 4-socket dual core machine with 24GB of >>>>>>>>> memory running six two-vcpu 2GB domains, four hvm >>>>>>>> >>>>>>>> >>>> plus two pv. >>>> >>>> >>>>>>>>> All six guests were running LTP simultaneously. The four hvm >>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and >>>>>>>> >>>>>>>> >>>> RHEL4u5-64. >>>> >>>> >>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 32-bit guests. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> All four hvm guests experienced skew around -1%, >>>>>>>> >>>>>>>> >>>> even the 32-bit >>>> >>>> >>>>>>>>> guest. Less intensive testing didn't exhibit much >>>>>>>> >>>>>>>> >>>> skew at all. >>>> >>>> >>>>>>>>> A representative graph is attached. >>>>>>>>> >>>>>>>>> Dave, I wonder if some portion of your patches >>>>>>>> >>>>>>>> >>>> didn't end up in >>>> >>>> >>>>>>>>> the xen trees? >>>>>>>>> >>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>>> 14.318MHz HPET.) >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Dan >>>>>>>>> >>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests. >>>>>>>>> >>>>>>>>> > -----Original Message----- >>>>>>>>> > From: xen-devel-bounces@lists.xensource.com >>>>>>>>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>>> > Dave Winchell >>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>>>>>>> > To: Keir Fraser >>>>>>>>> > Cc: dan.magenheimer@oracle.com; >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> xen-devel@lists.xensource.com; Dave >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > Winchell >>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>>> > disables pending >>>>>>>>> > missed ticks >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Hi Keir, >>>>>>>>> > >>>>>>>>> > The latest change, c/s 16690, looks fine. >>>>>>>>> > I agree that the code in c/s 16690 is equivalent to >>>>>>>>> > the code I submitted. Also, your version is more >>>>>>>>> > concise. >>>>>>>>> > >>>>>>>>> > The error tests confirm the equivalence. With >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> overnight cpu loads, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > the checked in version was accurate to +.048% for sles >>>>>>>>> > and +.038% for red hat. My version was +.046% and >>>>>>>> >>>>>>>> >>>> +.032% in a >>>> >>>> >>>>>>>>> > 2 hour test. >>>>>>>>> > I don't think the difference is significant. >>>>>>>>> > >>>>>>>>> > i/o loads produced errors of +.01%. >>>>>>>>> > >>>>>>>>> > Thanks for all your efforts on this issue. >>>>>>>>> > >>>>>>>>> > Regards, >>>>>>>>> > Dave >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Keir Fraser wrote: >>>>>>>>> > >>>>>>>>> > >Applied as c/s 16690, although the checked-in patch is >>>>>>>>> > smaller. I think the >>>>>>>>> > >only important fix is to pt_intr_post() and the >>>>>>>> >>>>>>>> >>>> only bit of >>>> >>>> >>>>>>>>> > the patch I >>>>>>>>> > >totally omitted was the change to >>>>>>>> >>>>>>>> >>>> pt_process_missed_ticks(). >>>> >>>> >>>>>>>>> > I don't think >>>>>>>>> > >that change can be important, but let's see what >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> happens to the >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> error >>>>>>>>> > >percentage... >>>>>>>>> > > >>>>>>>>> > > -- Keir >>>>>>>>> > > >>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > >>Hi Dan and Keir, >>>>>>>>> > >> >>>>>>>>> > >>Attached is a patch that fixes some issues with the >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> SYNC policy >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>(no_missed_ticks_pending). >>>>>>>>> > >>I have not tried to make the change the minimal one, but, >>>>>>>>> > rather, just >>>>>>>>> > >>ported into >>>>>>>>> > >>the new code what I know to work well. The error for >>>>>>>>> > >>no_missed_ticks_pending goes from >>>>>>>>> > >>over 3% to .03% with this change according to my testing. >>>>>>>>> > >> >>>>>>>>> > >>Regards, >>>>>>>>> > >>Dave >>>>>>>>> > >> >>>>>>>>> > >>Dan Magenheimer wrote: >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >>>Hi Dave -- >>>>>>>>> > >>> >>>>>>>>> > >>>Did you get your correction ported? If so, it would be >>>>>>>>> > nice to see this get >>>>>>>>> > >>>into 3.1.3. >>>>>>>>> > >>> >>>>>>>>> > >>>Note that I just did some very limited testing with >>>>>>>>> > timer_mode=2(=SYNC=no >>>>>>>>> > >>>missed ticks pending) >>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv >>>>>>>> >>>>>>>> >>>> guest) and the >>>> >>>> >>>>>>>>> > worst error I've >>>>>>>>> > >>>seen so far >>>>>>>>> > >>>is 0.012%. But I haven't tried any exotic >>>>>>>> >>>>>>>> >>>> loads, just LTP. >>>> >>>> >>>>>>>>> > >>> >>>>>>>>> > >>>Thanks, >>>>>>>>> > >>>Dan >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> > >>>>-----Original Message----- >>>>>>>>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>>>>>> > >>>>To: dan.magenheimer@oracle.com >>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>>>>>>> > xen-devel@lists.xensource.com; Dong, >>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>>> > >>>>disables pending >>>>>>>>> > >>>>missed ticks >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>>Dan, >>>>>>>>> > >>>> >>>>>>>>> > >>>>I did some testing with the constant tsc offset >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> SYNC method >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>(now called >>>>>>>>> > >>>>no_missed_ticks_pending) >>>>>>>>> > >>>>and found the error to be very high, much larger >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> than 1 %, as >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>I recall. >>>>>>>>> > >>>>I have not had a chance to submit a correction. I >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> will try to >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>do it later >>>>>>>>> > >>>>this week or the first week in January. My version of >>>>>>>>> constant tsc >>>>>>>>> > >>>>offset SYNC method >>>>>>>>> > >>>>produces .02 % error, so I just need to port >>>>>>>> >>>>>>>> >>>> that into the >>>> >>>> >>>>>>>>> > >>>>current code. >>>>>>>>> > >>>> >>>>>>>>> > >>>>The error you got for both of those kernels is >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> what I would >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> expect >>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. >>>>>>>>> > >>>> >>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode. >>>>>>>>> > >>>> >>>>>>>>> > >>>>Regards, >>>>>>>>> > >>>>Dave >>>>>>>>> > >>>> >>>>>>>>> > >>>>Dan Magenheimer wrote: >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>>>Anyone make measurements on the final patch? >>>>>>>>> > >>>>> >>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and >>>>>>>> >>>>>>>> >>>> saw a loss of >>>> >>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>about 0.2% with no load. This was >>>>>>>> >>>>>>>> >>>> xen-unstable tip today >>>> >>>> >>>>>>>>> > >>>>with no options specified. 32-bit was about 0.01%. >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>>>I think I missed something... how do I run the various >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>accounting choices and which ones are known to be >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> appropriate >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>for which kernels? >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>>>Thanks, >>>>>>>>> > >>>>>Dan >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>>>-----Original Message----- >>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>Keir Fraser >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>>>> > >>>>>>To: Dave Winchell >>>>>>>>> > >>>>>>Cc: Shan, Haitao; >>>>>>>> >>>>>>>> >>>> xen-devel@lists.xensource.com; Dong, >>>> >>>> >>>>>>>>> > Eddie; Jiang, >>>>>>>>> > >>>>>>Yunhong >>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer >>>>>>>>> >>>>>>>> >>>>>>>> >>>> mode that >>>> >>>> >>>>>>>>> > >>>>>>disables pending >>>>>>>>> > >>>>>>missed ticks >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>-- Keir >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>> wrote: >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>Keir, >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o >>>>>>>> >>>>>>>> >>>> loads for the >>>> >>>> >>>>>>>>> > >>>>>>>various time protocols follows. In >>>>>>>> >>>>>>>> >>>> addition, the data >>>> >>>> >>>>>>>>> > >>>>>>>for cpu loads is shown. >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> processor AMD >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> box. >>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>>> > >>>>>>>(usex is available at >>>>>>>>> http://people.redhat.com/anderson/usex.) >>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> of=/dev/null. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>>> > >>>>>>>All three guests are 8vcpu. >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 >>>>>>>> >>>>>>>> >>>> instances of dd. >>>> >>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, +4.42 >>>>>>>> >>>>>>>> >>>> sec -.006%, >>>> >>>> >>>>>>>>> > +.005% cpu >>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 >>>>>>>> >>>>>>>> >>>> sec, -.001%, >>>> >>>> >>>>>>>>> > +.012% cpu >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, >>>>>>>>> -.004% cpu >>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -.005%, -.005% cpu >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -.008%, -.003% cpu >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -.040% cpu >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 >>>>>>>> >>>>>>>> >>>> sec, -.034%, >>>> >>>> >>>>>>>>> > -.031% cpu >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, >>>>>>>>> > -.09% i/o-8 >>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% >>>>>>>>> > -.14% i/o-8 >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, >>>>>>>>> > -.022% i/o-8 >>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -.017%, -.018% i/o-8 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, >>>>>>>>> > -.029% i/o-8 >>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 >>>>>>>> >>>>>>>> >>>> sec, -.023%, >>>> >>>> >>>>>>>>> > -.031% i/o-8 >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -.04% i/o-32 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, >>>>>>>>> > -.005% i/o-32 >>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -.11% i/o-32 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, >>>>>>>>> > .003% i/o-4/32 >>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, >>>>>>>>> > .01% i/o-4/32 >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>Overhead measurements: >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>Progress in terms of number of passes >>>>>>>> >>>>>>>> >>>> through a fixed >>>> >>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>system workload >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>>> > >>>>>>>The workload was usex -b48. >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>Conclusions: >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>>>>>>> > requirement for ntp >>>>>>>>> > >>>>>>>tracking under the loads >>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> accuracies for >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>SYNC, MIXED, >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>and ASYNC >>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC method by only >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>scheduling the extra >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>wakeups if a certain number >>>>>>>>> > >>>>>>>of ticks are missed. >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>Regards, >>>>>>>>> > >>>>>>>Dave >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>Keir Fraser wrote: >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>> >>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>> wrote: >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ASYNC method a >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>couple of days ago, >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have >>>>>>>>> > been something >>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of >>>>>>>> >>>>>>>> >>>> days ago for >>>> >>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>ASYNC. It may have been >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> after context >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>switch in. >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>acceptable accuracy, >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>each running consistently around or under >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> .01%. MIXED has >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>a fairly high >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>error of >>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>threshold for comfort. >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> plan to leave >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>SYNC running >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can >>>>>>>> >>>>>>>> >>>> leave MIXED >>>> >>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>running instead. >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I can run >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>more overnight tests >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>>next week. >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>> >>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> effects of the >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>SYNC+run_timer >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> cause higher >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>system-wide CPU >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>contention. I find it easier to think through the >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>implications of ASYNC. I'm >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> accurate than >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>ASYNC. Perhaps it >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> approaches, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>and each interrupt >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>event causes a small accumulated error? >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> favourites and >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>if the latter is >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>actually more accurate then I can simply revert the >>>>>>>>> > changeset that >>>>>>>>> > >>>>>>>>implemented MIXED. >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> workloads you >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>could try idle >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> large disc reads >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>to /dev/null)? We >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> CPU bound, so >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>that's really an >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>>>>obvious place to put any further effort imo. >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>>-- Keir >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>>> >>>>>>>>> > >>>>>>_______________________________________________ >>>>>>>>> > >>>>>>Xen-devel mailing list >>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com >>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>>> >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> > >>> >>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 >>>>>>>>> >>>>>>>> >>>>>>>> >>>> 2007 +0000 >>>> >>>> >>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 >>>>>>>>> >>>>>>>> >>>>>>>> >>>> 2008 -0500 >>>> >>>> >>>>>>>>> > >>@@ -58,7 +58,7 @@ static void >>>>>>>> >>>>>>>> >>>> pt_process_missed_ticks(stru >>>> >>>> >>>>>>>>> > >> >>>>>>>>> > >> missed_ticks = missed_ticks / (s_time_t) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> pt->period + 1; >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >> if ( mode_is(pt->vcpu->domain, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> no_missed_ticks_pending) ) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>>>>>> > >>+ pt->do_not_freeze = 1; >>>>>>>>> > >> else >>>>>>>>> > >> pt->pending_intr_nr += missed_ticks; >>>>>>>>> > >> pt->scheduled += missed_ticks * pt->period; >>>>>>>>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >>>>>>>>> > >> >>>>>>>>> > >> pt_lock(pt); >>>>>>>>> > >> >>>>>>>>> > >>- pt->pending_intr_nr++; >>>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> no_missed_ticks_pending) ) { >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >>+ pt->pending_intr_nr = 1; >>>>>>>>> > >>+ pt->do_not_freeze = 0; >>>>>>>>> > >>+ } >>>>>>>>> > >>+ else >>>>>>>>> > >>+ pt->pending_intr_nr++; >>>>>>>>> > >> >>>>>>>>> > >> if ( !pt->one_shot ) >>>>>>>>> > >> { >>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct >>>>>>>> >>>>>>>> >>>> vcpu *v, struct >>>> >>>> >>>>>>>>> > >> return; >>>>>>>>> > >> } >>>>>>>>> > >> >>>>>>>>> > >>- pt->do_not_freeze = 0; >>>>>>>>> > >>- >>>>>>>>> > >> if ( pt->one_shot ) >>>>>>>>> > >> { >>>>>>>>> > >> pt->enabled = 0; >>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *v, struct >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> > >> pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>>> > >> pt->pending_intr_nr = 0; /* 'collapse' all >>>>>>>>> > missed ticks */ >>>>>>>>> > >> } >>>>>>>>> > >>+ else if ( mode_is(v->domain, >>>>>>>> >>>>>>>> >>>> no_missed_ticks_pending) ) { >>>> >>>> >>>>>>>>> > >>+ pt->pending_intr_nr--; >>>>>>>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>>> > >>+ } >>>>>>>>> > >> else >>>>>>>>> > >> { >>>>>>>>> > >> pt->last_plt_gtime += pt->period_cycles; >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > _______________________________________________ >>>>>>>>> > Xen-devel mailing list >>>>>>>>> > Xen-devel@lists.xensource.com >>>>>>>>> > http://lists.xensource.com/xen-devel >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >> > --------------070202080006050007040301 Content-Type: image/png; name="rh4u5.64.2.11.png" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename="rh4u5.64.2.11.png" iVBORw0KGgoAAAANSUhEUgAAAoAAAAHgCAMAAAACDyzWAAAAA3NCSVQICAjb4U/gAAABKVBM VEX///8AAACgoKD/AAAAwAAAgP/AAP8A7u7AQADu7gAgIMD/wCAAgECggP+AQAD/gP8AwGAA wMAAYIDAYIAAgABA/4AwYICAYABAQEBAgAAAAICAYBCAYGCAYIAAAMAAAP8AYADjsMBAwIBg oMBgwABgwKCAAACAAIBgIIBgYGAgICAgQEAgQIBggCBggGBggICAgEAggCCAgICgoKCg0ODA ICAAgIDAYACAwODAYMDAgADAgGD/QAD/QECAwP//gGD/gIDAoADAwMDA/8D/AAD/AP//gKDA wKD/YGAA/wD/gAD/oACA4OCg4OCg/yDAAADAAMCgICCgIP+AIACAICCAQCCAQICAYMCAYP+A gADAwAD/gED/oED/oGD/oHD/wMD//wD//4D//8BUJrxzAAAAR3RFWHRTb2Z0d2FyZQBnbnVw bG90IHZlcnNpb24gNC4wIHBhdGNobGV2ZWwgMCBvbiBMaW51eCAyLjYuMTUtMS4xODMwX0ZD NHNtcPL2wVMAAA2fSURBVHic7d2Llpu4EkBRWP7/f87qmId42EalR6mqzr53Vk8ygSadEwlh 2kwTAAAAAAAAAABALPM8Jx+Avub3P8sHQME7QIZAaHjPvfP//+0/BZR53N/aXDIFlw6FxUMp BxDnAOb9AwFyAP0PYB0u00Ezzu+eAxjyAGL/7jkA9QOI/bvnANQPgOsxKEWA/s3pH9TtGfz5 esj6g+PPHf64D79ilqdAgP7dBzjfVnn44fmXHJI9fPiSwuvXwRUgQBPmpZ45+bD95PLi1l2O l6zm678TIB6a9xdSt/+n41oyNy8X3J5PwZ+9/nw9rGdH32Bj9HV8JWE567sP8PBLpvN/vuzu O0ZA/Hd5KWvrLBkM5+svSX5y/xEBItdtgH89vctLWttn2zTA7ZWw9Jdk3Ezw/bAUNgYmAoQy AoQqAoQqAoQqAoQqAoQqAoQqAoQqAoQqAoQqAoQqAoQqAkSh7zec/kKAKPLrjudfCBByr/ct 9yUJEiDEXhv5PggQUhXyI0BIvWqMfwQIqRr1TQQImSqj3x8ChESd+XciQIhUOgGcsho6fc9o 3sbwZGuveAbOaOj8XfNZG8OTfezrGeBEgPiv0vLjrWwKznrQA3yo1l9uPoyAmCqPf0zByFS5 v6xFyOX9kAgwnMr5cR0QWWqPfwSIHNVegNsRIJ6q9upHigDxUJP+CBAPVXz5LUWAeKTB4Pcf AeKJVv0RIJ5olR8B4ol2/REgHmjXHwHit4YDIAHip2YLkD8EiB9aXH7eESC+a9sfAeK7Ni/A 7QgQ3yztNRsACRDfVPz2tw8IEJ81Pft7I0B81KE/AsRHPfojQHzSpT8CxAdtL/9tCBC3OvVH gLjT+vLzjgBx1S0/AsTV69VvACRAnO3tdRgACRAnydhHgOit38nfggCR6t0fASLRPT8CRIoA oan7CeBEgEj0z0/wLvkTb9HrlMb4l/+gmsMGBOiIRn2T7F3yJRtjcCrD3yQIkAfVeKQz+/Kg Giy0xj8eVINJMz8eVINJbf3xH9cBodkfAUK1PwIMT/MEcCLA0Dreef8RAcaVfu8HIyB6S+Jj BERvrxOt4yDAiM7pMQKin5thjwDRzwDzboIAgxkrPwIMZ4SFR4oAQ9kvvIxSIAHGkWY3Sn8E GMhx5h2jPwIM43zmN0Z/BBjFYGuPDQHGMGh+BBhEsvrVPpQTAoxgzMHvPwIMYOD+CDCAgfMj wACG7o8A3Rt5/p0I0Llhr75sCNCv9MYXAkRvSXzD5keAbh1GPwJEb+NPvm8E6NBh8CNAdGZk 7HsjQHcMLH0TBOiNqfwkzwlJNiHA0byG+663X7KfE8KDakb2Su780z6WZ7JHwJkRcFiHkc9p gPNhCuY5IYOxM/VO0ueEpNvQ3mAMnfutWIQ4YrC//OeEEOCwLPbHdUBHTF1+WRGgG0t7pvIj QDe2sY8AocHc3LsgQBfsnfutCNADg4uPFQE6YLg/ArTP4sWXHQEaZ/LiX4IALUtv/jNaIAEa Zj8/AjQsrY8A0Z390e8PARrlIb4/BGiTl/4I0CYv+RGgTW7GPwI0ycfy440A7Vnbc5AfARpk 9dbTewRojZe5d0GA1vjqjwCtcdYfARrjZvW7IkBL3OVHgJY4uvq3I0A7HNz9d0WAVji49+8O AVqxX38mwCobI4e7qXdFgCa47Y8ATfDbHwFa4Lg/wbvk85iGzvxdeknlPidkTrchwPZ85yd6 k3IC7MjhpeejsimY54Q0lrz44S9B2XNCLj+Ddvb2/OW3IMCR7WMfAS5zL09K6sfr1HvAdcBB vWL0R4CD8rv0OCHAITle+54Q4IjC5EeAQ4oz/hHggOLE94cAh7Peea99HH0Q4GBijX8EOJpo /RHgWAKtPhYEOJJw+RHgSCJdfdkQ4DC2+iL1R4DDSO89DZQgAY4hLS9QfgQ4iMPIR4BdNsYq 4sy7IUBtofMjQHUxlx47AtQVdO27I0BVe3sx8yNARYehjxGw+8bRnWfekPkRoJrLmR8Bdt44 tNgLjxQBagi+8k0RoALa2xFgf/SXIMDu6C9FgL1x8ndAgJ2x+jgiwL7I74QAu9pe+dU+kGEQ YE8Mfxc8qKYj+rviQTX90N8Nwbvk8yblMvR3Jz/AORkBeVDNY+R3IXtQzcwULMHlvw8kD6oh wFzc/PJR7oNqWAXnS26+IsAzrgO2lsRHflcE2Fg69hHgFQE2xdz7CwG2RH8/EWBD1PcbATbD 6PcEATbCpZdnCLAJ8nuKABtI6yPA7wiwPka/DARYG/llIcDKyC8PAdZFfpkIsCryy0WA9TD7 ChBgNfQnQYCVsPiVIcBKeO1DhgCr2OMjvzwEWEMy9BFgHgIsxsxbggBLcepXhADLsPQoRIBF 6K8UARZILr0QoBAByiXXXkhQigDFDmMf+QkRoNRx6iVAIQKU4dyvEgKUIL9qCDAfa9+KCDDb 60D7aKwjwEyH9MivWHZDod8h9Tz0EWCx3IYCv0f0ZeJlCq6AAJ+6O/Ejv2JlU3Cg54TcLjwI sIQon6gjICvfRgjwES69tMIq+Im0O/KriuuAv3HbQUME+AMTb1sE+AMBtkWAX5FfawT4DUvf 5gjwsyQ+8muFAD85DH4E2AoB3mPm7YQA73Dm1w0B3mHt0Q0BXrH27YgArwiwIwK82Msjv/YI 8Ogw8hFgewSYYurtjgATnPv1R4Arlh4qCHCVvPBLgP0Q4H8MfloIkG94UxU+wBf9qQoc4OuG 9jHFEzdA+htC1ADv8iNABQTIN/yqihng7bhHgBqCB8hFZ20hA0wGPgJUFjDAZOJl5aEuXoCc +A0lWoDnpQcBKosV4HntyxSsLlSANxdfyE9ZpACT9nif+1HkN2T2LXoPc6/2wWCR/x7RJRsr 4vXeMUneJV+8sR7yG1V2gCYfVMP9LiOK86Aa7rgaV4AAufNgZP4fVJOOfQQ4HO/XAZl6B+c6 QM78xhciQO3jwGeOA2Tpa0GIALWPBJ/5DZAB0AS3AXLx2QavAXLjixEECFU+A2TmNcNlgJz6 2eEwQJYelvgLkP5McRcg115s8RbgVh8B2uAswP3WU8ZAG3wFmEy+5GeDqwDTkz8CtMFRgKw+ LPISILceGOUkQPKzykeAe34EaIyLABn+7PIQIP0Z5iZA7YOAjP0AGf1MI0CoMh8g+dlmPEDW H9bZDpDrz+aZDpD87DMc4F4fAdplNkBmXx8EDY3wDqnk54XNB9WQnxuCB9WoB8jw54jgXfK1 nxNCfU7InhOSbqMyAnLvsyvmFiHceuqLtQC3/BgBfTB2HTCZesnPBVMBHk79CNAFQwGy9vDI ToD055KVAJOLLxToib0A+31OdGAjwCS+PUBS9MBEgPdjHwF6YCDA+7mX6diH0QP8cu5Hfh6M HeDXtS8BejB0gF8vvdCfC+MG+Dq5/PeWnxy9DBtgmh5rEL+GDPA88jEC+jVigDfzLueAXg0Y 4N2JH7V5NVyA6alfg91jMKMFmA5+BBjAWAF+vfAHj0YKkPwCGifAry+7watRAvz6ogf8GiHA Y3z0F4p+gJeXPWrsFFaoB8jcG5t2gPQXnHKA9BedboBJefQXk2qA6chHfzFpBsjEC80AOfOD YoDkhz/575Jf6TENrH3xR/Au+fs28gC3193Ee4APkobKA9wGPwKMrmwKlj0nhLkXf2TPCfn4 g6c490Oid4CsPXDQ+UlJ9IejrtcBue8AZ0oBUiDeOga4x0d+WPULcM+PERCbbgFy6xXu9Arw RYC40ydA1h74oEuALH7xSY8AyQ8fdQiQW1/wWfsAGf3wResAyQ9fNQ2Qkz/80jBA1r74rVmA yY0HBIiPWgX4IkA80ShA8sMz9QN8HQc/AsQ3LQMs2TWCqB0gUy+yVA6Qcz/kqRsg+SFT1QDT tUfJfhFHgwBL9ohoKgbI5It81QKkPkgQIFRVCpDlL2SqB1h8RAilToC88AuhKgFuYx/9IVON AJl7IVYeIOd+KFAYIIsPlCl7l3yuvqBQ2YNq6A+FBAFuQ+CLAFEqP8B5OgfI1RdIyB5UM5+m 4L+PBAipwnNA2kMZtce1An8Un5gOECCUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBU ESBUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBUESBU ESBUqQZYHDAHEPsAYv/uOQCVA6j5Dqn2fvccgPYBHN8juvMnr74DDsDkARAgB9D/ANZnOhym YKBUq2CBFggQqiSjJgAAAADUIl2RrGuZZfvs3czpZxfsZN1AehC3nzljB9snNnsA+9dQdgDL i2oVOhAVOB+2z97NcriHrXN2sm8vO4jbz5yxg2QLowewfA3lfwT79tIDWDaUDYFr+NP6BOzM 3awBJVtn7WQbAeUHcfnMmTuYbR/APF23zNnBvI2AwgPY9iIKcFo+63v7/N3MyWeX7GRO/kW0 /fJ1kx7APtdIvwqFBzAVHsA2XgkPYB9+yzpI9pPt8rcnb9u7v3uyAPO3X2cO6QHMh38VfhVu R5+8EVB+AMvJm/wrsL+gUdqBKMB13D18yNy+ZCeF2+9fOtkOSre/3zLzAEq2T/Yh3kHp9u+d SF+XK1g9fdo6ew1YtP3N4u35Dkq31/8KTOtf4rJVcGEHAAAAAAAAAMz6By1fNwzf7b/AAAAA AElFTkSuQmCC --------------070202080006050007040301 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------070202080006050007040301-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 14 Feb 2008 10:59:43 -0500 Message-ID: <47B4656F.40306@virtualiron.com> References: <20080201153111687.00000002384@djm-pc> <47A77075.3010707@virtualiron.com> <47ACC7CC.5030103@virtualiron.com> <47B07D66.7030307@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47B07D66.7030307@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: Dave Winchell , Deepak Patel , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Hi Dan, I ran the ltp tests with 3.2 and found the errors for a 16 hour run to be: rh4u564 -9.9 sec (-.017%) rh4u464 -7.3 sec (-.013%) There were no cliffs and the drift was linear. I think the problem you had may be due to the use of the pm timer. If you still have the boot log, it would tell you. When I first tried a guest on 3.2 with "clocksource=pit nohpet" I noticed that it picked the pm timer. Adding "nopmtimer", the guest will pick the pit. The reason I didn't have the problem with our 3.1 base is that I had disabled the hpet and the pmtimer by not advertising them in the acpi tables. I did this so long ago, I forgot that I had to disable pmtimer as well as hpet. So, can you re-run your test with "clocksource=pit nohpet nopmtimer"? You should see this in the boot messages: time.c: Using PIT/TSC based timekeeping. Thanks, Dave Dave Winchell wrote: > Hi Dan, > > Over the weekend the drift was +18 seconds for each guest (no ntp). > The duration was 3900 minutes, so the error for each was +.0077%. > Looking back through the data, it appears to drift linearly at > this rate. I've attached a plot for rh4u5-64. > > This accuracy is better than what I've seen before (.03-.05%). > This may be due to the different load (ltp vs usex) or to one of the > changes I've made recently. I'll do some experimentation to see if > there is > a fix I should propose. > > This still doesn't address the radical drift you saw. > The next step for me is to run 3.2 and see if I can reproduce it. > > Regards, > Dave > > > > > > Dave Winchell wrote: > >> Hi Dan, >> >> Sorry it took me so long, but I finally ran an ltp test today. >> Its on rh4u4-64. I'm using the defaults for ltp and using a script >> called runltp. I had a usex load on rh4u5-64. No ntpd. >> virtual processors / physical processors = 2. >> >> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes >> for -.005% and .008%. >> >> I'm running a 3.1 based hypervisor with some clock related tweaks that >> I haven't submitted, because I'm still characterizing them. >> >> I'm stopping the usex load on 4u5-64 now and replacing it with ltp >> and will leave the two guests running ltp over the weekend. >> >> Regards, >> Dave >> >> >> Dave Winchell wrote: >> >>> Hi Dan, Deepak: >>> >>> Thanks for the data. Those drifts are severe - no wonder ntp couldn't >>> keep then in synch. I'll try to reproduce that behaviour here, with >>> my code base. >>> If I can't reproduce it, I'll try 3.2. >>> >>> If you can isolate what ltp is doing during the cliffs, that would >>> be very >>> helpful. >>> >>> thanks, >>> Dave >>> >>> >>> >>> >>> Dan Magenheimer wrote: >>> >>>> OK, Deepak repeated the test without ntpd and using ntpdate -b before >>>> the test. >>>> >>>> The attached graph shows his results: el5u1-64 (best=~0.07%), >>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%). >>>> >>>> We will continue to look at LTP to try to isolate. >>>> >>>> Thanks, >>>> Dan >>>> >>>> P.S. elXuY is essentially RHEL XuY with some patches. >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>> Sent: Wednesday, January 30, 2008 2:45 PM >>>>> To: Deepak Patel >>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser; >>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; Dave Winchell >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>>> pending >>>>> missed ticks >>>>> >>>>> >>>>> Dan, Deeepak, >>>>> >>>>> It may be that the underlying clock error is too great for ntp >>>>> to handle. It would be useful if you did not run ntpd >>>>> and, instead did ntpdate -b at the start of the test >>>>> for each guest. Then capture the data as you have been doing. >>>>> If the drift is greater than .05%, then we need to address that. >>>>> >>>>> Another option is, when running ntpd, to enable loop statistics in >>>>> /etc/ntp.conf >>>>> by adding this to the file: >>>>> >>>>> statistics loopstats >>>>> statsdir /var/lib/ntp/ >>>>> >>>>> Then you will see loop data in that directory. >>>>> Correlating the data in the loopstats files with the >>>>> peaks in skew would be interesting. You will see entries of the form >>>>> >>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 239.735511 10 >>>>> >>>>> Where the second to last column is the Allan Deviation. When that >>>>> gets over 1000, ntpd is working pretty hard. However, I have not >>>>> seen ntpd >>>>> completely loose it like you have. >>>>> >>>>> I'm on vacation until Monday, and won't be reading >>>>> email. >>>>> >>>>> Thanks for all your work on this! >>>>> >>>>> -Dave >>>>> >>>>> Deepak Patel wrote: >>>>> >>>>> >>>>> >>>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.) >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> I do not know which graph was attached with this. But I saw this >>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm guests when I >>>>>> was running ltp tests continuously. >>>>>> >>>>>> >>>>>> >>>>>>> What was the behaviour of the other guests running? >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> All pvm guests are fine. But behavior of most of the hvm guests were >>>>>> as described. >>>>>> >>>>>> >>>>>> >>>>>>> If they had spikes, were they at the same wall time? >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> No. They are not at the same wall time. >>>>>> >>>>>> >>>>>> >>>>>>> Were the other guests running ltp as well? >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp >>>>>> continuously. >>>>>> >>>>>> >>>>>> >>>>>>> How are you measuring skew? >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> I was collecting output of "ntpdate -q every >>>>> >>>>> >>>>> >>>>> 300 seconds >>>>> >>>>> >>>>>> (5 minutes) and have created graph based on that. >>>>>> >>>>>> >>>>>> >>>>>>> Are you running ntpd? >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Yes. ntp was running on all the guests. >>>>>> >>>>>> I am investigating what causes this spikes and let everyone >>>>> >>>>> >>>>> >>>>> know what >>>>> >>>>> >>>>>> are my findings. >>>>>> >>>>>> Thanks, >>>>>> Deepak >>>>>> >>>>>> >>>>>> >>>>>>> Anything that you can discover that would be in sync with >>>>>>> the spikes would be very helpful! >>>>>>> >>>>>>> The code that I test with is our product code, which is based >>>>>>> on 3.1. So it is possible that something in 3.2 other than vpt.c >>>>>>> is the cause. I can test with 3.2, if necessary. >>>>>>> >>>>>>> thanks, >>>>>>> Dave >>>>>>> >>>>>>> >>>>>>> >>>>>>> Dan Magenheimer wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Hi Dave (Keir, see suggestion below) -- >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> Turning off vhpet certainly helps a lot (though see below). >>>>>>>> >>>>>>>> I wonder if timekeeping with vhpet is so bad that it should be >>>>>>>> turned off by default (in 3.1, 3.2, and unstable) until it is >>>>>>>> fixed? (I have a patch that defaults it off, can post it if >>>>>>>> there is agreement on the above point.) The whole point of an >>>>>>>> HPET is to provide more precise timekeeping and if vhpet is >>>>>>>> worse than vpit, it can only confuse users. Comments? >>>>>>>> >>>>>>>> >>>>>>>> In your testing, are you just measuring % skew over a long >>>>>>>> period of time? >>>>>>>> We are graphing the skew continuously and >>>>>>>> seeing periodic behavior that is unsettling, even with pit. >>>>>>>> See attached. Though your algorithm recovers, the "cliffs" >>>>>>>> could still cause real user problems. I wonder if there is >>>>>>>> anything that can be done to make the "recovery" more >>>>>>>> responsive? >>>>>>>> >>>>>>>> We are looking into what part(s) of LTP is causing the cliffs. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dan >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM >>>>>>>>> To: dan.magenheimer@oracle.com >>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>>>>>>> deepak.patel@oracle.com; >>>>>>>>> akira.ijuin@oracle.com; Dave Winchell >>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>>>>>>> pending >>>>>>>>> missed ticks >>>>>>>>> >>>>>>>>> >>>>>>>>> Dan, >>>>>>>>> >>>>>>>>> I guess I'm a bit out of date calling for clock= usage. >>>>>>>>> Looking at linux 2.6.20.4 sources, I think you should specify >>>>>>>>> "clocksource=pit nohpet" on the linux guest bootline. >>>>>>>>> >>>>>>>>> You can leave the xen and dom0 bootlines as they are. >>>>>>>>> The xen and guest clocksources do not need to be the same. >>>>>>>>> In my tests, xen is using the hpet for its timekeeping and >>>>>>>>> that appears to be the default. >>>>>>>>> >>>>>>>>> When you boot the guests you should see >>>>>>>>> time.c: Using PIT/TSC based timekeeping. >>>>>>>>> on the rh4u5-64 guest, and something similar on the others. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>>>> 14.318MHz HPET.) >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> This appears to be the xen state, which is fine. >>>>>>>>> I was wrongly assuming that this was the guest state. >>>>>>>>> You might want to look in your guest logs and see what they were >>>>>>>>> picking >>>>>>>>> for a clock source. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Dave >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Dan Magenheimer wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, I hadn't realized that! No wonder we didn't >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> see the same >>>>> >>>>> >>>>>>>>>> improvement you saw! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Try specifying clock=pit on the linux boot line... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> command line or >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the guest (or >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> dom0?) command >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> line? Or both places? Since the tests take awhile, it >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> would be nice >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> to get this right the first time. Do the Xen and guest >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> clocksources need >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> to be the same? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Dan >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM >>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser >>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> that disables >>>>> >>>>> >>>>>>>>>> pending missed ticks >>>>>>>>>> >>>>>>>>>> Hi Dan, >>>>>>>>>> >>>>>>>>>> Hpet timer does have a fairly large error, as I >>>>>>>>>> was >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> trying this >>>>> >>>>> >>>>>>>>>> one recently. >>>>>>>>>> I don't remember what I got for error, but 1% sounds >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> about right. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> The problem is that hpet is not built on top of vpt.c, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> the module >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Keir and I did >>>>>>>>>> all the recent work in, for its periodic timer needs. Try >>>>>>>>>> specifying clock=pit >>>>>>>>>> on the linux boot line. If it still picks the hpet, which it >>>>>>>>>> might, let me know >>>>>>>>>> and I'll tell you how to get around this. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Dave >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> ---------- >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> *From:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] >>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM >>>>>>>>>> *To:* Dave Winchell; Keir Fraser >>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; deepak.patel@oracle.com; >>>>>>>>>> akira.ijuin@oracle.com >>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> that disables >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> pending missed ticks >>>>>>>>>> >>>>>>>>>> Sorry for the very late followup on this but we finally >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> were able >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> to get our testing set up again on stable 3.1 bits and have >>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the order of 1%. >>>>>>>>>> >>>>>>>>>> Test enviroment was a 4-socket dual core machine with 24GB of >>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> plus two pv. >>>>> >>>>> >>>>>>>>>> All six guests were running LTP simultaneously. The four hvm >>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> RHEL4u5-64. >>>>> >>>>> >>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 32-bit guests. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> All four hvm guests experienced skew around -1%, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> even the 32-bit >>>>> >>>>> >>>>>>>>>> guest. Less intensive testing didn't exhibit much >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> skew at all. >>>>> >>>>> >>>>>>>>>> A representative graph is attached. >>>>>>>>>> >>>>>>>>>> Dave, I wonder if some portion of your patches >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> didn't end up in >>>>> >>>>> >>>>>>>>>> the xen trees? >>>>>>>>>> >>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>>>> 14.318MHz HPET.) >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Dan >>>>>>>>>> >>>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests. >>>>>>>>>> >>>>>>>>>> > -----Original Message----- >>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com >>>>>>>>>> > [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>>>> > Dave Winchell >>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>>>>>>>> > To: Keir Fraser >>>>>>>>>> > Cc: dan.magenheimer@oracle.com; >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> xen-devel@lists.xensource.com; Dave >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > Winchell >>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>>>> > disables pending >>>>>>>>>> > missed ticks >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Hi Keir, >>>>>>>>>> > >>>>>>>>>> > The latest change, c/s 16690, looks fine. >>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to >>>>>>>>>> > the code I submitted. Also, your version is more >>>>>>>>>> > concise. >>>>>>>>>> > >>>>>>>>>> > The error tests confirm the equivalence. With >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> overnight cpu loads, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > the checked in version was accurate to +.048% for sles >>>>>>>>>> > and +.038% for red hat. My version was +.046% >>>>>>>>>> and >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> +.032% in a >>>>> >>>>> >>>>>>>>>> > 2 hour test. >>>>>>>>>> > I don't think the difference is significant. >>>>>>>>>> > >>>>>>>>>> > i/o loads produced errors of +.01%. >>>>>>>>>> > >>>>>>>>>> > Thanks for all your efforts on this issue. >>>>>>>>>> > >>>>>>>>>> > Regards, >>>>>>>>>> > Dave >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Keir Fraser wrote: >>>>>>>>>> > >>>>>>>>>> > >Applied as c/s 16690, although the checked-in patch is >>>>>>>>>> > smaller. I think the >>>>>>>>>> > >only important fix is to pt_intr_post() and the >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> only bit of >>>>> >>>>> >>>>>>>>>> > the patch I >>>>>>>>>> > >totally omitted was the change to >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> pt_process_missed_ticks(). >>>>> >>>>> >>>>>>>>>> > I don't think >>>>>>>>>> > >that change can be important, but let's see what >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> happens to the >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> error >>>>>>>>>> > >percentage... >>>>>>>>>> > > >>>>>>>>>> > > -- Keir >>>>>>>>>> > > >>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > >>Hi Dan and Keir, >>>>>>>>>> > >> >>>>>>>>>> > >>Attached is a patch that fixes some issues with the >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> SYNC policy >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>(no_missed_ticks_pending). >>>>>>>>>> > >>I have not tried to make the change the minimal one, but, >>>>>>>>>> > rather, just >>>>>>>>>> > >>ported into >>>>>>>>>> > >>the new code what I know to work well. The error for >>>>>>>>>> > >>no_missed_ticks_pending goes from >>>>>>>>>> > >>over 3% to .03% with this change according to my testing. >>>>>>>>>> > >> >>>>>>>>>> > >>Regards, >>>>>>>>>> > >>Dave >>>>>>>>>> > >> >>>>>>>>>> > >>Dan Magenheimer wrote: >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> > >>>Hi Dave -- >>>>>>>>>> > >>> >>>>>>>>>> > >>>Did you get your correction ported? If so, it would be >>>>>>>>>> > nice to see this get >>>>>>>>>> > >>>into 3.1.3. >>>>>>>>>> > >>> >>>>>>>>>> > >>>Note that I just did some very limited testing with >>>>>>>>>> > timer_mode=2(=SYNC=no >>>>>>>>>> > >>>missed ticks pending) >>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> guest) and the >>>>> >>>>> >>>>>>>>>> > worst error I've >>>>>>>>>> > >>>seen so far >>>>>>>>>> > >>>is 0.012%. But I haven't tried any exotic >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> loads, just LTP. >>>>> >>>>> >>>>>>>>>> > >>> >>>>>>>>>> > >>>Thanks, >>>>>>>>>> > >>>Dan >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>>>-----Original Message----- >>>>>>>>>> > >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com >>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>>>>>>>> > xen-devel@lists.xensource.com; Dong, >>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>>>> > >>>>disables pending >>>>>>>>>> > >>>>missed ticks >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>Dan, >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>I did some testing with the constant tsc offset >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> SYNC method >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>(now called >>>>>>>>>> > >>>>no_missed_ticks_pending) >>>>>>>>>> > >>>>and found the error to be very high, much larger >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> than 1 %, as >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>I recall. >>>>>>>>>> > >>>>I have not had a chance to submit a correction. I >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> will try to >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>do it later >>>>>>>>>> > >>>>this week or the first week in January. My version of >>>>>>>>>> constant tsc >>>>>>>>>> > >>>>offset SYNC method >>>>>>>>>> > >>>>produces .02 % error, so I just need to port >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> that into the >>>>> >>>>> >>>>>>>>>> > >>>>current code. >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>The error you got for both of those kernels is >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> what I would >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> expect >>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode. >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>Regards, >>>>>>>>>> > >>>>Dave >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>Dan Magenheimer wrote: >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>>Anyone make measurements on the final patch? >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> saw a loss of >>>>> >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>about 0.2% with no load. This was >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> xen-unstable tip today >>>>> >>>>> >>>>>>>>>> > >>>>with no options specified. 32-bit was about 0.01%. >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>>I think I missed something... how do I run the various >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>accounting choices and which ones are known to be >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> appropriate >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>for which kernels? >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>>Thanks, >>>>>>>>>> > >>>>>Dan >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>>>-----Original Message----- >>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>Keir Fraser >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>>>>> > >>>>>>To: Dave Winchell >>>>>>>>>> > >>>>>>Cc: Shan, Haitao; >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> xen-devel@lists.xensource.com; Dong, >>>>> >>>>> >>>>>>>>>> > Eddie; Jiang, >>>>>>>>>> > >>>>>>Yunhong >>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> mode that >>>>> >>>>> >>>>>>>>>> > >>>>>>disables pending >>>>>>>>>> > >>>>>>missed ticks >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>Please take a look at xen-unstable changeset 16545. >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>-- Keir >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>> wrote: >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>Keir, >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> loads for the >>>>> >>>>> >>>>>>>>>> > >>>>>>>various time protocols follows. In >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> addition, the data >>>>> >>>>> >>>>>>>>>> > >>>>>>>for cpu loads is shown. >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> processor AMD >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> box. >>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 vcpu each. >>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>>>> > >>>>>>>(usex is available at >>>>>>>>>> http://people.redhat.com/anderson/usex.) >>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> of=/dev/null. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>>>> > >>>>>>>All three guests are 8vcpu. >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same as i/o-32 >>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> instances of dd. >>>>> >>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, >>>>>>>>>> +4.42 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> sec -.006%, >>>>> >>>>> >>>>>>>>>> > +.005% cpu >>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> sec, -.001%, >>>>> >>>>> >>>>>>>>>> > +.012% cpu >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 sec, -.009%, >>>>>>>>>> -.004% cpu >>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -.005%, -.005% cpu >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -.008%, -.003% cpu >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -.040% cpu >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> sec, -.034%, >>>>> >>>>> >>>>>>>>>> > -.031% cpu >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 sec,-55.7 sec, -.01%, >>>>>>>>>> > -.09% i/o-8 >>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 sec,-14.0 sec, -.015% >>>>>>>>>> > -.14% i/o-8 >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 sec, -.017%, >>>>>>>>>> > -.022% i/o-8 >>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -.017%, -.018% i/o-8 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 sec, -.020%, >>>>>>>>>> > -.029% i/o-8 >>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> sec, -.023%, >>>>> >>>>> >>>>>>>>>> > -.031% i/o-8 >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -.04% i/o-32 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 sec, -.011%, >>>>>>>>>> > -.005% i/o-32 >>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -.11% i/o-32 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, 13. sec -.07%, >>>>>>>>>> > .003% i/o-4/32 >>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 sec, -.017%, >>>>>>>>>> > .01% i/o-4/32 >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>Overhead measurements: >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>Progress in terms of number of passes >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> through a fixed >>>>> >>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>system workload >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>>>> > >>>>>>>The workload was usex -b48. >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>Conclusions: >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>>>>>>>> > requirement for ntp >>>>>>>>>> > >>>>>>>tracking under the loads >>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> accuracies for >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>SYNC, MIXED, >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>and ASYNC >>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC method by only >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>scheduling the extra >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>wakeups if a certain number >>>>>>>>>> > >>>>>>>of ticks are missed. >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>Regards, >>>>>>>>>> > >>>>>>>Dave >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>Keir Fraser wrote: >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>> >>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>> wrote: >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ASYNC method a >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>couple of days ago, >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think there may have >>>>>>>>>> > been something >>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> days ago for >>>>> >>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>ASYNC. It may have been >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> after context >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>switch in. >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC or ASYNC give >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>acceptable accuracy, >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>each running consistently around or under >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> .01%. MIXED has >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>a fairly high >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>error of >>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close to .05% ntp >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>threshold for comfort. >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> plan to leave >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>SYNC running >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> leave MIXED >>>>> >>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>running instead. >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I can run >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>more overnight tests >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>>next week. >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>> >>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> effects of the >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>SYNC+run_timer >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> cause higher >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>system-wide CPU >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>contention. I find it easier to think through the >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>implications of ASYNC. I'm >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> accurate than >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>ASYNC. Perhaps it >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> approaches, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>and each interrupt >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>event causes a small accumulated error? >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> favourites and >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>if the latter is >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>actually more accurate then I can simply revert the >>>>>>>>>> > changeset that >>>>>>>>>> > >>>>>>>>implemented MIXED. >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> workloads you >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>could try idle >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> large disc reads >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>to /dev/null)? We >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> CPU bound, so >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>that's really an >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo. >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>>-- Keir >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>>> >>>>>>>>>> > >>>>>>_______________________________________________ >>>>>>>>>> > >>>>>>Xen-devel mailing list >>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com >>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>>> >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> 2007 +0000 >>>>> >>>>> >>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> 2008 -0500 >>>>> >>>>> >>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> pt_process_missed_ticks(stru >>>>> >>>>> >>>>>>>>>> > >> >>>>>>>>>> > >> missed_ticks = missed_ticks / (s_time_t) >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> pt->period + 1; >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >> if ( mode_is(pt->vcpu->domain, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> no_missed_ticks_pending) ) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>>>>>>> > >>+ pt->do_not_freeze = 1; >>>>>>>>>> > >> else >>>>>>>>>> > >> pt->pending_intr_nr += missed_ticks; >>>>>>>>>> > >> pt->scheduled += missed_ticks * pt->period; >>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void pt_timer_fn(void *data) >>>>>>>>>> > >> >>>>>>>>>> > >> pt_lock(pt); >>>>>>>>>> > >> >>>>>>>>>> > >>- pt->pending_intr_nr++; >>>>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> no_missed_ticks_pending) ) { >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >>+ pt->pending_intr_nr = 1; >>>>>>>>>> > >>+ pt->do_not_freeze = 0; >>>>>>>>>> > >>+ } >>>>>>>>>> > >>+ else >>>>>>>>>> > >>+ pt->pending_intr_nr++; >>>>>>>>>> > >> >>>>>>>>>> > >> if ( !pt->one_shot ) >>>>>>>>>> > >> { >>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> vcpu *v, struct >>>>> >>>>> >>>>>>>>>> > >> return; >>>>>>>>>> > >> } >>>>>>>>>> > >> >>>>>>>>>> > >>- pt->do_not_freeze = 0; >>>>>>>>>> > >>- >>>>>>>>>> > >> if ( pt->one_shot ) >>>>>>>>>> > >> { >>>>>>>>>> > >> pt->enabled = 0; >>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *v, struct >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> > >> pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>>>> > >> pt->pending_intr_nr = 0; /* 'collapse' all >>>>>>>>>> > missed ticks */ >>>>>>>>>> > >> } >>>>>>>>>> > >>+ else if ( mode_is(v->domain, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>> no_missed_ticks_pending) ) { >>>>> >>>>> >>>>>>>>>> > >>+ pt->pending_intr_nr--; >>>>>>>>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>>>> > >>+ } >>>>>>>>>> > >> else >>>>>>>>>> > >> { >>>>>>>>>> > >> pt->last_plt_gtime += pt->period_cycles; >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > _______________________________________________ >>>>>>>>>> > Xen-devel mailing list >>>>>>>>>> > Xen-devel@lists.xensource.com >>>>>>>>>> > http://lists.xensource.com/xen-devel >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>> >> > > > ------------------------------------------------------------------------ > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 14 Feb 2008 12:55:57 -0500 Message-ID: <47B480AD.4050308@virtualiron.com> References: <20080214092114625.00000001516@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080214092114625.00000001516@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , Deepak Patel , Dave Winchell List-Id: xen-devel@lists.xenproject.org Dan, Here are some boot snipets for rh4u564 on xen 3.2. #1: Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro root=LABEL=/ console=ttyS0 clocksource=pit nohpet) Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 ... Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/ console=ttyS0 clocksource=pit nohpet Feb 14 10:44:59 vs076 kernel: Initializing CPU#0 Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11, 65536 bytes) Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer. Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor. ... Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8 CPUs: passed. Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use of PM timer Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping. #2: Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer) Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 ... Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer Feb 14 10:47:58 vs076 kernel: Initializing CPU#0 Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11, 65536 bytes) Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer. Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor. ... Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8 CPUs: passed. Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping. As you can see, I only get the pit if I specify nopmtimer. Dan Magenheimer wrote: >Hi Dave -- > >Thanks for continuing to run tests! > >Hmmm... I thought I had noticed that even though Linux will acknowledge >the existence of the pmtimer, it still prints: > >time.c: Using PIT/TSC based timekeeping. > >I will check again, but assuming the clocksource for our tests is >indeed pit, the huge difference in the results (yours vs ours) is >baffling. I wonder if the difference may be the underlying hardware. >Maybe we will try to ensure we can duplicate the results on a different >box. > > >So your testing was with stock 3.2.0 xen bits (what cset?) without >any of your [quote from below] "clock related tweaks that I haven't >submitted, because I'm still characterizing them"? > > None of the tweaks I mentioned are in this test. It was stock with some patches. However, none of the patches are time related to my knowledge and I checked vpt.c to make sure that it is the same as what's in unstable. The only difference is in pt_intr_post, where I set the timer mode. I don't have timer mode tied into our config process yet, which is different than official xen method. (In pt_intr_post) else { + if(v->arch.paging.mode->guest_levels == 4) + v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = HVMPTM_no_missed_ticks_pending; + else + v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = HVMPTM_delay_for_missed_ticks; if ( mode_is(v->domain, one_missed_tick_pending) || mode_is(v->domain, no_missed_ticks_pending) ) { >Could you also send detail on the rhel4u4-64 kernel you >are testing with, just to ensure we are not comparing apples >and oranges? (Perhaps there's some way we can even share the >identical disk image and vm.cfg file?) > >And if our problem is indeed the pmtimer, I will need to submit >another patch to Keir to add an hvm pmtimer platform variable. >(Hmmm... I don't think he's even accepted the hpet variable patch >yet. I'll have to check.) > >Thanks, >Dan > > > > >>-----Original Message----- >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>Sent: Thursday, February 14, 2008 9:00 AM >>To: dan.magenheimer@oracle.com >>Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak >>Patel >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Hi Dan, >> >>I ran the ltp tests with 3.2 and found the errors >>for a 16 hour run to be: >> >>rh4u564 -9.9 sec (-.017%) >>rh4u464 -7.3 sec (-.013%) >> >>There were no cliffs and the drift was linear. >> >>I think the problem you had may be due to the use of the >>pm timer. If you still have the boot log, it would tell you. >> >>When I first tried a guest on 3.2 with "clocksource=pit nohpet" >>I noticed that it picked the pm timer. Adding "nopmtimer", the >>guest will pick the pit. >> >>The reason I didn't have the problem with our 3.1 base is that >>I had disabled the hpet and the pmtimer by not advertising them >>in the acpi tables. I did this so long ago, I forgot that I had to >>disable pmtimer as well as hpet. >> >>So, can you re-run your test with "clocksource=pit nohpet nopmtimer"? >>You should see this in the boot messages: >> >>time.c: Using PIT/TSC based timekeeping. >> >>Thanks, >>Dave >> >> >>Dave Winchell wrote: >> >> >> >>>Hi Dan, >>> >>>Over the weekend the drift was +18 seconds for each guest (no ntp). >>>The duration was 3900 minutes, so the error for each was +.0077%. >>>Looking back through the data, it appears to drift linearly at >>>this rate. I've attached a plot for rh4u5-64. >>> >>>This accuracy is better than what I've seen before (.03-.05%). >>>This may be due to the different load (ltp vs usex) or to one of the >>>changes I've made recently. I'll do some experimentation to see if >>>there is >>>a fix I should propose. >>> >>>This still doesn't address the radical drift you saw. >>>The next step for me is to run 3.2 and see if I can reproduce it. >>> >>>Regards, >>>Dave >>> >>> >>> >>> >>> >>>Dave Winchell wrote: >>> >>> >>> >>>>Hi Dan, >>>> >>>>Sorry it took me so long, but I finally ran an ltp test today. >>>>Its on rh4u4-64. I'm using the defaults for ltp and using a script >>>>called runltp. I had a usex load on rh4u5-64. No ntpd. >>>>virtual processors / physical processors = 2. >>>> >>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes >>>>for -.005% and .008%. >>>> >>>>I'm running a 3.1 based hypervisor with some clock related >>>> >>>> >>tweaks that >> >> >>>>I haven't submitted, because I'm still characterizing them. >>>> >>>>I'm stopping the usex load on 4u5-64 now and replacing it with ltp >>>>and will leave the two guests running ltp over the weekend. >>>> >>>>Regards, >>>>Dave >>>> >>>> >>>>Dave Winchell wrote: >>>> >>>> >>>> >>>>>Hi Dan, Deepak: >>>>> >>>>>Thanks for the data. Those drifts are severe - no wonder >>>>> >>>>> >>ntp couldn't >> >> >>>>>keep then in synch. I'll try to reproduce that behaviour >>>>> >>>>> >>here, with >> >> >>>>>my code base. >>>>>If I can't reproduce it, I'll try 3.2. >>>>> >>>>>If you can isolate what ltp is doing during the cliffs, that would >>>>>be very >>>>>helpful. >>>>> >>>>>thanks, >>>>>Dave >>>>> >>>>> >>>>> >>>>> >>>>>Dan Magenheimer wrote: >>>>> >>>>> >>>>> >>>>>>OK, Deepak repeated the test without ntpd and using >>>>>> >>>>>> >>ntpdate -b before >> >> >>>>>>the test. >>>>>> >>>>>>The attached graph shows his results: el5u1-64 (best=~0.07%), >>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%). >>>>>> >>>>>>We will continue to look at LTP to try to isolate. >>>>>> >>>>>>Thanks, >>>>>>Dan >>>>>> >>>>>>P.S. elXuY is essentially RHEL XuY with some patches. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>-----Original Message----- >>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM >>>>>>>To: Deepak Patel >>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser; >>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; >>>>>>> >>>>>>> >>Dave Winchell >> >> >>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>>>>>pending >>>>>>>missed ticks >>>>>>> >>>>>>> >>>>>>>Dan, Deeepak, >>>>>>> >>>>>>>It may be that the underlying clock error is too great for ntp >>>>>>>to handle. It would be useful if you did not run ntpd >>>>>>>and, instead did ntpdate -b at the start >>>>>>> >>>>>>> >>of the test >> >> >>>>>>>for each guest. Then capture the data as you have been doing. >>>>>>>If the drift is greater than .05%, then we need to address that. >>>>>>> >>>>>>>Another option is, when running ntpd, to enable loop >>>>>>> >>>>>>> >>statistics in >> >> >>>>>>>/etc/ntp.conf >>>>>>>by adding this to the file: >>>>>>> >>>>>>>statistics loopstats >>>>>>>statsdir /var/lib/ntp/ >>>>>>> >>>>>>>Then you will see loop data in that directory. >>>>>>>Correlating the data in the loopstats files with the >>>>>>>peaks in skew would be interesting. You will see >>>>>>> >>>>>>> >>entries of the form >> >> >>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776 >>>>>>> >>>>>>> >>239.735511 10 >> >> >>>>>>>Where the second to last column is the Allan Deviation. >>>>>>> >>>>>>> >>When that >> >> >>>>>>>gets over 1000, ntpd is working pretty hard. However, I have not >>>>>>>seen ntpd >>>>>>>completely loose it like you have. >>>>>>> >>>>>>>I'm on vacation until Monday, and won't be reading >>>>>>>email. >>>>>>> >>>>>>>Thanks for all your work on this! >>>>>>> >>>>>>>-Dave >>>>>>> >>>>>>>Deepak Patel wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested this one.) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>I do not know which graph was attached with this. But >>>>>>>> >>>>>>>> >>I saw this >> >> >>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm >>>>>>>> >>>>>>>> >>guests when I >> >> >>>>>>>>was running ltp tests continuously. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>What was the behaviour of the other guests running? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>All pvm guests are fine. But behavior of most of the >>>>>>>> >>>>>>>> >>hvm guests were >> >> >>>>>>>>as described. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>If they had spikes, were they at the same wall time? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>No. They are not at the same wall time. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>Were the other guests running ltp as well? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp >>>>>>>>continuously. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>How are you measuring skew? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>I was collecting output of "ntpdate -q every >>>>>>>> >>>>>>>> >>>>>>> >>>>>>>300 seconds >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>(5 minutes) and have created graph based on that. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>Are you running ntpd? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>>Yes. ntp was running on all the guests. >>>>>>>> >>>>>>>>I am investigating what causes this spikes and let everyone >>>>>>>> >>>>>>>> >>>>>>> >>>>>>>know what >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>are my findings. >>>>>>>> >>>>>>>>Thanks, >>>>>>>>Deepak >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>Anything that you can discover that would be in sync with >>>>>>>>>the spikes would be very helpful! >>>>>>>>> >>>>>>>>>The code that I test with is our product code, which is based >>>>>>>>>on 3.1. So it is possible that something in 3.2 other >>>>>>>>> >>>>>>>>> >>than vpt.c >> >> >>>>>>>>>is the cause. I can test with 3.2, if necessary. >>>>>>>>> >>>>>>>>>thanks, >>>>>>>>>Dave >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>Dan Magenheimer wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>Hi Dave (Keir, see suggestion below) -- >>>>>>>>>> >>>>>>>>>>Thanks! >>>>>>>>>> >>>>>>>>>>Turning off vhpet certainly helps a lot (though see below). >>>>>>>>>> >>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it >>>>>>>>>> >>>>>>>>>> >>should be >> >> >>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable) until it is >>>>>>>>>>fixed? (I have a patch that defaults it off, can post it if >>>>>>>>>>there is agreement on the above point.) The whole >>>>>>>>>> >>>>>>>>>> >>point of an >> >> >>>>>>>>>>HPET is to provide more precise timekeeping and if vhpet is >>>>>>>>>>worse than vpit, it can only confuse users. Comments? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>In your testing, are you just measuring % skew over a long >>>>>>>>>>period of time? >>>>>>>>>>We are graphing the skew continuously and >>>>>>>>>>seeing periodic behavior that is unsettling, even with pit. >>>>>>>>>>See attached. Though your algorithm recovers, the "cliffs" >>>>>>>>>>could still cause real user problems. I wonder if there is >>>>>>>>>>anything that can be done to make the "recovery" more >>>>>>>>>>responsive? >>>>>>>>>> >>>>>>>>>>We are looking into what part(s) of LTP is causing >>>>>>>>>> >>>>>>>>>> >>the cliffs. >> >> >>>>>>>>>>Thanks, >>>>>>>>>>Dan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>-----Original Message----- >>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM >>>>>>>>>>>To: dan.magenheimer@oracle.com >>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>>>>>>>>>deepak.patel@oracle.com; >>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>> >>>>>>>>>>> >>that disables >> >> >>>>>>>>>>>pending >>>>>>>>>>>missed ticks >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>Dan, >>>>>>>>>>> >>>>>>>>>>>I guess I'm a bit out of date calling for clock= usage. >>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you >>>>>>>>>>> >>>>>>>>>>> >>should specify >> >> >>>>>>>>>>>"clocksource=pit nohpet" on the linux guest bootline. >>>>>>>>>>> >>>>>>>>>>>You can leave the xen and dom0 bootlines as they are. >>>>>>>>>>>The xen and guest clocksources do not need to be the same. >>>>>>>>>>>In my tests, xen is using the hpet for its timekeeping and >>>>>>>>>>>that appears to be the default. >>>>>>>>>>> >>>>>>>>>>>When you boot the guests you should see >>>>>>>>>>> time.c: Using PIT/TSC based timekeeping. >>>>>>>>>>>on the rh4u5-64 guest, and something similar on the others. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>>>>>>14.318MHz HPET.) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>This appears to be the xen state, which is fine. >>>>>>>>>>>I was wrongly assuming that this was the guest state. >>>>>>>>>>>You might want to look in your guest logs and see >>>>>>>>>>> >>>>>>>>>>> >>what they were >> >> >>>>>>>>>>>picking >>>>>>>>>>>for a clock source. >>>>>>>>>>> >>>>>>>>>>>Regards, >>>>>>>>>>>Dave >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>Dan Magenheimer wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>Thanks, I hadn't realized that! No wonder we didn't >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>see the same >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>improvement you saw! >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>Try specifying clock=pit on the linux boot line... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>I'm confused... do you mean "clocksource=pit" on the Xen >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>command line or >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the guest (or >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>dom0?) command >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>line? Or both places? Since the tests take awhile, it >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>would be nice >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>to get this right the first time. Do the Xen and guest >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>clocksources need >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>to be the same? >>>>>>>>>>>> >>>>>>>>>>>>Thanks, >>>>>>>>>>>>Dan >>>>>>>>>>>> >>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM >>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser >>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com; >>>>>>>>>>>> >>>>>>>>>>>> >>deepak.patel@oracle.com; >> >> >>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>that disables >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>pending missed ticks >>>>>>>>>>>> >>>>>>>>>>>> Hi Dan, >>>>>>>>>>>> >>>>>>>>>>>> Hpet timer does have a fairly large error, as I >>>>>>>>>>>>was >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>trying this >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> one recently. >>>>>>>>>>>> I don't remember what I got for error, but 1% sounds >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>about right. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> The problem is that hpet is not built on top of vpt.c, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>the module >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Keir and I did >>>>>>>>>>>> all the recent work in, for its periodic timer needs. Try >>>>>>>>>>>> specifying clock=pit >>>>>>>>>>>> on the linux boot line. If it still picks the >>>>>>>>>>>> >>>>>>>>>>>> >>hpet, which it >> >> >>>>>>>>>>>> might, let me know >>>>>>>>>>>> and I'll tell you how to get around this. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Dave >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>-------------------------------------------------------------- >> >> >>>>>>>>>>>---------- >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> *From:* Dan Magenheimer >>>>>>>>>>>> >>>>>>>>>>>> >>[mailto:dan.magenheimer@oracle.com] >> >> >>>>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM >>>>>>>>>>>> *To:* Dave Winchell; Keir Fraser >>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; >>>>>>>>>>>> >>>>>>>>>>>> >>deepak.patel@oracle.com; >> >> >>>>>>>>>>>> akira.ijuin@oracle.com >>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>that disables >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> pending missed ticks >>>>>>>>>>>> >>>>>>>>>>>> Sorry for the very late followup on this but we finally >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>were able >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> to get our testing set up again on stable 3.1 >>>>>>>>>>>> >>>>>>>>>>>> >>bits and have >> >> >>>>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the >>>>>>>>>>>> >>>>>>>>>>>> >>order of 1%. >> >> >>>>>>>>>>>> Test enviroment was a 4-socket dual core machine >>>>>>>>>>>> >>>>>>>>>>>> >>with 24GB of >> >> >>>>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>plus two pv. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> All six guests were running LTP simultaneously. >>>>>>>>>>>> >>>>>>>>>>>> >>The four hvm >> >> >>>>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>RHEL4u5-64. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>32-bit guests. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> All four hvm guests experienced skew around -1%, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>even the 32-bit >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> guest. Less intensive testing didn't exhibit much >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>skew at all. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> A representative graph is attached. >>>>>>>>>>>> >>>>>>>>>>>> Dave, I wonder if some portion of your patches >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>didn't end up in >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> the xen trees? >>>>>>>>>>>> >>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, >>>>>>>>>>>> >>>>>>>>>>>> >>Platform timer >> >> >>>>>>>>>>>> 14.318MHz HPET.) >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Dan >>>>>>>>>>>> >>>>>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests. >>>>>>>>>>>> >>>>>>>>>>>> > -----Original Message----- >>>>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >> >> >>>>>>>>>>>> > Dave Winchell >>>>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>>>>>>>>>> > To: Keir Fraser >>>>>>>>>>>> > Cc: dan.magenheimer@oracle.com; >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>xen-devel@lists.xensource.com; Dave >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > Winchell >>>>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>>>>>> > disables pending >>>>>>>>>>>> > missed ticks >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > Hi Keir, >>>>>>>>>>>> > >>>>>>>>>>>> > The latest change, c/s 16690, looks fine. >>>>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to >>>>>>>>>>>> > the code I submitted. Also, your version is more >>>>>>>>>>>> > concise. >>>>>>>>>>>> > >>>>>>>>>>>> > The error tests confirm the equivalence. With >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>overnight cpu loads, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > the checked in version was accurate to +.048% for sles >>>>>>>>>>>> > and +.038% for red hat. My version was +.046% >>>>>>>>>>>>and >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>+.032% in a >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > 2 hour test. >>>>>>>>>>>> > I don't think the difference is significant. >>>>>>>>>>>> > >>>>>>>>>>>> > i/o loads produced errors of +.01%. >>>>>>>>>>>> > >>>>>>>>>>>> > Thanks for all your efforts on this issue. >>>>>>>>>>>> > >>>>>>>>>>>> > Regards, >>>>>>>>>>>> > Dave >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > Keir Fraser wrote: >>>>>>>>>>>> > >>>>>>>>>>>> > >Applied as c/s 16690, although the checked-in patch is >>>>>>>>>>>> > smaller. I think the >>>>>>>>>>>> > >only important fix is to pt_intr_post() and the >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>only bit of >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > the patch I >>>>>>>>>>>> > >totally omitted was the change to >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>pt_process_missed_ticks(). >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > I don't think >>>>>>>>>>>> > >that change can be important, but let's see what >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>happens to the >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> error >>>>>>>>>>>> > >percentage... >>>>>>>>>>>> > > >>>>>>>>>>>> > > -- Keir >>>>>>>>>>>> > > >>>>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > >>Hi Dan and Keir, >>>>>>>>>>>> > >> >>>>>>>>>>>> > >>Attached is a patch that fixes some issues with the >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>SYNC policy >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>(no_missed_ticks_pending). >>>>>>>>>>>> > >>I have not tried to make the change the >>>>>>>>>>>> >>>>>>>>>>>> >>minimal one, but, >> >> >>>>>>>>>>>> > rather, just >>>>>>>>>>>> > >>ported into >>>>>>>>>>>> > >>the new code what I know to work well. The error for >>>>>>>>>>>> > >>no_missed_ticks_pending goes from >>>>>>>>>>>> > >>over 3% to .03% with this change according >>>>>>>>>>>> >>>>>>>>>>>> >>to my testing. >> >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >>Regards, >>>>>>>>>>>> > >>Dave >>>>>>>>>>>> > >> >>>>>>>>>>>> > >>Dan Magenheimer wrote: >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >>>Hi Dave -- >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>>Did you get your correction ported? If so, >>>>>>>>>>>> >>>>>>>>>>>> >>it would be >> >> >>>>>>>>>>>> > nice to see this get >>>>>>>>>>>> > >>>into 3.1.3. >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>>Note that I just did some very limited testing with >>>>>>>>>>>> > timer_mode=2(=SYNC=no >>>>>>>>>>>> > >>>missed ticks pending) >>>>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>guest) and the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > worst error I've >>>>>>>>>>>> > >>>seen so far >>>>>>>>>>>> > >>>is 0.012%. But I haven't tried any exotic >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>loads, just LTP. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>>Thanks, >>>>>>>>>>>> > >>>Dan >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>>>-----Original Message----- >>>>>>>>>>>> > >>>>From: Dave Winchell >>>>>>>>>>>> >>>>>>>>>>>> >>[mailto:dwinchell@virtualiron.com] >> >> >>>>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com >>>>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>>>>>>>>>> > xen-devel@lists.xensource.com; Dong, >>>>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a >>>>>>>>>>>> >>>>>>>>>>>> >>timer mode that >> >> >>>>>>>>>>>> > >>>>disables pending >>>>>>>>>>>> > >>>>missed ticks >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>Dan, >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>I did some testing with the constant tsc offset >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>SYNC method >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>(now called >>>>>>>>>>>> > >>>>no_missed_ticks_pending) >>>>>>>>>>>> > >>>>and found the error to be very high, much larger >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>than 1 %, as >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>I recall. >>>>>>>>>>>> > >>>>I have not had a chance to submit a correction. I >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>will try to >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>do it later >>>>>>>>>>>> > >>>>this week or the first week in January. My >>>>>>>>>>>> >>>>>>>>>>>> >>version of >> >> >>>>>>>>>>>> constant tsc >>>>>>>>>>>> > >>>>offset SYNC method >>>>>>>>>>>> > >>>>produces .02 % error, so I just need to port >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>that into the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>current code. >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>The error you got for both of those kernels is >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>what I would >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> expect >>>>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode. >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>Regards, >>>>>>>>>>>> > >>>>Dave >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>Dan Magenheimer wrote: >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>>Anyone make measurements on the final patch? >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>saw a loss of >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>about 0.2% with no load. This was >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>xen-unstable tip today >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>with no options specified. 32-bit was about 0.01%. >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>>I think I missed something... how do I >>>>>>>>>>>> >>>>>>>>>>>> >>run the various >> >> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>accounting choices and which ones are known to be >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>appropriate >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>for which kernels? >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>>Thanks, >>>>>>>>>>>> > >>>>>Dan >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>>>-----Original Message----- >>>>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>Keir Fraser >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>>>>>>> > >>>>>>To: Dave Winchell >>>>>>>>>>>> > >>>>>>Cc: Shan, Haitao; >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>xen-devel@lists.xensource.com; Dong, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > Eddie; Jiang, >>>>>>>>>>>> > >>>>>>Yunhong >>>>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>mode that >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>>>disables pending >>>>>>>>>>>> > >>>>>>missed ticks >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>Please take a look at xen-unstable >>>>>>>>>>>> >>>>>>>>>>>> >>changeset 16545. >> >> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>-- Keir >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>> wrote: >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>Keir, >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>loads for the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>>>>various time protocols follows. In >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>addition, the data >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>>>>for cpu loads is shown. >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>processor AMD >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> box. >>>>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 >>>>>>>>>>>> >>>>>>>>>>>> >>vcpu each. >> >> >>>>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>>>>>> > >>>>>>>(usex is available at >>>>>>>>>>>> http://people.redhat.com/anderson/usex.) >>>>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>of=/dev/null. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 instances of dd. >>>>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>>>>>> > >>>>>>>All three guests are 8vcpu. >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same >>>>>>>>>>>> >>>>>>>>>>>> >>as i/o-32 >> >> >>>>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>instances of dd. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, >>>>>>>>>>>>+4.42 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>sec -.006%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > +.005% cpu >>>>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>sec, -.001%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > +.012% cpu >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 >>>>>>>>>>>> >>>>>>>>>>>> >>sec, -.009%, >> >> >>>>>>>>>>>> -.004% cpu >>>>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>-.005%, -.005% cpu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>-.008%, -.003% cpu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>-.040% cpu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>sec, -.034%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > -.031% cpu >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 >>>>>>>>>>>> >>>>>>>>>>>> >>sec,-55.7 sec, -.01%, >> >> >>>>>>>>>>>> > -.09% i/o-8 >>>>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 >>>>>>>>>>>> >>>>>>>>>>>> >>sec,-14.0 sec, -.015% >> >> >>>>>>>>>>>> > -.14% i/o-8 >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 >>>>>>>>>>>> >>>>>>>>>>>> >>sec, -.017%, >> >> >>>>>>>>>>>> > -.022% i/o-8 >>>>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>-.017%, -.018% i/o-8 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 >>>>>>>>>>>> >>>>>>>>>>>> >>sec, -.020%, >> >> >>>>>>>>>>>> > -.029% i/o-8 >>>>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>sec, -.023%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > -.031% i/o-8 >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>-.04% i/o-32 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 >>>>>>>>>>>> >>>>>>>>>>>> >>sec, -.011%, >> >> >>>>>>>>>>>> > -.005% i/o-32 >>>>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>-.11% i/o-32 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, >>>>>>>>>>>> >>>>>>>>>>>> >>13. sec -.07%, >> >> >>>>>>>>>>>> > .003% i/o-4/32 >>>>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 >>>>>>>>>>>> >>>>>>>>>>>> >>sec, -.017%, >> >> >>>>>>>>>>>> > .01% i/o-4/32 >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>Overhead measurements: >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>Progress in terms of number of passes >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>through a fixed >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>system workload >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>>>>>> > >>>>>>>The workload was usex -b48. >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>Conclusions: >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>>>>>>>>>> > requirement for ntp >>>>>>>>>>>> > >>>>>>>tracking under the loads >>>>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>accuracies for >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>SYNC, MIXED, >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>and ASYNC >>>>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC >>>>>>>>>>>> >>>>>>>>>>>> >>method by only >> >> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>scheduling the extra >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>wakeups if a certain number >>>>>>>>>>>> > >>>>>>>of ticks are missed. >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>Regards, >>>>>>>>>>>> > >>>>>>>Dave >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>Keir Fraser wrote: >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>> wrote: >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>ASYNC method a >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>couple of days ago, >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think >>>>>>>>>>>> >>>>>>>>>>>> >>there may have >> >> >>>>>>>>>>>> > been something >>>>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>days ago for >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>ASYNC. It may have been >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>after context >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>switch in. >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC >>>>>>>>>>>> >>>>>>>>>>>> >>or ASYNC give >> >> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>acceptable accuracy, >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>each running consistently around or under >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>.01%. MIXED has >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>a fairly high >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>error of >>>>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close >>>>>>>>>>>> >>>>>>>>>>>> >>to .05% ntp >> >> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>threshold for comfort. >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>plan to leave >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>SYNC running >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>leave MIXED >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>running instead. >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>I can run >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>more overnight tests >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>>next week. >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>effects of the >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>SYNC+run_timer >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>cause higher >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>system-wide CPU >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>contention. I find it easier to think >>>>>>>>>>>> >>>>>>>>>>>> >>through the >> >> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>implications of ASYNC. I'm >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>accurate than >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>ASYNC. Perhaps it >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>approaches, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>and each interrupt >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>event causes a small accumulated error? >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>favourites and >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>if the latter is >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>actually more accurate then I can >>>>>>>>>>>> >>>>>>>>>>>> >>simply revert the >> >> >>>>>>>>>>>> > changeset that >>>>>>>>>>>> > >>>>>>>>implemented MIXED. >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>workloads you >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>could try idle >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>large disc reads >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>to /dev/null)? We >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>CPU bound, so >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>that's really an >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo. >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>>-- Keir >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>> > >>>>>>_______________________________________________ >>>>>>>>>>>> > >>>>>>Xen-devel mailing list >>>>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com >>>>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>2007 +0000 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>2008 -0500 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>pt_process_missed_ticks(stru >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> missed_ticks = missed_ticks / (s_time_t) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>pt->period + 1; >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >> if ( mode_is(pt->vcpu->domain, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>no_missed_ticks_pending) ) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>>>>>>>>> > >>+ pt->do_not_freeze = 1; >>>>>>>>>>>> > >> else >>>>>>>>>>>> > >> pt->pending_intr_nr += missed_ticks; >>>>>>>>>>>> > >> pt->scheduled += missed_ticks * pt->period; >>>>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void >>>>>>>>>>>> >>>>>>>>>>>> >>pt_timer_fn(void *data) >> >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> pt_lock(pt); >>>>>>>>>>>> > >> >>>>>>>>>>>> > >>- pt->pending_intr_nr++; >>>>>>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>no_missed_ticks_pending) ) { >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >>+ pt->pending_intr_nr = 1; >>>>>>>>>>>> > >>+ pt->do_not_freeze = 0; >>>>>>>>>>>> > >>+ } >>>>>>>>>>>> > >>+ else >>>>>>>>>>>> > >>+ pt->pending_intr_nr++; >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> if ( !pt->one_shot ) >>>>>>>>>>>> > >> { >>>>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>vcpu *v, struct >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >> return; >>>>>>>>>>>> > >> } >>>>>>>>>>>> > >> >>>>>>>>>>>> > >>- pt->do_not_freeze = 0; >>>>>>>>>>>> > >>- >>>>>>>>>>>> > >> if ( pt->one_shot ) >>>>>>>>>>>> > >> { >>>>>>>>>>>> > >> pt->enabled = 0; >>>>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>*v, struct >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > >> pt->last_plt_gtime = >>>>>>>>>>>> >>>>>>>>>>>> >>hvm_get_guest_time(v); >> >> >>>>>>>>>>>> > >> pt->pending_intr_nr = 0; /* >>>>>>>>>>>> >>>>>>>>>>>> >>'collapse' all >> >> >>>>>>>>>>>> > missed ticks */ >>>>>>>>>>>> > >> } >>>>>>>>>>>> > >>+ else if ( mode_is(v->domain, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>no_missed_ticks_pending) ) { >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>> > >>+ pt->pending_intr_nr--; >>>>>>>>>>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>>>>>> > >>+ } >>>>>>>>>>>> > >> else >>>>>>>>>>>> > >> { >>>>>>>>>>>> > >> pt->last_plt_gtime += pt->period_cycles; >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > _______________________________________________ >>>>>>>>>>>> > Xen-devel mailing list >>>>>>>>>>>> > Xen-devel@lists.xensource.com >>>>>>>>>>>> > http://lists.xensource.com/xen-devel >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>> >>> >>> >>-------------------------------------------------------------- >>---------- >> >> >> >> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Fri, 15 Feb 2008 12:28:22 -0500 Message-ID: <47B5CBB6.5020505@virtualiron.com> References: <20080215094650484.00000001516@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080215094650484.00000001516@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , Deepak Patel , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Dan, Mine was oversubscribed. 8 physical cpu, 2 guests, each with 8 vcpu. I ran one instance of ltp on each guest, continuously. I hope ltp loaded up all the vcpus. I seem to recall that it did, but I could be wrong. If it didn't, that would be a major difference between our tests. I'll verify this afternoon and run multiple instances, if necessary. Thanks, Dave Dan Magenheimer wrote: >Hi Dave -- > >No new results yet but one other question: > >The problems we've seen with our testing have been with a heavily >oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests >running LTP simultaneously. > >Was your LTP testing oversubscribed or just a single guest? > >Thanks, >Dan > > > >>-----Original Message----- >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>Sent: Thursday, February 14, 2008 10:56 AM >>To: dan.magenheimer@oracle.com >>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave >>Winchell >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >>Dan, >> >>Here are some boot snipets for rh4u564 on xen 3.2. >> >> >>#1: >> >>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro >>root=LABEL=/ console=ttyS0 clocksource=pit nohpet) >>Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp >>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 >>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 >>... >>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/ >>console=ttyS0 clocksource=pit nohpet >>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0 >>Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11, >>65536 bytes) >>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer. >>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor. >>... >>Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8 >>CPUs: passed. >>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs >>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use >>of PM timer >>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping. >> >> >> >>#2: >> >>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro >>root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer) >>Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp >>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 >>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 >>... >>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/ >>console=ttyS0 clocksource=pit nohpet nopmtimer >>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0 >>Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11, >>65536 bytes) >>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer. >>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor. >>... >>Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8 >>CPUs: passed. >>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs >>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping. >> >> >>As you can see, I only get the pit if I specify nopmtimer. >> >>Dan Magenheimer wrote: >> >> >> >>>Hi Dave -- >>> >>>Thanks for continuing to run tests! >>> >>>Hmmm... I thought I had noticed that even though Linux will >>> >>> >>acknowledge >> >> >>>the existence of the pmtimer, it still prints: >>> >>>time.c: Using PIT/TSC based timekeeping. >>> >>>I will check again, but assuming the clocksource for our tests is >>>indeed pit, the huge difference in the results (yours vs ours) is >>>baffling. I wonder if the difference may be the underlying hardware. >>>Maybe we will try to ensure we can duplicate the results on >>> >>> >>a different >> >> >>>box. >>> >>> >>>So your testing was with stock 3.2.0 xen bits (what cset?) without >>>any of your [quote from below] "clock related tweaks that I haven't >>>submitted, because I'm still characterizing them"? >>> >>> >>> >>> >>None of the tweaks I mentioned are in this test. >>It was stock with some patches. However, none of the patches are time >>related to >>my knowledge and I checked vpt.c to make sure that it is the same as >>what's in unstable. >>The only difference is in pt_intr_post, where I set the timer mode. >>I don't have timer mode tied into our config process yet, which >>is different than official xen method. >> >> >>(In pt_intr_post) >> else >> { >>+ if(v->arch.paging.mode->guest_levels == 4) >>+ v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = >>HVMPTM_no_missed_ticks_pending; >>+ else >>+ v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = >>HVMPTM_delay_for_missed_ticks; >> if ( mode_is(v->domain, one_missed_tick_pending) || >> mode_is(v->domain, no_missed_ticks_pending) ) >> { >> >> >> >>>Could you also send detail on the rhel4u4-64 kernel you >>>are testing with, just to ensure we are not comparing apples >>>and oranges? (Perhaps there's some way we can even share the >>>identical disk image and vm.cfg file?) >>> >>>And if our problem is indeed the pmtimer, I will need to submit >>>another patch to Keir to add an hvm pmtimer platform variable. >>>(Hmmm... I don't think he's even accepted the hpet variable patch >>>yet. I'll have to check.) >>> >>>Thanks, >>>Dan >>> >>> >>> >>> >>> >>> >>>>-----Original Message----- >>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>Sent: Thursday, February 14, 2008 9:00 AM >>>>To: dan.magenheimer@oracle.com >>>>Cc: Dave Winchell; Keir Fraser; >>>> >>>> >>xen-devel@lists.xensource.com; Deepak >> >> >>>>Patel >>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>disables pending >>>>missed ticks >>>> >>>> >>>>Hi Dan, >>>> >>>>I ran the ltp tests with 3.2 and found the errors >>>>for a 16 hour run to be: >>>> >>>>rh4u564 -9.9 sec (-.017%) >>>>rh4u464 -7.3 sec (-.013%) >>>> >>>>There were no cliffs and the drift was linear. >>>> >>>>I think the problem you had may be due to the use of the >>>>pm timer. If you still have the boot log, it would tell you. >>>> >>>>When I first tried a guest on 3.2 with "clocksource=pit nohpet" >>>>I noticed that it picked the pm timer. Adding "nopmtimer", the >>>>guest will pick the pit. >>>> >>>>The reason I didn't have the problem with our 3.1 base is that >>>>I had disabled the hpet and the pmtimer by not advertising them >>>>in the acpi tables. I did this so long ago, I forgot that I had to >>>>disable pmtimer as well as hpet. >>>> >>>>So, can you re-run your test with "clocksource=pit nohpet >>>> >>>> >>nopmtimer"? >> >> >>>>You should see this in the boot messages: >>>> >>>>time.c: Using PIT/TSC based timekeeping. >>>> >>>>Thanks, >>>>Dave >>>> >>>> >>>>Dave Winchell wrote: >>>> >>>> >>>> >>>> >>>> >>>>>Hi Dan, >>>>> >>>>>Over the weekend the drift was +18 seconds for each guest (no ntp). >>>>>The duration was 3900 minutes, so the error for each was +.0077%. >>>>>Looking back through the data, it appears to drift linearly at >>>>>this rate. I've attached a plot for rh4u5-64. >>>>> >>>>>This accuracy is better than what I've seen before (.03-.05%). >>>>>This may be due to the different load (ltp vs usex) or to >>>>> >>>>> >>one of the >> >> >>>>>changes I've made recently. I'll do some experimentation to see if >>>>>there is >>>>>a fix I should propose. >>>>> >>>>>This still doesn't address the radical drift you saw. >>>>>The next step for me is to run 3.2 and see if I can reproduce it. >>>>> >>>>>Regards, >>>>>Dave >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>Dave Winchell wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>Hi Dan, >>>>>> >>>>>>Sorry it took me so long, but I finally ran an ltp test today. >>>>>>Its on rh4u4-64. I'm using the defaults for ltp and using a script >>>>>>called runltp. I had a usex load on rh4u5-64. No ntpd. >>>>>>virtual processors / physical processors = 2. >>>>>> >>>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes >>>>>>for -.005% and .008%. >>>>>> >>>>>>I'm running a 3.1 based hypervisor with some clock related >>>>>> >>>>>> >>>>>> >>>>>> >>>>tweaks that >>>> >>>> >>>> >>>> >>>>>>I haven't submitted, because I'm still characterizing them. >>>>>> >>>>>>I'm stopping the usex load on 4u5-64 now and replacing it with ltp >>>>>>and will leave the two guests running ltp over the weekend. >>>>>> >>>>>>Regards, >>>>>>Dave >>>>>> >>>>>> >>>>>>Dave Winchell wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>Hi Dan, Deepak: >>>>>>> >>>>>>>Thanks for the data. Those drifts are severe - no wonder >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>ntp couldn't >>>> >>>> >>>> >>>> >>>>>>>keep then in synch. I'll try to reproduce that behaviour >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>here, with >>>> >>>> >>>> >>>> >>>>>>>my code base. >>>>>>>If I can't reproduce it, I'll try 3.2. >>>>>>> >>>>>>>If you can isolate what ltp is doing during the cliffs, >>>>>>> >>>>>>> >>that would >> >> >>>>>>>be very >>>>>>>helpful. >>>>>>> >>>>>>>thanks, >>>>>>>Dave >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>Dan Magenheimer wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>OK, Deepak repeated the test without ntpd and using >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>ntpdate -b before >>>> >>>> >>>> >>>> >>>>>>>>the test. >>>>>>>> >>>>>>>>The attached graph shows his results: el5u1-64 (best=~0.07%), >>>>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%). >>>>>>>> >>>>>>>>We will continue to look at LTP to try to isolate. >>>>>>>> >>>>>>>>Thanks, >>>>>>>>Dan >>>>>>>> >>>>>>>>P.S. elXuY is essentially RHEL XuY with some patches. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>-----Original Message----- >>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM >>>>>>>>>To: Deepak Patel >>>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser; >>>>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>Dave Winchell >>>> >>>> >>>> >>>> >>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>>>>>>>pending >>>>>>>>>missed ticks >>>>>>>>> >>>>>>>>> >>>>>>>>>Dan, Deeepak, >>>>>>>>> >>>>>>>>>It may be that the underlying clock error is too great for ntp >>>>>>>>>to handle. It would be useful if you did not run ntpd >>>>>>>>>and, instead did ntpdate -b at the start >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>of the test >>>> >>>> >>>> >>>> >>>>>>>>>for each guest. Then capture the data as you have been doing. >>>>>>>>>If the drift is greater than .05%, then we need to >>>>>>>>> >>>>>>>>> >>address that. >> >> >>>>>>>>>Another option is, when running ntpd, to enable loop >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>statistics in >>>> >>>> >>>> >>>> >>>>>>>>>/etc/ntp.conf >>>>>>>>>by adding this to the file: >>>>>>>>> >>>>>>>>>statistics loopstats >>>>>>>>>statsdir /var/lib/ntp/ >>>>>>>>> >>>>>>>>>Then you will see loop data in that directory. >>>>>>>>>Correlating the data in the loopstats files with the >>>>>>>>>peaks in skew would be interesting. You will see >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>entries of the form >>>> >>>> >>>> >>>> >>>>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>239.735511 10 >>>> >>>> >>>> >>>> >>>>>>>>>Where the second to last column is the Allan Deviation. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>When that >>>> >>>> >>>> >>>> >>>>>>>>>gets over 1000, ntpd is working pretty hard. However, >>>>>>>>> >>>>>>>>> >>I have not >> >> >>>>>>>>>seen ntpd >>>>>>>>>completely loose it like you have. >>>>>>>>> >>>>>>>>>I'm on vacation until Monday, and won't be reading >>>>>>>>>email. >>>>>>>>> >>>>>>>>>Thanks for all your work on this! >>>>>>>>> >>>>>>>>>-Dave >>>>>>>>> >>>>>>>>>Deepak Patel wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested this one.) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>I do not know which graph was attached with this. But >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>I saw this >>>> >>>> >>>> >>>> >>>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>guests when I >>>> >>>> >>>> >>>> >>>>>>>>>>was running ltp tests continuously. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>What was the behaviour of the other guests running? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>All pvm guests are fine. But behavior of most of the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>hvm guests were >>>> >>>> >>>> >>>> >>>>>>>>>>as described. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>If they had spikes, were they at the same wall time? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>No. They are not at the same wall time. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>Were the other guests running ltp as well? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp >>>>>>>>>>continuously. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>How are you measuring skew? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>I was collecting output of "ntpdate -q every >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>300 seconds >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>(5 minutes) and have created graph based on that. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>Are you running ntpd? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>Yes. ntp was running on all the guests. >>>>>>>>>> >>>>>>>>>>I am investigating what causes this spikes and let everyone >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>know what >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>are my findings. >>>>>>>>>> >>>>>>>>>>Thanks, >>>>>>>>>>Deepak >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>Anything that you can discover that would be in sync with >>>>>>>>>>>the spikes would be very helpful! >>>>>>>>>>> >>>>>>>>>>>The code that I test with is our product code, which is based >>>>>>>>>>>on 3.1. So it is possible that something in 3.2 other >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>than vpt.c >>>> >>>> >>>> >>>> >>>>>>>>>>>is the cause. I can test with 3.2, if necessary. >>>>>>>>>>> >>>>>>>>>>>thanks, >>>>>>>>>>>Dave >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>Dan Magenheimer wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>Hi Dave (Keir, see suggestion below) -- >>>>>>>>>>>> >>>>>>>>>>>>Thanks! >>>>>>>>>>>> >>>>>>>>>>>>Turning off vhpet certainly helps a lot (though see below). >>>>>>>>>>>> >>>>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>should be >>>> >>>> >>>> >>>> >>>>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable) >>>>>>>>>>>> >>>>>>>>>>>> >>until it is >> >> >>>>>>>>>>>>fixed? (I have a patch that defaults it off, can post it if >>>>>>>>>>>>there is agreement on the above point.) The whole >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>point of an >>>> >>>> >>>> >>>> >>>>>>>>>>>>HPET is to provide more precise timekeeping and if vhpet is >>>>>>>>>>>>worse than vpit, it can only confuse users. Comments? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>In your testing, are you just measuring % skew over a long >>>>>>>>>>>>period of time? >>>>>>>>>>>>We are graphing the skew continuously and >>>>>>>>>>>>seeing periodic behavior that is unsettling, even with pit. >>>>>>>>>>>>See attached. Though your algorithm recovers, the "cliffs" >>>>>>>>>>>>could still cause real user problems. I wonder if there is >>>>>>>>>>>>anything that can be done to make the "recovery" more >>>>>>>>>>>>responsive? >>>>>>>>>>>> >>>>>>>>>>>>We are looking into what part(s) of LTP is causing >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>the cliffs. >>>> >>>> >>>> >>>> >>>>>>>>>>>>Thanks, >>>>>>>>>>>>Dan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM >>>>>>>>>>>>>To: dan.magenheimer@oracle.com >>>>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>>>>>>>>>>>deepak.patel@oracle.com; >>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>that disables >>>> >>>> >>>> >>>> >>>>>>>>>>>>>pending >>>>>>>>>>>>>missed ticks >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>Dan, >>>>>>>>>>>>> >>>>>>>>>>>>>I guess I'm a bit out of date calling for clock= usage. >>>>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>should specify >>>> >>>> >>>> >>>> >>>>>>>>>>>>>"clocksource=pit nohpet" on the linux guest bootline. >>>>>>>>>>>>> >>>>>>>>>>>>>You can leave the xen and dom0 bootlines as they are. >>>>>>>>>>>>>The xen and guest clocksources do not need to be the same. >>>>>>>>>>>>>In my tests, xen is using the hpet for its timekeeping and >>>>>>>>>>>>>that appears to be the default. >>>>>>>>>>>>> >>>>>>>>>>>>>When you boot the guests you should see >>>>>>>>>>>>> time.c: Using PIT/TSC based timekeeping. >>>>>>>>>>>>>on the rh4u5-64 guest, and something similar on the others. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>>>>>>>>14.318MHz HPET.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>This appears to be the xen state, which is fine. >>>>>>>>>>>>>I was wrongly assuming that this was the guest state. >>>>>>>>>>>>>You might want to look in your guest logs and see >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>what they were >>>> >>>> >>>> >>>> >>>>>>>>>>>>>picking >>>>>>>>>>>>>for a clock source. >>>>>>>>>>>>> >>>>>>>>>>>>>Regards, >>>>>>>>>>>>>Dave >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>Dan Magenheimer wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>Thanks, I hadn't realized that! No wonder we didn't >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>see the same >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>improvement you saw! >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>Try specifying clock=pit on the linux boot line... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>I'm confused... do you mean "clocksource=pit" on the Xen >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>command line or >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>guest (or >> >> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>dom0?) command >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>line? Or both places? Since the tests take awhile, it >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>would be nice >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>to get this right the first time. Do the Xen and guest >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>clocksources need >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>to be the same? >>>>>>>>>>>>>> >>>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>>Dan >>>>>>>>>>>>>> >>>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>>*From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM >>>>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser >>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com; >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>deepak.patel@oracle.com; >>>> >>>> >>>> >>>> >>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>that disables >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>pending missed ticks >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Dan, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hpet timer does have a fairly large error, as I >>>>>>>>>>>>>>was >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>trying this >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> one recently. >>>>>>>>>>>>>> I don't remember what I got for error, but 1% sounds >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>about right. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> The problem is that hpet is not built on top of vpt.c, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>the module >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Keir and I did >>>>>>>>>>>>>> all the recent work in, for its periodic timer >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>needs. Try >> >> >>>>>>>>>>>>>> specifying clock=pit >>>>>>>>>>>>>> on the linux boot line. If it still picks the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>hpet, which it >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> might, let me know >>>>>>>>>>>>>> and I'll tell you how to get around this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Dave >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>-------------------------------------------------------------- >>>> >>>> >>>> >>>> >>>>>>>>>>>>>---------- >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> *From:* Dan Magenheimer >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>[mailto:dan.magenheimer@oracle.com] >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM >>>>>>>>>>>>>> *To:* Dave Winchell; Keir Fraser >>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>deepak.patel@oracle.com; >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> akira.ijuin@oracle.com >>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>that disables >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> pending missed ticks >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sorry for the very late followup on this but we finally >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>were able >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> to get our testing set up again on stable 3.1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>bits and have >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>order of 1%. >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> Test enviroment was a 4-socket dual core machine >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>with 24GB of >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>plus two pv. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> All six guests were running LTP simultaneously. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>The four hvm >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>RHEL4u5-64. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>32-bit guests. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> All four hvm guests experienced skew around -1%, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>even the 32-bit >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> guest. Less intensive testing didn't exhibit much >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>skew at all. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> A representative graph is attached. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dave, I wonder if some portion of your patches >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>didn't end up in >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> the xen trees? >>>>>>>>>>>>>> >>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>Platform timer >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> 14.318MHz HPET.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Dan >>>>>>>>>>>>>> >>>>>>>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests. >>>>>>>>>>>>>> >>>>>>>>>>>>>> > -----Original Message----- >>>>>>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > Dave Winchell >>>>>>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>>>>>>>>>>>> > To: Keir Fraser >>>>>>>>>>>>>> > Cc: dan.magenheimer@oracle.com; >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>xen-devel@lists.xensource.com; Dave >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > Winchell >>>>>>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>>>>>>>> > disables pending >>>>>>>>>>>>>> > missed ticks >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Hi Keir, >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > The latest change, c/s 16690, looks fine. >>>>>>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to >>>>>>>>>>>>>> > the code I submitted. Also, your version is more >>>>>>>>>>>>>> > concise. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > The error tests confirm the equivalence. With >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>overnight cpu loads, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > the checked in version was accurate to +.048% for sles >>>>>>>>>>>>>> > and +.038% for red hat. My version was +.046% >>>>>>>>>>>>>>and >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>+.032% in a >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > 2 hour test. >>>>>>>>>>>>>> > I don't think the difference is significant. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > i/o loads produced errors of +.01%. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Thanks for all your efforts on this issue. >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Regards, >>>>>>>>>>>>>> > Dave >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > Keir Fraser wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >Applied as c/s 16690, although the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>checked-in patch is >> >> >>>>>>>>>>>>>> > smaller. I think the >>>>>>>>>>>>>> > >only important fix is to pt_intr_post() and the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>only bit of >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > the patch I >>>>>>>>>>>>>> > >totally omitted was the change to >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>pt_process_missed_ticks(). >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > I don't think >>>>>>>>>>>>>> > >that change can be important, but let's see what >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>happens to the >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> error >>>>>>>>>>>>>> > >percentage... >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > -- Keir >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > >>Hi Dan and Keir, >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >>Attached is a patch that fixes some issues with the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>SYNC policy >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>(no_missed_ticks_pending). >>>>>>>>>>>>>> > >>I have not tried to make the change the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>minimal one, but, >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > rather, just >>>>>>>>>>>>>> > >>ported into >>>>>>>>>>>>>> > >>the new code what I know to work well. The error for >>>>>>>>>>>>>> > >>no_missed_ticks_pending goes from >>>>>>>>>>>>>> > >>over 3% to .03% with this change according >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>to my testing. >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >>Regards, >>>>>>>>>>>>>> > >>Dave >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >>Dan Magenheimer wrote: >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >>>Hi Dave -- >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>>Did you get your correction ported? If so, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>it would be >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > nice to see this get >>>>>>>>>>>>>> > >>>into 3.1.3. >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>>Note that I just did some very limited testing with >>>>>>>>>>>>>> > timer_mode=2(=SYNC=no >>>>>>>>>>>>>> > >>>missed ticks pending) >>>>>>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>guest) and the >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > worst error I've >>>>>>>>>>>>>> > >>>seen so far >>>>>>>>>>>>>> > >>>is 0.012%. But I haven't tried any exotic >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>loads, just LTP. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>>Thanks, >>>>>>>>>>>>>> > >>>Dan >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>>>-----Original Message----- >>>>>>>>>>>>>> > >>>>From: Dave Winchell >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>[mailto:dwinchell@virtualiron.com] >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>>>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com >>>>>>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>>>>>>>>>>>> > xen-devel@lists.xensource.com; Dong, >>>>>>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>>>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>timer mode that >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>disables pending >>>>>>>>>>>>>> > >>>>missed ticks >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>Dan, >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>I did some testing with the constant tsc offset >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>SYNC method >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>(now called >>>>>>>>>>>>>> > >>>>no_missed_ticks_pending) >>>>>>>>>>>>>> > >>>>and found the error to be very high, much larger >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>than 1 %, as >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>I recall. >>>>>>>>>>>>>> > >>>>I have not had a chance to submit a correction. I >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>will try to >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>do it later >>>>>>>>>>>>>> > >>>>this week or the first week in January. My >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>version of >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> constant tsc >>>>>>>>>>>>>> > >>>>offset SYNC method >>>>>>>>>>>>>> > >>>>produces .02 % error, so I just need to port >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>that into the >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>current code. >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>The error you got for both of those kernels is >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>what I would >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> expect >>>>>>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode. >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>Regards, >>>>>>>>>>>>>> > >>>>Dave >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>Dan Magenheimer wrote: >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>>Anyone make measurements on the final patch? >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>saw a loss of >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>about 0.2% with no load. This was >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>xen-unstable tip today >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>with no options specified. 32-bit was >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>about 0.01%. >> >> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>>I think I missed something... how do I >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>run the various >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>accounting choices and which ones are known to be >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>appropriate >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>for which kernels? >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>>Thanks, >>>>>>>>>>>>>> > >>>>>Dan >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>>>-----Original Message----- >>>>>>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>> >>>> >>>> >>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>Keir Fraser >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>>>>>>>>> > >>>>>>To: Dave Winchell >>>>>>>>>>>>>> > >>>>>>Cc: Shan, Haitao; >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>xen-devel@lists.xensource.com; Dong, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > Eddie; Jiang, >>>>>>>>>>>>>> > >>>>>>Yunhong >>>>>>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>mode that >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>disables pending >>>>>>>>>>>>>> > >>>>>>missed ticks >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>Please take a look at xen-unstable >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>changeset 16545. >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>-- Keir >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>> wrote: >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>Keir, >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>loads for the >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>various time protocols follows. In >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>addition, the data >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>for cpu loads is shown. >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>processor AMD >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> box. >>>>>>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>vcpu each. >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>>>>>>>> > >>>>>>>(usex is available at >>>>>>>>>>>>>> http://people.redhat.com/anderson/usex.) >>>>>>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>of=/dev/null. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>instances of dd. >> >> >>>>>>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>>>>>>>> > >>>>>>>All three guests are 8vcpu. >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>as i/o-32 >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>instances of dd. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, >>>>>>>>>>>>>>+4.42 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>sec -.006%, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > +.005% cpu >>>>>>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>sec, -.001%, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > +.012% cpu >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>sec, -.009%, >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> -.004% cpu >>>>>>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>-.005%, -.005% cpu >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>-.008%, -.003% cpu >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>-.040% cpu >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>sec, -.034%, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > -.031% cpu >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>sec,-55.7 sec, -.01%, >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > -.09% i/o-8 >>>>>>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>sec,-14.0 sec, -.015% >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > -.14% i/o-8 >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>sec, -.017%, >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > -.022% i/o-8 >>>>>>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>-.017%, -.018% i/o-8 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>sec, -.020%, >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > -.029% i/o-8 >>>>>>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>sec, -.023%, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > -.031% i/o-8 >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>-.04% i/o-32 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>sec, -.011%, >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > -.005% i/o-32 >>>>>>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>-.11% i/o-32 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>13. sec -.07%, >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > .003% i/o-4/32 >>>>>>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>sec, -.017%, >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > .01% i/o-4/32 >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>Overhead measurements: >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>Progress in terms of number of passes >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>through a fixed >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>system workload >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>>>>>>>> > >>>>>>>The workload was usex -b48. >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>Conclusions: >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>>>>>>>>>>>> > requirement for ntp >>>>>>>>>>>>>> > >>>>>>>tracking under the loads >>>>>>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>accuracies for >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>SYNC, MIXED, >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>and ASYNC >>>>>>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>method by only >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>scheduling the extra >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>wakeups if a certain number >>>>>>>>>>>>>> > >>>>>>>of ticks are missed. >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>Regards, >>>>>>>>>>>>>> > >>>>>>>Dave >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>Keir Fraser wrote: >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>> wrote: >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>ASYNC method a >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>couple of days ago, >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>there may have >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > been something >>>>>>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>days ago for >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>ASYNC. It may have been >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>after context >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>switch in. >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>or ASYNC give >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>acceptable accuracy, >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>each running consistently around or under >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>.01%. MIXED has >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>a fairly high >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>error of >>>>>>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>to .05% ntp >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>threshold for comfort. >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>plan to leave >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>SYNC running >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>leave MIXED >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>running instead. >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>I can run >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>more overnight tests >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>>next week. >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>effects of the >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>SYNC+run_timer >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>cause higher >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>system-wide CPU >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>contention. I find it easier to think >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>through the >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>implications of ASYNC. I'm >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>accurate than >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>ASYNC. Perhaps it >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>approaches, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>and each interrupt >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>event causes a small accumulated error? >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>favourites and >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>if the latter is >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>actually more accurate then I can >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>simply revert the >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > changeset that >>>>>>>>>>>>>> > >>>>>>>>implemented MIXED. >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>workloads you >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>could try idle >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>large disc reads >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>to /dev/null)? We >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>CPU bound, so >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>that's really an >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo. >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>>-- Keir >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>> > >>>>>>_______________________________________________ >>>>>>>>>>>>>> > >>>>>>Xen-devel mailing list >>>>>>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com >>>>>>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>>>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>2007 +0000 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>2008 -0500 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>pt_process_missed_ticks(stru >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> missed_ticks = missed_ticks / (s_time_t) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>pt->period + 1; >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >> if ( mode_is(pt->vcpu->domain, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>no_missed_ticks_pending) ) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>>>>>>>>>>> > >>+ pt->do_not_freeze = 1; >>>>>>>>>>>>>> > >> else >>>>>>>>>>>>>> > >> pt->pending_intr_nr += missed_ticks; >>>>>>>>>>>>>> > >> pt->scheduled += missed_ticks * pt->period; >>>>>>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>pt_timer_fn(void *data) >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> pt_lock(pt); >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >>- pt->pending_intr_nr++; >>>>>>>>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>no_missed_ticks_pending) ) { >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >>+ pt->pending_intr_nr = 1; >>>>>>>>>>>>>> > >>+ pt->do_not_freeze = 0; >>>>>>>>>>>>>> > >>+ } >>>>>>>>>>>>>> > >>+ else >>>>>>>>>>>>>> > >>+ pt->pending_intr_nr++; >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> if ( !pt->one_shot ) >>>>>>>>>>>>>> > >> { >>>>>>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>vcpu *v, struct >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >> return; >>>>>>>>>>>>>> > >> } >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >>- pt->do_not_freeze = 0; >>>>>>>>>>>>>> > >>- >>>>>>>>>>>>>> > >> if ( pt->one_shot ) >>>>>>>>>>>>>> > >> { >>>>>>>>>>>>>> > >> pt->enabled = 0; >>>>>>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>*v, struct >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> > >> pt->last_plt_gtime = >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>hvm_get_guest_time(v); >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > >> pt->pending_intr_nr = 0; /* >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>'collapse' all >>>> >>>> >>>> >>>> >>>>>>>>>>>>>> > missed ticks */ >>>>>>>>>>>>>> > >> } >>>>>>>>>>>>>> > >>+ else if ( mode_is(v->domain, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>no_missed_ticks_pending) ) { >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> > >>+ pt->pending_intr_nr--; >>>>>>>>>>>>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>>>>>>>> > >>+ } >>>>>>>>>>>>>> > >> else >>>>>>>>>>>>>> > >> { >>>>>>>>>>>>>> > >> pt->last_plt_gtime += >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>pt->period_cycles; >> >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > _______________________________________________ >>>>>>>>>>>>>> > Xen-devel mailing list >>>>>>>>>>>>>> > Xen-devel@lists.xensource.com >>>>>>>>>>>>>> > http://lists.xensource.com/xen-devel >>>>>>>>>>>>>> > >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>> >>>>> >>>>> >>>>-------------------------------------------------------------- >>>>---------- >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 19 Feb 2008 10:55:57 -0700 Message-ID: <20080219105557046.00000002392@djm-pc> References: <47BAF542.2020507@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <47BAF542.2020507@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org Hi Dave -- Thanks for that observation on ltp running on one vcpu! With "clocksource=3Dpit nohpet nopmtimer" our clock skew problems seem to have been reduced to a reasonable percentage. However, our 32-bit clock skew seems to show a measureable problem now. As a result, I've been doing some digging into kernel sources and have observed the following relative to RHEL4 (2.6.9-based) kernels and RHEL5 (2.6.18-based) kernels and thought I would document them for posterity. Some of our confusion arises from the fact that invalid command line parameters are silently ignored. RHEL4: - clock=3D is a valid parameter for RHEL4-32 - clocksource=3D is not a valid parameter for RHEL4-xx - nohpet is a valid parameter for RHEL4-64, not RHEL4-32 - nopmtimer is not a valid parameter for RHEL4-xx - notsc is a valid parameter for RHEL4-32, not RHEL4-64 - SMP vs UP RHEL4-64 reports timekeeping in dmesg differently For Xen RHEL4 HVM guests: - I *think* clock=3Dpit is sufficient for RHEL4-32 [1] - I *think* nohpet is sufficient for RHEL4-64 [1] RHEL5: - there are two kinds of timekeeping, WALL and gtod - clocksource=3D is a valid parameter for RHEL5-xx - clock=3D is a valid but deprecated parameter for RHEL5-xx - clock=3D and clocksource=3D are essentially equivalent - nohpet is a valid parameter for RHEL5-64, not RHEL5-32 - nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32 - notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1] - clock=3Dpit changes the gtod source but not the WALL source[2] - nohpet nopmtimer changes the WALL source to PIT - /sys/devices/system/clocksource/clocksource0/... available_clocksource lists the possible clock sources current_clocksource lists the chosen clock source ..but neither of these works in a RHEL5 guest! For Xen RHEL5 HVM guests: - I *think* clock=3Dpit is sufficient for RHEL5-32 - I *think* clock=3Dpit nohpet nopmtimer is sufficient for RHEL5-64 Other info: - As of 2.6.24.2, clock=3D is still valid (though still deprecated) So, some open questions: [1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? (I *think* not as it has never come up in any email.) [2] In RHEL5, I *think* it is the WALL source that we care about? And finally, since invalid command line parameters are ignored. I think specifying: clock=3Dpit nohpet nopmtimer will force the guest clock sources into the optimal state for all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the question above on tsc). And we should keep an eye on kernel/time/clocksource.c to ensure the __setup("clock=3D"...) line doesn't go away before RHEL6. Note that if hpet=3D0 and pmtimer=3D0 were the default hvm platform parameters for all xen hvm guests (on all versions of xen), specifying kernel command line parameters would be unnecessary, but c'est la vie. Oh, and to be complete, timer_mode=3D0 for 32-bit RHEL guests and timer_mode=3D2 for 64-bit RHEL guests. Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Tuesday, February 19, 2008 8:27 AM > To: dan.magenheimer@oracle.com > Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak > Patel > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > = > Hi Dan, > = > ltp runs by default loading up only one vcpu. > The -x option can be used to run multiple instances, though > in this mode you will get test failures. > I ran 8 instances on each guest for 16 hours, 25 min > and the time error was -11 sec (-.019%) on each guest. > = > Regards, > Dave > = > Dave Winchell wrote: > = > > Hi Dan, > > > > Mine was oversubscribed. > > 8 physical cpu, 2 guests, each with 8 vcpu. > > I ran one instance of ltp on each guest, continuously. I hope ltp > > loaded up all the vcpus. I seem to recall that it did, but I > > could be wrong. If it didn't, that would be a major difference > > between our tests. I'll verify this afternoon and run = > multiple instances, > > if necessary. > > > > Thanks, > > Dave > > > > > > > > Dan Magenheimer wrote: > > > >> Hi Dave -- > >> > >> No new results yet but one other question: > >> > >> The problems we've seen with our testing have been with a heavily > >> oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests > >> running LTP simultaneously. > >> > >> Was your LTP testing oversubscribed or just a single guest? > >> > >> Thanks, > >> Dan > >> > >> > >> > >>> -----Original Message----- > >>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>> Sent: Thursday, February 14, 2008 10:56 AM > >>> To: dan.magenheimer@oracle.com > >>> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave > >>> Winchell > >>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > >>> missed ticks > >>> > >>> > >>> Dan, > >>> > >>> Here are some boot snipets for rh4u564 on xen 3.2. > >>> > >>> > >>> #1: > >>> > >>> Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro > >>> root=3DLABEL=3D/ console=3DttyS0 clocksource=3Dpit nohpet) > >>> Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp > >>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version = > 3.4.6 20060404 > >>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 > >>> ... > >>> Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=3DLABEL=3D= / > >>> console=3DttyS0 clocksource=3Dpit nohpet > >>> Feb 14 10:44:59 vs076 kernel: Initializing CPU#0 > >>> Feb 14 10:44:59 vs076 kernel: PID hash table entries: = > 2048 (order: 11, > >>> 65536 bytes) > >>> Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer. > >>> Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 = > MHz processor. > >>> ... > >>> Feb 14 10:45:00 vs076 kernel: checking TSC = > synchronization across 8 > >>> CPUs: passed. > >>> Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs > >>> Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to = > use of PM timer > >>> Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping. > >>> > >>> > >>> > >>> #2: > >>> > >>> Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro > >>> root=3DLABEL=3D/ console=3DttyS0 clocksource=3Dpit nohpet nopmtimer) > >>> Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp > >>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version = > 3.4.6 20060404 > >>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 > >>> ... > >>> Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=3DLABEL=3D= / > >>> console=3DttyS0 clocksource=3Dpit nohpet nopmtimer > >>> Feb 14 10:47:58 vs076 kernel: Initializing CPU#0 > >>> Feb 14 10:47:58 vs076 kernel: PID hash table entries: = > 2048 (order: 11, > >>> 65536 bytes) > >>> Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz = > PIT timer. > >>> Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 = > MHz processor. > >>> ... > >>> Feb 14 10:47:59 vs076 kernel: checking TSC = > synchronization across 8 > >>> CPUs: passed. > >>> Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs > >>> Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based = > timekeeping. > >>> > >>> > >>> As you can see, I only get the pit if I specify nopmtimer. > >>> > >>> Dan Magenheimer wrote: > >>> > >>> > >>> > >>>> Hi Dave -- > >>>> > >>>> Thanks for continuing to run tests! > >>>> > >>>> Hmmm... I thought I had noticed that even though Linux will > >>> > >>> acknowledge > >>> > >>> > >>>> the existence of the pmtimer, it still prints: > >>>> > >>>> time.c: Using PIT/TSC based timekeeping. > >>>> > >>>> I will check again, but assuming the clocksource for our tests is > >>>> indeed pit, the huge difference in the results (yours vs ours) is > >>>> baffling. I wonder if the difference may be the = > underlying hardware. > >>>> Maybe we will try to ensure we can duplicate the results on > >>> > >>> a different > >>> > >>> > >>>> box. > >>>> > >>>> > >>>> So your testing was with stock 3.2.0 xen bits (what = > cset?) without > >>>> any of your [quote from below] "clock related tweaks = > that I haven't > >>>> submitted, because I'm still characterizing them"? > >>>> > >>>> > >>>> > >>> > >>> None of the tweaks I mentioned are in this test. > >>> It was stock with some patches. However, none of the = > patches are time > >>> related to > >>> my knowledge and I checked vpt.c to make sure that it is = > the same as > >>> what's in unstable. > >>> The only difference is in pt_intr_post, where I set the = > timer mode. > >>> I don't have timer mode tied into our config process yet, which > >>> is different than official xen method. > >>> > >>> > >>> (In pt_intr_post) > >>> else > >>> { > >>> + if(v->arch.paging.mode->guest_levels =3D=3D 4) > >>> + = > v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =3D > >>> HVMPTM_no_missed_ticks_pending; > >>> + else > >>> + = > v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =3D > >>> HVMPTM_delay_for_missed_ticks; > >>> if ( mode_is(v->domain, one_missed_tick_pending) || > >>> mode_is(v->domain, no_missed_ticks_pending) ) > >>> { > >>> > >>> > >>> > >>>> Could you also send detail on the rhel4u4-64 kernel you > >>>> are testing with, just to ensure we are not comparing apples > >>>> and oranges? (Perhaps there's some way we can even share the > >>>> identical disk image and vm.cfg file?) > >>>> > >>>> And if our problem is indeed the pmtimer, I will need to submit > >>>> another patch to Keir to add an hvm pmtimer platform variable. > >>>> (Hmmm... I don't think he's even accepted the hpet variable patch > >>>> yet. I'll have to check.) > >>>> > >>>> Thanks, > >>>> Dan > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>> -----Original Message----- > >>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>> Sent: Thursday, February 14, 2008 9:00 AM > >>>>> To: dan.magenheimer@oracle.com > >>>>> Cc: Dave Winchell; Keir Fraser; > >>>> > >>> xen-devel@lists.xensource.com; Deepak > >>> > >>> > >>>>> Patel > >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>> disables pending > >>>>> missed ticks > >>>>> > >>>>> > >>>>> Hi Dan, > >>>>> > >>>>> I ran the ltp tests with 3.2 and found the errors > >>>>> for a 16 hour run to be: > >>>>> > >>>>> rh4u564 -9.9 sec (-.017%) > >>>>> rh4u464 -7.3 sec (-.013%) > >>>>> > >>>>> There were no cliffs and the drift was linear. > >>>>> > >>>>> I think the problem you had may be due to the use of the > >>>>> pm timer. If you still have the boot log, it would tell you. > >>>>> > >>>>> When I first tried a guest on 3.2 with "clocksource=3Dpit nohpet" > >>>>> I noticed that it picked the pm timer. Adding "nopmtimer", the > >>>>> guest will pick the pit. > >>>>> > >>>>> The reason I didn't have the problem with our 3.1 base is that > >>>>> I had disabled the hpet and the pmtimer by not advertising them > >>>>> in the acpi tables. I did this so long ago, I forgot = > that I had to > >>>>> disable pmtimer as well as hpet. > >>>>> > >>>>> So, can you re-run your test with "clocksource=3Dpit nohpet > >>>> > >>> nopmtimer"? > >>> > >>> > >>>>> You should see this in the boot messages: > >>>>> > >>>>> time.c: Using PIT/TSC based timekeeping. > >>>>> > >>>>> Thanks, > >>>>> Dave > >>>>> > >>>>> > >>>>> Dave Winchell wrote: > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> Hi Dan, > >>>>>> > >>>>>> Over the weekend the drift was +18 seconds for each = > guest (no ntp). > >>>>>> The duration was 3900 minutes, so the error for each = > was +.0077%. > >>>>>> Looking back through the data, it appears to drift linearly at > >>>>>> this rate. I've attached a plot for rh4u5-64. > >>>>>> > >>>>>> This accuracy is better than what I've seen before (.03-.05%). > >>>>>> This may be due to the different load (ltp vs usex) or to > >>>>> > >>> one of the > >>> > >>> > >>>>>> changes I've made recently. I'll do some = > experimentation to see if > >>>>>> there is > >>>>>> a fix I should propose. > >>>>>> > >>>>>> This still doesn't address the radical drift you saw. > >>>>>> The next step for me is to run 3.2 and see if I can = > reproduce it. > >>>>>> > >>>>>> Regards, > >>>>>> Dave > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> Dave Winchell wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> Hi Dan, > >>>>>>> > >>>>>>> Sorry it took me so long, but I finally ran an ltp test today. > >>>>>>> Its on rh4u4-64. I'm using the defaults for ltp and = > using a script > >>>>>>> called runltp. I had a usex load on rh4u5-64. No ntpd. > >>>>>>> virtual processors / physical processors =3D 2. > >>>>>>> > >>>>>>> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in = > 300 minutes > >>>>>>> for -.005% and .008%. > >>>>>>> > >>>>>>> I'm running a 3.1 based hypervisor with some clock related > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> tweaks that > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>> I haven't submitted, because I'm still characterizing them. > >>>>>>> > >>>>>>> I'm stopping the usex load on 4u5-64 now and = > replacing it with ltp > >>>>>>> and will leave the two guests running ltp over the weekend. > >>>>>>> > >>>>>>> Regards, > >>>>>>> Dave > >>>>>>> > >>>>>>> > >>>>>>> Dave Winchell wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Hi Dan, Deepak: > >>>>>>>> > >>>>>>>> Thanks for the data. Those drifts are severe - no wonder > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> ntp couldn't > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>> keep then in synch. I'll try to reproduce that behaviour > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> here, with > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>> my code base. > >>>>>>>> If I can't reproduce it, I'll try 3.2. > >>>>>>>> > >>>>>>>> If you can isolate what ltp is doing during the cliffs, > >>>>>>>> > >>>>>>> > >>> that would > >>> > >>> > >>>>>>>> be very > >>>>>>>> helpful. > >>>>>>>> > >>>>>>>> thanks, > >>>>>>>> Dave > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Dan Magenheimer wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> OK, Deepak repeated the test without ntpd and using > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>> ntpdate -b before > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>> the test. > >>>>>>>>> > >>>>>>>>> The attached graph shows his results: el5u1-64 = > (best=3D~0.07%), > >>>>>>>>> el4u5-64 (middle=3D~0.2%), and el4u5-32 (worst=3D~0.3%). > >>>>>>>>> > >>>>>>>>> We will continue to look at LTP to try to isolate. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Dan > >>>>>>>>> > >>>>>>>>> P.S. elXuY is essentially RHEL XuY with some patches. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> -----Original Message----- > >>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>>>> Sent: Wednesday, January 30, 2008 2:45 PM > >>>>>>>>>> To: Deepak Patel > >>>>>>>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser; > >>>>>>>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> Dave Winchell > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode = > that disables > >>>>>>>>>> pending > >>>>>>>>>> missed ticks > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Dan, Deeepak, > >>>>>>>>>> > >>>>>>>>>> It may be that the underlying clock error is too = > great for ntp > >>>>>>>>>> to handle. It would be useful if you did not run ntpd > >>>>>>>>>> and, instead did ntpdate -b at the start > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> of the test > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>> for each guest. Then capture the data as you have = > been doing. > >>>>>>>>>> If the drift is greater than .05%, then we need to > >>>>>>>>>> > >>>>>>>>> > >>> address that. > >>> > >>> > >>>>>>>>>> Another option is, when running ntpd, to enable loop > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> statistics in > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>> /etc/ntp.conf > >>>>>>>>>> by adding this to the file: > >>>>>>>>>> > >>>>>>>>>> statistics loopstats > >>>>>>>>>> statsdir /var/lib/ntp/ > >>>>>>>>>> > >>>>>>>>>> Then you will see loop data in that directory. > >>>>>>>>>> Correlating the data in the loopstats files with the > >>>>>>>>>> peaks in skew would be interesting. You will see > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> entries of the form > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> 239.735511 10 > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>> Where the second to last column is the Allan Deviation. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> When that > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>> gets over 1000, ntpd is working pretty hard. However, > >>>>>>>>>> > >>>>>>>>> > >>> I have not > >>> > >>> > >>>>>>>>>> seen ntpd > >>>>>>>>>> completely loose it like you have. > >>>>>>>>>> > >>>>>>>>>> I'm on vacation until Monday, and won't be reading > >>>>>>>>>> email. > >>>>>>>>>> > >>>>>>>>>> Thanks for all your work on this! > >>>>>>>>>> > >>>>>>>>>> -Dave > >>>>>>>>>> > >>>>>>>>>> Deepak Patel wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>> Is the graph for RHEL5u1-64? (I've never tested = > this one.) > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I do not know which graph was attached with this. But > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>> I saw this > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>> guests when I > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>> was running ltp tests continuously. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> What was the behaviour of the other guests running? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> All pvm guests are fine. But behavior of most of the > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>> hvm guests were > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>> as described. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> If they had spikes, were they at the same wall time? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> No. They are not at the same wall time. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> Were the other guests running ltp as well? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are = > running ltp > >>>>>>>>>>> continuously. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> How are you measuring skew? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I was collecting output of "ntpdate -q every > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> 300 seconds > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> (5 minutes) and have created graph based on that. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> Are you running ntpd? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Yes. ntp was running on all the guests. > >>>>>>>>>>> > >>>>>>>>>>> I am investigating what causes this spikes and = > let everyone > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> know what > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> are my findings. > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Deepak > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> Anything that you can discover that would be in sync with > >>>>>>>>>>>> the spikes would be very helpful! > >>>>>>>>>>>> > >>>>>>>>>>>> The code that I test with is our product code, = > which is based > >>>>>>>>>>>> on 3.1. So it is possible that something in 3.2 other > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>> than vpt.c > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>> is the cause. I can test with 3.2, if necessary. > >>>>>>>>>>>> > >>>>>>>>>>>> thanks, > >>>>>>>>>>>> Dave > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Dan Magenheimer wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Dave (Keir, see suggestion below) -- > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>> > >>>>>>>>>>>>> Turning off vhpet certainly helps a lot (though = > see below). > >>>>>>>>>>>>> > >>>>>>>>>>>>> I wonder if timekeeping with vhpet is so bad that it > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>> should be > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> turned off by default (in 3.1, 3.2, and unstable) > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>> until it is > >>> > >>> > >>>>>>>>>>>>> fixed? (I have a patch that defaults it off, = > can post it if > >>>>>>>>>>>>> there is agreement on the above point.) The whole > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>> point of an > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> HPET is to provide more precise timekeeping and = > if vhpet is > >>>>>>>>>>>>> worse than vpit, it can only confuse users. Comments? > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> In your testing, are you just measuring % skew = > over a long > >>>>>>>>>>>>> period of time? > >>>>>>>>>>>>> We are graphing the skew continuously and > >>>>>>>>>>>>> seeing periodic behavior that is unsettling, = > even with pit. > >>>>>>>>>>>>> See attached. Though your algorithm recovers, = > the "cliffs" > >>>>>>>>>>>>> could still cause real user problems. I wonder = > if there is > >>>>>>>>>>>>> anything that can be done to make the "recovery" more > >>>>>>>>>>>>> responsive? > >>>>>>>>>>>>> > >>>>>>>>>>>>> We are looking into what part(s) of LTP is causing > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>> the cliffs. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Dan > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM > >>>>>>>>>>>>>> To: dan.magenheimer@oracle.com > >>>>>>>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; > >>>>>>>>>>>>>> deepak.patel@oracle.com; > >>>>>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell > >>>>>>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>> that disables > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>> pending > >>>>>>>>>>>>>> missed ticks > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Dan, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I guess I'm a bit out of date calling for clock=3D usage. > >>>>>>>>>>>>>> Looking at linux 2.6.20.4 sources, I think you > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>> should specify > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>> "clocksource=3Dpit nohpet" on the linux guest bootline. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> You can leave the xen and dom0 bootlines as they are. > >>>>>>>>>>>>>> The xen and guest clocksources do not need to = > be the same. > >>>>>>>>>>>>>> In my tests, xen is using the hpet for its = > timekeeping and > >>>>>>>>>>>>>> that appears to be the default. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> When you boot the guests you should see > >>>>>>>>>>>>>> time.c: Using PIT/TSC based timekeeping. > >>>>>>>>>>>>>> on the rh4u5-64 guest, and something similar = > on the others. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, = > Platform timer > >>>>>>>>>>>>>>> 14.318MHz HPET.) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This appears to be the xen state, which is fine. > >>>>>>>>>>>>>> I was wrongly assuming that this was the guest state. > >>>>>>>>>>>>>> You might want to look in your guest logs and see > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>> what they were > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>> picking > >>>>>>>>>>>>>> for a clock source. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>> Dave > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Dan Magenheimer wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, I hadn't realized that! No wonder we didn't > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> see the same > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> improvement you saw! > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Try specifying clock=3Dpit on the linux boot line... > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'm confused... do you mean "clocksource=3Dpit" = > on the Xen > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> command line or > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> "nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>> guest (or > >>> > >>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> dom0?) command > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> line? Or both places? Since the tests take = > awhile, it > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> would be nice > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> to get this right the first time. Do the Xen = > and guest > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> clocksources need > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> to be the same? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> Dan > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>>>> *From:* Dave Winchell = > [mailto:dwinchell@virtualiron.com] > >>>>>>>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM > >>>>>>>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser > >>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> deepak.patel@oracle.com; > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell > >>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> that disables > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> pending missed ticks > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Dan, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hpet timer does have a fairly large error, as I > >>>>>>>>>>>>>>> was > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> trying this > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> one recently. > >>>>>>>>>>>>>>> I don't remember what I got for error, but 1% sounds > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> about right. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The problem is that hpet is not built on top of vpt.c, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> the module > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Keir and I did > >>>>>>>>>>>>>>> all the recent work in, for its periodic timer > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>> needs. Try > >>> > >>> > >>>>>>>>>>>>>>> specifying clock=3Dpit > >>>>>>>>>>>>>>> on the linux boot line. If it still picks the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> hpet, which it > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> might, let me know > >>>>>>>>>>>>>>> and I'll tell you how to get around this. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>> Dave > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>> -------------------------------------------------------------- > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>> ---------- > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> *From:* Dan Magenheimer > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> [mailto:dan.magenheimer@oracle.com] > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM > >>>>>>>>>>>>>>> *To:* Dave Winchell; Keir Fraser > >>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> deepak.patel@oracle.com; > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> akira.ijuin@oracle.com > >>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> that disables > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> pending missed ticks > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Sorry for the very late followup on this but = > we finally > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> were able > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> to get our testing set up again on stable 3.1 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> bits and have > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> order of 1%. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> Test enviroment was a 4-socket dual core machine > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> with 24GB of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> plus two pv. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> All six guests were running LTP simultaneously. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> The four hvm > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> RHEL4u5-64. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 32-bit guests. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> All four hvm guests experienced skew around -1%, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> even the 32-bit > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> guest. Less intensive testing didn't exhibit much > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> skew at all. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> A representative graph is attached. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Dave, I wonder if some portion of your patches > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> didn't end up in > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> the xen trees? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> Platform timer > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> 14.318MHz HPET.) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> Dan > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> P.S. Many thanks to Deepak and Akira for = > running tests. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > -----Original Message----- > >>>>>>>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > Dave Winchell > >>>>>>>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM > >>>>>>>>>>>>>>> > To: Keir Fraser > >>>>>>>>>>>>>>> > Cc: dan.magenheimer@oracle.com; > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> xen-devel@lists.xensource.com; Dave > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Winchell > >>>>>>>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a = > timer mode that > >>>>>>>>>>>>>>> > disables pending > >>>>>>>>>>>>>>> > missed ticks > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > Hi Keir, > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > The latest change, c/s 16690, looks fine. > >>>>>>>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to > >>>>>>>>>>>>>>> > the code I submitted. Also, your version is more > >>>>>>>>>>>>>>> > concise. > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > The error tests confirm the equivalence. With > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> overnight cpu loads, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > the checked in version was accurate to = > +.048% for sles > >>>>>>>>>>>>>>> > and +.038% for red hat. My version was +.046% > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> +.032% in a > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > 2 hour test. > >>>>>>>>>>>>>>> > I don't think the difference is significant. > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > i/o loads produced errors of +.01%. > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > Thanks for all your efforts on this issue. > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > Regards, > >>>>>>>>>>>>>>> > Dave > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > Keir Fraser wrote: > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > >Applied as c/s 16690, although the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>> checked-in patch is > >>> > >>> > >>>>>>>>>>>>>>> > smaller. I think the > >>>>>>>>>>>>>>> > >only important fix is to pt_intr_post() and the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> only bit of > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > the patch I > >>>>>>>>>>>>>>> > >totally omitted was the change to > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> pt_process_missed_ticks(). > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > I don't think > >>>>>>>>>>>>>>> > >that change can be important, but let's see what > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> happens to the > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> error > >>>>>>>>>>>>>>> > >percentage... > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > -- Keir > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > >>Hi Dan and Keir, > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >>Attached is a patch that fixes some = > issues with the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> SYNC policy > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>(no_missed_ticks_pending). > >>>>>>>>>>>>>>> > >>I have not tried to make the change the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> minimal one, but, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > rather, just > >>>>>>>>>>>>>>> > >>ported into > >>>>>>>>>>>>>>> > >>the new code what I know to work well. = > The error for > >>>>>>>>>>>>>>> > >>no_missed_ticks_pending goes from > >>>>>>>>>>>>>>> > >>over 3% to .03% with this change according > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> to my testing. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >>Regards, > >>>>>>>>>>>>>>> > >>Dave > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >>Dan Magenheimer wrote: > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >>>Hi Dave -- > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>>Did you get your correction ported? If so, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> it would be > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > nice to see this get > >>>>>>>>>>>>>>> > >>>into 3.1.3. > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>>Note that I just did some very limited = > testing with > >>>>>>>>>>>>>>> > timer_mode=3D2(=3DSYNC=3Dno > >>>>>>>>>>>>>>> > >>>missed ticks pending) > >>>>>>>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> guest) and the > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > worst error I've > >>>>>>>>>>>>>>> > >>>seen so far > >>>>>>>>>>>>>>> > >>>is 0.012%. But I haven't tried any exotic > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> loads, just LTP. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>>Thanks, > >>>>>>>>>>>>>>> > >>>Dan > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>>>-----Original Message----- > >>>>>>>>>>>>>>> > >>>>From: Dave Winchell > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> [mailto:dwinchell@virtualiron.com] > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>>>>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com > >>>>>>>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; > >>>>>>>>>>>>>>> > xen-devel@lists.xensource.com; Dong, > >>>>>>>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>>>>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> timer mode that > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>disables pending > >>>>>>>>>>>>>>> > >>>>missed ticks > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>Dan, > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>I did some testing with the constant tsc offset > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> SYNC method > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>(now called > >>>>>>>>>>>>>>> > >>>>no_missed_ticks_pending) > >>>>>>>>>>>>>>> > >>>>and found the error to be very high, much larger > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> than 1 %, as > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>I recall. > >>>>>>>>>>>>>>> > >>>>I have not had a chance to submit a = > correction. I > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> will try to > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>do it later > >>>>>>>>>>>>>>> > >>>>this week or the first week in January. My > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> version of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> constant tsc > >>>>>>>>>>>>>>> > >>>>offset SYNC method > >>>>>>>>>>>>>>> > >>>>produces .02 % error, so I just need to port > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> that into the > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>current code. > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>The error you got for both of those kernels is > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> what I would > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> expect > >>>>>>>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>I'll let Keir answer on how to set the = > time mode. > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>Regards, > >>>>>>>>>>>>>>> > >>>>Dave > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>Dan Magenheimer wrote: > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>>Anyone make measurements on the final patch? > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> saw a loss of > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>about 0.2% with no load. This was > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> xen-unstable tip today > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>with no options specified. 32-bit was > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>> about 0.01%. > >>> > >>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>>I think I missed something... how do I > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> run the various > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>accounting choices and which ones are = > known to be > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> appropriate > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>for which kernels? > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>>Thanks, > >>>>>>>>>>>>>>> > >>>>>Dan > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>>-----Original Message----- > >>>>>>>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>Keir Fraser > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>>>>>>>>>>>> > >>>>>>To: Dave Winchell > >>>>>>>>>>>>>>> > >>>>>>Cc: Shan, Haitao; > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> xen-devel@lists.xensource.com; Dong, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > Eddie; Jiang, > >>>>>>>>>>>>>>> > >>>>>>Yunhong > >>>>>>>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> mode that > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>disables pending > >>>>>>>>>>>>>>> > >>>>>>missed ticks > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>Please take a look at xen-unstable > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> changeset 16545. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>-- Keir > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>> wrote: > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>Keir, > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> loads for the > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>various time protocols follows. In > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> addition, the data > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>for cpu loads is shown. > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> processor AMD > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> box. > >>>>>>>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> vcpu each. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. > >>>>>>>>>>>>>>> > >>>>>>>(usex is available at > >>>>>>>>>>>>>>> http://people.redhat.com/anderson/usex.) > >>>>>>>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> of=3D/dev/null. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>> instances of dd. > >>> > >>> > >>>>>>>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. > >>>>>>>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. > >>>>>>>>>>>>>>> > >>>>>>>All three guests are 8vcpu. > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> as i/o-32 > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> instances of dd. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, > >>>>>>>>>>>>>>> +4.42 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> sec -.006%, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > +.005% cpu > >>>>>>>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> sec, -.001%, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > +.012% cpu > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> sec, -.009%, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> -.004% cpu > >>>>>>>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -.005%, -.005% cpu > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -.008%, -.003% cpu > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -.040% cpu > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> sec, -.034%, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > -.031% cpu > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> sec,-55.7 sec, -.01%, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > -.09% i/o-8 > >>>>>>>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> sec,-14.0 sec, -.015% > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > -.14% i/o-8 > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> sec, -.017%, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > -.022% i/o-8 > >>>>>>>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -.017%, -.018% i/o-8 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> sec, -.020%, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > -.029% i/o-8 > >>>>>>>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> sec, -.023%, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > -.031% i/o-8 > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 = > sec, -.12%, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -.04% i/o-32 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> sec, -.011%, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > -.005% i/o-32 > >>>>>>>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 = > sec -.10%, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -.11% i/o-32 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> 13. sec -.07%, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > .003% i/o-4/32 > >>>>>>>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> sec, -.017%, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > .01% i/o-4/32 > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>Overhead measurements: > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>Progress in terms of number of passes > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> through a fixed > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>system workload > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu = > sles idle. > >>>>>>>>>>>>>>> > >>>>>>>The workload was usex -b48. > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>>>>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>>>>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>>>>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>Conclusions: > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>The only protocol which meets the = > .05% accuracy > >>>>>>>>>>>>>>> > requirement for ntp > >>>>>>>>>>>>>>> > >>>>>>>tracking under the loads > >>>>>>>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> accuracies for > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>SYNC, MIXED, > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>and ASYNC > >>>>>>>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> method by only > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>scheduling the extra > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>wakeups if a certain number > >>>>>>>>>>>>>>> > >>>>>>>of ticks are missed. > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>Regards, > >>>>>>>>>>>>>>> > >>>>>>>Dave > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>Keir Fraser wrote: > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> ASYNC method a > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>couple of days ago, > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> there may have > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > been something > >>>>>>>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> days ago for > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>ASYNC. It may have been > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> after context > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>switch in. > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> or ASYNC give > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>acceptable accuracy, > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>each running consistently around or under > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> .01%. MIXED has > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>a fairly high > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>error of > >>>>>>>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> to .05% ntp > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>threshold for comfort. > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> plan to leave > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>SYNC running > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> leave MIXED > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>running instead. > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>It may be too early to pick the = > protocol and > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I can run > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>more overnight tests > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>next week. > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> effects of the > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>SYNC+run_timer > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> cause higher > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>system-wide CPU > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>contention. I find it easier to think > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> through the > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>implications of ASYNC. I'm > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> accurate than > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>ASYNC. Perhaps it > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>delivers more timer interrupts than = > the other > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> approaches, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>and each interrupt > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>event causes a small accumulated error? > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> favourites and > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>if the latter is > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>actually more accurate then I can > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> simply revert the > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > changeset that > >>>>>>>>>>>>>>> > >>>>>>>>implemented MIXED. > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> workloads you > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>could try idle > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> large disc reads > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>to /dev/null)? We > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> CPU bound, so > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>that's really an > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo. > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>-- Keir > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > = > >>>>>>_______________________________________________ > >>>>>>>>>>>>>>> > >>>>>>Xen-devel mailing list > >>>>>>>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com > >>>>>>>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>> > >>>>>>>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>>>>>>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> 2007 +0000 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> 2008 -0500 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> pt_process_missed_ticks(stru > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >> missed_ticks =3D missed_ticks / (s_time_t) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> pt->period + 1; > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >> if ( mode_is(pt->vcpu->domain, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> no_missed_ticks_pending) ) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>>>>>>>>>>>>>> > >>+ pt->do_not_freeze =3D 1; > >>>>>>>>>>>>>>> > >> else > >>>>>>>>>>>>>>> > >> pt->pending_intr_nr +=3D missed_ticks; > >>>>>>>>>>>>>>> > >> pt->scheduled +=3D missed_ticks * pt->period; > >>>>>>>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> pt_timer_fn(void *data) > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >> pt_lock(pt); > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >>- pt->pending_intr_nr++; > >>>>>>>>>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> no_missed_ticks_pending) ) { > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr =3D 1; > >>>>>>>>>>>>>>> > >>+ pt->do_not_freeze =3D 0; > >>>>>>>>>>>>>>> > >>+ } > >>>>>>>>>>>>>>> > >>+ else > >>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr++; > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >> if ( !pt->one_shot ) > >>>>>>>>>>>>>>> > >> { > >>>>>>>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> vcpu *v, struct > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >> return; > >>>>>>>>>>>>>>> > >> } > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >>- pt->do_not_freeze =3D 0; > >>>>>>>>>>>>>>> > >>- > >>>>>>>>>>>>>>> > >> if ( pt->one_shot ) > >>>>>>>>>>>>>>> > >> { > >>>>>>>>>>>>>>> > >> pt->enabled =3D 0; > >>>>>>>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> *v, struct > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >> pt->last_plt_gtime =3D > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> hvm_get_guest_time(v); > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > >> pt->pending_intr_nr =3D 0; /* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>> 'collapse' all > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>> > missed ticks */ > >>>>>>>>>>>>>>> > >> } > >>>>>>>>>>>>>>> > >>+ else if ( mode_is(v->domain, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>> no_missed_ticks_pending) ) { > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr--; > >>>>>>>>>>>>>>> > >>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>>>>>>>>>>>>>> > >>+ } > >>>>>>>>>>>>>>> > >> else > >>>>>>>>>>>>>>> > >> { > >>>>>>>>>>>>>>> > >> pt->last_plt_gtime +=3D > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>> pt->period_cycles; > >>> > >>> > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > >> > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > _______________________________________________ > >>>>>>>>>>>>>>> > Xen-devel mailing list > >>>>>>>>>>>>>>> > Xen-devel@lists.xensource.com > >>>>>>>>>>>>>>> > http://lists.xensource.com/xen-devel > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> -------------------------------------------------------------- > >>>>> ---------- > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>> > >>> > >> > >> > >> > >> > > > = > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 19 Feb 2008 19:29:39 +0000 Message-ID: References: <20080219105557046.00000002392@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080219105557046.00000002392@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" , Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org On 19/2/08 17:55, "Dan Magenheimer" wrote: > Note that if hpet=0 and pmtimer=0 were the default hvm platform > parameters for all xen hvm guests (on all versions of xen), > specifying kernel command line parameters would be unnecessary, > but c'est la vie. I will get round to pmtimer at some point. I haven't put the hpet patch into 3.2 or 3.1 because it did not trivially backport. Someone else will have to do that work if they want it (including 3.2, if you want 3.1!). -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 19 Feb 2008 16:38:06 -0700 Message-ID: <20080219163806078.00000002392@djm-pc> References: <47BB410F.8000904@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <47BB410F.8000904@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org > >percentage. However, our 32-bit clock skew seems to > >show a measureable problem now. > > > For the 32 bit guest, which timesource did it pick? The dmesg output is hard to interpret on a 32-bit guest, but based on what we've seen, I think it was selecting hpet as timesource (because we specified clocksource=3Dpit which would have been ignored on RHEL4-32). We are running another test with "clock=3Dpit" to see if the skew goes away. > >For Xen RHEL5 HVM guests: > >- I *think* clock=3Dpit is sufficient for RHEL5-32 > > > But still poor accuracy, right? Unproven yet but I hope not. The nohpet and nopmtimer parameters are ignored on RHEL5-32 so the clock=3Dpit (or clocksource=3Dpit) is the only way to choose the clock source, and thus the only way to get good accuracy on RHEL5-32. Oops, I see from my long list that I neglected to say that the two clocks (WALL and GTOD) on RHEL5 are only reported on RHEL5-64. RHELx-32 looks to have only the one, which is overridden with clock=3D. > >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? > > (I *think* not as it has never come up in any email.) > > > I have not investigated this yet. My *think* is based on: 1) observation of dmesg output for RHEL5-64 where specifying "nohpet nopmtimer" seems to select PIT for WALL timer; 2) no mention of tsc in the generic clocksource.c code nor in the i386-specific time code. Still I would sleep better if this were definitive. > >[2] In RHEL5, I *think* it is the WALL source that we care about? > > > I'll have to check on this too. My *think* is based on our observations to date that clock=3Dpit is insufficient to fix the skew problem (and doesn't change the dmesg WALL source output on RHELx-64)... nohpet and nopmtimer is required to change the WALL source output and fix the skew. Again I would sleep better if this were definitive. Thanks, Dan > -----Original Message----- > From: Dave Winchell [mailto:dwinchell@virtualiron.com] > Sent: Tuesday, February 19, 2008 1:50 PM > To: dan.magenheimer@oracle.com > Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave > Winchell > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Hi Dan, > = > Thanks for all the investigation you've done! > = > -Dave > = > = > Dan Magenheimer wrote: > = > >Hi Dave -- > > > >Thanks for that observation on ltp running on one vcpu! > > > >With "clocksource=3Dpit nohpet nopmtimer" our clock skew > >problems seem to have been reduced to a reasonable > >percentage. However, our 32-bit clock skew seems to > >show a measureable problem now. > > > For the 32 bit guest, which timesource did it pick? > = > > As a result, > >I've been doing some digging into kernel sources and > >have observed the following relative to RHEL4 (2.6.9-based) > >kernels and RHEL5 (2.6.18-based) kernels and thought I > >would document them for posterity. Some of > >our confusion arises from the fact that invalid command > >line parameters are silently ignored. > > > >RHEL4: > >- clock=3D is a valid parameter for RHEL4-32 > >- clocksource=3D is not a valid parameter for RHEL4-xx > >- nohpet is a valid parameter for RHEL4-64, not RHEL4-32 > >- nopmtimer is not a valid parameter for RHEL4-xx > >- notsc is a valid parameter for RHEL4-32, not RHEL4-64 > >- SMP vs UP RHEL4-64 reports timekeeping in dmesg differently > > > >For Xen RHEL4 HVM guests: > >- I *think* clock=3Dpit is sufficient for RHEL4-32 [1] > >- I *think* nohpet is sufficient for RHEL4-64 [1] > > > >RHEL5: > >- there are two kinds of timekeeping, WALL and gtod > >- clocksource=3D is a valid parameter for RHEL5-xx > >- clock=3D is a valid but deprecated parameter for RHEL5-xx > >- clock=3D and clocksource=3D are essentially equivalent > >- nohpet is a valid parameter for RHEL5-64, not RHEL5-32 > >- nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32 > >- notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1] > >- clock=3Dpit changes the gtod source but not the WALL source[2] > >- nohpet nopmtimer changes the WALL source to PIT > >- /sys/devices/system/clocksource/clocksource0/... > > available_clocksource lists the possible clock sources > > current_clocksource lists the chosen clock source > > ..but neither of these works in a RHEL5 guest! > > > >For Xen RHEL5 HVM guests: > >- I *think* clock=3Dpit is sufficient for RHEL5-32 > > > > > But still poor accuracy, right? > = > >- I *think* clock=3Dpit nohpet nopmtimer is sufficient for RHEL5-64 > > > >Other info: > >- As of 2.6.24.2, clock=3D is still valid (though still deprecated) > > > >So, some open questions: > >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? > > (I *think* not as it has never come up in any email.) > > > > > I have not investigated this yet. > = > >[2] In RHEL5, I *think* it is the WALL source that we care about? > > > > > I'll have to check on this too. > = > >And finally, since invalid command line parameters are ignored. > >I think specifying: > > clock=3Dpit nohpet nopmtimer > >will force the guest clock sources into the optimal state for > >all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the > >question above on tsc). And we should keep an eye on > >kernel/time/clocksource.c to ensure the __setup("clock=3D"...) > >line doesn't go away before RHEL6. > > > >Note that if hpet=3D0 and pmtimer=3D0 were the default hvm platform > >parameters for all xen hvm guests (on all versions of xen), > >specifying kernel command line parameters would be unnecessary, > >but c'est la vie. > > > >Oh, and to be complete, timer_mode=3D0 for 32-bit RHEL guests > >and timer_mode=3D2 for 64-bit RHEL guests. > > > >Thanks, > >Dan > > > > > > > >>-----Original Message----- > >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>Sent: Tuesday, February 19, 2008 8:27 AM > >>To: dan.magenheimer@oracle.com > >>Cc: Dave Winchell; Keir Fraser; = > xen-devel@lists.xensource.com; Deepak > >>Patel > >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>disables pending > >>missed ticks > >> > >> > >> > >>Hi Dan, > >> > >>ltp runs by default loading up only one vcpu. > >>The -x option can be used to run multiple instances, though > >>in this mode you will get test failures. > >>I ran 8 instances on each guest for 16 hours, 25 min > >>and the time error was -11 sec (-.019%) on each guest. > >> > >>Regards, > >>Dave > >> > >>Dave Winchell wrote: > >> > >> > >> > >>>Hi Dan, > >>> > >>>Mine was oversubscribed. > >>>8 physical cpu, 2 guests, each with 8 vcpu. > >>>I ran one instance of ltp on each guest, continuously. I hope ltp > >>>loaded up all the vcpus. I seem to recall that it did, but I > >>>could be wrong. If it didn't, that would be a major difference > >>>between our tests. I'll verify this afternoon and run > >>> > >>> > >>multiple instances, > >> > >> > >>>if necessary. > >>> > >>>Thanks, > >>>Dave > >>> > >>> > >>> > >>>Dan Magenheimer wrote: > >>> > >>> > >>> > >>>>Hi Dave -- > >>>> > >>>>No new results yet but one other question: > >>>> > >>>>The problems we've seen with our testing have been with a heavily > >>>>oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests > >>>>running LTP simultaneously. > >>>> > >>>>Was your LTP testing oversubscribed or just a single guest? > >>>> > >>>>Thanks, > >>>>Dan > >>>> > >>>> > >>>> > >>>> > >>>> > >>>>>-----Original Message----- > >>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>Sent: Thursday, February 14, 2008 10:56 AM > >>>>>To: dan.magenheimer@oracle.com > >>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak = > Patel; Dave > >>>>>Winchell > >>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>> > >>>>> > >>disables pending > >> > >> > >>>>>missed ticks > >>>>> > >>>>> > >>>>>Dan, > >>>>> > >>>>>Here are some boot snipets for rh4u564 on xen 3.2. > >>>>> > >>>>> > >>>>>#1: > >>>>> > >>>>>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro > >>>>>root=3DLABEL=3D/ console=3DttyS0 clocksource=3Dpit nohpet) > >>>>>Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp > >>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version > >>>>> > >>>>> > >>3.4.6 20060404 > >> > >> > >>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 > >>>>>... > >>>>>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro = > root=3DLABEL=3D/ > >>>>>console=3DttyS0 clocksource=3Dpit nohpet > >>>>>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0 > >>>>>Feb 14 10:44:59 vs076 kernel: PID hash table entries: > >>>>> > >>>>> > >>2048 (order: 11, > >> > >> > >>>>>65536 bytes) > >>>>>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz = > PM timer. > >>>>>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 > >>>>> > >>>>> > >>MHz processor. > >> > >> > >>>>>... > >>>>>Feb 14 10:45:00 vs076 kernel: checking TSC > >>>>> > >>>>> > >>synchronization across 8 > >> > >> > >>>>>CPUs: passed. > >>>>>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs > >>>>>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to > >>>>> > >>>>> > >>use of PM timer > >> > >> > >>>>>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping. > >>>>> > >>>>> > >>>>> > >>>>>#2: > >>>>> > >>>>>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro > >>>>>root=3DLABEL=3D/ console=3DttyS0 clocksource=3Dpit nohpet nopmtimer)= > >>>>>Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp > >>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version > >>>>> > >>>>> > >>3.4.6 20060404 > >> > >> > >>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 > >>>>>... > >>>>>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro = > root=3DLABEL=3D/ > >>>>>console=3DttyS0 clocksource=3Dpit nohpet nopmtimer > >>>>>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0 > >>>>>Feb 14 10:47:58 vs076 kernel: PID hash table entries: > >>>>> > >>>>> > >>2048 (order: 11, > >> > >> > >>>>>65536 bytes) > >>>>>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz > >>>>> > >>>>> > >>PIT timer. > >> > >> > >>>>>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 > >>>>> > >>>>> > >>MHz processor. > >> > >> > >>>>>... > >>>>>Feb 14 10:47:59 vs076 kernel: checking TSC > >>>>> > >>>>> > >>synchronization across 8 > >> > >> > >>>>>CPUs: passed. > >>>>>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs > >>>>>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based > >>>>> > >>>>> > >>timekeeping. > >> > >> > >>>>>As you can see, I only get the pit if I specify nopmtimer. > >>>>> > >>>>>Dan Magenheimer wrote: > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>Hi Dave -- > >>>>>> > >>>>>>Thanks for continuing to run tests! > >>>>>> > >>>>>>Hmmm... I thought I had noticed that even though Linux will > >>>>>> > >>>>>> > >>>>>acknowledge > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>the existence of the pmtimer, it still prints: > >>>>>> > >>>>>>time.c: Using PIT/TSC based timekeeping. > >>>>>> > >>>>>>I will check again, but assuming the clocksource for = > our tests is > >>>>>>indeed pit, the huge difference in the results (yours = > vs ours) is > >>>>>>baffling. I wonder if the difference may be the > >>>>>> > >>>>>> > >>underlying hardware. > >> > >> > >>>>>>Maybe we will try to ensure we can duplicate the results on > >>>>>> > >>>>>> > >>>>>a different > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>box. > >>>>>> > >>>>>> > >>>>>>So your testing was with stock 3.2.0 xen bits (what > >>>>>> > >>>>>> > >>cset?) without > >> > >> > >>>>>>any of your [quote from below] "clock related tweaks > >>>>>> > >>>>>> > >>that I haven't > >> > >> > >>>>>>submitted, because I'm still characterizing them"? > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>None of the tweaks I mentioned are in this test. > >>>>>It was stock with some patches. However, none of the > >>>>> > >>>>> > >>patches are time > >> > >> > >>>>>related to > >>>>>my knowledge and I checked vpt.c to make sure that it is > >>>>> > >>>>> > >>the same as > >> > >> > >>>>>what's in unstable. > >>>>>The only difference is in pt_intr_post, where I set the > >>>>> > >>>>> > >>timer mode. > >> > >> > >>>>>I don't have timer mode tied into our config process yet, which > >>>>>is different than official xen method. > >>>>> > >>>>> > >>>>>(In pt_intr_post) > >>>>> else > >>>>> { > >>>>>+ if(v->arch.paging.mode->guest_levels =3D=3D 4) > >>>>>+ > >>>>> > >>>>> > >>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =3D > >> > >> > >>>>>HVMPTM_no_missed_ticks_pending; > >>>>>+ else > >>>>>+ > >>>>> > >>>>> > >>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] =3D > >> > >> > >>>>>HVMPTM_delay_for_missed_ticks; > >>>>> if ( mode_is(v->domain, one_missed_tick_pending) || > >>>>> mode_is(v->domain, no_missed_ticks_pending) ) > >>>>> { > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>Could you also send detail on the rhel4u4-64 kernel you > >>>>>>are testing with, just to ensure we are not comparing apples > >>>>>>and oranges? (Perhaps there's some way we can even share the > >>>>>>identical disk image and vm.cfg file?) > >>>>>> > >>>>>>And if our problem is indeed the pmtimer, I will need to submit > >>>>>>another patch to Keir to add an hvm pmtimer platform variable. > >>>>>>(Hmmm... I don't think he's even accepted the hpet = > variable patch > >>>>>>yet. I'll have to check.) > >>>>>> > >>>>>>Thanks, > >>>>>>Dan > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>-----Original Message----- > >>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>Sent: Thursday, February 14, 2008 9:00 AM > >>>>>>>To: dan.magenheimer@oracle.com > >>>>>>>Cc: Dave Winchell; Keir Fraser; > >>>>>>> > >>>>>>> > >>>>>xen-devel@lists.xensource.com; Deepak > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>Patel > >>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that > >>>>>>>disables pending > >>>>>>>missed ticks > >>>>>>> > >>>>>>> > >>>>>>>Hi Dan, > >>>>>>> > >>>>>>>I ran the ltp tests with 3.2 and found the errors > >>>>>>>for a 16 hour run to be: > >>>>>>> > >>>>>>>rh4u564 -9.9 sec (-.017%) > >>>>>>>rh4u464 -7.3 sec (-.013%) > >>>>>>> > >>>>>>>There were no cliffs and the drift was linear. > >>>>>>> > >>>>>>>I think the problem you had may be due to the use of the > >>>>>>>pm timer. If you still have the boot log, it would tell you. > >>>>>>> > >>>>>>>When I first tried a guest on 3.2 with "clocksource=3Dpit nohpet" > >>>>>>>I noticed that it picked the pm timer. Adding "nopmtimer", the > >>>>>>>guest will pick the pit. > >>>>>>> > >>>>>>>The reason I didn't have the problem with our 3.1 base is that > >>>>>>>I had disabled the hpet and the pmtimer by not advertising them > >>>>>>>in the acpi tables. I did this so long ago, I forgot > >>>>>>> > >>>>>>> > >>that I had to > >> > >> > >>>>>>>disable pmtimer as well as hpet. > >>>>>>> > >>>>>>>So, can you re-run your test with "clocksource=3Dpit nohpet > >>>>>>> > >>>>>>> > >>>>>nopmtimer"? > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>You should see this in the boot messages: > >>>>>>> > >>>>>>>time.c: Using PIT/TSC based timekeeping. > >>>>>>> > >>>>>>>Thanks, > >>>>>>>Dave > >>>>>>> > >>>>>>> > >>>>>>>Dave Winchell wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>Hi Dan, > >>>>>>>> > >>>>>>>>Over the weekend the drift was +18 seconds for each > >>>>>>>> > >>>>>>>> > >>guest (no ntp). > >> > >> > >>>>>>>>The duration was 3900 minutes, so the error for each > >>>>>>>> > >>>>>>>> > >>was +.0077%. > >> > >> > >>>>>>>>Looking back through the data, it appears to drift linearly at > >>>>>>>>this rate. I've attached a plot for rh4u5-64. > >>>>>>>> > >>>>>>>>This accuracy is better than what I've seen before (.03-.05%). > >>>>>>>>This may be due to the different load (ltp vs usex) or to > >>>>>>>> > >>>>>>>> > >>>>>one of the > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>changes I've made recently. I'll do some > >>>>>>>> > >>>>>>>> > >>experimentation to see if > >> > >> > >>>>>>>>there is > >>>>>>>>a fix I should propose. > >>>>>>>> > >>>>>>>>This still doesn't address the radical drift you saw. > >>>>>>>>The next step for me is to run 3.2 and see if I can > >>>>>>>> > >>>>>>>> > >>reproduce it. > >> > >> > >>>>>>>>Regards, > >>>>>>>>Dave > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>Dave Winchell wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>Hi Dan, > >>>>>>>>> > >>>>>>>>>Sorry it took me so long, but I finally ran an ltp = > test today. > >>>>>>>>>Its on rh4u4-64. I'm using the defaults for ltp and > >>>>>>>>> > >>>>>>>>> > >>using a script > >> > >> > >>>>>>>>>called runltp. I had a usex load on rh4u5-64. No ntpd. > >>>>>>>>>virtual processors / physical processors =3D 2. > >>>>>>>>> > >>>>>>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in > >>>>>>>>> > >>>>>>>>> > >>300 minutes > >> > >> > >>>>>>>>>for -.005% and .008%. > >>>>>>>>> > >>>>>>>>>I'm running a 3.1 based hypervisor with some clock related > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>tweaks that > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>I haven't submitted, because I'm still characterizing them. > >>>>>>>>> > >>>>>>>>>I'm stopping the usex load on 4u5-64 now and > >>>>>>>>> > >>>>>>>>> > >>replacing it with ltp > >> > >> > >>>>>>>>>and will leave the two guests running ltp over the weekend. > >>>>>>>>> > >>>>>>>>>Regards, > >>>>>>>>>Dave > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>Dave Winchell wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>>Hi Dan, Deepak: > >>>>>>>>>> > >>>>>>>>>>Thanks for the data. Those drifts are severe - no wonder > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>ntp couldn't > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>keep then in synch. I'll try to reproduce that behaviour > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>here, with > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>my code base. > >>>>>>>>>>If I can't reproduce it, I'll try 3.2. > >>>>>>>>>> > >>>>>>>>>>If you can isolate what ltp is doing during the cliffs, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>that would > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>be very > >>>>>>>>>>helpful. > >>>>>>>>>> > >>>>>>>>>>thanks, > >>>>>>>>>>Dave > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>Dan Magenheimer wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>OK, Deepak repeated the test without ntpd and using > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>ntpdate -b before > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>the test. > >>>>>>>>>>> > >>>>>>>>>>>The attached graph shows his results: el5u1-64 > >>>>>>>>>>> > >>>>>>>>>>> > >>(best=3D~0.07%), > >> > >> > >>>>>>>>>>>el4u5-64 (middle=3D~0.2%), and el4u5-32 (worst=3D~0.3%). > >>>>>>>>>>> > >>>>>>>>>>>We will continue to look at LTP to try to isolate. > >>>>>>>>>>> > >>>>>>>>>>>Thanks, > >>>>>>>>>>>Dan > >>>>>>>>>>> > >>>>>>>>>>>P.S. elXuY is essentially RHEL XuY with some patches. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>>-----Original Message----- > >>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM > >>>>>>>>>>>>To: Deepak Patel > >>>>>>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser; > >>>>>>>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>Dave Winchell > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>> > >>>>>>>>>>>> > >>that disables > >> > >> > >>>>>>>>>>>>pending > >>>>>>>>>>>>missed ticks > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>Dan, Deeepak, > >>>>>>>>>>>> > >>>>>>>>>>>>It may be that the underlying clock error is too > >>>>>>>>>>>> > >>>>>>>>>>>> > >>great for ntp > >> > >> > >>>>>>>>>>>>to handle. It would be useful if you did not run ntpd > >>>>>>>>>>>>and, instead did ntpdate -b at the start > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>of the test > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>for each guest. Then capture the data as you have > >>>>>>>>>>>> > >>>>>>>>>>>> > >>been doing. > >> > >> > >>>>>>>>>>>>If the drift is greater than .05%, then we need to > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>address that. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>Another option is, when running ntpd, to enable loop > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>statistics in > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>/etc/ntp.conf > >>>>>>>>>>>>by adding this to the file: > >>>>>>>>>>>> > >>>>>>>>>>>>statistics loopstats > >>>>>>>>>>>>statsdir /var/lib/ntp/ > >>>>>>>>>>>> > >>>>>>>>>>>>Then you will see loop data in that directory. > >>>>>>>>>>>>Correlating the data in the loopstats files with the > >>>>>>>>>>>>peaks in skew would be interesting. You will see > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>entries of the form > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>239.735511 10 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>Where the second to last column is the Allan Deviation. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>When that > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>gets over 1000, ntpd is working pretty hard. However, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>I have not > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>seen ntpd > >>>>>>>>>>>>completely loose it like you have. > >>>>>>>>>>>> > >>>>>>>>>>>>I'm on vacation until Monday, and won't be reading > >>>>>>>>>>>>email. > >>>>>>>>>>>> > >>>>>>>>>>>>Thanks for all your work on this! > >>>>>>>>>>>> > >>>>>>>>>>>>-Dave > >>>>>>>>>>>> > >>>>>>>>>>>>Deepak Patel wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>this one.) > >> > >> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>I do not know which graph was attached with this. But > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>I saw this > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>guests when I > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>was running ltp tests continuously. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>>What was the behaviour of the other guests running? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>All pvm guests are fine. But behavior of most of the > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>hvm guests were > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>as described. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>>If they had spikes, were they at the same wall time? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>No. They are not at the same wall time. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>>Were the other guests running ltp as well? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>running ltp > >> > >> > >>>>>>>>>>>>>continuously. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>>How are you measuring skew? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>I was collecting output of "ntpdate -q = > every > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>300 seconds > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>(5 minutes) and have created graph based on that. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>>Are you running ntpd? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>Yes. ntp was running on all the guests. > >>>>>>>>>>>>> > >>>>>>>>>>>>>I am investigating what causes this spikes and > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>let everyone > >> > >> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>know what > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>are my findings. > >>>>>>>>>>>>> > >>>>>>>>>>>>>Thanks, > >>>>>>>>>>>>>Deepak > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>>Anything that you can discover that would be in = > sync with > >>>>>>>>>>>>>>the spikes would be very helpful! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>The code that I test with is our product code, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>which is based > >> > >> > >>>>>>>>>>>>>>on 3.1. So it is possible that something in 3.2 other > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>than vpt.c > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>is the cause. I can test with 3.2, if necessary. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>thanks, > >>>>>>>>>>>>>>Dave > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>Dan Magenheimer wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>>Hi Dave (Keir, see suggestion below) -- > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>Thanks! > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>Turning off vhpet certainly helps a lot (though > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>see below). > >> > >> > >>>>>>>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>should be > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>until it is > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>>fixed? (I have a patch that defaults it off, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>can post it if > >> > >> > >>>>>>>>>>>>>>>there is agreement on the above point.) The whole > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>point of an > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>HPET is to provide more precise timekeeping and > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>if vhpet is > >> > >> > >>>>>>>>>>>>>>>worse than vpit, it can only confuse users. Comments? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>In your testing, are you just measuring % skew > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>over a long > >> > >> > >>>>>>>>>>>>>>>period of time? > >>>>>>>>>>>>>>>We are graphing the skew continuously and > >>>>>>>>>>>>>>>seeing periodic behavior that is unsettling, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>even with pit. > >> > >> > >>>>>>>>>>>>>>>See attached. Though your algorithm recovers, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>the "cliffs" > >> > >> > >>>>>>>>>>>>>>>could still cause real user problems. I wonder > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>if there is > >> > >> > >>>>>>>>>>>>>>>anything that can be done to make the "recovery" more > >>>>>>>>>>>>>>>responsive? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>We are looking into what part(s) of LTP is causing > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>the cliffs. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>Thanks, > >>>>>>>>>>>>>>>Dan > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>-----Original Message----- > >>>>>>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] > >>>>>>>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM > >>>>>>>>>>>>>>>>To: dan.magenheimer@oracle.com > >>>>>>>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; > >>>>>>>>>>>>>>>>deepak.patel@oracle.com; > >>>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell > >>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>that disables > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>pending > >>>>>>>>>>>>>>>>missed ticks > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>Dan, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>I guess I'm a bit out of date calling for = > clock=3D usage. > >>>>>>>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>should specify > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>"clocksource=3Dpit nohpet" on the linux guest bootline. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>You can leave the xen and dom0 bootlines as they are. > >>>>>>>>>>>>>>>>The xen and guest clocksources do not need to > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>be the same. > >> > >> > >>>>>>>>>>>>>>>>In my tests, xen is using the hpet for its > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>timekeeping and > >> > >> > >>>>>>>>>>>>>>>>that appears to be the default. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>When you boot the guests you should see > >>>>>>>>>>>>>>>>time.c: Using PIT/TSC based timekeeping. > >>>>>>>>>>>>>>>>on the rh4u5-64 guest, and something similar > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>on the others. > >> > >> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>Platform timer > >> > >> > >>>>>>>>>>>>>>>>>14.318MHz HPET.) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>This appears to be the xen state, which is fine. > >>>>>>>>>>>>>>>>I was wrongly assuming that this was the guest state. > >>>>>>>>>>>>>>>>You might want to look in your guest logs and see > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>what they were > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>picking > >>>>>>>>>>>>>>>>for a clock source. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>Regards, > >>>>>>>>>>>>>>>>Dave > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>Dan Magenheimer wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Thanks, I hadn't realized that! No wonder we didn't > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>see the same > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>improvement you saw! > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>Try specifying clock=3Dpit on the linux boot line... > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>I'm confused... do you mean "clocksource=3Dpit" > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>on the Xen > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>command line or > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>"nohpet" / "clock=3Dpit" / "clocksource=3Dpit" on the > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>guest (or > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>dom0?) command > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>line? Or both places? Since the tests take > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>awhile, it > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>would be nice > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>to get this right the first time. Do the Xen > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>and guest > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>clocksources need > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>to be the same? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Thanks, > >>>>>>>>>>>>>>>>>Dan > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>-----Original Message----- > >>>>>>>>>>>>>>>>>*From:* Dave Winchell > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>[mailto:dwinchell@virtualiron.com] > >> > >> > >>>>>>>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM > >>>>>>>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser > >>>>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com; > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>deepak.patel@oracle.com; > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell > >>>>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>that disables > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>pending missed ticks > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Hi Dan, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Hpet timer does have a fairly large error, as I > >>>>>>>>>>>>>>>>>was > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>trying this > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>one recently. > >>>>>>>>>>>>>>>>>I don't remember what I got for error, but 1% sounds > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>about right. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>The problem is that hpet is not built on top = > of vpt.c, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>the module > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Keir and I did > >>>>>>>>>>>>>>>>>all the recent work in, for its periodic timer > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>needs. Try > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>>>>specifying clock=3Dpit > >>>>>>>>>>>>>>>>>on the linux boot line. If it still picks the > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>hpet, which it > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>might, let me know > >>>>>>>>>>>>>>>>>and I'll tell you how to get around this. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Regards, > >>>>>>>>>>>>>>>>>Dave > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>-------------------------------------------------------------- > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>---------- > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>*From:* Dan Magenheimer > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>[mailto:dan.magenheimer@oracle.com] > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>*Sent:* Fri 1/25/2008 6:50 PM > >>>>>>>>>>>>>>>>>*To:* Dave Winchell; Keir Fraser > >>>>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com; > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>deepak.patel@oracle.com; > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>akira.ijuin@oracle.com > >>>>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>that disables > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>pending missed ticks > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Sorry for the very late followup on this but > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>we finally > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>were able > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>to get our testing set up again on stable 3.1 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>bits and have > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>seen some very bad results on 3.1.3-rc1, on the > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>order of 1%. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>Test enviroment was a 4-socket dual core machine > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>with 24GB of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>memory running six two-vcpu 2GB domains, four hvm > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>plus two pv. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>All six guests were running LTP simultaneously. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>The four hvm > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>RHEL4u5-64. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Timer_mode was set to 2 for 64-bit guests and 0 for > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>32-bit guests. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>All four hvm guests experienced skew around -1%, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>even the 32-bit > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>guest. Less intensive testing didn't exhibit much > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>skew at all. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>A representative graph is attached. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Dave, I wonder if some portion of your patches > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>didn't end up in > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>the xen trees? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>Platform timer > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>14.318MHz HPET.) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>Thanks, > >>>>>>>>>>>>>>>>>Dan > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>P.S. Many thanks to Deepak and Akira for > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>running tests. > >> > >> > >>>>>>>>>>>>>>>>>>-----Original Message----- > >>>>>>>>>>>>>>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>Dave Winchell > >>>>>>>>>>>>>>>>>>Sent: Wednesday, January 09, 2008 9:53 AM > >>>>>>>>>>>>>>>>>>To: Keir Fraser > >>>>>>>>>>>>>>>>>>Cc: dan.magenheimer@oracle.com; > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dave > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>Winchell > >>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>timer mode that > >> > >> > >>>>>>>>>>>>>>>>>>disables pending > >>>>>>>>>>>>>>>>>>missed ticks > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>Hi Keir, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>The latest change, c/s 16690, looks fine. > >>>>>>>>>>>>>>>>>>I agree that the code in c/s 16690 is equivalent to > >>>>>>>>>>>>>>>>>>the code I submitted. Also, your version is more > >>>>>>>>>>>>>>>>>>concise. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>The error tests confirm the equivalence. With > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>overnight cpu loads, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>the checked in version was accurate to > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>+.048% for sles > >> > >> > >>>>>>>>>>>>>>>>>>and +.038% for red hat. My version was +.046% > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>and > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>+.032% in a > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>2 hour test. > >>>>>>>>>>>>>>>>>>I don't think the difference is significant. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>i/o loads produced errors of +.01%. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>Thanks for all your efforts on this issue. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>Regards, > >>>>>>>>>>>>>>>>>>Dave > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>Keir Fraser wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>Applied as c/s 16690, although the > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>checked-in patch is > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>>>>>smaller. I think the > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>only important fix is to pt_intr_post() and the > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>only bit of > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>the patch I > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>totally omitted was the change to > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>pt_process_missed_ticks(). > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>I don't think > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>that change can be important, but let's see what > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>happens to the > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>error > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>percentage... > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>-- Keir > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>On 4/1/08 23:24, "Dave Winchell" > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>Hi Dan and Keir, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>Attached is a patch that fixes some > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>issues with the > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>SYNC policy > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>(no_missed_ticks_pending). > >>>>>>>>>>>>>>>>>>>>I have not tried to make the change the > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>minimal one, but, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>rather, just > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>ported into > >>>>>>>>>>>>>>>>>>>>the new code what I know to work well. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>The error for > >> > >> > >>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending goes from > >>>>>>>>>>>>>>>>>>>>over 3% to .03% with this change according > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>to my testing. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>Regards, > >>>>>>>>>>>>>>>>>>>>Dave > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>Dan Magenheimer wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>Hi Dave -- > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>Did you get your correction ported? If so, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>it would be > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>nice to see this get > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>into 3.1.3. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>Note that I just did some very limited > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>testing with > >> > >> > >>>>>>>>>>>>>>>>>>timer_mode=3D2(=3DSYNC=3Dno > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>missed ticks pending) > >>>>>>>>>>>>>>>>>>>>>on tip of xen-3.1-testing (64-bit Linux hv > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>guest) and the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>worst error I've > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>seen so far > >>>>>>>>>>>>>>>>>>>>>is 0.012%. But I haven't tried any exotic > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>loads, just LTP. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>Thanks, > >>>>>>>>>>>>>>>>>>>>>Dan > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>-----Original Message----- > >>>>>>>>>>>>>>>>>>>>>>From: Dave Winchell > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>[mailto:dwinchell@virtualiron.com] > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>Sent: Wednesday, December 19, 2007 12:33 PM > >>>>>>>>>>>>>>>>>>>>>>To: dan.magenheimer@oracle.com > >>>>>>>>>>>>>>>>>>>>>>Cc: Keir Fraser; Shan, Haitao; > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dong, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>Eddie; Jiang, Yunhong; Dave Winchell > >>>>>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>timer mode that > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>disables pending > >>>>>>>>>>>>>>>>>>>>>>missed ticks > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>Dan, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>I did some testing with the constant tsc offset > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>SYNC method > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>(now called > >>>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending) > >>>>>>>>>>>>>>>>>>>>>>and found the error to be very high, much larger > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>than 1 %, as > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>I recall. > >>>>>>>>>>>>>>>>>>>>>>I have not had a chance to submit a > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>correction. I > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>will try to > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>do it later > >>>>>>>>>>>>>>>>>>>>>>this week or the first week in January. My > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>version of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>constant tsc > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>offset SYNC method > >>>>>>>>>>>>>>>>>>>>>>produces .02 % error, so I just need to port > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>that into the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>current code. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>The error you got for both of those kernels is > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>what I would > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>expect > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>for the default mode, delay_for_missed_ticks. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>I'll let Keir answer on how to set the > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>time mode. > >> > >> > >>>>>>>>>>>>>>>>>>>>>>Regards, > >>>>>>>>>>>>>>>>>>>>>>Dave > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>Dan Magenheimer wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>Anyone make measurements on the final patch? > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>saw a loss of > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>about 0.2% with no load. This was > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>xen-unstable tip today > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>with no options specified. 32-bit was > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>about 0.01%. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>I think I missed something... how do I > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>run the various > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>accounting choices and which ones are > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>known to be > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>appropriate > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>for which kernels? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>Thanks, > >>>>>>>>>>>>>>>>>>>>>>>Dan > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>-----Original Message----- > >>>>>>>>>>>>>>>>>>>>>>>>From: xen-devel-bounces@lists.xensource.com > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>Keir Fraser > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>Sent: Thursday, December 06, 2007 4:57 AM > >>>>>>>>>>>>>>>>>>>>>>>>To: Dave Winchell > >>>>>>>>>>>>>>>>>>>>>>>>Cc: Shan, Haitao; > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>xen-devel@lists.xensource.com; Dong, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>Eddie; Jiang, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>Yunhong > >>>>>>>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>mode that > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>disables pending > >>>>>>>>>>>>>>>>>>>>>>>>missed ticks > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>Please take a look at xen-unstable > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>changeset 16545. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>-- Keir > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>On 26/11/07 20:57, "Dave Winchell" > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Keir, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>The accuracy data I've collected for i/o > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>loads for the > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>various time protocols follows. In > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>addition, the data > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>for cpu loads is shown. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled cpu and i/o-8 are on an 8 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>processor AMD > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>box. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Two guests, red hat and sles 64 bit, 8 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>vcpu each. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>The cpu load is usex -e36 on each guest. > >>>>>>>>>>>>>>>>>>>>>>>>>(usex is available at > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>http://people.redhat.com/anderson/usex.) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>i/o load is 8 instances of dd if=3D/dev/hda6 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>of=3D/dev/null. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled i/o-32 are 32 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>instances of dd. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Also, these are run on 4 cpu AMD box. > >>>>>>>>>>>>>>>>>>>>>>>>>In addition, there is an idle rh-32bit guest. > >>>>>>>>>>>>>>>>>>>>>>>>>All three guests are 8vcpu. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled i/o-4/32 are the same > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>as i/o-32 > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>except that the redhat-64 guest has 4 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>instances of dd. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Date Duration Protocol sles, rhat error load > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>+4.42 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>sec -.006%, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>+.005% cpu > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>sec, -.001%, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>+.012% cpu > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>sec, -.009%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>-.004% cpu > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>-.005%, -.005% cpu > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>-.008%, -.003% cpu > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>-.040% cpu > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>sec, -.034%, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>-.031% cpu > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/14 17 hrs 17 min ASYNC -6.1 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>sec,-55.7 sec, -.01%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>-.09% i/o-8 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/15 2 hrs 44 min ASYNC -1.47 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>sec,-14.0 sec, -.015% > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>-.14% i/o-8 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>sec, -.017%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>-.022% i/o-8 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>-.017%, -.018% i/o-8 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>sec, -.020%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>-.029% i/o-8 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>sec, -.023%, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>-.031% i/o-8 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>sec, -.12%, > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>-.04% i/o-32 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>sec, -.011%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>-.005% i/o-32 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>sec -.10%, > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>-.11% i/o-32 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/26 113 hrs 46 min MIXED -297. sec, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>13. sec -.07%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>.003% i/o-4/32 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>sec, -.017%, > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>.01% i/o-4/32 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Overhead measurements: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Progress in terms of number of passes > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>through a fixed > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>system workload > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>on an 8 vcpu red hat with an 8 vcpu > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>sles idle. > >> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>The workload was usex -b48. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>ASYNC 167 min 145 passes .868 passes/min > >>>>>>>>>>>>>>>>>>>>>>>>>SYNC 167 min 144 passes .862 passes/min > >>>>>>>>>>>>>>>>>>>>>>>>>SYNC 1065 min 919 passes .863 passes/min > >>>>>>>>>>>>>>>>>>>>>>>>>MIXED 221 min 196 passes .887 passes/min > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Conclusions: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>The only protocol which meets the > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>.05% accuracy > >> > >> > >>>>>>>>>>>>>>>>>>requirement for ntp > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>tracking under the loads > >>>>>>>>>>>>>>>>>>>>>>>>>above is the SYNC protocol. The worst case > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>accuracies for > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>SYNC, MIXED, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>and ASYNC > >>>>>>>>>>>>>>>>>>>>>>>>>are .022%, .12%, and .14%, respectively. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>We could reduce the cost of the SYNC > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>method by only > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>scheduling the extra > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>wakeups if a certain number > >>>>>>>>>>>>>>>>>>>>>>>>>of ticks are missed. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Regards, > >>>>>>>>>>>>>>>>>>>>>>>>>Dave > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>Keir Fraser wrote: > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>On 9/11/07 19:22, "Dave Winchell" > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>Since I had a high error (~.03%) for the > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>ASYNC method a > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>couple of days ago, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>I ran another ASYNC test. I think > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>there may have > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>been something > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>wrong with the code I used a couple of > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>days ago for > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>ASYNC. It may have been > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>missing the immediate delivery of interrupt > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>after context > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>switch in. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>My results indicate that either SYNC > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>or ASYNC give > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>acceptable accuracy, > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>each running consistently around or under > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>.01%. MIXED has > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>a fairly high > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>error of > >>>>>>>>>>>>>>>>>>>>>>>>>>>greater than .03%. Probably too close > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>to .05% ntp > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>threshold for comfort. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>I don't have an overnight run with SYNC. I > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>plan to leave > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>SYNC running > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>over the weekend. If you'd rather I can > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>leave MIXED > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>running instead. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>It may be too early to pick the > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>protocol and > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>I can run > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>more overnight tests > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>next week. > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>I'm a bit worried about any unwanted side > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>effects of the > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>SYNC+run_timer > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>approach -- e.g., whether timer wakeups will > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>cause higher > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>system-wide CPU > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>contention. I find it easier to think > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>through the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>implications of ASYNC. I'm > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>surprised that MIXED loses time, and is less > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>accurate than > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>ASYNC. Perhaps it > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>delivers more timer interrupts than > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>the other > >> > >> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>approaches, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>and each interrupt > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>event causes a small accumulated error? > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>Overall I would consider MIXED and ASYNC as > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>favourites and > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>if the latter is > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>actually more accurate then I can > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>simply revert the > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>changeset that > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>implemented MIXED. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>Perhaps rather than running more of the same > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>workloads you > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>could try idle > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>large disc reads > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>to /dev/null)? We > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>don't have any data on workloads that aren't > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>CPU bound, so > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>that's really an > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>obvious place to put any further effort imo. > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>>-- Keir > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>_______________________________________________ > >>>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>Xen-devel mailing list > >>>>>>>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com > >>>>>>>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c > >>>>>>>>>>>>>>>>>>>>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>2007 +0000 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>2008 -0500 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>@@ -58,7 +58,7 @@ static void > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>pt_process_missed_ticks(stru > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> missed_ticks =3D missed_ticks / (s_time_t) > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>pt->period + 1; > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> if ( mode_is(pt->vcpu->domain, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>no_missed_ticks_pending) ) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>- pt->do_not_freeze =3D !pt->pending_intr_nr; > >>>>>>>>>>>>>>>>>>>>+ pt->do_not_freeze =3D 1; > >>>>>>>>>>>>>>>>>>>> else > >>>>>>>>>>>>>>>>>>>> pt->pending_intr_nr +=3D missed_ticks; > >>>>>>>>>>>>>>>>>>>> pt->scheduled +=3D missed_ticks * pt->period; > >>>>>>>>>>>>>>>>>>>>@@ -127,7 +127,12 @@ static void > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>pt_timer_fn(void *data) > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>> pt_lock(pt); > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>- pt->pending_intr_nr++; > >>>>>>>>>>>>>>>>>>>>+ if ( mode_is(pt->vcpu->domain, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>no_missed_ticks_pending) ) { > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>+ pt->pending_intr_nr =3D 1; > >>>>>>>>>>>>>>>>>>>>+ pt->do_not_freeze =3D 0; > >>>>>>>>>>>>>>>>>>>>+ } > >>>>>>>>>>>>>>>>>>>>+ else > >>>>>>>>>>>>>>>>>>>>+ pt->pending_intr_nr++; > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> if ( !pt->one_shot ) > >>>>>>>>>>>>>>>>>>>> { > >>>>>>>>>>>>>>>>>>>>@@ -221,8 +226,6 @@ void pt_intr_post(struct > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>vcpu *v, struct > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> return; > >>>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>- pt->do_not_freeze =3D 0; > >>>>>>>>>>>>>>>>>>>>- > >>>>>>>>>>>>>>>>>>>> if ( pt->one_shot ) > >>>>>>>>>>>>>>>>>>>> { > >>>>>>>>>>>>>>>>>>>> pt->enabled =3D 0; > >>>>>>>>>>>>>>>>>>>>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>*v, struct > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> pt->last_plt_gtime =3D > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>hvm_get_guest_time(v); > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>>> pt->pending_intr_nr =3D 0; /* > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>'collapse' all > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>>>>>>>>>missed ticks */ > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>>>>>+ else if ( mode_is(v->domain, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>no_missed_ticks_pending) ) { > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>+ pt->pending_intr_nr--; > >>>>>>>>>>>>>>>>>>>>+ pt->last_plt_gtime =3D hvm_get_guest_time(v); > >>>>>>>>>>>>>>>>>>>>+ } > >>>>>>>>>>>>>>>>>>>> else > >>>>>>>>>>>>>>>>>>>> { > >>>>>>>>>>>>>>>>>>>> pt->last_plt_gtime +=3D > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>pt->period_cycles; > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>_______________________________________________ > >>>>>>>>>>>>>>>>>>Xen-devel mailing list > >>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com > >>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>-------------------------------------------------------------- > >>>>>>>---------- > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> > >> > >> > > > > > > > = > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 20 Feb 2008 16:40:47 -0700 Message-ID: <20080220164047500.00000002392@djm-pc> References: <20080219163806078.00000002392@djm-pc> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20080219163806078.00000002392@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" , Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org > > >percentage. However, our 32-bit clock skew seems to > > >show a measureable problem now. > > > > > For the 32 bit guest, which timesource did it pick? > = > The dmesg output is hard to interpret on a 32-bit guest, > but based on what we've seen, I think it was selecting > hpet as timesource (because we specified clocksource=3Dpit > which would have been ignored on RHEL4-32). We are > running another test with "clock=3Dpit" to see if the > skew goes away. Yep, that was the problem. No clock skew with clock=3Dpit on RHEL4-32 anymore. > > >For Xen RHEL5 HVM guests: > > >- I *think* clock=3Dpit is sufficient for RHEL5-32 > > > > > But still poor accuracy, right? > = > Unproven yet but I hope not. The nohpet and nopmtimer > parameters are ignored on RHEL5-32 so the clock=3Dpit > (or clocksource=3Dpit) is the only way to choose the > clock source, and thus the only way to get good > accuracy on RHEL5-32. We had an interesting firedrill with RHEL5-32. It seems that the RHEL-specific "divider=3D10" doesn't get along very well with "clock=3Dpit" and causes the system clock to go whacko, gaining literally HOURS every second. Until Deepak discovered this, it was impossible to boot. So we haven't tested this hypothesis yet, but the dmesg output looks promising. Dan From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 19 Feb 2008 10:26:58 -0500 Message-ID: <47BAF542.2020507@virtualiron.com> References: <20080215094650484.00000001516@djm-pc> <47B5CBB6.5020505@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47B5CBB6.5020505@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: Dave Winchell , Deepak Patel , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Hi Dan, ltp runs by default loading up only one vcpu. The -x option can be used to run multiple instances, though in this mode you will get test failures. I ran 8 instances on each guest for 16 hours, 25 min and the time error was -11 sec (-.019%) on each guest. Regards, Dave Dave Winchell wrote: > Hi Dan, > > Mine was oversubscribed. > 8 physical cpu, 2 guests, each with 8 vcpu. > I ran one instance of ltp on each guest, continuously. I hope ltp > loaded up all the vcpus. I seem to recall that it did, but I > could be wrong. If it didn't, that would be a major difference > between our tests. I'll verify this afternoon and run multiple instances, > if necessary. > > Thanks, > Dave > > > > Dan Magenheimer wrote: > >> Hi Dave -- >> >> No new results yet but one other question: >> >> The problems we've seen with our testing have been with a heavily >> oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests >> running LTP simultaneously. >> >> Was your LTP testing oversubscribed or just a single guest? >> >> Thanks, >> Dan >> >> >> >>> -----Original Message----- >>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>> Sent: Thursday, February 14, 2008 10:56 AM >>> To: dan.magenheimer@oracle.com >>> Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave >>> Winchell >>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables pending >>> missed ticks >>> >>> >>> Dan, >>> >>> Here are some boot snipets for rh4u564 on xen 3.2. >>> >>> >>> #1: >>> >>> Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro >>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet) >>> Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp >>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 >>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 >>> ... >>> Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/ >>> console=ttyS0 clocksource=pit nohpet >>> Feb 14 10:44:59 vs076 kernel: Initializing CPU#0 >>> Feb 14 10:44:59 vs076 kernel: PID hash table entries: 2048 (order: 11, >>> 65536 bytes) >>> Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer. >>> Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 MHz processor. >>> ... >>> Feb 14 10:45:00 vs076 kernel: checking TSC synchronization across 8 >>> CPUs: passed. >>> Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs >>> Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to use of PM timer >>> Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping. >>> >>> >>> >>> #2: >>> >>> Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro >>> root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer) >>> Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp >>> (brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version 3.4.6 20060404 >>> (Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 >>> ... >>> Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/ >>> console=ttyS0 clocksource=pit nohpet nopmtimer >>> Feb 14 10:47:58 vs076 kernel: Initializing CPU#0 >>> Feb 14 10:47:58 vs076 kernel: PID hash table entries: 2048 (order: 11, >>> 65536 bytes) >>> Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz PIT timer. >>> Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 MHz processor. >>> ... >>> Feb 14 10:47:59 vs076 kernel: checking TSC synchronization across 8 >>> CPUs: passed. >>> Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs >>> Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based timekeeping. >>> >>> >>> As you can see, I only get the pit if I specify nopmtimer. >>> >>> Dan Magenheimer wrote: >>> >>> >>> >>>> Hi Dave -- >>>> >>>> Thanks for continuing to run tests! >>>> >>>> Hmmm... I thought I had noticed that even though Linux will >>> >>> acknowledge >>> >>> >>>> the existence of the pmtimer, it still prints: >>>> >>>> time.c: Using PIT/TSC based timekeeping. >>>> >>>> I will check again, but assuming the clocksource for our tests is >>>> indeed pit, the huge difference in the results (yours vs ours) is >>>> baffling. I wonder if the difference may be the underlying hardware. >>>> Maybe we will try to ensure we can duplicate the results on >>> >>> a different >>> >>> >>>> box. >>>> >>>> >>>> So your testing was with stock 3.2.0 xen bits (what cset?) without >>>> any of your [quote from below] "clock related tweaks that I haven't >>>> submitted, because I'm still characterizing them"? >>>> >>>> >>>> >>> >>> None of the tweaks I mentioned are in this test. >>> It was stock with some patches. However, none of the patches are time >>> related to >>> my knowledge and I checked vpt.c to make sure that it is the same as >>> what's in unstable. >>> The only difference is in pt_intr_post, where I set the timer mode. >>> I don't have timer mode tied into our config process yet, which >>> is different than official xen method. >>> >>> >>> (In pt_intr_post) >>> else >>> { >>> + if(v->arch.paging.mode->guest_levels == 4) >>> + v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = >>> HVMPTM_no_missed_ticks_pending; >>> + else >>> + v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = >>> HVMPTM_delay_for_missed_ticks; >>> if ( mode_is(v->domain, one_missed_tick_pending) || >>> mode_is(v->domain, no_missed_ticks_pending) ) >>> { >>> >>> >>> >>>> Could you also send detail on the rhel4u4-64 kernel you >>>> are testing with, just to ensure we are not comparing apples >>>> and oranges? (Perhaps there's some way we can even share the >>>> identical disk image and vm.cfg file?) >>>> >>>> And if our problem is indeed the pmtimer, I will need to submit >>>> another patch to Keir to add an hvm pmtimer platform variable. >>>> (Hmmm... I don't think he's even accepted the hpet variable patch >>>> yet. I'll have to check.) >>>> >>>> Thanks, >>>> Dan >>>> >>>> >>>> >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>> Sent: Thursday, February 14, 2008 9:00 AM >>>>> To: dan.magenheimer@oracle.com >>>>> Cc: Dave Winchell; Keir Fraser; >>>> >>> xen-devel@lists.xensource.com; Deepak >>> >>> >>>>> Patel >>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>> disables pending >>>>> missed ticks >>>>> >>>>> >>>>> Hi Dan, >>>>> >>>>> I ran the ltp tests with 3.2 and found the errors >>>>> for a 16 hour run to be: >>>>> >>>>> rh4u564 -9.9 sec (-.017%) >>>>> rh4u464 -7.3 sec (-.013%) >>>>> >>>>> There were no cliffs and the drift was linear. >>>>> >>>>> I think the problem you had may be due to the use of the >>>>> pm timer. If you still have the boot log, it would tell you. >>>>> >>>>> When I first tried a guest on 3.2 with "clocksource=pit nohpet" >>>>> I noticed that it picked the pm timer. Adding "nopmtimer", the >>>>> guest will pick the pit. >>>>> >>>>> The reason I didn't have the problem with our 3.1 base is that >>>>> I had disabled the hpet and the pmtimer by not advertising them >>>>> in the acpi tables. I did this so long ago, I forgot that I had to >>>>> disable pmtimer as well as hpet. >>>>> >>>>> So, can you re-run your test with "clocksource=pit nohpet >>>> >>> nopmtimer"? >>> >>> >>>>> You should see this in the boot messages: >>>>> >>>>> time.c: Using PIT/TSC based timekeeping. >>>>> >>>>> Thanks, >>>>> Dave >>>>> >>>>> >>>>> Dave Winchell wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Hi Dan, >>>>>> >>>>>> Over the weekend the drift was +18 seconds for each guest (no ntp). >>>>>> The duration was 3900 minutes, so the error for each was +.0077%. >>>>>> Looking back through the data, it appears to drift linearly at >>>>>> this rate. I've attached a plot for rh4u5-64. >>>>>> >>>>>> This accuracy is better than what I've seen before (.03-.05%). >>>>>> This may be due to the different load (ltp vs usex) or to >>>>> >>> one of the >>> >>> >>>>>> changes I've made recently. I'll do some experimentation to see if >>>>>> there is >>>>>> a fix I should propose. >>>>>> >>>>>> This still doesn't address the radical drift you saw. >>>>>> The next step for me is to run 3.2 and see if I can reproduce it. >>>>>> >>>>>> Regards, >>>>>> Dave >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Dave Winchell wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Hi Dan, >>>>>>> >>>>>>> Sorry it took me so long, but I finally ran an ltp test today. >>>>>>> Its on rh4u4-64. I'm using the defaults for ltp and using a script >>>>>>> called runltp. I had a usex load on rh4u5-64. No ntpd. >>>>>>> virtual processors / physical processors = 2. >>>>>>> >>>>>>> The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in 300 minutes >>>>>>> for -.005% and .008%. >>>>>>> >>>>>>> I'm running a 3.1 based hypervisor with some clock related >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> tweaks that >>>>> >>>>> >>>>> >>>>> >>>>>>> I haven't submitted, because I'm still characterizing them. >>>>>>> >>>>>>> I'm stopping the usex load on 4u5-64 now and replacing it with ltp >>>>>>> and will leave the two guests running ltp over the weekend. >>>>>>> >>>>>>> Regards, >>>>>>> Dave >>>>>>> >>>>>>> >>>>>>> Dave Winchell wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Hi Dan, Deepak: >>>>>>>> >>>>>>>> Thanks for the data. Those drifts are severe - no wonder >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> ntp couldn't >>>>> >>>>> >>>>> >>>>> >>>>>>>> keep then in synch. I'll try to reproduce that behaviour >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> here, with >>>>> >>>>> >>>>> >>>>> >>>>>>>> my code base. >>>>>>>> If I can't reproduce it, I'll try 3.2. >>>>>>>> >>>>>>>> If you can isolate what ltp is doing during the cliffs, >>>>>>>> >>>>>>> >>> that would >>> >>> >>>>>>>> be very >>>>>>>> helpful. >>>>>>>> >>>>>>>> thanks, >>>>>>>> Dave >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Dan Magenheimer wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> OK, Deepak repeated the test without ntpd and using >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> ntpdate -b before >>>>> >>>>> >>>>> >>>>> >>>>>>>>> the test. >>>>>>>>> >>>>>>>>> The attached graph shows his results: el5u1-64 (best=~0.07%), >>>>>>>>> el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%). >>>>>>>>> >>>>>>>>> We will continue to look at LTP to try to isolate. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Dan >>>>>>>>> >>>>>>>>> P.S. elXuY is essentially RHEL XuY with some patches. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>> Sent: Wednesday, January 30, 2008 2:45 PM >>>>>>>>>> To: Deepak Patel >>>>>>>>>> Cc: dan.magenheimer@oracle.com; Keir Fraser; >>>>>>>>>> xen-devel@lists.xensource.com; akira.ijuin@oracle.com; >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> Dave Winchell >>>>> >>>>> >>>>> >>>>> >>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode that disables >>>>>>>>>> pending >>>>>>>>>> missed ticks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Dan, Deeepak, >>>>>>>>>> >>>>>>>>>> It may be that the underlying clock error is too great for ntp >>>>>>>>>> to handle. It would be useful if you did not run ntpd >>>>>>>>>> and, instead did ntpdate -b at the start >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> of the test >>>>> >>>>> >>>>> >>>>> >>>>>>>>>> for each guest. Then capture the data as you have been doing. >>>>>>>>>> If the drift is greater than .05%, then we need to >>>>>>>>>> >>>>>>>>> >>> address that. >>> >>> >>>>>>>>>> Another option is, when running ntpd, to enable loop >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> statistics in >>>>> >>>>> >>>>> >>>>> >>>>>>>>>> /etc/ntp.conf >>>>>>>>>> by adding this to the file: >>>>>>>>>> >>>>>>>>>> statistics loopstats >>>>>>>>>> statsdir /var/lib/ntp/ >>>>>>>>>> >>>>>>>>>> Then you will see loop data in that directory. >>>>>>>>>> Correlating the data in the loopstats files with the >>>>>>>>>> peaks in skew would be interesting. You will see >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> entries of the form >>>>> >>>>> >>>>> >>>>> >>>>>>>>>> 54495 76787.701 -0.045153303 -132.569229 0.020806776 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> 239.735511 10 >>>>> >>>>> >>>>> >>>>> >>>>>>>>>> Where the second to last column is the Allan Deviation. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> When that >>>>> >>>>> >>>>> >>>>> >>>>>>>>>> gets over 1000, ntpd is working pretty hard. However, >>>>>>>>>> >>>>>>>>> >>> I have not >>> >>> >>>>>>>>>> seen ntpd >>>>>>>>>> completely loose it like you have. >>>>>>>>>> >>>>>>>>>> I'm on vacation until Monday, and won't be reading >>>>>>>>>> email. >>>>>>>>>> >>>>>>>>>> Thanks for all your work on this! >>>>>>>>>> >>>>>>>>>> -Dave >>>>>>>>>> >>>>>>>>>> Deepak Patel wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> Is the graph for RHEL5u1-64? (I've never tested this one.) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I do not know which graph was attached with this. But >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>> I saw this >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>> behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>> guests when I >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>> was running ltp tests continuously. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> What was the behaviour of the other guests running? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> All pvm guests are fine. But behavior of most of the >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>> hvm guests were >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>> as described. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> If they had spikes, were they at the same wall time? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> No. They are not at the same wall time. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Were the other guests running ltp as well? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes all 6 guests (4 hvm and 2 pvm) the guests are running ltp >>>>>>>>>>> continuously. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> How are you measuring skew? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I was collecting output of "ntpdate -q every >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 300 seconds >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> (5 minutes) and have created graph based on that. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Are you running ntpd? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes. ntp was running on all the guests. >>>>>>>>>>> >>>>>>>>>>> I am investigating what causes this spikes and let everyone >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> know what >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> are my findings. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Deepak >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Anything that you can discover that would be in sync with >>>>>>>>>>>> the spikes would be very helpful! >>>>>>>>>>>> >>>>>>>>>>>> The code that I test with is our product code, which is based >>>>>>>>>>>> on 3.1. So it is possible that something in 3.2 other >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>> than vpt.c >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>> is the cause. I can test with 3.2, if necessary. >>>>>>>>>>>> >>>>>>>>>>>> thanks, >>>>>>>>>>>> Dave >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Dan Magenheimer wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Hi Dave (Keir, see suggestion below) -- >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> >>>>>>>>>>>>> Turning off vhpet certainly helps a lot (though see below). >>>>>>>>>>>>> >>>>>>>>>>>>> I wonder if timekeeping with vhpet is so bad that it >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>> should be >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>> turned off by default (in 3.1, 3.2, and unstable) >>>>>>>>>>>>> >>>>>>>>>>>> >>> until it is >>> >>> >>>>>>>>>>>>> fixed? (I have a patch that defaults it off, can post it if >>>>>>>>>>>>> there is agreement on the above point.) The whole >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>> point of an >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>> HPET is to provide more precise timekeeping and if vhpet is >>>>>>>>>>>>> worse than vpit, it can only confuse users. Comments? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> In your testing, are you just measuring % skew over a long >>>>>>>>>>>>> period of time? >>>>>>>>>>>>> We are graphing the skew continuously and >>>>>>>>>>>>> seeing periodic behavior that is unsettling, even with pit. >>>>>>>>>>>>> See attached. Though your algorithm recovers, the "cliffs" >>>>>>>>>>>>> could still cause real user problems. I wonder if there is >>>>>>>>>>>>> anything that can be done to make the "recovery" more >>>>>>>>>>>>> responsive? >>>>>>>>>>>>> >>>>>>>>>>>>> We are looking into what part(s) of LTP is causing >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>> the cliffs. >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Dan >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>>>>>> Sent: Monday, January 28, 2008 8:21 AM >>>>>>>>>>>>>> To: dan.magenheimer@oracle.com >>>>>>>>>>>>>> Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>>>>>>>>>>>> deepak.patel@oracle.com; >>>>>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>>>>>> Subject: Re: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>> that disables >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>> pending >>>>>>>>>>>>>> missed ticks >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dan, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I guess I'm a bit out of date calling for clock= usage. >>>>>>>>>>>>>> Looking at linux 2.6.20.4 sources, I think you >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>> should specify >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>> "clocksource=pit nohpet" on the linux guest bootline. >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can leave the xen and dom0 bootlines as they are. >>>>>>>>>>>>>> The xen and guest clocksources do not need to be the same. >>>>>>>>>>>>>> In my tests, xen is using the hpet for its timekeeping and >>>>>>>>>>>>>> that appears to be the default. >>>>>>>>>>>>>> >>>>>>>>>>>>>> When you boot the guests you should see >>>>>>>>>>>>>> time.c: Using PIT/TSC based timekeeping. >>>>>>>>>>>>>> on the rh4u5-64 guest, and something similar on the others. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, Platform timer >>>>>>>>>>>>>>> 14.318MHz HPET.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> This appears to be the xen state, which is fine. >>>>>>>>>>>>>> I was wrongly assuming that this was the guest state. >>>>>>>>>>>>>> You might want to look in your guest logs and see >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>> what they were >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>> picking >>>>>>>>>>>>>> for a clock source. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Dave >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dan Magenheimer wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, I hadn't realized that! No wonder we didn't >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> see the same >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> improvement you saw! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Try specifying clock=pit on the linux boot line... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm confused... do you mean "clocksource=pit" on the Xen >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> command line or >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> "nohpet" / "clock=pit" / "clocksource=pit" on the >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> guest (or >>> >>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> dom0?) command >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> line? Or both places? Since the tests take awhile, it >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> would be nice >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> to get this right the first time. Do the Xen and guest >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> clocksources need >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> to be the same? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Dan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>> *From:* Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>>>>>>> *Sent:* Sunday, January 27, 2008 2:22 PM >>>>>>>>>>>>>>> *To:* dan.magenheimer@oracle.com; Keir Fraser >>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> deepak.patel@oracle.com; >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> that disables >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> pending missed ticks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Dan, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hpet timer does have a fairly large error, as I >>>>>>>>>>>>>>> was >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> trying this >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> one recently. >>>>>>>>>>>>>>> I don't remember what I got for error, but 1% sounds >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> about right. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> The problem is that hpet is not built on top of vpt.c, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> the module >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Keir and I did >>>>>>>>>>>>>>> all the recent work in, for its periodic timer >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> needs. Try >>> >>> >>>>>>>>>>>>>>> specifying clock=pit >>>>>>>>>>>>>>> on the linux boot line. If it still picks the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> hpet, which it >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> might, let me know >>>>>>>>>>>>>>> and I'll tell you how to get around this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Dave >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>> -------------------------------------------------------------- >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>> ---------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> *From:* Dan Magenheimer >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> [mailto:dan.magenheimer@oracle.com] >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> *Sent:* Fri 1/25/2008 6:50 PM >>>>>>>>>>>>>>> *To:* Dave Winchell; Keir Fraser >>>>>>>>>>>>>>> *Cc:* xen-devel@lists.xensource.com; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> deepak.patel@oracle.com; >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> akira.ijuin@oracle.com >>>>>>>>>>>>>>> *Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> that disables >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> pending missed ticks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sorry for the very late followup on this but we finally >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> were able >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> to get our testing set up again on stable 3.1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> bits and have >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> seen some very bad results on 3.1.3-rc1, on the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> order of 1%. >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> Test enviroment was a 4-socket dual core machine >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> with 24GB of >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> memory running six two-vcpu 2GB domains, four hvm >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> plus two pv. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> All six guests were running LTP simultaneously. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> The four hvm >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> RHEL4u5-64. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> Timer_mode was set to 2 for 64-bit guests and 0 for >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 32-bit guests. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> All four hvm guests experienced skew around -1%, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> even the 32-bit >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> guest. Less intensive testing didn't exhibit much >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> skew at all. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> A representative graph is attached. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dave, I wonder if some portion of your patches >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> didn't end up in >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> the xen trees? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> (xm dmesg shows 8x Xeon 3.2GHz stepping 04, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> Platform timer >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> 14.318MHz HPET.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Dan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> P.S. Many thanks to Deepak and Akira for running tests. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > -----Original Message----- >>>>>>>>>>>>>>> > From: xen-devel-bounces@lists.xensource.com >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > Dave Winchell >>>>>>>>>>>>>>> > Sent: Wednesday, January 09, 2008 9:53 AM >>>>>>>>>>>>>>> > To: Keir Fraser >>>>>>>>>>>>>>> > Cc: dan.magenheimer@oracle.com; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> xen-devel@lists.xensource.com; Dave >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > Winchell >>>>>>>>>>>>>>> > Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>>>>>>>>> > disables pending >>>>>>>>>>>>>>> > missed ticks >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Hi Keir, >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > The latest change, c/s 16690, looks fine. >>>>>>>>>>>>>>> > I agree that the code in c/s 16690 is equivalent to >>>>>>>>>>>>>>> > the code I submitted. Also, your version is more >>>>>>>>>>>>>>> > concise. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > The error tests confirm the equivalence. With >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> overnight cpu loads, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > the checked in version was accurate to +.048% for sles >>>>>>>>>>>>>>> > and +.038% for red hat. My version was +.046% >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> +.032% in a >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > 2 hour test. >>>>>>>>>>>>>>> > I don't think the difference is significant. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > i/o loads produced errors of +.01%. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Thanks for all your efforts on this issue. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Regards, >>>>>>>>>>>>>>> > Dave >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Keir Fraser wrote: >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >Applied as c/s 16690, although the >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> checked-in patch is >>> >>> >>>>>>>>>>>>>>> > smaller. I think the >>>>>>>>>>>>>>> > >only important fix is to pt_intr_post() and the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> only bit of >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > the patch I >>>>>>>>>>>>>>> > >totally omitted was the change to >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> pt_process_missed_ticks(). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > I don't think >>>>>>>>>>>>>>> > >that change can be important, but let's see what >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> happens to the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> error >>>>>>>>>>>>>>> > >percentage... >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > -- Keir >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > >On 4/1/08 23:24, "Dave Winchell" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > >>Hi Dan and Keir, >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >>Attached is a patch that fixes some issues with the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> SYNC policy >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>(no_missed_ticks_pending). >>>>>>>>>>>>>>> > >>I have not tried to make the change the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> minimal one, but, >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > rather, just >>>>>>>>>>>>>>> > >>ported into >>>>>>>>>>>>>>> > >>the new code what I know to work well. The error for >>>>>>>>>>>>>>> > >>no_missed_ticks_pending goes from >>>>>>>>>>>>>>> > >>over 3% to .03% with this change according >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> to my testing. >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >>Regards, >>>>>>>>>>>>>>> > >>Dave >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >>Dan Magenheimer wrote: >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >>>Hi Dave -- >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>>Did you get your correction ported? If so, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> it would be >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > nice to see this get >>>>>>>>>>>>>>> > >>>into 3.1.3. >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>>Note that I just did some very limited testing with >>>>>>>>>>>>>>> > timer_mode=2(=SYNC=no >>>>>>>>>>>>>>> > >>>missed ticks pending) >>>>>>>>>>>>>>> > >>>on tip of xen-3.1-testing (64-bit Linux hv >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> guest) and the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > worst error I've >>>>>>>>>>>>>>> > >>>seen so far >>>>>>>>>>>>>>> > >>>is 0.012%. But I haven't tried any exotic >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> loads, just LTP. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>>Thanks, >>>>>>>>>>>>>>> > >>>Dan >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>>>-----Original Message----- >>>>>>>>>>>>>>> > >>>>From: Dave Winchell >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> [mailto:dwinchell@virtualiron.com] >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>>>>>>>>>>>> > >>>>To: dan.magenheimer@oracle.com >>>>>>>>>>>>>>> > >>>>Cc: Keir Fraser; Shan, Haitao; >>>>>>>>>>>>>>> > xen-devel@lists.xensource.com; Dong, >>>>>>>>>>>>>>> > >>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>>>>>>>>>>>> > >>>>Subject: Re: [Xen-devel] [PATCH] Add a >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> timer mode that >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>disables pending >>>>>>>>>>>>>>> > >>>>missed ticks >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>Dan, >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>I did some testing with the constant tsc offset >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> SYNC method >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>(now called >>>>>>>>>>>>>>> > >>>>no_missed_ticks_pending) >>>>>>>>>>>>>>> > >>>>and found the error to be very high, much larger >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> than 1 %, as >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>I recall. >>>>>>>>>>>>>>> > >>>>I have not had a chance to submit a correction. I >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> will try to >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>do it later >>>>>>>>>>>>>>> > >>>>this week or the first week in January. My >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> version of >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> constant tsc >>>>>>>>>>>>>>> > >>>>offset SYNC method >>>>>>>>>>>>>>> > >>>>produces .02 % error, so I just need to port >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> that into the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>current code. >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>The error you got for both of those kernels is >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> what I would >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> expect >>>>>>>>>>>>>>> > >>>>for the default mode, delay_for_missed_ticks. >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>I'll let Keir answer on how to set the time mode. >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>Regards, >>>>>>>>>>>>>>> > >>>>Dave >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>Dan Magenheimer wrote: >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>>Anyone make measurements on the final patch? >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>>I just ran a 64-bit RHEL5.1 pvm kernel and >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> saw a loss of >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>about 0.2% with no load. This was >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> xen-unstable tip today >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>with no options specified. 32-bit was >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> about 0.01%. >>> >>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>>I think I missed something... how do I >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> run the various >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>accounting choices and which ones are known to be >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> appropriate >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>for which kernels? >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>>Thanks, >>>>>>>>>>>>>>> > >>>>>Dan >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>>>-----Original Message----- >>>>>>>>>>>>>>> > >>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> [mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>Keir Fraser >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>>>>>>>>>> > >>>>>>To: Dave Winchell >>>>>>>>>>>>>>> > >>>>>>Cc: Shan, Haitao; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> xen-devel@lists.xensource.com; Dong, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > Eddie; Jiang, >>>>>>>>>>>>>>> > >>>>>>Yunhong >>>>>>>>>>>>>>> > >>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> mode that >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>disables pending >>>>>>>>>>>>>>> > >>>>>>missed ticks >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>Please take a look at xen-unstable >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> changeset 16545. >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>-- Keir >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>> wrote: >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>Keir, >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>The accuracy data I've collected for i/o >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> loads for the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>various time protocols follows. In >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> addition, the data >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>for cpu loads is shown. >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> processor AMD >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> box. >>>>>>>>>>>>>>> > >>>>>>>Two guests, red hat and sles 64 bit, 8 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> vcpu each. >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>>>>>>>>> > >>>>>>>(usex is available at >>>>>>>>>>>>>>> http://people.redhat.com/anderson/usex.) >>>>>>>>>>>>>>> > >>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> of=/dev/null. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-32 are 32 >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> instances of dd. >>> >>> >>>>>>>>>>>>>>> > >>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>>>>>>>>> > >>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>>>>>>>>> > >>>>>>>All three guests are 8vcpu. >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>The loads labeled i/o-4/32 are the same >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> as i/o-32 >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>>>>except that the redhat-64 guest has 4 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> instances of dd. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>Date Duration Protocol sles, rhat error load >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, >>>>>>>>>>>>>>> +4.42 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> sec -.006%, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > +.005% cpu >>>>>>>>>>>>>>> > >>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> sec, -.001%, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > +.012% cpu >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sec, -.009%, >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> -.004% cpu >>>>>>>>>>>>>>> > >>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -.005%, -.005% cpu >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -.008%, -.003% cpu >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -.040% cpu >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> sec, -.034%, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > -.031% cpu >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/14 17 hrs 17 min ASYNC -6.1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sec,-55.7 sec, -.01%, >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > -.09% i/o-8 >>>>>>>>>>>>>>> > >>>>>>>11/15 2 hrs 44 min ASYNC -1.47 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sec,-14.0 sec, -.015% >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > -.14% i/o-8 >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sec, -.017%, >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > -.022% i/o-8 >>>>>>>>>>>>>>> > >>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -.017%, -.018% i/o-8 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sec, -.020%, >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > -.029% i/o-8 >>>>>>>>>>>>>>> > >>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> sec, -.023%, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > -.031% i/o-8 >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/21 28 min MIXED -2.01 sec, -.67 sec, -.12%, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -.04% i/o-32 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sec, -.011%, >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > -.005% i/o-32 >>>>>>>>>>>>>>> > >>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 sec -.10%, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -.11% i/o-32 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>11/26 113 hrs 46 min MIXED -297. sec, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> 13. sec -.07%, >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > .003% i/o-4/32 >>>>>>>>>>>>>>> > >>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> sec, -.017%, >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > .01% i/o-4/32 >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>Overhead measurements: >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>Progress in terms of number of passes >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> through a fixed >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>system workload >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>on an 8 vcpu red hat with an 8 vcpu sles idle. >>>>>>>>>>>>>>> > >>>>>>>The workload was usex -b48. >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>>>>>>>>> > >>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>>>>>>>>> > >>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>>>>>>>>> > >>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>Conclusions: >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>The only protocol which meets the .05% accuracy >>>>>>>>>>>>>>> > requirement for ntp >>>>>>>>>>>>>>> > >>>>>>>tracking under the loads >>>>>>>>>>>>>>> > >>>>>>>above is the SYNC protocol. The worst case >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> accuracies for >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>SYNC, MIXED, >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>and ASYNC >>>>>>>>>>>>>>> > >>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>We could reduce the cost of the SYNC >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> method by only >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>scheduling the extra >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>wakeups if a certain number >>>>>>>>>>>>>>> > >>>>>>>of ticks are missed. >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>Regards, >>>>>>>>>>>>>>> > >>>>>>>Dave >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>Keir Fraser wrote: >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>> wrote: >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>Since I had a high error (~.03%) for the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ASYNC method a >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>couple of days ago, >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>I ran another ASYNC test. I think >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> there may have >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > been something >>>>>>>>>>>>>>> > >>>>>>>>>wrong with the code I used a couple of >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> days ago for >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>ASYNC. It may have been >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>missing the immediate delivery of interrupt >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> after context >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>switch in. >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>My results indicate that either SYNC >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> or ASYNC give >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>acceptable accuracy, >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>each running consistently around or under >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> .01%. MIXED has >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>a fairly high >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>error of >>>>>>>>>>>>>>> > >>>>>>>>>greater than .03%. Probably too close >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> to .05% ntp >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>threshold for comfort. >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>I don't have an overnight run with SYNC. I >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> plan to leave >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>SYNC running >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>over the weekend. If you'd rather I can >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> leave MIXED >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>running instead. >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>It may be too early to pick the protocol and >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I can run >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>more overnight tests >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>>next week. >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>I'm a bit worried about any unwanted side >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> effects of the >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>SYNC+run_timer >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>approach -- e.g., whether timer wakeups will >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> cause higher >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>system-wide CPU >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>contention. I find it easier to think >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> through the >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>implications of ASYNC. I'm >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>surprised that MIXED loses time, and is less >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> accurate than >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>ASYNC. Perhaps it >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>delivers more timer interrupts than the other >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> approaches, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>and each interrupt >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>event causes a small accumulated error? >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>Overall I would consider MIXED and ASYNC as >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> favourites and >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>if the latter is >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>actually more accurate then I can >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> simply revert the >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > changeset that >>>>>>>>>>>>>>> > >>>>>>>>implemented MIXED. >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>Perhaps rather than running more of the same >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> workloads you >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>could try idle >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> large disc reads >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>to /dev/null)? We >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>don't have any data on workloads that aren't >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> CPU bound, so >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>that's really an >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>>>>obvious place to put any further effort imo. >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>>-- Keir >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>>> >>>>>>>>>>>>>>> > >>>>>>_______________________________________________ >>>>>>>>>>>>>>> > >>>>>>Xen-devel mailing list >>>>>>>>>>>>>>> > >>>>>>Xen-devel@lists.xensource.com >>>>>>>>>>>>>>> > >>>>>>http://lists.xensource.com/xen-devel >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>>>>>>>>>>>> > >>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> 2007 +0000 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> 2008 -0500 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>@@ -58,7 +58,7 @@ static void >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> pt_process_missed_ticks(stru >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> missed_ticks = missed_ticks / (s_time_t) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> pt->period + 1; >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >> if ( mode_is(pt->vcpu->domain, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> no_missed_ticks_pending) ) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>>>>>>>>>>>> > >>+ pt->do_not_freeze = 1; >>>>>>>>>>>>>>> > >> else >>>>>>>>>>>>>>> > >> pt->pending_intr_nr += missed_ticks; >>>>>>>>>>>>>>> > >> pt->scheduled += missed_ticks * pt->period; >>>>>>>>>>>>>>> > >>@@ -127,7 +127,12 @@ static void >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> pt_timer_fn(void *data) >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> pt_lock(pt); >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >>- pt->pending_intr_nr++; >>>>>>>>>>>>>>> > >>+ if ( mode_is(pt->vcpu->domain, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> no_missed_ticks_pending) ) { >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr = 1; >>>>>>>>>>>>>>> > >>+ pt->do_not_freeze = 0; >>>>>>>>>>>>>>> > >>+ } >>>>>>>>>>>>>>> > >>+ else >>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr++; >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> if ( !pt->one_shot ) >>>>>>>>>>>>>>> > >> { >>>>>>>>>>>>>>> > >>@@ -221,8 +226,6 @@ void pt_intr_post(struct >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> vcpu *v, struct >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >> return; >>>>>>>>>>>>>>> > >> } >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >>- pt->do_not_freeze = 0; >>>>>>>>>>>>>>> > >>- >>>>>>>>>>>>>>> > >> if ( pt->one_shot ) >>>>>>>>>>>>>>> > >> { >>>>>>>>>>>>>>> > >> pt->enabled = 0; >>>>>>>>>>>>>>> > >>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *v, struct >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> > >> pt->last_plt_gtime = >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> hvm_get_guest_time(v); >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > >> pt->pending_intr_nr = 0; /* >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>> 'collapse' all >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>> > missed ticks */ >>>>>>>>>>>>>>> > >> } >>>>>>>>>>>>>>> > >>+ else if ( mode_is(v->domain, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> no_missed_ticks_pending) ) { >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>>>> > >>+ pt->pending_intr_nr--; >>>>>>>>>>>>>>> > >>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>>>>>>>>> > >>+ } >>>>>>>>>>>>>>> > >> else >>>>>>>>>>>>>>> > >> { >>>>>>>>>>>>>>> > >> pt->last_plt_gtime += >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>> pt->period_cycles; >>> >>> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > _______________________________________________ >>>>>>>>>>>>>>> > Xen-devel mailing list >>>>>>>>>>>>>>> > Xen-devel@lists.xensource.com >>>>>>>>>>>>>>> > http://lists.xensource.com/xen-devel >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>> >>>>>> >>>>> >>>>> -------------------------------------------------------------- >>>>> ---------- >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >> >> > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 19 Feb 2008 15:50:23 -0500 Message-ID: <47BB410F.8000904@virtualiron.com> References: <20080219105557046.00000002392@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080219105557046.00000002392@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , Deepak Patel , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Dan, Thanks for all the investigation you've done! -Dave Dan Magenheimer wrote: >Hi Dave -- > >Thanks for that observation on ltp running on one vcpu! > >With "clocksource=pit nohpet nopmtimer" our clock skew >problems seem to have been reduced to a reasonable >percentage. However, our 32-bit clock skew seems to >show a measureable problem now. > For the 32 bit guest, which timesource did it pick? > As a result, >I've been doing some digging into kernel sources and >have observed the following relative to RHEL4 (2.6.9-based) >kernels and RHEL5 (2.6.18-based) kernels and thought I >would document them for posterity. Some of >our confusion arises from the fact that invalid command >line parameters are silently ignored. > >RHEL4: >- clock= is a valid parameter for RHEL4-32 >- clocksource= is not a valid parameter for RHEL4-xx >- nohpet is a valid parameter for RHEL4-64, not RHEL4-32 >- nopmtimer is not a valid parameter for RHEL4-xx >- notsc is a valid parameter for RHEL4-32, not RHEL4-64 >- SMP vs UP RHEL4-64 reports timekeeping in dmesg differently > >For Xen RHEL4 HVM guests: >- I *think* clock=pit is sufficient for RHEL4-32 [1] >- I *think* nohpet is sufficient for RHEL4-64 [1] > >RHEL5: >- there are two kinds of timekeeping, WALL and gtod >- clocksource= is a valid parameter for RHEL5-xx >- clock= is a valid but deprecated parameter for RHEL5-xx >- clock= and clocksource= are essentially equivalent >- nohpet is a valid parameter for RHEL5-64, not RHEL5-32 >- nopmtimer is a valid parameter for RHEL5-64, not RHEL5-32 >- notsc is a valid parameter for RHEL5-64, not RHEL5-32 [1] >- clock=pit changes the gtod source but not the WALL source[2] >- nohpet nopmtimer changes the WALL source to PIT >- /sys/devices/system/clocksource/clocksource0/... > available_clocksource lists the possible clock sources > current_clocksource lists the chosen clock source > ..but neither of these works in a RHEL5 guest! > >For Xen RHEL5 HVM guests: >- I *think* clock=pit is sufficient for RHEL5-32 > > But still poor accuracy, right? >- I *think* clock=pit nohpet nopmtimer is sufficient for RHEL5-64 > >Other info: >- As of 2.6.24.2, clock= is still valid (though still deprecated) > >So, some open questions: >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? > (I *think* not as it has never come up in any email.) > > I have not investigated this yet. >[2] In RHEL5, I *think* it is the WALL source that we care about? > > I'll have to check on this too. >And finally, since invalid command line parameters are ignored. >I think specifying: > clock=pit nohpet nopmtimer >will force the guest clock sources into the optimal state for >all RHEL4 and RHEL5 both 32-bit and 64-bit guests (though see the >question above on tsc). And we should keep an eye on >kernel/time/clocksource.c to ensure the __setup("clock="...) >line doesn't go away before RHEL6. > >Note that if hpet=0 and pmtimer=0 were the default hvm platform >parameters for all xen hvm guests (on all versions of xen), >specifying kernel command line parameters would be unnecessary, >but c'est la vie. > >Oh, and to be complete, timer_mode=0 for 32-bit RHEL guests >and timer_mode=2 for 64-bit RHEL guests. > >Thanks, >Dan > > > >>-----Original Message----- >>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>Sent: Tuesday, February 19, 2008 8:27 AM >>To: dan.magenheimer@oracle.com >>Cc: Dave Winchell; Keir Fraser; xen-devel@lists.xensource.com; Deepak >>Patel >>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>disables pending >>missed ticks >> >> >> >>Hi Dan, >> >>ltp runs by default loading up only one vcpu. >>The -x option can be used to run multiple instances, though >>in this mode you will get test failures. >>I ran 8 instances on each guest for 16 hours, 25 min >>and the time error was -11 sec (-.019%) on each guest. >> >>Regards, >>Dave >> >>Dave Winchell wrote: >> >> >> >>>Hi Dan, >>> >>>Mine was oversubscribed. >>>8 physical cpu, 2 guests, each with 8 vcpu. >>>I ran one instance of ltp on each guest, continuously. I hope ltp >>>loaded up all the vcpus. I seem to recall that it did, but I >>>could be wrong. If it didn't, that would be a major difference >>>between our tests. I'll verify this afternoon and run >>> >>> >>multiple instances, >> >> >>>if necessary. >>> >>>Thanks, >>>Dave >>> >>> >>> >>>Dan Magenheimer wrote: >>> >>> >>> >>>>Hi Dave -- >>>> >>>>No new results yet but one other question: >>>> >>>>The problems we've seen with our testing have been with a heavily >>>>oversubscribed system: 8 physical CPU, six 2-vcpu 2GB guests >>>>running LTP simultaneously. >>>> >>>>Was your LTP testing oversubscribed or just a single guest? >>>> >>>>Thanks, >>>>Dan >>>> >>>> >>>> >>>> >>>> >>>>>-----Original Message----- >>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>Sent: Thursday, February 14, 2008 10:56 AM >>>>>To: dan.magenheimer@oracle.com >>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; Deepak Patel; Dave >>>>>Winchell >>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>> >>>>> >>disables pending >> >> >>>>>missed ticks >>>>> >>>>> >>>>>Dan, >>>>> >>>>>Here are some boot snipets for rh4u564 on xen 3.2. >>>>> >>>>> >>>>>#1: >>>>> >>>>>Feb 14 10:44:59 vs076 kernel: Bootdata ok (command line is ro >>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet) >>>>>Feb 14 10:44:59 vs076 kernel: Linux version 2.6.9-55.ELsmp >>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version >>>>> >>>>> >>3.4.6 20060404 >> >> >>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 >>>>>... >>>>>Feb 14 10:44:59 vs076 kernel: Kernel command line: ro root=LABEL=/ >>>>>console=ttyS0 clocksource=pit nohpet >>>>>Feb 14 10:44:59 vs076 kernel: Initializing CPU#0 >>>>>Feb 14 10:44:59 vs076 kernel: PID hash table entries: >>>>> >>>>> >>2048 (order: 11, >> >> >>>>>65536 bytes) >>>>>Feb 14 10:44:59 vs076 kernel: time.c: Using 3.579545 MHz PM timer. >>>>>Feb 14 10:44:59 vs076 kernel: time.c: Detected 1992.050 >>>>> >>>>> >>MHz processor. >> >> >>>>>... >>>>>Feb 14 10:45:00 vs076 kernel: checking TSC >>>>> >>>>> >>synchronization across 8 >> >> >>>>>CPUs: passed. >>>>>Feb 14 10:45:00 vs076 kernel: Brought up 8 CPUs >>>>>Feb 14 10:45:00 vs076 kernel: Disabling vsyscall due to >>>>> >>>>> >>use of PM timer >> >> >>>>>Feb 14 10:45:00 vs076 kernel: time.c: Using PM based timekeeping. >>>>> >>>>> >>>>> >>>>>#2: >>>>> >>>>>Feb 14 10:47:57 vs076 kernel: Bootdata ok (command line is ro >>>>>root=LABEL=/ console=ttyS0 clocksource=pit nohpet nopmtimer) >>>>>Feb 14 10:47:57 vs076 kernel: Linux version 2.6.9-55.ELsmp >>>>>(brewbuilder@hs20-bc2-4.build.redhat.com) (gcc version >>>>> >>>>> >>3.4.6 20060404 >> >> >>>>>(Red Hat 3.4.6-3)) #1 SMP Fri Apr 20 16:36:54 EDT 2007 >>>>>... >>>>>Feb 14 10:47:58 vs076 kernel: Kernel command line: ro root=LABEL=/ >>>>>console=ttyS0 clocksource=pit nohpet nopmtimer >>>>>Feb 14 10:47:58 vs076 kernel: Initializing CPU#0 >>>>>Feb 14 10:47:58 vs076 kernel: PID hash table entries: >>>>> >>>>> >>2048 (order: 11, >> >> >>>>>65536 bytes) >>>>>Feb 14 10:47:58 vs076 kernel: time.c: Using 1.193182 MHz >>>>> >>>>> >>PIT timer. >> >> >>>>>Feb 14 10:47:58 vs076 kernel: time.c: Detected 1991.959 >>>>> >>>>> >>MHz processor. >> >> >>>>>... >>>>>Feb 14 10:47:59 vs076 kernel: checking TSC >>>>> >>>>> >>synchronization across 8 >> >> >>>>>CPUs: passed. >>>>>Feb 14 10:47:59 vs076 kernel: Brought up 8 CPUs >>>>>Feb 14 10:47:59 vs076 kernel: time.c: Using PIT/TSC based >>>>> >>>>> >>timekeeping. >> >> >>>>>As you can see, I only get the pit if I specify nopmtimer. >>>>> >>>>>Dan Magenheimer wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>Hi Dave -- >>>>>> >>>>>>Thanks for continuing to run tests! >>>>>> >>>>>>Hmmm... I thought I had noticed that even though Linux will >>>>>> >>>>>> >>>>>acknowledge >>>>> >>>>> >>>>> >>>>> >>>>>>the existence of the pmtimer, it still prints: >>>>>> >>>>>>time.c: Using PIT/TSC based timekeeping. >>>>>> >>>>>>I will check again, but assuming the clocksource for our tests is >>>>>>indeed pit, the huge difference in the results (yours vs ours) is >>>>>>baffling. I wonder if the difference may be the >>>>>> >>>>>> >>underlying hardware. >> >> >>>>>>Maybe we will try to ensure we can duplicate the results on >>>>>> >>>>>> >>>>>a different >>>>> >>>>> >>>>> >>>>> >>>>>>box. >>>>>> >>>>>> >>>>>>So your testing was with stock 3.2.0 xen bits (what >>>>>> >>>>>> >>cset?) without >> >> >>>>>>any of your [quote from below] "clock related tweaks >>>>>> >>>>>> >>that I haven't >> >> >>>>>>submitted, because I'm still characterizing them"? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>None of the tweaks I mentioned are in this test. >>>>>It was stock with some patches. However, none of the >>>>> >>>>> >>patches are time >> >> >>>>>related to >>>>>my knowledge and I checked vpt.c to make sure that it is >>>>> >>>>> >>the same as >> >> >>>>>what's in unstable. >>>>>The only difference is in pt_intr_post, where I set the >>>>> >>>>> >>timer mode. >> >> >>>>>I don't have timer mode tied into our config process yet, which >>>>>is different than official xen method. >>>>> >>>>> >>>>>(In pt_intr_post) >>>>> else >>>>> { >>>>>+ if(v->arch.paging.mode->guest_levels == 4) >>>>>+ >>>>> >>>>> >>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = >> >> >>>>>HVMPTM_no_missed_ticks_pending; >>>>>+ else >>>>>+ >>>>> >>>>> >>v->domain->arch.hvm_domain.params[HVM_PARAM_TIMER_MODE] = >> >> >>>>>HVMPTM_delay_for_missed_ticks; >>>>> if ( mode_is(v->domain, one_missed_tick_pending) || >>>>> mode_is(v->domain, no_missed_ticks_pending) ) >>>>> { >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>Could you also send detail on the rhel4u4-64 kernel you >>>>>>are testing with, just to ensure we are not comparing apples >>>>>>and oranges? (Perhaps there's some way we can even share the >>>>>>identical disk image and vm.cfg file?) >>>>>> >>>>>>And if our problem is indeed the pmtimer, I will need to submit >>>>>>another patch to Keir to add an hvm pmtimer platform variable. >>>>>>(Hmmm... I don't think he's even accepted the hpet variable patch >>>>>>yet. I'll have to check.) >>>>>> >>>>>>Thanks, >>>>>>Dan >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>-----Original Message----- >>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>Sent: Thursday, February 14, 2008 9:00 AM >>>>>>>To: dan.magenheimer@oracle.com >>>>>>>Cc: Dave Winchell; Keir Fraser; >>>>>>> >>>>>>> >>>>>xen-devel@lists.xensource.com; Deepak >>>>> >>>>> >>>>> >>>>> >>>>>>>Patel >>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode that >>>>>>>disables pending >>>>>>>missed ticks >>>>>>> >>>>>>> >>>>>>>Hi Dan, >>>>>>> >>>>>>>I ran the ltp tests with 3.2 and found the errors >>>>>>>for a 16 hour run to be: >>>>>>> >>>>>>>rh4u564 -9.9 sec (-.017%) >>>>>>>rh4u464 -7.3 sec (-.013%) >>>>>>> >>>>>>>There were no cliffs and the drift was linear. >>>>>>> >>>>>>>I think the problem you had may be due to the use of the >>>>>>>pm timer. If you still have the boot log, it would tell you. >>>>>>> >>>>>>>When I first tried a guest on 3.2 with "clocksource=pit nohpet" >>>>>>>I noticed that it picked the pm timer. Adding "nopmtimer", the >>>>>>>guest will pick the pit. >>>>>>> >>>>>>>The reason I didn't have the problem with our 3.1 base is that >>>>>>>I had disabled the hpet and the pmtimer by not advertising them >>>>>>>in the acpi tables. I did this so long ago, I forgot >>>>>>> >>>>>>> >>that I had to >> >> >>>>>>>disable pmtimer as well as hpet. >>>>>>> >>>>>>>So, can you re-run your test with "clocksource=pit nohpet >>>>>>> >>>>>>> >>>>>nopmtimer"? >>>>> >>>>> >>>>> >>>>> >>>>>>>You should see this in the boot messages: >>>>>>> >>>>>>>time.c: Using PIT/TSC based timekeeping. >>>>>>> >>>>>>>Thanks, >>>>>>>Dave >>>>>>> >>>>>>> >>>>>>>Dave Winchell wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>Hi Dan, >>>>>>>> >>>>>>>>Over the weekend the drift was +18 seconds for each >>>>>>>> >>>>>>>> >>guest (no ntp). >> >> >>>>>>>>The duration was 3900 minutes, so the error for each >>>>>>>> >>>>>>>> >>was +.0077%. >> >> >>>>>>>>Looking back through the data, it appears to drift linearly at >>>>>>>>this rate. I've attached a plot for rh4u5-64. >>>>>>>> >>>>>>>>This accuracy is better than what I've seen before (.03-.05%). >>>>>>>>This may be due to the different load (ltp vs usex) or to >>>>>>>> >>>>>>>> >>>>>one of the >>>>> >>>>> >>>>> >>>>> >>>>>>>>changes I've made recently. I'll do some >>>>>>>> >>>>>>>> >>experimentation to see if >> >> >>>>>>>>there is >>>>>>>>a fix I should propose. >>>>>>>> >>>>>>>>This still doesn't address the radical drift you saw. >>>>>>>>The next step for me is to run 3.2 and see if I can >>>>>>>> >>>>>>>> >>reproduce it. >> >> >>>>>>>>Regards, >>>>>>>>Dave >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>Dave Winchell wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>Hi Dan, >>>>>>>>> >>>>>>>>>Sorry it took me so long, but I finally ran an ltp test today. >>>>>>>>>Its on rh4u4-64. I'm using the defaults for ltp and >>>>>>>>> >>>>>>>>> >>using a script >> >> >>>>>>>>>called runltp. I had a usex load on rh4u5-64. No ntpd. >>>>>>>>>virtual processors / physical processors = 2. >>>>>>>>> >>>>>>>>>The clocks drifted -1 sec (4u5) and +1.5 sec (4u4) in >>>>>>>>> >>>>>>>>> >>300 minutes >> >> >>>>>>>>>for -.005% and .008%. >>>>>>>>> >>>>>>>>>I'm running a 3.1 based hypervisor with some clock related >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>tweaks that >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>I haven't submitted, because I'm still characterizing them. >>>>>>>>> >>>>>>>>>I'm stopping the usex load on 4u5-64 now and >>>>>>>>> >>>>>>>>> >>replacing it with ltp >> >> >>>>>>>>>and will leave the two guests running ltp over the weekend. >>>>>>>>> >>>>>>>>>Regards, >>>>>>>>>Dave >>>>>>>>> >>>>>>>>> >>>>>>>>>Dave Winchell wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>>Hi Dan, Deepak: >>>>>>>>>> >>>>>>>>>>Thanks for the data. Those drifts are severe - no wonder >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>ntp couldn't >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>keep then in synch. I'll try to reproduce that behaviour >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>here, with >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>my code base. >>>>>>>>>>If I can't reproduce it, I'll try 3.2. >>>>>>>>>> >>>>>>>>>>If you can isolate what ltp is doing during the cliffs, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>that would >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>be very >>>>>>>>>>helpful. >>>>>>>>>> >>>>>>>>>>thanks, >>>>>>>>>>Dave >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>Dan Magenheimer wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>OK, Deepak repeated the test without ntpd and using >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>ntpdate -b before >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>the test. >>>>>>>>>>> >>>>>>>>>>>The attached graph shows his results: el5u1-64 >>>>>>>>>>> >>>>>>>>>>> >>(best=~0.07%), >> >> >>>>>>>>>>>el4u5-64 (middle=~0.2%), and el4u5-32 (worst=~0.3%). >>>>>>>>>>> >>>>>>>>>>>We will continue to look at LTP to try to isolate. >>>>>>>>>>> >>>>>>>>>>>Thanks, >>>>>>>>>>>Dan >>>>>>>>>>> >>>>>>>>>>>P.S. elXuY is essentially RHEL XuY with some patches. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>>>>Sent: Wednesday, January 30, 2008 2:45 PM >>>>>>>>>>>>To: Deepak Patel >>>>>>>>>>>>Cc: dan.magenheimer@oracle.com; Keir Fraser; >>>>>>>>>>>>xen-devel@lists.xensource.com; akira.ijuin@oracle.com; >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>Dave Winchell >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>> >>>>>>>>>>>> >>that disables >> >> >>>>>>>>>>>>pending >>>>>>>>>>>>missed ticks >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>Dan, Deeepak, >>>>>>>>>>>> >>>>>>>>>>>>It may be that the underlying clock error is too >>>>>>>>>>>> >>>>>>>>>>>> >>great for ntp >> >> >>>>>>>>>>>>to handle. It would be useful if you did not run ntpd >>>>>>>>>>>>and, instead did ntpdate -b at the start >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>of the test >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>for each guest. Then capture the data as you have >>>>>>>>>>>> >>>>>>>>>>>> >>been doing. >> >> >>>>>>>>>>>>If the drift is greater than .05%, then we need to >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>address that. >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>Another option is, when running ntpd, to enable loop >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>statistics in >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>/etc/ntp.conf >>>>>>>>>>>>by adding this to the file: >>>>>>>>>>>> >>>>>>>>>>>>statistics loopstats >>>>>>>>>>>>statsdir /var/lib/ntp/ >>>>>>>>>>>> >>>>>>>>>>>>Then you will see loop data in that directory. >>>>>>>>>>>>Correlating the data in the loopstats files with the >>>>>>>>>>>>peaks in skew would be interesting. You will see >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>entries of the form >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>54495 76787.701 -0.045153303 -132.569229 0.020806776 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>239.735511 10 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>Where the second to last column is the Allan Deviation. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>When that >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>gets over 1000, ntpd is working pretty hard. However, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>I have not >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>seen ntpd >>>>>>>>>>>>completely loose it like you have. >>>>>>>>>>>> >>>>>>>>>>>>I'm on vacation until Monday, and won't be reading >>>>>>>>>>>>email. >>>>>>>>>>>> >>>>>>>>>>>>Thanks for all your work on this! >>>>>>>>>>>> >>>>>>>>>>>>-Dave >>>>>>>>>>>> >>>>>>>>>>>>Deepak Patel wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>Is the graph for RHEL5u1-64? (I've never tested >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>this one.) >> >> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>I do not know which graph was attached with this. But >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>I saw this >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>behavior in EL4u5 - 32, EL4U5 - 64 and EL5U1 - 64 hvm >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>guests when I >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>was running ltp tests continuously. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>What was the behaviour of the other guests running? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>All pvm guests are fine. But behavior of most of the >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>hvm guests were >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>as described. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>If they had spikes, were they at the same wall time? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>No. They are not at the same wall time. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>Were the other guests running ltp as well? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>Yes all 6 guests (4 hvm and 2 pvm) the guests are >>>>>>>>>>>>> >>>>>>>>>>>>> >>running ltp >> >> >>>>>>>>>>>>>continuously. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>How are you measuring skew? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>I was collecting output of "ntpdate -q every >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>300 seconds >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>(5 minutes) and have created graph based on that. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>Are you running ntpd? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>Yes. ntp was running on all the guests. >>>>>>>>>>>>> >>>>>>>>>>>>>I am investigating what causes this spikes and >>>>>>>>>>>>> >>>>>>>>>>>>> >>let everyone >> >> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>know what >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>are my findings. >>>>>>>>>>>>> >>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>Deepak >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>Anything that you can discover that would be in sync with >>>>>>>>>>>>>>the spikes would be very helpful! >>>>>>>>>>>>>> >>>>>>>>>>>>>>The code that I test with is our product code, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>which is based >> >> >>>>>>>>>>>>>>on 3.1. So it is possible that something in 3.2 other >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>than vpt.c >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>is the cause. I can test with 3.2, if necessary. >>>>>>>>>>>>>> >>>>>>>>>>>>>>thanks, >>>>>>>>>>>>>>Dave >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>Dan Magenheimer wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>>Hi Dave (Keir, see suggestion below) -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>Turning off vhpet certainly helps a lot (though >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>see below). >> >> >>>>>>>>>>>>>>>I wonder if timekeeping with vhpet is so bad that it >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>should be >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>turned off by default (in 3.1, 3.2, and unstable) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>until it is >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>>fixed? (I have a patch that defaults it off, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>can post it if >> >> >>>>>>>>>>>>>>>there is agreement on the above point.) The whole >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>point of an >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>HPET is to provide more precise timekeeping and >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>if vhpet is >> >> >>>>>>>>>>>>>>>worse than vpit, it can only confuse users. Comments? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>In your testing, are you just measuring % skew >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>over a long >> >> >>>>>>>>>>>>>>>period of time? >>>>>>>>>>>>>>>We are graphing the skew continuously and >>>>>>>>>>>>>>>seeing periodic behavior that is unsettling, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>even with pit. >> >> >>>>>>>>>>>>>>>See attached. Though your algorithm recovers, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>the "cliffs" >> >> >>>>>>>>>>>>>>>could still cause real user problems. I wonder >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>if there is >> >> >>>>>>>>>>>>>>>anything that can be done to make the "recovery" more >>>>>>>>>>>>>>>responsive? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>We are looking into what part(s) of LTP is causing >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>the cliffs. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>>>Dan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>>>>From: Dave Winchell [mailto:dwinchell@virtualiron.com] >>>>>>>>>>>>>>>>Sent: Monday, January 28, 2008 8:21 AM >>>>>>>>>>>>>>>>To: dan.magenheimer@oracle.com >>>>>>>>>>>>>>>>Cc: Keir Fraser; xen-devel@lists.xensource.com; >>>>>>>>>>>>>>>>deepak.patel@oracle.com; >>>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>that disables >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>pending >>>>>>>>>>>>>>>>missed ticks >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>Dan, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>I guess I'm a bit out of date calling for clock= usage. >>>>>>>>>>>>>>>>Looking at linux 2.6.20.4 sources, I think you >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>should specify >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>"clocksource=pit nohpet" on the linux guest bootline. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>You can leave the xen and dom0 bootlines as they are. >>>>>>>>>>>>>>>>The xen and guest clocksources do not need to >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>be the same. >> >> >>>>>>>>>>>>>>>>In my tests, xen is using the hpet for its >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>timekeeping and >> >> >>>>>>>>>>>>>>>>that appears to be the default. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>When you boot the guests you should see >>>>>>>>>>>>>>>>time.c: Using PIT/TSC based timekeeping. >>>>>>>>>>>>>>>>on the rh4u5-64 guest, and something similar >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>on the others. >> >> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>Platform timer >> >> >>>>>>>>>>>>>>>>>14.318MHz HPET.) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>This appears to be the xen state, which is fine. >>>>>>>>>>>>>>>>I was wrongly assuming that this was the guest state. >>>>>>>>>>>>>>>>You might want to look in your guest logs and see >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>what they were >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>picking >>>>>>>>>>>>>>>>for a clock source. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>Regards, >>>>>>>>>>>>>>>>Dave >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>Dan Magenheimer wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Thanks, I hadn't realized that! No wonder we didn't >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>see the same >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>improvement you saw! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Try specifying clock=pit on the linux boot line... >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>I'm confused... do you mean "clocksource=pit" >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>on the Xen >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>command line or >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>"nohpet" / "clock=pit" / "clocksource=pit" on the >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>guest (or >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>dom0?) command >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>line? Or both places? Since the tests take >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>awhile, it >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>would be nice >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>to get this right the first time. Do the Xen >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>and guest >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>clocksources need >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>to be the same? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>>>>>Dan >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>>>>>*From:* Dave Winchell >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>[mailto:dwinchell@virtualiron.com] >> >> >>>>>>>>>>>>>>>>>*Sent:* Sunday, January 27, 2008 2:22 PM >>>>>>>>>>>>>>>>>*To:* dan.magenheimer@oracle.com; Keir Fraser >>>>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com; >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>deepak.patel@oracle.com; >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>akira.ijuin@oracle.com; Dave Winchell >>>>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>that disables >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>pending missed ticks >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Hi Dan, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Hpet timer does have a fairly large error, as I >>>>>>>>>>>>>>>>>was >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>trying this >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>one recently. >>>>>>>>>>>>>>>>>I don't remember what I got for error, but 1% sounds >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>about right. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>The problem is that hpet is not built on top of vpt.c, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>the module >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Keir and I did >>>>>>>>>>>>>>>>>all the recent work in, for its periodic timer >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>needs. Try >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>>>>specifying clock=pit >>>>>>>>>>>>>>>>>on the linux boot line. If it still picks the >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>hpet, which it >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>might, let me know >>>>>>>>>>>>>>>>>and I'll tell you how to get around this. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Regards, >>>>>>>>>>>>>>>>>Dave >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>-------------------------------------------------------------- >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>---------- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>*From:* Dan Magenheimer >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>[mailto:dan.magenheimer@oracle.com] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>*Sent:* Fri 1/25/2008 6:50 PM >>>>>>>>>>>>>>>>>*To:* Dave Winchell; Keir Fraser >>>>>>>>>>>>>>>>>*Cc:* xen-devel@lists.xensource.com; >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>deepak.patel@oracle.com; >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>akira.ijuin@oracle.com >>>>>>>>>>>>>>>>>*Subject:* RE: [Xen-devel] [PATCH] Add a timer mode >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>that disables >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>pending missed ticks >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Sorry for the very late followup on this but >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>we finally >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>were able >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>to get our testing set up again on stable 3.1 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>bits and have >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>seen some very bad results on 3.1.3-rc1, on the >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>order of 1%. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>Test enviroment was a 4-socket dual core machine >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>with 24GB of >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>memory running six two-vcpu 2GB domains, four hvm >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>plus two pv. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>All six guests were running LTP simultaneously. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>The four hvm >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>guests were: RHEL5u1-64, RHEL4u5-32, RHEL5-64, and >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>RHEL4u5-64. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>Timer_mode was set to 2 for 64-bit guests and 0 for >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>32-bit guests. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>All four hvm guests experienced skew around -1%, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>even the 32-bit >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>guest. Less intensive testing didn't exhibit much >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>skew at all. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>A representative graph is attached. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Dave, I wonder if some portion of your patches >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>didn't end up in >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>the xen trees? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>(xm dmesg shows 8x Xeon 3.2GHz stepping 04, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>Platform timer >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>14.318MHz HPET.) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>>>>>Dan >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>P.S. Many thanks to Deepak and Akira for >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>running tests. >> >> >>>>>>>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>Dave Winchell >>>>>>>>>>>>>>>>>>Sent: Wednesday, January 09, 2008 9:53 AM >>>>>>>>>>>>>>>>>>To: Keir Fraser >>>>>>>>>>>>>>>>>>Cc: dan.magenheimer@oracle.com; >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dave >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Winchell >>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>timer mode that >> >> >>>>>>>>>>>>>>>>>>disables pending >>>>>>>>>>>>>>>>>>missed ticks >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Hi Keir, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>The latest change, c/s 16690, looks fine. >>>>>>>>>>>>>>>>>>I agree that the code in c/s 16690 is equivalent to >>>>>>>>>>>>>>>>>>the code I submitted. Also, your version is more >>>>>>>>>>>>>>>>>>concise. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>The error tests confirm the equivalence. With >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>overnight cpu loads, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>the checked in version was accurate to >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>+.048% for sles >> >> >>>>>>>>>>>>>>>>>>and +.038% for red hat. My version was +.046% >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>and >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>+.032% in a >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>2 hour test. >>>>>>>>>>>>>>>>>>I don't think the difference is significant. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>i/o loads produced errors of +.01%. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Thanks for all your efforts on this issue. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Regards, >>>>>>>>>>>>>>>>>>Dave >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Keir Fraser wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>Applied as c/s 16690, although the >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>checked-in patch is >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>>>>>smaller. I think the >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>only important fix is to pt_intr_post() and the >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>only bit of >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>the patch I >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>totally omitted was the change to >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>pt_process_missed_ticks(). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>I don't think >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>that change can be important, but let's see what >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>happens to the >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>error >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>percentage... >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>-- Keir >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>On 4/1/08 23:24, "Dave Winchell" >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>Hi Dan and Keir, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>Attached is a patch that fixes some >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>issues with the >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>SYNC policy >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>(no_missed_ticks_pending). >>>>>>>>>>>>>>>>>>>>I have not tried to make the change the >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>minimal one, but, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>rather, just >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>ported into >>>>>>>>>>>>>>>>>>>>the new code what I know to work well. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>The error for >> >> >>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending goes from >>>>>>>>>>>>>>>>>>>>over 3% to .03% with this change according >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>to my testing. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>Regards, >>>>>>>>>>>>>>>>>>>>Dave >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>Dan Magenheimer wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>Hi Dave -- >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>Did you get your correction ported? If so, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>it would be >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>nice to see this get >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>into 3.1.3. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>Note that I just did some very limited >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>testing with >> >> >>>>>>>>>>>>>>>>>>timer_mode=2(=SYNC=no >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>missed ticks pending) >>>>>>>>>>>>>>>>>>>>>on tip of xen-3.1-testing (64-bit Linux hv >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>guest) and the >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>worst error I've >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>seen so far >>>>>>>>>>>>>>>>>>>>>is 0.012%. But I haven't tried any exotic >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>loads, just LTP. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>>>>>>>>>Dan >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>>>>>>>>>>From: Dave Winchell >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>[mailto:dwinchell@virtualiron.com] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>Sent: Wednesday, December 19, 2007 12:33 PM >>>>>>>>>>>>>>>>>>>>>>To: dan.magenheimer@oracle.com >>>>>>>>>>>>>>>>>>>>>>Cc: Keir Fraser; Shan, Haitao; >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>xen-devel@lists.xensource.com; Dong, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>Eddie; Jiang, Yunhong; Dave Winchell >>>>>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>timer mode that >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>disables pending >>>>>>>>>>>>>>>>>>>>>>missed ticks >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>Dan, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>I did some testing with the constant tsc offset >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>SYNC method >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>(now called >>>>>>>>>>>>>>>>>>>>>>no_missed_ticks_pending) >>>>>>>>>>>>>>>>>>>>>>and found the error to be very high, much larger >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>than 1 %, as >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>I recall. >>>>>>>>>>>>>>>>>>>>>>I have not had a chance to submit a >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>correction. I >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>will try to >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>do it later >>>>>>>>>>>>>>>>>>>>>>this week or the first week in January. My >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>version of >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>constant tsc >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>offset SYNC method >>>>>>>>>>>>>>>>>>>>>>produces .02 % error, so I just need to port >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>that into the >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>current code. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>The error you got for both of those kernels is >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>what I would >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>expect >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>for the default mode, delay_for_missed_ticks. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>I'll let Keir answer on how to set the >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>time mode. >> >> >>>>>>>>>>>>>>>>>>>>>>Regards, >>>>>>>>>>>>>>>>>>>>>>Dave >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>Dan Magenheimer wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>Anyone make measurements on the final patch? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>I just ran a 64-bit RHEL5.1 pvm kernel and >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>saw a loss of >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>about 0.2% with no load. This was >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>xen-unstable tip today >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>with no options specified. 32-bit was >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>about 0.01%. >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>I think I missed something... how do I >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>run the various >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>accounting choices and which ones are >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>known to be >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>appropriate >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>for which kernels? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>Thanks, >>>>>>>>>>>>>>>>>>>>>>>Dan >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>-----Original Message----- >>>>>>>>>>>>>>>>>>>>>>>>From: xen-devel-bounces@lists.xensource.com >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>[mailto:xen-devel-bounces@lists.xensource.com]On Behalf Of >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>Keir Fraser >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>Sent: Thursday, December 06, 2007 4:57 AM >>>>>>>>>>>>>>>>>>>>>>>>To: Dave Winchell >>>>>>>>>>>>>>>>>>>>>>>>Cc: Shan, Haitao; >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>xen-devel@lists.xensource.com; Dong, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>Eddie; Jiang, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>Yunhong >>>>>>>>>>>>>>>>>>>>>>>>Subject: Re: [Xen-devel] [PATCH] Add a timer >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>mode that >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>disables pending >>>>>>>>>>>>>>>>>>>>>>>>missed ticks >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>Please take a look at xen-unstable >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>changeset 16545. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>-- Keir >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>On 26/11/07 20:57, "Dave Winchell" >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Keir, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>The accuracy data I've collected for i/o >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>loads for the >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>various time protocols follows. In >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>addition, the data >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>for cpu loads is shown. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled cpu and i/o-8 are on an 8 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>processor AMD >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>box. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Two guests, red hat and sles 64 bit, 8 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>vcpu each. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>The cpu load is usex -e36 on each guest. >>>>>>>>>>>>>>>>>>>>>>>>>(usex is available at >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>http://people.redhat.com/anderson/usex.) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>i/o load is 8 instances of dd if=/dev/hda6 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>of=/dev/null. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled i/o-32 are 32 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>instances of dd. >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Also, these are run on 4 cpu AMD box. >>>>>>>>>>>>>>>>>>>>>>>>>In addition, there is an idle rh-32bit guest. >>>>>>>>>>>>>>>>>>>>>>>>>All three guests are 8vcpu. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>The loads labeled i/o-4/32 are the same >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>as i/o-32 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>except that the redhat-64 guest has 4 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>instances of dd. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Date Duration Protocol sles, rhat error load >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/07 23 hrs 40 min ASYNC -4.96 sec, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>+4.42 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>sec -.006%, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>+.005% cpu >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/09 3 hrs 19 min ASYNC -.13 sec, +1.44 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>sec, -.001%, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>+.012% cpu >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/08 2 hrs 21 min SYNC -.80 sec, -.34 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>sec, -.009%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>-.004% cpu >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/08 1 hr 25 min SYNC -.24 sec, -.26 sec, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>-.005%, -.005% cpu >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/12 65 hrs 40 min SYNC -18 sec, -8 sec, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>-.008%, -.003% cpu >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/08 28 min MIXED -.75 sec, -.67 sec -.045%, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>-.040% cpu >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/08 15 hrs 39 min MIXED -19. sec,-17.4 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>sec, -.034%, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>-.031% cpu >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/14 17 hrs 17 min ASYNC -6.1 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>sec,-55.7 sec, -.01%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>-.09% i/o-8 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/15 2 hrs 44 min ASYNC -1.47 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>sec,-14.0 sec, -.015% >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>-.14% i/o-8 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/13 15 hrs 38 min SYNC -9.7 sec,-12.3 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>sec, -.017%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>-.022% i/o-8 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/14 48 min SYNC - .46 sec, - .48 sec, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>-.017%, -.018% i/o-8 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/14 4 hrs 2 min MIXED -2.9 sec, -4.15 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>sec, -.020%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>-.029% i/o-8 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/20 16 hrs 2 min MIXED -13.4 sec,-18.1 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>sec, -.023%, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>-.031% i/o-8 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/21 28 min MIXED -2.01 sec, -.67 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>sec, -.12%, >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>-.04% i/o-32 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/21 2 hrs 25 min SYNC -.96 sec, -.43 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>sec, -.011%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>-.005% i/o-32 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/21 40 min ASYNC -2.43 sec, -2.77 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>sec -.10%, >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>-.11% i/o-32 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/26 113 hrs 46 min MIXED -297. sec, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>13. sec -.07%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>.003% i/o-4/32 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>11/26 4 hrs 50 min SYNC -3.21 sec, 1.44 >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>sec, -.017%, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>.01% i/o-4/32 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Overhead measurements: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Progress in terms of number of passes >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>through a fixed >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>system workload >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>on an 8 vcpu red hat with an 8 vcpu >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>sles idle. >> >> >>>>>>>>>>>>>>>>>>>>>>>>>The workload was usex -b48. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>ASYNC 167 min 145 passes .868 passes/min >>>>>>>>>>>>>>>>>>>>>>>>>SYNC 167 min 144 passes .862 passes/min >>>>>>>>>>>>>>>>>>>>>>>>>SYNC 1065 min 919 passes .863 passes/min >>>>>>>>>>>>>>>>>>>>>>>>>MIXED 221 min 196 passes .887 passes/min >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Conclusions: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>The only protocol which meets the >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>.05% accuracy >> >> >>>>>>>>>>>>>>>>>>requirement for ntp >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>tracking under the loads >>>>>>>>>>>>>>>>>>>>>>>>>above is the SYNC protocol. The worst case >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>accuracies for >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>SYNC, MIXED, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>and ASYNC >>>>>>>>>>>>>>>>>>>>>>>>>are .022%, .12%, and .14%, respectively. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>We could reduce the cost of the SYNC >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>method by only >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>scheduling the extra >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>wakeups if a certain number >>>>>>>>>>>>>>>>>>>>>>>>>of ticks are missed. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Regards, >>>>>>>>>>>>>>>>>>>>>>>>>Dave >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>Keir Fraser wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>On 9/11/07 19:22, "Dave Winchell" >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>Since I had a high error (~.03%) for the >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>ASYNC method a >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>couple of days ago, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>I ran another ASYNC test. I think >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>there may have >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>been something >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>wrong with the code I used a couple of >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>days ago for >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>ASYNC. It may have been >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>missing the immediate delivery of interrupt >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>after context >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>switch in. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>My results indicate that either SYNC >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>or ASYNC give >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>acceptable accuracy, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>each running consistently around or under >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>.01%. MIXED has >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>a fairly high >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>error of >>>>>>>>>>>>>>>>>>>>>>>>>>>greater than .03%. Probably too close >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>to .05% ntp >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>threshold for comfort. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>I don't have an overnight run with SYNC. I >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>plan to leave >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>SYNC running >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>over the weekend. If you'd rather I can >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>leave MIXED >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>running instead. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>It may be too early to pick the >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>protocol and >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>I can run >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>more overnight tests >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>next week. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>I'm a bit worried about any unwanted side >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>effects of the >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>SYNC+run_timer >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>approach -- e.g., whether timer wakeups will >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>cause higher >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>system-wide CPU >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>contention. I find it easier to think >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>through the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>implications of ASYNC. I'm >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>surprised that MIXED loses time, and is less >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>accurate than >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>ASYNC. Perhaps it >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>delivers more timer interrupts than >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>the other >> >> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>approaches, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>and each interrupt >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>event causes a small accumulated error? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>Overall I would consider MIXED and ASYNC as >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>favourites and >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>if the latter is >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>actually more accurate then I can >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>simply revert the >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>changeset that >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>implemented MIXED. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>Perhaps rather than running more of the same >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>workloads you >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>could try idle >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>VCPUs and I/O bound VCPUs (e.g., repeated >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>large disc reads >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>to /dev/null)? We >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>don't have any data on workloads that aren't >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>CPU bound, so >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>that's really an >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>obvious place to put any further effort imo. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>-- Keir >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>_______________________________________________ >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>Xen-devel mailing list >>>>>>>>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com >>>>>>>>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>diff -r cfdbdca5b831 xen/arch/x86/hvm/vpt.c >>>>>>>>>>>>>>>>>>>>--- a/xen/arch/x86/hvm/vpt.c Thu Dec 06 15:36:07 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>2007 +0000 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>+++ b/xen/arch/x86/hvm/vpt.c Fri Jan 04 17:58:16 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>2008 -0500 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>@@ -58,7 +58,7 @@ static void >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>pt_process_missed_ticks(stru >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> missed_ticks = missed_ticks / (s_time_t) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>pt->period + 1; >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> if ( mode_is(pt->vcpu->domain, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>no_missed_ticks_pending) ) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>- pt->do_not_freeze = !pt->pending_intr_nr; >>>>>>>>>>>>>>>>>>>>+ pt->do_not_freeze = 1; >>>>>>>>>>>>>>>>>>>> else >>>>>>>>>>>>>>>>>>>> pt->pending_intr_nr += missed_ticks; >>>>>>>>>>>>>>>>>>>> pt->scheduled += missed_ticks * pt->period; >>>>>>>>>>>>>>>>>>>>@@ -127,7 +127,12 @@ static void >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>pt_timer_fn(void *data) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> pt_lock(pt); >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>- pt->pending_intr_nr++; >>>>>>>>>>>>>>>>>>>>+ if ( mode_is(pt->vcpu->domain, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>no_missed_ticks_pending) ) { >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>+ pt->pending_intr_nr = 1; >>>>>>>>>>>>>>>>>>>>+ pt->do_not_freeze = 0; >>>>>>>>>>>>>>>>>>>>+ } >>>>>>>>>>>>>>>>>>>>+ else >>>>>>>>>>>>>>>>>>>>+ pt->pending_intr_nr++; >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> if ( !pt->one_shot ) >>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>>@@ -221,8 +226,6 @@ void pt_intr_post(struct >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>vcpu *v, struct >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> return; >>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>- pt->do_not_freeze = 0; >>>>>>>>>>>>>>>>>>>>- >>>>>>>>>>>>>>>>>>>> if ( pt->one_shot ) >>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>> pt->enabled = 0; >>>>>>>>>>>>>>>>>>>>@@ -235,6 +238,10 @@ void pt_intr_post(struct vcpu >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>*v, struct >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> pt->last_plt_gtime = >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>hvm_get_guest_time(v); >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>>> pt->pending_intr_nr = 0; /* >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>'collapse' all >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>>>>>>>>>>>missed ticks */ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>+ else if ( mode_is(v->domain, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>no_missed_ticks_pending) ) { >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>+ pt->pending_intr_nr--; >>>>>>>>>>>>>>>>>>>>+ pt->last_plt_gtime = hvm_get_guest_time(v); >>>>>>>>>>>>>>>>>>>>+ } >>>>>>>>>>>>>>>>>>>> else >>>>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>>>> pt->last_plt_gtime += >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>pt->period_cycles; >>>>> >>>>> >>>>> >>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>_______________________________________________ >>>>>>>>>>>>>>>>>>Xen-devel mailing list >>>>>>>>>>>>>>>>>>Xen-devel@lists.xensource.com >>>>>>>>>>>>>>>>>>http://lists.xensource.com/xen-devel >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>-------------------------------------------------------------- >>>>>>>---------- >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >> >> > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Mon, 25 Feb 2008 09:42:02 -0700 Message-ID: <20080225094202546.00000003172@djm-pc> References: <47BB410F.8000904@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <47BB410F.8000904@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org Hi Dave -- I've looked into RHEL5-64 a bit more and would appreciate any thoughts you might have: > >So, some open questions: > >[2] In RHEL5, I *think* it is the WALL source that we care about? > > > > > I'll have to check on this too. On second look, I may be wrong. The GTOD clock seems to be the one associated with vxtime.mode and vxtime.mode is used in linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to determine if a tick is lost or not. Is this the code that we want timer_mode=3D2 to influence? > >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? > > (I *think* not as it has never come up in any email.) > > > > > I have not investigated this yet. Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c, it appears whether notsc is required or not depends in part on the underlying virtual AND physical system. notsc definitely is involved in the selection of GTOD time but notsc can get set not only by kernel command line parameter but also by the result of unsynchronized_tsc(): if (apic_is_clustered_box) notsc=3D1 if (box_is_Intel) { /* C3 delay is worst case HW latency to enter/exit C3 state */ if (C3 delay < 1000) notsc=3D1 } else { /* e.g. AMD */ if (num_present_cpus() > 1) notsc=3D1 } I'm not sure what constitutes a clustered HVM guest or how that C3 state latency is determined under Xen, but its clear that the clocksource can be influenced not only by what clock hardware is present and command-line parameters but also by the physical CPU and number of guest vcpus. Yuck! Thanks, Dan From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 26 Feb 2008 08:26:05 +0000 Message-ID: References: <47C31E8F.90805@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47C31E8F.90805@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org If this is an integration of hpet into the vpt.c infrastructure then that would be very welcome. -- Keir On 25/2/08 20:01, "Dave Winchell" wrote: > Hi Dan, > > I've added a few comments below ... > > Lately I've been busy getting the hpet emulation to be accurate. > Right now I have the hpet based clock error down to under .02% with > a limited amount of testing. (The error was 1% or more before my changes.) > I hope to submit a patch soon so others can try this out. I need to do more > testing and reduce my changes to the minimal set required. > > One thing I really like about the hpet is that a test designed to see if > the clock will go backwards passes on the hpet where it fails on > the pit/tsc clock. I tried for quite a while to keep pit/tsc from going > backwards, but was unsuccessful. It ended up being easier to fix the hpet > emulation for accuracy. > > The clock going backwards test is simply a call to gettimeofday() in a loop, > with no delay between calls except to check for new_time < prev_time. > I run this from a driver calling do_gettimeofday() to make it > more stressful. Then I run an instance of this on all (8) vcpus on > 8 physical * 2 guests for a total of 16 vcpus running the test. > The test does a yield now and then so you can actually log into the > system, etc. > > Regards, > Dave > > Dan Magenheimer wrote: > >> Hi Dave -- >> >> I've looked into RHEL5-64 a bit more and would appreciate >> any thoughts you might have: >> >> >> >>>> So, some open questions: >>>> [2] In RHEL5, I *think* it is the WALL source that we care about? >>>> >>>> >>>> >>>> >>> I'll have to check on this too. >>> >>> >> >> On second look, I may be wrong. The GTOD clock seems to be >> the one associated with vxtime.mode and vxtime.mode is used in >> linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to >> determine if a tick is lost or not. Is this the code that we >> want timer_mode=2 to influence? >> >> > Yes, it is. > >> >> >>>> [1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? >>>> (I *think* not as it has never come up in any email.) >>>> >>>> >>>> >>>> >>> I have not investigated this yet. >>> >>> >> >> Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c, >> it appears whether notsc is required or not depends in part on the >> underlying virtual AND physical system. >> >> notsc definitely is involved in the selection of GTOD time >> but notsc can get set not only by kernel command line parameter >> but also by the result of unsynchronized_tsc(): >> >> > According to the comment in time.c, unsynchronized_tsc() is guessing > whether the tsc is synchronized across processors. On most physical > platforms they are, > and, on those physical platforms the virtualization layer will present the > same error as the hardware exibits. Right now we don't have code in xen to > make a unsynchronized host look synchronized to the guest, though that > wouldn't be very hard to do as long as the error between physical processors > remained constant. > >> if (apic_is_clustered_box) notsc=1 >> if (box_is_Intel) { >> /* C3 delay is worst case HW latency to enter/exit C3 state */ >> if (C3 delay < 1000) notsc=1 >> } >> else { /* e.g. AMD */ >> if (num_present_cpus() > 1) notsc=1 >> } >> >> I'm not sure what constitutes a clustered HVM guest or how that >> C3 state latency is determined under Xen, but its clear that the >> clocksource can be influenced not only by what clock hardware is >> present and command-line parameters but also by the physical CPU >> and number of guest vcpus. >> >> > I haven't looked into > >> Yuck! >> >> Thanks, >> Dan >> >> >> >> >> > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Mon, 25 Feb 2008 15:01:19 -0500 Message-ID: <47C31E8F.90805@virtualiron.com> References: <20080225094202546.00000003172@djm-pc> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20080225094202546.00000003172@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , Deepak Patel , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Dan, I've added a few comments below ... Lately I've been busy getting the hpet emulation to be accurate. Right now I have the hpet based clock error down to under .02% with a limited amount of testing. (The error was 1% or more before my changes.) I hope to submit a patch soon so others can try this out. I need to do more testing and reduce my changes to the minimal set required. One thing I really like about the hpet is that a test designed to see if the clock will go backwards passes on the hpet where it fails on the pit/tsc clock. I tried for quite a while to keep pit/tsc from going backwards, but was unsuccessful. It ended up being easier to fix the hpet emulation for accuracy. The clock going backwards test is simply a call to gettimeofday() in a loop, with no delay between calls except to check for new_time < prev_time. I run this from a driver calling do_gettimeofday() to make it more stressful. Then I run an instance of this on all (8) vcpus on 8 physical * 2 guests for a total of 16 vcpus running the test. The test does a yield now and then so you can actually log into the system, etc. Regards, Dave Dan Magenheimer wrote: >Hi Dave -- > >I've looked into RHEL5-64 a bit more and would appreciate >any thoughts you might have: > > > >>>So, some open questions: >>>[2] In RHEL5, I *think* it is the WALL source that we care about? >>> >>> >>> >>> >>I'll have to check on this too. >> >> > >On second look, I may be wrong. The GTOD clock seems to be >the one associated with vxtime.mode and vxtime.mode is used in >linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to >determine if a tick is lost or not. Is this the code that we >want timer_mode=2 to influence? > > Yes, it is. > > >>>[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? >>> (I *think* not as it has never come up in any email.) >>> >>> >>> >>> >>I have not investigated this yet. >> >> > >Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c, >it appears whether notsc is required or not depends in part on the >underlying virtual AND physical system. > >notsc definitely is involved in the selection of GTOD time >but notsc can get set not only by kernel command line parameter >but also by the result of unsynchronized_tsc(): > > According to the comment in time.c, unsynchronized_tsc() is guessing whether the tsc is synchronized across processors. On most physical platforms they are, and, on those physical platforms the virtualization layer will present the same error as the hardware exibits. Right now we don't have code in xen to make a unsynchronized host look synchronized to the guest, though that wouldn't be very hard to do as long as the error between physical processors remained constant. >if (apic_is_clustered_box) notsc=1 >if (box_is_Intel) { > /* C3 delay is worst case HW latency to enter/exit C3 state */ > if (C3 delay < 1000) notsc=1 >} >else { /* e.g. AMD */ > if (num_present_cpus() > 1) notsc=1 >} > >I'm not sure what constitutes a clustered HVM guest or how that >C3 state latency is determined under Xen, but its clear that the >clocksource can be influenced not only by what clock hardware is >present and command-line parameters but also by the physical CPU >and number of guest vcpus. > > I haven't looked into >Yuck! > >Thanks, >Dan > > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 26 Feb 2008 14:56:30 +0000 Message-ID: References: <47C42608.1090501@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47C42608.1090501@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org On 26/2/08 14:45, "Dave Winchell" wrote: >> If this is an integration of hpet into the vpt.c infrastructure then that >> would be very welcome. >> > So far it is not, however I may head in that direction. > Last night's test had an error of .065% on one guest so I still have some > work to do. Then I hope that the accuracy improvements have come from a set of changes whose effects are both explicable and generally for the good (rather than artefacts of how a particular sub-point release of Linux drives the HPET). -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 26 Feb 2008 09:45:28 -0500 Message-ID: <47C42608.1090501@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Deepak Patel , Dave Winchell List-Id: xen-devel@lists.xenproject.org Hi Keir - Keir Fraser wrote: >If this is an integration of hpet into the vpt.c infrastructure then that >would be very welcome. > > So far it is not, however I may head in that direction. Last night's test had an error of .065% on one guest so I still have some work to do. -Dave > -- Keir > >On 25/2/08 20:01, "Dave Winchell" wrote: > > > >>Hi Dan, >> >>I've added a few comments below ... >> >>Lately I've been busy getting the hpet emulation to be accurate. >>Right now I have the hpet based clock error down to under .02% with >>a limited amount of testing. (The error was 1% or more before my changes.) >>I hope to submit a patch soon so others can try this out. I need to do more >>testing and reduce my changes to the minimal set required. >> >>One thing I really like about the hpet is that a test designed to see if >>the clock will go backwards passes on the hpet where it fails on >>the pit/tsc clock. I tried for quite a while to keep pit/tsc from going >>backwards, but was unsuccessful. It ended up being easier to fix the hpet >>emulation for accuracy. >> >>The clock going backwards test is simply a call to gettimeofday() in a loop, >>with no delay between calls except to check for new_time < prev_time. >>I run this from a driver calling do_gettimeofday() to make it >>more stressful. Then I run an instance of this on all (8) vcpus on >>8 physical * 2 guests for a total of 16 vcpus running the test. >>The test does a yield now and then so you can actually log into the >>system, etc. >> >>Regards, >>Dave >> >>Dan Magenheimer wrote: >> >> >> >>>Hi Dave -- >>> >>>I've looked into RHEL5-64 a bit more and would appreciate >>>any thoughts you might have: >>> >>> >>> >>> >>> >>>>>So, some open questions: >>>>>[2] In RHEL5, I *think* it is the WALL source that we care about? >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>I'll have to check on this too. >>>> >>>> >>>> >>>> >>>On second look, I may be wrong. The GTOD clock seems to be >>>the one associated with vxtime.mode and vxtime.mode is used in >>>linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to >>>determine if a tick is lost or not. Is this the code that we >>>want timer_mode=2 to influence? >>> >>> >>> >>> >>Yes, it is. >> >> >> >>> >>> >>> >>> >>>>>[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? >>>>> (I *think* not as it has never come up in any email.) >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>I have not investigated this yet. >>>> >>>> >>>> >>>> >>>Well for RHEL5-64, looking at linux-2.6.18.8/arch/x86_64/kernel/time.c, >>>it appears whether notsc is required or not depends in part on the >>>underlying virtual AND physical system. >>> >>>notsc definitely is involved in the selection of GTOD time >>>but notsc can get set not only by kernel command line parameter >>>but also by the result of unsynchronized_tsc(): >>> >>> >>> >>> >>According to the comment in time.c, unsynchronized_tsc() is guessing >>whether the tsc is synchronized across processors. On most physical >>platforms they are, >>and, on those physical platforms the virtualization layer will present the >>same error as the hardware exibits. Right now we don't have code in xen to >>make a unsynchronized host look synchronized to the guest, though that >>wouldn't be very hard to do as long as the error between physical processors >>remained constant. >> >> >> >>>if (apic_is_clustered_box) notsc=1 >>>if (box_is_Intel) { >>>/* C3 delay is worst case HW latency to enter/exit C3 state */ >>>if (C3 delay < 1000) notsc=1 >>>} >>>else { /* e.g. AMD */ >>>if (num_present_cpus() > 1) notsc=1 >>>} >>> >>>I'm not sure what constitutes a clustered HVM guest or how that >>>C3 state latency is determined under Xen, but its clear that the >>>clocksource can be influenced not only by what clock hardware is >>>present and command-line parameters but also by the physical CPU >>>and number of guest vcpus. >>> >>> >>> >>> >>I haven't looked into >> >> >> >>>Yuck! >>> >>>Thanks, >>>Dan >>> >>> >>> >>> >>> >>> >>> > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Tue, 26 Feb 2008 10:49:01 -0500 Message-ID: <47C434ED.4090907@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org Keir Fraser wrote: >On 26/2/08 14:45, "Dave Winchell" wrote: > > > >>>If this is an integration of hpet into the vpt.c infrastructure then that >>>would be very welcome. >>> >>> >>> >>So far it is not, however I may head in that direction. >>Last night's test had an error of .065% on one guest so I still have some >>work to do. >> >> > >Then I hope that the accuracy improvements have come from a set of changes >whose effects are both explicable and generally for the good (rather than >artefacts of how a particular sub-point release of Linux drives the HPET). > > I believe that my changes meet these requirements. Your suggestion to layer on vpt.c is a good one that I will pursue. - Dave > -- Keir > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 5 Mar 2008 10:53:32 -0700 Message-ID: <20080305105332234.00000002164@djm-pc> References: <47CEDBA0.20103@virtualiron.com> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <47CEDBA0.20103@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , Keir Fraser Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org > >Do you see and simply ignore warning messages from 64-bit = > Linux when using > >hpet (or otherwise not doing missed-tick handling in Xen), = > by the way? I > >know 64-bit Linux is keen to warn about missed ticks, = > although it does look > >like at least the warning is one shot. > > > > > I see the one shot messages on 64 bit and no complaints at = > all on 4u432 > Linux. > And I have been ignoring them. Code to print the complaints is x86_64-specific in 2.6.9 and 2.6.18. On 64-bit, a command line parameter of "report_lost_ticks" will enable a complaint for every lost tick. As of 2.6.24, the lost ticks message seems to have gone away completely. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] Add a timer mode that disables pending missed ticks Date: Thu, 6 Mar 2008 16:36:16 -0700 Message-ID: <20080306163616812.00000002164@djm-pc> References: <20080225094202546.00000003172@djm-pc> Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <20080225094202546.00000003172@djm-pc> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: Keir Fraser , "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org (One more time on this old thread...) Hi Dave -- We have a strange but alarming result: Running rhel5ga-64 with timer_mode=3D0 has no skew but with timer_mode=3D2 has very bad skew. With rhel5u1-64 (and all rhel4-64 guests) the opposite is true. We repeated the tests to verify. In all cases, dmesg shows: time.c: Using 1.193182 MHz WALL PIT GTOD PIT/TSC timer. Could you try your heavy load tests with all four combinations of rhel5ga-64/rhel5u1-64 and timer_mode=3D0/2 to see if you can reproduce our results. Thanks, Dan > -----Original Message----- > From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > Sent: Monday, February 25, 2008 9:42 AM > To: 'Dave Winchell' > Cc: 'Keir Fraser'; 'xen-devel@lists.xensource.com'; 'Deepak Patel' > Subject: RE: [Xen-devel] [PATCH] Add a timer mode that = > disables pending > missed ticks > = > = > Hi Dave -- > = > I've looked into RHEL5-64 a bit more and would appreciate > any thoughts you might have: > = > > >So, some open questions: > > >[2] In RHEL5, I *think* it is the WALL source that we care about? > > > > > > > > I'll have to check on this too. > = > On second look, I may be wrong. The GTOD clock seems to be > the one associated with vxtime.mode and vxtime.mode is used in > linux-2.6.18.8/arch/x86_64/kernel/time.c:main_timer_handler() to > determine if a tick is lost or not. Is this the code that we > want timer_mode=3D2 to influence? > = > > >[1] Is notsc necessary for proper ticks for RHEL4-32/RHEL5-64? > > > (I *think* not as it has never come up in any email.) > > > > > > > > I have not investigated this yet. > = > Well for RHEL5-64, looking at = > linux-2.6.18.8/arch/x86_64/kernel/time.c, > it appears whether notsc is required or not depends in part on the > underlying virtual AND physical system. > = > notsc definitely is involved in the selection of GTOD time > but notsc can get set not only by kernel command line parameter > but also by the result of unsynchronized_tsc(): > = > if (apic_is_clustered_box) notsc=3D1 > if (box_is_Intel) { > /* C3 delay is worst case HW latency to enter/exit C3 state */ > if (C3 delay < 1000) notsc=3D1 > } > else { /* e.g. AMD */ > if (num_present_cpus() > 1) notsc=3D1 > } > = > I'm not sure what constitutes a clustered HVM guest or how that > C3 state latency is determined under Xen, but its clear that the > clocksource can be influenced not only by what clock hardware is > present and command-line parameters but also by the physical CPU > and number of guest vcpus. > = > Yuck! > = > Thanks, > Dan > = > = > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 05 Mar 2008 10:06:55 -0500 Message-ID: <47CEB70F.9010101@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser , "dan.magenheimer@oracle.com" Cc: Dave Winchell , "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org Keir, Dan: This is an update on the hpet work. With limited testing, the accuracy of this method appears to be at least 10 times better than the pit/tsc method for usex loads. Also, as mentioned before, it does not have the time going backwards problem. Another interesting property is that the same policy is used for a 32 bit Linux guest and 64 bit Linux. Two recent 14-16 hour tests were run, one with usex e48 and the other with usex b48 as loads. The same load was run on three 8 vcpu guests on an 8 cpu platform. No ntpd running. ntpdate -b used for initial setting of clock and ntpdate -q used for monitoring drift. The guests were 4u464, 4u564, 4u432 Linux. The results are: test duration drift (secs) 4u464,4u564,4u432 drift % usex b48 16 hrs -.68, -.60, -.68 -.0012 usex e48 15 hrs -.58, -.55, -.58 -.0011 The drift with pit/tsc as checked in reported a few months ago: >> The error tests confirm the equivalence. With overnight cpu loads, >> the checked in version was accurate to +.048% for sles >> and +.038% for red hat. Recall that .05% is the goal for accuracy so that ntpd can synchronize. With pit/tsc we are rather close to that limit. So far, with hpet, we are well under that goal. As for the code structure, I tried layering on vpt.c and had trouble doing so. Therefore, I focused on a direct approach. I'll describe the approach shortly, after I run tests with other loads and guests. Regards, Dave Keir Fraser wrote: >On 26/2/08 14:45, "Dave Winchell" wrote: > > > >>>If this is an integration of hpet into the vpt.c infrastructure then that >>>would be very welcome. >>> >>> >>> >>So far it is not, however I may head in that direction. >>Last night's test had an error of .065% on one guest so I still have some >>work to do. >> >> > >Then I hope that the accuracy improvements have come from a set of changes >whose effects are both explicable and generally for the good (rather than >artefacts of how a particular sub-point release of Linux drives the HPET). > > -- Keir > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 05 Mar 2008 15:20:20 +0000 Message-ID: References: <47CEB70F.9010101@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47CEB70F.9010101@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell , "dan.magenheimer@oracle.com" Cc: "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org On 5/3/08 15:06, "Dave Winchell" wrote: > As for the code structure, I tried layering on vpt.c and had trouble > doing so. Therefore, I focused on a direct approach. I'll describe the > approach > shortly, after I run tests with other loads and guests. Are you targeting only 64-bit Linux guests? 32-bit Linux cannot tolerate missed ticks, and I want only one missed-tick algorithm in Xen: currently that is vpt.c. -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 05 Mar 2008 12:25:24 -0500 Message-ID: <47CED784.3090403@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Deepak Patel , Dave Winchell List-Id: xen-devel@lists.xenproject.org Keir Fraser wrote: >On 5/3/08 15:06, "Dave Winchell" wrote: > > > >>As for the code structure, I tried layering on vpt.c and had trouble >>doing so. Therefore, I focused on a direct approach. I'll describe the >>approach >>shortly, after I run tests with other loads and guests. >> >> > >Are you targeting only 64-bit Linux guests? 32-bit Linux cannot tolerate >missed ticks, and I want only one missed-tick algorithm in Xen: currently >that is vpt.c. > > No, my results are for 32 bit Linux as well, at least 4u432. It turns out that for hpet, (rh4u4) 32 bit Linux does handle missed ticks. I need to try other 32 bit Linux versions, as well as Windows, to see how broad this behaviour is. Looking at the code for 2.6.20, it appears to me that if CONFIG_GENERIC_TIME is set then missed ticks are handled for hpet for 32 bit and 64 bit Linux (see update_wall_time() and read_hpet()). In 2.6.9, it looks like cur_timer->mark_offset() call from timer_interrupt in 32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(), which computes missed ticks based on hpet counter. mark_offset_pit() does nothing. mark_offset_tsc() does compute missed ticks. In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks. So from the code perspective, it looks like missed ticks are computed for 32 and 64 bit Linux using hpet clocksource. regards, Dave > -- Keir > > > > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 05 Mar 2008 17:21:09 +0000 Message-ID: References: <47CED784.3090403@virtualiron.com> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <47CED784.3090403@virtualiron.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dave Winchell Cc: "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Deepak Patel List-Id: xen-devel@lists.xenproject.org On 5/3/08 17:25, "Dave Winchell" wrote: > In 2.6.9, it looks like cur_timer->mark_offset() call from > timer_interrupt in > 32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(), > which computes > missed ticks based on hpet counter. mark_offset_pit() does nothing. > mark_offset_tsc() does compute missed ticks. > > In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet > reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks. > > So from the code perspective, it looks like missed ticks are computed > for 32 > and 64 bit Linux using hpet clocksource. Ah. I looked at 2.6.18 which seems to have neither the mark_offset nor the GENERIC_TIME approach in its arch/i386 time code. But yeah, it does look like in general Linux 2.6 is robust to missed ticks when using hpet. That's good. Do you see and simply ignore warning messages from 64-bit Linux when using hpet (or otherwise not doing missed-tick handling in Xen), by the way? I know 64-bit Linux is keen to warn about missed ticks, although it does look like at least the warning is one shot. -- Keir From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Winchell Subject: Re: (progress on hpet accuracy) and Re: [PATCH] Add a timer mode that disables pending missed ticks Date: Wed, 05 Mar 2008 12:42:56 -0500 Message-ID: <47CEDBA0.20103@virtualiron.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: "dan.magenheimer@oracle.com" , "xen-devel@lists.xensource.com" , Deepak Patel , Dave Winchell List-Id: xen-devel@lists.xenproject.org Keir Fraser wrote: >On 5/3/08 17:25, "Dave Winchell" wrote: > > > >>In 2.6.9, it looks like cur_timer->mark_offset() call from >>timer_interrupt in >>32 bit arch/i386/kernel/time.c invokes, for hpet, mark_offset_hpet(), >>which computes >>missed ticks based on hpet counter. mark_offset_pit() does nothing. >>mark_offset_tsc() does compute missed ticks. >> >>In 64 bit 2.6.9, the timer_interrupt() in arch/x86_64/time.c does hpet >>reads directly HPET_T0_CMP, HPET_COUNTER to calculate missed ticks. >> >>So from the code perspective, it looks like missed ticks are computed >>for 32 >>and 64 bit Linux using hpet clocksource. >> >> > >Ah. I looked at 2.6.18 which seems to have neither the mark_offset nor the >GENERIC_TIME approach in its arch/i386 time code. But yeah, it does look >like in general Linux 2.6 is robust to missed ticks when using hpet. That's >good. > >Do you see and simply ignore warning messages from 64-bit Linux when using >hpet (or otherwise not doing missed-tick handling in Xen), by the way? I >know 64-bit Linux is keen to warn about missed ticks, although it does look >like at least the warning is one shot. > > I see the one shot messages on 64 bit and no complaints at all on 4u432 Linux. And I have been ignoring them. > -- Keir > > > >