From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dan Magenheimer" Subject: RE: [PATCH] rendezvous-based local time calibration WOW! Date: Mon, 4 Aug 2008 09:24:46 -0600 Message-ID: <20080804092446812.00000008444@djm-pc> References: Reply-To: "dan.magenheimer@oracle.com" Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=-------3e3e6c4c3e3e6c4c Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser , "Xen-Devel (E-mail)" Cc: Ian Pratt , Dave Winchell List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format ---------3e3e6c4c3e3e6c4c Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable OK, how about this version. The rendezvous only collects the key per-cpu time data then sets up a per-cpu 1ms timer to later update the timestamp record and vcpu system time, so neither should have racing issues. I've only run it for about an hour but still haven't seen any skew over 600nsec so apparently it is the collection of the key time data that must be closely synchronized (probably to ensure the slope is correct) while exact synchronization of setting the timestamp records is less important. Note that I'm not positive I got the clocksource=3Dtsc part correct... but am interested in your opinion on whether clocksource=3Dtsc can now be eliminated anyway (as the main reason I pushed for it was because of unacceptable skew which with this patch appears to be fixed). Signed-off-by: Dan Magenheimer > -----Original Message----- > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > Sent: Sunday, August 03, 2008 11:25 AM > To: dan.magenheimer@oracle.com; Xen-Devel (E-mail) > Cc: Ian Pratt; Dave Winchell > Subject: Re: [PATCH] rendezvous-based local time calibration WOW! > = > = > It's not safe to poke a new timestamp record from an interrupt handler > (which is what the smp_call_function() callback functions = > are). Users of the > timestamp records (e.g., get_s_time) need = > local_irq_save/restore() or an > equivalent of the Linux seqlock. The latter is likely faster. = > I'm dubious > about update_vcpu_system_time() from an interrupt handler = > too. It needs > thought about how it might race with a context switch (change = > of 'current') > or if it interrupts an existing invocation of = > update_vcpu_system_time(). > = > -- Keir > = > On 3/8/08 17:50, "Dan Magenheimer" wrote: > = > > The synchronization of local_time_calibration (l_t_c) via > > round-to-nearest-epoch provided some improvement, but I was > > still seeing skew up to 16usec and higher. I measured the > > temporal distance between the rounded-epoch vs when ltc > > was actually running to ensure there wasn't some kind of > > bug and found that l_t_c was running up to 150us after the > > round-epoch and sometimes up to 50us before. I guess this > > is the granularity of setting a Xen timer. While it seemed > > that +/- 100us shouldn't cause that much skew, I finally > > decided to try synchronization-via-rendezvous, as suggested > > by Ian here: > > > > = > http://lists.xensource.com/archives/html/xen-devel/2008-07/msg 01074.html > http://lists.xensource.com/archives/html/xen-devel/2008-07/msg01080.html > > The result is phenomenal... using this approach (in attached > patch), I have yet to see a skew exceed 1usec!!! So this is > about a 10-fold increase in accuracy vs the rounded-epoch > method and about 20-fold over the one-epoch-from-NOW() method. > > The platform time is now read once for all processors rather > than once per processor. (Actually, it is read once again > in platform_time_calibration()... by "inlining" that routine > into master_local_time_calibration() that extra read can > be -- and probably should be -- avoided too.) > > It may be too late to get this into 3.3.0 but, if so, please > consider it asap for 3.3.1 rather than just xen-unstable/3.4. > > Dan > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Thanks... for the memory > I really could use more / My throughput's on the floor > The balloon is flat / My swap disk's fat / I've OOM's in store > Overcommitted so much > (with apologies to the late great Bob Hope) ---------3e3e6c4c3e3e6c4c Content-Type: application/octet-stream; name="rendezcalib3.patch" Content-Disposition: attachment; filename="rendezcalib3.patch" Content-Transfer-Encoding: base64 ZGlmZiAtciA4OTUxYzNiODRlMmEgeGVuL2FyY2gveDg2L3RpbWUuYwotLS0gYS94ZW4vYXJj aC94ODYvdGltZS5jCUZyaSBBdWcgMDEgMDk6NTQ6NTQgMjAwOCArMDEwMAorKysgYi94ZW4v YXJjaC94ODYvdGltZS5jCU1vbiBBdWcgMDQgMDk6MTA6MTQgMjAwOCAtMDYwMApAQCAtNTYs NyArNTYsNiBAQCBzdHJ1Y3QgY3B1X3RpbWUgewogICAgIHNfdGltZV90IHN0aW1lX21hc3Rl cl9zdGFtcDsKICAgICBzdHJ1Y3QgdGltZV9zY2FsZSB0c2Nfc2NhbGU7CiAgICAgdTY0IGNz dGF0ZV9wbHRfY291bnRfc3RhbXA7Ci0gICAgc3RydWN0IHRpbWVyIGNhbGlicmF0aW9uX3Rp bWVyOwogfTsKIAogc3RydWN0IHBsYXRmb3JtX3RpbWVzb3VyY2UgewpAQCAtNjYsNyArNjUs MTggQEAgc3RydWN0IHBsYXRmb3JtX3RpbWVzb3VyY2UgewogICAgIGludCBjb3VudGVyX2Jp dHM7CiB9OwogCitzdHJ1Y3QgY3Vycl9jcHVfdGltZSB7CisgICAgc190aW1lX3QgbG9jYWxf c3RpbWU7CisgICAgc190aW1lX3QgbWFzdGVyX3N0aW1lOworICAgIHU2NCBsb2NhbF90c2M7 CisgICAgc3RydWN0IHRpbWVyIGNhbGlicmF0aW9uX3RpbWVyOworfTsKKworc3RydWN0IHRp bWVyIG1hc3Rlcl9jYWxpYnJhdGlvbl90aW1lcjsKKwogc3RhdGljIERFRklORV9QRVJfQ1BV KHN0cnVjdCBjcHVfdGltZSwgY3B1X3RpbWUpOworLyogc2F2ZSB0aW1lIHZhbHVlcyBvYnRh aW5lZCBkdXJpbmcgaXJxIGZvciBuZXh0IHRpbWVyICovCitzdGF0aWMgREVGSU5FX1BFUl9D UFUoc3RydWN0IGN1cnJfY3B1X3RpbWUsIGN1cnJfY3B1X3RpbWUpOwogCiAvKiBUU0MgaXMg aW52YXJpYW50IG9uIEMgc3RhdGUgZW50cnk/ICovCiBzdGF0aWMgYm9vbF90IHRzY19pbnZh cmlhbnQ7CkBAIC04NDgsOSArODU4LDExIEBAIGludCBjcHVfZnJlcXVlbmN5X2NoYW5nZSh1 NjQgZnJlcSkKICAgICBsb2NhbF9pcnFfZW5hYmxlKCk7CiAKICAgICAvKiBBIGZ1bGwgZXBv Y2ggc2hvdWxkIHBhc3MgYmVmb3JlIHdlIGNoZWNrIGZvciBkZXZpYXRpb24uICovCi0gICAg c2V0X3RpbWVyKCZ0LT5jYWxpYnJhdGlvbl90aW1lciwgTk9XKCkgKyBFUE9DSCk7CiAgICAg aWYgKCBzbXBfcHJvY2Vzc29yX2lkKCkgPT0gMCApCisgICAgeworICAgICAgICBzZXRfdGlt ZXIoJm1hc3Rlcl9jYWxpYnJhdGlvbl90aW1lciwgTk9XKCkgKyBFUE9DSCk7CiAgICAgICAg IHBsYXRmb3JtX3RpbWVfY2FsaWJyYXRpb24oKTsKKyAgICB9CiAKICAgICByZXR1cm4gMDsK IH0KQEAgLTg3OSw2ICs4OTEsNyBAQCBzdGF0aWMgdm9pZCBsb2NhbF90aW1lX2NhbGlicmF0 aW9uKHZvaWQgCiBzdGF0aWMgdm9pZCBsb2NhbF90aW1lX2NhbGlicmF0aW9uKHZvaWQgKnVu dXNlZCkKIHsKICAgICBzdHJ1Y3QgY3B1X3RpbWUgKnQgPSAmdGhpc19jcHUoY3B1X3RpbWUp OworICAgIHN0cnVjdCBjdXJyX2NwdV90aW1lICpjID0gJnRoaXNfY3B1KGN1cnJfY3B1X3Rp bWUpOwogCiAgICAgLyoKICAgICAgKiBTeXN0ZW0gdGltZXN0YW1wcywgZXh0cmFwb2xhdGVk IGZyb20gbG9jYWwgYW5kIG1hc3RlciBvc2NpbGxhdG9ycywKQEAgLTkxMyw3ICs5MjYsNyBA QCBzdGF0aWMgdm9pZCBsb2NhbF90aW1lX2NhbGlicmF0aW9uKHZvaWQgCiAgICAgewogICAg ICAgICBtYWtlX3RzY3RpbWVyX3JlY29yZCgpOyAKICAgICAgICAgdXBkYXRlX3ZjcHVfc3lz dGVtX3RpbWUoY3VycmVudCk7Ci0gICAgICAgIHNldF90aW1lcigmdC0+Y2FsaWJyYXRpb25f dGltZXIsIE5PVygpICsgTUlMTElTRUNTKDEwKjEwMDApKTsKKyAgICAgICAgc2V0X3RpbWVy KCZtYXN0ZXJfY2FsaWJyYXRpb25fdGltZXIsIE5PVygpICsgTUlMTElTRUNTKDEwKjEwMDAp KTsKICAgICAgICAgcmV0dXJuOwogICAgIH0KIApAQCAtOTIxLDE1ICs5MzQsOSBAQCBzdGF0 aWMgdm9pZCBsb2NhbF90aW1lX2NhbGlicmF0aW9uKHZvaWQgCiAgICAgcHJldl9sb2NhbF9z dGltZSAgPSB0LT5zdGltZV9sb2NhbF9zdGFtcDsKICAgICBwcmV2X21hc3Rlcl9zdGltZSA9 IHQtPnN0aW1lX21hc3Rlcl9zdGFtcDsKIAotICAgIC8qCi0gICAgICogRGlzYWJsZSBJUlFz IHRvIGdldCAnaW5zdGFudGFuZW91cycgY3VycmVudCB0aW1lc3RhbXBzLiBXZSByZWFkIHBs YXRmb3JtCi0gICAgICogdGltZSBmaXJzdCwgYXMgd2UgbWF5IGJlIGRlbGF5ZWQgd2hlbiBh Y3F1aXJpbmcgcGxhdGZvcm1fdGltZXJfbG9jay4KLSAgICAgKi8KLSAgICBsb2NhbF9pcnFf ZGlzYWJsZSgpOwotICAgIGN1cnJfbWFzdGVyX3N0aW1lID0gcmVhZF9wbGF0Zm9ybV9zdGlt ZSgpOwotICAgIGN1cnJfbG9jYWxfc3RpbWUgID0gZ2V0X3NfdGltZSgpOwotICAgIHJkdHNj bGwoY3Vycl90c2MpOwotICAgIGxvY2FsX2lycV9lbmFibGUoKTsKKyAgICBjdXJyX2xvY2Fs X3N0aW1lID0gYy0+bG9jYWxfc3RpbWU7CisgICAgY3Vycl9tYXN0ZXJfc3RpbWUgPSBjLT5t YXN0ZXJfc3RpbWU7CisgICAgY3Vycl90c2MgPSBjLT5sb2NhbF90c2M7CiAKICNpZiAwCiAg ICAgcHJpbnRrKCJQUkUlZDogdHNjPSUiUFJJdTY0IiBzdGltZT0lIlBSSXU2NCIgbWFzdGVy PSUiUFJJdTY0IlxuIiwKQEAgLTEwMjEsMTYgKzEwMjgsNjMgQEAgc3RhdGljIHZvaWQgbG9j YWxfdGltZV9jYWxpYnJhdGlvbih2b2lkIAogCiAgICAgdXBkYXRlX3ZjcHVfc3lzdGVtX3Rp bWUoY3VycmVudCk7CiAKLSBvdXQ6Ci0gICAgc2V0X3RpbWVyKCZ0LT5jYWxpYnJhdGlvbl90 aW1lciwgTkVYVF9FUE9DSChjdXJyX2xvY2FsX3N0aW1lKSk7CitvdXQ6CisgICAgaWYgKCBz bXBfcHJvY2Vzc29yX2lkKCkgPT0gMCApCisgICAgeworICAgICAgICBwbGF0Zm9ybV90aW1l X2NhbGlicmF0aW9uKCk7CisgICAgICAgIHNldF90aW1lcigmbWFzdGVyX2NhbGlicmF0aW9u X3RpbWVyLCBORVhUX0VQT0NIKGN1cnJfbG9jYWxfc3RpbWUpKTsKKyAgICB9Cit9CiAKLSAg ICBpZiAoIHNtcF9wcm9jZXNzb3JfaWQoKSA9PSAwICkKLSAgICAgICAgcGxhdGZvcm1fdGlt ZV9jYWxpYnJhdGlvbigpOworc3RhdGljIGNwdW1hc2tfdCBsb2NhbF90aW1lX2NhbGlicmF0 ZV9jcHVtYXNrID0gQ1BVX01BU0tfTk9ORTsKK3NfdGltZV90IGN1cnJfbWFzdGVyX3N0aW1l OworCitzdGF0aWMgdm9pZCBzbGF2ZV90aW1lX2NhbGlicmF0aW9uKHZvaWQgKnVudXNlZCkK K3sKKyAgICB1bnNpZ25lZCBpbnQgY3B1ID0gc21wX3Byb2Nlc3Nvcl9pZCgpOworICAgIHN0 cnVjdCBjdXJyX2NwdV90aW1lICpjID0gJnRoaXNfY3B1KGN1cnJfY3B1X3RpbWUpOworCisg ICAgbG9jYWxfaXJxX2Rpc2FibGUoKTsKKyAgICB3aGlsZSAoICFjcHVfaXNzZXQoY3B1LCBs b2NhbF90aW1lX2NhbGlicmF0ZV9jcHVtYXNrKSApCisgICAgICAgIGNwdV9yZWxheCgpOwor ICAgIGMtPmxvY2FsX3N0aW1lID0gZ2V0X3NfdGltZSgpOworICAgIHJkdHNjbGwoYy0+bG9j YWxfdHNjKTsKKyAgICBjLT5tYXN0ZXJfc3RpbWUgPSBjdXJyX21hc3Rlcl9zdGltZTsKKyAg ICBjcHVfY2xlYXIoY3B1LCBsb2NhbF90aW1lX2NhbGlicmF0ZV9jcHVtYXNrKTsKKyAgICBz ZXRfdGltZXIoJmMtPmNhbGlicmF0aW9uX3RpbWVyLCBjLT5sb2NhbF9zdGltZSArIE1JTExJ U0VDUygxKSk7CisgICAgbG9jYWxfaXJxX2VuYWJsZSgpOworfQorCitzdGF0aWMgdm9pZCBt YXN0ZXJfdGltZV9jYWxpYnJhdGlvbih2b2lkICp1bnVzZWQpCit7CisgICAgdW5zaWduZWQg aW50IGNwdSA9IHNtcF9wcm9jZXNzb3JfaWQoKTsKKyAgICBzdHJ1Y3QgY3Vycl9jcHVfdGlt ZSAqYyA9ICZ0aGlzX2NwdShjdXJyX2NwdV90aW1lKTsKKworICAgIGlmICggcGxhdGZvcm1f dGltZXJfaXNfdHNjKCkgKQorICAgIHsKKyAgICAgICAgc21wX2NhbGxfZnVuY3Rpb24oc2xh dmVfdGltZV9jYWxpYnJhdGlvbiwgTlVMTCwgMCwgMCk7CisgICAgICAgIG1ha2VfdHNjdGlt ZXJfcmVjb3JkKCk7IAorICAgICAgICB1cGRhdGVfdmNwdV9zeXN0ZW1fdGltZShjdXJyZW50 KTsKKyAgICAgICAgc2V0X3RpbWVyKCZtYXN0ZXJfY2FsaWJyYXRpb25fdGltZXIsIE5PVygp ICsgTUlMTElTRUNTKDEwKjEwMDApKTsKKyAgICAgICAgcmV0dXJuOworICAgIH0KKworICAg IHNtcF9jYWxsX2Z1bmN0aW9uKHNsYXZlX3RpbWVfY2FsaWJyYXRpb24sIE5VTEwsIDAsIDAp OworCisgICAgbG9jYWxfaXJxX2Rpc2FibGUoKTsKKyAgICBjdXJyX21hc3Rlcl9zdGltZSA9 IGMtPm1hc3Rlcl9zdGltZSA9IHJlYWRfcGxhdGZvcm1fc3RpbWUoKTsKKyAgICBsb2NhbF90 aW1lX2NhbGlicmF0ZV9jcHVtYXNrID0gY3B1X29ubGluZV9tYXA7CisgICAgYy0+bG9jYWxf c3RpbWUgID0gZ2V0X3NfdGltZSgpOworICAgIHJkdHNjbGwoYy0+bG9jYWxfdHNjKTsKKyAg ICBjcHVfY2xlYXIoY3B1LCBsb2NhbF90aW1lX2NhbGlicmF0ZV9jcHVtYXNrKTsKKyAgICBz ZXRfdGltZXIoJmMtPmNhbGlicmF0aW9uX3RpbWVyLCBjLT5sb2NhbF9zdGltZSArIE1JTExJ U0VDUygxKSk7CisgICAgbG9jYWxfaXJxX2VuYWJsZSgpOwogfQogCiB2b2lkIGluaXRfcGVy Y3B1X3RpbWUodm9pZCkKIHsKICAgICBzdHJ1Y3QgY3B1X3RpbWUgKnQgPSAmdGhpc19jcHUo Y3B1X3RpbWUpOworICAgIHN0cnVjdCBjdXJyX2NwdV90aW1lICpjID0gJnRoaXNfY3B1KGN1 cnJfY3B1X3RpbWUpOwogICAgIHVuc2lnbmVkIGxvbmcgZmxhZ3M7CiAgICAgc190aW1lX3Qg bm93OwogCkBAIC0xMDQ5LDkgKzExMDMsMTQgQEAgdm9pZCBpbml0X3BlcmNwdV90aW1lKHZv aWQpCiAgICAgdC0+c3RpbWVfbG9jYWxfc3RhbXAgID0gbm93OwogCiAgb3V0OgotICAgIGlu aXRfdGltZXIoJnQtPmNhbGlicmF0aW9uX3RpbWVyLCBsb2NhbF90aW1lX2NhbGlicmF0aW9u LAorICAgIGlmICggc21wX3Byb2Nlc3Nvcl9pZCgpID09IDAgKQorICAgIHsKKyAgICAgICAg aW5pdF90aW1lcigmbWFzdGVyX2NhbGlicmF0aW9uX3RpbWVyLCBtYXN0ZXJfdGltZV9jYWxp YnJhdGlvbiwKICAgICAgICAgICAgICAgIE5VTEwsIHNtcF9wcm9jZXNzb3JfaWQoKSk7Ci0g ICAgc2V0X3RpbWVyKCZ0LT5jYWxpYnJhdGlvbl90aW1lciwgTkVYVF9FUE9DSChOT1coKSkp OworICAgICAgICBzZXRfdGltZXIoJm1hc3Rlcl9jYWxpYnJhdGlvbl90aW1lciwgTkVYVF9F UE9DSChOT1coKSkpOworICAgIH0KKyAgICBpbml0X3RpbWVyKCZjLT5jYWxpYnJhdGlvbl90 aW1lciwgbG9jYWxfdGltZV9jYWxpYnJhdGlvbiwKKyAgICAgICAgICAgICAgIE5VTEwsIHNt cF9wcm9jZXNzb3JfaWQoKSk7CiB9CiAKIC8qIExhdGUgaW5pdCBmdW5jdGlvbiAoYWZ0ZXIg YWxsIENQVXMgYXJlIGJvb3RlZCkuICovCkBAIC0xMTcwLDcgKzEyMjksNyBAQCBpbnQgdGlt ZV9zdXNwZW5kKHZvaWQpCiAgICAgfQogCiAgICAgLyogQmV0dGVyIHRvIGNhbmNlbCBjYWxp YnJhdGlvbiB0aW1lciBmb3IgYWNjdXJhY3kuICovCi0gICAga2lsbF90aW1lcigmdGhpc19j cHUoY3B1X3RpbWUpLmNhbGlicmF0aW9uX3RpbWVyKTsKKyAgICBraWxsX3RpbWVyKCZtYXN0 ZXJfY2FsaWJyYXRpb25fdGltZXIpOwogCiAgICAgcmV0dXJuIDA7CiB9Cg== ---------3e3e6c4c3e3e6c4c Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel ---------3e3e6c4c3e3e6c4c--