From mboxrd@z Thu Jan 1 00:00:00 1970 From: Philipp Hahn Subject: [BUG 2.6.32.y] Broken PV migration between hosts with different uptime, non-monotonic time? Date: Fri, 4 May 2012 10:54:03 +0200 Message-ID: <201205041054.04079.hahn@univention.de> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4798092387412600323==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Xen-devel@lists.xen.org Cc: Thomas Gleixner , Jeremy Fitzhardinge List-Id: xen-devel@lists.xenproject.org --===============4798092387412600323== Content-Type: multipart/signed; boundary="nextPart13234650.LQF3ZOgi9e"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit --nextPart13234650.LQF3ZOgi9e Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hello, I encountered the following bug when migrating a Linux-2.6.32.54 PV domain = on=20 Xen-3.4.3 between different hosts, whose uptime differs by several minutes = (3=20 hosts, each ~5 minutes apart): When migrating from a host with lower uptime= =20 to a host with higher uptime, the VM looses it's network connection for som= e=20 time and then continues after some minutes (roughly equivalent to the=20 difference in uptime?). There are two different symptoms: Either the VM becomes unpingable, or the = VM=20 is pingable but the ssh-connection freezes: a while-loop dumping /proc/upti= me=20 freezes and continues without a jump after the freeze is over. When looking at the output of dmesg of the domU, I also see a jump in the=20 timestamp: [1967742.320218] eth0: no IPv6 routers present [1968779.217256] suspending xenstore... [1968779.217358] trying to map vcpu_info 0 at ffff88000bcbc020, mfn 85e61e, offset 32 [1968779.217358] cpu 0 using vcpu_info at ffff88000bcbc020 [ 5655.842391] suspending xenstore... [ 5655.842477] trying to map vcpu_info 0 at ffff88000bcbc020, mfn d5e61e, offset 32 [ 5655.842477] cpu 0 using vcpu_info at ffff88000bcbc020 [ 7745.941585] suspending xenstore... [ 7745.941667] trying to map vcpu_info 0 at ffff88000bcbc020, mfn be4163, offset 32 [ 7745.941667] cpu 0 using vcpu_info at ffff88000bcbc020 [342272.197261] suspending xenstore... If I revert the following commit (original from 2.6.36-rc1), the problem do= es=20 not show in 2.6.32.y: commit 8a22b9996b001c88f2bfb54c6de6a05fc39e177a Author: Jeremy Fitzhardinge Date: Mon Jul 12 11:49:59 2010 -0700 xen: drop xen_sched_clock in favour of using plain wallclock time =20 xen_sched_clock only counts unstolen time. In principle this should be useful to the Linux scheduler so that it knows how much time a proce= ss actually consumed. But in practice this doesn't work very well as the scheduler expects the sched_clock time to be synchronized between cpus. It also uses sched_clock to measure the time a task spends sleeping, in which case "unstolen time" isn't meaningful. =20 So just use plain xen_clocksource_read to return wallclock nanoseconds for sched_clock. 2.6.36 does not work, since 489fb49 and e7a3481 are missing: Without=20 the "global synchonization point for pvclock" (AKA last_value) plus the fix= =20 to "reset it to 0 on resume" VMs migrate fine in the opposite direction=20 (older=3Dhigher uptime =E2=86=92 newer=3Dlower uptime), but the original di= rection=20 (lower=E2=86=92higher) now stalls for 5 minutes. 2.6.37 (which includes above patches) works fine in both directions (I only= =20 see a 2 second network dropout for 2 VMs going lower=E2=86=92higher). So so= mething=20 other must have changed also, which is missing in 2.6.32.59 so far. I tried to unserstand all those clockevent, timer, pvclock, sched_clock()=20 details, but now I'm stuck. To me it looks like xen_clocksource_read() is n= ot=20 monotonic over migration, which seems to break some assumtion of=20 sched_clock() being monotonic. Has sombody else observed a similar problem and can provide a helpful hint? Is there anything I can look at to get this issue solved? Sincerely Philipp PS: bisecting did not help much, since 2.6.32.59 contains a lot of back-por= ts=20 from 2.6.33, 35, 36 and 37. 2.6.33 needs 281ff33 # x86_64, cpa: Don't work hard in preserving kernel 2M= =20 mappings when using 4K already 2.6.33-rc1: c5cae66 fixes 65f6338 # do_suspend error handling 2.6.35-rc1: e7a3481 fixes 489fb49 # global sync point 2.6.37 needs ceff1a7 # /proc/kcore: fix seeking =2D-=20 Philipp Hahn Open Source Software Engineer hahn@univention.de Univention GmbH be open. fon: +49 421 22 232- 0 Mary-Somerville-Str.1 D-28359 Bremen fax: +49 421 22 232-99 http://www.univention.de/ --nextPart13234650.LQF3ZOgi9e Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEABECAAYFAk+jmSwACgkQYPlgoZpUDjkQjwCfXx7naF4FJqPmVxb90peYuItO qPgAn0BQiYLhoCo2839rE8Hx7LVZRMjQ =zwQo -----END PGP SIGNATURE----- --nextPart13234650.LQF3ZOgi9e-- --===============4798092387412600323== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============4798092387412600323==--