From mboxrd@z Thu Jan 1 00:00:00 1970 From: Olaf Hering Subject: PV performance degraded after live migration Date: Wed, 15 Mar 2017 12:20:44 +0100 Message-ID: <20170315112044.GA14366@aepfle.de> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7903504874652743383==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============7903504874652743383== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="G4iJoqBmSsgzjUCe" Content-Disposition: inline --G4iJoqBmSsgzjUCe Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable After reports about degraded performance after a PV domU was migrated =66rom one dom0 to another it turned out that this issue happens with every version of Xen and every version of domU kernel. The used benchmark is 'sysbench memory'. I hacked it up to show how long the actual work takes, and that loop takes longer to execute after the domU is migrated. In my testing the loop (memory_execute_event) takes about 1200ns, after migration it takes about 1500ns. It just writes 0 to an array of memory. In total sysbench reports 6500 MiB/sec, after migration its just 3350 MiB/sec. The source of the modified test can be found here: https://github.com/olafhering/sysbench/compare/master...pv This happens on several hosts. NUMA or not makes no difference. CPU pinning or not makes no difference. Pinning of the test pthreads makes no difference. It was initially reported with xen-4.4, I see it with staging too. The guest kernel makes no difference, several variants of xenlinux or pvops based show the slowdown. Also live migration to localhost is affected. The domU.cfg looks like that: name=3D'pv' memory=3D1024 vcpus=3D4 cpus=3D[ "4", "5", "6", "7" ] disk=3D[ 'file:/disk0.raw,xvda,w', ] vif=3D[ 'bridge=3Dbr0,type=3Dnetfront', ] vfb=3D[ 'type=3Dvnc,vncunused=3D1,keymap=3Dde', ] serial=3D"pty" kernel=3D"/usr/lib/grub2/x86_64-xen/grub.xen" Xen is booted with "console=3Dcom1 com1=3D115200 loglvl=3Dall guest_loglvl= =3Dall dom0_max_vcpus=3D2 dom0_vcpus_pin=3Don". I wonder what the cause might be, and how to check where the time is spent. Olaf --G4iJoqBmSsgzjUCe Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQSkRyP6Rn//f03pRUBdQqD6ppg2fgUCWMkjiAAKCRBdQqD6ppg2 fiwiAJ4jolkxz3bI3gzqgyjYcPc2GN2YygCguYI+ZdGS22/Mzul5u9jcS/kJ6Xs= =zDrJ -----END PGP SIGNATURE----- --G4iJoqBmSsgzjUCe-- --===============7903504874652743383== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============7903504874652743383==--