From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: How to migrate vCPUs based on Credit Scheduler Date: Tue, 4 Apr 2017 12:59:00 +0200 Message-ID: <1491303540.18721.1.camel@citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7174496471376072247==" Return-path: Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cvMBH-0002AF-Tt for xen-devel@lists.xenproject.org; Tue, 04 Apr 2017 10:59:24 +0000 In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Lars Kurth , =?UTF-8?Q?=E7=94=98=E6=B8=85=E7=94=9C?= Cc: "xen-devel@lists.xenproject.org" , Anshul Makkar , George Dunlap List-Id: xen-devel@lists.xenproject.org --===============7174496471376072247== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-YG1cMkLT9Ff4iHPgG+fG" --=-YG1cMkLT9Ff4iHPgG+fG Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2017-04-03 at 18:18 +0100, Lars Kurth wrote: > Adding George, Dario & Anshul > Hey, hello, Thanks Lars for the heads up... > > On 3 Apr 2017, at 17:21, =E7=94=98=E6=B8=85=E7=94=9C wrote: > >=20 > > Hi, > >=20 > > =C2=A0I'm now designing new vCPU scheduler in Xen, and trying to > > implement=C2=A0 > > the scheduler based on the Credit scheduler in Xen-4.5.1. > Can I ask what the purpose and the end goal of this is? 99% of the times, knowing that helps giving good advices. Also, are you forced to use 4.5.x for some specific reason? Because, if that is not the case, it's always better to base on the most possible recent code base (which, ideally, would be upstream git repo or, if that's not possible or convenient, the latest released version, i.e., 4.8, and soon enough, 4.9-rc). > > But I encountered > > =C2=A0come problems when debuging the code. > >=20 > > Most of the code modification is done in function csched_schedule() > > in=C2=A0 > > file: xen/common/csched_schedule.c . And the core code is as > > followed: > >=20 > > if( vcpu_runnable(current) ) > > { > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if( match the migration= contition ) > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0{ > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0cpu_affinity =3D pick_pcpu_runq();=C2=A0=C2=A0// this function is > > defined by myself > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0pcpulock =3D pcpu_schedule_lock_irqsave(cpu_affinity,=C2=A0 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0&pcpulock_flag); > >=20 Err... It's rather hard to comment without seeing the code. All of it, I mean. For instance, you're already in csched_schedule(), called, say, on cpu X, and hence you hold the scheduler lock of pcpu X. Calling pcpu_schedule_lock() like above, is at high risk of deadlock, depending of what "match the migration condition" actually means, and on how pick_pcpu_runq() is defined. In fact, if you look, for instance, in csched_load_balance(), you'll see that it does a trylock, exactly for this very reason. > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0TRACE_3D(TRC_CSCHED_STOLEN_VCPU, cpu_affinity=C2=A0=C2=A0, > > domain_id, > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0vcpu_id); > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0SCHED_VCPU_STAT_CRANK(scurr, migrate_q); > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0SCHED_STAT_CRANK(migrate_queued); > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0WARN_ON(scurr->vcpu->is_urgent); > > =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0scurr->vcpu->processor= =3D cpu_affinity;=09 > >=20 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0__runq_insert(cpu_affinity, scurr); > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0pcpu_schedule_unlock_irqrestore(pcpulock, > > pcpulock_flag, cpu_affinity=C2=A0=C2=A0); > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0} > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0else=C2=A0 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0__runq_insert(cpu, scurr); > > } > > else=C2=A0 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0BUG_ON( is_idle_vcpu(cu= rrent) || list_empty(runq) ); > >=20 > >=20 > > I try to run the modified Xen. But according to the log I found > > that,=C2=A0 > > although I insert the vCPU into the runqueue=C2=A0=C2=A0of another pCPU= , the=C2=A0 > > vCPU still appears at the old pCPU in the following scheduling > > period.=C2=A0 > Again: it's impossible to tell why this is happening, only looking at the code snipped above. For instance, what does csched_schedule() returns, in the case you'd want the migration to occur? That influences what will be run in the next scheduling period. > > Now I have a few questions: > >=20 > > 1. Does the Xen scheduler framework support changing the pCPU of a=C2= =A0 > > vCPU after using out the scheduling time slice, but not just to > > steal one=C2=A0 > > vCPU from runqueue of other pCPU in load_balance period? > >=20 It does such a thing already (at least, if I understood correctly what you're asking). Look at csched_load_balance() and csched_runq_steal(). They do exactly this. > > 2. If yes, what status of the vCPU should be changed before > > inserting=C2=A0 > > the vCPU into the destination pCPU? > >=20 The answer varies, depending on whether the vCPU is currently running on a pCPU, whether it is in a pCPU's runqueue, or whether it is blocked/sleeping. Looking closely at what csched_runq_steal() does is the best source of information, but really, in order to be able to say anything useful, we need to see the code, and to know what the end goal is. :-) Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-YG1cMkLT9Ff4iHPgG+fG Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJY43yDAAoJEBZCeImluHPuWkYP/2llAVP/CZSss+3dfLb2Bwt0 PJF2a4eIYh34sNbonfpI5eIe/ATonWZsh5+WHKCM1o1rCFEfi/EV5O+sPga+eN4M gwqbLCXh+ol+lPxgW1roVeV2or9zxlu00OQS+2NzAepatSmlmyBcsDl0CTCAeySA c0F5v2VaAGBshrbnBJjH8IPyjhTNdpMXLQ8SO56Tu18H4kladwHhqIs5Z7Ej5rU8 ++OQmIkb12ESHWHtgBvXikME+dQt+DLUj0KDtvPVU/rWwcE1L7PVpQyyBZvuLhgY 2kR30R55UY9qXVGsLV0qdh3vRPwSJlfXvswLIPdEWDiPBIRRytOyKEwkXzEUgqmM w8T4dL4pEqxSWp6g9a2P2V5QyYu6UKxPJyN1On2ImFsO2Rmgt9x40sIKz/ihXJhn p8IVXRVS6BoPagO9aHNW1C6j6QPVzO3Ljs3i2IMIn86YrHkAUoL7YuwNYxsx+BTc iKgNj2NEXL0RKdnDijSVpNxxIXej9Gl+n9anSMYmuhahMv882cJMhqtnglWHAZvU DkFw9M1hRkVtM7OWgyIY7Ho2nDsG+3dWNtoPGl0n2/8BcpdEt2hZ3tQL/0NC8PKn IsyaF/gAGdEbzww2q2zz8r7ZTwBjXyW1Qvj8pZHcUP7o4qAY1xp470SgH0WG9k4W MOV00JlvJSIf+ZI47nc0 =YXbm -----END PGP SIGNATURE----- --=-YG1cMkLT9Ff4iHPgG+fG-- --===============7174496471376072247== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============7174496471376072247==--