From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [PATCH v3 08/11] xen: sched: allow for choosing credit2 runqueues configuration at boot Date: Fri, 8 Apr 2016 09:35:25 +0200 Message-ID: <1460100925.13871.6.camel@citrix.com> References: <20160408011204.10762.14241.stgit@Solace.fritz.box> <20160408012420.10762.61178.stgit@Solace.fritz.box> <57073128.3030803@suse.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8050715208792021030==" Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aoQxE-0008IQ-6f for xen-devel@lists.xenproject.org; Fri, 08 Apr 2016 07:35:44 +0000 In-Reply-To: <57073128.3030803@suse.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Juergen Gross , xen-devel@lists.xenproject.org Cc: George Dunlap , Uma Sharma List-Id: xen-devel@lists.xenproject.org --===============8050715208792021030== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-DoybO45B+YD8S4moJpr8" --=-DoybO45B+YD8S4moJpr8 Content-Type: multipart/mixed; boundary="=-N8m7xUDYDvhU66ye/kNx" --=-N8m7xUDYDvhU66ye/kNx Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, 2016-04-08 at 06:18 +0200, Juergen Gross wrote: > On 08/04/16 03:24, Dario Faggioli wrote: > >=20 > > In fact, credit2 uses CPU topology to decide how to arrange > > its internal runqueues. Before this change, only 'one runqueue > > per socket' was allowed. However, experiments have shown that, > > for instance, having one runqueue per physical core improves > > performance, especially in case hyperthreading is available. > >=20 > > In general, it makes sense to allow users to pick one runqueue > > arrangement at boot time, so that: > > =C2=A0- more experiments can be easily performed to even better > > =C2=A0=C2=A0=C2=A0assess and improve performance; > > =C2=A0- one can select the best configuration for his specific > > =C2=A0=C2=A0=C2=A0use case and/or hardware. > >=20 > > This patch enables the above. > >=20 > > Note that, for correctly arranging runqueues to be per-core, > > just checking cpu_to_core() on the host CPUs is not enough. > > In fact, cores (and hyperthreads) on different sockets, can > > have the same core (and thread) IDs! We, therefore, need to > > check whether the full topology of two CPUs matches, for > > them to be put in the same runqueue. > >=20 > > Note also that the default (although not functional) for > > credit2, since now, has been per-socket runqueue. This patch > > leaves things that way, to avoid mixing policy and technical > > changes. > >=20 > > Finally, it would be a nice feature to be able to select > > a particular runqueue arrangement, even when creating a > > Credit2 cpupool. This is left as future work. > >=20 > > Signed-off-by: Dario Faggioli > > Signed-off-by: Uma Sharma > > Some nits below. >=20 Thanks for the quick review! A revised version of this patch is provided here (both inlined and attached), and a branch with the remaining to be committed patches of this series, and with this patch changed as you suggest, is available at: =C2=A0git://xenbits.xen.org/people/dariof/xen.git=C2=A0rel/sched/credit2/fi= x-runq-and-haff-v4 =C2=A0http://xenbits.xen.org/gitweb/?p=3Dpeople/dariof/xen.git;a=3Dshortlog= ;h=3Drefs/heads/rel/sched/credit2/fix-runq-and-haff-v4 Regards, Dario --- commit 7f491488bbff1cc3af021cd29fca7e0fba321e02 Author: Dario Faggioli Date:=C2=A0=C2=A0=C2=A0Tue Sep 29 14:05:09 2015 +0200 =C2=A0=C2=A0=C2=A0=C2=A0xen: sched: allow for choosing credit2 runqueues co= nfiguration at boot =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0In fact, credit2 uses CPU topology to decide how to= arrange =C2=A0=C2=A0=C2=A0=C2=A0its internal runqueues. Before this change, only 'o= ne runqueue =C2=A0=C2=A0=C2=A0=C2=A0per socket' was allowed. However, experiments have = shown that, =C2=A0=C2=A0=C2=A0=C2=A0for instance, having one runqueue per physical core= improves =C2=A0=C2=A0=C2=A0=C2=A0performance, especially in case hyperthreading is a= vailable. =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0In general, it makes sense to allow users to pick o= ne runqueue =C2=A0=C2=A0=C2=A0=C2=A0arrangement at boot time, so that: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- more experiments can be easily performed to= even better =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0assess and improve performance; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- one can select the best configuration for h= is specific =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0use case and/or hardware. =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0This patch enables the above. =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0Note that, for correctly arranging runqueues to be = per-core, =C2=A0=C2=A0=C2=A0=C2=A0just checking cpu_to_core() on the host CPUs is not= enough. =C2=A0=C2=A0=C2=A0=C2=A0In fact, cores (and hyperthreads) on different sock= ets, can =C2=A0=C2=A0=C2=A0=C2=A0have the same core (and thread) IDs! We, therefore,= need to =C2=A0=C2=A0=C2=A0=C2=A0check whether the full topology of two CPUs matches= , for =C2=A0=C2=A0=C2=A0=C2=A0them to be put in the same runqueue. =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0Note also that the default (although not functional= ) for =C2=A0=C2=A0=C2=A0=C2=A0credit2, since now, has been per-socket runqueue. T= his patch =C2=A0=C2=A0=C2=A0=C2=A0leaves things that way, to avoid mixing policy and = technical =C2=A0=C2=A0=C2=A0=C2=A0changes. =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0Finally, it would be a nice feature to be able to s= elect =C2=A0=C2=A0=C2=A0=C2=A0a particular runqueue arrangement, even when creati= ng a =C2=A0=C2=A0=C2=A0=C2=A0Credit2 cpupool. This is left as future work. =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0Signed-off-by: Dario Faggioli =C2=A0=C2=A0=C2=A0=C2=A0Signed-off-by: Uma Sharma =C2=A0=C2=A0=C2=A0=C2=A0--- =C2=A0=C2=A0=C2=A0=C2=A0Cc: George Dunlap =C2=A0=C2=A0=C2=A0=C2=A0Cc: Uma Sharma =C2=A0=C2=A0=C2=A0=C2=A0Cc: Juergen Gross =C2=A0=C2=A0=C2=A0=C2=A0--- =C2=A0=C2=A0=C2=A0=C2=A0Changes from v3: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0* fix type and other issue in comments; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0use ARRAY_SIZE when iterating the= parameter string array. =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0Changes from v2: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0* valid strings=C2=A0=C2=A0are now in an arra= y, that we scan during =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0parameter parsing, as suggested d= uring review. =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0Cahnges from v1: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0* fix bug in parameter parsing, and start usi= ng strcmp() =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0for that, as requested during rev= iew. diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-li= ne.markdown index ca77e3b..0047f94 100644 --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -469,6 +469,25 @@ combination with the `low_crashinfo` command line opti= on. =C2=A0### credit2\_load\_window\_shift =C2=A0> `=3D ` =C2=A0 +### credit2\_runqueue +> `=3D core | socket | node | all` + +> Default: `socket` + +Specify how host CPUs are arranged in runqueues. Runqueues are kept +balanced with respect to the load generated by the vCPUs running on +them. Smaller runqueues (as in with `core`) means more accurate load +balancing (for instance, it will deal better with hyperthreading), +but also more overhead. + +Available alternatives, with their meaning, are: +* `core`: one runqueue per each physical core of the host; +* `socket`: one runqueue per each physical socket (which often, +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bu= t not always, matches a NUMA node) of the host; +* `node`: one runqueue per each NUMA node of the host; +* `all`: just one runqueue shared by all the logical pCPUs of +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0the host + =C2=A0### dbgp =C2=A0> `=3D ehci[ | @pci:. ]` =C2=A0 diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index a61a45a..d43f67a 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -81,10 +81,6 @@ =C2=A0 * Credits are "reset" when the next vcpu in the runqueue is less tha= n =C2=A0 * or equal to zero.=C2=A0=C2=A0At that point, everyone's credits are= "clipped" =C2=A0 * to a small value, and a fixed credit is added to everyone. - * - * The plan is for all cores that share an L2 will share the same - * runqueue.=C2=A0=C2=A0At the moment, there is one global runqueue for al= l - * cores. =C2=A0 */ =C2=A0 =C2=A0/* @@ -193,6 +189,63 @@ static int __read_mostly opt_overload_balance_toleranc= e =3D -3; =C2=A0integer_param("credit2_balance_over", opt_overload_balance_tolerance)= ; =C2=A0 =C2=A0/* + * Runqueue organization. + * + * The various cpus are to be assigned each one to a runqueue, and we + * want that to happen basing on topology. At the moment, it is possible + * to choose to arrange runqueues to be: + * + * - per-core: meaning that there will be one runqueue per each physical + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0core of the host. This will happen if the opt_runqueue + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0parameter is set to 'core'; + * + * - per-socket: meaning that there will be one runqueue per each physical + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0socket (AKA package, which often, but not always, also + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0matches a NUMA node) of the host; This will happen if + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0the opt_runqueue parameter is set to 'socket'; + * + * - per-node: meaning that there will be one runqueue per each physical + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0NUMA node of the host. This will happen if the opt_runqueue + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0parameter is set to 'node'; + * + * - global: meaning that there will be only one runqueue to which all the + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0(logic= al) processors of the host belong. This will happen if + *=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0the op= t_runqueue parameter is set to 'all'. + * + * Depending on the value of opt_runqueue, therefore, cpus that are part o= f + * either the same physical core, the same physical socket, the same NUMA + * node, or just all of them, will be put together to form runqueues. + */ +#define OPT_RUNQUEUE_CORE=C2=A0=C2=A0=C2=A00 +#define OPT_RUNQUEUE_SOCKET 1 +#define OPT_RUNQUEUE_NODE=C2=A0=C2=A0=C2=A02 +#define OPT_RUNQUEUE_ALL=C2=A0=C2=A0=C2=A0=C2=A03 +static const char *const opt_runqueue_str[] =3D { +=C2=A0=C2=A0=C2=A0=C2=A0[OPT_RUNQUEUE_CORE] =3D "core", +=C2=A0=C2=A0=C2=A0=C2=A0[OPT_RUNQUEUE_SOCKET] =3D "socket", +=C2=A0=C2=A0=C2=A0=C2=A0[OPT_RUNQUEUE_NODE] =3D "node", +=C2=A0=C2=A0=C2=A0=C2=A0[OPT_RUNQUEUE_ALL] =3D "all" +}; +static int __read_mostly opt_runqueue =3D OPT_RUNQUEUE_SOCKET; + +static void parse_credit2_runqueue(const char *s) +{ +=C2=A0=C2=A0=C2=A0=C2=A0unsigned int i; + +=C2=A0=C2=A0=C2=A0=C2=A0for ( i =3D 0; i < ARRAY_SIZE(opt_runqueue_str); i= ++ ) +=C2=A0=C2=A0=C2=A0=C2=A0{ +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if ( !strcmp(s, opt_runque= ue_str[i]) ) +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0{ +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0op= t_runqueue =3D i; +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0re= turn; +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0} +=C2=A0=C2=A0=C2=A0=C2=A0} + +=C2=A0=C2=A0=C2=A0=C2=A0printk("WARNING, unrecognized value of credit2_run= queue option!\n"); +} +custom_param("credit2_runqueue", parse_credit2_runqueue); + +/* =C2=A0 * Per-runqueue data =C2=A0 */ =C2=A0struct csched2_runqueue_data { @@ -1974,6 +2027,22 @@ static void deactivate_runqueue(struct csched2_priva= te *prv, int rqi) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0cpumask_clear_cpu(rqi, &prv->active_queues); =C2=A0} =C2=A0 +static inline bool_t same_node(unsigned int cpua, unsigned int cpub) +{ +=C2=A0=C2=A0=C2=A0=C2=A0return cpu_to_node(cpua) =3D=3D cpu_to_node(cpub); +} + +static inline bool_t same_socket(unsigned int cpua, unsigned int cpub) +{ +=C2=A0=C2=A0=C2=A0=C2=A0return cpu_to_socket(cpua) =3D=3D cpu_to_socket(cp= ub); +} + +static inline bool_t same_core(unsigned int cpua, unsigned int cpub) +{ +=C2=A0=C2=A0=C2=A0=C2=A0return same_socket(cpua, cpub) && +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0cpu_to_c= ore(cpua) =3D=3D cpu_to_core(cpub); +} + =C2=A0static unsigned int =C2=A0cpu_to_runqueue(struct csched2_private *prv, unsigned int cpu) =C2=A0{ @@ -2006,7 +2075,10 @@ cpu_to_runqueue(struct csched2_private *prv, unsigne= d int cpu) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0BUG_ON(cpu_to_socket(= cpu) =3D=3D XEN_INVALID_SOCKET_ID || =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0cpu_to_socket(peer_cpu) =3D=3D XEN_INVALID_SOCKET_ID); =C2=A0 -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if ( cpu_to_socket(cpumask= _first(&rqd->active)) =3D=3D cpu_to_socket(cpu) ) +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if ( opt_runqueue =3D=3D O= PT_RUNQUEUE_ALL || +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0(opt_runqueue =3D=3D OPT_RUNQUEUE_CORE && same_core(peer_cpu, cpu)) |= | +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0(opt_runqueue =3D=3D OPT_RUNQUEUE_SOCKET && same_socket(peer_cpu, cpu= )) || +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0(opt_runqueue =3D=3D OPT_RUNQUEUE_NODE && same_node(peer_cpu, cpu)) ) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0break; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0} =C2=A0 @@ -2170,6 +2242,7 @@ csched2_init(struct scheduler *ops) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0printk(" load_window_shift: %d\n", opt_load_w= indow_shift); =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0printk(" underload_balance_tolerance: %d\n", = opt_underload_balance_tolerance); =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0printk(" overload_balance_tolerance: %d\n", o= pt_overload_balance_tolerance); +=C2=A0=C2=A0=C2=A0=C2=A0printk(" runqueues arrangement: %s\n", opt_runqueu= e_str[opt_runqueue]); =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if ( opt_load_window_shift < LOADAVG_WINDOW_S= HIFT_MIN ) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0{ --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-N8m7xUDYDvhU66ye/kNx Content-Disposition: attachment; filename="xen-sched-allow-for-choosing-credit2-runqueue.patch" Content-Transfer-Encoding: base64 Content-Type: text/x-patch; name="xen-sched-allow-for-choosing-credit2-runqueue.patch"; charset="UTF-8" Y29tbWl0IDdmNDkxNDg4YmJmZjFjYzNhZjAyMWNkMjlmY2E3ZTBmYmEzMjFlMDIKQXV0aG9yOiBE YXJpbyBGYWdnaW9saSA8ZGFyaW8uZmFnZ2lvbGlAY2l0cml4LmNvbT4KRGF0ZTogICBUdWUgU2Vw IDI5IDE0OjA1OjA5IDIwMTUgKzAyMDAKCiAgICB4ZW46IHNjaGVkOiBhbGxvdyBmb3IgY2hvb3Np bmcgY3JlZGl0MiBydW5xdWV1ZXMgY29uZmlndXJhdGlvbiBhdCBib290CiAgICAKICAgIEluIGZh Y3QsIGNyZWRpdDIgdXNlcyBDUFUgdG9wb2xvZ3kgdG8gZGVjaWRlIGhvdyB0byBhcnJhbmdlCiAg ICBpdHMgaW50ZXJuYWwgcnVucXVldWVzLiBCZWZvcmUgdGhpcyBjaGFuZ2UsIG9ubHkgJ29uZSBy dW5xdWV1ZQogICAgcGVyIHNvY2tldCcgd2FzIGFsbG93ZWQuIEhvd2V2ZXIsIGV4cGVyaW1lbnRz IGhhdmUgc2hvd24gdGhhdCwKICAgIGZvciBpbnN0YW5jZSwgaGF2aW5nIG9uZSBydW5xdWV1ZSBw ZXIgcGh5c2ljYWwgY29yZSBpbXByb3ZlcwogICAgcGVyZm9ybWFuY2UsIGVzcGVjaWFsbHkgaW4g Y2FzZSBoeXBlcnRocmVhZGluZyBpcyBhdmFpbGFibGUuCiAgICAKICAgIEluIGdlbmVyYWwsIGl0 IG1ha2VzIHNlbnNlIHRvIGFsbG93IHVzZXJzIHRvIHBpY2sgb25lIHJ1bnF1ZXVlCiAgICBhcnJh bmdlbWVudCBhdCBib290IHRpbWUsIHNvIHRoYXQ6CiAgICAgLSBtb3JlIGV4cGVyaW1lbnRzIGNh biBiZSBlYXNpbHkgcGVyZm9ybWVkIHRvIGV2ZW4gYmV0dGVyCiAgICAgICBhc3Nlc3MgYW5kIGlt cHJvdmUgcGVyZm9ybWFuY2U7CiAgICAgLSBvbmUgY2FuIHNlbGVjdCB0aGUgYmVzdCBjb25maWd1 cmF0aW9uIGZvciBoaXMgc3BlY2lmaWMKICAgICAgIHVzZSBjYXNlIGFuZC9vciBoYXJkd2FyZS4K ICAgIAogICAgVGhpcyBwYXRjaCBlbmFibGVzIHRoZSBhYm92ZS4KICAgIAogICAgTm90ZSB0aGF0 LCBmb3IgY29ycmVjdGx5IGFycmFuZ2luZyBydW5xdWV1ZXMgdG8gYmUgcGVyLWNvcmUsCiAgICBq dXN0IGNoZWNraW5nIGNwdV90b19jb3JlKCkgb24gdGhlIGhvc3QgQ1BVcyBpcyBub3QgZW5vdWdo LgogICAgSW4gZmFjdCwgY29yZXMgKGFuZCBoeXBlcnRocmVhZHMpIG9uIGRpZmZlcmVudCBzb2Nr ZXRzLCBjYW4KICAgIGhhdmUgdGhlIHNhbWUgY29yZSAoYW5kIHRocmVhZCkgSURzISBXZSwgdGhl cmVmb3JlLCBuZWVkIHRvCiAgICBjaGVjayB3aGV0aGVyIHRoZSBmdWxsIHRvcG9sb2d5IG9mIHR3 byBDUFVzIG1hdGNoZXMsIGZvcgogICAgdGhlbSB0byBiZSBwdXQgaW4gdGhlIHNhbWUgcnVucXVl dWUuCiAgICAKICAgIE5vdGUgYWxzbyB0aGF0IHRoZSBkZWZhdWx0IChhbHRob3VnaCBub3QgZnVu Y3Rpb25hbCkgZm9yCiAgICBjcmVkaXQyLCBzaW5jZSBub3csIGhhcyBiZWVuIHBlci1zb2NrZXQg cnVucXVldWUuIFRoaXMgcGF0Y2gKICAgIGxlYXZlcyB0aGluZ3MgdGhhdCB3YXksIHRvIGF2b2lk IG1peGluZyBwb2xpY3kgYW5kIHRlY2huaWNhbAogICAgY2hhbmdlcy4KICAgIAogICAgRmluYWxs eSwgaXQgd291bGQgYmUgYSBuaWNlIGZlYXR1cmUgdG8gYmUgYWJsZSB0byBzZWxlY3QKICAgIGEg cGFydGljdWxhciBydW5xdWV1ZSBhcnJhbmdlbWVudCwgZXZlbiB3aGVuIGNyZWF0aW5nIGEKICAg IENyZWRpdDIgY3B1cG9vbC4gVGhpcyBpcyBsZWZ0IGFzIGZ1dHVyZSB3b3JrLgogICAgCiAgICBT aWduZWQtb2ZmLWJ5OiBEYXJpbyBGYWdnaW9saSA8ZGFyaW8uZmFnZ2lvbGlAY2l0cml4LmNvbT4K ICAgIFNpZ25lZC1vZmYtYnk6IFVtYSBTaGFybWEgPHVtYS5zaGFybWE1MjNAZ21haWwuY29tPgog ICAgLS0tCiAgICBDYzogR2VvcmdlIER1bmxhcCA8Z2VvcmdlLmR1bmxhcEBldS5jaXRyaXguY29t PgogICAgQ2M6IFVtYSBTaGFybWEgPHVtYS5zaGFybWE1MjNAZ21haWwuY29tPgogICAgQ2M6IEp1 ZXJnZW4gR3Jvc3MgPGpncm9zc0BzdXNlLmNvbT4KICAgIC0tLQogICAgQ2hhbmdlcyBmcm9tIHYz OgogICAgICogZml4IHR5cGUgYW5kIG90aGVyIGlzc3VlIGluIGNvbW1lbnRzOwogICAgICAgdXNl IEFSUkFZX1NJWkUgd2hlbiBpdGVyYXRpbmcgdGhlIHBhcmFtZXRlciBzdHJpbmcgYXJyYXkuCiAg ICAKICAgIENoYW5nZXMgZnJvbSB2MjoKICAgICAqIHZhbGlkIHN0cmluZ3MgIGFyZSBub3cgaW4g YW4gYXJyYXksIHRoYXQgd2Ugc2NhbiBkdXJpbmcKICAgICAgIHBhcmFtZXRlciBwYXJzaW5nLCBh cyBzdWdnZXN0ZWQgZHVyaW5nIHJldmlldy4KICAgIAogICAgQ2FobmdlcyBmcm9tIHYxOgogICAg ICogZml4IGJ1ZyBpbiBwYXJhbWV0ZXIgcGFyc2luZywgYW5kIHN0YXJ0IHVzaW5nIHN0cmNtcCgp CiAgICAgICBmb3IgdGhhdCwgYXMgcmVxdWVzdGVkIGR1cmluZyByZXZpZXcuCgpkaWZmIC0tZ2l0 IGEvZG9jcy9taXNjL3hlbi1jb21tYW5kLWxpbmUubWFya2Rvd24gYi9kb2NzL21pc2MveGVuLWNv bW1hbmQtbGluZS5tYXJrZG93bgppbmRleCBjYTc3ZTNiLi4wMDQ3Zjk0IDEwMDY0NAotLS0gYS9k b2NzL21pc2MveGVuLWNvbW1hbmQtbGluZS5tYXJrZG93bgorKysgYi9kb2NzL21pc2MveGVuLWNv bW1hbmQtbGluZS5tYXJrZG93bgpAQCAtNDY5LDYgKzQ2OSwyNSBAQCBjb21iaW5hdGlvbiB3aXRo IHRoZSBgbG93X2NyYXNoaW5mb2AgY29tbWFuZCBsaW5lIG9wdGlvbi4KICMjIyBjcmVkaXQyXF9s b2FkXF93aW5kb3dcX3NoaWZ0CiA+IGA9IDxpbnRlZ2VyPmAKIAorIyMjIGNyZWRpdDJcX3J1bnF1 ZXVlCis+IGA9IGNvcmUgfCBzb2NrZXQgfCBub2RlIHwgYWxsYAorCis+IERlZmF1bHQ6IGBzb2Nr ZXRgCisKK1NwZWNpZnkgaG93IGhvc3QgQ1BVcyBhcmUgYXJyYW5nZWQgaW4gcnVucXVldWVzLiBS dW5xdWV1ZXMgYXJlIGtlcHQKK2JhbGFuY2VkIHdpdGggcmVzcGVjdCB0byB0aGUgbG9hZCBnZW5l cmF0ZWQgYnkgdGhlIHZDUFVzIHJ1bm5pbmcgb24KK3RoZW0uIFNtYWxsZXIgcnVucXVldWVzIChh cyBpbiB3aXRoIGBjb3JlYCkgbWVhbnMgbW9yZSBhY2N1cmF0ZSBsb2FkCitiYWxhbmNpbmcgKGZv ciBpbnN0YW5jZSwgaXQgd2lsbCBkZWFsIGJldHRlciB3aXRoIGh5cGVydGhyZWFkaW5nKSwKK2J1 dCBhbHNvIG1vcmUgb3ZlcmhlYWQuCisKK0F2YWlsYWJsZSBhbHRlcm5hdGl2ZXMsIHdpdGggdGhl aXIgbWVhbmluZywgYXJlOgorKiBgY29yZWA6IG9uZSBydW5xdWV1ZSBwZXIgZWFjaCBwaHlzaWNh bCBjb3JlIG9mIHRoZSBob3N0OworKiBgc29ja2V0YDogb25lIHJ1bnF1ZXVlIHBlciBlYWNoIHBo eXNpY2FsIHNvY2tldCAod2hpY2ggb2Z0ZW4sCisgICAgICAgICAgICBidXQgbm90IGFsd2F5cywg bWF0Y2hlcyBhIE5VTUEgbm9kZSkgb2YgdGhlIGhvc3Q7CisqIGBub2RlYDogb25lIHJ1bnF1ZXVl IHBlciBlYWNoIE5VTUEgbm9kZSBvZiB0aGUgaG9zdDsKKyogYGFsbGA6IGp1c3Qgb25lIHJ1bnF1 ZXVlIHNoYXJlZCBieSBhbGwgdGhlIGxvZ2ljYWwgcENQVXMgb2YKKyAgICAgICAgIHRoZSBob3N0 CisKICMjIyBkYmdwCiA+IGA9IGVoY2lbIDxpbnRlZ2VyPiB8IEBwY2k8YnVzPjo8c2xvdD4uPGZ1 bmM+IF1gCiAKZGlmZiAtLWdpdCBhL3hlbi9jb21tb24vc2NoZWRfY3JlZGl0Mi5jIGIveGVuL2Nv bW1vbi9zY2hlZF9jcmVkaXQyLmMKaW5kZXggYTYxYTQ1YS4uZDQzZjY3YSAxMDA2NDQKLS0tIGEv eGVuL2NvbW1vbi9zY2hlZF9jcmVkaXQyLmMKKysrIGIveGVuL2NvbW1vbi9zY2hlZF9jcmVkaXQy LmMKQEAgLTgxLDEwICs4MSw2IEBACiAgKiBDcmVkaXRzIGFyZSAicmVzZXQiIHdoZW4gdGhlIG5l eHQgdmNwdSBpbiB0aGUgcnVucXVldWUgaXMgbGVzcyB0aGFuCiAgKiBvciBlcXVhbCB0byB6ZXJv LiAgQXQgdGhhdCBwb2ludCwgZXZlcnlvbmUncyBjcmVkaXRzIGFyZSAiY2xpcHBlZCIKICAqIHRv IGEgc21hbGwgdmFsdWUsIGFuZCBhIGZpeGVkIGNyZWRpdCBpcyBhZGRlZCB0byBldmVyeW9uZS4K LSAqCi0gKiBUaGUgcGxhbiBpcyBmb3IgYWxsIGNvcmVzIHRoYXQgc2hhcmUgYW4gTDIgd2lsbCBz aGFyZSB0aGUgc2FtZQotICogcnVucXVldWUuICBBdCB0aGUgbW9tZW50LCB0aGVyZSBpcyBvbmUg Z2xvYmFsIHJ1bnF1ZXVlIGZvciBhbGwKLSAqIGNvcmVzLgogICovCiAKIC8qCkBAIC0xOTMsNiAr MTg5LDYzIEBAIHN0YXRpYyBpbnQgX19yZWFkX21vc3RseSBvcHRfb3ZlcmxvYWRfYmFsYW5jZV90 b2xlcmFuY2UgPSAtMzsKIGludGVnZXJfcGFyYW0oImNyZWRpdDJfYmFsYW5jZV9vdmVyIiwgb3B0 X292ZXJsb2FkX2JhbGFuY2VfdG9sZXJhbmNlKTsKIAogLyoKKyAqIFJ1bnF1ZXVlIG9yZ2FuaXph dGlvbi4KKyAqCisgKiBUaGUgdmFyaW91cyBjcHVzIGFyZSB0byBiZSBhc3NpZ25lZCBlYWNoIG9u ZSB0byBhIHJ1bnF1ZXVlLCBhbmQgd2UKKyAqIHdhbnQgdGhhdCB0byBoYXBwZW4gYmFzaW5nIG9u IHRvcG9sb2d5LiBBdCB0aGUgbW9tZW50LCBpdCBpcyBwb3NzaWJsZQorICogdG8gY2hvb3NlIHRv IGFycmFuZ2UgcnVucXVldWVzIHRvIGJlOgorICoKKyAqIC0gcGVyLWNvcmU6IG1lYW5pbmcgdGhh dCB0aGVyZSB3aWxsIGJlIG9uZSBydW5xdWV1ZSBwZXIgZWFjaCBwaHlzaWNhbAorICogICAgICAg ICAgICAgY29yZSBvZiB0aGUgaG9zdC4gVGhpcyB3aWxsIGhhcHBlbiBpZiB0aGUgb3B0X3J1bnF1 ZXVlCisgKiAgICAgICAgICAgICBwYXJhbWV0ZXIgaXMgc2V0IHRvICdjb3JlJzsKKyAqCisgKiAt IHBlci1zb2NrZXQ6IG1lYW5pbmcgdGhhdCB0aGVyZSB3aWxsIGJlIG9uZSBydW5xdWV1ZSBwZXIg ZWFjaCBwaHlzaWNhbAorICogICAgICAgICAgICAgICBzb2NrZXQgKEFLQSBwYWNrYWdlLCB3aGlj aCBvZnRlbiwgYnV0IG5vdCBhbHdheXMsIGFsc28KKyAqICAgICAgICAgICAgICAgbWF0Y2hlcyBh IE5VTUEgbm9kZSkgb2YgdGhlIGhvc3Q7IFRoaXMgd2lsbCBoYXBwZW4gaWYKKyAqICAgICAgICAg ICAgICAgdGhlIG9wdF9ydW5xdWV1ZSBwYXJhbWV0ZXIgaXMgc2V0IHRvICdzb2NrZXQnOworICoK KyAqIC0gcGVyLW5vZGU6IG1lYW5pbmcgdGhhdCB0aGVyZSB3aWxsIGJlIG9uZSBydW5xdWV1ZSBw ZXIgZWFjaCBwaHlzaWNhbAorICogICAgICAgICAgICAgTlVNQSBub2RlIG9mIHRoZSBob3N0LiBU aGlzIHdpbGwgaGFwcGVuIGlmIHRoZSBvcHRfcnVucXVldWUKKyAqICAgICAgICAgICAgIHBhcmFt ZXRlciBpcyBzZXQgdG8gJ25vZGUnOworICoKKyAqIC0gZ2xvYmFsOiBtZWFuaW5nIHRoYXQgdGhl cmUgd2lsbCBiZSBvbmx5IG9uZSBydW5xdWV1ZSB0byB3aGljaCBhbGwgdGhlCisgKiAgICAgICAg ICAgKGxvZ2ljYWwpIHByb2Nlc3NvcnMgb2YgdGhlIGhvc3QgYmVsb25nLiBUaGlzIHdpbGwgaGFw cGVuIGlmCisgKiAgICAgICAgICAgdGhlIG9wdF9ydW5xdWV1ZSBwYXJhbWV0ZXIgaXMgc2V0IHRv ICdhbGwnLgorICoKKyAqIERlcGVuZGluZyBvbiB0aGUgdmFsdWUgb2Ygb3B0X3J1bnF1ZXVlLCB0 aGVyZWZvcmUsIGNwdXMgdGhhdCBhcmUgcGFydCBvZgorICogZWl0aGVyIHRoZSBzYW1lIHBoeXNp Y2FsIGNvcmUsIHRoZSBzYW1lIHBoeXNpY2FsIHNvY2tldCwgdGhlIHNhbWUgTlVNQQorICogbm9k ZSwgb3IganVzdCBhbGwgb2YgdGhlbSwgd2lsbCBiZSBwdXQgdG9nZXRoZXIgdG8gZm9ybSBydW5x dWV1ZXMuCisgKi8KKyNkZWZpbmUgT1BUX1JVTlFVRVVFX0NPUkUgICAwCisjZGVmaW5lIE9QVF9S VU5RVUVVRV9TT0NLRVQgMQorI2RlZmluZSBPUFRfUlVOUVVFVUVfTk9ERSAgIDIKKyNkZWZpbmUg T1BUX1JVTlFVRVVFX0FMTCAgICAzCitzdGF0aWMgY29uc3QgY2hhciAqY29uc3Qgb3B0X3J1bnF1 ZXVlX3N0cltdID0geworICAgIFtPUFRfUlVOUVVFVUVfQ09SRV0gPSAiY29yZSIsCisgICAgW09Q VF9SVU5RVUVVRV9TT0NLRVRdID0gInNvY2tldCIsCisgICAgW09QVF9SVU5RVUVVRV9OT0RFXSA9 ICJub2RlIiwKKyAgICBbT1BUX1JVTlFVRVVFX0FMTF0gPSAiYWxsIgorfTsKK3N0YXRpYyBpbnQg X19yZWFkX21vc3RseSBvcHRfcnVucXVldWUgPSBPUFRfUlVOUVVFVUVfU09DS0VUOworCitzdGF0 aWMgdm9pZCBwYXJzZV9jcmVkaXQyX3J1bnF1ZXVlKGNvbnN0IGNoYXIgKnMpCit7CisgICAgdW5z aWduZWQgaW50IGk7CisKKyAgICBmb3IgKCBpID0gMDsgaSA8IEFSUkFZX1NJWkUob3B0X3J1bnF1 ZXVlX3N0cik7IGkrKyApCisgICAgeworICAgICAgICBpZiAoICFzdHJjbXAocywgb3B0X3J1bnF1 ZXVlX3N0cltpXSkgKQorICAgICAgICB7CisgICAgICAgICAgICBvcHRfcnVucXVldWUgPSBpOwor ICAgICAgICAgICAgcmV0dXJuOworICAgICAgICB9CisgICAgfQorCisgICAgcHJpbnRrKCJXQVJO SU5HLCB1bnJlY29nbml6ZWQgdmFsdWUgb2YgY3JlZGl0Ml9ydW5xdWV1ZSBvcHRpb24hXG4iKTsK K30KK2N1c3RvbV9wYXJhbSgiY3JlZGl0Ml9ydW5xdWV1ZSIsIHBhcnNlX2NyZWRpdDJfcnVucXVl dWUpOworCisvKgogICogUGVyLXJ1bnF1ZXVlIGRhdGEKICAqLwogc3RydWN0IGNzY2hlZDJfcnVu cXVldWVfZGF0YSB7CkBAIC0xOTc0LDYgKzIwMjcsMjIgQEAgc3RhdGljIHZvaWQgZGVhY3RpdmF0 ZV9ydW5xdWV1ZShzdHJ1Y3QgY3NjaGVkMl9wcml2YXRlICpwcnYsIGludCBycWkpCiAgICAgY3B1 bWFza19jbGVhcl9jcHUocnFpLCAmcHJ2LT5hY3RpdmVfcXVldWVzKTsKIH0KIAorc3RhdGljIGlu bGluZSBib29sX3Qgc2FtZV9ub2RlKHVuc2lnbmVkIGludCBjcHVhLCB1bnNpZ25lZCBpbnQgY3B1 YikKK3sKKyAgICByZXR1cm4gY3B1X3RvX25vZGUoY3B1YSkgPT0gY3B1X3RvX25vZGUoY3B1Yik7 Cit9CisKK3N0YXRpYyBpbmxpbmUgYm9vbF90IHNhbWVfc29ja2V0KHVuc2lnbmVkIGludCBjcHVh LCB1bnNpZ25lZCBpbnQgY3B1YikKK3sKKyAgICByZXR1cm4gY3B1X3RvX3NvY2tldChjcHVhKSA9 PSBjcHVfdG9fc29ja2V0KGNwdWIpOworfQorCitzdGF0aWMgaW5saW5lIGJvb2xfdCBzYW1lX2Nv cmUodW5zaWduZWQgaW50IGNwdWEsIHVuc2lnbmVkIGludCBjcHViKQoreworICAgIHJldHVybiBz YW1lX3NvY2tldChjcHVhLCBjcHViKSAmJgorICAgICAgICAgICBjcHVfdG9fY29yZShjcHVhKSA9 PSBjcHVfdG9fY29yZShjcHViKTsKK30KKwogc3RhdGljIHVuc2lnbmVkIGludAogY3B1X3RvX3J1 bnF1ZXVlKHN0cnVjdCBjc2NoZWQyX3ByaXZhdGUgKnBydiwgdW5zaWduZWQgaW50IGNwdSkKIHsK QEAgLTIwMDYsNyArMjA3NSwxMCBAQCBjcHVfdG9fcnVucXVldWUoc3RydWN0IGNzY2hlZDJfcHJp dmF0ZSAqcHJ2LCB1bnNpZ25lZCBpbnQgY3B1KQogICAgICAgICBCVUdfT04oY3B1X3RvX3NvY2tl dChjcHUpID09IFhFTl9JTlZBTElEX1NPQ0tFVF9JRCB8fAogICAgICAgICAgICAgICAgY3B1X3Rv X3NvY2tldChwZWVyX2NwdSkgPT0gWEVOX0lOVkFMSURfU09DS0VUX0lEKTsKIAotICAgICAgICBp ZiAoIGNwdV90b19zb2NrZXQoY3B1bWFza19maXJzdCgmcnFkLT5hY3RpdmUpKSA9PSBjcHVfdG9f c29ja2V0KGNwdSkgKQorICAgICAgICBpZiAoIG9wdF9ydW5xdWV1ZSA9PSBPUFRfUlVOUVVFVUVf QUxMIHx8CisgICAgICAgICAgICAgKG9wdF9ydW5xdWV1ZSA9PSBPUFRfUlVOUVVFVUVfQ09SRSAm JiBzYW1lX2NvcmUocGVlcl9jcHUsIGNwdSkpIHx8CisgICAgICAgICAgICAgKG9wdF9ydW5xdWV1 ZSA9PSBPUFRfUlVOUVVFVUVfU09DS0VUICYmIHNhbWVfc29ja2V0KHBlZXJfY3B1LCBjcHUpKSB8 fAorICAgICAgICAgICAgIChvcHRfcnVucXVldWUgPT0gT1BUX1JVTlFVRVVFX05PREUgJiYgc2Ft ZV9ub2RlKHBlZXJfY3B1LCBjcHUpKSApCiAgICAgICAgICAgICBicmVhazsKICAgICB9CiAKQEAg LTIxNzAsNiArMjI0Miw3IEBAIGNzY2hlZDJfaW5pdChzdHJ1Y3Qgc2NoZWR1bGVyICpvcHMpCiAg ICAgcHJpbnRrKCIgbG9hZF93aW5kb3dfc2hpZnQ6ICVkXG4iLCBvcHRfbG9hZF93aW5kb3dfc2hp ZnQpOwogICAgIHByaW50aygiIHVuZGVybG9hZF9iYWxhbmNlX3RvbGVyYW5jZTogJWRcbiIsIG9w dF91bmRlcmxvYWRfYmFsYW5jZV90b2xlcmFuY2UpOwogICAgIHByaW50aygiIG92ZXJsb2FkX2Jh bGFuY2VfdG9sZXJhbmNlOiAlZFxuIiwgb3B0X292ZXJsb2FkX2JhbGFuY2VfdG9sZXJhbmNlKTsK KyAgICBwcmludGsoIiBydW5xdWV1ZXMgYXJyYW5nZW1lbnQ6ICVzXG4iLCBvcHRfcnVucXVldWVf c3RyW29wdF9ydW5xdWV1ZV0pOwogCiAgICAgaWYgKCBvcHRfbG9hZF93aW5kb3dfc2hpZnQgPCBM T0FEQVZHX1dJTkRPV19TSElGVF9NSU4gKQogICAgIHsK --=-N8m7xUDYDvhU66ye/kNx-- --=-DoybO45B+YD8S4moJpr8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEABECAAYFAlcHXz4ACgkQk4XaBE3IOsQZvQCfRYki0Qn2AVBDnpKzOgJParB6 pCgAn0IoGWlOvaqsDOv5LjDwOOVTCVSt =K1rV -----END PGP SIGNATURE----- --=-DoybO45B+YD8S4moJpr8-- --===============8050715208792021030== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwOi8vbGlzdHMueGVuLm9y Zy94ZW4tZGV2ZWwK --===============8050715208792021030==--