From mboxrd@z Thu Jan 1 00:00:00 1970 From: Duyck, Alexander H Date: Thu, 16 Nov 2017 21:09:12 +0000 Subject: [Intel-wired-lan] [PATCH 0/7] [RESEND] [net] intel: Use smp_rmb rather than read_barrier_depends In-Reply-To: References: <1510846675-15169-1-git-send-email-brking@linux.vnet.ibm.com> <20171116113358.00001d5a@intel.com> Message-ID: <1510866549.28435.26.camel@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On Thu, 2017-11-16 at 14:03 -0600, Brian King wrote: > On 11/16/2017 01:33 PM, Jesse Brandeburg wrote: > > On Thu, 16 Nov 2017 09:37:48 -0600 > > Brian King wrote: > > > > > Resending as the first attempt is not showing up in the list archive. > > > > > > This patch converts several network drivers to use smp_rmb > > > rather than read_barrier_depends. The initial issue was > > > discovered with ixgbe on a Power machine which resulted > > > in skb list corruption due to fetching a stale skb pointer. > > > More details can be found in the ixgbe patch description. > > > > Thanks for the fix Brian, I bet it was a tough debug. > > > > The only users in the entire kernel of read_barrier_depends() (not > > smp_read_barrier_depends) are the Intel network drivers. > > > > Wouldn't it be better for power to just fix read_barrier_depends to do > > the right thing on power? The question I'm not sure of the answer to is: > > Is it really the wrong barrier to be using or is the implementation in > > the kernel powerpc wrong? > > > > So I think the right thing might actually to be to: > > Fix arch powerpc read_barrier_depends to not be a noop, as the > > semantics of the read_barrier_depends seems to be sufficient to solve > > this problem, but it seems not to work for powerpc? > > Jesse, > > Thanks for the quick response. > > Cc'ing linuxppc-dev as well. > > I did think about changing the powerpc definition of read_barrier_depends, > but after reading up on that barrier, decided it was not the correct barrier > to be used in this context. Here is some good historical background on > read_barrier_depends that I found, along with an example. > > https://lwn.net/Articles/5159/ > > Since there is no data-dependency in the code in question here, I think > the smp_rmb is the proper barrier to use. > > For background, the code in question looks like this: > > CPU 1 CPU2 > ============================ ============================ > 1: ixgbe_xmit_frame_ring ixgbe_clean_tx_irq > 2: first->skb = skb eop_desc = tx_buffer->next_to_watch > if (!eop_desc) > break; > 3: ixgbe_tx_map read_barrier_depends() > if (!(eop_desc->wb.status) ... ) > break; > 4: wmb > 5: first->next_to_watch = tx_desc napi_consume_skb(tx_buffer->skb ..); > 6: writel(i, tx_ring->tail); > > What we see on powerpc is that tx_buffer->skb on CPU2 is getting loaded > prior to tx_buffer->next_to_watch. Changing the read_barrier_depends > to a smp_rmb solves this and prevents us from dereferencing old pointer. > > -Brian So the barrier part I am okay with for all the drivers. I hadn't accounted for the skb being read before next_to_watch. I was more concerned about the descriptor ring versus buffer_info structure at the time I made use of that. The updates to clear tx_buffer->skb in ixgbe I am not okay with. Basically the tell-tale sign for skb present is next_to_watch being non-null. The extra writes add overhead and I want to avoid that at all costs since I want to avoid as much bouncing between the xmit path and the Tx clean-up as possible. - Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3ydDQ84MkTzDr8l for ; Fri, 17 Nov 2017 08:09:17 +1100 (AEDT) From: "Duyck, Alexander H" To: "Brandeburg, Jesse" , "brking@linux.vnet.ibm.com" CC: "dipankar@linux.vnet.ibm.com" , "michaele@au1.ibm.com" , "linuxppc-dev@lists.ozlabs.org" , "intel-wired-lan@lists.osuosl.org" , "stable@vger.kernel.org" , "brking@pobox.com" Subject: Re: [Intel-wired-lan] [PATCH 0/7] [RESEND] [net] intel: Use smp_rmb rather than read_barrier_depends Date: Thu, 16 Nov 2017 21:09:12 +0000 Message-ID: <1510866549.28435.26.camel@intel.com> References: <1510846675-15169-1-git-send-email-brking@linux.vnet.ibm.com> <20171116113358.00001d5a@intel.com> In-Reply-To: Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , T24gVGh1LCAyMDE3LTExLTE2IGF0IDE0OjAzIC0wNjAwLCBCcmlhbiBLaW5nIHdyb3RlOg0KPiBP biAxMS8xNi8yMDE3IDAxOjMzIFBNLCBKZXNzZSBCcmFuZGVidXJnIHdyb3RlOg0KPiA+IE9uIFRo dSwgMTYgTm92IDIwMTcgMDk6Mzc6NDggLTA2MDANCj4gPiBCcmlhbiBLaW5nIDxicmtpbmdAbGlu dXgudm5ldC5pYm0uY29tPiB3cm90ZToNCj4gPiANCj4gPiA+IFJlc2VuZGluZyBhcyB0aGUgZmly c3QgYXR0ZW1wdCBpcyBub3Qgc2hvd2luZyB1cCBpbiB0aGUgbGlzdCBhcmNoaXZlLg0KPiA+ID4g DQo+ID4gPiBUaGlzIHBhdGNoIGNvbnZlcnRzIHNldmVyYWwgbmV0d29yayBkcml2ZXJzIHRvIHVz ZSBzbXBfcm1iDQo+ID4gPiByYXRoZXIgdGhhbiByZWFkX2JhcnJpZXJfZGVwZW5kcy4gVGhlIGlu aXRpYWwgaXNzdWUgd2FzDQo+ID4gPiBkaXNjb3ZlcmVkIHdpdGggaXhnYmUgb24gYSBQb3dlciBt YWNoaW5lIHdoaWNoIHJlc3VsdGVkDQo+ID4gPiBpbiBza2IgbGlzdCBjb3JydXB0aW9uIGR1ZSB0 byBmZXRjaGluZyBhIHN0YWxlIHNrYiBwb2ludGVyLg0KPiA+ID4gTW9yZSBkZXRhaWxzIGNhbiBi ZSBmb3VuZCBpbiB0aGUgaXhnYmUgcGF0Y2ggZGVzY3JpcHRpb24uDQo+ID4gDQo+ID4gVGhhbmtz IGZvciB0aGUgZml4IEJyaWFuLCBJIGJldCBpdCB3YXMgYSB0b3VnaCBkZWJ1Zy4NCj4gPiANCj4g PiBUaGUgb25seSB1c2VycyBpbiB0aGUgZW50aXJlIGtlcm5lbCBvZiByZWFkX2JhcnJpZXJfZGVw ZW5kcygpIChub3QNCj4gPiBzbXBfcmVhZF9iYXJyaWVyX2RlcGVuZHMpIGFyZSB0aGUgSW50ZWwg bmV0d29yayBkcml2ZXJzLg0KPiA+IA0KPiA+IFdvdWxkbid0IGl0IGJlIGJldHRlciBmb3IgcG93 ZXIgdG8ganVzdCBmaXggcmVhZF9iYXJyaWVyX2RlcGVuZHMgdG8gZG8NCj4gPiB0aGUgcmlnaHQg dGhpbmcgb24gcG93ZXI/IFRoZSBxdWVzdGlvbiBJJ20gbm90IHN1cmUgb2YgdGhlIGFuc3dlciB0 byBpczoNCj4gPiBJcyBpdCByZWFsbHkgdGhlIHdyb25nIGJhcnJpZXIgdG8gYmUgdXNpbmcgb3Ig aXMgdGhlIGltcGxlbWVudGF0aW9uIGluDQo+ID4gdGhlIGtlcm5lbCBwb3dlcnBjIHdyb25nPw0K PiA+IA0KPiA+IFNvIEkgdGhpbmsgdGhlIHJpZ2h0IHRoaW5nIG1pZ2h0IGFjdHVhbGx5IHRvIGJl IHRvOg0KPiA+IEZpeCBhcmNoIHBvd2VycGMgcmVhZF9iYXJyaWVyX2RlcGVuZHMgdG8gbm90IGJl IGEgbm9vcCwgYXMgdGhlDQo+ID4gc2VtYW50aWNzIG9mIHRoZSByZWFkX2JhcnJpZXJfZGVwZW5k cyBzZWVtcyB0byBiZSBzdWZmaWNpZW50IHRvIHNvbHZlDQo+ID4gdGhpcyBwcm9ibGVtLCBidXQg aXQgc2VlbXMgbm90IHRvIHdvcmsgZm9yIHBvd2VycGM/DQo+IA0KPiBKZXNzZSwNCj4gDQo+IFRo YW5rcyBmb3IgdGhlIHF1aWNrIHJlc3BvbnNlLg0KPiANCj4gQ2MnaW5nIGxpbnV4cHBjLWRldiBh cyB3ZWxsLiANCj4gDQo+IEkgZGlkIHRoaW5rIGFib3V0IGNoYW5naW5nIHRoZSBwb3dlcnBjIGRl ZmluaXRpb24gb2YgcmVhZF9iYXJyaWVyX2RlcGVuZHMsDQo+IGJ1dCBhZnRlciByZWFkaW5nIHVw IG9uIHRoYXQgYmFycmllciwgZGVjaWRlZCBpdCB3YXMgbm90IHRoZSBjb3JyZWN0IGJhcnJpZXIN Cj4gdG8gYmUgdXNlZCBpbiB0aGlzIGNvbnRleHQuIEhlcmUgaXMgc29tZSBnb29kIGhpc3Rvcmlj YWwgYmFja2dyb3VuZCBvbg0KPiByZWFkX2JhcnJpZXJfZGVwZW5kcyB0aGF0IEkgZm91bmQsIGFs b25nIHdpdGggYW4gZXhhbXBsZS4NCj4gDQo+IGh0dHBzOi8vbHduLm5ldC9BcnRpY2xlcy81MTU5 Lw0KPiANCj4gU2luY2UgdGhlcmUgaXMgbm8gZGF0YS1kZXBlbmRlbmN5IGluIHRoZSBjb2RlIGlu IHF1ZXN0aW9uIGhlcmUsIEkgdGhpbmsNCj4gdGhlIHNtcF9ybWIgaXMgdGhlIHByb3BlciBiYXJy aWVyIHRvIHVzZS4NCj4gDQo+IEZvciBiYWNrZ3JvdW5kLCB0aGUgY29kZSBpbiBxdWVzdGlvbiBs b29rcyBsaWtlIHRoaXM6DQo+IA0KPiBDUFUgMSAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgQ1BVMg0KPiA9PT09PT09PT09PT09PT09PT09PT09PT09PT09ICAgICAgICAgICAgPT09 PT09PT09PT09PT09PT09PT09PT09PT09PQ0KPiAxOiBpeGdiZV94bWl0X2ZyYW1lX3JpbmcgICAg ICAgICAgICAgICAgaXhnYmVfY2xlYW5fdHhfaXJxDQo+IDI6ICBmaXJzdC0+c2tiID0gc2tiICAg ICAgICAgICAgICAgICAgICAgZW9wX2Rlc2MgPSB0eF9idWZmZXItPm5leHRfdG9fd2F0Y2gNCj4g ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZiAoIWVvcF9kZXNjKQ0K PiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBicmVhazsNCj4g MzogIGl4Z2JlX3R4X21hcCAgICAgICAgICAgICAgICAgICAgICAgICByZWFkX2JhcnJpZXJfZGVw ZW5kcygpDQo+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaWYgKCEo ZW9wX2Rlc2MtPndiLnN0YXR1cykgLi4uICkNCj4gICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgYnJlYWs7DQo+IDQ6ICAgd21iICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgDQo+IDU6ICAgZmlyc3QtPm5leHRfdG9fd2F0Y2ggPSB0eF9kZXNjICAgICAg bmFwaV9jb25zdW1lX3NrYih0eF9idWZmZXItPnNrYiAuLik7DQo+IDY6ICAgd3JpdGVsKGksIHR4 X3JpbmctPnRhaWwpOw0KPiANCj4gV2hhdCB3ZSBzZWUgb24gcG93ZXJwYyBpcyB0aGF0IHR4X2J1 ZmZlci0+c2tiIG9uIENQVTIgaXMgZ2V0dGluZyBsb2FkZWQNCj4gcHJpb3IgdG8gdHhfYnVmZmVy LT5uZXh0X3RvX3dhdGNoLiBDaGFuZ2luZyB0aGUgcmVhZF9iYXJyaWVyX2RlcGVuZHMNCj4gdG8g YSBzbXBfcm1iIHNvbHZlcyB0aGlzIGFuZCBwcmV2ZW50cyB1cyBmcm9tIGRlcmVmZXJlbmNpbmcg b2xkIHBvaW50ZXIuDQo+IA0KPiAtQnJpYW4NCg0KU28gdGhlIGJhcnJpZXIgcGFydCBJIGFtIG9r YXkgd2l0aCBmb3IgYWxsIHRoZSBkcml2ZXJzLiBJIGhhZG4ndA0KYWNjb3VudGVkIGZvciB0aGUg c2tiIGJlaW5nIHJlYWQgYmVmb3JlIG5leHRfdG9fd2F0Y2guIEkgd2FzIG1vcmUNCmNvbmNlcm5l ZCBhYm91dCB0aGUgZGVzY3JpcHRvciByaW5nIHZlcnN1cyBidWZmZXJfaW5mbyBzdHJ1Y3R1cmUg YXQgdGhlDQp0aW1lIEkgbWFkZSB1c2Ugb2YgdGhhdC4NCg0KVGhlIHVwZGF0ZXMgdG8gY2xlYXIg dHhfYnVmZmVyLT5za2IgaW4gaXhnYmUgSSBhbSBub3Qgb2theSB3aXRoLg0KQmFzaWNhbGx5IHRo ZSB0ZWxsLXRhbGUgc2lnbiBmb3Igc2tiIHByZXNlbnQgaXMgbmV4dF90b193YXRjaCBiZWluZw0K bm9uLW51bGwuIFRoZSBleHRyYSB3cml0ZXMgYWRkIG92ZXJoZWFkIGFuZCBJIHdhbnQgdG8gYXZv aWQgdGhhdCBhdCBhbGwNCmNvc3RzIHNpbmNlIEkgd2FudCB0byBhdm9pZCBhcyBtdWNoIGJvdW5j aW5nIGJldHdlZW4gdGhlIHhtaXQgcGF0aCBhbmQNCnRoZSBUeCBjbGVhbi11cCBhcyBwb3NzaWJs ZS4NCg0KLSBBbGV4DQoNCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga02.intel.com ([134.134.136.20]:38333 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S935459AbdKPVJO (ORCPT ); Thu, 16 Nov 2017 16:09:14 -0500 From: "Duyck, Alexander H" To: "Brandeburg, Jesse" , "brking@linux.vnet.ibm.com" CC: "dipankar@linux.vnet.ibm.com" , "michaele@au1.ibm.com" , "linuxppc-dev@lists.ozlabs.org" , "intel-wired-lan@lists.osuosl.org" , "stable@vger.kernel.org" , "brking@pobox.com" Subject: Re: [Intel-wired-lan] [PATCH 0/7] [RESEND] [net] intel: Use smp_rmb rather than read_barrier_depends Date: Thu, 16 Nov 2017 21:09:12 +0000 Message-ID: <1510866549.28435.26.camel@intel.com> References: <1510846675-15169-1-git-send-email-brking@linux.vnet.ibm.com> <20171116113358.00001d5a@intel.com> In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-ID: <5DB3B93666FBA64798AA7F45F3D4E1C2@intel.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 Sender: stable-owner@vger.kernel.org List-ID: T24gVGh1LCAyMDE3LTExLTE2IGF0IDE0OjAzIC0wNjAwLCBCcmlhbiBLaW5nIHdyb3RlOg0KPiBP biAxMS8xNi8yMDE3IDAxOjMzIFBNLCBKZXNzZSBCcmFuZGVidXJnIHdyb3RlOg0KPiA+IE9uIFRo dSwgMTYgTm92IDIwMTcgMDk6Mzc6NDggLTA2MDANCj4gPiBCcmlhbiBLaW5nIDxicmtpbmdAbGlu dXgudm5ldC5pYm0uY29tPiB3cm90ZToNCj4gPiANCj4gPiA+IFJlc2VuZGluZyBhcyB0aGUgZmly c3QgYXR0ZW1wdCBpcyBub3Qgc2hvd2luZyB1cCBpbiB0aGUgbGlzdCBhcmNoaXZlLg0KPiA+ID4g DQo+ID4gPiBUaGlzIHBhdGNoIGNvbnZlcnRzIHNldmVyYWwgbmV0d29yayBkcml2ZXJzIHRvIHVz ZSBzbXBfcm1iDQo+ID4gPiByYXRoZXIgdGhhbiByZWFkX2JhcnJpZXJfZGVwZW5kcy4gVGhlIGlu aXRpYWwgaXNzdWUgd2FzDQo+ID4gPiBkaXNjb3ZlcmVkIHdpdGggaXhnYmUgb24gYSBQb3dlciBt YWNoaW5lIHdoaWNoIHJlc3VsdGVkDQo+ID4gPiBpbiBza2IgbGlzdCBjb3JydXB0aW9uIGR1ZSB0 byBmZXRjaGluZyBhIHN0YWxlIHNrYiBwb2ludGVyLg0KPiA+ID4gTW9yZSBkZXRhaWxzIGNhbiBi ZSBmb3VuZCBpbiB0aGUgaXhnYmUgcGF0Y2ggZGVzY3JpcHRpb24uDQo+ID4gDQo+ID4gVGhhbmtz IGZvciB0aGUgZml4IEJyaWFuLCBJIGJldCBpdCB3YXMgYSB0b3VnaCBkZWJ1Zy4NCj4gPiANCj4g PiBUaGUgb25seSB1c2VycyBpbiB0aGUgZW50aXJlIGtlcm5lbCBvZiByZWFkX2JhcnJpZXJfZGVw ZW5kcygpIChub3QNCj4gPiBzbXBfcmVhZF9iYXJyaWVyX2RlcGVuZHMpIGFyZSB0aGUgSW50ZWwg bmV0d29yayBkcml2ZXJzLg0KPiA+IA0KPiA+IFdvdWxkbid0IGl0IGJlIGJldHRlciBmb3IgcG93 ZXIgdG8ganVzdCBmaXggcmVhZF9iYXJyaWVyX2RlcGVuZHMgdG8gZG8NCj4gPiB0aGUgcmlnaHQg dGhpbmcgb24gcG93ZXI/IFRoZSBxdWVzdGlvbiBJJ20gbm90IHN1cmUgb2YgdGhlIGFuc3dlciB0 byBpczoNCj4gPiBJcyBpdCByZWFsbHkgdGhlIHdyb25nIGJhcnJpZXIgdG8gYmUgdXNpbmcgb3Ig aXMgdGhlIGltcGxlbWVudGF0aW9uIGluDQo+ID4gdGhlIGtlcm5lbCBwb3dlcnBjIHdyb25nPw0K PiA+IA0KPiA+IFNvIEkgdGhpbmsgdGhlIHJpZ2h0IHRoaW5nIG1pZ2h0IGFjdHVhbGx5IHRvIGJl IHRvOg0KPiA+IEZpeCBhcmNoIHBvd2VycGMgcmVhZF9iYXJyaWVyX2RlcGVuZHMgdG8gbm90IGJl IGEgbm9vcCwgYXMgdGhlDQo+ID4gc2VtYW50aWNzIG9mIHRoZSByZWFkX2JhcnJpZXJfZGVwZW5k cyBzZWVtcyB0byBiZSBzdWZmaWNpZW50IHRvIHNvbHZlDQo+ID4gdGhpcyBwcm9ibGVtLCBidXQg aXQgc2VlbXMgbm90IHRvIHdvcmsgZm9yIHBvd2VycGM/DQo+IA0KPiBKZXNzZSwNCj4gDQo+IFRo YW5rcyBmb3IgdGhlIHF1aWNrIHJlc3BvbnNlLg0KPiANCj4gQ2MnaW5nIGxpbnV4cHBjLWRldiBh cyB3ZWxsLiANCj4gDQo+IEkgZGlkIHRoaW5rIGFib3V0IGNoYW5naW5nIHRoZSBwb3dlcnBjIGRl ZmluaXRpb24gb2YgcmVhZF9iYXJyaWVyX2RlcGVuZHMsDQo+IGJ1dCBhZnRlciByZWFkaW5nIHVw IG9uIHRoYXQgYmFycmllciwgZGVjaWRlZCBpdCB3YXMgbm90IHRoZSBjb3JyZWN0IGJhcnJpZXIN Cj4gdG8gYmUgdXNlZCBpbiB0aGlzIGNvbnRleHQuIEhlcmUgaXMgc29tZSBnb29kIGhpc3Rvcmlj YWwgYmFja2dyb3VuZCBvbg0KPiByZWFkX2JhcnJpZXJfZGVwZW5kcyB0aGF0IEkgZm91bmQsIGFs b25nIHdpdGggYW4gZXhhbXBsZS4NCj4gDQo+IGh0dHBzOi8vbHduLm5ldC9BcnRpY2xlcy81MTU5 Lw0KPiANCj4gU2luY2UgdGhlcmUgaXMgbm8gZGF0YS1kZXBlbmRlbmN5IGluIHRoZSBjb2RlIGlu IHF1ZXN0aW9uIGhlcmUsIEkgdGhpbmsNCj4gdGhlIHNtcF9ybWIgaXMgdGhlIHByb3BlciBiYXJy aWVyIHRvIHVzZS4NCj4gDQo+IEZvciBiYWNrZ3JvdW5kLCB0aGUgY29kZSBpbiBxdWVzdGlvbiBs b29rcyBsaWtlIHRoaXM6DQo+IA0KPiBDUFUgMSAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgQ1BVMg0KPiA9PT09PT09PT09PT09PT09PT09PT09PT09PT09ICAgICAgICAgICAgPT09 PT09PT09PT09PT09PT09PT09PT09PT09PQ0KPiAxOiBpeGdiZV94bWl0X2ZyYW1lX3JpbmcgICAg ICAgICAgICAgICAgaXhnYmVfY2xlYW5fdHhfaXJxDQo+IDI6ICBmaXJzdC0+c2tiID0gc2tiICAg ICAgICAgICAgICAgICAgICAgZW9wX2Rlc2MgPSB0eF9idWZmZXItPm5leHRfdG9fd2F0Y2gNCj4g ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZiAoIWVvcF9kZXNjKQ0K PiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBicmVhazsNCj4g MzogIGl4Z2JlX3R4X21hcCAgICAgICAgICAgICAgICAgICAgICAgICByZWFkX2JhcnJpZXJfZGVw ZW5kcygpDQo+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaWYgKCEo ZW9wX2Rlc2MtPndiLnN0YXR1cykgLi4uICkNCj4gICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgYnJlYWs7DQo+IDQ6ICAgd21iICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgDQo+IDU6ICAgZmlyc3QtPm5leHRfdG9fd2F0Y2ggPSB0eF9kZXNjICAgICAg bmFwaV9jb25zdW1lX3NrYih0eF9idWZmZXItPnNrYiAuLik7DQo+IDY6ICAgd3JpdGVsKGksIHR4 X3JpbmctPnRhaWwpOw0KPiANCj4gV2hhdCB3ZSBzZWUgb24gcG93ZXJwYyBpcyB0aGF0IHR4X2J1 ZmZlci0+c2tiIG9uIENQVTIgaXMgZ2V0dGluZyBsb2FkZWQNCj4gcHJpb3IgdG8gdHhfYnVmZmVy LT5uZXh0X3RvX3dhdGNoLiBDaGFuZ2luZyB0aGUgcmVhZF9iYXJyaWVyX2RlcGVuZHMNCj4gdG8g YSBzbXBfcm1iIHNvbHZlcyB0aGlzIGFuZCBwcmV2ZW50cyB1cyBmcm9tIGRlcmVmZXJlbmNpbmcg b2xkIHBvaW50ZXIuDQo+IA0KPiAtQnJpYW4NCg0KU28gdGhlIGJhcnJpZXIgcGFydCBJIGFtIG9r YXkgd2l0aCBmb3IgYWxsIHRoZSBkcml2ZXJzLiBJIGhhZG4ndA0KYWNjb3VudGVkIGZvciB0aGUg c2tiIGJlaW5nIHJlYWQgYmVmb3JlIG5leHRfdG9fd2F0Y2guIEkgd2FzIG1vcmUNCmNvbmNlcm5l ZCBhYm91dCB0aGUgZGVzY3JpcHRvciByaW5nIHZlcnN1cyBidWZmZXJfaW5mbyBzdHJ1Y3R1cmUg YXQgdGhlDQp0aW1lIEkgbWFkZSB1c2Ugb2YgdGhhdC4NCg0KVGhlIHVwZGF0ZXMgdG8gY2xlYXIg dHhfYnVmZmVyLT5za2IgaW4gaXhnYmUgSSBhbSBub3Qgb2theSB3aXRoLg0KQmFzaWNhbGx5IHRo ZSB0ZWxsLXRhbGUgc2lnbiBmb3Igc2tiIHByZXNlbnQgaXMgbmV4dF90b193YXRjaCBiZWluZw0K bm9uLW51bGwuIFRoZSBleHRyYSB3cml0ZXMgYWRkIG92ZXJoZWFkIGFuZCBJIHdhbnQgdG8gYXZv aWQgdGhhdCBhdCBhbGwNCmNvc3RzIHNpbmNlIEkgd2FudCB0byBhdm9pZCBhcyBtdWNoIGJvdW5j aW5nIGJldHdlZW4gdGhlIHhtaXQgcGF0aCBhbmQNCnRoZSBUeCBjbGVhbi11cCBhcyBwb3NzaWJs ZS4NCg0KLSBBbGV4DQoNCg==