From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="a/b5xmrK" Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1BBDE193 for ; Wed, 6 Dec 2023 11:06:08 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-5c69bf92447so50874a12.2 for ; Wed, 06 Dec 2023 11:06:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701889567; x=1702494367; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=I409PBeV0vI0BSi/sNEko+BMZt4s00+VHGZ9Cxi/uGs=; b=a/b5xmrKgk/7uyzJAxjbIr2Xb7wyTKzTfLsjFb6oWW05EP6YlIjaNmUcBT0I1x8yq5 TEXhXYDpVmPJELmA6eoz0fCoelVWgMttaxPwpp2tsf+r+jVlJvsoeSEE54lur5ShfBs9 vWWXsgclEYLdYu/0MGT2L1AoCRq8ijqhLD8eQUf69H0NWlAR3WSH6EujcCYZmMaQNSsA RxiBPVWWa8zcJmmenYgCRNJWLyyIt8hm1H/4IR5q36cszbwoBHZzPaCwcWVv4MYTSRth RnwkyJJdnvKS5+Pj+G0Re10FUOBe8MuenPHZF2JltBxQGtxDUfQclF6ORuX7TPEyRPZK KxSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701889567; x=1702494367; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=I409PBeV0vI0BSi/sNEko+BMZt4s00+VHGZ9Cxi/uGs=; b=Z4GzSsgdmLHki4rFHPrfJefhYyG+CBkH7BSXHi9ft6qQO/Q4JJB0MzZcE3fq38rmxw 9gscDMeetoxGNsa0xjb8gEqq8pF/gbE3NTjx+PcJj1792ILY5Qk+7nN0KkoEyGDJjLf2 CXp+GYygvqdRf+XzpSCXr8A3JCcGWBJ8Yjrf/IT6GWcNO3SxWdEbmENyk3Q05MgUGcK2 qoBZ40ZO9TcM1K741rUkICOT7Ywm+3ITzVLDBiXLtixQaInQBVZud98goAhUbeeeLVc7 eeCITNodbN9AqulwgOIznUu5ptxUSow0uIWFuYC8qqu8UP66hcnMWjtD1QsKZX9hqP6a rWKg== X-Gm-Message-State: AOJu0YwTXFZmcd5ehiGOBvoToKDCoYScG/YnTWzdNFsx3huvBKZKmQtD UVgTbCYykuHnPAhj8kmPcnq2ITg= X-Google-Smtp-Source: AGHT+IEqS0IGgMlSKmdLV1hbJusVKwZ8hKL8byL5eF/gGsSa+b+3KizaW4xrrV9KIs8i0M+jEvdFFF0= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a65:67d2:0:b0:5c6:a4e5:2d6a with SMTP id b18-20020a6567d2000000b005c6a4e52d6amr14804pgs.7.1701889567399; Wed, 06 Dec 2023 11:06:07 -0800 (PST) Date: Wed, 6 Dec 2023 11:06:05 -0800 In-Reply-To: Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20231203165129.1740512-1-yoong.siang.song@intel.com> <20231203165129.1740512-3-yoong.siang.song@intel.com> <43b01013-e78b-417e-b169-91909c7309b1@kernel.org> <656de830e8d70_2e983e294ca@willemb.c.googlers.com.notmuch> <5a0faf8cc9ec3ab0d5082c66b909c582c8f1eae6.camel@siemens.com> <656f66023f7bd_3dd6422942a@willemb.c.googlers.com.notmuch> Message-ID: Subject: Re: [xdp-hints] Re: [PATCH bpf-next v3 2/3] net: stmmac: add Launch Time support to XDP ZC From: Stanislav Fomichev To: Magnus Karlsson Cc: Willem de Bruijn , Florian Bezdeka , yoong.siang.song@intel.com, Jesper Dangaard Brouer , davem@davemloft.net, Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Bjorn Topel , magnus.karlsson@intel.com, maciej.fijalkowski@intel.com, Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Lorenzo Bianconi , Tariq Toukan , Willem de Bruijn , Maxime Coquelin , Andrii Nakryiko , Mykola Lysenko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Hao Luo , Jiri Olsa , Shuah Khan , Alexandre Torgue , Jose Abreu , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-doc@vger.kernel.org" , "bpf@vger.kernel.org" , "xdp-hints@xdp-project.net" , "linux-stm32@st-md-mailman.stormreply.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kselftest@vger.kernel.org" Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable On 12/06, Magnus Karlsson wrote: > On Tue, 5 Dec 2023 at 20:39, Stanislav Fomichev wrote: > > > > On 12/05, Willem de Bruijn wrote: > > > Stanislav Fomichev wrote: > > > > On Tue, Dec 5, 2023 at 7:34=E2=80=AFAM Florian Bezdeka > > > > wrote: > > > > > > > > > > On Tue, 2023-12-05 at 15:25 +0000, Song, Yoong Siang wrote: > > > > > > On Monday, December 4, 2023 10:55 PM, Willem de Bruijn wrote: > > > > > > > Jesper Dangaard Brouer wrote: > > > > > > > > > > > > > > > > > > > > > > > > On 12/3/23 17:51, Song Yoong Siang wrote: > > > > > > > > > This patch enables Launch Time (Time-Based Scheduling) su= pport to XDP zero > > > > > > > > > copy via XDP Tx metadata framework. > > > > > > > > > > > > > > > > > > Signed-off-by: Song Yoong Siang > > > > > > > > > --- > > > > > > > > > drivers/net/ethernet/stmicro/stmmac/stmmac.h | 2 = ++ > > > > > > > > > > > > > > > > As requested before, I think we need to see another driver = implementing > > > > > > > > this. > > > > > > > > > > > > > > > > I propose driver igc and chip i225. > > > > > > > > > > > > Sure. I will include igc patches in next version. > > > > > > > > > > > > > > > > > > > > > > The interesting thing for me is to see how the LaunchTime m= ax 1 second > > > > > > > > into the future[1] is handled code wise. One suggestion is = to add a > > > > > > > > section to Documentation/networking/xsk-tx-metadata.rst per= driver that > > > > > > > > mentions/documents these different hardware limitations. I= t is natural > > > > > > > > that different types of hardware have limitations. This is= a close-to > > > > > > > > hardware-level abstraction/API, and IMHO as long as we docu= ment the > > > > > > > > limitations we can expose this API without too many limitat= ions for more > > > > > > > > capable hardware. > > > > > > > > > > > > Sure. I will try to add hardware limitations in documentation. > > > > > > > > > > > > > > > > > > > > I would assume that the kfunc will fail when a value is passe= d that > > > > > > > cannot be programmed. > > > > > > > > > > > > > > > > > > > In current design, the xsk_tx_metadata_request() dint got retur= n value. > > > > > > So user won't know if their request is fail. > > > > > > It is complex to inform user which request is failing. > > > > > > Therefore, IMHO, it is good that we let driver handle the error= silently. > > > > > > > > > > > > > > > > If the programmed value is invalid, the packet will be "dropped" = / will > > > > > never make it to the wire, right? > > > > > > Programmable behavior is to either drop or cap to some boundary > > > value, such as the farthest programmable time in the future: the > > > horizon. In fq: > > > > > > /* Check if packet timestamp is too far in the future= . */ > > > if (fq_packet_beyond_horizon(skb, q, now)) { > > > if (q->horizon_drop) { > > > q->stat_horizon_drops++; > > > return qdisc_drop(skb, sch, t= o_free); > > > } > > > q->stat_horizon_caps++; > > > skb->tstamp =3D now + q->horizon; > > > } > > > fq_skb_cb(skb)->time_to_send =3D skb->tstamp; > > > > > > Drop is the more obviously correct mode. > > > > > > Programming with a clock source that the driver does not support will > > > then be a persistent failure. > > > > > > Preferably, this driver capability can be queried beforehand (rather > > > than only through reading error counters afterwards). > > > > > > Perhaps it should not be a driver task to convert from possibly > > > multiple clock sources to the device native clock. Right now, we do > > > use per-device timecounters for this, implemented in the driver. > > > > > > As for which clocks are relevant. For PTP, I suppose the device PHC, > > > converted to nsec. For pacing offload, TCP uses CLOCK_MONOTONIC. > > > > Do we need to expose some generic netdev netlink apis to query/adjust > > nic clock sources (or maybe there is something existing already)? > > Then the userspace can be responsible for syncing/converting the > > timestamps to the internal nic clocks. +1 to trying to avoid doing > > this in the drivers. > > > > > > > That is clearly a situation that the user should be informed abou= t. For > > > > > RT systems this normally means that something is really wrong reg= arding > > > > > timing / cycle overflow. Such systems have to react on that situa= tion. > > > > > > > > In general, af_xdp is a bit lacking in this 'notify the user that t= hey > > > > somehow messed up' area :-( > > > > For example, pushing a tx descriptor with a wrong addr/len in zc mo= de > > > > will not give any visible signal back (besides driver potentially > > > > spilling something into dmesg as it was in the mlx case). > > > > We can probably start with having some counters for these events? > > > > > > This is because the AF_XDP completion queue descriptor format is only > > > a u64 address? > > > > Yeah. XDP_COPY mode has the descriptor validation which is exported via > > recvmsg errno, but zerocopy path seems to be too deep in the stack > > to report something back. And there is no place, as you mention, > > in the completion ring to report the status. > > > > > Could error conditions be reported on tx completion in the metadata, > > > using xsk_tx_metadata_complete? > > > > That would be one way to do it, yes. But then the error reporting depen= ds > > on the metadata opt-in. Having a separate ring to export the errors, > > or having a v2 tx-completions layout with extra 'status' field would al= so > > work. >=20 > There are error counters for the non-metadata and offloading cases > above that can be retrieved with the XDP_STATISTICS getsockopt(). From > if_xdp.h: >=20 > struct xdp_statistics { > __u64 rx_dropped; /* Dropped for other reasons */ > __u64 rx_invalid_descs; /* Dropped due to invalid descriptor */ > __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */ > __u64 rx_ring_full; /* Dropped due to rx ring being full */ > __u64 rx_fill_ring_empty_descs; /* Failed to retrieve item > from fill ring */ > __u64 tx_ring_empty_descs; /* Failed to retrieve item from tx rin= g */ > }; >=20 > Albeit, these are aggregate statistics and do not say anything about > which packet that caused it. Works well for things that are > programming bugs that should not occur (such as rx_invalid_descs and > tx_invalid_descs) and requires the programmer to debug and fix his or > her program, but it does not work for requests that might fail even > though the program is correct and need to be handled on a packet by > packet basis. So something needs to be added for that as you both say. >=20 > Would prefer if we could avoid a v2 completion descriptor format or > another ring that needs to be checked all the time, so if we could > live with providing the error status in the metadata field of the > packet at completion time, that would be good. Though having the error > status in the completion ring would be faster as that cache line is > hot, while the metadata section of the packet is likely not at > completion time. So that speaks for a v2 completion ring format. Just > thinking out loud here. In this case, maybe adding tx_over_horizon_dropped to XDP_STATISTICS is all we need here? We can have some new api to query this horizon per netdev. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 592E8C4167B for ; Wed, 6 Dec 2023 19:06:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=ZVizcxJvCvdPBmRJPHKFS4c4B76/VSRBDv4gHo+SgrQ=; b=VplEcZ7MIIWFko7bDokUXtTezQ qy+BufHfGtLDXCpn+clWi5DuCNVpWP6EBHv8GtZk/cgGeNqD26o9k2Ffe3zTXBEGxsJFHBcE6gMMM AAUhj3pLDlua4sZauyGqWYhj0SY2ugcm4L+e2qJyB2XGJOo56ZxR8RzNYsHFGZPK6htdHTfYigOej Uxtj0PS92Bl4EgFvTnNgQoW1DakOJE3kIPlHkaNo25XCxyTvrQ4CxuYR2LXlMD52XBr9H3deqbO/A YqW8Bt9hKX/Nni0MJNXcAoPGj9PZ5JG/dMdBYUxrpqljULQytthDp3aRnkAvMnW5hA/D/oqnQM2LK pezCITDg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rAxDp-00B6Ls-0R; Wed, 06 Dec 2023 19:06:13 +0000 Received: from mail-pg1-x54a.google.com ([2607:f8b0:4864:20::54a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rAxDl-00B6LG-0L for linux-arm-kernel@lists.infradead.org; Wed, 06 Dec 2023 19:06:10 +0000 Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-5c69bf92447so50871a12.2 for ; Wed, 06 Dec 2023 11:06:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701889567; x=1702494367; darn=lists.infradead.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=I409PBeV0vI0BSi/sNEko+BMZt4s00+VHGZ9Cxi/uGs=; b=4vb4LxJi9Vs17+8zeapliPRswxaev7zjkwa+VGlPhrxXBfqZ6E3FvCdqTJhoFzcOCd 2tJOvolse+q1zH/UHC6UrzNFQqd91O2pf/26lmJMIlYxbuEqzp9JRdB/3AHv91eaANi9 uwqa1EPOt6hnEFCFETwQeAhT0yIt3ZmDjGi6LcwQlGlKmPxVklimroCOAlGTU7Me1TCp h7cFUneMjk0aI4hfjfcYZHTD7OzLii0LZWjXILOFMJVH5BKIWp0H3y/KJPRNVvungmwN HbECGzmD2m8m7Fwf+iOVYGJvUWGLQeXw0TWzzVRhosWmRyQrbugtbgenfg4RVmi4OkOI 3Cjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701889567; x=1702494367; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=I409PBeV0vI0BSi/sNEko+BMZt4s00+VHGZ9Cxi/uGs=; b=WfpNmILr3rjv7LEJGTt2sqdSl/ojgTf0WmrU7pRTl8S2kp8JnjylWsrL9WTzqqp2EW NVCgGtPl+X3RO9bCOcbqlLe/PafuRJifXoKc9PoFYhFK7+S9wodJy3F3iRlGH65nA3uk jEvCG6sUKfS5lCoIc5FhZ/AQRM+BUwhyDZF6GzJGz2n62G6Y+BQ1HiXhXmSVKsM444yX AkexmwRsq2l9Wh8aBgIyq4S1jcbeZ07flK+cQWZyrBY020/vUn8njwUEzrTUNL1BARd1 xt+RaYmddYUNGvYmhxNZUReUm9rqoWFKoyV5JV603OxRnqueL19VOm+Rx12aRILVIVQ3 +R1g== X-Gm-Message-State: AOJu0YzifSLLNJTd/O/q2tmkCpCU8yd529zgkiLYH0IYxwaX/+WGpkMt 8D1arKUt3Np2whUtYp919kRFh1g= X-Google-Smtp-Source: AGHT+IEqS0IGgMlSKmdLV1hbJusVKwZ8hKL8byL5eF/gGsSa+b+3KizaW4xrrV9KIs8i0M+jEvdFFF0= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a65:67d2:0:b0:5c6:a4e5:2d6a with SMTP id b18-20020a6567d2000000b005c6a4e52d6amr14804pgs.7.1701889567399; Wed, 06 Dec 2023 11:06:07 -0800 (PST) Date: Wed, 6 Dec 2023 11:06:05 -0800 In-Reply-To: Mime-Version: 1.0 References: <20231203165129.1740512-1-yoong.siang.song@intel.com> <20231203165129.1740512-3-yoong.siang.song@intel.com> <43b01013-e78b-417e-b169-91909c7309b1@kernel.org> <656de830e8d70_2e983e294ca@willemb.c.googlers.com.notmuch> <5a0faf8cc9ec3ab0d5082c66b909c582c8f1eae6.camel@siemens.com> <656f66023f7bd_3dd6422942a@willemb.c.googlers.com.notmuch> Message-ID: Subject: Re: [xdp-hints] Re: [PATCH bpf-next v3 2/3] net: stmmac: add Launch Time support to XDP ZC From: Stanislav Fomichev To: Magnus Karlsson Cc: Willem de Bruijn , Florian Bezdeka , yoong.siang.song@intel.com, Jesper Dangaard Brouer , davem@davemloft.net, Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Bjorn Topel , magnus.karlsson@intel.com, maciej.fijalkowski@intel.com, Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Lorenzo Bianconi , Tariq Toukan , Willem de Bruijn , Maxime Coquelin , Andrii Nakryiko , Mykola Lysenko , Martin KaFai Lau , Song Liu , Yonghong Song , KP Singh , Hao Luo , Jiri Olsa , Shuah Khan , Alexandre Torgue , Jose Abreu , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-doc@vger.kernel.org" , "bpf@vger.kernel.org" , "xdp-hints@xdp-project.net" , "linux-stm32@st-md-mailman.stormreply.com" , "linux-arm-kernel@lists.infradead.org" , "linux-kselftest@vger.kernel.org" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231206_110609_147893_60F33F57 X-CRM114-Status: GOOD ( 53.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gMTIvMDYsIE1hZ251cyBLYXJsc3NvbiB3cm90ZToKPiBPbiBUdWUsIDUgRGVjIDIwMjMgYXQg MjA6MzksIFN0YW5pc2xhdiBGb21pY2hldiA8c2RmQGdvb2dsZS5jb20+IHdyb3RlOgo+ID4KPiA+ IE9uIDEyLzA1LCBXaWxsZW0gZGUgQnJ1aWpuIHdyb3RlOgo+ID4gPiBTdGFuaXNsYXYgRm9taWNo ZXYgd3JvdGU6Cj4gPiA+ID4gT24gVHVlLCBEZWMgNSwgMjAyMyBhdCA3OjM04oCvQU0gRmxvcmlh biBCZXpkZWthCj4gPiA+ID4gPGZsb3JpYW4uYmV6ZGVrYUBzaWVtZW5zLmNvbT4gd3JvdGU6Cj4g PiA+ID4gPgo+ID4gPiA+ID4gT24gVHVlLCAyMDIzLTEyLTA1IGF0IDE1OjI1ICswMDAwLCBTb25n LCBZb29uZyBTaWFuZyB3cm90ZToKPiA+ID4gPiA+ID4gT24gTW9uZGF5LCBEZWNlbWJlciA0LCAy MDIzIDEwOjU1IFBNLCBXaWxsZW0gZGUgQnJ1aWpuIHdyb3RlOgo+ID4gPiA+ID4gPiA+IEplc3Bl ciBEYW5nYWFyZCBCcm91ZXIgd3JvdGU6Cj4gPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+ID4K PiA+ID4gPiA+ID4gPiA+IE9uIDEyLzMvMjMgMTc6NTEsIFNvbmcgWW9vbmcgU2lhbmcgd3JvdGU6 Cj4gPiA+ID4gPiA+ID4gPiA+IFRoaXMgcGF0Y2ggZW5hYmxlcyBMYXVuY2ggVGltZSAoVGltZS1C YXNlZCBTY2hlZHVsaW5nKSBzdXBwb3J0IHRvIFhEUCB6ZXJvCj4gPiA+ID4gPiA+ID4gPiA+IGNv cHkgdmlhIFhEUCBUeCBtZXRhZGF0YSBmcmFtZXdvcmsuCj4gPiA+ID4gPiA+ID4gPiA+Cj4gPiA+ ID4gPiA+ID4gPiA+IFNpZ25lZC1vZmYtYnk6IFNvbmcgWW9vbmcgU2lhbmc8eW9vbmcuc2lhbmcu c29uZ0BpbnRlbC5jb20+Cj4gPiA+ID4gPiA+ID4gPiA+IC0tLQo+ID4gPiA+ID4gPiA+ID4gPiAg IGRyaXZlcnMvbmV0L2V0aGVybmV0L3N0bWljcm8vc3RtbWFjL3N0bW1hYy5oICAgICAgfCAgMiAr Kwo+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+IEFzIHJlcXVlc3RlZCBiZWZvcmUsIEkg dGhpbmsgd2UgbmVlZCB0byBzZWUgYW5vdGhlciBkcml2ZXIgaW1wbGVtZW50aW5nCj4gPiA+ID4g PiA+ID4gPiB0aGlzLgo+ID4gPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPiA+IEkgcHJvcG9zZSBk cml2ZXIgaWdjIGFuZCBjaGlwIGkyMjUuCj4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+IFN1cmUuIEkg d2lsbCBpbmNsdWRlIGlnYyBwYXRjaGVzIGluIG5leHQgdmVyc2lvbi4KPiA+ID4gPiA+ID4KPiA+ ID4gPiA+ID4gPiA+Cj4gPiA+ID4gPiA+ID4gPiBUaGUgaW50ZXJlc3RpbmcgdGhpbmcgZm9yIG1l IGlzIHRvIHNlZSBob3cgdGhlIExhdW5jaFRpbWUgbWF4IDEgc2Vjb25kCj4gPiA+ID4gPiA+ID4g PiBpbnRvIHRoZSBmdXR1cmVbMV0gaXMgaGFuZGxlZCBjb2RlIHdpc2UuIE9uZSBzdWdnZXN0aW9u IGlzIHRvIGFkZCBhCj4gPiA+ID4gPiA+ID4gPiBzZWN0aW9uIHRvIERvY3VtZW50YXRpb24vbmV0 d29ya2luZy94c2stdHgtbWV0YWRhdGEucnN0IHBlciBkcml2ZXIgdGhhdAo+ID4gPiA+ID4gPiA+ ID4gbWVudGlvbnMvZG9jdW1lbnRzIHRoZXNlIGRpZmZlcmVudCBoYXJkd2FyZSBsaW1pdGF0aW9u cy4gIEl0IGlzIG5hdHVyYWwKPiA+ID4gPiA+ID4gPiA+IHRoYXQgZGlmZmVyZW50IHR5cGVzIG9m IGhhcmR3YXJlIGhhdmUgbGltaXRhdGlvbnMuICBUaGlzIGlzIGEgY2xvc2UtdG8KPiA+ID4gPiA+ ID4gPiA+IGhhcmR3YXJlLWxldmVsIGFic3RyYWN0aW9uL0FQSSwgYW5kIElNSE8gYXMgbG9uZyBh cyB3ZSBkb2N1bWVudCB0aGUKPiA+ID4gPiA+ID4gPiA+IGxpbWl0YXRpb25zIHdlIGNhbiBleHBv c2UgdGhpcyBBUEkgd2l0aG91dCB0b28gbWFueSBsaW1pdGF0aW9ucyBmb3IgbW9yZQo+ID4gPiA+ ID4gPiA+ID4gY2FwYWJsZSBoYXJkd2FyZS4KPiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gU3VyZS4g SSB3aWxsIHRyeSB0byBhZGQgaGFyZHdhcmUgbGltaXRhdGlvbnMgaW4gZG9jdW1lbnRhdGlvbi4K PiA+ID4gPiA+ID4KPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPiA+IEkgd291bGQgYXNzdW1lIHRo YXQgdGhlIGtmdW5jIHdpbGwgZmFpbCB3aGVuIGEgdmFsdWUgaXMgcGFzc2VkIHRoYXQKPiA+ID4g PiA+ID4gPiBjYW5ub3QgYmUgcHJvZ3JhbW1lZC4KPiA+ID4gPiA+ID4gPgo+ID4gPiA+ID4gPgo+ ID4gPiA+ID4gPiBJbiBjdXJyZW50IGRlc2lnbiwgdGhlIHhza190eF9tZXRhZGF0YV9yZXF1ZXN0 KCkgZGludCBnb3QgcmV0dXJuIHZhbHVlLgo+ID4gPiA+ID4gPiBTbyB1c2VyIHdvbid0IGtub3cg aWYgdGhlaXIgcmVxdWVzdCBpcyBmYWlsLgo+ID4gPiA+ID4gPiBJdCBpcyBjb21wbGV4IHRvIGlu Zm9ybSB1c2VyIHdoaWNoIHJlcXVlc3QgaXMgZmFpbGluZy4KPiA+ID4gPiA+ID4gVGhlcmVmb3Jl LCBJTUhPLCBpdCBpcyBnb29kIHRoYXQgd2UgbGV0IGRyaXZlciBoYW5kbGUgdGhlIGVycm9yIHNp bGVudGx5Lgo+ID4gPiA+ID4gPgo+ID4gPiA+ID4KPiA+ID4gPiA+IElmIHRoZSBwcm9ncmFtbWVk IHZhbHVlIGlzIGludmFsaWQsIHRoZSBwYWNrZXQgd2lsbCBiZSAiZHJvcHBlZCIgLyB3aWxsCj4g PiA+ID4gPiBuZXZlciBtYWtlIGl0IHRvIHRoZSB3aXJlLCByaWdodD8KPiA+ID4KPiA+ID4gUHJv Z3JhbW1hYmxlIGJlaGF2aW9yIGlzIHRvIGVpdGhlciBkcm9wIG9yIGNhcCB0byBzb21lIGJvdW5k YXJ5Cj4gPiA+IHZhbHVlLCBzdWNoIGFzIHRoZSBmYXJ0aGVzdCBwcm9ncmFtbWFibGUgdGltZSBp biB0aGUgZnV0dXJlOiB0aGUKPiA+ID4gaG9yaXpvbi4gSW4gZnE6Cj4gPiA+Cj4gPiA+ICAgICAg ICAgICAgICAgICAvKiBDaGVjayBpZiBwYWNrZXQgdGltZXN0YW1wIGlzIHRvbyBmYXIgaW4gdGhl IGZ1dHVyZS4gKi8KPiA+ID4gICAgICAgICAgICAgICAgIGlmIChmcV9wYWNrZXRfYmV5b25kX2hv cml6b24oc2tiLCBxLCBub3cpKSB7Cj4gPiA+ICAgICAgICAgICAgICAgICAgICAgICAgIGlmIChx LT5ob3Jpem9uX2Ryb3ApIHsKPiA+ID4gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgIHEtPnN0YXRfaG9yaXpvbl9kcm9wcysrOwo+ID4gPiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgcmV0dXJuIHFkaXNjX2Ryb3Aoc2tiLCBzY2gsIHRvX2ZyZWUp Owo+ID4gPiAgICAgICAgICAgICAgICAgICAgICAgICB9Cj4gPiA+ICAgICAgICAgICAgICAgICAg ICAgICAgIHEtPnN0YXRfaG9yaXpvbl9jYXBzKys7Cj4gPiA+ICAgICAgICAgICAgICAgICAgICAg ICAgIHNrYi0+dHN0YW1wID0gbm93ICsgcS0+aG9yaXpvbjsKPiA+ID4gICAgICAgICAgICAgICAg IH0KPiA+ID4gICAgICAgICAgICAgICAgIGZxX3NrYl9jYihza2IpLT50aW1lX3RvX3NlbmQgPSBz a2ItPnRzdGFtcDsKPiA+ID4KPiA+ID4gRHJvcCBpcyB0aGUgbW9yZSBvYnZpb3VzbHkgY29ycmVj dCBtb2RlLgo+ID4gPgo+ID4gPiBQcm9ncmFtbWluZyB3aXRoIGEgY2xvY2sgc291cmNlIHRoYXQg dGhlIGRyaXZlciBkb2VzIG5vdCBzdXBwb3J0IHdpbGwKPiA+ID4gdGhlbiBiZSBhIHBlcnNpc3Rl bnQgZmFpbHVyZS4KPiA+ID4KPiA+ID4gUHJlZmVyYWJseSwgdGhpcyBkcml2ZXIgY2FwYWJpbGl0 eSBjYW4gYmUgcXVlcmllZCBiZWZvcmVoYW5kIChyYXRoZXIKPiA+ID4gdGhhbiBvbmx5IHRocm91 Z2ggcmVhZGluZyBlcnJvciBjb3VudGVycyBhZnRlcndhcmRzKS4KPiA+ID4KPiA+ID4gUGVyaGFw cyBpdCBzaG91bGQgbm90IGJlIGEgZHJpdmVyIHRhc2sgdG8gY29udmVydCBmcm9tIHBvc3NpYmx5 Cj4gPiA+IG11bHRpcGxlIGNsb2NrIHNvdXJjZXMgdG8gdGhlIGRldmljZSBuYXRpdmUgY2xvY2su IFJpZ2h0IG5vdywgd2UgZG8KPiA+ID4gdXNlIHBlci1kZXZpY2UgdGltZWNvdW50ZXJzIGZvciB0 aGlzLCBpbXBsZW1lbnRlZCBpbiB0aGUgZHJpdmVyLgo+ID4gPgo+ID4gPiBBcyBmb3Igd2hpY2gg Y2xvY2tzIGFyZSByZWxldmFudC4gRm9yIFBUUCwgSSBzdXBwb3NlIHRoZSBkZXZpY2UgUEhDLAo+ ID4gPiBjb252ZXJ0ZWQgdG8gbnNlYy4gRm9yIHBhY2luZyBvZmZsb2FkLCBUQ1AgdXNlcyBDTE9D S19NT05PVE9OSUMuCj4gPgo+ID4gRG8gd2UgbmVlZCB0byBleHBvc2Ugc29tZSBnZW5lcmljIG5l dGRldiBuZXRsaW5rIGFwaXMgdG8gcXVlcnkvYWRqdXN0Cj4gPiBuaWMgY2xvY2sgc291cmNlcyAo b3IgbWF5YmUgdGhlcmUgaXMgc29tZXRoaW5nIGV4aXN0aW5nIGFscmVhZHkpPwo+ID4gVGhlbiB0 aGUgdXNlcnNwYWNlIGNhbiBiZSByZXNwb25zaWJsZSBmb3Igc3luY2luZy9jb252ZXJ0aW5nIHRo ZQo+ID4gdGltZXN0YW1wcyB0byB0aGUgaW50ZXJuYWwgbmljIGNsb2Nrcy4gKzEgdG8gdHJ5aW5n IHRvIGF2b2lkIGRvaW5nCj4gPiB0aGlzIGluIHRoZSBkcml2ZXJzLgo+ID4KPiA+ID4gPiA+IFRo YXQgaXMgY2xlYXJseSBhIHNpdHVhdGlvbiB0aGF0IHRoZSB1c2VyIHNob3VsZCBiZSBpbmZvcm1l ZCBhYm91dC4gRm9yCj4gPiA+ID4gPiBSVCBzeXN0ZW1zIHRoaXMgbm9ybWFsbHkgbWVhbnMgdGhh dCBzb21ldGhpbmcgaXMgcmVhbGx5IHdyb25nIHJlZ2FyZGluZwo+ID4gPiA+ID4gdGltaW5nIC8g Y3ljbGUgb3ZlcmZsb3cuIFN1Y2ggc3lzdGVtcyBoYXZlIHRvIHJlYWN0IG9uIHRoYXQgc2l0dWF0 aW9uLgo+ID4gPiA+Cj4gPiA+ID4gSW4gZ2VuZXJhbCwgYWZfeGRwIGlzIGEgYml0IGxhY2tpbmcg aW4gdGhpcyAnbm90aWZ5IHRoZSB1c2VyIHRoYXQgdGhleQo+ID4gPiA+IHNvbWVob3cgbWVzc2Vk IHVwJyBhcmVhIDotKAo+ID4gPiA+IEZvciBleGFtcGxlLCBwdXNoaW5nIGEgdHggZGVzY3JpcHRv ciB3aXRoIGEgd3JvbmcgYWRkci9sZW4gaW4gemMgbW9kZQo+ID4gPiA+IHdpbGwgbm90IGdpdmUg YW55IHZpc2libGUgc2lnbmFsIGJhY2sgKGJlc2lkZXMgZHJpdmVyIHBvdGVudGlhbGx5Cj4gPiA+ ID4gc3BpbGxpbmcgc29tZXRoaW5nIGludG8gZG1lc2cgYXMgaXQgd2FzIGluIHRoZSBtbHggY2Fz ZSkuCj4gPiA+ID4gV2UgY2FuIHByb2JhYmx5IHN0YXJ0IHdpdGggaGF2aW5nIHNvbWUgY291bnRl cnMgZm9yIHRoZXNlIGV2ZW50cz8KPiA+ID4KPiA+ID4gVGhpcyBpcyBiZWNhdXNlIHRoZSBBRl9Y RFAgY29tcGxldGlvbiBxdWV1ZSBkZXNjcmlwdG9yIGZvcm1hdCBpcyBvbmx5Cj4gPiA+IGEgdTY0 IGFkZHJlc3M/Cj4gPgo+ID4gWWVhaC4gWERQX0NPUFkgbW9kZSBoYXMgdGhlIGRlc2NyaXB0b3Ig dmFsaWRhdGlvbiB3aGljaCBpcyBleHBvcnRlZCB2aWEKPiA+IHJlY3Ztc2cgZXJybm8sIGJ1dCB6 ZXJvY29weSBwYXRoIHNlZW1zIHRvIGJlIHRvbyBkZWVwIGluIHRoZSBzdGFjawo+ID4gdG8gcmVw b3J0IHNvbWV0aGluZyBiYWNrLiBBbmQgdGhlcmUgaXMgbm8gcGxhY2UsIGFzIHlvdSBtZW50aW9u LAo+ID4gaW4gdGhlIGNvbXBsZXRpb24gcmluZyB0byByZXBvcnQgdGhlIHN0YXR1cy4KPiA+Cj4g PiA+IENvdWxkIGVycm9yIGNvbmRpdGlvbnMgYmUgcmVwb3J0ZWQgb24gdHggY29tcGxldGlvbiBp biB0aGUgbWV0YWRhdGEsCj4gPiA+IHVzaW5nIHhza190eF9tZXRhZGF0YV9jb21wbGV0ZT8KPiA+ Cj4gPiBUaGF0IHdvdWxkIGJlIG9uZSB3YXkgdG8gZG8gaXQsIHllcy4gQnV0IHRoZW4gdGhlIGVy cm9yIHJlcG9ydGluZyBkZXBlbmRzCj4gPiBvbiB0aGUgbWV0YWRhdGEgb3B0LWluLiBIYXZpbmcg YSBzZXBhcmF0ZSByaW5nIHRvIGV4cG9ydCB0aGUgZXJyb3JzLAo+ID4gb3IgaGF2aW5nIGEgdjIg dHgtY29tcGxldGlvbnMgbGF5b3V0IHdpdGggZXh0cmEgJ3N0YXR1cycgZmllbGQgd291bGQgYWxz bwo+ID4gd29yay4KPiAKPiBUaGVyZSBhcmUgZXJyb3IgY291bnRlcnMgZm9yIHRoZSBub24tbWV0 YWRhdGEgYW5kIG9mZmxvYWRpbmcgY2FzZXMKPiBhYm92ZSB0aGF0IGNhbiBiZSByZXRyaWV2ZWQg d2l0aCB0aGUgWERQX1NUQVRJU1RJQ1MgZ2V0c29ja29wdCgpLiBGcm9tCj4gaWZfeGRwLmg6Cj4g Cj4gc3RydWN0IHhkcF9zdGF0aXN0aWNzIHsKPiAgICAgICAgIF9fdTY0IHJ4X2Ryb3BwZWQ7IC8q IERyb3BwZWQgZm9yIG90aGVyIHJlYXNvbnMgKi8KPiAgICAgICAgIF9fdTY0IHJ4X2ludmFsaWRf ZGVzY3M7IC8qIERyb3BwZWQgZHVlIHRvIGludmFsaWQgZGVzY3JpcHRvciAqLwo+ICAgICAgICAg X191NjQgdHhfaW52YWxpZF9kZXNjczsgLyogRHJvcHBlZCBkdWUgdG8gaW52YWxpZCBkZXNjcmlw dG9yICovCj4gICAgICAgICBfX3U2NCByeF9yaW5nX2Z1bGw7IC8qIERyb3BwZWQgZHVlIHRvIHJ4 IHJpbmcgYmVpbmcgZnVsbCAqLwo+ICAgICAgICAgX191NjQgcnhfZmlsbF9yaW5nX2VtcHR5X2Rl c2NzOyAvKiBGYWlsZWQgdG8gcmV0cmlldmUgaXRlbQo+IGZyb20gZmlsbCByaW5nICovCj4gICAg ICAgICBfX3U2NCB0eF9yaW5nX2VtcHR5X2Rlc2NzOyAvKiBGYWlsZWQgdG8gcmV0cmlldmUgaXRl bSBmcm9tIHR4IHJpbmcgKi8KPiB9Owo+IAo+IEFsYmVpdCwgdGhlc2UgYXJlIGFnZ3JlZ2F0ZSBz dGF0aXN0aWNzIGFuZCBkbyBub3Qgc2F5IGFueXRoaW5nIGFib3V0Cj4gd2hpY2ggcGFja2V0IHRo YXQgY2F1c2VkIGl0LiBXb3JrcyB3ZWxsIGZvciB0aGluZ3MgdGhhdCBhcmUKPiBwcm9ncmFtbWlu ZyBidWdzIHRoYXQgc2hvdWxkIG5vdCBvY2N1ciAoc3VjaCBhcyByeF9pbnZhbGlkX2Rlc2NzIGFu ZAo+IHR4X2ludmFsaWRfZGVzY3MpIGFuZCByZXF1aXJlcyB0aGUgcHJvZ3JhbW1lciB0byBkZWJ1 ZyBhbmQgZml4IGhpcyBvcgo+IGhlciBwcm9ncmFtLCBidXQgaXQgZG9lcyBub3Qgd29yayBmb3Ig cmVxdWVzdHMgdGhhdCBtaWdodCBmYWlsIGV2ZW4KPiB0aG91Z2ggdGhlIHByb2dyYW0gaXMgY29y cmVjdCBhbmQgbmVlZCB0byBiZSBoYW5kbGVkIG9uIGEgcGFja2V0IGJ5Cj4gcGFja2V0IGJhc2lz LiBTbyBzb21ldGhpbmcgbmVlZHMgdG8gYmUgYWRkZWQgZm9yIHRoYXQgYXMgeW91IGJvdGggc2F5 Lgo+IAo+IFdvdWxkIHByZWZlciBpZiB3ZSBjb3VsZCBhdm9pZCBhIHYyIGNvbXBsZXRpb24gZGVz Y3JpcHRvciBmb3JtYXQgb3IKPiBhbm90aGVyIHJpbmcgdGhhdCBuZWVkcyB0byBiZSBjaGVja2Vk IGFsbCB0aGUgdGltZSwgc28gaWYgd2UgY291bGQKPiBsaXZlIHdpdGggcHJvdmlkaW5nIHRoZSBl cnJvciBzdGF0dXMgaW4gdGhlIG1ldGFkYXRhIGZpZWxkIG9mIHRoZQo+IHBhY2tldCBhdCBjb21w bGV0aW9uIHRpbWUsIHRoYXQgd291bGQgYmUgZ29vZC4gVGhvdWdoIGhhdmluZyB0aGUgZXJyb3IK PiBzdGF0dXMgaW4gdGhlIGNvbXBsZXRpb24gcmluZyB3b3VsZCBiZSBmYXN0ZXIgYXMgdGhhdCBj YWNoZSBsaW5lIGlzCj4gaG90LCB3aGlsZSB0aGUgbWV0YWRhdGEgc2VjdGlvbiBvZiB0aGUgcGFj a2V0IGlzIGxpa2VseSBub3QgYXQKPiBjb21wbGV0aW9uIHRpbWUuIFNvIHRoYXQgc3BlYWtzIGZv ciBhIHYyIGNvbXBsZXRpb24gcmluZyBmb3JtYXQuIEp1c3QKPiB0aGlua2luZyBvdXQgbG91ZCBo ZXJlLgoKSW4gdGhpcyBjYXNlLCBtYXliZSBhZGRpbmcgdHhfb3Zlcl9ob3Jpem9uX2Ryb3BwZWQg dG8gWERQX1NUQVRJU1RJQ1MKaXMgYWxsIHdlIG5lZWQgaGVyZT8gV2UgY2FuIGhhdmUgc29tZSBu ZXcgYXBpIHRvIHF1ZXJ5IHRoaXMgaG9yaXpvbgpwZXIgbmV0ZGV2LgoKX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5n IGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5p bmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=