From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5B20C433E0 for ; Sat, 30 Jan 2021 00:27:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7394C64E02 for ; Sat, 30 Jan 2021 00:27:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231183AbhA3A0D (ORCPT ); Fri, 29 Jan 2021 19:26:03 -0500 Received: from mga01.intel.com ([192.55.52.88]:38336 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230009AbhA3AZa (ORCPT ); Fri, 29 Jan 2021 19:25:30 -0500 IronPort-SDR: ihODjFYr1ag+oXqn21UF3cNlZMyxdzBQwqFEzZGnXp2FsT0NIqxcNw6KABQJbdWMVuHBQv8jyp 0OvEI9mvsCVg== X-IronPort-AV: E=McAfee;i="6000,8403,9879"; a="199350680" X-IronPort-AV: E=Sophos;i="5.79,387,1602572400"; d="scan'208";a="199350680" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jan 2021 16:24:45 -0800 IronPort-SDR: 2Qr93ZP+aH/uhPOAc0J5zom4DzflXQWWzztfofYSrASo6XXKWnzKVo9aQbDuqJbLce75q5/+gP 6Qi46MABmQiQ== X-IronPort-AV: E=Sophos;i="5.79,387,1602572400"; d="scan'208";a="370591638" Received: from jambrizm-mobl1.amr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.133.15]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jan 2021 16:24:44 -0800 From: Ben Widawsky To: linux-cxl@vger.kernel.org Cc: Ben Widawsky , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-pci@vger.kernel.org, Bjorn Helgaas , Chris Browy , Christoph Hellwig , Dan Williams , Ira Weiny , Jon Masters , Jonathan Cameron , Rafael Wysocki , Randy Dunlap , Vishal Verma , daniel.lll@alibaba-inc.com, "John Groves (jgroves)" , "Kelley, Sean V" Subject: [PATCH 00/14] CXL 2.0 Support Date: Fri, 29 Jan 2021 16:24:24 -0800 Message-Id: <20210130002438.1872527-1-ben.widawsky@intel.com> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org # Changes since RFC v3 [1] * Added error message when payload size is too small. (Ben) * Fix includes in UAPI for Clang (LKP) * Reorder CXL in MAINTAINERS (Joe Perches) * Kconfig whitespace and spelling fixes (Randy) * Remove excess frees controlled by devm, introduced in v3 (Jonathan, Dan) * Use 'PCI Express' instead of 'PCI-E' in Kconfig (Jonathan) * Fail when mailbox commands return value is an error (Jonathan) * Add comment to mailbox protocol to explain ordering of operations (Jonathan, Ben) * Fail mailbox xfer when doorbell is busy. (Jonathan) * Remove extraneous SHIFT defines. (Jonathan) * Change kdocs for mbox_cmd size_out to output only. (Jonathan) * Fix transient bug (ENOTTY) in CXL_MEM_QUERY_COMMANDS (Jonathan) * Add some comments and code beautification to mbox commands (Jonathan) * Add some comments and code beautification to user commands (Jonathan) * Fix bogus check of memcpy return value (Ben) * Add concept of blocking certain RAW opcodes (Dan) * Add debugfs knob to allow all RAW opcodes (Vishal) * Move docs to driver-api/ (Dan) * Use bounce buffer again like in v2 (Jonathan) * Use kvzalloc instead of memdup (Ben) * Wordsmith some changelogs and documentation (Dan) * Use a percpu_ref counter to protect devm allocated data in the ioctl path (Dan) * Rework cdev registration and lookup to use inode->i_cdev (Dan) * Drop mutex_lock_interruptible() from ioctl path (Dan) * Convert add_taint() to WARN_TAINT_ONCE() * Drop ACPI coordination for pure mailbox driver milestone (Dan) * Permit GET_LOG with CEL_UUID (Ben) * Cover letter overhaul (Ben) * Use info.id instead of CXL_COMMAND_INDEX (Dan) * Add several new commands to the mailbox interface (Ben) --- In addition to the mailing list, please feel free to use #cxl on oftc IRC for discussion. --- # Summary Introduce support for “type-3” memory devices defined in the Compute Express Link (CXL) 2.0 specification [2]. Specifically, these are the memory devices defined by section 8.2.8.5 of the CXL 2.0 spec. A reference implementation emulating these devices has been submitted to the QEMU mailing list [3] and is available on gitlab [4], but will move to a shared tree on kernel.org after initial acceptance. “Type-3” is a CXL device that acts as a memory expander for RAM or Persistent Memory. The device might be interleaved with other CXL devices in a given physical address range. In addition to the core functionality of discovering the spec defined registers and resources, introduce a CXL device model that will be the foundation for translating CXL capabilities into existing Linux infrastructure for Persistent Memory and other memory devices. For now, this only includes support for the management command mailbox the surfacing of type-3 devices. These control devices fill the role of “DIMMs” / nmemX memory-devices in LIBNVDIMM terms. ## Userspace Interaction Interaction with the driver and type-3 devices via the CXL drivers is introduced in this patch series and considered stable ABI. They include * sysfs - Documentation/ABI/testing/sysfs-bus-cxl * IOCTL - Documentation/driver-api/cxl/memory-devices.rst * debugfs - Documentation/ABI/testing/debugfs-debug Work is in process to add support for CXL interactions to the ndctl project [5] ### Development plans One of the unique challenges that CXL imposes on the Linux driver model is that it requires the operating system to perform physical address space management interleaved across devices and bridges. Whereas LIBNVDIMM handles a list of established static persistent memory address ranges (for example from the ACPI NFIT), CXL introduces hotplug and the concept of allocating address space to instantiate persistent memory ranges. This is similar to PCI in the sense that the platform establishes the MMIO range for PCI BARs to be allocated, but it is significantly complicated by the fact that a given device can optionally be interleaved with other devices and can participate in several interleave-sets at once. LIBNVDIMM handled something like this with the aliasing between PMEM and BLOCK-WINDOW mode, but CXL adds flexibility to alias DEVICE MEMORY through up to 10 decoders per device. All of the above needs to be enabled with respect to PCI hotplug events on Type-3 memory device which needs hooks to determine if a given device is contributing to a "System RAM" address range that is unable to be unplugged. In other words CXL ties PCI hotplug to Memory Hotplug and PCI hotplug needs to be able to negotiate with memory hotplug. In the medium term the implications of CXL hotplug vs ACPI SRAT/SLIT/HMAT need to be reconciled. One capability that seems to be needed is either the dynamic allocation of new memory nodes, or default initializing extra pgdat instances beyond what is enumerated in ACPI SRAT to accommodate hot-added CXL memory. Patches welcome, questions welcome as the development effort on the post v5.12 capabilities proceeds. ## Running in QEMU The incantation to get CXL support in QEMU [4] is considered unstable at this time. Future readers of this cover letter should verify if any changes are needed. For the novice QEMU user, the following can be copy/pasted into a working QEMU commandline. It is enough to make the simplest topology possible. The topology would consist of a single memory window, single type3 device, single root port, and single host bridge. +-------------+ | CXL PXB | | | | +-------+ |<----------+ | |CXL RP | | | +--+-------+--+ v | +----------+ | | "window" | | +----------+ v ^ +-------------+ | | CXL Type 3 | | | Device |<----------+ +-------------+ // Memory backend -object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M // Host Bridge -device pxb-cxl id=cxl.0,bus=pcie.0,bus_nr=52,uid=0 len-window-base=1,window-base[0]=0x4c0000000 memdev[0]=cxl-mem1 // Single root port -device cxl rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0,memdev=cxl-mem1 // Single type3 device -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M -device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M --- [1]: https://lore.kernel.org/linux-cxl/20201209002418.1976362-1-ben.widawsky@intel.com/ [2]: https://www.computeexpresslink.org/](https://www.computeexpresslink.org/ [3]: https://lore.kernel.org/qemu-devel/20210105165323.783725-1-ben.widawsky@intel.com/T/#t [4]: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v* [5]: https://github.com/pmem/ndctl/tree/cxl-2.0v* Ben Widawsky (12): cxl/mem: Map memory device registers cxl/mem: Find device capabilities cxl/mem: Implement polled mode mailbox cxl/mem: Add basic IOCTL interface cxl/mem: Add send command taint: add taint for direct hardware access cxl/mem: Add a "RAW" send command cxl/mem: Create concept of enabled commands cxl/mem: Use CEL for enabling commands cxl/mem: Add set of informational commands cxl/mem: Add limited Get Log command (0401h) MAINTAINERS: Add maintainers of the CXL driver Dan Williams (2): cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints cxl/mem: Register CXL memX devices .clang-format | 1 + Documentation/ABI/testing/debugfs-cxl | 10 + Documentation/ABI/testing/sysfs-bus-cxl | 26 + Documentation/admin-guide/sysctl/kernel.rst | 1 + Documentation/admin-guide/tainted-kernels.rst | 6 +- Documentation/driver-api/cxl/index.rst | 12 + .../driver-api/cxl/memory-devices.rst | 46 + Documentation/driver-api/index.rst | 1 + .../userspace-api/ioctl/ioctl-number.rst | 1 + MAINTAINERS | 11 + drivers/Kconfig | 1 + drivers/Makefile | 1 + drivers/base/core.c | 14 + drivers/cxl/Kconfig | 49 + drivers/cxl/Makefile | 7 + drivers/cxl/bus.c | 29 + drivers/cxl/cxl.h | 140 ++ drivers/cxl/mem.c | 1603 +++++++++++++++++ drivers/cxl/pci.h | 34 + include/linux/device.h | 1 + include/linux/kernel.h | 3 +- include/uapi/linux/cxl_mem.h | 180 ++ kernel/panic.c | 1 + 23 files changed, 2176 insertions(+), 2 deletions(-) create mode 100644 Documentation/ABI/testing/debugfs-cxl create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl create mode 100644 Documentation/driver-api/cxl/index.rst create mode 100644 Documentation/driver-api/cxl/memory-devices.rst create mode 100644 drivers/cxl/Kconfig create mode 100644 drivers/cxl/Makefile create mode 100644 drivers/cxl/bus.c create mode 100644 drivers/cxl/cxl.h create mode 100644 drivers/cxl/mem.c create mode 100644 drivers/cxl/pci.h create mode 100644 include/uapi/linux/cxl_mem.h -- 2.30.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2900C433DB for ; Sat, 30 Jan 2021 00:24:50 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F2FBD64E00 for ; Sat, 30 Jan 2021 00:24:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F2FBD64E00 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvdimm-bounces@lists.01.org Received: from ml01.vlan13.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id 67267100EB85D; Fri, 29 Jan 2021 16:24:49 -0800 (PST) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=192.55.52.43; helo=mga05.intel.com; envelope-from=ben.widawsky@intel.com; receiver= Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 3BF88100ED4A7 for ; Fri, 29 Jan 2021 16:24:46 -0800 (PST) IronPort-SDR: YoBGgewsaw7bSSBvfqZQWq47yb1SLZgqZoOi1ybVhnfg6jtws7LuRGYJ8vh2KDMju3kUUeLKUk bufwAaCaChLQ== X-IronPort-AV: E=McAfee;i="6000,8403,9879"; a="265333126" X-IronPort-AV: E=Sophos;i="5.79,387,1602572400"; d="scan'208";a="265333126" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jan 2021 16:24:45 -0800 IronPort-SDR: 2Qr93ZP+aH/uhPOAc0J5zom4DzflXQWWzztfofYSrASo6XXKWnzKVo9aQbDuqJbLce75q5/+gP 6Qi46MABmQiQ== X-IronPort-AV: E=Sophos;i="5.79,387,1602572400"; d="scan'208";a="370591638" Received: from jambrizm-mobl1.amr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.133.15]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Jan 2021 16:24:44 -0800 From: Ben Widawsky To: linux-cxl@vger.kernel.org Subject: [PATCH 00/14] CXL 2.0 Support Date: Fri, 29 Jan 2021 16:24:24 -0800 Message-Id: <20210130002438.1872527-1-ben.widawsky@intel.com> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 Message-ID-Hash: TW4NTVSIG3OT3JG4O4XLH4F45UU6SDRT X-Message-ID-Hash: TW4NTVSIG3OT3JG4O4XLH4F45UU6SDRT X-MailFrom: ben.widawsky@intel.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation CC: Ben Widawsky , linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, linux-pci@vger.kernel.org, Bjorn Helgaas , Chris Browy , Christoph Hellwig , Jon Masters , Jonathan Cameron , Rafael Wysocki , Randy Dunlap , daniel.lll@alibaba-inc.com, "John Groves (jgroves)" , "Kelley, Sean V" X-Mailman-Version: 3.1.1 Precedence: list List-Id: "Linux-nvdimm developer list." Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 IyBDaGFuZ2VzIHNpbmNlIFJGQyB2MyBbMV0NCiAgICogQWRkZWQgZXJyb3IgbWVzc2FnZSB3aGVu IHBheWxvYWQgc2l6ZSBpcyB0b28gc21hbGwuIChCZW4pDQogICAqIEZpeCBpbmNsdWRlcyBpbiBV QVBJIGZvciBDbGFuZyAoTEtQKQ0KICAgKiBSZW9yZGVyIENYTCBpbiBNQUlOVEFJTkVSUyAoSm9l IFBlcmNoZXMpDQogICAqIEtjb25maWcgd2hpdGVzcGFjZSBhbmQgc3BlbGxpbmcgZml4ZXMgKFJh bmR5KQ0KICAgKiBSZW1vdmUgZXhjZXNzIGZyZWVzIGNvbnRyb2xsZWQgYnkgZGV2bSwgaW50cm9k dWNlZCBpbiB2MyAoSm9uYXRoYW4sIERhbikNCiAgICogVXNlICdQQ0kgRXhwcmVzcycgaW5zdGVh ZCBvZiAnUENJLUUnIGluIEtjb25maWcgKEpvbmF0aGFuKQ0KICAgKiBGYWlsIHdoZW4gbWFpbGJv eCBjb21tYW5kcyByZXR1cm4gdmFsdWUgaXMgYW4gZXJyb3IgKEpvbmF0aGFuKQ0KICAgKiBBZGQg Y29tbWVudCB0byBtYWlsYm94IHByb3RvY29sIHRvIGV4cGxhaW4gb3JkZXJpbmcgb2Ygb3BlcmF0 aW9ucw0KICAgICAoSm9uYXRoYW4sIEJlbikNCiAgICogRmFpbCBtYWlsYm94IHhmZXIgd2hlbiBk b29yYmVsbCBpcyBidXN5LiAoSm9uYXRoYW4pDQogICAqIFJlbW92ZSBleHRyYW5lb3VzIFNISUZU IGRlZmluZXMuIChKb25hdGhhbikNCiAgICogQ2hhbmdlIGtkb2NzIGZvciBtYm94X2NtZCBzaXpl X291dCB0byBvdXRwdXQgb25seS4gKEpvbmF0aGFuKQ0KICAgKiBGaXggdHJhbnNpZW50IGJ1ZyAo RU5PVFRZKSBpbiBDWExfTUVNX1FVRVJZX0NPTU1BTkRTIChKb25hdGhhbikNCiAgICogQWRkIHNv bWUgY29tbWVudHMgYW5kIGNvZGUgYmVhdXRpZmljYXRpb24gdG8gbWJveCBjb21tYW5kcyAoSm9u YXRoYW4pDQogICAqIEFkZCBzb21lIGNvbW1lbnRzIGFuZCBjb2RlIGJlYXV0aWZpY2F0aW9uIHRv IHVzZXIgY29tbWFuZHMgKEpvbmF0aGFuKQ0KICAgKiBGaXggYm9ndXMgY2hlY2sgb2YgbWVtY3B5 IHJldHVybiB2YWx1ZSAoQmVuKQ0KICAgKiBBZGQgY29uY2VwdCBvZiBibG9ja2luZyBjZXJ0YWlu IFJBVyBvcGNvZGVzIChEYW4pDQogICAqIEFkZCBkZWJ1Z2ZzIGtub2IgdG8gYWxsb3cgYWxsIFJB VyBvcGNvZGVzIChWaXNoYWwpDQogICAqIE1vdmUgZG9jcyB0byBkcml2ZXItYXBpLyAoRGFuKQ0K ICAgKiBVc2UgYm91bmNlIGJ1ZmZlciBhZ2FpbiBsaWtlIGluIHYyIChKb25hdGhhbikNCiAgICAg ICAqIFVzZSBrdnphbGxvYyBpbnN0ZWFkIG9mIG1lbWR1cCAoQmVuKQ0KICAgKiBXb3Jkc21pdGgg c29tZSBjaGFuZ2Vsb2dzIGFuZCBkb2N1bWVudGF0aW9uIChEYW4pDQogICAqIFVzZSBhIHBlcmNw dV9yZWYgY291bnRlciB0byBwcm90ZWN0IGRldm0gYWxsb2NhdGVkIGRhdGEgaW4gdGhlIGlvY3Rs IHBhdGgNCiAgICAgKERhbikNCiAgICogUmV3b3JrIGNkZXYgcmVnaXN0cmF0aW9uIGFuZCBsb29r dXAgdG8gdXNlIGlub2RlLT5pX2NkZXYgKERhbikNCiAgICogRHJvcCBtdXRleF9sb2NrX2ludGVy cnVwdGlibGUoKSBmcm9tIGlvY3RsIHBhdGggKERhbikNCiAgICogQ29udmVydCBhZGRfdGFpbnQo KSB0byBXQVJOX1RBSU5UX09OQ0UoKQ0KICAgKiBEcm9wIEFDUEkgY29vcmRpbmF0aW9uIGZvciBw dXJlIG1haWxib3ggZHJpdmVyIG1pbGVzdG9uZSAoRGFuKQ0KICAgKiBQZXJtaXQgR0VUX0xPRyB3 aXRoIENFTF9VVUlEIChCZW4pDQogICAqIENvdmVyIGxldHRlciBvdmVyaGF1bCAoQmVuKQ0KICAg KiBVc2UgaW5mby5pZCBpbnN0ZWFkIG9mIENYTF9DT01NQU5EX0lOREVYIChEYW4pDQogICAqIEFk ZCBzZXZlcmFsIG5ldyBjb21tYW5kcyB0byB0aGUgbWFpbGJveCBpbnRlcmZhY2UgKEJlbikNCg0K LS0tDQoNCkluIGFkZGl0aW9uIHRvIHRoZSBtYWlsaW5nIGxpc3QsIHBsZWFzZSBmZWVsIGZyZWUg dG8gdXNlICNjeGwgb24gb2Z0YyBJUkMgZm9yDQpkaXNjdXNzaW9uLg0KDQotLS0NCg0KIyBTdW1t YXJ5DQoNCkludHJvZHVjZSBzdXBwb3J0IGZvciDigJx0eXBlLTPigJ0gbWVtb3J5IGRldmljZXMg ZGVmaW5lZCBpbiB0aGUgQ29tcHV0ZSBFeHByZXNzDQpMaW5rIChDWEwpIDIuMCBzcGVjaWZpY2F0 aW9uIFsyXS4gU3BlY2lmaWNhbGx5LCB0aGVzZSBhcmUgdGhlIG1lbW9yeSBkZXZpY2VzDQpkZWZp bmVkIGJ5IHNlY3Rpb24gOC4yLjguNSBvZiB0aGUgQ1hMIDIuMCBzcGVjLiBBIHJlZmVyZW5jZSBp bXBsZW1lbnRhdGlvbg0KZW11bGF0aW5nIHRoZXNlIGRldmljZXMgaGFzIGJlZW4gc3VibWl0dGVk IHRvIHRoZSBRRU1VIG1haWxpbmcgbGlzdCBbM10gYW5kIGlzDQphdmFpbGFibGUgb24gZ2l0bGFi IFs0XSwgYnV0IHdpbGwgbW92ZSB0byBhIHNoYXJlZCB0cmVlIG9uIGtlcm5lbC5vcmcgYWZ0ZXIN CmluaXRpYWwgYWNjZXB0YW5jZS4g4oCcVHlwZS0z4oCdIGlzIGEgQ1hMIGRldmljZSB0aGF0IGFj dHMgYXMgYSBtZW1vcnkgZXhwYW5kZXIgZm9yDQpSQU0gb3IgUGVyc2lzdGVudCBNZW1vcnkuIFRo ZSBkZXZpY2UgbWlnaHQgYmUgaW50ZXJsZWF2ZWQgd2l0aCBvdGhlciBDWEwgZGV2aWNlcw0KaW4g YSBnaXZlbiBwaHlzaWNhbCBhZGRyZXNzIHJhbmdlLg0KDQpJbiBhZGRpdGlvbiB0byB0aGUgY29y ZSBmdW5jdGlvbmFsaXR5IG9mIGRpc2NvdmVyaW5nIHRoZSBzcGVjIGRlZmluZWQgcmVnaXN0ZXJz DQphbmQgcmVzb3VyY2VzLCBpbnRyb2R1Y2UgYSBDWEwgZGV2aWNlIG1vZGVsIHRoYXQgd2lsbCBi ZSB0aGUgZm91bmRhdGlvbiBmb3INCnRyYW5zbGF0aW5nIENYTCBjYXBhYmlsaXRpZXMgaW50byBl eGlzdGluZyBMaW51eCBpbmZyYXN0cnVjdHVyZSBmb3IgUGVyc2lzdGVudA0KTWVtb3J5IGFuZCBv dGhlciBtZW1vcnkgZGV2aWNlcy4gRm9yIG5vdywgdGhpcyBvbmx5IGluY2x1ZGVzIHN1cHBvcnQg Zm9yIHRoZQ0KbWFuYWdlbWVudCBjb21tYW5kIG1haWxib3ggdGhlIHN1cmZhY2luZyBvZiB0eXBl LTMgZGV2aWNlcy4gVGhlc2UgY29udHJvbA0KZGV2aWNlcyBmaWxsIHRoZSByb2xlIG9mIOKAnERJ TU1z4oCdIC8gbm1lbVggbWVtb3J5LWRldmljZXMgaW4gTElCTlZESU1NIHRlcm1zLg0KDQojIyBV c2Vyc3BhY2UgSW50ZXJhY3Rpb24NCg0KSW50ZXJhY3Rpb24gd2l0aCB0aGUgZHJpdmVyIGFuZCB0 eXBlLTMgZGV2aWNlcyB2aWEgdGhlIENYTCBkcml2ZXJzIGlzIGludHJvZHVjZWQNCmluIHRoaXMg cGF0Y2ggc2VyaWVzIGFuZCBjb25zaWRlcmVkIHN0YWJsZSBBQkkuIFRoZXkgaW5jbHVkZQ0KDQog ICAqIHN5c2ZzIC0gRG9jdW1lbnRhdGlvbi9BQkkvdGVzdGluZy9zeXNmcy1idXMtY3hsDQogICAq IElPQ1RMIC0gRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2N4bC9tZW1vcnktZGV2aWNlcy5yc3QN CiAgICogZGVidWdmcyAtIERvY3VtZW50YXRpb24vQUJJL3Rlc3RpbmcvZGVidWdmcy1kZWJ1Zw0K DQoNCldvcmsgaXMgaW4gcHJvY2VzcyB0byBhZGQgc3VwcG9ydCBmb3IgQ1hMIGludGVyYWN0aW9u cyB0byB0aGUgbmRjdGwgcHJvamVjdCBbNV0NCg0KIyMjIERldmVsb3BtZW50IHBsYW5zDQoNCk9u ZSBvZiB0aGUgdW5pcXVlIGNoYWxsZW5nZXMgdGhhdCBDWEwgaW1wb3NlcyBvbiB0aGUgTGludXgg ZHJpdmVyIG1vZGVsIGlzIHRoYXQNCml0IHJlcXVpcmVzIHRoZSBvcGVyYXRpbmcgc3lzdGVtIHRv IHBlcmZvcm0gcGh5c2ljYWwgYWRkcmVzcyBzcGFjZSBtYW5hZ2VtZW50DQppbnRlcmxlYXZlZCBh Y3Jvc3MgZGV2aWNlcyBhbmQgYnJpZGdlcy4gV2hlcmVhcyBMSUJOVkRJTU0gaGFuZGxlcyBhIGxp c3Qgb2YNCmVzdGFibGlzaGVkIHN0YXRpYyBwZXJzaXN0ZW50IG1lbW9yeSBhZGRyZXNzIHJhbmdl cyAoZm9yIGV4YW1wbGUgZnJvbSB0aGUgQUNQSQ0KTkZJVCksIENYTCBpbnRyb2R1Y2VzIGhvdHBs dWcgYW5kIHRoZSBjb25jZXB0IG9mIGFsbG9jYXRpbmcgYWRkcmVzcyBzcGFjZSB0bw0KaW5zdGFu dGlhdGUgcGVyc2lzdGVudCBtZW1vcnkgcmFuZ2VzLiBUaGlzIGlzIHNpbWlsYXIgdG8gUENJIGlu IHRoZSBzZW5zZSB0aGF0DQp0aGUgcGxhdGZvcm0gZXN0YWJsaXNoZXMgdGhlIE1NSU8gcmFuZ2Ug Zm9yIFBDSSBCQVJzIHRvIGJlIGFsbG9jYXRlZCwgYnV0IGl0IGlzDQpzaWduaWZpY2FudGx5IGNv bXBsaWNhdGVkIGJ5IHRoZSBmYWN0IHRoYXQgYSBnaXZlbiBkZXZpY2UgY2FuIG9wdGlvbmFsbHkg YmUNCmludGVybGVhdmVkIHdpdGggb3RoZXIgZGV2aWNlcyBhbmQgY2FuIHBhcnRpY2lwYXRlIGlu IHNldmVyYWwgaW50ZXJsZWF2ZS1zZXRzIGF0DQpvbmNlLiBMSUJOVkRJTU0gaGFuZGxlZCBzb21l dGhpbmcgbGlrZSB0aGlzIHdpdGggdGhlIGFsaWFzaW5nIGJldHdlZW4gUE1FTSBhbmQNCkJMT0NL LVdJTkRPVyBtb2RlLCBidXQgQ1hMIGFkZHMgZmxleGliaWxpdHkgdG8gYWxpYXMgREVWSUNFIE1F TU9SWSB0aHJvdWdoIHVwIHRvDQoxMCBkZWNvZGVycyBwZXIgZGV2aWNlLg0KDQpBbGwgb2YgdGhl IGFib3ZlIG5lZWRzIHRvIGJlIGVuYWJsZWQgd2l0aCByZXNwZWN0IHRvIFBDSSBob3RwbHVnIGV2 ZW50cyBvbg0KVHlwZS0zIG1lbW9yeSBkZXZpY2Ugd2hpY2ggbmVlZHMgaG9va3MgdG8gZGV0ZXJt aW5lIGlmIGEgZ2l2ZW4gZGV2aWNlIGlzDQpjb250cmlidXRpbmcgdG8gYSAiU3lzdGVtIFJBTSIg YWRkcmVzcyByYW5nZSB0aGF0IGlzIHVuYWJsZSB0byBiZSB1bnBsdWdnZWQuIEluDQpvdGhlciB3 b3JkcyBDWEwgdGllcyBQQ0kgaG90cGx1ZyB0byBNZW1vcnkgSG90cGx1ZyBhbmQgUENJIGhvdHBs dWcgbmVlZHMgdG8gYmUNCmFibGUgdG8gbmVnb3RpYXRlIHdpdGggbWVtb3J5IGhvdHBsdWcuICBJ biB0aGUgbWVkaXVtIHRlcm0gdGhlIGltcGxpY2F0aW9ucyBvZg0KQ1hMIGhvdHBsdWcgdnMgQUNQ SSBTUkFUL1NMSVQvSE1BVCBuZWVkIHRvIGJlIHJlY29uY2lsZWQuIE9uZSBjYXBhYmlsaXR5IHRo YXQNCnNlZW1zIHRvIGJlIG5lZWRlZCBpcyBlaXRoZXIgdGhlIGR5bmFtaWMgYWxsb2NhdGlvbiBv ZiBuZXcgbWVtb3J5IG5vZGVzLCBvcg0KZGVmYXVsdCBpbml0aWFsaXppbmcgZXh0cmEgcGdkYXQg aW5zdGFuY2VzIGJleW9uZCB3aGF0IGlzIGVudW1lcmF0ZWQgaW4gQUNQSQ0KU1JBVCB0byBhY2Nv bW1vZGF0ZSBob3QtYWRkZWQgQ1hMIG1lbW9yeS4NCg0KUGF0Y2hlcyB3ZWxjb21lLCBxdWVzdGlv bnMgd2VsY29tZSBhcyB0aGUgZGV2ZWxvcG1lbnQgZWZmb3J0IG9uIHRoZSBwb3N0IHY1LjEyDQpj YXBhYmlsaXRpZXMgcHJvY2VlZHMuDQoNCiMjIFJ1bm5pbmcgaW4gUUVNVQ0KDQpUaGUgaW5jYW50 YXRpb24gdG8gZ2V0IENYTCBzdXBwb3J0IGluIFFFTVUgWzRdIGlzIGNvbnNpZGVyZWQgdW5zdGFi bGUgYXQgdGhpcw0KdGltZS4gRnV0dXJlIHJlYWRlcnMgb2YgdGhpcyBjb3ZlciBsZXR0ZXIgc2hv dWxkIHZlcmlmeSBpZiBhbnkgY2hhbmdlcyBhcmUNCm5lZWRlZC4gRm9yIHRoZSBub3ZpY2UgUUVN VSB1c2VyLCB0aGUgZm9sbG93aW5nIGNhbiBiZSBjb3B5L3Bhc3RlZCBpbnRvIGENCndvcmtpbmcg UUVNVSBjb21tYW5kbGluZS4gSXQgaXMgZW5vdWdoIHRvIG1ha2UgdGhlIHNpbXBsZXN0IHRvcG9s b2d5IHBvc3NpYmxlLg0KVGhlIHRvcG9sb2d5IHdvdWxkIGNvbnNpc3Qgb2YgYSBzaW5nbGUgbWVt b3J5IHdpbmRvdywgc2luZ2xlIHR5cGUzIGRldmljZSwNCnNpbmdsZSByb290IHBvcnQsIGFuZCBz aW5nbGUgaG9zdCBicmlkZ2UuDQoNCiAgICArLS0tLS0tLS0tLS0tLSsNCiAgICB8ICAgQ1hMIFBY QiAgIHwNCiAgICB8ICAgICAgICAgICAgIHwNCiAgICB8ICArLS0tLS0tLSsgIHw8LS0tLS0tLS0t LSsNCiAgICB8ICB8Q1hMIFJQIHwgIHwgICAgICAgICAgIHwNCiAgICArLS0rLS0tLS0tLSstLSsg ICAgICAgICAgIHYNCiAgICAgICAgICAgfCAgICAgICAgICAgICstLS0tLS0tLS0tKw0KICAgICAg ICAgICB8ICAgICAgICAgICAgfCAid2luZG93IiB8DQogICAgICAgICAgIHwgICAgICAgICAgICAr LS0tLS0tLS0tLSsNCiAgICAgICAgICAgdiAgICAgICAgICAgICAgICAgIF4NCiAgICArLS0tLS0t LS0tLS0tLSsgICAgICAgICAgIHwNCiAgICB8ICBDWEwgVHlwZSAzIHwgICAgICAgICAgIHwNCiAg ICB8ICAgRGV2aWNlICAgIHw8LS0tLS0tLS0tLSsNCiAgICArLS0tLS0tLS0tLS0tLSsNCg0KLy8g TWVtb3J5IGJhY2tlbmQNCi1vYmplY3QgbWVtb3J5LWJhY2tlbmQtZmlsZSxpZD1jeGwtbWVtMSxz aGFyZSxtZW0tcGF0aD1jeGwtdHlwZTMsc2l6ZT01MTJNDQoNCi8vIEhvc3QgQnJpZGdlDQotZGV2 aWNlIHB4Yi1jeGwgaWQ9Y3hsLjAsYnVzPXBjaWUuMCxidXNfbnI9NTIsdWlkPTAgbGVuLXdpbmRv dy1iYXNlPTEsd2luZG93LWJhc2VbMF09MHg0YzAwMDAwMDAgbWVtZGV2WzBdPWN4bC1tZW0xDQoN Ci8vIFNpbmdsZSByb290IHBvcnQNCi1kZXZpY2UgY3hsIHJwLGlkPXJwMCxidXM9Y3hsLjAsYWRk cj0wLjAsY2hhc3Npcz0wLHNsb3Q9MCxtZW1kZXY9Y3hsLW1lbTENCg0KLy8gU2luZ2xlIHR5cGUz IGRldmljZQ0KLWRldmljZSBjeGwtdHlwZTMsYnVzPXJwMCxtZW1kZXY9Y3hsLW1lbTEsaWQ9Y3hs LXBtZW0wLHNpemU9MjU2TSAtZGV2aWNlIGN4bC10eXBlMyxidXM9cnAxLG1lbWRldj1jeGwtbWVt MSxpZD1jeGwtcG1lbTEsc2l6ZT0yNTZNDQoNCi0tLQ0KDQpbMV06IGh0dHBzOi8vbG9yZS5rZXJu ZWwub3JnL2xpbnV4LWN4bC8yMDIwMTIwOTAwMjQxOC4xOTc2MzYyLTEtYmVuLndpZGF3c2t5QGlu dGVsLmNvbS8NClsyXTogaHR0cHM6Ly93d3cuY29tcHV0ZWV4cHJlc3NsaW5rLm9yZy9dKGh0dHBz Oi8vd3d3LmNvbXB1dGVleHByZXNzbGluay5vcmcvDQpbM106IGh0dHBzOi8vbG9yZS5rZXJuZWwu b3JnL3FlbXUtZGV2ZWwvMjAyMTAxMDUxNjUzMjMuNzgzNzI1LTEtYmVuLndpZGF3c2t5QGludGVs LmNvbS9ULyN0DQpbNF06IGh0dHBzOi8vZ2l0bGFiLmNvbS9id2lkYXdzay9xZW11Ly0vdHJlZS9j eGwtMi4wdioNCls1XTogaHR0cHM6Ly9naXRodWIuY29tL3BtZW0vbmRjdGwvdHJlZS9jeGwtMi4w dioNCg0KDQpCZW4gV2lkYXdza3kgKDEyKToNCiAgY3hsL21lbTogTWFwIG1lbW9yeSBkZXZpY2Ug cmVnaXN0ZXJzDQogIGN4bC9tZW06IEZpbmQgZGV2aWNlIGNhcGFiaWxpdGllcw0KICBjeGwvbWVt OiBJbXBsZW1lbnQgcG9sbGVkIG1vZGUgbWFpbGJveA0KICBjeGwvbWVtOiBBZGQgYmFzaWMgSU9D VEwgaW50ZXJmYWNlDQogIGN4bC9tZW06IEFkZCBzZW5kIGNvbW1hbmQNCiAgdGFpbnQ6IGFkZCB0 YWludCBmb3IgZGlyZWN0IGhhcmR3YXJlIGFjY2Vzcw0KICBjeGwvbWVtOiBBZGQgYSAiUkFXIiBz ZW5kIGNvbW1hbmQNCiAgY3hsL21lbTogQ3JlYXRlIGNvbmNlcHQgb2YgZW5hYmxlZCBjb21tYW5k cw0KICBjeGwvbWVtOiBVc2UgQ0VMIGZvciBlbmFibGluZyBjb21tYW5kcw0KICBjeGwvbWVtOiBB ZGQgc2V0IG9mIGluZm9ybWF0aW9uYWwgY29tbWFuZHMNCiAgY3hsL21lbTogQWRkIGxpbWl0ZWQg R2V0IExvZyBjb21tYW5kICgwNDAxaCkNCiAgTUFJTlRBSU5FUlM6IEFkZCBtYWludGFpbmVycyBv ZiB0aGUgQ1hMIGRyaXZlcg0KDQpEYW4gV2lsbGlhbXMgKDIpOg0KICBjeGwvbWVtOiBJbnRyb2R1 Y2UgYSBkcml2ZXIgZm9yIENYTC0yLjAtVHlwZS0zIGVuZHBvaW50cw0KICBjeGwvbWVtOiBSZWdp c3RlciBDWEwgbWVtWCBkZXZpY2VzDQoNCiAuY2xhbmctZm9ybWF0ICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgfCAgICAxICsNCiBEb2N1bWVudGF0aW9uL0FCSS90ZXN0aW5nL2RlYnVn ZnMtY3hsICAgICAgICAgfCAgIDEwICsNCiBEb2N1bWVudGF0aW9uL0FCSS90ZXN0aW5nL3N5c2Zz LWJ1cy1jeGwgICAgICAgfCAgIDI2ICsNCiBEb2N1bWVudGF0aW9uL2FkbWluLWd1aWRlL3N5c2N0 bC9rZXJuZWwucnN0ICAgfCAgICAxICsNCiBEb2N1bWVudGF0aW9uL2FkbWluLWd1aWRlL3RhaW50 ZWQta2VybmVscy5yc3QgfCAgICA2ICstDQogRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2N4bC9p bmRleC5yc3QgICAgICAgIHwgICAxMiArDQogLi4uL2RyaXZlci1hcGkvY3hsL21lbW9yeS1kZXZp Y2VzLnJzdCAgICAgICAgIHwgICA0NiArDQogRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2luZGV4 LnJzdCAgICAgICAgICAgIHwgICAgMSArDQogLi4uL3VzZXJzcGFjZS1hcGkvaW9jdGwvaW9jdGwt bnVtYmVyLnJzdCAgICAgIHwgICAgMSArDQogTUFJTlRBSU5FUlMgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgICAxMSArDQogZHJpdmVycy9LY29uZmlnICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgICAgMSArDQogZHJpdmVycy9NYWtlZmlsZSAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgICAgMSArDQogZHJpdmVycy9iYXNlL2NvcmUuYyAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgICAxNCArDQogZHJpdmVycy9jeGwvS2NvbmZpZyAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgICA0OSArDQogZHJpdmVycy9jeGwvTWFrZWZpbGUgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgICAgNyArDQogZHJpdmVycy9jeGwvYnVzLmMgICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgICAyOSArDQogZHJpdmVycy9jeGwvY3hsLmggICAgICAgICAgICAg ICAgICAgICAgICAgICAgIHwgIDE0MCArKw0KIGRyaXZlcnMvY3hsL21lbS5jICAgICAgICAgICAg ICAgICAgICAgICAgICAgICB8IDE2MDMgKysrKysrKysrKysrKysrKysNCiBkcml2ZXJzL2N4bC9w Y2kuaCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgIDM0ICsNCiBpbmNsdWRlL2xpbnV4 L2RldmljZS5oICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAxICsNCiBpbmNsdWRlL2xpbnV4 L2tlcm5lbC5oICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAzICstDQogaW5jbHVkZS91YXBp L2xpbnV4L2N4bF9tZW0uaCAgICAgICAgICAgICAgICAgIHwgIDE4MCArKw0KIGtlcm5lbC9wYW5p Yy5jICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgIDEgKw0KIDIzIGZpbGVzIGNo YW5nZWQsIDIxNzYgaW5zZXJ0aW9ucygrKSwgMiBkZWxldGlvbnMoLSkNCiBjcmVhdGUgbW9kZSAx MDA2NDQgRG9jdW1lbnRhdGlvbi9BQkkvdGVzdGluZy9kZWJ1Z2ZzLWN4bA0KIGNyZWF0ZSBtb2Rl IDEwMDY0NCBEb2N1bWVudGF0aW9uL0FCSS90ZXN0aW5nL3N5c2ZzLWJ1cy1jeGwNCiBjcmVhdGUg bW9kZSAxMDA2NDQgRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2N4bC9pbmRleC5yc3QNCiBjcmVh dGUgbW9kZSAxMDA2NDQgRG9jdW1lbnRhdGlvbi9kcml2ZXItYXBpL2N4bC9tZW1vcnktZGV2aWNl cy5yc3QNCiBjcmVhdGUgbW9kZSAxMDA2NDQgZHJpdmVycy9jeGwvS2NvbmZpZw0KIGNyZWF0ZSBt b2RlIDEwMDY0NCBkcml2ZXJzL2N4bC9NYWtlZmlsZQ0KIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2 ZXJzL2N4bC9idXMuYw0KIGNyZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2N4bC9jeGwuaA0KIGNy ZWF0ZSBtb2RlIDEwMDY0NCBkcml2ZXJzL2N4bC9tZW0uYw0KIGNyZWF0ZSBtb2RlIDEwMDY0NCBk cml2ZXJzL2N4bC9wY2kuaA0KIGNyZWF0ZSBtb2RlIDEwMDY0NCBpbmNsdWRlL3VhcGkvbGludXgv Y3hsX21lbS5oDQoNCi0tIA0KMi4zMC4wDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXwpMaW51eC1udmRpbW0gbWFpbGluZyBsaXN0IC0tIGxpbnV4LW52ZGlt bUBsaXN0cy4wMS5vcmcKVG8gdW5zdWJzY3JpYmUgc2VuZCBhbiBlbWFpbCB0byBsaW51eC1udmRp bW0tbGVhdmVAbGlzdHMuMDEub3JnCg==