From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753848AbcGTJ4s (ORCPT ); Wed, 20 Jul 2016 05:56:48 -0400 Received: from eu-smtp-delivery-143.mimecast.com ([146.101.78.143]:24369 "EHLO eu-smtp-delivery-143.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753639AbcGTJ4m convert rfc822-to-8bit (ORCPT ); Wed, 20 Jul 2016 05:56:42 -0400 Date: Wed, 20 Jul 2016 17:56:25 +0800 From: Dennis Chen To: Eric Auger CC: , , , , , , , , , , , , , , , , , , Subject: Re: [PATCH v11 0/8] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes Message-ID: <20160720095624.GA1915@arm.com> References: <1468932911-23062-1-git-send-email-eric.auger@redhat.com> MIME-Version: 1.0 In-Reply-To: <1468932911-23062-1-git-send-email-eric.auger@redhat.com> User-Agent: Mutt/1.5.24 (2015-08-30) X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:217.140.96.140;IPV:CAL;SCL:-1;CTRY:GB;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(7916002)(2980300002)(438002)(189002)(24454002)(199003)(57704003)(8676002)(246002)(26826002)(50466002)(7846002)(7696003)(5003600100003)(106466001)(86362001)(2950100001)(356003)(33656002)(8936002)(77096005)(46406003)(36756003)(54356999)(15975445007)(97756001)(50986999)(110136002)(19580405001)(189998001)(87936001)(19580395003)(6806005)(92566002)(2906002)(4326007)(1076002)(104016004)(4001350100001)(586003)(47776003)(76176999)(83506001)(305945005)(23726003)(11100500001)(18370500001);DIR:OUT;SFP:1101;SCL:1;SRVR:HE1PR0801MB1436;H:nebula.arm.com;FPR:;SPF:Pass;PTR:fw-tnat.cambridge.arm.com;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;AM1FFO11FD055;1:ERqZBjlMvXv9Kz8zlPvUrvfC3lTjCYA+FwAF2AbqBIbMXHWgpaPpkI4FfsMMWS37MvkirNjWJzglDm90HPNymDA2l1AHmMWyF06b+177MHWIg06m7B0NW/kyyCRZO27lBI+KJrGkPQQFiGLY5Izs1WumbFmvN2y2E9gg27xFV7yPMdr2mlGs/EOhHm2NTYthoGNcmSfzuAtjZil5t2YgP5P221AehPn1JhcFtdlwDfBwxqLExkSNmxcsQw+qATWnk5CMl2j7eNTz0M0cewqf73c73aTPloaZprd+HaUF790we1JWlHQv5xJgsk1mTWnOKfTC+yk1f9HXtzHEbItSFzQ0iSm7zFQfN1sCAEMFLAFhZUcNbS4ubdi7AbrMPF1n8vOtc8g+0w5dnFYOYrvaUp4U7ahEBT08Cc1a+PLWv328fY9GlWJsTDNx3XDvDn/xt/+JSqlmsEiqmQRWWmyMyKg3Ar4B9m1gmK17zlziLdcj5kJYjxksrilarFkJxRCIN7Mwn5O+OQM+sQwfeo1Ktu4t72wvso0ipV0i4FI7Dda24JVzKfBjdi80BYkCUpehiD3a2j7BNy6IktLw7RB5iVoWQARjKwGu2AKxXKswTAEgTBJ6e+gXk+TLfrmGN2ol X-MS-Office365-Filtering-Correlation-Id: 55c2ae09-c1de-4c77-4554-08d3b084220c X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1436;2:gT4yCDeh/p7nsDoJ5s81VZl/Fw/H+LoG0OYbx0pLfLNWfymizkoev9M90+B6U7/u303TTx5hUB6ztKSUP4SRO/jwSWW8GNUC5p3YYcOYwnNhFBg3PCrn5Ljf6/XCyzVm7jqy3uArGnGu0cNnaYwums4PDVB0Qu5aDgR7m7Ku8XwCBpFI8qu2ytJ+ZiKZ7xIZ;3:F/vnIRe0SP+ZZjDx5zOAr8zthiJ/Vn9gbTWb+thThgdC4Nrk9eMxD4N67CgNgaF5CvtXE2feY+XpKhsYh+lK74l8ebtEPTtC6gCBcW2V0t1nmiykxwoyK0fJH64DvdK32b5PfBOGI+oouyAfpybNlZwFx1FZzp5btbkZr529PLuDPwQrN/Vuub6c9XS/5KXQSBemChTPuWN7BLpfaMxiXPAwa6yFdK+PLrOiCXVU5kTSshikO7zwXGTqLIu/8v/Ejk6jPK8lRA6uG276elxvuw==;25:eBCF9+DA+7yWX1QMrdYIqYOF9tcSoNC9O83g2HdbrCUPxCV8SoX1FetZ5PMPTCZfGAM75aw03nJSbWi28su1ncJ0ZlVL8VIqoqGRJB8nrHHFDyO1ggOH1MYksXe6HT7kMhgGePogTcYwK4ZEZ/ugk0Q6pjpyyFPlEgnWxzeqntHVTNjX6aAI53cMdehzoIsBSgSvpW9q+L2GGwcwFB7h1shPhrQL2bb6MGRdRtCRzK8UM1WGuRdVEdLdmUwwc5sqR2xQ858H+cdPYdbhugb7aS+BDlDWSfCyKVkui3midlQBpMQw8fN0d/nTzyVW0frjaKUwm5Vlh7HVL1po/0YZtiYjLni30OetH4iBT34e8wNCRzSRkJbWCJ+KYklPx/bTSk+5Fu4V8tTvMTpmpzDpgRILqmwmbju2M4+5loDV65Y= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501002);SRVR:HE1PR0801MB1436; X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1436;31:pRKtudv8j7LwJ/7FX36x+YFwuraN1QoslDMFFsooLGm+T++qSVdGL5C1jXkJzIy46Uq+KTbRsE7ZY69aRPWDpqrw93hpd98ciNickiMHrP+JBEu3uxXqynb4Mg5lTJJTHAjFP+2HCkO5bCyVUX0eFRBapMr5NskA9/dvdZf3/aB2f4hNugFDb4lEu8ax/L9R/wqWrkr2lKHqD/CwUJqgnQ==;20:z4UynIaeeKvpf8jezJqpzXF+cPGjBwkvB6eOrT2aZilMMEgz817QqwpTavW43tjrl9fBalkes+IVHrEQ8zWSJn7t1JvH9zjdkzFE0I3yZBoPgvihAcSaCZ6u5iLsduN2IbgdMMvD0ULC2UJQBKNaHqJR8ogMA1ICcgs6H+xSOecRblJNDk2m9DLeYW1lW3uAiddJmTCWyRJwGZdOlBPddkDSPd8QYUo287LLGOGL/tDV6FMBiYcX56czSX/V9Bao NoDisclaimer: True X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(166708455590820)(85170053105377); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(13023025)(13024025)(13020025)(13013025)(5005006)(8121501046)(10201501046)(3002001)(6055026);SRVR:HE1PR0801MB1436;BCL:0;PCL:0;RULEID:;SRVR:HE1PR0801MB1436; X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1436;4:NCQYymNEQG8C87seHBBTf+dWDB3DMfhSROXhRDPfWDJoynAjFlNfRME4ts+Ua2YMY0IqPIvwBdYRshuhgCYRXExJNxP+ClhTNe3bXxsybfKCvHZgR+ji3aXaxbVoiS4SCvaybx/cN3jUtTfXW8BazNbMw2Bv/mDL9UXFQzgxMOVBOGtSUumKmOBpSO0D4tp4jO5+thZVBTsm24AAewnoADPFiBRj9oeW2GSl433NC1kmh+j/cosOKiQAmaOcLRqYMl/ngPBEEZfmTMA3zSQvH1BS/h/dZaqV8J7bttIiPq5oHVFXQNCSTfu8ilVMpLsXgPT5fmeXzQNJOCup0oebp0ujYJ5Bznn3QMZAVJeBHJ2LxZZRaNb9zTvkZU8VupoPxpaXtPCvxNH/9veUeFAlKcRBs0jPHUqOXluesRHizeYyyxVF1HrOpkHYZvTRELgnfpFZuaGO7+BufdjpBalvQu+lQXvY04KEi7mj/4b+YlUAU9SAIRDnYkcaCJXiihn8h6Uh3efD8UjIurouE0ZbYh3JoNj4HQWXsXp7DnJFfe4= X-Forefront-PRVS: 000947967F X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1;HE1PR0801MB1436;23:d2WmGmFqQmMXQ4E0LokpEpBryUV/kJhDd1UGoyc?= =?us-ascii?Q?p8ySxbEso0iJJguEaRrHWh9mHSfg4iFS9m1G3YXNVijeqAizyvmIG8AqVy5/?= =?us-ascii?Q?TTojjPhTsg/FV+s1QmJS9pH0aELmKIsYFCS17uhhyyiUMzrHEZrWqEOfGImj?= =?us-ascii?Q?Xc51s87DcGkLp8IbQue6OuCKokewD8dwIDE5lSLxDToLso5AMPDihBwwhdBO?= =?us-ascii?Q?H74qHhA5j54byh3wsBAUeAfu+ESlP5wr9OETj0jdUalpHV/hnUN27QgcQ4Eg?= =?us-ascii?Q?xa/8ad+fiRrHQocvdK/2AJXDmKnkMIkorOrVXb6NIoAm1ve6dUgxD5+Ba56Y?= =?us-ascii?Q?zf2IJUf0xCOB8/9JHEqGliYSzpHFYKvpIjp6hmroO+v8YbYuw7/pabJjhQ9R?= =?us-ascii?Q?WuizzGKUiBcRwNp6QTwoOP+1+USnO6qQnwGCMAaHvqYLT8DJrOJybuVqvZP9?= =?us-ascii?Q?hgaGVRaybaN9eMJchSKquZRU8uRuEaijnsbbuiomGdCI+1v0Y1IzVVU1UmYm?= =?us-ascii?Q?c78OPSFT9dpkyM6qESJpkV8qqYBj2T5fZfk9Wa50PJO9q0mX/YDPz5dxbkhQ?= =?us-ascii?Q?TrfmbsjMSaAhNz+2Sdn2j/HHIWdGVzmVz2nbEPFL3GviZTIpPY6CoatSnWHy?= =?us-ascii?Q?gCaKMqReZd+10/5NmCIrXhKzKUrT5LFLhKUHHQe1D24Vyv4YXZvu3ph0tHZG?= =?us-ascii?Q?rbxhIDnYlBU13ZiANy9blJCAq9kRO0v0v7Sj7HhBHeJFqKak46/wX/vDoFZr?= =?us-ascii?Q?u6jHzupCOF6mIxADdBxDjB+CGNHlq6aLKMtMYaEdzm8FnQGDB7i6VurDcvfM?= =?us-ascii?Q?DaRS+ZDaIm8jGAIL0Fl0wmZI+lsTXmiP949swZ58ZxnzIcuc/SqcWhyRLG6p?= =?us-ascii?Q?cJpcAYQjmEO9VR6i0EbP3RSfAypC4fsG4nqgR6XtyeB+6FXImiWc1RwvcAaS?= =?us-ascii?Q?rpIsFOg6RJn/cpnaJ7mYaPIyQpYAsnZkXxktcYtJfY8U/SH8u4w5pYmRCjcQ?= =?us-ascii?Q?LlKfxbTnX2LgByTtwqEZ0Hn+1Z/fLATNhn8Gr+rYQln8mStkd6dJ7hFASxk0?= =?us-ascii?Q?j/e8SJYWs/r+DgHCpgrnhXIq7uOhDf4YvKBqhuW8AtDvIIGSdp74f1iRhxVQ?= =?us-ascii?Q?TMtpvV1ACJxW2AsY1X624tFX26Yv/nO42i54boS4IH9HXDOGVNKuWeXsCdCf?= =?us-ascii?Q?Eoxd2qIhlmd8hQI9ooLpeEa9pfDpNOmgTmAeR?= X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1436;6:xcYJE2PPLB/6otsdq+3Yyw/VqGJk8inSR/V0KKEz+sizKw0+kurm/Al5oWceb9IyPTBMYHZGQtLnNrRee2XX4RXWiY8UF/K/AoekBbiGVRTQ5Dw4F7jK/Qf+198SCAlcAQuzlhAs7IxxQSZgv48ksfhDZ0ATpWJ5CNEtK8ZWvAgA6r0w8XLrE7172+BA3ojyNpHJ8c5B/nPD+9FfD6FnBELZ6GjYGlUI08kUWGpz+UDarMb9iFiB3mlzsSKGhjeg6q70NllEj5QMZ7cUGLXxhLvfgnlfwKrywAqDTshYnKmkIiHx8PqVm7gVjuGCIBgiEaKvrVV1U/kgEqHfeGWf3Q==;5:AfGGmZzgROZqwrUCB+5lk6QI7I4sW5pOTsrNherng2p0vsBm6mclKnOyEHLs8HVQFUQet4DZ/Zo5xxNK0CePfmjd+D/KHYmVIopwPLQBcb5DT5d9xHtR7Pg7Cb2z5p5A1M3tH+hG3PNyE/0FAhmWbw==;24:GJib8bd/L5dqeJoyDxukK92NtQqRnl8iKbWzk7Glxtu3g5q01lLbsj2H5XJ8YhaCg98gqgu4eU5xc8euS1nXu1eByjlnqgYAa87bRnI/7ig=;7:iEYEHyX1C2s4xOJR10/dz1NQb/EzX4YHEPK94XeYV98IdhpSKFWYhh6NSFu0TRBmTu0WmYKBwFWHDudu07ShSe1lPhF/LQe05AZ+aHDkwsUxt6ARCfofWsqBtbYcsRXQOJaEq9TJ9pP1NPz61JYR5oFsadc8HHOCTWn2WHwKmuhBKPqMEE02xgKxhkI51IakYgAwUx+fUig9z/k4P5LAjgg9bI+/olY+EtkE3Z3+FBpIaeM2PmwtnGa5GJxSGhop SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;HE1PR0801MB1436;20:+9RD2ZS32KA1GeaVLM6wLfXtYLWlDPZBMcTe3UElfP2Id/FaNaQ6wD0sRibbKE6I3H3ko6C137bsHotqMu0G2upZqTmV7N9rMqFG2xtZDZ7+pzoKkIYa9Skm86vpfAa7RhmpbBGqO4TwJl0VP1tjpQThz4o80hglXgOXFHCkvf0= X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Jul 2016 09:56:34.0277 (UTC) X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d;Ip=[217.140.96.140];Helo=[nebula.arm.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0801MB1436 X-MC-Unique: SBCdIpfVO7mNEmYXKII69w-1 Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: 8BIT Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Eric, On Tue, Jul 19, 2016 at 12:55:03PM +0000, Eric Auger wrote: > This series introduces the msi-iommu api used to: > > - allocate/free resources for MSI IOMMU mapping > - set the MSI iova window aperture > - map/unmap physical addresses onto MSI IOVAs. > - determine whether an msi needs to be iommu mapped > - overwrite an msi_msg PA address with its pre-allocated/mapped IOVA > > Also a new iommu domain attribute, DOMAIN_ATTR_MSI_GEOMETRY is introduced > to report the MSI iova window geometry (aperture and iommu-msi API support). > > Currently: > - iommu driver is supposed to allocate/free MSI mapping resources > - VFIO subsystem is supposed to set the MSI IOVA aperture. > - The MSI layer is supposed to allocate/free iova mappings and overwrite > msi_msg with IOVA at composition time > > More details & context can be found at: > http://www.linaro.org/blog/core-dump/kvm-pciemsi-passthrough-armarm64/ > > Best Regards > > Eric > > Git: complete series available at > https://github.com/eauger/linux/tree/v4.7-rc7-passthrough-v11 > Why can't I find this new series on your git tree: https://git.linaro.org/people/eric.auger/linux.git ? Also, do I need to download all the 3-part patches to test the PCIe NIC passthru as I did on your v9 series? Thanks, Dennis > > see part III for wrap-up details. > > History: > v10 -> v11: > - no change in the series, just incremented for consistency > - added a temporary patch in the branch: > "iommu/iova: FIXUP! validate iova_domain input to put_iova_domain" > originally sent by Nate and adapted for this use case. This is currently > under discussion on the ML. The crash typically occurs in case unsafe > interrupts are discovered while allow_unsafe_interrupts is not set. > > v9 -> v10: > - split error management in iommu_msi_set_aperture > > v8 -> v9: > - rename iommu_domain_msi_geometry programmable flag into iommu_msi_supported > - introduce msi_apperture_valid helper and use this instead of is_aperture_set > > v7 -> v8: > - The API is retargetted for MSI: renamed msi-iommu > all "dma-reserved" namings removed > - now implemented upon dma-iommu (get, put, init), ie. reuse iova_cookie, > and iova API > - msi mapping resources now are guaranteed to exist during the whole iommu > domain's lifetime. No need to lock to garantee the cookie integrity > - removed alloc/free_reserved_reserved_iova_domain. We now have a single > function that sets the aperture, looking like iommu_dma_init_domain. > - we now use a list instead of an RB-tree > - prot is not propagated anymore at domain creation due to the retargetting > for MSI > - iommu_domain pointer removed from doorbell_mapping struct > - replaced DOMAIN_ATTR_MSI_MAPPING by DOMAIN_ATTR_MSI_GEOMETRY > > v6 -> v7: > - fixed known lock bugs and multiple page sized slots matching > (I only have a single MSI frame made of a single page) > - reserved_iova_cookie now pointing to a struct that encapsulates the > iova domain handle + protection attribute passed from VFIO (Alex' req) > - 2 new functions exposed: iommu_msi_mapping_translate_msg, > iommu_msi_mapping_desc_to_domain: not sure this is the right location/proto > though > - iommu_put_reserved_iova now takes a phys_addr_t > - everything now is cleanup on iommu_domain destruction > > RFC v5 -> patch v6: > - split to ease the review process > - in dma-reserved-api use a spin lock instead of a mutex (reported by > Jean-Philippe) > - revisit iommu_get_reserved_iova API to pass a size parameter upon > Marc's request > - Consistently use the page order passed when creating the iova domain. > - init reserved_binding_list (reported by Julien) > > RFC v4 -> RFC v5: > - take into account Thomas' comments on MSI related patches > - split "msi: IOMMU map the doorbell address when needed" > - increase readability and add comments > - fix style issues > - split "iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute" > - platform ITS now advertises IOMMU_CAP_INTR_REMAP > - fix compilation issue with CONFIG_IOMMU API unset > - arm-smmu-v3 now advertises DOMAIN_ATTR_MSI_MAPPING > > RFC v3 -> v4: > - Move doorbell mapping/unmapping in msi.c > - fix ref count issue on set_affinity: in case of a change in the address > the previous address is decremented > - doorbell map/unmap now is done on msi composition. Should allow the use > case for platform MSI controllers > - create dma-reserved-iommu.h/c exposing/implementing a new API dedicated > to reserved IOVA management (looking like dma-iommu glue) > - series reordering to ease the review: > - first part is related to IOMMU > - second related to MSI sub-system > - third related to VFIO (except arm-smmu IOMMU_CAP_INTR_REMAP removal) > - expose the number of requested IOVA pages through VFIO_IOMMU_GET_INFO > [this partially addresses Marc's comments on iommu_get/put_single_reserved > size/alignment problematic - which I did not ignore - but I don't know > how much I can do at the moment] > > RFC v2 -> RFC v3: > - should fix wrong handling of some CONFIG combinations: > CONFIG_IOVA, CONFIG_IOMMU_API, CONFIG_PCI_MSI_IRQ_DOMAIN > - fix MSI_FLAG_IRQ_REMAPPING setting in GICv3 ITS (although not tested) > > PATCH v1 -> RFC v2: > - reverted to RFC since it looks more reasonable ;-) the code is split > between VFIO, IOMMU, MSI controller and I am not sure I did the right > choices. Also API need to be further discussed. > - iova API usage in arm-smmu.c. > - MSI controller natively programs the MSI addr with either the PA or IOVA. > This is not done anymore in vfio-pci driver as suggested by Alex. > - check irq remapping capability of the group > > RFC v1 [2] -> PATCH v1: > - use the existing dma map/unmap ioctl interface with a flag to register a > reserved IOVA range. Use the legacy Rb to store this special vfio_dma. > - a single reserved IOVA contiguous region now is allowed > - use of an RB tree indexed by PA to store allocated reserved slots > - use of a vfio_domain iova_domain to manage iova allocation within the > window provided by the userspace > - vfio alloc_map/unmap_free take a vfio_group handle > - vfio_group handle is cached in vfio_pci_device > - add ref counting to bindings > - user modality enabled at the end of the series > > > Eric Auger (8): > iommu: Add iommu_domain_msi_geometry and DOMAIN_ATTR_MSI_GEOMETRY > iommu/arm-smmu: initialize the msi geometry and advertise iommu-msi > support > iommu: introduce an msi cookie > iommu/msi-iommu: initialization > iommu/msi-iommu: iommu_msi_[get,put]_doorbell_iova > iommu/msi-iommu: iommu_msi_domain > iommu/msi-iommu: iommu_msi_msg_pa_to_va > iommu/arm-smmu: get/put the msi cookie > > drivers/iommu/Kconfig | 7 + > drivers/iommu/Makefile | 1 + > drivers/iommu/arm-smmu-v3.c | 18 ++- > drivers/iommu/arm-smmu.c | 18 ++- > drivers/iommu/iommu.c | 5 + > drivers/iommu/msi-iommu.c | 322 ++++++++++++++++++++++++++++++++++++++++++++ > include/linux/iommu.h | 15 +++ > include/linux/msi-iommu.h | 144 ++++++++++++++++++++ > 8 files changed, 522 insertions(+), 8 deletions(-) > create mode 100644 drivers/iommu/msi-iommu.c > create mode 100644 include/linux/msi-iommu.h > > -- > 1.9.1 > > _______________________________________________ > kvmarm mailing list > kvmarm@lists.cs.columbia.edu > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm >