From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerin Jacob Subject: Re: [RFC][PATCH 2/5] pci: allow shared device instances. Date: Wed, 10 May 2017 17:48:10 +0530 Message-ID: <20170510121809.GA25209@jerin> References: <1494341879-18718-1-git-send-email-radu.nicolau@intel.com> <1571308.M4Vgp8n44D@xps> <18252037.5gudFzihFo@xps> <1c20ee02-530f-a39a-47c5-15cc5ce1a973@intel.com> <20170510110843.GA22712@jerin> <73b94d12-f1cf-295f-284f-0ee1bd5f0d15@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Thomas Monjalon , Radu Nicolau , dev@dpdk.org To: Declan Doherty Return-path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0062.outbound.protection.outlook.com [104.47.34.62]) by dpdk.org (Postfix) with ESMTP id 3E12D234 for ; Wed, 10 May 2017 14:18:38 +0200 (CEST) Content-Disposition: inline In-Reply-To: <73b94d12-f1cf-295f-284f-0ee1bd5f0d15@intel.com> List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" -----Original Message----- > Date: Wed, 10 May 2017 12:31:48 +0100 > From: Declan Doherty > To: Jerin Jacob > CC: Thomas Monjalon , Radu Nicolau > , dev@dpdk.org > Subject: Re: [dpdk-dev] [RFC][PATCH 2/5] pci: allow shared device instances. > User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 > Thunderbird/45.8.0 > > On 10/05/2017 12:08 PM, Jerin Jacob wrote: > > -----Original Message----- > > > Date: Wed, 10 May 2017 11:52:45 +0100 > > > From: Declan Doherty > > > To: Thomas Monjalon , Radu Nicolau > > > > > > CC: dev@dpdk.org > > > Subject: Re: [dpdk-dev] [RFC][PATCH 2/5] pci: allow shared device instances. > > > User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 > > > Thunderbird/45.8.0 > > > > > > Hey Thomas, I've been working on this with Radu, so see my take below > > > > > > On 10/05/2017 11:28 AM, Thomas Monjalon wrote: > > > > 10/05/2017 12:11, Radu Nicolau: > > > > > Hi > > > > > > > > > > > > > > > On 5/10/2017 10:09 AM, Thomas Monjalon wrote: > > > > > > Hi, > > > > > > > > > > > > 09/05/2017 16:57, Radu Nicolau: > > > > > > > Updated PCI initialization code to allow devices to be shared across multiple PMDs. > > > > > > > > > > > > > > Signed-off-by: Radu Nicolau > > > > > > I was waiting the day when we have a device shared > > > > > > by two different interfaces. > > > > > > Note that some Mellanox and Chelsio devices already instantiate > > > > > > two ethdev ports per PCI device. > > > > > > > > > > > > Please explain your idea behind this "shared" flag. > > > > > > What is your exact need? > > > > > > > > > > Currently for each pci device a look-up into a list of PMDs is > > > > > performed, and when a match is found the system moves to the next > > > > > device. Having this flag will allow a PMD to inform the system that > > > > > there may be more matches, more PMDs that can be used for this > > > > > particular device. > > > > > There is a difference when comparing to the devices you mentioned above, > > > > > in this case the PMDs are totally different types, one network and one > > > > > cryptodev PMD for each IXGBE network card. > > > > > > > > Yes I know it is a lack in DPDK. > > > > Linux introduced MultiFunction Device in 2005: > > > > http://events.linuxfoundation.org/sites/events/files/slides/belloni-mfd-regmap-syscon_0.pdf > > > > > > > > > > So at the most basic level the intention is to allow more than one device of > > > different types, in our case a net PMD and a crypto PMD, to be instantiated > > > on a single PCI bar, in essence to share the bar. I'm not familiar with the > > > approaches taken in the Mellanox and Chelsio devices but I assume they are > > > handled with the driver probe/create functions independently from the EAL > > > infrastructure? > > > > > > For the initial proto-typing of this RFC we only implemented the > > > multi-device creation but I envisage that there will be a requirement for > > > sharing state between drivers, or at a minimum implementing locking around > > > shared resources, registers etc. And I would like to see this done in a > > > generic fashion that can me leverage by any driver and not have each driver > > > having to solve this independently. > > > > Cavium's next generation PCI based NW devices has similar scheme where we > > need to share the same BAR with multiple DPDK subsystems(ethdev, > > eventdev etc) unlike current generation(OcteonTX). > > > > Have you done investigation into how you would like to support this, and are > you trending to any particular approach. The rte_bus approach as you outline > below does sound like it would suit this multi-function device. Not much investigation has been done as its for next generation. It is no PCIe multi function device. There will be a lot shared functions between these shared DPDK devices and there should be place holder for this in code.I thought driver/bus/foo may a option. In additional to this, If we expose new function pointer based interfaces in bus for the shared device register access and other shared resource alloc/free between these two DPDK devices, it can be centralized to one place(driver/bus/foo) and generalized. Just 2c. We haven't done any prototype. > > > I think, Another possible way to handle this in generic way is to: > > Register a new rte_bus for the shared PCI access which sits on top PCIe bus. > > With new bus's scan and probe scheme, it can probe the two devices. > > > > > > Yes, this would work and I think it makes a lot of sense in the case where > you have logically independent hardware functional blocks on a shared bus. > In our particular case, we only have a single physical device, which we are > presenting as 2 logical devices purely to improve the sw model through DPDK > existing infrastructure. We may also need to implement some shared context > for protecting access to shared resources such as register and to > synchronized exposure of capabilities. In the case of the IXBGE family of > devices they can support MACsec or IPsec functionality but not both at the > same time, so some mechanism of passing this state between the net and > crypto PMDs will be required. I guess it should be possible to do this > through the bus model as well but we'll need to have another look, although > my initial feeling is they are slightly different problems.