Re: [PATCH v2 1/5] drivers/accel: Introduce subsystem

Re: [PATCH v2 1/5] drivers/accel: Introduce subsystem

[Andrew Donnellan]

[+ linuxppc-dev, because cxl/ocxl are handled through powerpc - please 
cc on future versions of this series]

On 26/1/19 8:13 am, Olof Johansson wrote:
> We're starting to see more of these kind of devices, the current
> upcoming wave will likely be around machine learning and inference
> engines. A few drivers have been added to drivers/misc for this, but
> it's timely to make it into a separate group of drivers/subsystem, to
> make it easier to find them, and to encourage collaboration between
> contributors.
> 
> Over time, we expect to build shared frameworks that the drivers will
> make use of, but how that framework needs to look like to fill the needs
> is still unclear, and the best way to gain that knowledge is to give the
> disparate implementations a shared location.
> 
> There has been some controversy around expectations for userspace
> stacks being open. The clear preference is to see that happen, and any
> driver and platform stack that is delivered like that will be given
> preferential treatment, and at some point in the future it might
> become the requirement. Until then, the bare minimum we need is an
> open low-level userspace such that the driver and HW interfaces can be
> exercised if someone is modifying the driver, even if the full details
> of the workload are not always available.
> 
> Bootstrapping this with myself and Greg as maintainers (since the current
> drivers will be moving out of drivers/misc). Looking forward to expanding
> that group over time.
> 

[snip]

> +
> +Hardware offload accelerator subsystem
> +======================================
> +
> +This is a brief overview of the subsystem (grouping) of hardware
> +accelerators kept under drivers/accel
> +
> +Types of hardware supported
> +---------------------------
> +
> +  The general types of hardware supported are hardware devices that has
> +  general interactions of sending commands and buffers to the hardware,
> +  returning completions and possible filled buffers back, together
> +  with the usual driver pieces around hardware control, setup, error
> +  handling, etc.
> +
> +  Drivers that fit into other subsystems are expected to be merged
> +  there, and use the appropriate userspace interfaces of said functional
> +  areas. We don't expect to see drivers for network, storage, graphics
> +  and similar hardware implemented by drivers here.
> +
> +Expectations for contributions
> +------------------------------
> +
> + - Platforms and hardware that has fully open stacks, from Firmware to
> +   Userspace, are always going to be given preferential treatment. These
> +   platforms give the best insight for behavior and interaction of all
> +   layers, including ability to improve implementation across the stack
> +   over time.
> +
> + - If a platform is partially proprietary, it is still expected that the
> +   portions that interact the driver can be shared in a form that allows
> +   for exercising the hardware/driver and evolution of the interface over
> +   time. This could be separated into a shared library and test/sample
> +   programs, for example.
> +
> + - Over time, there is an expectation to converge drivers over to shared
> +   frameworks and interfaces. Until then, the general rule is that no
> +   more than one driver per vendor will be acceptable. For vendors that
> +   aren't participating in the work towards shared frameworks over time,
> +   we reserve the right to phase out support for the hardware.
How exactly do generic drivers for interconnect protocols, such as 
cxl/ocxl, fit in here?

cxl and ocxl are not drivers for a specific device, they are generic 
drivers which can be used with any device implementing the CAPI or 
OpenCAPI protocol respectively - many of which will be FPGA boards 
flashed with customer-designed accelerator cores for specific workloads, 
some will be accelerators using ASICs or using FPGA images supplied by 
vendors, some will be driven from userspace, others using the cxl/ocxl 
kernel API, etc.

-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan@au1.ibm.com  IBM Australia Limited

Re: [PATCH v2 1/5] drivers/accel: Introduce subsystem

[Frederic Barrat]

Le 27/01/2019 à 05:31, Andrew Donnellan a écrit :
> [+ linuxppc-dev, because cxl/ocxl are handled through powerpc - please 
> cc on future versions of this series]
> 
> On 26/1/19 8:13 am, Olof Johansson wrote:
>> We're starting to see more of these kind of devices, the current
>> upcoming wave will likely be around machine learning and inference
>> engines. A few drivers have been added to drivers/misc for this, but
>> it's timely to make it into a separate group of drivers/subsystem, to
>> make it easier to find them, and to encourage collaboration between
>> contributors.
>>
>> Over time, we expect to build shared frameworks that the drivers will
>> make use of, but how that framework needs to look like to fill the needs
>> is still unclear, and the best way to gain that knowledge is to give the
>> disparate implementations a shared location.
>>
>> There has been some controversy around expectations for userspace
>> stacks being open. The clear preference is to see that happen, and any
>> driver and platform stack that is delivered like that will be given
>> preferential treatment, and at some point in the future it might
>> become the requirement. Until then, the bare minimum we need is an
>> open low-level userspace such that the driver and HW interfaces can be
>> exercised if someone is modifying the driver, even if the full details
>> of the workload are not always available.
>>
>> Bootstrapping this with myself and Greg as maintainers (since the current
>> drivers will be moving out of drivers/misc). Looking forward to expanding
>> that group over time.
>>
> 
> [snip]
> 
>> +
>> +Hardware offload accelerator subsystem
>> +======================================
>> +
>> +This is a brief overview of the subsystem (grouping) of hardware
>> +accelerators kept under drivers/accel
>> +
>> +Types of hardware supported
>> +---------------------------
>> +
>> +  The general types of hardware supported are hardware devices that has
>> +  general interactions of sending commands and buffers to the hardware,
>> +  returning completions and possible filled buffers back, together
>> +  with the usual driver pieces around hardware control, setup, error
>> +  handling, etc.
>> +
>> +  Drivers that fit into other subsystems are expected to be merged
>> +  there, and use the appropriate userspace interfaces of said functional
>> +  areas. We don't expect to see drivers for network, storage, graphics
>> +  and similar hardware implemented by drivers here.
>> +
>> +Expectations for contributions
>> +------------------------------
>> +
>> + - Platforms and hardware that has fully open stacks, from Firmware to
>> +   Userspace, are always going to be given preferential treatment. These
>> +   platforms give the best insight for behavior and interaction of all
>> +   layers, including ability to improve implementation across the stack
>> +   over time.
>> +
>> + - If a platform is partially proprietary, it is still expected that the
>> +   portions that interact the driver can be shared in a form that allows
>> +   for exercising the hardware/driver and evolution of the interface 
>> over
>> +   time. This could be separated into a shared library and test/sample
>> +   programs, for example.
>> +
>> + - Over time, there is an expectation to converge drivers over to shared
>> +   frameworks and interfaces. Until then, the general rule is that no
>> +   more than one driver per vendor will be acceptable. For vendors that
>> +   aren't participating in the work towards shared frameworks over time,
>> +   we reserve the right to phase out support for the hardware.
> How exactly do generic drivers for interconnect protocols, such as 
> cxl/ocxl, fit in here?
> 
> cxl and ocxl are not drivers for a specific device, they are generic 
> drivers which can be used with any device implementing the CAPI or 
> OpenCAPI protocol respectively - many of which will be FPGA boards 
> flashed with customer-designed accelerator cores for specific workloads, 
> some will be accelerators using ASICs or using FPGA images supplied by 
> vendors, some will be driven from userspace, others using the cxl/ocxl 
> kernel API, etc.


I have the same reservation as Andrew. While my first reaction was to 
think that cxl and ocxl should be part of the accel subsystem, they 
hardly seem to fit the stated goals.
Furthermore, there are implications there, as all the distros currently 
shipping cxl and ocxl as modules on powerpc would need to have their 
config modified to enable CONFIG_ACCEL.

   Fred