Re: [Ksummit-discuss] [CORE TOPIC] Core Kernel support for Compute-Offload Devices

[Rik van Riel <riel@redhat.com>]

On 07/30/2015 09:00 AM, Joerg Roedel wrote:

> 	(1) Do we need the concept of an off-CPU task in the kernel
> 	    together with a common interface to create and manage them
> 	    and probably a (collection of) batch scheduler(s) for these
> 	    tasks?

Given that some of these compute offload devices share the
same address space (mm_struct) as the threads running on
CPUs, it would be easiest if there was a reference on the
mm_struct for the threads that are running off-CPU.

I do not know if a generic scheduler would work, since
it is common to have N threads on compute devices all bound
to the same address space, etc.

Different devices might even require different schedulers,
but having a common data structure that pins mm_struct,
provides for a place to have state (like register content)
stored, and has pointers to scheduler, driver, and cleanup
functions could be really useful.

> 	(2) Changes in memory management for devices accessing user
> 	    address spaces:
> 	    
> 	    (2.1) How can we best support the different memory models
> 	          these devices support?
> 	    
> 	    (2.2) How do we handle the off-CPU users of an mm_struct?
> 	    
> 	    (2.3) How can we attach common state for off-CPU tasks to
> 	          mm_struct (and what needs to be in there)?

Jerome has a bunch of code for this already.

> 	(3) Does it make sense to implement automatic migration of
> 	    system memory to device memory (when available) and vice
> 	    versa? How do we decide what and when to migrate?

I believe he has looked at migration too, but not implemented
it yet.

If compute-offload devices are a kernel summit topic this year,
it would be useful to invite Jerome Glisse.

> 	(4) What features do we require in the hardware to support it
> 	    with a common interface?
> 
> I think it would be great if the kernel would have a common interface
> for these kind of devices. Currently every vendor develops its own
> interface with various hacks to work around core code behavior.
> 
> I am particularily interested in this topic because on PCIe newer IOMMUs
> are often an integral part in supporting these devices (ARM-SMMUv3,
> Intel VT-d with SVM, AMD IOMMUv2). so that core work here will also
> touch the IOMMU code.
> 
> Probably (uncomplete list of) interested people:
> 
> 	David Woodhouse
> 	Jesse Barnes
> 	Will Deacon
> 	Paul E. McKenney
> 	Rik van Riel
> 	Mel Gorman
> 	Andrea Arcangeli
> 	Christoph Lameter
> 	Jérôme Glisse

-- 
All rights reversed