Re: [PATCH/RFC 0/5] HW accel subsystem

[Kenneth Lee <liguozhu@hisilicon.com>]

On Fri, Jan 25, 2019 at 10:16:11AM -0800, Olof Johansson wrote:
> Date: Fri, 25 Jan 2019 10:16:11 -0800
> From: Olof Johansson <olof@lixom.net>
> To: linux-kernel@vger.kernel.org
> CC: ogabbay@habana.ai, Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
>  jglisse@redhat.com, Andrew Donnellan <andrew.donnellan@au1.ibm.com>,
>  Frederic Barrat <fbarrat@linux.ibm.com>, airlied@redhat.com,
>  linux-accelerators@lists.ozlabs.org
> Subject: [PATCH/RFC 0/5] HW accel subsystem
> X-Mailer: git-send-email 2.11.0
> Message-ID: <20190125181616.62609-1-olof@lixom.net>
> 
> Per discussion in on the Habana Labs driver submission
> (https://lore.kernel.org/lkml/20190123000057.31477-1-oded.gabbay@gmail.com/),
> there seems to be time to create a separate subsystem for hw accellerators
> instead of letting them proliferate around the tree (and/or in misc).
> 
> There's difference in opinion on how stringent the requirements are for
> a fully open stack for these kind of drivers. I've documented the middle
> road approach in the first patch (requiring some sort of open low-level
> userspace for the kernel interaction, and a way to use/test it).
> 
> Comments and suggestions for better approaches are definitely welcome.

Dear Olof,

How are you? Let me introduce myself. My name is Kenenth Lee, working for
Hisilicon. Our company provide server, AI, networking and terminal SoCs to the
market. We tried to create an accelerator framework a year back and now we are
working on the branch here (There is document in Documentation/warpdrive
directory):

https://github.com/Kenneth-Lee/linux-kernel-warpdrive/tree/wdprd-v1

The user space framework is here:

https://github.com/Kenneth-Lee/warpdrive/tree/wdprd-v1

We have tried to create it on VFIO at the very beginning. The RFCv1 is here:

https://lwn.net/Articles/763990/

But it seems it is not fit. There are two major issues:

1. The VFIO framework enforces the concept of separating the resource into
   devices before using it. This is not an accelerator style. Accelerator is
   another CPU to let the others to share it.
2. The way VFIO used to pin memory in place, has some flaw. In the current
   kernel, if you fork a sub-rpcess after pin the dma memory, you may lost the
   physical pages. (You can get more detail in the threads)

So we tried RFCv2 and build the solution directly on IOMMU. We call our solution
as WarpDrive and the kernel module is called uacce. Our assumption is that:

1. Most of users of the accelerator are in user space.
2. An accelerator is always another heterogeneous processor. It is waiting and
   processing work load sent from CPU.
3. The data structure in the CPU may be complex. It is no good to wrap the data
   and send it to hardware again and again. The better way is to keep the data
   in place and give a pointer to the accelerator, leaving it to finish the job.

So we create a pipe (we called it queue) between the user process and the
hardware directly. It is presented as a file to the user space. The user process
mmap the queue file to address the mmio space of the hardware, share memory and 
so on. With the capability of IOMMU, we can share the whole or part of process
space with the hardware. This can make the software solution easier.

After the RFCv2 was sent to the lkml, we do not get much feedback. But the
Infini-band guys said they did not like it. They think the solution is
re-invention of ib-verbs.

But we do not think so. ib-verbs maintains semantics of "REMOTE memory". But
UACCE maintains semantics of "LOCAL memory". We don't need to send, or sync
memory with other parties. We share those memory with all processes who share
the local bus.

But we know we need more "complete" solution to let people understand and accept
our idea. So now we are working on it with our Compression and RSA accelerator
on Hi1620 Server SoC. We are also planning to port our AI framework on it.

Do you think we can cooperate to create an framework in Linux together? Please
feel free to ask for more information. We are happy to answer it.


Cheers
-- 
			-Kenneth(Hisilicon)