Re: [MAINTAINER SUMMIT] User-space requirements for accelerator drivers

Arnd Bergmann
2021-09-13T13:25:41Z

On Mon, Sep 13, 2021 at 12:51 AM Linus Walleij <linus.walleij@linaro.org> wrote:
> On Sun, Sep 12, 2021 at 11:13 PM Dave Airlie <airlied@gmail.com> wrote:
>
> > For userspace components as well these communities of experts need to
> > exist for each domain, and we need to encourage upstream first
> > processes across the board for these split kernel/userspace stacks.
> >
> > The habanalabs compiler backend is an LLVM fork, I'd like to see the
> > effort to upstream that LLVM backend into LLVM proper.
>
> I couldn't agree more.
>
> A big part of the problem with inference engines / NPU:s is that of no
> standardized userspace. Several of the machine learning initiatives
> from some years back now have stale git repositories and are
> visibly unmaintained, c.f. Caffe https://github.com/BVLC/caffe
> last commit 2 years ago.

Caffe as a standalone project was abandoned and merged into
PyTorch, see https://caffe2.ai/. I think this is the kind of consolidation
of those projects that you are looking for.

> Habanalabs propose an LLVM fork as compiler, yet the Intel
> logo is on the Apache TVM website, and no sign of integrating with
> that project. They claim to support also TensorFlow.
>
> The way I perceive it is that there simply isn't any GCC/LLVM or
> Gallium 3D of NPU:s, these people haven't yet decided that "here
> is that userspace we are all going to use". Or have they?
>
> LLVM? TVM? TensorFlow? PyTorch? Some other one?
>
> What worries me is that I don't see one single developer being
> able to say "this one definitely, and they will work with the kernel
> community", and that is what we need to hear.

I don't actually think this is a decision we can possibly wait for.
The ones you listed all work on different levels, some build on top
of others, and some may get replaced by new ones over time.

For a generic kernel interface, we need something that can be
supported as a back-end for multiple such libraries, and that
works on more than just one hardware. Most likely we will need
both higher-level and lower-level interfaces, so that a
framework (or an application directly) may target one interface,
but some hardware may not be able to implement this.

One straightforward hardware independent low-level API would
be the traditional BLAS GEMM call[1] for matrix multiplication
and its variants (integer, float, bfloat16, ...). Most of the frameworks
are able to use SGEMM to do the actual calculation since that
has optimized versions for most CPUs and GPUs, and most
hardware accelerators should be able to provide an
implementation of this that doesn't completely suck. This
can be used for both inferencing and training.

On the kernel side, this could probably be done inside the
existing crypto (async), media (mem2mem), or gpu/drm
interfaces that all provide ways to offload computational
functions on blocks of memory potentially backed by a dmabuf,
but having a new top-level chardev interface may be a better
fit.

A completely different interface would something that lets you
compile a model into a hardware specific blob in user space
and then submit that blob into the kernel, using further commands
to send and receive model specific data. As I understand it,
this method is roughly what habanalabs and some of the
other ones do for inferencing. The performance is almost
certainly better here, but it requires a high degree of integration
between model, framework, user space driver, compiler and
kernel driver.
We already do similar things in the gpu, fpga and remoteproc
frameworks, all of which could be used here, or we add a more
specialized interface.

What the actual interfaces should be I have no clue, those
two are just examples of what it could be, being completely
ignorant of what drivers do today. As Dave said, this really
needs a maintainer that understands both the kernel side
and what kind of hardware and frameworks exist and
what interfaces both sides actually require.

Arnd

[1] http://www.netlib.org/lapack/explore-html/db/dc9/group__single__blas__level3_gafe51bacb54592ff5de056acabd83c260.html