Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

Laurent Pinchart
2024-08-13T10:27:05Z

Hi Tomasz,

On Tue, Aug 13, 2024 at 07:17:07PM +0900, Tomasz Figa wrote:
> 2024年7月31日(水) 22:16 Daniel Vetter <daniel.vetter@ffwll.ch>:
> >
> > On Wed, 31 Jul 2024 at 13:55, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > > This is also very different from GPUs or accel devices that are built to be
> > > user-programmable. If I'd compare ISPs to different devices, then the
> > > closest match would probably be video codecs -- which also use V4L2.
> >
> > Really just aside, but I figured I should correct this. DRM supports
> > plenty of video codecs. They're all tied to gpus, but the real reason
> > really is that the hw has decent command submission support so that
> > running the entire codec in userspace except the basic memory and
> > batch execution and synchronization handling in the kernel is a
> > feasible design.
>
> FWIW, V4L2 also has an interface for video decoders that require
> bitstream processing in software, it's called the V4L2 Stateless
> Decoder interface [1]. It defines low level data structures that map
> directly to the particular codec specification, so the kernel
> interface is generic and the userspace doesn't need to have
> hardware-specific components. Hardware that consumes command buffers
> can be supported simply by having the kernel driver fill the command
> buffers as needed (as opposed to writing the registers directly).
> On the other hand, DRM also has the fixed function (i.e. V4L2-alike)
> KMS interface for display controllers, rather than a command buffer
> passthrough, even though some display controllers actually are driven
> by command buffers.
> So arguably it's possible and practical to do both command
> buffer-based and fixed interfaces for both display controllers and
> video codecs. Do you happen to know some background behind why one or
> the other was chosen for each of them in DRM?
>
> For how it applies to ISPs, there are both types of ISPs out in the
> wild, some support command buffers, while some are programmed directly
> via registers.

Could you provide examples of ISPs that use command buffers ? The
discussion has remained fairly vague so far, which I think hinders
progress.

> For the former, I can see some loss of flexibility if
> the command buffers are hidden behind a fixed function API, because
> the userspace would only be able to do what the kernel driver supports
> internally, which could make some use case-specific optimizations very
> challenging if not impossible.

Let's try to discuss this with specific examples.

> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-stateless-decoder.html
>
> > And actually good, because your kernel wont ever blow
> > up trying to parse complex media formats because it just doesn't.

--
Regards,

Laurent Pinchart