Re: [PATCH] drm: fourcc byteorder: brings header file comments in line with reality.

[Gerd Hoffmann <kraxel@redhat.com>]

  Hi,

> > I guess that makes changing drm_mode_legacy_fb_format + drivers a
> > reasonable option ...
> 
> Yeah, I came to the same conclusion after chatting with some
> folks on irc.
> 
> So my current idea is that we change any driver that wants to follow the
> CPU endianness to declare support for big endian formats if the CPU is
> big endian. Presumably these are mostly the virtual GPU drivers.

Agree.  Easy.

> Additonally we'll make the mapping performed by drm_mode_legacy_fb_format()
> driver controlled.

I don't think this is useful.  IMO drm_mode_legacy_fb_format should
return host endian formats unconditionally.

> That way drivers that got changed to follow CPU
> endianness can return a framebuffer that matches CPU endianness. And
> drivers that expect the GPU endianness to not depend on the CPU
> endianness will keep working as they do now. The downside is that users
> of the legacy addfb ioctl will need to magically know which endianness
> they will get, but that is apparently already the case.

Existing userspace expects host endian, and IMO we should maintain that
behavior.

> And users of
> addfb2 will keep on specifying the endianness explicitly with
> DRM_FORMAT_BIG_ENDIAN vs. 0.

I'd drop DRM_FORMAT_BIG_ENDIAN.

At least for the virt drivers it doesn't buy us anything.  They support
32bpp / 8 bpc formats only[1], and for those I can specify the
byteswapped format version without a bigendian flag because we have
fourccs for everything we need.


There is a WIP patch series at
https://www.kraxel.org/cgit/linux/log/?h=drm-byteorder

Needs more testing and better commit messages.  /me plans to polish &
post next week, but feel free to look and comment.

cheers,
  Gerd

[1] Everything else is a PITA to deal with on the host side because
    I can't offload that to pixman.  There is no support for
    PIXMAN_r5g6b5 or PIXMAN_x2b10g10r10 in non-host byte order.