[RFC] new uapi policy for drm

[RFC] new uapi policy for drm

[Dave Airlie]

I've kicked this around in my head over the past few weeks but wanted
to get some feedback on whether it's a good idea or what impact it
might have that I haven't considered.

We are getting requests via both amdgpu/amdkfd and i915 for new user
APIs for userspace drivers that throw code over the wall instead of
being open source developed projects, but we are also seeing it for
android drivers and kms properties, and we had that i915 crappy crtc
background thing that was for Chrome but Chrome didn't want it.

Now this presents a couple of issues:

a) these projects don't seem to that good at following our development
guidelines, avoid developing userspace features in parallel in the
open and having good development implementations before submitting
upstream.

b) these projects don't have experienced userspace developers
reviewing their kernel uapis. One big advantage of adding uapis with
mesa developers is they have a lot of experience in the area as well.

It's leading me to think I want to just stop all uapi submissions via
driver trees, and instead mandate that all driver uapi changes are
sent in separate git pull requests to dri-devel, I'd try (with some
help) to catch all uapi modifications in normal trees, and refuse
pulls that modified uapi.

At least I'm considered writing the script and refusing and pulls that
have a uapi change that doesn't contain a link to the userspace
changes required for it in a public developed repo.

Thoughts?

Dave.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Rodrigo Vivi]

On Tue, Oct 15, 2019 at 04:16:00AM +1000, Dave Airlie wrote:
> I've kicked this around in my head over the past few weeks but wanted
> to get some feedback on whether it's a good idea or what impact it
> might have that I haven't considered.
> 
> We are getting requests via both amdgpu/amdkfd and i915 for new user
> APIs for userspace drivers that throw code over the wall instead of
> being open source developed projects, but we are also seeing it for
> android drivers and kms properties, and we had that i915 crappy crtc
> background thing that was for Chrome but Chrome didn't want it.
> 
> Now this presents a couple of issues:
> 
> a) these projects don't seem to that good at following our development
> guidelines, avoid developing userspace features in parallel in the
> open and having good development implementations before submitting
> upstream.
> 
> b) these projects don't have experienced userspace developers
> reviewing their kernel uapis. One big advantage of adding uapis with
> mesa developers is they have a lot of experience in the area as well.
> 
> It's leading me to think I want to just stop all uapi submissions via
> driver trees, and instead mandate that all driver uapi changes are
> sent in separate git pull requests to dri-devel, I'd try (with some
> help) to catch all uapi modifications in normal trees, and refuse
> pulls that modified uapi.

I truly see your reass and a separated pull request would even
give more visibility to the UAPI changes for everyone. My only concern
would be the flow of merging this on different repositories, etc...

So I'd prefer if we could keep on the simplest side.

> 
> At least I'm considered writing the script and refusing and pulls that
> have a uapi change that doesn't contain a link to the userspace
> changes required for it in a public developed repo.

This is a great idea.

Probably better if we could enforce that on "dim" so we couldn't even
merge a uapi without a link.

Would you consider a different tag for that:

UAPI: https://gitlab.../code.c

"Reference:" should be enough, but that could very easily bypass any script
and a new tag would make the changes even more visible in a way that
the separate pull request wouldn't be needed.

> 
> Thoughts?
> 
> Dave.
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Rodrigo Vivi]

On Tue, Oct 15, 2019 at 01:55:41PM -0700, Rodrigo Vivi wrote:
> On Tue, Oct 15, 2019 at 04:16:00AM +1000, Dave Airlie wrote:
> > I've kicked this around in my head over the past few weeks but wanted
> > to get some feedback on whether it's a good idea or what impact it
> > might have that I haven't considered.
> > 
> > We are getting requests via both amdgpu/amdkfd and i915 for new user
> > APIs for userspace drivers that throw code over the wall instead of
> > being open source developed projects, but we are also seeing it for
> > android drivers and kms properties, and we had that i915 crappy crtc
> > background thing that was for Chrome but Chrome didn't want it.
> > 
> > Now this presents a couple of issues:
> > 
> > a) these projects don't seem to that good at following our development
> > guidelines, avoid developing userspace features in parallel in the
> > open and having good development implementations before submitting
> > upstream.
> > 
> > b) these projects don't have experienced userspace developers
> > reviewing their kernel uapis. One big advantage of adding uapis with
> > mesa developers is they have a lot of experience in the area as well.
> > 
> > It's leading me to think I want to just stop all uapi submissions via
> > driver trees, and instead mandate that all driver uapi changes are
> > sent in separate git pull requests to dri-devel, I'd try (with some
> > help) to catch all uapi modifications in normal trees, and refuse
> > pulls that modified uapi.
> 
> I truly see your reass

I truly see your reasons....

(and I can't even blame an auto-corrector... sorry)

> and a separated pull request would even
> give more visibility to the UAPI changes for everyone. My only concern
> would be the flow of merging this on different repositories, etc...
> 
> So I'd prefer if we could keep on the simplest side.
> 
> > 
> > At least I'm considered writing the script and refusing and pulls that
> > have a uapi change that doesn't contain a link to the userspace
> > changes required for it in a public developed repo.
> 
> This is a great idea.
> 
> Probably better if we could enforce that on "dim" so we couldn't even
> merge a uapi without a link.
> 
> Would you consider a different tag for that:
> 
> UAPI: https://gitlab.../code.c
> 
> "Reference:" should be enough, but that could very easily bypass any script
> and a new tag would make the changes even more visible in a way that
> the separate pull request wouldn't be needed.
> 
> > 
> > Thoughts?
> > 
> > Dave.
> > _______________________________________________
> > dri-devel mailing list
> > dri-devel@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/dri-devel
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Matt Roper]

On Tue, Oct 15, 2019 at 04:16:00AM +1000, Dave Airlie wrote:
> I've kicked this around in my head over the past few weeks but wanted
> to get some feedback on whether it's a good idea or what impact it
> might have that I haven't considered.
> 
> We are getting requests via both amdgpu/amdkfd and i915 for new user
> APIs for userspace drivers that throw code over the wall instead of
> being open source developed projects, but we are also seeing it for
> android drivers and kms properties, and we had that i915 crappy crtc
> background thing that was for Chrome but Chrome didn't want it.

Having submitted "that i915 crappy crtc background thing" it's unclear
to me what you're trying to say here.  If you're trying to claim that
this was "thrown over the wall" or that there was an attempt to try to
sneak this work in under the radar and circumvent uapi policy then you're
way off track and you probably need to review the (long) history of this
feature.  We've very religiously followed both the spirit and the letter
of your upstream uapi policy with regards to that feature and taken great
care to ensure it would *not* land upstream without an actual opensource
userspace consumer.  There was some confusion recently about how
ChromeOS was/wasn't using this interface, but the Google guys helped
clear that up, and the kernel changes have now returned to their holding
pattern and will not go upstream unless new userspace emerges.  To
refresh your memory on the history:

 * The proposed interface was originally posted in 2015/2016 as an FYI
   to find out if any opensource userspace projects might have a use for
   this functionality (we already had the patches written since there's
   non-opensource software that requires it).  The cover letters at the
   time clearly stated that it should not be merged upstream since there
   was no userspace:

        "Note that this series isn't mergeable yet since we don't (yet)
        have an open source userspace that can make use of it"
        (https://lists.freedesktop.org/archives/intel-gfx/2016-February/087382.html)

 * About 2.5 years later there was suddenly interest expressed both from
   people involved in Weston and people involved in ChromeOS, and they
   independently asked for refreshed kernel patches.  When the refreshed
   patches were posted  they again included a disclaimer that we needed
   proper userspace before it could land:

        "As always, we'll still need the patches for at least one of
        those projects to get posted (and reviewed) somewhere public
        before we actually merge these kernel patches."
        (https://lists.freedesktop.org/archives/intel-gfx/2018-October/178202.html)

 * The Weston effort apparently fizzled out, but at the beginning of
   2019 we were directed to some patches to the ChromeOS ozone layer
   (https://chromium-review.googlesource.com/c/chromium/src/+/1278858)
   that utilized the interface.  I'm not at all familiar with the
   ChromeOS userspace stack, but since the Ozone code was reviewed by
   non-Intel folks and shows as 'merged' that sounded like a legitimate
   open source userspace consumer.  From that point on the Chromium
   review links were clearly advertised in the cover letter each time
   the series was respun so that it would be very clear what the
   expected userspace was and so any concerns could be raised.

 * Daniele and Sean from Google recently brought it to our attention
   that even though that ozone code made use of the background color
   property, the ozone interfaces themselves don't get used at higher
   levels of the ChromeOS stack (and had probably been merged to the
   ozone layer in error).  Once they helped us understand that the
   properties wouldn't ultimately be utilized in a meaningful way with a
   real ChromeOS setup today, we put a halt to the upstreaming effort:

        "Hmm, in that case it sounds like we probably don't actually
        have enough userspace support yet to justify merging the kernel
        changes at this time."
        (https://lists.freedesktop.org/archives/dri-devel/2019-October/239436.html)

So long story short, we've been very transparent about what's going on
with this feature for the whole ~4 years it's been floating around.  I
would hope that you, as DRM maintainer, would have spoken up at some
point if you felt there was something else that needed to be handled
differently.  No code for this api has ever landed in any of the driver
trees, so I'm not sure why you called out this work in an email
proposing a change to uapi pull requests.

> Now this presents a couple of issues:
> 
> a) these projects don't seem to that good at following our development
> guidelines, avoid developing userspace features in parallel in the
> open and having good development implementations before submitting
> upstream.
> 
> b) these projects don't have experienced userspace developers
> reviewing their kernel uapis. One big advantage of adding uapis with
> mesa developers is they have a lot of experience in the area as well.
> 
> It's leading me to think I want to just stop all uapi submissions via
> driver trees, and instead mandate that all driver uapi changes are
> sent in separate git pull requests to dri-devel, I'd try (with some
> help) to catch all uapi modifications in normal trees, and refuse
> pulls that modified uapi.
> 
> At least I'm considered writing the script and refusing and pulls that
> have a uapi change that doesn't contain a link to the userspace
> changes required for it in a public developed repo.

FWIW, requiring a link to the corresponding userspace in the kernel
patch itself and not just the series cover letter seems like a
reasonable request to me.


Matt

> 
> Thoughts?
> 
> Dave.
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

-- 
Matt Roper
Graphics Software Engineer
VTT-OSGC Platform Enablement
Intel Corporation
(916) 356-2795
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Stéphane Marchesin]

On Wed, Oct 16, 2019 at 10:28 AM Matt Roper <matthew.d.roper@intel.com> wrote:
>
> On Tue, Oct 15, 2019 at 04:16:00AM +1000, Dave Airlie wrote:
> > I've kicked this around in my head over the past few weeks but wanted
> > to get some feedback on whether it's a good idea or what impact it
> > might have that I haven't considered.
> >
> > We are getting requests via both amdgpu/amdkfd and i915 for new user
> > APIs for userspace drivers that throw code over the wall instead of
> > being open source developed projects, but we are also seeing it for
> > android drivers and kms properties, and we had that i915 crappy crtc
> > background thing that was for Chrome but Chrome didn't want it.
>
> Having submitted "that i915 crappy crtc background thing" it's unclear
> to me what you're trying to say here.  If you're trying to claim that
> this was "thrown over the wall" or that there was an attempt to try to
> sneak this work in under the radar and circumvent uapi policy then you're
> way off track and you probably need to review the (long) history of this
> feature.  We've very religiously followed both the spirit and the letter
> of your upstream uapi policy with regards to that feature and taken great
> care to ensure it would *not* land upstream without an actual opensource
> userspace consumer.  There was some confusion recently about how
> ChromeOS was/wasn't using this interface, but the Google guys helped
> clear that up, and the kernel changes have now returned to their holding
> pattern and will not go upstream unless new userspace emerges.  To
> refresh your memory on the history:
>
>  * The proposed interface was originally posted in 2015/2016 as an FYI
>    to find out if any opensource userspace projects might have a use for
>    this functionality (we already had the patches written since there's
>    non-opensource software that requires it).  The cover letters at the
>    time clearly stated that it should not be merged upstream since there
>    was no userspace:
>
>         "Note that this series isn't mergeable yet since we don't (yet)
>         have an open source userspace that can make use of it"
>         (https://lists.freedesktop.org/archives/intel-gfx/2016-February/087382.html)
>
>  * About 2.5 years later there was suddenly interest expressed both from
>    people involved in Weston and people involved in ChromeOS, and they
>    independently asked for refreshed kernel patches.  When the refreshed
>    patches were posted  they again included a disclaimer that we needed
>    proper userspace before it could land:
>
>         "As always, we'll still need the patches for at least one of
>         those projects to get posted (and reviewed) somewhere public
>         before we actually merge these kernel patches."
>         (https://lists.freedesktop.org/archives/intel-gfx/2018-October/178202.html)
>
>  * The Weston effort apparently fizzled out, but at the beginning of
>    2019 we were directed to some patches to the ChromeOS ozone layer
>    (https://chromium-review.googlesource.com/c/chromium/src/+/1278858)
>    that utilized the interface.  I'm not at all familiar with the
>    ChromeOS userspace stack, but since the Ozone code was reviewed by
>    non-Intel folks and shows as 'merged' that sounded like a legitimate
>    open source userspace consumer.  From that point on the Chromium
>    review links were clearly advertised in the cover letter each time
>    the series was respun so that it would be very clear what the
>    expected userspace was and so any concerns could be raised.
>
>  * Daniele and Sean from Google recently brought it to our attention
>    that even though that ozone code made use of the background color
>    property, the ozone interfaces themselves don't get used at higher
>    levels of the ChromeOS stack (and had probably been merged to the
>    ozone layer in error).


Yes I think we (at Google) more or less all assumed that someone else
was going to consume the ozone API (at least that's what I did). And
because everyone assumed someone else might make use of it, that code
ended up landing. I am sorry about that; we'll try to be more careful
in the future. As you say the ozone code doesn't provide any feature
in itself, it's just plumbing. Obviously a feature on top didn't
materialize.

The exact process to fix this isn't really relevant to me, but I think
maybe we should have more open discussions when new features like this
happen. Ask ourselves "Who will make use of this feature?" and check
with that person seems like a good start. At least then, we have a
contact point to ask further questions. Maybe through that process we
would find that this is just ozone code, and there is no consumer on
top. Or we can dig into exactly what's needed to allow shaping the
APIs.

Best,
Stéphane


>
> Once they helped us understand that the
>    properties wouldn't ultimately be utilized in a meaningful way with a
>    real ChromeOS setup today, we put a halt to the upstreaming effort:
>
>         "Hmm, in that case it sounds like we probably don't actually
>         have enough userspace support yet to justify merging the kernel
>         changes at this time."
>         (https://lists.freedesktop.org/archives/dri-devel/2019-October/239436.html)
>
> So long story short, we've been very transparent about what's going on
> with this feature for the whole ~4 years it's been floating around.  I
> would hope that you, as DRM maintainer, would have spoken up at some
> point if you felt there was something else that needed to be handled
> differently.  No code for this api has ever landed in any of the driver
> trees, so I'm not sure why you called out this work in an email
> proposing a change to uapi pull requests.
>
> > Now this presents a couple of issues:
> >
> > a) these projects don't seem to that good at following our development
> > guidelines, avoid developing userspace features in parallel in the
> > open and having good development implementations before submitting
> > upstream.
> >
> > b) these projects don't have experienced userspace developers
> > reviewing their kernel uapis. One big advantage of adding uapis with
> > mesa developers is they have a lot of experience in the area as well.
> >
> > It's leading me to think I want to just stop all uapi submissions via
> > driver trees, and instead mandate that all driver uapi changes are
> > sent in separate git pull requests to dri-devel, I'd try (with some
> > help) to catch all uapi modifications in normal trees, and refuse
> > pulls that modified uapi.
> >
> > At least I'm considered writing the script and refusing and pulls that
> > have a uapi change that doesn't contain a link to the userspace
> > changes required for it in a public developed repo.
>
> FWIW, requiring a link to the corresponding userspace in the kernel
> patch itself and not just the series cover letter seems like a
> reasonable request to me.
>
>
> Matt
>
> >
> > Thoughts?
> >
> > Dave.
> > _______________________________________________
> > dri-devel mailing list
> > dri-devel@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/dri-devel
>
> --
> Matt Roper
> Graphics Software Engineer
> VTT-OSGC Platform Enablement
> Intel Corporation
> (916) 356-2795
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Alex Deucher]

On Mon, Oct 14, 2019 at 2:16 PM Dave Airlie <airlied@gmail.com> wrote:
>
> I've kicked this around in my head over the past few weeks but wanted
> to get some feedback on whether it's a good idea or what impact it
> might have that I haven't considered.
>
> We are getting requests via both amdgpu/amdkfd and i915 for new user
> APIs for userspace drivers that throw code over the wall instead of
> being open source developed projects, but we are also seeing it for
> android drivers and kms properties, and we had that i915 crappy crtc
> background thing that was for Chrome but Chrome didn't want it.
>
> Now this presents a couple of issues:
>
> a) these projects don't seem to that good at following our development
> guidelines, avoid developing userspace features in parallel in the
> open and having good development implementations before submitting
> upstream.
>
> b) these projects don't have experienced userspace developers
> reviewing their kernel uapis. One big advantage of adding uapis with
> mesa developers is they have a lot of experience in the area as well.
>
> It's leading me to think I want to just stop all uapi submissions via
> driver trees, and instead mandate that all driver uapi changes are
> sent in separate git pull requests to dri-devel, I'd try (with some
> help) to catch all uapi modifications in normal trees, and refuse
> pulls that modified uapi.
>
> At least I'm considered writing the script and refusing and pulls that
> have a uapi change that doesn't contain a link to the userspace
> changes required for it in a public developed repo.
>
> Thoughts?

This seems like more hassle for questionable benefits.  I don't know
that mesa is really any better than any other driver teams with
respect to UAPI.  This just seems like sort of a arbitrary political
decision.  The people working on mesa have as much of an agenda as
those working on other projects.  Moreover, especially with the
migration to gitlab and MRs, I feel that mesa development has gotten
more opaque.  Say what you will about mailing lists, but at least you
could have a drive by view of what's going on.  With MRs, you sort of
have to seek out what to review; if stuff is not tagged with something
you feel is relevant, you probably won't look at it, so the only
people likely to review it are the people involved in writing it in
the first place, which would be the same whether it's mesa or some
other project.  I think all of the projects generally have the best
intentions at heart, but for better or worse they just have different
development models.  In the case of the AMD throw it over the wall
stuff, it's not really an anti-open source or community engagement
issue, it's more of how to we support several OSes, tons of new
products, several custom projects, etc. while leveraging as much
shared code as possible.  There are ways to make it work, but they are
usually a pretty heavy lift that not all teams can make.

All of that said, I think providing a link to the userspace user of
the API is reasonable, but I don't think there have been any egregious
cases of badly designed UAPI that were not caught using the existing
processes.

Alex
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Sean Paul]

On Tue, Oct 15, 2019 at 04:16:00AM +1000, Dave Airlie wrote:
> I've kicked this around in my head over the past few weeks but wanted
> to get some feedback on whether it's a good idea or what impact it
> might have that I haven't considered.
> 
> We are getting requests via both amdgpu/amdkfd and i915 for new user
> APIs for userspace drivers that throw code over the wall instead of
> being open source developed projects, but we are also seeing it for
> android drivers and kms properties, and we had that i915 crappy crtc
> background thing that was for Chrome but Chrome didn't want it.

The Chrome issue this past week was really a communication breakdown between
everyone involved. The Chrome patch at first glance looks like it implements the
feature, so this type of thing really just needs to be caught by the left hand
talking to the right (which is what happened when Daniele and I talked at XDC).
At least in our (Chrome's) case, we'll catch this type of thing much earlier
next time and avoid the situation entirely. I think even under your new proposal
we would have been in the same spot (since this patchset is still in review).

> 
> Now this presents a couple of issues:
> 
> a) these projects don't seem to that good at following our development
> guidelines, avoid developing userspace features in parallel in the
> open and having good development implementations before submitting
> upstream.
> 
> b) these projects don't have experienced userspace developers
> reviewing their kernel uapis. One big advantage of adding uapis with
> mesa developers is they have a lot of experience in the area as well.
> 
> It's leading me to think I want to just stop all uapi submissions via
> driver trees, and instead mandate that all driver uapi changes are
> sent in separate git pull requests to dri-devel, I'd try (with some
> help) to catch all uapi modifications in normal trees, and refuse
> pulls that modified uapi.

We did this a separate pull with the initial HDCP support and it went Ok. I
think there was a bit of conflict pain between hdcp topic branch and drm-intel,
but we mostly got along.

That said, I do want to be sensitive to HW vendors trying to upstream their HW
features. We shit on them for keeping everything downstream, but then we shit on
them for missing the bar when they try to upstream. With Android moving to kms,
we're going to have more vendors coming forward with boutique HW features they
are looking to standardize (or maybe they'll just stay downstream -- I'll try to
stay optimistic). Nothing in your proposal suggests that it will be more
difficult for vendors to upstream the right way, just that we'll have more
oversight and a more consistent voice for UAPI. So yeah, I think that's
completely reasonable.

Sean

> 
> At least I'm considered writing the script and refusing and pulls that
> have a uapi change that doesn't contain a link to the userspace
> changes required for it in a public developed repo.
> 
> Thoughts?
> 
> Dave.
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel

-- 
Sean Paul, Software Engineer, Google / Chromium OS
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Dave Airlie]

On Thu, 17 Oct 2019 at 06:00, Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Mon, Oct 14, 2019 at 2:16 PM Dave Airlie <airlied@gmail.com> wrote:
> >
> > I've kicked this around in my head over the past few weeks but wanted
> > to get some feedback on whether it's a good idea or what impact it
> > might have that I haven't considered.
> >
> > We are getting requests via both amdgpu/amdkfd and i915 for new user
> > APIs for userspace drivers that throw code over the wall instead of
> > being open source developed projects, but we are also seeing it for
> > android drivers and kms properties, and we had that i915 crappy crtc
> > background thing that was for Chrome but Chrome didn't want it.
> >
> > Now this presents a couple of issues:
> >
> > a) these projects don't seem to that good at following our development
> > guidelines, avoid developing userspace features in parallel in the
> > open and having good development implementations before submitting
> > upstream.
> >
> > b) these projects don't have experienced userspace developers
> > reviewing their kernel uapis. One big advantage of adding uapis with
> > mesa developers is they have a lot of experience in the area as well.
> >
> > It's leading me to think I want to just stop all uapi submissions via
> > driver trees, and instead mandate that all driver uapi changes are
> > sent in separate git pull requests to dri-devel, I'd try (with some
> > help) to catch all uapi modifications in normal trees, and refuse
> > pulls that modified uapi.
> >
> > At least I'm considered writing the script and refusing and pulls that
> > have a uapi change that doesn't contain a link to the userspace
> > changes required for it in a public developed repo.
> >
> > Thoughts?
>
> This seems like more hassle for questionable benefits.  I don't know
> that mesa is really any better than any other driver teams with
> respect to UAPI.  This just seems like sort of a arbitrary political
> decision.  The people working on mesa have as much of an agenda as
> those working on other projects.  Moreover, especially with the
> migration to gitlab and MRs, I feel that mesa development has gotten
> more opaque.  Say what you will about mailing lists, but at least you
> could have a drive by view of what's going on.  With MRs, you sort of
> have to seek out what to review; if stuff is not tagged with something
> you feel is relevant, you probably won't look at it, so the only
> people likely to review it are the people involved in writing it in
> the first place, which would be the same whether it's mesa or some
> other project.  I think all of the projects generally have the best
> intentions at heart, but for better or worse they just have different
> development models.  In the case of the AMD throw it over the wall
> stuff, it's not really an anti-open source or community engagement
> issue, it's more of how to we support several OSes, tons of new
> products, several custom projects, etc. while leveraging as much
> shared code as possible.  There are ways to make it work, but they are
> usually a pretty heavy lift that not all teams can make.
>
> All of that said, I think providing a link to the userspace user of
> the API is reasonable, but I don't think there have been any egregious
> cases of badly designed UAPI that were not caught using the existing
> processes.

One advantage I forgot to state was that it gives me less reason to
drop a complete pull request on the grounds that it contains some new
uAPI that didn't follow the rules. This reduces the pressure on me and
all the submaintainers. At the moment I'm inclined to let some simpler
uapi changes past, because I don't want to drop complete feature pulls
on the ground. I'm going to get stricter on this I suspect.

Btw you can subscribe to all Mesa MR activity, I do it, I just ignore
the ones I don't want to look at, just like I did for mailing lists.
Even using MRs for Mesa still puts it streets ahead of any other
userspace project in terms of insight into development, and hence
trust of the developers involved. This is mostly about trust,
currently I'd don't trust those teams to care enough or do the right
thing. When we write down a bunch of rules about how this should work
we get rules lawyering instead. I really don't want uapi development
to be me with a magic eight ball, but it seems like that is where we
will end up unless companies follow even the basic directions.

Dave.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Dave Airlie]

On Thu, 17 Oct 2019 at 03:28, Matt Roper <matthew.d.roper@intel.com> wrote:
>
> On Tue, Oct 15, 2019 at 04:16:00AM +1000, Dave Airlie wrote:
> > I've kicked this around in my head over the past few weeks but wanted
> > to get some feedback on whether it's a good idea or what impact it
> > might have that I haven't considered.
> >
> > We are getting requests via both amdgpu/amdkfd and i915 for new user
> > APIs for userspace drivers that throw code over the wall instead of
> > being open source developed projects, but we are also seeing it for
> > android drivers and kms properties, and we had that i915 crappy crtc
> > background thing that was for Chrome but Chrome didn't want it.
>
> Having submitted "that i915 crappy crtc background thing" it's unclear
> to me what you're trying to say here.  If you're trying to claim that
> this was "thrown over the wall" or that there was an attempt to try to
> sneak this work in under the radar and circumvent uapi policy then you're
> way off track and you probably need to review the (long) history of this
> feature.  We've very religiously followed both the spirit and the letter
> of your upstream uapi policy with regards to that feature and taken great
> care to ensure it would *not* land upstream without an actual opensource
> userspace consumer.  There was some confusion recently about how
> ChromeOS was/wasn't using this interface, but the Google guys helped
> clear that up, and the kernel changes have now returned to their holding
> pattern and will not go upstream unless new userspace emerges.  To
> refresh your memory on the history:
>
>  * The proposed interface was originally posted in 2015/2016 as an FYI
>    to find out if any opensource userspace projects might have a use for
>    this functionality (we already had the patches written since there's
>    non-opensource software that requires it).  The cover letters at the
>    time clearly stated that it should not be merged upstream since there
>    was no userspace:
>
>         "Note that this series isn't mergeable yet since we don't (yet)
>         have an open source userspace that can make use of it"
>         (https://lists.freedesktop.org/archives/intel-gfx/2016-February/087382.html)
>
>  * About 2.5 years later there was suddenly interest expressed both from
>    people involved in Weston and people involved in ChromeOS, and they
>    independently asked for refreshed kernel patches.  When the refreshed
>    patches were posted  they again included a disclaimer that we needed
>    proper userspace before it could land:
>
>         "As always, we'll still need the patches for at least one of
>         those projects to get posted (and reviewed) somewhere public
>         before we actually merge these kernel patches."
>         (https://lists.freedesktop.org/archives/intel-gfx/2018-October/178202.html)
>
>  * The Weston effort apparently fizzled out, but at the beginning of
>    2019 we were directed to some patches to the ChromeOS ozone layer
>    (https://chromium-review.googlesource.com/c/chromium/src/+/1278858)
>    that utilized the interface.  I'm not at all familiar with the
>    ChromeOS userspace stack, but since the Ozone code was reviewed by
>    non-Intel folks and shows as 'merged' that sounded like a legitimate
>    open source userspace consumer.  From that point on the Chromium
>    review links were clearly advertised in the cover letter each time
>    the series was respun so that it would be very clear what the
>    expected userspace was and so any concerns could be raised.
>
>  * Daniele and Sean from Google recently brought it to our attention
>    that even though that ozone code made use of the background color
>    property, the ozone interfaces themselves don't get used at higher
>    levels of the ChromeOS stack (and had probably been merged to the
>    ozone layer in error).  Once they helped us understand that the
>    properties wouldn't ultimately be utilized in a meaningful way with a
>    real ChromeOS setup today, we put a halt to the upstreaming effort:
>
>         "Hmm, in that case it sounds like we probably don't actually
>         have enough userspace support yet to justify merging the kernel
>         changes at this time."
>         (https://lists.freedesktop.org/archives/dri-devel/2019-October/239436.html)
>
> So long story short, we've been very transparent about what's going on
> with this feature for the whole ~4 years it's been floating around.  I
> would hope that you, as DRM maintainer, would have spoken up at some
> point if you felt there was something else that needed to be handled
> differently.  No code for this api has ever landed in any of the driver
> trees, so I'm not sure why you called out this work in an email
> proposing a change to uapi pull requests.

I understand that but someone merged this to ozone without going
through proper procedures, I'm not sure who that was, you probably do.

The procedure is develop in the open in parallel, get everyone to sign
off on it, merge to kernel, then merge to userspace, in this case it
looked like
it got merged to ozone before the kernel, and then the kernel was left
thinking it was a done deal from the userspace side so it should be
merged, without anyone really checking if there was a userspace stack
on top.

This is just one of the examples of where it's impossible for me as a
kernel maintainer to follow all the pieces of the puzzle, and I have
to trust the process is being followed, but when I find out I can't
trust it I have to adjust the process to catch this sort of thing next
time.

Dave.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Daniel Vetter]

On Wed, Oct 16, 2019 at 04:00:25PM -0400, Alex Deucher wrote:
> On Mon, Oct 14, 2019 at 2:16 PM Dave Airlie <airlied@gmail.com> wrote:
> >
> > I've kicked this around in my head over the past few weeks but wanted
> > to get some feedback on whether it's a good idea or what impact it
> > might have that I haven't considered.
> >
> > We are getting requests via both amdgpu/amdkfd and i915 for new user
> > APIs for userspace drivers that throw code over the wall instead of
> > being open source developed projects, but we are also seeing it for
> > android drivers and kms properties, and we had that i915 crappy crtc
> > background thing that was for Chrome but Chrome didn't want it.
> >
> > Now this presents a couple of issues:
> >
> > a) these projects don't seem to that good at following our development
> > guidelines, avoid developing userspace features in parallel in the
> > open and having good development implementations before submitting
> > upstream.
> >
> > b) these projects don't have experienced userspace developers
> > reviewing their kernel uapis. One big advantage of adding uapis with
> > mesa developers is they have a lot of experience in the area as well.
> >
> > It's leading me to think I want to just stop all uapi submissions via
> > driver trees, and instead mandate that all driver uapi changes are
> > sent in separate git pull requests to dri-devel, I'd try (with some
> > help) to catch all uapi modifications in normal trees, and refuse
> > pulls that modified uapi.
> >
> > At least I'm considered writing the script and refusing and pulls that
> > have a uapi change that doesn't contain a link to the userspace
> > changes required for it in a public developed repo.
> >
> > Thoughts?
>
> This seems like more hassle for questionable benefits.  I don't know
> that mesa is really any better than any other driver teams with
> respect to UAPI.  This just seems like sort of a arbitrary political
> decision.  The people working on mesa have as much of an agenda as
> those working on other projects.  Moreover, especially with the
> migration to gitlab and MRs, I feel that mesa development has gotten
> more opaque.  Say what you will about mailing lists, but at least you
> could have a drive by view of what's going on.  With MRs, you sort of
> have to seek out what to review; if stuff is not tagged with something
> you feel is relevant, you probably won't look at it, so the only
> people likely to review it are the people involved in writing it in
> the first place, which would be the same whether it's mesa or some
> other project.  I think all of the projects generally have the best
> intentions at heart, but for better or worse they just have different
> development models.  In the case of the AMD throw it over the wall
> stuff, it's not really an anti-open source or community engagement
> issue, it's more of how to we support several OSes, tons of new
> products, several custom projects, etc. while leveraging as much
> shared code as possible.  There are ways to make it work, but they are
> usually a pretty heavy lift that not all teams can make.

I think there's a difference between All Tools Sucks (tm) and the
discussions not even being accessible at all. I do agree that generally
everyone screws up uapi once in a while, and we seem to overall do a not
too shoddy job. So code is probably all ok enough.

But imo long term code is fungible and really doesn't matter much, the
important stuff is the people and teams who create it, and all the shared
knowledge. That's also were I see the benefit in upstream (for customers
and vendors and everyone), we can learn from each another. As an example,
I've spent lots of time recently reading amdgpu code and how it's used in
userspace. Understanding that without having access to the discussion or
being able to ping people on irc and mailing lists would have been
impossible - lots of questions where I just plain guessed wrong. For the
code-over-wall projects that stuff all simply doesn't exist. It's nigh
impossible to figure out whether uapi makes sense or not if you can't see
all the tradeoffs and discussions that influenced it and why.

That's also why I think the separate pull won't help at all, since Dave
will still have incomplete information. All he can do with more pulls is
roll the die more often, that's not helping.

Now short term "moar hw support" is cool and all that, but long term I do
think it's missing the point of upstreaming. It's not that mesa (or any
other cross vendor project, we have a bunch of those on the kms side) is
better at uapi, it's that it's more open and so _much_ easier to
understand how we ended up at a specific place. That's at least my take on
all this.

> All of that said, I think providing a link to the userspace user of
> the API is reasonable, but I don't think there have been any egregious
> cases of badly designed UAPI that were not caught using the existing
> processes.

Imo the problem isn't the lack of links, but lack of (public) discussions.
One idea I toyed around with could be to require uapi review for new uapi
by someone outside the wall. That's defacto what we do for everything
pushed through cross-vendor userspace anyway, and it would make sure that
all the design considerations relevant to the uapi would bubble over the
wall too, not just the code. The people and their expertise would still be
in hiding, so still far from nirvana, but I think this would at least move
things meaningfully.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Alex Deucher]

On Thu, Oct 17, 2019 at 9:58 AM Daniel Vetter <daniel@ffwll.ch> wrote:
>
> On Wed, Oct 16, 2019 at 04:00:25PM -0400, Alex Deucher wrote:
> > On Mon, Oct 14, 2019 at 2:16 PM Dave Airlie <airlied@gmail.com> wrote:
> > >
> > > I've kicked this around in my head over the past few weeks but wanted
> > > to get some feedback on whether it's a good idea or what impact it
> > > might have that I haven't considered.
> > >
> > > We are getting requests via both amdgpu/amdkfd and i915 for new user
> > > APIs for userspace drivers that throw code over the wall instead of
> > > being open source developed projects, but we are also seeing it for
> > > android drivers and kms properties, and we had that i915 crappy crtc
> > > background thing that was for Chrome but Chrome didn't want it.
> > >
> > > Now this presents a couple of issues:
> > >
> > > a) these projects don't seem to that good at following our development
> > > guidelines, avoid developing userspace features in parallel in the
> > > open and having good development implementations before submitting
> > > upstream.
> > >
> > > b) these projects don't have experienced userspace developers
> > > reviewing their kernel uapis. One big advantage of adding uapis with
> > > mesa developers is they have a lot of experience in the area as well.
> > >
> > > It's leading me to think I want to just stop all uapi submissions via
> > > driver trees, and instead mandate that all driver uapi changes are
> > > sent in separate git pull requests to dri-devel, I'd try (with some
> > > help) to catch all uapi modifications in normal trees, and refuse
> > > pulls that modified uapi.
> > >
> > > At least I'm considered writing the script and refusing and pulls that
> > > have a uapi change that doesn't contain a link to the userspace
> > > changes required for it in a public developed repo.
> > >
> > > Thoughts?
> >
> > This seems like more hassle for questionable benefits.  I don't know
> > that mesa is really any better than any other driver teams with
> > respect to UAPI.  This just seems like sort of a arbitrary political
> > decision.  The people working on mesa have as much of an agenda as
> > those working on other projects.  Moreover, especially with the
> > migration to gitlab and MRs, I feel that mesa development has gotten
> > more opaque.  Say what you will about mailing lists, but at least you
> > could have a drive by view of what's going on.  With MRs, you sort of
> > have to seek out what to review; if stuff is not tagged with something
> > you feel is relevant, you probably won't look at it, so the only
> > people likely to review it are the people involved in writing it in
> > the first place, which would be the same whether it's mesa or some
> > other project.  I think all of the projects generally have the best
> > intentions at heart, but for better or worse they just have different
> > development models.  In the case of the AMD throw it over the wall
> > stuff, it's not really an anti-open source or community engagement
> > issue, it's more of how to we support several OSes, tons of new
> > products, several custom projects, etc. while leveraging as much
> > shared code as possible.  There are ways to make it work, but they are
> > usually a pretty heavy lift that not all teams can make.
>
> I think there's a difference between All Tools Sucks (tm) and the
> discussions not even being accessible at all. I do agree that generally
> everyone screws up uapi once in a while, and we seem to overall do a not
> too shoddy job. So code is probably all ok enough.
>
> But imo long term code is fungible and really doesn't matter much, the
> important stuff is the people and teams who create it, and all the shared
> knowledge. That's also were I see the benefit in upstream (for customers
> and vendors and everyone), we can learn from each another. As an example,
> I've spent lots of time recently reading amdgpu code and how it's used in
> userspace. Understanding that without having access to the discussion or
> being able to ping people on irc and mailing lists would have been
> impossible - lots of questions where I just plain guessed wrong. For the
> code-over-wall projects that stuff all simply doesn't exist. It's nigh
> impossible to figure out whether uapi makes sense or not if you can't see
> all the tradeoffs and discussions that influenced it and why.
>
> That's also why I think the separate pull won't help at all, since Dave
> will still have incomplete information. All he can do with more pulls is
> roll the die more often, that's not helping.
>
> Now short term "moar hw support" is cool and all that, but long term I do
> think it's missing the point of upstreaming. It's not that mesa (or any
> other cross vendor project, we have a bunch of those on the kms side) is
> better at uapi, it's that it's more open and so _much_ easier to
> understand how we ended up at a specific place. That's at least my take on
> all this.

Fair points.  I guess I should clarify my thinking.  I was assuming
that the throw it over the wall teams would discuss the interface
publicly as a precondition and i.e., provide patches against the throw
it over the wall code to show it's usage even if those changes
ultimately would get become publicly available via some throw it over
the wall update once the interface lands.  My point was just that the
both mesa and throw it over the wall teams are equally capable of
doing good or bad interfaces and it shouldn't matter who develops it
as long as you provide reasonable review.

Alex

>
> > All of that said, I think providing a link to the userspace user of
> > the API is reasonable, but I don't think there have been any egregious
> > cases of badly designed UAPI that were not caught using the existing
> > processes.
>
> Imo the problem isn't the lack of links, but lack of (public) discussions.
> One idea I toyed around with could be to require uapi review for new uapi
> by someone outside the wall. That's defacto what we do for everything
> pushed through cross-vendor userspace anyway, and it would make sure that
> all the design considerations relevant to the uapi would bubble over the
> wall too, not just the code. The people and their expertise would still be
> in hiding, so still far from nirvana, but I think this would at least move
> things meaningfully.
> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: [RFC] new uapi policy for drm

[Daniel Vetter]

On Thu, Oct 17, 2019 at 4:12 PM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Thu, Oct 17, 2019 at 9:58 AM Daniel Vetter <daniel@ffwll.ch> wrote:
> >
> > On Wed, Oct 16, 2019 at 04:00:25PM -0400, Alex Deucher wrote:
> > > On Mon, Oct 14, 2019 at 2:16 PM Dave Airlie <airlied@gmail.com> wrote:
> > > >
> > > > I've kicked this around in my head over the past few weeks but wanted
> > > > to get some feedback on whether it's a good idea or what impact it
> > > > might have that I haven't considered.
> > > >
> > > > We are getting requests via both amdgpu/amdkfd and i915 for new user
> > > > APIs for userspace drivers that throw code over the wall instead of
> > > > being open source developed projects, but we are also seeing it for
> > > > android drivers and kms properties, and we had that i915 crappy crtc
> > > > background thing that was for Chrome but Chrome didn't want it.
> > > >
> > > > Now this presents a couple of issues:
> > > >
> > > > a) these projects don't seem to that good at following our development
> > > > guidelines, avoid developing userspace features in parallel in the
> > > > open and having good development implementations before submitting
> > > > upstream.
> > > >
> > > > b) these projects don't have experienced userspace developers
> > > > reviewing their kernel uapis. One big advantage of adding uapis with
> > > > mesa developers is they have a lot of experience in the area as well.
> > > >
> > > > It's leading me to think I want to just stop all uapi submissions via
> > > > driver trees, and instead mandate that all driver uapi changes are
> > > > sent in separate git pull requests to dri-devel, I'd try (with some
> > > > help) to catch all uapi modifications in normal trees, and refuse
> > > > pulls that modified uapi.
> > > >
> > > > At least I'm considered writing the script and refusing and pulls that
> > > > have a uapi change that doesn't contain a link to the userspace
> > > > changes required for it in a public developed repo.
> > > >
> > > > Thoughts?
> > >
> > > This seems like more hassle for questionable benefits.  I don't know
> > > that mesa is really any better than any other driver teams with
> > > respect to UAPI.  This just seems like sort of a arbitrary political
> > > decision.  The people working on mesa have as much of an agenda as
> > > those working on other projects.  Moreover, especially with the
> > > migration to gitlab and MRs, I feel that mesa development has gotten
> > > more opaque.  Say what you will about mailing lists, but at least you
> > > could have a drive by view of what's going on.  With MRs, you sort of
> > > have to seek out what to review; if stuff is not tagged with something
> > > you feel is relevant, you probably won't look at it, so the only
> > > people likely to review it are the people involved in writing it in
> > > the first place, which would be the same whether it's mesa or some
> > > other project.  I think all of the projects generally have the best
> > > intentions at heart, but for better or worse they just have different
> > > development models.  In the case of the AMD throw it over the wall
> > > stuff, it's not really an anti-open source or community engagement
> > > issue, it's more of how to we support several OSes, tons of new
> > > products, several custom projects, etc. while leveraging as much
> > > shared code as possible.  There are ways to make it work, but they are
> > > usually a pretty heavy lift that not all teams can make.
> >
> > I think there's a difference between All Tools Sucks (tm) and the
> > discussions not even being accessible at all. I do agree that generally
> > everyone screws up uapi once in a while, and we seem to overall do a not
> > too shoddy job. So code is probably all ok enough.
> >
> > But imo long term code is fungible and really doesn't matter much, the
> > important stuff is the people and teams who create it, and all the shared
> > knowledge. That's also were I see the benefit in upstream (for customers
> > and vendors and everyone), we can learn from each another. As an example,
> > I've spent lots of time recently reading amdgpu code and how it's used in
> > userspace. Understanding that without having access to the discussion or
> > being able to ping people on irc and mailing lists would have been
> > impossible - lots of questions where I just plain guessed wrong. For the
> > code-over-wall projects that stuff all simply doesn't exist. It's nigh
> > impossible to figure out whether uapi makes sense or not if you can't see
> > all the tradeoffs and discussions that influenced it and why.
> >
> > That's also why I think the separate pull won't help at all, since Dave
> > will still have incomplete information. All he can do with more pulls is
> > roll the die more often, that's not helping.
> >
> > Now short term "moar hw support" is cool and all that, but long term I do
> > think it's missing the point of upstreaming. It's not that mesa (or any
> > other cross vendor project, we have a bunch of those on the kms side) is
> > better at uapi, it's that it's more open and so _much_ easier to
> > understand how we ended up at a specific place. That's at least my take on
> > all this.
>
> Fair points.  I guess I should clarify my thinking.  I was assuming
> that the throw it over the wall teams would discuss the interface
> publicly as a precondition and i.e., provide patches against the throw
> it over the wall code to show it's usage even if those changes
> ultimately would get become publicly available via some throw it over
> the wall update once the interface lands.  My point was just that the
> both mesa and throw it over the wall teams are equally capable of
> doing good or bad interfaces and it shouldn't matter who develops it
> as long as you provide reasonable review.

I think the discussions/patches generally happen and so the strict
wording of our rules are followed, at least by intel/amd (there's some
other stuff in uapi land that was a lot more skitting the edge, on
both sides of the edge). But usually by the time things go public, the
discussion is about implementation details. And the big questions like
whether we're solving the right problem, or in the right place are all
long ago decided already. So not really the useful/interesting stuff
is discussed in public.

Of course with hw enabling that can't really be avoided, but then once
you've fixed the hw design for a new generation the uapi adjustments
(new golden reg state, a few more flags perhaps) are all pretty clear
and there's not really two fundamentally different ways to do things.

And like I said, I'm not sure how we could actually change much here
without just creating useless paperwork to keep everyone busy while
still having no real benefit.
-Daniel

>
> Alex
>
> >
> > > All of that said, I think providing a link to the userspace user of
> > > the API is reasonable, but I don't think there have been any egregious
> > > cases of badly designed UAPI that were not caught using the existing
> > > processes.
> >
> > Imo the problem isn't the lack of links, but lack of (public) discussions.
> > One idea I toyed around with could be to require uapi review for new uapi
> > by someone outside the wall. That's defacto what we do for everything
> > pushed through cross-vendor userspace anyway, and it would make sure that
> > all the design considerations relevant to the uapi would bubble over the
> > wall too, not just the code. The people and their expertise would still be
> > in hiding, so still far from nirvana, but I think this would at least move
> > things meaningfully.
> > -Daniel
> > --
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > http://blog.ffwll.ch



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel