Re: DRM changes

Re: DRM changes

[Stuart Bennett]

Stephane Marchesin wrote:
> Hi,
> 
> As part of aiming at upstreaming our code, I suppose we have to
> discuss the DRM situation a little. In order to aim for merging, I
> think we'd better be working on a linux kernel tree layout. And
> considering we're technically the only ones still working in drm.git,
> it doesn't really make sense to keep doing that for the sake of
> sharing code with other drivers.
> 
> As we discussed on irc, there would be multiple changes related to this:
> - we move to a linux kernel-like tree, that should make it easier when
> upstreaming the code.
> - this new tree is hosted in freedekstop.org in the nouveau/ git so we
> don't need additional accounts for everyone all around and people can
> keep pushing things (even better, all nouveau people can push to the
> drm, which used to require mesa rights before)
> - we keep a(some) branch(es) in the tree for backwards compat with
> older kernels. Either in the form of separate kernel versions
> including nouveau, or in the form of an out-of-tree-compilable
> drm/nouveau module.
> 
> So, does that plan sound sane? Do you have better plans?
> 
> Stephane

(apologies for the crappy mailman archive->email forgery)

I have reservations about a full Linux kernel tree, if that's the proposal.

Assuming it is a relatively important goal that people should test 
Nouveau, then it makes sense to make it as easy as possible for them to 
do so.  Certainly as long as Nouveau has not merged to the Linux kernel, 
our repository is the only (non-packaged) source for those wishing to 
test.  Post merge, some repository other than Linus' kernel is still 
going to be the place to point people at for new development and fixes 
which are not in a kernel.org stable release.

I'd suggest that a) requiring 0.5 - 1 hour to git clone the half-gig or 
so of our new kernel-based tree (longer if the fd.o machinery is having 
a bad day), and b) needing people to then checkout some random branch in 
order for it to work with their kernel ( c) would be rebuild their 
kernel, if the out-of-tree compilable thing doesn't happen ), are all 
barriers to testers that are not currently there.  Although my usage may 
be atypical (or git outdated), the repository size issue also bites 
developers; cold-cache git diff and git status on the linux kernel takes 
a noticeable time, and gitk, even if regularly invoked, eats disk and 
processor for ages on each execution.

If the goal of these changes is to have a more Linux kernel-like layout, 
could we not just change drm/linux-core, or make a new repo, so that it 
mirrored everything necessary under drivers/drm?  I believe airlied 
suggested this ages ago, and it would make moving code from Linux to 
Nouveau or the other way round simpler.  Admittedly this doesn't work if 
for some reason your development demands changes elsewhere in the Linux 
tree, but in that case you may well have already failed the back-compat 
aspect.

As for branching and back-compat, I'd have thought it more useful to 
keep compat on the main branch (again, helping testers), and split 
specific kernel-merge-aimed branches off from that, as automating ifdef 
removal on a branch ought to be easier than manually adding compat to a 
bunch of branches, but I guess in some way the solutions here are all 
equivalent, and it's just which branch you bless as being called master.

Stuart

Re: DRM changes

[Alexey Dobriyan]

On Wed, Mar 11, 2009 at 05:15:08PM +0000, Stuart Bennett wrote:
> Stephane Marchesin wrote:
> > Hi,
> > 
> > As part of aiming at upstreaming our code, I suppose we have to
> > discuss the DRM situation a little. In order to aim for merging, I
> > think we'd better be working on a linux kernel tree layout. And
> > considering we're technically the only ones still working in drm.git,
> > it doesn't really make sense to keep doing that for the sake of
> > sharing code with other drivers.
> > 
> > As we discussed on irc, there would be multiple changes related to this:
> > - we move to a linux kernel-like tree, that should make it easier when
> > upstreaming the code.
> > - this new tree is hosted in freedekstop.org in the nouveau/ git so we
> > don't need additional accounts for everyone all around and people can
> > keep pushing things (even better, all nouveau people can push to the
> > drm, which used to require mesa rights before)
> > - we keep a(some) branch(es) in the tree for backwards compat with
> > older kernels. Either in the form of separate kernel versions
> > including nouveau, or in the form of an out-of-tree-compilable
> > drm/nouveau module.
> > 
> > So, does that plan sound sane? Do you have better plans?
> > 
> > Stephane
> 
> (apologies for the crappy mailman archive->email forgery)
> 
> I have reservations about a full Linux kernel tree, if that's the proposal.
> 
> Assuming it is a relatively important goal that people should test 
> Nouveau, then it makes sense to make it as easy as possible for them to 
> do so.  Certainly as long as Nouveau has not merged to the Linux kernel, 
> our repository is the only (non-packaged) source for those wishing to 
> test.  Post merge, some repository other than Linus' kernel is still 
> going to be the place to point people at for new development and fixes 
> which are not in a kernel.org stable release.
> 
> I'd suggest that a) requiring 0.5 - 1 hour to git clone the half-gig or 
> so of our new kernel-based tree (longer if the fd.o machinery is having 
> a bad day)

If user already has Linus's tree (or some tree based on it), then there is
no need to fetch most of it again.

> and b) needing people to then checkout some random branch in 
> order for it to work with their kernel ( c) would be rebuild their 
> kernel, if the out-of-tree compilable thing doesn't happen ), are all 
> barriers to testers that are not currently there.  Although my usage may 
> be atypical (or git outdated), the repository size issue also bites 
> developers; cold-cache git diff and git status on the linux kernel takes 
> a noticeable time,

You don't do just git-diff, you do git-diff drivers/gpu/drm/ .

> and gitk, even if regularly invoked, eats disk and 
> processor for ages on each execution.
> 
> If the goal of these changes is to have a more Linux kernel-like layout, 
> could we not just change drm/linux-core, or make a new repo, so that it 
> mirrored everything necessary under drivers/drm?  I believe airlied 
> suggested this ages ago, and it would make moving code from Linux to 
> Nouveau or the other way round simpler.  Admittedly this doesn't work if 
> for some reason your development demands changes elsewhere in the Linux 
> tree, but in that case you may well have already failed the back-compat 
> aspect.
> 
> As for branching and back-compat, I'd have thought it more useful to 
> keep compat on the main branch (again, helping testers), and split 
> specific kernel-merge-aimed branches off from that, as automating ifdef 
> removal on a branch ought to be easier than manually adding compat to a 
> bunch of branches, but I guess in some way the solutions here are all 
> equivalent, and it's just which branch you bless as being called master.

DRM changes

[Stephane Marchesin]

Hi,

As part of aiming at upstreaming our code, I suppose we have to
discuss the DRM situation a little. In order to aim for merging, I
think we'd better be working on a linux kernel tree layout. And
considering we're technically the only ones still working in drm.git,
it doesn't really make sense to keep doing that for the sake of
sharing code with other drivers.

As we discussed on irc, there would be multiple changes related to this:
- we move to a linux kernel-like tree, that should make it easier when
upstreaming the code.
- this new tree is hosted in freedekstop.org in the nouveau/ git so we
don't need additional accounts for everyone all around and people can
keep pushing things (even better, all nouveau people can push to the
drm, which used to require mesa rights before)
- we keep a(some) branch(es) in the tree for backwards compat with
older kernels. Either in the form of separate kernel versions
including nouveau, or in the form of an out-of-tree-compilable
drm/nouveau module.

So, does that plan sound sane? Do you have better plans?

Stephane

Re: DRM changes

[Maarten Maathuis]

Questions:

A tree based on what? (linus, drm-next, etc)

What happens to shared-core?

Opinions (assuming this happens):

I'd do a branch for each kernel version (or less if needed), and let
people who care cherry-pick fixes.

Maarten.

Re: DRM changes

[Stephane Marchesin]

On Mon, Mar 9, 2009 at 23:23, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Questions:
>
> A tree based on what? (linus, drm-next, etc)
>

I would follow drm-next, but I don't think it is very relevant at this
point, since anyway drm-next periodically merges to linus. If there
are reasons to choose one over the other, I'm interested in hearing
them.

> What happens to shared-core?

It will probably meet its creator.

>
> Opinions (assuming this happens):
>
> I'd do a branch for each kernel version (or less if needed), and let
> people who care cherry-pick fixes.
>

Yeah that's what I initially thought, but Pekka said you could also
get an out-of-tree build from a full kernel tree. So that's the two
paths to explore.

Stephane

Re: DRM changes

[Peter Hjalmarsson]

mån 2009-03-09 klockan 22:49 +0100 skrev Stephane Marchesin:
> Hi,
> 
> So, does that plan sound sane? Do you have better plans?
> 
> Stephane

I must say that for me it does not give that big problems, since I
already are using my own hem-brewed kernel. But I think it will be
harder getting testers if they need to not just compile som modules for
their kernel, but also try to configure the beast on their own.

Just my 2-cents.


_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: DRM changes

[Renato Caldas]

On Thu, Mar 12, 2009 at 7:21 PM, Peter Hjalmarsson <xake-ca8b1Tj3MiseIZ0/mPfg9Q@public.gmane.org> wrote:
> mån 2009-03-09 klockan 22:49 +0100 skrev Stephane Marchesin:
>> Hi,
>>
>> So, does that plan sound sane? Do you have better plans?
>>
>> Stephane
>
> I must say that for me it does not give that big problems, since I
> already are using my own hem-brewed kernel. But I think it will be
> harder getting testers if they need to not just compile som modules for
> their kernel, but also try to configure the beast on their own.

Lets face it, users who can't configure their own kernel will probably
turn out to be bad testers. And besides, configuring a kernel is not
that hard, and users can just copy their distro's configuration and
only tweak the drm related parts. No show stoppers here.

Cheers,
  Renato

Re: DRM changes

[Peter Hjalmarsson]

lör 2009-03-14 klockan 11:18 +0000 skrev Renato Caldas:
> On Thu, Mar 12, 2009 at 7:21 PM, Peter Hjalmarsson <xake@rymdraket.net> wrote:
> > mån 2009-03-09 klockan 22:49 +0100 skrev Stephane Marchesin:
> >> Hi,
> >>
> >> So, does that plan sound sane? Do you have better plans?
> >>
> >> Stephane
> >
> > I must say that for me it does not give that big problems, since I
> > already are using my own hem-brewed kernel. But I think it will be
> > harder getting testers if they need to not just compile som modules for
> > their kernel, but also try to configure the beast on their own.
> 
> Lets face it, users who can't configure their own kernel will probably
> turn out to be bad testers. And besides, configuring a kernel is not
> that hard, and users can just copy their distro's configuration and
> only tweak the drm related parts. No show stoppers here.
> 
> Cheers,
>   Renato

I know many who are good enought to be good testers and are programmers
but could care less about having to update their kernel, patch it and/or
reconfigure it every time someone finds a security-related bug or broken
feature and WANT someone (i.e. the distribution) to do that work for
them. They just want to use their computer to what they want: code
programs, not spend time on their configuration if they do not have to.

Even if they can copy the configuration from their distribution, the
question is if they want to spend time on tracking the kernel, patch it
and so on if they had to. 

I do not know if I have hadthe time to do that if it was not for having
a good working configuration and having gentoo tracking and patching it
for me. 
And with the current development-speed of nouveau: do we want testers
that only update when they have that amount of time, it would take long
time to bisect those historys if/when something breaks.
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: DRM changes

[Maarten Maathuis]

Is there a final verdict yet?

Re: DRM changes

[Pekka Paalanen]

On Mon, 9 Mar 2009 22:49:32 +0100
Stephane Marchesin <marchesin-YynmYXH4S6491FlJP1ih0VAUjnlXr6A1@public.gmane.org> wrote:

> Hi,
> 
> As part of aiming at upstreaming our code, I suppose we have to
> discuss the DRM situation a little. In order to aim for merging, I
> think we'd better be working on a linux kernel tree layout. And
> considering we're technically the only ones still working in drm.git,
> it doesn't really make sense to keep doing that for the sake of
> sharing code with other drivers.
> 
> As we discussed on irc, there would be multiple changes related to this:
> - we move to a linux kernel-like tree, that should make it easier when
> upstreaming the code.
> - this new tree is hosted in freedekstop.org in the nouveau/ git so we
> don't need additional accounts for everyone all around and people can
> keep pushing things (even better, all nouveau people can push to the
> drm, which used to require mesa rights before)
> - we keep a(some) branch(es) in the tree for backwards compat with
> older kernels. Either in the form of separate kernel versions
> including nouveau, or in the form of an out-of-tree-compilable
> drm/nouveau module.
> 
> So, does that plan sound sane? Do you have better plans?

I had a discussion with darktama (Ben) about this.
Dave, there are questions specifically for you.
Marcheu, we need your verdict after Dave's comments.

Using a kernel tree would free Ben and others from backporting
things to drm.git. My "Building rawhide+darktama on 2.6.24"
experiment shows, that it is possible to reach practically the
same level of ease as drm.git offers to users. More on this later.
One major point here is to keep the Nouveau DRM development in
one place.

The downside is that BSD devels will probably get hurt. The
backporting work Ben and others are doing now will fall onto them.
Yet we cannot really stick to drm.git if we want upstream, since
de facto upstream is Linux. Someone has to do the porting and it's
pretty much a trade-off between helping BSDs or concentrating on
core development.

Why have a full kernel tree, and not just drivers/gpu/?
Because then it is easy to merge from upstream back to our tree.

My proposition follows:

Step 1: Nouveau development tree

This will be a complete kernel tree hosted at freedesktop.org.
Ben is ready to request it from fd.o. This tree will be free for all
developers to commit, adhering to kernel coding style. This is
NOT a tree that will be sent upstream, but all (future) patches
must be as upstream-quality as possible. This tree will
completely replace drm.git's shared-core and linux-core directories.

This tree is never rebased. Ever. Period. Upstream kernel
development will be marged back here, but we do not rebase.
Whatever gets pushed into the repo, stays there. This offers
a solid ground for devels to base their private work on.
Somewhat related, read also this:
http://sourceforge.net/mailarchive/message.php?msg_name=alpine.LFD.2.00.0903291409030.3994%40localhost.localdomain

The $1M question: which kernel tree will this be based on?
Ben tells me Dave's drm-next[1] gets rebased sometimes.

Dave, what are your plans on drm-next now after Linus'
comments? Are you able to offer a DRM kernel tree that will
not be rebased, and that we could use as the base?

Since this is not an upstreaming tree, it should not matter
too much when we pull updates from upstream, but we probably
need a policy for it.

Compatibility for older kernels can be implemented as a
branch in this tree, let's call it compat. Compat starts as a
fork of master, gets the basic compatibility patches from me,
and then we periodically just merge master into compat to keep
it up-to-date. If compatibility needs some fixes, they are just
committed into the branch. This keeps the main development and
backwards compat separate. I can maintain the compat stuff for
now, and I will also have personal interest as long as I use
2.6.24. The single compat branch will serve "all" old kernels
using #ifdefs. The compat branch is used with out-of-tree compiling.

There are several ways to offer the branches to users:
- cgit.fd.o can produce tar-balls of the complete tree, i.e. snapshots
- someone could set up an automatic snapshot service, that packages
only the drivers/gpu/ directory and other relevant files instead of
the full kernel tree
- users can clone the kernel tree without history, and still later
use git-fetch or git-pull to stay up-to-date. (See git-clone --depth)

It is possible to build drm.ko and nouveau.ko out-of-tree. This means
the modules are built from the Nouveau kernel tree sources, but
for (with) whatever kernel the user is already running (provided the
configuration supports Nouveau, and we have compat stuff for the
kernel version). The commands are a bit long,
but we can offer a script that nicely automates it, and also checks
the kernel config. I think we can get pretty close to how easy
drm.git is for building kernel modules. Currently there is one
nasty obstacle: some older kernels are missing
EXPORT_SYMBOL_GPL(shmem_file_setup), which means the user needs to
patch that in. Ben says we should be able to work around that in near
future.


Step 2: Nouveau upstreaming tree

Because the development tree is a development tree, and from the
kernel's perspective is a private tree, it will not be used for
upstreaming. As you know, accidents happen in development trees, and
the patches do not get reviewed before being pushed. A separate tree
is made for upstreaming.

This tree will live in kernel.org and have a dedicated maintainer,
who will cherry-pick patches from the Nouveau development tree for
upstreaming. The upstreaming tree is based on whatever Dave wants,
as the patches (or maybe pull-requests, when the practices have
stabilized) will be sent to Dave.

There is no need to create the upstreaming tree until Nouveau is
in the shape that it could start pushing for upstream. Hopefully
by then we have also found the maintainer (or two). Personally
I cannot promise to take this task, since I am not convinced I
have the time to do it well.

In the meanwhile, we should rehearse making upstream-quality
patches in the development tree.


Marcheu, what do you think about this? Let's give Ben the green light?
Dave, do you think the git use policy I described is feasible?

Comments, objections?
I have read the earlier comments on this thread and hope to have
answered all concerns.

A hint for kernel git trees: gitk -n 1000


Thanks.


[1] http://git.kernel.org/?p=linux/kernel/git/airlied/drm-2.6.git;a=summary

-- 
Pekka Paalanen
http://www.iki.fi/pq/