Git, merging, and News/Relnotes files

Git, merging, and News/Relnotes files

[Edward Z. Yang]

As a policy on a project that I manage, almost every commit warrants a
change to our NEWS (changelog) file, which end-users can browse to get
an in-depth idea of the changes that have happened from the last
release. If it's an added feature, the changelog includes a description
of how to use it; if it's a fixed bug, it briefly describes what
happened. Internal changes may or may not get added, depending on the
visibility of the APIs affected.

Something that I've noticed recently, as we've started migrating away
from the ghetto SVN development model to the Git branchy model, is that
this NEWS file ends up being the source of a lot of conflicts. Granted,
they're easy conflicts to resolve, but still, they make a pull a little
more complicated than it should be.

What would you guys, as experienced Git users, recommend in this case?
Scrapping a NEWS file and simply drawing up the release-notes shortly
before release (as the Git project does)? Aggregating the Git commit
messages into one monster release log? Having the release manager add
the NEWS entries himself, and mandate that no patch have it in them?

Thanks!

Re: Git, merging, and News/Relnotes files

[Linus Torvalds]

On Sat, 5 Jul 2008, Edward Z. Yang wrote:
> 
> Something that I've noticed recently, as we've started migrating away
> from the ghetto SVN development model to the Git branchy model, is that
> this NEWS file ends up being the source of a lot of conflicts. Granted,
> they're easy conflicts to resolve, but still, they make a pull a little
> more complicated than it should be.

I don't think anybody really _uses_ the functionality, but git does have 
the capability to specify special merge drivers based on filenames.

So you can

 (a) create a merge strategy that automaticaly does what you want. There's 
     a built-in driver called "union" that may or may not work for your 
     use case.

     See "Defining a custom merge driver" in "man gitattributes" for more 
     details about this.

 (b) Say which files you want to merge with this driver, by having 
     something like

	NEWS merge=news-merge

     in your .gitattributes file (or in ".git/info/attributes", if you 
     want to keep this all private to your own setup rather than in a 
     committed file that gets distributed to everybody else too).

and now your NEWS file will be merged using your special "news-merge" 
custom merge function.

Of course, the custom merge driver is only done for non-trivial merges. 
Git will do all the trivial fast-forward merges on its own, and only call 
the custom merge driver for things that have actual possible data 
conflicts (ie changes in both branches).

NOTE! Keeping an ordered list (like a ChangeLog or a NEWS file) is 
fundamentally not an easy thing to do in a distributed environment. The 
"union" merge strategy may well work for you (and if it does, this is all 
going to be very easy), but it's also entirely possible that you will find 
that the ordering in a distributed environment is so unspecified, you'll 
prefer to do the merges by hand _anyway_ in the end.

So the first thing you should do is probably to just *try* adding that

	NEWS merge=union

line to your .gitattributes file, and see if it works for you. My personal 
guess is that you'll realize that you really prefer doing the trivial 
merges manually after all, but hey, maybe not. And as mentioned, you *can* 
create your very own merge strategy that knows about the particular rules 
of the files in question, but that gets more complex.

For example, the default 'union' merge will literally _duplicate_ 
identical that were added in both branches. So if you cherry-pick a commit 
so that it exists both in the branch you are merging _and_ the branch you 
are merging into, then any additions to the NEWS file will basically show 
up twice, and yet auto-merge "cleanly".

Which is very understandable, but is almost certainly not what you want.

			Linus

Re: Git, merging, and News/Relnotes files

[Edward Z. Yang]

Linus Torvalds wrote:
> So the first thing you should do is probably to just *try* adding that
> 
> 	NEWS merge=union
> 
> line to your .gitattributes file, and see if it works for you.

Sounds like a good first-step. It's very unlikely that we're going to
bother writing our own merge strategy for the NEWS file, so if union
ends up being more trouble than its worth, we'll probably end sticking
with manual merges.

Pieter also suggested (for some reason, I don't see the post on this
list) the git-merge-changelog driver from Gnu Savannah. Unfortunately,
the log format is a little different from ours (entries are sorted into
BC-incompatible, features, bugfixes and internal changes), so the driver
may not work (it's still worth a try, I imagine).

I'm slightly surprised no one suggested that I can the file, given that
both Git and the Linux kernel don't have one.

> For example, the default 'union' merge will literally _duplicate_ 
> identical that were added in both branches. So if you cherry-pick a commit 
> so that it exists both in the branch you are merging _and_ the branch you 
> are merging into, then any additions to the NEWS file will basically show 
> up twice, and yet auto-merge "cleanly".

I suppose that's why we have git reset --hard HEAD~. :-) I will
certainly keep this gotcha in mind.

Re: Git, merging, and News/Relnotes files

[Linus Torvalds]

On Sat, 5 Jul 2008, Edward Z. Yang wrote:
> 
> I'm slightly surprised no one suggested that I can the file, given that
> both Git and the Linux kernel don't have one.

Well, I personally think ChangeLog files are a total waste of time. You're 
much better off autogenerating those from the real logs, I think.

[ Although in all honesty, I also think we could improve on our reporting 
  tools, and have ways to perhaps highlight big or important changes some 
  way ]

But a NEWS file that actually talks about new features is a different 
thing. It can make lots of sense to maintain something like that, so I 
wouldn't suggest canning it if it works for you. I'm not convinced it 
would work for the kernel, but I suspect it can work really well for other 
projects.

> > For example, the default 'union' merge will literally _duplicate_ 
> > identical that were added in both branches. So if you cherry-pick a commit 
> > so that it exists both in the branch you are merging _and_ the branch you 
> > are merging into, then any additions to the NEWS file will basically show 
> > up twice, and yet auto-merge "cleanly".
> 
> I suppose that's why we have git reset --hard HEAD~. :-) I will
> certainly keep this gotcha in mind.

Well, the real problem with a clean automatic merge is not that it can't 
be undone (or better yet - fixed: just edit the NEWS file and then do a 
"git commit --amend NEWS" to fix up the atomatic merge), but the fact that 
most of the time you'll simply never even notice.

IOW, when something merges cleanly (and the 'union' merge will basically 
always do so), the most common case is probably that people won't even 
_look_ at the end result - especially if it works fine most of the time. 

That's why a trivial conflict can often be better than a silently clean 
merge: at least it forces people to spend a small amount of brainpower to 
look at the obvious fix.

But hey, give it a try. Maybe you'll like the union merge, together with 
occasional manual fixups. Or maybe you'll decide that a specialized merge 
strategy isn't that painful after all (or can find somebody who already 
went through the pain and wrote one you can use).

		Linus

Re: Git, merging, and News/Relnotes files

[Dmitry Potapov]

On Sat, Jul 5, 2008 at 11:24 AM, Edward Z. Yang
<edwardzyang@thewritingpot.com> wrote:
> As a policy on a project that I manage, almost every commit warrants a
> change to our NEWS (changelog) file, which end-users can browse to get
> an in-depth idea of the changes that have happened from the last
> release. If it's an added feature, the changelog includes a description
> of how to use it; if it's a fixed bug, it briefly describes what
> happened. Internal changes may or may not get added, depending on the
> visibility of the APIs affected.

I believe it is better to put all this information directly to the commit
message using some special tagging, so you can extract it automatically
at the release time and generate the changelog file for users. You may
edit the generated changelog and commit it directly before release.

Having one file changed on almost every commit is not a good idea, and
not only because it will cause unnecessary conflicts but also it may
considerable increase the size of the whole repository. By default, the
delta compression has limit 50, which means that every 50 change of file
will become its full copy. If the changelog file is changed very often
and it is long, it may turn out that changelog alone takes as much space
as the rest of the source tree.

Dmitry

Re: Git, merging, and News/Relnotes files

[Edward Z. Yang]

Dmitry Potapov wrote:
> Having one file changed on almost every commit is not a good idea, and
> not only because it will cause unnecessary conflicts but also it may
> considerable increase the size of the whole repository. By default, the
> delta compression has limit 50, which means that every 50 change of file
> will become its full copy. If the changelog file is changed very often
> and it is long, it may turn out that changelog alone takes as much space
> as the rest of the source tree.

That is certainly a good technical point, and I will certainly look into
building a log parser after we wrap up our next release cycle.

P.S. Linus, we ended up manually merging the NEWS file; in some cases
there were branch specific changes in the file which would have been
completely inappropriate with a union merge. Thank you for the
suggestion, however.