offtopic: how to break huge patch into smaller independent patches?

offtopic: how to break huge patch into smaller independent patches?

[Chris Friesen]

Its kind of offtopic, but I hoped that someone might have some pointers since 
the kernel developers deal with so many patches.

I've been given a massive kernel patch that makes a whole bunch of conceptually 
independent changes.

Does anyone have any advice on how to break it up into independent patches?


Chris

Re: offtopic: how to break huge patch into smaller independent patches?

[William Lee Irwin III]

On Tue, Sep 14, 2004 at 01:48:37PM -0600, Chris Friesen wrote:
> Its kind of offtopic, but I hoped that someone might have some pointers 
> since the kernel developers deal with so many patches.
> I've been given a massive kernel patch that makes a whole bunch of 
> conceptually independent changes.
> Does anyone have any advice on how to break it up into independent patches?

It's hard work. It's sometimes even harder than writing the patch
itself. To handle this, I would:

(a) identify the conceptually independent changes
(b) rewrite the conceptually independent changes separately in series
(c) look for the results of (b) that are too large, and try to find
	some way to represent them as a series conceptually independent
	changes.
(d) if (c) returned a nonzero number of results, recurse to (b)

Yes, this means you have to rewrite the stuff altogether. This is not
quite as hard as it sounds, as you can largely "discover" what the
rewrites should be by filtering changes. But it's still relatively hard.

Identifying conceptually independent changes has no universal rules, so
that may also be hard.


-- wli

Re: offtopic: how to break huge patch into smaller independent patches?

[Dave Jones]

On Tue, Sep 14, 2004 at 01:48:37PM -0600, Chris Friesen wrote:
 > 
 > Its kind of offtopic, but I hoped that someone might have some pointers 
 > since the kernel developers deal with so many patches.
 > 
 > I've been given a massive kernel patch that makes a whole bunch of 
 > conceptually independent changes.
 > 
 > Does anyone have any advice on how to break it up into independent patches?

diffsplit will split it into a patch-per-file, which could be
a good start. If you have multiple changes touching the same file
however, things get a bit more fun, and you get to spend a lot
of time in your favorite text editor glueing bits together.

		Dave

Re: offtopic: how to break huge patch into smaller independent patches?

[J. Bruce Fields]

On Tue, Sep 14, 2004 at 09:12:10PM +0100, Dave Jones wrote:
> On Tue, Sep 14, 2004 at 01:48:37PM -0600, Chris Friesen wrote:
>  > 
>  > Its kind of offtopic, but I hoped that someone might have some pointers 
>  > since the kernel developers deal with so many patches.
>  > 
>  > I've been given a massive kernel patch that makes a whole bunch of 
>  > conceptually independent changes.
>  > 
>  > Does anyone have any advice on how to break it up into independent patches?
> 
> diffsplit will split it into a patch-per-file, which could be
> a good start. If you have multiple changes touching the same file
> however, things get a bit more fun, and you get to spend a lot
> of time in your favorite text editor glueing bits together.

When I've done this I've started by taking big patch P against tree T,
finding the simplest, easiest to understand change in it, writing a
small patch P_0 which makes that one change, then diffing T + P against

	T + P_0

(where "T + P_0" means "the result of applying patch P_0 to tree T").
Repeat, write a second patch P_1, and diff T + P against

	T + P_0 + P_1

So at step n+1, I find out what's left to do by diffing T + P against

	T + P_0 + ... + P_n

(where I'm actually maintaining the patches P_i using Andrew Morton's
patch scripts).

Given a tree, a big patch, and a series of little patches (more
generally, given *two* series of patches) I'd like to be able to
calculate the diff between the two quickly.  Not having any automated
way to do that makes this all more of a pain than it should be.

It seems like it shouldn't be hard to do this with some shell scripting,
but I haven't sat down and tried it yet.  I'd be interested to hear if
anyone else as.

--Bruce Fields

Re: offtopic: how to break huge patch into smaller independent patches?

[Matthias Urlichs]

Hi, Dave Jones wrote:

> diffsplit will split it into a patch-per-file, which could be
> a good start. If you have multiple changes touching the same file
> however, things get a bit more fun, and you get to spend a lot
> of time in your favorite text editor glueing bits together.

You can rip the bits apart instead, and leave the glueing and rip-patching
to the computer.

- edit patch file:
  - delete all the parts you don't want applied; freely hand-edit stuff,
    and don't worry about the pesky line numbers
  - save to new patch file
- run "rediff" to fix up the new file
- run "interdiff" to create a second, clean patch file
  containing just the deleted parts
- iterate until finished

All of this is part of the nice patchutils package.

NB: if all else fails, use espdiff(1).
-- 
Matthias Urlichs

Re: offtopic: how to break huge patch into smaller independent patches?

[Herbert Poetzl]

On Thu, Sep 16, 2004 at 11:11:48PM +0200, Matthias Urlichs wrote:
> Hi, Dave Jones wrote:
> 
> > diffsplit will split it into a patch-per-file, which could be
> > a good start. If you have multiple changes touching the same file
> > however, things get a bit more fun, and you get to spend a lot
> > of time in your favorite text editor glueing bits together.
> 
> You can rip the bits apart instead, and leave the glueing and rip-patching
> to the computer.
> 
> - edit patch file:
>   - delete all the parts you don't want applied; freely hand-edit stuff,
>     and don't worry about the pesky line numbers
>   - save to new patch file
> - run "rediff" to fix up the new file
> - run "interdiff" to create a second, clean patch file
>   containing just the deleted parts
> - iterate until finished
> 
> All of this is part of the nice patchutils package.
> 
> NB: if all else fails, use espdiff(1).

yes, this often helps me to get it right, where the
other patchutils seem to fail, but the option parsing
code seems to be broken :)

# espdiff -h
espdiff: invalid option -- h
Try `cat --help' for more information.
     ~~~

> -- 
> Matthias Urlichs
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: offtopic: how to break huge patch into smaller independent patches?

[Anton Blanchard]

> Its kind of offtopic, but I hoped that someone might have some pointers 
> since the kernel developers deal with so many patches.
> 
> I've been given a massive kernel patch that makes a whole bunch of 
> conceptually independent changes.
> 
> Does anyone have any advice on how to break it up into independent patches?

dirdiff is a great tool for this. I think its on samba.org somewhere,
but you can definitely find it in debian.

The new version is even better, I think Paul should do a release :)

Anton

Re: offtopic: how to break huge patch into smaller independent patches?

[Martin J. Bligh]

>> Its kind of offtopic, but I hoped that someone might have some pointers 
>> since the kernel developers deal with so many patches.
>> 
>> I've been given a massive kernel patch that makes a whole bunch of 
>> conceptually independent changes.
>> 
>> Does anyone have any advice on how to break it up into independent patches?
> 
> dirdiff is a great tool for this. I think its on samba.org somewhere,
> but you can definitely find it in debian.
> 
> The new version is even better, I think Paul should do a release :)

If the changes are in fairly independant files, just vi'ing the diff is
normally very effective. If they're all intertangled, then starting again
from scratch is prob easier ;-)

M.

Re: offtopic: how to break huge patch into smaller independent patches?

[Chris Friesen]

Martin J. Bligh wrote:

> If the changes are in fairly independant files, just vi'ing the diff is
> normally very effective. If they're all intertangled, then starting again
> from scratch is prob easier ;-)

Unfortunately I've got over 550 files being changed, in probably about 50 
conceptual areas.

It's not going to be fun.

Chris

Re: offtopic: how to break huge patch into smaller independent patches?

[Andreas Dilger]

On Sep 14, 2004  19:24 -0600, Chris Friesen wrote:
> >If the changes are in fairly independant files, just vi'ing the diff is
> >normally very effective. If they're all intertangled, then starting again
> >from scratch is prob easier ;-)
> 
> Unfortunately I've got over 550 files being changed, in probably about 50 
> conceptual areas.

Start with the smallest logical change(s), possibly just extracting them
from the big diff with vi.  Assuming you have a decent diff format (-up
is what I find the most useful) you should just be able to copy the
original patch, delete all the hunks that are unrelated, and you are left
with logical change.

Now that you have it as a separate patch, apply it to the reference tree
and rediff to get a reduced-size combined patch.  For some changes that
stomp on overlapping lines of code you don't have much hope but to recreate
them by hand, but hopefully those are in relatively few areas.

Consider using a source-control tool next time ;-/.  

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/

Re: offtopic: how to break huge patch into smaller independent patches?

[Chris Friesen]

Andreas Dilger wrote:

> Consider using a source-control tool next time ;-/.  

We used a source control tool.  Its just not very useful when people do a port 
from one kernel version to the next and submit it as one giant patch against the 
new kernel rather than new versions of the original individual patches.

I'm the one planning how to avoid this problem in our next development cycle.

Chris

Re: offtopic: how to break huge patch into smaller independent patches?

[Martin J. Bligh]

>> Consider using a source-control tool next time ;-/.  
> 
> We used a source control tool.  Its just not very useful when people do a port from one kernel version to the next and submit it as one giant patch against the new kernel rather than new versions of the original individual patches.
> 
> I'm the one planning how to avoid this problem in our next development cycle.

Having maintained my own tree for a while, what I'd suggest is to keep a
sequence of flat patches, then apply and port then ONE AT A TIME to each
new kernel version. Once you get a system down, it really doesn't take
long to do. 

Personally I find source control tools utterly useless for exactly this 
reason. Akpm uses his own set of tools which someone packaged up as
"quilt" I think. I have a similar set that works by patch number, his
are controlled by a series file. Suggest you look at his tools - if he
can manage a release every few days with several hundred patches in, they
must work pretty well ;-)

M.

Re: offtopic: how to break huge patch into smaller independent patches?

[Arnd Bergmann]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

On Mittwoch, 15. September 2004 07:27, Martin J. Bligh wrote:
> Personally I find source control tools utterly useless for exactly this 
> reason. Akpm uses his own set of tools which someone packaged up as
> "quilt" I think. I have a similar set that works by patch number, his
> are controlled by a series file. Suggest you look at his tools - if he
> can manage a release every few days with several hundred patches in, they
> must work pretty well ;-)

Seconded. I just inherited an equally sized patch from another party
and started looking into quilt for this. It doesn't save you from
understanding the patch, but it works wonders reducing the overhead
of doing the split once you know what you want to end up with.

What I'm doing now is keeping the patch files from quilt in the
CVS tree that also contains the patched kernel itself.

	Arnd <><

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: offtopic: how to break huge patch into smaller independent patches?

[Andrew Morton]

Chris Friesen <cfriesen@nortelnetworks.com> wrote:
>
> Andreas Dilger wrote:
> 
> > Consider using a source-control tool next time ;-/.  
> 
> We used a source control tool.  Its just not very useful when people do a port 
> from one kernel version to the next and submit it as one giant patch against the 
> new kernel rather than new versions of the original individual patches.
> 
> I'm the one planning how to avoid this problem in our next development cycle.
> 

What others said.

Once you apply those patches to your baseline tree you're dead.  If your
primary revision-controlled objects are baseline+patch1+patch2+...+patchN
then life is much simpler when some smarty decides to uprev baseline.