[ANNOUNCE] kpatchup 0.02 kernel patching script

[ANNOUNCE] kpatchup 0.02 kernel patching script

[Matt Mackall]

This is the first release of kpatchup, a script for managing switching
between kernel releases via patches with some smarts:

 - understands -pre and -rc version numbering
 - aware of various external trees
 - automatically patch between any tree in an x.y release
 - automatically download and cache patches on demand
 - automatically determine the latest patch in various series
 - optionally print version strings or URLs for patches

Currently it knows about 2.4, 2.4-pre, 2.6, 2.6-pre, 2.6-bk, 2.6-mm,
and 2.6-tiny.

Example usage:

 $ head Makefile
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 2
 EXTRAVERSION =-rc2
 [...]
 $ kpatchup 2.6-mm
 2.6.2-rc2 -> 2.6.4-rc1-mm1
 Applying patch-2.6.2-rc2.bz2 -R
 Applying patch-2.6.2.bz2
 Applying patch-2.6.3.bz2
 Downloading patch-2.6.4-rc1.bz2...
 Applying patch-2.6.4-rc1.bz2
 Downloading 2.6.4-rc1-mm1.bz2...
 Applying 2.6.4-rc1-mm1.bz2
 $ head Makefile
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 4
 EXTRAVERSION =-rc1-mm1
 NAME=Feisty Dunnart
 [...] 
 $ kpatchup -q 2.6.3-rc1
 $ head Makefile
 VERSION = 2
 PATCHLEVEL = 6
 SUBLEVEL = 3
 EXTRAVERSION =-rc1
 NAME=Feisty Dunnart
 [...]
 $ kpatchup -s 2.6-bk
 2.6.4-rc1-bk3
 $ kpatchup -u 2.4-pre
 http://www.kernel.org/pub/linux/kernel/v2.4/testing/patch-2.4.26-pre1.bz2


This is an alpha release for people to experiment with. Feedback and
patches encouraged. Grab your copy today at:

 http://selenic.com/kpatchup/

-- 
Matt Mackall : http://www.selenic.com : Linux development and consulting

Re: [ANNOUNCE] kpatchup 0.02 kernel patching script

[Zwane Mwaikambo]

On Tue, 2 Mar 2004, Matt Mackall wrote:

> This is the first release of kpatchup, a script for managing switching
> between kernel releases via patches with some smarts:
>
>  - understands -pre and -rc version numbering
>  - aware of various external trees
>  - automatically patch between any tree in an x.y release
>  - automatically download and cache patches on demand
>  - automatically determine the latest patch in various series
>  - optionally print version strings or URLs for patches
>
> Currently it knows about 2.4, 2.4-pre, 2.6, 2.6-pre, 2.6-bk, 2.6-mm,
> and 2.6-tiny.

Oh i definitely owe you one now, this is replacing the ugly shell script i
had before, i'm mostly using this now to download and patch up trees
before cvs import'ing them.

Thanks!

Re: [ANNOUNCE] kpatchup 0.02 kernel patching script

[Rusty Russell]

[-- Attachment #1: Type: text/plain, Size: 433 bytes --]

On Wed, 2004-03-03 at 13:24, Matt Mackall wrote: 
> This is the first release of kpatchup, a script for managing switching
> between kernel releases via patches with some smarts:

Hi Matt!

Please find below my grab_kernel, latest-kernel-version and lkvercmp
scripts, in case they are useful for you to merge.  Kinda
rough, like any unexposed code...

Rusty


-- 
Anyone who quotes me in their signature is an idiot -- Rusty Russell

[-- Attachment #2: lkvercmp --]
[-- Type: text/x-sh, Size: 2645 bytes --]

#! /bin/sh
# Compare multiple kernel versions

# Return more recent of two kernel versions.
if [ $# -eq 0 ]; then
    echo Usage: $0 kernelversion kernelversion ... >&2
    echo Returns greatest of the versions. >&2
    exit 1
fi

unknown()
{
    # Does a -dj kernel come before a -ac kernel?  Or a Linus kernel?
    # Undefined.
    echo Can\'t compare extension \'"$1"\'. >&2
    exit 1
}

firstpart()
{
    echo $1 | sed 's/[-.].*//'
}

lastpart()
{
    echo $1 | sed 's/.*[-.]//'
}

trimlastpart()
{
    echo $1 | sed 's/[-.][^-.]*$//'
}

trimfirstpart()
{
    echo $1 | sed 's/^[^-.]*[-.]//'
}

letters()
{
    echo $1 | tr -d '[0-9]'
}

numbers()
{
    echo $1 | tr -dc '[0-9]'
}

if [ $# -eq 1 ]; then
    echo "$1"
    exit 0
fi

if [ $# -gt 2 ]; then
    FIRST=$1
    shift
    set -- $FIRST $($0 "$@")
fi

VER1=$1
VER2=$2

# Vanity kernels (at end): handle 2.6.3-mm2 vs 2.6.3-rc1-mm3 by comparing
# base first.
TAIL1=$(lastpart $VER1)
TAIL2=$(lastpart $VER2)
if [ x"$(letters $TAIL1)" = x"$(letters $TAIL2)" ]; then
    case "$TAIL1" in
    bk*|rc*|pre*) : ;;
    *)
	BASE1=$(trimlastpart $VER1)
	BASE2=$(trimlastpart $VER2)
	if [ x"$BASE1" = x"$BASE2" ]; then
	    if [ $(numbers $TAIL1) -gt $(numbers $TAIL2) ]; then
		echo $1
	    else
		echo $2
	    fi
	else
	    if [ x$(lkvercmp $BASE1 $BASE2) = x$BASE1 ]; then
		echo $1
	    else
		echo $2
	    fi
	fi
	exit
	;;
   esac
fi

while true; do
    PART1=$(firstpart $VER1)
    PART2=$(firstpart $VER2)
    VER1=$(trimfirstpart $VER1)
    VER2=$(trimfirstpart $VER2)

    PART1LETTERS=$(letters $PART1)
    PART2LETTERS=$(letters $PART2)
    PART1NUMBERS=$(numbers $PART1)
    PART2NUMBERS=$(numbers $PART2)

    if [ "$PART1LETTERS" = "$PART2LETTERS" ]; then
	if [ "$PART1NUMBERS" = "$PART2NUMBERS" ]; then
	    continue;
	else
	    if [ "$PART1NUMBERS" -gt "$PART2NUMBERS" ]; then
		echo $1; exit
	    else
		echo $2; exit
	    fi
        fi
    fi

    case x"$PART1LETTERS" in
    xbk)
        # bk snapshots come after (ie. 2.6.1-bk1 is > 2.6.1).
	# So this always wins.
	case x"$PART2LETTERS" in
	x|xrc|xpre) echo $1; exit;;
	esac
	unknown $PART2LETTERS;;
    x)
	# Final beats everything except bk.
	case x"$PART2LETTERS" in
	xbk) echo $2; exit;;
	xrc|xpre) echo $1; exit;;
	esac
	unknown $PART2LETTERS;;
    xrc)
	# Only a final or bk can win.
	case x"$PART2LETTERS" in
        x|xbk) echo $2; exit;;
	xpre) echo $1; exit;;
        esac
	unknown $PART2LETTERS;;
    xpre)
        # Final, bk or rc can win.
	case x"$PART2LETTERS" in
        x|xbk|xrc) echo $2; exit;;
	xpre) echo $1; exit;;
        esac
	unknown $PART2LETTERS;;
    *)
	unknown $PART1LETTERS;;
    esac
done

[-- Attachment #3: latest-kernel-version --]
[-- Type: text/x-sh, Size: 1632 bytes --]

#! /bin/sh -e
# Latest kernel version of this vintage.

KERNELDIR=~/devel/kernel

# Return the latest kernel versions.
get_kernel_vers()
{
#     if [ x"$GK_KERNS" = x ]; then
# 	GK_KERNS=$(finger @finger.kernel.org | tail +2 | cut -s -d: -f2- | sed 's/[ 	]*//')
#     fi
    if [ x"$GK_KERNS" = x ]; then
	GK_KERNS=$(lynx -width=500 -dump http://www.kernel.org/kdist/finger_banner | tail +2 | cut -s -d: -f2- | sed 's/[ 	]*//')
    fi
    if [ x"$GK_KERNS" = x ]; then
	echo Failed to get kernel list from kernel.org >&2
	exit 1
    fi
    # Vanity are there too: only take recognized patches.
    # Ignore 2.4-bk versions: I don't know where they're coming from 8(
    echo "$GK_KERNS" # | egrep -- "$1|-rc|-bk|-pre|^[0-9]*\.[0-9]*\.[0-9]*\$" | grep -v '2\.4\.[0-9]*-bk[0-9]*'
}


PROBE=0
if [ x"$1" = x--probe ]; then
    PROBE=1
    shift
fi

case "$1" in --*)
    echo "Usage: $0 [--probe] [kernel-base-version]" >&2
    echo "Returns the latest version on the system, or with --probe," >&2
    echo "returns the latest version on network." >&2
    exit 1
    ;;
esac

if [ $# -eq 0 ]; then set .; fi

if [ $PROBE = 1 ]; then
    CANDIDATES=$(get_kernel_vers)
else
    CANDIDATES=$(cd $KERNELDIR && ls -d linux-* | fgrep -v '.tmp' | grep -v -- '-ppc' | sed 's/^linux-//')
fi

for w; do
    case "$w" in
	# By default, ignore any vanity kernels.
	.) FILTER="^[0-9.]*(-rc[0-9]*|-bk[0-9]*|-pre[0-9]*)*\$";;
	# eg -mm.
	-*) FILTER="^[0-9.]*(-rc[0-9]*|-bk[0-9]*|-pre[0-9]*)*$w[0-9]*\$";;
	# eg. 2.4
	*) FILTER="^$w[0-9.]*(-rc[0-9]*|-bk[0-9]*|-pre[0-9]*)*\$";;
    esac

    lkvercmp $(echo "$CANDIDATES" | egrep -- "$FILTER")
done



[-- Attachment #4: grab-kernel --]
[-- Type: text/x-sh, Size: 5453 bytes --]

#! /bin/sh -e
# Script to grab a given kernel version (stolen from Virtual Anton).

barf()
{
    echo "$@" >&2
    exit 1
}

PATCHDIR=~/devel/kernel/kernel-patches
MIRRORS="rsync://kernel::kernel http://kernel.linux.pacific.net.au/linux/kernel/ rsync://master.kernel.org:/pub/linux/kernel"
#MIRRORS="rsync.planetmirror.com::ftp.kernel.org/pub/linux/kernel master.kernel.org:/pub/linux/kernel"

# Takes destination directory, and filename, tries each mirror.
get_file()
{
    get_file_dir=$1
    get_file_name=$2

    # Patch there already?
    if [ -f "$get_file_dir"/"$(basename "$get_file_name")" ]; then
	echo " Already have patch..."
	return 0
    fi

    for get_file_server in $MIRRORS; do
	echo Trying $get_file_server...
	case $get_file_server in
	rsync://*) get_file_s=`echo $get_file_server | cut -c9-`;
		  rsync $get_file_s/$get_file_name $get_file_dir && return 0
		  ;;
	http://*) get_file_s=`echo $get_file_server | cut -c8-`;
		wget -O $get_file_dir/`basename $get_file_name` $get_file_s/$get_file_name && return 0
		# Leaves a zero-len file on failure 8(
		rm -f $get_file_dir/`basename $get_file_name`
		;;
	esac
    done
    return 1
}

if [ $# -ne 2 ]; then
    echo Usage: $0 version kerneldir >&2
    exit 1
fi

VERSION=$1
KDIR=$2

if [ -d $KDIR/linux-$VERSION ]; then exit 0; fi
MAJOR=`echo $VERSION | cut -d. -f1-2`

# Given vanity kernel name, major version, and previous version, produce pathname
vanity_source()
{
    case "$1" in
    *-ac*)
	echo people/alan/linux-$2/$3/patch-$1.bz2;;
    *-dj*)
	echo people/davej/patches/$2/$3/patch-$1.bz2;;
    *-mm*)
	echo people/akpm/patches/$2/$3/$1/$1.bz2;;
    *)
	barf "Unknown kernel series $1";;
    esac
}

# Given base name, new name and patch, do patch application.
apply_patch()
{
    rm -rf $KDIR/.linux-$2
    cp -al $KDIR/linux-$1 $KDIR/.linux-$2 ||
	    barf "Couldn't clone $1 dir"
    bunzip2 -t $3 || barf "Corrupt patch $3"
    bzcat $3 | patch -s -d $KDIR/.linux-$2 -p1 ||
	    barf "Problem patching new kernel $2"
    mv $KDIR/.linux-$2 $KDIR/linux-$2
}

echo Grabbing $VERSION
case "$VERSION" in
    # Bitkeeper snapshot
    *-bk*)
	if [ -d $KDIR/linux-$VERSION ]; then exit 0; fi

	PREVIOUS=`echo "$VERSION" | sed 's/-bk.*//'`
	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/snapshots/patch-$VERSION.bz2  || barf "Couldn't get v$MAJOR/snapshots/patch-$VERSION.bz2"
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;

    # Various vanity patches
    *-mm[0-9]*|*-ac[0-9]*|*-dj[0-9]*)
	# First make sure we have base tree
	BASE_KERNEL=`echo "$VERSION" | sed -e 's/-[a-z][a-z][0-9]*$//'`
	$0 "$BASE_KERNEL" $KDIR	||
	    barf "Couldn't get $BASE_KERNEL to build $VERSION"
	VANITY=`vanity_source $VERSION $MAJOR $BASE_KERNEL`
	get_file $PATCHDIR $VANITY ||
		barf "Couldn't fetch $VANITY"

	apply_patch $BASE_KERNEL $VERSION $PATCHDIR/`basename $VANITY`
	;;

    # *-pre*: This refers to the Linus or Marcelo pre-patches.
    *-pre*|*-rc*)
	# Linus or Marcelo pre-patch.
	if [ -d $KDIR/linux-$VERSION ]; then exit 0; fi

	PREVIOUS=`echo "$VERSION" | sed 's/-\(pre\|rc\).*//'`
	LAST_VER=`echo "$PREVIOUS" | cut -d. -f3`
	PREVIOUS=`echo "$PREVIOUS" | cut -d. -f1,2`.`expr $LAST_VER - 1 || true`
	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/testing/patch-$VERSION.bz2 ||
	    get_file $PATCHDIR v$MAJOR/testing/old/patch-$VERSION.bz2 ||
	    barf "Couldn't get v$MAJOR/testing/patch-$VERSION.bz2"
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;

    # Linus 2.6.0 test patch.
    2.6.0-test*)
	VER=`echo "$VERSION" | sed 's/.*test//'`
	PREVIOUS=2.6.0-test`expr $VER - 1 || true`

	if [ $PREVIOUS = 2.6.0-test0 ]; then
	    PREVIOUS=2.5.75
	fi

	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/patch-$VERSION.bz2 ||
	    get_file $PATCHDIR v$MAJOR/old/patch-$VERSION.bz2 ||
	    barf "Couldn't get v$MAJOR/patch-$VERSION.bz2"
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;

    2.6.0)
	PREVIOUS=2.6.0-test11

	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/patch-$VERSION.bz2 ||
	    get_file $PATCHDIR v$MAJOR/old/patch-$VERSION.bz2 ||
	    barf "Couldn't get v$MAJOR/patch-$VERSION.bz2"
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;

    # *.0: This refers to the base of the Linus kernels, fetched whole.
    *.0)
	# Linus' root of all trees.
	get_file $PATCHDIR v$MAJOR/linux-$VERSION.tar.bz2 || barf "Couldn't get  v$MAJOR/linux-$VERSION.tar.bz2"
	bzcat $PATCHDIR/linux-$VERSION.tar.bz2 | (cd $KDIR && tar xf -) ||
	    barf "Problem unpacking linux-$VERSION.tar.bz2"
	mv $KDIR/linux $KDIR/linux-$VERSION
	;;

    #   [1-9].*.*: This refers to the any other Linus kernel.
    [1-9].*.*[0-9])
	# Linus standard ?
	if [ -d $KDIR/linux-$VERSION ]; then exit 0; fi

	LAST_VER=`echo "$VERSION" | cut -d. -f3`
	PREVIOUS=`echo "$VERSION" | cut -d. -f1,2`.`expr $LAST_VER - 1 || true`
	$0 "$PREVIOUS" $KDIR || barf "Couldn't get $PREVIOUS to build $VERSION"

	# Grab patch
	get_file $PATCHDIR v$MAJOR/patch-$VERSION.bz2 || barf "Couldn't get v$MAJOR/patch-$VERSION.bz2"
	trap "rm -rf $KDIR/.linux-$VERSION" 0
	apply_patch $PREVIOUS $VERSION $PATCHDIR/patch-$VERSION.bz2
	;;
    *)
	barf "Unknown kernel version $VERSION"
	;;
esac

Re: [ANNOUNCE] kpatchup 0.02 kernel patching script

[Dave Hansen]

On Tue, 2004-03-02 at 18:24, Matt Mackall wrote:
> This is an alpha release for people to experiment with. Feedback and
> patches encouraged. Grab your copy today at:

First of all, very nice script.

But, it doesn't look like it properly handles empty directories.  I
tried this command, this morning, and it blew up.  I think it's because
this directory http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/ is
empty because of last night's 2.6.4-rc2 release.  I don't grok python
very well but is the "return p[-1]" there just to cause a fault like
this?  Would it be better if it just returned a "no version of that
patch right now" message and exited nicely?

[dave@nighthawk linux-2.6]$ kpatchup-0.02 2.6-bk
"Traceback (most recent call last):
  File "/home/dave/bin/kpatchup-0.02", line 283, in ?
    b = find_ver(args[0])
  File "/home/dave/bin/kpatchup-0.02", line 240, in find_ver
    return v[0](os.path.dirname(v[1]), v[2])
  File "/home/dave/bin/kpatchup-0.02", line 147, in latest_dir
    return p[-1]
IndexError: list index out of range

I think your script, combined with Rusty's latest-kernel-version could
make me a very happy person.  

-- dave

Re: [ANNOUNCE] kpatchup 0.02 kernel patching script

[Matt Mackall]

On Thu, Mar 04, 2004 at 09:22:02AM -0800, Dave Hansen wrote:
> On Tue, 2004-03-02 at 18:24, Matt Mackall wrote:
> > This is an alpha release for people to experiment with. Feedback and
> > patches encouraged. Grab your copy today at:
> 
> First of all, very nice script.
> 
> But, it doesn't look like it properly handles empty directories.  I
> tried this command, this morning, and it blew up.  I think it's because
> this directory http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/ is
> empty because of last night's 2.6.4-rc2 release.  I don't grok python
> very well but is the "return p[-1]" there just to cause a fault like
> this?  Would it be better if it just returned a "no version of that
> patch right now" message and exited nicely?

Python does a good job at falling over loudly whenever anything
unexpected happens. I hadn't noticed that the snapshot directory got
purged. Hmmm. The right thing is to make it fall back to checking old/
where the most recent -bk is to be found. Like this:

$ kpatchup -s 2.6-pre
2.6.4-rc2
$ kpatchup -s 2.6-bk
2.6.4-rc1-bk4

New version at http://selenic.com/kpatchup/kpatchup-0.03

I've added a couple other niceties for scripting purposes: a -p option
which will report the "base" version for a given version, -m which
will parse Makefile and print the version therein.

> I think your script, combined with Rusty's latest-kernel-version could
> make me a very happy person.  

I skimmed latest-kernel-version, is it doing something my -s option
doesn't do yet?

-- 
Matt Mackall : http://www.selenic.com : Linux development and consulting

Re: [ANNOUNCE] kpatchup 0.02 kernel patching script

[Dave Hansen]

On Thu, 2004-03-04 at 10:35, Matt Mackall wrote:
> I skimmed latest-kernel-version, is it doing something my -s option
> doesn't do yet?

$ ./kpatchup-0.03 -s 2.6-bk
2.6.4-rc1-bk4
$ ./kpatchup-0.03 -s 2.6
2.6.3
$ ./kpatchup-0.03 -s 2.6-pre
2.6.4-rc2
$ latest-kernel-version --probe
2.6.4-rc2

I might be misunderstanding the options, but 'kpatchup -s' I think is
limited to giving the latest version of a single tree, while
'latest-kernel-version' will look at several different trees.  This is a
tiny problem for kpatchup because it treats 2.6, 2.6-bk, and 2.6-pre as
separate trees.  For my use, I just want the latest snapshot, whether
it's a bk snapshot, or one of the real point releases.  

What I'm doing to work around it is this:
kpatchup `latest-kernel-version --probe`

-- dave

Re: [ANNOUNCE] kpatchup 0.02 kernel patching script

[Matt Mackall]

On Thu, Mar 04, 2004 at 10:50:10AM -0800, Dave Hansen wrote:
> On Thu, 2004-03-04 at 10:35, Matt Mackall wrote:
> > I skimmed latest-kernel-version, is it doing something my -s option
> > doesn't do yet?
> 
> $ ./kpatchup-0.03 -s 2.6-bk
> 2.6.4-rc1-bk4

This is actually broken at the moment as it won't know how to fetch
stuff from old/ when it's given a non-current bk version. I have a
design assumption that I can go from a version name to a URL without
any lookups, which the snapshot dir purging breaks.

ISTR Jeff maintains the snapshot scripts. Jeff, could you possibly
have it put a copy of the each snapshot in old/ at creation time
rather than moving stuff there at a later point? This will make URLs
pointing to old/ for -bk patches stable but shouldn't break anything
else.

> $ ./kpatchup-0.03 -s 2.6
> 2.6.3
> $ ./kpatchup-0.03 -s 2.6-pre
> 2.6.4-rc2
> $ latest-kernel-version --probe
> 2.6.4-rc2
> 
> I might be misunderstanding the options, but 'kpatchup -s' I think is
> limited to giving the latest version of a single tree, while
> 'latest-kernel-version' will look at several different trees.  This is a
> tiny problem for kpatchup because it treats 2.6, 2.6-bk, and 2.6-pre as
> separate trees.  For my use, I just want the latest snapshot, whether
> it's a bk snapshot, or one of the real point releases.  

I suppose I could add a 2.6-tip, which will return the greatest of
2.6, 2.6-pre, and 2.6-bk. Like this:

$ kpatchup -s 2.6-tip
2.6.4-rc2
$ kpatchup -u 2.6-tip
http://www.kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.4-rc2.bz2

-- 
Matt Mackall : http://www.selenic.com : Linux development and consulting

Re: [ANNOUNCE] kpatchup 0.02 kernel patching script

[Dave Hansen]

On Thu, 2004-03-04 at 11:09, Matt Mackall wrote:
> I suppose I could add a 2.6-tip, which will return the greatest of
> 2.6, 2.6-pre, and 2.6-bk. Like this:
> 
> $ kpatchup -s 2.6-tip
> 2.6.4-rc2
> $ kpatchup -u 2.6-tip
> http://www.kernel.org/pub/linux/kernel/v2.6/testing/patch-2.6.4-rc2.bz2

That would at least keep me from mixing different tools like I am now. 
It's not too big of a deal.

-- dave