[RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[Ian Campbell]

(CC-ing 2x QEMU maintainers and stable release manager)

The separate trees are a holdover from mercurial, which didn't (at the
time) have a good in-repo branching model.

I propose that Xen 4.6 should be the last release which uses these split
trees and that instead we combine them into just qemu-xen.git (upstream)
and qemu-xen-traditional.git (our old fork), with branches in those repos
to match our releases. I picked those names to match the libxl interface
names for these.

This will result in one place to get all the qemu stable branches for a
given qemu type, without having to remember the naming scheme (which is
counter-intuitive). It will allow easier cherry-picking and produce less
places for users to look for our code etc.

We could put all branches for both in the same tree but I think we want to
just let qemu-xen-trad sit quietly in its own corner. Keeping it separate
reinforces that it is almost completely frozen, also it saves me having to
think up a branch naming scheme within a single repo.

A proposed mapping for the two tree scheme which broadly follows the
xen.git scheme is (paths relative to xenbits git root):

qemu-xen-traditional:

    /staging/qemu-xen-unstable.git#master => /qemu-xen-traditional.git#staging
    /staging/qemu-xen-X.Y-testing.git#master => /qemu-xen-traditional.git#staging-X.Y
    /qemu-xen-unstable.git#master => /qemu-xen-traditional.git#master
    /qemu-xen-X.Y-testing.git#master => /qemu-xen-traditional.git#stable-X.Y

    NB the revision to use is still referenced in xen.git/Config.mk for
    each branch, no change here.

    XXX why do staging/* exist, and what pushes from staging to the other?
    Should we ditch one or the other?

qemu-xen:

    /staging/qemu-upstream-unstable.git#master => /qemu-xen.git#staging
    /staging/qemu-upstream-X.Y-testing.git#master => /qemu-xen.git#staging-X.Y
    /qemu-upstream-unstable.git#master => /qemu-xen.git#master
    /qemu-upstream-X.Y-testing.git#master => /qemu-xen.git#stable-X.Y

    These are push gated by their respective qemu-upstream-* osstest flights.

qemu-mainline:

    Upstream remains git://git.qemu.org/qemu.git. 

    /osstest/qemu.git#mainline/xen-tested-master => /qemu-xen.git#upstream-tested

    /osstest/qemu.git should be removed.

    NB this is the same qemu-xen.git as above.

Below is an osstest patch which implements this proposed scheme (I've not
actually created any trees).

The main open question is what to do about the existing split repos for
existing stable branches. We could:

      * Teach osstest (ap-push) to push to the old tree as well as the new
        for existing (<= 4.6) branches only.
      * Push a Config.mk update to every stable branch and retire the
        existing trees on the next relevant point release, if there is one.

In both cases perhaps combined with tweaking gitweb to omit the old names
from the project list (http://xenbits.xen.org/gitweb/).

Ian.

>From c87f628a7b505d46f2823252c0c3e3ae0fcb736b Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Thu, 30 Jul 2015 13:42:03 +0100
Subject: [PATCH] Switch to merged qemu-xen{,-traditional}.git trees

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 ap-common            |  7 +++----
 ap-fetch-version     | 10 ++++++++--
 ap-fetch-version-old | 12 ++++++++----
 ap-push              | 18 ++++++++++++++----
 4 files changed, 33 insertions(+), 14 deletions(-)

diff --git a/ap-common b/ap-common
index dfeab9e..eb0624e 100644
--- a/ap-common
+++ b/ap-common
@@ -26,9 +26,7 @@
 : ${TREE_XEN:=git://xenbits.xen.org/xen.git}
 : ${PUSH_TREE_XEN:=$XENBITS:/home/xen/git/xen.git}
 
-#: ${TREE_QEMU:=git://mariner.uk.xensource.com/qemu-$xenbranch.git}
-: ${TREE_QEMU:=git://xenbits.xen.org/staging/qemu-$xenbranch.git}
-
+: ${TREE_QEMU:=git://xenbits.xen.org/qemu-xen-traditional.git}
 
 : ${GIT_KERNEL_ORG:=git://git.kernel.org}
 : ${KERNEL_SCM:=${GIT_KERNEL_ORG}/pub/scm/linux/kernel/git}
@@ -87,7 +85,8 @@ fi
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
-: ${TREE_QEMU_UPSTREAM:=git://xenbits.xen.org/staging/qemu-upstream-${xenbranch#xen-}.git}
+: ${TREE_QEMU_UPSTREAM:=git://xenbits.xen.org/qemu-xen.git}
+: ${PUSH_TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-xen.git
 : ${LOCALREV_QEMU_UPSTREAM:=daily-cron.$branch}
 
 : ${TREE_QEMU_MAINLINE:=git://git.qemu.org/qemu.git}
diff --git a/ap-fetch-version b/ap-fetch-version
index 8b41acf..754a398 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -53,9 +53,15 @@ qemu-mainline)
 	repo_tree_rev_fetch_git $branch \
 		$TREE_QEMU_MAINLINE master $LOCALREV_QEMU_UPSTREAM
 	;;
-qemu-upstream-*)
+qemu-upstream-unstable)
         repo_tree_rev_fetch_git $branch \
-		$TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
+		$TREE_QEMU_UPSTREAM staging $LOCALREV_QEMU_UPSTREAM
+        ;;
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
+        repo_tree_rev_fetch_git $branch \
+		$TREE_QEMU_UPSTREAM staging-$branchcore $LOCALREV_QEMU_UPSTREAM
         ;;
 linux)
 	repo_tree_rev_fetch_git linux \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 0b4739b..8f7769a 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -32,8 +32,6 @@ select_xenbranch
 : ${BASE_LOCALREV_SEABIOS:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
 
-: ${BASE_TREE_QEMU_UPSTREAM:=${TREE_QEMU_UPSTREAM/\/staging\//\/}}
-
 if info_linux_tree "$branch"; then
 	repo_tree_rev_fetch_git linux \
 		$BASE_TREE_LINUX_THIS $BASE_TAG_LINUX_THIS $BASE_LOCALREV_LINUX
@@ -61,9 +59,15 @@ qemu-mainline)
         repo_tree_rev_fetch_git $branch \
 		$BASE_TREE_QEMU_MAINLINE mainline/xen-tested-master $LOCALREV_QEMU_MAINLINE
         ;;
-qemu-upstream-*)
+qemu-upstream-unstable)
+        repo_tree_rev_fetch_git $branch \
+		$TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
+        ;;
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
         repo_tree_rev_fetch_git $branch \
-		$BASE_TREE_QEMU_UPSTREAM master $LOCALREV_QEMU_UPSTREAM
+		$TREE_QEMU_UPSTREAM stable-$branchcore $LOCALREV_QEMU_UPSTREAM
         ;;
 linux)
 	repo_tree_rev_fetch_git linux \
diff --git a/ap-push b/ap-push
index 5967b42..9e5edad 100755
--- a/ap-push
+++ b/ap-push
@@ -30,8 +30,7 @@ select_xenbranch
 . ap-common
 
 TREE_LINUX=$PUSH_TREE_LINUX
-TREE_QEMU_MAINLINE=$PUSH_TREE_QEMU_MAINLINE
-TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-upstream-${xenbranch#xen-}.git
+TREE_QEMU_UPSTREAM=$PUSH_TREE_QEMU_UPSTREAM
 TREE_XEN=$PUSH_TREE_XEN
 TREE_LIBVIRT=$PUSH_TREE_LIBVIRT
 TREE_RUMPUSERXEN=$PUSH_TREE_RUMPUSERXEN
@@ -63,12 +62,23 @@ xen-*-testing)
 	;;
 qemu-mainline)
 	cd $repos/qemu-mainline
-	git push $TREE_QEMU_MAINLINE $revision:refs/heads/mainline/xen-tested-master
+	git push $TREE_QEMU_UPSTREAM $revision:refs/heads/upstream-tested
 	;;
-qemu-upstream-*)
+qemu-upstream-unstable)
 	cd $repos/$branch
         git push $TREE_QEMU_UPSTREAM $revision:master
         ;;
+qemu-upstream-*-testing)
+	branchcore=${branch#qemu-upstream-}
+	branchcore=${branchcore%-testing}
+	cd $repos/$branch
+        git push $TREE_QEMU_UPSTREAM $revision:stable-$branchcore
+	# Maybe push to historical stable branches too?
+	#case "$branch" in
+	#qemu-upstream-4.[23456]-testing)
+	#...XXX...
+	#esac
+        ;;
 linux-3.*)
 	cd $repos/linux
 	git push $TREE_LINUX $revision:tested/$branch
-- 
2.1.4

Re: [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[Ian Jackson]

Ian Campbell writes ("[RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad"):
...
> qemu-xen-traditional:
...
>     XXX why do staging/* exist, and what pushes from staging to the other?
>     Should we ditch one or the other?

Once upon a time, some of these had their own push gate.  They are
nowadays always pushed together.  The staging branches should be
abolished.

> The main open question is what to do about the existing split repos for
> existing stable branches. We could:
> 
>       * Teach osstest (ap-push) to push to the old tree as well as the new
>         for existing (<= 4.6) branches only.
>       * Push a Config.mk update to every stable branch and retire the
>         existing trees on the next relevant point release, if there is one.

We could do both of these, so we have a transitional period.

Ian.

Re: [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[Andrew Cooper]

On 30/07/15 15:22, Ian Campbell wrote:
> (CC-ing 2x QEMU maintainers and stable release manager)
>
> The separate trees are a holdover from mercurial, which didn't (at the
> time) have a good in-repo branching model.
>
> I propose that Xen 4.6 should be the last release which uses these split
> trees and that instead we combine them into just qemu-xen.git (upstream)
> and qemu-xen-traditional.git (our old fork), with branches in those repos
> to match our releases. I picked those names to match the libxl interface
> names for these.

Very much +1 with my XenServer hat on.

>
> This will result in one place to get all the qemu stable branches for a
> given qemu type, without having to remember the naming scheme (which is
> counter-intuitive). It will allow easier cherry-picking and produce less
> places for users to look for our code etc.
>
> We could put all branches for both in the same tree but I think we want to
> just let qemu-xen-trad sit quietly in its own corner. Keeping it separate
> reinforces that it is almost completely frozen, also it saves me having to
> think up a branch naming scheme within a single repo.
>
> A proposed mapping for the two tree scheme which broadly follows the
> xen.git scheme is (paths relative to xenbits git root):
>
> qemu-xen-traditional:
>
>     /staging/qemu-xen-unstable.git#master => /qemu-xen-traditional.git#staging
>     /staging/qemu-xen-X.Y-testing.git#master => /qemu-xen-traditional.git#staging-X.Y
>     /qemu-xen-unstable.git#master => /qemu-xen-traditional.git#master
>     /qemu-xen-X.Y-testing.git#master => /qemu-xen-traditional.git#stable-X.Y

And take the change to drop the other random branches in the repository 
By my observations:

$ git remote show upstream
* remote upstream
  ...
  Remote branches:
    ide-write-cache
    iwj.block-extendable-flag
    origin
    qemu
    upstream
    vga-reverse-merge
    xen.netscriptenv


>
>     NB the revision to use is still referenced in xen.git/Config.mk for
>     each branch, no change here.
>
>     XXX why do staging/* exist, and what pushes from staging to the other?
>     Should we ditch one or the other?

This is how staging used to work for the mecurial xen trees.  I presume
therefore that whatever used to be done for the Xen trees are still
being done for the current qemu trees.

>
> qemu-xen:
>
>     /staging/qemu-upstream-unstable.git#master => /qemu-xen.git#staging
>     /staging/qemu-upstream-X.Y-testing.git#master => /qemu-xen.git#staging-X.Y
>     /qemu-upstream-unstable.git#master => /qemu-xen.git#master
>     /qemu-upstream-X.Y-testing.git#master => /qemu-xen.git#stable-X.Y
>
>     These are push gated by their respective qemu-upstream-* osstest flights.
>
> qemu-mainline:
>
>     Upstream remains git://git.qemu.org/qemu.git. 
>
>     /osstest/qemu.git#mainline/xen-tested-master => /qemu-xen.git#upstream-tested
>
>     /osstest/qemu.git should be removed.
>
>     NB this is the same qemu-xen.git as above.
>
> Below is an osstest patch which implements this proposed scheme (I've not
> actually created any trees).

LGTM.

Also suitable adjustments to the landing page at http://xenbits.xen.org/

~Andrew

Re: [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[Ian Campbell]

On Thu, 2015-07-30 at 15:33 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[RFC for-4.7] Switching to a single qemu tree each 
> per qemu-xen and qemu-trad"):
> ...
> > qemu-xen-traditional:
> ...
> >     XXX why do staging/* exist, and what pushes from staging to the 
> > other?
> >     Should we ditch one or the other?
> 
> Once upon a time, some of these had their own push gate.  They are
> nowadays always pushed together.  The staging branches should be
> abolished.

Thanks, I suspect as much.

> > The main open question is what to do about the existing split repos for
> > existing stable branches. We could:
> > 
> >       * Teach osstest (ap-push) to push to the old tree as well as the 
> > new
> >         for existing (<= 4.6) branches only.
> >       * Push a Config.mk update to every stable branch and retire the
> >         existing trees on the next relevant point release, if there is 
> > one.
> 
> We could do both of these, so we have a transitional period.

That could work, yes.

e.g. perhaps do the dual push until the corresponding point release
happens?

Re: [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[Vitaly Kuznetsov]

Ian Campbell <ian.campbell@citrix.com> writes:

> (CC-ing 2x QEMU maintainers and stable release manager)
>
> The separate trees are a holdover from mercurial, which didn't (at the
> time) have a good in-repo branching model.
>
> I propose that Xen 4.6 should be the last release which uses these split
> trees and that instead we combine them into just qemu-xen.git (upstream)
> and qemu-xen-traditional.git (our old fork)

(sorry my question is a bit unrelated)

is there a proposed time frame for the retirement of
qemu-xen-traditional? http://wiki.xenproject.org/wiki/QEMU_Upstream only
mentions VGA passthrough as a missing feature but maybe it can be fixed
for 4.7?

-- 
  Vitaly

Re: [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[Ian Campbell]

On Thu, 2015-07-30 at 16:51 +0200, Vitaly Kuznetsov wrote:
> Ian Campbell <ian.campbell@citrix.com> writes:
> 
> > (CC-ing 2x QEMU maintainers and stable release manager)
> > 
> > The separate trees are a holdover from mercurial, which didn't (at the
> > time) have a good in-repo branching model.
> > 
> > I propose that Xen 4.6 should be the last release which uses these 
> > split
> > trees and that instead we combine them into just qemu-xen.git 
> > (upstream)
> > and qemu-xen-traditional.git (our old fork)
> 
> (sorry my question is a bit unrelated)
> 
> is there a proposed time frame for the retirement of
> qemu-xen-traditional? http://wiki.xenproject.org/wiki/QEMU_Upstream only
> mentions VGA passthrough as a missing feature but maybe it can be fixed
> for 4.7?

AIUI people (from Intel?) are working on that.

Due to Windows activation issues which are triggered by changing the h/w
platform (aka QEMU) under the guest we are taking a long tail approach here
and not actively looking to remove qemu-xen-traditional so that people with
VMs installed under old versions can choose the stick with that h/w
platform even on newer Xen.

Eventually, some long time down the line, we may decide that the chances
that anyone is still running such a VM is negligible and we can remove it.

In the meantime qemu-xen-traditional is essentially frozen, no new features
and only adjustments to e.g. keep up with changes to the control libraries
etc are made.

Ian.

Re: [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[Ian Jackson]

Andrew Cooper writes ("Re: [Xen-devel] [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad"):
> And take the change to drop the other random branches in the repository 
> By my observations:
> 
> $ git remote show upstream
> * remote upstream
>   ...
>   Remote branches:
>     ide-write-cache
>     iwj.block-extendable-flag
>     origin
>     qemu
>     upstream
>     vga-reverse-merge
>     xen.netscriptenv

Yes, all of that can be got rid of.  I'm not even sure how much of
that got there.

Ian.

Re: [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[George Dunlap]

On Thu, Jul 30, 2015 at 3:51 PM, Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
> Ian Campbell <ian.campbell@citrix.com> writes:
>
>> (CC-ing 2x QEMU maintainers and stable release manager)
>>
>> The separate trees are a holdover from mercurial, which didn't (at the
>> time) have a good in-repo branching model.
>>
>> I propose that Xen 4.6 should be the last release which uses these split
>> trees and that instead we combine them into just qemu-xen.git (upstream)
>> and qemu-xen-traditional.git (our old fork)
>
> (sorry my question is a bit unrelated)
>
> is there a proposed time frame for the retirement of
> qemu-xen-traditional? http://wiki.xenproject.org/wiki/QEMU_Upstream only
> mentions VGA passthrough as a missing feature but maybe it can be fixed
> for 4.7?

We also don't yet have a good story for stub domains; and there are
some rather hacky things related to PCI pass-through that still have
to be worked around in qemu-upstream. (Namely, hvmloader cannot resize
the MMIO hole because it can't tell qemu about the resize; so when
passing through a device that doesn't fit in the default hole size,
you have to manually specify the size in the config file.)

We're working on stub domains with rump kernels, but we're not there yet.

 -George

Re: [RFC for-4.7] Switching to a single qemu tree each per qemu-xen and qemu-trad

[Jan Beulich]

>>> On 30.07.15 at 16:22, <ian.campbell@citrix.com> wrote:
> (CC-ing 2x QEMU maintainers and stable release manager)
> 
> The separate trees are a holdover from mercurial, which didn't (at the
> time) have a good in-repo branching model.
> 
> I propose that Xen 4.6 should be the last release which uses these split
> trees and that instead we combine them into just qemu-xen.git (upstream)
> and qemu-xen-traditional.git (our old fork), with branches in those repos
> to match our releases.

Yes please!

Jan