ioemu

ioemu

[Christoph Egger]

Hi,

is it possible to get ioemu-remote into mercurial?

When I packaged Xen 3.3.0 for NetBSD, I run into the problem that
the git tree could not checked against a checksum because
a) the download happens during the build phase.
b) the checksum check works against one file but not against a whole tree
The checksum phase happens right after the download of the
Xen 3.3.0 source tarball.
I think, Gentoo Linux has the same problem.

I worked around this by using the in-tree ioemu.

In this respect, I would like to know if the move to the new ioemu can be done
by updating the ioemu code in the mercurial tree by taking over the sources 
from the ioemu-remote tree into mercural tree.

This would also allow to create snapshot packages from unstable
via hg archive -t tgz xen-snapshot.tar.gz in a half-automated way.

Christoph

-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy

Re: ioemu

[Ian Jackson]

Christoph Egger writes ("[Xen-devel] ioemu"):
> is it possible to get ioemu-remote into mercurial?

No.  Well, it would be possible, but it wouldn't be desirable.

> When I packaged Xen 3.3.0 for NetBSD, I run into the problem that
> the git tree could not checked against a checksum because
> a) the download happens during the build phase.
> b) the checksum check works against one file but not against a whole tree
> The checksum phase happens right after the download of the
> Xen 3.3.0 source tarball.
> I think, Gentoo Linux has the same problem.

I'm not sure I understand what checksum you're referring to here.  Is
it part of the NetBSD ports system ?  What does the ports system
expect ?

> I worked around this by using the in-tree ioemu.

How about using the official 3.3.0 tarball which contains a copy of
ioemu-remote ?

> In this respect, I would like to know if the move to the new ioemu
> can be done by updating the ioemu code in the mercurial tree by
> taking over the sources from the ioemu-remote tree into mercural
> tree.

We won't be doing this.  The point of using git rather than hg is to
much more easily manage the interactions and patch workflows between
the various branches of qemu, of which ioemu-remote is just one.

Dealing well with this kind of forked up trainwreck requires a lot of
heavy lifting from the revision control system.  git can do this
(despite the appalling user interface) and hg can't.

> This would also allow to create snapshot packages from unstable
> via hg archive -t tgz xen-snapshot.tar.gz in a half-automated way.

I agree it is a shame that our official tarball isn't made entirely
mechanically.  If you would care to contribute a script that
reproduces the 3.3.0 tarball when dropped into the appropriate
xen-3.3-testing changeset, and also does a sane thing in currently
xen-unstable, we'll consider including it and using it next time.

Ian.

Re: ioemu

[Christoph Egger]

On Monday 01 September 2008 12:05:36 Ian Jackson wrote:
> Christoph Egger writes ("[Xen-devel] ioemu"):
> > is it possible to get ioemu-remote into mercurial?
>
> No.  Well, it would be possible, but it wouldn't be desirable.

Why not? Everyone here is used to use mercurial.
Also the testing environments are bound on it. See below.

> > When I packaged Xen 3.3.0 for NetBSD, I run into the problem that
> > the git tree could not checked against a checksum because
> > a) the download happens during the build phase.
> > b) the checksum check works against one file but not against a whole tree
> > The checksum phase happens right after the download of the
> > Xen 3.3.0 source tarball.
> > I think, Gentoo Linux has the same problem.
>
> I'm not sure I understand what checksum you're referring to here.  Is
> it part of the NetBSD ports system ?  What does the ports system
> expect ?

The checksum is part of the package in pkgsrc. It needs an URL where
to download the source archive. After the download, the archive is verified
against SHA1 and RMD160 checksums and against its size in bytes.

Gentoo has taken over this concept when it was founded. And it is still
doing so.

> > I worked around this by using the in-tree ioemu.
>
> How about using the official 3.3.0 tarball which contains a copy of
> ioemu-remote ?

It doesn't compile on BSD (and hasn't got the testing).

> > In this respect, I would like to know if the move to the new ioemu
> > can be done by updating the ioemu code in the mercurial tree by
> > taking over the sources from the ioemu-remote tree into mercural
> > tree.
>
> We won't be doing this.  The point of using git rather than hg is to
> much more easily manage the interactions and patch workflows between
> the various branches of qemu, of which ioemu-remote is just one.
>
> Dealing well with this kind of forked up trainwreck requires a lot of
> heavy lifting from the revision control system.  git can do this
> (despite the appalling user interface) and hg can't.

Our testing infrastructure is based on changeset numbers. From reading
the number you immediately know is this an old or new changeset and
you immediately know how many changesets are between two changesets.
This makes it easy in finding a changeset which introduced a bug by bisecting.

With git using hash numbers no longer works.
(Ever tried to remember on the hash number you worked on at last for
one hour?)

> > This would also allow to create snapshot packages from unstable
> > via hg archive -t tgz xen-snapshot.tar.gz in a half-automated way.
>
> I agree it is a shame that our official tarball isn't made entirely
> mechanically.  If you would care to contribute a script that
> reproduces the 3.3.0 tarball when dropped into the appropriate
> xen-3.3-testing changeset, and also does a sane thing in currently
> xen-unstable, we'll consider including it and using it next time.

So you are planning to also move the hypervisor to git in the middle/long run?

Christoph

-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy

Re: ioemu

[Ian Jackson]

Christoph Egger writes ("Re: [Xen-devel] ioemu"):
> On Monday 01 September 2008 12:05:36 Ian Jackson wrote:
> > I'm not sure I understand what checksum you're referring to here.  Is
> > it part of the NetBSD ports system ?  What does the ports system
> > expect ?
> 
> The checksum is part of the package in pkgsrc. It needs an URL where
> to download the source archive. After the download, the archive is verified
> against SHA1 and RMD160 checksums and against its size in bytes.

I've spoken to a local FreeBSD expert and we don't understand this
part of your complaint.  The ports system expects to start with a
tarball.  If you're trying to use the Xen 3.3.0 release, that tarball
could be the official release tarball.  If you say that that doesn't
work for you then you need some other tarball but there are no other
official tarballs.  In particular there are no official tarballs of
xen-unstable or qemu-xen-unstable.

Are you producing this tarball yourselves somehow ?  (hg archive
perhaps).  And your complaint is that its hash isn't always the same
when you use git ?  git-archive seems to produce reproduceable
tarballs for me. 

> It doesn't compile on BSD (and hasn't got the testing).

Does qemu upstream compile on BSD ?  I think you should be looking
into that if it's a problem.

> Our testing infrastructure is based on changeset numbers. From reading
> the number you immediately know is this an old or new changeset and
> you immediately know how many changesets are between two changesets.

Yes, then your test infrastructure will need to be taught how to deal
with git changeset ids - just like ours did.  It's not very difficult
and I'm sure you can cope.

> > I agree it is a shame that our official tarball isn't made entirely
> > mechanically.  If you would care to contribute a script that
> > reproduces the 3.3.0 tarball when dropped into the appropriate
> > xen-3.3-testing changeset, and also does a sane thing in currently
> > xen-unstable, we'll consider including it and using it next time.
> 
> So you are planning to also move the hypervisor to git in the
> middle/long run?

That doesn't seem to relate to the paragraph of mine you quote, but:

This is not my decision to make, but I expect that Xen will stay with
hg for quite a while and I don't think it should change soon.

The situation with Xen is much less fraught than that with qemu.
After all Xen does not have three or four strange semi-forks, whereas
qemu does (ioemu is one), Xen upstream is already using a dvcs rather
than svn, etc.  So use of git's features for dealing with bad
situations much less necessary with Xen.  That means that git's
downsides (chiefly, the catastrophic and constantly changing UI) are a
decisive factor - and of course there is no particular incentive to
change.

Ian.

RE: ioemu

[Ian Pratt]

> So you are planning to also move the hypervisor to git in the
> middle/long run?

No plans, but it's something we're going to have to discuss at some
point. I hate git with a passion, but the thought of having to use a
combination of git and mercurial is arguably worse... 

Ian 

Re: ioemu

[John Levon]

On Mon, Sep 01, 2008 at 01:18:07PM +0100, Ian Pratt wrote:

> > So you are planning to also move the hypervisor to git in the
> > middle/long run?
> 
> No plans, but it's something we're going to have to discuss at some
> point. I hate git with a passion, but the thought of having to use a
> combination of git and mercurial is arguably worse... 

Some of us have a heavy infrastructure investment into a Mercurial based
xen tree. Moving makes absolutely no sense and will cause total havoc.

It's bad enough not having proper history back into BitKeeper days.

regards
john

Re: ioemu

[Christoph Egger]

On Monday 01 September 2008 14:16:33 Ian Jackson wrote:
> Christoph Egger writes ("Re: [Xen-devel] ioemu"):
> > On Monday 01 September 2008 12:05:36 Ian Jackson wrote:
> > > I'm not sure I understand what checksum you're referring to here.  Is
> > > it part of the NetBSD ports system ?  What does the ports system
> > > expect ?
> >
> > The checksum is part of the package in pkgsrc. It needs an URL where
> > to download the source archive. After the download, the archive is
> > verified against SHA1 and RMD160 checksums and against its size in bytes.
>
> I've spoken to a local FreeBSD expert and we don't understand this
> part of your complaint. The ports system expects to start with a 
> tarball.

FreeBSD ports and NetBSD pkgsrc are technically different, but there
are some overlapping parts like the UI, the tarball and the capability
to apply extra patches after extraction.

> If you're trying to use the Xen 3.3.0 release, that tarball
> could be the official release tarball.  If you say that that doesn't
> work for you then you need some other tarball but there are no other
> official tarballs.  In particular there are no official tarballs of
> xen-unstable or qemu-xen-unstable.

I started to create the packaging process before the official release.
I created an archive via hg archive and uploaded it to an NetBSD server.

When the official release was there, I just changed the download URL
and updated the checksums.

> Are you producing this tarball yourselves somehow ?  (hg archive
> perhaps).  And your complaint is that its hash isn't always the same
> when you use git ?  git-archive seems to produce reproduceable
> tarballs for me.

# man git
man: no entry for git in the manual.

git 's UI changes that fast, that it's not worth to provide a manpage?

"man hg" works and describes all available commands.

> > It doesn't compile on BSD (and hasn't got the testing).
>
> Does qemu upstream compile on BSD ?  I think you should be looking
> into that if it's a problem.

It does with the extra patches in pkgsrc (which also contain non-BSD specific
bugfixes). FreeBSD ports surely have these, too.

> > Our testing infrastructure is based on changeset numbers. From reading
> > the number you immediately know is this an old or new changeset and
> > you immediately know how many changesets are between two changesets.
>
> Yes, then your test infrastructure will need to be taught how to deal
> with git changeset ids - just like ours did.  It's not very difficult
> and I'm sure you can cope.

Is the format stable or does it change like the UI?

> > > I agree it is a shame that our official tarball isn't made entirely
> > > mechanically.  If you would care to contribute a script that
> > > reproduces the 3.3.0 tarball when dropped into the appropriate
> > > xen-3.3-testing changeset, and also does a sane thing in currently
> > > xen-unstable, we'll consider including it and using it next time.
> >
> > So you are planning to also move the hypervisor to git in the
> > middle/long run?
>
> That doesn't seem to relate to the paragraph of mine you quote, but:
>
> This is not my decision to make, but I expect that Xen will stay with
> hg for quite a while and I don't think it should change soon.
>
> The situation with Xen is much less fraught than that with qemu.
> After all Xen does not have three or four strange semi-forks, whereas
> qemu does (ioemu is one), Xen upstream is already using a dvcs rather
> than svn, etc.  So use of git's features for dealing with bad
> situations much less necessary with Xen.  That means that git's
> downsides (chiefly, the catastrophic and constantly changing UI) are a
> decisive factor - and of course there is no particular incentive to
> change.



-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy

Re: ioemu

[Ian Jackson]

Christoph Egger writes ("Re: [Xen-devel] ioemu"):
> Is the format stable or does it change like the UI?

I'm not authoritative on this question but I don't think there are any
relevant incompatibilities in repo formats between different git
versions.  AIUI that there are some fancy repo format features - that
we aren't using - which not all versions support.

Ian.

Re: ioemu

[Brendan Cully]

On Monday, 01 September 2008 at 11:05, Ian Jackson wrote:
> We won't be doing this.  The point of using git rather than hg is to
> much more easily manage the interactions and patch workflows between
> the various branches of qemu, of which ioemu-remote is just one.
> 
> Dealing well with this kind of forked up trainwreck requires a lot of
> heavy lifting from the revision control system.  git can do this
> (despite the appalling user interface) and hg can't.

As a mercurial developer I'd be very interested in knowing more
specifically what functionality hg is missing. The big two pieces I'm
aware of are git-style branches and rebase support. We've just gotten
a nice rebase extension out of the google summer of code, and I have
an extension called "localbranch" that provides a sort of git branch.