[PATCH 1/2] Introduce support for upstream qemu in the xen-unstable build system

[PATCH 1/2] Introduce support for upstream qemu in the xen-unstable build system

[stefano.stabellini]

In order to distinguish between upstream qemu and qemu-xen I am
introducing a new variable named "QEMU" that only if is equal to
"upstream" switches the build system to the new qemu.

Users that want to try the new qemu just have to export QEMU=upstream
before calling make in the xen-unstable top level directory.

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

diff -r 0ba816e077e6 Config.mk
--- a/Config.mk	Wed Aug 31 16:04:37 2011 +0000
+++ b/Config.mk	Wed Sep 07 11:13:17 2011 +0000
@@ -192,12 +192,21 @@ else
 QEMU_REMOTE=git://xenbits.xensource.com/qemu-xen-unstable.git
 endif
 
+# Only available through the git protocol at the moment
+QEMU_UPSTREAM_URL=git://xenbits.xen.org/people/sstabellini/qemu-dm.git
+QEMU_UPSTREAM_TAG=origin/xen-stable-0.15
+
 # Specify which qemu-dm to use. This may be `ioemu' to use the old
 # Mercurial in-tree version, or a local directory, or a git URL.
 # CONFIG_QEMU ?= `pwd`/$(XEN_ROOT)/../qemu-xen.git
+ifeq ($(QEMU),upstream)
+CONFIG_QEMU ?= $(QEMU_UPSTREAM_URL)
+QEMU_TAG ?= $(QEMU_UPSTREAM_TAG)
+else
 CONFIG_QEMU ?= $(QEMU_REMOTE)
+QEMU_TAG ?= cd776ee9408ff127f934a707c1a339ee600bc127
+endif
 
-QEMU_TAG ?= cd776ee9408ff127f934a707c1a339ee600bc127
 # Tue Jun 28 13:50:53 2011 +0100
 # qemu-char.c: fix incorrect CONFIG_STUBDOM handling
 
diff -r 0ba816e077e6 tools/Makefile
--- a/tools/Makefile	Wed Aug 31 16:04:37 2011 +0000
+++ b/tools/Makefile	Wed Sep 07 11:13:17 2011 +0000
@@ -106,7 +106,19 @@ ioemu-dir-find:
 	set -e; \
 		$(buildmakevars2shellvars); \
 		cd ioemu-dir; \
-		$(QEMU_ROOT)/xen-setup $(IOEMU_CONFIGURE_CROSS)
+		if [ "$(QEMU)" = upstream ]; then \
+			cd $(QEMU_ROOT); \
+			./configure --enable-xen --target-list=i386-softmmu \
+			--extra-cflags="-I$(XEN_ROOT)/tools/include \
+			-I$(XEN_ROOT)/tools/libxc \
+			-I$(XEN_ROOT)/tools/xenstore" \
+			--extra-ldflags="-L$(XEN_ROOT)/tools/libxc \
+			-L$(XEN_ROOT)/tools/libxenstore" \
+			--disable-kvm \
+			$(IOEMU_CONFIGURE_CROSS); \
+		else \
+			$(QEMU_ROOT)/xen-setup $(IOEMU_CONFIGURE_CROSS); \
+		fi
 
 .PHONY: ioemu-dir-force-update
 ioemu-dir-force-update:

[PATCH 2/2] Automatically select seabios when we are building upstream qemu

[stefano.stabellini]

Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

diff -r f1269b3216ef Config.mk
--- a/Config.mk	Wed Sep 07 13:28:05 2011 +0000
+++ b/Config.mk	Wed Sep 07 13:29:15 2011 +0000
@@ -195,6 +195,7 @@ endif
 # Only available through the git protocol at the moment
 QEMU_UPSTREAM_URL=git://xenbits.xen.org/people/sstabellini/qemu-dm.git
 QEMU_UPSTREAM_TAG=origin/xen-stable-0.15
+SEABIOS_UPSTREAM_URL=git://git.qemu.org/seabios.git
 
 # Specify which qemu-dm to use. This may be `ioemu' to use the old
 # Mercurial in-tree version, or a local directory, or a git URL.
@@ -202,9 +203,12 @@ QEMU_UPSTREAM_TAG=origin/xen-stable-0.15
 ifeq ($(QEMU),upstream)
 CONFIG_QEMU ?= $(QEMU_UPSTREAM_URL)
 QEMU_TAG ?= $(QEMU_UPSTREAM_TAG)
+CONFIG_SEABIOS ?= $(SEABIOS_UPSTREAM_URL)
+SEABIOS_TAG ?= 7fc039e9c262b4199fab497f3e12f4e425c37560
 else
 CONFIG_QEMU ?= $(QEMU_REMOTE)
 QEMU_TAG ?= cd776ee9408ff127f934a707c1a339ee600bc127
+CONFIG_SEABIOS ?=
 endif
 
 # Tue Jun 28 13:50:53 2011 +0100
@@ -213,15 +217,6 @@ endif
 # Short answer -- do not enable this unless you know what you are
 # doing and are prepared for some pain.
 
-# SeaBIOS integration is a work in progress. Before enabling this
-# option you must clone git://git.qemu.org/seabios.git/, possibly add
-# some development patches and then build it yourself before pointing
-# this variable to it (using an absolute path).
-#
-# Note that using SeaBIOS requires the use the upstream qemu as the
-# device model.
-SEABIOS_DIR ?= 
-
 # Optional components
 XENSTAT_XENTOP     ?= y
 VTPM_TOOLS         ?= n
diff -r f1269b3216ef tools/firmware/Makefile
--- a/tools/firmware/Makefile	Wed Sep 07 13:28:05 2011 +0000
+++ b/tools/firmware/Makefile	Wed Sep 07 13:29:15 2011 +0000
@@ -6,13 +6,35 @@ TARGET      := hvmloader/hvmloader
 INST_DIR := $(DESTDIR)$(XENFIRMWAREDIR)
 
 SUBDIRS :=
+ifneq ($(CONFIG_SEABIOS),)
+SEABIOS_DIR := seabios-dir
+SUBDIRS += $(SEABIOS_DIR)
+endif
 SUBDIRS += rombios
 SUBDIRS += vgabios
 SUBDIRS += etherboot
 SUBDIRS += hvmloader
 
+$(SEABIOS_DIR):
+	set -ex; \
+	if [ ! -d seabios-remote ]; then \
+		rm -rf seabios-remote seabios-remote.tmp; \
+		mkdir seabios-remote.tmp; rmdir seabios-remote.tmp; \
+		$(GIT) clone $(CONFIG_SEABIOS) seabios-remote.tmp; \
+		if [ "$(SEABIOS_TAG)" ]; then			\
+			cd seabios-remote.tmp;			\
+			$(GIT) branch -D dummy >/dev/null 2>&1 ||:; \
+			$(GIT) checkout -b dummy $(SEABIOS_TAG);	\
+			cd ..;					\
+		fi;						\
+		mv seabios-remote.tmp seabios-remote; \
+	fi; \
+	rm -f seabios-dir; \
+	ln -sf seabios-remote seabios-dir; \
+	mv seabios-config seabios-dir/.config;
+
 .PHONY: all
-all:
+all: $(SEABIOS_DIR)
 	@set -e; if [ $$((`( bcc -v 2>&1 | grep version || echo 0.0.0 ) | cut -d' ' -f 3 | awk -F. '{ printf "0x%02x%02x%02x", $$1, $$2, $$3}'`)) -lt $$((0x00100e)) ] ; then \
 	echo "==========================================================================="; \
 	echo "Require dev86 rpm or bin86 & bcc debs version >= 0.16.14 to build firmware!"; \
@@ -35,4 +57,7 @@ clean: subdirs-clean
 distclean: subdirs-distclean
 
 subdir-distclean-etherboot: .phony
-	$(MAKE) -C etherboot distclean
\ No newline at end of file
+	$(MAKE) -C etherboot distclean
+
+subdir-distclean-$(SEABIOS_DIR): .phony
+	rm -rf seabios-dir seabios-remote
diff -r f1269b3216ef tools/firmware/hvmloader/Makefile
--- a/tools/firmware/hvmloader/Makefile	Wed Sep 07 13:28:05 2011 +0000
+++ b/tools/firmware/hvmloader/Makefile	Wed Sep 07 13:29:15 2011 +0000
@@ -37,19 +37,17 @@ endif
 
 CIRRUSVGA_DEBUG ?= n
 
+ifneq ($(SEABIOS_DIR),)
+OBJS += seabios.o
+CFLAGS += -DENABLE_SEABIOS
+SEABIOS_ROM := ../$(SEABIOS_DIR)/out/bios.bin
+else
 ROMBIOS_DIR := ../rombios
-ifneq ($(ROMBIOS_DIR),)
 OBJS += rombios.o
 CFLAGS += -DENABLE_ROMBIOS
 ROMBIOS_ROM := $(ROMBIOS_DIR)/BIOS-bochs-latest
 endif
 
-ifneq ($(SEABIOS_DIR),)
-OBJS += seabios.o
-CFLAGS += -DENABLE_SEABIOS
-SEABIOS_ROM := $(SEABIOS_DIR)/out/bios.bin
-endif
-
 STDVGA_ROM    := ../vgabios/VGABIOS-lgpl-latest.bin
 ifeq ($(CIRRUSVGA_DEBUG),y)
 CIRRUSVGA_ROM := ../vgabios/VGABIOS-lgpl-latest.cirrus.debug.bin
diff -r f1269b3216ef tools/firmware/seabios-config
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tools/firmware/seabios-config	Wed Sep 07 13:29:15 2011 +0000
@@ -0,0 +1,73 @@
+#
+# Automatically generated make config: don't edit
+# SeaBIOS Configuration
+# Wed Sep  7 13:03:21 2011
+#
+
+#
+# General Features
+#
+# CONFIG_COREBOOT is not set
+CONFIG_XEN=y
+CONFIG_THREADS=y
+# CONFIG_THREAD_OPTIONROMS is not set
+CONFIG_RELOCATE_INIT=y
+CONFIG_BOOTMENU=y
+# CONFIG_BOOTSPLASH is not set
+CONFIG_BOOTORDER=y
+
+#
+# Hardware support
+#
+CONFIG_ATA=y
+CONFIG_ATA_DMA=y
+CONFIG_ATA_PIO32=y
+CONFIG_AHCI=y
+CONFIG_VIRTIO_BLK=y
+CONFIG_FLOPPY=y
+CONFIG_PS2PORT=y
+CONFIG_USB=y
+CONFIG_USB_UHCI=y
+CONFIG_USB_OHCI=y
+CONFIG_USB_EHCI=y
+CONFIG_USB_MSC=y
+CONFIG_USB_HUB=y
+CONFIG_USB_KEYBOARD=y
+CONFIG_USB_MOUSE=y
+CONFIG_SERIAL=y
+CONFIG_LPT=y
+# CONFIG_USE_SMM is not set
+CONFIG_MTRR_INIT=y
+
+#
+# BIOS interfaces
+#
+CONFIG_DRIVES=y
+CONFIG_CDROM_BOOT=y
+CONFIG_CDROM_EMU=y
+CONFIG_PCIBIOS=y
+CONFIG_APMBIOS=y
+CONFIG_PNPBIOS=y
+CONFIG_OPTIONROMS=y
+# CONFIG_OPTIONROMS_DEPLOYED is not set
+CONFIG_PMM=y
+CONFIG_BOOT=y
+CONFIG_KEYBOARD=y
+CONFIG_KBD_CALL_INT15_4F=y
+CONFIG_MOUSE=y
+CONFIG_S3_RESUME=y
+# CONFIG_DISABLE_A20 is not set
+
+#
+# BIOS Tables
+#
+CONFIG_PIRTABLE=y
+CONFIG_MPTABLE=y
+CONFIG_SMBIOS=y
+CONFIG_ACPI=y
+
+#
+# Debugging
+#
+CONFIG_DEBUG_LEVEL=1
+# CONFIG_DEBUG_SERIAL is not set

Re: [PATCH 1/2] Introduce support for upstream qemu in the xen-unstable build system

[Ian Campbell]

On Wed, 2011-09-07 at 10:39 -0400, stefano.stabellini@eu.citrix.com
wrote:
> In order to distinguish between upstream qemu and qemu-xen I am
> introducing a new variable named "QEMU" that only if is equal to
> "upstream" switches the build system to the new qemu.

Thanks Stefano, this integrated support is overdue.

Ultimately though I expect we will need a "both" mode since people will
want old qemu for compatibility with their existing installed guests and
new qemu for new ones. The allegation (and I don't really know how true
it is or if it is pessimism or realism) is that some OSes don't cope
with having the platform components etc changed under them.

> Users that want to try the new qemu just have to export QEMU=upstream
> before calling make in the xen-unstable top level directory.

In xl/libxl we call them "qemu-xen-traditional" and "qemu-xen". Perhaps
we should mirror that nomenclature here?

> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> 

> diff -r 0ba816e077e6 Config.mk
> --- a/Config.mk	Wed Aug 31 16:04:37 2011 +0000
> +++ b/Config.mk	Wed Sep 07 11:13:17 2011 +0000
> @@ -192,12 +192,21 @@ else
>  QEMU_REMOTE=git://xenbits.xensource.com/qemu-xen-unstable.git
>  endif
>  
> +# Only available through the git protocol at the moment
> +QEMU_UPSTREAM_URL=git://xenbits.xen.org/people/sstabellini/qemu-dm.git
> +QEMU_UPSTREAM_TAG=origin/xen-stable-0.15
> +
>  # Specify which qemu-dm to use. This may be `ioemu' to use the old
>  # Mercurial in-tree version, or a local directory, or a git URL.
>  # CONFIG_QEMU ?= `pwd`/$(XEN_ROOT)/../qemu-xen.git
> +ifeq ($(QEMU),upstream)
> +CONFIG_QEMU ?= $(QEMU_UPSTREAM_URL)
> +QEMU_TAG ?= $(QEMU_UPSTREAM_TAG)
> +else
>  CONFIG_QEMU ?= $(QEMU_REMOTE)
> +QEMU_TAG ?= cd776ee9408ff127f934a707c1a339ee600bc127
> +endif
>  
> -QEMU_TAG ?= cd776ee9408ff127f934a707c1a339ee600bc127
>  # Tue Jun 28 13:50:53 2011 +0100
>  # qemu-char.c: fix incorrect CONFIG_STUBDOM handling
>  
> diff -r 0ba816e077e6 tools/Makefile
> --- a/tools/Makefile	Wed Aug 31 16:04:37 2011 +0000
> +++ b/tools/Makefile	Wed Sep 07 11:13:17 2011 +0000
> @@ -106,7 +106,19 @@ ioemu-dir-find:
>  	set -e; \
>  		$(buildmakevars2shellvars); \
>  		cd ioemu-dir; \
> -		$(QEMU_ROOT)/xen-setup $(IOEMU_CONFIGURE_CROSS)
> +		if [ "$(QEMU)" = upstream ]; then \
> +			cd $(QEMU_ROOT); \
> +			./configure --enable-xen --target-list=i386-softmmu \
> +			--extra-cflags="-I$(XEN_ROOT)/tools/include \
> +			-I$(XEN_ROOT)/tools/libxc \
> +			-I$(XEN_ROOT)/tools/xenstore" \
> +			--extra-ldflags="-L$(XEN_ROOT)/tools/libxc \
> +			-L$(XEN_ROOT)/tools/libxenstore" \
> +			--disable-kvm \
> +			$(IOEMU_CONFIGURE_CROSS); \
> +		else \
> +			$(QEMU_ROOT)/xen-setup $(IOEMU_CONFIGURE_CROSS); \
> +		fi
>  
>  .PHONY: ioemu-dir-force-update
>  ioemu-dir-force-update:
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: [PATCH 1/2] Introduce support for upstream qemu in the xen-unstable build system

[Keir Fraser]

On 07/09/2011 15:39, "stefano.stabellini@eu.citrix.com"
<stefano.stabellini@eu.citrix.com> wrote:

> In order to distinguish between upstream qemu and qemu-xen I am
> introducing a new variable named "QEMU" that only if is equal to
> "upstream" switches the build system to the new qemu.
> 
> Users that want to try the new qemu just have to export QEMU=upstream
> before calling make in the xen-unstable top level directory.

I think we should build and install both, and select between them via
per-domain config option. That's basically what we need for supporting old
save images.

 -- Keir

> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> 
> diff -r 0ba816e077e6 Config.mk
> --- a/Config.mk Wed Aug 31 16:04:37 2011 +0000
> +++ b/Config.mk Wed Sep 07 11:13:17 2011 +0000
> @@ -192,12 +192,21 @@ else
>  QEMU_REMOTE=git://xenbits.xensource.com/qemu-xen-unstable.git
>  endif
>  
> +# Only available through the git protocol at the moment
> +QEMU_UPSTREAM_URL=git://xenbits.xen.org/people/sstabellini/qemu-dm.git
> +QEMU_UPSTREAM_TAG=origin/xen-stable-0.15
> +
>  # Specify which qemu-dm to use. This may be `ioemu' to use the old
>  # Mercurial in-tree version, or a local directory, or a git URL.
>  # CONFIG_QEMU ?= `pwd`/$(XEN_ROOT)/../qemu-xen.git
> +ifeq ($(QEMU),upstream)
> +CONFIG_QEMU ?= $(QEMU_UPSTREAM_URL)
> +QEMU_TAG ?= $(QEMU_UPSTREAM_TAG)
> +else
>  CONFIG_QEMU ?= $(QEMU_REMOTE)
> +QEMU_TAG ?= cd776ee9408ff127f934a707c1a339ee600bc127
> +endif
>  
> -QEMU_TAG ?= cd776ee9408ff127f934a707c1a339ee600bc127
>  # Tue Jun 28 13:50:53 2011 +0100
>  # qemu-char.c: fix incorrect CONFIG_STUBDOM handling
>  
> diff -r 0ba816e077e6 tools/Makefile
> --- a/tools/Makefile Wed Aug 31 16:04:37 2011 +0000
> +++ b/tools/Makefile Wed Sep 07 11:13:17 2011 +0000
> @@ -106,7 +106,19 @@ ioemu-dir-find:
> set -e; \
> $(buildmakevars2shellvars); \
> cd ioemu-dir; \
> -  $(QEMU_ROOT)/xen-setup $(IOEMU_CONFIGURE_CROSS)
> +  if [ "$(QEMU)" = upstream ]; then \
> +   cd $(QEMU_ROOT); \
> +   ./configure --enable-xen --target-list=i386-softmmu \
> +   --extra-cflags="-I$(XEN_ROOT)/tools/include \
> +   -I$(XEN_ROOT)/tools/libxc \
> +   -I$(XEN_ROOT)/tools/xenstore" \
> +   --extra-ldflags="-L$(XEN_ROOT)/tools/libxc \
> +   -L$(XEN_ROOT)/tools/libxenstore" \
> +   --disable-kvm \
> +   $(IOEMU_CONFIGURE_CROSS); \
> +  else \
> +   $(QEMU_ROOT)/xen-setup $(IOEMU_CONFIGURE_CROSS); \
> +  fi
>  
>  .PHONY: ioemu-dir-force-update
>  ioemu-dir-force-update:

Re: [PATCH 1/2] Introduce support for upstream qemu in the xen-unstable build system

[Stefano Stabellini]

On Wed, 7 Sep 2011, Ian Campbell wrote:
> On Wed, 2011-09-07 at 10:39 -0400, stefano.stabellini@eu.citrix.com
> wrote:
> > In order to distinguish between upstream qemu and qemu-xen I am
> > introducing a new variable named "QEMU" that only if is equal to
> > "upstream" switches the build system to the new qemu.
> 
> Thanks Stefano, this integrated support is overdue.
> 
> Ultimately though I expect we will need a "both" mode since people will
> want old qemu for compatibility with their existing installed guests and
> new qemu for new ones. The allegation (and I don't really know how true
> it is or if it is pessimism or realism) is that some OSes don't cope
> with having the platform components etc changed under them.

Yes. At that point we'll probably have to always build them both,
including two hvmloader...


> > Users that want to try the new qemu just have to export QEMU=upstream
> > before calling make in the xen-unstable top level directory.
> 
> In xl/libxl we call them "qemu-xen-traditional" and "qemu-xen". Perhaps
> we should mirror that nomenclature here?

I looked at libxl but I thought that QEMU=qemu-xen would be confusing for
users/developers.
Like you said, considering that we already have
device_model_version=qemu-xen in VM config files, maybe we should go for
DEVICE_MODEL=qemu-xen?

Re: [PATCH 1/2] Introduce support for upstream qemu in the xen-unstable build system

[Ian Campbell]

On Wed, 2011-09-07 at 12:54 -0400, Stefano Stabellini wrote:
> On Wed, 7 Sep 2011, Ian Campbell wrote:
> > On Wed, 2011-09-07 at 10:39 -0400, stefano.stabellini@eu.citrix.com
> > wrote:
> > > In order to distinguish between upstream qemu and qemu-xen I am
> > > introducing a new variable named "QEMU" that only if is equal to
> > > "upstream" switches the build system to the new qemu.
> > 
> > Thanks Stefano, this integrated support is overdue.
> > 
> > Ultimately though I expect we will need a "both" mode since people will
> > want old qemu for compatibility with their existing installed guests and
> > new qemu for new ones. The allegation (and I don't really know how true
> > it is or if it is pessimism or realism) is that some OSes don't cope
> > with having the platform components etc changed under them.
> 
> Yes. At that point we'll probably have to always build them both,

Why wait? Why not just do that now.

> including two hvmloader...

Only one hvmloader will be needed, it will include both rombios and
seabios support and it selects the right one at boot time, since the
toolstack pushes down the choice. This is what happens today if you turn
on the SEABIOS option.

> > > Users that want to try the new qemu just have to export QEMU=upstream
> > > before calling make in the xen-unstable top level directory.
> > 
> > In xl/libxl we call them "qemu-xen-traditional" and "qemu-xen". Perhaps
> > we should mirror that nomenclature here?
> 
> I looked at libxl but I thought that QEMU=qemu-xen would be confusing for
> users/developers.
> Like you said, considering that we already have
> device_model_version=qemu-xen in VM config files, maybe we should go for
> DEVICE_MODEL=qemu-xen?

With the (current) default being DEVICE_MODEL=qemu-xen-traditional? That
would makes sense to me (excepting that, as above, I think we shouldn't
offer it as a choice in the first place).

Ian.