Re: [PATCH]: Implement bzip2 and LZMA loaders / xen-unstable stubdom pvgrub support

["Pasi Kärkkäinen" <pasik@iki.fi>]

[-- Attachment #1: Type: text/plain, Size: 6285 bytes --]

On Tue, Aug 25, 2009 at 06:23:19PM +0300, Pasi Kärkkäinen wrote:
> On Mon, Aug 24, 2009 at 03:04:00PM +0100, Stefano Stabellini wrote:
> > On Fri, 21 Aug 2009, Pasi Kärkkäinen wrote:
> > > On Fri, Aug 21, 2009 at 11:12:00PM +0300, Pasi Kärkkäinen wrote:
> > > > On Fri, Aug 21, 2009 at 11:13:23AM +0100, Keir Fraser wrote:
> > > > > On 21/08/2009 10:54, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> > > > > 
> > > > > >> Good point. pvgrub should support these aswell.
> > > > > >> 
> > > > > >> Keir: Would you like to add those libs to stubdom?
> > > > > >> 
> > > > > >> (Note the lzma.h is coming from that "xz" package..)
> > > > > > 
> > > > > > No. :-) But feel free to make a patch! You'll just need to modify
> > > > > > libxc/Makefile too to force those libs on instead of off, as well as
> > > > > > download and build the libs of course.
> > > > > 
> > > > > Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> > > > > in that Makefile. So all you need to do is patch to download and build the
> > > > > libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> > > > > on xenbits and adjust the patch URLs appropriately.
> > > > > 
> > > > 
> > > > I can now see why you didn't want to do that.. :) I managed to get the libs to 
> > > > compile, but now I'm having other problems..
> > > > 
> > > > "make stubdom" gets to the point where it compiles ioemu-stubdom, but it fails 
> > > > to link because xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o has 
> > > > undefined references in xc_try_bzip2_decode and xc_try_lzma_decode functions.. 
> > > > 
> > > > I guess only pvgrub would need to have this stuff in? I've been trying to
> > > > figure out how the makefile magic works, but haven't really understood it yet.
> > > > 
> > > > Any tips how to make ioemu-stubdom compile/link without including bzip2/lzma stuff into it? 
> > > > 
> > > 
> > > Actually I guess I could add the missing libs to extras/mini-os/Makefile to
> > > APP_LDLIBS to get them included into ioemu-stubdom?
> > > 
> > > Is that the right way to do it?
> > > 
> > 
> > bzlib and lzma are libxc dependencies now, so firstly you have to add
> > the two libraries to the stubdom build system, take a look at
> > pciutils in stubdom/Makefile, that is a good example of how a new
> > library is added.
> 
> This is done, and works. libbz2.a and liblzma.a end up in
> stubdom/cross-root-i686/i686-xen-elf/lib/ directory on my 32bit host.
> 
> > Then you also need to add them both at least to the libxc target, take a
> > look at zlib (and cross-zlib), that should be close to what you need.
> 
> Done.
> 
> > Finally you need to tweak the libxc Makefile for the stubdom case so
> > that the two libraries are correctly added to zlib-options.
> > 
> 
> So you mean xen-unstable/tools/libxc/Makefile here? This part?
> 
> ifeq ($(CONFIG_MiniOS),y)
> zlib-options =
> 
> 
> 
> The problem I'm seeing now is "ioemu" target from xen-unstable/stubdom/Makefile fails to
> link:
> 
> ld -nostdlib -L/root/tem/xen-unstable.hg/stubdom/cross-root-i686/i686-xen-elf/lib  -m elf_i386 -T arch/x86/minios-x86_32.lds
> /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o  -o /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os
> ld: warning: section `.bss' type changed to PROGBITS 
> /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `xc_try_bzip2_decode':
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:42: undefined reference to `BZ2_bzDecompressInit'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:105: undefined reference to `BZ2_bzDecompressEnd'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:69: undefined reference to `BZ2_bzDecompress'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:105: undefined reference to `BZ2_bzDecompressEnd'
> /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `xc_try_lzma_decode':
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:158: undefined reference to `lzma_alone_decoder'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:253: undefined reference to `lzma_end'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:185: undefined reference to `lzma_code'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:253: undefined reference to `lzma_end'
> make[1]: ***
> [/root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os] Error 1
> make[1]: Leaving directory `/root/tem/xen-unstable.hg/extras/mini-os'
> make: *** [ioemu-stubdom] Error 2
> 
> So basicly I should add libbz2.a and liblzma.a to the list of files to link,
> but I'm a bit lost where this is happening. Should I edit
> xen-unstable/extras/mini-os/Makefile to add those libs or somewhere else? 
> 
> I tried adding those libs to xen-unstable/stubdom/Makefile to "ioemu" TARGET_LDFLAGS, but
> that didn't seem to help.
> 
> All tips welcome :)
> 

Attached are my current work-in-progress patches. It still doesn't link properly.. 

Stefano: Would you like to take a look? The patches are against current xen-unstable.

I'm getting weird linking error about stdin/stdout/stderr missing.. 
libc (newlib) is definitely linked in, so I'm not sure what's happening.. 

ld -nostdlib -L/root/xen-unstable.hg/stubdom/cross-root-i686/i686-xen-elf/lib -m elf_i386 -T arch/x86/minios-x86_32.lds
/root/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o -o /root/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os

xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `bzopen_or_bzdopen':
xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1411: undefined reference to `__ctype_b_loc'
xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1447: undefined reference to `stdin'
xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1447: undefined reference to `stdout'
...

In the earlier step mini-os.o was definitely linked against -lc .. 

"strings /path/mini-os.o | grep -i stdin" doesn't give anything though.. 
I wonder why the stuff doesn't get included there.

-- Pasi


[-- Attachment #2: stubdom-Makefile-bzip2-lzma-support-v1.patch --]
[-- Type: text/x-diff, Size: 3984 bytes --]

--- xen-unstable.hg/stubdom/Makefile.backup.orig	2009-09-01 18:30:06.000000000 +0300
+++ xen-unstable.hg/stubdom/Makefile	2009-09-01 20:32:20.000000000 +0300
@@ -11,6 +11,14 @@
 ZLIB_URL=$(XEN_EXTFILES_URL)
 ZLIB_VERSION=1.2.3
 
+#BZIP2_URL?=http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz
+BZIP2_URL=http://www.bzip.org/1.0.5
+BZIP2_VERSION=1.0.5
+
+#LZMA_URL?=http://tukaani.org/xz/xz-4.999.9beta.tar.gz
+LZMA_URL=http://tukaani.org/xz
+LZMA_VERSION=4.999.9beta
+
 #LIBPCI_URL?=http://www.kernel.org/pub/software/utils/pciutils
 LIBPCI_URL?=$(XEN_EXTFILES_URL)
 LIBPCI_VERSION=2.2.9
@@ -134,6 +142,51 @@
 	  $(MAKE) libz.a && \
 	  $(MAKE) install )
 
+#############
+# Cross-bzip2
+#############
+
+bzip2-$(BZIP2_VERSION).tar.gz:
+	$(WGET) $(BZIP2_URL)/$@
+
+bzip2-$(XEN_TARGET_ARCH): bzip2-$(BZIP2_VERSION).tar.gz 
+	tar xzf $<
+	mv bzip2-$(BZIP2_VERSION) $@
+
+BZIP2_STAMPFILE=$(CROSS_ROOT)/$(GNU_TARGET_ARCH)-xen-elf/lib/libbz2.a
+.PHONY: cross-bzip2
+cross-bzip2: $(BZIP2_STAMPFILE)
+$(BZIP2_STAMPFILE): bzip2-$(XEN_TARGET_ARCH) $(NEWLIB_STAMPFILE)
+	( cd $< && \
+	  CFLAGS="$(TARGET_CPPFLAGS) $(TARGET_CFLAGS)" CC=$(CC) $(MAKE) && \
+	  $(INSTALL_DATA) libbz2.a $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/lib && \
+	  $(INSTALL_DATA) bzlib.h $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/include && \
+	  $(INSTALL_DATA) bzlib_private.h $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/include )
+
+
+############
+# Cross-lzma
+############
+
+xz-$(LZMA_VERSION).tar.gz:
+	$(WGET) $(LZMA_URL)/$@
+
+xz-$(XEN_TARGET_ARCH): xz-$(LZMA_VERSION).tar.gz 
+	tar xzf $<
+	mv xz-$(LZMA_VERSION) $@
+
+LZMA_STAMPFILE=$(CROSS_ROOT)/$(GNU_TARGET_ARCH)-xen-elf/lib/liblzma.a
+.PHONY: cross-lzma
+cross-lzma: $(LZMA_STAMPFILE)
+$(LZMA_STAMPFILE): xz-$(XEN_TARGET_ARCH) $(NEWLIB_STAMPFILE)
+	( cd $< && \
+	  CFLAGS="$(TARGET_CPPFLAGS) $(TARGET_CFLAGS)" CC=$(CC) ./configure --prefix=$(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf --enable-shared=no && \
+	  cd src/liblzma && $(MAKE) && cd ../../ && \
+	  $(INSTALL_DATA) src/liblzma/.libs/liblzma.a $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/lib && \
+	  $(INSTALL_DATA) src/liblzma/api/lzma.h $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/include && \
+	  cp -r src/liblzma/api/lzma $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/include/ )
+
+
 ##############
 # Cross-libpci
 ##############
@@ -210,7 +263,7 @@
 #######
 
 .PHONY: $(CROSS_ROOT)
-$(CROSS_ROOT): cross-newlib cross-zlib cross-libpci
+$(CROSS_ROOT): cross-newlib cross-zlib cross-bzip2 cross-lzma cross-libpci
 
 $(XEN_ROOT)/tools/ioemu-dir:
 	$(MAKE) -C $(XEN_ROOT)/tools ioemu-dir-find
@@ -265,15 +318,15 @@
 
 .PHONY: libxc
 libxc: libxc-$(XEN_TARGET_ARCH)/libxenctrl.a libxc-$(XEN_TARGET_ARCH)/libxenguest.a
-libxc-$(XEN_TARGET_ARCH)/libxenctrl.a libxc-$(XEN_TARGET_ARCH)/libxenguest.a:: cross-zlib
-	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C libxc-$(XEN_TARGET_ARCH)
+libxc-$(XEN_TARGET_ARCH)/libxenctrl.a libxc-$(XEN_TARGET_ARCH)/libxenguest.a:: cross-zlib cross-bzip2 cross-lzma
+	CPPFLAGS="$(TARGET_CPPFLAGS) -DHAVE_BZLIB -DHAVE_LZMA" CFLAGS="$(TARGET_CFLAGS) -DHAVE_BZLIB -DHAVE_LZMA" $(MAKE) -C libxc-$(XEN_TARGET_ARCH)
 
 #######
 # ioemu
 #######
 
 .PHONY: ioemu
-ioemu: cross-zlib cross-libpci libxc
+ioemu: cross-zlib cross-bzip2 cross-lzma cross-libpci libxc
 	[ -f ioemu/config-host.mak ] || \
 	  ( $(absolutify_xen_root); \
 	    $(buildmakevars2shellvars); \
@@ -339,7 +392,7 @@
 
 .PHONY: pv-grub
 pv-grub: mini-os-$(XEN_TARGET_ARCH)-grub libxc grub
-	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS) -DHAVE_BZLIB -DHAVE_LZMA" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS) -DHAVE_BZLIB -DHAVE_LZMA" DEF_LDFLAGS="$(TARGET_LDFLAGS) -lbz2 -llzma" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
 
 #########
 # install

[-- Attachment #3: extras-minios-Makefile-bzip2-lzma-support-v1.patch --]
[-- Type: text/x-diff, Size: 316 bytes --]

--- xen-unstable.hg/extras/mini-os/Makefile.backup.orig	2009-09-01 18:30:06.000000000 +0300
+++ xen-unstable.hg/extras/mini-os/Makefile	2009-09-01 21:43:14.000000000 +0300
@@ -83,6 +83,8 @@
 APP_LDLIBS += -lpci
 APP_LDLIBS += -lz
 APP_LDLIBS += -lm
+APP_LDLIBS += -lbz2
+APP_LDLIBS += -llzma
 LDLIBS += -lc
 endif
 

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel