[PATCH]: Implement bzip2 and LZMA loaders

[PATCH]: Implement bzip2 and LZMA loaders

[Chris Lalancette]

[-- Attachment #1: Type: text/plain, Size: 692 bytes --]

All,
     Recent upstream kernels can be compressed using either gzip, bzip2, or
LZMA.  However, the PV kernel loader in Xen currently only understands gzip, and
will fail on the other two types.  The attached patch implements kernel
decompression for gzip, bzip2, and LZMA so that kernels compressed with any of
these methods can be launched.  Note that the patch is still a little bit rough,
but I thought I would post it to get feedback about what I'm doing wrong.
     I developed this against the RHEL-5 version of the xen tools, but a quick
look through xen-unstable shows that this should apply pretty easily to the
current code.

Signed-off-by: Chris Lalancette <clalance@redhat.com>

[-- Attachment #2: xen-loader-bzip2-lzma.patch --]
[-- Type: text/x-patch, Size: 10446 bytes --]

diff -urp xen-3.1.0-src/tools/libxc/Makefile xen-3.1.0-src.working/tools/libxc/Makefile
--- xen-3.1.0-src/tools/libxc/Makefile	2009-08-13 03:03:55.000000000 -0400
+++ xen-3.1.0-src.working/tools/libxc/Makefile	2009-08-12 11:56:38.000000000 -0400
@@ -161,7 +161,7 @@ libxenguest.so.$(MAJOR): libxenguest.so.
 	ln -sf $< $@
 
 libxenguest.so.$(MAJOR).$(MINOR): $(GUEST_PIC_OBJS) libxenctrl.so
-	$(CC) $(CFLAGS) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenguest.so.$(MAJOR) $(SHLIB_CFLAGS) -o $@ $(GUEST_PIC_OBJS) -lz -lxenctrl -lpthread
+	$(CC) $(CFLAGS) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenguest.so.$(MAJOR) $(SHLIB_CFLAGS) -o $@ $(GUEST_PIC_OBJS) -lz -llzma -lbz2 -lxenctrl -lpthread
 
 -include $(DEPS)
 
diff -urp xen-3.1.0-src/tools/libxc/xc_dom_bzimageloader.c xen-3.1.0-src.working/tools/libxc/xc_dom_bzimageloader.c
--- xen-3.1.0-src/tools/libxc/xc_dom_bzimageloader.c	2009-08-13 03:03:55.000000000 -0400
+++ xen-3.1.0-src.working/tools/libxc/xc_dom_bzimageloader.c	2009-08-13 03:06:19.000000000 -0400
@@ -11,15 +11,208 @@
  * written 2006 by Gerd Hoffmann <kraxel@suse.de>.
  * written 2007 by Jeremy Fitzhardinge <jeremy@xensource.com>
  * written 2008 by Ian Campbell <ijc@hellion.org.uk>
+ * written 2009 by Chris Lalancette <clalance@redhat.com>
  *
  */
 #include <stdio.h>
 #include <stdlib.h>
 #include <inttypes.h>
+#include <bzlib.h>
+#include <lzma.h>
 
 #include "xg_private.h"
 #include "xc_dom.h"
 
+static inline uint64_t physmem(void)
+{
+	uint64_t ret = 0;
+
+	const long pagesize = sysconf(_SC_PAGESIZE);
+	const long pages = sysconf(_SC_PHYS_PAGES);
+	if (pagesize != -1 || pages != -1)
+		// According to docs, pagesize * pages can overflow.
+		// Simple case is 32-bit box with 4 GiB or more RAM,
+		// which may report exactly 4 GiB of RAM, and "long"
+		// being 32-bit will overflow. Casting to uint64_t
+		// hopefully avoids overflows in the near future.
+		ret = (uint64_t)(pagesize) * (uint64_t)(pages);
+
+    return ret;
+}
+
+static int xc_try_bzip2_decode(struct xc_dom_image *dom, void **blob, size_t *size)
+{
+    bz_stream stream;
+    int ret;
+    char *out_buf;
+    int retval = -1;
+    int outsize;
+    uint64_t total;
+
+    stream.bzalloc = NULL;
+    stream.bzfree = NULL;
+    stream.opaque = NULL;
+
+    ret = BZ2_bzDecompressInit(&stream, 0, 0);
+    if (ret != BZ_OK) {
+        xc_dom_printf("Error initting bz2 stream\n");
+        return -1;
+    }
+
+    /* sigh.  We don't know up-front how much memory we are going to need
+     * for the output buffer.  Allocate the output buffer to be equal
+     * the input buffer to start, and we'll realloc as needed.
+     */
+    outsize = dom->kernel_size;
+    out_buf = malloc(outsize);
+    if (out_buf == NULL) {
+        xc_dom_printf("Failed to alloc memory\n");
+        goto bzip2_cleanup;
+    }
+
+    stream.next_in = dom->kernel_blob;
+    stream.avail_in = dom->kernel_size;
+
+    stream.next_out = out_buf;
+    stream.avail_out = dom->kernel_size;
+
+    while (1) {
+        ret = BZ2_bzDecompress(&stream);
+        if (stream.avail_out == 0 || ret != BZ_OK) {
+            out_buf = realloc(out_buf, outsize * 2);
+            if (out_buf == NULL) {
+                xc_dom_printf("Failed to realloc memory\n");
+                break;
+            }
+
+            stream.next_out = out_buf + outsize;
+            stream.avail_out = (outsize * 2) - outsize;
+            outsize *= 2;
+        }
+
+        if (ret != BZ_OK) {
+            if (ret == BZ_STREAM_END) {
+                xc_dom_printf("Saw data stream end\n");
+                retval = 0;
+                break;
+            }
+            xc_dom_printf("BZIP error\n");
+        }
+    }
+
+    total = (stream.total_out_hi32 << 31) | stream.total_out_lo32;
+
+    xc_dom_printf("%s: BZIP2 decompress OK, 0x%zx -> 0x%lx\n", __FUNCTION__, *size, total);
+
+    *blob = out_buf;
+    *size = total;
+
+bzip2_cleanup:
+    BZ2_bzDecompressEnd(&stream);
+
+    return retval;
+}
+
+static int xc_try_lzma_decode(struct xc_dom_image *dom, void **blob, size_t *size)
+{
+    lzma_stream stream = LZMA_STREAM_INIT;
+    lzma_ret ret;
+    lzma_action action = LZMA_RUN;
+    unsigned char *out_buf;
+    int retval = -1;
+    int outsize;
+    const char *msg;
+
+    ret = lzma_alone_decoder(&stream, physmem() / 3);
+    if (ret != LZMA_OK) {
+        xc_dom_printf("Failed to init lzma stream decoder\n");
+        return -1;
+    }
+
+    /* sigh.  We don't know up-front how much memory we are going to need
+     * for the output buffer.  Allocate the output buffer to be equal
+     * the input buffer to start, and we'll realloc as needed.
+     */
+    outsize = dom->kernel_size;
+    out_buf = malloc(outsize);
+    if (out_buf == NULL) {
+        xc_dom_printf("Failed to alloc memory\n");
+        goto lzma_cleanup;
+    }
+
+    stream.next_in = dom->kernel_blob;
+    stream.avail_in = dom->kernel_size;
+
+    stream.next_out = out_buf;
+    stream.avail_out = dom->kernel_size;
+
+    while (1) {
+        ret = lzma_code(&stream, action);
+        if (stream.avail_out == 0 || ret != LZMA_OK) {
+            out_buf = realloc(out_buf, outsize * 2);
+            if (out_buf == NULL) {
+                xc_dom_printf("Failed to realloc memory\n");
+                break;
+            }
+
+            stream.next_out = out_buf + outsize;
+            stream.avail_out = (outsize * 2) - outsize;
+            outsize *= 2;
+        }
+
+        if (ret != LZMA_OK) {
+            if (ret == LZMA_STREAM_END) {
+                xc_dom_printf("Saw data stream end\n");
+                retval = 0;
+                break;
+            }
+
+            switch (ret) {
+            case LZMA_MEM_ERROR:
+                msg = strerror(ENOMEM);
+                break;
+
+            case LZMA_MEMLIMIT_ERROR:
+                msg = "Memory usage limit reached";
+                break;
+
+            case LZMA_FORMAT_ERROR:
+                msg = "File format not recognized";
+                break;
+
+            case LZMA_OPTIONS_ERROR:
+                // FIXME: Better message?
+                msg = "Unsupported compression options";
+                break;
+
+            case LZMA_DATA_ERROR:
+                msg = "File is corrupt";
+                break;
+
+            case LZMA_BUF_ERROR:
+                msg = "Unexpected end of input";
+                break;
+
+            default:
+                msg = "Internal program error (bug)";
+                break;
+            }
+            xc_dom_printf("%s: LZMA decompression error %s\n", __FUNCTION__, msg);
+            break;
+        }
+    }
+
+    xc_dom_printf("%s: LZMA decompress OK, 0x%zx -> 0x%zx\n", __FUNCTION__, *size, stream.total_out);
+
+    *blob = out_buf;
+    *size = stream.total_out;
+
+ lzma_cleanup:
+    lzma_end(&stream);
+
+    return retval;
+}
+
 struct setup_header {
 	uint8_t		_pad0[0x1f1];		/* skip uninteresting stuff */
 	uint8_t		setup_sects;
@@ -70,22 +263,22 @@ static unsigned int payload_offset(struc
     return off;
 }
 
-static int check_bzimage_kernel(struct xc_dom_image *dom, int verbose)
+static int xc_dom_probe_bzimage_kernel(struct xc_dom_image *dom)
 {
     struct setup_header *hdr;
+    int ret;
 
     if ( dom->kernel_blob == NULL )
     {
-        if ( verbose )
-            xc_dom_panic(XC_INTERNAL_ERROR, "%s: no kernel image loaded\n",
-                         __FUNCTION__);
+        xc_dom_panic(XC_INTERNAL_ERROR, "%s: no kernel image loaded\n",
+                     __FUNCTION__);
         return -EINVAL;
     }
+
     if ( dom->kernel_size < sizeof(struct setup_header) )
     {
-        if ( verbose )
-            xc_dom_panic(XC_INTERNAL_ERROR, "%s: kernel image too small\n",
-                         __FUNCTION__);
+        xc_dom_panic(XC_INTERNAL_ERROR, "%s: kernel image too small\n",
+                     __FUNCTION__);
         return -EINVAL;
     }
 
@@ -93,39 +286,54 @@ static int check_bzimage_kernel(struct x
 
     if ( memcmp(&hdr->header, HDR_MAGIC, HDR_MAGIC_SZ) != 0 )
     {
-        if ( verbose )
-            xc_dom_panic(XC_INVALID_KERNEL, "%s: kernel is not a bzImage\n",
-                         __FUNCTION__);
+        xc_dom_panic(XC_INVALID_KERNEL, "%s: kernel is not a bzImage\n",
+                     __FUNCTION__);
         return -EINVAL;
     }
 
     if ( hdr->version < VERSION(2,8) )
     {
-        if ( verbose )
-            xc_dom_panic(XC_INVALID_KERNEL, "%s: boot protocol too old (%04x)\n",
-                         __FUNCTION__, hdr->version);
+        xc_dom_panic(XC_INVALID_KERNEL, "%s: boot protocol too old (%04x)\n",
+                     __FUNCTION__, hdr->version);
         return -EINVAL;
     }
 
     dom->kernel_blob = dom->kernel_blob + payload_offset(hdr);
     dom->kernel_size = hdr->payload_length;
 
-    if ( xc_dom_try_gunzip(dom, &dom->kernel_blob, &dom->kernel_size) == -1 )
-    {
-        if ( verbose )
-            xc_dom_panic(XC_INVALID_KERNEL, "%s: unable to decompress kernel\n",
+    if (memcmp(dom->kernel_blob, "\037\213", 2) == 0) {
+        ret = xc_dom_try_gunzip(dom, &dom->kernel_blob, &dom->kernel_size);
+        if (ret == -1) {
+            xc_dom_panic(XC_INVALID_KERNEL, "%s: unable to gzip decompress kernel\n",
+                         __FUNCTION__);
+            return -EINVAL;
+        }
+    }
+    else if (memcmp(dom->kernel_blob, "\102\132\150", 3) == 0) {
+        ret = xc_try_bzip2_decode(dom, &dom->kernel_blob, &dom->kernel_size);
+        if (ret < 0) {
+            xc_dom_panic(XC_INVALID_KERNEL, "%s unable to BZIP2 decompress kernel",
+                         __FUNCTION__);
+            return -EINVAL;
+        }
+    }
+    else if (memcmp(dom->kernel_blob, "\135\000", 2) == 0) {
+        ret = xc_try_lzma_decode(dom, &dom->kernel_blob, &dom->kernel_size);
+        if (ret < 0) {
+            xc_dom_panic(XC_INVALID_KERNEL, "%s unable to LZMA decompress kernel\n",
                          __FUNCTION__);
+            return -EINVAL;
+        }
+    }
+    else {
+        xc_dom_panic(XC_INVALID_KERNEL, "%s: unknown compression format\n",
+                     __FUNCTION__);
         return -EINVAL;
     }
 
     return elf_loader.probe(dom);
 }
 
-static int xc_dom_probe_bzimage_kernel(struct xc_dom_image *dom)
-{
-    return check_bzimage_kernel(dom, 0);
-}
-
 static int xc_dom_parse_bzimage_kernel(struct xc_dom_image *dom)
 {
     return elf_loader.parser(dom);

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

[-- Attachment #1: Type: text/plain, Size: 12191 bytes --]

On Thu, Aug 13, 2009 at 09:47:27AM +0200, Chris Lalancette wrote:
> All,
>      Recent upstream kernels can be compressed using either gzip, bzip2, or
> LZMA.  However, the PV kernel loader in Xen currently only understands gzip, and
> will fail on the other two types.  The attached patch implements kernel
> decompression for gzip, bzip2, and LZMA so that kernels compressed with any of
> these methods can be launched.  Note that the patch is still a little bit rough,
> but I thought I would post it to get feedback about what I'm doing wrong.
>      I developed this against the RHEL-5 version of the xen tools, but a quick
> look through xen-unstable shows that this should apply pretty easily to the
> current code.
> 

Attached are two patches:

- Patch by Chris modified to apply to Xen 3.4.1 (only a small Makefile change)
- Patch by me to fix compilation with gcc 4.4.0.

-- Pasi

> Signed-off-by: Chris Lalancette <clalance@redhat.com>

> diff -urp xen-3.1.0-src/tools/libxc/Makefile xen-3.1.0-src.working/tools/libxc/Makefile
> --- xen-3.1.0-src/tools/libxc/Makefile	2009-08-13 03:03:55.000000000 -0400
> +++ xen-3.1.0-src.working/tools/libxc/Makefile	2009-08-12 11:56:38.000000000 -0400
> @@ -161,7 +161,7 @@ libxenguest.so.$(MAJOR): libxenguest.so.
>  	ln -sf $< $@
>  
>  libxenguest.so.$(MAJOR).$(MINOR): $(GUEST_PIC_OBJS) libxenctrl.so
> -	$(CC) $(CFLAGS) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenguest.so.$(MAJOR) $(SHLIB_CFLAGS) -o $@ $(GUEST_PIC_OBJS) -lz -lxenctrl -lpthread
> +	$(CC) $(CFLAGS) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenguest.so.$(MAJOR) $(SHLIB_CFLAGS) -o $@ $(GUEST_PIC_OBJS) -lz -llzma -lbz2 -lxenctrl -lpthread
>  
>  -include $(DEPS)
>  
> diff -urp xen-3.1.0-src/tools/libxc/xc_dom_bzimageloader.c xen-3.1.0-src.working/tools/libxc/xc_dom_bzimageloader.c
> --- xen-3.1.0-src/tools/libxc/xc_dom_bzimageloader.c	2009-08-13 03:03:55.000000000 -0400
> +++ xen-3.1.0-src.working/tools/libxc/xc_dom_bzimageloader.c	2009-08-13 03:06:19.000000000 -0400
> @@ -11,15 +11,208 @@
>   * written 2006 by Gerd Hoffmann <kraxel@suse.de>.
>   * written 2007 by Jeremy Fitzhardinge <jeremy@xensource.com>
>   * written 2008 by Ian Campbell <ijc@hellion.org.uk>
> + * written 2009 by Chris Lalancette <clalance@redhat.com>
>   *
>   */
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <inttypes.h>
> +#include <bzlib.h>
> +#include <lzma.h>
>  
>  #include "xg_private.h"
>  #include "xc_dom.h"
>  
> +static inline uint64_t physmem(void)
> +{
> +	uint64_t ret = 0;
> +
> +	const long pagesize = sysconf(_SC_PAGESIZE);
> +	const long pages = sysconf(_SC_PHYS_PAGES);
> +	if (pagesize != -1 || pages != -1)
> +		// According to docs, pagesize * pages can overflow.
> +		// Simple case is 32-bit box with 4 GiB or more RAM,
> +		// which may report exactly 4 GiB of RAM, and "long"
> +		// being 32-bit will overflow. Casting to uint64_t
> +		// hopefully avoids overflows in the near future.
> +		ret = (uint64_t)(pagesize) * (uint64_t)(pages);
> +
> +    return ret;
> +}
> +
> +static int xc_try_bzip2_decode(struct xc_dom_image *dom, void **blob, size_t *size)
> +{
> +    bz_stream stream;
> +    int ret;
> +    char *out_buf;
> +    int retval = -1;
> +    int outsize;
> +    uint64_t total;
> +
> +    stream.bzalloc = NULL;
> +    stream.bzfree = NULL;
> +    stream.opaque = NULL;
> +
> +    ret = BZ2_bzDecompressInit(&stream, 0, 0);
> +    if (ret != BZ_OK) {
> +        xc_dom_printf("Error initting bz2 stream\n");
> +        return -1;
> +    }
> +
> +    /* sigh.  We don't know up-front how much memory we are going to need
> +     * for the output buffer.  Allocate the output buffer to be equal
> +     * the input buffer to start, and we'll realloc as needed.
> +     */
> +    outsize = dom->kernel_size;
> +    out_buf = malloc(outsize);
> +    if (out_buf == NULL) {
> +        xc_dom_printf("Failed to alloc memory\n");
> +        goto bzip2_cleanup;
> +    }
> +
> +    stream.next_in = dom->kernel_blob;
> +    stream.avail_in = dom->kernel_size;
> +
> +    stream.next_out = out_buf;
> +    stream.avail_out = dom->kernel_size;
> +
> +    while (1) {
> +        ret = BZ2_bzDecompress(&stream);
> +        if (stream.avail_out == 0 || ret != BZ_OK) {
> +            out_buf = realloc(out_buf, outsize * 2);
> +            if (out_buf == NULL) {
> +                xc_dom_printf("Failed to realloc memory\n");
> +                break;
> +            }
> +
> +            stream.next_out = out_buf + outsize;
> +            stream.avail_out = (outsize * 2) - outsize;
> +            outsize *= 2;
> +        }
> +
> +        if (ret != BZ_OK) {
> +            if (ret == BZ_STREAM_END) {
> +                xc_dom_printf("Saw data stream end\n");
> +                retval = 0;
> +                break;
> +            }
> +            xc_dom_printf("BZIP error\n");
> +        }
> +    }
> +
> +    total = (stream.total_out_hi32 << 31) | stream.total_out_lo32;
> +
> +    xc_dom_printf("%s: BZIP2 decompress OK, 0x%zx -> 0x%lx\n", __FUNCTION__, *size, total);
> +
> +    *blob = out_buf;
> +    *size = total;
> +
> +bzip2_cleanup:
> +    BZ2_bzDecompressEnd(&stream);
> +
> +    return retval;
> +}
> +
> +static int xc_try_lzma_decode(struct xc_dom_image *dom, void **blob, size_t *size)
> +{
> +    lzma_stream stream = LZMA_STREAM_INIT;
> +    lzma_ret ret;
> +    lzma_action action = LZMA_RUN;
> +    unsigned char *out_buf;
> +    int retval = -1;
> +    int outsize;
> +    const char *msg;
> +
> +    ret = lzma_alone_decoder(&stream, physmem() / 3);
> +    if (ret != LZMA_OK) {
> +        xc_dom_printf("Failed to init lzma stream decoder\n");
> +        return -1;
> +    }
> +
> +    /* sigh.  We don't know up-front how much memory we are going to need
> +     * for the output buffer.  Allocate the output buffer to be equal
> +     * the input buffer to start, and we'll realloc as needed.
> +     */
> +    outsize = dom->kernel_size;
> +    out_buf = malloc(outsize);
> +    if (out_buf == NULL) {
> +        xc_dom_printf("Failed to alloc memory\n");
> +        goto lzma_cleanup;
> +    }
> +
> +    stream.next_in = dom->kernel_blob;
> +    stream.avail_in = dom->kernel_size;
> +
> +    stream.next_out = out_buf;
> +    stream.avail_out = dom->kernel_size;
> +
> +    while (1) {
> +        ret = lzma_code(&stream, action);
> +        if (stream.avail_out == 0 || ret != LZMA_OK) {
> +            out_buf = realloc(out_buf, outsize * 2);
> +            if (out_buf == NULL) {
> +                xc_dom_printf("Failed to realloc memory\n");
> +                break;
> +            }
> +
> +            stream.next_out = out_buf + outsize;
> +            stream.avail_out = (outsize * 2) - outsize;
> +            outsize *= 2;
> +        }
> +
> +        if (ret != LZMA_OK) {
> +            if (ret == LZMA_STREAM_END) {
> +                xc_dom_printf("Saw data stream end\n");
> +                retval = 0;
> +                break;
> +            }
> +
> +            switch (ret) {
> +            case LZMA_MEM_ERROR:
> +                msg = strerror(ENOMEM);
> +                break;
> +
> +            case LZMA_MEMLIMIT_ERROR:
> +                msg = "Memory usage limit reached";
> +                break;
> +
> +            case LZMA_FORMAT_ERROR:
> +                msg = "File format not recognized";
> +                break;
> +
> +            case LZMA_OPTIONS_ERROR:
> +                // FIXME: Better message?
> +                msg = "Unsupported compression options";
> +                break;
> +
> +            case LZMA_DATA_ERROR:
> +                msg = "File is corrupt";
> +                break;
> +
> +            case LZMA_BUF_ERROR:
> +                msg = "Unexpected end of input";
> +                break;
> +
> +            default:
> +                msg = "Internal program error (bug)";
> +                break;
> +            }
> +            xc_dom_printf("%s: LZMA decompression error %s\n", __FUNCTION__, msg);
> +            break;
> +        }
> +    }
> +
> +    xc_dom_printf("%s: LZMA decompress OK, 0x%zx -> 0x%zx\n", __FUNCTION__, *size, stream.total_out);
> +
> +    *blob = out_buf;
> +    *size = stream.total_out;
> +
> + lzma_cleanup:
> +    lzma_end(&stream);
> +
> +    return retval;
> +}
> +
>  struct setup_header {
>  	uint8_t		_pad0[0x1f1];		/* skip uninteresting stuff */
>  	uint8_t		setup_sects;
> @@ -70,22 +263,22 @@ static unsigned int payload_offset(struc
>      return off;
>  }
>  
> -static int check_bzimage_kernel(struct xc_dom_image *dom, int verbose)
> +static int xc_dom_probe_bzimage_kernel(struct xc_dom_image *dom)
>  {
>      struct setup_header *hdr;
> +    int ret;
>  
>      if ( dom->kernel_blob == NULL )
>      {
> -        if ( verbose )
> -            xc_dom_panic(XC_INTERNAL_ERROR, "%s: no kernel image loaded\n",
> -                         __FUNCTION__);
> +        xc_dom_panic(XC_INTERNAL_ERROR, "%s: no kernel image loaded\n",
> +                     __FUNCTION__);
>          return -EINVAL;
>      }
> +
>      if ( dom->kernel_size < sizeof(struct setup_header) )
>      {
> -        if ( verbose )
> -            xc_dom_panic(XC_INTERNAL_ERROR, "%s: kernel image too small\n",
> -                         __FUNCTION__);
> +        xc_dom_panic(XC_INTERNAL_ERROR, "%s: kernel image too small\n",
> +                     __FUNCTION__);
>          return -EINVAL;
>      }
>  
> @@ -93,39 +286,54 @@ static int check_bzimage_kernel(struct x
>  
>      if ( memcmp(&hdr->header, HDR_MAGIC, HDR_MAGIC_SZ) != 0 )
>      {
> -        if ( verbose )
> -            xc_dom_panic(XC_INVALID_KERNEL, "%s: kernel is not a bzImage\n",
> -                         __FUNCTION__);
> +        xc_dom_panic(XC_INVALID_KERNEL, "%s: kernel is not a bzImage\n",
> +                     __FUNCTION__);
>          return -EINVAL;
>      }
>  
>      if ( hdr->version < VERSION(2,8) )
>      {
> -        if ( verbose )
> -            xc_dom_panic(XC_INVALID_KERNEL, "%s: boot protocol too old (%04x)\n",
> -                         __FUNCTION__, hdr->version);
> +        xc_dom_panic(XC_INVALID_KERNEL, "%s: boot protocol too old (%04x)\n",
> +                     __FUNCTION__, hdr->version);
>          return -EINVAL;
>      }
>  
>      dom->kernel_blob = dom->kernel_blob + payload_offset(hdr);
>      dom->kernel_size = hdr->payload_length;
>  
> -    if ( xc_dom_try_gunzip(dom, &dom->kernel_blob, &dom->kernel_size) == -1 )
> -    {
> -        if ( verbose )
> -            xc_dom_panic(XC_INVALID_KERNEL, "%s: unable to decompress kernel\n",
> +    if (memcmp(dom->kernel_blob, "\037\213", 2) == 0) {
> +        ret = xc_dom_try_gunzip(dom, &dom->kernel_blob, &dom->kernel_size);
> +        if (ret == -1) {
> +            xc_dom_panic(XC_INVALID_KERNEL, "%s: unable to gzip decompress kernel\n",
> +                         __FUNCTION__);
> +            return -EINVAL;
> +        }
> +    }
> +    else if (memcmp(dom->kernel_blob, "\102\132\150", 3) == 0) {
> +        ret = xc_try_bzip2_decode(dom, &dom->kernel_blob, &dom->kernel_size);
> +        if (ret < 0) {
> +            xc_dom_panic(XC_INVALID_KERNEL, "%s unable to BZIP2 decompress kernel",
> +                         __FUNCTION__);
> +            return -EINVAL;
> +        }
> +    }
> +    else if (memcmp(dom->kernel_blob, "\135\000", 2) == 0) {
> +        ret = xc_try_lzma_decode(dom, &dom->kernel_blob, &dom->kernel_size);
> +        if (ret < 0) {
> +            xc_dom_panic(XC_INVALID_KERNEL, "%s unable to LZMA decompress kernel\n",
>                           __FUNCTION__);
> +            return -EINVAL;
> +        }
> +    }
> +    else {
> +        xc_dom_panic(XC_INVALID_KERNEL, "%s: unknown compression format\n",
> +                     __FUNCTION__);
>          return -EINVAL;
>      }
>  
>      return elf_loader.probe(dom);
>  }
>  
> -static int xc_dom_probe_bzimage_kernel(struct xc_dom_image *dom)
> -{
> -    return check_bzimage_kernel(dom, 0);
> -}
> -
>  static int xc_dom_parse_bzimage_kernel(struct xc_dom_image *dom)
>  {
>      return elf_loader.parser(dom);

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


[-- Attachment #2: xen-loader-bzip2-lzma-xen34-fixed.patch --]
[-- Type: text/x-diff, Size: 10145 bytes --]

--- xen-3.4-testing.hg/tools/libxc/Makefile.orig	2009-08-20 20:02:53.000000000 +0300
+++ xen-3.4-testing.hg/tools/libxc/Makefile	2009-08-20 19:57:00.000000000 +0300
@@ -150,7 +150,7 @@
 	ln -sf $< $@
 
 libxenguest.so.$(MAJOR).$(MINOR): $(GUEST_PIC_OBJS) libxenctrl.so
-	$(CC) $(CFLAGS) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenguest.so.$(MAJOR) $(SHLIB_CFLAGS) -o $@ $(GUEST_PIC_OBJS) -lz -lxenctrl $(PTHREAD_LIBS)
+	$(CC) $(CFLAGS) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenguest.so.$(MAJOR) $(SHLIB_CFLAGS) -o $@ $(GUEST_PIC_OBJS) -lz -llzma -lbz2 -lxenctrl $(PTHREAD_LIBS)
 
 -include $(DEPS)
 
--- xen-3.4-testing.hg/tools/libxc/xc_dom_bzimageloader.c.orig	2009-08-20 20:02:53.000000000 +0300
+++ xen-3.4-testing.hg/tools/libxc/xc_dom_bzimageloader.c	2009-08-20 20:05:44.000000000 +0300
@@ -11,15 +11,208 @@
  * written 2006 by Gerd Hoffmann <kraxel@suse.de>.
  * written 2007 by Jeremy Fitzhardinge <jeremy@xensource.com>
  * written 2008 by Ian Campbell <ijc@hellion.org.uk>
+ * written 2009 by Chris Lalancette <clalance@redhat.com>
  *
  */
 #include <stdio.h>
 #include <stdlib.h>
 #include <inttypes.h>
+#include <bzlib.h>
+#include <lzma.h>
 
 #include "xg_private.h"
 #include "xc_dom.h"
 
+static inline uint64_t physmem(void)
+{
+	uint64_t ret = 0;
+
+	const long pagesize = sysconf(_SC_PAGESIZE);
+	const long pages = sysconf(_SC_PHYS_PAGES);
+	if (pagesize != -1 || pages != -1)
+		// According to docs, pagesize * pages can overflow.
+		// Simple case is 32-bit box with 4 GiB or more RAM,
+		// which may report exactly 4 GiB of RAM, and "long"
+		// being 32-bit will overflow. Casting to uint64_t
+		// hopefully avoids overflows in the near future.
+		ret = (uint64_t)(pagesize) * (uint64_t)(pages);
+
+    return ret;
+}
+
+static int xc_try_bzip2_decode(struct xc_dom_image *dom, void **blob, size_t *size)
+{
+    bz_stream stream;
+    int ret;
+    char *out_buf;
+    int retval = -1;
+    int outsize;
+    uint64_t total;
+
+    stream.bzalloc = NULL;
+    stream.bzfree = NULL;
+    stream.opaque = NULL;
+
+    ret = BZ2_bzDecompressInit(&stream, 0, 0);
+    if (ret != BZ_OK) {
+        xc_dom_printf("Error initting bz2 stream\n");
+        return -1;
+    }
+
+    /* sigh.  We don't know up-front how much memory we are going to need
+     * for the output buffer.  Allocate the output buffer to be equal
+     * the input buffer to start, and we'll realloc as needed.
+     */
+    outsize = dom->kernel_size;
+    out_buf = malloc(outsize);
+    if (out_buf == NULL) {
+        xc_dom_printf("Failed to alloc memory\n");
+        goto bzip2_cleanup;
+    }
+
+    stream.next_in = dom->kernel_blob;
+    stream.avail_in = dom->kernel_size;
+
+    stream.next_out = out_buf;
+    stream.avail_out = dom->kernel_size;
+
+    while (1) {
+        ret = BZ2_bzDecompress(&stream);
+        if (stream.avail_out == 0 || ret != BZ_OK) {
+            out_buf = realloc(out_buf, outsize * 2);
+            if (out_buf == NULL) {
+                xc_dom_printf("Failed to realloc memory\n");
+                break;
+            }
+
+            stream.next_out = out_buf + outsize;
+            stream.avail_out = (outsize * 2) - outsize;
+            outsize *= 2;
+        }
+
+        if (ret != BZ_OK) {
+            if (ret == BZ_STREAM_END) {
+                xc_dom_printf("Saw data stream end\n");
+                retval = 0;
+                break;
+            }
+            xc_dom_printf("BZIP error\n");
+        }
+    }
+
+    total = (stream.total_out_hi32 << 31) | stream.total_out_lo32;
+
+    xc_dom_printf("%s: BZIP2 decompress OK, 0x%zx -> 0x%lx\n", __FUNCTION__, *size, total);
+
+    *blob = out_buf;
+    *size = total;
+
+bzip2_cleanup:
+    BZ2_bzDecompressEnd(&stream);
+
+    return retval;
+}
+
+static int xc_try_lzma_decode(struct xc_dom_image *dom, void **blob, size_t *size)
+{
+    lzma_stream stream = LZMA_STREAM_INIT;
+    lzma_ret ret;
+    lzma_action action = LZMA_RUN;
+    unsigned char *out_buf;
+    int retval = -1;
+    int outsize;
+    const char *msg;
+
+    ret = lzma_alone_decoder(&stream, physmem() / 3);
+    if (ret != LZMA_OK) {
+        xc_dom_printf("Failed to init lzma stream decoder\n");
+        return -1;
+    }
+
+    /* sigh.  We don't know up-front how much memory we are going to need
+     * for the output buffer.  Allocate the output buffer to be equal
+     * the input buffer to start, and we'll realloc as needed.
+     */
+    outsize = dom->kernel_size;
+    out_buf = malloc(outsize);
+    if (out_buf == NULL) {
+        xc_dom_printf("Failed to alloc memory\n");
+        goto lzma_cleanup;
+    }
+
+    stream.next_in = dom->kernel_blob;
+    stream.avail_in = dom->kernel_size;
+
+    stream.next_out = out_buf;
+    stream.avail_out = dom->kernel_size;
+
+    while (1) {
+        ret = lzma_code(&stream, action);
+        if (stream.avail_out == 0 || ret != LZMA_OK) {
+            out_buf = realloc(out_buf, outsize * 2);
+            if (out_buf == NULL) {
+                xc_dom_printf("Failed to realloc memory\n");
+                break;
+            }
+
+            stream.next_out = out_buf + outsize;
+            stream.avail_out = (outsize * 2) - outsize;
+            outsize *= 2;
+        }
+
+        if (ret != LZMA_OK) {
+            if (ret == LZMA_STREAM_END) {
+                xc_dom_printf("Saw data stream end\n");
+                retval = 0;
+                break;
+            }
+
+            switch (ret) {
+            case LZMA_MEM_ERROR:
+                msg = strerror(ENOMEM);
+                break;
+
+            case LZMA_MEMLIMIT_ERROR:
+                msg = "Memory usage limit reached";
+                break;
+
+            case LZMA_FORMAT_ERROR:
+                msg = "File format not recognized";
+                break;
+
+            case LZMA_OPTIONS_ERROR:
+                // FIXME: Better message?
+                msg = "Unsupported compression options";
+                break;
+
+            case LZMA_DATA_ERROR:
+                msg = "File is corrupt";
+                break;
+
+            case LZMA_BUF_ERROR:
+                msg = "Unexpected end of input";
+                break;
+
+            default:
+                msg = "Internal program error (bug)";
+                break;
+            }
+            xc_dom_printf("%s: LZMA decompression error %s\n", __FUNCTION__, msg);
+            break;
+        }
+    }
+
+    xc_dom_printf("%s: LZMA decompress OK, 0x%zx -> 0x%zx\n", __FUNCTION__, *size, stream.total_out);
+
+    *blob = out_buf;
+    *size = stream.total_out;
+
+ lzma_cleanup:
+    lzma_end(&stream);
+
+    return retval;
+}
+
 struct setup_header {
 	uint8_t		_pad0[0x1f1];		/* skip uninteresting stuff */
 	uint8_t		setup_sects;
@@ -70,22 +263,22 @@
     return off;
 }
 
-static int check_bzimage_kernel(struct xc_dom_image *dom, int verbose)
+static int xc_dom_probe_bzimage_kernel(struct xc_dom_image *dom)
 {
     struct setup_header *hdr;
+    int ret;
 
     if ( dom->kernel_blob == NULL )
     {
-        if ( verbose )
-            xc_dom_panic(XC_INTERNAL_ERROR, "%s: no kernel image loaded\n",
-                         __FUNCTION__);
+        xc_dom_panic(XC_INTERNAL_ERROR, "%s: no kernel image loaded\n",
+                     __FUNCTION__);
         return -EINVAL;
     }
+
     if ( dom->kernel_size < sizeof(struct setup_header) )
     {
-        if ( verbose )
-            xc_dom_panic(XC_INTERNAL_ERROR, "%s: kernel image too small\n",
-                         __FUNCTION__);
+        xc_dom_panic(XC_INTERNAL_ERROR, "%s: kernel image too small\n",
+                     __FUNCTION__);
         return -EINVAL;
     }
 
@@ -93,39 +286,54 @@
 
     if ( memcmp(&hdr->header, HDR_MAGIC, HDR_MAGIC_SZ) != 0 )
     {
-        if ( verbose )
-            xc_dom_panic(XC_INVALID_KERNEL, "%s: kernel is not a bzImage\n",
-                         __FUNCTION__);
+        xc_dom_panic(XC_INVALID_KERNEL, "%s: kernel is not a bzImage\n",
+                     __FUNCTION__);
         return -EINVAL;
     }
 
     if ( hdr->version < VERSION(2,8) )
     {
-        if ( verbose )
-            xc_dom_panic(XC_INVALID_KERNEL, "%s: boot protocol too old (%04x)\n",
-                         __FUNCTION__, hdr->version);
+        xc_dom_panic(XC_INVALID_KERNEL, "%s: boot protocol too old (%04x)\n",
+                     __FUNCTION__, hdr->version);
         return -EINVAL;
     }
 
     dom->kernel_blob = dom->kernel_blob + payload_offset(hdr);
     dom->kernel_size = hdr->payload_length;
 
-    if ( xc_dom_try_gunzip(dom, &dom->kernel_blob, &dom->kernel_size) == -1 )
-    {
-        if ( verbose )
-            xc_dom_panic(XC_INVALID_KERNEL, "%s: unable to decompress kernel\n",
+    if (memcmp(dom->kernel_blob, "\037\213", 2) == 0) {
+        ret = xc_dom_try_gunzip(dom, &dom->kernel_blob, &dom->kernel_size);
+        if (ret == -1) {
+            xc_dom_panic(XC_INVALID_KERNEL, "%s: unable to gzip decompress kernel\n",
+                         __FUNCTION__);
+            return -EINVAL;
+        }
+    }
+    else if (memcmp(dom->kernel_blob, "\102\132\150", 3) == 0) {
+        ret = xc_try_bzip2_decode(dom, &dom->kernel_blob, &dom->kernel_size);
+        if (ret < 0) {
+            xc_dom_panic(XC_INVALID_KERNEL, "%s unable to BZIP2 decompress kernel",
+                         __FUNCTION__);
+            return -EINVAL;
+        }
+    }
+    else if (memcmp(dom->kernel_blob, "\135\000", 2) == 0) {
+        ret = xc_try_lzma_decode(dom, &dom->kernel_blob, &dom->kernel_size);
+        if (ret < 0) {
+            xc_dom_panic(XC_INVALID_KERNEL, "%s unable to LZMA decompress kernel\n",
                          __FUNCTION__);
+            return -EINVAL;
+        }
+    }
+    else {
+        xc_dom_panic(XC_INVALID_KERNEL, "%s: unknown compression format\n",
+                     __FUNCTION__);
         return -EINVAL;
     }
 
     return elf_loader.probe(dom);
 }
 
-static int xc_dom_probe_bzimage_kernel(struct xc_dom_image *dom)
-{
-    return check_bzimage_kernel(dom, 0);
-}
-
 static int xc_dom_parse_bzimage_kernel(struct xc_dom_image *dom)
 {
     return elf_loader.parser(dom);

[-- Attachment #3: xen-loader-bzip2-lzma-xen34-fixed-compilefix-gcc440.patch --]
[-- Type: text/x-diff, Size: 845 bytes --]

--- xen-3.4-testing.hg/tools/libxc/xc_dom_bzimageloader.c.prev	2009-08-20 20:05:44.000000000 +0300
+++ xen-3.4-testing.hg/tools/libxc/xc_dom_bzimageloader.c	2009-08-20 21:05:11.000000000 +0300
@@ -102,7 +102,7 @@
 
     total = (stream.total_out_hi32 << 31) | stream.total_out_lo32;
 
-    xc_dom_printf("%s: BZIP2 decompress OK, 0x%zx -> 0x%lx\n", __FUNCTION__, *size, total);
+    xc_dom_printf("%s: BZIP2 decompress OK, 0x%zx -> 0x%lx\n", __FUNCTION__, *size, (long unsigned int) total);
 
     *blob = out_buf;
     *size = total;
@@ -202,7 +202,7 @@
         }
     }
 
-    xc_dom_printf("%s: LZMA decompress OK, 0x%zx -> 0x%zx\n", __FUNCTION__, *size, stream.total_out);
+    xc_dom_printf("%s: LZMA decompress OK, 0x%zx -> 0x%zx\n", __FUNCTION__, *size, (size_t) stream.total_out);
 
     *blob = out_buf;
     *size = stream.total_out;

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Thu, Aug 20, 2009 at 09:34:08PM +0300, Pasi Kärkkäinen wrote:
> On Thu, Aug 13, 2009 at 09:47:27AM +0200, Chris Lalancette wrote:
> > All,
> >      Recent upstream kernels can be compressed using either gzip, bzip2, or
> > LZMA.  However, the PV kernel loader in Xen currently only understands gzip, and
> > will fail on the other two types.  The attached patch implements kernel
> > decompression for gzip, bzip2, and LZMA so that kernels compressed with any of
> > these methods can be launched.  Note that the patch is still a little bit rough,
> > but I thought I would post it to get feedback about what I'm doing wrong.
> >      I developed this against the RHEL-5 version of the xen tools, but a quick
> > look through xen-unstable shows that this should apply pretty easily to the
> > current code.
> > 
> 
> Attached are two patches:
> 
> - Patch by Chris modified to apply to Xen 3.4.1 (only a small Makefile change)
> - Patch by me to fix compilation with gcc 4.4.0.
> 

Hmm, actually this patch seems to break stubdom build/linking:

ld -nostdlib -L/root/tem/xen-3.4-testing.hg/stubdom/cross-root-i686/i686-xen-elf/lib  -m elf_i386 
-T arch/x86/minios-x86_32.lds /root/tem/xen-3.4-testing.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o  
-o /root/tem/xen-3.4-testing.hg/stubdom/mini-os-x86_32-ioemu/mini-os
ld: warning: section `.bss' type changed to PROGBITS 
/root/tem/xen-3.4-testing.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `xc_try_bzip2_decode':
/root/tem/xen-3.4-testing.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:56: undefined reference to `BZ2_bzDecompressInit'
/root/tem/xen-3.4-testing.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:111: undefined reference to `BZ2_bzDecompressEnd'
/root/tem/xen-3.4-testing.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:80: undefined reference to `BZ2_bzDecompress'
/root/tem/xen-3.4-testing.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:111: undefined reference to `BZ2_bzDecompressEnd'
/root/tem/xen-3.4-testing.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `xc_try_lzma_decode':
/root/tem/xen-3.4-testing.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:126: undefined reference to `lzma_alone_decoder'
/root/tem/xen-3.4-testing.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:211: undefined reference to `lzma_end'
/root/tem/xen-3.4-testing.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:150: undefined reference to `lzma_code'
/root/tem/xen-3.4-testing.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:211: undefined reference to `lzma_end'
make[2]: ***
[/root/tem/xen-3.4-testing.hg/stubdom/mini-os-x86_32-ioemu/mini-os] Error 1
make[2]: Leaving directory `/root/tem/xen-3.4-testing.hg/extras/mini-os'
make[1]: *** [ioemu-stubdom] Error 2
make[1]: Leaving directory `/root/tem/xen-3.4-testing.hg/stubdom'
make: *** [install-stubdom] Error 2


Any ideas how to fix that? Do we need to include bzip2 and lzma libs in stubdom/ ?

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 20/08/2009 21:14, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>> Attached are two patches:
>> 
>> - Patch by Chris modified to apply to Xen 3.4.1 (only a small Makefile
>> change)
>> - Patch by me to fix compilation with gcc 4.4.0.
>> 
> 
> Hmm, actually this patch seems to break stubdom build/linking:

I checked in a variant as c/s 20103, which at least for non-stubdom build of
libxenguest should do the right thing if liblzma-devel or libbz2-devel
packages (or whatever they are called) are not installed. See how that goes
with stubdom. We may just have to massage the Makefile a little more to kill
off the bz2/lzma stuff for stubdom. I'm not sure why libxenguest is getting
built for stubdom in the first place though. It's not likely to be very
useful!

 -- Keir

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 20/08/2009 22:15, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

>> Hmm, actually this patch seems to break stubdom build/linking:
> 
> I checked in a variant as c/s 20103, which at least for non-stubdom build of
> libxenguest should do the right thing if liblzma-devel or libbz2-devel
> packages (or whatever they are called) are not installed. See how that goes
> with stubdom. We may just have to massage the Makefile a little more to kill
> off the bz2/lzma stuff for stubdom. I'm not sure why libxenguest is getting
> built for stubdom in the first place though. It's not likely to be very
> useful!

Okay, I tested and fixed the stubdom case, so hopefully c/s 20104 will
actually work for you.

 -- Keir

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Thu, Aug 20, 2009 at 10:27:29PM +0100, Keir Fraser wrote:
> On 20/08/2009 22:15, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> 
> >> Hmm, actually this patch seems to break stubdom build/linking:
> > 
> > I checked in a variant as c/s 20103, which at least for non-stubdom build of
> > libxenguest should do the right thing if liblzma-devel or libbz2-devel
> > packages (or whatever they are called) are not installed. See how that goes
> > with stubdom. We may just have to massage the Makefile a little more to kill
> > off the bz2/lzma stuff for stubdom. I'm not sure why libxenguest is getting
> > built for stubdom in the first place though. It's not likely to be very
> > useful!
> 
> Okay, I tested and fixed the stubdom case, so hopefully c/s 20104 will
> actually work for you.
> 

xen-unstable "make tools && make stubdom" compiles for me with and without
libbz2-devel/liblzma-devel installed.

But even when I have liblzma-devel installed it doesn't seem to use it. 
ie. I only see -DHAVE_BZLIB in the build log and libxenguest.so.3.4.0 gets
only linked against -lbz2.

Investigating more..

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 21/08/2009 10:22, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>> Okay, I tested and fixed the stubdom case, so hopefully c/s 20104 will
>> actually work for you.
>> 
> 
> xen-unstable "make tools && make stubdom" compiles for me with and without
> libbz2-devel/liblzma-devel installed.
> 
> But even when I have liblzma-devel installed it doesn't seem to use it.
> ie. I only see -DHAVE_BZLIB in the build log and libxenguest.so.3.4.0 gets
> only linked against -lbz2.
> 
> Investigating more..

With that package installed, do you have a header /usr/include/lzma.h, or
otherwise have lzma.h on the standard include path? If not then we're doing
the right thing, since Chris's patch wants to #include <lzma.h>. If it is,
then the has_header lzma.h check in the libxc Makefile is broken somehow.

 -- Keir

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Chris Lalancette]

Keir Fraser wrote:
> On 21/08/2009 10:22, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
>>> Okay, I tested and fixed the stubdom case, so hopefully c/s 20104 will
>>> actually work for you.
>>>
>> xen-unstable "make tools && make stubdom" compiles for me with and without
>> libbz2-devel/liblzma-devel installed.
>>
>> But even when I have liblzma-devel installed it doesn't seem to use it.
>> ie. I only see -DHAVE_BZLIB in the build log and libxenguest.so.3.4.0 gets
>> only linked against -lbz2.
>>
>> Investigating more..
> 
> With that package installed, do you have a header /usr/include/lzma.h, or
> otherwise have lzma.h on the standard include path? If not then we're doing
> the right thing, since Chris's patch wants to #include <lzma.h>. If it is,
> then the has_header lzma.h check in the libxc Makefile is broken somehow.

Hm, there's also a somewhat confusing situation with LZMA.  I'm not sure I
completely understand it, but from what I can tell the lzma.h header file should
be coming from the new "xz" project, not from the lzma project.  That is, on my
Fedora 10 box, in order to get LZMA support compiled, I needed to install the
xz-devel package; lzma-devel was not sufficient.

-- 
Chris Lalancette

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Gerd Hoffmann]

On 08/20/09 22:14, Pasi Kärkkäinen wrote:
>
> Any ideas how to fix that? Do we need to include bzip2 and lzma libs in stubdom/ ?
>

Yes.  Otherwise bzip2+lzma support will not work in pvgrub.

cheers,
   Gerd

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 21/08/2009 10:38, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

>> xen-unstable "make tools && make stubdom" compiles for me with and without
>> libbz2-devel/liblzma-devel installed.
>> 
>> But even when I have liblzma-devel installed it doesn't seem to use it.
>> ie. I only see -DHAVE_BZLIB in the build log and libxenguest.so.3.4.0 gets
>> only linked against -lbz2.
>> 
>> Investigating more..
> 
> With that package installed, do you have a header /usr/include/lzma.h, or
> otherwise have lzma.h on the standard include path? If not then we're doing
> the right thing, since Chris's patch wants to #include <lzma.h>. If it is,
> then the has_header lzma.h check in the libxc Makefile is broken somehow.

Did you do a 'make clean' in libxc directory before rebuilding after
installing liblzma-devel? That would be necessary.

 -- Keir

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 12:22:43PM +0300, Pasi Kärkkäinen wrote:
> On Thu, Aug 20, 2009 at 10:27:29PM +0100, Keir Fraser wrote:
> > On 20/08/2009 22:15, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> > 
> > >> Hmm, actually this patch seems to break stubdom build/linking:
> > > 
> > > I checked in a variant as c/s 20103, which at least for non-stubdom build of
> > > libxenguest should do the right thing if liblzma-devel or libbz2-devel
> > > packages (or whatever they are called) are not installed. See how that goes
> > > with stubdom. We may just have to massage the Makefile a little more to kill
> > > off the bz2/lzma stuff for stubdom. I'm not sure why libxenguest is getting
> > > built for stubdom in the first place though. It's not likely to be very
> > > useful!
> > 
> > Okay, I tested and fixed the stubdom case, so hopefully c/s 20104 will
> > actually work for you.
> > 
> 
> xen-unstable "make tools && make stubdom" compiles for me with and without
> libbz2-devel/liblzma-devel installed.
> 
> But even when I have liblzma-devel installed it doesn't seem to use it. 
> ie. I only see -DHAVE_BZLIB in the build log and libxenguest.so.3.4.0 gets
> only linked against -lbz2.
> 
> Investigating more..
> 

Ok, that was a stupid problem. I tried to build on Ubuntu box now (instead
of Fedora), and it seems lzma-dev was not the correct/needed package, 
but I had to install liblzma-dev.

So it seems to build OK for me. I'll verify also on Fedora later today.

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 11:42:46AM +0200, Chris Lalancette wrote:
> Keir Fraser wrote:
> > On 21/08/2009 10:22, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> > 
> >>> Okay, I tested and fixed the stubdom case, so hopefully c/s 20104 will
> >>> actually work for you.
> >>>
> >> xen-unstable "make tools && make stubdom" compiles for me with and without
> >> libbz2-devel/liblzma-devel installed.
> >>
> >> But even when I have liblzma-devel installed it doesn't seem to use it.
> >> ie. I only see -DHAVE_BZLIB in the build log and libxenguest.so.3.4.0 gets
> >> only linked against -lbz2.
> >>
> >> Investigating more..
> > 
> > With that package installed, do you have a header /usr/include/lzma.h, or
> > otherwise have lzma.h on the standard include path? If not then we're doing
> > the right thing, since Chris's patch wants to #include <lzma.h>. If it is,
> > then the has_header lzma.h check in the libxc Makefile is broken somehow.
> 
> Hm, there's also a somewhat confusing situation with LZMA.  I'm not sure I
> completely understand it, but from what I can tell the lzma.h header file should
> be coming from the new "xz" project, not from the lzma project.  That is, on my
> Fedora 10 box, in order to get LZMA support compiled, I needed to install the
> xz-devel package; lzma-devel was not sufficient.
> 

Yeah I noticed this too. In debian/ubuntu liblzma-dev is built from "xz"
source package.

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 11:43:27AM +0200, Gerd Hoffmann wrote:
> On 08/20/09 22:14, Pasi Kärkkäinen wrote:
> >
> >Any ideas how to fix that? Do we need to include bzip2 and lzma libs in 
> >stubdom/ ?
> >
> 
> Yes.  Otherwise bzip2+lzma support will not work in pvgrub.
> 

Good point. pvgrub should support these aswell.

Keir: Would you like to add those libs to stubdom? 

(Note the lzma.h is coming from that "xz" package..)

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 21/08/2009 10:49, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>>> 
>>> Any ideas how to fix that? Do we need to include bzip2 and lzma libs in
>>> stubdom/ ?
>>> 
>> 
>> Yes.  Otherwise bzip2+lzma support will not work in pvgrub.
>> 
> 
> Good point. pvgrub should support these aswell.
> 
> Keir: Would you like to add those libs to stubdom?
> 
> (Note the lzma.h is coming from that "xz" package..)

No. :-) But feel free to make a patch! You'll just need to modify
libxc/Makefile too to force those libs on instead of off, as well as
download and build the libs of course.

 -- Keir

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 21/08/2009 10:54, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

>> Good point. pvgrub should support these aswell.
>> 
>> Keir: Would you like to add those libs to stubdom?
>> 
>> (Note the lzma.h is coming from that "xz" package..)
> 
> No. :-) But feel free to make a patch! You'll just need to modify
> libxc/Makefile too to force those libs on instead of off, as well as
> download and build the libs of course.

Actually I just checked in c/s 20105 to get rid of the minios-specific hack
in that Makefile. So all you need to do is patch to download and build the
libs. If you make a patch, I will arrange for the lib tarballs to be hosted
on xenbits and adjust the patch URLs appropriately.

 -- Keir

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 11:13:23AM +0100, Keir Fraser wrote:
> On 21/08/2009 10:54, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> 
> >> Good point. pvgrub should support these aswell.
> >> 
> >> Keir: Would you like to add those libs to stubdom?
> >> 
> >> (Note the lzma.h is coming from that "xz" package..)
> > 
> > No. :-) But feel free to make a patch! You'll just need to modify
> > libxc/Makefile too to force those libs on instead of off, as well as
> > download and build the libs of course.
> 
> Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> in that Makefile. So all you need to do is patch to download and build the
> libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> on xenbits and adjust the patch URLs appropriately.
> 

Ok :)

c/s 20105 stubdom fails to build now, but I guess that's expected until the
patch for adding the libs is done.

I'll take a look at it today.

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 21/08/2009 11:57, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>> Actually I just checked in c/s 20105 to get rid of the minios-specific hack
>> in that Makefile. So all you need to do is patch to download and build the
>> libs. If you make a patch, I will arrange for the lib tarballs to be hosted
>> on xenbits and adjust the patch URLs appropriately.
>> 
> 
> Ok :)
> 
> c/s 20105 stubdom fails to build now, but I guess that's expected until the
> patch for adding the libs is done.

No, that isn't expected. It builds okay for me.

 -- Keir

> I'll take a look at it today.

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 02:09:04PM +0100, Keir Fraser wrote:
> On 21/08/2009 11:57, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> 
> >> Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> >> in that Makefile. So all you need to do is patch to download and build the
> >> libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> >> on xenbits and adjust the patch URLs appropriately.
> >> 
> > 
> > Ok :)
> > 
> > c/s 20105 stubdom fails to build now, but I guess that's expected until the
> > patch for adding the libs is done.
> 
> No, that isn't expected. It builds okay for me.
> 

For me it builds _without_ libbz2-devel/liblzma-devel.

With those installed stubdom build fails with errors about not finding the
headers for those libs.

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 05:01:49PM +0300, Pasi Kärkkäinen wrote:
> On Fri, Aug 21, 2009 at 02:09:04PM +0100, Keir Fraser wrote:
> > On 21/08/2009 11:57, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> > 
> > >> Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> > >> in that Makefile. So all you need to do is patch to download and build the
> > >> libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> > >> on xenbits and adjust the patch URLs appropriately.
> > >> 
> > > 
> > > Ok :)
> > > 
> > > c/s 20105 stubdom fails to build now, but I guess that's expected until the
> > > patch for adding the libs is done.
> > 
> > No, that isn't expected. It builds okay for me.
> > 
> 
> For me it builds _without_ libbz2-devel/liblzma-devel.
> 
> With those installed stubdom build fails with errors about not finding the
> headers for those libs.
> 

c/s 20104: 
	- "make tools && make stubdom" without libbz2-devel/liblzma-devel succeeds
	- "make tools && make stubdom" with libbz2-devel/liblzma-devel succeeds

c/s 20105:
	- "make tools && make stubdom" without libbz2-devel/liblzma-devel succeeds
	- "make tools && make stubdom" with libbz2-devel/liblzma-devel:	tools succeeds, stubdom fails


xc_dom_bzimageloader.c:26:19: error: bzlib.h: No such file or directory
xc_dom_bzimageloader.c: In function 'xc_try_bzip2_decode':

xc_dom_bzimageloader.c:124:18: error: lzma.h: No such file or directory
xc_dom_bzimageloader.c: In function 'xc_try_lzma_decode':


-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 06:03:29PM +0300, Pasi Kärkkäinen wrote:
> On Fri, Aug 21, 2009 at 05:01:49PM +0300, Pasi Kärkkäinen wrote:
> > On Fri, Aug 21, 2009 at 02:09:04PM +0100, Keir Fraser wrote:
> > > On 21/08/2009 11:57, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:
> > > 
> > > >> Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> > > >> in that Makefile. So all you need to do is patch to download and build the
> > > >> libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> > > >> on xenbits and adjust the patch URLs appropriately.
> > > >> 
> > > > 
> > > > Ok :)
> > > > 
> > > > c/s 20105 stubdom fails to build now, but I guess that's expected until the
> > > > patch for adding the libs is done.
> > > 
> > > No, that isn't expected. It builds okay for me.
> > > 
> > 
> > For me it builds _without_ libbz2-devel/liblzma-devel.
> > 
> > With those installed stubdom build fails with errors about not finding the
> > headers for those libs.
> > 
> 
> c/s 20104: 
> 	- "make tools && make stubdom" without libbz2-devel/liblzma-devel succeeds
> 	- "make tools && make stubdom" with libbz2-devel/liblzma-devel succeeds
> 
> c/s 20105:
> 	- "make tools && make stubdom" without libbz2-devel/liblzma-devel succeeds
> 	- "make tools && make stubdom" with libbz2-devel/liblzma-devel:	tools succeeds, stubdom fails
> 
> 
> xc_dom_bzimageloader.c:26:19: error: bzlib.h: No such file or directory
> xc_dom_bzimageloader.c: In function 'xc_try_bzip2_decode':
> 
> xc_dom_bzimageloader.c:124:18: error: lzma.h: No such file or directory
> xc_dom_bzimageloader.c: In function 'xc_try_lzma_decode':
> 

And I'm working on a patch to add these libs to stubdom atm..

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 21/08/2009 15:01, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>>> c/s 20105 stubdom fails to build now, but I guess that's expected until the
>>> patch for adding the libs is done.
>> 
>> No, that isn't expected. It builds okay for me.
>> 
> 
> For me it builds _without_ libbz2-devel/liblzma-devel.
> 
> With those installed stubdom build fails with errors about not finding the
> headers for those libs.

Uh, yeah, I should have left the Makefile hack in. The has_header check will
go check for a standard host header, not for the header within the stubdom
build environment. I'll revert 20105: it was a bad idea.

 -- Keir

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 11:13:23AM +0100, Keir Fraser wrote:
> On 21/08/2009 10:54, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> 
> >> Good point. pvgrub should support these aswell.
> >> 
> >> Keir: Would you like to add those libs to stubdom?
> >> 
> >> (Note the lzma.h is coming from that "xz" package..)
> > 
> > No. :-) But feel free to make a patch! You'll just need to modify
> > libxc/Makefile too to force those libs on instead of off, as well as
> > download and build the libs of course.
> 
> Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> in that Makefile. So all you need to do is patch to download and build the
> libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> on xenbits and adjust the patch URLs appropriately.
> 

I can now see why you didn't want to do that.. :) I managed to get the libs to 
compile, but now I'm having other problems..

"make stubdom" gets to the point where it compiles ioemu-stubdom, but it fails 
to link because xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o has 
undefined references in xc_try_bzip2_decode and xc_try_lzma_decode functions.. 

I guess only pvgrub would need to have this stuff in? I've been trying to
figure out how the makefile magic works, but haven't really understood it yet.

Any tips how to make ioemu-stubdom compile/link without including bzip2/lzma stuff into it? 

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Fri, Aug 21, 2009 at 11:12:00PM +0300, Pasi Kärkkäinen wrote:
> On Fri, Aug 21, 2009 at 11:13:23AM +0100, Keir Fraser wrote:
> > On 21/08/2009 10:54, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> > 
> > >> Good point. pvgrub should support these aswell.
> > >> 
> > >> Keir: Would you like to add those libs to stubdom?
> > >> 
> > >> (Note the lzma.h is coming from that "xz" package..)
> > > 
> > > No. :-) But feel free to make a patch! You'll just need to modify
> > > libxc/Makefile too to force those libs on instead of off, as well as
> > > download and build the libs of course.
> > 
> > Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> > in that Makefile. So all you need to do is patch to download and build the
> > libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> > on xenbits and adjust the patch URLs appropriately.
> > 
> 
> I can now see why you didn't want to do that.. :) I managed to get the libs to 
> compile, but now I'm having other problems..
> 
> "make stubdom" gets to the point where it compiles ioemu-stubdom, but it fails 
> to link because xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o has 
> undefined references in xc_try_bzip2_decode and xc_try_lzma_decode functions.. 
> 
> I guess only pvgrub would need to have this stuff in? I've been trying to
> figure out how the makefile magic works, but haven't really understood it yet.
> 
> Any tips how to make ioemu-stubdom compile/link without including bzip2/lzma stuff into it? 
> 

Actually I guess I could add the missing libs to extras/mini-os/Makefile to
APP_LDLIBS to get them included into ioemu-stubdom?

Is that the right way to do it?

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Keir Fraser]

On 21/08/2009 22:22, "Pasi Kärkkäinen" <pasik@iki.fi> wrote:

>> I guess only pvgrub would need to have this stuff in? I've been trying to
>> figure out how the makefile magic works, but haven't really understood it
>> yet.
>> 
>> Any tips how to make ioemu-stubdom compile/link without including bzip2/lzma
>> stuff into it? 
>> 
> 
> Actually I guess I could add the missing libs to extras/mini-os/Makefile to
> APP_LDLIBS to get them included into ioemu-stubdom?
> 
> Is that the right way to do it?

Stefano can probably best advise you.

 -- Keir

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2374 bytes --]

On Fri, 21 Aug 2009, Pasi Kärkkäinen wrote:
> On Fri, Aug 21, 2009 at 11:12:00PM +0300, Pasi Kärkkäinen wrote:
> > On Fri, Aug 21, 2009 at 11:13:23AM +0100, Keir Fraser wrote:
> > > On 21/08/2009 10:54, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> > > 
> > > >> Good point. pvgrub should support these aswell.
> > > >> 
> > > >> Keir: Would you like to add those libs to stubdom?
> > > >> 
> > > >> (Note the lzma.h is coming from that "xz" package..)
> > > > 
> > > > No. :-) But feel free to make a patch! You'll just need to modify
> > > > libxc/Makefile too to force those libs on instead of off, as well as
> > > > download and build the libs of course.
> > > 
> > > Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> > > in that Makefile. So all you need to do is patch to download and build the
> > > libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> > > on xenbits and adjust the patch URLs appropriately.
> > > 
> > 
> > I can now see why you didn't want to do that.. :) I managed to get the libs to 
> > compile, but now I'm having other problems..
> > 
> > "make stubdom" gets to the point where it compiles ioemu-stubdom, but it fails 
> > to link because xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o has 
> > undefined references in xc_try_bzip2_decode and xc_try_lzma_decode functions.. 
> > 
> > I guess only pvgrub would need to have this stuff in? I've been trying to
> > figure out how the makefile magic works, but haven't really understood it yet.
> > 
> > Any tips how to make ioemu-stubdom compile/link without including bzip2/lzma stuff into it? 
> > 
> 
> Actually I guess I could add the missing libs to extras/mini-os/Makefile to
> APP_LDLIBS to get them included into ioemu-stubdom?
> 
> Is that the right way to do it?
> 

bzlib and lzma are libxc dependencies now, so firstly you have to add
the two libraries to the stubdom build system, take a look at
pciutils in stubdom/Makefile, that is a good example of how a new
library is added.
Then you also need to add them both at least to the libxc target, take a
look at zlib (and cross-zlib), that should be close to what you need.
Finally you need to tweak the libxc Makefile for the stubdom case so
that the two libraries are correctly added to zlib-options.

Have fun! ;)

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH]: Implement bzip2 and LZMA loaders / fixed patch for Xen 3.4.1

[Pasi Kärkkäinen]

On Mon, Aug 24, 2009 at 03:04:00PM +0100, Stefano Stabellini wrote:
> On Fri, 21 Aug 2009, Pasi Kärkkäinen wrote:
> > On Fri, Aug 21, 2009 at 11:12:00PM +0300, Pasi Kärkkäinen wrote:
> > > On Fri, Aug 21, 2009 at 11:13:23AM +0100, Keir Fraser wrote:
> > > > On 21/08/2009 10:54, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> > > > 
> > > > >> Good point. pvgrub should support these aswell.
> > > > >> 
> > > > >> Keir: Would you like to add those libs to stubdom?
> > > > >> 
> > > > >> (Note the lzma.h is coming from that "xz" package..)
> > > > > 
> > > > > No. :-) But feel free to make a patch! You'll just need to modify
> > > > > libxc/Makefile too to force those libs on instead of off, as well as
> > > > > download and build the libs of course.
> > > > 
> > > > Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> > > > in that Makefile. So all you need to do is patch to download and build the
> > > > libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> > > > on xenbits and adjust the patch URLs appropriately.
> > > > 
> > > 
> > > I can now see why you didn't want to do that.. :) I managed to get the libs to 
> > > compile, but now I'm having other problems..
> > > 
> > > "make stubdom" gets to the point where it compiles ioemu-stubdom, but it fails 
> > > to link because xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o has 
> > > undefined references in xc_try_bzip2_decode and xc_try_lzma_decode functions.. 
> > > 
> > > I guess only pvgrub would need to have this stuff in? I've been trying to
> > > figure out how the makefile magic works, but haven't really understood it yet.
> > > 
> > > Any tips how to make ioemu-stubdom compile/link without including bzip2/lzma stuff into it? 
> > > 
> > 
> > Actually I guess I could add the missing libs to extras/mini-os/Makefile to
> > APP_LDLIBS to get them included into ioemu-stubdom?
> > 
> > Is that the right way to do it?
> > 
> 
> bzlib and lzma are libxc dependencies now, so firstly you have to add
> the two libraries to the stubdom build system, take a look at
> pciutils in stubdom/Makefile, that is a good example of how a new
> library is added.

This is done, and works. libbz2.a and liblzma.a end up in
stubdom/cross-root-i686/i686-xen-elf/lib/ directory on my 32bit host.

> Then you also need to add them both at least to the libxc target, take a
> look at zlib (and cross-zlib), that should be close to what you need.

Done.

> Finally you need to tweak the libxc Makefile for the stubdom case so
> that the two libraries are correctly added to zlib-options.
> 

So you mean xen-unstable/tools/libxc/Makefile here? This part?

ifeq ($(CONFIG_MiniOS),y)
zlib-options =



The problem I'm seeing now is "ioemu" target from xen-unstable/stubdom/Makefile fails to
link:

ld -nostdlib -L/root/tem/xen-unstable.hg/stubdom/cross-root-i686/i686-xen-elf/lib  -m elf_i386 -T arch/x86/minios-x86_32.lds
/root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o  -o /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os
ld: warning: section `.bss' type changed to PROGBITS 
/root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `xc_try_bzip2_decode':
/root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:42: undefined reference to `BZ2_bzDecompressInit'
/root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:105: undefined reference to `BZ2_bzDecompressEnd'
/root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:69: undefined reference to `BZ2_bzDecompress'
/root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:105: undefined reference to `BZ2_bzDecompressEnd'
/root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `xc_try_lzma_decode':
/root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:158: undefined reference to `lzma_alone_decoder'
/root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:253: undefined reference to `lzma_end'
/root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:185: undefined reference to `lzma_code'
/root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:253: undefined reference to `lzma_end'
make[1]: ***
[/root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os] Error 1
make[1]: Leaving directory `/root/tem/xen-unstable.hg/extras/mini-os'
make: *** [ioemu-stubdom] Error 2

So basicly I should add libbz2.a and liblzma.a to the list of files to link,
but I'm a bit lost where this is happening. Should I edit
xen-unstable/extras/mini-os/Makefile to add those libs or somewhere else? 

I tried adding those libs to xen-unstable/stubdom/Makefile to "ioemu" TARGET_LDFLAGS, but
that didn't seem to help.

All tips welcome :)

-- Pasi

Re: [PATCH]: Implement bzip2 and LZMA loaders / xen-unstable stubdom pvgrub support

[Pasi Kärkkäinen]

[-- Attachment #1: Type: text/plain, Size: 6285 bytes --]

On Tue, Aug 25, 2009 at 06:23:19PM +0300, Pasi Kärkkäinen wrote:
> On Mon, Aug 24, 2009 at 03:04:00PM +0100, Stefano Stabellini wrote:
> > On Fri, 21 Aug 2009, Pasi Kärkkäinen wrote:
> > > On Fri, Aug 21, 2009 at 11:12:00PM +0300, Pasi Kärkkäinen wrote:
> > > > On Fri, Aug 21, 2009 at 11:13:23AM +0100, Keir Fraser wrote:
> > > > > On 21/08/2009 10:54, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> > > > > 
> > > > > >> Good point. pvgrub should support these aswell.
> > > > > >> 
> > > > > >> Keir: Would you like to add those libs to stubdom?
> > > > > >> 
> > > > > >> (Note the lzma.h is coming from that "xz" package..)
> > > > > > 
> > > > > > No. :-) But feel free to make a patch! You'll just need to modify
> > > > > > libxc/Makefile too to force those libs on instead of off, as well as
> > > > > > download and build the libs of course.
> > > > > 
> > > > > Actually I just checked in c/s 20105 to get rid of the minios-specific hack
> > > > > in that Makefile. So all you need to do is patch to download and build the
> > > > > libs. If you make a patch, I will arrange for the lib tarballs to be hosted
> > > > > on xenbits and adjust the patch URLs appropriately.
> > > > > 
> > > > 
> > > > I can now see why you didn't want to do that.. :) I managed to get the libs to 
> > > > compile, but now I'm having other problems..
> > > > 
> > > > "make stubdom" gets to the point where it compiles ioemu-stubdom, but it fails 
> > > > to link because xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o has 
> > > > undefined references in xc_try_bzip2_decode and xc_try_lzma_decode functions.. 
> > > > 
> > > > I guess only pvgrub would need to have this stuff in? I've been trying to
> > > > figure out how the makefile magic works, but haven't really understood it yet.
> > > > 
> > > > Any tips how to make ioemu-stubdom compile/link without including bzip2/lzma stuff into it? 
> > > > 
> > > 
> > > Actually I guess I could add the missing libs to extras/mini-os/Makefile to
> > > APP_LDLIBS to get them included into ioemu-stubdom?
> > > 
> > > Is that the right way to do it?
> > > 
> > 
> > bzlib and lzma are libxc dependencies now, so firstly you have to add
> > the two libraries to the stubdom build system, take a look at
> > pciutils in stubdom/Makefile, that is a good example of how a new
> > library is added.
> 
> This is done, and works. libbz2.a and liblzma.a end up in
> stubdom/cross-root-i686/i686-xen-elf/lib/ directory on my 32bit host.
> 
> > Then you also need to add them both at least to the libxc target, take a
> > look at zlib (and cross-zlib), that should be close to what you need.
> 
> Done.
> 
> > Finally you need to tweak the libxc Makefile for the stubdom case so
> > that the two libraries are correctly added to zlib-options.
> > 
> 
> So you mean xen-unstable/tools/libxc/Makefile here? This part?
> 
> ifeq ($(CONFIG_MiniOS),y)
> zlib-options =
> 
> 
> 
> The problem I'm seeing now is "ioemu" target from xen-unstable/stubdom/Makefile fails to
> link:
> 
> ld -nostdlib -L/root/tem/xen-unstable.hg/stubdom/cross-root-i686/i686-xen-elf/lib  -m elf_i386 -T arch/x86/minios-x86_32.lds
> /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o  -o /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os
> ld: warning: section `.bss' type changed to PROGBITS 
> /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `xc_try_bzip2_decode':
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:42: undefined reference to `BZ2_bzDecompressInit'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:105: undefined reference to `BZ2_bzDecompressEnd'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:69: undefined reference to `BZ2_bzDecompress'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:105: undefined reference to `BZ2_bzDecompressEnd'
> /root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `xc_try_lzma_decode':
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:158: undefined reference to `lzma_alone_decoder'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:253: undefined reference to `lzma_end'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:185: undefined reference to `lzma_code'
> /root/tem/xen-unstable.hg/stubdom/libxc-x86_32/xc_dom_bzimageloader.c:253: undefined reference to `lzma_end'
> make[1]: ***
> [/root/tem/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os] Error 1
> make[1]: Leaving directory `/root/tem/xen-unstable.hg/extras/mini-os'
> make: *** [ioemu-stubdom] Error 2
> 
> So basicly I should add libbz2.a and liblzma.a to the list of files to link,
> but I'm a bit lost where this is happening. Should I edit
> xen-unstable/extras/mini-os/Makefile to add those libs or somewhere else? 
> 
> I tried adding those libs to xen-unstable/stubdom/Makefile to "ioemu" TARGET_LDFLAGS, but
> that didn't seem to help.
> 
> All tips welcome :)
> 

Attached are my current work-in-progress patches. It still doesn't link properly.. 

Stefano: Would you like to take a look? The patches are against current xen-unstable.

I'm getting weird linking error about stdin/stdout/stderr missing.. 
libc (newlib) is definitely linked in, so I'm not sure what's happening.. 

ld -nostdlib -L/root/xen-unstable.hg/stubdom/cross-root-i686/i686-xen-elf/lib -m elf_i386 -T arch/x86/minios-x86_32.lds
/root/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o -o /root/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os

xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `bzopen_or_bzdopen':
xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1411: undefined reference to `__ctype_b_loc'
xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1447: undefined reference to `stdin'
xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1447: undefined reference to `stdout'
...

In the earlier step mini-os.o was definitely linked against -lc .. 

"strings /path/mini-os.o | grep -i stdin" doesn't give anything though.. 
I wonder why the stuff doesn't get included there.

-- Pasi


[-- Attachment #2: stubdom-Makefile-bzip2-lzma-support-v1.patch --]
[-- Type: text/x-diff, Size: 3984 bytes --]

--- xen-unstable.hg/stubdom/Makefile.backup.orig	2009-09-01 18:30:06.000000000 +0300
+++ xen-unstable.hg/stubdom/Makefile	2009-09-01 20:32:20.000000000 +0300
@@ -11,6 +11,14 @@
 ZLIB_URL=$(XEN_EXTFILES_URL)
 ZLIB_VERSION=1.2.3
 
+#BZIP2_URL?=http://www.bzip.org/1.0.5/bzip2-1.0.5.tar.gz
+BZIP2_URL=http://www.bzip.org/1.0.5
+BZIP2_VERSION=1.0.5
+
+#LZMA_URL?=http://tukaani.org/xz/xz-4.999.9beta.tar.gz
+LZMA_URL=http://tukaani.org/xz
+LZMA_VERSION=4.999.9beta
+
 #LIBPCI_URL?=http://www.kernel.org/pub/software/utils/pciutils
 LIBPCI_URL?=$(XEN_EXTFILES_URL)
 LIBPCI_VERSION=2.2.9
@@ -134,6 +142,51 @@
 	  $(MAKE) libz.a && \
 	  $(MAKE) install )
 
+#############
+# Cross-bzip2
+#############
+
+bzip2-$(BZIP2_VERSION).tar.gz:
+	$(WGET) $(BZIP2_URL)/$@
+
+bzip2-$(XEN_TARGET_ARCH): bzip2-$(BZIP2_VERSION).tar.gz 
+	tar xzf $<
+	mv bzip2-$(BZIP2_VERSION) $@
+
+BZIP2_STAMPFILE=$(CROSS_ROOT)/$(GNU_TARGET_ARCH)-xen-elf/lib/libbz2.a
+.PHONY: cross-bzip2
+cross-bzip2: $(BZIP2_STAMPFILE)
+$(BZIP2_STAMPFILE): bzip2-$(XEN_TARGET_ARCH) $(NEWLIB_STAMPFILE)
+	( cd $< && \
+	  CFLAGS="$(TARGET_CPPFLAGS) $(TARGET_CFLAGS)" CC=$(CC) $(MAKE) && \
+	  $(INSTALL_DATA) libbz2.a $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/lib && \
+	  $(INSTALL_DATA) bzlib.h $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/include && \
+	  $(INSTALL_DATA) bzlib_private.h $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/include )
+
+
+############
+# Cross-lzma
+############
+
+xz-$(LZMA_VERSION).tar.gz:
+	$(WGET) $(LZMA_URL)/$@
+
+xz-$(XEN_TARGET_ARCH): xz-$(LZMA_VERSION).tar.gz 
+	tar xzf $<
+	mv xz-$(LZMA_VERSION) $@
+
+LZMA_STAMPFILE=$(CROSS_ROOT)/$(GNU_TARGET_ARCH)-xen-elf/lib/liblzma.a
+.PHONY: cross-lzma
+cross-lzma: $(LZMA_STAMPFILE)
+$(LZMA_STAMPFILE): xz-$(XEN_TARGET_ARCH) $(NEWLIB_STAMPFILE)
+	( cd $< && \
+	  CFLAGS="$(TARGET_CPPFLAGS) $(TARGET_CFLAGS)" CC=$(CC) ./configure --prefix=$(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf --enable-shared=no && \
+	  cd src/liblzma && $(MAKE) && cd ../../ && \
+	  $(INSTALL_DATA) src/liblzma/.libs/liblzma.a $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/lib && \
+	  $(INSTALL_DATA) src/liblzma/api/lzma.h $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/include && \
+	  cp -r src/liblzma/api/lzma $(CROSS_PREFIX)/$(GNU_TARGET_ARCH)-xen-elf/include/ )
+
+
 ##############
 # Cross-libpci
 ##############
@@ -210,7 +263,7 @@
 #######
 
 .PHONY: $(CROSS_ROOT)
-$(CROSS_ROOT): cross-newlib cross-zlib cross-libpci
+$(CROSS_ROOT): cross-newlib cross-zlib cross-bzip2 cross-lzma cross-libpci
 
 $(XEN_ROOT)/tools/ioemu-dir:
 	$(MAKE) -C $(XEN_ROOT)/tools ioemu-dir-find
@@ -265,15 +318,15 @@
 
 .PHONY: libxc
 libxc: libxc-$(XEN_TARGET_ARCH)/libxenctrl.a libxc-$(XEN_TARGET_ARCH)/libxenguest.a
-libxc-$(XEN_TARGET_ARCH)/libxenctrl.a libxc-$(XEN_TARGET_ARCH)/libxenguest.a:: cross-zlib
-	CPPFLAGS="$(TARGET_CPPFLAGS)" CFLAGS="$(TARGET_CFLAGS)" $(MAKE) -C libxc-$(XEN_TARGET_ARCH)
+libxc-$(XEN_TARGET_ARCH)/libxenctrl.a libxc-$(XEN_TARGET_ARCH)/libxenguest.a:: cross-zlib cross-bzip2 cross-lzma
+	CPPFLAGS="$(TARGET_CPPFLAGS) -DHAVE_BZLIB -DHAVE_LZMA" CFLAGS="$(TARGET_CFLAGS) -DHAVE_BZLIB -DHAVE_LZMA" $(MAKE) -C libxc-$(XEN_TARGET_ARCH)
 
 #######
 # ioemu
 #######
 
 .PHONY: ioemu
-ioemu: cross-zlib cross-libpci libxc
+ioemu: cross-zlib cross-bzip2 cross-lzma cross-libpci libxc
 	[ -f ioemu/config-host.mak ] || \
 	  ( $(absolutify_xen_root); \
 	    $(buildmakevars2shellvars); \
@@ -339,7 +392,7 @@
 
 .PHONY: pv-grub
 pv-grub: mini-os-$(XEN_TARGET_ARCH)-grub libxc grub
-	DEF_CPPFLAGS="$(TARGET_CPPFLAGS)" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS)" DEF_LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
+	DEF_CPPFLAGS="$(TARGET_CPPFLAGS) -DHAVE_BZLIB -DHAVE_LZMA" DEF_CFLAGS="-DCONFIG_GRUB $(TARGET_CFLAGS) -DHAVE_BZLIB -DHAVE_LZMA" DEF_LDFLAGS="$(TARGET_LDFLAGS) -lbz2 -llzma" $(MAKE) -C $(MINI_OS) OBJ_DIR=$(CURDIR)/$< APP_OBJS=$(CURDIR)/grub-$(XEN_TARGET_ARCH)/main.a
 
 #########
 # install

[-- Attachment #3: extras-minios-Makefile-bzip2-lzma-support-v1.patch --]
[-- Type: text/x-diff, Size: 316 bytes --]

--- xen-unstable.hg/extras/mini-os/Makefile.backup.orig	2009-09-01 18:30:06.000000000 +0300
+++ xen-unstable.hg/extras/mini-os/Makefile	2009-09-01 21:43:14.000000000 +0300
@@ -83,6 +83,8 @@
 APP_LDLIBS += -lpci
 APP_LDLIBS += -lz
 APP_LDLIBS += -lm
+APP_LDLIBS += -lbz2
+APP_LDLIBS += -llzma
 LDLIBS += -lc
 endif
 

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH]: Implement bzip2 and LZMA loaders / xen-unstable stubdom pvgrub support

[Pasi Kärkkäinen]

Hello again,

Stefano: Have you ever seen this kind of linking failures with
stubdoms/ioemu? 

Any ideas?

-- Pasi

On Tue, Sep 01, 2009 at 10:08:55PM +0300, Pasi Kärkkäinen wrote:
> 
> Attached are my current work-in-progress patches. It still doesn't link properly.. 
> 
> Stefano: Would you like to take a look? The patches are against current xen-unstable.
> 
> I'm getting weird linking error about stdin/stdout/stderr missing.. 
> libc (newlib) is definitely linked in, so I'm not sure what's happening.. 
> 
> ld -nostdlib -L/root/xen-unstable.hg/stubdom/cross-root-i686/i686-xen-elf/lib -m elf_i386 -T arch/x86/minios-x86_32.lds
> /root/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o -o /root/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os
> 
> xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `bzopen_or_bzdopen':
> xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1411: undefined reference to `__ctype_b_loc'
> xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1447: undefined reference to `stdin'
> xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1447: undefined reference to `stdout'
> ...
> 
> In the earlier step mini-os.o was definitely linked against -lc .. 
> 
> "strings /path/mini-os.o | grep -i stdin" doesn't give anything though.. 
> I wonder why the stuff doesn't get included there.
> 

Re: [PATCH]: Implement bzip2 and LZMA loaders / xen-unstable stubdom pvgrub support

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Thu, 8 Oct 2009, Pasi Kärkkäinen wrote:
> 
> Hello again,
> 
> Stefano: Have you ever seen this kind of linking failures with
> stubdoms/ioemu? 
> 
> Any ideas?
> 
> -- Pasi
> 
> On Tue, Sep 01, 2009 at 10:08:55PM +0300, Pasi Kärkkäinen wrote:
> > 
> > Attached are my current work-in-progress patches. It still doesn't link properly.. 
> > 
> > Stefano: Would you like to take a look? The patches are against current xen-unstable.
> > 
> > I'm getting weird linking error about stdin/stdout/stderr missing.. 
> > libc (newlib) is definitely linked in, so I'm not sure what's happening.. 
> > 
> > ld -nostdlib -L/root/xen-unstable.hg/stubdom/cross-root-i686/i686-xen-elf/lib -m elf_i386 -T arch/x86/minios-x86_32.lds
> > /root/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o -o /root/xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os
> > 
> > xen-unstable.hg/stubdom/mini-os-x86_32-ioemu/mini-os.o: In function `bzopen_or_bzdopen':
> > xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1411: undefined reference to `__ctype_b_loc'

__ctype_b_loc is missing from newlib so you need to implement it
yourself.
See for example "minios: implement ffs, ffsl and ffsll.".


> > xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1447: undefined reference to `stdin'
> > xen-unstable.hg/stubdom/bzip2-x86_32/bzlib.c:1447: undefined reference to `stdout'
> > ...

bzip2 doesn't use autotools and the bzip2 Makefile seems to ignore any
environmental variable for CFLAG so you are not compiling it against
newlib at all.


[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel