[PATCH OSSTEST RFC 0/5] Improve README and enable Wheezy

[PATCH OSSTEST RFC 0/5] Improve README and enable Wheezy

[Wei Liu]

Hi Ian and Ian

With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run
build-* jobs and test-amd64-amd64-{pv,xl}.

Wei Liu (5):
  README: add curl and netcat to requirement list
  README: describe job dependence
  Debian: install libc6-dev-i386 for Wheezy amd64
  ts-debian-fixup: remove dead code
  ts-debian-fixup: workaround Debian bug #691320

 README            |    8 ++++++++
 ts-debian-fixup   |   32 ++++++++++++++------------------
 ts-xen-build-prep |    4 ++++
 3 files changed, 26 insertions(+), 18 deletions(-)

-- 
1.7.10.4

[PATCH OSSTEST RFC 1/5] README: add curl and netcat to requirement list

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 README |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/README b/README
index e9aed52..c845d4d 100644
--- a/README
+++ b/README
@@ -5,6 +5,8 @@ To run osstest in standalone mode:
      tcl8.4 tclx8.4 libsqlite3-tcl
      libdbi-perl libdbd-sqlite3-perl
      pax rsync
+     curl
+     netcat
 
  - Write a config file
     ~/.xen-osstest/config
-- 
1.7.10.4

[PATCH OSSTEST RFC 2/5] README: describe job dependence

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 README |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/README b/README
index c845d4d..7f90696 100644
--- a/README
+++ b/README
@@ -33,6 +33,12 @@ To run osstest in standalone mode:
                              (strictly, the final `_HOST' is the
 			      uppercase of the host's `ident' in the job)
 
+   However, as test-amd64-{i386,amd64}-xl and other tests depends on
+   some runtime variables generated by build-* jobs, you can run
+   build-* jobs before running actual test jobs. If you don't want to do
+   so you need to insert those missing runvars into standalone.db with
+   sqlite3.
+
  - Don't forget to set the machine to pxeboot in the BIOS.
 
  - Currently you need a serial console at 115200 8n1; see TODO.
-- 
1.7.10.4

[PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

[Wei Liu]

Otherwise build-amd64 fails with missing header predefs.h.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build-prep |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index b1e6a6c..66ba000 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -175,6 +175,10 @@ sub prep () {
     target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
     # workaround for Debian #595728
 
+    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
+	target_install_packages($ho, 'libc6-dev-i386');
+    }
+
     if ($ho->{Suite} !~ m/lenny/) {
         target_putfilecontents_stash($ho,60,<<'END','/home/osstest/.hgrc');
 [extensions]
-- 
1.7.10.4

[PATCH OSSTEST RFC 4/5] ts-debian-fixup: remove dead code

[Wei Liu]

Per c7910040 ("Osstest.pm, toolstack()->; always use "cfg", not separate
xl configs"), we only pick the config file with .cfg suffix.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-fixup |   18 ------------------
 1 file changed, 18 deletions(-)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index d8b51b7..f001418 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -163,23 +163,6 @@ sub writecfg () {
     target_putfile_root($ho,10, $cfgstash, $cfgfile);
 }
 
-sub xlcfg () {
-    my $cfg_xend= target_var($gho,'cfgpath');
-    my $cfg_xl= "/etc/xen/$gho->{Name}.xl";
-    store_runvar("$gho->{Guest}_xlpath", $cfg_xl);
-    target_editfile_root($ho, $cfg_xend, undef, $cfg_xl, sub {
-        local ($/)= undef;
-        $_= <EI>;
-        s/^ [ \t]* memory [ \t]* \= [ \t]* ['"] (.+) ['"] [ \t]*
-          $/memory=$1/mx;
-        s/([[,])\s*\n/$1/g;
-        s/,\s*\]/\]/g;
-        s/([^#\n].*)\n/$1;\n/mg;
-        s/\n\n+/\n/g;
-        print EO or die $!;
-    });
-}
-
 savecfg();
 ether();
 target_kernkind_check($gho);
@@ -188,5 +171,4 @@ console();
 filesystems();
 otherfixupcfg();
 writecfg();
-xlcfg();
 unmount();
-- 
1.7.10.4

[PATCH OSSTEST RFC 5/5] ts-debian-fixup: workaround Debian bug #691320

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-debian-fixup |   14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/ts-debian-fixup b/ts-debian-fixup
index f001418..f62bb37 100755
--- a/ts-debian-fixup
+++ b/ts-debian-fixup
@@ -118,6 +118,20 @@ sub otherfixupcfg () {
     $cfg =~ s/^vcpus.*//mg;
     $cfg .= "\nvcpus = $vcpus\n";
 
+    # Debian bug #691320
+    # Wheezy's xen-create-image writes incorrect memory= option in
+    # config file. This is fixed in newer version of xen-tools but it
+    # is not likely to go in in short term.
+    # On the other hand we should consider making xl support '512M',
+    # '2G' etc. But xend will still need this workaround until Wheezy
+    # is fixed.
+    # The default value 512 comes from ts-debian-install.
+    if ($ho->{Suite} =~ m/wheezy/) {
+	my $memory= guest_var($gho,'memory',512);
+	$cfg =~ s/^memory.*//mg;
+	$cfg .= "\nmemory = $memory\n";
+    }
+
     # PCI passthrough
     # Look for runvars   <gn>_pcipassthrough_<devtype>=<hostident>
     # and pass through all matching devices from the specified host.
-- 
1.7.10.4

Re: [PATCH OSSTEST RFC 5/5] ts-debian-fixup: workaround Debian bug #691320

[Ian Campbell]

On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  ts-debian-fixup |   14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/ts-debian-fixup b/ts-debian-fixup
> index f001418..f62bb37 100755
> --- a/ts-debian-fixup
> +++ b/ts-debian-fixup
> @@ -118,6 +118,20 @@ sub otherfixupcfg () {
>      $cfg =~ s/^vcpus.*//mg;
>      $cfg .= "\nvcpus = $vcpus\n";
>  
> +    # Debian bug #691320
> +    # Wheezy's xen-create-image writes incorrect memory= option in
> +    # config file. This is fixed in newer version of xen-tools but it
> +    # is not likely to go in in short term.
> +    # On the other hand we should consider making xl support '512M',
> +    # '2G' etc. But xend will still need this workaround until Wheezy
> +    # is fixed.
> +    # The default value 512 comes from ts-debian-install.

Wouldn't it be better to fix the invocation of xne-create-image to use a
Mb suffix instead of M, AFAICT that would work on both Squeeze and
Wheezy.

I think the fact that Squeeze's accepted just M was unintentional,
although it's good that they are willing to fix it to be acceptable
again...

> +    if ($ho->{Suite} =~ m/wheezy/) {
> +	my $memory= guest_var($gho,'memory',512);
> +	$cfg =~ s/^memory.*//mg;
> +	$cfg .= "\nmemory = $memory\n";
> +    }
> +
>      # PCI passthrough
>      # Look for runvars   <gn>_pcipassthrough_<devtype>=<hostident>
>      # and pass through all matching devices from the specified host.

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

[Ian Campbell]

On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> Otherwise build-amd64 fails with missing header predefs.h.

This package exists in Squeeze too, it might simplify things to make it
unconditional?

I wonder why it apparently isn't need for a Squeeze build though. What
32-bit stuff is getting built? 

> 
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  ts-xen-build-prep |    4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/ts-xen-build-prep b/ts-xen-build-prep
> index b1e6a6c..66ba000 100755
> --- a/ts-xen-build-prep
> +++ b/ts-xen-build-prep
> @@ -175,6 +175,10 @@ sub prep () {
>      target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
>      # workaround for Debian #595728
>  
> +    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
> +	target_install_packages($ho, 'libc6-dev-i386');
> +    }
> +
>      if ($ho->{Suite} !~ m/lenny/) {
>          target_putfilecontents_stash($ho,60,<<'END','/home/osstest/.hgrc');
>  [extensions]

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

[Wei Liu]

On Thu, Sep 19, 2013 at 06:16:15PM +0100, Ian Campbell wrote:
> On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > Otherwise build-amd64 fails with missing header predefs.h.
> 
> This package exists in Squeeze too, it might simplify things to make it
> unconditional?
> 

No, it doesn't exist in Lenny -- I see there's branches conditional on
Lenny, do we still use that?

> I wonder why it apparently isn't need for a Squeeze build though. What
> 32-bit stuff is getting built? 
> 

I guess that's because Wheezy introduces multiarch? The thing being
built is QEMU.

Wei.

> > 
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> >  ts-xen-build-prep |    4 ++++
> >  1 file changed, 4 insertions(+)
> > 
> > diff --git a/ts-xen-build-prep b/ts-xen-build-prep
> > index b1e6a6c..66ba000 100755
> > --- a/ts-xen-build-prep
> > +++ b/ts-xen-build-prep
> > @@ -175,6 +175,10 @@ sub prep () {
> >      target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates");
> >      # workaround for Debian #595728
> >  
> > +    if ($r{arch} eq 'amd64' && $ho->{Suite} =~ m/wheezy/) {
> > +	target_install_packages($ho, 'libc6-dev-i386');
> > +    }
> > +
> >      if ($ho->{Suite} !~ m/lenny/) {
> >          target_putfilecontents_stash($ho,60,<<'END','/home/osstest/.hgrc');
> >  [extensions]
> 

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

[Ian Campbell]

On Thu, 2013-09-19 at 19:31 +0100, Wei Liu wrote:
> On Thu, Sep 19, 2013 at 06:16:15PM +0100, Ian Campbell wrote:
> > On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > > Otherwise build-amd64 fails with missing header predefs.h.
> > 
> > This package exists in Squeeze too, it might simplify things to make it
> > unconditional?
> > 
> 
> No, it doesn't exist in Lenny -- I see there's branches conditional on
> Lenny, do we still use that?

I think it is all Squeeze these days.

> > I wonder why it apparently isn't need for a Squeeze build though. What
> > 32-bit stuff is getting built? 
> > 
> 
> I guess that's because Wheezy introduces multiarch? The thing being
> built is QEMU.

We're building 64-bit qemu and failing because of a missing 32-bit
header? That sounds like a bug somewhere, perhaps in the compiler/libc
packaging or maybe in some part of our build system. How does the actual
error look?

Ian.

Re: [PATCH OSSTEST RFC 0/5] Improve README and enable Wheezy

[Ian Campbell]

On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run
> build-* jobs and test-amd64-amd64-{pv,xl}.

So I guess you've just been through the process of setting up osstest in
standalone mode? I wonder if you could put that into a short blog or
wiki doc as an illustration for people?

Ian.

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

[Wei Liu]

On Thu, Sep 19, 2013 at 08:03:17PM +0100, Ian Campbell wrote:
> On Thu, 2013-09-19 at 19:31 +0100, Wei Liu wrote:
> > On Thu, Sep 19, 2013 at 06:16:15PM +0100, Ian Campbell wrote:
> > > On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > > > Otherwise build-amd64 fails with missing header predefs.h.
> > > 
> > > This package exists in Squeeze too, it might simplify things to make it
> > > unconditional?
> > > 
> > 
> > No, it doesn't exist in Lenny -- I see there's branches conditional on
> > Lenny, do we still use that?
> 
> I think it is all Squeeze these days.
> 

OK.

> > > I wonder why it apparently isn't need for a Squeeze build though. What
> > > 32-bit stuff is getting built? 
> > > 
> > 
> > I guess that's because Wheezy introduces multiarch? The thing being
> > built is QEMU.
> 
> We're building 64-bit qemu and failing because of a missing 32-bit
> header? That sounds like a bug somewhere, perhaps in the compiler/libc
> packaging or maybe in some part of our build system. How does the actual
> error look?
> 

Oh sorrrry, it's not QEMU, -j32 and interleaving output fooled me. :-(

XEN_TARGET_ARCH=x86_32 make -f blowfish.mk all
make[5]: Entering directory `/local/scratch/xen.git/tools/tests/x86_emulator'
gcc  -O1 -fno-omit-frame-pointer -m32 -march=i686 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable -Wno-unused-local-typedefs   -D__XEN_TOOLS__ -MMD -MF .blowfish.bin.d -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -fno-optimize-sibling-calls -mno-tls-direct-seg-refs  -fno-stack-protector -fno-exceptions -fno-builtin -msoft-float -c blowfish.c
In file included from /usr/include/stdint.h:26:0,
                 from /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stdint.h:3,
                 from blowfish.c:19:
/usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or directory
compilation terminated.

Wei.

> Ian.

Re: [PATCH OSSTEST RFC 0/5] Improve README and enable Wheezy

[Wei Liu]

On Thu, Sep 19, 2013 at 08:05:38PM +0100, Ian Campbell wrote:
> On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run
> > build-* jobs and test-amd64-amd64-{pv,xl}.
> 
> So I guess you've just been through the process of setting up osstest in
> standalone mode? I wonder if you could put that into a short blog or
> wiki doc as an illustration for people?
> 

I'm actually writing a blog post on this topic, but still far from
complete.

Wei.

> Ian.

Re: [PATCH OSSTEST RFC 3/5] Debian: install libc6-dev-i386 for Wheezy amd64

[Ian Campbell]

On Thu, 2013-09-19 at 20:12 +0100, Wei Liu wrote:
> On Thu, Sep 19, 2013 at 08:03:17PM +0100, Ian Campbell wrote:
> > On Thu, 2013-09-19 at 19:31 +0100, Wei Liu wrote:
> > > On Thu, Sep 19, 2013 at 06:16:15PM +0100, Ian Campbell wrote:
> > > > On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > > > > Otherwise build-amd64 fails with missing header predefs.h.
> > > > 
> > > > This package exists in Squeeze too, it might simplify things to make it
> > > > unconditional?
> > > > 
> > > 
> > > No, it doesn't exist in Lenny -- I see there's branches conditional on
> > > Lenny, do we still use that?
> > 
> > I think it is all Squeeze these days.
> > 
> 
> OK.
> 
> > > > I wonder why it apparently isn't need for a Squeeze build though. What
> > > > 32-bit stuff is getting built? 
> > > > 
> > > 
> > > I guess that's because Wheezy introduces multiarch? The thing being
> > > built is QEMU.
> > 
> > We're building 64-bit qemu and failing because of a missing 32-bit
> > header? That sounds like a bug somewhere, perhaps in the compiler/libc
> > packaging or maybe in some part of our build system. How does the actual
> > error look?
> > 
> 
> Oh sorrrry, it's not QEMU, -j32 and interleaving output fooled me. :-(
> 
> XEN_TARGET_ARCH=x86_32 make -f blowfish.mk all
> make[5]: Entering directory `/local/scratch/xen.git/tools/tests/x86_emulator'
> gcc  -O1 -fno-omit-frame-pointer -m32 -march=i686 -g -fno-strict-aliasing -std=gnu99 -Wall -Wstrict-prototypes -Wdeclaration-after-statement -Wno-unused-but-set-variable -Wno-unused-local-typedefs   -D__XEN_TOOLS__ -MMD -MF .blowfish.bin.d -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -fno-optimize-sibling-calls -mno-tls-direct-seg-refs  -fno-stack-protector -fno-exceptions -fno-builtin -msoft-float -c blowfish.c
> In file included from /usr/include/stdint.h:26:0,
>                  from /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stdint.h:3,
>                  from blowfish.c:19:
> /usr/include/features.h:323:26: fatal error: bits/predefs.h: No such file or directory
> compilation terminated.

Ah, OK, this is test code for the 32-bit instruction decoder/emulator.
In principal this probably shouldn't be built if the compiler is not
available, but in the meantime your fix looks sensible.

Ian

Re: [PATCH OSSTEST RFC 0/5] Improve README and enable Wheezy

[Ian Campbell]

On Thu, 2013-09-19 at 20:13 +0100, Wei Liu wrote:
> On Thu, Sep 19, 2013 at 08:05:38PM +0100, Ian Campbell wrote:
> > On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > > With the changes to Debian.pm and ts-debian-fixup I can now use Wheezy to run
> > > build-* jobs and test-amd64-amd64-{pv,xl}.
> > 
> > So I guess you've just been through the process of setting up osstest in
> > standalone mode? I wonder if you could put that into a short blog or
> > wiki doc as an illustration for people?
> > 
> 
> I'm actually writing a blog post on this topic, but still far from
> complete.

Perfect, thanks! ;-)

Re: [PATCH OSSTEST RFC 5/5] ts-debian-fixup: workaround Debian bug #691320

[Wei Liu]

On Thu, Sep 19, 2013 at 06:12:26PM +0100, Ian Campbell wrote:
> On Thu, 2013-09-19 at 17:02 +0100, Wei Liu wrote:
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> >  ts-debian-fixup |   14 ++++++++++++++
> >  1 file changed, 14 insertions(+)
> > 
> > diff --git a/ts-debian-fixup b/ts-debian-fixup
> > index f001418..f62bb37 100755
> > --- a/ts-debian-fixup
> > +++ b/ts-debian-fixup
> > @@ -118,6 +118,20 @@ sub otherfixupcfg () {
> >      $cfg =~ s/^vcpus.*//mg;
> >      $cfg .= "\nvcpus = $vcpus\n";
> >  
> > +    # Debian bug #691320
> > +    # Wheezy's xen-create-image writes incorrect memory= option in
> > +    # config file. This is fixed in newer version of xen-tools but it
> > +    # is not likely to go in in short term.
> > +    # On the other hand we should consider making xl support '512M',
> > +    # '2G' etc. But xend will still need this workaround until Wheezy
> > +    # is fixed.
> > +    # The default value 512 comes from ts-debian-install.
> 
> Wouldn't it be better to fix the invocation of xne-create-image to use a
> Mb suffix instead of M, AFAICT that would work on both Squeeze and
> Wheezy.
> 

OK. I tested passing 'Mb', it did work in Wheezy.

I will drop this one.

Wei.

> I think the fact that Squeeze's accepted just M was unintentional,
> although it's good that they are willing to fix it to be acceptable
> again...
> 
> > +    if ($ho->{Suite} =~ m/wheezy/) {
> > +	my $memory= guest_var($gho,'memory',512);
> > +	$cfg =~ s/^memory.*//mg;
> > +	$cfg .= "\nmemory = $memory\n";
> > +    }
> > +
> >      # PCI passthrough
> >      # Look for runvars   <gn>_pcipassthrough_<devtype>=<hostident>
> >      # and pass through all matching devices from the specified host.
>