Xen 4.2.2 / fixed vanilla kernel 3.7.10 / GPLPV r956 passes my torture test

Xen 4.2.2 / fixed vanilla kernel 3.7.10 / GPLPV r956 passes my torture test

[Andreas Kinzler]

[-- Attachment #1.1: Type: text/plain, Size: 1959 bytes --]

Hello,

this is to report that Xen 4.2.2 (from Gentoo), a fixed 3.7.10 vanilla kernel and GPLPV r956 has passed my quite intensive torture test. I stopped it after 49 hours.

Kernel is vanilla 3.7.10 with the following netback patches applied.
   core fixes:
     2810e5b9 xen-netback: coalesce slots in TX path and fix regressions
     03393fd5 xen-netback: don't disconnect frontend when seeing oversize packet  

   preconditions:
     27f85228 xen-netback: remove skb in xen_netbk_alloc_page
     35876b5f xen-netback: correctly return errors from netbk_count_requests()
     9eaee8be xen-netback: fix sparse warning

Test itself is as follows:

Have 2 HVMs "win1" and "win2". Each HVM has:
   * 2 VCPUs
   * Windows 7 SP1 x64 fully patched, GPLPV r956
   * iometer 2006.07.27
     * 2 workers with QD 4 each, pattern "all in one". Test runs on (virtual) physical disk
   * prime95 x64 27.7 build2, In-place large FFTs, Windows affinity set to VCPU1 on "win1"
     and to VCPU0 on "win2"
   * apache 2.4.4 for wget test as sender
   * ActiveState Perl x64, wget 1.14 from
     https://code.google.com/p/osspack32/downloads/detail?name=wget-1.14.exe
   * test-via-wget.pl as receiver

The mentioned wget test uses wget to recursively download 3376 files in 2101 directories via http (apache server in HVMs, nginx on Linux test driver box). The attached script also hashes the file content of downloaded files to make sure that every bit is transferred correctly. In the whole test setup the script is running in 4 instances (once on every HVM and twice on the test driver box). So I make sure that each HVM is tortured in both send/receive directions. All 3376/2101 objects are on RAM disks in both the HVMs and on the test driver box. I am using OSFMount from PassMark as RAM disk on Windows.

Please feel free to use the test-via-wget script for your own testing.
@James Harper: you might find the script helpful for your GPLPV tests

Regards Andreas




[-- Attachment #1.2: Type: text/html, Size: 2570 bytes --]

[-- Attachment #2: test-via-wget.pl --]
[-- Type: text/plain, Size: 3417 bytes --]

#!/usr/bin/perl

use IPC::Run3;
use Digest::SHA1;
use File::Spec;

# config
# $mode: 0 = normal cleardir/wget/hash using $tmpBase
#        1 = calc hash of $htdocsBase
$mode       = 0;
$url        = 'http://10.0.0.253:81';
$tmpBase    = '/mnt/fs2';
$htdocsBase = '/var/www/port81/htdocs';
$cmpHash    = '04e28506af6a4d00f308647dac7c5842bfa79f80';
$sleepSec   = 5;

# global vars
$tmpPath = File::Spec->catfile($tmpBase, "wget");
%hashes = ();

sub ensureEmptyDir
{
    my ($dir) = @_;
    
    die if ($dir eq "" or $dir eq "/");
    if (-d $dir)
    {
        recursiveEmpty($dir, 0);
    }
    else
    {
        mkdir $dir;
    }
}

sub recursiveEmpty
{
        my ($dir, $level) = @_;
	local *DIR;

	opendir DIR, $dir or die "opendir $dir: $!";
	my $found = 0;
	while ($_ = readdir DIR) {
	        next if /^\.{1,2}$/;
	        my $path = File::Spec->catfile($dir, $_);
		unlink $path if -f $path;
		recursiveEmpty($path, $level + 1) if -d $path;
	}
	closedir DIR;
	if ($level >= 1)
	{
		rmdir $dir or print "error - $!";
	}
}

sub recursiveHash
{
    my ($dir, $level, $relDir) = @_;
    local *DIR;
    local *FILE;
    
    opendir DIR, $dir or die "opendir $dir: $!";
    my $found = 0;
    while ($_ = readdir DIR) {
        next if /^\.{1,2}$/;
        next if m/^index\.html?$/im;
        my $path = File::Spec->catfile($dir, $_);
        my $newRelDir = "";
        $newRelDir = $relDir . "/" . $_ if $level >= 1;
        if (-f $path && $level >= 1)
        {
            open(FILE, "<", $path) or die "cannot hash file";
            binmode FILE;
            my $sha1 = Digest::SHA1->new;
            $sha1->addfile(FILE);
            close FILE;
            #print "$newRelDir\n" if ($mode);
            $hashes{$newRelDir} = $sha1->hexdigest;
        }
	recursiveHash($path, $level + 1, $newRelDir) if -d $path;
    }
    closedir DIR;
}

sub runWget()
{
    my ($stdout, $stderr);
    my @args = ("wget", "-nv", "-P", $tmpPath, "-r", $url);
    IPC::Run3::run3(\@args, undef, \$stdout, \$stderr);
    $r = $?;
    open FILE, ">last-stdout" or die;
    print FILE $stdout;
    close FILE;
    open FILE, ">last-stderr" or die;
    print FILE $stderr;
    close FILE;
    return $r;
}

sub runHash()
{
    my ($dir, $level);
    
    if ($mode)
    {
        $dir = $htdocsBase;
        $level = 1;
    }
    else
    {
        $dir = $tmpPath;
        $level = 0;
    }
    %hashes = ();
    recursiveHash($dir, $level, "");
    my $sha1 = Digest::SHA1->new;
    foreach my $path (sort keys %hashes)
    {
        my $d = $hashes{$path};
        $sha1->add($d);
    }
    return $sha1->hexdigest;
}

sub printFlush
{
    my ($txt) = @_;
    print $txt;
    STDOUT->flush;
}

$iter = 1;

if ($mode)
{
    my $r;
    
    $r = runHash();
    print "calculated hash: $r\n";
    exit 0;
}

while(1)
{
    my ($r);

    printFlush("iteration #$iter: cleardir... ");
    ensureEmptyDir($tmpPath);
    printFlush("wget...");
    $r = runWget;
    if ($r)
    {
        print "\nwget returned an error!\n";
        exit 1;
    }
    if (length($cmpHash) >= 1)
    {
        printFlush(" hashing...");
        $r = runHash;
        if (! ((lc $r) eq (lc $cmpHash)))
        {
            print ": SHA1 hash verification failed!\n";
            print "  expected: $cmpHash\n";
            print "calculated: $r\n";
            exit 1;
        }
    }
    print ": OK\n";
    $iter++;
    sleep($sleepSec);
}




[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.2.2 / fixed vanilla kernel 3.7.10 / GPLPV r956 passes my torture test

[James Harper]

> Hello,
> 
> this is to report that Xen 4.2.2 (from Gentoo), a fixed 3.7.10 vanilla kernel and
> GPLPV r956 has passed my quite intensive torture test. I stopped it after 49
> hours.
> 

Thanks for the feedback. If you feel like doing some more testing, the latest commit includes a very experimental tmem driver that does write-through caching on the pagefile. The only real testing I've done is applying 100+ updates on a brand new 2008R2 install on a DomU with an unrealistically small 384mb of memory assigned, but it did get a reasonable hit rate. I haven't had a chance to do much more testing than that though.

Let me know before you test as the very latest commit has had even less testing and was reporting a tmem error somewhere that I haven't had a chance to follow up.

James

Re: Xen 4.2.2 / fixed vanilla kernel 3.7.10 / GPLPV r956 passes my torture test

[Andreas Kinzler]

Hello James,

>  Thanks for the feedback. If you feel like doing some more testing, the latest
>  commit includes a very experimental tmem driver that does write-through caching
>  on the pagefile.

Sorry, I don't have time to test/validate experimental features. I think 
it would be great if you could introduce some "stable release concept" 
in GPLPV. I think most users (including me) will find it difficult to 
identify the latest stable release of GPLPV.

My tests always try to validate (as far as this is possible at all) a 
stable Xen setup.

Regards Andreas

Re: Xen 4.2.2 / fixed vanilla kernel 3.7.10 / GPLPV r956 passes my torture test

[Wei Liu]

On Fri, Jun 07, 2013 at 08:29:13PM +0200, Andreas Kinzler wrote:
> Hello,
> 
> this is to report that Xen 4.2.2 (from Gentoo), a fixed 3.7.10 vanilla
> kernel and GPLPV r956 has passed my quite intensive torture test. I
> stopped it after 49 hours.
> 
> Kernel is vanilla 3.7.10 with the following netback patches applied.
>   core fixes:
>     2810e5b9 xen-netback: coalesce slots in TX path and fix regressions
>     03393fd5 xen-netback: don't disconnect frontend when seeing
> oversize packet
> 

I would suggest you apply one more patch to keep the user visible kernel
parameter in sync with upstream.

  3764149 xen-netback: better names for thresholds

And, thanks for the testing. :-)


Wei.