Configuration parameters get "sometimes" omitted

Configuration parameters get "sometimes" omitted

[Christian Ehrhardt]

Hi,
we have the problem that our xen configuration settings are not correctly evaluated when creating a xen guests.

When creating domains with "xm create domainname" the system sometimes has all configuration parameters out of the config files, but sometimes not.
We did some tests and saw that the parameters are only certainly used if we specify them via the commandline like "xm create domainname parameter=value,...".

If the parameters are only specified in the configuration files in /etc/xen/ they are parsed and used "sometimes".
Another fact we found is that if the "xm create" is called by a script the error occurs very often ~70% of the testcases, while calling the command from a ssh shell manually only fails in ~10% of the testcases. We stripped down the script and it does nothing but calling "xm create" and the normal shell where it works mostly has no .profile/.bashrc/... that change the environment.
Additionally there is some persistent behavior - if it failed once it will fail until the host is rebooted (maybe some config parts are stored and not re-evaluated). I checked xenstore with the script from http://wiki.xensource.com/xenwiki/XenStore and saw that xenstore only has the parameters if a good-case guest is running.

An interesting fact is that it does not fail initializing memory or pci passthrough in host or guest - somehow it fails the initial config parsing. You can see below that already the output of XendDomainInfo.create in the log is different in good/bad case.

Because of the unstable behavior I expect some kind of race condition while parsing from the files.



### See below for more verbose details on what we found in the log's ,system setup and configuration files ###



The command "xm create lnudb1 pci=04:01.0 memory=2048" works always successful.
A "xm create lnudb1" creates only *sometimes* a system with a pci adapter and 2 GB memory.
Looking in the /var/log/xend.log reveals a different XendDomainInfo.create statement 
for good and bad case.

(I added the >>> <<< signs to point to the important statement)

XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 2048], ['vcpus', 1], 
['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', ['linux', 
['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk', '/etc/xen/initrd-xen'], ['root', '/dev/xvda1'], 
['args', 'TERM=xterm selinux=off xencons=tty']]], ['device', ['vbd', ['uname', 'phy:sde5'], 
['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/hda'], ['dev', 'hdc:cdrom'], 
['mode', 'r']]], 
>>> ['device', ['pci', ['dev', ['domain', '0x0'], ['bus', '0x04'], ['slot', '0x01'], ['func', '0x0']]]], <<<
['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]], ['device', ['vif', ['bridge', 'xenbr1'],
['mac', '00:16:3e:00:00:31']]], ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type', 'vnc'],
['display', 'localhost:10.0'], ['xauthority', '/root/.Xauthority']]]])


The xm create with the profile described above produces following statement 
XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 1024], ['vcpus', 1], 
['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', ['linux', 
['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk', '/etc/xen/initrd-xen'], ['root', '/dev/xvda1'], 
['args', 'TERM=xterm selinux=off xencons=tty']]], ['device', ['vbd', ['uname', 'phy:sde5'], 
['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev/hda'], ['dev', 'hdc:cdrom'], 
['mode', 'r']]], 
>>>  <<<<
['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]], ['device', ['vif', ['bridge', 'xenbr1'],
['mac', '00:16:3e:00:00:31']]], ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type', 'vnc'],
['display', 'localhost:10.0'], ['xauthority', '/root/.Xauthority']]]])

Also note the different values for the memory statement

System Description
Hardware IBM x3950, 4 Intel Xeon CPUs (dual core) 3.5GHz, 23Gb memory.
The operating system is Linux SLES10 SP1, with the following rpms:
o xen-kmp-smp-3.1.0_2.6.16.46_0.12-0.1
o xen-libs-32bit-3.0.4_13138-0.40
o xen-doc-pdf-3.1.0-0.1
o kernel-xen-2.6.16.46-0.12
o xen-3.1.0-0.1
o xen-tools-3.1.0-0.1
o xen-doc-html-3.1.0-0.1
o xen-libs-3.1.0-0.1
o xen-tools-ioemu-3.1.0-0.1

the kernel is booted with the parameters 
pciback.hide=(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1) 
and /etc/modprobe.conf constains the statements 
install lpfc /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install lpfc
options  pciback hide=(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1)
to hide the pci adapters for the Dom0.

The xen profile used is:
builder='linux'
memory = 2048
name = "lnudb1"
vcpus = 1
vif = [ 'mac=00:16:3e:00:00:10, bridge=xenbr0', 'mac=00:16:3e:00:00:31, bridge=xenbr1' ]
disk = [ 'phy:sde5,xvda1,w', 'phy:/dev/hda,hdc:cdrom,r' ]
root = "/dev/xvda1"
extra = "TERM=xterm selinux=off xencons=tty"
vfb=[ "type=vnc,vncunused=1" ]
pci= [ '0000:04:01.0' ]

The critical parameters are memory and pci, we never saw the issue with e.g. vif's.

P.S. It was hard to decide between xen-devel and xen-users by topic for that mail.
Eventually I coose xen-devel becasue I'm already member of this list and thereby get also responses that may lack a cc to me. Please redirect if needed.

-- 

Grüsse / regards, 
Christian Ehrhardt

IBM Linux Technology Center, Open Virtualization
+49 7031/16-3385
Ehrhardt@linux.vnet.ibm.com
Ehrhardt@de.ibm.com

IBM Deutschland Entwicklung GmbH
Vorsitzender des Aufsichtsrats: Johann Weihen 
Geschäftsführung: Herbert Kircher 
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: Configuration parameters get "sometimes" omitted

[Mark Williamson]

> we have the problem that our xen configuration settings are not correctly
> evaluated when creating a xen guests.

Congratulations - I think you just won my "weird problem of the week" 
award ;-)

> When creating domains with "xm create domainname" the system sometimes has
> all configuration parameters out of the config files, but sometimes not. We
> did some tests and saw that the parameters are only certainly used if we
> specify them via the commandline like "xm create domainname
> parameter=value,...".

Now that's ... really strange.

What version of Xen is this?  Did you build it from source, or was it 
packaged?  What distro is this?

Are you using any Xend managed domains or are you doing everything by config 
files?

> If the parameters are only specified in the configuration files in
> /etc/xen/ they are parsed and used "sometimes". Another fact we found is
> that if the "xm create" is called by a script the error occurs very often
> ~70% of the testcases, while calling the command from a ssh shell manually
> only fails in ~10% of the testcases. We stripped down the script and it
> does nothing but calling "xm create" and the normal shell where it works
> mostly has no .profile/.bashrc/... that change the environment.

Good troubleshooting.  And good analysis...

Are you 100% sure that you don't have another copy of your domain config file 
somewhere on the system (e.g. maybe a backup) that might have different 
values in?  Is it possible you're running the commands from different 
directories in each case?

xm will use a config file in ./ if it's there, or it can search /etc/xen.  I'm 
not sure what the order of precedence is but I'm thinking it might be 
sometimes using one copy and sometimes the other?

Another way to test: try specifying the full path, e.g "xm 
create /etc/xen/lnudb1".  Does this behave consistently?

> Additionally there is some persistent behavior - if it failed once it will
> fail until the host is rebooted (maybe some config parts are stored and not
> re-evaluated). I checked xenstore with the script from
> http://wiki.xensource.com/xenwiki/XenStore and saw that xenstore only has
> the parameters if a good-case guest is running.

That's strange too...

> An interesting fact is that it does not fail initializing memory or pci
> passthrough in host or guest - somehow it fails the initial config parsing.
> You can see below that already the output of XendDomainInfo.create in the
> log is different in good/bad case.
>
> Because of the unstable behavior I expect some kind of race condition while
> parsing from the files.

This seems very odd.  Although it sounds buggy, it seems to me that others 
would have noticed it if it was common.  Therefore, it suggests that there's 
something perculiar to your local system which is provoking this behaviour.

Let us know what you find out, and maybe we can crank through this!

Cheers,
mark

> The command "xm create lnudb1 pci=04:01.0 memory=2048" works always
> successful. A "xm create lnudb1" creates only *sometimes* a system with a
> pci adapter and 2 GB memory. Looking in the /var/log/xend.log reveals a
> different XendDomainInfo.create statement for good and bad case.
>
> (I added the >>> <<< signs to point to the important statement)
>
> XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 2048],
> ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
> ['image', ['linux', ['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk',
> '/etc/xen/initrd-xen'], ['root', '/dev/xvda1'], ['args', 'TERM=xterm
> selinux=off xencons=tty']]], ['device', ['vbd', ['uname', 'phy:sde5'],
> ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> 'phy:/dev/hda'], ['dev', 'hdc:cdrom'], ['mode', 'r']]],
>
> >>> ['device', ['pci', ['dev', ['domain', '0x0'], ['bus', '0x04'], ['slot',
> >>> '0x01'], ['func', '0x0']]]], <<<
>
> ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]],
> ['device', ['vif', ['bridge', 'xenbr1'], ['mac', '00:16:3e:00:00:31']]],
> ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type',
> 'vnc'], ['display', 'localhost:10.0'], ['xauthority',
> '/root/.Xauthority']]]])
>
>
> The xm create with the profile described above produces following statement
> XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 1024],
> ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
> ['image', ['linux', ['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk',
> '/etc/xen/initrd-xen'], ['root', '/dev/xvda1'], ['args', 'TERM=xterm
> selinux=off xencons=tty']]], ['device', ['vbd', ['uname', 'phy:sde5'],
> ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> 'phy:/dev/hda'], ['dev', 'hdc:cdrom'], ['mode', 'r']]],
>
> >>>  <<<<
>
> ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]],
> ['device', ['vif', ['bridge', 'xenbr1'], ['mac', '00:16:3e:00:00:31']]],
> ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type',
> 'vnc'], ['display', 'localhost:10.0'], ['xauthority',
> '/root/.Xauthority']]]])
>
> Also note the different values for the memory statement
>
> System Description
> Hardware IBM x3950, 4 Intel Xeon CPUs (dual core) 3.5GHz, 23Gb memory.
> The operating system is Linux SLES10 SP1, with the following rpms:
> o xen-kmp-smp-3.1.0_2.6.16.46_0.12-0.1
> o xen-libs-32bit-3.0.4_13138-0.40
> o xen-doc-pdf-3.1.0-0.1
> o kernel-xen-2.6.16.46-0.12
> o xen-3.1.0-0.1
> o xen-tools-3.1.0-0.1
> o xen-doc-html-3.1.0-0.1
> o xen-libs-3.1.0-0.1
> o xen-tools-ioemu-3.1.0-0.1
>
> the kernel is booted with the parameters
> pciback.hide=(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1)
> and /etc/modprobe.conf constains the statements
> install lpfc /sbin/modprobe pciback ; /sbin/modprobe --first-time
> --ignore-install lpfc options  pciback
> hide=(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1) to hide the pci
> adapters for the Dom0.
>
> The xen profile used is:
> builder='linux'
> memory = 2048
> name = "lnudb1"
> vcpus = 1
> vif = [ 'mac=00:16:3e:00:00:10, bridge=xenbr0', 'mac=00:16:3e:00:00:31,
> bridge=xenbr1' ] disk = [ 'phy:sde5,xvda1,w', 'phy:/dev/hda,hdc:cdrom,r' ]
> root = "/dev/xvda1"
> extra = "TERM=xterm selinux=off xencons=tty"
> vfb=[ "type=vnc,vncunused=1" ]
> pci= [ '0000:04:01.0' ]
>
> The critical parameters are memory and pci, we never saw the issue with
> e.g. vif's.
>
> P.S. It was hard to decide between xen-devel and xen-users by topic for
> that mail. Eventually I coose xen-devel becasue I'm already member of this
> list and thereby get also responses that may lack a cc to me. Please
> redirect if needed.



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Configuration parameters get "sometimes" omitted

[Juergen Doelle]

Mark, I am sorry, but with your help I have to give back the "weird problem
of the week". The
problem cause is very simple. We were not aware that xm will use the config
file from the local
directory. We had one directory with all scripts for cloning, start,
reboot, and shutdown. And it
contained the templates for the profiles. After moving them to a different
location, the real profile
in /etc/xen/vm were used and it works well.

Thanks for your help,
Juergen

Juergen Doelle
Linux End to End Performance Project Lead
Dept. 3235, Linux on System z System & Performance Evaluation, IBM Lab
Boeblingen
http://www.ibm.com/developerworks/linux/linux390/perf

IBM Deutschland Entwicklung GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Herbert Kircher
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294







                                                                       
             Mark Williamson                                           
             <mark.williamson@                                         
             cl.cam.ac.uk>                                              To
             Sent by: "M.A.            xen-devel@lists.xensource.com   
             Williamson"                                                cc
             <maw48@hermes.cam         Christian Ehrhardt              
             .ac.uk>                   <ehrhardt@linux.vnet.ibm.com>,  
                                       Juergen Doelle/Germany/IBM@IBMDE
                                                                   Subject
             10/18/2007 04:23          Re: [Xen-devel] Configuration   
             AM                        parameters get "sometimes" omitted
                                                                       
                                                                       
                                                                       
                                                                       
                                                                       
                                                                       




> we have the problem that our xen configuration settings are not correctly
> evaluated when creating a xen guests.

Congratulations - I think you just won my "weird problem of the week"
award ;-)

> When creating domains with "xm create domainname" the system sometimes
has
> all configuration parameters out of the config files, but sometimes not.
We
> did some tests and saw that the parameters are only certainly used if we
> specify them via the commandline like "xm create domainname
> parameter=value,...".

Now that's ... really strange.

What version of Xen is this?  Did you build it from source, or was it
packaged?  What distro is this?

Are you using any Xend managed domains or are you doing everything by
config
files?

> If the parameters are only specified in the configuration files in
> /etc/xen/ they are parsed and used "sometimes". Another fact we found is
> that if the "xm create" is called by a script the error occurs very often
> ~70% of the testcases, while calling the command from a ssh shell
manually
> only fails in ~10% of the testcases. We stripped down the script and it
> does nothing but calling "xm create" and the normal shell where it works
> mostly has no .profile/.bashrc/... that change the environment.

Good troubleshooting.  And good analysis...

Are you 100% sure that you don't have another copy of your domain config
file
somewhere on the system (e.g. maybe a backup) that might have different
values in?  Is it possible you're running the commands from different
directories in each case?

xm will use a config file in ./ if it's there, or it can search /etc/xen.
I'm
not sure what the order of precedence is but I'm thinking it might be
sometimes using one copy and sometimes the other?

Another way to test: try specifying the full path, e.g "xm
create /etc/xen/lnudb1".  Does this behave consistently?

> Additionally there is some persistent behavior - if it failed once it
will
> fail until the host is rebooted (maybe some config parts are stored and
not
> re-evaluated). I checked xenstore with the script from
> http://wiki.xensource.com/xenwiki/XenStore and saw that xenstore only has
> the parameters if a good-case guest is running.

That's strange too...

> An interesting fact is that it does not fail initializing memory or pci
> passthrough in host or guest - somehow it fails the initial config
parsing.
> You can see below that already the output of XendDomainInfo.create in the
> log is different in good/bad case.
>
> Because of the unstable behavior I expect some kind of race condition
while
> parsing from the files.

This seems very odd.  Although it sounds buggy, it seems to me that others
would have noticed it if it was common.  Therefore, it suggests that
there's
something perculiar to your local system which is provoking this behaviour.

Let us know what you find out, and maybe we can crank through this!

Cheers,
mark

> The command "xm create lnudb1 pci=04:01.0 memory=2048" works always
> successful. A "xm create lnudb1" creates only *sometimes* a system with a
> pci adapter and 2 GB memory. Looking in the /var/log/xend.log reveals a
> different XendDomainInfo.create statement for good and bad case.
>
> (I added the >>> <<< signs to point to the important statement)
>
> XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 2048],
> ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
> ['image', ['linux', ['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk',
> '/etc/xen/initrd-xen'], ['root', '/dev/xvda1'], ['args', 'TERM=xterm
> selinux=off xencons=tty']]], ['device', ['vbd', ['uname', 'phy:sde5'],
> ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> 'phy:/dev/hda'], ['dev', 'hdc:cdrom'], ['mode', 'r']]],
>
> >>> ['device', ['pci', ['dev', ['domain', '0x0'], ['bus', '0x04'],
['slot',
> >>> '0x01'], ['func', '0x0']]]], <<<
>
> ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]],
> ['device', ['vif', ['bridge', 'xenbr1'], ['mac', '00:16:3e:00:00:31']]],
> ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type',
> 'vnc'], ['display', 'localhost:10.0'], ['xauthority',
> '/root/.Xauthority']]]])
>
>
> The xm create with the profile described above produces following
statement
> XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 1024],
> ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
> ['image', ['linux', ['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk',
> '/etc/xen/initrd-xen'], ['root', '/dev/xvda1'], ['args', 'TERM=xterm
> selinux=off xencons=tty']]], ['device', ['vbd', ['uname', 'phy:sde5'],
> ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> 'phy:/dev/hda'], ['dev', 'hdc:cdrom'], ['mode', 'r']]],
>
> >>>  <<<<
>
> ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]],
> ['device', ['vif', ['bridge', 'xenbr1'], ['mac', '00:16:3e:00:00:31']]],
> ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type',
> 'vnc'], ['display', 'localhost:10.0'], ['xauthority',
> '/root/.Xauthority']]]])
>
> Also note the different values for the memory statement
>
> System Description
> Hardware IBM x3950, 4 Intel Xeon CPUs (dual core) 3.5GHz, 23Gb memory.
> The operating system is Linux SLES10 SP1, with the following rpms:
> o xen-kmp-smp-3.1.0_2.6.16.46_0.12-0.1
> o xen-libs-32bit-3.0.4_13138-0.40
> o xen-doc-pdf-3.1.0-0.1
> o kernel-xen-2.6.16.46-0.12
> o xen-3.1.0-0.1
> o xen-tools-3.1.0-0.1
> o xen-doc-html-3.1.0-0.1
> o xen-libs-3.1.0-0.1
> o xen-tools-ioemu-3.1.0-0.1
>
> the kernel is booted with the parameters
> pciback.hide=(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1)
> and /etc/modprobe.conf constains the statements
> install lpfc /sbin/modprobe pciback ; /sbin/modprobe --first-time
> --ignore-install lpfc options  pciback
> hide=(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1) to hide the
pci
> adapters for the Dom0.
>
> The xen profile used is:
> builder='linux'
> memory = 2048
> name = "lnudb1"
> vcpus = 1
> vif = [ 'mac=00:16:3e:00:00:10, bridge=xenbr0', 'mac=00:16:3e:00:00:31,
> bridge=xenbr1' ] disk = [ 'phy:sde5,xvda1,w', 'phy:/dev/hda,hdc:cdrom,r'
]
> root = "/dev/xvda1"
> extra = "TERM=xterm selinux=off xencons=tty"
> vfb=[ "type=vnc,vncunused=1" ]
> pci= [ '0000:04:01.0' ]
>
> The critical parameters are memory and pci, we never saw the issue with
> e.g. vif's.
>
> P.S. It was hard to decide between xen-devel and xen-users by topic for
> that mail. Eventually I coose xen-devel becasue I'm already member of
this
> list and thereby get also responses that may lack a cc to me. Please
> redirect if needed.



--
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: Configuration parameters get "sometimes" omitted

[Mark Williamson]

Aha!  Glad that cleared things up.

Possibly we should consider making the behaviour of xm more explicit (e.g. 
requiring a ./ before  choosing a a config from the current directory).

Cheers,
Mark

On Friday 19 October 2007, Juergen Doelle wrote:
> Mark, I am sorry, but with your help I have to give back the "weird problem
> of the week". The
> problem cause is very simple. We were not aware that xm will use the config
> file from the local
> directory. We had one directory with all scripts for cloning, start,
> reboot, and shutdown. And it
> contained the templates for the profiles. After moving them to a different
> location, the real profile
> in /etc/xen/vm were used and it works well.
>
> Thanks for your help,
> Juergen
>
> Juergen Doelle
> Linux End to End Performance Project Lead
> Dept. 3235, Linux on System z System & Performance Evaluation, IBM Lab
> Boeblingen
> http://www.ibm.com/developerworks/linux/linux390/perf
>
> IBM Deutschland Entwicklung GmbH
> Vorsitzender des Aufsichtsrats: Martin Jetter
> Geschäftsführung: Herbert Kircher
> Sitz der Gesellschaft: Böblingen
> Registergericht: Amtsgericht Stuttgart, HRB 243294
>
>
>
>
>
>
>
>
>              Mark Williamson
>              <mark.williamson@
>              cl.cam.ac.uk>                                              To
>              Sent by: "M.A.            xen-devel@lists.xensource.com
>              Williamson"                                                cc
>              <maw48@hermes.cam         Christian Ehrhardt
>              .ac.uk>                   <ehrhardt@linux.vnet.ibm.com>,
>                                        Juergen Doelle/Germany/IBM@IBMDE
>                                                                    Subject
>              10/18/2007 04:23          Re: [Xen-devel] Configuration
>              AM                        parameters get "sometimes" omitted
>
> > we have the problem that our xen configuration settings are not correctly
> > evaluated when creating a xen guests.
>
> Congratulations - I think you just won my "weird problem of the week"
> award ;-)
>
> > When creating domains with "xm create domainname" the system sometimes
>
> has
>
> > all configuration parameters out of the config files, but sometimes not.
>
> We
>
> > did some tests and saw that the parameters are only certainly used if we
> > specify them via the commandline like "xm create domainname
> > parameter=value,...".
>
> Now that's ... really strange.
>
> What version of Xen is this?  Did you build it from source, or was it
> packaged?  What distro is this?
>
> Are you using any Xend managed domains or are you doing everything by
> config
> files?
>
> > If the parameters are only specified in the configuration files in
> > /etc/xen/ they are parsed and used "sometimes". Another fact we found is
> > that if the "xm create" is called by a script the error occurs very often
> > ~70% of the testcases, while calling the command from a ssh shell
>
> manually
>
> > only fails in ~10% of the testcases. We stripped down the script and it
> > does nothing but calling "xm create" and the normal shell where it works
> > mostly has no .profile/.bashrc/... that change the environment.
>
> Good troubleshooting.  And good analysis...
>
> Are you 100% sure that you don't have another copy of your domain config
> file
> somewhere on the system (e.g. maybe a backup) that might have different
> values in?  Is it possible you're running the commands from different
> directories in each case?
>
> xm will use a config file in ./ if it's there, or it can search /etc/xen.
> I'm
> not sure what the order of precedence is but I'm thinking it might be
> sometimes using one copy and sometimes the other?
>
> Another way to test: try specifying the full path, e.g "xm
> create /etc/xen/lnudb1".  Does this behave consistently?
>
> > Additionally there is some persistent behavior - if it failed once it
>
> will
>
> > fail until the host is rebooted (maybe some config parts are stored and
>
> not
>
> > re-evaluated). I checked xenstore with the script from
> > http://wiki.xensource.com/xenwiki/XenStore and saw that xenstore only has
> > the parameters if a good-case guest is running.
>
> That's strange too...
>
> > An interesting fact is that it does not fail initializing memory or pci
> > passthrough in host or guest - somehow it fails the initial config
>
> parsing.
>
> > You can see below that already the output of XendDomainInfo.create in the
> > log is different in good/bad case.
> >
> > Because of the unstable behavior I expect some kind of race condition
>
> while
>
> > parsing from the files.
>
> This seems very odd.  Although it sounds buggy, it seems to me that others
> would have noticed it if it was common.  Therefore, it suggests that
> there's
> something perculiar to your local system which is provoking this behaviour.
>
> Let us know what you find out, and maybe we can crank through this!
>
> Cheers,
> mark
>
> > The command "xm create lnudb1 pci=04:01.0 memory=2048" works always
> > successful. A "xm create lnudb1" creates only *sometimes* a system with a
> > pci adapter and 2 GB memory. Looking in the /var/log/xend.log reveals a
> > different XendDomainInfo.create statement for good and bad case.
> >
> > (I added the >>> <<< signs to point to the important statement)
> >
> > XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 2048],
> > ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
> > ['image', ['linux', ['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk',
> > '/etc/xen/initrd-xen'], ['root', '/dev/xvda1'], ['args', 'TERM=xterm
> > selinux=off xencons=tty']]], ['device', ['vbd', ['uname', 'phy:sde5'],
> > ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> > 'phy:/dev/hda'], ['dev', 'hdc:cdrom'], ['mode', 'r']]],
> >
> > >>> ['device', ['pci', ['dev', ['domain', '0x0'], ['bus', '0x04'],
>
> ['slot',
>
> > >>> '0x01'], ['func', '0x0']]]], <<<
> >
> > ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]],
> > ['device', ['vif', ['bridge', 'xenbr1'], ['mac', '00:16:3e:00:00:31']]],
> > ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type',
> > 'vnc'], ['display', 'localhost:10.0'], ['xauthority',
> > '/root/.Xauthority']]]])
> >
> >
> > The xm create with the profile described above produces following
>
> statement
>
> > XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 1024],
> > ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'],
> > ['image', ['linux', ['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk',
> > '/etc/xen/initrd-xen'], ['root', '/dev/xvda1'], ['args', 'TERM=xterm
> > selinux=off xencons=tty']]], ['device', ['vbd', ['uname', 'phy:sde5'],
> > ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname',
> > 'phy:/dev/hda'], ['dev', 'hdc:cdrom'], ['mode', 'r']]],
> >
> > >>>  <<<<
> >
> > ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]],
> > ['device', ['vif', ['bridge', 'xenbr1'], ['mac', '00:16:3e:00:00:31']]],
> > ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type',
> > 'vnc'], ['display', 'localhost:10.0'], ['xauthority',
> > '/root/.Xauthority']]]])
> >
> > Also note the different values for the memory statement
> >
> > System Description
> > Hardware IBM x3950, 4 Intel Xeon CPUs (dual core) 3.5GHz, 23Gb memory.
> > The operating system is Linux SLES10 SP1, with the following rpms:
> > o xen-kmp-smp-3.1.0_2.6.16.46_0.12-0.1
> > o xen-libs-32bit-3.0.4_13138-0.40
> > o xen-doc-pdf-3.1.0-0.1
> > o kernel-xen-2.6.16.46-0.12
> > o xen-3.1.0-0.1
> > o xen-tools-3.1.0-0.1
> > o xen-doc-html-3.1.0-0.1
> > o xen-libs-3.1.0-0.1
> > o xen-tools-ioemu-3.1.0-0.1
> >
> > the kernel is booted with the parameters
> > pciback.hide=(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1)
> > and /etc/modprobe.conf constains the statements
> > install lpfc /sbin/modprobe pciback ; /sbin/modprobe --first-time
> > --ignore-install lpfc options  pciback
> > hide=(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1) to hide the
>
> pci
>
> > adapters for the Dom0.
> >
> > The xen profile used is:
> > builder='linux'
> > memory = 2048
> > name = "lnudb1"
> > vcpus = 1
> > vif = [ 'mac=00:16:3e:00:00:10, bridge=xenbr0', 'mac=00:16:3e:00:00:31,
> > bridge=xenbr1' ] disk = [ 'phy:sde5,xvda1,w', 'phy:/dev/hda,hdc:cdrom,r'
>
> ]
>
> > root = "/dev/xvda1"
> > extra = "TERM=xterm selinux=off xencons=tty"
> > vfb=[ "type=vnc,vncunused=1" ]
> > pci= [ '0000:04:01.0' ]
> >
> > The critical parameters are memory and pci, we never saw the issue with
> > e.g. vif's.
> >
> > P.S. It was hard to decide between xen-devel and xen-users by topic for
> > that mail. Eventually I coose xen-devel becasue I'm already member of
>
> this
>
> > list and thereby get also responses that may lack a cc to me. Please
> > redirect if needed.
>
> --
> Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
> Mark: To answer a question with a question: What use is a skateboard?
> Dave: Skateboards have wheels.
> Mark: My wheel has a wheel!



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!