From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christian Ehrhardt Subject: Configuration parameters get "sometimes" omitted Date: Mon, 15 Oct 2007 14:08:36 +0200 Message-ID: <47135844.70701@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel , jdoelle@de.ibm.com List-Id: xen-devel@lists.xenproject.org Hi, we have the problem that our xen configuration settings are not correctly= evaluated when creating a xen guests. When creating domains with "xm create domainname" the system sometimes ha= s all configuration parameters out of the config files, but sometimes not= . We did some tests and saw that the parameters are only certainly used if = we specify them via the commandline like "xm create domainname parameter=3D= value,...". If the parameters are only specified in the configuration files in /etc/x= en/ they are parsed and used "sometimes". Another fact we found is that if the "xm create" is called by a script th= e error occurs very often ~70% of the testcases, while calling the comman= d from a ssh shell manually only fails in ~10% of the testcases. We strip= ped down the script and it does nothing but calling "xm create" and the n= ormal shell where it works mostly has no .profile/.bashrc/... that change= the environment. Additionally there is some persistent behavior - if it failed once it wil= l fail until the host is rebooted (maybe some config parts are stored and= not re-evaluated). I checked xenstore with the script from http://wiki.x= ensource.com/xenwiki/XenStore and saw that xenstore only has the paramete= rs if a good-case guest is running. An interesting fact is that it does not fail initializing memory or pci p= assthrough in host or guest - somehow it fails the initial config parsing= . You can see below that already the output of XendDomainInfo.create in t= he log is different in good/bad case. Because of the unstable behavior I expect some kind of race condition whi= le parsing from the files. ### See below for more verbose details on what we found in the log's ,sys= tem setup and configuration files ### The command "xm create lnudb1 pci=3D04:01.0 memory=3D2048" works always s= uccessful. A "xm create lnudb1" creates only *sometimes* a system with a pci adapter= and 2 GB memory. Looking in the /var/log/xend.log reveals a different XendDomainInfo.creat= e statement=20 for good and bad case. (I added the >>> <<< signs to point to the important statement) XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 2048], ['vcpu= s', 1],=20 ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', ['linu= x',=20 ['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk', '/etc/xen/initrd-xen'], [= 'root', '/dev/xvda1'],=20 ['args', 'TERM=3Dxterm selinux=3Doff xencons=3Dtty']]], ['device', ['vbd'= , ['uname', 'phy:sde5'],=20 ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev= /hda'], ['dev', 'hdc:cdrom'],=20 ['mode', 'r']]],=20 >>> ['device', ['pci', ['dev', ['domain', '0x0'], ['bus', '0x04'], ['slot= ', '0x01'], ['func', '0x0']]]], <<< ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]], = ['device', ['vif', ['bridge', 'xenbr1'], ['mac', '00:16:3e:00:00:31']]], ['device', ['vkbd']], ['device', ['vfb', = ['vncunused', '1'], ['type', 'vnc'], ['display', 'localhost:10.0'], ['xauthority', '/root/.Xauthority']]]]) The xm create with the profile described above produces following stateme= nt=20 XendDomainInfo.create(['vm', ['name', 'lnudb1'], ['memory', 1024], ['vcpu= s', 1],=20 ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', ['linu= x',=20 ['kernel', '/etc/xen/vmlinuz-xen'], ['ramdisk', '/etc/xen/initrd-xen'], [= 'root', '/dev/xvda1'],=20 ['args', 'TERM=3Dxterm selinux=3Doff xencons=3Dtty']]], ['device', ['vbd'= , ['uname', 'phy:sde5'],=20 ['dev', 'xvda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:/dev= /hda'], ['dev', 'hdc:cdrom'],=20 ['mode', 'r']]],=20 >>> <<<< ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:00:00:10']]], = ['device', ['vif', ['bridge', 'xenbr1'], ['mac', '00:16:3e:00:00:31']]], ['device', ['vkbd']], ['device', ['vfb', = ['vncunused', '1'], ['type', 'vnc'], ['display', 'localhost:10.0'], ['xauthority', '/root/.Xauthority']]]]) Also note the different values for the memory statement System Description Hardware IBM x3950, 4 Intel Xeon CPUs (dual core) 3.5GHz, 23Gb memory. The operating system is Linux SLES10 SP1, with the following rpms: o xen-kmp-smp-3.1.0_2.6.16.46_0.12-0.1 o xen-libs-32bit-3.0.4_13138-0.40 o xen-doc-pdf-3.1.0-0.1 o kernel-xen-2.6.16.46-0.12 o xen-3.1.0-0.1 o xen-tools-3.1.0-0.1 o xen-doc-html-3.1.0-0.1 o xen-libs-3.1.0-0.1 o xen-tools-ioemu-3.1.0-0.1 the kernel is booted with the parameters=20 pciback.hide=3D(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:01.1)=20 and /etc/modprobe.conf constains the statements=20 install lpfc /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignor= e-install lpfc options pciback hide=3D(04:01.0)(04:01.1)(06:01.0)(06:01.1)(08:01.0)(08:= 01.1) to hide the pci adapters for the Dom0. The xen profile used is: builder=3D'linux' memory =3D 2048 name =3D "lnudb1" vcpus =3D 1 vif =3D [ 'mac=3D00:16:3e:00:00:10, bridge=3Dxenbr0', 'mac=3D00:16:3e:00:= 00:31, bridge=3Dxenbr1' ] disk =3D [ 'phy:sde5,xvda1,w', 'phy:/dev/hda,hdc:cdrom,r' ] root =3D "/dev/xvda1" extra =3D "TERM=3Dxterm selinux=3Doff xencons=3Dtty" vfb=3D[ "type=3Dvnc,vncunused=3D1" ] pci=3D [ '0000:04:01.0' ] The critical parameters are memory and pci, we never saw the issue with e= .g. vif's. P.S. It was hard to decide between xen-devel and xen-users by topic for t= hat mail. Eventually I coose xen-devel becasue I'm already member of this list and = thereby get also responses that may lack a cc to me. Please redirect if n= eeded. --=20 Gr=FCsse / regards,=20 Christian Ehrhardt IBM Linux Technology Center, Open Virtualization +49 7031/16-3385 Ehrhardt@linux.vnet.ibm.com Ehrhardt@de.ibm.com IBM Deutschland Entwicklung GmbH Vorsitzender des Aufsichtsrats: Johann Weihen=20 Gesch=E4ftsf=FChrung: Herbert Kircher=20 Sitz der Gesellschaft: B=F6blingen Registergericht: Amtsgericht Stuttgart, HRB 243294